Shy dogs! Do I ever know about 'em!! Well, okay - one.

2 years ago, I adopted a chihuahua from a rescue. She was a little over a year old and came from a 'backyard breeder' - kept in a crate her whole life - just for breeding. As a result, she was (literally) scared of her own shadow, unhealthy, undersocialized, not housetrained, had never seen a toy or a leash in her life and afraid of everyone and everything.
She was A LOT of work and will never be 100% confident. But she is EONS from the dog I first saw at the rescue place, shivering and cowering from everyone.
The first thing I did was read "Help for Your Shy Dog" by Deborah Wood. Awesome book. Invaluable!
First, I taught her the basics, to keep her mind engaged: sit, down, stay. When a shy dog has something else to focus on, they are less distracted by whatever makes them scared.
I would strongly recommend teaching 'Watch me'. Say it to your dog and the SECOND they make eye contact with you, reward. I use it all the time on walks, now - if only to practice. I used it whenever I knew something would/was frightening her...to distract her attention from the 'trigger'. This was especially great when she was new to a leash - sometimes she'd get frightened - and we'd find ourselves with her leash wrapped around a pole and her, struggling to get out and away. Then I'd say, "Watch Me" - and she'd look at me and forget (temporarily) about her struggle. It was just enough time for me to calmy untangle her and move on.
With shy dogs, it is always baby steps. It took me a whole summer to work her up to walking around the block. I live in a major city with lots of traffic/people and city buses. It was a lot to deal with for a tiny, shy one - but we worked at HER pace. If we walked too far out of her comfort zone, that would be it for the day. If a bus stopped beside us, I'd move a little away from it and give her treats, so she got used to the noise. Then, the next day - we'd get a little further and further. Believe it or not - NOW, she LOVES her walks! She walks past people/kids/dogs and is generally alright. Now and then, we have setbacks here and there - but if I look ahead and see that something will scare her (skateboad, or construction for instance), I move to the other side of the street OR we move out onto someone's front lawn and practice our sit, down, stay stuff until the noisy thing has passed.
Keep having him meet strangers that give him yummy treats! Flea was terrified of everyone - especially if they reached for her (she still gets scared of two hands - except for me). NOW, she sees them as treat dispensers and will always say hello to a calm, friendly person. I would have your visitors only pet your dog on the chest or under the chin. Shy dogs often fear hands above their heads. She is also still more afraid of my boyfriend than me (and he has been there since the beginning)..so have everyone in the family take turns with the feeding and the happy activities - so he can expand his trust for more than one person.
If your dog becomes afraid outside, do not pet him. What I do, is start talking in a happy, upbeat voice. "Just a noise, let's go!" It seems to let her know, it's no biggie.
I TRIED clicker training - which will be great if it works for you, but my dog was TERRIFIED of the 'click' sound (and I was using the lid from a juice bottle which is a very light click. And I mean, terrified to the point where she would run and hide. didn't need that regression!). So, we stopped that quickly.
I agree, if you think your dog will get along with other dogs, do the obedience thing. I did not do this with my dog - because she was terrified of other dogs barking!! And I didn't want to throw her right into the mix without first doing some work with her on my own. When I got my 2nd dog, she seemed to come out of her shell more. And now, she will meet ANY dog on the street (she especially adores small ones) - if they approach her nicely.
Flea still has her moments - every day - when she gets startled by the most innocent things. You can place a glass down too hard on the kitchen counter and she scrambles to run out of the way. But we don't coddle her about it. We move on, like everything's cool. All I can do is hope that her continued exposure to the outside world will put her more at ease in her own environment. It's a good thing your dog has the crate to go to. Flea will do that too, sometimes if she feels overwhelmed. It is important that they have a place which makes them feel safe.
I might suggest that you shorten the length of your leash with this dog. It seems that he has a lot of leashroom to work with if he 'runs out' ahead of you. If he's closer, you can better control his movements. If he starts to twist and wrench out of the collar - remain calm - put him in a 'sit' - give him treats and wait until he becomes stable again.
Sometimes when I'm out with Flea, and her tail is up high and she's having the best time on walk, sniffing and running - and stopping to meet a stranger - I just can't believe she is the SAME little dog that I first met...cowering and wide-eyed and panicked about the world. But she has taught me that dogs like this CAN change. For the first time in her life, she gets to be a dog!
Believe me, I KNOW it's a slow slow process...but for the person with the patience and love to give a dog like this...it is very rewarding! She was my diamond in the rough and I wouldn't trade her for anything.
And one more thing..kudos to you for doing the foster thing. We need more kind-hearted souls like you in the world. Good luck!