Similar but different problem of "Sniffing Everything"

Didn't want to hijack another thread, so I thought I'd make a separate one... thanks in advance for all suggestions!
 
I walk Oliver (10 month old lab mix, about 54 pounds, graduated basic and intermediate obedience) for 40 minutes in the morning and 45 minutes in the evening.  I do not have a yard for him to use for a potty, so the walks are two-fold: potty and exercise.  He heels pretty well, although we are continuing to progress on a solid heel in any situation.  I've taught him that an arm sweep to my right and a verbal "OK" means he can go sniff and potty, etc.  When he is done, he recovers to a heel position or I recall.
 
My problem right now is that when he is on his "potty" time and sniffing around, he'll find a smell and throw his body weight against me and the leash to get to it.  He has a full 6' leash and I don't mind roaming around a bit, but I HATE being yanked off balance.  When this happens, I've been doing a recall at this point and moving on. 
 
How to I teach him is is not OK to pull like that on his "recreational sniffing" time?  This is the only time he pulls, so I'm looking more for techniques/commands than equipment recommendations.
 
Thanks!