I don't think the driving thing is a big deal considering he's 50..
It's everything else that sets up red flags and doesn't sound right. This guy sounds like a weirdo. You would *have* to move there and live in a city you're not comfortable in.. I've never been to CA but I know there's areas around here I'd never want to live in. Neither one of you should have to "Change" but should be able to come to a compromise..Like you said, move somewhere he can easily get to work and you're not in an area you feel uncomfortable in. If he's in any way making you feel guilty for this then that's another reason to dump this guy.
I'd be afraid of him telling you not to drive and obsess over every little financial thing that would end up breaking you up in the end anyway. 2 different backgrounds, 2 different beliefs on how one should "live in a situation" (whatever that comment of his meant) .. You'd be fighting over it forever. Imagine xmas shopping or furniture shopping and you picking out nice things you want and can afford while he's behind you saying "too expensive lets save money". LOL.
And maybe it's just me, but I'd be weary of a guy who says "you don't have to work", like one of those weirdos that believe the woman should be home raising the kids and in the kitchen while the guy works. And to take your car away so you're really stuck at home. Seriously anyone else get the weirdo impression?