Irrational fears of driving

I am 18 1/2 years old and I have yet to learn how to drive. I've gotten my learners permit, about 20 different people have taken me at least once, and as soon as I start to warm up to the idea of going faster than 30 mph and I don't have a panic attack when some one else is driving on the same road as me I don't go driving again for another 4..5..6 months and then have to start all over.
My mom never taught me how to because I wouldn't let her. (seriously, I love my mom to death but she's the kind of mother who when I tried to learn to ride a bike, she pushed me down a hill and yelled "JUST GO!"...not to mention she makes this brutal angry face that I would never wish on anyone else.)
My husband isn't allowed to teach me because 1. He's too busy and 2. he's freaking crazy and tries to teach me at night when I can't see anything.
So, here I am a stay at home mother with no way of going anywhere during the day because I can not drive.
I find myself day dreaming of the things Rowyn and I would do if I could drive. Oh, the interesting and exciting life we would have that doesn't consist of sitting in an apartment all day or going to the complex's park with only one swing, a slide, and some weird digging things that make large holes that my poor baby is constantly falling in.
We would start some kind of business, either as some sort of consultant or one where I made an Etsy shop to sell furniture I've re-finished(you have to leave the house to find cool crap at Goodwill you know.) We would go thrifting and antiquing, to mommy and me music classes, we'd venture to Harwin in Houston to do some shopping, go to Traders Village, have numerous play dates, go to the Zoo, visit my friends in my home town, and maybe get some grocery shopping done without my husband calling me every 5 minutes asking me if I'm done yet.
It's very frustrating thinking of all these fun and interesting things we could do if I had a car while I sit in my home, fighting off the boredom. It's even more frustrating knowing that even on the weekend when my husband IS home we are only able to go do things that are only located 10 minutes away because he is busy working on start ups and isn't able to come with us (not like he'd want to do any of the stuff on that list anyway.) and if I want to go any further I have to hear the same old long speech about how "It's 20 minutes there, 20 minutes back to the house then 20 minutes back there to get you and 20 minutes to bring us back the house house again..THAT'S OVER AN HOUR OF DRIVING!"
I really don't know what it is with him and his obsession with not spending more than 40 minutes a day driving. I don't think he likes it very much.

Have any of you had these same problems as me? Have you concurred a fear of driving? How did you learn to drive without anyone to help you and are you still afraid when you get out there on the road?
Any and all advice is welcome.

p.s. check out my nifty new music player on the left right above my archive. It's playing the songs I'm currently obsessed with. :]