Mandy, you could probably start very small and work your way up. If you just see one private client every once in a while, you wouldn't have to do the whole shebang.
I carry liability insurance on my own, even though I'm covered by my hospital. It just seems like a good idea. I've always done it. I've provided first aid at private events, I stop at car accidents, I help kids on playgrounds, strangers show me their rashes or tell me about their bowel movements. It's around $100/year. I also have business cards, just randomly, that do not have any professional designation on them, just contact information. I got them for cheap along with a Christmas card order from VistaPrint one year. I carry a few around in my wallet and they are helpful when I hit it off with another mom at a playground (they are actually marketing "Mom" cards now) and we promise to stay in touch and have a playdate sometime, or like when we were at the rocket launch and someone promised to send me the pictures he was taking of the scene, or when my mailman asked if he should call before he came over when I helped him with getting his health insurance set up. I got a few boxes (hundreds!) for around $5, I think. Those two things are very easy to get.
The only things I think would be more complicated at this point would be a business license (I would check with your friends who do it--what do you actually need for private sessions?), some way to bill insurance, and a way to get clients. You already had one that was interested. You have contacts. If you don't need to sustain an entire job or income on this, I wouldn't worry about doing full-blown marketing with ads or custom pens or anything. I would just let word of mouth start letting people trickle in. Let your local contacts in the field know that you are accepting clients for private sessions.
I would be wary of accepting clients that came to you via Craigslist, too, just for security's sake. You never know if it's a real client, you know?
Your DH is most likely worried that this is just some wild idea that has come to you and is giving you a lot of things to think about, but if you are just seeing an occasional private client and not doing this full-time, I don't think it has to be as intimidating as he is making it sound, and it doesn't have to be the make-it or break-it scenario. If you only see one client a week for a while, you will still be establishing a routine, and bringing in a little bit of extra cash to help alleviate some of the pretty minimal outlay, and you will hopefully get more later on. Remember, all of these expenses will be tax deductions, too.