Do you think I'm being selfish?

You're not being selfish. If he was to get in an accident or be pulled over while driving the car with no licence, you could lose your insurance AND be fined for knowingly allowing an unlicensed driver to drive your car. He needs to grow up, and if he wants to drive, he needs to go and get a driver's licence.