Every back is a risk. The longer the back, the bigger the risk. He is choosing your driveway because its safer, but he shouldn't have a problem if you tell him to stop doing it unless he has a stop for you. I personally don't think what you're asking is unreasonable. As you said, you didn't mind if it was weather, or overweights or a stop for you.
Just be courteous, and I dont think you'll have a problem. If you have an issue confronting people, you could always get a sign that says no turn around in driveway. He should get the hint. Its really a minor inconvenience, if it takes him longer to back slowly down the street so be it. More overtime imo