You already know loading is not a piece of cake. You've done it for long enough, and if you have no issues with your joints, back, tendons, or muscles from the repetitive stress, perhaps you can hang in there long enough to make it work for you. I've done it for quite a while. 4 or 5 years is a very long time with this kind of repetitive stress workout.
I'm not a spring chicken either, but I was in terrific shape when I started loading. Yet, it has worn on my body over the years. It's kept me in shape, but the wear and tear is not kind. However, unlike you, I have a full time job and only do it to keep fit. For me, I want some seniority so I am not constantly loading, but can do Pick-Off once in a while. Until then, it's a forced workout for me, with benefits, that I get paid for. Not much, but sure better than paying a Health Club for 1/2 the exercise.
I hope you can do it, as it sounds like you want to. Hang in there and hopefully, you will get that driving job you want. 40 is not that old any more. We have guys in their early 50's loading. And they are not that bad. A little slower, but they cut the mustard.
Good luck!