If your driver points with his head "that house", just look at the box and address no matter what he/she says. It doesn't hurt to be sure.
Also, if you are unsure of what the driver means, clarify. It might irritate him, but it will really irritate him if you do the wrong thing.
If you stop and have a free minute, write some things down that will help you. When I get home I can stop and think about what I did wrong and how to fix it the next day.
I ask a ton of questions and my driver seems cool with it. If I understand why I'm doing something, I do a better job at it. Now I'm even sorting the back when I can.
Good luck