"I've never had the opportunity to work with physical labor"
Yes you have. You could go out today and find a job doing physical labor unless you are significantly physically disabled. It's not like there is an educational barrier to all of those jobs. (Yes, I know there is to SOME of them because they require skills and performing physical labor)
As someone that has done plenty of both, you aren't contributing anything to this discussion unless you try it.