there's also no reason to condemn him for using it to live the best life he can.
Actually, there is. Simply accepting the benefits of this systemic injustice and not actively working to remove it, especially when in a position of power, is not acceptable. If you just use white privilege to live the best life you can, that is participating in the system and is not correct. You have a duty to not only disavow 'white preference' as you call it, but also remedy the inequity of your privilege, not simply use it.