I think the the appropriation can still be said to be a problem in that the people appropriating music might not even be racist, there is nothing inherently wrong with playing or selling another cultures music.
But by doing so they are "aiding and abetting" a racist society by enabling racists to listen to music they desire, from people they desire. If they choose not to it makes it harder for racists to be racist.