But the second I need to use an IDE for Windows development, it's too slow to bear.
This very much depends on your machine. For example, I have an office machine running an older i7 with 16 GB of RAM. I give my VM's 6 cores and 8 GB of ram and I don't run into the issues you describe.
I also precreate the disk (instead of using dynamic) and shut off things that tend to be I/O intensive in Visual Studio, but once done I've not noticed a performance difference between it and native.