I switched to vim after years of IDE use, and I find it painful to go back to the IDE when I'm working on projects that require it.
The biggest advantage I find in using something like vim or emacs is that you never have to switch between the mouse and the keyboard — in fact its awkward to use the mouse and that makes it difficult to get accustomed to. It allows you to seamlessly transition between code navigation and code writing, and you develop a kind of rythme that I never felt like I achieved with a graphical text editor.
Other nice features are the ability to write custom scripts and repeat actions, but these can generally also be done in other editors. One huge reason I made the switch is I end up writing code on remote machines on occasion, and I can almost always do so in my everyday editor: vim. All my personalization can be transferred over to the remote machine in a single scp command and the experience remains the same no matter where the files live that I'm working on.