You don't have to use all Hibernate features. You can write your own SQLs too if you want to. Hibernate can be super fast if you use the correct strategies [1] at your bottlenecks. Any other performance issues are typically ORM related.
[1] http://docs.jboss.org/hibernate/core/3.3/reference/en/html/p...