The only part the compiler actually enforces is that you mark the specific spots where you make use of the operations that 'unsafe' allows.
Yes, I wish there was a way to mark code as unsafe without also allowing unsafe operations. Its quite common I have some "safe" code that generates values which are eventually used in an unsafe block. If the "safe" code is wrong, my unsafe code will fail in memory-unsafe ways. But there's currently no way to annotate this sort of "safe" code. Rust provides the unsafe keyword - but that keyword is generally reserved for code which needs to actually make unsafe operations (like derefing pointers or calling other unsafe functions).