The dangers are real. I've known multiple recreational users come close to death from ketamine addiction, either from the effects or from general self-neglect.
The problems are threefold: firstly, it does not impair you to the extent that you cannot "live," you can still limp along through a normal life high out of your mind on ketamine every day. This means it is not self-limiting in any way, and people are not forced to change until they are seriously ill.
Secondly, the tolerance curve is very steep. If you use it just a few times in a week, the very next week you will need to consume noticeably more. It continues in a sort of hockey-stick arrangement. This is how people suddenly end up on 5g a day, destroying their urinary tract with this caustic, highly alkaline stuff.
Thirdly, some people seem to just really like it. Ketamine has a subtle psychological effect that really seems to resonate with some people. I guess this is why its connected to long term depression treatment. Unfortunately when combined with the other two factors I have listed, it becomes a trap into which people can fall for a long time. Especially the young.
It's basically halfway between marijuana and heroin. It's as destructive as that sounds.
For the record, I'm in favor of the freedom to use drugs, but ketamine is nasty stuff.