It's not that bad, I'm on my second Elixir job in 3 years and in Asia which is even harder to find.
There are a good amount of jobs in the US or in Europe but you have to actively look for them...
I can however say that working with Elixir has been really pleasant and even enjoyable, and I'm not someone who codes outside of work. I'd definitely recommend it!