You’re right; you can make this claim about anything. We have to be careful about which things we spend resources worrying about, and if we’re overzealous with restrictions we pay a lot of opportunity cost.
The thing is, AI is not a normal technology; it is already vastly more powerful than any technology we could compare it to, and more resources are being poured into its development right now than the development of any other technology; we are in uncharted waters. If there is any new technology to be careful with, it is this one. In this case, it is worth paying the opportunity cost.