I think you’re using the typical aesthetic definition of fantasy vs. sci-fi. You’re right that under that one, Tower of Babylon would be considered fantasy.
Ted Chiang has an alternate definition though, I prefer that one to be honest. His definition is about whether there are certain “special people” to whom the general laws of the universe don’t apply [0]. Under that definition, even what we would colloquially call magic (ex. turning lead to gold) would be called sci-fi, as long as everyone could do it; once you have that, you can do things like mechanize it and make factories to do it at scale, and there’s where you get the interesting second order problems.
Under that definition, I think Tower of Babylon is better considered sci-fi, because there are no “special people”. The new rules of the universe also lead interesting second order effects: the tower gets so tall that entire families live in the tower, and people are born and die in the tower [1].
[0] - better explained him here: https://boingboing.net/2010/07/22/ted-chiang-interview.html, see “You have very specific views on the difference between magic and science. Can you talk about that?”
[1] - I don’t know if Chiang intended this, but I think you could probably draw a parallel to missionaries to the new world.