The best understanding of matter is in the form of quantum field theory (QFT). QFT is always formulated with some background metric (geometry) of spacetime as an input.
One idea is that the metric (geometry, gravity) could be a field just as matter is a field and people tried to apply the standard rules of (perturbative) QFT to gravity but failed. This is because the theory of gravity is unrenormalizable [0]. An interesting avenue in saving this line of thought is asymptotic safety where the idea is that gravity coupled to the standard model could actually be renormalizable in a certain sense [1].
In any case general relativity and quantum theory have so far been irreconcilable and there is now consensus on how to bridge the gap between those two theories. It is exactly because of this that most physicists will think of gravity and the other forces to be of a different nature.
When people say that gravity is not a force they mean that there is no known particle which acts as the intermediary of said force. For all the other forces we have a theory that explains the exertion of force via a particle.
Your first point states that "general relativity is just a model". Many physicists believe that it is more than a model but a true description of what the world really is like. I understand your urge to label theories as models, but ultimately the question is whether or not there is some level of ground truth that can be accessed in the form of mathematical theories.
[0] https://en.wikipedia.org/wiki/Renormalization#Renormalizabil... [1] https://en.wikipedia.org/wiki/Asymptotic_safety_in_quantum_g...