That doesn't work in real life. In order to know what level of force is necessary, one has to have a perfect knowledge of the opposition. That isn't possible, so you always respond with more force than you think in necessary - overengineering the carnage.
That's exactly the kind of thinking that leads to global nuclear annihilation, and when that consequence became clear, intelligent people finally realized how wrong it is.
But your argument is wrong anyhow. Either you're trying to end the war, or you're not. If you are, you try all avenues, including negotiation. If you don't try negotiation, you aren't trying to end the war, you're trying to prolong it. The US leadership knew they couldn't lose at that point, so they weren't trying to end it.
The what if game is pretty boring.
It sure is when you lose right away, due to the US never trying to negotiate. Earlier you said that it was "risky" to do so. How?
And It doesn't matter that we don't know what would have happened if the US had attempted negotiations with Japan. It only matters that they never did.