Even in the summer, when it was scorching hot?
I have high-twist wool(!) trousers that are comfortable up to 90 F. Most people couldn't tell they're not normal wool dress trousers. I have linen trousers that aren't really any warmer than shorts (and they keep the sun off you). Ditto for shirts. I'd rather be in long sleeves and trousers, all light-woven linen, in high heat, than in shorts and a T. Doesn't really insulate more, does keep the sun off.
I have an unlined wool navy blazer in a Summer weight that breaths so well it's basically totally ineffective at warming a person up, LOL. Put it on, take it off, makes basically no difference, aside from the color being dark enough that it does absorb heat from the sun. If I wore that kind of stuff more, I'd have other options that're even better in the heat.
I have some sweaters that're completely comfortable into the high 70s F (50/50 silk/linen blend, thin) and one that won't make you feel like your clothes are making you warmer up to at least 85F (that one's 100% linen and open-weave).
I dunno where people got the idea that you have to be uncomfortable if you're wearing much more than a swimsuit, in the heat.