Is this a by-product of some compliance or self-regulatory requirement like ESRB?
Yes, Steam addressed this an announcement a few years ago:
Q: Why do you KEEP asking my damn age throughout the store?
A: We're with you on this. Unfortunately, many rating agencies have rules that stipulate that we cannot save your age for longer than a single browsing session. It's frustrating, but know we're filling out those age gates too.
https://steamcommunity.com/games/593110/announcements/detail...