Learning how to use documentation for/with AI has been the story of my last year+. I don't have the technical background you do but I've worked in product and project management, so I was never deep in code the way you've been.
I think the journey of figuring out how to actually create with AI has been as interesting as what I'm able to create. I wish I had documented more of my lessons-learned the way you have. The biggest challenge I've found is that, as you mentioned, AI often thinks more equals better. So I had to learn when "comprehensive" meant subtracting over adding. Of course, there was also the practical reason, that more documentation meant more tokens used up - more context consumed - for a given task.
As far as files and rules, since I like seeing things visually structured, I've had luck creating temporary html files that visually capture the structure (files and rules) of a given effort. I use that as my visual map and it's what I have Claude use for its own understanding. It reduces the need for reading entire docs but it also forces me and the AI to stay structured. I also mirror this in that (as of late) I use Cowork for the planning (i.e. the maintenance of that visual mapping) and Claude Code (in a completely separate instance) to run the task(s) and, in the end, pass the lessons-learned over to Cowork manually. Otherwise if they were one "mind" e.g. when I would use Claude Code to coordinate AND code, it would always add more information to its memory and it would get out of hand very quickly. Ultimately, the key for me was understanding how AI uses structure. For example, when I started using Linear a couple of months ago - prior to which I had no "formal" project management system, all of the tags and labels and 'projects' (as Linear uses the term) really helped manage efficiency in productivity. I use Linear's MCP to connect to Cowork directly and always start sessions with reading 'project' titles, labels, priorities, etc. before diving into details for a given effort.