I've read portions of volumes 1, 3 and 4A so I haven't finished the books. The thing about these books is that while you can _Read_ them like any other book, reference or otherwise, what sets them apart are the questions at the end that have always been very thought provoking.
4A to me was the most useful - combinatorial algorithms, talking about generating trees, permutations et al. Reading it really cleared out some concepts that relate to sorting and searching. It's almost like someone condensed all known information about those algorithms (till the last 2-3 years) into one dense manual. (oh wait..)
There's no better time to read them than right now IMHO. And I don't think the right way to read them is volume wise. What I've done is to go through the table of contents and pick something that looks interesting, and chew on it for a bit. But to each their own.