Hmm the issue is that there might be some correlation between the syntactic and lexical similarity and the actual subject matter the authors are talking about.
I've been working on a similar project.
Three things that I would suggest would be
- add documents definitively written by the authors (Maddison, John jay, etc) from outside the federalist papers to your train set.
http://oll.libertyfund.org/titles/jay-the-correspondence-and...
http://www.gutenberg.org/ebooks/author/14, etc
- Add another feature which looks at the frequency of the function (closed class words) such as articles, prepositions etc these are very stylistic and hard for an author to control, they are also independent of the content, this is a classical feature in forensics.
- Add a distractor case to your train and validation set I.e documents written by a non federalist such as Thomas Jefferson and confirm that they don't get clustered into one of the other federalist authors.
If you have questions feel free to tweet me it seems like a cool project @pythiccoder