Ask HN: Are there any NLP projects that use context?
https://news.ycombinator.com/item?id=9401973I was thinking about natural language processing today. I am by no means educated on the subject, but it seems like an interesting field. Most NLP libraries can parse text into its parts-of-speech, but often times it can be wrong because it's a word like "project" where the verb form is a completely different word than the noun form, etc.
In all the libaries I looked at, "project" was interpreted as a noun despite the sentence being "We will project our earnings".
Could we not provide context to the parsing process? For instance: a database could store the frequency of parts of speech for a given word's N nearest neighbours (ie. the n-1 neighbour of "project" is a definite article 90% of the time), as well as the frequency for each part-of-speech for the given word. So if the n-1 neighbouring word of "project" was not a definite article, it may indicate with a certain weight that "project" is not being used in its typical form either. Then you could do the reverse lookup (the n+1 neighbour of "will") and see that that neighbour is a verb 50% of the time. Putting those two pieces of information together gives you pretty good confidence that "project" probably isn't a noun like it usually is.
So is this just too computationally expensive or is there something else that I'm missing?