Stop Words as Social Signals (a result that goes against IR common knowledge)

http://www.cpdiehl.org/2010/04/social-signaling-and-language-use.html

by hamilton • 16 years ago

85 11 16 years ago

conanite2• 16 years ago

Amazing that [to, to, on, my, to, on, be, on, to, to ... etc] can reveal so much. One of the linked articles ( http://homepage.psy.utexas.edu/homepage/faculty/pennebaker/r... ) describes how the relative frequency of the use of "I" in communication is a powerful indicator of status (less "I" => higher status).

patio112• 16 years ago

Stop word distributions are also sufficient for identifying gender in most English corpora with about 80% accuracy -- and you don't need much text, either.

[Edit to add linky: http://u.cs.biu.ac.il/~koppel/papers/male-female-text-final.... ]

conanite2• 16 years ago

I wonder would James Chartrand have been found out ... http://news.ycombinator.com/item?id=994284 [Why James Chartrand Wears Women’s Underpants]

klochner• 16 years ago

One of the two linked papers (AnnualReview.pdf) discusses formal settings, including those with disparities in power:

'Brown & Levinson’s (1987) polite-ness theory takes into account an individual’s efforts to preserve the “face(s)” of others with whom one communicates. For example, they propose impersonalizing the speaker and hearer by avoiding the pronouns I and you, using past tense to create distance and time, diminishing the force of speech by using hedge words such as perhaps, using slang to convey ingroup membership, and using inclusive forms (we and let’s) to include speaker and hearer.'

rozim• 16 years ago

See also SpotSigs ("...stopwords may however be very good indicators of the actual interesting parts of a web page...")

http://infoblog.stanford.edu/2008/08/spotsigs-are-stopwords-...

lallysingh• 16 years ago

Outside the old post-2001 TIA/NSA efforts, what can this be used for?

Maybe for determining social hierarchies for mind share in product recommendations or advertising?

Yeah, I can't think of one noncreepy way to use this.

Vivtek• 16 years ago

Good Lord that is counterintuitive!

tlb• 16 years ago

He seems to be claiming that in the phrase I don't feel comfortable signing up for a $7,000 extra flight without talking to you guys , it's the word to that signals the manager/subordinate relationship. It seems to me that ... without talking ... you is the best clue.

In English, subjunctive case usually requires a to, and people asking permission use the subjunctive a lot. Is that the signal he's finding? Is that the best NLP can do?

sesqu• 16 years ago

I was always uncomfortable with stopwords. Ignoring some data smells like overfitting, or perhaps a naïve algorithm (as in, it has no concept of structure – which is often the case, for computational reasons).

That said, the last time I dabbled in IR, my algorithm choked on stopwords. I should revisit it.

akshaybhat• 16 years ago

I think the following phrase is unnncessary "a result that goes against IR common knowledge"

IR aims to retrieve information and not sentiments from the text. This kind of work is related to sentiment analysis, where use of stops words as signal is common. Thus the whole argument that some holy criteria in IR has been shown to false is clearly incorrect. Even original authors don't make such assertions.

Stop Words as Social Signals (a result that goes against IR common knowledge)

Related Stories