Sentiment Symposium Tutorial: Stemming

1. Overview
2. Porter stemmer
3. Lancaster stemmer
4. WordNet stemmer
5. Assessment
   1. Classification accuracy
   2. Speed
6. Summary of conclusions

Overview

Stemming is a method for collapsing distinct word forms. This could help reduce the vocabulary size, thereby sharpening one's results, especially for small data sets.

This section reviews three common stemming algorithms in the context of sentiment: the Porter stemmer, the Lancaster stemmer, and the WordNet stemmer.

My overall conclusion is that the Porter and Lancaster stemmers destroy too many sentiment distinctions. The WordNet stemmer does not have this problem nearly so severely, but it doesn't do enough collapsing to be worth the resources necessary to run it.

Porter stemmer

The Porter stemmer is one of the earliest and best-known stemming algorithms. It works by heuristically identifying word suffixes (endings) and stripping them off, with some regularization of the endings.

The Porter stemmer often collapses sentiment distinctions, by mapping two words with different sentiment into the same stemmed form. Table tab:porter provides examples of such collapsing relative to the disjoint Positiv/Negativ classess of the Harvard General Inquirer, a large gold-standard semantic resource containing extensive sentiment information.

Lancaster stemmer

The Lancaster stemmer is another widely used stemming algorithm. However, for sentiment analysis, it is arguably even more problematic than the Porter stemmer, since it collapses even more words of differing sentiment. Table tab:lancaster illustrates with a randomly chosen selection of such collapses, again using the Harvard General Inquirer's Positiv/Negativ distinction as a gold standard.

WordNet stemmer

WordNet (Fellbaum 1998) has high-precision stemming functionality, but it is probably of limited use for sentiment analysis. To effect real change, it requires (word, part-of-speech tag) pairs, where the part-of-speech is a, n, r (adverb), or v. When given such pairs, it collapses tense, aspect, and number marking.

The only danger I know of for sentiment analysis is that it collapses base, comparative, and superlative adjective forms. Table tab:wordnet provides some illustrations.

Assessment

Classification accuracy

To assess the impact of the stemming algorithms, I use the experimental set-up as I used when assessing tokenizers. Here, though, I compare just the plain sentiment tokenizer with the sentiment tokenizer plus the Porter and Lancaster stemmers, applied to the output of sentiment tokenization. (Since the WordNet stemmer requires part-of-speech tagged data, and since its changes are minimal, I don't assess it here.)

The results of the experiment are given in figure fig:stemmer_accuracy. It looks like the stemmers benefit somewhat from their reduced vocabulary size when the amount of training data is small, though not enough to improve on the sentiment-aware tokenizer, and they are quickly out-paced as the training data grows.

Figure fig:stemmer_accuracy

Assessing stemming algorithms via classification. (Details on the experimental design.)

Speed

Table tab:tokenizer_speed extends the tokenizer speed assessment given earlier with comparable numbers for the stemming algorithms.

Summary of conclusions

1. My central conclusion is that, for sentiment analysis, running a stemmer is costly in terms of resources and performance accuracy.
2. I can imagine running the WordNet stemmer if matching against a restricted vocabulary became important, but in that case it would be better to run the algorithm in reverse to expand the word list.
3. I don't mean to suggest that stemming could never help with sentiment analysis, but rather only that these off-the-shelf stemming algorithms can weaken sentiment systems.