importance

The last decade has seen an unprecedented surge in the growth and proliferation of online social media platforms such

as Twitter, Facebook, and Tumblr. For individuals, social media is a uniquely powerful platform for communication,

self-expression, microblogging, social networking, image and video sharing, and opinion sharing through online word-

of-mouth. As of January 2020, the world has more than 3.8 billion active social media users [Kepios Pte. Ltd. et al.,

2020] generating prodigious volumes of public opinion data. For businesses and organizations, social media provides

access to consumer bases for targeted advertising and public opinion mining on products and services to inform

posts, called ’tweets’. Tweets have been limited to 140 Unicode characters until 2017 when the limit has doubled to

A PREPRINT - JULY 28, 2020

280 characters [Boot et al., 2019]. Due to the character limitation, Twitter users prioritize efficiency when composing

tweets by employing informal language, abbreviations, unconventional word spellings, symbols and numerals, as well

as omitting parts-of-speech (POS) to convey the desired message [Boot et al., 2019]. Twitter’s condensed format and

associated usage characteristics make it a challenging domain for SA unlike mining standard literature and documents.

This paper investigates the efficacy of common text processing and classification steps on Twitter-specific SA. We

hypothesize that efficient use of word embedding with data-driven optimization can be robust to the selection of

standard text processing routines in Twitter sentiment classification.

et al., 2015]. Some popular lexicons include WordNet [Princeton University, 2010], SentiWordNet [Esuli, 2013], and

opinion lexicon [Liu, 2017]. Since lexicon-based methods depend on the words used in the social media post being

present in the sentiment dictionary, these methods suffer from low recall [MART

´

INEZ-C

´

AMARA et al., 2014]. Two

words of different polarity (good and bad) can appear in the same tweet that complicates SA in a particular context.

Furthermore, the noisy nature of tweets and ambiguous usage of words across contexts makes the automatic generation

of lexicons from Twitter data unreliable [Kolchyna et al., 2015]. On the other hand, the ML approach uses supervised

optimization to learn a function that performs the classification task based upon labeled training examples. However,

the primary limitation of ML methods is the availability of labeled social media posts for training SA models.

SA of social media posts, especially from character-limited tweets, remains a nontrivial task. For example, senti-

linguistics.

Early ML approaches to Twitter SA rely on raw word representations called n-grams as the primary input fea-

tures [MART

´

INEZ-C

´

and pairs of words are referred to as unigrams and bigrams, respectively. A tweet is represented in a feature vector

of length equal to the number of n-grams such that each n-gram has its unique position in the vector. The vector

positions contain frequency of corresponding n-grams or simply hold ’1’ or ’0’ depending on the presence or absence

of the n-gram in the tweet, respectively. This construction is heavily reliant on substantial text processing and careful

feature selection steps to optimize the classifier model performance. For example, Jianqiang et al. have extensively

studied the effect of six well known text processing steps on Twitter sentiment classification using n-gram representa-

report no effect of removing numbers, URL, and stop words on sentiment classification [Jianqiang, 2015].

Recently, ML-derived contextual vector representations of text (text embedding) have created a new paradigm for ML-

based SA research [Tang et al., 2016]. Several highly efficient text-to-numeric word embedding algorithms have been

developed by field experts and industrial giants, including skip gram [Balikas and Amini, 2016a], GloVe [Pennington

et al., 2014a], and fastText of facebook [Balikas and Amini, 2016b]. For example, the popular word2vec routine,

developed by Mikolov et al. [Mikolov et al., 2013, CHURCH, 2017], is a highly optimized Python package that

implements the skip gram model for obtaining a contextually meaningful word embedding that enables subsequent

text is important, it requires careful selection of text processing steps.

embedding-based models for Twitter SA. We hypothesize that the importance of these text processing steps depends on

A PREPRINT - JULY 28, 2020

Table 1: Effects of text processing steps on sentiment classification performance reported in [Jianqiang and Xiaolin,

2017] using Bag-of-Words (BoW) n-gram features.

the text domain (Tweets) and the contextual (sentiment analysis) representation of words in the feature vector obtained

via word embedding. A robust word embedding is expected to be less sensitive to the selection of text processing steps

and classifier models.

A common text processing step is stop word removal using off-the-shelf stop word dictionaries. For example, the sev-

eral decades old Van stop list [van, 1979] is used in recent Twitter SA literature [Jianqiang et al., 2018]. In contrast, a

Furthermore, there are many non-dictionary words, symbols, and short and colloquial language expressions on so-

cial media that can complicate SA, and hence, are typically discarded from analysis. For example, ’wow’, ’lol’,

’omg’, emoticons, and many other non-dictionary expressions frequently appear in tweets. The treatment of these

The successors to emoticons, Unicode ideograms called emoji, such as , are used by Twitter users as multimodal

markers to convey affect, gestures, or repetition, and/or to function semantically in a sentence by replacing parts of

speech [Na’aman et al., 2017, Bai et al., 2019]. However, since text analysis is commonly performed only on ASCII

characters, Unicode emojis are often filtered out. In Redmond et al., emojis have been shown to outperform emoticons

in sentiment classification [Redmond et al., 2017]. Manual [Kralj Novak et al., 2015] and automatic [Kimura and

Katsurai, 2017] methods have been proposed to build emoji lexicons by labeling emojis with their corresponding

sentiment. However, emoji lexicons are limited by the corpora without learning their contextual representations in

word embeddings. We postulate that emojis may be directly represented using their individual UTF-8 encodings for

learning the word embedding. In general, the use of off-the-shelf dictionaries or lexicons, including those used for

Last but not least, the strategy for aggregating multiple word vectors from word embeddings into a constant length

tweet vector is an open research problem. In machine learning, the ordering of constituent words in the tweet vector

different word vector aggregation strategies needs to be evaluated on Twitter SA.

A PREPRINT - JULY 28, 2020

Given the limitations in the current literature, this paper aims to investigate the effects of text processing and classifi-

cation steps on Twitter SA. The contributions of this paper are as follows.

(representing a single word) to a lower K-dimensional encoded space that

A PREPRINT - JULY 28, 2020

has a

K-dimensional vector representation for each of N words corresponding to N rows.

deals with many rare and domain-specific clinical terminologies. To strictly separate the training and test data, only

the training data set (90%) is used to yield the word embedding from the skip gram model. This word-embedding is

classification accuracy.

A PREPRINT - JULY 28, 2020

validation.

: a word vector

x = W[t] and then a = xx

b = variance (x)

SL2 [t] = a and then Svar [t] = b

Each tweet of variable word-length is required to be represented in a Tweet vector of constant length. The baseline

approach is to average (Eq. 4) across n-dimensional m word vectors to yield an n-dimensional tweet vector. We

A PREPRINT - JULY 28, 2020

Figure 2: Effect of Skip gram parameters: (a) word vector dimensionality and (b) n-gram window size on Twitter

sentiment classification performance.

vector. We can obtain the tweet vector as below.

(5)

importance is measured using two metrics in an unsupervised way, i.e., being unaware of the relationship between

the tweet and its sentiment class. The two metrics, the variance and L2 norm of individual word vector, are used to

most important words.

curve (AUC) values for classification. We use three one-versus-all classification models to perform the proposed three-

class sentiment classification. For each sentiment class, the resultant AUC is obtained by averaging across ten AUC

values corresponding to ten test data folds.

tweets, respectively. The label imbalance in the data set can bias the 3-way classification performance due to the

higher proportion of tweets with negative sentiment. With an imbalanced data set, the accuracy metric appears inflated

F1-score and its constituent precision-recall values are significantly affected by data imbalance. The ROC includes

A PREPRINT - JULY 28, 2020

Figure 3: Effect of selecting N most important words on sentiment analysis. Cumulative contribution of top N words

selected from descending or ascending rank order of variance and L2-norm of word vectors. Superior performance of

descending order suggests higher variance and dot products infer more informative words.

sensitivity and specificity values for varying thresholds on classification scores and is not affected by data imbalance.

An F1-score or a confusion matrix can be obtained for each point on the ROC. Thus, the ROC captures the trade-

with hyperparameters : the c value (inverse of regularization weight) and regularization type (L1 and L2 norms). Note

that more sophisticated nonlinear classifier models, such as random forest or gradient boosting trees, do not improve

the performance over the linear classifier model at the cost of higher computational steps as discussed later. We discuss

our findings in the following sections.

We identify eight text processing cases (Table 3) using four text processing steps that are shown in Table 2.

Case 0 generates the word-embedding and performs the classification without involving any text processing, which

equivalent to simple tokenization of the raw tweet. Case 1 cleans raw tweets using regular expression (regEx) based

matching patterns as shown in Table 1. This RegEx-based cleaning isolates punctuation and punctuation-spliced

words such as ’broken.i’ while retaining hashtag words and single characters including the Unicode emoji character

There appears to be no significant improvement following the word vector dimensionality of 40. Hence, we report

Figure 2(b) shows the effect of increasing the window size of neighboring words on Twitter sentiment classification