DATA SETS. What were the data collection process. When, where, and how? PRE-PROCESSING. Natural language data pre-processing steps

Size: px

Start display at page:

Download "DATA SETS. What were the data collection process. When, where, and how? PRE-PROCESSING. Natural language data pre-processing steps"

Felicia Newman
6 years ago
Views:

2 01 DATA SETS What were the data collection process. When, where, and how? 02 PRE-PROCESSING Natural language data pre-processing steps

3 03 RULE-BASED CLASSIFICATIONS Without using machine learning, can we use some basic rules to classify sentiments? 04 MACHINE LEARNING BINARY CLASSIFICATIONS Using machine learning algorithms, we attempt to classify positive and negative sentiments 05 MACHINE LEARNING MULTI-CLASS CLASSIFICATIONS Using machine learning to predict positive, neutral and negative sentiments and determine the best model for data with unknown sentiments

4 06 REASULTS The summary and statistics of the collected airline tweets 07 CLOSING THOUGHTS FEEDBACK Q&A

5 Where, when and how?

@AmericanAir @delta @southwestair @united AA

6 AA DELTA SOUTHWEST UNITED #americanairlines #deltaairlines #southwestairlines #unitedairlines #americanair #deltaair #southwestair #unitedair

7 k 80,121 Tweets TWITTER API Using Python and twython to retrieve tweets through Twitter s API during 7 days period. 14,640 Tweets KAGGLE Reformatted/cleaned tweets with graded sentiment of Major Airlines from Feb ,000 Tweets Newsroom Commercial datasets provided by Newsroom with machine graded tweets

9 Natural language processes to prepare and transform the tweet content for various classification models and sentiment analysis

10 Major processing steps Tokenize all tweet Perform regular Remove all the English For machine learning contents expression stop words Vector Matrix get it?

11 tweet.lower() re.sub('((www\.[^\s]+) (https?://[^\s]+))','url',tweet) r'\1', tweet) Lower Cases Eliminate URLs Turn all letters to lower cases. One Turn all and www format to Convert to display just option is to retain original upper cases display just the word URL username without sign re.sub(r'#([^\s]+)', r'\1', tweet) Replace #hastags & re.sub(r'&', r' and', tweet) Replace &amp Signs re.sub('[\s]+', ' ', tweet) Remove Additional Spaces Convert all #hashtagwords to just Replacing all &amp symbols with the Remove all annoying white spaces hashtagwords actual words and with a space

12 RULE BASED First we will use two simple rule-based techniques to conduct our sentiment analysis to see what happens

13 (pos) x (pos) = (pos) This movie is very good (pos) x (neg) = (neg) This movie is not good at all (neg) x (neg) = (pos) This movie was not bad

15 VADER is a lexicon and rulebased sentiment analysis tool that is especially attuned to sentiments expressed in social media.

16 Examples VADER is smart, handsome and funny. { neg : 0.0, neu : 0.254, pos : 0.746, compound : } Today SUX! { neg : 0.779, neu : 0.221, pos : 0.0, compound : } VADER is very smart, handsome, and funny! { neg : 0.0, neu : 0.293, pos : 0.707, compound : } Today kinda sux! But I ll get by, lol :) { neg : 0.154, neu : 0.42, pos : 0.426, compound : } VADER is VERY SMART, handsome, and FUNNY!!! { neg : 0.0, neu : 0.233, pos : 0.767, compound : } At least it isn t a horrible book. { neg : 0.0, neu : 0.637, pos : 0.363, compound : 0.431}

18 Part I Binary Classifications

19 Data Prep Remove All Neutral Sentiment Random Forest Perform Random Forest Model Linear SVM Perform Support Vector Classifier Hyperplane

20 k k TRAINING: 9,266 TESTING: 2,275 Training Accuracy: 99.87% Testing Accuracy: 89.58% k TRAINING: 1,124 TESTING: 290 Training Accuracy: 99.91% Testing Accuracy: 84.48% TRAINING: 11,541 TESTING: 1,843 Training Accuracy: 99.88% Testing Accuracy: 86.49%

21 k TRAINING: 1,124 TESTING: 290 Training Accuracy: 99.56% Testing Accuracy: 84.14% TRAINING: 11,541 TESTING: 1,843 Training Accuracy: 99.12% Testing Accuracy: 86.38% k k TRAINING: 9,266 TESTING: 2,275 Training Accuracy: 99.18% Testing Accuracy: 90.95%

22 MACHINE LEARING Part II Multi-Class Classifications

23 k TRAINING: 2,040 TESTING: 360 TRAINING: 14,640 TESTING: 2,400

24 CV Accuracy 67.80% 64.86% Test Accuracy 60.88% 66.94% [+1] Precision 33.85% 60.34% [ne] Precision 1.43% 50.00% [ -1] Precision 65.25% 96.89% 0% 20% 40% 60% 80% 100% 120% Kaggle->Tweet Tweet->Tweet While the test accuracy is slightly lower, using Kaggle data as training set proved to be a stronger predictor based on the crossvalidation accuracy results. In addition, it is superior in positive and neutral precision, not to mention much higher training volume.

33% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Kaggle->Tweet Tweet->Tweet Again, while we see the precision of the twitter data

25 CV Accuracy 71.91% 66.39% Test Accuracy 69.08% 69.72% [+1] Precision 47.69% 60.34% [ne] Precision 14.29% 49.10% [ -1] Precision 79.34% 93.33% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Kaggle->Tweet Tweet->Tweet Again, while we see the precision of the twitter data is superior, the Kaggle data yields a more balanced results for all three sentiments. Because the overall test accuracy is about the same, we chose to use Kaggle data as training due to its superior CV accuracy as well as its large sum of data points (~8X).

26 Support Vector Classifier Model Parameters: decision_fun= ovo, kernel= linear, cost=0.4, gamma=0 Use Entire Kaggle Dataset as Training Set (14,640 data points) Apply Model to Tweets of Unknown Sentiment

27 Now we have the model, let s take a look at the results of collected tweets!

28 American 10,516 18,762 Delta 5,728 11,718 Southwest 5,478 9,647 United 5,705 12, ,000 4,000 6,000 8,000 10,000 12,000 14,000 16,000 18,000 20,000 Original Tweets Re-Tweets

29 Negative 52,694 Original Tweets Neutral Positive

30 20, % 18,000 90% 16,000 80% 14,000 70% 12,000 60% 10,000 50% 8,000 40% 6,000 30% 4,000 20% 2,000 10% 0 American Delta Southwest United 0% American Delta Southwest United Negative Netual Positive Negative Netual Positive

31 Negative Sentiment Cenk Uygur Positive Sentiment -

32 Negative Sentiment Delta Assist Positive Sentiment -

33 Negative Sentiment Dalton Rapattoni Positive Sentiment Dalton Rapattoni

34 Negative Sentiment Muslim, Shame Positive Sentiment Happy Birthday, 90th"

35 59.5% 49.2% If the sentiment is classify accurately, people are more likely to retweet a neutral or positive tweets than a negative tweet. 26.0% 31.7% 14.5% 19.1% NEGATIVE NEUTRAL POSITIVE Original Tweets Retweets

36 10,260 I know it's crazy #DFWsnow I'm just excited 2see my baby girl! Man freaking out next 2 me isn't helping 3,795 Retweets The and Lucian Grainge #MusicIsUniversal #UMGShowcase 17,093 Retweets We re partnering to send a lucky fan to games 1 & 2! RT for your chance to win #DeltaMetsSweeps 3,535 Retweets for adding our movie to all your flights! If any of you are traveling and bored make sure u peep it 3,362 Retweets Yo #Kimye, we're really happy for you, we'll let you finish, but South is one of the best names of all time. 1,952 Retweets Been home in Dallas for hours now but still in the PLEASE find my bag. Please RT. #SWAFindDalto 3,329 Retweets this skank bitch was SO RUDE to me on my I asked for her name and she goes, "it's Britney bitch. 352 Retweets Arab-American family of 5 going on vacation - 3 young kids in tow - removed plane for "safety reasons"

37 iphone 60% 50% 40% 30% 20% Others 10% 0% Android The majority of the tweets are from iphone. There is also almost 20% of tweets from Android devices. The rest from the Web client and other applications. There is little difference between the source of negative and positive tweets; but there seems to be more alternative sources for the neutral sentiment. Web Negative Neutral Positive

38 Photo Photo Link Video Link Negative Neutral Positive

39 ENDING THOUGHTS Discussing final thoughts and some future steps

COORDINATES MULTIPLE VALIDATION OTHER MODELS CLUSTERING Current coordinates data Each data point being Naïve Bayes Unsupervised model to isn t great

40 COORDINATES MULTIPLE VALIDATION OTHER MODELS CLUSTERING Current coordinates data Each data point being Naïve Bayes Unsupervised model to isn t great for Geo-location validated by multiple people Lemmatization discover words association analysis

42 Project Description Prior to machine learning, the project conducted two rule-based sentiment classifications. One of which simply count the number of positive and negative words, and the other utilizes VADER lexicon, a pre-built sentiment analysis tool for Python. Both rule-based sentiment classifications achieved an accuracy of 47% to 55%. The project aims to use rule-based classification and machine learning algorithms to predict content sentiments of Twitter feeds that mentioned four major airlines: American Airline, United, Delta, and Southwest. Next, the project used machine learning algorithms Random Forest, and Support Vector Classifier to conduct binary classification on positive and negative sentiments by removing all known neutral tweets. 85% to 90% accuracy were achieved on the test dataset. The project collected 80,121 tweets during a 7-day period via Twitter s API using Twython, a Python package; among these tweets, 2,400 randomly selected tweets were given a sentiment rating manually (test). Through Kaggle, the project also obtained a cleaned dataset contains 14,640 airline tweets with already rated sentiments (training). Finally, the project used Ada Boost and Support Vector Classifier to conduct multi-class classifications by including back all neutral tweets. Through grid-search and cross-validation, the project achieved a 70% overall accuracy on prediction and an 80% precision on negative sentiment prediction on the test dataset. To start off the project, the tweets were processed using tokenization, regular expression, stop words removals, and vectorization. In regular expression, the project eliminated all sign, # sign, and replaced & sign with the word and. Using the best model and parameters found, the model is applied to tweets with unknown sentiments. The statistics summary were included in the later half of the presentation. The presented word clouds also captured interesting stories during the time the tweets were collected.

Predicting the Odds of Getting Retweeted

Predicting the Odds of Getting Retweeted Arun Mahendra Stanford University arunmahe@stanford.edu 1. Introduction Millions of people tweet every day about almost any topic imaginable, but only a small percent