Data Mining Applications with R

Size: px

Start display at page:

Download "Data Mining Applications with R"

Shanon Marshall
6 years ago
Views:

1 Data Mining Applications with R Yanchang Zhao Senior Data Miner, RDataMining.com, Australia Associate Professor, Yonghua Cen Nanjing University of Science and Technology, China AMSTERDAM BOSTON HEIDELBERG LONDON NEW YORK OXFORD PARIS SAN DIEGO ELSEVIER SAN FRANCISCO SYDNEY TOKYO Academic Prcsi is in imprint of Elsevier

2 Contents Preface Acknowledgments Review Committee Foreword Chapter 1: Power Grid Data Analysis with R and Hadoop Introduction A Brief Overview of the Power Grid Introduction to MapReduce, Hadoop, and RHIPE MapReduce Hadoop RHIPE: R with Hadoop Other Parallel R Packages Power Grid Analytical Approach Data Preparation Exploratory Analysis and Data Cleaning Event Extraction Discussion and Conclusions 31 Appendix 32 References 34 xiii xv xvii xix Chapter 2: Picturing Bayesian Classifiers: A Visual Data Mining Approach to Parameters Optimization Introduction Related Works Motivations and Requirements R Packages Requirements Probabilistic Framework of NB Classifiers Choosing the Model Estimating the Parameters Two-Dimensional Visualization System Design Choices Visualization Design 49 v

3 vi Contents 2.6 A Case Study: Text Classification Description of the Dataset Creating Document-Term Matrices Loading Existing Term-Document Matrices Running the Program Conclusions 59 Acknowledgments 60 References 60 Chapter 3: Discovery ofemergent Anthropology Using Text Mining, Topic Modeling, Network Analysis ofmicroblog Issues and Controversies in and Social Content Introduction How Many Messages and How Many Twitter-Users in the Sample? Who Is Writing All These Twitter Messages? Who Are the Influential Twitter-Users in This Sample? What Is the Community Structure of These Twitter-Users? What Were Twitter-Users Writing About During the Meeting? What Do the Twitter Messages Reveal About the Opinions of Their Authors? What Can Be Discovered in the Less Frequently Used Words in the Sample? What Are the Topics That Can Be Algorithmically Discovered in This Sample? Conclusion 88 References 91 Chapter 4: Text Mining and Network Analysis of Digital Libraries in R Introduction Dataset Preparation Manipulating the Document-Term Matrix The Document-Term Matrix Term Frequency-Inverse Document Frequency Exploring the Document-Term Matrix Clustering Content by Topics Using the LDA The Latent Dirichlet Allocation Learning the Various Distributions for LDA Using the Log-Likelihood for Model Validation Topics Representation Plotting the Topics Associations Using Similarity Between Documents to Explore Document Cohesion Computing Similarities Between Documents Using a Heatmap to Illustrate Clusters of Documents 109

4 Contents vii 4.6 Social Network Analysis of Authors Constructing the Network as a Graph Author Importance Using Centrality Measures Conclusion 115 References 115 Chapter 5: Recommender Systems in R Introduction Business Case Evaluation Collaborative Filtering Methods Latent Factor Collaborative Filtering Simplified Approach Roll Your Own Final Thoughts 149 References 151 Chapter 6: Response Modeling in Direct Marketing: A Data Mining-Based Approach for Target Selection Introduction/B ackground Business Problem Proposed Response Model Modeling Detail Data Collection Data Preprocessing Feature Construction Feature Selection Data Sampling for Training and Test Class Balancing Classifier (SVM) Prediction Result Model Evaluation Conclusion 177 References 178 Chapter 7: Caravan Insurance Customer Profile Modeling with R Introduction Data Description and Initial Exploratory Data Analysis Variable Correlations and Logistic Regression Analysis Classifier Models of Caravan Insurance Holders Overview of Model Building and Validating Review of Four Classifier Methods RP Model Bagging Ensemble 192

5 viii Contents Support Vector Machine LR Classification Comparison of Four Classifier Models: ROC and AUC Model Comparison: Recall-Precision, Accuracy-v-Cut-off, and Computation Times Discussion of Results and Conclusion 206 Appendix Appendix B Customer Profile Data-Frequency of Binary Appendix C Proportion of Caravan Insurance Holders vis-a-vis other A Details of the Full Data Set Variables 209 Values 212 Customer Profile Variables 220 Appendix D LR Model Details 222 Appendix E R Commands for Computation of ROC Curves for Each Model Using Validation Dataset 225 Appendix F Commands for Cross-Validation Analysis of Classifier Models 225 References 226 Chapter 8: Selecting Best Features for Predicting Bank Loan Default Introduction Business Problem Data Extraction Data Exploration and Preparation Null Value Detection Outlier Detection Missing Imputation Relevance Analysis Data Set Balancing Feature Selection Modeling Model Evaluation Finding and Model Deployment Lessons and Discussions 244 Appendix Selecting Best Features for Predicting Bank Loan Default 244 References 245 Chapter 9: A Choquet Integral Toolbox and Its Application in Customer Preference Analysis Introduction Background Aggregation Functions Choquet Integral Fuzzy Measure Representation Shapley Value and Interaction Index 252

6 Contents ix 9.3 Rfmtool Package Installation Toolbox Description Preference Analysis Example Case Study Traveler Preference Study and Hotel Management Data Collection and Experiment Design Model Evaluation Result Analysis Discussion Conclusions 270 References 271 Chapter 10: A Real-Time Property Value Index Based on Web Data Introduction Housing Prices and Indices A Data Mining Approach Data Capture Geocoding Price Evolution Real Estate Pricing Models Model 1: Hedonic Model Plus Smooth Term Model 2: GWR Plus a Smooth Term Relationship to Other Work Conclusion 295 Acknowledgments 295 References 295 Chapter 11: Predicting Seabed Hardness Using Random Forest in R Introduction Study Region and Data Processing Study Region Data Processing of Seabed Hardness Predictors Dataset Manipulation and Exploratory Analyses Features of the Dataset Exploratory Data Analyses Application of RF for Predicting Seabed Hardness Model Validation Using rfcv Optimal Predictive Model Application of the Optimal Predictive Model Discussion and Conclusions Selection of Relevant Predictors and the Consequences of Missing the Most Important Predictors Issues with Searching for the Most Accurate Predictive Model Using RF 323

7 x Contents Predictive Accuracy of RF and Prediction Maps of Seabed Hardness Limitations 325 Acknowledgments 326 Appendix AA Dataset of Seabed Hardness and 15 Predictors 326 Appendix BA R Function, if.cv, Shows the Cross-Validated Prediction Performance of a Predictive Model 326 References 327 Chapter 12: Supervised Classification ofimages, Applied to Plankton Samples Using R and Zooimage Background Challenges Data Extraction and Exploration Data Preprocessing Modeling Model Evaluation Model Deployment Lessons, Discussion, and Conclusions 359 Acknowledgments 362 References 363 Chapter 13: Crime Analyses Using R Introduction Problem Definition Data Extraction Data Exploration and Preprocessing Visualizations Modeling Model Evaluation Discussions and Improvements 394 References 395 Chapter 14: Football Mining with R Introduction to the Case Study and Organization of the Analysis Background of the Analysis: The Italian Football Championship Data Extraction and Exploration Data Extraction Data Exploration Data Preprocessing Variable Importance Evaluation Composite Indicators Construction Model Development: Building Classifiers Learning Step 413

8 Contents xi Model Selection Model Refinement Model Deployment Concluding Remarks 430 Acknowledgments 431 References, 431 Chapter 15: Analyzing Internet DNS(SEC) Traffic with R for Resolving Platform Optimization Introduction Data Extraction from PCAP to CSV File Data Importation from CSV File to R Dimension Reduction Via PCA Initial Data Exploration Via Graphs Variables Scaling and Samples Selection Clustering for Segmenting the FQDN Building Routing Table Thanks to Clustering Building Routing Table Thanks to Mixed Integer Linear Programming Building Routing Table Via a Heuristic Final Evaluation Conclusion 454 References 455 Index 457

Preface to the third edition Preface to the first edition Acknowledgments

Contents Foreword Preface to the third edition Preface to the first edition Acknowledgments Part I PRELIMINARIES XXI XXIII XXVII XXIX CHAPTER 1 Introduction 3 1.1 What Is Business Analytics?................