Data Mining. Chapter 7: Score Functions for Data Mining Algorithms. Fall Ming Li

Transcription

1 Data Mining Chapter 7: Score Functions for Data Mining Algorithms Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University

2 The merit of score function Score function indicates what is a good instantiation of a model structure. Ultimately, score function should be able to rank the models (with different complexity or different parameters) according to the usefulness of the model to the data miners Practically, simple generic score functions with well-understood properties and being relatively easy to work with are usually used to derive particular score functions for different applications.

3 Score functions for predictive models (I) Notation: Let training set be Let be the current model to be evaluated 0-1 loss (misclassification loss) Directly measures classification error Can hardly applied to regression

4 Score functions for predictive models (II) Least square loss: The output value should be as close to the observed value as possible Can be applied to both regression and classification problem

5 Score functions for predictive models (III) Hinge loss Enforce to separate the data with a margin using the model as the boundary Output value should have the same sign as the observed value The data should locate in the correct side of the boundary and be away from the boundary at least certain distance. 1 Can be applied to classification problem 0 1

6 Score functions for predictive models (IV) ε-insensitive Enforce the difference between the output value and observed value should be less than ε Can be applied to both classification and regression -ε 0 ε

7 Remarks on score functions for predictive models Strong assumption of previous score function: Every instance and observed labels are equally important to the tasks Input space: Can hardly emphasize some hard or important instances Output space: Can hardly emphasize different cost for making incorrect predictions Different choice of score function may be different upper (lower) bound of the true score functions, minimize (maximize) those score function can implicitly optimize the true score function

8 Score functions for descriptive models (I) Probabilistic models Better models assign higher probability to observed data Likelihood Can be regarded as error term Negative log-likelihood i.e., the over-head that are needed for describe the data

9 Score functions for descriptive models (II) Non-probablistic descriptive models Compactness in clustering Data within cluster should be close to each other Compressed quality in dimensionality reduction Recovered data should be similar to its original one

10 Considering model complexity Modeling in another word Reduce the complexity of the data to something that is more comprehensible High goodness of fit: Can fit the training data reasonably well Low model complexity: model should not be too difficult to describe Score function should reflects the difficulties of describing the current system S I (θ, M) = # bits to describe the data given the model + # bits to describe the model (and parameter)

11 Why consider model complexity? Simple models Model complexity complex models Do not fit data Goodness of fit Perfectly fit data Underfitting Only catch the vague picture Poor generalization Overfitting Catches every individual details Improving generalization ability MSE = Variance + Bias 2 Low complexity Well-fit of data

12 Penalize complexity To achieve better compromise, the score function should penalizes: Error made be by underfitting of the model Complexity of the model Score (model) = error(model) + penalty(model)

13 Score functions using external validation Split data into mutually exclusive parts Design part: used for training Validation part: used for evaluate score function. Different types of splits Random split at a fixed ratio and repeat multiple times Cross-validation: e.g., 10-fold cross validation Leave one out The validation set should be different from the test set for evaluate the performance of the model to achieve unbiased estimate

14 Score functions for pattern structure Two important issues on scoring pattern Coverage: the pattern that frequently appears in the data set should be of more interest Accuracy: the discovered patterns should provide accurate information Should describe the interestingness of the pattern

15 Let s move to Chapter 8