Machine Learning Models for Pattern Classification. Comp 473/6731

Transcription

1 Machine Learning Models for Pattern Classification Comp 473/6731 November 17 th 2016 Prof. Neamat El Gayar

2 Topics Introduction on Machine Learning Neural Networks Single Layer Perceptron and Discriminant Functions Multilayer Perceptron and Backpropagation Support Vector Machines Decision Trees Classifier testing and Evaluation 12/7/2016 2

3 What is Machine learning? Algorithms and techniques that allow computers to "learn" extract information from data automatically (find model parameters) Algorithm 12/7/2016 3

4 Why do Machines Need to Learn? Human expertise does not exist (navigating on Mars) Humans are unable to explain their expertise (speech recognition) Solution changes in time (routing on a computer network) Solution needs to be adapted to particular cases (user biometrics) 12/7/2016 4

5 What is Machine learning? use examples to find model 12/7/2016 5

6 Examples ( Data patterns) Face images Fingerprints Speech Signals Documents Satellite Images Medical Images Outlook Temp. Humidity Windy Play sunny hot high false no sunnyoutlook hot Temp. highhumiditytrue Windy no Play overcastsunny hot hot high high false false yes no rainy sunnyoutlook mild hot Temp. high highhumidity false true Windy yes no Play rainyovercastsunny cool hot normal hot high false high false false yes yes no rainy rainy sunny cool Outlook mild normal hot Temp. high true highhumidity false true no Windy yes no Play overcastrainyovercast cool sunny cool normal hot normal hot true high false high false yes false yes yes no sunny rainy rainy mild sunny cool Outlook mild high normal hot Temp. false high true highhumidity false no true no Windy yes no Play sunnyovercastrainy coolovercast normal sunny cool normal hot normal false hot true high false high yes false yes false yes yes no rainy sunny rainy mild rainy mild normal sunny cool mild high normal false hot false high true high yes false no true no yes no sunny sunnyovercast mild rainy coolovercast normal normal cool normal true hot normal false true high yes false yes false yes yes yes overcastrainy sunny mild rainy mild rainy mild high normal cool true mild high normal false false high yes true yes false no no yes overcastsunny sunny hot overcast mild normal rainy cool normal cool normal false cool normal true normal false yes true yes false yes yes yes rainyovercastrainy mild sunny mild rainy mild high mild high normal true cool true high normal false false yes true yes no no overcastsunny sunny hot overcast mild normal cool normal cool normal false normal true false yes true yes yes yes rainyovercastrainy mild sunny mild mild high mild high normal true true high false no false yes yes no overcastsunny sunny hot mild normal cool normal normal false true false yes yes yes rainyovercastrainy mild mild mild high high normal true true false no yes yes overcastsunny hot mild normal normal false true yes yes rainyovercastmild mild high high true true no yes overcast hot normal false yes rainy mild high true no 12/7/2016 6

7 ..more Examples ( Data patterns) Social Network Data Sensor Data Enterprise Data... 12/7/2016 7

8 The Information Age Data Intelligence 90% of world s data in last 2 years Wisdom ----> Enlightenment 2020, 40% of data from sensors 500 million tweets per day Analytics is the KEY!!! YouTube: 48 hours of video every minute Hospitals: 665 terabytes of patient data, 80% unstructured (CT scans and x-rays) Over 100bn s per day Facebook: 100 terabytes of data daily. 571 new websites per minute. More data created in 2012 than in prior 5,000 years Now: More than 5 billion people virtually interacting. 12/7/2016 8

9 Trends in Machine Learning ML research 2005 Semi-supervised Learning Ensemble Networks Support Vector Machines Online Learning ML research 2015 Deep Learning Recurrent Networks Large scale Learning & Big Data Challenges Biologically inspired learning IAPR TC Neural Networks and Computational Intelligence 12/7/2016 9

10 Applications of Machine Learning Learning Association Supervised Learning Classification Regression Unsupervised Learning Reinforcement Learning 12/7/

11 Basic topics in ML Bayesian Decision Theory More on supervised learning: KNN, SVM, Bayesian networks, Decision Trees Classifier combination Neural Networks: MLP, RBF, CLN, SOM Classifier evaluation and testing Clustering 12/7/

12 Recent topics in ML Neural Networks... Large Scale Machine Learning... Deep Learning 12/7/

13 Popular topics in ML Bayesian Decision Theory More on supervised learning: KNN, SVM, Bayesian networks, Decision Trees Classifier combination Neural Networks: MLP, RBF, CLN, SOM Classifier evaluation and testing Clustering 12/7/

14 Scope of Work Applications Emotion Analysis(Speech &Text) Bioinformatics Biometric Identification Security Ubiquitous Computing Medical Imaging Hotel Revenue Management Mining Social Network Data Ensembles and Neural Networks Machine Learning Research Semi-Supervised Learning Prediction 12/7/

15 Outline today s talk Introduction to Artificial Neural Networks Single Layer Neural Networks Representational Power Learning Algorithms Limitation Multi Layer Perceptron Representational Power Backpropagation Variations and design choices 12/7/

16 Introduction to Neural Networks

17 What are Artificial NN? Neural Networks are massively connected networks of elementary processors (simple computational elements). ANN are biologically inspired NN are also called connectionism or parallel distributed processing. 12/7/

18 Why Artificial NN? There are two basic reasons why we are interested in building artificial neural networks (ANNs): Technical viewpoint: Some problems such as character recognition or the prediction of future states of a system require massively parallel and adaptive processing. Biological viewpoint: ANNs can be used to replicate and simulate components of the human (or animal) brain, thereby giving us insight into natural information processing. 12/7/

19 Neural Computing Neuroscience The objective is to understand the human brain Biologically realistic models of neurons Biologically realistic connection topologies Neural networks The objective is to develop computation methods Highly simplified artificial neurons Connection topologies that are aimed at computational effectiveness 12/7/

20 Neural Computing Neural Computing can be defined as: the study of networks of adaptable nodes which through a process of learning from task examples, store experiential knowledge and make it available for use. 12/7/

21 Neural Computing Neural Computing can be defined as: the study of networks of adaptable nodes which through a process of learning from task examples, store experiential knowledge and make it available for use. 12/7/

22 Biological Neuron Main parts Cell body (soma) Dendrites Axon Synapses 100 billion neurons Each neuron connected to 1000 other 100 trillion synaptic junctions excitatory inhibitory Adaptable = learning 12/7/

23 Simple Neuroscience Messages passed or not are based on the neuron s excitation status. From Neuroscience for Kids The Synapse 12/7/

24 Artificial Neuron x 1 w 1 x 2 w2 w 0 v y w n linear transfer x n combiner function 12/7/

25 Advantages of ANNs Inherent massively parallel high computation rates User-friendly Learning instead of programming Ability to generalize Learn by example (no algorithm required) Adaptation and continuous learning Robust Can deal with incomplete and/or noisy data Fault-tolerant Still works when part of the net fails 12/7/

26 Applications Pattern association Pattern recognition (speech, character & text, image & signal processing) Clustering Function approximation Control Adaptive filtering Data compression Financial Analysis 12/7/

27 Neural Networks A neural network (NN) is a machine learning approach inspired by the way in which the brain performs a particular learning task. A NN is specified by: an architecture: a set of neurons and links connecting neurons. Each link has a weight, a neuron model: the information processing unit of the NN, a learning algorithm: used for training the NN by modifying the weights in order to model the particular learning task correctly on the training examples. The aim is to obtain a NN that generalizes well, that is, that behaves correctly on new instances of the learning task. 12/7/

28 Neural Networks A neural network (NN) is a machine learning approach inspired by the way in which the brain performs a particular learning task. A NN is specified by: an architecture: a set of neurons and links connecting neurons. Each link has a weight, a neuron model: the information processing unit of the NN, a learning algorithm: used for training the NN by modifying the weights in order to model the particular learning task correctly on the training examples. The aim is to obtain a NN that generalizes well, that is, that behaves correctly on new instances of the learning task. 12/7/

29 Neural Architecture

30 Network architectures Three different classes of network architectures single-layer feed-forward neurons are organized multi-layer feed-forward in acyclic layers recurrent The architecture of a neural network is linked with the learning algorithm used to train 12/7/

31 Single Layer Feed-forward Input layer of source nodes Output layer of neurons 12/7/

32 Multi layer feed-forward Network Input layer Output layer Hidden Layer 12/7/

33 Recurrent network Recurrent Network with hidden neuron: unit delay operator z -1 is used to model a dynamic system z -1 z -1 input hidden output z -1 12/7/

34 The Artificial Neuron

35 The Neuron Bias b x 1 w 1 Input values x 2 w 2 Summing function Local Field v Activation function ( ) 1 sgn ( x) Output y 0.5 x m w m weights /7/

36 The Neuron The neuron is the basic information processing unit of a NN. It consists of: 1 A set of links, describing the neuron inputs, with weights W 1, W 2,, W m 2 An adder function (linear combiner) for computing the weighted sum of the inputs (real numbers): u m wjxj j 1 3 Activation function (squashing function) for limiting the amplitude of the neuron output. y (u b) 12/7/ Machine Learning, by Neamat El Gayar, Spring 2009

37 The Neuron Bias b x 1 w 1 Input values x 2 w 2 Summing function Local Field v Activation function ( ) c = 0.5 c = 1.0 c = 2.0 Output y x m w m weights /7/

38 Bias of a Neuron The bias b u m wjxj j 1 y (u b) x2 x1-x2= -1 x1-x2=0 x1-x2= 1 y x1 x2 x1 Machine Learning, by Neamat El Gayar, 12/7/ Spring 2009

39 Learning in a ANN

40 Classification Example x 2 x 1 12/7/

41 Equation of a Line x 2 2x 1 + 3x 2-6 = 0 2 2x 1 + 3x 2-6 > 0 2x 1 + 3x 2-6 < x 1 12/7/

42 Neural Classifier x 0 =1 w 0 = -6 x 1 w 1 = 2 w 2 = 3 s= x i w i y= +1 y=sgn(s) y= -1 x 2 12/7/

43 Equation of a Line is Wrong x 2 2x 1 + 2x 2-4 = x 1 12/7/

44 Need to Learn Tweak parameter w i to see its effect Modify w i in the direction which reduces the error

45 Learning rules How should you adapt the weights to meet the desired performance (i.e to perform a certain task) Synchronous vrs. Asynchronus Itrerative vrs. Batch Supervised vrs. Unsupervised Early learning models: Hebb: w jk = x j x k Perceptron w jk = d x; Widrow-Hoff Delta Learning rule: w jk = x (d x); 12/7/

46 History Early stages 1943 McCulloch-Pitts: neuron as comp. elem Wiener: cybernatics 1949 Hebb: learning rule 1958 Rosenblatt: perceptron 1960 Widrow-Hoff: least mean square algorithm Recession 1969 Minsky-Papert: limitations perceptron model Revival 1982 Hopfield: recurrent network model 1982 Kohonen: self-organizing maps 1986 Rumelhart et. al.: backpropagation 12/7/

47 Single layer Network Perceptron and Widrow-Hoff learning 12/7/

48 Single layer NN We consider the simple architecture consisting of just one neuron. Generalization to a single layer with more neurons is easy because: The output units are independent among each other Each weight only affects one of the outputs 12/7/

49 Perceptron: Neuron Model The (McCulloch-Pitts) perceptron is a single layer NN with a non-linear, the sign function ( v) x 1 1if 1if v v b (bias) 0 0 x 2 w 1 w 2 v (v) y w n x n 12/7/

50 Perceptron for Classification The perceptron is used for binary classification. Given training examples of classes C 1, C 2 train the perceptron in such a way that it classifies correctly the training examples: If the output of the perceptron is +1 then the input is assigned to class C 1 If the output is -1 then the input is assigned to C 2 12/7/

51 Perceptron Training How can we train a perceptron for a classification task? Find suitable values for the weights in such a way that the training examples are correctly classified. Geometrically, we try to find a hyper-plane that separates the examples of the two classes. 12/7/

52 Perceptron Geometric View The equation below describes a (hyper-)plane in the input space consisting of real valued m-dimensional vectors. The plane splits the input space into two regions, each of them describing one class. m x 2 decision region for C1 w 1 x 1 + w 2 x 2 + w 0 >= 0 i 1 w x w i i 0 0 decision boundary C 1 C 2 x 1 w 1 x 1 + w 2 x 2 + w 0 = 0 12/7/

53 The fixed-increment learning algorithm n=1; initialize w(n) randomly; while (there are misclassified training examples) Select a misclassified augmented example (x(n),d(n)) w(n+1) = w(n) + d(n)x(n); n = n+1; end-while; = learning rate parameter (real number) 12/7/

54 v (voice freq.) Single-Layer Perceptrons Example: Gender classification Learning rule wi = k t xi Network Arch. Training data h v w1 w2 w0 y y = signum(hw1+vw2+w0) -1 if female = 1 if male h (hair length) 12/7/

55 Perceptron: Limitations The perceptron can only model linearly separable classes, like (those described by) the following Boolean functions: AND OR COMPLEMENT It cannot model the XOR. 12/7/

56 Example 2 Draw the discriminant function for the following two classes: Class A: (0.5,1.5),(0.5,0.5),(2,1.5). Class B: (-0.5,0.5),(0.5,-0.25),(1.5,-1). Where the weights w1, w2,, are 1,2, and 2 respectively. (verify your results analytically) 12/7/

57 Example 1 12/7/

58 Example 3 Draw the discriminant function for the following two classes: Class A: (-1,-1), (1,1). Class B: (-1,1), (1,-1). (a) Using single layer perceptron. 12/7/

59 Example 3 12/7/

60 Adaline: Adaptive Linear Element When the two classes are not linearly separable, it may be desirable to obtain a linear separator that minimizes the mean squared error. Adaline (Adaptive Linear Element): uses a linear neuron model and the Least-Mean-Square (LMS) learning algorithm useful for robust linear classification and regression For an example (x,d) the error e(w) of the network is e(w) d m j 0 x j w j and the squared error is 2 E( w) 1 e 2 12/7/

61 Adaline The total error E_tot is the mean of the squared errors of all the examples. E_tot is a quadratic function of the weights whose derivative exists everywhere. Then incremental gradient descent may be used to minimize E_tot. At each iteration LMS algorithm selects an example and decreases the network error E of that example, even when the example is correctly classified by the network. 12/7/

62 Incremental Gradient Descent start from an arbitrary point in the weight space the direction in which the error E of an example is decreasing most rapidly is the opposite of the gradient of E: take a small step (of size ) in that direction w(t 1) w(t) (gradient of E(w(t))) 12/7/

63 Weights Update Rule Computation of Gradient(E): E(w) w e e w Delta rule for weight update: w(t 1) w(t) e(p,t)x(p) 12/7/

64 LMS learning algorithm n=t; initialize w(t) randomly; while (E_tot unsatisfactory and t<max_iterations) t = t+1; Select an example (x(p),d(p)) end-while; = learning rate parameter (real number) A modification uses w( t 1) w( t) e( p, t) x( p) T e( p, t) d( p) w( p, t) x( p) w( t 1) w( t) e( p) x( p) x( p) 12/7/

65 Comparison Perceptron and Adaline Perceptron Adaline Architecture Single-layer Single-layer Neuron model Learning algorithm Non-linear Minimze number of misclassified examples linear Minimize total error Application Linear classification Linear classification regression 12/7/

66 Multi Layer Perceptron Backpropagation

67 Multilayer Perceptrons (MLPs) Example: XOR problem Training data Network Arch. x1 x2 y x1 x2 y x2 x1 x2 y x1 12/7/

68 MLP Decision Boundaries XOR Interwined General 1-layer: Half planes A B B A A B 2-layer: Convex A B B A A B 3-layer: Arbitrary A B B A A B 12/7/

69 Goal: Classify objects by learning nonlinearity There are many problems for which linear discriminants are insufficient for minimum error In using the multilayer Neural Networks, the form of the nonlinearity is learned from the training data Multilayer neural networks can, in principle, provide the optimal solution to any classification problem (universal approximator) 12/7/

70 Feedforward Operation and Classification A three-layer neural network consists of an input layer, a hidden layer and an output layer interconnected by modifiable weights represented by links between layers 12/7/

71 12/7/

72 Multilayer Perceptron Expressive Power The ability of the MLP to generalize is performed by the hidden layer neurons The knowledge stored in the hidden layer neurons is abstracted from the information contained in the input pattern 12/7/

73 Network Learning Let t k be the k-th target (or desired) output and z k be the k- th computed output with k = 1,, c and w represents all the weights of the network The per-pattern training error: J( w ) 1 2 c k 1 ( t k z k ) t z 2 The backpropagation learning rule is based on gradient descent 12/7/2016 The weights are initialized with random values and are changed in a direction that will reduce the error: w J w 73

74 Backpropagation algorithm is so-called because during training an error must be propagated from the output layer back to the hidden layer in order to perform the learning of the input-to-hidden weights If the weights w kj to the output layer were ever all zero, the back-propagated error would also be zero and the input-to-hidden weights would never change! 12/7/

75 The weight update (or learning rule) for the hidden-tooutput weights is: w kj = k y j = (t k z k ) f (net k )y j The sensitivity at a hidden unit is simply the sum of the individual where sensitivities the sensitivity at the output of unit units k, k weighted describes by how the the hidden-to-output overall error changes weights with wthe activation of the unit s net kj ; all multipled by f (net j ) The learning rule for the input-to-hidden weights is: j 12/7/

76 12/7/

77 Activation Function Desired properties Nonlinear Continuous and differentiable Saturate: have maximum and minimum output value --- This will keep the weights and activations bounded and thus keep training time limited. Monotonic: if not monotonic and has multiple local maxima, additional local extrema in the error surface may become introduced. Linear for a small value of net --- enable the system to implement a linear model 12/7/

78 Activation Function The class of sigmoid is the most widely used activation function which has all the desired properties The logistic sigmoid f f 1 ( net ) net 1 e '( net ) f ( net )(1 ( net )) It is best to keep the function centered on zero and antisymmetric or as an odd function, that is, f(-net)=-f(net). f 12/7/

79 Activation Function For faster learning, use sigmoid functions of the form of a hyperbolic tangent f ( net ) a tanh( bnet ) e a e a=1.716, b=1/3 f(net) is nearly linear in the range -1<net<+1. See Figure 6.14 bnet bnet e e bnet bnet 12/7/

80 12/7/

81 Main Problems with BP learning Slow Learning: Long number of iterations to reach a solution Gradient descent trapped in local minimum (network paralysis) Parameter Settings / choosing network structure 12/7/

82 Some Implementation Issues Learning rate update Weight Initialization Stopping Criteria Input scaling Incremental vrs. Batch training Network Parameters/topology (hidden neurons, ) 12/7/

83 Stopping criterion The algorithm terminates when the change in the criterion function J(w) is smaller than some preset value There are other stopping criteria that lead to better performance than this one 12/7/

84 Learning rate update Learning rate = size of step down the error gradient ; η is usually chosen [0,1] η v. small long training but better chance to find a local minimum η large unstable training as you might step over the minima To speed up learning Use an adaptive learning rate (if error decreases increase step size; if error increases- decrease step size) Very common choose η as monotonically decreasing function of the number of training iterations 12/7/

85 12/7/

86 Weight Initialization Usually are set to small random weights Goal: have fast and uniform learning -- all weights reach their final equilibrium values at about the same time 12/7/

87 Weight Initialization Use a uniform distribution -w<w ji <+w, that place the neurons in the linear range -1<net<+1 The net activation from standardized input is distributed roughly between w d net j w d 1 d w ji 1 d 1 n H w kj 1 n H 12/7/

88 Input scaling Why? Limit the range of weights so as to prevent premature saturation of sigmoidal fn. Give each input/ feature equal importance Methods of scaling: 1. Linear Scaling : map input to range [a,b] x k i ( b a) x x k i max i bx min i min i x ax max i 12/7/

89 89 Input scaling(cont.) 2. Mean- Variance Scaling input The values of one feature may be orders of magnitude larger than that of another Standardize the training patterns i i k i k i n k k i k i i n k k k i i x x n x x n ˆ ˆ 1 ) ˆ ( ˆ ; 1 ˆ /7/2016

90 Batch vrs. Iterative BP In the batch training, all the training patterns are presented first and their corresponding weight updates summed; only then are the actual weights in the network updated 12/7/

91 Online/Stochastic or batch learning? For most applications especially ones employing large redundant training sets stochastic learning is typically faster than batch learning (+ less chance to get stuck in local minimum) Online learning : (every pattern is presented only once): Used when large training patterns and memory constraints 12/7/

92 Overfitting Learning Curves Before training starts, the error on the training set is high; through the learning process, the error becomes smaller --- tends to decrease monotonically Excessive training can lead to poor generalization as the network implements a complex decision boundary tuned to the specific training data rather than the general properties of the underlying distributions --- overfitting 12/7/

93 Other Stopping criterion The average error on an independent test set can decrease as well as increase A validation set is used in order to decide when to stop training ; we do not want to overfit the network and decrease the power of the classifier generalization we stop training at a minimum of the error on the validation set 12/7/

94 12/7/

95 m-fold cross-validation: The training set is randomly divided into m disjoint sets of equal size n/m The training procedure is run m times, each time with a different set held out as a validation set. On each run the number of epochs is determined that yield the best performance on the validation set. The mean k of these estimates is calculated, and a final run of backpropagation algorithm is performed on all n examples for k epochs. 12/7/

96 Some Tips Introducing noise Generate virtual training patterns Add Gaussian noise to true training points with category label unchanged e.g, for the standardized inputs: noise ~ N(0,0.01) Improve generalization, helps avoid over-fitting 12/7/

97 Some Tips Manufacture training data Use knowledge such as geometrical invariances among patterns Manufacture patterns by translating or rotating given training patterns if translation and rotation invariant recognition is desired 12/7/

98 Some Tips Number of hidden units The number of hidden units governs the expressive power of the net --- and thus the complexity of the decision boundary A convenient rule of thumb is to choose the number of hidden units such that the total number of weights in the net is roughly n/10 A more principled method is to adjust the complexity of the network in response to the training data 12/7/

99 12/7/

100 Some Tips More than one hidden layer? Unless under special problem requirements Empirically, more prone to getting caught in undesirable local minima The backpropagation algorithm is easily generalized 12/7/

101 Steps of NN implementation How to select training and test set How to select Network Architecture Data scaling/weight initialization Error goal How to avoid overfitting 12/7/

102 WHY??? 1. Local Minimum 2. Net has not enough degree of freedom 3. Training data is poor Change Weights or Increase NN size No Collect Data Training and test data should cover the entire space where Select Training and the Test net Sets is expected to operate Start with a large net and shrink it or start with a small Select Network Architecture net and grow it Initialize Weights Error goal met?e Yes Run on Test Set WHY??? Train Poor Net Generalization 1. Incomplete training set 2. Over fitting due to large degree Reselect of freedom Training or collect more data Error goal met?e Yes 12/7/ END No

103 Overfitting! When does overfitting happen? Network error level on training data is significantly better than on test data Why? Peculiarities of training data are learned; such as noise. Not the functional relationship of the model 12/7/

104 How to avoid overfitting! Limiting the number of free parameters (no. Of hidden neurons) Increasing training set size Use a realistic error goal Use cross validation to stop learning 12/7/

105 Textbooks Introduction to Machine Learning (2004), by Ethem ALPAYDIN, The MIT Press, ISBN Statistical Pattern Recognition, 2 nd Edition, Andrew Webb, 2005 Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wiley & Sons, 2000 Pattern Recognition and Machine Learning (2006), by Christopher M. Bishop, Springer (2006). Data Mining: Practical Machine Learning Tools and Techniques (2005), by Ian Witten and Eibe Frank(2005) 2nd Edition, Morgan Kaufmann, ISBN Machine Learning (1997), Tom Mitchell, McGraw Hill,, ISBN Principles of Data Mining (2001)., by David Hand, Heikki Mannila, and Padhraic Smyth MIT Press. 12/7/2016 Machine Learning, by Neamat El Gayar, 105 Spring 2010