Class Presentation of CSC481 - Artificial Neural Networks. Outline

Transcription

1 page 1 of 34 Class Presentation of CSC481 - Artificial Neural Networks Tiegeng Ren Dept. of Computer Science 04/19/2004 page 2 of 34 Outline Problems in classification systems Introduction of neural networks How to use neural networks Applications Summary 1

2 Classification Systems page 3 of 34 Observation space Mapping Relationship Methods: Statistical classifier ANN-based classifier Solution space Category 1 Category.. Category Category Category Category Category N Landsat TM Band1 Band2 Band3 Band4 Band5 Band6 Band7 Classification Process for Remote Sensing Image Observation space Mapping Relationship Methods: Statistical classifier ANN-based classifier Solution space Category 1 Category.. Category Category Category Category Category N page 4 of 34 Water wetland Forest Agri. Urban Residential (Pattern) Category: Forest 2

3 page 5 of 34 Supervised Remote Sensing Image Classification Vegetation Soil Band 2 (0 ~ 255) Band 1 (0 ~ 255) Water page 6 of 34 Statistical Methods - Need Gaussian (Normal) distribution on the input data which is required by Bayesian classifier. - Restrictions about the format of input data. Band 2 (0 ~ 255) Vegetation Water Soil Band 1 (0 ~ 255) 3

4 page 7 of 34 Statistical Methods: How to find a boundary for the following patterns? Conifer forest Deciduous forest Mixed forest Non-forest Wetland Forest Wetland Brush Land page 8 of 34 Artificial Neural Network Approach No need for normal distribution on input data Flexibility on input data format Improved classification accuracy Robust and reliability 4

5 Introduction to Neural Networks -- Artificial Neural Network Is Defined by... page 9 of 34 Processing elements Organized topological structure Training/Learning algorithms page 10 of 34 Processing Element (PE) Artificial counterparts of neurons in a brain W j1 PE Input W j2 W j3 W j4 f(x) Output W j5 5

6 Function of Processing Elements page 11 of 34 Receive outputs from each PEs locate in previous layer. Compute the output with a Sigmoid activation function PE s Inputs o 1 o 2 o 3 w j1 w j2 w j3 unit j f PE s Output o j F(Sumof(Oi*Wji)) Transfer the output to all the PEs in next layer page 12 of 34 Organized topological structures 6

7 page 13 of 34 ANN Structure - A multi-layer feedforward NN Input layer Hidden layer Output layer Input vector i(x 1, x 2, x n ) Output vector i(o 1, o 2, o m ) page 14 of 34 Training/Learning Algorithm - Backpropagation (BP) Input layer Hidden layer Output layer Input vector i(x 1, x 2, x n ) Output vector i(o 1, o 2, o m ) Feed Information Analysis Result 7

8 page 15 of 34 Training/Learning Algorithm Back-propagation Mechanism Compute total error 1 2 p P Compute the partial derivatives E w ij Update the weights and go to next epoch W (t+1) = W(t) + n ( t p n s ) p 2 n W page 16 of 34 Back-propagation Mechanism Error w Global minimum Wij 8

9 page 17 of 34 BPANN Error Space, Weight Adaptive Steps Total Error Wkl Wij page 18 of 34 How to use a neural network Analysis the problem domain ANN design What structure of ANN to choose What Algorithm to use Input and Output Training Applying the well-trained neural network to your problem 9

10 Band 2 (0 ~ 255) Vegetation Water Pattern Recognition - Understand the problem Vegetation: Soil Band 1 (0 ~ 255) What ANN structure to choose? Multi-layer feed-forward What ANN training algorithm? Back-propagation (10, 89) > (1,0,0) (11,70) > (1,0,0) (1,0,0) Water: (10, 21) > (0,1,0) (15, 32) > (0,1,0) (0,1,0) Soil: (50, 40) > (0,0,1) (52, 40) > (0,0,1) (0,0,1) page 19 of 34 page 20 of 34 ANN Design Band 2 (0 ~ 255) Vegetation Water Pattern Soil Band 1 (0 ~ 255) Input layer Hidden layer Output layer (0 ~ 255) (0 ~ 255) Feed Forward Back-Propagate (0 ~ 1) (0 ~ 1) (0 ~ 1) How many PEs we need - Basic rules in designing an ANN. Input layer PEs - by dimension of input vector Output layer PEs - by total number of patterns (classes) 10

11 page 21 of 34 ANN Training - From Pattern to Land Cover Category Vegetation: (10, 89) > (1,0,0) (11,70) > (1,0,0) (1,0,0) Water: (10, 21) > (0,1,0) (15, 32) > (0,1,0) (0,1,0) Soil: (50, 40) > (0,0,1) (52, 40) > (0,0,1) (0,0,1) Pattern A vegetation pixel (10, 89) > (1,0,0) Feed Forward Back-Propagate Land Cover Category 1 (Vegetation) 0 (Water) 0 (Soil) page 22 of 34 After Training Vegetation Soil A Well-trained Neural Network water A new pixel (x,y), x in band 1, y in band 2 x y 1 (Vegetation) 0 (Water) 0 (Soil) 11

12 page 23 of 34 Real-world Applications Pattern recognition - Remote sensing image classification Banking credit evaluation $tock market data analysis and prediction page 24 of 34 Remote sensing image classification Rhode Island 1999 Landsat-7 Enhanced Thematic Mapper Plus (ETM+) Image Band 4,3,2 In RGB Band 5,4,3 In RGB 12

13 page 25 of 34 ANN Design What ANN structure to choose? Multi-layer feed-forward What ANN training algorithm? Back-propagation & RPROP PE in each layer? -- 6 X 10 page 26 of 34 Training and Testing Pattern Training sample and Testing sample Class Name Training Sample Size (pixels) Testing Sample Size (pixels) Agriculture Barren Land Conifer Forest Deciduous Forest Mixed Forest Brush Land Urban Area Water Non-forest Wetland Forest Wetland Total Pixels

14 page 27 of 34 Classification Result Turf / Grass Barren land Conifer forest Deciduous forest Mixed forest Brush Land Urban area Water Wetland 1 Wetland 2 Some stats on the classification: Training: 1943 pixel, ANN structure: , Training time: 5 hours, final error: (100% %) Classification: over 9 Million pixels, takes 6 hours to get the land-cover map. page 28 of 34 Classification Result - A Close Look Turf / Grass Barren land Conifer forest Deciduous forest Mixed forest Brush Land Urban area Water Wetland 1 Wetland 2 Rhode Island 1999 ETM+ Rhode Island 1999 Land-use and Land-cover map 14

15 page 29 of 34 Banking credit evaluation 10 ~ 20 attributes as input Yearly income, marriage status, credit history, residence, children, etc Expert to choose typical training data set Choose NN structure and training algorithm A dynamic NN structure applied Self-growing algorithm page 30 of 34 Stock market data analysis and prediction a world full of patterns 10 ~ 20 attributes P/E, Weekly Volume, Book Value etc Expert to choose training data set Typically upon a certain pattern/theory NN structure this time, it is different, time plays an important rule. 15

16 page 31 of 34 NN structure and training algorithm in stock market Recurrent network, Generalized neural network Hopfield Network (Recurrent network) Normal BP, Conjugate Gradient Method(CG), Quick-Prop page 32 of 34 Summary A Neural Network consists Processing Element(PE) Topological structure Multi-layer feed-forward Training/learning algorithm Back-propagation Numbers of PEs in each layer 16

17 page 33 of 34 Summary cont. Basic Back-propagation has the following shortcomings: Time-consuming Black box - uncontrollable training Training result unpredictable page 34 of 34 Some reference A good starting point Timothy Master s Practical neural network Recipes in C++ Other site: IEEE Neural Network community 17

18 page 35 of 34 Thank You! 18