Metaheuristics applied to the feature selection problem

Transcription

1 Metaheuristics applied to the feature selection problem Tom Fredrik B. Department of Mathematics University of Oslo Master presentation 28th June

2 What we will see Motivation Feature selection I will present 2 new methods for the FSP I will present a number of adapted methods, from other scientific fields. I will present a visualization algorithm for the FSP

3 What we will see Motivation Feature selection I will present 2 new methods for the FSP I will present a number of adapted methods, from other scientific fields. I will present a visualization algorithm for the FSP

4 What we will see Motivation Feature selection I will present 2 new methods for the FSP I will present a number of adapted methods, from other scientific fields. I will present a visualization algorithm for the FSP

5 Introduction What we will see Motivation Feature selection Segmentation We want to separate a set of data into distinct subregions: Figure: Ground truth for Fontainebleu dataset Curse of dimensionality Natural to add more data to get better separation, but results often deteriorate Solution: Try to use the best set of data

6 Introduction What we will see Motivation Feature selection Segmentation We want to separate a set of data into distinct subregions: Figure: Ground truth for Fontainebleu dataset Curse of dimensionality Natural to add more data to get better separation, but results often deteriorate Solution: Try to use the best set of data

7 Introduction What we will see Motivation Feature selection Segmentation We want to separate a set of data into distinct subregions: Figure: Ground truth for Fontainebleu dataset Curse of dimensionality Natural to add more data to get better separation, but results often deteriorate Solution: Try to use the best set of data

8 Feature selection Introduction What we will see Motivation Feature selection The Feature Selection is: An Image Analysis A Parameter Estimation A Discrete Optimization

9 Feature selection Introduction What we will see Motivation Feature selection The Feature Selection is: An Image Analysis A Parameter Estimation A Discrete Optimization

10 Feature selection Introduction What we will see Motivation Feature selection The Feature Selection is: An Image Analysis A Parameter Estimation A Discrete Optimization

11 Feature selection Introduction What we will see Motivation Feature selection The Feature Selection is: An Image Analysis A Parameter Estimation A Discrete Optimization

12 Feature selection Introduction What we will see Motivation Feature selection The Feature Selection is: An Image Analysis A Parameter Estimation A Discrete Optimization

13 formulation Regularization Motivation for metaheuristics Standard formulation max p(x ω c ) = c Cholesky factorization 1 (2π) d/2 Σ c 1/2 exp( 1 2 (x ˆµ c) T Σ 1 c (x ˆµ c )) L c L T c = Σ c Equivalent, more easily calculated formulation max c p (x ω c ) = 1 2 L 1 c N (x ˆµ c ) 2 log( i=1 (L 1 c ) i,i )

14 formulation Regularization Motivation for metaheuristics Standard formulation max p(x ω c ) = c Cholesky factorization 1 (2π) d/2 Σ c 1/2 exp( 1 2 (x ˆµ c) T Σ 1 c (x ˆµ c )) L c L T c = Σ c Equivalent, more easily calculated formulation max c p (x ω c ) = 1 2 L 1 c N (x ˆµ c ) 2 log( i=1 (L 1 c ) i,i )

15 formulation Regularization Motivation for metaheuristics Standard formulation max p(x ω c ) = c Cholesky factorization 1 (2π) d/2 Σ c 1/2 exp( 1 2 (x ˆµ c) T Σ 1 c (x ˆµ c )) L c L T c = Σ c Equivalent, more easily calculated formulation max c p (x ω c ) = 1 2 L 1 c N (x ˆµ c ) 2 log( i=1 (L 1 c ) i,i )

16 formulation Regularization Motivation for metaheuristics Motivation Inverting nearly singular matrices is numerically very unstable. How can we get more stability? Regularization Side effect: stabilizes the parameter estimation on its own merit. Choice of regularizer ˆΣ c (α) = αˆσ c + (1 α)ˆσ (1) which combined yields: ˆΣ(λ) = λˆσ + (1 λ)ˆσ 2 I (2) ˆΣ c (α, λ) = αˆσ c + (1 α)ˆσ(λ) (3)

17 formulation Regularization Motivation for metaheuristics Motivation Inverting nearly singular matrices is numerically very unstable. How can we get more stability? Regularization Side effect: stabilizes the parameter estimation on its own merit. Choice of regularizer ˆΣ c (α) = αˆσ c + (1 α)ˆσ (1) which combined yields: ˆΣ(λ) = λˆσ + (1 λ)ˆσ 2 I (2) ˆΣ c (α, λ) = αˆσ c + (1 α)ˆσ(λ) (3)

18 formulation Regularization Motivation for metaheuristics Motivation Inverting nearly singular matrices is numerically very unstable. How can we get more stability? Regularization Side effect: stabilizes the parameter estimation on its own merit. Choice of regularizer ˆΣ c (α) = αˆσ c + (1 α)ˆσ (1) which combined yields: ˆΣ(λ) = λˆσ + (1 λ)ˆσ 2 I (2) ˆΣ c (α, λ) = αˆσ c + (1 α)ˆσ(λ) (3)

19 formulation Regularization Motivation for metaheuristics Motivation Inverting nearly singular matrices is numerically very unstable. How can we get more stability? Regularization Side effect: stabilizes the parameter estimation on its own merit. Choice of regularizer ˆΣ c (α) = αˆσ c + (1 α)ˆσ (1) which combined yields: ˆΣ(λ) = λˆσ + (1 λ)ˆσ 2 I (2) ˆΣ c (α, λ) = αˆσ c + (1 α)ˆσ(λ) (3)

20 formulation Regularization Motivation for metaheuristics Running time of exact algorithms Exact algorithms are available for the Feature Selection, but running time is exponential. Local search Local search is an easy way, to obtain relatively good solutions, but cannot exit a local optimum. Metaheuristics Metaheuristics are a class of algorithms that uses knowledge of the problem topology in order to move from one place in the search-space to another place in hopefully an intelligent manner. Common to them all are the fact that they use more or less local information to decide where to go next. This use of local information, while maybe carefully devised, and often effective, can in general not guarantee that we find the best solution to our problem. In fact we can only hope that we are in the vicinity of a good solution.

21 formulation Regularization Motivation for metaheuristics Running time of exact algorithms Exact algorithms are available for the Feature Selection, but running time is exponential. Local search Local search is an easy way, to obtain relatively good solutions, but cannot exit a local optimum. Metaheuristics Metaheuristics are a class of algorithms that uses knowledge of the problem topology in order to move from one place in the search-space to another place in hopefully an intelligent manner. Common to them all are the fact that they use more or less local information to decide where to go next. This use of local information, while maybe carefully devised, and often effective, can in general not guarantee that we find the best solution to our problem. In fact we can only hope that we are in the vicinity of a good solution.

22 formulation Regularization Motivation for metaheuristics Running time of exact algorithms Exact algorithms are available for the Feature Selection, but running time is exponential. Local search Local search is an easy way, to obtain relatively good solutions, but cannot exit a local optimum. Metaheuristics Metaheuristics are a class of algorithms that uses knowledge of the problem topology in order to move from one place in the search-space to another place in hopefully an intelligent manner. Common to them all are the fact that they use more or less local information to decide where to go next. This use of local information, while maybe carefully devised, and often effective, can in general not guarantee that we find the best solution to our problem. In fact we can only hope that we are in the vicinity of a good solution.

23 Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search Floating searches Floating searches are a collection of essentially greedy construction heuristics. Genetic (GA) Genetic Algorithm a population based metaheuristic. Based on evolution theory. Simulated Annealing (SA) Simulated Annealing is perhaps the best known metaheuristic. Based on the notion of cooling an alloy to minimize tension.

24 Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search Floating searches Floating searches are a collection of essentially greedy construction heuristics. Genetic (GA) Genetic Algorithm a population based metaheuristic. Based on evolution theory. Simulated Annealing (SA) Simulated Annealing is perhaps the best known metaheuristic. Based on the notion of cooling an alloy to minimize tension.

25 Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search Floating searches Floating searches are a collection of essentially greedy construction heuristics. Genetic (GA) Genetic Algorithm a population based metaheuristic. Based on evolution theory. Simulated Annealing (SA) Simulated Annealing is perhaps the best known metaheuristic. Based on the notion of cooling an alloy to minimize tension.

26 Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search Huff-Puff Based on path relinking, is designed to increase the diversity compared to GA. Variable Neigborhood Search (VNS) Variable Neigbhorhood Search is based on the fact that local optimum are not necessarily local optimum in all neighborhoods. A global optimum is however a global optimum in every neighborhood. Roaming Search Roaming search is based on VNS, it is designed to give added diversity compared to VNS.

27 Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search Huff-Puff Based on path relinking, is designed to increase the diversity compared to GA. Variable Neigborhood Search (VNS) Variable Neigbhorhood Search is based on the fact that local optimum are not necessarily local optimum in all neighborhoods. A global optimum is however a global optimum in every neighborhood. Roaming Search Roaming search is based on VNS, it is designed to give added diversity compared to VNS.

28 Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search Huff-Puff Based on path relinking, is designed to increase the diversity compared to GA. Variable Neigborhood Search (VNS) Variable Neigbhorhood Search is based on the fact that local optimum are not necessarily local optimum in all neighborhoods. A global optimum is however a global optimum in every neighborhood. Roaming Search Roaming search is based on VNS, it is designed to give added diversity compared to VNS.

29 Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search Define the projection and signum function: 0 k 0 1 x < 0 P(k) = k 0 k N sgn(x) = 0 x = 0 N k N 1 x > 0 we may specify a move probability with respect to the guidance function g(k) p(θ t, k, r) = 1 + sgn( g(k)) ( ) r/t g(p(k+1))+g(p(k 1)) 2 2

30 Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search Define the projection and signum function: 0 k 0 1 x < 0 P(k) = k 0 k N sgn(x) = 0 x = 0 N k N 1 x > 0 we may specify a move probability with respect to the guidance function g(k) p(θ t, k, r) = 1 + sgn( g(k)) ( ) r/t g(p(k+1))+g(p(k 1)) 2 2

31 Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search RoamingSA RoamingSA uses a guidance function to guide the search a particular number of features. Defined as the low probability parts of the guidance function. Guided RoamingSA(GRSA) Guided Roaming SA is a hyper heuristic that repeatedly call RoamingSA, but as time passes by, alters the guidance function as more data is collected.

32 Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search RoamingSA RoamingSA uses a guidance function to guide the search a particular number of features. Defined as the low probability parts of the guidance function. Guided RoamingSA(GRSA) Guided Roaming SA is a hyper heuristic that repeatedly call RoamingSA, but as time passes by, alters the guidance function as more data is collected.

33 Iterated Locals Search (ILS) Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search Motivation When local searches get stuck in a local optimum, an iterated local search uses small perturbations in the search space to avoid a local baisin of attraction. If perturbation is too small, risks returning to the same baisin. Tabulist Keep a list of the most recently visisted local optimums, if we get a certain number of revisits, increase perturbation.

34 Iterated Locals Search (ILS) Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search Motivation When local searches get stuck in a local optimum, an iterated local search uses small perturbations in the search space to avoid a local baisin of attraction. If perturbation is too small, risks returning to the same baisin. Tabulist Keep a list of the most recently visisted local optimums, if we get a certain number of revisits, increase perturbation.

35 Iterated Locals Search (ILS) Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search Motivation When local searches get stuck in a local optimum, an iterated local search uses small perturbations in the search space to avoid a local baisin of attraction. If perturbation is too small, risks returning to the same baisin. Tabulist Keep a list of the most recently visisted local optimums, if we get a certain number of revisits, increase perturbation.

36 No regularization, CV Introduction Bar plots Convergence Discussion Regularized, CV No regularization, test Regularized, test

37 Bar plots Convergence Discussion Figure: Selected runs of 3 algorithms, to demonstrate different convergence functions. The runs are for the Rosis set, without regularization.

38 Bar plots Convergence Discussion Inconsistency between crossvalidation and test It is difficult to conclude with a single best methdod. Since we see a large discrepancy between test and CV. However deviance is low, which suggests underlaying effects. Best methods GRSA, and the add-l-rem-r method do well in all tests. The ILS is very good on CV, but seems to overadapt.

39 Bar plots Convergence Discussion Inconsistency between crossvalidation and test It is difficult to conclude with a single best methdod. Since we see a large discrepancy between test and CV. However deviance is low, which suggests underlaying effects. Best methods GRSA, and the add-l-rem-r method do well in all tests. The ILS is very good on CV, but seems to overadapt.

40 Summary I have presented 2 new methods for the FSP I have presented a number of adapted methods, from other scientific fields. The visualization algorithm will be shown to you now.

41 Summary I have presented 2 new methods for the FSP I have presented a number of adapted methods, from other scientific fields. The visualization algorithm will be shown to you now.

42 Summary I have presented 2 new methods for the FSP I have presented a number of adapted methods, from other scientific fields. The visualization algorithm will be shown to you now.