Survey of Kolmogorov Complexity and its Applications

Transcription

1 Survey of Kolmogorov Complexity and its Applications Andrew Berni University of Illinois at Chicago 1 Abstract In this paper, a survey of Kolmogorov complexity is reported. The basics are briefly covered to give the reader an introductory understanding of the topic. This includes necessary definitions followed by some interesting properties and theorems. Following the brief introduction to the topic is some current research in the area, mainly focusing on the application of Kolmogorov complexity to quantum states and quantum computers. We finish with a discussion of current applications of Kolmogorov complexity to highlight how it can be used to solve practical problems. I. INTRODUCTION Kolmogorov complexity is a branch of information theory that deals with the complexity contained in a single object or string. It is another measure of randomness, as opposed to entropy, and it is not based on probability, but instead considers the method or algorithm used to compute the string. Not being tied to a probability distribution allows the measurement of randomness in a single object. The notion was developed independently by Andrey Kolmogorov in 1965 [3], Gregory Chaitin in 1965, and Ray Solomonoff [9]. In this paper, the definition of Kolmogorov complexity and many of its important properties will be examined. This will shed some light on some interesting features, as well as show some analogs to other areas of information theory. Keep in mind that only a small subset of the theory is presented. Just enough to show the reader some interesting properties and to give a short introduction. Newer developments in the field will be discussed, mainly in the area of quantum Kolmogorov complexity. This has become a topic of interest due to the rising popularity of the quantum computer. Finally, we will look at several applications of Kolmogorov complexity to get an idea of its practicality. We will see how Kolmogorov complexity can be used to solve a wide variety of problems. Its unique properties can be exploited to use it as a tool in problem solving. II. DEFINITION AND EXAMPLES Kolmogorov complexity came about as a way to describe the randomness in an object based on the length of the computer program used to describe the object [1]. This object could be a string, which is generated by a computer program. In general, that is what will usually be considered in definitions and proofs. It is different from Shannon s way of measuring randomness (entropy) since it does not rely on probability at all [1]. The mathematical definition of Kolmogorov complexity is as follow: K U (x) = min p:u(p)=x l(p)

2 where we find the minimum the length of computer program p over all programs p, such that computer U with input p produces the output string x. It is shown in [1] that we cannot find the exact value of K(x) in practice, but we can upper bound it. We can find these upper bounds by giving examples of programs that will produce the string x. The shorter the program that is found, the closer the length of that program is to the actual Kolmogorov complexity. A few examples to get the idea of the bounding are as follows: 1.The sequence 12a12a12a12a12a12a12a12a12a12a12a12a12a12a12a12a12a12a12a12a 12a12a12a12a12a12a12a12a12a12a has fairly low Kolmogorov complexity. It can be described as print out the 3 character string 12a 30 times. Although the string is 90 characters long, its Kolmogorov complexity is 45 characters. We could conceivably make a string thousands or millions of characters long, which we can describe with a relatively short program. As we can see in the string, its redundant structure allows this compression. Looking at this from an entropy point of view, we would see this string as not very random as well. 2.The sequence 3vo1i3z9qm09ij3j4j6ncb1mfjh3890vg56n5md9s19va3 has a higher Kolmogorov complexity. It probably cannot be described as simply as the previous example. The shortest computer program to print this string would most likely look like print out the 46 character string 3vo1i3z9qm09ij3j4j6ncb1mfjh3890vg56n5md9s19va3. We need to know how many characters to print so that we do not extend beyond the end of the program when executing it. The program to print the string is just slightly longer than the string itself! Again, if we were to look at this in a probabilistic perspective, we would say that this string is pretty random. We could probably find shorter programs for to generate the examples above. The example programs given just represent upper bounds. If we found a shorter program to produce the output, then that would tighten the upper bound. 2 III. INTERESTING THEOREMS AND PROPERTIES A. Upper Bound on Kolmogorov Complexity One interesting theorem that is that the conditional Kolmogorov complexity (given the length of the string) is less than the length of the string plus an additive constant. We can almost deduce this by looking at how the string was described in the second example above. Since we are given the length of the string from the conditioning, we do not have to consider this in the Kolmogorov complexity. We only need to consider the string given, plus the length of the rest of the program, which is constant. If we want the unconditional Kolmogorov complexity, then a term is added to account for the length of the integer needed to describe how long the sequence is supposed to be. This term can be represented in 2*log(l(x)) bits if we are describing a binary string using a binary program [1]. This term which represents the length of the string can be more tightly bounded, but that will not be discussed here. B. Lower Bound on Kolmogorov Complexity It is also interesting to look at the lower bound of Kolmogorov complexity. This lower bound is stated as follows: The number of strings x with complexity K(x) < k satisfies

3 3 {x {0, 1} : K(x) < k} < 2 k This is proven by listing all binary programs with length less than k. This results in 1 null program, 2 1-bit programs, 4 2-bit programs, 8 16-bit programs, 2 k 1 (k-1)-bit programs. Counting these programs results in 2 k 1 possible programs, which is less than 2 k. In other words, there are not many sequences with low Kolmogorov complexity. Most sequences are not simple [1]. C. Computer Independence of Kolmogorov Complexity It can also be shown that the Kolmogorov complexity is computer independent except for an additive constant, which becomes negligible for high complexity sequences [1]. The notion of a universal computer is used here. Before diving too much into this explanation, we first need to consider how a machine can compute algorithms. A useful tool for analyzing computability is the Turing machine. A Turing machine is a theoretical device that operates on an infinite length tape that is fed into the machine. The machine has a tape head which can read an instruction from the tape, change states, and possibly write something to a work tape [12]. Further work has shown that a Turing machine can simulate any other computer, and a Turing machine could be simulated by any other computer [1]. This idea is used to prove the following theorem. Suppose we have a universal computer U that can simulate another computer. The idea is that the universal computer can mimic the behavior of another computer by running a fixed length program instructing the universal computer how to translate instructions for the other computer. Now, given K(x) for an arbitrary computer A, K U (x), the complexity of x for the universal computer U, is equal to K A (x), plus the length of the translation program. This shows that the Kolmogorov complexity is computer independent. As K A (x) rises, this constant becomes negligible, and the Kolmogorov complexity is the same no matter which computer outputs string x. D. Relationship Between K(x) and H(x) Another interesting property is the relationship between Kolmogorov complexity and entropy H(x) [1]. Which leads to: H(X) 1 f(x n )K(x n n) H(X) + n x n E 1 n K(Xn n) H(X) ( X 1) log n n This relationship shows that Kolmogorov complexity is closely related to Entropy, even though they are formed from very different foundations. E. Kolmogorov Complexity and Occam s Razor Next, we look at how we can apply Kolmogorov complexity as a formalization of Occam s razor [1]. Occam s razor states that The simplest explanation is best [11]. Kolmogorov complexity aims to find the minimum length program that produces the string x. This minimum length program is essentially the simplest program. Using Occam s razor as a guideline, we can conclude that this program is the best explanation. It seems to be a logical and natural result. + c n

4 4 A. Algorithmic Test for Randomness IV. FURTHER DEVELOPMENTS Martin Lof extended the theory of Kolmogorov complexity by defining algorithmically random sequences [7]. Before his work, there were only vague definitions of algorithmically randomness sequences. Lof introduced specific tests to find if a sequence is truly algorithmically random. B. Recent Developments In more recent developments in the field, Quantum Kolmogorov complexity is explored. Vitanyi explores this idea in [13]. The theory of Quantum Kolmogorov complexity describes the amount of information contained in a pure quantum state. Vitanyi shows that Quantum Kolmogorov complexity has analogs to classical Kolmogorov complexity. One of these analogs is that quantum Kolmogorov complexity is upper bounded and can be approximated from above. Vitanyi further develops the theory of Quantum Kolmogorov complexity in [14]. He explains that there is increasing interest in the subject due to the rising interest in the quantum computer. Vitayni also explains the idea of quantum Turing machines in this paper. He explains how it is based on probabilistic computation. The quantum Turing machine is useful when considering how a machine would operate on quantum states. He then goes on to explain quantum algorithms and how they apply to this theory. In 2008, this idea of quantum Turing machines is explored further by Muller. Muller shows in [8] the existence of a universal quantum Turing machine, which can simulate every other quantum Turing machine. Using this result, he shows that quantum Kolmogorov complexity is independent of the machine considered when computing quantum Kolmogorov complexity, up to an additive constant. This is analogous to the same property of classical Kolmogorov complexity. A. Information Assurance V. APPLICATIONS Kolmogorov complexity has some interesting applications. One such application is using Kolmogorov complexity to detect abnormal behavior in an information system. The paper [2] introduces this idea. The author describes how information security issues are usually dealt with after the fact that security has been breached. They propose instead to deal with a security breach as it happens. Other systems are presented as examples where abnormalities are detected and corrected before any damage can be caused. One example is a thermodynamic system monitoring and correcting pressure. A breach in pressure can be detected and isolated to a small portion of the entire system. In order to deal with a breach as it is happening, the information in the system needs to be somehow verified. One suggestion of how to monitor an information system is shown in the figure below. The author explains that a lower complexity of K(X,Y) will make it easier for an attacker to understand what the system is doing (since its inputs and outputs are not very complex). The other metric to monitor is K(Y X). If the system adds complexity to X to produce Y, then the system will be less vulnerable. If the system subtracts complexity to X to produce Y, then the

5 5 Fig. 1. system will be more vulnerable. Monitoring these two metrics, and classifying vulnerability based on the regions shown is a proposed method of using Kolmogorov complexity to monitor information in a system. This is just one example of several indicators presented in this paper. The paper concludes that that Kolmogorov complexity is a good candidate for further research in this area. B. Spam Filtering Another interesting application of Kolmogorov Complexity is using it to filter spam. [10] gives details of how this can be done. Using an adaptive filter and Kolmogorov complexity estimates, is checked to be spam or normal . They use estimates because, as explained earlier, the exact Kolmogorov complexity is impossible to compute. It can be bounded from above, so they use a compression algorithm to find out how much the string can be compressed, and use this as an upper bound on the Kolmogorov complexity [5]. A low Kolmogorov complexity indicates a higher probability of the message being spam. Statistical data is gathered showing the relation between Kolmogorov complexity and spam. An example is presented below in tabular form. In order to generate this estimate of the Kolmogorov complexity, a run-length compression algorithm is performed on the message. We can see that, for this particular database of example s, that most of the spam has very low Kolmogorov complexity. They show that this method can filter spam about twice as fast as normal methods based on Bayesian filters, which is a common implementation of a spam filter, while achieving an accuracy rate of 80% to 96%. They also find that their method is less likely to be fooled by a statistical attack, which is performed by adding words to the spam to make it look like normal . They project that in future work, better accuracy could be obtained by using different compression methods. This could possibly provide a better estimate of the actual Kolmogorov complexity

6 6 Fig. 2. of the message, which may lead to better performance. They also speculate on other possible applications of their method, including plaigairism detection by matching texts with their respective authors. C. Mental Fatigue The applications of Kolmogorov complexity seem to be limitless. Since Kolmogorov complexity operates on strings, it can really be applied to anything that can be described as a string. The next application introduced is using Kolmogorov complexity to assess mental fatigue. In [6], the author applies Kolmogorov complexity to EEG signals obtained from patients in order to measure the level of fatigue in that patient. In this case, electrical brain signals are measured and transformed into discrete sequences. The discrete sequence can then be analyzed using any choice of Kolmogorov complexity estimation methods. In this paper the method used is very different from the one used in the spam filtering application paper. The method used for the EEG signals was based on a method described in [4] in which the complexity of the object is estimated by the number of steps in the generating process. The goal of their work was solely to see if there was any connection between Kolmogorov complexity of the EEG, and mental fatigue. According to the conclusion of the paper, they achieved this goal, and found that there was a strong correlation. The value of Kolmogorov complexity on the EEG signal was found to decrease as mental fatigue increases. VI. CONCLUSION Kolmogorov complexity and a handful of its applications were surveyed in this paper. Kolmogorov complexity presents a different view of information and randomness than entropy methods of information theory. The properties of Kolmogorov complexity are interesting and at times unexpected. It is apparent that Kolmogorov complexity is an elegant and useful branch of information theory. New research is continuing, especially in the area of Quantum Kolmogorov complexity. The applications presented all took pieces of classical Kolmogorov complexity and put them to practical use. It will be interesting to see how quantum Kolmogorov complexity will be put to practical use in future works, as well as new applications of the classical Kolmogorov complexity. REFERENCES [1] T. A. Cover and J. A. Thomas. Elements of information theory. John Wiley and Sons, Inc, 2006.

7 [2] S. Evans, S.F. Bush, and J. Hershey. Information assurance through kolmogorov complexity. DISCEX 01. Proceedings, 2: , [3] A. N. Kolmogorov. Three approaches to the quantitative definition of information. Probl. Inf. Transm. (USSR), 1:4 7, [4] A. Lempel and J. Ziv. On the complexity of finite sequences. IEEE Trans. Inf. Theory, 22(1):75 81, [5] M. Li and P. Vitanyi. An introduction to kolmogorov complexity and its applications. 2nd ed. Springer-Verlag, New York, [6] Zhang Lian-yi and Zheng Chong-xun. Analysis of kolmogorov complexity in spontaneous eeg signal and it s application to assessment of mental fatigue. Bioinformatics and Biomedical Engineering, pages , [7] P. Martin-Lof. The definition of random sequences. Inf. Control, 9: , [8] M. Muller. Strongly universal quantum turing machines and invariance of kolmogorov complexity. Information Theory, IEEE Transactions on, 54(2): , [9] R. J. Solomonoff. A formal theory of inductive inference. Inf. Control, 7: , [10] L.M. Spracklin and L.V. Saxton. Filtering spam using kolmogorov complexity estimates. Advanced Information Networking and Applications Workshops, 1: , [11] S. C. Tornay. Ockham: Studies and selections (chapter commentarium in sententias, i, 27). Proceedings of the London Mathematical Society, [12] A. M. Turing. On computable numbers, with an application to the entscheidungsproblem. Proceedings of the London Mathematical Society, 2 42:230 65, [13] P. Vitanyi. Three approaches to the quantitative definition of information in an individual pure quantum state. 15th Annual IEEE Conference on Computational Complexity, pages , [14] Paul M. B. Vitanyi. Quantum kolmogorov complexity based on classical descriptions. IEEE Transactions on Information Theory, 47(6),