Introduction to Machine Learning 67577 - Fall, 2008 Amnon Shashua School of Computer Science and Engineering The Hebrew University of Jerusalem Jerusalem, Israel ar X iv :0 90 4. 36 64 v1 [ cs .L G ] 2 3 A pr 2 00 9 Contents 1 Bayesian Decision Theory page 1 1.1 Independence Constraints 5 1.1.1 Example: Coin Toss 7 1.1.2 Example: Gaussian Density Estimation 7 1.2 Incremental Bayes Classifier 9 1.3 Bayes Classifier for 2-class Normal Distributions 10 2 Maximum Likelihood/ Maximum Entropy Duality 12 2.1 ML and Empirical Distribution 12 2.2 Relative Entropy 14 2.3 Maximum Entropy and Duality ML/MaxEnt 15 3 EM Algorithm: ML over Mixture of Distributions 19 3.1 The EM Algorithm: General 21 3.2 EM with i.i.d. Data 24 3.3 Back to the Coins Example 24 3.4 Gaussian Mixture 26 3.5 Application Examples 27 3.5.1 Gaussian Mixture and Clustering 27 3.5.2 Multinomial Mixture and ”bag of words” Application 27 4 Support Vector Machines and Kernel Fu
