Publication

Online EM with weight-based forgetting

Journal Article (2015)

Journal

Neural Computation

Pages

1142–1157

Volume

27

Number

5

Doc link

http://dx.doi.org/10.1162/NECO_a_00723

File

Download the digital copy of the doc pdf document

Authors

Abstract

In the on-line version of the EM algorithm introduced by Sato and Ishii (2000), a time-dependent discount factor is introduced for forgetting the effect of the old posterior values obtained with an earlier, inaccurate estimator. In their approach, forgetting is uniformly applied to the estimators of each mixture component depending exclusively on time, irrespective of the weight attributed to each unit for the observed sample. This causes an excessive forgetting in the less frequently sampled regions. To address this problem we propose a modification of the algorithm that involves a weight-dependent forgetting, different for each mixture component, in which old observations are forgotten according to the actual weight of the new samples used to replace older values. A comparison of the time-dependent versus the weight-dependent approach shows that the last one improves the accuracy of the approximation and exhibits a much greater stability.

Categories

learning (artificial intelligence), stochastic programming.

Scientific reference

E. Celaya and A. Agostini. Online EM with weight-based forgetting. Neural Computation, 27(5): 1142–1157, 2015.