Activity Recognition within a Manufacturing System: A Comparison of Logic Programming, Machine Learning, and Combinatorial Optimization Based Methods

Abstract

A systematic investigation of machine learning and ``domain knowledge'' based Activity Recognition (AR) methods was undertaken for a manufacturing based used case involving the assembly of computers. Experimental data was generated by combining a stochastic, discrete event simulator of the computer assembly process with a generative model used to synthesize the trajectories of the simulated workers. These trajectories were used to classify a set of complex work activities associated with the assembly process. Sequence based neural networks, which are currently the foundation of state of the art AR methods, were compared to two proposed ``domain knowledge'' approaches that encode prior knowledge of work classes using symbolic probabilistic logic programming (PLP) or mixed integer linear programming (MILP) methods.Manufacturing production often involves highly structured tasks that are repetitive in nature. This was the case for the computer assembly process where work activity classes were represented as sequences of lower level actions (primitive actions) performed at particular factory locations. It was shown that a linear MILP classifier that represents this class knowledge outperformed non-linear sequence based neural networks. Moreover, whilst the PLP approach achieved inferior performance to the MILP and neural network classifiers, unlike standard logic programming, the PLP classifier was shown to be similarly robust to input noise and uncertainty as the other classifiers. Finally, flat AR approaches were shown to clearly outperform hierarchical AR approaches.  Unlike the flat, single stage classification, hierarchical classification introduced additional error by propagating noisy labels between different classification stages.

Keywords: Human Activity Recognition, Neural Networks, probabilistic logic programming, mixed integer linear programming, Manufacturing, work task classification