Treffer: Unsupervised encoding and extraction of video sequences using Beta-Variational autoencoders

This research addresses two significant challenges in sequential task understanding from video data: the requirement for extensive labelled data in supervised learning approaches and the need for specialised neural architectures to differentiate between highly similar states. This research proposes an unsupervised learning approach utilising a disentangled β-Variational Autoencoder (β-VAE) to identify and differentiate steps in sequential tasks without requiring labelled data. The approach was evaluated on both synthetic and real-world datasets of increasing complexity, demonstrating high accuracy in state identification (88-100%) using only eight training examples. The encoder network successfully reduced high-dimensional visual data to eightdimensional latent vectors, enabling effective comparison between current frames and expected task states (waypoints) extracted through unsupervised motion detection. Results show that the system can accurately identify task progression and detect deviations from expected sequences while maintaining robustness to moderate noise and environmental variations. This research contributes a practical solution for operator training and automated quality control in sequential tasks, significantly reducing the manual labelling burden while maintaining high accuracy in state identification and deviation detection. ; Thesis (MSc) -- Faculty of Science, School of Computer Science, Mathematics, Physics and Statistics, 2025