From Aug 2025 - May 2026 Zasha Benites helped get this working. This is the end of the project state from when she left:
ENVIRONMENT INFORMATION
Python virtual machine in Zasha's home directory -
Using python 3.12.5 - There is a py_12_5 folder in Quartz/memLab_deepLabCut/VAME_copy
Files are on Quartz/memLab_deepLabCut/VAME_copy
NEW_VAME/ - videos from the original paper describing VAME
NEW_VAME/results - replication of original paper
NEW_VAME/experiment_1/results/* videos folders
e.g. NEW_VAME/experiment_1/results/91_mouse_2/VAME/
contains hmm-20, kmeans-20 (20 is number of cluster) - These contains examples of video snips from each of the clusters
lab_videos/videos* - 8-arm maze videos from lab she analyzed
lab_videos/experiment_logs - files for each of the 'experiments' that reports what each was used and the performance in the cases when there was no error
lab_videos/Logs - 'tensor word files' that contain reports of how the training is going
lab_videos/notebooks - python scripts for plotting and evaluating performance
lab_videos/Utils - python scripts for moving files
experiment_XX folders are different sets of parameters created 'pipeline_experimenting_with_vame.py'
This used parameters in 'instructions.csv' contained in the same folder
'slurms/new_parallel_slurm.sh' - runs the instructions.csv file
Evaluating performance
"graphs are not indicative of the performance" - reconstruction and prediction graphs were not predictive of motif video qualities
Her answer to "what do you look for in the videos to evaluate performance?" - Clear boundaries between different behaviors. Consistencies across examples of a behavior (i.e., look similar). 'One behavior for each motif' She also looked at the 'communities it generated'
Communities - NEW_VAME/experiment_1/results/91_mouse_2/VAME/hmm_20/community-videos
RE: Replicating the main paper
She thought that it was possible to replicate the main results of the paper using their videos but that the actual performance was very mixed. Zasha says - The weakness of the paper is that benchmarks are just losses and not the ground truth of the videos contents.
She thinks that there is a weakness in how they tried to solve the problem. She said that the current method is just trying to represent the sequence and that this is not necessarily the best way to separate behaviors. The core issue is that the 'loss function' is not well aligned with the overall objective to separate behaviors. The embedding formed from this loss function is thus not useful for identifying behaviors.
RE: Our lab videos
experiment_33_good may have been the best performing
To get ours working, it had very little information because it only had a few points on the head and one on the tail. Once converted to egocentric coordinates, this had very little information. The overhead camera angle didn't provide rich data.