R, OpenPoseR, & PURRR

Comparing DeepLabCut and OpenPose Performance

• DeepLabCut outputs a .csv file with the 2D coordinates of each point of interest in each video frame and a confidence score for each coordinate estimated.

• OpenPose outputs one .JSON file for each video frame with its XY coordinates for each point of interest along with its confidence in the estimated coordinates. These .JSON files are saved to a folder names for the video being analyzed.

• To compare the relative accuracy of DeepLabCut and OpenPose, their outputs have to be standardized.

• OpenPoseR (Trettenbrein et al., 2021)  is an R package that reduces the multiple .JSON files produced by OpenPose and saves them to a single .csv file.

• OpenPoseR was developed for sign language, hence it assumes a single subject, though OpenPose itself is designed to identify points of interest in multiple subjects.

• Github resources:  https://github.com/trettenbrein/OpenPoseR

• If your work involves a comparison of DLC and OpenPose, you will need to get their output into the same format. OpenPoseR is good for this, but requires modification.

• Modification is needed because there are often errors in writing the .JSON file syntax from OpenPose. Errors need to be "caught" otherwise the conversion process aborts.

R, Rstudio

• To run OpenPoseR code, you will need to install R and Rstudio.

• R is freely available  from the R Project.  Tutorials and documentation are likewise available here.

• R Studio is also free to download and install.  Tutorials and documentation are also available at this link.

Error Handling - Manual Work-Around

• This is a work in progress.  This code represents the current strategy.  It is modified from the OpenPoseR package and conceived to be stepped through, running each code segment separately (R-typical). 

• The code creates a dataframe for a very short video. 

• Because the video is short (i.e.. hundreds versus thousands of frames) when a .JSON file is corrupt, the strategy is to manually remove it from the processing queue and save it to an error folder to keep track of where in the sequence the error occurred. 

• Then, the process is restarted and goes forward until the next corrupt .JSON file is discovered, or to the end of the queue.

• Once the dataframe is complete, the process to convert it to a .csv file is invoked. This .csv file can then be manipulated (in Excel) to compare it with the .csv file generated by DeepLabCut for estimating poses in the same video.

I am exploring the PURRR package to do the error handling versus the manual approach. 

PURRR

• Purrr is part of the functional programming movement in R. 

• Part of the innovation is its ability to avoid for loops, which create complexity and opportunity for greater error.

• The "possibly" and "safely" functions within the Purrr package are probably the way to go to handle the errors in OpenPose .JSON files that currently thwart OpenPoseR processing, .JSON to .csv files.

Tutorial: Bruno Rodrigues