Video Visualization: My Journey so far

With my research interest in volume graphics and visualization in the 1990s, naturally I started to speculate the possibility of rendering a video, which is also in the form of volumetric data, into a single visualization image. During the conference dinner of EGUK 2000 at Swansea, As the local chair, I sat next to the keynote speaker Andrew Glassner, a leading computer graphics guru at Microsoft that time. I talked about my ambition to visualize videos. Andrew, who worked on spacetime (4D) ray tracing for animation, positively encouraged this idea.

Sometime in 2001, Rhodri Williams, a leading expert of rheology at Swansea, showed me a few exciting research experiments in his laboratory. Several such experiments involved the use of high-speed cameras to record the behaviour of non-Newtonian fluid. It usually took a postdoctoral a week to watch a video, take some measurement frame-by-frame, transcript the data to a spreadsheet, and plot the data as time series curves. Rhodri kindly gave me two videos. I converted them to a volume data format, and fed the data into my volume ray-casting software. A few minutes later, two volume-rendered images showed two distinctive curves. I took the visualization images to Rhodri and suggested some possible research directions.

A year later, another opportunity arose. Gareth Daniel, a very talented MPhil student, finally decided to stay on to do a PhD at Swansea. As he worked on video surveillance for his MPhil, he was immediately attracted by the idea of video visualization. Gareth and two of his friends and fellow PhD students, David Clark and Justin Biddle, together with two staff members Mark Kiddell and Emma Jeffery, made some playful videos in the context of video surveillance. Gareth and I tried many change-detection filters and modified my existing volume renderer to produce the horseshoe visual layout. These led to the first video visualization paper in IEEE Vis2003 [1].

Building on the Vis2003 paper, Gareth quickly finished his PhD and went to the industry. Meanwhile, there were a few new directions emerged from the paper. One idea was to visualize optical flow in the video, which was suggested by Deborah Silver (Rutgers), a leading expert of time-varying volume visualization and volume animation. Deborah and I collaborated on volume deformation for a number of years, and she often offered ingenious ideas and nifty tips like a magician pulling a rabbit out of the hat. A new PhD student at Swansea was assigned to work on this idea, but did not manage to make adequate progress after one year or more.

One day Tom Ertl and I were talking about video visualization. He invited me to spend some time with his team in Stuttgart. Royal Society provided me with a travel grant after turning down my initial application for having a post-doctoral researcher working on the problem. With the travel grant, I spent 3 months in Stuttgart in the summer of 2005. Tom had been leading one of the largest and most productive visualization teams in the world. I met many top-quality researchers there, including Daniel Weiskopf, who was about to take up a faculty position in Simon Fraser. Tom asked Ralf Botchen to work with me closely. Although Ralf was still at the early stage of his PhD, his ability surpassed many third-year PhD students. Within a month, Ralf delivered a GPU implementation of the horseshoe volume renderer, while I implemented the pre-processing code for creating the optical flow field in addition to the difference volume resulting from change detection. Daniel suggested to use animated glyphs for depicting the flow field. Together we designed three new visual representations in addition to the basic one in [1]. By the end of my three-month visit, we had all technical implementation completed, and were ready for an empirical study to be done when I returned to Swansea. I was totally overwhelmed by the intellectual capability and teamwork ethos of the Stuttgart team.

At Swansea, my PhD student Rudy Hashim, Ian Thornton (a psychology professor), and I conducted an empirical study, which confirmed that humans could recognise visual signatures of different motions using video visualization, and could learn and retain such skills easily. We wrapped the work done in Stuttgart and at Swansea together into a paper and submitted it to a TVCG special issue on Visual Analytics. It was rejected as some reviewers considered that this was not within the scope of visual analytics. We then submitted the GPU part of the work to Volume Graphics 2006 [3], and the rest of the work to Vis 2006 [4]. This work brought me much confidence that video visualization should be able to fill in many technological gaps, for which automated computer vision could not deliver adequate solutions currently, and in many cases for the foreseeable future.

However, the papers in Vis 2003 and Vis 2006 did not translate immediately to any funded project where some young PhDs or postdoctoral researchers could continue to take this new adventure forward. Typically reviewers of my grant proposals commented that this was really a computer vision problem and surely visualization could not detect those events which computer vision could not. Interestingly, in the next decade, I was asked frequently asked by potential users of visualization about how visualization would detect this or that. These questions forced me to ponder on what visualization was really for, leading to my meddling with theories of visualization.

One day, Ralf Botchen contacted me about his ongoing effort in advising some MSc projects on video visualization. At the same time, Daniel Weiskopf started to work with his Simon Fraser colleague, Greg Mori, on the interface between video visualization and computer vision. In an email, Ralf explained that he was looking for something that his MSc students could realistically do. This was followed by a long telephone call between us. I described a visual design in one of my failed grant proposal, which would capture the "continuous" and "endless" notion of electrocardiograms (ECGs) and seismographs. I suggested that it would be wonderful if we could name it as something -gram or -graph. Within a short period, the most talented Ralf not only delivered a whole implementation with two MSc students, but also came up a name, VideoPerpetuoGram (VPG), for the visual design [7]. We also filed a patent for the new technique [8].

2009 and 2010 finally saw some research funding for video visualization arrived at Swansea from several sources. These include an EPSRC project, a Welsh Assembly Government A4B project, a work package in the RIVIC project, and two relatively small project grants. These projects explored a number of new applications, including sports videos (e.g., match videos in Rugby and training videos in Snooker) [11,13, 15, 16, 18, 20, 22, 25, 26], images sequences (e.g., satellite images in glaciology) [9], sound track in videos [10], facial expression videos [14, 19], traffic videos [23], and biomedical videos [21, 24]. There were a number of memorable and fruitful collaborations with many domain experts, including Iwan Griffiths (Swansea), Adrian Moris (Swansea), Wyn Griffiths (Terry Griffiths Matchroom), and Rhys Long (WRU) in sports science, Tavi Murray (Swansea) and her team in glaciology, Paul Rosin (Cardiff), Dave Marshall (Cardiff), and Nei Mac Parthaláin (Aberystwyth) in computer vision and machine learning, Eamonn Gaffney (Oxford) in bioinformatics and his collaborators at at Birmingham Women’s Fertility Centre, and Cameron Holloway (Oxford) in medicine. Other academic colleagues who also made contribution to the research include David Ebert (Purdue), Daniel Weiskopf (Stuggart), Phil Grant (Swansea), Bob Laramee (Swansea), Mark Jones (Swansea), and Ian Thornton (Swansea). Several research officers who were fully or partly funded by these projects are now faculty members, including Rita Borgo (Swansea and King's College London), Heike Leitte (Heidelberg and Kaiserslautern), Phil Legg (UWE, Bristol), Gary Tam (Swansea), Hui Fang (Edge Hill), and Jeyan Thiyagalingam (Liverpool), In fact, those researchers who were funded by these projects and then chose to have a career in the industry are also brilliant in their technical ability, including Simon Walton, David Chung, Matthew Perry, and Brian Duffy.

In comparison to the investment in automated techniques for video processing, analysis, and understanding, the investment in video visualization technology has been tiny. This is perhaps partly because the overoptimistic anticipation over the last a few decades that AI would soon deliver automated computer vision, and partly because the scientific community did not understand the real values of visualization or how it works as much as many in the field of visualization do now (for the latter, see also Theories of Visualization).


  1. G.W. Daniel and M. Chen. Video visualization, Proc. IEEE Visualization 2003, 409-416, Seattle, WA, October 2003. PDF(10M).

  2. G.W. Daniel and M. Chen. Visualising video sequences using direct volume rendering, Proc. the 1st International Conference on Vision, Video and Graphics (VVG2003), 103-110, Eurographics/ACM Workshop Series, Bath, July 2003.

  3. R. P. Botchen, M. Chen, D. Weiskopf and T. Ertl. GPU-based multi-field video volume visualization, Proc. Volume Graphics, 47-54, Boston, MA, July, 2006.

  4. M. Chen, R.P. Botchen, R.R. Hashim, D. Weiskopf, T. Ertl and I.M. Thornton. Visual signatures in video visualization, IEEE Transactions on Visualization and Computer Graphics, 12(5):1093-1100, 2006. (Presented in IEEE Visualization 2006.) DOI.

  5. M. Chen, J. Robinson and D. Silver. Illuminating the path of video visualization in the shadow of video processing. Invited survey in Proc. International Symposium of Multimedia, 219-226, San Diego, CA, 2006.

  6. C. Correa, D. Silver and M. Chen. Illustrative deformation for data exploration, IEEE Transactions on Visualization and Computer Graphics, 13(6):1320-1327, 2007. (Presented in IEEE Visualization 2007.) DOI.

  7. R. P. Botchen, S. Bachthaler, F. Schick, M. Chen, G. Mori, D. Weiskopf and T. Ertl. Action-based multi-field video visualization, IEEE Transactions on Visualization and Computer Graphics, 14(4):885-899, 2008. DOI.

  8. R. Botchen, D. Weiskopf, T. Ertl, and M. Chen. Video Data Processing, Patent: Europe EP2112619, US 20090278937 A1, 2009.

  9. R. Borgo, K. Proctor, M. Chen, H. Jaenicke, T. Murray and I. M. Thornton, Evaluating the impact of task demands and block resolution on the effectiveness of pixel-based visualization, IEEE Transactions on Visualization and Computer Graphics, 16(6):963-972, 2010. (Presented in IEEE VisWeek 2010.) DOI.

  10. H. Jaenicke, R. Borgo, J. S. D. Mason and M. Chen. SoundRiver: semantically-rich sound illustration. Computer Graphics Forum, 29(2):357-366, 2010. (Presented in EG 2010.) DOI.

  11. M. Hoeferlin, E. Grundy, R. Borgo, D. Weiskopf, M. Chen, I. W. Griffiths and W. Griffiths. Video visualization for snooker skill training, Computer Graphics Forum, 29(3):1053-1062, 2010. (Presented in EuroVis 2010.) DOI.

  12. R. Borgo., M. Chen, B. Daubney, E. Grundy, H. Jaenicke, G. Heidemann, B. Hoeferlin, M. Hoeferlin, D. Weiskopf and X. Xie. A survey on video-based graphics and video visualization. Eurographics 2011 STAR, 2011.

  13. P. A. Legg, M. L. Parry, D. H. S. Chung, R. M. Jiang, A. Morris, I. W. Griffiths, D. Marshall and M. Chen. Intelligent filtering by semantic importance for single-view 3D reconstruction from snooker video. Proc. IEEE International Conference on Image Processing (ICIP), 2433-2436, 2011.

  14. G. K. L. Tam, H. Fang, A. J. Aubrey, P. W. Grant, P. L. Rosin, D. Marshall and M. Chen. Visualization of time-series data in parameter space for understanding facial dynamics, Computer Graphics Forum, 30(3):901-910, 2011. (Presented in EuroVis 2011.) DOI.

  15. M. L. Parry, P. A. Legg, D. H. S. Chung, I. W. Griffiths, M. Chen. Hierarchical event selection for video storyboards with a case study on snooker video visualization, IEEE Transactions on Visualization and Computer Graphics, 17(12):1747-1756, 2011. (Presented in IEEE VisWeek 2011.) DOI.

  16. P. Legg, D. Chung, M. Parry, M. Jones, R. Long, I. Griffiths and M. Chen. MatchPad: interactive glyph-Based visualization for real-time sports performance analysis, Computer Graphics Forum, 31(3):1255-1264, 2012. (Presented in EuroVis 2012.) DOI.

  17. R. Borgo., M. Chen, B. Daubney, E. Grundy, H. Jaenicke, G. Heidemann, B. Hoeferlin, M. Hoeferlin, D. Weiskopf and X. Xie. State of the art report on video-based graphics and video visualization, Computer Graphics Forum, 31(8):2450-2477, 2012. DOI.

  18. P. A. Legg, D. H. S. Chung, M. L. Parry, R. Bown, M. W. Jones, I. W. Griffiths, M. Chen. Transformation of an uncertain video search pipeline to a sketch-based visual analytics loop. IEEE Transactions on Visualization and Computer Graphics, 19(12):2109-2118, 2013. (Presented in IEEE VIS 2013.) DOI.

  19. H. Fang, N. Mac Parthaláin, A. J. Aubrey, G. K. L. Tam, R. Borgo, P. L. Rosin, P. W. Grant, D. Marshall, and M. Chen. Facial expression recognition in dynamic sequences: An integrated approach. Pattern Recognition, Elsevier, 47(3):1271-1281, 2014. DOI.

  20. D. H.S. Chung, M. L. Parry, P. A. Legg, I. W. Griffiths, R. S. Laramee, and M. Chen. Visualizing multiple error-sensitivity fields for single camera positioning. Computing and Visualization in Science, Springer, 15(6):303-317. DOI.

  21. S. Walton., K. Berger, J. Thiyagalingam, B. Duffy, H. Fang, C. Holloway, A. E. Trefethen, and M. Chen, Visualizing cardiovascular magnetic resonance (CMR) imagery: Challenges and opportunities. Progress in Biophysics and Molecular Biology, Elsevier, 115(2-3):349-358, 2014. DOI.

  22. D. H. S. Chung, I. W. Griffiths, P. A. Legg, M. L. Parry, A. Morris, M. Chen, W. Griffiths, and A. Thomas. Systematic snooker skills test to analyze player performance. International Journal of Sports Science and Coaching, 9(5):1083-1106, 2014. DOI.

  23. S. Walton, K. Berger, D. Ebert and M. Chen. Vehicle object retargeting from dynamic traffic videos for real-time visualisation, The Visual Computer, Springer, 30(5):493-505, 2014. DOI.

  24. B. Duffy, J. Thiyagalingam, S. Walton, D. J. Smith, A. Trefethen, J. C. Kirkman-Brown, E. A. Gaffney and. M. Chen. Glyph-based video visualization for semen analysis. IEEE Transactions on Visualization and Computer Graphics, 21(8):980-993, 2015. (Presented in IEEE VIS 2014.) DOI.

  25. D. H. S. Chung, P. A. Legg, M. L. Parry, R. Bown, I. W. Griffiths, R. S. Laramee and M. Chen. Glyph sorting: interactive visualization for multi-dimensional Data. Information Visualization, Sage Journal, 14(1):76-90, 2015. DOI.

  26. D. H. S. Chung, M. L. Parry, I. W. Griffiths, and R. S. Laramee, R. Bown, P. A. Legg and M. Chen. Knowledge-assisted ranking: A visual analytic application for sport event data. IEEE Computer Graphics and Applications, 36(3):72-82, 2016. (Presented in IEEE VIS 2015.) DOI, Video.