Introduction To Video

For some reason, I *love* video technicalities. I know that it's just the nuts and bolts behind what should be a creative medium, but it's the technicalities that intruige me.

Some of the basics of video files on computer are good to know for anyone fiddling with video files.

I haven't seen them clearly put in one place before. Hence this. The overview is really short, don't panic.

Playing A Video File

Most media players don't actually decode the file, they actually ask Windows to decode and display it! How it works is that there's part of the operating system called a media framework ("DirectShow", part of "DirectX" in Windows). The media player asks the framework to please give it some pixels and some audio samples. The framework looks at the file to identify if it has the right splitters, then looks at the "FourCC" (Four Character Code) of the video to see if it has the right codecs, then plugs together all the right parts to start decoding the video for you. You can use a tool called GraphEdit (start looking here) or any of several alternatives (like GraphStudio) to see what's going on.

Here's the DirectShow graph of my my computer playing a DVD, you can see:

• The DVD source splitting out the video, audio, and subtitles,

• The MPEG-2 decoder (part of a codec, short for "coder-decoder") decoding the video,

• An AC3 codec for the audio,

• An ffdshow codec writing the subtitles onto the video, and

• The video and audio renderers/devices which show the video and play the sound:

I said Windows before, but what about other platforms? In case you're curious:

• In Vista and Windows7, DirectShow is slowly being replaced with "Microsoft Media Foundation",

  • ...and it's the source of a lot of nuisance problems for all of the old DirectShow tools!

• On Macs, it's QuickTime (it's not just a video player, it's the whole media framework)

• In the Linux world, mostly GStreamer and Phonon,

• The popular VLC Media Player has/is its own media framework

  • Which makes it very practically useful as a tool, and a huge nuisance because it has allowed the spread of a lot of mangled video files which should have simply been put in a better format originally.

Well, that was it! A minimum amount of information on the concepts above. The next section is beyond the basic concepts, and covers the basic containers and codecs that you're likely to meet.

Bonus - Know Your Containers and Codecs

Popular Container Formats

There are a lot around: http://en.wikipedia.org/wiki/Comparison_of_container_formats, however, there are few which are way more popular than the others:

AVI

• AVI = Audio Video Interleave, meaning that there's a chunk of video, then a chunk of audio typically 500ms at a time (originally so that you would only need to buffer half a second of video/audio at a time since RAM used to be so expensive).

• The grand-daddy of video containers, around since 1992! It is super-reliable but slightly limited:

  • AVI files can hold some modern video formats like h264... but not according to the official spec so most software won't work with that.

  • AVI files can hold multiple audio streams (eg: two languages)... but a lot of software won't work with that. Also, these days, almost all AVI files have mp3 audio.

• AVI requires you to interleave / multiplex ("mux") the audio and video correctly. A badly muxed AVI file (eg: one where all the audio is at the start) will cause trouble for hardware devices which can't read ahead (see the section below) or computers without much RAM. This shouldn't be an issue, but there seems to be some software out there which doesn't mux correctly so it's a relatively common problem.

Matroska

• Around since 2002, you may not have heard of .mkv files but they're very widely compatible and gaining popularity because they're freely and openly licensed (while many other formats are caught up in patent wars at the moment).

MPEG-4 / MP4

• Based on an original Quicktime format, standardised in 2003

• Supports a lot of modern features, but occasionally software will treat it strangely because it was slow to standardise, and only supports one particular format of subtitle.

• Apple devices use this container but iTunes insists on calling them .m4v files.

...And Many Many More

If I wrote about all of these, you'd be reading all day. Worth at least a mention are:

• ASF - one flavour of which is WMV (Windows Media Video),

• MPG - since 1988 and still around! Arguably not a container but that's a long story,

• Ogg,

• Flash,

• Quicktime,

• WebM, the newest rising star now used by YouTube (replacing flash and mp4). Only released in May 2010, still gaining support.

Popular Video Formats

The problem with video codecs is that most of the popular ones are patented. Heavily. There's a company for managing the licenses for most popular formats, called MPEG LA (the MPEG Licensing Authority), and they have been the subject of some significant controversy. Some popular software uses these patents without licensing (illegal) focussing on the fact that it doesn't infringe on copyright, which is true but a separate matter to patents.

Patent-free video codecs exist, but are not popular or widely supported.

This has made the current state of video codecs a bit of a mess. Significant video codecs include:

MPEG-2

• Used by DVDs

• Covered by MPEG-LA which is why for so long you needed to buy separate DVD-playing software to play DVDs on your computer.

• Not supported by the Microsoft Media Foundation codecs included in Windows 7 for some reason.

MPEG-4

• Very popular now, but being replaced by h.264. Used by a lot of web video.

• Covered by MPEG-LA

• Widely supported in hardware. You may have heard of DivX, because they set themselves up as a standardisation and certification body and hence you see the logo on lots of equipment. The DivX software makes / reads MPEG-4 files.

• Other software/codecs to make MPEG-4 include XVid (DivX's open-source twin), Microsoft-MPEG-4, and the ffmpeg libraries which are used in tools like ffmpeg, VLC, Handbrake, AutoGordianKnot, and AVIDemux.

H264 / AVC

• Used on Blu-Ray discs and supported by some hardware

• Covered by MPEG-LA

• You may have heard of h264 because it's all the rage these days. Files are generally smaller than MPEG-4 files, and the quality is much higher, mainly because it has a solution to blocking artefacts (Google that if you're interested).

• h265 / HEVC ("High Efficiency Video Codec") will come out in 2013 and will probably take over very quickly because it should be easier (computationally cheaper) to decode than h264 and will have much smaller files.

VP8

• Essentially a patent-free version of h264.

• Openly-licensed and used in WebM files, now in YouTube. Generally better than MPEG-4 but not as good as H264, the open licensing is the really interesting part.

• Grows from previous versions, including VP3 which was Ogg Theora, the other popular open-source codec.

MJPEG

• "Motion-JPEG" = a jpg image for every frame

• Not really a fixed standard as much as it is a common workaround or tool

• Quite a lot of old digital cameras and several current but very cheaply-manufactured ones use MJPEG because JPEG exists as a hardware implementation meaning that saving JPEG files is very fast and battery-friendly.

• Also worth mentioning because MJPEG with lossless JPEG-2000 images is used in digital cinema formats, and because the concept is interesting: An h264 file with all I-Frames (Google to find out what I mean) is essentially a modern MJPEG video, and at least some modern video cameras make these.

At the time of me writing this, MPEG-4 is probably the dominant video format, but:

• The failings of MPEG-2 (DVD) have been highlighted by advertising campaigns for h264 (Blu-Ray),

• The popularity of MPEG-4 is failing thanks to the highly improved visual quality of h264,

• h264 isn't as widely supported as it could be because it was never popularly certified like DivX did for MPEG-4, and using h264 in hardware is very hit-and-miss,

• h264 isn't becoming more standardised because h265 is just around the corner promising smaller files,

• All of MPEG-2, MPEG-4, and h264 are suffering thanks to the patent wars surrounding them, but the alternatives have not been up to scratch.

• Webm is rising to the challenge of being a patent-free format, and is being adopted very quickly, but is still new and unfamiliar to most people, not widely supported in software, and lower quality than h264. MPEG-LA are attacking WebM and its precursor VP8.

Popular Audio Formats

There are many more popular audio codecs than video codecs, but there is a very clear winner: MP3.

MP3

• Actually a very old standard (1993)

• Not bad, but not very good quality or compression compared to other modern codecs, but...

• Extremely widely supported, plays on most hardware, and was around at the birth of internet sharing (legal and non-legal) so is used everywhere

• Popular in part because it does not support DRM ("Digital Rights Management"). Formats which arose to compete with MP3 were supported by their respective backing companies to introduce DRM to try to control illegal file sharing. Some DRM schemes meant having to have a constant internet connection to be able to play your files, some DRM files became un-playable when the software or companies went out of business, some meant lower quality, and some caused technical problems. The extreme public unpopularity of DRM has kept mp3 files popular until today.

AAC

• A very popular replacement for mp3, it's the audio-equivalent of AVC (Advanced Audio Codec vs Advanced Video Codec).

• Used as the standard type on iPods, iPhones, and many other media players

• Like HEVC, there's a standard for HE-AAC (actual "HEAC" - High Efficiency Audio Codec) which is already in use, but AAC will be around for a while yet. I spoke with an engineer from Dolby once who told me that the current HEAC will encode perceptually-lossless music at 64kbps! In non-nerd speak, that means about half the size and twice the quality of mp3s.

AC3 / DTS / PCM

• These formats are not related to each other, but they are all used in DVDs and Blu-Ray discs and are the main formats currently used in media files besides MP3.

• These are directly supported by most sound systems (eg: the 5.1 speaker system plugged into your DVD player).

• PCM isn't a codec, it's just raw, uncompressed "Pulse Code Modulation" audio.

Ogg Vorbis

• Patent-free, used in webm files.

Playing a Video File on Hardware

I have a cheap "Soniq"-brand Blu-Ray player and media center. Media-center PCs and streaming media boxes are increasingly popular. Many new TVs support media playback, and several of my friends have WD Live media player boxes. You can plug the newest iPhones and iPads directly into your TV with the right cable.

How do you make sure that your media files will play on your hardware?

• Make sure that it can read your container format

• Make sure that it supports your audio format

• Find out what video formats it supports, and what resolutions, and what "profiles".

  • Video codecs like MPEG-4 and h264 have "profiles", which specify essentially how much memory and how much processing they will require to play.

  • It's hard to find out what profile a file was made with, but it's good to know if you're encoding any files to copy onto it.

• Ignore all of that research and just give it a try!

  • The keen eye of the MPEG-LA and other patent holders, combined with the liberal approach to patents and copyright in the countries where hardware is mass-produced (mainly China), means that a lot of hardware will support a lot more than the manual will admit to!

  • On the flip side, the manual might claim that it should work, but you'll find that you have trouble making it work in practice. Even something as simple as a DVD can cause you trouble if there's too much audio data and the video has too much action, meaning that your DVD player might stutter or even crash while playing some scenes.