Media Servers

Video Codecs, Containers, and Files

Why are there so many different standards and file extensions pertaining to digital video? This article explores a high-level understanding of common digital video and multimedia formats, with an emphasis video/multimedia formats, file extensions, containers, and codecs. Multimedia formats (codecs, containers, and standards) can be broken down into two (2) distinct groups: Video + Audio (this article) or Audio only (this article).

Why are there so many different file extensions? Why do some video formats work on a given device and some don't? Which format or file extension is best, and why? At the root of all these questions are the technology that makes the multimedia magic happen: multimedia codecs (video codecs in particular). Digital video codecs have evolved over time due to demand-side market forces. Notably:

  1. Video resolution
  2. File size (compression)
  3. Operating system compatibility
  4. Licensing standard availability

Incumbent Standards Organizations

The audio and video standards we are accustomed to are not arbitrary. In order to effectively share information, it is necessary all the parties involved agree on how the shared data is organized and accessed. These standards emanate from one of several incumbent international standards bodies:

  • ITU: International Telecommunication Union (1847); communications
  • IEC: International Electrotechnical Commission (1906); electrical, electronic and related technologies
  • SMTPE: Society of Motion Picture and Television Engineers (1916); media and entertainment industry experts group
  • ISO: International Organization for Standardization (1947); representatives from various national standards organizations
  • ITU-T: ITU Telecommunication Standardization Sector (1956); telecommunications and information communication
  • IEEE: Institute of Electrical and Electronics Engineers (1963)
  • MPEG: Moving Picture Experts Group (1988); working group of ISO/IEC to set standards for audio and video compression
  • IETF: Internet Engineering Task Force (1993); develops and promotes voluntary Internet standards

The majority of video codecs and containers are sponsored by MPEG or ITU-T. The ITU is an international body sanctioned by the United Nations.1 ITU-T is a sub-group of the ITU, and is responsible for coordinating worldwide standards for telecommunications and cybersecurity. The so-called "H-" standards are video compression codecs endorsed by the ITU-T. These are not independent standards. Rather, the H. designation should be thought of as a short-hand way of referring to video codec standards created by other bodies and adopted by the ITU-T. For example, H.264 was developed by MPEG.

This entire subject becomes confusing very easily, particularly considering the fact these organizations tend to overlap in their scope. SMTPE tends to be an exception. However, it rarely weighs in on standards. When it does, it is normally to clarify or provide additional guidance, such as recommended use case scenarios for particular formats.

Some file extensions, containers, and codecs have similar names to one another. This can make things very confusing for software developers, content creators, and end users to figure out what the heck is what. To prevent this from happening, a few well-represented organizations have sprung up over time to address these kind of things. Notable organizations include MPEG (Motion Picture Experts Group), ISO/IEC (), and ITU-T (). The vast majority of file extensions we are all familiar with stem from adoption by one of these key organizations. However, there are a few historical cases where poor communication and/or questionable marketing practices on the part of a small number of large tech companies has created the perfect storm of confusion over file extensions and file formats. Thankfully, these cases are few and for the most part files associated with unpublished or poorly executed codecs and containers have died out, but that is not the case 100% of the time. Thus, on occassion, someone (typically an end user) may encounter a file that does not play as expected. In the case of the file formats described below, when this occurs it may be because the encoder used on the file is not the one the user would expect.

Multimedia File Components

Digital multimedia content is normally stored as individual files. The video and/or audio content on those files may be disseminated either via the file itself or by transmitting its contents (data) from one point to another (streaming). The process that allows content to go from creation to consumption involves several key concepts that are important to understand if you want to maintain a digital library of any kind. And even if you don't care about that, it's good to have a high-level understanding of how these things work, so that at some point when you're frustrated why you can't watch or listen to some particular multimedia content, you will have an idea of why.

The factors determining whether or not you're able to watch or listen to a paricular file or stream depend on:

  • Device hardware
  • File extensions
  • Containers
  • Codecs

Devices

The device you're using to consume multimedia content matters to the extent it is capable of interpreting the data in any given digital file or stream. While this might seem obvious, it's important to note that some multimedia codecs require specific types of hardware capabilities to either encode and/or decode content. Why would anyone use a codec with hardware limitations? Such solutions are typically one of the following reasons:

  1. Exclusion; limit access to a particular class of devices and exclude all others
  2. Efficiency; taking advantage of hardware-based, advanced data compression capabilities
  3. Copy Protection; encrypting content via hardware that makes decrypting it difficult or impossible without identical hardware
  4. Speed; using specific hardware to boost data transmission via a method other than greater data compression
  5. Quality; improving perceived quality of consumed content with a higher qualified result, such as higher video resolution

Examples of corporate entities that use these techniques to influence consumer behavior abound. Apple, Inc. is notorious for using exclusion and improved quality techniques either via hardware or software licensing restrictions to limit access to multimedia content created by their devices. Cellular phone companies use efficiency to limit access to specific classes of content to mobile devices with associated hardware, driving consumers toward more frequent handheld device upgrades and exchanges. Those are just two examples of how consumers are frequently manipulated through hardware exclusion. As you will see further in this discussion, these restrictions are enforced by the clever and purposeful implementation of codecs.

Video File Extensions

They are not the key factor, but file extensions are what most people want to know about! Files are simply storage media for digital data. A file extensions refers to the suffix of a filename, and usually (but doesn't always) indicate the type of content inside a file. It's wise to consider them to be hints about what type of data is likely contained inside a file. As you'll see below, a number of popular file extensions are shared by different media types. File extensions suggest what type of content the file contains. Through this suggestion, a starting point is provided to guide the operating system of the playback device. It is an indicator of how the file was encoded.

Most file extensions follow a legacy pattern of a period followed by three (3) alphanumeric characters. These four (4) characters are appended to the end of a filename. The trend began with two (2) alphabetical characters, when software developers wanted a simple method of locating file types for a specific programming language or function. Four (4) character extensions began to become more prevalent in the late 2000's, though three (3) character extensions remains the norm. Theoretically, there is no limit and the practice is primarily a convenience for humans. From a technical perspective, they are irrelevant.

File extensions are arbitrary. Contrary to popular belief, they are not accurate indicators of a file's content type.

Certain file extensions denote proprietary codecs or file format containers. Their native support may be restricted by operating system, device, or intellectual property limitations. If you are creating content, proprietary formats should generally be avoided when possible, in favor of open source solutions.

Reference Standards for Filename Extensions

There are two widely regarded standards for filename extensions pertainting to multimedia files: the MPEG-4 standard for Coding of audio-visual objects — Part 12: ISO base media file format (ISO/IEC 14496-12:2015) and the older JPEG 2000 ISO base media file format (ISO/IEC 15444-12:2012), which has now been consolidated into the former. These ISO/IEC standards dictate how file extensions should be used to represent media types, including but not limited to file structures.

Codecs vs. Containers and Why It Matters

Codecs are software programs that describe how multimedia content (video and/or audio) should be stored (called "encoding") and played back (called "decoding"). Containers are shells that provide additional structure to "contain" codecs. Codecs may be stand-alone or placed inside a container. Multimedia files are digital representations of audio and/or video. Most multimedia files are containers, but some consist of encoded files only (via a codec).

Most codecs and many containers compress the underlying original multimedia data in order to use less storage space and to make it more efficient to transfer multimedia information (streaming) via a network, such as the Internet. Doing so requires the receiving device to be able to accurately identify how the content was created (by which codec), be able to decode the information, and play it back to the end user in its original or intended form. This process is called decoding.

Audio and video files may also hold compressed or uncompressed material with no identifiers or metadata. This is called a "raw" file. Raw files still require codecs to interpret their content. Their difference is the playback device must know how to restore and playback the information. There are no clues in the file or data stream. Raw files are very rarely found in the wild as they are inducive to sharing information. Raw files require pre-shared knowledge exchanged between the sending and receiving device, or the playback device will be unable to restore the information for the end user.

Codecs

a Native operating system support (Win=Windows; And=Android)2
b Partial
Codecs
Format Type File Extensions Containers Operating Systema Open
Source
Standard
Win macOS iOS And Linux
H.266
(VVC)
Storage
Streaming
? ? ? ? ? ? ? ? MPEG-I
H.265
(HEVC)
Storage
Streaming
.mp4, .mkv, .mka MPEG-4
Matroska
10 10.13 11 4.4 5.0 No MPEG-H
H.264 Storage
Streaming
.avci, .avcs
.flv, .h264
MPEG-4
Matroska
7+ 10.4 Yes Yes3 Yes Yes MPEG-4
H.263 Storage
Streaming
.mp4, .3gp MPEG-4 Yes Yes No Yes Yes No MPEG-4
H.262 Storage
Streaming
.mpg, .mpeg -- Yes Yes No No Yes No MPEG-2
H.261 Broadcast .mpg, .mpeg -- Yes No4 No Yes Yes No MPEG-1
AV1 Streaming .mkv, .mp4, .webm HEIF No No No Yes Yes Yesb AV1
VP9 Web browser .mkv, .mp4, .webm WebM 10 No4 14 4.4 Yes Yesb TrueMotion
VP8 Web browser .mkv, .webm WebM 10 No4 12.1 2.3.3 Yes Yesb TrueMotion
VP7 Storage .mkv Matroska 10 No4 12.1 2.3.3 Yes Yesb TrueMotion
VC-1 Blu-ray
HD DVD
.vc1 ASF Yes Yes No No Yes Yesb VC-1
WMV Storage .wmv ASF Yes No No No Yes No WMV, M4S2
AVI Storage .avi AVI Yes Partial Yes No Yes No AVI
FFV1 Storage .avi AVI Yes No No No Yes No AVI
RealVideo Storage .rm, .rv -- Yes No No No No No RealVideo
ProRes Storage -- -- No Yes No No No No ProRes
Theora Storage .mkv Matroska Yes Yes No No Yes Yes Theora

ITU-T Video Compression Standards

ITU stands for the International Telecommunication Union. ITU-T stands for the ITU Telecommunication Standardization Sector. The ITU is an international body sanctioned by the United Nations.1 ITU-T is a sub-group of the ITU, and is responsible for coordinating worldwide standards for telecommunications and cybersecurity.

The so-called "H-" standards are video compression codecs endorsed by the ITU-T. These are not independent standards. Rather, the H. designation should be thought of as a short-hand way of referring to video codec standards created by other bodies and adopted by the ITU-T. For example, H.264 was developed by MPEG.

H.120 (1984)

I'd be shocked if you ever encounter H.120 video. It is the oldest video codec standard, and its video quality was terrible. It would be considered unusable by today's standards. H.120 is not an MPEG standard. Its publishing organization became the ITU-T.

MPEG Codecs

The most popular video codecs were created wholly by MPEG or jointly between MPEG and another organization (usually the ITU-T).

EVC | MPEG-5 Part 1 (2020+)

Essential Video Coding (EVC) - also known as MPEG-5 EVC - is Part 1 of ISO/IEC 23094; a video compression standard still in development that is rumored to be planned as an open standard, though whether or not that will be the case remains to be seen. MPEG does not have a strong history in that regard (quite the opposite). EVC is presumed to be an alternative to HEVC (H.265/MPEG-H) and AVC (H.264/MPEG-4 AVC). At this time, it seems more likely the best-case scenario is MPEG-5 will be released as a partially open source platform, with its free version sporting compression equivalent to AVC High 10 (a profile of MPEG-4 AVC) and roughly 31% greater than the H.264. Meanwhile, the licensed (paid) version will likely be benchmarked to H.265 and reportedly provide about 25% better compression. EVC will almost undoubtedly be hardware-dependent and support 8K video, 4:4:4 and 4:2:2 chroma sub-sampling.

LCEVC | MPEG-5 Part 2 (2020+)

LCEVC stands for Low Complexity Enhancement Video Coding. It is defined under MPEG-5 Part 2 and is an extension to the EVC codec. LCEVC will prioritize implementation speed over encoding/compression efficiency, making it an ideal tool for use cases that require rapid data transfer with less concern for compression and storage, such as live streaming.

H.266 | MPEG-I (2020+)

H.266 - also known as VVC (Versatile Video Coding) - and technically, MPEG-I Part 3 (ISO/IEC 23090) is a planned, unfinalized video codec being defined by a joint ITU-T/MPEG research team, expected to be finalized and released by the end of 2020.5 H.266 purportedly will sport a minimum 30% lower bitrate than H.265 while maintaining the same quality standards. Normally, the notion of a lower bitrate equates to lower video resolution and quality, but that is not always the case. The transition from H.264 to H.265 brought about the same thing. Improvements in data compression between the standards (thanks largely to improved hardware device compression codecs) are what makes this possible. As long as one is not changing the resolution and color depth, the newer standards will deliver the same experience with less shared information. Intead, the standards' algorithms perform the heavy lifting.

As of this writing, the 2nd draft (2019) of the standard is under review. The pending standard is intended to support video resolutions up to 16K and 360 degree videos, and will be the first MPEG video codec to support the ITU's high-end video recommended standard called BT.2100 resolution standard (also known as Rec. 2100). H.266 is hardware-dependent.

File extension: .mp4

H.265 | HEVC v2 (2017)

There are two (2) versions of the HEVC codec. Released in a 2017 update to the MPEG-H HEVC standard (ISO/IEC 23008-2:2017), HEVC continues to suffer from poor adoption in spite of its enhanced capabilities over its predecessor. This likely results from two (2) factors: 1) HEVC's complex licensing structure; and 2) no one cares about 8K video at the moment (true 8K is not yet a practical reality, in spite of occasional media hype).

File extensions:
Common: .mp4, .mkv, .mka, .mks
Less Common: .3gp, .3g2
Rare: .mj2, .dvb, .dcf, .m21, .mk3d, .ts, .tsv, .tsa

H.265 | HEVC v1 (2003)

Better known as H.265, High Efficiency Video Encoding (HEVC) is part of the MPEG-H standard (ISO/IEC 23008-2). Four (4) components to the MPEG-H standard impact HEVC. They are:

  • ISO/IEC 23008 Part 1 (Streaming standard)
  • ISO/IEC 23008 Part 2 (HEVC standard)
  • ISO/IEC 23008 Part 3 (3D audio)
  • ISO/IEC 23008 Part 12 (Image File Format for HEVC standard)

H.265 ups the ante on video resolution over H.264 with a maximum supported of so-called "8K" (i.e., 8192 x 4320 pixels) or roughly 16 times the depth of its predecessor. Compared with H.264-AVC, H.265 has 25-50% higher data compression, resulting in smaller files with equal video quality.

File extensions:
Common: .mp4, .mkv, .mka, .mks
Less Common: .3gp, .3g2
Rare: .mj2, .dvb, .dcf, .m21, .mk3d, .ts, .tsv, .tsa

H.264-AVC (2003)

H.264-AVC or Advanced Video Codec is a hybrid format, consisting of an AVC (H.264) codec inside a HEIF container. AVCI is specified under MPEG-H Part 12 (ISO/IEC 23008-12) and the 2003 revision of MPEG-4 Part 10 (ISO/IEC 14496-10:2003 Advanced Video Coding). It is sometimes referred to as MPEG-4:10 AVC.

File extensions: .avci, .3gp, .mp4

H.264 (2001)

The original High Efficiency Video Encoding (HEVC) standard - also known as H.264 is part of the MPEG-4 standard (ISO/IEC 14496 Part 10). HEVC (version 1) is known by several different monikers:

  • H.264
  • Advanced Video Coding (AVC)
  • HEVC-AVC
  • MPEG-4 AVC
  • AVC (Advanced Video Coding)

As if that's not confusing enough, there are two (2) HEVC standards: the MPEG-4 - or original - variant, and the modern, MPEG-H version. This unfortunately can create quite a bit of confusion. It is also why many media afficianados and professionals refer to these standards by their "H." name;6 H.264 for HEVC v1 and H.265 for HEVC v2. This method is helpful for other reasons as well. Taking a look at HEVC v2 for example, you'll find there are a number of related container types. While most of these containers can be read by most playback devices and software, not all of them can be (even though they all use the HEVC v2 codec).

Well, that sounds like a mess, doesn't it? How can consumers tell them apart? While it's not foolproof, one may be able to deduce the codec based on a file's extension. If a HEVC-encoded file has a file extension of .mp4, it's impossible to tell without examining the file. If it's a .3gp or .3gpp file, it's HEVC version 1. If it is any other file extension, it is HEVC version 2.

H.264 is generally regarded as the baseline for what most people think of as "high definition" video.

Conflicting information will be found via online searches inquiring about the maximum video resolution of the H.264 standard. Is it 2K (1080p), 4K, or 8K? MPEG's documentation states it is 1080p (1080x1920), progressive scan.7 Meanwhile, other sources report a 4K boundary, and still others claim it's possible to get an 8K throughput from custom modified H.264 streams.8 One additional caveat: it should be noted the definition of "8K" is somewhat fluid. Some sources claim 7680 x 4320 and others 8192 x 4320. I'm not going to go down this rabbit-hole now, but I'll briefly add the 7680 figure does have merit. It comes from removing the scan lines which are not visible due to how television signals are drawn on a screen, making use of something called overscan. When one factors out the overscan of an 8K image, 7680 lines is what you're left with. However, again it depends on one's perspective and what device, dimensions, and a host of other variables. Still, in a general sense it does make sense at some level.

Mobile device hardware must explicitly support HEVC v1 or v2 playback or it won't work.

H.263 (1996)

H.263 is a low-bitrate video codec designed specifically for terrestrial-based video conferencing, such as video-telephony over PSTN lines (Public Switched Telephone Networks; i.e. traditional 2-wire analog phone lines) and SIP (Session Initiation Protocol) connections. Built off MPEG-1 and MPEG-2 standards, it has no relationship to the better-known H.264 standard. H.263 became the de-facto standard protocol for multimedia delivery to so-called "3G" ("3rd generation") wireless mobile devices in the 2000's, replacing the venerable H.261 standard.9

File extensions: .mp4, .3gp

H.262 (1995)

H.262 is an MPEG-2 video codec (MPEG-2 Part 2; ISO/IEC 13818-2:2013) more commonly known by it's ITU-T name. Even though it is over 25 years old,it is still supported today, having been last ratified in 2019.

H.262 codecs do not need to be stored inside a container, but they can be. They may optionally be stored in .ts or .ps container types, both of which are also part of the MPEG-2 standard.

File extensions: .mpg, .mpeg, .ps, .ts

H.261 (1988)

H.261 is a very old video codec based on MPEG-1 (ISO/IEC 11172). With a maximum throughput of 1.5 Mbps, H.261 supports a maximium resolution of 352x288 (CIF vide frames) with 4:2:0 sampling. It was used primarily used to deliver audio-visual services at a more modest 64 kbps typical bitrate.

File extensions: .mpg, .mpeg

Google and TrueMotion Codecs

TrueMotion is the basis for the "VPx" video codecs developed by The Duck Corporation. The company renamed itself to On2 Technologies before being acquired by Google in 2010. TrueMotion was the product name of the company's codecs that attracted Google's attention. Notably, TrueMotion VP7 and VP8. The TrueMotion codecs are better known by their version numbers, such as VP6, VP7, nd VP8 to name a few. VP8 was the last version produced solely by On2. Google acquired the company in 2010, and subsequently released VP9 (2013), followed by AV1 (2018).

AV1 (2018)

AV1 is a terribly named open source streaming multimedia codec with strong promise. Developed by the Alliance for Open Media (AOMedia) - a non-profit organization supported by Mozilla, Netflix, and Google - AV1 is the successor to VP9. Released at the very end of 2018, AV1 offers better data compression (with like-kind quality) compared with VP9 and HEVC-based codecs (e.g. H.264 and H.265).

It is common for AV1 codecs to be embedded inside HEIF containers. AV1 supports up to 12-bit color depth, 4:2:0/4:2:2/4:4:4 chroma sub-sampling, HDR, and both lossless and lossy compression. It is sometimes erroneously referred to as VP10, which may stem from confusion regarding Google's involvement in the codec's development. VP9 was the last TrueMotion codec.

File extensions: .3gp, .3g2, .m21, .mj2, .mkv, .mov, .mp4
Found in Containers:
AVIF, Matroska

AV1 ("ay-vee-wuhn") is video codec and bitstream format. The nearly identically named AV1F ("ay-vee-wuhn-ɛf"), also known as AVIF ("ay-vee-i-ɛf"), is an image file format.

AVIF | AV1F (2018)

The AV1 Image Format (AVIF or AV1F) is an image file format (not a video codec) consisting of AV1 encoded (AV-one) frames embedded in a HEIF container. This is an important distinction, because AVIF is not intended to replace AV1, but rather to co-exist with it.

Just like it's cousin, the AV1 codec (pronounced "ay-vee-wuhn"), AVIF is another poorly named file format that arguably does just as much to confuse users (especially consumers) as it does to move the bar forward in video compression formats. AVIF files and streams may contain still images, video, or synchronized video and audio. Technically released in very late 2018, AVIF did not hit the consumer market until 2019, and even now it is relatively unknown.

As of June 2020, AVIF is not supported by any web browser, though it is likely to happen in late 2020. Mozilla quietly included experimental support for it in early April 2020, but removed it two months later.10 And Google has leaked hints it is planning to add support in its Chromium-related products in the 2nd half of 2020.11

File extensions: .avif, .heif, .hif

VP9 (2013)

VP9 succeeded VP8. Designed and released by Google as an open-source alternative to the proprietary H.265 (HEVC v2), VP9 is geared largely toward web browsers and streaming multimedia content. Google seized on an opportunity to sway the market adoption of 4K video standards by offering a free and open source video codec as good or better than H.265. At the time, the MPEG Licensing Authority (MPEG LA) was struggling to gain widespread adoption for the H.265 standard due to its onerous licensing fee structure and aggressive stance on enforcing its related patents,12 not the least of which was a high-profile lawsuit brought against Apple, Inc. for patent infringement of H.265.13

File extensions: .ikv, .mkv, .webm
Found in Containers: Matroska, WebM

VP8 (2008)

TrueMotion VP8 was a proprietary codec competing directly against H.264. In 2010, Google acquired the intellectual property of On2 Technologies, including the VP8 standard. Almost immediately, Google migrated VP8 to an open-source status shortly after the aquisition. VP8 formed the basis for VP9 and AV1, which are also open-source.

VP8 encoded video must reside in a WebM or Matroska container.14>

File extensions: .ikv, .mkv, .webm
Found in Containers: Matroska, WebM

VP7 (2005)

VP7 is one of the TrueMotion compressed video codecs created by On2 Technologies. It is comparable to H.264 in terms of video quality and compression. VP7 is widely regarded as having slightly superior compression (smaller files) compared with H.264. However, evidence to support this position is scant.

VP6 (2003)

VP7's predessors: TrueMotion VP6 and its prior versions are mentioned here briefly in the interest of completeness. These codecs are outdated, obsolete, and generally not supported. VP6 is supported well only by Adobe's original Flash container type, which is itself on the way out.

Microsoft's Legacy of Video Codecs

If you can't buy them, beat them to market; or try to. That seems to have been Microsoft's strategy around video codecs and containers in the 1990's and early to mid 2000's.

VC-1 (2006)

Still hell-bent on outdoing MPEG, in 2006, Microsoft released VC-1, an open-source evolution of its WMV codec after waiting three (3) years to attain support from the Society of Motion Picture and Television Engineers (SMPTE) - another international multimedia standards organization - for its codec format (SMPTE 421M-2006). Simultaneously, Microsoft abandoned the WMV format. However, VC-1's backward compatability matrix mirrors that of its predecessor. Like WMV 9, VC-1 has two (2) different profiles called Simple and Advanced. The Simple VC-1 profile is equivalent to ST 421-2013 (2013).

File extensions: .vc1

Windows Media Video (2003)

There is a series of little known versions of WMV released by Microsoft from 2003-2005 when the company attempted to dominate the emerging high-definition video codec market. Also called, Windows Media Video (like its predecessor), these were proprietary MPEG 4 codecs are supported by Windows-based operating systems only. Microsoft attempted to out-do MPEG by beating the latter's market launch of MPEG-4 to the masses. Microsoft accomplished this by bundling a series of proprietary, exclusively Windows-based MPEG-4 codecs into Microsoft's Windows Media Player in the form of three (3) different codecs, released over time. Creatively dubbed, Microsoft MPEG 4 codec versions 1, 2 and 3 (yes, how creative is that)? Microsoft MPEG 4 Codec Version 3 was the last iteration; later to be superceded by Microsoft Media Foundation, a Software Development Kit (SDK) consisting of a collection of Windows-based codecs supported by the Windows Media Player15 and which did not sound quite as plagiarizing as the prior versions.

Playback of video encoded with these codecs requires the Mpeg4s Decoder DMO video decoder, an add-on extension to the Windows Media Player.

These pre-release MPEG-4 codecs (what they really are) were also known as:

  • MPG4: Microsoft MPEG-4 version 1
  • M4S2: Microsoft MPEG-4 version 1.1
  • MP42: Microsoft MPEG-4 version 2
  • MP43: Microsoft MPEG-4 version 3

While obsolete to the vast majority of users, these codecs are still supported by ASF, DivX, QuickTime, Xvid containers, and are H.263 compatible, though their maximum packet size is rather limited at 64 KB.33

Found in container types: ASF, AVI, WMV
File Extension: .mp4s

WMV (1999)

Windows Media Video codec (WMV) is actually a group of codecs built on a proprietary implementation of MPEG-4. Developed by Microsoft, the first iteration - WMV 7 - was released in 1999. The last version was WMV 9 (2003) before it was transitioned to an open-source platform called VC-1.

WMV 9 has three (3) iterations:

  1. WMV 9
  2. WMV 9 Professional (Pro)
  3. WMV 9 Advanced Profile (AP)

WMV 9 and WMV 9 Pro are interoperable. The Pro codec is invoked when the stream bitrate exceeds 1 Mbps or video resolution is 720p or higher. The WMV 9 AP codec is incompatible with the others. VC-1 is based on the main and Pro formats.

WMV content may be found inside a .wmv file (ASF type container), .mkv (Matroska), .avi (AVI container), or an .asf file (ASF container).

File extensions: .asf, .avi, .mkv, .wmv

AV1 ("ay-vee-wuhn") is not the same thing as AVI ("ay-vee-ai"). While appearing nearly identical in print and are both multimedia related, they are different generations of video codecs.

AVI (1992)

A legacy codec introduced in 1992, Audio Video Interleave (AVI) is one of the oldest multimedia container formats. Although uncommon today, if you've been using the Internet for a long time, you likely have some of these files somewhere.

The AVI standard has an overall file size limit of 4 GB. However, larger files are possible through a hack. RIFF's architecture uses a chunk style format. The process of expanding an AVI maximum file size involves writing a sequence of nested RIFF chunks. However, early AVI implementations contain bugs in the code which obviate this process, so if you are looking for a multimedia display solution on very old legacy hardware, be warned this technique might not work. When that is the case, the outer chunks will be limited to 1-2 GB in size, so make sure if you do have larger sized chunks they are in the nested portion. Either way, AVI is not a good format for large files (> 4 GB), period.16

File extension: .avi

Future Codecs

While they aren't yet released or officially usable, these video codecs are likely to come to fruition between 2020-2022. They are listed here because they are not under the auspices of the most influential video codec standards bodies, such as MPEG, yet they are likely to have enough support to still become mainstream sooner or later.

RFC 8761: Next Generation Video Codec? (2021+)

RFC 8761: Video Codec Requirements and Evaluation Methodology is an IETF effort to define a new video codec standard geared toward streaming high resolution (4K or higher) across low and high latency networks. Time will tell if it goes anywhere or not.

Daala (2020?)

Daala is a royalty free, lossy video compression format based on the AOMedia Video 1 (AV1) codec and developed by the Xiph.Org Foundation, a non-profit organization known for very popular audio codecs such as Vorbis, Opus, Speex, and FLAC. At its conception, Daala was intended to compete with H.265. Going full open source, Daala will be wrapped inside an Ogg container.

While it is yet to be released as an official codec, the IETF was impressed enough with Daala to explicitly mention it as the objective metrics implementation standard in the January 2020 draft of RFC-8761 (updated 25 March 2020). Interestingly, references to Daala were removed prior to the public release of final standard, RFC 8761: Video Codec Requirements and Evaluation Methodology. It's not clear why, but a good guess would be the fact it doesn't exist yet from a practical standpoint. RFC 8761 is an IETF effort to define a new video codec standard geared toward streaming high resolution (4K or higher) across low and high latency networks.

Will Daala ever come to fruition? Despite its bold intentions, the project has been 'in progress' for quite some time. In fact, interest in the platform has waned since early 2015 (with the exception of a paper published by the IEEE in 2016).17 Perusing the project's roadmap and GitHub page, it's apparent no formal plans have evolved post-2015 and since 2017 only cosmetic development efforts have been made. Information on the Xiph website indicates the project is active, but the available evidence gives one the impression this is only marginally true. While it seems promising, Daala appears dead-in-the-water at this time. On-the-other-hand, all hope may not be lost for it. The video codec space is quite crowded, however there is still plenty of room for improvements; particularly for software-based, open-source formats such as Daala. And with Mozilla putting its weight behind it, there are sufficient reasons to believe it may eventually get off the ground and fulfill its potential as a next-generation codec. Time will tell.

File extension: .ogv

Oddball Codecs

Codecs you've likely never heard of, and will probably never use. Still, you never know. One or more of these may intrigue you. These audio, visual, and audio/visual multimedia file formats are not officially recognized under ISO/IEC, ITU-T, or RFC documentation standards, but were adopted by various enterprise organizations at one point in time or another. The fact these were never registered with a 3rd party authority such as IANA, ISO, or MPEG suggests the companies didn't foresee there was a need to do so for one reason or another. While it is rare to find these file extensions in use, it does happen from time to time.

NUT (?)

NUT is perhaps the antithesis of FFV1. NUT is a nearly-universal container open-source audio/video container format that has never existed, but has been worked on for almost 10 years. It probably deserves a prize for the longest running codec that no one uses (at least no one with any common sense). A series of upstart attempts to complete the standard have all consistently failed since its inception. While the original mission of its founders may have been admirable, sadly the end goal has continued to elude its supporters. I suspect this is largely due to the fact anyone is hard-pressed to come up with a reason why the world needs NUT. Either way, currently HEIF. If you want an open-source and versatile container format, there really is no reason to look far beyond Matroska for 99% of requirements.

Dirac Research (2008)

Dirac is an open-source broadcast video codec developed by the British Broadcasting Commission (BBC) in 2008. Supporting both lossy and lossless video compression, Dirac was designed for HDTV broadcasting and has similar data compression capabilities to H.264. Dirac is a relatively obscure codec. It's most recent update was released in 2009.

Dirac codec files may be found as stand-alone .vc2 files or (more commonly) embedded inside an AVI, MPEG-2 TS, MPEG-4, or Matroska container.

VC-2 is a standard adopted by the SMPTE in 2009 which embodies Dirac's sister codec, called Dirac Pro. The latter is a lossless, low latency, low-complexity version of the original codec. Dirac Pro is in many ways the veritable opposite of Dirac; a bandwidth hog capable of 8K video with 10-bit chroma (4:2:2) depth with no sub-sampling. Dirac Pro prioritizes low latency and high resolution above all else.18 Despite being a respected authority within the theater and film industry, SMPTE doesn't carry much weight in the digital media world, generally speaking.

File extension: .vc2

Apple ProRes (2007/2018)

Apple has a tendency to introduce new technologies and impact consumer behavior in very polarizing ways. Technologically, it often does this in such a way as to simultaneously promote other products it owns and admonish non-Apple standards. Apple's ProRes is a good example of this. ProRes (and its cousin ProRes Raw introduced in 2018) were cutting edge, high performance codecs designed for video editing and to elevate the capability of average users. The downside to them, of course, is they only work on Apple products.

ProRes is distinctive in part because it is a lossy video codec with significantly faster encoding and decoding times compared with H.264, while supporting 5K resolution (5,120 x 2,880), chroma subsampling of 4:2:2, and 10-bit color depth. That was very impressive in 2007.

Apple has released several iterations of the ProRes series (for a total of six, currently), culminating with ProRes RAW in 2018. ProRes RAW sacrifices some performance (encoding speed) in exchange for greater color depth and resolution quality. Since 2009, the format family has supported 8K video resolution, including 4:4:4 chroma subsampling and 12-bit color depth. That doesn't sound like much until you consider the fact an 8K resolution picture requires almost 34 Mbps per bit per pixel, times the display frequency. At 30 frames-per-second (fps) and 12-bit color depth that is about 12 gbps (giga-bits-per-second) uncompressed. Good luck with that.

Supported container formats: 3GP, AVI, MP4, M2TS, Quicktime
File extensions: .3gp, .avi, .m2ts, .mp4, .qt, .ts

Theora (2004)

Theora is a free lossy video compression format developed by the Xiph.Org Foundation, a non-profit organization known for very popular audio codecs such as Vorbis, Opus, Speex, and FLAC. Unfortunately for Xiph, Theora has always been a relatively unknown format. Based on On2 Technologies' very old VP3 codec, today it is quite obsolete.

Daala is the planned successor to Theora, and is purportedly under active development by Xiph, along with some assistance from the Mozilla Corporation - the owner and producer of another heavyweight open-source platform - Firefox.

Supported container formats: Ogg
File extensions: .ogg, .ogv

FFV1 (2003)

FFV1 or FF Video Codec 1 is a lossless ffmpeg codec. Relatively unknown to consumers and even to film professionals, in spite of its superior function for the archival of high resolution digital video material, such as film and broadcast master copies.

FFV1 is used by a number of governmental organizations, perhaps due to lesser emphasis on being cost-conscious and a greater emphasis on quality of data retention. For example, the city of Vancouver, Canada uses FFV1 to encode and preserve a portion of its archvies, which are in turn stored using Matroska containers.19

For the purpose of long-term archival storage of digital video, FFV1 is potentially attractive for this purpose due to the fact it is lossless, yet still offers data compression. This is particularly poignant for digital film storage of high-bitrate master copies, such as those used in the professional film industry. Interestingly, there is ample evidence that FFV1 - although primarily designed for lossless video archiving - makes for an excellent long-term storage medium for analog audio.20,21

File extensions: .avi, .mkv, .mov, .nut

RealVideo (1997)

RealVideo comprises a set of proprietary video formats produced by Realnetworks, Inc. While there are multiple iterations of these codecs and containers, they all share something in common. You must have a RealPlayer application to play them. The company rigorously enforces its intellectual property rights. As crowded as the video codec and container landscape is, I don't understand why anyone still uses them, but they do. So be it.

RealVideo and its predecessor RealMedia are notable for only one thing: the fact they still exist and someone is actually paying to use them (OK, that's two things, but no doubt you get my point).

File extensions: .rm, .rv

Containers

a Native operating system support (Win=Windows; And=Android)22
b Partial
Containers
Format Type File Extensions Codecs Operating Systema Meta
data
Open
Source
Standard
Win macOS iOS And Linux
HEIF Storage .avci, .avcs, .avif
.heic, .heif
H.265 (HEVC) 10 10.13 11 9 Yes Yes Yesb HEVC
HEIC Storage .3gp codecs 10 10.13 11 9 No Yes Yesb HEIF
MPEG-DASH Storage
Streaming
-- MPEG-H Yes No No No No Yes No MPEG-DASH
WebM Streaming .webm | .mp4 VP8, VP9
AV1
Yes No No Yes Yes Yes Yes Matroska
AVIF Storage .avif | .heif | .hif codecs 10 No No 10 No No Yes AV1
Matroska Storage .mkv Many
(see list)
Yes No No Yes Yes Yes Yes Matroska
MPEG-4 Storage
Streaming
.mp4 MPEG-4 Yes Yes Yes Yes Yes Yes No MPEG-4
(Part 14)
Ogg Storage .ogg | .ogv Theora, Daala Yes Yes No No Yes Yes No Ogg
IVF Streaming .ivf VP8 Yes No4 No No Yes No No VP8
MPEG-2 PS Storage .mpeg, .ps H.262, WMA 9
WMA 9 Pro, VC-1
Yes No No No No No No MPEG-2
MPEG-2 TS Broadcast .m2ts, .ts H.261, H.262
H.263, H.264
Yes Yes Yes Yes Yes No No MPEG-2
Flash Video v1 Streaming .flv FLV, VP6
H.264c
Yes Yes No No Yes Yes No Flash
Flash Video v2 Streaming .flv FLV, H.264 Yes Yes No No Yes Yes No Flash
3GP Mobile .3gp | .3gpp H.263, H.264
AVC1
No No Yes Yes No Yes No MPEG-4
Quicktime Storage .mov, .qt Quicktime
MPEG-2, MPEG-4
No Yes Yes No No Yes No Quicktime
ASF Storage .asf | .wmv WMV Yes No No No Yes Yes No ASF

Modern Container Standards

Current video/multimedia standards. These are the container types you are most likely to encounter unless you're working with rather old files.

HEIC (2019)

File extension(s): .heic

HEIC is a proprietary container owned by Apple consisting of a HEVC codec inside another (HEIF) proprietary container not owned by Apple. Confused yet?

High Efficiency Image Container (HEIC) was created by Apple. A derivative of the HEIF container standard, Apple took the liberty of modifying the HEIF container standard (owned by Nokia) just enough to confuse standard HEIF readers and prevent them opening HEIC files. Typical Apple tactic. At least that would appears to have been Apple's intent. Thankfully for Microsoft Windows 10 users, an optional (and free) Windows 10 codec patch in April 2018 happens to provide support for HEIC. It is also supported by the Android operating system as of Android Pie.23 For once, it seems Apple decided to "play nice" in this regard, though it's quite possible Nokia had something to do with this, since they own the intellectual property rights to HEIF. Either way, .heic files are only created by Apple devices.

File Extension: .3gp

HEIF (2015)

High Efficiency Image File Format (HEIF) is a high-compression, lossy image container standard developed by MPEG (MPEG-H, Part 12 or ISO/IEC 23008-12). Freely available for personal use, it is based on the High Efficiency Video Encoding (HEVC) codec. There are nuances to HEIF files that vary based on operating system and application, and based on these variations you will find the container standard represented as different file extensions. If that sounds a little confusing, what it boils down to is all of these file standards use a form of HEVC codec stored inside a form of the HEIF container type. In spite of their differences in implementation, they all adhere to a very similar architecture and benefit from the pros and cons of HEIF.

MPEG-AVC is a hybrid codec combining AVC (H.264) inside a HEIF combination that sports ~2x data compression compared with native H.264 codecs. HEIF containers also often contain AV1 encoded content.

Technical information on HEIF may be found on Nokia's website.

File Extensions: .avci, .avcs; .avif, .heic, .heif

WebM (2010-2019)

Ostensibly shorthand for, "Web Matroska," WebM was created by Google after it acquired the intellectual property for the TrueMotion video codecs from On2 Technologies in 2010. Google applied its newly acquired IP to create the VP9 codec, followed a few years later by AV1. However, while these codecs were and are very competitive with comparable ITU-T and SMTPE standard codecs, they lacked an accompanying delivery mechanism, particuarly one geared toward web browsers and HTML5, as was applicable particularly to Google's long-term agenda. Thus, in 2010 Google also released the WebM container and file format.

Based heavily on the Matroska open-source standard, WebM is able to encapsulate VP8, VP9, and AV1 encoded video streams and combine them with synchronized audio using the (Vorbis or Opus) audio codecs.14>

File Extension: .webm

Ogg (2008/2002)

Ogg is an open source multimedia container format. It's difficult to pinpoint the timeline of when Ogg originated, as it depends on one's perspective. While the Ogg format was born in 1993, it was originally an audio-only container type. In 2002, a hacked version was created that supported video, but only a specific type of framework. Later, in mid-2003 it began to gain traction when two Internet RFCs were published related to the format. And finally, in 2008 RFC 5334 was published, making Ogg an officially sponsored multimedia container.

File Extensions: .ogg, .ogv [macOS: .ogx for multimedia storage file, .OgxX for multi-stream Ogg file]

MP4 (2003)

MP4 is a proprietary MPEG-4 multimedia container and file format. Technically, MPEG-4 Part 14 (ISO/IEC 14496-14:2003); it is best known by its official file extension: MP4. The most familiar multimedia container and file format, MP4 is so well known and popular that most modern codec and container types continue to work well with it, including MPEG's own more recent standards such as MPEG-H.

File Extension: .mp4

Matroska (2002)

Matroska is the gold standard of open source digital multimedia container types. It is a spin-off of the very first attempt to create an open-source multimedia container format that could capture, encode, and decode both video and synchronized audio in a single file. The original project was called the Multimedia Container Format (MCF). However, MCF was never completed, and as it foundered some of its early contributors left the project and created what would become the Matroska standard; the first truly open-source, flexible multimedia container format.

The platform's name is a play on words meant to reference Russian matryoshka dolls; hollow, wooden dolls that open to expose another doll that in turn opens to expose another, repeatedly. Just like the dolls of its namesake, Matroska containers may hold multiple other containers, which in turn may hold other containers or codecs. The system is purposefully designed to be very flexible and as agnostic as possible to the underlying content.

A relatively unique characteristic of Matroska - and one that has led to its longevity - is its inherent ability to remain future-proof to a large extent, due to its flexibile design.

Matroska containers are versatile when it comes to handling audio content as well, such as MP3, AAC, AC3, and DTS.

Matroska supports the following video codecs: VP8, VP9, AV1, H.262, H.263, H.264, FFV1, MPEG-4, MPEG-2, MPEG-1, RealVideo (5,8,9,G2), Quicktime, Theora (Xiph), ProRes24
File Extensions: .mkv

3GP (1998)

3GP (3rd Generation Protocol) is a MPEG-4 streaming container type optimized for low-bandwidth applications, such as mobile devices. It emphasizes small file sizes over quality. The 3GP standard belongs to the 3rd Generation Partnership Project.

3GP files may stream HEVC version 1 and version 2, H.264, H.263, MPEG-4 video, AMR narrow-band speech, AMR wide-band speech, and/or MPEG-4 AAC audio.25

File Extension: .3gp

.3gp may contain encrypted media

Legacy Container Formats

Video and multimedia container formats from the "Web 1.0" era. Remember the phrase, "surfing the Internet" and AOL? If you know what I'm talking about, that should be an appropriate mental frame to wrap your head around these video container formats. If you don't know what I'm talking about, carry on.

Adobe Flash Video (2007)

Adobe Flash Video or F4V was an evolution of Flash Video produced wholly by Adobe. Revised in 2007, the format supports 26

File Extensions: .flv, .f4v, .f4p, .f4a, .f4b

ASF (1998)

Advanced Systems Format (ASF) is a proprietary audio/video streaming container format released by Microsoft, Inc. in 1998. It is the de-facto successor to RIFF, and was one of the first video container formats to employ the use of Digital Rights Management; i.e., the practice of encrypting content in an effort to prevent its unauthorized duplication and distribution without a license.

ASF files may contain Windows Media (audio and/or video), AMR, and/or MPEG-4 video. ASF may also be used as a vehicle to record or play back H.32x content (e.g. H.323 or H.324); a family of multimedia protocols designed to encode and playback multimedia content across an IP packet network.

File Extensions: .asf, .wmv
ASF History and Version Confusion

There's a bit of history surrounding ASF. Advanced Systems Format (sometimes erroneously referred to as "ASF version 2") was finalized by Microsoft in 2004. The container's original name was Advanced Streaming Format (also sometimes referred to as "ASF version 1"), which appears in a 1998 draft standard published by the IETF. They are in fact the same thing. The fact is simply that between the 1998 draft and 2004, Microsoft changed the name.

The entire subject of the 1998 ASF draft standard is compounded by the fact Microsoft and Real Networks jointly published a draft of the same standard on the same day. Yet, the IETF version makes no mention of Real Networks, and instead mentions one of the authors - Eric Fleischman - who is not mentioned in the Microsoft/Real Networks document. Fleischman was a contributor to the standard, having spent time in 1997 working on it with Anders Klemets developing what would become the draft of the IETF/Microsoft/Real Network standards released in 1998.27

Inextricably, when Microsoft filed a patent application for ASF, it made no mention of Fleishman, Klemets, or Real Networks. Submitted on September 4, 2002 (4-1/2 years after the first draft was released), the patent was not granted nearly six (6) years later in mid-2008.31 The fact the company chose to pursue a patent for ASF in the first place tells you how competitive Microsoft believed the multimedia container landscape would become. It seems they viewed factors such as a file header as intellectual property, even in spite of the fact open source container formats such as Ogg and Au already existed well before ASF was even in its infancy.

Flash Video (1996)

Flash Video is an obsolete browser-based, proprietary streaming video container owned by Adobe, Inc. Developed by Macromedia before it was acquired by Adobe, Flash video is currently slated for deprecation on December 31, 2020.26 It is considered obsolete and prone to security risks. It's use is strongly discouraged.

Aside from Adobe's Flash Video format, Flash containers also support VP6, and a legacy video container called Shockwave (deprecated in 2019) and it may be stored in files per the MPEG-4 Part 12.

File Extension: .flv

Quicktime (1991)

In terms of popularity, Quicktime is to Mac and iOS what RIFF, ASF, and WMV are to Windows machines. Also known as Quicktime File Format or QTFF, it is sometimes referred to by old-school Windows users as the ".mov file" codec.

Quicktime is a digital multimedia container and file format. Originally, the format supported only Apple's proprietary MOV (.mov) file format, for use with Apple's Quicktime Player. Over time, Apple's QT Player has added the capability to also decode MPEG-2 (MPEG) and MPEG-4 (MP4) files.

MPEG-4's first edition was based largely on Quicktime, though both containers types have diverged since then.

File Extensions: .mov, .qt

Resource Interchange File Format (1991)

Resource Interchange File Format (RIFF) is the first proprietary audio/video streaming container format released by Microsoft, Inc. in 1991 and developed jointly by Microsoft and IBM. Based on IFF, it is the predecessor to AVI.

Interestingly, Audio Interchange File Format (AIFF) is a container format invented by Apple, intended to compete directly with RIFF. While nearly identical to RIFF, a minor structural change to its architecture makes the two formats incompatible. Also, unlike RIFF, the AIFF file format is limited to audio files only. You may learn more about it here.

File Extension: .riff

IFF (1985)

Interchange File Format (IFF) is the first generic digital multimedia container standard ever created. A joint effort between Electronic Arts and Commodore (one of the first consumer personal computer manufacturers). IFF is special. It's existence marks a watershed event in the history of personal computer systems. IFF was one of the first concerted attempt at rival corporations to make an effort to develop a standard of sharing data between software produced by different companies and on different platforms, and the first pertaining to consumer-grade personal computer systems.

The file extension of an IFF file is arbitrary. IFF uses so-called chunks to segment data in the file. Each chunk identifies itself as one of several pre-determined chunk types via a header, followed by a 32-bit length specification, followed by the actual data. Subsequent early container formats such as RIFF also follow this model.

A relatively unique characteristic of IFF is that it was never intended to be used as a multimedia container. Rather, it is a generic container format. Today, it is very rarely found in the wild.

File Extension: .iff

Oddball Container Standards

Container standards that probably never should have been created. While very rare, they occassionally surface and often confuse the general population; or at least their media players.

MPEG-DASH (2012): The Standard No One Cares About

MPEG-DASH is a veritable history lesson for multimedia standards bodies in how not to define a standard. MPEG-DASH is a modern container that delivers content via an HTTP transport stream rather than via a file. The "DASH" portion stands for Dynamic Adaptive Streaming over HTTP. The premise of DASH centers around the notion that traditional content delivery methodology is a poor model. To wit, DASH presumes:

  1. HTTP is an appropriate, universal method of content delivery to virtually any type of device
  2. Devices that receive content via HTTP are typically bandwidth-constrained
  3. The nuances of pre-existing content delivery methods have a tendency to dominate what limited bandwidth the device has
  4. All these factors taken together often result in a poor end-user experience

So, how does DASH solve these problems?

  • Client-server content delivery model
  • Content is chopped up into portions
  • Content portions are delivered to the client on-demand

When DASH was released in 2010, it presented a paradigm shift in thinking about content delivery. Particularly targetted at broadcast or streaming providers, the industry had previously viewed the control of content presentation as something owned solely by the client device. This is not a big deal when the client device is designed to accept, store, and process a single monolithic file - like a download - but it doesn't work well when the device has limited resources, such as most mobile devices.

Despite its name, the DASH standard does contain provisions for file storage, as obviously at some point the content is going to be stored in a file (at least temporarily). When that is the case, the data must be stored in one of two MPEG container types: MPEG-4 (MP4) or MPEG-2 Transport Stream (M2TS). MPEG-DASH is and has been pushed hard by MPEG, but the reality is it's still not used much and has proven to be a generally unpopular streaming format. Perhaps due to the fact Windows 10 is the only operating system that supports it natively. MPEG-DASH is defined under ISO/IEC 23009 Part 1.

Unfortunately for MPEG, today MPEG-DASH - just like RealVideo - seems like a solution in search of a problem. There really isn't a strong need for it in any respect, and there hasn't been since its inception. Why? Device manufacturers and content providers discovered that tried-and-true MPEG-4 works just as well, and is effectively capable of the same behavior when the devices on either end are setup for it. Furtheremore, there are plenty of RPC/WebRPC/HTTP-RPC formats28 which have been used for decades that effectively already exhibit the same type of behavior. While MPEG did quantify similar requirements under the umbrella of a defined standard specifically geared toward multimedia, it did so presuming the broadcast industry was not going to take steps to innvote solving these problems on its own. The result was by the time MPEG-DASH arrived, it wasn't really needed anymore. The world didn't ask for DASH from MPEG, and it still doesn't need it. Thus, as one might expect, it hasn't caught on.

MEPG-DASH is an excellent example of what happens to adoption when a market is already over-crowded with existing standards and yet-another-standard doesn't offer any substantially reason for incumbent users to change. Result: null set. Recently (in 2019), MPEG revised its DASH criteria to attempt to boost popularity once again, with the mentality of, "it didn't work between 2010-2019, but maybe it will work now." In an effort to gussie up support, MPEG has been driving MPEG-DASH hard with 3GPP since 2019. We shall see if it results in any appreciable adoption going forward. Admittedly, the mobile (cellular data) space is probably the only industry that might find it useful when compard to other formats, though it must be an uphill battle for MPEG to try and convince the industry that their own venerable MP4 format isn't good enough to begin with. Ater all, MP4 (MPEG-4) already has a streaming component with good compression options suitable for mobile devices, and to boot it offers good storage/higher quality options as well for non-mobile devices. What's not to like? Again, the world does not need DASH, and this is why the format has floundered for 10 years.

Even if one takes into consideration MPEG-DASH's attention toward the burgeoning HTML5 standard, Google beat it to the punch by aggressively promoting in situ web browser support for the TrueMotion codec intellect it acquired from On2 Technologies at the same time DASH launched in 2010. Google, however, has been far more successful (due in no small part to its considerable influence in the web browser space). But, aside from potential gerrymandering on Google's part, the fact is VP8, VP9, AV1, and Matroska are like chocolate and peanut butter. No one knows why, but gosh darn it they taste great together. Why mess with a good thing?

IVF (2011)

Indeo Video Format (IVF). Google resurrected this ancient (circa 1992) family of multimedia formats and codecs - in name only - for the purpose of transporting VP8 encoded data. In this context, it is sort of an ad-hoc container, acting as a very simple transport conduit supporting raw VP8 video only (i.e. no audio).29

File Extension: .ivf

PIFF: Protected Interoperable File Format (2009)

Perhaps the epitome of Microsoft's continuing efforts to fail at unseating MPEG as the leading authority in video codecs and containers, Protected Interoperable File Format (PIFF) bears the dubious honor of being the only container type to fall under not one, but two independent standards. Well, truthfully one of them isn't a standard, but what multi-billion dollar corporation is going to allow a minor detail like that to get in its way of attempted world domination??? It seems Microsoft still hadn't learned its lesson a few years earlier when it lost a battle with MPEG as it tried cheating MPEG out of market share with Microsoft's artificial MPEG-4 codecs.

Protected Interoperable File Format (PIFF) is an odd and confusing story about two tech giants who decided to release different container formats with the same name and same ISO file extension. Only, you can't have more than one use for the same file extension if they are ISO certified, right? I mean, isn't that the point of having standards organizations? Indeed, that's true.

MPEG registered a .piff file extension with ISO/IEC in 2010. However, prior to that time, Microsoft promoted a proprietary file format it created with the same name, called PIFF or .piff. In 2009, Microsoft claimed its PIFF format was ISO base media file format compliant and released developer documentation claiming this as fact, even though they had not cleared the concept with ISO/IEC at the time. Making matters even more confusing for end users, Microsoft released a developer specification document for their version of RIFF in 2010 - the same year MPEG received the green light to adopt the .riff file extension for its MPEG-4 files (MPEG-4: Part 12) from - you guessed it - the ISO.

Microsoft went on to apply their RIFF standard to Windows Media Player Professional files (with file extension .wma). Any way you look at it, RIFF was a disaster from the perspective of software developers and end users. To this day, the usage of WMA compression in ISO base media file format has never been publicly documented (e.g. via ISO/IEC), meaning platform support for any given WMA file is effectively a crapshoot. While .wma files are likely to be encoded with the lossy or lossless Microsoft WMA encoder, it's always possible they are actually RIFF encoded or MPEG-4 encoded files. This means .wma files have a higher probability of not being playable on any given decoder a user may happen to attempt to use to play them. It also means multiple files from multiple sources could potentially all have the .wma file extension and yet they could have been encoded with any of several different encoders resulting in files that are incompatible with some other decoders. As I said, WMA is a crapshoot (from a listener's perspective).

File Extension: .piff

AVIX (1996)

AVIX (AVI Extended) is a RIFF based container file format introduced as an extension to the AVI container standard in 1996.30 AVIX was a more concerted attempt than RNMP to codify a RIFF-based container file format capable of handling file sizes exceeding 1 gigabyte (1 GB).

File Extensions: .avi, .avix

VOB (1992)

VOB stands for Video OBject. It is an old MPEG-2 container format designed specifically for DVDs. VOB files (.vob) contain digital video, audio, sub-titles, DVD menus, and DVD navigation metadata. It's easiest to simply think of a .vob file as DVD content presented as a file. Due to its age and the time in which it was developed, VOB has some limitations that seem archaic by today's standards. For example, a VOB file is limited in size to 1 gigabyte (1 GB).

If you encounter .vob files, you might also find .ifo and/or .bup files with them. IFO files can be thought of metadata and navigation data for a DVD, and often accompany sets of VOB files. IFO files normally describe the content and are text-based. BUP files are simply backups of IFO files.

File Extension: .vob

RNMP (1992)

RIFF Multimedia Movie or RNMP is a RIFF-based video/animation file format. An obscure adaptation of the RIFF container format, RNMP could be a predecessor of AVI. The MMM Movie file extension (.mmm) is a related (and also highly obscure) RNMP file type.

File Extensions: .mmm, .rnmp

Endnotes

1 ITU-T. (17 April 2020). Wikipedia. Wikimedia Foundation.

2 Mobile device support for audio and video codecs is limited, due to their inherenet limited hardware resources, relative to PC platforms.

3 Hardware limited. Android KitKat or later required on Qualcomm chipsets. Devices using Samsung Exynos require Android Lollipop or later operating system. Source: Ganesan, Robert. (20 June 2019). Chrome on Android does not work with H.264 codec projects. Vonage.

4 macOS previously supported, but dropped support as of October 2019 (macOS version 10.15, aka Catalina). Source: Formats compatible with macOS Catalina. (10 April 2020). Apple, Incorporated (corporate website).

5 Beyond HEVC: Versatile Video Coding project starts strongly in Joint Video Experts Team. (27 April 2018). ITUNews.

6 The so-called "H. series" is a set of multimedia format standards derived from the ITU Telecommunication Standardization Sector (ITU-T).

7 Ohm, Jens-Rainer and Sullivan, Gary. (July 2005). Advanced Video Coding. The Moving Picture Experts Group.

8 Bilodeau, Samuel. (5 April 2017). Delivering 8K using AVC/H.264.

9 Syed, Mahbubur Rahman. (30 June 2008). Multimedia Technologies: Concepts, Methodologies, Tools, and Applications. p. 1,428.

10 AVIF (AV1 Image File Format): experimental support. (May 2020). Bugzilla.

11 Francis, Dinsan. (24 June 2020). Chrome to Add Support for AVIF Image Format. Chrome Story.

12 Ozer, Jan. (27 June 2017). The HEVC Soap Opera: Keeping Track of Players and Costs. Streaming Media Magazine.

13 REALTIME ADAPTIVE STREAMING LLC v. APPLE INC. (30 November 2017). Case 1:17-cv-02869-MSK-STV. United States District Court for the District of Colorado.

14 WebM Project. (n.d.). "WebM FAQ".

15 Microsoft Media Foundation. (31 May 2018). Microsoft Corporation.

16 XMP Specification Part 3: Storage in Files. (2020). Adobe.

17 Daede, Thomas; Egge, Nathan E.; Montgomery, Christopher; Terriberry, Timothy B.; Valin, Jean-Marc. (2016). Daala: A perceptually-driven still picture codec. 2016 IEEE International Conference on Image Processing (ICIP). Institute of Electrical and Electronics Engineers (IEEE).

18 VC-2 HQ. (2019). Silex Insight.

19McLellan, Cindy. (3 November 2011). AuthentiCity: The City of Vancouver Archives Blog. City of Vancouver Archives. Vancouver, Canada.

20B., Peter; Jaks, Marion; and Lewetz, Hermann. (4 September 2013). The archivist's video codec and container FAQ.

21Purnell, Graham. (3 May 2019). Comparative analysis of uncompressed AVI and FFV1 video. Digital Preservation Coalition.

22 Protected Interoperable File Format. (8 September 2009). Microsoft Corporation.

23 HEIF Imaging. (4 May 2020). Android Open Source Project.

24 Codec Mappings. (n.d.). Matroska.

25 3GPP TS 26.244 version 2.0.0 “3GPP file format (3GP)" (Release 6). (March 2004). 3rd Generation Partnership Project (3GPP).

26 Warren, Tom. (25 July 2017). Adobe will finally kill Flash in 2020. The Verge.

27 Fleischman, Eric and Klemets, Anders. (14 November 1997). Recording MBone Sessions to ASF Files.

28 Remote Procedure Call/Web Remote Procedure Call/HTTP Remote Procedure Call all refer to a process of data exchange between a remote device (client) and a data repository and delivery device (server). RPC uses a schema or language pre-agreed to by both parties that allows the client device to request content from the server. The concept was invented in the 1960's, in the early years of networking and the Internet.

29 Hassey, Phil. (2 February 2012). How to create and play IVF / VP8 / WebM / libvpx video in OpenGL. Phil Hassey blog.

30 OpenDML AVI File Format Extensions. (28 February 1996). OpenDML AVI M-JPEG File Format Subcommittee.

31 ADVANCED STREAM FORMAT (ASF) DATA STREAM HEADER OBJECT PROTECTION (US Patent number US 7,401,221 B2).

References