Video formats comparison

Introduction

This study compares 4 differents video encoders, AOM AV1 libaom, Google VP9 libvpx, x264 and x265. We use five algorithms in order to compare each format:

• VMAF: Video Multi-Method Assessment Fusion: http://techblog.netflix.com/2016/06/toward-practical-perceptual-video.html
• Y-SSIM: Structural Similarity algorithm applied to the luma channel
• RGB-SSIM: Structural Similarity algorithm applied to the R, G and B channels
• Y-MSSIM: Multi-Scale Structural Similarity algorithm applied to the luma channel
• PSNR-HVS-M: Peak Signal to Noise Ratio taking into account Contrast Sensitivity Function (CSF) and between-coefficient contrast masking of DCT basis functions

Materials

Video set

The video set is comprised of 30 videos from objective-1-fast by Xiph. All videos are YCbCr 4:2:0 Y4M files.

Encoders

• Alliance for Open Media AV1 libaom: https://aomedia.googlesource.com/aom/. The versions used is built from GIT revision 935d6d479231c6b5f11d19e56aa931c2976262e7 (april 2018).
• Google VP9 libvpx: https://chromium.googlesource.com/webm/libvpx. The version used is 1.7.0.
• x264: https://www.videolan.org/developers/x264.html. The version used is built from GIT revision 7d0ff22e8c96de126be9d3de4952edd6d1b75a8c.
• x265: https://www.videolan.org/developers/x265.html. The version used is built from GIT revision 3cf3839f82bb177c43449ab10792c184c4485d8b.

Metrics

• The VMAF (Video Multi-Method Assessment Fusion) metric is computed using vmafossexec, provided by Netflix: https://github.com/Netflix/vmaf. The version used is built from GIT revision 7ebbde0c64493af978da66cb7ebe2946fb12dec2.

  vmafossexec compares two YUV files, given their subsampling and dimensions.

• Y-MSSIM, Y-SSIM, RGB-SSIM and PNSR-HVS-M are computed by the tools dump_msssim, dump_ssim and dump_psnrhvs, provided by the Daala repository: https://git.xiph.org/?p=daala.git. The version used is built from GIT revision 05243557bc3e59872fd043c99dc4c17ca33bcb1b.

  Each metric compares two Y4M files.

Tools

• ffmpeg is used for video formats conversion. The version used is ffmpeg 3.3.5.
• FFMPEG's ffprobe is used to determine the width, height and number of frames of the videos.
• Python 3 timeit is used to mesure computing times.
• Python 3 Pandas and Numpy are used to compute means.
• Python 3 Matplot is used to plot graphs.

Methods

Video conversion

Each Y4M videos is exported to 4:2:0 10 bits Y4M with FFMPEG:

ffmpeg -y -i [input] -pix_fmt yuv420p10le -strict -1 [output]

Image compression

All videos are compressed over a range of qualities for each codec:

• AV1 libaom:

  • between q=12 and q=60, with a step of 4:

aomenc --cpu-used=4 --tile-columns=4 --passes=2 --pass=1 --bit-depth=10 --input-bit-depth=10 --end-usage=q --cq-level=$q --fpf=[output].log -o [output] [input(Y4M_10bits)]

aomenc --cpu-used=4 --tile-columns=4 --passes=2 --pass=2 --bit-depth=10 --input-bit-depth=10 --end-usage=q --cq-level=$q --fpf=[output].log -o [output] [input(Y4M_10bits)]

• VP9 libvpx:

  • between q=12 and q=60, with a step of 4:

vpxenc --tile-columns=4 --row-mt=1 --passes=2 --cpu-used=2 --bit-depth=10 --input-bit-depth=10 --profile=2 --end-usage=q --cq-level=$ -o [output] [input(Y4M_10bits)]

• x264:

  • between q=12 and q=40, with a step of 2:

x264 --profile high10 --preset slower --input-depth=10 --output-depth=10 --crf $q -o [output] [input(Y4M_10bits)]

• x265:

  • between q=12 and q=40, with a step of 2:

x265 --profile main10 --preset slower --input-depth=10 --output-depth=10 --crf $q -o [output] [input(Y4M_10bits)]

The Python script used to generate the compressed videos are available on the GIT repository.

Encoding and decoding speeds:

For each codec and videos, the encoding and decoding duration are mesured using Python timeit. It is then converted in frames per minute.

metrics

For each codec and videos, we apply the following metrics, Y-SSIM, RGB-SSIM, Y-MSSSIM, PSNR-HVS-M and VMAF, over videos samples of increasing quality. For VMAF, we use the trained model vmaf_v0.6.1.pkl given by Netflix.

For each sample, we first decode the compressed videos to 4:2:0 10bits Y4M then export to YUV format using FFMPEG (ffmpeg -y -i [input] -pix_fmt yuv420p10le -strict -1 [output]). Finally we apply the metrics over each sample, comparing it to the original video.

For each codec, we calculate the arithmetic mean of each metric over the entire set of videos, weighted by the number of pixels of the corresponding video, for the samples of increasing quality:

$$\overline{metric}_{quality\ q} = \frac{ \sum\limits_{videos=1}^{30} metric_{qp} \times pixels_{p}}{\sum\limits_{p=1}^{50} area_{p}}$$

We also determine the average bits per pixel for each quality sample:

$$\overline{bpp}_{quality\ q} = \frac{ \sum\limits_{videos=1}^{30} filesize_{qp}}{\sum\limits_{videos=1}^{30} pixels_{p}}$$

Results

Raw data

The following archives contain the raw data in csv format for objective-1-fast:

• Subset1 (updated April 2018)

Compression speed at 1080p:

Metrics

For each comparison algorithms, we plot the quality in dB in function of the mean bits per pixel on a logarithmic scale. We can then visualize which codec gives the best quality at a given bit per pixel (top left is better).

Bits per pixel at equivalent quality according to VMAF

360p

720p

1080p

Bits per pixel at equivalent quality according to Y-PSNR-HVS-M

360p

720p

1080p

Bits per pixel at equivalent quality according to Y-MSSSIM

360p

720p

1080p

Bits per pixel at equivalent quality according to Y-SSIM

360p

720p

1080p

Bits per pixel at equivalent quality according to RGB-SSIM

360p

720p

1080p

CRF equivalences and bitrate reduction at 1080p

In the following table, we try to find equivalences between the CRF values of the various encoders according to a selected metric. We then calculate the expected bitrate reduction for this comparable CRF value.

For example, a x264 encode at CRF 20 would give a PSNR-HVS-M of 42.80. To obtain the same quality with VP9, one should look at the intersection of CRF 20 and "libvpx crf according to psnr-hvs-m", which gives an equivent CRF of 30.60. The column "libvpx % reduction according to psnr-hvs-m" then gives the expected reduction of bitrate, -21.294%.