Image formats comparison

Introduction

This study compares 8 differents image formats, AOM AV1, BPG, Daala, FLIF, JPEG XR, JPEG 2000, JPEG and WebP. We use five algorithms in order to compare each format:

Materials

Image set

The image set is comprised of 50 images from the subset 1 and subset 2 maintened by Xiph. All images are YCbCr 4:2:0 Y4M files.

Codecs

Metrics

Tools

Methods

Image conversion

Each Y4M image is exported to 4:2:0 PNG, YUV and PPM files with FFMPEG:

ffmpeg -loglevel quiet -y -i [input] -pix_fmt yuv420p [output]

Image compression

All images are compressed losslessly and over a range of qualities for each codec:

The Python script used to generate the compressed images are available on the GIT repository.

Image selection

The images which will be displayed on the website are then chosen among all compressed images, using the following criteria:

The Python script used to select the compressed images are available on the GIT repository.

Encoding and decoding speeds:

Lossless compression and speed

For each codec and image, the encoding and decoding speeds for lossless compression are sampled using Python timeit. The arithmetic mean of encoding and decoding speeds are calculated over the entire image set. We then determine a Weissman score for each codec using the following formula:

where r is the compression ratio over PPM filesize, T the time required to compress, ̅r and ̅T the same metrics for the standard compressor, and alpha is a scaling constant.

The standard compressor used is the compression of a JPG image using mozjpeg.

Lossy metrics

For each codec and image, we apply the following metrics, Y-SSIM, RGB-SSIM, Y-MSSSIM, PSNR-HVS-M and VMAF, over 15 image samples of increasing quality. For VMAF, we use the trained model nflxall_vmafv4.pkl given by Netflix.

For each sample, we first decode the compressed image, then export the resulting file to 4:2:0 Y4M and YUV format using FFMPEG (ffmpeg -loglevel quiet -y -i [input] -pix_fmt yuv420p [output]). Finally we apply the metrics over each sample, comparing it to the original image.

For each codec, we calculate the arithmetic mean of each metric over the entire set of images, weighted by the area of the corresponding picture, for the 15 samples of increasing quality:

We also determine the average bits per pixel for each quality sample:

Results

Raw data

The following archives contain the raw data in csv format for subset1 and subset2:

Lossless compression ratio and Weissman score:

codec avg. compression ratio avg. space saving wavg. encode time wavg. decode time Weissman score
daala 2.798 64.26% 0.8049 0.7280 3.3381
vp9 2.905 65.58% 3.9375 0.4193 2.8011
av1-20160930 2.912 65.66% 4.7511 0.4838 2.7455
av1-20170809 2.984 66.49% 20.4157 0.6900 2.4003
kdu 1.564 36.06% 0.3875 0.2892 2.0946
jxr 1.560 35.89% 0.4058 0.3628 2.0730
av1-20180222 2.943 66.02% 97.6492 1.0156 2.0444
flif 2.473 59.57% 22.1088 4.3042 1.9732
openjpeg 1.564 36.06% 2.1181 1.3794 1.6300
webp 2.124 52.93% 37.9675 2.4799 1.6079
bpg 1.150 13.03% 3.7503 3.5251 1.1151
mozjpeg 1.137 12.05% 8.7385 0.4144 1.0000

Lossy compression and speed

Encoding time in function of bits per pixel

Lossy metrics

For each comparison algorithms, we plot the quality in dB in function of the mean bits per pixel on a logarithmic scale. We can then visualize which codec gives the best quality at a given bit per pixel (top left is better).

Bits per pixel at equivalent quality according to VMAF

Bits per pixel at equivalent quality according to VMAF

Bits per pixel at equivalent quality according to Y-PSNR-HVS-M

Bits per pixel at equivalent quality according to Y-PSNR-HVS-M

Bits per pixel at equivalent quality according to Y-MSSSIM

Bits per pixel at equivalent quality according to Y-MSSSIM

Bits per pixel at equivalent quality according to Y-SSIM

Bits per pixel at equivalent quality according to Y-SSIM

Bits per pixel at equivalent quality according to RGB-SSIM

Bits per pixel at equivalent quality according to RGB-SSIM