Inverse tone mapping - upscaling SDR content to HDR

Introduction

The explosive growth of HDR display devices has yet to be matched by an equivalent growth in HDR content despite considerable efforts from studios, operators and more recently from live sports distributors.
Each time there has been a significant improvement in Television technology, the content industry has found a way to upconvert from the old technology to the new, such as spatially upconverting Standard to High-Definition Television.
Still, most existing legacy video content is either recorded or mastered in a lower standard dynamic range (SDR) quality. To show standard dynamic range content on High Dynamic Range displays, the content needs to be upconverted using a so-called ‘inverse tone mapping’ algorithm.  You may also see this process referred to as ‘upmapping’, or less scientifically as ‘upconverting’ or ‘upscaling’ but they are all referring to the process of creating HDR from an SDR source.

There are two primary use cases for needing to convert SDR content to HDR

• Where an archive production is only available as SDR, and the entire production needs to be remastered for HDR distribution/rerelease
• Where selected shots are only available in SDR, such as where archive footage or specialist cameras are in SDR, and these need to be integrated into an otherwise HDR-native production

Whether this is converting a complete production, or just certain shots, in order to use SDR content in an HDR production then the following steps may be necessary depending on the source content format

• Deinterlacing
• Resolution scaling
• Color space conversion
• Inverse tone mapping

 

Technical Characteristics of Professional HD SDR

High Definition SDR content usually has one of the following characteristics:

• 1920 x 1080 pixels
  • 50/59.94Hz Interlaced
  • 23.98/24/25Hz Progressive or PSF (progressive segmented frame)
  • 50/59.94Hz Progressive
• 1280 x 720 pixels
  • 50/59.94Hz Progressive

Apart from studio content mastered as full range RGB or YCbCr 4:4:4, the most likely source format comes from a 10bit YCbCr 4:2:2 source, often that has passed through HD-SDI, using the SMPTE code range representing black at code value 64 and nominal peak at code value 940 (10bit). Note that signals between 4 and 64 (the super-blacks) and 940 and 1019 (super whites) are valid in HD-SDI and often contain information which will allow a higher quality HDR conversion.

NOTE:  Any equipment which limits (clips) the video signal to 64 to 940 should be avoided.

Irrespective of the resolution, frame rate, field order or color representation, all SDR content is assumed to conform to the non-linear transfer characteristic defined in ITU-R BT.1886 with color primaries according to ITU-R BT.709or ITU-R BT.2020.

 

Converting from SDR to HDR

De-interlacing & upscaling HD SDR content

HD SDR content was often captured in the interlaced format and this content must be converted to progressive before any further processing takes place. When dealing with 59,94Hz sources that were originally created from native movie or episodic 23.98Hz content, one would need to remove the 3:2 pulldown to track down to its source content.
As HDR usually comes with the need for an UHD deliverable, HD content must be upscaled accordingly.
 

Characteristics of SDR Archive Content Types

SDR legacy content can reasonably be divided into 3 main content types:

• Mastered (post produced) SDR content (Movies, Episodic, Documentaries, etc.)
• SDR live captured content with controlled lighting (Studio content, Concerts, Indoor Sports, etc.)
• SDR live captured content with uncontrolled lighting (Live Outdoor Sports, etc.)

 

Mastered SDR Content

Such content was traditionally shot on film or more recently using digital motion picture cameras and was color graded in post-production referring to a given SDR display technology (cinema or TV). Usually, this content is available in RGB or YCbCr 4:4:4 (or 4:2:2) format on videotape or digital storage.
 

SDR Live Content with Controlled Lighting

Shot on three sensor broadcast cameras usually in interlaced format.
Studio lighting is adjusted to ensure that the dynamic range of the scene approximates that of an SDR camera. Highlights are often compressed to create a more pleasing look.

 

SDR Live Content with Uncontrolled Lighting (outdoors)

Shot on three sensor broadcast cameras usually in interlaced format
Lighting may vary dramatically over time both in terms of brightness and the color of the illumination (clouds, rain, sun going down, floodlights...)
As multiple cameras are usually operated by a single shader, highlights are often clipped and the sky is frequently desaturated.
 

Up-mapping SDR Archive Content to HDR

There are two fundamentally different ways to approach this issue:

1. The accurate (mathematical) conversion from the SDR to HDR format is defined in the document “MovieLabs Best Practices for Mapping BT.709 Content to HDR10 for Consumer Distribution” https://www.movielabs.com/ngvideo/MovieLabs_Mapping_BT.709_to_HDR10_v1.0.pdf
   This preserves the original look of the SDR image within HDR. This means that the resulting HDR watched on an HDR display should look identical to the original SDR watched on an SDR display.  However, it does not change the look of the image and therefore may not blend well with other native HDR content. Such conversions are also referred to as direct mapping conversions, a terminology used in the Ultra HD Forum Guidelines.

 

2. To perform an inverse tone-mapping of the SDR content to create an “HDR look”.  The Ultra HD Forum refers to this type of conversion as up-mapping. This is often the desired approach as it will enable SDR clips to more seamlessly blend into an HDR production, or in the case of a complete SDR archive production creating this “HDR look” is often the primary reason for performing the remastering. 

 

SDR Round-Tripping through HDR

Whenever a production is created in HDR, it is important to remember that this content will still be viewed in SDR by some consumers.  This usually occurs when:

• The studio requires an SDR deliverable to be produced in post-production
• When the OTT/broadcast service derives an SDR version automatically, using the Dolby Vision metadata, for distribution to SDR-only devices or on SDR-only services

Where SDR content has been inverse tone mapped to HDR, and then is subsequently tone mapped back to SDR, this is known as “SDR round-tripping through HDR”.  In most cases, the aim is for the up-mapping process used to convert SDR to HDR to be visually opposite of the down-mapping process used to convert HDR to SDR, in order for both the original SDR and the derived SDR to have the same look.
For option 1 above, as no non-linear image processing has been applied, the process can be mathematically inverted and the SDR source will be identical to the SDR output.

For option 2, it is more complicated with the desire to achieve symmetrical non-linear up and down-mapping in order to maintain good SDR quality and even then there will be some minor differences, but in a well-designed and correctly used system these can be almost lossless visually.  Careful use of the Dolby Vision trims can ensure that the derived SDR can look very similar to the original SDR source.
The following section looks at what techniques can be used to optimize the up-mapping performance.

 

Up-mapping Mastered SDR Content

As existing SDR mastered content was often creatively color graded from a higher dynamic range source with a wider color gamut, be that digital or film, it is preferable to use the original source materials or a scene referred archival master but this is often not practical or simply not available. However, a manual regrading for a pleasing picture at a higher luminous output is very time consuming and labor-intensive. Stretching gamma or expanding just the highlights would distort the tonal balance and the relation between the colors and therefore distort the creative intent. It is usually required to maintain all color relationships and to recreate an image that matches the essence of the original one, therefore any SDR originated content must be converted from gamma to PQ first. In PQ the image can be scaled to be brighter overall while maintaining the tonal balance, preserving the original creative intent.

Not surprisingly, due to the time-consuming nature of this process, and the huge amount of SDR library content, automatic processes are being added to a range of tools to assist with this conversion.  We include some examples below, but whether this conversion is done fully manual, or in-part fully automatic, it is recommended that extreme care is taken to monitor both the HDR output and the round-tripped SDR.

Movie and episodic broadcast deliverables were often mastered in the following formats:

• 1920x1080 23.98/24/25Hz RGB or YCbCr 4:4:4 progressive or segmented frame 10 or 12bit RGB full range
• 1920x1080 23.98/24/25Hz YCbCr 4:2:2 10bit either progressive or segmented frame. Either full range or SMPTE range versions of this type of content (or a mixture of the two) can be found in archives.

If only an 8bit version is available, then it is likely that some additional processing will be required in order to avoid introducing artifacts, in particular banding / contouring artifacts in the HDR output. This might include noise reduction of the source and subsequent dithering of the output.

When the source uses the SMPTE video range, additional information may be present below code value 64 and above code value 940 (in 10 bits). This out-of-range information should not be clipped to the SMPTE range as it may contain not only luminance highlights but can also allow for the recovery of some color information beyond ITU-R BT.709. 

 

Up-mapping Live SDR Content created under Controlled Lighting

In a studio production environment, the lighting director normally takes great care to illuminate the studio in such a way that the dynamic range on set closely matches the capabilities of the SDR cameras in use.  Cameras are often set up to use a “knee” function to allow creative control over the highlights while keeping the skin tones at the nominal level. More dynamic range can be obtained by undoing the “knee” function. Unfortunately, there is no standardized knee function so finding an optimum undo function requires experimentation.
 

Up-mapping Live SDR Content created under Uncontrolled Lighting

Sport is the most likely content in this category. Not only does the lighting vary dependent on weather and time of day but each shader / vision engineer has to control multiple cameras. Under these circumstances, the shader only has time to vary the basic camera controls to keep the skin and mid-tones (e.g. grass) consistent between the cameras under his control. In this situation, it is very unusual for a shader to adjust the camera knee (they have no time to do it), so consequently there is normally a hard clip at the peak white level.

 

Examples of equipment commonly used for SDR to HDR Up-mapping

The following list is for information only and does not imply an endorsement of any of the manufacturers mentioned below.

Non-realtime, software based HDR to SDR converters 

Non-realtime inverse tone-mapping algorithms require high computation time but usually come with improved output quality. Such solutions are often part of a wider set of color grading and mastering functionalities and hence offer flexible parametric color processing capabilities with manual adjustment capabilities and allow for the mapping of various camera original (scene-referred) and graded (display-referred) sources to the desired HDR output. Such software- solutions are available from vendors like Colorfront and others.

 

Realtime HDR to SDR converters

Realtime converters are usually designed for live/broadcast use cases but can be used wherever needed.  Multiple solutions are available from vendors like Sony, Aja, b<>com or Cobalt. Those tools work either by using a set of pre-defined LUTs or let the operator choose from numerous options for the input, conversion and output parameters. Some solutions even provide a content adaptive conversion from SDR to HDR and adapt the conversion parameters dependent upon the source content and some manual “look” control.
 

Summary

It is recommended that whichever solution is chosen for conversion from SDR to HDR, that the following process is followed

·      Deinterlace first if necessary
·      Convert to the required frame rate and resolution
·      Then convert from SDR to HDR

Care should be taken to monitor both the HDR results, and the round tripped SDR, to ensure that the results are acceptable and fit alongside any HDR native content that is being used in the production.

Using the conversion method described in the MovieLabs document is a safe solution, but this may not be the preferred approach as it does not add any “HDR-ness” to the SDR up-mapped image. This content may stand out particularly when SDR sections are regularly cut into HDR native elements.

Where controls are available to increase the HDR effect from SDR sources, it is recommended that these are used conservatively, especially for live-captured content where bright areas (such as the sky) are usually overexposed. In the up-mapped HDR version, these SDR overexposed areas can visually dominate the image, not only changing the creative intent but also creating an unpleasant user experience. Achieving acceptable results will require testing of the chosen solutions with relevant content types to find the right set-up that works for your production but taking this time to experiment can mean that even these SDR sources can be used to add impact to your HDR productions.