Premultiplication

I’ve just published a new video tutorial explaining the concept and maths behind premultiplication, and premultiplied images. This post is basically the text version of the video. I wrote it as a script before I recorded the video and I know some people find it easier to follow written text so I thought I’d publish this too.

Here is the video version:

Understanding what premultiplication is, and how it works in compositing is important. It is a fundamental concept that is a big part of the most basic compositing operation, merging one image over another image.

Luckily, the concept is pretty simple and uses very basic maths.

When the concept of premultiplication and how mattes actually work was explained to me on the first day of my compositing course at Escape Studios, a light bulb went off in my head. I had spent years using After Effects and I had always wondered how it was possible to have a ‘transparent’ pixel, but I just assumed it was some complicated, or proprietary, process that each piece of software did differently. But once it was explained to me, it blew my mind that not only was it so simple I could fully understand it, but I could actually do the maths and calculate the resulting pixel values myself.

Once I realised this, it meant I had the confidence to start trying to understand and troubleshoot other issues I faced as I learned more about compositing.

So in this article I am going to explain exactly what a premultiplied image is, and how we create one. And then in the next article I will explain the steps that Nuke goes through to composite a premultiplied image on top of a background.

So let’s get started with premultiplication.

The transparency of an image in compositing is controlled by a matte. A matte is a single channel black and white image, where the white pixels represent the areas of the image that are to be fully opaque, black pixels represent the completely transparent areas, and grey pixels represent semi transparency.

A premultiplied image is an image that has had the matte applied to the colour information AND has the same matte included in the alpha channel.

Lets see how we can create this.

The matte can be generated in multiple ways. It could come from rotoscoping, keying, or it could be generated by the render software if we are working with a CG image.

Once we have the matte for the foreground, we need to apply it to the image.

But before we do that, we need to understand how these black, white, and grey values are represented in the software.

If you are used to working in Photoshop or After Effects, you might be used to seeing pixel values that range from 0 to 255 if you are working on an 8 bit image, or 0 to 32,768 for a 16 bit image. In these cases, 0 represents black and the maximum number, either 255 or 32,768, represents white. If you are working in Nuke, or if you switch After Effects to 32 bit mode, and you sample pixels, you will see that 0 still represents black, but white is now 1, and this is the same regardless of the bit depth of the image you bring into the software.

Representing the values from black to white as 0 to 1 makes what we do next much easier to understand and follow.

This 0-1 representation works for each colour channel too. 0 in the red channel is no red at all, 1 is full red. Same for the green and blue.

Now all we need to do is remember some primary school multiplication maths.

If we multiply a number by 1 it stays the same. So 1 x 1 = 1.

If we multiply a number by 0 we get 0. So 1 x 0 = 0.

And if we multiply something by a fraction we get that fraction of the original number. So 1 x 0.5 = 0.5, and 0.5 x 0.5 = 0.25.

So you should now be able to see where we’re going with this.

To apply the matte to the image we need to multiply each of the red, green and blue channels by the alpha channel

Let’s use this simple matte and the colour bars below as our first example.

This matte is rendered only into the alpha channel for now, but if we look at that alpha, we can see there are four distinct blocks. At the top, the matte is completely solid, with values of 1. The next blocks down has a value of 0.5, then 0.25, and the bottom block is 0.

If we look at the colour bars below, they have max values of 0.5 by default, but I have graded them up to 1 for now. So the red bar has values of 1 in the red and 0 in green and blue for example.

So now we need to somehow multiply the values from the matte with the values in the colourbars. There are multiple ways to do this, but let’s start by using a multiply merge operation. We can create a merge node set to multiply, and hook up the matte to the A input and the colour bars to the B input. 

Now if we look at the merge node, we’ll see that everything has gone black. This is because the merge node is multiplying each pixel of the red channel of the A input, with the corresponding red pixel of the B input, and the green by green and blue by blue, and in this case the matte only has information in the alpha channel. We can change this with a shuffle node. 

Connect the shuffle node between the matte and the merge node, and shuffle the alpha channel into the R, G, and B channels.

Now we can view merge node again and see how our matte is affecting the image.

We can see that at the top of the image, where the matte is 1, nothing has changed, because anything multiplied by 1 stays the same.

In the next section, where the matte is 0.5, everything has halved.

Below that, everything has quartered, because the matte is 0.25

And at the bottom, where the matte is 0, everything has become 0, or effectively vanishes.

So, now we have applied the matte to the colour information, we almost have a premultiplied image. But we said that a premultiplied image has to include the matte in the alpha channel, and if we look at the alpha channel below the merge node it is empty. That’s because there was no information in the alpha channel of the B input to start with, and 0 multiplied by anything is still 0.

So we need to copy the matte from the A input into the alpha channel of the B pipe with a copy node. And now we have a premultiplied image!

But…. we can get the same result with far fewer nodes in Nuke. Let me show you how I would actually do this when I’m working on a composite.

We can actually start with the copy node. By default, this copies the alpha channel from the A input into the B stream, so we can see that we now have the matte in the alpha channel of the image we are working with.

Now we need to multiply the red, green and blue channel by the alpha channel. For that there is the premult node. By default this takes the alpha channel already in the stream, and multiplies it with the R, G, and B channels, giving us the same result as the merge node in the previous setup, but with fewer nodes and fewer clicks.

And here is an example using a more realistic set of images. The bird is a photo, and the matte was created with a combination of keying and roto shapes and rendered out into a separate file. In this case the matte was rendered into all four channels, the R, G, B, and A.

If we sample the pixel values of this image we can see that where the alpha channel is 1, the RGB values don’t change after the premult because anything x 1 stays the same.

The pixel values where the mask is 0 become 0, because anything multiplied by 0 becomes 0.

If we sample a pixel on the edge of the bird, where the alpha is almost exactly 0.5, we can see that after the premult node the R, G, and B values have basically halved. Where the red channel was 0.09 it becomes 0.046.

We could render this image out and, as long as we choose a format that supports an alpha channel, like an EXR or TIFF, we could import this render into any app that supports premultiplied images.

So now we have a premultiplied image. And hopefully you understand how we got it. As compositors, we will often be creating our own premultiplied images, when we are working with roto, or pulling keys. But we will also work with images that have already been premultiplied, like CG renders or motion graphics.

Once we have a premultiplied image, we can use an over operation to just place that image in front of another one. And in the next video I will show you that the over operation relies on premultiplication, and is almost as simple as the premultiplication process.

I hope this video helps you understand one of the fundamentals of compositing. If you have any questions, please post them in the comments below.