Gradient Domain Fusion

Written on September 28th, 2018 by Andrew Campbell

This post explores an interesting application of gradient-domain processing known as Poisson blending. The goal is to seamlessly blend an object or texture from a source image into a target image. The simplest method would be to just copy and paste the pixels from one image directly into the other, but this would result in obvious seams. One approach is to use multiresolution blending (a technique discussed in the last blog post).

We will look at another approach using the gradient of the image. The idea is that the gradient of the image is more important than the overall intensity or individual colors. The problem is then finding values for the target pixels that maximally preserve the gradient of the source region without changing any of the background pixels.

Note: This was originally done as part of a project for CS194-26, a course on computational photography at UC Berkeley.

Poisson Blending

Background

Poisson Blending formulates the gradient blending problem as a least squares problem.

Given the pixel intensities of the source image \( s \) and of the target image \( t \), we want to solve for new intensity values \( v \) within the source region \( S \) (as specified by a mask):

$$\mathbf{v} = \text{argmin}_{\mathbf{v}} \sum_{i \in S, j \in N_i \cap S} \left( (v_i - v_j)^2 - (s_i - s_j)^2 \right) + \sum_{i \in S, j \in N_i \cap \neg S} \left( (v_i - t_j)^2 - (s_i - s_j)^2 \right)$$

Here, each \( i \) is a pixel in the source region \( S \) and each \( j \) is a 4-neighbor of \( i \) (left, right, up, and down). The first term expresses that we want an image whose gradients inside the region \( S \) are similar to the gradients of the cutout we’re trying to paste in. The least squares solver will take any hard edges of the cutout at the boundary and smooth them by spreading the error over the gradients inside \( S \). The second term handles the boundary of \( S \); we just pluck the intensity value right out of the target image.

Results

Below we show the source image, mask, and target image (top row) as well as the result of directly copying source pixels onto the target region, and finally the result of Poisson blending (bottom row) for a few successful cases.

---

---

A few more decent results are shown below:

We now show a failure case:

What went wrong here? The problem is that the gradient of the target region is very rough while the gradient of the source region is smooth. The inevitable error from trying to match the gradients is spread across the source image, and the intensity values are noticeably off as a result.

---

We can also compare the results of Poisson blending with multiresolution blending. Shown below are the results of multiresolution blending (left) and Poisson blending (right).

The choice of appropriate blending method depends on the desired result. Multiresolution blending is ideal when the colors and intensities of the source and target should be preserved. It is also much faster than Poisson blending. However, Poisson blending is great for the case in which the target and source regions have similar textures and we wish to seamlessly clone the source onto the target without a noticeable change in intensity.

---

Because this project may be reused in the future, I cannot make the code public. I will make it available if I obtain permission in the future.