Tuesday, May 21, 2013

Under the Hood: How we Mask our Images

Polyvore users create over 3 million sets every month, mixing and matching their favorite products to express their style. They clip in images from all over the web, like this teal T-shirt from nordstrom.com

As you can see, the shirt image has a light gray background that creates an eyesore when layered with other items in a set:

How do we strip this background away in order to achieve a cleaner look? More importantly, how can we do it for not just this T-shirt, but for any of the 2.2 million images our users clip in every month?


Ok, not quite. But we do use ImageMagick software as well as some nifty tricks to do the transformation. Let’s walk through the basic steps.

First, since the background is usually a neutral color, we use ImageMagick's Modulate to boost the color saturation to highlight the difference between the product and the background. In this shirt, we see the teal turn a bright blue:

Next, we replace the background with white, starting at the pixel that is at the xy position of 2,2. We start there since it's almost certainly part of the background and not the product. ImageMagick has a Draw command that replaces pixels that are the same color as the start pixel with a new color. We use Draw to replace background pixels (those the same color as our start pixel) with white pixels. We also use Draw with a fuzz factor so that it also replaces pixels that are similar in color to the starting pixel to account for subtle gradients and variations in the background.

Negate flips all the colors in the image, which renders our white background black.

Threshold changes lighter-colored pixels to white and darker-colored pixels to black, giving us a black and white mask:

Laying this mask on top of our original image using ImageMagick's Composite with CopyOpacity will keep the parts of the original image that are white in the mask and discard the parts that are black. In essence, the mask acts as a stencil to "cut out" the shirt from the original image:

To demonstrate the background removal effect, here is that same final image laid over a dark gray background:

Neat, right?

This technique works great at smaller image sizes, and this was sufficient when the maximum Polyvore set image was 500x500. But with a general trend towards UIs with larger images and retina displays, we needed to render sets at 1024x1024. At this size, that jagged white outline became much more visible.

We figured there had to be a better way. After some research and a lot of trial and error, here’s what we came up with:

We start with the same teal T-shirt, and we do the same Modulate and Draw steps as before:

Here is where we start to stray from our original masking algorithm. The step after this one works better with grayscale input, so we apply Threshold first:

Then we use Potrace, an open-source utility which can convert bitmaps into vectors (SVG in our case). We use it to "trace" our mask, which produces smooth edges instead of jagged ones. It’s already looking better!

Negate to get a mask with white shirt and black background:

And here's what the masked image looks like on a gray background:

We probably could have stopped here. The white outline around the T-shirt is much smoother than in the original, but wouldn't it be better to not have that outline at all?

Let's go back to the mask we made with Potrace.

Potrace draws such perfect lines that the transition between the black background and white foreground is very sharp. When we use Composite with a mask, ImageMagick interprets the black pixels as areas that should be transparent, white pixels as areas that should be opaque, and gray pixels as areas that should be semi-transparent. If we can replace the sudden black to white transition at the mask's edges with a gradient, we should be able to get the white outline in the final image to fade away.

Using the feathering technique described here, we can smooth the boundary between the black and white by Blurring the edges, then applying this gradient to the Blur'd regions:

It's hard to see the difference in the final mask, so here's a closeup of the right sleeve area before and after the feathering:

Notice that the jagged edges have been replaced with smoother lines and curves and the transition between black and white is more gradual.

And the final payoff, our original masking on the left, our current masking on the right:

We're constantly looking for ways to improve every aspect of our product. Even though this may seem like a subtle feature, it represents the kind of detail and polish that we strive for and the lengths to which we go to make Polyvore great.

Got any of your own image processing tips to share?