Tuesday, May 21, 2013

Under the Hood: How we Mask our Images

Polyvore users create over 3 million sets every month, mixing and matching their favorite products to express their style. They clip in images from all over the web, like this teal T-shirt from nordstrom.com


As you can see, the shirt image has a light gray background that creates an eyesore when layered with other items in a set:


How do we strip this background away in order to achieve a cleaner look? More importantly, how can we do it for not just this T-shirt, but for any of the 2.2 million images our users clip in every month?

Magic!

Ok, not quite. But we do use ImageMagick software as well as some nifty tricks to do the transformation. Let’s walk through the basic steps.

First, since the background is usually a neutral color, we use ImageMagick's Modulate to boost the color saturation to highlight the difference between the product and the background. In this shirt, we see the teal turn a bright blue:


Next, we replace the background with white, starting at the pixel that is at the xy position of 2,2. We start there since it's almost certainly part of the background and not the product. ImageMagick has a Draw command that replaces pixels that are the same color as the start pixel with a new color. We use Draw to replace background pixels (those the same color as our start pixel) with white pixels. We also use Draw with a fuzz factor so that it also replaces pixels that are similar in color to the starting pixel to account for subtle gradients and variations in the background.


Negate flips all the colors in the image, which renders our white background black.


Threshold changes lighter-colored pixels to white and darker-colored pixels to black, giving us a black and white mask:


Laying this mask on top of our original image using ImageMagick's Composite with CopyOpacity will keep the parts of the original image that are white in the mask and discard the parts that are black. In essence, the mask acts as a stencil to "cut out" the shirt from the original image:


To demonstrate the background removal effect, here is that same final image laid over a dark gray background:


Neat, right?

This technique works great at smaller image sizes, and this was sufficient when the maximum Polyvore set image was 500x500. But with a general trend towards UIs with larger images and retina displays, we needed to render sets at 1024x1024. At this size, that jagged white outline became much more visible.


We figured there had to be a better way. After some research and a lot of trial and error, here’s what we came up with:

We start with the same teal T-shirt, and we do the same Modulate and Draw steps as before:


Here is where we start to stray from our original masking algorithm. The step after this one works better with grayscale input, so we apply Threshold first:


Then we use Potrace, an open-source utility which can convert bitmaps into vectors (SVG in our case). We use it to "trace" our mask, which produces smooth edges instead of jagged ones. It’s already looking better!


Negate to get a mask with white shirt and black background:


And here's what the masked image looks like on a gray background:


We probably could have stopped here. The white outline around the T-shirt is much smoother than in the original, but wouldn't it be better to not have that outline at all?

Let's go back to the mask we made with Potrace.


Potrace draws such perfect lines that the transition between the black background and white foreground is very sharp. When we use Composite with a mask, ImageMagick interprets the black pixels as areas that should be transparent, white pixels as areas that should be opaque, and gray pixels as areas that should be semi-transparent. If we can replace the sudden black to white transition at the mask's edges with a gradient, we should be able to get the white outline in the final image to fade away.

Using the feathering technique described here, we can smooth the boundary between the black and white by Blurring the edges, then applying this gradient to the Blur'd regions:


It's hard to see the difference in the final mask, so here's a closeup of the right sleeve area before and after the feathering:


Notice that the jagged edges have been replaced with smoother lines and curves and the transition between black and white is more gradual.

And the final payoff, our original masking on the left, our current masking on the right:


We're constantly looking for ways to improve every aspect of our product. Even though this may seem like a subtle feature, it represents the kind of detail and polish that we strive for and the lengths to which we go to make Polyvore great.

Got any of your own image processing tips to share?

65 comments:

  1. Your post included such an awesome and in-depth information. I am totally impressed from your post. Superb Work.
    west london electrician services

    ReplyDelete
  2. This is the best post to understood about how to mask images. Really worthy post to read. Nice post.
    t shirt embroidery

    ReplyDelete
  3. It's my first time visit here. And I found so many entertaining stuff in this blog, especially its discussion. birkin

    ReplyDelete
  4. Stunning post. I loved this post and bookmarked it for further good reading. Keep Sharing. nyc cfp

    ReplyDelete
  5. Cool post!
    I've never thought about that until reading this... And I must say it IS magic.. I mean, doing that for so many images every day can take quite some time.

    Battlefield 4

    ReplyDelete
  6. "Using the feathering technique described here, we can smooth the boundary between the black and white by Blurring the edges, then applying this gradient to the Blur'd regions:"

    What command do you need to do this?

    ReplyDelete
  7. Thanks for great share! This is the best post to understood about how to mask images. Really worthy post to read. Proweb365 provide custom web design at reasonable prices and is always up-front about our affordable web design services. We list our web design prices right on the website, so you can do price comparisons. Our Minnesota web design firm custom builds professional websites – including eCommerce, Blog, and Mobile Websites - that are easy to self-update and manage, with integrated content management systems (CMS) such as Drupal, Joomla, Magento, and WordPress - Web Designer MN

    ReplyDelete
  8. This is how we can do about thse design and how to get new images with Twin cities web design. Make sure you also check the full sources in order to get good tips and tricks from Minneapolis Internet Marketing

    ReplyDelete
  9. Good design with nice ideas, I will check to apply and repost on Faribault Web Design

    ReplyDelete
  10. Thank for sharing. I am try to implement in your way in Node and work perfect. It can get more correct result if user can adjust the fuzz, color saturation, the coordinate of target replace point.

    ReplyDelete
  11. Really impressed! Everything is very open and very clear clarification of issues. It contains truly information. Your website is very beneficial. Thanks for sharing. best dehumidifiers

    ReplyDelete
  12. This was a great review of the digital publishing models. This was really very interesting and fun to read. free candy crush

    ReplyDelete
  13. I really do appreciate this so much. This was a very good and interesting article to say the least. Thanks! I work for a design company and I have really learned so much from this great site. gummy prenatal vitamins

    ReplyDelete
  14. I really do appreciate this so much. This was a very good and interesting article to say the least. Thanks! I work for. important source

    ReplyDelete
  15. I am currently developing a website of a export quality garment and that's why I am currently taking image masking services to give all those garment's related photos professional look.

    ReplyDelete
  16. One would think a dental professional gained enough cash that no more money would be required. But Antonio is not your common dental professional. http://www.edbernstein.com

    ReplyDelete
  17. In today's world this industry involving dissertation producing will be changing daily and the requirements involving producing are changing quite easily. site

    ReplyDelete
  18. One would think a dental professional gained enough cash that no more money would be required. But Antonio. my link

    ReplyDelete
  19. This was a terrific review of this digital camera creating products. This was definitely quite fascinating and also exciting to read.Buy yacon cambogia

    ReplyDelete
  20. This was a terrific review of this digital camera creating products. This was definitely quite fascinating and also.
    find people

    ReplyDelete
  21. Nowadays this kind of market regarding Best Ibeacon Online
    dissertation creating is going to be altering day-to-day along with the prerequisites regarding creating are generally altering without difficulty.

    ReplyDelete
  22. However, the powerful reveals that offer a real value to the market ongoing to flourish and some have now exceeded their pre-recession levels. A excellent very good example is AHR Expo, the world’s biggest HVACR-focused display and meeting. search

    ReplyDelete
  23. Appreciate this one, you are a great author. It's really written in depth analysis and have urged visitors to look forward more on your writing Healthcare

    ReplyDelete
  24. This blog contains so many interesting stuff that makes me want to visit again and again. Look forward for more interesting updates columbus ms bed and breakfast

    ReplyDelete
  25. A unique partnership was forged, which not only guaranteed survival, but allowed us to have what may have been our most. over here

    ReplyDelete
  26. However, the powerful reveals that offer a real value to the market ongoing to flourish and some have now exceeded their. shemale cams

    ReplyDelete
  27. This is best site to spent time on .I just stumbled upon your chatty blog and desired to say that I have really enjoyed reading your very well written blog posts. I will be your frequent visitor, that's for sure. Herpes treatment 2014

    ReplyDelete
  28. That's pretty powerful stuff. I like the way your logic follows, and how you're able to put it into words with such a powerful post! I am enthralled. Plantar Fasciitis treatment 2014

    ReplyDelete
  29. We are existing because we are not dead; though our daily way of life is packed with the emotional jumble of concepts, feelings, and applications, at another stage we have a low-level careful system that is always looking out for existential threats. Klik4D

    ReplyDelete
  30. However, the powerful reveals that offer a real value to the market ongoing to flourish and some have now exceeded. Best tattoo shops near Chicago

    ReplyDelete
  31. Our Minnesota web design firm custom builds professional websites – including eCommerce, Blog, and Mobile Websites - that are easy to self-update and manage, with integrated content management systems. Unbiased Reviews

    ReplyDelete
  32. Next, we replace the background with white, starting at the pixel that is at the xy position of 2,2. We start there since it's almost certainly part of the background and not the product. ImageMagick has a Draw command that replaces. professional house cleaning

    ReplyDelete
  33. Not many! Robichaud is a great example of a contractor on the cutting-edge of technology.If the technology works, and in this case it seems that it does, it’s a win for everyone. check out

    ReplyDelete
  34. However, the powerful reveals that offer a real value to the market ongoing to flourish and some have now.
    bounce house rentals pensacola fl

    ReplyDelete
  35. Thanks a lot for being our mentor on this niche. I enjoyed your current article very much and most of all cherished how you really handled the aspect I considered to be controversial. www.plumlivinghealth.org |

    www.provitalhealth.org |

    www.northernhealthplan.org |

    www.oralhealthexam.org |

    www.cwyzehealth.org |

    www.cwyzehealthcare.org |

    www.healthykskids.org |

    www.biohealthchip.org |

    www.meridianwomenhealth.org |

    www.heroeshealthproject.org |

    ReplyDelete
  36. Great buy for the price! I received the item in a day or so and I was able to complete a photo project I had been working on! It also shuts the feeding ports when they put enough weight on the ring. www.QuoteCorner.com

    ReplyDelete
  37. This is my first opportunity to visit this website I found some interesting things and I will apply to the development of my blog. hacker un compte facebook

    ReplyDelete
  38. The urgency to drive digital revenues increased this year as media companies sought new ways to offset declining print revenues or to build sustainable digital-only publishing brands.Thanks Clash of Clans Hack

    ReplyDelete
  39. Another great example of innovation, I am glad to locate it. There are so several developers working on this segment but this is one of the best innovative idea ever. Thanks for sharing it here. tax fraud attorney

    ReplyDelete
  40. I just stumbled upon your informative blog and desired to say that I have really enjoyed reading your very well written blog posts. convenience

    ReplyDelete
  41. It employs weapons unused by its predecessor, like wardrobe designer Jason Alper, who concocted a hundred jaw-dropping looks for the character. Treatment For Vitiligo

    ReplyDelete
  42. Really impressed! Everything is very open and very clear clarification of issues. It contains truly information. Your website is very beneficial. Thanks for sharing. golf cart service Columbia

    ReplyDelete
  43. That's pretty powerful stuff. I like the way your logic follows, and how you're able to put it into words with such a powerful post! I am enthralled. fast weight loss Boca Raton

    ReplyDelete
  44. This is quite wonderful post. The article affects a lot of urgent challenges of our society. We can not be indifferent to these challenges. Your post gives the light in which we can observe our real life. Very professional. free iTunes cards

    ReplyDelete
  45. I do think that your brand new goods website is included along with some top quality details post. I've got this sort of brilliant on the net shopping web site, their supply best products like bracelets, house requires, youngsters apparel, mens grooming products, style products, for example. venus factor negative reviews

    ReplyDelete
  46. Carter Brownish, CEO of Boulder Power Automobile and a Perfection client, developed the electrical support vehicle, and assured Robichaud to test-drive the vehicle that he experienced would be an ideal fit for the support market because of the brief tracks support specialists generate everyday. http://ireport.cnn.com/docs/DOC-1152784

    ReplyDelete
  47. One of my family members actually proved helpful at a discarded steel supplier (for only a few several weeks, thank God) in the Metro-Detroit area. Stephen Williams wine

    ReplyDelete
  48. We are current because we are not dead; though our everyday way of lifestyle is loaded with the psychological jumble of ideas, emotions, and programs, at another level we have a low-level cautious program that is always looking out for existential risks. Brian Torchin

    ReplyDelete
  49. He says, “I don’t ideas the regarded of deficiency of way of lifestyle. I just don’t want to be there when it happens to me.” For most of us who are not ill or extremely neurotic, worry of deficiency of way of lifestyle seems remote and theoretical. The Dorchester Collection

    ReplyDelete
  50. I surprised with the analysis you made to create this actual put up incredible. Wonderful task! https://www.behance.net/christophercowdray

    ReplyDelete
  51. This post is helpful with an analysis I am doing for a specific group of people. Do you have any other articles to suggest on this topic? Thanks educational technology research and development jobs

    ReplyDelete
  52. Robichaud also hired Bella Energy of Louisville, Colo., to install solar panels on Precision headquarters to help recharge the batteries for eight hours at night after technicians drive them 120 miles during the day. Igor Cornelsen

    ReplyDelete
  53. It has no way of it. But there is another aspect of interest that not only can understand it, but already knows about it. There is a saying in the Zen customized, “Birth and lack of way of life are the outstanding problem.” That is where actual Buddhist work out needs primary. Mike Livak

    ReplyDelete
  54. Moreover to monitoring moment-to-moment threats such as an beginning car or a decrease banister, our threat verifying starts to intuit a distant but progressively approaching dark thinking. Susan McGall

    ReplyDelete
  55. By cooperating with Syngenta, we believe we can enhance the development of lovely sorghum plants control methods and offer a more finish program of innovative compounds and major plants security items to our common customers," said Eileen Stephenson, Vice Chief executive of Functions for Ceres. Jason Halpern

    ReplyDelete
  56. Many company activities, like contractors’ businesses, knowledgeable considerably during the excellent economic economic downturn of the last few years. Sultan Alhokai

    ReplyDelete
  57. Thanks to an wide range of mobile cellphone programs and thinking handling, companies can often find out out most of the facts they need, and handle their day, without ever speaking with the house company office. Sam Tabar

    ReplyDelete
  58. Why not let the bubble burst, the recession linger for a year or two and then let the market and entrepreneurs get us out of this hole. http://ireport.cnn.com/docs/DOC-1149856

    ReplyDelete
  59. Stay at the impressive of new technological innovation – make sure you have educational sessions and members introducing considers like geothermal power energy, biomass heat power, etc. http://www.cbsnews.com/news/steve-jobs-family-photo-album/

    ReplyDelete
  60. Colo., to set up residential solar energy sections on Perfection secret headquarters to help renew the battery energy for eight time in the evening after specialists generate them 120 kilometers during the day. Lee G. Lovett

    ReplyDelete
  61. If the technological innovation performs, and in this situation it seems that it does, it’s a win for everyone. One nice aspect about being the first organization to buy the vehicles is that Robichaud is fairly much engaged in any style changes that may be created to your vehicle. Marc Sparks

    ReplyDelete
  62. Great buy for the price! I received the item in a day or so and I was able to complete a photo project I had been working on! It also shuts the feeding ports when they put enough weight on the ring. mercedes benz repair shop

    ReplyDelete
  63. Fine reporting? If the last case was in February, then what is the current urgency?
    Or does our community just disregard science if it interferes with our hysteria?invitations click here

    ReplyDelete
  64. Disappointed that it did not include a miter guide. I have even made adjustments so I can throw in a little ground flavored coffee while still using the grinder. does truth about cellulite work

    ReplyDelete