Before launching or changing any major feature on Polyvore we run live experiments to find the best design, wording, flow or algorithm.
Our typical flow is to design a few variations, pick the most promising ones, run them as experiments against a percentage of our site traffic, measure the difference in how each performs to iterate on designs and finally pick the best performing version.
Running experiments is baked into our product development process. We have run hundreds of experiments since launch and have a few live at any given time. Once we realized we’d be running lots of experiments, we took the time to build a bucket testing system that makes it easy and streamlined to define, run and report on experiments.