Tuesday, August 28, 2012

Bucket Testing System

Before launching or changing any major feature on Polyvore we run live experiments to find the best design, wording, flow or algorithm.

Our typical flow is to design a few variations, pick the most promising ones, run them as experiments against a percentage of our site traffic, measure the difference in how each performs to iterate on designs and finally pick the best performing version.

Running experiments is baked into our product development process. We have run hundreds of experiments since launch and have a few live at any given time. Once we realized we’d be running lots of experiments, we took the time to build a bucket testing system that makes it easy and streamlined to define, run and report on experiments.

Measure Everything

Measuring and acting on stats is an essential part of building successful products. There are many direct and indirect benefits to pervasive measurement and tracking of stats:

  • Accurate, real-time data enables better and faster decisions.
  • It empowers a data-driven culture where ideas can come from anyone -- ideas can be easily tested, and the best ones can be chosen based on their merit, instead of pure intuition, or because it’s the HiPPO.
  • Tracking stats and watching them improve in response to our iterations helps teams stay focused and is incredibly motivating.
  • Historical stats are a great way to keep an eye on how code changes affect the health of a product and processes.

Early on in the life of Polyvore, we decided to weave stats into everything we did. One of our engineering philosophies is to invest a fair amount of our resources into making ourselves more efficient. This approach introduces some latency into our projects but this is made up in the future by an increase in overall team bandwidth. Consistent with thie philosophy, the first thing we did was to invest in building a system that made it easy to collect stats and made it immediately useful.

Monday, August 13, 2012

Hello World!

print 'Hello world!';