Test and learn trend continues into 2014

Looking at trends for this year can be done with guidance from developments made last year. Econsultancy have recently predicted a number of digital trends for 2014, including a greater focus on experimentation and agility. They say that the value of ‘test and learn’ approaches are increasingly being appreciated by an ever wider audience.
This is nothing new and can actually be broken down into more detailed categories:

  • Doing lots of prototyping (whether high or low fidelity)…
  • Releasing prototypes early in the development process and
  • Looking to analogous situations and ethnography for inspiration

Experimentation and agility in 2013

Among the various talks I attended in 2013, there were some highlights around experimentation and agility being put into practice by various companies:

How the news feed is continuously developed – Mark Tonkelowitz, Facebook

The Facebook development process is built on the ethos of prototype a lot and launch early. Facebook believe that ‘code wins arguments’, so they create high fidelity prototypes of all their different ideas to test them out. There are three possible test stages prototypes can go through, depending on how well they work:

  1. Facebook employees
  2. A slightly wider but still select audience which often includes power users and countries where users are less likely to have international friends
  3. Everyone

At each stage quantitative multivariate testing is done and results are compared. They compare things like: number of posts, feedback, increase in number of friends, time spent, revenue, spam reports.
However, numbers don’t always give you the whole story, context is important. So they also do qualitative research, such as usability sessions, open ended interviews and help centre feedback analysis.

Anthropology before technology – Matthew Cockerill, Seymourpowel

Seymourpowell’s methodology is to start their research phase of design and development by talking to ‘progressive users’ and ‘experts’, rather than the intended users, as they don’t know what they’ll want in the future.
Part of this is to look at analogous situations, situations or fields of interest that are in some way related to the situation they are designing for. For example, when designing a mountaineering survival jacket they looked at polar explorers & SAS troops before focussing on mountain climbers themselves.
Ethnographic techniques are used as “what we say we do” and “what we actually do” are often different – see Usability Testing vs Focus Groups.

Helping organisations innovate through design – Juan Parviainen, IDEO

IDEO as an organisation are interested in where new ideas come from. They use a combination of ethnography, analogous situations and low fidelity prototyping.
The analogous situation looked at was Motorsport Pit Stop Stations. Pit Stop workers had worked out that 80% of their problems related to the same thing – not having the right tools together in the same place. To stop them running around wasting time finding what they needed, they designed tool packaging to keep everything together. A&E workers have the same kind of problem with nurses’ running back & forth to get different tools needed in emergency situations.
IDEO have a slightly different stance on prototyping compared with Facebook. IDEO tend to do low fidelity prototyping. They feel that this can make you more creative, as you are not attached to a design that you’ve spent a lot of time on, so you can have more open conversations to find the best solution and have multiple prototypes.

How 2014 might be different?

One of the biggest reasons that experimentation and agility might not yet have taken hold is the perceived ‘fear of failure’, and this might be holding some companies back.
Greg Linden, of Amazon, recently said

“To find high impact experiments, you need to try a lot of things. Genius is born from a thousand failures. In each failed test, you learn something that helps you find something that will work. Constant, continuous, ubiquitous experimentation is the most important thing.”

So hopefully 2014 is the year we give up on the ‘fear of failure’ and focus on the ‘power of failure’.

Want to learn more?

  • Training Course: Rapid Prototyping with