Retail search & compare
Ryan VanDyke
Product design leader & manager

Retail search & compare

Shopping is a journey, not a destination

Challenge

Amazon customers find it difficult to find and compare products that have many options and competitive offerings. For example, research conducted in Q4 2017 indicated that out of 100 people that start searching for a TV on Amazon 5 will actually make the purchase. 

Objective

Improve the Amazon search and comparison experience for technical products where customers are overwhelmed with variety and technical details, ultimately accelerating the decision making process and enabling customers to find and purchase the product that fits their needs wants with confidence.

Team

Myself as UX lead, several product managers, a technical product manager, a software development manager, a junior UX designer and stakeholders in the Amazon search team that included a design manager, front-end developer, and a design technologist.

Discovery

First page of screener for study
First page of screener for study

Generative research & usability study

The initiative got its start as a result of a study I secured funding for and contracted to Blink UX to conduct in Q4 of 2015.

  • I worked closely with Blink UX in the creation of the screener and study plan, making sure the right questions were being asked and the structure of the plan led to the insights we sought.
  • The study consisted of 60 minute sessions with up to 16 participants that consisted of both generative shopping behavior questions along with a task-based observational approach.
  • Participants were screened for shopping and electronics purchase activity.
Q1 2016 mobile web search experience, similar to what was tested
Q1 2016 mobile web search experience, similar to what was tested

What was tested

Both desktop and mobile experiences of Amazon.com was tested, allowing participants to self direct their shopping journey. A few competitor websites were also tested in the study.

Q1 2016 desktop search experience, similar to what was tested
Q1 2016 desktop search experience, similar to what was tested
Retail search & compare

The findings

Blink UX created a thorough report of their findings and presented it to leadership within the Product & Service Discovery team. While some of the findings confirmed what we already knew to be challenges and aligned with roadmap objectives, others opened up completely new opportunities to improve the customer experience. The research made the team confident heading into 2016 that our roadmap priorities were addressing customers needs.

“What I do is if I’m looking for something, I’ll usually Google it first… so if I’m looking for a TV or if I’m looking for usually for clothes I’ll look for long jeans or black dress… I look for the product first …”

Mary

Final research report created by Blink UX
Final research report created by Blink UX

Definition

Early design exploration with Search

While the Product & Service Discovery teams roadmap scope didn’t include the retail search experience, that team reached out to us to partner on an initiative after I had shared our research with them a few months prior. Aside from attributable sales revenue, the effectiveness of product comparison features has historically been measured through reduction in click-thru-rates in search, with detail page comparison table features typically reducing that metric by 2-3%. New metrics would eventually be needed though to measure the effectiveness of product comparison features within the search experience itself.

Photo of early whiteboard collaboration session
Photo of early whiteboard collaboration session
Retail search & compare

Wireframes

At this stage of the project a recently hired Junior UX Designer joined the team who I challenged with driving the mobile experience (with my lead) while I focused on the desktop experience where most of our customer shopped.

Wireframe flow demonstrating mobile experience
Wireframe flow demonstrating mobile experience
Retail search & compare

While the wireframe explorations on mobile led to some novel interaction patterns that we really wanted to test, it was desktop that would eventually become the primary focus as most customers do the predominate amount of their product comparison shopping there. Technical constraints also dictate that desktop be the first experience to test.

“I guess that I do prefer to do research activities on a desktop or laptop, it’s more ergonomic, I don’t feel like I’m losing my eyesight. I can use my keyboard shortcuts, you know those little comforts make it so much easier to do.”

Christopher

Develop

Desktop search and compare experience
Desktop search and compare experience
Final report created by Blink UX
Final report created by Blink UX

Usability study

Working with Blink UX again, a second usability study was conducted on both a desktop and mobile prototype.

  • 10 participants
  • Two prototypes created, 1 for desktop in Invision, another for mobile built using Framer by a UX Design Technologist who worked closely with the Junior UX Designer on the team.


“If I were to be purchasing a laptop, what I would end up doing is that I would look at the price. And then immediately I would look over and would look, I don’t care about weight, I would look at the memory...Very easy, right in front of me. ”

Jesse

Deliver

Retail search & compare

Design refinements & specification 

Along with the feedback from the usability study the designs were refined leveraging data points such as average product title lengths across various languages. Sketch measure along with a limited UI specification was provided to the front-end developer within the Amazon search team.

Sketch measure was used to give developers what they needed
Sketch measure was used to give developers what they needed
Retail search & compare

2016 A/B test

After launching in an A/B test the experience was a success with some customers, but was a miss with many others. Hypothesis for failure included too broad of an experiment with too many product categories included along with a lack of data quality in those categories.

Diagram created for patent submission
Diagram created for patent submission

Evolve

Retail search & compare

It returns, the 2018 A/B test

While myself and a number of other team members from the original project moved on to other teams in 2017, the original technical product manager kept the project alive and it returned in 2018 to be tested again after mechanisms were put in place to improve data quality and limit which product categories and search queries would result in the comparison experience being shown to customers.

The treatments tested in the 2018 A/B test.
The treatments tested in the 2018 A/B test.

Learnings

  • The best customer experience design in the world will still fail without quality data and data coverage.
  • Don’t let politics or the scope of your roles responsibilities limit what you can accomplish, by being customer obsessed and sharing my findings freely with the Amazon design community I and my team was able to accomplish far more than we would have ever been able to alone.
  • The lines between UX and product management are blurry ones and don’t be hesitant to push for an experience you believe in.