Launch Lessons with Carla Pellicano: How Airbnb Scaled Trust

The pricing problem space

When I joined Airbnb in late 2014, our product teams were still pretty lean. There were maybe ten product pods and just a few product leads. The company was starting to build out teams and organize around key areas, and I was brought in to lead the pricing and availability team.

The kind of feedback that we were hearing from hosts was Why am I not getting more bookings? or I'm not showing up in search enough—generally, a frustration that they wanted to get more bookings on Airbnb. But they were not telling us they didn’t think their price was right. You'll always hear good product managers say, yes, you need to understand what problems your customers are facing and what they want you to solve, but you can't rely on them to tell you exactly what to build. Like Henry Ford said, if you ask people what they want, they're going to tell you they want faster horses, right?

So our hosts weren’t getting all the bookings they wanted, and we saw a marketplace with lots of potential guests visiting who weren’t making bookings. Figuring out how to make more matches, that was a problem that we needed to solve. We needed to better understand what impacts a listing’s ability to get booked at any given moment in time. This was the beginning of our work with marketplace dynamics, really understanding how supply and demand are matching at any given time and the underlying drivers of those matches.

One of the strongest contributing factors to bookings was price. Hotels have had dynamic pricing for years, but they rely on software to do it. How were our hosts pricing their listings by themselves? How could we better support hosts in making great pricing decisions? We knew that to provide this kind of support, we needed to break down both what impacts price and how hosts think about price.

Listening to hosts

We started discovery on two tracks. One, was there an objectively good price for every night? We had a team of economics, stats, and computer science PhDs who set out to model the likelihood that each listing would get booked at a range of prices for each individual calendar night using all the data that we had.

Two, just as importantly, we set out to understand how hosts were thinking about price, because this was going to inform the design of a pricing product that they ultimately needed to trust. We looked at pricing actions and behaviors. For example, if hosts aren't getting booked and it's a month out, do they go in and update their prices? What do they do when they have an unbooked night just two weeks away? Behavior varied a lot. Some hosts were taking a very active role in their pricing, and they were essentially being their own dynamic pricing algorithms and doing the work themselves. And others were far more lackadaisical about it; seemingly, it wasn't as important to them to get those additional bookings or they just didn't have the time to put into it.

Having actual conversations with hosts revealed there was a real range of motivation. Some hosts were more casual; they weren’t looking to maximize bookings and they had a higher price floor they wouldn’t go below. Other hosts wanted to increase bookings and had some amount of price wiggle room, but just didn’t have the tooling they needed to make smart updates. When hosts did put in work, one of the behaviors we observed most often was hosts setting their price by looking at prices of listings similar to theirs. It affirmed this idea that, okay, hosts are looking for data. It allowed us to move forward with the conviction that if we put more data in front of them, it would be helpful to them. 

At the same time, it was a really big opportunity because there was a gap, a flaw, in this type of price research. If a host was looking at similar listings for a given night, they inherently were looking at nights that had not been booked; that’s all that would show up in search for them. So that's not necessarily the most useful comparison price for them. They’d actually rather know how somebody who’s booked for that night priced their Airbnb, but they couldn’t access that data themselves. We knew that there was power in using our data to give hosts actionable pricing insights and that we could definitely build a product that takes away manual effort.

The regret metric

From the start, we knew trust would be paramount. We knew that hosts were going to tune us out if they felt like we were putting a tool in front of them that was trying to achieve what hosts perceived as Airbnb's goals as opposed to theirs. Hosts assumed that Airbnb wants to maximize bookings, but Airbnb's goal was never to race to the bottom and have hosts’ nights sold at bargain prices. One of the most wonderfully challenging things about working on marketplace problems is that you need to think about second order impacts of everything that you do. We recognized that there would be very real tradeoffs and long term harm if we optimized for something that didn’t balance the goals of both hosts and guests.

Say we had a host who trusted us and used our pricing recommendations, but set a low minimum price without realizing the implications. As an unbooked date drew nearer, the algorithm might lower the listing price to reduce the risk of not getting booked, optimizing for maximum bookings versus revenue maximization. The host might get booked—which seems like a win on an A/B test—but with lower prices than what they could have gotten if they’d been booked for fewer nights at a higher price. Over time that host may decide it’s not worth it. They're going to take availability offline, they're going to stop posting entirely, or maybe they don't even trust Airbnb as a platform and they take their business elsewhere. Those are the kinds of things that we were sensitive to.

We were also aware that our models were not always going to get it right, so we developed long-term tracking and evaluation metrics to keep ourselves honest. One of the most important metrics we developed was what we called ‘booking regret.’ When evaluating a model and its goodness, we tracked the difference between what a host priced and successfully booked vs. the price we would have recommended. Ideally those numbers are the same. The difference, or how much money a host would’ve left on the table if they’d used our price suggestion instead, is the regret metric. Our goal with every subsequent model release was to reduce the amount of booking regret.

Simplicity versus (false) precision

One thing we grappled with was just how much data to put in front of hosts versus how much ease to build into our pricing products. We ultimately rolled out a tool that allowed hosts to flip a switch and always set their price to whatever the algorithm had as its prevailing price tip at that moment. We left the host with plenty of control to override prices for individual nights and to set minimums and maximums. The whole point was for this to be something that made hosts’ lives a little bit easier and helped hosts get more or better bookings. We very quickly got to 40% of hosts who had turned on Smart Pricing, and we continued to do this research and understand how they were using it and how that could be used to improve the product.

The aesthetic factor

It was easy to capture metadata on things like the number of bedrooms, whether the host is going to be present, does it have a hot tub… But trickier things like design aesthetic, the more difficult-to-quantify things, also started to show up in the data at a certain point. Our engineers and our data scientists who worked on the algorithm had a weekly meeting, and it was always somebody's job to pull up a couple of listings that we over- or underpriced and dissect why. Say two places with the same technical amenities, across the street from each other, are priced differently. You look at the photos and you totally see it. One place is kind of drab. You don't really want to sit on that furniture. There's not a lot of light. But the other place has thoughtfully chosen furnishings, and you can envision yourself there and feel like it’s a vacation. Those things will come through in photos and the reviews and ultimately in the bookings, so our pricing models would pick it up too. Those are some delightful affirming moments where you get to see the qualitative show up in the data.

Trust at the core

Recognizing and appreciating the importance of our hosts, we were always focused on how these products would come across to them. Like I mentioned, we were monitoring booking regret because it was a good proxy for how much hosts were going to trust us. We recognized that our ability to be impactful with this work was going to be dependent upon how accurate we could get with price and how well we could contextualize it and tailor it for each individual host.

There was a lot of fear and skepticism about how much trust we were building or losing with hosts at any time. We tried to increase trust by infusing the product with a sense that we aren’t telling you what to do, we’re just trying to tell you that if you want to get this night booked, there is high probability that this price will get you booked.

It’s an emotional issue as well. For a hotel, it's not personal, but for an Airbnb, a host might feel that a price is a reflection of how much people like them and their home and their hosting. There were also perception issues that lower prices attracted guests who weren’t going to treat your home as respectfully. Whether or not that's true, there were a lot of complex customer sentiment issues to navigate with these products.

Internally as an organization, we had a lot of healthy, challenging conversations. Our platform is based on the belief that by and large, we can all trust each other. The main problem that Airbnb solved wasn’t just improving online accommodations booking. The real thing that Airbnb did was normalize the act of random strangers staying in the home of other random strangers. Trust was really at the core of everything we did, and it was no less so with pricing.