Kaiser Fung, writing for fivethirtyeight.com:
Kayak’s wait recommendations for the 10 flights for which prices never fell below their initial amounts were false-positive errors in airfare prediction. Hoping to capture more savings opportunities — or in Target’s example, identify the pregnant shoppers — the algorithm overshot, asking travelers to wait to buy on more flights than it should have. As we saw earlier, only a fraction of the routes offered potential savings. How companies tune their predictive models depends on how they judge the costs of these false-positive errors.
I asked Farecast’s Etzioni and Kayak’s Zacharia about this. Interestingly, they didn’t agree on strategy. Etzioni believes showing too many buy recommendations creates distrust amongst users, an impression that Farecast is not working hard enough to find better prices. Farecast carefully managed the fraction of buy recommendations to be between 67 and 80 percent. Kayak has the opposite concern. When a buy recommendation errs, users don’t know about it because they don’t typically track subsequent price changes. But when users see fares continue to rise while they’re waiting, the error is highly visible, and they may even cancel their trips. So, Zacharia said, “Kayak wants to make sure our ‘wait’ recommendations are as accurate as possible, even if that means we sacrifice a little bit of the ‘buy’ accuracy.”
I’ve often wondered how useful the fare predictor is on Kayak. Now I know!
Have I mentioned that I’m really enjoying the new FiveThirtyEight?