How Yelp evaluated competing LLMs to ensure correctness, relevance and voice in order to develop its AI assistant

Review app Yelp provides helpful information to consumers and diners for decades. It had been experimenting with machine learning ever since its early days. Despite the recent explosion of AI technology, it still encountered stumbling block when it tried to use modern large language models for some features.

Yelp discovered that customers, particularly those who used the app only occasionally, had difficulty connecting with its AI features. This included its AI-powered Assistant. Craig Saldanha told VentureBeat that “we learned that it is easy to create something that looks cool but difficult to create something that is both cool and useful.”

It wasn’t easy. Yelp’s AI-powered assistant for service searches, Yelp Assistant was launched in April 2024. After the launch, Yelp’s AI tools began to decline.

The one that caught us by surprise was the launch of this beta version to consumers – a few users, and people who are familiar with the app – [and they] they loved it. Saldanha said that after we rolled out the app to everyone [and] performance dropped. “It was a long time before we figured out why.”

Turns out, Yelp’s casual users, who visited the site or mobile app occasionally to find a new plumber or tailor, didn’t expect to be able to talk to an AI representative.

From simple to more complex AI features

The majority of people know Yelp for its website and app that allows them to search restaurant reviews and view menu photos. I use Yelp for pictures of food at new restaurants and to see if other people feel the same way about a bland dish. It also tells me whether a coffee shop that I’m planning to use for a day as a workplace has WiFi, plugs, and seating, which is rare in Manhattan.

Saldanha remembered that Yelp invested in AI “for a better part of a century.”

We were in a different generation of AI back then, so we focused on building our models to do things such as query understanding. He said that helping people refine their search intent is a key part of making meaningful connections.

As AI evolved, so did Yelp’s needs. The company invested in AI technology to identify popular dishes in photos submitted by users. It then launched new ways for users to find tradespeople, services and to guide their searches on the platform.Yelp Assistant can help users find the best “Pro” for their project. The chatbox allows users to type in their task or use the prompts. The assistant asks further questions to narrow down the list of potential service providers, before sending a message to Pros that might be interested in bidding for the job.

Saldanha says that Pros are encouraged by Yelp to respond directly to users, but he admits that many larger brands have call centers to handle messages generated by the AI Assistant.

In conjunction with Yelp Assistant and Review Insights, Yelp also launched Highlights. LLMs analyze the sentiment of users and reviewers, which Yelp then collects to create sentiment scores. Yelp generates a dataset using a detailed GPT-4o question. Then it is fine-tuned using a GPT-4o mini model.

Review highlights, which displays information from reviews, uses an LLM prompt for generating a dataset. It is based on GPT-4 with fine-tunings from GPT-3.5 turbo. Yelp announced that it will update this feature with GPT-4o, and o1.

Yelp has joined other companies in using LLMs to enhance the usefulness and quality of reviews. They have added better search functions based upon customer comments. For example, Amazon has launched Rufusan AI-powered assistant to help people find recommended products.

Big models and performance requirements

Yelp used OpenAI’s GPT-4o for many of its new AI functions, including its AI assistant. But Saldanha said that, no matter what model it uses, Yelp data is the secret sauce behind its assistants. Yelp didn’t want to be locked into a single model, and was open-minded about which LLMs could provide the best service to its customers. Saldanha stated that “we use models from OpenAI Anthropic, and other models on AWS Bedrock.”

Saldanha explained Yelp created rubrics to test the performance models in correctness and relevance, consciousness, safety of customers, and compliance. He said “it’s really the high-end models” that perform best. The company runs a pilot for each model before considering iteration costs and response latency.

Teaching users

Yelp has also made a concerted attempt to educate both casual users and power users in order to become comfortable with the AI features. Saldanha said that one of the things they realized was the need for a human tone, especially when it came to the AI assistant. It couldn’t react too quickly or too slowly, it couldn’t overly encourage or be too brusque.

We put a lot of effort into making people feel comfortable with the first response. It took us nearly four months to perfect this second piece. Saldanha said that as soon as they did so, you could immediately see the hockey stick.

A part of the process involved teaching the Yelp assistant to use positive words and sound. Saldanha says that after all the fine-tuning they are finally seeing higher usage rates for Yelp AI features.