By Sara Guaglione * January 30, 2025 *
Ivy Liu
Ivy Liu Publishers who want to experiment with generative AI technology for building products and features such as creating chatbots and analysing data must evaluate which large language model best fits the bill. According to three publishing executives,
one of the most important factors in these evaluations was how easy it was to integrate an LLM with their companies’ technology systems – such as different product suites or content management platforms – This often means selecting LLMs from companies with whom they already have enterprise software or content licensing agreements.
A spokesperson at a publisher, who asked to remain anonymized, told Digiday that their company doesn’t experiment with a variety of different LLMs but uses OpenAI’s model. The company has a licensing agreement with OpenAI, which was a result of a successful project that involved building a chatbot based on OpenAI’s GPT. The publisher continues to use GPT, including for productivity tools.
A second publishing executive, speaking on condition of anonymity said that they use OpenAI’s GPT for internal use, instead of other LLMs. This is because they have an OpenAI content deal.
A third publishing executive, who also requested anonymity said that their company uses LLMs owned and operated by companies such as Microsoft and Google, because they already pay for their enterprise software including Microsoft Office 365, and Google Workspace.
The third publishing executive said that this makes it “very simple for us to integrate into our entire development ecosystem”. The company, which is a Microsoft Office 365 customer, can “integrate [Copilot] into the tools we already use like Outlook, Excel [and] Word,” the executives added. Nate Landau is the chief product and technology officers at TheSkimm. He said that using technology provided by companies with whom they already have agreements means they do not have to build their solutions. “Some of our partners like [data storage company] Snowflake offer their own AI Integrations and we prioritise those over building solutions completely from scratch when applicable,” he explained. Landau said that because TheSkimm does not have an exclusive deal with an AI tech firm, they are able to work with a variety of models.
Given the rapidly evolving nature and the different strengths and weaknesses of the models, he said that they have not centralized on one provider. “For each use-case, we evaluate multiple vendors to ensure the best match.”
Other factors media companies consider when evaluating the LLMs they use include cost, performance and quality of models’ outputs, and privacy and security concerns.
Vadim Supitskiy is Forbes’ chief digital officer and said that it was important for the company that models were not trained using user inputs or information.
Testing Performance
For LLMs to be worthwhile, they must work well with publishers’ use cases.
Before choosing which LLMs to use for various processes, TheSkimm tests the models side-by-side and compares their results “to ensure that they align with our editorial standards and brand voice,” Landau said. He said that the main differences between these models were their “voice, tonality and accuracy across different use cases.” Landau prefers Claude because it has a “softer tone” and “is particularly accurate.” For tasks that require a response voice and tone to be important, Landau prefers Claude. For tasks that require less creativity, such as working with TheSkimm datasets, he prefers Meta’s Llama models “for their reliability to provide accurate, actionable answers and avoid hallucinations.” Landau says that TheSkimm uses LLMs primarily for data analysis, audience acquisition, and experimentation. These models allow TheSkimm to identify key trends and cohorts in its data and create audience segments for targeted messaging and products. This is especially useful for its shopping and commerce business and sponsored content.
A third publishing executive said that their company uses Google Gemini for product development. This includes building chatbots. Microsoft Copilot is used by the publisher for internal efficiency tools. The company is currently evaluating Google Gemini. Google Workspace includes products such as Gmail, Docs, and Meet.
Cost considerations
Mark Howard said he considered the financial incentives of working with one LLM versus another (Time has an agreement with OpenAI for content licensing) as well as having a seat at a table to help shape future products that can be beneficial to publishers.
Some [these deals] focus more on being a part of the new marketplaces they are developing. To me, this is the most interesting part — companies that are looking to build something new. “I’d rather be a member of those as they build them,” Howard said. Time has also signed deals with Perplexity, ProRata and other companies to participate in their ad revenue sharing programs and per-use compensation structures.
Currently, TheSkimm uses a mix of open-source products (such as Meta’s Llama or French-based Mistral), and commercial products such as Anthropic’s Claude and OpenAI’s GPT models.
Landau said that the company weighs “pros” and “cons” of paying fees for access to private LLMs APIs, and the more cost-efficient alternative of hosting their open-source models.
https://digiday.com/?p=566832
