Published on the 24/01/2024 | Written by Heather Wright

And how to get ahead of the game…

A UNSW Business School associate professor is recommending Australian and Kiwi companies up their game when it comes to understanding how large language models such as ChatGPT make decisions, after his research revealed a discernible bias in how ChatGPT operates and its impact on product recommendations.

Sam Kirshner, an associate professor in the School of Information Systems and Technology Management at UNSW Business School, says because of the way large language models (LLMs) operate, descriptions of product features and product reviews could become even more influential if consumers begin shifting towards using LLMs to guide or automate purchases.

“Understanding how LLMs make decisions could lead to firms reaping substantial advantages.”

He believes how LLMs will evaluate products moving forward will likely become a ‘substantial’ issue for businesses.

“My research focuses on the specific tradeoff between a product’s desirability (ie, why we use a product, which is typically for its performance) and its feasibility (ie, how we use a product, which is typically about its usability or convenience),” Kirshner told iStart.

“Overall, I find that even in scenarios where consumers prefer convenience, ChatGPT exhibits a strong preference for recommending products with better performance. Therefore, businesses have to be conscious about how they position their products.”

He says if this trend continues, then how products are described and reviewed and how they compare within product lines and against competitors’ products could significantly impact how they are perceived and recommended by ChatGPT.

Products are combinations of attributes, ranging from various dimensions of performance, aesthetics, usability, and convenience, he notes.

“So, for LLMs like ChatGPT to recommend a product, they have to weigh different aspects of the products based on the descriptions of these features.”

That means their recommendations will vary in how they weigh the importance of product features.

“Moreover, if LLMs understand a consumer’s preferences, they will be able to readjust weights to make person-specific customised recommendations. Not only descriptions of product features but product reviews could be even more influential,” Kirshner told iStart.

While consumers currently only use a handful of reviews and aggregate information for recommendations, LLMs ability to absorb large bodies of text mean it could leverage substantially more reviews to adjust the weighting it provides to product features.

While there’s ‘tremendous uncertainty’ around how quickly consumers will move to using LLMs to guide or automate purchases (some e-business sites are starting to use ChatGPT tools to summarise product feedback threads), Kirshner says the fierce competition between tech giants like Google, Microsoft, Amazon, Meta and Apple mean it will only take one accessible breakthrough, similar to the introduction of ChatGPT, to completely revolutionise the shopping experience.

“If consumers begin to use LLM models for product recommendations, or if these tech giants successfully implement their LLMs as recommenders on their platforms, then misunderstanding how LLMs evaluate product assortments could create additional barriers for firms operating in online marketplaces.”

He’s warning that the competition to get product in front of potential buyers online could become even more cutthroat.

“In previous iterations of e-commerce unless a product or website was on the first page of Amazon or Google, it was unlikely that consumers would purchase or find the product. With LLM recommendations… all consumers will likely see the selected recommendations.

“This could make it more difficult for many vendors. It also means that consumers may have difficulties finding the precise product, experience or service they seek.”

In terms of the LLMs’ recommendation preferences between high-performance products – for example a 15-inch high-powered but heavy laptop – and high-convenience products, such as say an 11-inch lightweight laptop, while the AI models underpinning LLMs may change over time, this is likely to remain a feature of LLMs.

“These preferences are likely driven by an abstraction bias that is inherently a part of how these models are trained,” he says.

“This is a fascinating time for vendors and consumers. Although ChatGPT could create significant challenges for firms, understanding how LLMs make decisions could lead to firms reaping substantial advantages as the models and adoption of LLMs like ChatGPT continue progressing.”

Kirshner offers up these three tips for Australian and New Zealand businesses:

1. Use ChatGPT to understand how GPT interprets your products. For example, by telling GPT about your product range and close competitors (using website product descriptions and reviews), you can ask for recommendations on how to price your products. Prices should indicate how ChatGPT values your products. Ensure the use of appropriate prompt engineering techniques to avoid GPT over-relying on its training data (e.g., historical prices of your products).
2. Take a similar approach by using ChatGPT as a consumer. Provide ChatGPT with several options across your product line and/or against competitors. Also, be sure to provide consumer characteristics (eg, preferences and budgets) to provide ChatGPT with the appropriate information needed to make decisions. From here, you can get ChatGPT to provide a ranking of recommendations or recommendations on willingness to pay for the products. Ideally, there is an alignment between your products and their target market groups.
3. Different LLM models – ChatGPT (GPT-3.5), ChatGPT Plus (GPT-4), and Bard – will provide different recommendations. Similarly, people’s heterogeneity in user sophistication will also impact the received recommendations. So it is crucial to understand: Are your potential consumers using LLMs for purchasing decisions? Are your consumers sophisticated LLM users? Are they using advanced LLMs (currently GPT-4 models) or free versions (Bard and GPT-3.5)? As LLMs shift consumers’ purchasing decisions, firms will need to respond by updating how they market and stock their products.