October 24, 2023 Rushi Luhar

Navigating the Uncertainty Around using AI

Some thoughts around building on top of Large Language Models

Image of a rope bridge over a steep valley signifying the difficulty in making decisions.

Understanding and leveraging the potential of Large Language Models (LLMs) is an essential goal for many technology leaders.

However, in my discussions with our clients, I often come across a common misunderstanding of how these systems work. While it’s easy to lump LLMs with traditional computing systems, they are fundamentally different. Computers are deterministic. They follow a chain of logic (i.e., a program) to generate output from an input. In contrast, LLMs, or Machine-learning-based systems generally, are prediction machines. They predict an output based on their input, often with varying degrees of uncertainty. As a result, trying to decide whether to use LLMs can be a difficult decision.

The key to effectively leveraging LLMs is understanding this inherent uncertainty. When building systems that use LLMs, measuring your appetite for uncertainty against the model’s current capabilities is critical. Evaluating the performance of LLMs can be challenging. Still, one helpful heuristic is the Best Available Human (BAH) standard, as coined by Ethan Mollick in a recent post

It asks whether a given AI can outperform the best available human in solving a particular problem at a specific moment and location.

For example, a GPT-4 level LLM can summarize and extract insights from a large amount of text data faster and, given the time constraints, as accurately as a human analyst. Therefore, utilizing LLMs in such a context could be helpful as a stand-alone solution or a tool to augment human capabilities.

Decisions under conditions of uncertainty are best made through empirical experimentation. The most effective way to understand the utility of LLMs for your domain is to experiment. 

Using APIs from OpenAI or HuggingFace, build prototypes to identify the problems LLMs can solve in your area. At Jeavio, we understood this when we organized a two-week hackathon allowing our engineers to build anything, provided they utilized LLM capabilities. The outcome was so promising that we created an Applied AI team.

In summary, while AI is not a panacea, it holds transformative potential when its capabilities surpass or augment those of the best available human resources. 

If you want to learn more about our AI hackathon, check out this post. If you are interested in how other companies use LLMs, here is a post that you might find helpful. 

If you are curious about how to use LLMs to solve your problems, drop me a line!

, , , ,

Rushi Luhar

CTO at Jeavio