How to Supercharge Your AI Agents with Function Calling

Photo by Dominik Scythe on Unsplash

TLDR: Large Language Models (LLMs) excel at text-based tasks but struggle with real-time data, math, and specialized expertise. Function calling extends LLMs by enabling access to external tools, APIs, and databases. This supercharges AI Agents for complex, job-specific automation.

Chat-based LLM apps like ChatGPT, Grok, Copilot, and DeepSeek are becoming commonplace for today’s consumers. Whether it is asking AI to summarize a lengthy email or write a paper for civics class, AI is being used across many domains. While they can be used for many tasks, they do not excel at everything they’re asked to do. Let’s look at the underlying technology of LLMs and how they can be used with function calling to supercharge AI Agents.

What is an LLM?

Before we can dive into what LLMs are good at, we need to understand what they are. LLM stands for large language model. It is the tech that generative pre-trained transformers (GPTs) and other transformers are built on. An LLM is a computer program trained on billions of words from texts, which it uses to understand and generate human-like responses to your questions or prompts.

Figure 1. LLM receiving a prompt and creating output.

LLMs may seem to understand what users are asking for, but are really just a large predictive model for text output based on text input. A user could give an LLM some text input like “The capital of Arkansas is.”

The LLM does not retrieve the value “Little Rock” by mapping the question and the answer. It breaks the question into tokens, then creates a new set of tokens with the highest likelihood of being correct based on the input pattern.

Figure 2. LLM tokenization process.

The answer could be “Little Rock,” or the answer could be “the most populous city in the state.” Both outputs are technically correct according to the test data and model training, but only one of these answers is what the user was asking for.

What are LLMs good at?

Now that we understand what LLMs are, let’s talk about what the basic technology is good at. LLMs are great at generating text-based responses given some input. This input takes many forms, including text, audio, and video. The output includes text, audio, video, and images.

Figure 3. Different possible outputs from an LLM.

Some of the tasks that LLMs excel at include:

1. Text generation
2. Questions and answers
3. Language translation
4. Summarization
5. Code assistance
6. Conversational tasks
7. Text analysis
8. Text-based task automation

LLMs handle analysis, summarization, and data generation very well. If your task falls into one of these categories, a base LLM without tools works well. We will see in the next section that there are many tasks LLMs are not natively good at. That’s where extension with tools and function calling comes in.

What are LLMs not good at?

We know what LLMs are good at by themselves. Where do they struggle, and what tools would we want to introduce to extend their capabilities? The tasks that LLMs usually struggle with include:

1. Tasks requiring real-world experience
2. Highly specialized expertise
3. Real-time data access
4. Complex math and calculations
5. Tasks requiring perfect accuracy
6. Ethical and moral judgments
7. Handling sensitive data

LLMs are not people. They do not reflect a society’s ethics or morality. In fact, they are flawed in these categories, as the HITL (human in the loop) used for training can bias models with their own beliefs.

They do not usually have access to or knowledge of current events because they are trained on old data. They do not understand what is happening, instead, they only predict responses based on what they’ve been trained on.

Surprisingly, to many users, LLMs are not good at math. When you understand that they don’t perform calculations, but instead guess responses, this makes more sense.

While LLMs are limited in these domains, that doesn’t mean other programs can’t perform desired tasks. This is where function calling becomes very useful. As we see in the next section, function calling allows LLMs to leverage tools for tasks beyond text analysis and manipulation.

What is function calling with LLMs?

Function calling is the ability of LLMs to access programs and data that they have not been trained on to further their capabilities. As mentioned above, LLMs are great at some things and incapable of other actions. Sometimes the issue is not even about ability.

An AI Agent built on an LLM may need access to an API or database for live data it has not been trained with. This is another great time for function calling. To keep it simple, function calling allows LLMs to :

1. Access external tools or code
2. Access APIs or databases

These actions extend the ability and knowledge of an LLM to complete your tasks. With function calling, your LLM can read customer order values from an Airtable table, call a math tool to compute some tax or extra charge, and then write this value to a Google Sheets document.

Figure 4. Syncing data from Airtable to Google Sheets with LLM mod intermediary.

Without function calling, you hope the LLM understood the task instead of making up numbers to transcribe from one data source to another. Function calling opens up the ability to access massive, unique data sources and perform complex actions on them.

An AI Agent may look like another basic LLM with transformers (ChatGPT or Grok), but is equipped with serious power. With function calling, you can solve more complex problems for your users while leaving their UI simple and easy to understand.

Need Help Building Your AI Agents?

AI Agents are powerful tools for modern web apps. They can simplify and automate processes like customer service, order fulfillment, and content generation. It can be tricky to get your AI Agents to the point of being job-specific, measurable, and reasonably priced. If you have an idea for a business automation or tool you need built, reach out to me and we can discuss the best solutions for your company or business.

Key Takeaways

• LLMs are predictive models great for text generation, summarization, and translation but lack real-world experience and real-time data access.
• Function calling allows LLMs to access external tools, APIs, and databases, enabling tasks like live data retrieval and complex calculations.
• AI Agents with function calling can automate tasks like syncing data across platforms, enhancing efficiency while maintaining a simple UI.
• LLMs struggle with perfect accuracy, ethical judgments, and math, making function calling essential for precise, specialized tasks.