9  Miscellaneous Techniques

These are additional techniques in utilizing local LLMs that I want to learn, explore, and share.

9.1 Tool Calling

9.1.1 About

[deepseek-r1:7b]

Tool calling in the context of Large Language Models (LLMs) refers to a method where the model delegates specific subtasks or data retrieval processes to external tools, databases, or services. This approach allows LLMs to handle complex tasks more efficiently by leveraging specialized resources while maintaining higher-level reasoning capabilities internally.

Key Points:

1. Delegation of Tasks: Tool calling enables an LLM to break down intricate problems into manageable parts, delegating complex subtasks to external tools. For example, for a weather forecast request, the model might delegate data retrieval to a reliable meteorological service.

2. Efficiency and Resource Management: By offloading tasks that require external knowledge or resources, tool calling allows the LLM to stay focused on higher-level reasoning without being bogged down by internal processing limitations.

3. Access to External Resources: This method often involves accessing databases, APIs, or other systems, which can provide real-time data and specialized information crucial for accurate responses.

4. Privacy Considerations: Safeguards are in place to ensure that access to external tools does not expose sensitive information, maintaining data security and anonymization.

5. Task Limitations and Scalability: There is a boundary between tasks the LLM can handle natively and those it must outsource. Tool calling contributes to scalability but may be constrained by processing power and network capabilities when handling multiple complex tasks simultaneously.

In essence, tool calling enhances an LLM’s functionality by expanding its problem-solving capabilities beyond basic text generation, allowing it to address a broader range of tasks efficiently and effectively.

9.1.2 Models with Tool Calling

For Ollama, you can identify LLMs with tool calling capability from the description of the models in the Ollama’s model page, or from the template link on the page. For example, the template page for LLama3.2:3b is https://ollama.com/library/llama3.2/blobs/966de95ca8a6 with the following template text:

<|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023

{{ if .System }}{{ .System }}
{{- end }}
{{- if .Tools }}When you receive a tool call response, use the output to format an answer to the orginal user question.

You are a helpful assistant with tool calling capabilities.
{{- end }}<|eot_id|>
{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1 }}
{{- if eq .Role "user" }}<|start_header_id|>user<|end_header_id|>
{{- if and $.Tools $last }}

Given the following functions, please respond with a JSON for a function call with its proper arguments that best answers the given prompt.

Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. Do not use variables.

{{ range $.Tools }}
{{- . }}
{{ end }}
{{ .Content }}<|eot_id|>
{{- else }}

{{ .Content }}<|eot_id|>
{{- end }}{{ if $last }}<|start_header_id|>assistant<|end_header_id|>

{{ end }}
{{- else if eq .Role "assistant" }}<|start_header_id|>assistant<|end_header_id|>
{{- if .ToolCalls }}
{{ range .ToolCalls }}
{"name": "{{ .Function.Name }}", "parameters": {{ .Function.Arguments }}}{{ end }}
{{- else }}

{{ .Content }}
{{- end }}{{ if not $last }}<|eot_id|>{{ end }}
{{- else if eq .Role "tool" }}<|start_header_id|>ipython<|end_header_id|>

{{ .Content }}<|eot_id|>{{ if $last }}<|start_header_id|>assistant<|end_header_id|>

{{ end }}
{{- end }}
{{- end }}

9.1.3 Tool Calling R Functions

For tool calling, we will use ellmer package, which allows using tool calling of R functions (https://ellmer.tidyverse.org/reference/tool.html and https://ellmer.tidyverse.org/articles/tool-calling.html for details).

We start with the first example from https://ellmer.tidyverse.org/reference/tool.html, we need to define the metadata that the model will use to understand when it needs to call the tool,

library(ellmer)
# define the metadata
tool_rnorm <- tool(
  .fun = rnorm,
  .description = "Draw numbers from a random normal distribution",
  n = type_integer("The number of observations. Must be a positive integer."),
  mean = type_number("The mean value of the distribution."),
  sd = type_number("The standard deviation of the distribution. Must be a non-negative number.")
)
tool_rnorm

<ellmer::ToolDef>
 @ name       : chr "rnorm"
 @ fun        : function (n, mean = 0, sd = 1)  
 @ description: chr "Draw numbers from a random normal distribution"
 @ arguments  : <ellmer::TypeObject>
 .. @ description          : NULL
 .. @ required             : logi TRUE
 .. @ properties           :List of 3
 .. .. $ n   : <ellmer::TypeBasic>
 .. ..  ..@ description: chr "The number of observations. Must be a positive integer."
 .. ..  ..@ required   : logi TRUE
 .. ..  ..@ type       : chr "integer"
 .. .. $ mean: <ellmer::TypeBasic>
 .. ..  ..@ description: chr "The mean value of the distribution."
 .. ..  ..@ required   : logi TRUE
 .. ..  ..@ type       : chr "number"
 .. .. $ sd  : <ellmer::TypeBasic>
 .. ..  ..@ description: chr "The standard deviation of the distribution. Must be a non-negative number."
 .. ..  ..@ required   : logi TRUE
 .. ..  ..@ type       : chr "number"
 .. @ additional_properties: logi FALSE

Then, we must redefine the provider’s base URL to Ollama’s OpenAI compatible endpoint as base_url = "http://localhost:11434/v1" according to https://ollama.com/blog/tool-support, specify the local LLM that we want to use, followed by registering the tool.

Provider(base_url = "http://localhost:11434/v1")

<ellmer::Provider>
 @ base_url  : chr "http://localhost:11434/v1"
 @ extra_args: list()

chat = chat_ollama(model = "llama3.2:3b", seed = 111)
chat$register_tool(tool_rnorm)

Finally, we can now try out the tool calling ability of the LLM,

chat$chat("Give me ten numbers from a random normal distribution with mean of 120 and standard deviation of 15.")

The ten numbers generated from a normal distribution with a mean of 120 and 
standard deviation of 15 are:

1. 125.0977
2. 140.8486
3. 131.5419
4. 140.3376
5. 125.3349
6. 134.1422
7. 111.2048
8. 126.6184
9. 129.9178
10. 123.316

We compare that to the response without tool calling using rollama’s query,

rollama::query("Give me ten numbers from a random normal distribution with mean of 120 and standard deviation of 15.",
               model = "llama3.2", output = "text", screen = FALSE,
               model_params = list(seed = 111)) |> cat()

Here are ten numbers randomly drawn from a normal distribution with a mean of 120 and a standard deviation of 15:

1. 108
2. 145
3. 115
4. 131
5. 121
6. 139
7. 124
8. 148
9. 113
10. 127

If you were to ask any LLM this simple question, “What’s the time now?”, you will be greeted with a standard example, i.e. “I don’t know” or its variants,

rollama::query("What's the time now?", "llama3.2",
               output = "text", screen = FALSE,
               model_params = list(seed = 111)) |> cat()

I'm not currently able to share the time.

Now, we simplify the example from https://ellmer.tidyverse.org/articles/tool-calling.html by asking the current time.

# Gets the current time in local PC time zone.
get_current_time <- function() {
  Sys.time()
}
get_current_time()

[1] "2025-02-06 16:16:06 +08"

Now, we register the tool,

tool_get_current_time = tool(
  get_current_time,
  "Gets the current time in local PC.")
tool_get_current_time

<ellmer::ToolDef>
 @ name       : chr "get_current_time"
 @ fun        : function ()  
 @ description: chr "Gets the current time in local PC."
 @ arguments  : <ellmer::TypeObject>
 .. @ description          : NULL
 .. @ required             : logi TRUE
 .. @ properties           : list()
 .. @ additional_properties: logi FALSE

Provider(base_url = "http://localhost:11434/v1")

<ellmer::Provider>
 @ base_url  : chr "http://localhost:11434/v1"
 @ extra_args: list()

chat = chat_ollama(model = "llama3.2:3b", seed = 123)
chat$register_tool(tool_get_current_time)

Note that for this step, so far I could not manage to get the code create_tool_metadata(get_current_time) running to automatically generate the metadata. This might change in the future version of ellmer.

Then, lastly, we try the query,

chat$chat("What's the time now?")

The current time is 4:16 PM.

which is the correct time in my PC at the time of writing.

9.2 AI Agents

• Web search
• LR search and summary
• more …

In progress …