Benchmarking Ollama Over Tailscale & LAN: Squeezing Sub-Second Latency Out of the Grav AI Chatbot

In Part III of this series, we built a 5-tier pipeline for grav-ai-chatbot, running local AI search on a 6-Watt Intel Celeron laptop.

But once real traffic started hitting it from different devices on different networks — the local LAN, and a Tailscale mesh VPN for access from elsewhere — a new question showed up: does the network path itself add latency on top of everything we already fixed?

Turns out, mostly no — but proving that took real telemetry, a custom benchmark script, and a few genuinely useful lessons about failover and caching along the way. Here's what the numbers actually said.

The Network Setup: Two Paths to the Same Laptop

There's a common assumption that anything routed over a Tailscale mesh VPN has to hop through a relay server somewhere out on the internet, adding real latency to every request. That's true when two devices can't reach each other directly — but it's not what happens when both machines are sitting on the same local network.

Worth being precise about what actually causes overhead here, since it's easy to blame the wrong layer: Tailscale itself isn't the bottleneck. Its own protocol overhead is negligible regardless of where two devices sit. What actually determines whether a connection is fast or slow is (1) whether both devices can establish a direct peer-to-peer path at all — which mostly comes down to what kind of NAT each device is sitting behind, since some NAT types make direct hole-punching easy and others force a fallback to a relay — and (2) the underlying quality of whatever network is carrying the traffic, whether that's your home Wi-Fi, your ISP's routing, or the public internet in between if the devices aren't local. Two devices on the same LAN skip both of those variables entirely, which is exactly why the numbers below look the way they do.

When that's the case, Tailscale is smart enough to skip the relay entirely:

1. Direct peer-to-peer negotiation. Tailscale detects that both devices share a local subnet and negotiates a direct, encrypted UDP connection over its own virtual IP range — no relay server involved.
2. Sub-millisecond ping. Hitting the Ollama API's version endpoint responds in under 0.1 milliseconds, whether you go through the physical LAN address or the Tailscale address. Practically speaking, they're the same path wearing two different name tags.

Client / Grav Web Server
       │
       ├────► Primary:  Tailscale mesh address   (~0.09 ms)
       │
       └────► Fallback: Physical LAN address     (~0.09 ms)

(For illustration, think of these as something like 100.x.x.x for the Tailscale interface and 192.168.x.x for the LAN — the actual addresses will obviously be specific to your own network.)

Since either path works equally well, we built in automatic dual-endpoint failover in OpenAiCompatibleClient.php: if the primary address doesn't respond within 3 seconds — say, a device rejoins Wi-Fi after sleep and its interface takes a moment to rebind — the client quietly retries the fallback address before the visitor ever sees so much as a hiccup.

The Benchmarks: Three Real-World Prompt Sizes

Using telemetry pulled straight from Ollama's own server logs, we ran three prompt scenarios — a short question, a medium summary request, and a long RAG-context query — each once cold (uncached) and once with a warm prompt cache.

Short prompt (38 tokens)

Medium summary (51 tokens)

Long RAG context (92 tokens)

The pattern across all three: the network itself is a rounding error. Every meaningful win here came from the prompt cache, not from which IP address the request happened to travel over.

What We Actually Learned

1. The prompt cache is doing almost all the work

On the long RAG-context query, prompt evaluation dropped from 13.6 seconds to 0.23 seconds — a 98.3% cut — purely because Ollama's KV-cache kept the shared system-prompt prefix warm in RAM. Anyone asking a follow-up question, or any two visitors whose questions happen to route through similar context, gets to skip almost the entire "thinking" delay. This is the same prefix-caching trick from Part I, just now measured under real, messy production traffic instead of a clean benchmark.

2. Nobody should ever see a raw cURL error

Under heavy load, or right after a cold start while the model was still warming up, visitors were occasionally shown something like Operation timed out after 120002 ms — a perfectly accurate, perfectly useless error message for a human being. OpenAiCompatibleClient.php now catches these transport failures and shows a friendly "Our AI assistant is currently warming up…" message instead, while the real error still gets written to a log file for whoever's actually debugging it. Small fix, disproportionately better first impression.

3. Answers were getting cut off mid-sentence

Complex page summaries were occasionally running into the token budget and stopping abruptly, mid-thought. Bumping max_tokens from 128 to 256 in ai-chatbot.yaml, plus adding proper context truncation bounds in ContextIndexer.php, fixed the abrupt endings without meaningfully hurting response speed — turns out most answers just needed a little more room to finish their sentence.

What's Next

Real token-level cost tracking. Parsing the actual prompt_eval_count and eval_count numbers Ollama already reports, straight into the interactions log, for accurate cost and throughput numbers in the admin dashboard instead of estimates.

• Part I: From 120 Seconds to Under 1 Second
• Part II: Setting Up Your Own AI Server
• Part III: Building an AI Chatbot for Grav CMS
• Part IV: Benchmarking Tailscale, LAN, and Chatbot Optimizations (you are here)