Ollama on Intel Celeron, Part V: Deconstructing Chain-of-Thought (CoT) Pitfalls, KV-Cache Latencies, and System Rule Defenses

ollama chain-of-thought qwen grav-cms php benchmarking

In Part IV of this series, we benchmarked local LAN and Tailscale mesh network latency for grav-ai-chatbot, proving that network transport was fast and reliable.

But shortly after deploying reasoning models like qwen3.5:2b to production, users began reporting a perplexing bug: sending a simple greeting like "hello" repeatedly returned the generic error message:

"An unexpected connection error occurred. Please try again later."

Investigating this failure revealed a deep interaction between Chain-of-Thought (CoT) model reasoning, low max_tokens limits, Ollama KV-cache behavior, and client-side fallback defenses. Here is what was actually happening under the hood and how we fixed it step-by-step.

1. The Root Cause: CoT Monologue Leaks & Empty Content Errors

Modern local reasoning models (such as Qwen 3.5 and DeepSeek R1) output responses through two channels in Ollama's API:

  1. reasoning: An internal thinking stream / monologue (`Thinking Process:
  2. Analyze the Request...`).
  3. content: The final answer text meant for the user.

When a user sent a short input like "hello", qwen3.5:2b initiated an extensive internal monologue:

{
  "id": "chatcmpl-801",
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "message": {
        "content": "",
        "reasoning": "Thinking Process:

1. **Analyze the Request:**
   * Input: "hello"
   * Intent: Greeting and checking in...
2. **Determine Appropriate Response:**...
3. **Drafting the Response:**...
4. **Refining the Response:**..."
      }
    }
  ],
  "usage": {
    "completion_tokens": 512,
    "prompt_tokens": 18,
    "total_tokens": 530
  }
}

This produced two critical failure modes:

  • Token Budget Exhaustion: When output tokens (max_tokens) were set low (256/512 tokens), the model consumed 100% of its budget inside "reasoning", hitting finish_reason: "length" with "content": "". The PHP client (OpenAiCompatibleClient.php) evaluated !empty($content) as false, causing ChatbotHandler.php to throw the generic custom connection error message to the visitor.
  • Monologue Leaking: When max_tokens was larger, the model completed reasoning and outputted text, but the raw monologue string (`Thinking Process:
    1. Analyze...`) was stored in conversation history and shown in chat UI.

2. The KV-Cache Pitfall: How Prompt Caching Masked the Problem

During developer testing, repeating the exact same test input ("hello") produced inconsistent results. Sometimes it failed immediately, and other times it returned a fast answer.

This inconsistency was caused by Ollama KV-Cache (Context Prompt Caching):

  1. Context Reuse: Ollama reuses pre-computed Key-Value (KV) attention states for identical system + user prompt prefixes.
  2. Masked Latency & Behavior: When repeating identical prompts, Ollama skipped full prompt evaluation and served cached tokens. This hid the true token allocation behavior during repeated manual testing.
  3. Cache Invalidation Requirement: To uncover the true model behavior, testing had to be conducted using unique, non-repeating prompts (or by restarting Ollama to purge the KV-cache).

Varying prompt inputs during benchmarking was essential to isolate the actual CoT behavior.

3. What We Actually Implemented: The Architecture & Code Changes

To solve CoT leakage, token exhaustion, and provide configurable admin controls, we made the following specific updates to the codebase:

A. Custom System Prompt Rules & System Rules Prepending (ChatbotHandler.php)

We added a new configuration option custom_system_prompt in Grav Admin and plugin settings.

In ChatbotHandler.php, we implemented logic to resolve system prompt rules:

  • If custom_system_prompt is populated in Grav Admin, the plugin executes the user's custom instructions.
  • If custom_system_prompt is empty/blank, it automatically falls back to the exact default system rule: 
    Answer the user directly. Do not include any thinking process, reasoning steps, scratchpad, or internal monologue. Keep your total response under approximately {max_response_chars} characters.
  • The {max_response_chars} placeholder dynamically interpolates the soft character limit setting.

Prepending System Rules to Site Context

Crucially, system prompt rules are prepended at the very top of the system prompt context ($siteContext) before any RAG page documentation or site context is attached:

// Code implemented in ChatbotHandler.php
$customSystemPrompt = trim($this->config['custom_system_prompt'] ?? '');
$maxChars = (int)($this->config['max_response_chars'] ?? 1000);
$defaultSystemPrompt = "Answer the user directly. Do not include any thinking process, reasoning steps, scratchpad, or internal monologue. Keep your total response under approximately {max_response_chars} characters.";

$template = !empty($customSystemPrompt) ? $customSystemPrompt : $defaultSystemPrompt;
$systemRule = str_replace('{max_response_chars}', (string)$maxChars, $template);

// PREPEND SYSTEM RULES TO SITE CONTEXT
$siteContext = trim($systemRule . "

" . $siteContext);

Why Prepending Rules is Essential: Placing system directives (CoT suppression & response length constraints) at the very beginning of the system prompt ensures the LLM reads and prioritizes these instruction constraints before processing large RAG documentation context blocks. This guarantees that reasoning models obey output formatting rules regardless of how much RAG context follows.

B. Soft Response Character Limit Setting (max_response_chars)

We added a new setting max_response_chars (default 1000) across all plugin configuration layers:

  • blueprints.yaml: Added Admin UI input field for soft character response limits.
  • ai-chatbot.yaml & user/config/plugins/ai-chatbot.yaml: Added default key max_response_chars: 1000.
  • languages.yaml: Added English and multilingual labels (MAX_RESPONSE_CHARS).

C. Defense-in-Depth Sanitization & Error Logging (OpenAiCompatibleClient.php)

In OpenAiCompatibleClient.php, we added two backup defenses:

  1. <think> Tag Sanitization: Strips any <think>...</think> XML tags from response content.
  2. Reasoning Fallback & Audit Logging: If the model returns empty content and fallback to reasoning occurs, it records an explicit warning/error entry in user/data/ai-chatbot/error.log: 
    // Code implemented in OpenAiCompatibleClient.php
$answer = trim($content);
if (empty($answer) && !empty($reasoning)) {
   try {
       if (class_exists('Grav\Common\Grav')) {
           $logger = new Logger(\Grav\Common\Grav::instance());
           $logger->logError("AI Model returned empty content; fallback to reasoning applied [Model: {$this->model}].", 'AI_MODEL_API');
       }
   } catch (\Throwable $t) {}
   $answer = trim(preg_replace('/<think>.*?<\/think>/s', '', $reasoning));
}

4. Token Usage & Resource Savings Matrix

Below is the empirical benchmark comparison measured before and after implementing our changes:

Metric / Benchmark BEFORE Implementation AFTER Implementation Savings / Improvement
Test Query Prompt "Greetings, how are you today?" "Greetings! What assistance can you offer me today?" Unique prompts to bypass KV-cache
Completion Tokens Generated 512 - 2048 tokens ~60 tokens ~88% - 97% Token Reduction
Monologue Token Waste 512 tokens (100%) 0 tokens (0%) 100% Monologue Eliminated
Useful Answer Tokens 0 tokens (content: "") ~60 tokens (content populated) Infinite Efficiency Gain
Response Latency (CPU) >45.0s (or HTTP Timeout) ~0.4s - 1.2s ~98% Faster Response Time
CPU Hardware Compute Energy 100% CPU load for 45s+ Minimal load for ~0.4s ~90% CPU Compute Savings

Output Comparison Sample

BEFORE (Token Exhaustion Failure):

  • Generated Text: `"Thinking Process:
    1. Analyze the Request: Input is hello..."`
  • User Result: An unexpected connection error occurred. Please try again later. (HTTP 500 error due to empty "content").

AFTER (Direct & Token-Efficient):

  • Generated Text: "Hello! I am here to help you navigate this website, summarize blog pages, answer questions regarding software engineering and DevOps, or assist you with custom Grav CMS and AI integration details. How can I assist you today?"
  • User Result: Instant, clean answer with zero CoT leakage.

5. Key Takeaways for Local SLM/LLM Deployments

  1. Prepend System Rules to System Prompts: Always place CoT suppression directives at the top of the system prompt context so the model parses formatting constraints before reading large RAG documentation.
  2. Provide Configurable System Prompts & Fallbacks: Provide site admins with a custom_system_prompt setting, but always fall back to a robust default system rule if left empty.
  3. Beware KV-Cache Masking: During LLM testing, vary test prompts to avoid false positives caused by KV-cache hit rates.
  4. Log Fallback Events: Log warning events when empty content forces fallback to reasoning streams so administrator logs remain actionable.

Previous Post Next Post