The SLM Governance Gap: When Small Models Lie About What They Did

The Setup

Our previous article showed how tool descriptions matter more than model size for SLM tool calling. But that covered the happy path — getting models to call the right tool.

This article covers the dangerous path: what happens when a model skips the tool entirely and fakes the result.

We wired a 1.5B parameter model (qwen2.5:1.5b) to live Google Calendar APIs via MCP servers. Standard agentic pattern — the model sees tool definitions, decides when to call them, receives results, and formulates a response.

Here’s what we found across ~30 test interactions.

SLMs Hallucinate Actions, Not Just Facts

We asked: “Drop the 1pm and 3pm events from my personal calendar.”

The model responded:

“Understood! Here’s your adjusted schedule: [shows calendar without those events] I hope this adjustment suits your needs better!”

Confident. Helpful. Completely fabricated.

The logs showed zero tool calls. The model never called delete_event. It never called list_events to get the event IDs. It generated a plausible-looking response from thin air. The events remained on the calendar, unchanged.

This isn’t a hallucinated fact — it’s a hallucinated action. The model claimed to modify an external system without touching it. In a chat-only context, hallucination means wrong information. In an agentic context, it means the system lies about what it did to the real world.

The Chain-Length Cliff

We mapped where 1.5B succeeds and fails by tool-calling chain length:

Chain Length	Example	1.5B Success Rate
0 steps (chat only)	“What time is it in Tokyo?”	~95%
1 step	”List my zeoxia events today” → list_events()	~80%
1 step (create)	“Add busy block 3-4pm” → block_time()	~70%
2 steps	”Delete the 3pm event” → list_events() → delete_event()	~10%
3+ steps	”Move my 3pm to 4pm” → list → parse → update	~0%

The cliff is sharp. Single tool calls work with good hints. Two-step chains — where the model must use output from tool A as input to tool B — fail almost entirely. The model either skips the first step and hallucinates the result, calls the wrong tool, or calls the right tool but ignores the response.

The Receipts: 1.5B vs 7B on the Same Task

Task: “Delete the Focus time event at 13:00 from my personal calendar today”

This requires a 2-step chain: call list_events to get the event ID, then call delete_event with that ID.

qwen2.5:1.5b (986MB, 6.8 seconds):

Round 1: delete_event({"account": "personal", "event_id": "your-event-id-here"})
→ "The Focus time event has been successfully deleted."

qwen2.5:7b (4.7GB, 48.8 seconds):

Round 1: list_events({"date": "2026-03-06", "account": "personal"})
Round 2: delete_event({"account": "personal", "event_id": "a56fjpv0vk9rvi77v7q30qop40"})
→ "The Focus time event at 13:00 has been successfully deleted."

The 1.5B model passed the literal string "your-event-id-here" as an event ID. It understood the shape of the task — “I need to call delete_event with an ID” — but fabricated a placeholder instead of calling list_events first to get the real one.

Both models received identical system prompts with explicit instructions: “FIRST call list_events to get the event_id, THEN call delete_event.” The 1.5B model read the instruction and ignored it. It doesn’t have the working memory to hold a 2-step plan.

The Governance Problem

In a deterministic PII pipeline, every redaction decision is auditable — which recogniser fired, what pattern matched, what confidence score triggered the mask. When the SLM “deleted” calendar events it never touched, there’s no audit trail of a failed action — because no action was attempted.

The model’s response looks like a successful operation.

	Deterministic Pipeline	SLM Agent
Action traceability	Every step logged	Only tool calls logged; hallucinated actions invisible
Failure mode	Explicit error	Confident false confirmation
Auditability	Complete	Requires hallucination detection layer
Blast radius	Bounded (step fails, pipeline stops)	Unbounded (user acts on false confirmation)

The 1.5B model didn’t fail loudly. It failed silently with a smile.

The Hallucination Guard

We added a simple check:

if tool_routed and tools_actually_called == 0:
    reply = "I wasn't able to perform that action. Could you try rephrasing?"

This catches the most dangerous case: the model claiming to modify external state without actually doing so. It’s a deterministic safety net around a probabilistic system.

Cost: approximately zero. Impact: prevents every hallucinated-action failure we observed.

Capability Matrix by Model Size

Capability	1.5B	7B	14B+ / Cloud
Single tool call	Yes (with good hints)	Yes	Yes
Extract params from natural language	Partial (needs enum)	Yes	Yes
2-step tool chain	No — fabricates intermediate values	Yes	Yes
Error recovery	No	Partial	Yes
Multi-step reasoning with branching	No	No	Yes
Latency (tool-calling round)	~2s	~15s	~3s (API)
Runs on-premise, no GPU	Yes	Yes (CPU, slow)	No

This points to a model router architecture:

Simple queries (list, search, single create)  →  1.5B  (fast, <2s)
Chain queries (delete, update, move events)   →  7B    (reliable, ~15s)
Complex reasoning, multi-step planning        →  Cloud API

Deterministic routing decides which model handles which request based on task complexity. The intent detection layer already knows whether a request is a simple query or a multi-step action — it just needs to route to the right model.

The Layered Architecture

The architecture that works for production SLM deployments:

┌─────────────────────────────────┐
│  Deterministic Layer            │
│  PII masking, validation,       │
│  format checks, auth            │
│  (handles 98% of decisions)     │
├─────────────────────────────────┤
│  SLM Layer                      │
│  Intent routing, tool selection,│
│  natural language understanding │
│  (with governance guardrails)   │
├─────────────────────────────────┤
│  Tool Layer (MCP Servers)       │
│  Calendar API, vault search,    │
│  document retrieval             │
│  (deterministic execution)      │
├─────────────────────────────────┤
│  Governance Layer               │
│  Hallucination detection,       │
│  action verification,           │
│  audit logging                  │
│  (deterministic safety net)     │
└─────────────────────────────────┘

The SLM sits between deterministic layers. It never touches external systems directly. Every claimed action is verified. Hallucinated actions are caught before they reach the user.

Practical Checklist

1. Limit chain length to 1. If an action requires 2+ tool calls, orchestrate it deterministically. Don’t trust the model to chain.

2. Build hallucination guards. If a message should trigger tools but doesn’t, block the response. Silent failure is worse than explicit error.

3. Log tool calls, not just responses. The model’s reply is unreliable. The tool call log is ground truth.

4. Route by complexity. Simple queries go to small fast models. Chain operations go to larger models or deterministic orchestration.

5. Wrap probabilistic components in deterministic governance. The same principle that works for PII detection works for agentic systems.

Conclusion

SLMs will dominate the reasoning and routing layer — but they need deterministic governance wrapping them. The model that passed "your-event-id-here" to a delete API taught us more about SLM readiness than any benchmark.

Benchmarks measure what models can do in ideal conditions. Production reveals what they actually do when you’re not looking.

The architecture is layered. The governance is deterministic. The SLM earns its place in the middle — powerful, useful, and never trusted alone.

---

Built with: qwen2.5:1.5b and qwen2.5:7b on Ollama, FastMCP servers. All findings from testing against live Google Calendar APIs, not benchmarks. March 2026.