LLM Fallback

When no skill or WASM handler matches a task, the broker falls back to an LLM (Language Model) API.

How It Works

The LLM fallback is the last resort in the routing cascade:

  1. Skill match — check registered patterns (free)
  2. WASM handler — try local WASM execution (~$0.001)
  3. LLM fallback — send to language model API (~$0.03+)

Configuration

broker:
  enabled: true
  routing:
    llmAsLastResort: true
    llmProvider: anthropic
    llmModel: claude-3-haiku

Set your API key:

export ANTHROPIC_API_KEY="your-key-here"

Cost Tracking

Every LLM call is tracked in the cost system:

naos broker stats

Output:

BROKER STATISTICS (today)
─────────────────────────
Total tasks:     142
Skill matches:   89 (63%)
WASM handlers:   31 (22%)
LLM fallbacks:   22 (15%)

Cost savings:    $3.51 (vs all-LLM)
Avg latency:     45ms (vs 2.1s all-LLM)

Reducing LLM Usage

To minimize expensive LLM calls:

  1. Add more skills — cover common task patterns
  2. Lower confidence threshold — match more tasks to skills
  3. Add WASM handlers — for complex but deterministic tasks
  4. Enable caching — avoid duplicate LLM calls
broker:
  routing:
    confidenceThreshold: 0.6   # lower = more skill matches
  cache:
    enabled: true
    ttl: 3600