Local vs Cloud LLM Routing from Production Experience

I build tools for K-12 school districts. The data I work with -- student names, grades, attendance records, emergency contacts -- is protected under FERPA, the federal law that governs student education records. That means every design decision has a compliance dimension that most AI application developers never have to think about.

This post is about a routing decision I made in a chat assistant for school office staff, why I made it, and what the numbers actually look like.

The Problem: Every Query Was a Third-Party Disclosure

The assistant is simple. Office staff type a question in plain English -- "pull up student 10001", "who's failing math?", "show me Maria Garcia's contacts" -- and the system looks up the answer from Aeries, the student information system that about 3,000 California school districts use.

The first version sent every query through Claude's tool-use API. User message goes to Anthropic, Claude decides which Aeries API calls to make, backend executes them, results go back to Claude, Claude formats a response. Standard agent loop, six rounds max.

It worked. But there was a problem I'd underestimated.

Under FERPA, sending student data to a third-party service counts as a disclosure. Even if Anthropic's enterprise API terms say they don't train on the data, even if they delete it after processing -- the act of sending Maria Garcia's attendance records to an external API is itself a recordable event. A district's Data Processing Agreement needs to cover it. A compliance auditor can ask about it.

And I was doing this for queries like "pull up student 10001." A simple key-value lookup. No reasoning needed, no fuzzy matching, no analysis. Sending that through an LLM is like calling a taxi to walk across the street -- you get there, but the trip creates paperwork you didn't need.

The latency was the other issue. A locally-handled ID lookup against the Aeries API takes about 3 milliseconds on localhost. The same query through Claude's tool-use loop takes 7 to 12 seconds. For an office worker who just needs to pull up a student before a parent walks through the door, that's the difference between useful and frustrating.

The Fix: A Regex Router in Front of the AI

I added a two-tier routing layer. Before any query touches Claude, a regex router tries to match it to a direct API call:

def route_query(message: str) -> dict | None:
    msg = message.strip().lower()

    # Bail early on anything analytical -- these need AI
    ai_keywords = ['who has', 'which students', 'how many', 'compare',
                    'worst', 'best', 'most', 'least', 'failing',
                    'average', 'trend', 'all students', 'every student',
                    'across', 'between', 'summary', 'report on',
                    'analyze', 'why', 'should i', 'what do you think']
    if any(kw in msg for kw in ai_keywords):
        return None

    # Student by ID: "pull up student 12345", "student #12345"
    m = re.search(
        r'(?:student|pull\s*up|look\s*up|get|who\s+is|info\s+(?:on|for)|id)'
        r'\s*#?\s*(\d{3,})', msg
    )
    if m:
        sid = int(m.group(1))
        return {"tool": "get_student", "args": {"student_id": sid},
                "pii_detected": ["student_id"]}

    # ... patterns for attendance, grades, contacts, enrollment,
    #     name search, list schools, list teachers, marking periods

    return None  # not matched -- send to AI

If the router matches, the query goes through a local execution path that calls the Aeries API directly, formats the response with Python string templates, and logs an audit entry with processing_mode="local" and third_party_disclosure=false. Student data never leaves the server.

If the router doesn't match -- the user asked something analytical, ambiguous, or multi-step -- the query falls through to Claude. The audit entry records processing_mode="ai" and third_party_disclosure=true.

There's one escalation path worth mentioning. If a user searches by name and the local search returns zero results (maybe they typed "Marya" instead of "Maria"), the router automatically escalates to the AI path so Claude can try fuzzy matching. The failure mode is slower, not broken.

The Numbers

I wrote a benchmark script that runs 10 locally-routable queries and 4 AI queries against a mock Aeries server. These are the results:

Path	Avg latency	Min	Max
Local (regex router)	3ms	2ms	9ms
AI (Claude Sonnet)	9,319ms	6,716ms	12,492ms

The local path is roughly 3,100x faster. That's not a typo -- it's the difference between a regex match plus one HTTP call versus a multi-round LLM conversation with tool use.

Cost scales similarly. Each AI query consumes roughly 1,500 tokens. Ten locally-routed queries save about 15,000 tokens. At Claude Sonnet's pricing, that's not life-changing money for a small deployment, but it adds up at scale, and more importantly, those are ten queries where student PII stayed on your server.

PII Filtering: The Deny-by-Default Layer

The routing decision is only half the story. Even on the local path, the raw Aeries API returns 80+ fields per student -- including SSN placeholders, medical flags, income data, and disciplinary records that have no business appearing in a chat response.

I handle this with a simple allowlist:

SAFE_STUDENT_FIELDS = [
    "StudentID", "FirstName", "LastName", "Grade", "SchoolCode",
    "SchoolName", "Sex", "Birthdate", "StudentEmailAddress",
    "InactiveStatusCode", "LanguageFluency",
]

def _filter(record: dict, fields: list[str]) -> dict:
    return {k: v for k, v in record.items() if k in fields}

This is a deny-by-default pattern. If Aeries adds a new field to their API response tomorrow -- say, DisciplinaryActionCode -- it gets silently dropped because it's not on the list. You have to explicitly opt in to expose a field. I prefer this over a blocklist, where a new sensitive field could leak through if you forget to add it.

The Audit Trail

Every query -- local or AI -- writes a row to a SQLite database:

CREATE TABLE audit_log (
    id                     INTEGER PRIMARY KEY AUTOINCREMENT,
    timestamp              TEXT NOT NULL,
    user_email             TEXT NOT NULL DEFAULT 'anonymous',
    session_id             TEXT NOT NULL,
    tool_name              TEXT NOT NULL,
    parameters             TEXT NOT NULL,
    student_ids            TEXT NOT NULL DEFAULT '',
    result_count           INTEGER NOT NULL DEFAULT 0,
    processing_mode        TEXT NOT NULL DEFAULT 'ai',
    third_party_disclosure INTEGER NOT NULL DEFAULT 1
);

The two columns that matter most are processing_mode and third_party_disclosure. A FERPA auditor can run one query to answer the question that keeps compliance officers up at night: "Which student records were sent to a third party, when, and by whom?"

SELECT DISTINCT student_ids, timestamp, user_email
FROM audit_log
WHERE third_party_disclosure = 1
  AND student_ids != '[]'
ORDER BY timestamp DESC;

The frontend surfaces this too. Each chat response shows a colored badge -- green for "Answered locally, no data sent to third parties," orange for "AI-assisted, data processed by Anthropic." It's transparent enough that a non-technical administrator can see what's happening without touching a database.

What I Got Wrong

The regex router is brittle. It handles about 15 phrasings per tool, which covers most of what office staff actually type, but it will miss anything creative. "Yo what's the deal with student twelve-three-four-five" is a valid student lookup that the router will send to AI. I originally thought I'd need to keep expanding the patterns, but in practice the failure mode is fine -- you get a slower, more expensive response, not a wrong one.

I also didn't build a unified test suite before migrating from CSV audit logs to SQLite. The existing test file still references the old CSV format, which means pytest fails on the audit tests. The validation and rate-limiting tests pass, but the gap is embarrassing for a project that's supposed to demonstrate compliance rigor.

If I were starting over, I'd also consider whether the PII allowlist should live in configuration rather than code. Right now, adding a safe field means editing app.py and redeploying. A YAML or JSON config file would let a district admin adjust the field list without touching source code. I didn't do this because the allowlist should change rarely and I wanted zero config complexity, but it's the kind of thing that matters when you're selling to multiple districts with different comfort levels.

What's Next

This assistant is Phase 1 of a broader K-12 AI orchestrator. The same routing pattern -- local for simple lookups, AI for reasoning -- applies across other school systems. Phase 2 adds Google Admin (user provisioning, Chromebook management) and Mosyle MDM (device policies, app deployment) as additional tool sources. The goal is queries like "which students got new Chromebooks this month and are failing math?" that span three APIs and require the kind of cross-domain reasoning that justifies an LLM round-trip.

The local router won't scale to that. When you're coordinating across systems, the queries are inherently complex. But the audit infrastructure -- tracking which data went where, which third parties saw it, and whether the disclosure was necessary -- carries over directly. And the principle is the same: don't send student data to a third party unless you actually need to.

For anyone building AI features on regulated data -- healthcare, education, financial -- the takeaway isn't "use regex instead of LLMs." It's that the decision of where to process data is itself a compliance-relevant event, and your system should make that decision visible, auditable, and defensible. The regex router is a 120-line function. The audit trail is a few SQLite columns. Neither is technically impressive. But together they turned "we send everything to Anthropic" into "84% of queries never leave our server" -- and that's the number that matters in a procurement conversation.