Career Analysis

STAR Interview Report — AI Architect

Joel Johnston 2026-06-18 Post-stroke design

STAR Interview Report — AI Architect

Joel Johnston | AI Systems Architect

General Purpose — Adaptable to Any AI/ML Architecture Role

Candidate Summary


Name	Joel Johnston
Title	AI Systems Architect
Experience	20+ years — US Bank (10yr), Wells Fargo (10yr), WCTC Adjunct (7yr)
AI Stack	Python (10yr), distributed inference routing, multi-model orchestration (8+ providers), local LLM mesh, agent framework design
Core Stack	C#/.NET (25yr), SQL Server (20yr), Java (15yr), Linux (30yr)
AI Portfolio	robonet-forge (631 tests), Sentinel anomaly detection (62 tests), LLM discovery/routing, MCP server (11 tools), wire protocol with HCTH signing
GitHub	github.com/xsubi — 23 repos, ~500 commits, 3,248 tests
Education	B.S., Computer Science
Current Focus	Distributed local AI inference mesh, autonomous agent framework, zero-cloud AI architecture

Situational Responses

Q1. How would you design a system that orchestrates multiple AI models for different tasks?

Situation: Building robonet-forge — an autonomous AI development pipeline where different LLM models have different strengths. Claude Opus excels at architecture and planning. Claude Sonnet excels at code generation. GPT handles narrative prose. Grok handles scientific documentation. No single model is best at everything.

Task: Design a multi-model orchestration system that routes tasks to the optimal model based on task class, manages provider failover, and maintains consistent output quality across heterogeneous AI backends.

Action: Built a tier-based provider abstraction with capability routing:

Provider abstraction layer (LLMProvider ABC) — unified interface across 8+ providers (Claude, GPT, Ollama, Groq, HuggingFace, Cloudflare Workers AI, xAI, local models)
Tier routing — tasks classified by type (architecture → Opus, implementation → Sonnet, documentation → GPT/Grok), each tier maps to a provider+model configuration
ProviderCapability enum — TEXT, CODE, VISION, EMBEDDING — providers declare what they can do, router matches task requirements to capabilities
Failover chain — if primary provider fails, request cascades to next capable provider
OpenAI-compatible provider — single adapter speaks to any OpenAI-API-compatible endpoint (Ollama, Groq, vLLM, etc.)
Forgemaster decomposition — breaks use cases into task envelopes, assigns tier based on task complexity, dispatches to appropriate model

Result: 631 tests across the forge product. The system transparently routes inference across cloud and local models. A task submitted to forge gets decomposed, classified, routed to the optimal model, executed in an isolated worktree, tested, and pushed to PR — fully autonomous. Model selection is invisible to the workflow; the architecture handles it.

AI Architecture Relevance: Multi-model orchestration is the emerging pattern in enterprise AI. Single-model deployments are giving way to model routing based on cost, capability, latency, and data sensitivity. This system demonstrates production-grade routing with failover, not just prompt chaining.

Q2. Describe how you would build an AI system that keeps sensitive data on-premises while still leveraging AI capabilities.

Situation: Building distributed AI inference for robonet mesh — a system where nodes running local LLM servers (Ollama) auto-discover available models, advertise capabilities to the mesh, and route inference requests to the optimal node. The core requirement: zero cloud dependency, customer data never leaves the local network.

Task: Design and build a distributed local AI inference layer that provides the same capabilities as cloud AI (model selection, load balancing, transparent routing) without any data leaving the premises.

Action: Built three components on the existing mesh infrastructure:

LLMDiscovery — probes local Ollama instance via stdlib HTTP, enumerates available models (name, parameters, quantization, family, loaded status), advertises capabilities to the mesh's CapabilityRegistry. Periodic refresh via Timer ensures model availability stays current.
InferenceRouter — routes inference requests to the optimal node using CapabilityRegistry.best_match() with AND-matching on capability strings (["llm.inference", "llm.model.llama3.1:70b"]) and least-loaded strategy. Uses correlated request/reply (drone.request()) for non-streaming inference.
InferenceHandler — registered message handler on each node that executes inference locally via stdlib http.client to Ollama's OpenAI-compatible endpoint. Results return through mesh routing.

Key design decisions:

Zero external dependencies in core — stdlib http.client and json only
Capability strings encode model identity: llm.model.{name}:{tag}, llm.provider.ollama, llm.gpu.metal
Nodes without Ollama start normally (guarded imports, no LLM features, no errors)
Wire v2 protocol provides priority queuing, hop-count TTL, and HCTH cryptographic signing for all inference traffic

Result: 24 tests for LLM discovery, 34 tests for inference routing. A user can query any model on any node from any node — the mesh handles routing transparently. Adding inference capacity means plugging in hardware and joining the mesh. Infinite horizontal scaling. Zero cloud. Zero data exfiltration risk.

AI Architecture Relevance: On-premises AI is the fastest-growing segment of enterprise AI deployment, driven by data sovereignty requirements (GDPR, HIPAA, banking regulations). This system demonstrates that cloud-equivalent AI infrastructure can run on local hardware with zero external dependencies.

Q3. How would you approach building an AI-powered anomaly detection system?

Situation: Robonet mesh — distributed autonomous nodes operating independently across a network. Nodes can behave anomalously due to compromise, misconfiguration, resource exhaustion, or software faults. Traditional threshold-based alerting produces excessive false positives and misses subtle behavioral drift. No centralized authority to make detection decisions.

Task: Design a distributed anomaly detection system (Sentinel) that identifies behavioral deviations across mesh nodes without centralized authority, with consensus-based response.

Action: Built a three-layer distributed immune system:

Layer 1 — SentinelDrone (local behavioral baseline): Each node maintains a rolling statistical profile of its own behavior. Deviations measured using z-score analysis against the node's own baseline. Configurable sensitivity thresholds and sliding time windows.
Layer 2 — SentinelQueen (cross-node correlation): When a local anomaly is detected, the drone publishes an AnomalySignal to the mesh. The SentinelQueen correlates signals across multiple nodes to distinguish isolated events from systemic patterns. CorrelationResult captures the multi-signal analysis.
Layer 3 — Consensus eviction: If correlated anomalies exceed a confidence threshold, Sentinel initiates a consensus vote to quarantine or evict the affected node. No single node can unilaterally act. Trust dynamics adjust node reputation based on historical accuracy.

Pluggable AI provider interface (SentinelAIProvider) allows the correlation layer to optionally use LLM analysis for complex pattern detection. ZScoreProvider implements the statistical baseline as a concrete provider.

Result: 62 tests covering the full detection pipeline. Zero false positive rate in testing. The system distinguishes between "node is slow today" (local variance, no action) and "node is exhibiting a pattern consistent with compromise" (correlated signals, consensus response). Patent candidate for the distributed immune system approach.

AI Architecture Relevance: Anomaly detection at the edge without centralized processing. Each node is both sensor and analyst. The consensus model prevents single-point-of-failure decisions. The AI provider abstraction means detection can run on pure statistics (z-score) or escalate to LLM-based analysis — the architecture supports both without modification.

Behavioral Responses

Q4. Tell me about a time you had to make a critical architectural decision with incomplete information.

Situation: Designing the trust model for robonet mesh communication. Nodes join and leave the mesh dynamically. Any node could be compromised. Standard approaches (PKI, certificate authority) require centralized trust — incompatible with a decentralized mesh. No established pattern existed for the specific requirements.

Task: Design a trust protocol that works without centralized authority, scales to arbitrary mesh size, and provides tamper-evident audit trails.

Action: Designed HCTH (Hash-Chain Trust Handshake) — a blockchain-adjacent protocol where trust is earned through sustained cryptographic chain continuity:

Each message extends a hash chain between two nodes
Trust score increases with chain length (sustained honest communication)
Chain breaks (tampering, replay, impersonation) are immediately detectable
No central authority issues or revokes trust — it emerges from behavior
Full wire format specification, trust scoring algorithm, and attack surface analysis completed before any code was written

The decision was made with incomplete information about real-world attack patterns. The mitigation: designed the protocol to be conservative (trust grows slowly, breaks instantly) and documented all known attack vectors with their defenses.

Result: HCTH implemented in Wire v2 protocol. Patent candidate — no known prior art for behavioral trust accumulation via hash chains in mesh networks. The protocol provides tamper-evident logging of all mesh communication, including AI inference requests — directly applicable to AI audit trail requirements in regulated industries.

AI Architecture Relevance: AI systems in production need trust and audit infrastructure. HCTH provides cryptographic proof of what was requested, by whom, when, and what was returned — without a central logging server. For banking or healthcare AI deployments, this addresses the "how do we audit AI decisions?" requirement at the protocol level.

Q5. How do you ensure AI systems are reliable and maintainable in production?

Situation: Building robonet — 20 packages, 209,000+ lines of code, sole architect with AI-augmented execution. Output averaged approximately 5,846 lines per day. No team for code review. The AI components (forge, Sentinel, LLM routing) needed to be production-grade despite the development velocity.

Task: Maintain production-grade reliability for AI components at sustained high throughput without peer review.

Action: Implemented specification-first development with automated structural integrity:

201 formal use cases serve as source of truth — all code derived from specifications, including all AI components
Gap scanner enforces nine cross-reference integrity rules: every code file traces to a specification, every specification traces to code. Violations caught at build time.
3,248 tests across the product family (2,590 robonet core + 631 forge + 27 stamp). Tests run on every push — if tests fail, nothing merges.
Zero external dependencies in core (stdlib only) — eliminates supply chain risk, version conflicts, and upstream breaking changes
Forge autonomous pipeline — scan → lock → worktree → AI worker → test → push → PR → cleanup. The AI development tool is itself tested and specification-driven.

Quality bar: The entire codebase can be regenerated from the use case specifications alone using AI. If the specs are complete enough to rebuild the product, the engineering discipline is sound.

Result: Gap scanner: 0 failures, 0 warnings, 0 gaps. All AI components have dedicated test suites: Sentinel (62), LLM Discovery (24), Inference Router (34), Forge (631). The specification-first approach means AI components are as rigorously defined as infrastructure components — no "experimental" code in production.

Q6. Describe a situation where you had to integrate AI capabilities into an existing system without disrupting it.

Situation: Robonet mesh existed as a fully operational distributed system — networking, pub/sub, heartbeat, election, query engine — before any AI capabilities were added. Production nodes were running. The AI inference layer (LLM Discovery, Inference Router) needed to be added without breaking existing functionality.

Task: Add distributed AI inference capabilities to a running mesh without modifying existing components or requiring coordinated deployment.

Action: Followed the existing handler registration pattern in DroneRuntime:

Guarded imports — LLM modules import conditionally. If a node doesn't have Ollama, it starts normally with no AI features and no errors.
Capability advertisement — AI capabilities use the same CapabilityRegistry that handles all other mesh capabilities. No special AI registry.
Message handling — InferenceHandler registers on the llm:request channel using the same drone.add_message_handler() mechanism as every other handler.
Request/reply — InferenceRouter uses the same drone.request() correlated request/reply pattern used by QueryEngine and other mesh services.
Zero new dependencies — inference calls use stdlib http.client, same as existing HTTP providers.

No existing code was modified. The AI layer was purely additive — new files, new handlers, new capability strings. Existing tests continued to pass without modification.

Result: 120 AI tests pass alongside all existing tests. Nodes with Ollama gain inference capabilities automatically. Nodes without Ollama are unaffected. Mixed-capability meshes work transparently — inference routes only to nodes that can handle it.

AI Architecture Relevance: Enterprise AI adoption fails most often at integration, not at model selection. The ability to add AI capabilities to existing infrastructure without disruption, downtime, or forced upgrades is the difference between a demo and a deployment.

Technical Responses

Q7. How would you design an AI agent framework for autonomous task execution?

Current implementation: robonet-forge — autonomous AI development pipeline.

Architecture:

Component	Function
Forgemaster	Decomposes use cases into task envelopes with UC references, acceptance criteria, and tier classification
Striker workers	AI agents executing in isolated git worktrees — each task gets a clean environment
Task lifecycle	Scan repository → lock target → create worktree → dispatch striker → execute → run tests → push → create PR → cleanup worktree
Provider abstraction	LLMProvider ABC with ProviderCapability enum, 8+ backend providers
MeshBridge	Sync-to-async bridge for forge → mesh communication
Tier routing	Tasks classified by complexity, routed to appropriate model (Opus for architecture, Sonnet for implementation)

Design principles:

Isolation — each agent operates in its own worktree. No shared state, no race conditions, no contamination between tasks.
Specification-driven — agents work from use case specifications, not vague instructions. The spec IS the acceptance criteria.
Test-gated — agents must pass tests before pushing. Failed tests block the pipeline.
Human-in-the-loop — PR-based review. Agents propose, humans approve. No autonomous deployment to production.
Provider-agnostic — swap Claude for GPT for Ollama without changing the framework. The agent logic doesn't know or care which model executes it.

Result: 631 tests. The framework has autonomously produced code, created PRs, and maintained specification traceability across the entire robonet product family. Operational since March 2026.

Q8. What is your approach to prompt engineering and LLM integration?

Philosophy: Bimodal prompting — structured specifications for complex tasks, bare directives for tactical work.

Structured (complex tasks):

Use case specifications serve as prompts — formal documents with preconditions, postconditions, acceptance criteria
The spec is self-contained: an agent receiving only the spec can produce the correct output without additional context
UC-first development means the prompt engineering happens at the specification level, not at the API call level

Tactical (quick tasks):

Short directive prompts with full context compressed into minimal tokens
"Fix the test in test_inference_router.py line 42 — the mock is missing a return value" — no preamble, no context setting, just the instruction

Multi-model validation:

Critical outputs are validated across models (Claude, GPT, Grok)
Each model has known strengths and failure modes — the prompt is adapted to the model's characteristics
Error rates tracked per model — corrections always feed back into prompt refinement

Integration patterns:

OpenAI-compatible endpoint — single adapter for any provider that speaks the OpenAI API format (Ollama, Groq, vLLM, Azure OpenAI)
Streaming vs batch — streaming for interactive use, batch for autonomous pipeline
Token management — context window awareness, automatic chunking for large specifications
Rate limiting — RateGovernor primitive caps per-model request rates and spend

Anti-patterns I avoid:

No prompt injection vulnerabilities — user input never interpolated into system prompts without sanitization
No hallucination tolerance in critical paths — outputs validated against specifications
No single-model dependency — every integration point has a fallback provider

Q9. How do you approach AI security and governance in production systems?

Current implementations:

Sentinel — Distributed Immune System (Patent Candidate):

Three-layer anomaly detection (local behavioral baseline → cross-node correlation → consensus eviction)
AI provider interface allows LLM-assisted threat analysis while keeping statistical detection as the primary layer
No single node can unilaterally quarantine another — consensus prevents weaponized false positives

HCTH — Hash-Chain Trust Handshake (Patent Candidate):

Tamper-evident audit trail for all mesh communication, including AI inference requests
Every inference request and response is cryptographically chained
Provides regulatory-grade audit capability: who requested what model, with what prompt, when, and what was returned
No central logging server — audit trail is distributed and tamper-evident by protocol design

ResourceBudget + RateGovernor:

Per-model spend caps prevent runaway inference costs
Rate limiting prevents abuse of AI endpoints
Budget primitives are first-class mesh objects — configurable per node, per model, per user

Plugin Tier Security:

5-tier trust hierarchy for code that interacts with AI components
Kernel-level (tier 0) through raw scripts (tier 4)
AI inference capabilities require appropriate tier authorization

Data Sovereignty:

Zero-cloud inference architecture — all AI processing on local hardware
Customer data never leaves the network perimeter
No telemetry, no training data exfiltration, no cloud API calls for inference

Governance philosophy: Security and governance are infrastructure, not afterthoughts. They're built into the protocol layer (HCTH), the detection layer (Sentinel), the resource layer (Budget/Rate), and the trust layer (Plugin Tiers). Adding AI to a system doesn't get a security exemption — it gets the same rigor as every other component.

Q10. How would you evaluate and select AI models for an enterprise deployment?

Framework: Task-class routing with empirical validation.

Selection criteria by task class:

Task Class	Primary Criteria	Secondary Criteria
Architecture/planning	Reasoning depth, context retention	Cost is secondary — quality matters
Code generation	Speed, syntax accuracy, framework knowledge	Cost matters at scale
Documentation	Prose quality, domain accuracy	Model-specific (GPT for narrative, Grok for technical)
Anomaly detection	Consistency, low hallucination rate	Must be deterministic or nearly so
Classification/routing	Latency, cost per token	Accuracy threshold, then optimize for speed

Evaluation methodology:

Benchmark on real tasks — not synthetic benchmarks. Run actual use cases from the specification library through candidate models.
Cross-model validation — same task, multiple models, compare outputs. Divergence reveals model-specific biases or weaknesses.
Error rate tracking — catalog failures per model per task class. Build an empirical capability matrix, not a marketing-claims matrix.
Cost modeling — tokens per task class × volume × price per token. The cheapest model that meets the quality threshold wins.
Fallback testing — verify the failover chain works. If the primary model is down, does the secondary produce acceptable output?

Current model routing (production):

Model	Assignment	Rationale
Claude Opus	Architecture, strategy, evaluation	Deepest reasoning, best at nuanced analysis
Claude Sonnet	Code generation, implementation	Fastest code output, best syntax accuracy
GPT	Narrative, resumes, human-facing prose	Superior emotional tone and flow
Grok	Scientific documentation, research	Direct, analytical, no social padding
Ollama (local)	Privacy-sensitive inference, bulk processing	Zero data exfiltration, unlimited tokens

Key insight: Model selection is not a one-time decision. It's a routing table that evolves as models improve, costs change, and new providers emerge. The architecture must support model swapping without code changes — hence the provider abstraction layer.

Supplemental: AI Architecture Portfolio

Systems Built

System	Description	Evidence
Distributed AI Inference Mesh	Auto-discovery, capability routing, local execution across mesh nodes	58 tests (24 discovery + 34 routing)
Autonomous Agent Framework	Scan-lock-worktree-striker pipeline for AI-driven development	631 tests
Distributed Anomaly Detection	Three-layer behavioral analysis with consensus response	62 tests, patent candidate
Multi-Model Orchestration	8+ providers, tier routing, failover, capability matching	Production since March 2026
Cryptographic Trust Protocol	HCTH — hash-chain behavioral trust for AI audit trails	Patent candidate, no known prior art
MCP Server	Model Context Protocol server: 11 tools, 4 structured resources	Production
Wire v2 Protocol	Length-prefixed JSON with priority queues, hop-count TTL, HCTH signing	All AI inference traffic uses this
Voice/Persona System	26 AI personas with mutable voice profiles, TTS provider abstraction	Post-stroke accessibility layer

Zero External Dependencies

The entire AI infrastructure in robonet core runs on Python stdlib only. No numpy, no torch, no transformers, no langchain, no vector databases. This is a deliberate architectural decision:

No supply chain risk — no upstream package can break the system
No version conflicts — stdlib is the Python version
Portable — runs on any Python 3.9+ installation without pip install
Auditable — every line of code is visible, no black-box dependencies

External dependencies exist only at the boundary: Ollama (local LLM server), cloud provider APIs (optional, gated by API key). The framework itself is pure stdlib.

Banking-Relevant AI Use Cases

Use Case	Approach
On-premises AI (data sovereignty)	Local inference on bank hardware — no data transmitted externally
Audit trail for AI decisions	HCTH hash chain provides tamper-evident logging of all inference requests
Document processing at scale	Distributed inference across nodes — workload fans out automatically
Anomaly/fraud detection	Sentinel: behavioral baseline per entity, z-score deviation, consensus escalation
Legacy system modernization	UC-first: define legacy behavior as specifications, derive modern code from specs
AI cost governance	ResourceBudget + RateGovernor primitives cap per-model spend and request rates
Regulatory compliance	Specification traceability: every AI component traces to a formal use case

References

Name	Relationship	Notes
Jim Kloetske	WF Manager	Cold-called Joel into Wells Fargo. Overnighted offer. Talks to Joel daily. Says Joel would be a great manager. Providing reference.
Scott Schlatter	WF Principal Eng / Manager	Claimed Joel from interview: "I want Joel." Quote: "Your educated guesses are better than most people's knowledge." Potential reference.
EJ (DewWow)	20+ year colleague	First job together at Beta Systems. Flexible on title. Providing reference.

Prepared June 18, 2026