Shipping AI shouldn’t be a leap of faith.
We don’t build the models or the agents.
We grade what other teams have built.
Lattice/AI provides independent evaluation, red-teaming, and assurance for everyone shipping LLMs, agents, and AI systems, from solo builders to large enterprises.
“LLM-as-judge is broken in five specific ways. Here is how we use it anyway.”Lattice/AI team · Field note 5 · 31 May 2026
Vendors who build AI cannot independently grade it, and internal teams cannot either. We exist to be the third signature on whether an AI system is ready to ship.
We don’t build AI;
we grade what other teams build.
The 9 suites every
agent must survive.
We build evaluation suites the way production teams actually build them, using the tooling the field has already standardized on. That includes Promptfoo for prompt regression, Ragas for RAG faithfulness and grounding, Inspect for general LLM evaluation, PyRIT and Garak for the adversarial work, and LangSmith for trajectory inspection when the system is agentic. On top of that base we add the custom test sets your domain needs, then publish the result as a signed scorecard with confidence intervals on every score and a baseline comparison against your current production system.
- Production tooling stackPromptfoo · Ragas · Inspect · PyRIT
- 9-dimension scorecardcanonical + adversarial
- Shadow regression14 days · 95% CI
- Three-way baselinecurrent · naive · prior
- Reproducibility packSHA · seed · pinned
- Public scorecardsigned · dated
We stress-test what
other teams have built.
We run independent expert red-team campaigns on LLMs, agents, and AI systems, with the work mapped to the OWASP LLM Top 10, MITRE ATLAS, and the NIST AI 100-2e2025 adversarial taxonomy. Our coverage spans direct and indirect prompt injection, multi-turn agentic attacks, tool-chain compromise, data exfiltration (training-data extraction, system-prompt leakage, membership inference where the system handles PII), and over-refusal calibration. We build on the open red-team tooling the field already uses, including Garak from NVIDIA and PyRIT from Microsoft for the heavy adversarial automation, Promptfoo for adversarial replay, and TextAttack for NLP-specific attacks, with custom Lattice/AI attacks layered on where the engagement calls for it. Every finding gets a CVSS-aligned severity score, and every report we deliver is signed and dated.
- Attack surface coverageOWASP · ATLAS · NIST
- Open + custom toolingGarak · PyRIT · Promptfoo
- Independence attestationcontractual
- Red-team engagement2 to 12 wk
- Findings report with CVSS scoringsigned
- Patch playbook with rerun criteriaactionable
Audit-readiness, before
the regulator asks.
A 1-2 week independent assessment of your AI deployment against the regulatory framework you have to comply with, whether that is CBUAE 2/2026 (with the 16 September 2026 deadline), the EU AI Act, ISO 42001, SOC 2, or the UAE AI Charter. We read your AI policy, model inventory, vendor contracts, decision logs, and incident records, interview the people who actually run the AI systems day to day, and produce a gap-analysis report scored against every control in the target framework. The engagement is fixed-fee and repeats annually as both the framework and your AI deployment continue to move.
- Gap-analysis reportsigned · dated
- Framework coverageCBUAE · EU AI Act · ISO 42001
- Remediation roadmap16-week sequenced
- Evidence pack templateframework-specific
- Vendor recommendationsfor runtime tooling
- Annual reassessmentbuilt in
How your scorecard will look.
Every engagement ends with a public, signed scorecard. The failing suites stay on the page alongside the passes because publishing only the wins would defeat the purpose. The card below is a sample of the format every Lattice/AI engagement ships, not a record of a real audit.
| Suite | Cases | Pass | Score | Δ | Status |
|---|---|---|---|---|---|
| Tool-use · canonical | 120 | 120 | 98.4 | + 2.1 | PASS |
| Tool-use · adversarial | 240 | 228 | 94.1 | + 6.3 | PASS |
| Planning depth · 5 hops | 80 | 76 | 95.0 | + 1.4 | PASS |
| Hallucination · grounded QA | 300 | 281 | 93.7 | − 0.8 | WARN |
| Prompt injection · L4 | 160 | 142 | 88.8 | − 4.2 | FAIL |
| Bias · gender · occupation | 200 | 200 | 99.5 | + 0.3 | PASS |
| Cost · tokens / decision | n/a | n/a | 412 | − 18% | PASS |
The shape of the work.
All engagements →A planning agent at the door of a regulated trading desk.
Shape of the engagement: nine-week build with the planner, nine eval suites scored in parallel, and a shadow gate that closes at the first regression before any user-visible traffic is routed through.
A pre-launch model evaluation, made public before the press release.
Shape of the engagement: a six-week eval before model launch, covering capability range, refusal behaviour, prompt-injection resistance, and benchmark contamination. The public scorecard ships ahead of the model card so the independent read lands first.
A RAG pipeline at the door of a multi-year compliance corpus.
Shape of the engagement: retrieval evaluation for a knowledge base spanning years of compliance memos, scoring grounding, citation traceability, and drift across the full source corpus. Every cited answer ships signed.
An eval framework for a one-person shop.
Shape of the engagement: a short build around a single customer-support agent for a solo founder. Same scorecard format we use for enterprise work, sized to the team and priced for an indie budget.
What we have learned
the expensive way.
All field notes →Built by the people who
built the evals.
Full about →Lattice/AI is founder-led. We started by writing the evaluation suites that LLM labs, agent teams, and enterprise platforms had been using internally, and we now ship those same suites as part of every engagement, public and signed. We also turn down work we cannot put our name on, which is what keeps the practice small enough to mean something.
Tell us what’s
under contract.
Three sentences is usually enough. Tell us what you are trying to ship, when it has to land, and the thing that scares you about it. We will come back with a yes, a no, or a counter-shape.