Guides

Practical, hands-on guides on testing AI and LLM systems.

• Flaky AI tests are usually bad tests: how to stabilize a non-deterministic suite
  Nov 1, 2026

  Most AI test flakiness isn't the model's fault — it's exact-match assertions on output that was never guaranteed to stay the same. Fix the assertion, not the AI.

• Prompt injection is a QA problem now — here are the tests you should be writing
  Oct 31, 2026

  Prompt injection attacks hijack LLM behavior through malicious user input or retrieved documents. These are the four test categories every AI application needs.

• How to catch hallucinations in tests without a human reading every answer
  Oct 30, 2026

  Grounding checks, NLI contradiction detection, and a narrow LLM-as-judge rubric automate hallucination detection — no human in the loop required.

• Codex or Claude wrote my tests — how do I actually trust them?
  Oct 30, 2026

  A green AI-generated suite proves the code does what it already does, bugs and all. The only way to earn confidence is to break the code on purpose and watch the tests fail.

• 90% coverage and the bug still shipped: fixing your signal-to-noise
  Oct 29, 2026

  Coverage counts line executions, not correctness. A 90% score built on weak assertions is noise. Here's how to measure signal instead.

• From experienced tester to AI testing: the roadmap, in order
  Oct 29, 2026

  You don't need a data-science degree to test AI. Here's the order to actually learn it, built on what you already know as a tester.

• How to test an LLM or RAG app when the same prompt gives different valid answers
  Oct 29, 2026

  Use property checks, scored evals, and a golden dataset — never exact-match assertions. Non-determinism isn't the problem; asserting the wrong thing is.

• Real bug or model failure? How to tell the difference in AI tests
  Oct 27, 2026

  When an AI test goes red, it's either a genuine defect or the model behaving differently. Here's a practical way to separate the two instead of guessing.