In partnership with
AI Spotlight · The Failure Log
Build a Failure Log for Your AI Workflows
Everyone tracks their AI wins. Nobody tracks the hallucinations, the broken automations, or the outputs that quietly went out wrong. The people who get genuinely good at AI are the ones who log the failures and actually learn from them.
There is a pattern that shows up again and again among people who use AI tools seriously. Early on, they are excited. The outputs feel like magic. They build workflows, automate tasks, and tell everyone how much time they are saving. Then, somewhere around month three or four, the cracks start showing. An AI-generated email goes out with the wrong client name. An automation silently produces garbage for two weeks before someone notices. A research summary confidently cites a source that does not exist.
Most people respond to these failures by patching the immediate problem and moving on. A small number of people do something different. They write it down. Not in a vague, this-was-annoying way. In a structured, what-exactly-happened, why-did-it-happen, what-will-I-do-differently way. Those people compound. The others just accumulate the same frustrations on a longer timeline.
This issue is a practical guide to building a personal AI failure log: what it is, why it works, exactly what to put in it, and how to use it in a way that actually makes you sharper over time rather than just creating another document you never look at.
* * *
Why this matters
Why your AI failures are more valuable than your wins
When an AI workflow succeeds, it confirms what you already believed about how it works. When it fails, it reveals something true that you did not know. The hallucination tells you which topics the model confidently fabricates. The broken automation tells you which edge cases your prompt did not anticipate. The bad output tells you exactly where the gap is between what you asked for and what the model understood.
AI hallucination rates remain a genuine problem in 2026 across all major models, with different failure modes emerging depending on the type of task, the specificity of the prompt, and the domain being covered. Legal professionals report that a significant majority of AI-assisted document reviews require verification steps precisely because confident fabrication is a structural feature of language models, not a bug that gets patched away. The question is not whether your AI tools will fail. It is whether you will be paying attention when they do.
|
The compounding problem with unlogged failures When you patch a failure without logging it, you fix the symptom but not the pattern. The same class of error appears three months later in a slightly different form, and you have no record that you have seen this before. Over time, unlogged failures produce a false sense of mastery: you feel experienced with AI tools because you have been using them for a year, but your actual understanding has not compounded because you have not been systematically extracting lessons from the times they went wrong. |
* * *
Know what you are logging
The four types of AI failures worth tracking
Not every AI failure is the same, and treating them as interchangeable is one of the reasons most people do not extract useful lessons from them. There are four distinct failure types that show up repeatedly across AI workflows, and each one requires a different fix. Your failure log works best when you know which type you are dealing with before you write the entry.
|
The four failure types Type 1 — Hallucination: The model confidently stated something false. A statistic that does not exist. A source that was never published. A person attributed to a quote they never said. The failure is in the output. The fix is usually a verification step, a constraint in the prompt, or grounding the model with a reliable source document rather than asking it to recall from memory. Type 2 — Misunderstanding: The model understood the words but not the intent. You asked for a summary and got a transcript. You asked for a formal email and got a casual one. You asked for three options and got one long answer. The failure is in the gap between what you said and what the model interpreted. The fix is almost always a better prompt: more context, clearer constraints, an example of what good looks like. Type 3 — Drift: The workflow worked fine for weeks, then quietly started producing worse outputs. Nobody changed the prompt. Nothing obvious changed. But the model was updated, the input data shifted slightly, or an integration changed a format. The failure is invisible until you are comparing outputs over time. The fix is a regular review of your automation outputs against a small set of known-good reference examples. Type 4 — Brittleness: The workflow handles the common case perfectly but breaks on any variation. Works with English inputs, fails silently on French ones. Works on short documents, produces garbage on anything over a certain length. Works in the morning, times out in the evening. The failure is in the edge cases your setup was never designed to handle. The fix is either narrowing the workflow scope or adding explicit handling for the edge cases you have now identified. |
Every entry in your failure log should be labelled with one of these four types before anything else. The label is not just categorisation. It points directly at the class of fix you need, which cuts the time between noticing a problem and actually solving it.
* * *
The structure
Exactly what to put in each failure log entry
The most common reason failure logs do not work is that the entries are too vague to be useful later. "AI got it wrong on the report" tells you nothing when you read it three months later. A useful entry is specific enough that a future version of you can understand exactly what happened, why it happened, and what changed as a result. That specificity does not require a lot of time. It requires a consistent structure.
|
The seven-field failure log entry 1. Date and tool When it happened and which AI tool or automation was involved. This matters more than it seems: patterns often emerge by tool or by time period that you would never notice without a date column. 2. Failure type One of the four types: Hallucination, Misunderstanding, Drift, or Brittleness. Label it before you write anything else. 3. What I asked for The exact prompt, instruction, or workflow trigger. Not a paraphrase. The actual thing. This is the single most skipped field and the most important one. Without it, you cannot diagnose whether the problem was in the prompt or the model. 4. What I got The actual output or the description of what went wrong. As specific as possible. Include the exact wrong sentence, the exact broken output, or the exact error message if there was one. 5. What I expected One or two sentences describing what a correct output would have looked like. This forces you to articulate the gap precisely, which is the first step toward closing it. 6. Root cause hypothesis Your best guess at why it happened. You do not need to be certain. Prompt was ambiguous. Model does not have reliable data on this topic. Input format changed. Edge case I never considered. Even a wrong hypothesis is useful because it commits you to a theory you can test. 7. Fix applied or planned What you changed in response. New prompt language. Added a verification step. Narrowed the workflow scope. Changed the model. If you have not fixed it yet, write what you plan to do and by when. This field closes the loop and is what transforms the log from a complaint journal into a learning system. |
A complete entry should take three to five minutes to write. If it is taking longer, you are being too comprehensive. The goal is not a forensic report. It is a concise record that future you can read in thirty seconds and immediately understand.
Want to get the most out of ChatGPT?
ChatGPT is a superpower if you know how to use it correctly.
Discover how HubSpot's guide to AI can elevate both your productivity and creativity to get more things done.
Learn to automate tasks, enhance decision-making, and foster innovation with the power of AI.
See it in action
What a real failure log entry actually looks like
Here is what a properly completed entry looks like for a common failure type. This is the kind of entry that is genuinely useful six months later when you are diagnosing why a similar problem keeps appearing.
|
Failure Log Entry — Example Date and Tool June 2026 · ChatGPT research summary workflow Failure Type Hallucination What I Asked For "Summarise the key findings from recent studies on AI adoption rates in mid-size businesses and cite the sources." What I Got Three convincing statistics with specific percentages attributed to a McKinsey report (2025) and a Gartner study (Q1 2026). Neither report existed when I checked. The statistics were plausible but fabricated entirely. What I Expected Accurate citations to real, verifiable sources, or an honest acknowledgement that it does not have current data on this topic. Root Cause Hypothesis The model has a strong tendency to comply with citation requests even when it does not have reliable source data. Asking it to cite forces it to generate citation-shaped text rather than acknowledge uncertainty. Fix Applied Changed workflow: now provide the source documents directly in context and ask the model to summarise from those documents only. Added instruction: "If you cannot find the answer in the provided documents, say so explicitly rather than drawing on general knowledge." Also added a manual spot-check step before any statistics go into external-facing content. |
Notice what makes this entry useful: it is specific enough to diagnose, the root cause hypothesis is testable, and the fix is concrete. It took about four minutes to write. The next time this class of failure appears, this entry takes thirty seconds to find and immediately suggests the fix.
* * *
Making it a habit
How to actually use the log so it compounds over time
A failure log that only gets entries but never gets reviewed is just a more organised version of ignoring your failures. The review is where the compounding happens. Here is the cadence that works without creating another calendar commitment you will eventually ignore.
|
The three-layer review system Immediate — within 24 hours of the failure: Write the entry while the context is fresh. Focus entirely on fields three through seven: what you asked, what you got, what you expected, your hypothesis, and your fix. This is the capture step. Do not try to identify patterns yet. Just record accurately. Monthly — fifteen minutes, once a month: Scan all entries from the past thirty days. Look for: repeated failure types, the same tool appearing multiple times, fixes that were planned but never applied, and any patterns in when failures cluster. Update any entries where you now know the root cause was different from your original hypothesis. This is the pattern recognition step. Quarterly — thirty minutes, once per quarter: Read the full log. Write three sentences summarising the biggest patterns you see. Which tools fail most? Which failure type appears most often? Which fixes have actually stuck, and which ones required multiple attempts? These three sentences are your AI judgment memo for the quarter. They are also the most valuable thing you will write all year, because they represent compounded learning that almost nobody else is doing. |
* * *
Setup in five minutes
Where to keep it and how to set it up today
The best failure log is the one you will actually use. That means it needs to be low-friction to open, low-friction to add to, and easy to scan. Here are three formats that work, in order from simplest to most powerful.
| Format | Best for | Setup time |
|---|---|---|
| Notion table | Solo operators who want filtering by tool, date, or failure type | Five minutes to create seven columns and a template row |
| Spreadsheet | Anyone who wants to sort, filter, and see patterns at a glance across many entries | Two minutes to set up seven column headers in a new sheet |
| Voice memo | People who process better by talking; transcribe to a doc weekly | Zero setup; speak the seven fields immediately after a failure occurs |
The format matters far less than the consistency. Pick the one that requires the least activation energy for you personally. A mediocre format that you use every time beats a perfect format that you use once a month. The friction is the enemy, not the structure.
* * *
What separates people who get good from people who just get experienced
There is a difference between using AI tools for two years and learning from two years of using AI tools. The first produces familiarity. The second produces judgment. Familiarity tells you which tool to open. Judgment tells you when not to trust what it gives you, which prompts need a verification step, which automations require a human spot-check, and which failure types you keep hitting because you have not actually fixed the underlying cause.
The failure log is what converts experience into judgment. It closes the loop that most people leave permanently open. Every entry is a small investment in a compounding asset: a personalised, evolving record of exactly how your AI tools fail and exactly what to do about it. Nobody else's failure log will look like yours, because nobody else uses the exact combination of tools, prompts, and workflows that you do. That specificity is precisely what makes it valuable.
|
Everyone who uses AI tools fails. The people who get genuinely sharp are the ones who treat each failure as a data point rather than an inconvenience. Three to five minutes after the next time an AI gets it wrong. That is all it takes to start compounding. |
* * *
|
Before you go What is the worst AI failure you have had in the last three months — and did you log it? Hit reply and describe it in two sentences. The most instructive failures will be featured anonymously in a follow-up issue on the most common AI failure patterns and exactly how to fix each one. |
Until next time,
AI Spotlight
Practical AI, translated into real work, once a week.