Three failures stood out in my AI workflows this week, and none of them called for a different model. A loose prompt turned a September audit note into a stale outline of current code. A slide parser dropped vocabulary examples while a unit test asserted the empty output. And mobile memory recall broke while the client parsed the server’s response.
Why a Sunday Lab Notebook
On weekdays, this site covers spreadsheets, BI tools, and the checks an operator runs before trusting a number. Once a week, preferably every Sunday, I’ll look back at a week of working with AI across my own projects: the AI assistant on this site (its code is public on GitHub), CareerTalkLab, the learning platform I’m building, and the Obsidian and Open Brain memory setup that ties them together.
Every incident follows the same pattern. What I expected. What happened. Why it broke. What changed. How I checked it. What’s still open.
This first issue covers incidents from November 11 to 14. I checked everything below by November 14.
The Week at a Glance
| Date | Where | What broke | Cause | What changed |
|---|---|---|---|---|
| Nov 12 | Vault assistant and Open Brain | An outline listed already-built features as open gaps | A loose prompt on a September audit note | Check the source’s date and scope; write explicit comparison prompts |
| Nov 11 | Lesson slide pipeline | Slides 2 to 5 lost their example sentences | The parser dropped them, and a test asserted the empty result | Fixed the parser, rewrote the tests, checked the slides by eye |
| Nov 11–12 | Android memory recall | Every recall call I tested failed with a JSON parse error | The client missed a header because of letter case and parsed a stream as JSON | Added an opt-in JSON response mode on my server; tested on the phone |
| Nov 13–14 | Publishing workflow | Extra rework between stages | I changed one part of the usual sequence on purpose | Caught the mistakes, corrected them, and went back to the usual sequence |
A Quick Prompt Turned a September Baseline into “Current” Gaps
What I expected. I was planning an article about how my projects use models, prompts, and harnesses. In my vault I had an audit note dated September 11, 2026. I knew it was two months old. I wanted Gemini Helper, the AI plugin I use inside Obsidian, to read that note as a baseline, check what had changed since September, and then list the gaps that were still open for model evaluation, prompts, and harnesses.
What happened. I didn’t ask for that. I wrote a quick, loose prompt and didn’t spell out the two steps clearly enough, so the model had to guess what I wanted. The assistant gave me a clean, confident outline built on the September numbers: 34 test cases for the assistant and 15 for retrieval. It treated bounded agent loops, tool calling, and classical machine-learning baselines as work still to do.
The September note itself was more careful than the outline. It already mentioned some ad-hoc tool calling. But its gap list was two months old, and the outline read it as today’s list.
What the code showed. I compared the outline against the live repositories:
- The same test suites had grown. The assistant suite went from 34 to 37 cases. Retrieval went from 15 to 17.
- Across all the evaluation suites in the site’s repository, there are now 91 versioned cases: 37 assistant, 17 retrieval, 12 CV, 12 agent-loop, 10 tool-calling, and 3 production follow-ups. That’s how many cases are defined. It isn’t a count of evaluations I ran this week.
- The site’s bounded agent loop already existed. Its last change was on October 13. Its configured limits are 12 steps, 2 retrieval rounds, 3 tool calls, a 45-second deadline, and an estimated-cost ceiling of $0.05.
- CareerTalkLab already had classical machine-learning rules for predicting learner errors. I built them in June from a logistic regression and a depth-4 decision tree, trained offline on 2.62 million tokens from the Duolingo SLAM dataset.
None of that was new this week. It matters here only because the outline said it didn’t exist.
What changed. The mistake was on both sides. The assistant relied on a dated snapshot without checking any project notes in my vault. It could not check the live code since it is restricted to the vault. I was on my mobile, trying to brainstorm. Otherwise, I’d be in the CLI and would be able to search all surfaces. But I had asked a vague question about a source I knew was old. As I put it afterward: “quick and dirty prompts produce quick and dirty responses.”
I came away with two habits. First, when an answer cites a source, I check that source’s date and scope before I trust the answer. Second, when the answer depends on comparing an old snapshot with current work, I write the comparison out as steps. For this case, the prompt should have said something like: “Read the September 11 audit as a baseline. Then check the relevant project directories in the vault for newer notes and changes since that date. List each September gap as closed, partly closed, or still open, with the file that shows it.”
Still open. The audit note still shows September numbers at the top. I run the audit every three months, so the next one is in the first week of December. I haven’t decided whether to add a clear date warning, note when the next audit is due, or both. Until I do, it can mislead the AI unless the prompt spells out every facet of the task.
The Slide Parser That Dropped Examples, and the Test That Protected the Bug
What I expected. CareerTalkLab turns each lesson into a set of marketing slides. For A1 Lesson 21, “The Daily Routine,” slides 2 to 5 should show each vocabulary word with a definition, a Turkish translation, and an example sentence that starts with “Use:”.
What happened. On November 11, the slides came out with empty example blocks. Every single-line vocabulary entry lost its example sentence. The pattern slides also showed a stray “Pattern:” label at the start. I caught both by looking at the rendered slides.
Why it broke. The parser lives in marketing/scripts/generate-a1-assets.py. Its split_vocab() function handled two input shapes. Multi-line entries already looked for an example line. Single-line A1 entries fell back to splitting the text on periods, and that branch always set the example to an empty string.
Here is one real entry from the lesson:
spell. To say letters in order. Türkçe karşılığı: Harf harf söylemek. Use: "Please spell your name."
Before the fix, split_vocab() returned an empty example for that line. After the fix, it returns "Please spell your name." with the quotation marks kept.
The bug inside the safety net. An existing unit test, test_vocab_slide_single_line_a1_format_unchanged, ran that exact input and asserted that the example was empty. The test passed every time. It was protecting the bug.
A passing test only tells you the code does what the test asserts. If the assertion describes broken output, a green run locks the broken output in place.
What changed. The fix in pull request #339 extracts “Use:” and “Example:” sentences before the period split. It also strips the leading “Pattern:” label in render_pattern_slide(). I replaced the old test with three:
- One checks that the example sentence in a single-line entry comes through.
- One checks that an entry with no example still parses, with an empty example.
- One checks that the rendered pattern slide no longer shows “Pattern:”.
How I checked it. The three tests pass. I also re-rendered slides 2 to 5 and looked at each one myself. I always do. Tests check the fields; the eye check confirms the slide a learner will actually see.
Still open. Earlier A1 lessons went through the same single-line rule. I still need to check whether any of their slides shipped with empty example blocks.
Mobile Memory Recall Broke While the Client Parsed the Response
What I expected. I use Open Brain as a shared memory for my AI tools. My server is built on the public Open Brain (OB1) project and speaks the Model Context Protocol (MCP). On desktop, Obsidian’s Gemini Helper plugin connects to it without trouble. I expected the same plugin, version 1.26.3, to work in Obsidian Mobile on my Android phone.
What happened. Every memory call I tested from the phone failed with a JSON parse error. The error pointed at text starting with event: message. That’s the first line of a server-sent events (SSE) frame, not JSON.
Why it broke. My Open Brain server returned SSE by default, with a Content-Type of text/event-stream. The MCP transport specification allows a server to answer a request with either plain JSON or SSE, and it requires clients to handle both.
The plugin had code to handle SSE. It decided which path to take by reading the content-type header in lowercase. On Android, the app’s HTTP layer, Capacitor, kept the header name in title case as Content-Type. HTTP header names are case-insensitive, so both spellings are valid. But the lookup missed, the plugin assumed JSON, and it ran JSON.parse on the SSE text.
What changed. On November 12, I released version 8 of my server, open-brain-mcp. I didn’t change the default, because desktop clients were working with SSE. Instead, I added an opt-in plain-JSON mode. A client can turn it on with a ?format=json query parameter or an x-mcp-response: json request header. These are switches on my server, not standard MCP options.
The change sits in supabase/functions/open-brain-mcp/index.ts. It passes a JSON-response setting to StreamableHTTPTransport and adds the new header to the CORS allow list. There were no database or schema changes.
How I checked it. On the server side, the change passed deno check and an isolated handshake test. Probes against the live server confirmed three things:
- A request without the opt-in still gets
text/event-stream. - Both opt-in switches return
application/jsonfor theinitializerequest, the tool list (6 tools), and a stats call. - The
notifications/initializedmessage is accepted with a 202 status and no body, as the specification describes for notifications.
Then I set the Android client to use JSON mode and tried recall on the phone. It worked.
Still open. My server flag is a workaround. The underlying fix is a case-insensitive header lookup in the client plugin. Until that exists, any phone client that hits this needs the opt-in set by hand.
A Workflow Lesson That Wasn’t an AI Failure
Not every rough patch this week came from a model. On November 13 and 14, I pushed several articles through my publishing pipeline: brief, brief review, testing, draft, and draft review. AI models from different providers handle different stages. On part of that run, I changed one thing in the usual sequence. I wanted to see if I could get the same quality if I cut one of the models out of the process. I do this once in a while to evaluate the process for both cost and quality, and to check whether the usual sequence really is better before I make the decision to keep it as the default.
It created extra rework between stages. That wasn’t an AI problem. Each model I use is better at certain tasks, at least for my taste. Because I had changed only one thing, I was able to catch mistakes and review quality. I caught the fact that the target audience had changed on the articles, went back, made the corrections, and returned to the usual sequence.
Four Checks I’m Carrying into Next Week
- Did I write a real comparison prompt against a dated note, or a quick shortcut? If the answer depends on what changed since a snapshot, I ask for the comparison in steps.
- Is my test checking the output people need, or freezing today’s bug? Read the assertion, not just the green check mark.
- Did the failure happen in the model, or in the code around it? Parsers, headers, and transport settings fail too, and a stronger model won’t fix them.
- Am I debugging an AI failure, or a change I made to my own process? If I changed the workflow, I count that first.
The thread through all four is the same one I use with dashboards: check the source and its date before blaming the screen. If you work with reports, the same idea shows up in why your Data Studio report shows old data.
Next Sunday, I’ll look back at week two.