My AI assistant has been quietly broken for about a month. Depending on which part you count, a lot longer.
It’s an assistant called Hermes that lives on my laptop and runs on open models through Ollama’s cloud. I set it up months ago and then mostly stopped looking at it. This weekend I finally looked.
The backup model had been retired by the provider back in July. The helper model, the one that handled its small side jobs, was retired in late September. Nobody told me any of this. The assistant kept calling model names that no longer existed, failed, and carried on waiting for the next job like nothing had happened.
And every morning it still had three scheduled jobs: an AI news digest, a social media brief, and a “system update”. All three switched on, failed to load a model, and logged their errors to Discord. Or tried to: the bot itself had lost access to that Discord channel at some point. I never pinned down exactly when, and I’ve decided against doing archaeology on my own negligence. Weeks earlier I’d moved to Slack. As I put it this weekend: “Discord is no longer the main channel, we use Slack.” Somewhere in that move, nobody updated the old bot. So: three jobs a day, every one of them broken, reporting to nobody, for about a month.
When I went digging, my first reaction was to be annoyed at the assistant. The fairer answer is that nothing in the setup was ever designed to tell me when something died. The error handling existed. It pointed at a channel I’d chosen to stop reading. The monitoring was me, and I’d stopped.
The next question was whether those jobs were even worth saving. The answer was mine from the start: “I was testing automations, and I have better ones now.” Deleted all three without much ceremony. A month of daily failing, and the only casualty was my belief that things were working, which is honestly the part that bugs me most.
The fixes came after staring at that for a while. First, context. I use Claude Code on my laptop, Claude Code in the cloud when I’m travelling, and a Grok bot for data jobs, and each new session lately has started with me re-explaining the same things about myself. So now there’s a private repo of plain text files: who I am, what I’m working on right now, which agent owns which job. Every session reads it first. Handoffs between agents go into one Notion table, and Slack is where the ping arrives.
Then the models, picked the boring way. I wrote three small tasks (call the right tool, write a three-bullet morning brief, extract tasks from a note as JSON) and ran them across about 25 models on Ollama and OpenRouter. Some big-name models failed the easy stuff: picked the wrong tool, or spent their entire budget “thinking” and never gave an answer at all. The cheapest, fastest ones were often fine.
Two things I wasn’t expecting. The model I’d been using as the main one sat in the most expensive usage tier on my plan, and cheaper models passed the same three tasks, so the everyday jobs moved to those instead. And eleven of the models ran through OpenRouter, which cost about 4 cents total. That was the whole lab bill for deciding who does what in my setup. The standing preference around here is: use free as much as possible.
The third change is the least interesting and probably the most important. A monthly script, no AI anywhere in it, asking one boring question: is anything dead, or is anything new? A model retired, a provider changed what it offers, a job I now count on that didn’t exist last month. If the answer changed, the script leaves me a note, in the one place I still look.
After building it, I realised it’s a small version of something ops people solved ages ago, the dead man’s switch. The system has to check in on a schedule, and the alarm fires the moment check-ins stop showing up. Healthchecks.io is built entirely on this: your cron job pings it when it succeeds, and you get notified when the ping doesn’t arrive. Their homepage states the failure mode plainly: “A failed cron job often has no immediate visible consequences and can go unnoticed for a long time.” My script is a monthly, slower cousin of the same instinct.
I don’t fully trust any of this to stay fixed, either. The repo itself can go stale the same way the Discord bot did: I change projects, the files keep describing me from months ago, and each agent reads old context with total confidence. The monthly check is a hedge against my own not-noticing. First run is in November. If it finds something dead, good, the system worked. If it says everything is fine, I’ll be suspicious of that too, which is probably the correct amount of trust to have after a month like this.
References
- Healthchecks.io - Cron job monitoring built on the dead man’s switch idea: a job pings it when it succeeds, and you get alerted when the ping stops.
- Healthchecks.io docs, monitoring cron jobs - The failure modes of unmonitored cron jobs, and how ping-based monitoring catches them.
- Ollama - Runs open models, locally or in its cloud; most of the model tests ran on its cloud models.
- OpenRouter - One API in front of hundreds of models; the eleven models tested here cost about 4 cents total.
- Claude Code - The coding agent used on the laptop and in the cloud.