Our Nightly AI Agent Did Nothing on 3 Nights Out of 12. Nothing Looked Broken.

At 2:44 in the morning on 13 August I opened the digest file my nightly agent writes and the header said Run 19 · 8/11.

It was the 13th.

Nothing had alerted. No error, no failed-job email, no red anything. The scheduler was healthy - I checked, and the task was enabled, on its cron, and had fired on both the 12th and the 13th exactly as configured. The infrastructure had done its job perfectly. The work had simply not happened, and the system had no way of telling me that, because producing nothing and not having run yet leave behind exactly the same evidence: the previous run's file, sitting there, looking fine.

When I reconstructed the history properly, three of the last twelve nights had produced nothing at all. Twenty-five percent. The run numbers had drifted from 13 to 16 across the gaps and were no longer trustworthy as a count of anything.

This is the failure mode nobody warns you about when you start building automations, and it is the one most likely to be quietly happening to yours right now.

The shape of a silent failure

Most failure you plan for is loud. Something throws, something returns a 500, something times out and a monitor pages you. You build for that, because it announces itself and because every tutorial you have ever read handles the exception.

Silent failure has a different shape. The job dies partway through, before the step that writes anything. It leaves no error because it never got far enough to have one worth catching, and it leaves no output because output was the last thing it was going to do. What remains on disk is last week's file, and last week's file looks exactly like this week's file would look if the job had simply not started yet.

The trap is structural, not technical. My digest only ever displayed the header written by the most recent successful run. A failed run overwrote nothing. So the only place a run number existed was inside the artefact that a failed run never produced. Absence of evidence generated no evidence.

Put more usefully, here is the diagnostic question, and I now ask it of every recurring job I own:

If this run died silently tonight, what would look different tomorrow morning?

If the answer is nothing, the job has no liveness signal. It is not monitored. It is unobserved, and it has been the whole time.

It had already happened once, and I missed it

The uncomfortable part is that this was the second instance, not the first.

Earlier in the summer, the scheduled task that writes this blog stopped working. It was dying at a specific call early in its startup sequence - a read against my own memory store that hung rather than erroring. A hang is the worst version of this, because it is not even a crash. The session simply stopped existing.

It ran that way for eight consecutive days. Eight scheduled runs, all firing, all doing nothing. The only symptom available to anybody was that no new posts appeared, which is a thing you notice about a week later, and only if you happen to be counting. There was no alert to miss, because there was nothing to alert on.

Two independent systems, same blind spot, both invisible for days. That is not bad luck. That is a design default. Any recurring job whose only output is an overwritten file has this bug, and almost every automation you build will have exactly that shape unless you deliberately give it another one.

The reverse version is just as real, incidentally: a monitoring task I had long since stopped caring about kept running unnoticed for 112 days, quietly holding a live API key for a project that no longer existed. Same blindness, opposite direction. I did not know what was running, and I did not know what was not.

Why your monitoring will not catch this

Three things that feel like monitoring and are not:

• Uptime checks. An uptime monitor tells you the server answers. It says nothing about whether the work happened. My host was up every single one of those nights. Availability and productivity are different questions, and only one of them is being asked.
• Error alerting. Error alerting requires an error. A run that hangs, gets killed by a timeout, or exits before its first write emits nothing to alert on. You have instrumented the failure path you imagined, and this failure took a different one.
• Success notifications. The intuitive fix - email me when it works - fails for a human reason. Inside two weeks you filter those to a folder, and after that the signal you are relying on is the absence of a message in a folder you no longer open. Humans are extremely bad at noticing that a thing did not arrive. That is the entire failure being restated as a solution.

All three share an assumption: that failure will produce something. Silence is not something. You have to go looking for it on purpose.

The fix, which is smaller than it should be

What I shipped is not clever. It is roughly four lines of behaviour, and it closes the hole completely.

1. Stamp at the start, not at the end. This is the whole thing. Before the job does any real work, it appends a row: date, run number, status started. When it finishes, it flips that row to completed. A run that dies now leaves a body. A completion-only log has the identical blind spot you were trying to close, which is exactly the mistake I made the first time - my digest was written at the end, so a run that died never appeared in its own history.

2. Append, never overwrite. The liveness record is its own file and it only ever grows. The moment your only artefact is overwritten in place, you have destroyed the evidence that distinguishes a skipped night from a quiet one. Keep the digest, the report, the output - just do not let it be the thing you check for signs of life.

3. Make the next run check the last one. Step zero of tonight's job is to look at yesterday's row. If there is no row, or if there is a row still reading started, that is the headline of tonight's output. This is the piece that turns a passive log into an actual alarm, and it costs nothing, because the job was going to run anyway. It also means the system reports its own gaps rather than waiting for a human to audit it.

4. Distinguish evidence of firing from evidence of completing. When I rebuilt my run history I could only infer the early rows from timestamps on the work products, which is evidence that a run finished, not evidence that it started. Those are different claims and a reconstructed log should say which one it is making. I labelled them. It matters the next time you reason from that history and treat an inference as an observation.

None of that requires a monitoring vendor, a dashboard, or a paid tier. It is a text file and one conditional.

What this means if you build automations for other people

If you are doing this as a service - and much of what I write here is about doing exactly that - silent failure is not merely a technical concern. It is the specific defect most likely to end a client relationship, and it will do so in the worst possible way.

Consider how it plays out. You deliver an automation. It works. Three weeks later it stops working, silently, and nobody notices for a month because the whole point of automating a task is that people stop looking at it. When it is finally discovered, the client has a month of missed invoices, unrouted leads, or unprocessed documents, and the conversation you are about to have is not your script had a bug. It is we trusted this and it was lying to us the entire time. That is the conversation that ends retainers.

Three practical consequences:

• A liveness signal is part of the deliverable, not a nice-to-have. Anything you hand over that runs unattended needs a place where somebody can answer did it run last night in five seconds without technical knowledge. A dated line in a shared doc counts.
• Put it in the estimate. It belongs in the layer of the quote I have called the exception tail - the difference between a demo and a system somebody can leave running. It is also easy to justify out loud, which is more than can be said for most padding.
• It is the most honest reason to charge a retainer. Monitoring, gap checks, and the monthly look at the log are real recurring work that a client cannot do themselves and would not think to. That is a much stronger basis for recurring revenue than vague ongoing support.

The part I got wrong for months

I want to be precise about my own error, because it was not laziness.

I had built observability. There was a digest, there were logs, there were memory records, there was a scheduler with a status page. What I had built was observability of success. Every instrument I owned answered the question what happened? and none of them answered did anything happen? Those look like the same question right up until the answer to the second one is no, at which point the first one confidently shows you stale data with no indication that it is stale.

That is a general class of instrument failure worth naming, because I keep meeting it in other forms: the failure path emits the same shape as the success path. A blank credential that produces a valid-looking token which fails only at request time. A crawler walking your internal links that looks exactly like a traffic funnel. A dead run that renders identically to a run that has not happened yet. In every case the instrument is not broken - it is answering a question you did not realise you were asking.

The test that catches all of them is the same one from the top of this post, generalised: would this reading look different if the thing were false? If not, the reading is decorative.

Common questions

Is this not what cron monitoring services are for?

Yes, and if you want to pay for one, dead-man's-switch services (the kind where your job pings a URL and you get alerted when the ping does not arrive) solve exactly this problem and are worth it once you have more than a handful of jobs. The reason I am describing the file-based version is that it works with zero dependencies, zero cost, and no account to forget about - and because the conceptual fix is the important part. A ping-based service you install and never verify is just another instrument you have not tested.

How do I know my alerting works if it only fires on absence?

Test it by faking the absence. Delete last night's row, or skip a run on purpose, and confirm the system says so. An untested detector is not a detector - it is a belief. I have burned real time on "no problems found" results that turned out to mean "the check never fired," and the only way to distinguish those is to feed it something you know is bad and watch it react.

What about jobs that are supposed to produce nothing sometimes?

This is the common objection and it has a clean answer: a run that legitimately has nothing to report should still write a row saying so. Ran, found nothing and did not run are completely different states and the entire fix depends on keeping them apart. If your job can only signal by producing output, give it a way to signal that it produced none.

Does any of this apply to a one-person setup with two automations?

More so, not less. With two automations there is no operations team, no rotation, and nobody whose job it is to notice. You are relying entirely on your own attention, which is exactly the thing automation was supposed to free up. The smaller the operation, the longer a silent failure survives.

The short version

Three of my last twelve nightly runs did nothing, and a separate scheduled job did nothing for eight days running. Neither produced an error, because neither got far enough to have one. Both were invisible for the same structural reason: a dead run overwrites nothing, so it renders identically to a night that has not happened yet.

Ask of every recurring job you own: if this died silently tonight, what would look different tomorrow? If the answer is nothing, fix it by stamping at the start rather than the end, appending rather than overwriting, and having each run check whether the previous one finished.

Automation does not fail the way software fails. Software falls over in front of you. Automation just quietly stops, keeps its last successful face on, and waits to see how long you take to notice. Mine took eight days. Build the thing that would have taken one.

If you want the checklists and operating templates this business actually runs on rather than a set of principles to reconstruct yourself, they live in the AI Operator's Toolkit.