The Scheduler That Only Fired After a Restart: an APScheduler Debugging Story

We shipped a monitoring feature — scheduled jobs that re-run saved analyses and email you when the results change. In testing, a bug report came in that sounded impossible:

"I set a schedule's next_run_at in the database and waited. Nothing fired. Then I stopped the app, started it again — and it fired immediately."

Fires after a restart but not during normal operation. If you've run APScheduler inside a busy async app, you may already be grinning. If not, this post will save you the afternoon it cost us.

The setup

Our design is deliberately boring — one APScheduler interval job (a "master tick") that polls the database:

_scheduler = AsyncIOScheduler()
_scheduler.add_job(
    _tick, "interval",
    seconds=60,
    id="master_tick",
    max_instances=1,
    coalesce=True,
)

Every 60 seconds, _tick queries for due rows (next_run_at <= now), takes a per-row Redis lock, and dispatches each one. No per-schedule APScheduler jobs, no state loaded at startup — the database is the source of truth on every tick.

That last property made the bug report confusing. Our first hypothesis (and the reporter's) was "the scheduler must be caching schedules at startup, and the restart reloads them." But the code plainly queries fresh each tick. Caching wasn't the story. Something was preventing the tick from running at all.

The trap: misfire_grace_time defaults to 1 second

APScheduler computes the next fire time for each job. When that moment arrives, the scheduler's internal loop wakes up and submits the job. But if the wake-up is late — because, say, the asyncio event loop was busy — APScheduler checks how late:

If the job is more than misfire_grace_time seconds past its scheduled time, it is not run. It's counted as a misfire and skipped.

And the default misfire_grace_time is 1 second.

Now consider what shares our event loop: a FastAPI app doing LLM-assisted data analysis. The scheduled jobs themselves run pandas operations in-process. Requests do CPU-bound dataframe work. It is completely routine for the event loop to be occupied for more than one second past a scheduled fire time.

So the failure sequence was:

1. master_tick is due at 12:00:00.
2. The event loop is busy for a few seconds (a user request, a pandas op).
3. APScheduler wakes at 12:00:07, sees the job is 7 seconds late — beyond the 1-second grace — and silently discards that run.
4. interval triggers don't queue missed runs; the next chance is 12:01:00. If the loop is busy again... goto 3.

On an idle dev server, ticks run like clockwork. Under real load, they skip — intermittently, which is the worst kind of bug.

Why the restart "fixed" it every time

This is the beautiful part. On startup, the app was moments old: the event loop was empty, the first tick fired dead on time, well within any grace window. The tick queried the database, found the overdue row the user had been staring at, and ran it.

So the restart wasn't reloading cached schedules — the user's hypothesis was reasonable but wrong. The restart was simply the only moment the event loop was reliably quiet enough for a tick to beat a 1-second deadline. Every restart manufactured one guaranteed tick. That's why the behavior read as "loads schedules at startup."

Debugging lesson worth keeping: when a symptom pattern-matches a familiar cause ("must be startup caching"), verify against the code before acting on it. We confirmed the tick queried fresh per-tick, then went hunting for what could suppress the tick itself — and checked the live job's actual config:

>>> scheduler.get_job("master_tick").__getstate__()
{'misfire_grace_time': 1, 'coalesce': True, 'max_instances': 1, ...}

There it was.

The fix (one line, plus a lesson)

_scheduler.add_job(
    _tick, "interval",
    seconds=_TICK_SECONDS,
    id="master_tick",
    max_instances=1,
    coalesce=True,
    # APScheduler's default misfire_grace_time=1s silently skips late
    # ticks on a busy event loop; allow a full interval of lateness.
    misfire_grace_time=_TICK_SECONDS,
)

A late tick within one full interval now runs anyway; coalesce=True ensures a pile-up collapses into a single run; max_instances=1 prevents overlap. For a polling tick, running late is strictly better than not running — the tick reads the database, so it always does the current right thing whenever it fires.

While in there, the same review caught a second, quieter bug: the per-row Redis lock (SET NX EX, 300s TTL) taken before dispatching a schedule was never explicitly released — a schedule couldn't fire twice within 5 minutes even when due. Locks now release in a finally. Two bugs, one review; both of the "works on an idle laptop" species.

Takeaways for your own APScheduler setup

1. Set misfire_grace_time explicitly, always. The 1-second default is almost never what you want in an async web app. For a polling tick, misfire_grace_time = interval is a sane choice.
2. "Works after restart" ≠ "state loaded at startup." It can simply mean "startup is the only quiet moment." Distinguish stale state from suppressed execution early — they lead to opposite fixes.
3. Interrogate the live scheduler, not the docs. scheduler.get_job(id) shows the actual effective config; ours took 30 seconds to check and ended the mystery.
4. Log misfires. APScheduler emits an EVENT_JOB_MISSED event — wire a listener to it and this whole class of bug becomes a log line instead of a ghost story.
5. If you take distributed locks around jobs, release them in finally. TTL expiry is a safety net, not a release strategy.

This bug turned up while building Qunta's data monitors — scheduled, zero-LLM re-runs of saved analyses that email you when results change. If your team checks the same dashboard every morning, that's the feature that stops it. Try Qunta free →