A full account of Phase 2.5 through today: one real production outage, four separate bugs hiding behind the fix for it, a language expansion honest about its own limits, and the session that finally made the alarms real.
Byte is a mobile PWA that turns EU news into 30-second swipeable cards — AI-summarised, sentiment-tagged, running on a Next.js frontend, a FastAPI backend, and a Postgres database, held together for under €6 a month by exactly one person. It has a real pipeline: RSS feeds and Perplexity search get fetched every two hours, Claude Haiku summarises and classifies each story, a scoring pass ranks them, and an archive job cleans up after itself. It is small, it is scrappy, and for most of its life it has quietly worked.
In the second week of August, it quietly stopped.
The first sign of trouble wasn't an error. It was an absence. Somewhere in early August, the fetch pipeline — which had been running every two hours on GitHub Actions since the earliest sprints — just stopped showing up in the run history. Not a failed run. No run at all.
The likely cause, once you do the arithmetic, is almost boring: GitHub Free private repos get 2,000 Actions minutes a month. The core fetch cron alone runs every two hours at roughly three minutes a run — about 1,080 minutes a month, on its own, before counting the marketing-content cron, CI on every active-sprint push, or the self-healing feed-repair bot. An active development month plausibly blows past 2,000 minutes without anyone doing anything wrong. And once the allowance is gone, scheduled workflows don't fail loudly — they simply stop being scheduled. No billing alert, because the default spend limit is €0. No in-app warning unless you've gone and configured one. The allowance resets on your billing-cycle date, not the calendar month, which makes “when did this actually break” its own small mystery.
The diagnostic tell, for anyone hitting this later: a gap in your Actions run history is this failure mode. A red ✕ is a different problem — that means the workflow ran and failed, which at least tells you something happened.
The actual fix was almost anticlimactic: Railway, where the backend already lived, has a native Cron Schedule per service. Move the fetch → discover → summarise → score → archive chain there, and the entire minutes-cap failure mode disappears — no free-tier ceiling to quietly walk into a second time.
// railway.json
{
"build": { "builder": "NIXPACKS" },
"deploy": {
"startCommand": "python fetch_news.py && python discover_news.py && python summarize_news.py && python pipeline/score_articles.py && python archive_old_news.py",
"cronSchedule": "0 */2 * * *",
"restartPolicyType": "NEVER"
}
}Alongside the migration, a dead man's switch: a tiny function that pings healthchecks.io at the end of every run, success or failure, and does nothing at all if it isn't configured — monitoring must never become a hard dependency for the pipeline to run in the first place.
def ping_healthcheck(success: bool = True):
if not HEALTHCHECK_PING_URL:
return # no-op if not configured
try:
suffix = "" if success else "/fail"
requests.get(f"{HEALTHCHECK_PING_URL}{suffix}", timeout=5)
except Exception:
pass # a failed ping must never crash a successful runOne decision worth naming: an earlier draft of this fix considered making the repo public, on the theory that public repos don't have the same Actions-minutes ceiling. That got explicitly declined. The Railway migration alone removes the actual failure mode; reopening a private repo's API-key hygiene for a fix that would've been redundant wasn't worth the risk. Reliability work isn't optional roadmap polish here — it gates every other investment, which is exactly the same reasoning that put Phase 0 first, months earlier.
Before any of Sprint 2.5.1's siblings got built, the plan for them went through a direct review — and the review caught something real. The next two sprints were meant to add a “Flow” mode that layers lighter, non-political content (culture, sport, lifestyle) alongside the existing hard-news feed. The original draft scored all of it — news and lifestyle alike — with one EU-relevance scorer.
An anchor-term list built to detect “the Commission” and “member states” will never fire on a Eurovision recap or a Vienna Christmas-market piece — so Flow mode would have shipped functionally empty.Phase 2.5 review, byte-growth-engineering-plan.md
The fix wasn't a lower threshold on the same scorer — that would have let genuinely irrelevant content through just as easily as it let lifestyle content through. It was two scorers: _score_institutional(), unchanged, for hard news, and a new _score_cultural() that anchors on country names, European cities, and cultural markers like Eurovision or the Champions League instead of “Brussels” and “the Commission.” Two different signals for two different reasons a story might matter to a European reader.
The same review pass also collapsed content_category from four overlapping, undefinable values — was a Eurovision recap “entertainment” or “pop_culture”? nobody could answer that consistently, which means Haiku couldn't either — down to a clean binary: news or lifestyle. The existing topic field already carried the display-level granularity (Politics, Sport, Culture); the new field only had one job — decide which scorer applies, and whether sentiment/framing classification even makes sense for that story at all. And category itself got moved from an LLM decision to a fact known at fetch time, from which feed group an article came from — simpler, cheaper, and impossible to drift.
The implementation PR carried all three Sprint 2.5 sprints at once, and — as usual with a plan doc's illustrative pseudocode versus the actual codebase — a handful of real corrections surfaced along the way:
fetch → discover → summarize → score_articles → archive. Moving only two would have left most of the GitHub Actions minute consumption exactly where it was — the actual root cause of the outage, untouched.eu_relevance_score already existed on news_summaries, from an earlier sprint, typed as an unused INT DEFAULT 0 that had never been populated. A naive ADD COLUMN IF NOT EXISTS migration would have silently no-op'd against it. The real migration retypes it to DOUBLE PRECISION and nulls out the placeholder zeros first.content_category, language, and eu_relevance_score turned out to be needed on news_raw too, not just news_summaries — the summariser reads unprocessed rows in a separate later pass and has no other way to know a row's category or language by the time it gets there.user_preferences table and GET/POST /api/preferences from scratch, following the same anonymous-uid pattern the existing streak endpoint already used — no localStorage, per the project's own standing rule.The honest line in this PR's own test plan is worth keeping: the real FastAPI backend and full pytest suite could not be run locally — the only Python available was 3.12/3.14, and the backend's pinned fastapi==0.95.2/pydantic==1.10.21 raise a hard ConfigError on import under both. Verification that session came down to clean syntax checks, eighteen real unit tests actually executed against the new scorers, and a live Next.js dev server driven in-browser to confirm the Settings sheet fired the right query parameters. CI was the real signal for everything else.
Real-device testing on Android found the new Focus/Flow and language toggles simply didn't respond to a tap — despite local testing, driven by JavaScript's own .click(), showing them working fine.
The reason is a small, instructive HTML mistake: the shared settings-row component always renders a <button> as its root element. The new rows correctly didn't pass their own click handler to that wrapper — the wrapper itself isn't what should be clickable, only the segmented control inside it is — which left that outer button in a disabled state. But the segmented control's own buttons were nested inside it, invalid HTML on its own, and a disabled ancestor suppresses everything nested under it once a real touch event tries to hit-test it. element.click() from a console bypasses native hit-testing entirely, which is exactly why local verification never caught it.
The fix mirrors a pattern already sitting one row above in the same file — a plain <div> container with the control as a sibling, never a descendant. Verified this time with a real pointerdown → mousedown → pointerup → mouseup → click sequence, not a shortcut.
The Railway cron migration went live. The build failed at first — Nixpacks couldn't find a requirements.txt at the repo root, because the existing web service actually runs with Root Directory set to backend/, and the new pipeline service needed the same setting. A one-line path fix, and the build went green.
Two days later, the numbers didn't add up. news_raw's unprocessed backlog had grown past 160 rows. news_summaries and pipeline_runs — both only ever written by the summariser — had gone completely silent since 22:39 UTC on the 13th. Adding the missing Anthropic API key, the first suspect, changed nothing.
The pipeline service was building via backend/Dockerfile, which Railway had silently preferred over the NIXPACKS builder railway.json actually declared. Dockerfile-built services aren't guaranteed to run a custom start command through a shell the way Nixpacks ones are. Without a shell, the &&-chained command was passed to Python as a literal string of extra arguments — and fetch_news.py doesn't read sys.argv, so it silently ignored them, ran completely normally, printed a clean “Saved N new items to DB” summary, and exited 0.
Nothing after it in the chain ever ran. No crash. No traceback. Every single cron execution's deploy log, for two straight days, showed fetch_news.py's output and then simply stopped.
The fix is one line — wrap the whole chain in sh -c "…", so the command Railway actually executes is a shell, which will correctly interpret && regardless of which builder ends up in charge. But chasing that bug down opened two more doors that had been sitting quietly closed the entire time.
archive_old_news.py was supposed to purge only disposable fetch-staging data. It was also running an explicit DELETE FROM news_summaries against anything whose underlying news_raw record had aged out — and even after removing that line, a second, sneakier mechanism was still live: news_summaries.id carried its own ON DELETE CASCADE foreign key back to news_raw, so purging old raw rows would have kept quietly cascading the deletion of finished, published summaries regardless. Fixing this properly meant a migration to drop the constraint outright, not just editing the script — a good reminder that “delete the line that does the bad thing” and “the bad thing can no longer happen” aren't always the same fix.
The Railway pipeline service's IDEOGRAM_RESOLUTION environment variable was set to RESOLUTION_1280_720 — a value that doesn't exist in Ideogram's actual resolution enum. Every single AI image generation call had been returning 400 Bad Request since the service was created, and the only thing anyone could see in the logs was the bare status line — the error handler logged str(e) and nothing else, never the response body, which is the only place Ideogram actually explained what was wrong. Fixed by validating the configured resolution against the real enum at import time (copied verbatim from Ideogram's own rejection message, since no documentation page enumerates it), falling back loudly to a known-good default instead of failing forever, and logging the full response body on every future HTTP error. As the founder put it reviewing the fix: “I have faced this resolution mismatch previously.” Some bugs come back for a second visit.
One more thing surfaced mid-session and is worth writing down once, here, so it doesn't have to be relearned: calling conn.set_session(readonly=True) against Neon's pooled connection string can leak a read-only session through pgbouncer into a completely unrelated live connection. It crashed one production cron run in the middle of this exact debugging session. The fix going forward is autocommit=True plus strict SELECT-only discipline for any ad-hoc production diagnostics — the tool used to investigate the outage briefly caused a smaller one of its own.
Reports described infinite scroll stalling at inconsistent, seemingly random card counts — nine one day, sixteen the next, fifty another — and appearing to “recover” only after leaving the Feed tab and coming back, which was actually a full component remount starting over from card zero, not a real resume.
Two separate mechanisms turned out to be involved. First: a failed /api/feed call during loadMore() had no retry at all — it logged the error and gave up, and the effect that's supposed to trigger the next page only re-fires when hasMore, index, or items.length change, none of which can change after a failed load, since the user can't advance past the last card that's already rendered. One transient network blip, and scrolling halted permanently with zero user-facing signal.
Second, and the one that actually explained the inconsistent numbers: the app has a mechanism for widening its lookback window when the current one runs dry, but that logic lived entirely inside loadMore() — which itself is gated on hasMore already being true. Whenever the very first fetch's 72-hour window happened to return fewer than a full page (verified live: 72 hours returned 15 items with has_more=false on one check; 144 hours returned a full 20 with has_more=true), that expansion logic never got the chance to run at all. The user was stuck at whatever count the first page happened to contain that particular day — which is precisely why the number was different every time.
Fixed with a retry (three attempts, 1.5-second backoff, resetting on any success) and by making the initial load report that it can still expand even when its first page came back short. A small unrelated safety net went in alongside it: a 600ms bounded watchdog on the single flag that gates all swipe and keyboard navigation, which had exactly one reset path and no recovery if that callback was ever skipped for any reason.
The language-selector foundation had shipped with three DE feeds and three FR feeds against English's fourteen. This round added seven German news sources and one lifestyle feed — Tagesschau, Süddeutsche Zeitung, FAZ, Deutschlandfunk, WDR, taz, n-tv, plus Kicker for Bundesliga coverage — and eight French sources and one lifestyle feed: RFI, Libération, Le Figaro, France Info, Ouest-France, BFMTV, 20 Minutes, Euronews FR, plus Le Monde Sport.
Every single URL was live-tested via feedparser immediately before merging, not assumed. Six candidates were tried and dropped for real, documented reasons rather than silently skipped: Handelsblatt returned a 301 with no working endpoint behind it; Die Welt, BR24, and Zeit Kultur all 404'd; Le Point and Les Echos were both bot-blocked behind Cloudflare. Two outlets — n-tv and Kicker — raised a benign CharacterEncodingOverride warning (declaring us-ascii while actually serving UTF-8); confirmed the text decoded cleanly with no data loss before leaving them in.
Admin visibility followed immediately after — exact per-language article counts added to the Pipeline tab, reusing the same query pattern the existing topic breakdown already used, so DE/FR volume could be checked directly instead of inferred from public API probes.
Then the harder, more honest question got asked directly: with all those new feeds live, why was Focus mode still returning zero DE and zero FR results, across a full thirty-day window? The answer, diagnosed live against real production data rather than assumed: it wasn't a scoring bug. Both languages already had working native-anchor scorers. It was source density. English's news list has three outlets built specifically around EU-institutional coverage — Politico, Euractiv, EUobserver. German and French each had exactly one. Everything else was a general national daily, and a general daily simply doesn't repeat “die Kommission” or “la Commission” densely enough in a short blurb to clear the same relevance bar three dedicated Brussels-press outlets do without trying.
No DE/FR equivalent to EN's three-deep institutional-press roster exists yet — this narrows the gap, it doesn't close it.PR #117
Two sources were added that live-tested as genuinely clearing the bar — Touteleurope.eu (0.67 on an EU-housing-policy piece, against 0.0 for most general-daily coverage of the same story) and de.euronews.com, closing a parity gap since French already had a Euronews feed. Two more candidates, Tagesspiegel and Der Standard, were tested live and deliberately not added — neither outperformed what was already there, and Der Standard is Austrian, which raised a scope question left open rather than quietly decided either way.
The fifth-swipe subscribe sheet's button rendered in flat browser-default grey — not disabled, just colorless, which reads as broken to anyone looking at it. The actual bug: two components referenced four color tokens — accent, accentSubtle, text, textMuted — that had simply never been defined anywhere in the theme file. Every reference silently resolved to undefined. A full-codebase scan confirmed these were the only undefined color references anywhere in the frontend. Four lines added to both palettes, and the button had a color again.
Byte's Take — the app's one real editorial voice on every card — had been structured as fact-then-consequence: sentence one restated the key fact, sentence two drew the EU implication. Read back-to-back with the summary directly above it, it landed as a compressed rewrite of the same article, not an opinion — nothing pulled a reader toward Byte's take over the plain headline. Restructured verdict-first: a blunt judgment or aphorism about what the story actually means, then the fact that grounds it. Genuinely serious stories — death, disaster, real harm — explicitly skip the verdict framing and lead with plain factual impact instead, so the new voice never reads as flippant on tragedy.
That voice had existed since the app's earliest sprints and was, structurally, invisible to search engines and uncopyable by an actual human reader. The entire app disabled text selection at the root level — every word on every screen. The one server-rendered story page, the only page built to be indexable at all, never even selected the byte_take column from the database in its own query; it was blank on that page regardless of what existed in the feed. There was no robots.txt anywhere on the site, and no structured data of any kind. And the bot that auto-posts new stories to Twitter had been linking through a dead query parameter since the second sprint of the app's life — not the one the homepage actually reads — meaning every automated tweet since had been silently sending readers to the generic homepage instead of the story it claimed to be sharing.
Fixed as a set: text selection scoped down from the whole app to just swipe-gesture chrome, so headlines, summaries, and the take itself are selectable again; the story page's query fixed to actually select byte_take and published; the take rendered as a real <blockquote cite> on that indexable page, not just a client-side card chip; NewsArticle JSON-LD added; a real robots.txt written, with explicit allow rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot; and the tweet bot repointed at the actual canonical story URL, with UTM parameters attached, matching a rule the project had already written down for itself and then not followed.
It started with one plain question: why does the live site sometimes show no stories at all — and if the pipeline just had a quiet day, why doesn't the UI simply fall back to whatever's already sitting in the database?
Reproduced live, the actual mechanism turned out to be worse than a quiet day. Focus mode, for a reader with German or French set as their language, returned exactly zero results at every single lookback window up to thirty days — not sparse, not slow, permanently empty, for as long as that reader stayed in Focus mode. The app does have a mechanism designed to widen its search window automatically when the current one comes up empty. It just wasn't wired to fire when the very first page returned nothing — and after unblocking that, a second, deeper issue surfaced: even once unblocked, the retry chain stalled after exactly one attempt, because hasMore going true-to-true and items.length staying at zero-to-zero don't register as a “change” to the effect that's supposed to notice and keep going. With genuinely zero items on screen, there's also nothing left for a user to scroll past that would otherwise nudge it forward. Fixed by adding a small counter outside React's own state entirely, so the retry chain can drive itself through every window step without waiting on a render that would never come — and by adding an honest empty state for the case where DE/FR really is exhausted at every window: an explanation, and a one-tap switch to Flow mode, instead of a dead end.
Verifying this properly meant standing up a small local mock backend, since the real one is too slow and too far away to iterate against directly. For most of an hour, the fix appeared to simply not work — every request kept coming back with a flat {"error":"offline"}, regardless of what the mock was actually configured to return. The culprit wasn't the fix. It was a service worker left registered from an earlier testing session on the same local port, silently intercepting every fetch call and answering on the mock's behalf before it ever got the chance. Even your own test harness can lie to you with a perfectly straight face.
Separately: bytenews.eu/sitemap.xml had been 404ing this entire time — the actual route only ever existed at /api/sitemap, and nothing rewrote the conventional path to reach it. Fixing the routing surfaced a second bug hiding directly underneath it: the route itself returned 200 with valid XML, but with zero story URLs in it, just the homepage. It had been asking the feed API for a hundred items at once; the feed API validates that same parameter at a hard ceiling of fifty and rejects anything higher outright. The request had been failing with a 422 on every single call since the SEO work shipped, silently, and the code's own fallback path — quietly designed for exactly this kind of failure — had been doing precisely what it was told: serving a homepage-only sitemap rather than a broken one. Google may never have discovered a single Byte story through its sitemap since that work went live. Fixed with two fifty-item calls instead of one hundred-item call that was never going to succeed.
Two more findings, both examples of code behaving exactly as written and still leaving a real gap. Perplexity discovery calls three separate search queries every pipeline run, each wrapped in its own try/except so one bad query can't take the other two down with it — sound, deliberate design. All three had been failing with 401 Unauthorized on every single run, and because each failure was caught, logged, and quietly skipped exactly as designed, the script never crashed, the healthcheck ping reported success every time, and nothing anywhere ever escalated “these are all failing” into something a human would actually see. Added an immediate alert specifically for the case where one hundred percent of queries fail in the same run — a signal distinct from “this one query had a bad day.”
And separately, a genuinely legitimate zero — one run saving no new articles, because a quiet news cycle is a completely normal thing to have — has no way of being distinguished automatically from three or more zero-save runs in a row, which is not normal and deserves a look. Added a streak check against the pipeline's own existing run log, no new table required, that alerts once the moment a streak is freshly crossed rather than paging anyone on every single run after the first.
The last finding of the day was the best one. A self-healing mechanism has existed since an early sprint: track consecutive failures per RSS feed, and once a feed crosses three in a row, mark it auto-disabled. Its own docstring says plainly that this “stops fetch_news.py from attempting the feed.” Nothing, anywhere in the codebase, ever actually read that flag before attempting a feed — only ever wrote it.
| Feed | Consecutive fails | Last success |
|---|---|---|
| brusselstimes.com | 678 | never |
| euractiv.com | 452 | 2026-06-26 |
| lefigaro.fr | 13 | never |
Three feeds had been retried, and failed, on every single two-hour cron cycle for months. One of them had failed six hundred and seventy-eight times in a row without succeeding even once. Wired the fix in — build the disabled-URL set once per run, filter it out before the fetch loop even starts — and then did the less obvious work of actually checking whether any of the three could be repaired rather than just muted.
Brussels Times turned out to already be a documented, deliberate tradeoff from months earlier — it needs a Cloudflare-bypass sidecar the Railway cron environment doesn't have, an accepted loss on record already. Le Figaro's self-healing tool exhaustively tried twenty-eight candidate URL patterns and found nothing valid. Euractiv looked genuinely fixable — a trailing-slash variant of the URL tested clean — until cross-checking eight straight weeks of the actual scheduled healing job's real run logs showed that exact same candidate had already been tried, and had already failed, from that same execution environment, every single week since the feed broke. A URL that works perfectly well from one network and fails consistently from another is not a URL bug. It's very likely an IP-range block on cloud hosting providers, which is a different, harder problem — and shipping a fix that only looked correct because it was tested from the wrong network would have been worse than leaving the honest note in the code that says so.
Six calendar days, not five continuous ones — there's a real two-day gap in the middle where nothing shipped, because reliability work happens when it happens, not on a sprint clock. That gap is itself part of the story: most of what's above isn't a single feature launch, it's the unglamorous, compounding work of a small product staying alive under one person's attention. Bugs that revealed other bugs. A review process that occasionally worked exactly the way it's supposed to. Decisions made once and written down, so nobody has to re-litigate them at 2am during the next incident.
The DE/FR gap is narrower now, not closed. The alerting is real now, but only once an email address actually gets set in Railway. Euractiv is still down, and honestly documented as such. None of that is a failure to report — it's what an accurate changelog looks like when the work is still going.