Service fully restored

Incident Report — Activity & Audit History Unavailability

Status: Resolved.
Component affected: Activity & audit history (admin "Activities" / audit views) and the
BigQuery-backed analytics dashboards.
Severity: Major — feature unavailable; no data loss, no impact to payments/checkout.

Start	2026-06-21 22:44 UTC (23:44 WEST)
Service restored	2026-06-22 07:00 UTC (08:00 WEST — the daily limit reset)
Permanent fix deployed	2026-06-22 ~08:30 UTC
Duration (unavailability)	~8h 16m (overnight)
Data lost	None

---

Summary

On Saturday, 20 June we changed where our activity and audit history is stored, moving it to a
new data platform built for sustainable long-term retention. The change introduced an unexpected
behavior in how that data was queried: routine audit lookups began consuming far more of our
data-warehouse capacity than expected and exceeded our daily query limit.

Once that daily limit is reached, further queries are rejected until it resets — and the limit
resets on US Pacific time, which for us is only once a day, at 08:00 (WEST / UTC+1). As a result,
audit and activity views were unavailable until we reorganized the logical structure of the data
and the daily limit was restored.

The audit trail itself was never lost or corrupted — every record remained safely stored and new
activity continued to be recorded throughout. Because the daily query limit is shared across our
whole analytics warehouse, the impact reached beyond audit history to the analytics dashboards
that read from it (listed below) while the limit was exhausted. Core transactional functions —
payments, checkout, and subscription billing — were unaffected, as they do not depend on the
analytics warehouse.

Impact

• Activity / audit history (admin) — failed to load or returned errors. (Primary impact.)
• Analytics dashboards that query the shared warehouse were unavailable while the daily limit was exhausted:
  • Tracking / checkout analytics — click & conversion stats, lead funnels, campaign funnels, and the trackings dashboards.
  • Subscription analytics — LTV, churn, cash flow, renewals (including retries / reissues / payment attempts), trials, products, marketplace breakdowns, and decline-reason reports.
• Renewal charge-date optimization — the model that picks the best weekday to charge a renewal fell back to default scheduling during the window. No charges failed as a result; only the optimization was temporarily unavailable.
• No customer data was lost, exposed, or altered. New audit records continued to be written.

Root cause

The new storage location is organized well for storing the data cheaply, but — as initially
configured — not for the specific, narrow way our audit screens query it ("show the activity for
this one webhook / transaction / subscription / user"). Each of those lookups ended up reading far
more data than it needed, which rapidly exhausted our daily query limit.

Because that limit resets only once per day on US Pacific time (08:00 WEST), the service could
not fully recover on its own before that reset — which is what extended the impact window.

Resolution

We reorganized the logical structure of the information so each lookup reads only the small slice
it needs, instead of the whole dataset — reducing the data scanned per query by orders of magnitude —
and restored the daily limit. After deploying, we reconciled the reorganized data against the
complete source to guarantee every historical and in-flight record is present.

Once live, audit views load normally and the underlying query volume returned to a small fraction of
the previous baseline.

Preventive measures

• Per-query cost ceiling: queries on this data are now hard-capped — a single runaway query is stopped, not allowed to read unbounded data.
• Right-sized daily limit: the daily query budget is sized to real need, so a bad pattern is bounded instead of open-ended.
• Design-for-reads review: changes to how analytics data is stored now require an explicit review of the expected query/access pattern, not just storage cost, before rollout.

Timeline (UTC unless noted)

• Sat 20 Jun — Activity/audit history moved to the new storage platform.
• 21 Jun 22:44 UTC — Daily query limit exhausted; audit/activity views and analytics dashboards begin failing.
• overnight — The limit is per-day and resets only at 08:00 WEST, so the service could not recover on its own before then.
• 22 Jun 07:00 UTC (08:00 WEST) — Daily limit resets; service restored. Limit temporarily raised to prevent immediate re-exhaustion while the fix was deployed.
• 22 Jun ~07:45 UTC — Reorganized read structure (clustered copies) deployed; per-query cap applied.
• 22 Jun ~08:30 UTC — Two-pass reads deployed — each lookup now reads only the slice it needs (~30× less data); reconciliation completed; incident closed.