Kubian Kubian Status
Checking…

Kubian Platform Status

Live operational status for our Attestation, Device, Integrity, Internal and Web systems, monitored by Uptime Kuma. System components are checked continuously over the public transport path so these results reflect real-world availability of each system at the time you view this page.

0
Components Operational
0
Components Degraded
0
Components Monitored
5/5
Core Services
kubian.app · api.kubian.app Contacting status monitor…

Core Services

Database Operational
PostgreSQL available
Redis Cache Operational
Redis available
Device Ban Store Operational
device_bans reachable
Projects Store Operational
projects reachable
Attestation Log Store Operational
attestation_logs reachable

System Components

Your status feed is being initialized or is temporarily unreachable. Please try again shortly.
How this works: availability for each component is collected continuously by Uptime Kuma against the public endpoints of each system. A component is Operational when its latest check returned a healthy response, Down when the status monitor could not reach it, and Checking while a monitor is being measured for the first time. Region tags show where each system is served, and the wide bar beneath each component is a 24-hour timeline, one vertical bar per hour. Green means that hour was fully operational, orange partly degraded, and red means it was down. Bars that are grey (dim) are hours that have not occurred yet, and the bar on the far right is the component's current live status - it turns red the moment the monitor reports the component as down. Live status is stored to a local database so the timeline persists and survives restarts.

Incident History

Post-Mortem: PostgreSQL Advisory Lock Contention & Gevent Worker Stalls

Resolved
Incident Post-Mortem — Resolved

Executive Summary

During a routine server restart of the main backend process, an active transaction failed to release an exclusive PostgreSQL advisory lock. Subsequent incoming requests attempting to acquire the same lock became indefinitely blocked. Because synchronous C-level database socket calls bypassed gevent's event loop, the blocked queries froze the backend worker processes, preventing them from serving other traffic and eventually triggering Gunicorn worker timeouts (SIGKILL).

Root Cause Analysis

  • Unreleased Advisory Locks (pg_advisory_lock): When the main process restarted mid-execution, a worker process held an exclusive advisory lock acquired via pg_advisory_lock() inside shared.py (specifically within generate_android_unique_id()). Because the lock was not bound to a transaction boundary (pg_advisory_xact_lock) and lacked a guaranteed finally release block, the lock remained held in the database state by PID 1346355 / 1346499.
  • Blocking Socket I/O under gevent: By default, psycopg2 executes underlying C-level database calls synchronously. When subsequent greenlets attempted to acquire the same advisory lock, pg_advisory_lock() blocked indefinitely. Because the C socket calls were not yielding execution back to the gevent event loop, a single blocked database call effectively froze the entire worker process.
  • Cascading Worker Failures: As inbound traffic queued up, available connections in the pool were exhausted by blocked greenlets. Gunicorn workers ceased responding to health checks and were periodically killed via SIGKILL after hitting the 60-second worker timeout, causing widespread API timeouts and response latency across all integrated services.

Resolution & Fixes Applied

The incident was mitigated immediately by manually terminating the blocking database backends in PostgreSQL (SELECT pg_terminate_backend(...)). To permanently prevent recurrence, systemic updates were applied across the database driver configuration, connection pooling, and locking logic:

  • Non-Blocking Asynchronous I/O (psycogreen Integration): Added psycogreen.gevent.patch_psycopg() at application startup in backend.py. This patches psycopg2's C-level socket operations to cooperate with gevent's event loop, ensuring a waiting database query yields control so other greenlets can execute concurrently.
  • Transaction-Bound Advisory Locks with Guaranteed Cleanup: Refactored db_advisory_lock in shared.py to use pg_advisory_xact_lock, ensuring locks automatically release upon transaction commit or rollback. Explicit try...except handling was implemented around context exits to invoke conn.rollback() on failures.
  • Circuit Breakers (Statement & Lock Timeouts):
    • Implemented a 3s local lock timeout (SET LOCAL lock_timeout = '3s';) inside db_advisory_lock to abort lock acquisition attempts rather than hanging indefinitely.
    • Added a 5s statement timeout (SET statement_timeout = '5s';) across all connection checkouts in shared.py to act as an automated circuit breaker against rogue queries.
    • Applied user-level fallbacks (ALTER USER kubian_user SET statement_timeout = '5s'; and SET lock_timeout = '3s';) on the PostgreSQL instance.
  • Connection Pool Scaling: Maintained dedicated execution thread capacity using ThreadedConnectionPool(1, 55) per worker process alongside 16 dedicated DB executor threads to handle high-concurrency request bursts without pool starvation.

Downtime Fixed — Edge Traffic Service Reconfigured

Resolved
Our edge traffic service experienced connection timeouts and stalled requests due to upstream transport layer packet loss.

We resolved the service disruption by reconfiguring our Cloudflare edge tunnel transport protocol from UDP (QUIC) to TCP (HTTP/2) with active connection heartbeats.

This change bypasses upstream network-level UDP packet drops at our hosting provider's edge filtering tier. All systems integrated with Kubian are now fully operational, and we have implemented high-availability TCP configurations to prevent future transport stalls.

Unknown Downtime — Slow / Timing-Out Request Handling

Fixed
It appears servers are taking way longer to respond, or timing out before the request hits our edge traffic service.

We ran tests and debugging to determine the cause. All applications integrated with Kubian were affected while we reviewed whether this was a server-side issue or a hosting provider issue.

  • Status: Fixed
  • Analysis: Service requests appeared to never hit our services and were being dropped below our application level.
  • Next steps: We investigated further to review whether this was a Cloudflared Reverse Proxy Transport Tunnel issue or a hosting provider issue.
Availability is measured by Uptime Kuma. A component that the monitor can no longer reach is flagged as Down here, matching the transport-layer failure profile documented above.