Live operational status for our Attestation, Device, Integrity, Internal and Web systems, monitored by Uptime Kuma. System components are checked continuously over the public transport path so these results reflect real-world availability of each system at the time you view this page.
During a routine server restart of the main backend process, an active transaction failed to release an exclusive PostgreSQL advisory lock. Subsequent incoming requests attempting to acquire the same lock became indefinitely blocked. Because synchronous C-level database socket calls bypassed gevent's event loop, the blocked queries froze the backend worker processes, preventing them from serving other traffic and eventually triggering Gunicorn worker timeouts (SIGKILL).
pg_advisory_lock):
When the main process restarted mid-execution, a worker process held an exclusive
advisory lock acquired via pg_advisory_lock() inside
shared.py (specifically within
generate_android_unique_id()). Because the lock was not bound to a
transaction boundary (pg_advisory_xact_lock) and lacked a guaranteed
finally release block, the lock remained held in the database state by
PID 1346355 / 1346499.
pg_advisory_lock() blocked indefinitely. Because the C socket calls were
not yielding execution back to the gevent event loop, a single blocked database call
effectively froze the entire worker process.
The incident was mitigated immediately by manually terminating the blocking database
backends in PostgreSQL (SELECT pg_terminate_backend(...)). To permanently
prevent recurrence, systemic updates were applied across the database driver
configuration, connection pooling, and locking logic:
psycogreen.gevent.patch_psycopg() at application startup in
backend.py. This patches psycopg2's C-level socket operations to
cooperate with gevent's event loop, ensuring a waiting database query yields control
so other greenlets can execute concurrently.
db_advisory_lock in shared.py to use
pg_advisory_xact_lock, ensuring locks automatically release upon
transaction commit or rollback. Explicit try...except handling was
implemented around context exits to invoke conn.rollback() on
failures.
SET LOCAL lock_timeout = '3s';)
inside db_advisory_lock to abort lock acquisition attempts rather
than hanging indefinitely.SET statement_timeout = '5s';) across
all connection checkouts in shared.py to act as an automated circuit
breaker against rogue queries.ALTER USER kubian_user SET statement_timeout = '5s'; and
SET lock_timeout = '3s';) on the PostgreSQL instance.
ThreadedConnectionPool(1, 55) per worker process alongside 16 dedicated
DB executor threads to handle high-concurrency request bursts without pool
starvation.
We resolved the service disruption by reconfiguring our Cloudflare edge tunnel transport protocol from UDP (QUIC) to TCP (HTTP/2) with active connection heartbeats.
This change bypasses upstream network-level UDP packet drops at our hosting provider's edge filtering tier. All systems integrated with Kubian are now fully operational, and we have implemented high-availability TCP configurations to prevent future transport stalls.
We ran tests and debugging to determine the cause. All applications integrated with Kubian were affected while we reviewed whether this was a server-side issue or a hosting provider issue.