[P1] KubeServiceDown - Backend NA - Experiencing Issues

Incident Report for RIA

Postmortem

Root Cause
One of the two nodes in the Production US cluster (aks-system-96920573-vmss000000) became unresponsive at approximately 04:52 UTC, most likely due to memory exhaustion. As a result, critical workloads, including ria-api and ria-services, were unable to continue running on the affected node. Although a second node remained available, it had already reached its maximum pod capacity and could not accommodate the impacted workloads. Service availability was therefore affected until the failed node was automatically rebooted by AKS and returned to a healthy state.

Resolution
AKS automatically rebooted the affected node, restoring cluster capacity and allowing the impacted workloads to recover. Service was fully restored at approximately 05:33 UTC once the node returned to a healthy state and application services resumed normal operation.

Posted Nov 09, 2026 - 15:13 JST

Resolved

The incident has been resolved. We apologize for any inconvenience this may have caused and thank you for your patience.
Posted Nov 09, 2026 - 15:13 JST

Investigating

Our automated monitoring has detected a major issue with the NA Core API Services. Our engineering team is currently investigating.
Posted Nov 09, 2026 - 14:29 JST
This incident affected: US Data Center (Core API Services).