Edge node latency in US-East
Created by Sarah Connor • 4 total messages
Sarah Connor
Today at 10:42 AM
Hi Team,
We're seeing significant latency spikes on our edge nodes specifically in the US-East region. The P99 is hitting over 1500ms for several customer clusters. Our internal health checks show the nodes are running but the handoff to the gateway seems delayed.
Attached are the current logs for the affected region.
We're seeing significant latency spikes on our edge nodes specifically in the US-East region. The P99 is hitting over 1500ms for several customer clusters. Our internal health checks show the nodes are running but the handoff to the gateway seems delayed.
Attached are the current logs for the affected region.
attachment
metrics_dump_us_east.json
stars
Today at 11:05 AM
Alex K. (Engineering)
Hello Sarah,
I've reviewed the logs you sent. It looks like the US-East-1 instance group was hit by a cold-start issue following the patch update last night. I've manually scaled up the warm pool to alleviate the P99 spikes.
Can you confirm if you see improvements on your end?
I've reviewed the logs you sent. It looks like the US-East-1 instance group was hit by a cold-start issue following the patch update last night. I've manually scaled up the warm pool to alleviate the P99 spikes.
Can you confirm if you see improvements on your end?
Sarah Connor
Today at 11:12 AM
The latency is dropping back to 200ms. Still slightly higher than our baseline of 80ms, but much better than before.