Another primary mongodb server has become unreachable...

Incident Report for brainCloud

Resolved

Amazon has identified an issue with EBS instances in one of the zones of us-east-1 where brainCloud is hosted. We have migrated servers out of the affected zone. It turns out that in 2 scenarios, we had multiple instances of the same shard in the same zone - which interfered with MongoDB's ability to fail over when the zone failed. We will put additional processes in place to avoid this configuration issue in the future. Apologies!
Posted Sep 27, 2021 - 00:47 EDT

Identified

It looks like this is an Amazon EBS issue - they are now showing a status that EBS connections in the North Virginia region (where we are) are impaired. This is what is impacting our database.
Posted Sep 26, 2021 - 23:19 EDT

Investigating

We believe there are networking issues happening within the cluster...
Posted Sep 26, 2021 - 23:06 EDT
This incident affected: Developer Services (Design Portal).
← Current Status Powered by Atlassian Statuspage