The new swapped in servers are performing well. We have re-enabled request+response logging for the platform. We will continue to monitor the situation - but performance is back up to par.
Posted Oct 09, 2020 - 09:37 EDT
Update
We have spun up a new load server environment with a brand new set of servers - and they seem to be working properly. We have swapped them into place.
Posted Oct 09, 2020 - 09:22 EDT
Update
We have temporarily disabled the request+response logging for the API servers to eek out more performance while we are unable to spin up new ones...
Posted Oct 09, 2020 - 08:46 EDT
Identified
It appears that and AWS issue is causing our load balancers to have difficulty talking to servers in certain zones. This may also be affecting the spinning of new relay and room servers (but doesn't seem to be yet?)
Posted Oct 09, 2020 - 08:25 EDT
This incident affected: API Platform (API Servers, Lobby Services, Hosting Services).