Summary: The Config Backend was down for around 1 hour (08:35 AM UTC to 09:35 AM UTC) earlier today June 10th, 2022. This was due to a mistake while initiating an infrastructure deploy that involved merging changes with main.
Impact: The Config Backend and Webapp were not reachable during this time. User Transformations - There was data loss if the connection included a user transformation. If the user transformation code was requested for the first time during this timeframe.
Client SDKs JavaScript - There would have been data loss for all the device mode destinations if the SDK was initialized during the downtime. i.e. If visitors first visited a page during this timeframe Mobile - There would have been data loss for all the device mode destinations. However, only for the first-time users of an app during the downtime.
Action Items: 1. Be more vigilant in checking our infrastructure deployment plan difference when reviewing and approving PRs. 2. We are escalating config-backend-error alert. 3. Developing more comprehensive run book for responding to an incident of this nature