Doorkeeper was down for ten minutes on October 4th, 2019

Friday, October 04, 2019

On October 4th, 2019, Doorkeeper went down three times for a total of ten minutes. Doorkeeper was down at the following times:

  • 9:37 to 9:47 JST
  • 9:48 to 9:51 JST
  • 9:59 to 10:04 JST

Doorkeeper's team was immediately alerted about the issue, and began investigating it at once. By 10:08, we had identified the cause of the issue, and applied a fix.

We apologize for the disruption that this downtime caused, and we'll review our handling of this issue internally to better deal with future incidents.

Root cause

At 9:10, we changed the method our application used to redirect HTTP traffic to HTTPS. Although we thought that this method would be equivalent, it was not.

This resulted in a previously unredirected HTTP request to check that the Doorkeeper application was up to be redirected to HTTPS.

This caused the check to fail, resulting in the Doorkeeper application processes to be restarted.

The constant restarting of these processes resulted in Doorkeeper's downtime.

Once we identified this as the cause, we rolled back the change to our handling of HTTPS redirects, which resolved the issue.