Tech:Incidents/2018-02-22-Stunnel

This incident report is currently a draft After an SSL change made (addition of a new custom domain), stunnel seems to have crashed, leading all wikis to be down for about two hours.

Summary

 * What services were affected?
 * Cp* and due to backends being down, the whole site as a whole was down
 * How long was there a visible outage?
 * 2018-02-22 15:34 UTC until 17:16 UTC (1 hour 42 minutes)
 * What was/were the response times by each Operations member?
 * Reception123 responded at 15:37 on IRC, only due to the occurrence of 503 errors at random times, did not realize that it was a "permanent" error
 * Southparkfan responded at 17:12 on IRC, asking Reception123 to investigate backends, and respectively to restart stunnel
 * Was it caused by human error, supply/demand issues or something unknown currently?
 * Unclear.
 * Was the incident aggravated by human contact, users or investigating?
 * Does not seem to be aggravated in any way by human contact
 * How could response time by improved?
 * Response time could be improved if me (Reception123) would have realized that it was related to the configuration change, rather than dismissing it as the reoccurring 503 errors.

Timeline
All times are in UTC.
 * 15:34: The backends are all sick and all wikis go down with 503 Backend Fetch Failed error
 * 17:16: With the instructions of Southparkfan, Reception123 restarts the stunnel service, and all wikis go back up.

Quick facts

 * After a routine custom domain addition (SSL change) stunnel appears to have crashed.

Conclusions
To be added

Reporting

 * What services/sites were used to report the downtime?
 * None at the right time.
 * What other services/sites were available for reporting, but were not used?
 * N/A

Actionables
To be added
 * Permanent fix

Meta

 * Who responded to this incident?
 * Reception123, Southparkfan
 * What services were affected?
 * All wikis were down, cp*
 * Who, therefore, needs to review this report?
 * All Operations members
 * Timestamp: ...