Tech:Incidents/2016-01-25-DNS

A syntax error in our GDNSD configuration caused the wikis to go down for 7 minutes.

Timeline

 * 17:35 Southparkfan: commited 14afa4fd3bf1362e320e11c97ccce025a0f88982 "Add DNS for publictestwiki.com"
 * 17:40 Southparkfan: puppet runs automatically. At this point GDNSD crashed due to the syntax error.
 * 17:42 Southparkfan: Icinga alerts on IRC that DNS crashed
 * 17:47 Southparkfan: revert DNS config to good version
 * 17:47 Southparkfan: deployed change on DNS servers, site confirmed back online

Conclusion
Southparkfan commited a file to GDNSD that contained a syntax error, and he takes full responsibility for this outage. It has made clear that we should treat DNS even more carefully!

Actionables

 * #54 "GDNSD should not crash upon reload if config contains syntax errors" - in progress.

Meta

 * Online during downtime: NDKilla, Southparkfan
 * Affected services: DNS (full site)
 * Signature: Southparkfan (talk) 18:03, 25 January 2016 (UTC)