Tech:Incidents/2016-01-25-DNS

From Meta
Jump to navigation Jump to search

A syntax error in our GDNSD configuration caused the wikis to go down for 7 minutes.

Timeline[edit source]

  • 17:35 Southparkfan: commited 14afa4fd3bf1362e320e11c97ccce025a0f88982 "Add DNS for publictestwiki.com"
  • 17:40 Southparkfan: puppet runs automatically. At this point GDNSD crashed due to the syntax error.
  • 17:42 Southparkfan: Icinga alerts on IRC that DNS crashed
  • 17:47 Southparkfan: revert DNS config to good version
  • 17:47 Southparkfan: deployed change on DNS servers, site confirmed back online

Conclusion[edit source]

Southparkfan commited a file to GDNSD that contained a syntax error, and he takes full responsibility for this outage. It has made clear that we should treat DNS even more carefully!

Actionables[edit source]

  • #64 "GDNSD should not crash upon reload if config contains syntax errors" - Yes check.svg Done

Meta[edit source]

  • Online during downtime: NDKilla, Southparkfan
  • Affected services: DNS (full site)
  • Signature: Southparkfan (talk) 18:03, 25 January 2016 (UTC)