Tech:Compromised Handling

If you're at this page just to learn things, good. If you're at this page because something has been compromised - less good but let's make the matter go away quickly and securely.

What private data is stored that can be compromised?
We store a fair amount of private things, not so much user facing all the time but service-level things like passwords, certificate keys, actual servers and so on. Here is a list of things we store which if compromised, should be dealt with as a serious incident:
 * Servers
 * MariaDB database(s)
 * HHVM admin server password
 * mirahezebots IRC account password
 * root, wikiadmin, piwik, mediawiki MariaDB passwords
 * Redis password
 * MediaWiki private settings (see section about this in detail)
 * DKIM private key (mail)
 * Private SSH key for user git on misc1
 * All SSL private keys

If you've got something to add to the list above, please add it and then add a section detailing how to handle a compromise of the data. If the private git repo itself gets compromised, assume nothing is secure anymore and assess the damage and follow the necessary stages of all the above settings.

Who to notify if something is compromised?
In general, an email to operations@undefinedmiraheze.org should always be sent if any data (or suspicion of data) is compromised. Specific services will have specific additional people to contact or people who should be notified to assess the situation (the person who is most knowledgeable about the component - see list of System administrators). The community should be notified via a public method that a compromise occurred, but this should be left until an investigation has completed or someone from operations has signed off to say the situation is secure.

Servers
The servers should be the most secure parts of the Miraheze operation. Firewalls exist to keep port access limited, failed SSH attempts eventually result in port bans for specific IPs, secure ciphers are used and the root user can not be logged into. Security updates are also ran regularly - this prospect should be very unlikely but things happens.

Once someone is aware that is a server has been compromised, their ability to handle the situation will be limited by their access. Operations should be notified immediately. In extreme cases, if the user discovering the possible compromise has root access on the server but is not operations, they may be able to handle parts of the plan below, but if they're unsure, crowd control is the best method by means of either shutting the server down, disallowing port 22 or shutting down SSH - Operations have back doors in, others do not.


 * Ensure no one can gain access to the system after the issue has been discovered. Be it stopping the ssh server, shutting it down (or a reboot to close connections), blocking all ports etc. just ensure no one can get into the system.
 * Begin to get a handle of how access may have been gained - through software? an SSH account? Compromised SSH key? If it can be tied to a certain and definitive SSH account - remove said account from all servers and check the rest for safety.
 * Evaluate how the hole can be closed, if a key was compromised - ensure it is gone and force a regeneration, if a service/software allowed access - was it a security issue upstream? Were we running old software? Could it be how we're using it? Thoroughly investigate this question and follow solutions up.
 * Discuss findings with other operations members or send findings to operations@undefinedmiraheze.org to get more eyes on it.

The above is a basic plan to follow. Certain situations require more in-depth investigations, they can take minutes to hours to days to even weeks.

MariaDB Database(s)
MariaDB databases include wiki databases and other misc. services. If a database itself is compromised, there is not much collateral from it in the long run. The integrity of the database must be checked, investigate how access could have been gained and patch the vulnerability and monitor. If the database contained passwords, a precautionary reset of all passwords stored in the database should happen. A standard acknowledgement of the compromise should be noted and if possible, all people notified if any data of theirs could have been in the database.

HHVM Admin Server
Nothing is stored behind the HHVM admin server besides the ability to restart the server from the web. As there is a fall back to PHP-FPM, any compromise of the password is not major but not preferred. The password can be reset in the private git repo and an investigation into how the password was obtained should be launched.

IRC Account(s)
The IRC account is just a cloaked account for bots to share to avoid disclosing which server they're stored on. Nothing is stored on the account and it has no access to private channels or extended privileges. The password should be reset in private git and an investigation launched into how the password was obtained.

MariaDB passwords
The level of severity depends on which password is assumed to have been compromised. An evaluation of all databases that may have been compromised should be done after immediately resetting the password that was compromised and all below (if root). The process for databases above applies mostly as a compromised password compromises databases.

Redis
Redis does not store anything which is sensitive, therefore an integrity check should be carried out and the password should be reset with the standard investigation of how the password was obtained.

MediaWiki Private Settings
The MediaWiki private settings file contains:
 * MariaDB passwords for wikiadmin and mediawiki.
 * Redis password
 * SMTP password for noreply
 * Upgrade/Secret Key
 * ReCaptcha Keys

See the relevant sections for Redis and MariaDB passwords. The SMTP password should be reset by using passwd on misc1. No data can be compromised by this method. The Upgrade/Secret keys should be randomly regenerated and deployed. The ReCaptcha keys aren't important, they're just a confirmation stage for Google.

DKIM Private Key
This key should be immediately re-generated and re-deployed. A DKIM key allows someone to authenticate and sign emails from the domain miraheze.org, as such it has a serious consequence as social engineering attacks can be viewed as legitimate by mail servers around the world. The public key is listed in the DNS record for @ miraheze.org and this should be pulled immediately from deployment. An investigation into how the key was obtained should be launched.

Git Private Key
If this key is compromised, it should be reset and effectively all passwords and keys should be viewed as being compromised. If there is evidence the repo was cloned or the account was accessed, operations should be notified immediately and with urgency and an investigation into how the key was obtained should become a priority.

SSL Private Keys
All traffic should be viewed as compromised (as traffic may be decrypted though man-in-the-middle attacks and other methods) and the relevant domain(s) should be pulled if they have the certs regenerated immediately. An investigation should be launched into how the key was obtained and wiki owners/users should be notified.