Tech:Organisation/mw-admins

This is a guide for all MediaWiki admins. MediaWiki admins have access to four Miraheze servers: mw1, mw2, mw3 and test1

In case of any doubts regarding a command or configuration change, mw-admins should ask either another mw-admin or Site Reliability Engineering before going through with it. In addition, mw-admins should test any new changes on test1 before adding them to production.

Rules

 * 1) Be respectful to other volunteers and users. You represent the Miraheze project.
 * 2) Don't suddenly change big parts of the MediaWiki infrastructure (e.g. way how things are done in the current style) without discussing it with the other sysadmins.
 * 3) Don't use the servers for non-Miraheze purposes.
 * 4) Don't put abnormally high load(s) on the server(s) if avoidable. (Grafana can be used for more details)
 * 5) Respect privacy. Don't publish access logs, IP addresses, content of private wikis, or other personally identifiable information. If in doubt, ask before publishing.
 * 6) Don't publish database passwords, private keys, etc as well.
 * 7) When in doubt ask questions first, act second

Violation of these rules can result into warnings or revocation of access.

Maintenance scripts

 * Always run them as www-data (prefix command with "sudo -u www-data")
 * Always !log maintenance script runs in #miraheze (unless you were running puppet or were using sql.php, and did not execute any queries that changed the database (e.g. SELECT/DESCRIBE queries)
 * If you need to run a script on all wikis, use the foreachwikiindblist wrapper: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/maintenance/yourscript.php --put-your-parameters --here
 * update.php is evil. If you need to run it while you're not doing a MediaWiki upgrade, then something is really wrong.
 * maintenance/update.php just automatically creates all required database tables on the wiki(s) it is run on. SQL files should be added to the CreateWiki config setting so that tables are automatically added to all new wikis. See Tech:Adding a new extension for enabling extensions or adding tables to existing wikis.

Installing and enabling new extensions
See Tech:Adding a new extension.

Pushing configuration changes
See for the rest.
 * Ensure the PHP syntax is correct, you don't want to deploy files with syntax errors
 * Although everyone makes mistakes sometimes, try to avoid them in the first place and be aware of how the changes you make can affect the entire site.
 * If a syntax error should occur, push a fixing-commit to the repo(s) and manually run puppet (sudo puppet agent -t) on any applicable servers.
 * When a configuration variable is changed, you should be completely aware what it does, and how it could impact a wiki security-wise.
 * There is more stuff in LocalSettings.php than just configuration variables, like this. You are allowed to edit such stuff, but if you are not familiar with its functionality, it's not recommended to touch it (or to merge pull requests that make changes to this area)

Deployment

 * When deploying a configuration change or extension, you are required to closely watch the change going live.
 * After commiting a change to the mediawiki or mw-config repo (and being sure it should work), run 'sudo puppet agent -t' on all MediaWiki servers. It can take a while before the change is actually deployed.
 * Watch the error logs:

Monitoring errors

 * We have a lot of log files in /var/log/mediawiki, but these are the most important ones:
 * /var/log/mediawiki/debuglogs/exception.log for all MWExceptions
 * /var/log/mediawiki/php-error.log for PHP errors
 * /var/log/mediawiki/database.log for database errors
 * Pro tip: try "tail -f /var/log/mediawiki/php-error.log", it will automatically output any changes to the specified log file, so you don't have to open the log file manually a few times. Useful during a deployment.

Debugging

 * Look at the error logs
 * Try to send the failing HTTP request with the header 'X-Miraheze-Debug: (mw[4567]|test1).miraheze.org' (yes, that is regex and you have to replace it with a valid server name), it could be an error that is cached in Varnish