Tech:Server admin log/2018

2018-07-01

 * 05:32 PuppyKun: closed ticket with bacula after acknowledging response and confirming i could ping and ssh the host
 * 05:03 PuppyKun: opened support ticket with backupsy (bacula1.mh.o host)
 * 00:01 paladox: ./upload.sh /srv/mediawiki/dblist/all.dblist on test1 (this will upload /mnt/mediawiki-static wiki containers to the container created.

2018-06-30

 * 19:35 MacFan4000: macfan@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki templatewiki /home/macfan/Wikipedia-20180630172333.xml
 * 13:25 Reception123: sudo -u www-data php /srv/mediawiki/w/maint*/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and and rebuild LC
 * 01:21 paladox: applied custom hack in /srv/mediawiki/config for swift file backend on test1
 * 01:15 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/setZoneAccess.php in a screen on test1

2018-06-29

 * 21:51 MacFan4000: paladox stopped icinga
 * 21:47 paladox: test
 * 19:06 paladox: systemctl enable ircrcbot on misc1 so that ircrcbot service starts on boot
 * 19:06 paladox: rebooted misc1

2018-06-28

 * 14:00 paladox: PURGE BINARY LOGS TO 'mysql-bin.000399'; PURGE BINARY LOGS TO 'mysql-bin.000400'; PURGE BINARY LOGS TO 'mysql-bin.000401'; PURGE BINARY LOGS TO 'mysql-bin.000402'; on db4
 * 13:56 paladox: PURGE BINARY LOGS BEFORE DATE(NOW - INTERVAL 1 DAY) + INTERVAL 0 SECOND; on db4
 * 01:28 paladox: signing swift1.miraheze.org puppet key on puppet1

2018-06-27

 * 14:55 paladox: rm -rf /srv/mariadb on bacula1
 * 14:55 paladox: removed mariadb from bacula1 as it kept crashing
 * 14:44 paladox: upgrading mariadb on bacula1
 * 14:40 paladox: upgrading phabricator
 * 14:39 paladox: upgrading grafana to 5.2
 * 05:36 Reception123: reception@mw2:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki jordepediawiki /home/reception/jorde_pages_full.xml

2018-06-26

 * 01:25 paladox: rebooting cp4 to clear ram
 * 00:55 paladox: disabled https://phabricator.miraheze.org/H34

2018-06-25

 * 20:25 MacFan4000: that is also in a screen
 * 20:25 MacFan4000: running jobs on all wikis
 * 20:00 paladox: apt-get --purge remove memcached on mw*
 * 19:58 paladox: apt-get --purge remove memcached on mw1
 * 19:20 paladox: repool mw1
 * 19:18 paladox: depool mw1
 * 19:18 paladox: reboot mw1
 * 18:48 paladox: restarting php7.2-fpm and nginx on mw1
 * 18:26 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw3
 * 18:25 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw2
 * 18:25 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw1

2018-06-24

 * 22:33 paladox: reboot cp4 due to OOM
 * 14:08 Reception123: manually run sudo -u www-data foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/runJobs.php on mw1 to alleviate queue
 * 14:07 Reception123: restarted mw2 jobrunner
 * 12:39 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 12:35 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw3

2018-06-22

 * 17:00 MacFan4000: restarted mw3 JobRunner and running jobs on all wikis
 * 15:54 MacFan4000: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw* and test1

2018-06-20

 * 18:08 paladox: installed openstack swift on nfs1 (will create a puppet class in the future to do this)

2018-06-19

 * 15:11 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/cleanupUsersWithNoId.php --prefix=m --assign on mw1
 * 15:09 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/cleanupUsersWithNoId.php --prefix=m on mw1
 * 15:05 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/cleanupUsersWithNoId.php --prefix=prefix on mw1
 * 12:41 MacFan4000: test
 * 12:41 paladox: reboot cp4
 * 01:12 John: archived H29, will not work as intended

2018-06-18

 * 22:50 John: dropped extloadwiki
 * 21:13 John: MariaDB [metawiki]> UPDATE cw_wikis SET wiki_category = "education" WHERE wiki_dbname LIKE "wright%";
 * 19:26 paladox: clearing ram on cp4 by rebooting
 * 16:20 Southparkfan: rebooting msic2
 * 15:55 Southparkfan: @misc2:/sbin# mv init init2 && ln -s /lib/systemd/system init && systemctl daemon-reload
 * 15:46 Southparkfan: removed redis-tools from misc2, reinstalled redis-server
 * 15:38 paladox: fixing up misc2 seems to be using init.d services instead of systemd

2018-06-17

 * 20:53 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/PageTriage/sql/PageTriageTagsPatch.sql on test1 preperation change for mw 1.31
 * 12:28 MacFan4000: macfan@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki project772wiki /home/macfan/Wikipedia-20180616192948.xml in a screen

2018-06-16

 * 23:06 paladox: reboot cp4 to try and regain some memory

2018-06-15

 * 20:46 MacFan4000: stopped and restarted in screen
 * 20:41 MacFan4000: test
 * 16:40 paladox: rebooting test1 to see if that will bring down cpu usage.
 * 16:17 paladox: upgrading redis on misc2 (package updates available)

2018-06-14

 * 20:54 paladox: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wkupwiki --report=1 dump.xml on mw1 T3235
 * 20:42 paladox: create paladox db (fake db to import and then export as a xml) on db4
 * 17:26 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/Math/db/patches/mathoid.add_png.mysql.sql on test1 (preperations for mw 1.31)

2018-06-12

 * 18:40 paladox: upgrading php7.2 on test1
 * 17:22 paladox: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=tapestrywiki --report=1 Wikipedia-20180612165309.xml on mw1

2018-06-11

 * 22:21 paladox: deleting uptimerobot account as moved to hund.io
 * 18:23 John: UPDATE cw_wikis SET wiki_inactive = 0 WHERE wiki_closed = 1;

2018-06-10

 * 23:48 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw3
 * 23:39 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw2
 * 23:27 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw1
 * 16:48 paladox: rebooting cp4 to free mem

2018-06-09

 * 23:23 paladox: upgrading phabricator
 * 00:09 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on test1

2018-06-08

 * 21:11 Southparkfan: disable mw1 puppet
 * 20:34 Southparkfan: disregard previous action
 * 20:33 Southparkfan: disable puppet on mw1
 * 18:45 Reception123: > UPDATE cw_wikis SET wiki_closed = '0' WHERE wiki_dbname = 'domusaureawiki'
 * 17:00 paladox: running sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=nonsensopediawiki on mw2
 * 01:14 paladox: restarting sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=nonsensopediawiki --report=1 nonsensopedia_pages_full.xml on mw2 in screen

2018-06-07

 * 23:12 paladox: restarting sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=nonsensopediawiki --report=1 nonsensopedia_pages_full.xml on mw2 in screen
 * 22:51 John: further purged from last 24 hours
 * 22:48 John: as an fyi ~40G gained back on db4
 * 22:48 John: removed binlogs before June 4th
 * 20:14 MacFan4000: macfan@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php rebuildrecentchanges.php --wiki wikinationswiki
 * 20:13 MacFan4000: macfan@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php initSiteStats.php --wiki wikinationswiki --update
 * 18:58 paladox: restarted varnish on cp4 to get puppet running
 * 18:17 MacFan4000: running manageInactiveWikis.php on test1
 * 17:55 MacFan4000: macfan@test1:/srv/mediawiki/w/maintenance$ sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki
 * 15:26 John: running manageInactiveWikis.php
 * 15:04 MacFan4000: stopped and restarted in a screen
 * 14:58 MacFan4000: macfan@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki wikinationswiki /home/macfan/carrington_pages_current.xml

2018-06-06

 * 22:52 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw3
 * 22:43 paladox: repool mw3
 * 22:29 paladox: depool mw3
 * 22:28 paladox: repool mw2
 * 22:28 paladox: depool mw2
 * 22:28 paladox: repool mw2
 * 22:27 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw2
 * 22:24 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw1
 * 22:13 paladox: depool mw2
 * 22:13 paladox: repool mw1
 * 21:52 paladox: depooling mw1
 * 21:47 paladox: re cloning mediawiki on mw* to fix a submodule
 * 20:48 paladox: stopping script as it needs to be run only once!
 * 20:21 paladox: running /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/CentralAuth/maintenance/resetGlobalUserTokens.php on mw1
 * 15:40 paladox: restarting sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=nonsensopediawiki --report=1 nonsensopedia_pages_full.xml on mw2 in screen

2018-06-05

 * 22:30 paladox: test
 * 20:56 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw[23]
 * 20:17 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw1
 * 18:46 paladox: fixing mw* starting with mw1 for T3204
 * 01:03 paladox: restarted sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=nonsensopediawiki --report=1 --file=/home/revi/nonsensopedia_pages_full.xml on mw2 (as mw2 was close to using all of it's ram)

2018-06-04

 * 23:15 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on test1
 * 21:46 MacFan4000: macfan@mw1:~$ sudo -u www-data cp -R /srv/mediawiki/w/extensions/SocialProfile/avatars /srv/mediawiki/w/extensions/SocialProfile/awards /mnt/mediawiki-static/popnmusicwiki
 * 21:13 MacFan4000: macfan@test1:/srv/mediawiki/w/maintenance$ sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki
 * 20:57 paladox: upgrading test1wiki to 1.31
 * 13:55 paladox: reboot cp2 to clean ram
 * 13:35 paladox: reboot cp4 to clean mem

2018-06-03

 * 23:10 paladox: install SecurityInfo on piwik
 * 19:59 paladox: restarting sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=nonsensopediawiki --report=1 nonsensopedia_pages_full.xml on mw2 in screen
 * 10:39 Reception123: reception@mw1:/srv/mediawiki/w/extensions/MirahezeMagic/maintenance$ sudo -u www-data php manageInactiveWikis.php --wiki metawiki --warn --close

2018-06-02

 * 22:40 MacFan4000: macfan@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php initSiteStats.php --wiki pirategalaxywiki --update
 * 22:02 MacFan4000: macfan@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki pirategalaxywiki /home/macfan/pirategalax_pages_full.xml

2018-06-01

 * 23:40 paladox: rm -rf amaninfowiki/thumb/6/6a/Camera.jpg on nfs1
 * 23:35 paladox: varnish> ban req.http.Host == static.miraheze.org && req.url ~ "^/amaninfowiki/6/6a/Camera.jpg" on cp*
 * 23:33 paladox: varnish> ban req.http.Host == aman.info.tm && req.url ~ "^/w/load.php" on cp*
 * 23:30 paladox: rm -rf /srv/mediawiki-static/amaninfowiki/6/6a/Camera.jpg on nfs1
 * 17:53 paladox: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=nonsensopediawiki --report=1 nonsensopedia_pages_full.xml on mw2 in screen
 * 11:50 paladox: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=nonsensopediawiki --report=1 nonsensopedia_pages_full.xml on mw2 in screen

2018-05-31

 * 23:49 MacFan4000: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw* and test1
 * 22:12 paladox: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=nonsensopediawiki --report=1 nonsensopedia_pages_full.xml on mw2 in a screen
 * 17:57 revi: restarted nonsensopedia import
 * 15:06 paladox: composer require geoip2/geoip2 on mw* and also install http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz in /srv/
 * 12:27 revi: revi@mw2:~$  again
 * 12:24 revi: did  on mw2
 * 12:15 revi:  on mw2 and mw3
 * 00:09 paladox: restart apache on puppet1
 * 00:09 paladox: restart puppetdb on puppet1

2018-05-30

 * 23:16 Southparkfan: installing mariadb-backup-10.2 on bacula1
 * 22:11 revi: revi@mw2:~$
 * 21:22 paladox: sudo -u www-data php deleteBatch.php --wiki pokemondustwiki -r "Per request via email" -i /home/macfan/delete.list.txt mw1
 * 21:09 paladox: composer install under OAuth ext on mw*
 * 18:30 paladox: sudo chown -R www-data:www-data /srv/mediawiki-static/pirategalaxywiki/avatars/
 * 18:29 paladox: mkdir /srv/mediawiki-static/pirategalaxywiki/avatars/
 * 18:29 paladox: sudo chown -R www-data:www-data /srv/mediawiki-static/pirategalaxywiki/
 * 18:29 paladox: mkdir /srv/mediawiki-static/pirategalaxywiki/
 * 15:35 Southparkfan: another attempt to import db4->bacula1 repl

2018-05-29

 * 23:22 paladox: upgrading git on all servers "salt -E ".*" cmd.run "apt-get -t stretch-backports install git -y"" per https://phabricator.miraheze.org/T3162
 * 21:53 Southparkfan: another attempt to import db4 on bacula1
 * 17:46 Southparkfan: deleted redis keys "*:translate-groups*" "*:gadgets-definition*" "*:flow*" " "*:LESS*" "*:template*" "*:file*"  "*:textextracts*" "*:revision*" "*:page*"
 * 17:35 Southparkfan: deleted redis keys matching patterns "*:resourceloader*", "*:MessageBlobStore*"
 * 17:21 Southparkfan: deleted redis keys matching patterns "*:diff:version:*", "*:filerepo:*"
 * 17:18 Southparkfan: restarted salt-minion on mw1
 * 14:32 paladox: disabling puppet on mw1 and misc1 to test some ssl changes (auto renewing ssl tests)
 * 11:02 paladox: restarted jobchron and jobrunner on mw 2 and mw3
 * 01:28 John: eventually moved wgFavicon to ManageWiki

2018-05-28

 * 23:11 John: cleared /backup/ on db4
 * 22:49 Southparkfan: installed package 'pv' on bacula1
 * 21:02 paladox: apt-get -t stretch-backports install redis-server on misc2
 * 20:31 paladox: stopping mysql on db4 pending db update
 * 20:12 paladox: install python3-mwclient on misc1
 * 17:53 John: restart jobchron and jobrunner on mw2 and mw3
 * 17:49 MacFan4000: running jobs manually on nltramswiki
 * 17:39 MacFan4000: macfan@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-data php runJobs.php --wiki test1wiki
 * 17:36 MacFan4000: restarted JobRunner on mw3
 * 17:21 Southparkfan: running mysqldump --skip-lock-tables --single-transaction --flush-logs --master-data=2 --all-databases | pigz | ssh -i /home/dbcopy/dbcopy/id_rsa dbcopy@bacula1.miraheze.org 'cat > /home/dbcopy/db4.sql.gz'
 * 16:51 Southparkfan: same for expire-logs-days
 * 16:50 Southparkfan: set db4's server_id dynamically (to avoid mysql restart)
 * 10:48 paladox: restarted jobrunner and jobcron on mw[23]
 * 02:00 paladox: running sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki test1
 * 02:00 paladox: running sudo -u www-data php /srv/mediawiki/w/maint*/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki test1
 * 01:55 MacFan4000: macfan@test1:/srv/mediawiki/w/maintenance$ sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki
 * 00:08 MacFan4000: paladox unlocked policy for some spammy phab tasks

2018-05-27

 * 21:22 revi: revi@mw3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildrecentchanges.php --wiki=anglishwiki
 * 21:21 revi: revi@mw3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=anglishwiki
 * 14:50 revi: revi@mw3:~$
 * 13:54 revi: halted previous operation
 * 12:33 revi: revi@mw3:~$

2018-05-26

 * 18:14 revi: revi@mw1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=metawiki
 * 18:13 revi: revi@mw1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildrecentchanges.php --wiki=metawiki
 * 18:07 revi:
 * 17:10 paladox: updating piwik to use geoip2 as geoip is now discontinued (https://matomo.org/faq/how-to/#faq_163)
 * 15:40 paladox: cleaning ram on misc3
 * 15:38 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki mw3
 * 15:38 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki mw3
 * 15:27 paladox: depool mw3
 * 15:25 paladox: repool mw2
 * 15:24 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki mw2
 * 15:24 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki mw2
 * 15:08 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw1
 * 15:08 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki on mw1
 * 15:04 paladox: depool mw2
 * 15:04 paladox: repool mw1
 * 14:36 paladox: depool mw1

2018-05-25

 * 20:49 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 17:55 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 17:55 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki on mw*
 * 17:54 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on test1
 * 17:54 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki on test1
 * 16:52 John: moved wiki.ciptimedia.org backups from nfs1->bacula1

2018-05-24

 * 22:08 paladox: updating matomo (piwik) to 3.5.0
 * 11:43 MacFan4000: sudo -u www-data cp -R /srv/mediawiki/w/extensions/SocialProfile/avatars /srv/mediawiki/w/extensions/SocialProfile/awards /mnt/mediawiki-static/credcowiki
 * 01:55 John: added MirahezeSSLBot as a collaborator to miraheze/ssl
 * 00:35 MacFan4000: CORRECTION: on puppet1
 * 00:32 MacFan4000: ... on ns1*
 * 00:31 paladox: clean up /var/lib/puppet/reports/ (was 28gb)

2018-05-23

 * 19:48 MacFan4000: macfan@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki lexwikiwiki /home/macfan/lexwiki_23-05-2018.xml

2018-05-22

 * 17:39 paladox: upgrade phabricator
 * 16:25 John: delete "app" repo (fork of all of Wikias code)
 * 16:13 paladox: manually updating prometheus-varnish-exporter on cp* following https://github.com/jonnenauha/prometheus_varnish_exporter/issues/29#issuecomment-391045216
 * 15:55 paladox: rm -rf /var/lib/ganglia*
 * 15:53 paladox: removing ganglia from misc2
 * 15:41 MacFan4000: paladox and I invited all other ops as admins
 * 15:41 paladox: <+MacFan4000>	!log created admin account for me using the account "admin"
 * 14:48 John: running estimates on bacula1 for all existing jobs to maximise disk usage as we currently backup more than we physically can
 * 14:48 John: DB3 pool on bacula1 as well
 * 14:46 John: deleting DB pool on bacula1
 * 14:31 revi: updated form 7 https://meta.miraheze.org/w/index.php?title=Tech:Phabricator/Form/Maniphest/7&diff=47923&oldid=47922
 * 11:35 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on test1
 * 02:21 paladox: running sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki and sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki on mw*
 * 02:17 John: running manageInactiveWikis.php

2018-05-21

 * 23:19 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki mw*
 * 23:19 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw1
 * 22:33 paladox: running sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on test1
 * 22:11 paladox: removing puppet from db2 following steps at https://meta.miraheze.org/wiki/Tech:Puppet#Removing_puppet_agent_.28server.29_on_the_Puppetmaster
 * 15:33 paladox: migrating cp5 from 20gb to 25gb
 * 01:12 Southparkfan: reversed previous ufw change
 * 00:53 Southparkfan: allow cp4.miraheze.org:9131 (ufw) from misc2
 * 00:28 Southparkfan: trying to reset grafana password

2018-05-20

 * 21:25 paladox: apt upgrade lilypond lilypond-data on mw*

2018-05-19

 * 21:09 paladox: upgrade phabricator

2018-05-17

 * 18:39 revi: Labster  in GitHub

2018-05-15

 * 21:43 John: made 1 security task (not security) public
 * 21:43 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on test1
 * 20:29 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 17:04 paladox: rebooted misc3 to relive memory presure
 * 15:05 paladox: restarted misc3 to clear the ram
 * 14:03 John: scratch that
 * 13:58 John: moving ciptimedia.org data to bacula1 from nfs1
 * 13:45 John: removed old dumps from nfs1
 * 12:32 John: includes reinstall to Deb9, no puppet association, will be a loan server
 * 12:31 John: claiming db3 as mine for testing pot. new service in a non prod environment
 * 11:08 paladox: rebooting cp2 (to see if that fixes "CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb")
 * 05:14 Reception123: sudo service icingabot restart
 * 00:16 paladox: systemctl mask mysql on db2

2018-05-14

 * 23:58 paladox: rebooted test1
 * 23:44 MacFan4000: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 23:36 John: looping INSERT INTO cw_comments ( cw_id, cw_comment, cw_comment_timestamp, cw_comment_user ) SELECT cw_id, cw_status_comment, cw_status_comment_timestamp, cw_status_comment_user FROM cw_requests WHERE cw_id = $id; between all missing IDs to find troublesome ID
 * 23:29 paladox: upgrading debian on db2 to latest jessie point release
 * 23:29 John: MariaDB [metawiki]> UPDATE cw_requests SET cw_status_comment = 'empty' WHERE cw_status_comment = '';
 * 23:28 paladox: stopped mysql on db2
 * 22:57 paladox: switching wikis from db2 to db4 https://git.io/vpQyD
 * 21:50 paladox: restoring db backups to db4 dbBackup.sh
 * 21:11 paladox: starting db2 -> db4 migration
 * 20:47 John: trained SA with my junk inbox
 * 19:18 paladox: delete db extloadwiki and loginwiki and testwiki on db2 as they were migrated to db4
 * 16:06 paladox: reboot misc3 to see if it will reclaim ram
 * 15:29 paladox: delete db airwiki aistudywiki aiswiki aklassewiki aktposwiki alacritysimwiki alaexploitdbwiki alberhillwiki albionwebwiki albustestwiki alewiki alfrescowiki algopediawiki alifloktawiki alittletroutywiki allsamemonsterswiki allthingswikiwiki almanachdebauswicwiki altenpflegewiki alternatehistorywiki alternatesporthistorywiki alterwikiwiki althistoriawiki altruniversewiki altversewiki alwikiwiki amaninfowiki ambirtestwiki amcdlwiki ameciclowiki amicitiawiki amicsdesmuseucawiki amicsdesmuseuwiki amirampwiki amntwiki amxwikiwiki analogypediawiki anchorhrgwiki andreasalexanderulrichwiki animangamespnpwiki from db2 (was migrated to db4)
 * 14:56 paladox: delete db afictionalworldonawiki, africavisionwiki, afrwiki, afterworldwiki, ag7732wiki, agathachristiewiki, agathinonwiki, ageloniawiki, ageofenlightenmentwiki, ageofwushuwiki, agriwikiwiki, agropediawiki, ahabariwiki, ahdwiki, ahmsaqibwiki, aibowiki, aidorupediawiki, aidyllicwiki, aimciawiki, airportsofwings900wiki from db2 (was migrated to db4)
 * 13:58 paladox: delete db activistresourceswiki, adadevelopersacademywiki, adamwiki, adantewiki, addicteddadswiki, adiaprojectwiki, adminbookclubwiki, adminbuswiki, adnovumwiki, advantagewiki, advisingwiki, aemanualwiki, aenasanwiki, aeromwiki, aerossprimewiki, aerowikiwiki, aesbasewiki, aescapeswiki, aetheriawiki from db2 (was migrated to db4)
 * 13:46 paladox: delete db adiapediawiki on db2 and db4 (was renamed to elerawiki) https://phabricator.miraheze.org/rMWCFe41b685975cabaed803b42452dee5cfc7150d020
 * 12:51 paladox: delete db 8stationwiki, 99pdwikiwiki, a360wiki, aapwiki, aaupaftlocal6075wiki, abainnovationwiki, abhirupghoshwiki, abitaregeawiki, absurdopediawiki, abundancewiki, abzewiki, acasrbijawiki, accademiadellebirrewiki, access7wiki, accorderiewiki, accountingwiki, acewikiwiki, achancetopursuewiki, actartletraswiki from db2 was migrated to db4 a few days ago and no issues reported
 * 12:41 paladox: delete db 690squadronwiki from db2 was migrated to db4 a few days ago and no issues reported
 * 12:41 paladox: delete db 4mindswiki from db2 was migrated to db4 a few days ago and no issues reported
 * 12:41 paladox: delete db 3dprinterscncwiki from db2 was migrated to db4 a few days ago and no issues reported
 * 12:40 paladox: delete db 2cvcupfrancewiki from db2 was migrated to db4 a few days ago and no issues reported
 * 12:40 paladox: delete db 1sttractionbrigadewiki from db2 was migrated to db4 a few days ago and no issues reported
 * 12:39 paladox: delete db 1cewiki from db2 was migrated to db4 a few days ago and no issues reported
 * 12:38 paladox: delete db 161647y2awiki from db2 was migrated to db4 a few days ago and no issues reported
 * 12:37 paladox: delete db 131parkhurstwiki from db2 was migrated to db4 a few days ago and no issues reported
 * 12:36 paladox: delete db 0x1winwiki from db2 was migrated to db4 a few days ago and no issues reported
 * 12:20 paladox: PURGE BINARY LOGS BEFORE '2018-05-14 13:20:26'; on db2
 * 00:08 MacFan4000: macfan@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php dumpBackup.php --wiki podpediawiki --full > /home/macfan/podpedia.xml

2018-05-13

 * 08:04 Reception123: sudo -u www-data php dumpBackup.php --current --wiki atlantisboardwiki > /home/reception/atlantisboardwiki13052018.xml on mw3

2018-05-12

 * 22:41 paladox: upgrade phabricator
 * 21:18 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw* and test1
 * 19:04 paladox: rebooted cp4 to try and gain more ram
 * 15:00 Reception123: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 12:44 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 12:40 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on test1
 * 11:24 John: MariaDB [metawiki]> UPDATE cw_wikis SET wiki_extensions=CONCAT(wiki_extensions,',zzzz');
 * 01:37 PuppyKun: force changed passwd for ndkilla@misc1
 * 00:49 paladox: puppet disabled on mw1 and mw2 by JohnLewis.

2018-05-11

 * 23:21 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on test1`
 * 22:56 John: fixed stylings of wiki_extension entries using MySQL replace
 * 22:29 John: populated wiki_extensions with transferred extensions data
 * 22:04 paladox: restarted parsoid after icinga message
 * 21:37 paladox: manually installed python-diamond and diamond debs on db2 and db3 as it's only in stretch dist. We are migrating to db4 which is stretch so this is not a problem.
 * 21:29 paladox: rolling python-diamond to all hosts using salt -E ".*" cmd.run "puppet agent -tv"

2018-05-10

 * 19:54 paladox: upgrading php on test1 to php7.2
 * 15:08 paladox: dbBackup.sh on db4 restoring backups
 * 15:07 paladox: dbMigration.sh on db2 (mgirating dbs to db4)
 * 13:45 paladox: rm -rf /srv/backup-root-mw1
 * 13:44 paladox: rm -rf /srv/backup-root-misc1
 * 13:43 paladox: rm -rf /srv/backup-home-mw1 (it was backed up to mw1)
 * 13:41 paladox: rm -rf /srv/backup-home-misc1
 * 13:40 paladox: deleted db backups on db4 in /home/dbcopy (have been working successfully for over a week)
 * 09:36 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw* and test1*
 * 01:13 paladox: composer install in /srv/mediawiki/w and /srv/mediawiki/w/extensions/Wikibase and /srv/mediawiki/w/extensions/Flow
 * 00:11 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on test1

2018-05-09

 * 23:51 MacFan4000: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on test1 earlier than when paladox did it
 * 23:48 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on test1
 * 23:38 MacFan4000: sudo -u www-data php update.php --wiki test1wiki --force on test1
 * 23:13 MacFan4000: php createLocalAccount.php --wiki testwiki --username NDKilla (going to assign rights on testwiki)
 * 22:03 MacFan4000: composer update on test1
 * 21:59 paladox: upgrading test1wiki to mediawiki 1.31 for testing
 * 21:46 MacFan4000: php createLocalAccount.php --wiki testwiki --username Example5
 * 21:46 MacFan4000: php createLocalAccount.php --wiki testwiki --username Example4
 * 21:41 MacFan4000: php createLocalAccount.php --wiki testwiki --username Example3
 * 18:14 paladox: upgraded phabricator

2018-05-08

 * 21:15 paladox: test
 * 18:26 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 13:57 John: converting all RequestWiki comments from old table to new table, will be a while!
 * 13:45 John: INSERT INTO cw_comments ( cw_id, cw_comment, cw_comment_timestamp, cw_comment_user ) SELECT cw_id, cw_status_comment, cw_status_comment_timestamp, cw_status_comment_user FROM cw_requests WHERE cw_id = 1; (request building from John here, logging because this is going to be ran a couple thousand times probably)
 * 13:42 John: created cw_comments on metawiki

2018-05-07

 * 21:48 paladox: delete test3wiki from db2 and also delete wiki following https://meta.miraheze.org/wiki/Tech:Delete_a_wiki
 * 21:46 paladox: deleted test2wiki from db2
 * 18:00 MacFan4000: adminbot was restarted on misc1
 * 01:48 PuppyKun: misc1: service icingabot restart

2018-05-05

 * 16:40 MacFan4000: sudo -u www-data cp -R /srv/mediawiki/w/extensions/SocialProfile/avatars /srv/mediawiki/w/extensions/SocialProfile/awards /mnt/mediawiki-static/test1wiki
 * 14:21 MacFan4000: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on test1
 * 12:51 MacFan4000: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*

2018-05-04

 * 22:36 paladox: upgraded phabricator
 * 00:39 MacFan4000: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 00:15 paladox: repool mw2
 * 00:03 paladox: repool mw1

2018-05-03

 * 23:52 paladox: depool mw1
 * 23:51 paladox: repool mw3
 * 23:36 paladox: depool mw3
 * 23:35 paladox: repool mw2
 * 23:27 paladox: depool mw2
 * 23:26 paladox: repool mw1
 * 23:18 paladox: depool mw1
 * 20:05 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 17:31 Reception123: dropped newmanwiki and argpwiki T2886 T2960
 * 17:28 Reception123: > DELETE FROM cw_wikis WHERE wiki_dbname = 'newmanwiki';
 * 17:28 Reception123: > DELETE FROM cw_wikis WHERE wiki_dbname = 'argpwiki';
 * 11:21 Reception123: sudo -u www-data php dumpBackup.php --current --wiki allthetropeswiki > /home/reception/allthetropeswiki03052018.xml on mw3
 * 07:26 Reception123: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 00:07 MacFan4000: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on test1

2018-05-02

 * 20:53 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 10:33 Reception123: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki
 * 07:34 Reception123: sudo -u www-data php refreshLinks.php --wiki allthetropeswiki on mw1

2018-05-01

 * 19:40 paladox: reinstalling misc3 (sudo is not working)
 * 14:55 paladox: clean logs on cp*

2018-04-30

 * 16:22 paladox: reboot db3
 * 16:19 paladox: upgraded db3 to latest jessie point release (made sure it had no stretch dist)
 * 16:18 paladox: removed mysql db3 service from icinga manually
 * 15:49 paladox: sudo service mysql stop on db3
 * 14:30 paladox: delete db backups on db4 /home/dbcopy (no reports of wikis braking)
 * 12:09 paladox: apt-get install python-pip on misc3 (i will puppitise this later)

2018-04-29

 * 23:25 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 22:41 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on test1wiki

2018-04-28

 * 23:31 paladox: restarted puppetdb on puppet1 for config change

2018-04-27

 * 19:50 MacFan4000: macfan@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki podpediawiki /home/macfan/Wikipedia-20180202192828.xml
 * 19:44 MacFan4000: macfan@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki podpediawiki /home/macfan/roosterteeth_pages_current.xml
 * 18:07 paladox: clean up logs in /var/log/** on cp5
 * 16:14 paladox: sudo chown -R www-data:www-data ../repos on misc1 (/srv/phab/repos)
 * 16:12 paladox: upgrade phabricator
 * 13:38 Reception123: reception@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki testwiki /home/reception/backups4/home/reception/wikibackups/testwiki.xml
 * 12:33 Reception123: DROP DATABASE testwiki; (re-importing)
 * 12:33 Reception123: DELETE FROM cw_wikis WHERE wiki_dbname = 'testwiki'; (re-importing)
 * 12:31 Reception123: sudo -u www-data php dumpBackup.php --wiki testwiki --full --uploads > /home/reception/testwiki_feb.xml on mw3
 * 11:53 Southparkfan: chmod 777 /tmp on bacula1
 * 10:30 Southparkfan: installed percona-xtrabackup-24_2.4.11-1.stretch_amd64.deb on db4
 * 00:14 PuppyKun: freed up ~5.6 GB on mw1

2018-04-26

 * 23:57 PuppyKun: wget february testwiki xml dumps to mw1 / import
 * 22:29 paladox: delete db testwiki from db4
 * 22:29 paladox: /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/removeDeletedWikis.php --wiki testwiki on mw1
 * 22:28 paladox: DELETE FROM cw_wikis WHERE wiki_dbname = "testwiki"; on db2
 * 22:20 paladox: delete networksdatabasewiki nextlevelwikiwiki nonbinarywiki nvcwiki omnipediawiki ordiswiki ourmxfestwiki plasticssongcontestwiki platprojectwiki pluspiwiki pocketmonsterswiki podpediawiki r2wiki recentiawiki revitwiki scruffy2wiki sdiywiki sentrieswiki sidemwiki sirikotwiki songcontestswiki speleowiki spiralwiki spritopiawiki summerolympicswiki sumroletaericwiki templatewik teriawiki from db3
 * 22:18 paladox: delete omnipediawiki from db3 and db4
 * 22:18 paladox: delete saganbackup from db3 and db4
 * 21:37 paladox: drop db testdeletewiki from db2 (is on db4 now)
 * 20:24 paladox: rm *.sql in /home/dbcopy on db4. Everything is working fine!
 * 20:12 paladox: delete db centralauth and loginwiki from db3 (it's now on db4)
 * 17:40 Reception123: re-created reception user on icinga (again)
 * 17:31 paladox: delete testdeletewiki and thegreatwarwiki and thelonsdalebattalionwiki and tiandiwiki and truecapitalistwiki and ubrwikiwiki and uncyclopediawiki and unionwiki and ussewiki and utamacrosswiki and valentinaprojectwiki and viagroupiawiki and whentheycrywiki and wiki1776wiki and wikiversitywiki and wikivywikiand worldpediawiki and ylscwiki and yoavfreundwiki and yourosongcontestwiki and zendariwiki and zhdelwiki from db3 (they are now on db4)
 * 17:16 Reception123: removed labster and imbophil from Phabricator Security group
 * 17:12 Reception123: removed ImBoPhil from @miraheze org
 * 16:23 paladox: deleting testdelete1wiki from db3 (wiki does not exist anymore)
 * 16:17 paladox: drop puppet from db3 (not needed anymore since we went with puppetdb)
 * 16:11 paladox: delete ciudadciclistawiki and constwiki and cpiwiki from db3 (it's on db4 now)
 * 16:09 paladox: delete christianmusictempwiki from db3 (it's on db4 now)
 * 16:09 paladox: delete christianmusicwiki from db3 (it's on db4 now)
 * 16:08 paladox: delete calexitwiki from db3 (it's on db4 now)
 * 16:08 paladox: delete bqwiki from db3 (it's on db4 now)
 * 16:07 paladox: delete bgowiki from db3 (it's on db4 now)
 * 16:07 paladox: delete beatstationwiki from db3 (it's on db4 now)
 * 16:07 paladox: delete ballaratpubswiki from db3 (it's on db4 now)
 * 16:06 paladox: delete ayrshirewiki from db3 (it's on db4 now)
 * 16:06 paladox: delete arefepvawiki from db3 (it's on db4 now)
 * 16:05 paladox: delete altaussieruleswiki from db3 (it's on db4 now)
 * 16:05 paladox: delete aleenghawiki from db3 (it's on db4 now)
 * 11:04 Reception123: moved cargodb from db3 to db4 and dropped on db3
 * 10:17 John: restarted phd

2018-04-25

 * 22:56 paladox: reboot misc1 to free up memory
 * 22:33 paladox: added some manual patches to /usr/share/php/Icinga/** for php7.2 (https://github.com/Icinga/icingaweb2/commit/5d100779ce8557095e0cfe1768780086b5865adf and https://github.com/Icinga/icingaweb2/commit/72ec132f25c868d9510e6d36a2d5c92fc8dd59d1)
 * 20:24 paladox: re enabling puppet on misc1 to upgrade to icinga2
 * 20:19 paladox: disabling puppet on misc1 in preperation for icinga1 -> icinga2 migration
 * 16:31 paladox: upgraded phabricator
 * 15:45 Reception123: re-create my mail account as reception123 (deleted with the reinstall)
 * 11:20 Reception123: sudo -u www-data php importDump.php --wiki conworldwiki /home/reception/conworld_pages_current.xml on mw3
 * 00:06 paladox: composer install on mw*

2018-04-24

 * 21:42 John: ensured all rDNS is correct for servers
 * 14:17 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw* and test1
 * 08:51 Reception123: UPDATE cw_wikis SET wiki_language = 'no' WHERE wiki_dbname = 'ecypewiki'; (issue with ManageWiki_)
 * 05:37 Reception123: sudo -u www-data php dumpBackup.php --wiki safiriawiki --full --uploads > /home/reception/safiriawiki240402018.xml on mw2

2018-04-23

 * 22:17 paladox: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw* and test1
 * 07:49 Reception123: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*

2018-04-21

 * 19:03 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 17:22 paladox: drop phabricator db from db3
 * 17:01 paladox: delete database conworldwiki on db3, it's on db4

2018-04-20

 * 22:02 paladox: ran drop database poserdazfreebieswiki; on db3
 * 21:36 paladox: rm piwik.sql on db4
 * 21:34 paladox: drop database testwiki on db3, it's on db4
 * 21:31 paladox: drop database piwik from db3, it's now on db4.
 * 21:03 paladox: deleted conworldwiki.sql backup from db4
 * 21:01 paladox: sudo service apache2 stop on misc2 for migration of piwik to db4
 * 19:44 paladox: rm -rf /srv/mediawiki-static/unionnorteamericanawiki on nfs1
 * 19:43 paladox: ran DROP DATABASE unionnorteamericanawiki; on db3
 * 19:42 paladox: ran /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/removeDeletedWikis.php --wiki unionnorteamericanawiki on mw1
 * 19:41 paladox: ran DELETE FROM cw_wikis WHERE wiki_dbname = "unionnorteamericanawiki"; on db2
 * 19:12 paladox: drop database conworldwiki from db2 after being migrated to db4
 * 19:01 paladox: removed backup backup.sql.gz from db4
 * 19:00 paladox: remove backups ciptamediawiki.sql and buswiki.sql and bpwiki.sql and jawp2chwiki.sql from db4.
 * 19:00 Reception123: upgraded phabricator

2018-04-19

 * 18:13 paladox: delete db bpwiki on db3 after migration to db4
 * 18:13 paladox: delete db buswiki on db3 after migration to db4.
 * 18:12 paladox: delete db ciptamediawiki on db3 after migration to db4
 * 18:12 paladox: delete db jawp2chwiki on db3 after migration to db4
 * 17:38 paladox: backing up bpwiki buswiki ciptamediawiki jawp2chwiki from db3 to db4 then will migrate to db4.

2018-04-18

 * 23:28 paladox: deleting backup-srv-nfs1 on db4 to clear space up
 * 22:38 paladox: migrating test1wiki to db4
 * 19:19 paladox: reboot cp2
 * 18:48 paladox: upgrading php on test1 to php7.2
 * 17:49 paladox: sudo chown -R www-data:www-data * in /srv/mediawiki-static on nfs1
 * 16:51 paladox: re mount /mnt/mediawiki-static as nfs1 has come back up.
 * 16:50 paladox: unmount /mnt/mediawiki-static on mw* for nfs1 reboot
 * 16:44 paladox: reboot nfs1 to see if that will fix custom images
 * 16:32 Reception123: running dumpBackup.php for all wikis (in batches) on test1wiki
 * 15:40 Southparkfan: disabled puppet on cp4
 * 15:28 paladox: delete backup-root-nfs1
 * 15:28 paladox: delete backup-home-nfs1 on db4
 * 14:40 paladox: upgrade nginx to 1.14 on cp4
 * 14:39 paladox: upgrade nginx to 1.14 on cp2
 * 14:39 paladox: upgrade nginx to 1.14 on cp5
 * 14:37 paladox: upgraded nginx to 1.14 on misc3
 * 14:36 paladox: upgraded nginx to 1.14 on test1
 * 14:27 paladox: repool mw3
 * 14:27 paladox: upgrade to nginx to 1.14 on mw3
 * 14:26 paladox: depool mw3
 * 14:26 paladox: repool mw2
 * 14:23 paladox: upgrade to nginx to 1.14 on mw2
 * 14:23 paladox: depool mw2
 * 14:23 paladox: repool mw1
 * 14:20 paladox: upgrade nginx on mw1 to 1.14
 * 14:20 paladox: depool mw1
 * 13:47 paladox: re enable puppet on mw* but keeping file uploads disabled until backup restored complete
 * 13:30 paladox: syncing nfs1 backup on db4 back to nfs1.
 * 12:56 paladox: reinstall nfs1 as stretch
 * 12:52 paladox: salt -E "mw.*" cmd.run "umount /mnt/mediawiki-static"
 * 12:52 paladox: downtiming nfs1 in icinga
 * 12:49 paladox: disabled puppet on mw*
 * 12:32 paladox: update piwik plugin LoginLdap to 4.0.4
 * 12:16 paladox: taking one last backup of /srv on nfs1
 * 00:50 paladox: rsync -avz --delete -e "ssh -i /home/paladox/id_rsa" /srv/ dbcopy@db4.miraheze.org:/srv/backup-srv-nfs1/ on nfs1 in a screen

2018-04-17

 * 23:54 paladox: restoring all backups onto nfs1, will use rsync
 * 21:54 paladox: deleting allthetropeswiki from nfs1 as it's backed up on db4
 * 20:58 Southparkfan: installed php-memcached on mw1
 * 20:35 paladox: rm -rf aktposwiki on nfs1 backup on db4
 * 17:26 paladox: rebooted nfs1
 * 16:57 paladox: apt-get upgrade on nfs1
 * 16:44 paladox: rm -rf * in /var/lib/puppet/clientbucket on db3 to clean some space
 * 16:43 paladox: rm -rf * in /var/lib/puppet/clientbucket on db2 to clean some space
 * 16:41 paladox: rm -rf * in /var/lib/puppet/clientbucket on nfs1 to clean some space
 * 16:39 paladox: removed apt public key EF0F382A1A7B6500 from nfs1, was warnning with "There is no public key available for the following key IDs:"
 * 16:36 paladox: installing puppet4 on nfs1
 * 16:30 paladox: upgrading puppet to puppet 4 on db3 and db2
 * 15:40 paladox: created database icingaweb2 on db4 for icingaweb2
 * 15:37 paladox: created database icinga on db4 for icinga2

2018-04-16

 * 22:23 paladox: updating acme-tiny in /root on mw1
 * 20:33 paladox: run sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki for mw* on misc3
 * 19:34 paladox: deleted backup backup-repos-misc1 on db4, phabricator may have recreated this as it was working in the ui and there is repos listed.
 * 19:33 paladox: deleted backup "backup-mail-misc1" on db4, was synced when we upgraded misc1 last week.
 * 19:33 paladox: deleted backup "backup-misc2-home" on db4, was synced just now to misc2
 * 19:23 paladox: deleted backup "backup-images-misc1" on db4, was synced to misc1 last week when we upgraded misc1
 * 19:22 paladox: deleted backup "backup-misc2-ganglia" was synced to misc2 on sunday
 * 19:06 paladox: deleted backup backup-misc2-root from db4, i've just synced it to misc2
 * 15:55 paladox: upgraded piwiki plugin "VisitorGenerator" to 3.1.0
 * 15:55 paladox: upgraded piwik plugin "QueuedTracking" to 3.2.0
 * 15:54 paladox: upgraded piwik to 3.4.0 (no db updates so safe to update)

2018-04-15

 * 21:44 paladox: clean up puppet cert for bacula1 pending reinstall
 * 21:42 John: installed bacula1 again with more than 20G allocated to / and about 500GB less to /home
 * 20:22 paladox: rebooted bacula1
 * 16:00 paladox: salt -E 'mw.*' cmd.run 'puppet agent -tv' on misc3
 * 16:00 paladox: salt -E 'mw.*' cmd.run 'puppet agent -tv' on mw*
 * 15:50 paladox: downtimed misc2 in icinga for maint at 4pm utc
 * 15:05 paladox: rebooting puppet1
 * 15:04 paladox: upgrading puppet1 to stretch 9.4
 * 14:57 paladox: rebooting db4 for kernel upgrade
 * 14:52 paladox: reboot db4
 * 14:44 paladox: reboot db4 to test mysql starts up
 * 14:42 paladox: reboot db4 to see if hostname changes correctly
 * 14:30 paladox: rebooting db4 to see if mysql comes back up
 * 13:14 paladox: rebooting db4
 * 12:43 Reception123: deleted temporary user from bacula1
 * 12:24 Reception123: signed bacula1.miraheze.org cert on puppet1

2018-04-14

 * 23:53 paladox: downtimed bacula1 until 5:52am
 * 19:15 Reception123: re-imaging bacula1 with Debian 9
 * 19:14 paladox: Reception123 is reinstalling bacula1 as stretch
 * 19:12 paladox: removed bacula1.miraheze.org puppet cert pending reinstall to stretch
 * 18:12 paladox: installing salt-master on misc3 to see if i can get a pub key for puppet
 * 14:40 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki mw*
 * 14:40 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw3
 * 13:43 paladox: reboot puppet1 to try and clear ram
 * 12:30 paladox: created user guest in icinga
 * 12:29 Reception123: restarted icinga
 * 12:26 Reception123: recreated user 'reception' for icinga
 * 00:48 paladox: upgraded test1 to stretch 9.4
 * 00:48 paladox: rebooting test1

2018-04-13

 * 23:52 Southparkfan: test
 * 23:52 paladox: <+JohnLewis>    !log changed mailname for misc1 to miraheze.org
 * 19:49 paladox: reimaging misc1 to stretch
 * 14:26 paladox: repool mw2
 * 14:19 paladox: rebooting mw2
 * 14:14 paladox: upgrading mw2 to stretch 9.4
 * 14:14 paladox: depooling mw2
 * 14:13 paladox: repooling mw3
 * 14:09 paladox: rebooting mw3
 * 14:08 paladox: upgrade mw3 to stretch 9.4
 * 14:07 paladox: depool mw3
 * 01:54 paladox: rebooting misc3
 * 01:50 paladox: upgrading misc3 to 9.4 (apt-get upgrade) also upgrade to nginx

2018-04-12

 * 23:31 paladox: reimaging cp5 to stretch
 * 22:56 paladox: reimaging cp2 as stretch
 * 22:36 paladox: reinstalling cp4 as stretch
 * 21:59 paladox: reimaging ns1 to stretch
 * 21:52 paladox: repooling mw1
 * 21:44 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw1
 * 20:52 paladox: reimaging mw1 as stretch
 * 20:51 paladox: depooling mw1
 * 20:42 MacFan4000: paladox ran puppet on mw* (logging failed the first time)
 * 20:11 Southparkfan: created backup of /root and /home on mw1, stored on db4 in /srv/backup-{home,root}-mw1
 * 19:52 paladox: rebooting misc1
 * 19:31 paladox: restarting puppetdb on puppet1
 * 19:28 paladox: restarting apache on misc1
 * 19:09 paladox: disabled visualeditor on metawiki due to when using it on the main page it causes an outage
 * 19:00 paladox: downgrading parsoid to 0.8 on misc3
 * 17:24 paladox: repooling mw2
 * 17:21 paladox: depooling mw2
 * 17:19 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw2
 * 17:13 paladox: repooling mw2
 * 17:11 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw3
 * 17:03 Southparkfan: depool mw1 [emergency]
 * 16:41 Reception123: signed puppet cert for mw2
 * 16:32 Reception123: reinstall mw2 on deb 9
 * 16:30 paladox: depooling mw2 from cp[245]
 * 16:26 Reception123: repool mw3
 * 15:41 Reception123: reinstall mw3
 * 15:39 Reception123: depool mw3
 * 04:48 Reception123: sudo -u www-data cp -R /srv/mediawiki/w/extensions/SocialProfile/avatars /srv/mediawiki/w/extensions/SocialProfile/awards /mnt/mediawiki-static/avalicearchiveswiki/

2018-04-11

 * 20:42 paladox: sudo service icingabot restart
 * 19:48 PuppyKun: made paladox a github org owner
 * 19:19 paladox: restart icinga
 * 19:19 paladox: removing parsoid1 from icinga on misc1
 * 19:04 paladox: sudo chown -R www-data:www-data /srv/mediawiki/w/cache/l10n/ on mw3
 * 17:53 paladox: running sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 17:51 paladox: running sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw1
 * 16:41 paladox: running puppet on mw*
 * 16:03 Reception123: upgraded phabricator
 * 15:45 paladox: running sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki
 * 15:45 paladox: running "sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki"
 * 14:58 paladox: redoing sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki
 * 14:58 paladox: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki
 * 14:56 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki
 * 14:18 Reception123: made paladox a phabricator administrator
 * 14:17 Reception123: added Paladox to @miraheze
 * 00:41 PuppyKun: added john / John Lewis / john@undefinedmiraheze.org as a Matamo superuser.

2018-04-10

 * 23:56 John: parsoid1->misc3, misc3->stretch, parsoid->0.9.0, revert-count->big, error-count->less-than-earlier, success?->questionable-but-it-works

2018-04-09

 * 22:17 John: made H29 only act once
 * 22:13 John: rebuilding LC on mw3
 * 18:45 Reception123: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki
 * 16:33 John: restarted jobrunner on mw2,3
 * 16:31 John: running jobs

2018-04-08

 * 20:28 John: deleted Mkraheze/Math

2018-04-06

 * 18:10 Reception123: repool mw1
 * 18:09 Reception123: clear out /srv/mediawiki/config on mw1 for reclone
 * 18:09 Reception123: depool mw1
 * 18:09 Reception123: repool mw3
 * 18:08 Reception123: clear out /srv/mediawiki/config for reclone
 * 18:08 Reception123: depool mw3
 * 17:49 Reception123: ran check_procs for php7.0-fpm
 * 14:43 Reception123: reboot test1
 * 13:52 Reception123: reinstall test1 on Debian 9
 * 13:46 Reception|away: reception@mw2:/srv/mediawiki/w/extensions/Flow/maintenance$ sudo -u www-data php convertNamespaceFromWikitext.php --wiki isvwiki 
 * 13:05 Reception|away: reception@mw2:/srv/mediawiki/w/maintenance$ sudo -u www-data php populateContentModel.php --wiki isvwiki --ns all --table revision

2018-04-05

 * 15:31 Reception|away: sudo service icingabot restart
 * 15:28 Reception|away: restarted puppetdb on puppet1
 * 13:37 John: db4 has SSL capabilities now

2018-04-04

 * 23:21 John: installed db4 as stretch
 * 22:47 John: reinstall db4 as stretch
 * 22:43 John: unsigned db4 from puppet1
 * 17:19 Reception123: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*

2018-04-03

 * 19:35 Southparkfan: restart ganglia-monitor on puppet1
 * 19:23 John: restart ganglia-monitor on mw1
 * 14:02 John: add cw_custom to cw_requests on metawiki
 * 13:20 John: rebooted db2, excessive server load and CPU usage
 * 09:07 revi: add @miraheze/i18n group to WikiDiscover repo

2018-04-02

 * 19:25 revi: changed phabricator global timezone settings to UTC instead of Europe/London
 * 18:44 Reception123: added mw-admins team to @miraheze/WikiDiscover
 * 18:36 Reception123: added IRC webhook to @miraheze/WikiDiscover
 * 14:45 Reception123: repooled mw3
 * 14:36 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw1 and mw2
 * 14:21 Reception123: clear /srv/mediawiki/config on mw3
 * 14:21 Reception123: depool mw3
 * 14:21 Reception123: repool mw2
 * 14:20 Reception123: manually pull changes on misc1
 * 13:57 Reception123: clear /srv/mediawiki/w on mw2
 * 13:57 Reception123: depool mw2
 * 13:57 Reception123: repool mw1
 * 12:08 Reception123: cleared /srv/mediawiki/w on mw1
 * 12:07 Reception123: depool mw1

2018-04-01

 * 19:49 Reception123: disabled puppet on puppet1
 * 19:19 Reception123: signed certs for all servers on puppet1 except mw* and cp*
 * 15:59 Reception123: reinstalling puppet1 on Debian 9
 * 13:41 Reception123: signed cp5 cert on puppet1
 * 06:54 Reception123: sudo -u www-data cp -r /srv/mediawiki/w/extensions/SocialProfile/avatars & awards /mnt/mediawiki-static/garrettcountyguidewiki

2018-03-31

 * 21:31 John: running manageInactiveWikis.php to close old wikis and mark inactive wikis as wiki_inactive
 * 14:56 Reception123: reinstall test1 to test puppet1 install
 * 06:33 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw*
 * 05:32 Reception123: sudo -u www-data php importDump.php --wiki wiki1776wiki /home/reception/Wikipedia-20180216182614.xml && sudo -u www-data php importDump.php --wiki unionwiki /home/reception/Wikipedia-20180313163344.xml && sudo -u www-data php importDump.php --wiki wiki1776wiki /home/reception/Wikipedia-20180313163344.xml on mw3

2018-03-30

 * 22:40 John: made security task public
 * 19:38 Reception123: made John a phab admin
 * 08:38 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki loraldwiki /home/reception/lorald_pages_current.xml
 * 08:04 Reception123: upgraded phabricator

2018-03-29

 * 17:54 Reception123: manually removed "debian" and "db5" from services file

2018-03-27

 * 16:21 revi: latest command for T2904
 * 16:21 revi: revi@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-admin php namespaceDupes.php --fix --wiki=tmewiki
 * 15:57 revi: sudo puppet agent -t on mw(1-3)

2018-03-25

 * 14:48 Reception123: cleared /tmp on mw2

2018-03-24

 * 18:57 Reception123: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 07:28 Reception123: repool mw1
 * 06:59 Reception123: refresh /srv/mediawiki on mw1
 * 06:59 Reception123: depool mw1

2018-03-23

 * 16:47 Reception123: restarted nginx on mw1

2018-03-21

 * 19:54 Reception123: upgraded phabricator
 * 19:51 Reception123: restarted icinga

2018-03-20

 * 07:12 Reception123: MariaDB [phabricator_herald]> UPDATE herald_rule SET isDisabled = '1' WHERE id = '24';

2018-03-12

 * 17:44 revi: revi@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php initSiteStats.php --update --wiki=reviwikiwiki
 * 17:35 Reception|away: reception@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki reviwikiwiki /home/reception/reviwiki.xml
 * 17:28 Reception|away: DELETED from cw_wikis and DROPPED reviwikiwiki (recreation)

2018-03-11

 * 12:49 revi: make https://phabricator.miraheze.org/H31 stricter

2018-03-10

 * 16:51 revi: phab project import was created while ago, forgot to log
 * 15:41 Reception123: created https://phabricator.miraheze.org/H32 for secure mail

2018-03-09

 * 17:05 Southparkfan: fixed ownership of www-data on mw1

2018-03-08

 * 17:20 Southparkfan: create global_preferences table on centralauth db, drop global_preferences table from metawiki db
 * 17:11 Reception123: ran /srv/mediawiki/w/extensions/GlobalPreferences/schema.sql on metawiki
 * 16:24 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw*
 * 12:35 Reception123: reception@mw3:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-d                                                                                       ata php migrateAccount.php --wiki metawiki --userlist /home/reception/whsapushli                                                                                        st.txt --auto

2018-03-07

 * 19:46 Reception123: deleted and dropped testdeletewiki
 * 15:22 Reception123: restart jobrunners on mw2 and mw3 and manually run jobs on mw1
 * 12:19 Reception123: sudo -u www-data foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/runJobs.php on mw1
 * 09:40 Reception123: sudo -u www-data php /srv/piwik/console core:archive on misc2

2018-03-06

 * 11:50 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw*
 * 06:23 Reception123: upgraded phabricator

2018-03-05

 * 08:09 Reception123: repool mw1
 * 08:06 Reception123: sudo -u www-data php importDump.php --wiki magnaversewiki [and enigmawiki] /home/reception/Infoboxes for Magnaverse - 1.xml
 * 07:53 Reception123: clear out /srv/mediawiki on mw1 and re-run puppet (failing to pull mediawiki)
 * 07:53 Reception123: depool mw1
 * 07:42 Reception123: on sudo -u www-data php importDump.php --wiki almanachdebauswicwiki /home/reception/Gelo+e+Fogo+wiki-20180303164425.xml mw2
 * 07:42 Reception123: sudo -u www-data php importDump.php --wiki almanachdebauswicwiki /home/reception/Gelo+e+Fogo+wiki-20180303163927.xml on mw3

2018-03-03

 * 22:22 Southparkfan: restoring bacula's db3 backup on db4

2018-03-01

 * 21:52 Southparkfan: removed nginx from nfs1
 * 21:02 Southparkfan: replicating db3 to db4 (without read lock, to prevent downtime)
 * 20:14 Southparkfan: downtimed db4 in icinga, moved spfwiki from db4 to db2
 * 00:55 Southparkfan: db2 was holding (deleted) 20GB file mysql-slow.log.1 open. truncated it to reclaim 20GB space
 * 00:52 Southparkfan: db2 was holding file mysql-slow.log.1 open. by truncating it using ":>/proc/763/fd/54" I reclaimed 12GB space
 * 00:29 Southparkfan: purged binary logs @ db3
 * 00:14 Southparkfan: drop db southparkfanwiki on db2

2018-02-28

 * 23:52 Southparkfan: reboot db4 after kernel upgrade
 * 23:42 Southparkfan: reboot db4
 * 17:28 Reception123: removed db5 and "debian" from storeconfigs on puppet and moved /etc/icinga/config/puppet_services.cfg to a backup file then ran puppet to regenerate the list of services
 * 16:43 Reception123: stopped then started apache2 on misc1
 * 16:38 Reception123: removed /var/lib/apt/lists/* and regenerated with apt-get update
 * 16:26 Reception123: sudo service icingabot restart
 * 10:14 Reception123: manually edit (renew) phab.miraheze.wiki on misc1

2018-02-27

 * 21:18 Southparkfan: sysctl vm.swappiness=1 on db4
 * 21:12 Southparkfan: restarted jobrunners
 * 18:55 Reception123: manually run sudo -u www-data foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/runJobs.php on mw1 to alleviate queue
 * 16:58 Southparkfan: truncate allthetropeswiki.objectcache

2018-02-26

 * 14:26 Reception123: chowned /etc/icinga/config/puppet_services.cfg to icinga:icinga
 * 14:15 Reception123: retry moving /etc/icinga/config/puppet_services.cfg to a backup file then ran puppet to regenerate the list of services
 * 08:38 Reception123: > UPDATE cw_wikis SET wiki_closed = '0' WHERE wiki_dbname = '131parkhurstwiki'; (requested, ManageWiki broken)

2018-02-25

 * 19:56 Southparkfan: ALTER TABLE cw_wikis ADD COLUMN wiki_inactive_timestamp BINARY(14) AFTER wiki_inactive;
 * 19:26 Southparkfan: ALTER TABLE cw_wikis ADD COLUMN wiki_closed_timestamp BINARY(14) AFTER wiki_closed; and ALTER TABLE cw_wikis ADD COLUMN wiki_inactive smallint(6) AFTER wiki_closed_timestamp;
 * 18:55 Southparkfan: deleted the following wikis: https://phabricator.miraheze.org/P43
 * 17:57 Reception123: manually restarted icinga as well (required for the changes to take effect)
 * 17:56 Reception123: manually reloaded icinga
 * 17:52 Reception123: moved /etc/icinga/config/puppet_services.cfg to a backup file then ran puppet to regenerate the list of services
 * 17:18 Southparkfan: decommissioned db4
 * 09:32 Reception123: sudo -u www-data php dumpBackup.php --wiki testwiki --full > /home/reception/testwiki25022018.xml on mw3
 * 09:31 Reception123: sudo -u www-data php dumpBackup.php --wiki metawiki --full > /home/reception/metawiki25022018.xml on mw2
 * 06:40 Reception123: reception@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki templatewiki /home/reception/Wikipedia-20180224165206.xml

2018-02-24

 * 14:38 Reception123: sudo service icingabot restart
 * 13:23 Reception123: sudo -u www-data php importDump.php --wiki templatewiki /home/reception/Wikipedia-20180224132217.xml on mw3

2018-02-23

 * 22:39 Southparkfan: restarted ganglia-monitor on parsoid1+bacula1
 * 21:53 Southparkfan: cleaned db entries, dropped databases and removed nfs dirs for wikis: https://phabricator.miraheze.org/P42
 * 21:18 Southparkfan: cleaned db entries, dropped databases and removed nfs dirs for wikis: https://phabricator.miraheze.org/P41
 * 20:40 Southparkfan: cleaned db entries, dropped databases and removed nfs dirs for wikis: https://phabricator.miraheze.org/P40
 * 20:15 Southparkfan: cleaned db entries, dropped databases and removed nfs dirs for wikis: https://phabricator.miraheze.org/P39
 * 16:52 Reception123: reinstalling test1 (for further testing)
 * 16:05 Reception123: added grants for icinga on db4
 * 14:32 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw*
 * 09:42 Reception123: purge binary logs before '2018-02-23 00:00:00'; on db3
 * 07:36 Reception123: sudo service icingabot restart
 * 06:28 Reception123: moved to db3; dropped on db2 aerosswiki diagnostikwiki encyclopaediawiki networksdatabasewiki pocketmonsterswiki scruffy2wiki summerolympicswiki ussewiki

2018-02-22

 * 18:08 Reception123: DELETED from cw_wikis and DROPPED on db2 contextwiki contigoconnectwiki conuconwiki  coopwiki coorwiki coreywiki cpswiki crwikiwiki crxtrdudewiki cswiki ctuvnwiki cubewiki cultofparetowiki customerdesignwiki cwlagusawiki

2018-02-20

 * 20:30 Reception|away: purge binary logs before '2018-02-20 20:00:00'; on db2
 * 09:02 revi: create https://phabricator.miraheze.org/H31 to autoclose spam
 * 06:19 Reception|away: purge binary logs before '2018-02-20 03:00:00'; on db3

2018-02-19

 * 18:12 Reception|away: DELETED FROM cw_wikis and DROPPED on db2 childrensinstitutewiki chocowiki chromiumrosewiki chronicleswiki cinquiemesaisonwiki clarifirewiki clarkfamilywiki classroomsonelightglobalwiki clearicewiki clementsworldbuildingwiki cmpcrswiki cobwebmudwiki code7wiki codebaseclubwiki cofradiawiki cognitowiki colaboratoriowiki colfaxknightswiki collarwiki columbiasjpiwki comidawiki commonsensewiki composerohswiki concertpowerswiki
 * 05:57 Reception123: moved to db3 and dropped on db2 revitwiki unionwiki wiki1776wiki

2018-02-17

 * 19:59 Southparkfan: purged binary logs on dbx
 * 12:37 Reception123: sudo service icingabot restart
 * 07:47 Reception123: brokenworldwiki brooklynrfbwiki brxdwiki bssentialswiki bsuirwiki budgetenvywiki bulldogsracingtestwiki burnoutwiki bzdetopediawikibzswiki cagencywikiwiki cancerwiki cantelli25wiki casinotermswiki caspwiki cassallenwiki  casswiki cbecwiki cecwiki celuwiki chakorduvorwiki chaldunwiki cherryitwiki
 * 07:47 Reception123: DELETED FROM CW_WIKIS and DROPPED bobwiki boftwiki bookswiki bootstrapwiki bottlefarmwiki answerswiki boytvofwiki braindumpwiki bremenwikiwiki bribriwiki broadwaveswiki brofactorywiki

2018-02-16

 * 06:13 Reception|away: cleared /tmp on mw2
 * 05:47 Reception|away: DELETED FROM cw_wikis and DROPPED automotivewiki avidkbwiki azurawiki backyardsportsfanonwiki2wiki banglawiki barigawiki bcmschinawiki beckleynotebookwiki becuzwiki beezerwiki begawktdwiki bellmarewiki bertelwiki besselingwiki bhenuawiki bialowiezawiki bibliowiki bilatwiki bimbowiki bio10a2016garwiki bitphoriawiki bizmodsimwiki blackdesertfanonwiki blackstumpwiki blindfoldstudioswiki

2018-02-15

 * 18:17 Reception|away: reception@mw2:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki templatewiki /home/reception/Wikipedia-20180215181624.xml
 * 18:01 Reception|away: moved to db3 and dropped on db2 nextlevelwikiwiki recentiawiki teriawiki zendariwiki
 * 17:40 Reception|away: moved supernamuwiki from db3 to (temporary) db4, dropped from db3

2018-02-14

 * 19:31 Reception|away: DELETED from CW_WIKIS and dropped ictjudikaturawiki (T2724)
 * 19:17 Reception|away: moved templatewiki and mc2wiki to db3 and dropped on db2

2018-02-12

 * 11:12 revi: https://phabricator.miraheze.org/H30 updated to automatically subscribe CoCC members

2018-02-11

 * 06:57 Reception123: purge binary logs before '2018-02-11 06:00:00'; on db2
 * 06:52 Reception123: sudo -u www-data foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/QuizGame/sql/quizgame.sql
 * 06:48 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw*

2018-02-10

 * 12:03 Reception123: DROPPED wikis below
 * 10:56 Reception123: DELETED FROM CW_WIKIS altenpflegewiki altwiki amningwiki amparodenaswiki annuwikiwiki anon4wiki antoinewiki apenlinjewiki apituwiki apparchitectwiki apslaxwiki aptotecwiki aquivawiki arabudlandwiki arbabilowiki archipediawiki argentinachicagowiki arguwikiwiki arkheonwiki armlaptopwiki askerwiki astawiki atkinswiki atplwiki audiowikiwiki autismadminswiki
 * 07:31 Reception123: sudo -u www-data php importDump.php --wiki podpediawiki /home/reception/roosterteeth_pages_current.xml on mw2

2018-02-08

 * 20:58 Southparkfan: mw1 repooled
 * 19:12 Southparkfan: mw1 emergency depool
 * 19:10 Reception123: purge binary logs before '2018-02-08 16:00:00'; on db2
 * 18:47 Southparkfan: on cp4: varnishlog -g raw -i Backend_health -a -w /var/log/varnish/varnish-backend.log -D
 * 16:37 Southparkfan: ran git submodule update --init in /srv/piwik

2018-02-06

 * 06:46 Reception123: moved liberartwiki to libreartwiki

2018-02-04

 * 13:27 Reception123: moved mistralcupwiki from db4 to db2 (created when metawiki was moved to db4; not declared in Database.php)
 * 09:40 Reception123: moved to db3 and dropped on db2 ctswiki fourleafficswiki karniarutheniawiki madciclistawiki nvcwiki songcontestswiki unionnorteamericanawiki valentinaprojectwiki
 * 05:55 Reception123: moved metawiki back to db2 (has to be on default or else there are CW errros)

2018-02-03

 * 23:53 Southparkfan: dropped worlduniversityandschoolwiki, tmewiki and vandalwiki, all wikis have been transferred to c3
 * 22:44 Southparkfan: moved metawiki to c3, dropped database on db2. backup is in /home/dbcopy for emergencies
 * 18:19 Reception123: upgraded phabricator

2018-02-02

 * 20:04 Reception123: purge binary logs before '2018-02-02 18:00:00'; on db2
 * 18:28 Southparkfan: MariaDB [metawiki]> update cw_wikis set wiki_dbcluster = 'c3' where wiki_dbname = 'testwiki';
 * 15:48 PuppyKun: destroyed db5 last night

2018-02-01

 * 22:03 PuppyKun: root@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance# sudo -u www-data php fixStuckGlobalRename.php --wiki=maiasongcontestwiki --logwiki=metawiki "WEMO666" "Chriskk01b"
 * 21:55 PuppyKun: MariaDB [centralauth]> delete from renameuser_status where ru_wiki="lothuialethwiki"; (query ok, 1 row affected)
 * 21:44 PuppyKun: removeDeletedWikis for lothuialethwiki
 * 18:59 Southparkfan: cleaned puppet certificate info about db5
 * 18:48 PuppyKun: hard rebooting db5
 * 17:29 revi: update phabricator maniphest form 14 ( https://phabricator.miraheze.org/transactions/editengine/maniphest.task/view/14/ ) preamble
 * 02:12 PuppyKun: rebooted db5
 * 02:05 PuppyKun: rebooted db5
 * 02:00 PuppyKun: started hard reboot of db5.miraheze.org (unresponsive to ssh)
 * 01:48 PuppyKun: doing first puppet run on db5.cloud.online.net (??? that was the name that showed up when i did --wait-for-cert on db5.mh.o and the fingerprint matched)

2018-01-29

 * 10:27 revi: create #spam in Phabricator, for spammy tasks

2018-01-28

 * 18:59 Reception123: moved to db3 and dropped on db2 madgendersciencewiki ubrwikiwiki nonbinarywiki linenwiki beatstationwiki daesocialwiki uncyclopediawiki musicarchivewiki thegreatwarwiki christianmusictempwiki hellointernetwiki christianmusicwiki spritopiawiki magnaversewiki
 * 18:24 Reception123: disabled puppet on db3
 * 15:33 Reception123: revi added to @miraheze puppet-users team

2018-01-27

 * 20:36 Reception123: purge binary logs before '2018-01-27 15:00:00'; on db2

2018-01-25

 * 17:04 Reception|away: installed db4 and signed cert on puppet1

2018-01-24

 * 18:48 Reception|away: upgraded phabricator

2018-01-23

 * 06:45 Reception|away: purge binary logs before '2018-01-23 00:00:00';

2018-01-22

 * 20:26 Reception|away: sudo rm -rf /var/lib/puppet/reports on puppet1 (disk space running out)

2018-01-21

 * 10:50 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw*
 * 08:31 Reception123: archived H2 (no longer used)

2018-01-20

 * 08:09 Reception123: sudo -u www-data php dumpBackup.php --wiki metawiki --full > /home/reception/metawiki.xml on mw3

2018-01-19

 * 18:22 Reception123: rebooted test1 (puppetmaster)
 * 17:43 Reception123: sudo -u www-data php importDump.php --wiki pwikiwiki /home/reception/arkcls-wiki-20180117.xml on mw1

2018-01-18

 * 17:54 Reception123: re-enabled puppet on db3
 * 17:54 Reception123: moved databases to db3 and dropped on db2 dariawikiwiki viagroupiawiki truecapitalistwiki emulationwiki cpiwiki ylscwiki hirapediawiki whentheycrywiki utamacrosswiki sumroletaericwiki
 * 17:30 Reception123: disabled puppet on db3

2018-01-16

 * 06:37 Reception123: renewed phab.miraheze.wiki on misc1

2018-01-09

 * 06:49 Reception123: upgraded Phabricator

2018-01-08

 * 06:02 Reception123: restarted Phabricator daemons

2018-01-07

 * 02:33 PuppyKun: rebooted mw2 to fix puppet hangs?

2018-01-06

 * 16:19 Reception123: reinstalling bacula1 (size change; backups not running)

2018-01-05

 * 01:34 PuppyKun: MariaDB [centralauth]> update globaluser set gu_hidden="suppressed" where gu_id in (9192,9193,9188,9190,13117,13118,13119,13140,12407,12405,12404,11327,12293,11296,11292,11280,11481,11716,11717,11196,11197,11198,11199,11201,11202,11203,11204,11205); Query OK, 28 rows affected (0.01 sec)

2018-01-04

 * 18:23 Reception123: sudo -u www-data php importImages.php --wiki dariawikiwiki --search-recursively /home/reception/images on mw1
 * 08:41 Reception123: re-reinstall test1 on debian 9

2018-01-03

 * 16:47 Reception123: reinstalling test1 on debian 9

2018-01-02

 * 15:13 Reception123: started jobrunner on mw2 (moved from test1)
 * 15:09 Reception123: restarted phabricator daemons
 * 08:24 Reception123: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*

2018-01-01

 * 23:26 PuppyKun: renewed publictestwiki.com
 * 15:05 Reception123: cleared /tmp of mw2 (full)
 * 09:08 Reception|away: sudo -u www-data php importDump.php --wiki dariawikiwiki /home/reception/DariaWikiDump_12.31.17.xml on mw2

Information
All times are in UTC+0/GMT.

Archives: 2015, 2016, 2017