Tech:Server admin log/2018

2018-04-21

 * 17:22 paladox: drop phabricator db from db3
 * 17:01 paladox: delete database conworldwiki on db3, it's on db4

2018-04-20

 * 22:02 paladox: ran drop database poserdazfreebieswiki; on db3
 * 21:36 paladox: rm piwik.sql on db4
 * 21:34 paladox: drop database testwiki on db3, it's on db4
 * 21:31 paladox: drop database piwik from db3, it's now on db4.
 * 21:03 paladox: deleted conworldwiki.sql backup from db4
 * 21:01 paladox: sudo service apache2 stop on misc2 for migration of piwik to db4
 * 19:44 paladox: rm -rf /srv/mediawiki-static/unionnorteamericanawiki on nfs1
 * 19:43 paladox: ran DROP DATABASE unionnorteamericanawiki; on db3
 * 19:42 paladox: ran /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/removeDeletedWikis.php --wiki unionnorteamericanawiki on mw1
 * 19:41 paladox: ran DELETE FROM cw_wikis WHERE wiki_dbname = "unionnorteamericanawiki"; on db2
 * 19:12 paladox: drop database conworldwiki from db2 after being migrated to db4
 * 19:01 paladox: removed backup backup.sql.gz from db4
 * 19:00 paladox: remove backups ciptamediawiki.sql and buswiki.sql and bpwiki.sql and jawp2chwiki.sql from db4.
 * 19:00 Reception123: upgraded phabricator

2018-04-19

 * 18:13 paladox: delete db bpwiki on db3 after migration to db4
 * 18:13 paladox: delete db buswiki on db3 after migration to db4.
 * 18:12 paladox: delete db ciptamediawiki on db3 after migration to db4
 * 18:12 paladox: delete db jawp2chwiki on db3 after migration to db4
 * 17:38 paladox: backing up bpwiki buswiki ciptamediawiki jawp2chwiki from db3 to db4 then will migrate to db4.

2018-04-18

 * 23:28 paladox: deleting backup-srv-nfs1 on db4 to clear space up
 * 22:38 paladox: migrating test1wiki to db4
 * 19:19 paladox: reboot cp2
 * 18:48 paladox: upgrading php on test1 to php7.2
 * 17:49 paladox: sudo chown -R www-data:www-data * in /srv/mediawiki-static on nfs1
 * 16:51 paladox: re mount /mnt/mediawiki-static as nfs1 has come back up.
 * 16:50 paladox: unmount /mnt/mediawiki-static on mw* for nfs1 reboot
 * 16:44 paladox: reboot nfs1 to see if that will fix custom images
 * 16:32 Reception123: running dumpBackup.php for all wikis (in batches) on test1wiki
 * 15:40 Southparkfan: disabled puppet on cp4
 * 15:28 paladox: delete backup-root-nfs1
 * 15:28 paladox: delete backup-home-nfs1 on db4
 * 14:40 paladox: upgrade nginx to 1.14 on cp4
 * 14:39 paladox: upgrade nginx to 1.14 on cp2
 * 14:39 paladox: upgrade nginx to 1.14 on cp5
 * 14:37 paladox: upgraded nginx to 1.14 on misc3
 * 14:36 paladox: upgraded nginx to 1.14 on test1
 * 14:27 paladox: repool mw3
 * 14:27 paladox: upgrade to nginx to 1.14 on mw3
 * 14:26 paladox: depool mw3
 * 14:26 paladox: repool mw2
 * 14:23 paladox: upgrade to nginx to 1.14 on mw2
 * 14:23 paladox: depool mw2
 * 14:23 paladox: repool mw1
 * 14:20 paladox: upgrade nginx on mw1 to 1.14
 * 14:20 paladox: depool mw1
 * 13:47 paladox: re enable puppet on mw* but keeping file uploads disabled until backup restored complete
 * 13:30 paladox: syncing nfs1 backup on db4 back to nfs1.
 * 12:56 paladox: reinstall nfs1 as stretch
 * 12:52 paladox: salt -E "mw.*" cmd.run "umount /mnt/mediawiki-static"
 * 12:52 paladox: downtiming nfs1 in icinga
 * 12:49 paladox: disabled puppet on mw*
 * 12:32 paladox: update piwik plugin LoginLdap to 4.0.4
 * 12:16 paladox: taking one last backup of /srv on nfs1
 * 00:50 paladox: rsync -avz --delete -e "ssh -i /home/paladox/id_rsa" /srv/ dbcopy@db4.miraheze.org:/srv/backup-srv-nfs1/ on nfs1 in a screen

2018-04-17

 * 23:54 paladox: restoring all backups onto nfs1, will use rsync
 * 21:54 paladox: deleting allthetropeswiki from nfs1 as it's backed up on db4
 * 20:58 Southparkfan: installed php-memcached on mw1
 * 20:35 paladox: rm -rf aktposwiki on nfs1 backup on db4
 * 17:26 paladox: rebooted nfs1
 * 16:57 paladox: apt-get upgrade on nfs1
 * 16:44 paladox: rm -rf * in /var/lib/puppet/clientbucket on db3 to clean some space
 * 16:43 paladox: rm -rf * in /var/lib/puppet/clientbucket on db2 to clean some space
 * 16:41 paladox: rm -rf * in /var/lib/puppet/clientbucket on nfs1 to clean some space
 * 16:39 paladox: removed apt public key EF0F382A1A7B6500 from nfs1, was warnning with "There is no public key available for the following key IDs:"
 * 16:36 paladox: installing puppet4 on nfs1
 * 16:30 paladox: upgrading puppet to puppet 4 on db3 and db2
 * 15:40 paladox: created database icingaweb2 on db4 for icingaweb2
 * 15:37 paladox: created database icinga on db4 for icinga2

2018-04-16

 * 22:23 paladox: updating acme-tiny in /root on mw1
 * 20:33 paladox: run sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki for mw* on misc3
 * 19:34 paladox: deleted backup backup-repos-misc1 on db4, phabricator may have recreated this as it was working in the ui and there is repos listed.
 * 19:33 paladox: deleted backup "backup-mail-misc1" on db4, was synced when we upgraded misc1 last week.
 * 19:33 paladox: deleted backup "backup-misc2-home" on db4, was synced just now to misc2
 * 19:23 paladox: deleted backup "backup-images-misc1" on db4, was synced to misc1 last week when we upgraded misc1
 * 19:22 paladox: deleted backup "backup-misc2-ganglia" was synced to misc2 on sunday
 * 19:06 paladox: deleted backup backup-misc2-root from db4, i've just synced it to misc2
 * 15:55 paladox: upgraded piwiki plugin "VisitorGenerator" to 3.1.0
 * 15:55 paladox: upgraded piwik plugin "QueuedTracking" to 3.2.0
 * 15:54 paladox: upgraded piwik to 3.4.0 (no db updates so safe to update)

2018-04-15

 * 21:44 paladox: clean up puppet cert for bacula1 pending reinstall
 * 21:42 John: installed bacula1 again with more than 20G allocated to / and about 500GB less to /home
 * 20:22 paladox: rebooted bacula1
 * 16:00 paladox: salt -E 'mw.*' cmd.run 'puppet agent -tv' on misc3
 * 16:00 paladox: salt -E 'mw.*' cmd.run 'puppet agent -tv' on mw*
 * 15:50 paladox: downtimed misc2 in icinga for maint at 4pm utc
 * 15:05 paladox: rebooting puppet1
 * 15:04 paladox: upgrading puppet1 to stretch 9.4
 * 14:57 paladox: rebooting db4 for kernel upgrade
 * 14:52 paladox: reboot db4
 * 14:44 paladox: reboot db4 to test mysql starts up
 * 14:42 paladox: reboot db4 to see if hostname changes correctly
 * 14:30 paladox: rebooting db4 to see if mysql comes back up
 * 13:14 paladox: rebooting db4
 * 12:43 Reception123: deleted temporary user from bacula1
 * 12:24 Reception123: signed bacula1.miraheze.org cert on puppet1

2018-04-14

 * 23:53 paladox: downtimed bacula1 until 5:52am
 * 19:15 Reception123: re-imaging bacula1 with Debian 9
 * 19:14 paladox: Reception123 is reinstalling bacula1 as stretch
 * 19:12 paladox: removed bacula1.miraheze.org puppet cert pending reinstall to stretch
 * 18:12 paladox: installing salt-master on misc3 to see if i can get a pub key for puppet
 * 14:40 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki mw*
 * 14:40 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw3
 * 13:43 paladox: reboot puppet1 to try and clear ram
 * 12:30 paladox: created user guest in icinga
 * 12:29 Reception123: restarted icinga
 * 12:26 Reception123: recreated user 'reception' for icinga
 * 00:48 paladox: upgraded test1 to stretch 9.4
 * 00:48 paladox: rebooting test1

2018-04-13

 * 23:52 Southparkfan: test
 * 23:52 paladox: <+JohnLewis>    !log changed mailname for misc1 to miraheze.org
 * 19:49 paladox: reimaging misc1 to stretch
 * 14:26 paladox: repool mw2
 * 14:19 paladox: rebooting mw2
 * 14:14 paladox: upgrading mw2 to stretch 9.4
 * 14:14 paladox: depooling mw2
 * 14:13 paladox: repooling mw3
 * 14:09 paladox: rebooting mw3
 * 14:08 paladox: upgrade mw3 to stretch 9.4
 * 14:07 paladox: depool mw3
 * 01:54 paladox: rebooting misc3
 * 01:50 paladox: upgrading misc3 to 9.4 (apt-get upgrade) also upgrade to nginx

2018-04-12

 * 23:31 paladox: reimaging cp5 to stretch
 * 22:56 paladox: reimaging cp2 as stretch
 * 22:36 paladox: reinstalling cp4 as stretch
 * 21:59 paladox: reimaging ns1 to stretch
 * 21:52 paladox: repooling mw1
 * 21:44 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw1
 * 20:52 paladox: reimaging mw1 as stretch
 * 20:51 paladox: depooling mw1
 * 20:42 MacFan4000: paladox ran puppet on mw* (logging failed the first time)
 * 20:11 Southparkfan: created backup of /root and /home on mw1, stored on db4 in /srv/backup-{home,root}-mw1
 * 19:52 paladox: rebooting misc1
 * 19:31 paladox: restarting puppetdb on puppet1
 * 19:28 paladox: restarting apache on misc1
 * 19:09 paladox: disabled visualeditor on metawiki due to when using it on the main page it causes an outage
 * 19:00 paladox: downgrading parsoid to 0.8 on misc3
 * 17:24 paladox: repooling mw2
 * 17:21 paladox: depooling mw2
 * 17:19 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw2
 * 17:13 paladox: repooling mw2
 * 17:11 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw3
 * 17:03 Southparkfan: depool mw1 [emergency]
 * 16:41 Reception123: signed puppet cert for mw2
 * 16:32 Reception123: reinstall mw2 on deb 9
 * 16:30 paladox: depooling mw2 from cp[245]
 * 16:26 Reception123: repool mw3
 * 15:41 Reception123: reinstall mw3
 * 15:39 Reception123: depool mw3
 * 04:48 Reception123: sudo -u www-data cp -R /srv/mediawiki/w/extensions/SocialProfile/avatars /srv/mediawiki/w/extensions/SocialProfile/awards /mnt/mediawiki-static/avalicearchiveswiki/

2018-04-11

 * 20:42 paladox: sudo service icingabot restart
 * 19:48 PuppyKun: made paladox a github org owner
 * 19:19 paladox: restart icinga
 * 19:19 paladox: removing parsoid1 from icinga on misc1
 * 19:04 paladox: sudo chown -R www-data:www-data /srv/mediawiki/w/cache/l10n/ on mw3
 * 17:53 paladox: running sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 17:51 paladox: running sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw1
 * 16:41 paladox: running puppet on mw*
 * 16:03 Reception123: upgraded phabricator
 * 15:45 paladox: running sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki
 * 15:45 paladox: running "sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki"
 * 14:58 paladox: redoing sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki
 * 14:58 paladox: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki
 * 14:56 paladox: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki
 * 14:18 Reception123: made paladox a phabricator administrator
 * 14:17 Reception123: added Paladox to @miraheze
 * 00:41 PuppyKun: added john / John Lewis / john@undefinedmiraheze.org as a Matamo superuser.

2018-04-10

 * 23:56 John: parsoid1->misc3, misc3->stretch, parsoid->0.9.0, revert-count->big, error-count->less-than-earlier, success?->questionable-but-it-works

2018-04-09

 * 22:17 John: made H29 only act once
 * 22:13 John: rebuilding LC on mw3
 * 18:45 Reception123: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki
 * 16:33 John: restarted jobrunner on mw2,3
 * 16:31 John: running jobs

2018-04-08

 * 20:28 John: deleted Mkraheze/Math

2018-04-06

 * 18:10 Reception123: repool mw1
 * 18:09 Reception123: clear out /srv/mediawiki/config on mw1 for reclone
 * 18:09 Reception123: depool mw1
 * 18:09 Reception123: repool mw3
 * 18:08 Reception123: clear out /srv/mediawiki/config for reclone
 * 18:08 Reception123: depool mw3
 * 17:49 Reception123: ran check_procs for php7.0-fpm
 * 14:43 Reception123: reboot test1
 * 13:52 Reception123: reinstall test1 on Debian 9
 * 13:46 Reception|away: reception@mw2:/srv/mediawiki/w/extensions/Flow/maintenance$ sudo -u www-data php convertNamespaceFromWikitext.php --wiki isvwiki 
 * 13:05 Reception|away: reception@mw2:/srv/mediawiki/w/maintenance$ sudo -u www-data php populateContentModel.php --wiki isvwiki --ns all --table revision

2018-04-05

 * 15:31 Reception|away: sudo service icingabot restart
 * 15:28 Reception|away: restarted puppetdb on puppet1
 * 13:37 John: db4 has SSL capabilities now

2018-04-04

 * 23:21 John: installed db4 as stretch
 * 22:47 John: reinstall db4 as stretch
 * 22:43 John: unsigned db4 from puppet1
 * 17:19 Reception123: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*

2018-04-03

 * 19:35 Southparkfan: restart ganglia-monitor on puppet1
 * 19:23 John: restart ganglia-monitor on mw1
 * 14:02 John: add cw_custom to cw_requests on metawiki
 * 13:20 John: rebooted db2, excessive server load and CPU usage
 * 09:07 revi: add @miraheze/i18n group to WikiDiscover repo

2018-04-02

 * 19:25 revi: changed phabricator global timezone settings to UTC instead of Europe/London
 * 18:44 Reception123: added mw-admins team to @miraheze/WikiDiscover
 * 18:36 Reception123: added IRC webhook to @miraheze/WikiDiscover
 * 14:45 Reception123: repooled mw3
 * 14:36 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw1 and mw2
 * 14:21 Reception123: clear /srv/mediawiki/config on mw3
 * 14:21 Reception123: depool mw3
 * 14:21 Reception123: repool mw2
 * 14:20 Reception123: manually pull changes on misc1
 * 13:57 Reception123: clear /srv/mediawiki/w on mw2
 * 13:57 Reception123: depool mw2
 * 13:57 Reception123: repool mw1
 * 12:08 Reception123: cleared /srv/mediawiki/w on mw1
 * 12:07 Reception123: depool mw1

2018-04-01

 * 19:49 Reception123: disabled puppet on puppet1
 * 19:19 Reception123: signed certs for all servers on puppet1 except mw* and cp*
 * 15:59 Reception123: reinstalling puppet1 on Debian 9
 * 13:41 Reception123: signed cp5 cert on puppet1
 * 06:54 Reception123: sudo -u www-data cp -r /srv/mediawiki/w/extensions/SocialProfile/avatars & awards /mnt/mediawiki-static/garrettcountyguidewiki

2018-03-31

 * 21:31 John: running manageInactiveWikis.php to close old wikis and mark inactive wikis as wiki_inactive
 * 14:56 Reception123: reinstall test1 to test puppet1 install
 * 06:33 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw*
 * 05:32 Reception123: sudo -u www-data php importDump.php --wiki wiki1776wiki /home/reception/Wikipedia-20180216182614.xml && sudo -u www-data php importDump.php --wiki unionwiki /home/reception/Wikipedia-20180313163344.xml && sudo -u www-data php importDump.php --wiki wiki1776wiki /home/reception/Wikipedia-20180313163344.xml on mw3

2018-03-30

 * 22:40 John: made security task public
 * 19:38 Reception123: made John a phab admin
 * 08:38 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki loraldwiki /home/reception/lorald_pages_current.xml
 * 08:04 Reception123: upgraded phabricator

2018-03-29

 * 17:54 Reception123: manually removed "debian" and "db5" from services file

2018-03-27

 * 16:21 revi: latest command for T2904
 * 16:21 revi: revi@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-admin php namespaceDupes.php --fix --wiki=tmewiki
 * 15:57 revi: sudo puppet agent -t on mw(1-3)

2018-03-25

 * 14:48 Reception123: cleared /tmp on mw2

2018-03-24

 * 18:57 Reception123: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 07:28 Reception123: repool mw1
 * 06:59 Reception123: refresh /srv/mediawiki on mw1
 * 06:59 Reception123: depool mw1

2018-03-23

 * 16:47 Reception123: restarted nginx on mw1

2018-03-21

 * 19:54 Reception123: upgraded phabricator
 * 19:51 Reception123: restarted icinga

2018-03-20

 * 07:12 Reception123: MariaDB [phabricator_herald]> UPDATE herald_rule SET isDisabled = '1' WHERE id = '24';

2018-03-12

 * 17:44 revi: revi@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php initSiteStats.php --update --wiki=reviwikiwiki
 * 17:35 Reception|away: reception@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki reviwikiwiki /home/reception/reviwiki.xml
 * 17:28 Reception|away: DELETED from cw_wikis and DROPPED reviwikiwiki (recreation)

2018-03-11

 * 12:49 revi: make https://phabricator.miraheze.org/H31 stricter

2018-03-10

 * 16:51 revi: phab project import was created while ago, forgot to log
 * 15:41 Reception123: created https://phabricator.miraheze.org/H32 for secure mail

2018-03-09

 * 17:05 Southparkfan: fixed ownership of www-data on mw1

2018-03-08

 * 17:20 Southparkfan: create global_preferences table on centralauth db, drop global_preferences table from metawiki db
 * 17:11 Reception123: ran /srv/mediawiki/w/extensions/GlobalPreferences/schema.sql on metawiki
 * 16:24 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw*
 * 12:35 Reception123: reception@mw3:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-d                                                                                       ata php migrateAccount.php --wiki metawiki --userlist /home/reception/whsapushli                                                                                        st.txt --auto

2018-03-07

 * 19:46 Reception123: deleted and dropped testdeletewiki
 * 15:22 Reception123: restart jobrunners on mw2 and mw3 and manually run jobs on mw1
 * 12:19 Reception123: sudo -u www-data foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/runJobs.php on mw1
 * 09:40 Reception123: sudo -u www-data php /srv/piwik/console core:archive on misc2

2018-03-06

 * 11:50 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw*
 * 06:23 Reception123: upgraded phabricator

2018-03-05

 * 08:09 Reception123: repool mw1
 * 08:06 Reception123: sudo -u www-data php importDump.php --wiki magnaversewiki [and enigmawiki] /home/reception/Infoboxes for Magnaverse - 1.xml
 * 07:53 Reception123: clear out /srv/mediawiki on mw1 and re-run puppet (failing to pull mediawiki)
 * 07:53 Reception123: depool mw1
 * 07:42 Reception123: on sudo -u www-data php importDump.php --wiki almanachdebauswicwiki /home/reception/Gelo+e+Fogo+wiki-20180303164425.xml mw2
 * 07:42 Reception123: sudo -u www-data php importDump.php --wiki almanachdebauswicwiki /home/reception/Gelo+e+Fogo+wiki-20180303163927.xml on mw3

2018-03-03

 * 22:22 Southparkfan: restoring bacula's db3 backup on db4

2018-03-01

 * 21:52 Southparkfan: removed nginx from nfs1
 * 21:02 Southparkfan: replicating db3 to db4 (without read lock, to prevent downtime)
 * 20:14 Southparkfan: downtimed db4 in icinga, moved spfwiki from db4 to db2
 * 00:55 Southparkfan: db2 was holding (deleted) 20GB file mysql-slow.log.1 open. truncated it to reclaim 20GB space
 * 00:52 Southparkfan: db2 was holding file mysql-slow.log.1 open. by truncating it using ":>/proc/763/fd/54" I reclaimed 12GB space
 * 00:29 Southparkfan: purged binary logs @ db3
 * 00:14 Southparkfan: drop db southparkfanwiki on db2

2018-02-28

 * 23:52 Southparkfan: reboot db4 after kernel upgrade
 * 23:42 Southparkfan: reboot db4
 * 17:28 Reception123: removed db5 and "debian" from storeconfigs on puppet and moved /etc/icinga/config/puppet_services.cfg to a backup file then ran puppet to regenerate the list of services
 * 16:43 Reception123: stopped then started apache2 on misc1
 * 16:38 Reception123: removed /var/lib/apt/lists/* and regenerated with apt-get update
 * 16:26 Reception123: sudo service icingabot restart
 * 10:14 Reception123: manually edit (renew) phab.miraheze.wiki on misc1

2018-02-27

 * 21:18 Southparkfan: sysctl vm.swappiness=1 on db4
 * 21:12 Southparkfan: restarted jobrunners
 * 18:55 Reception123: manually run sudo -u www-data foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/runJobs.php on mw1 to alleviate queue
 * 16:58 Southparkfan: truncate allthetropeswiki.objectcache

2018-02-26

 * 14:26 Reception123: chowned /etc/icinga/config/puppet_services.cfg to icinga:icinga
 * 14:15 Reception123: retry moving /etc/icinga/config/puppet_services.cfg to a backup file then ran puppet to regenerate the list of services
 * 08:38 Reception123: > UPDATE cw_wikis SET wiki_closed = '0' WHERE wiki_dbname = '131parkhurstwiki'; (requested, ManageWiki broken)

2018-02-25

 * 19:56 Southparkfan: ALTER TABLE cw_wikis ADD COLUMN wiki_inactive_timestamp BINARY(14) AFTER wiki_inactive;
 * 19:26 Southparkfan: ALTER TABLE cw_wikis ADD COLUMN wiki_closed_timestamp BINARY(14) AFTER wiki_closed; and ALTER TABLE cw_wikis ADD COLUMN wiki_inactive smallint(6) AFTER wiki_closed_timestamp;
 * 18:55 Southparkfan: deleted the following wikis: https://phabricator.miraheze.org/P43
 * 17:57 Reception123: manually restarted icinga as well (required for the changes to take effect)
 * 17:56 Reception123: manually reloaded icinga
 * 17:52 Reception123: moved /etc/icinga/config/puppet_services.cfg to a backup file then ran puppet to regenerate the list of services
 * 17:18 Southparkfan: decommissioned db4
 * 09:32 Reception123: sudo -u www-data php dumpBackup.php --wiki testwiki --full > /home/reception/testwiki25022018.xml on mw3
 * 09:31 Reception123: sudo -u www-data php dumpBackup.php --wiki metawiki --full > /home/reception/metawiki25022018.xml on mw2
 * 06:40 Reception123: reception@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki templatewiki /home/reception/Wikipedia-20180224165206.xml

2018-02-24

 * 14:38 Reception123: sudo service icingabot restart
 * 13:23 Reception123: sudo -u www-data php importDump.php --wiki templatewiki /home/reception/Wikipedia-20180224132217.xml on mw3

2018-02-23

 * 22:39 Southparkfan: restarted ganglia-monitor on parsoid1+bacula1
 * 21:53 Southparkfan: cleaned db entries, dropped databases and removed nfs dirs for wikis: https://phabricator.miraheze.org/P42
 * 21:18 Southparkfan: cleaned db entries, dropped databases and removed nfs dirs for wikis: https://phabricator.miraheze.org/P41
 * 20:40 Southparkfan: cleaned db entries, dropped databases and removed nfs dirs for wikis: https://phabricator.miraheze.org/P40
 * 20:15 Southparkfan: cleaned db entries, dropped databases and removed nfs dirs for wikis: https://phabricator.miraheze.org/P39
 * 16:52 Reception123: reinstalling test1 (for further testing)
 * 16:05 Reception123: added grants for icinga on db4
 * 14:32 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw*
 * 09:42 Reception123: purge binary logs before '2018-02-23 00:00:00'; on db3
 * 07:36 Reception123: sudo service icingabot restart
 * 06:28 Reception123: moved to db3; dropped on db2 aerosswiki diagnostikwiki encyclopaediawiki networksdatabasewiki pocketmonsterswiki scruffy2wiki summerolympicswiki ussewiki

2018-02-22

 * 18:08 Reception123: DELETED from cw_wikis and DROPPED on db2 contextwiki contigoconnectwiki conuconwiki  coopwiki coorwiki coreywiki cpswiki crwikiwiki crxtrdudewiki cswiki ctuvnwiki cubewiki cultofparetowiki customerdesignwiki cwlagusawiki

2018-02-20

 * 20:30 Reception|away: purge binary logs before '2018-02-20 20:00:00'; on db2
 * 09:02 revi: create https://phabricator.miraheze.org/H31 to autoclose spam
 * 06:19 Reception|away: purge binary logs before '2018-02-20 03:00:00'; on db3

2018-02-19

 * 18:12 Reception|away: DELETED FROM cw_wikis and DROPPED on db2 childrensinstitutewiki chocowiki chromiumrosewiki chronicleswiki cinquiemesaisonwiki clarifirewiki clarkfamilywiki classroomsonelightglobalwiki clearicewiki clementsworldbuildingwiki cmpcrswiki cobwebmudwiki code7wiki codebaseclubwiki cofradiawiki cognitowiki colaboratoriowiki colfaxknightswiki collarwiki columbiasjpiwki comidawiki commonsensewiki composerohswiki concertpowerswiki
 * 05:57 Reception123: moved to db3 and dropped on db2 revitwiki unionwiki wiki1776wiki

2018-02-17

 * 19:59 Southparkfan: purged binary logs on dbx
 * 12:37 Reception123: sudo service icingabot restart
 * 07:47 Reception123: brokenworldwiki brooklynrfbwiki brxdwiki bssentialswiki bsuirwiki budgetenvywiki bulldogsracingtestwiki burnoutwiki bzdetopediawikibzswiki cagencywikiwiki cancerwiki cantelli25wiki casinotermswiki caspwiki cassallenwiki  casswiki cbecwiki cecwiki celuwiki chakorduvorwiki chaldunwiki cherryitwiki
 * 07:47 Reception123: DELETED FROM CW_WIKIS and DROPPED bobwiki boftwiki bookswiki bootstrapwiki bottlefarmwiki answerswiki boytvofwiki braindumpwiki bremenwikiwiki bribriwiki broadwaveswiki brofactorywiki

2018-02-16

 * 06:13 Reception|away: cleared /tmp on mw2
 * 05:47 Reception|away: DELETED FROM cw_wikis and DROPPED automotivewiki avidkbwiki azurawiki backyardsportsfanonwiki2wiki banglawiki barigawiki bcmschinawiki beckleynotebookwiki becuzwiki beezerwiki begawktdwiki bellmarewiki bertelwiki besselingwiki bhenuawiki bialowiezawiki bibliowiki bilatwiki bimbowiki bio10a2016garwiki bitphoriawiki bizmodsimwiki blackdesertfanonwiki blackstumpwiki blindfoldstudioswiki

2018-02-15

 * 18:17 Reception|away: reception@mw2:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki templatewiki /home/reception/Wikipedia-20180215181624.xml
 * 18:01 Reception|away: moved to db3 and dropped on db2 nextlevelwikiwiki recentiawiki teriawiki zendariwiki
 * 17:40 Reception|away: moved supernamuwiki from db3 to (temporary) db4, dropped from db3

2018-02-14

 * 19:31 Reception|away: DELETED from CW_WIKIS and dropped ictjudikaturawiki (T2724)
 * 19:17 Reception|away: moved templatewiki and mc2wiki to db3 and dropped on db2

2018-02-12

 * 11:12 revi: https://phabricator.miraheze.org/H30 updated to automatically subscribe CoCC members

2018-02-11

 * 06:57 Reception123: purge binary logs before '2018-02-11 06:00:00'; on db2
 * 06:52 Reception123: sudo -u www-data foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/QuizGame/sql/quizgame.sql
 * 06:48 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw*

2018-02-10

 * 12:03 Reception123: DROPPED wikis below
 * 10:56 Reception123: DELETED FROM CW_WIKIS altenpflegewiki altwiki amningwiki amparodenaswiki annuwikiwiki anon4wiki antoinewiki apenlinjewiki apituwiki apparchitectwiki apslaxwiki aptotecwiki aquivawiki arabudlandwiki arbabilowiki archipediawiki argentinachicagowiki arguwikiwiki arkheonwiki armlaptopwiki askerwiki astawiki atkinswiki atplwiki audiowikiwiki autismadminswiki
 * 07:31 Reception123: sudo -u www-data php importDump.php --wiki podpediawiki /home/reception/roosterteeth_pages_current.xml on mw2

2018-02-08

 * 20:58 Southparkfan: mw1 repooled
 * 19:12 Southparkfan: mw1 emergency depool
 * 19:10 Reception123: purge binary logs before '2018-02-08 16:00:00'; on db2
 * 18:47 Southparkfan: on cp4: varnishlog -g raw -i Backend_health -a -w /var/log/varnish/varnish-backend.log -D
 * 16:37 Southparkfan: ran git submodule update --init in /srv/piwik

2018-02-06

 * 06:46 Reception123: moved liberartwiki to libreartwiki

2018-02-04

 * 13:27 Reception123: moved mistralcupwiki from db4 to db2 (created when metawiki was moved to db4; not declared in Database.php)
 * 09:40 Reception123: moved to db3 and dropped on db2 ctswiki fourleafficswiki karniarutheniawiki madciclistawiki nvcwiki songcontestswiki unionnorteamericanawiki valentinaprojectwiki
 * 05:55 Reception123: moved metawiki back to db2 (has to be on default or else there are CW errros)

2018-02-03

 * 23:53 Southparkfan: dropped worlduniversityandschoolwiki, tmewiki and vandalwiki, all wikis have been transferred to c3
 * 22:44 Southparkfan: moved metawiki to c3, dropped database on db2. backup is in /home/dbcopy for emergencies
 * 18:19 Reception123: upgraded phabricator

2018-02-02

 * 20:04 Reception123: purge binary logs before '2018-02-02 18:00:00'; on db2
 * 18:28 Southparkfan: MariaDB [metawiki]> update cw_wikis set wiki_dbcluster = 'c3' where wiki_dbname = 'testwiki';
 * 15:48 PuppyKun: destroyed db5 last night

2018-02-01

 * 22:03 PuppyKun: root@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance# sudo -u www-data php fixStuckGlobalRename.php --wiki=maiasongcontestwiki --logwiki=metawiki "WEMO666" "Chriskk01b"
 * 21:55 PuppyKun: MariaDB [centralauth]> delete from renameuser_status where ru_wiki="lothuialethwiki"; (query ok, 1 row affected)
 * 21:44 PuppyKun: removeDeletedWikis for lothuialethwiki
 * 18:59 Southparkfan: cleaned puppet certificate info about db5
 * 18:48 PuppyKun: hard rebooting db5
 * 17:29 revi: update phabricator maniphest form 14 ( https://phabricator.miraheze.org/transactions/editengine/maniphest.task/view/14/ ) preamble
 * 02:12 PuppyKun: rebooted db5
 * 02:05 PuppyKun: rebooted db5
 * 02:00 PuppyKun: started hard reboot of db5.miraheze.org (unresponsive to ssh)
 * 01:48 PuppyKun: doing first puppet run on db5.cloud.online.net (??? that was the name that showed up when i did --wait-for-cert on db5.mh.o and the fingerprint matched)

2018-01-29

 * 10:27 revi: create #spam in Phabricator, for spammy tasks

2018-01-28

 * 18:59 Reception123: moved to db3 and dropped on db2 madgendersciencewiki ubrwikiwiki nonbinarywiki linenwiki beatstationwiki daesocialwiki uncyclopediawiki musicarchivewiki thegreatwarwiki christianmusictempwiki hellointernetwiki christianmusicwiki spritopiawiki magnaversewiki
 * 18:24 Reception123: disabled puppet on db3
 * 15:33 Reception123: revi added to @miraheze puppet-users team

2018-01-27

 * 20:36 Reception123: purge binary logs before '2018-01-27 15:00:00'; on db2

2018-01-25

 * 17:04 Reception|away: installed db4 and signed cert on puppet1

2018-01-24

 * 18:48 Reception|away: upgraded phabricator

2018-01-23

 * 06:45 Reception|away: purge binary logs before '2018-01-23 00:00:00';

2018-01-22

 * 20:26 Reception|away: sudo rm -rf /var/lib/puppet/reports on puppet1 (disk space running out)

2018-01-21

 * 10:50 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw*
 * 08:31 Reception123: archived H2 (no longer used)

2018-01-20

 * 08:09 Reception123: sudo -u www-data php dumpBackup.php --wiki metawiki --full > /home/reception/metawiki.xml on mw3

2018-01-19

 * 18:22 Reception123: rebooted test1 (puppetmaster)
 * 17:43 Reception123: sudo -u www-data php importDump.php --wiki pwikiwiki /home/reception/arkcls-wiki-20180117.xml on mw1

2018-01-18

 * 17:54 Reception123: re-enabled puppet on db3
 * 17:54 Reception123: moved databases to db3 and dropped on db2 dariawikiwiki viagroupiawiki truecapitalistwiki emulationwiki cpiwiki ylscwiki hirapediawiki whentheycrywiki utamacrosswiki sumroletaericwiki
 * 17:30 Reception123: disabled puppet on db3

2018-01-16

 * 06:37 Reception123: renewed phab.miraheze.wiki on misc1

2018-01-09

 * 06:49 Reception123: upgraded Phabricator

2018-01-08

 * 06:02 Reception123: restarted Phabricator daemons

2018-01-07

 * 02:33 PuppyKun: rebooted mw2 to fix puppet hangs?

2018-01-06

 * 16:19 Reception123: reinstalling bacula1 (size change; backups not running)

2018-01-05

 * 01:34 PuppyKun: MariaDB [centralauth]> update globaluser set gu_hidden="suppressed" where gu_id in (9192,9193,9188,9190,13117,13118,13119,13140,12407,12405,12404,11327,12293,11296,11292,11280,11481,11716,11717,11196,11197,11198,11199,11201,11202,11203,11204,11205); Query OK, 28 rows affected (0.01 sec)

2018-01-04

 * 18:23 Reception123: sudo -u www-data php importImages.php --wiki dariawikiwiki --search-recursively /home/reception/images on mw1
 * 08:41 Reception123: re-reinstall test1 on debian 9

2018-01-03

 * 16:47 Reception123: reinstalling test1 on debian 9

2018-01-02

 * 15:13 Reception123: started jobrunner on mw2 (moved from test1)
 * 15:09 Reception123: restarted phabricator daemons
 * 08:24 Reception123: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*

2018-01-01

 * 23:26 PuppyKun: renewed publictestwiki.com
 * 15:05 Reception123: cleared /tmp of mw2 (full)
 * 09:08 Reception|away: sudo -u www-data php importDump.php --wiki dariawikiwiki /home/reception/DariaWikiDump_12.31.17.xml on mw2

Information
All times are in UTC+0/GMT.

Archives: 2015, 2016, 2017