Tech:Server admin log/2021

2021-03-17

 * 00:18 Southparkfan: downtimed load alerts for dbbackup[12] till March 31, per https://phabricator.miraheze.org/T6984

2021-03-16

 * 09:48 Reception123: deleted and dropped tscwiki

2021-03-15

 * 19:38 paladox: restart varnish on cp12
 * 18:43 Reception123: reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=pokemundowiki --r "Requested" /home/reception/deletebatch.txt
 * 18:39 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki maximusescwiki /home/reception/delbackups3/maximusescwiki.xml --username-prefix="maximusesc"
 * 18:19 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=awesomegameswiki --r "Requested" /home/reception/awesomegamesdel.txt
 * 17:08 Reception123: DROP DATABASE maximusescwiki;
 * 01:21 paladox: rm /etc/systemd/system/nutcracker.service.d/puppet-override.conf ; systemctl daemon-reload ; service nutcracker restart on mw* and jobrunner*

2021-03-14

 * 23:30 paladox: upgrade phabricator
 * 22:06 RhinosF1: sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/migrateAccount.php --username=Tronbing5 --wiki=metawiki --auto (DONE - END MIGRATIONS)
 * 22:06 RhinosF1: sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/migrateAccount.php --username=Lysander123456 --wiki=metawiki --auto
 * 22:05 RhinosF1: sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/migrateAccount.php --username=Glossuwyn --wiki=metawiki --auto
 * 22:04 RhinosF1: rhinos@jobrunner3:/$ sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/migrateAccount.php --username=Falfulynw200202 --wiki=metawiki --auto
 * 22:04 RhinosF1: fixing broken SUL accounts so @Void can lock
 * 17:32 paladox: create feraltradinghandbookwiki in matomo
 * 17:30 paladox: create uselesssoftwarewiki in matomo
 * 17:28 paladox: create wikipucwiki and androidwiki in matomo
 * 16:21 Reception123: reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=awfulmovieswiki --r "Requested" /home/reception/awfulmovies.txt
 * 15:39 paladox: created zymonicwiki in matomo
 * 12:45 Reception123: reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=greatcharacterswikiwiki --r "Requested" /home/reception/GCW.txt
 * 12:17 Reception123: reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=loathsomecharacterswiki --r "Requested" /home/reception/LSCW.txt
 * 11:00 Reception123: reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=crappygameswiki --r "Requested" /home/reception/CGW.txt
 * 10:58 Reception123: reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=besttvshowswiki --r "Requested" /home/reception/BTVSW.txt

2021-03-13

 * 18:52 Reception123: GDPR script
 * 10:30 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=trollpastawiki --r "Requested" /home/reception/trollpastadel.txt

2021-03-12

 * 22:11 paladox: cp10/11/3: rm varnishncsa.log*
 * 22:09 paladox: cp12: rm varnishncsa.log*
 * 21:14 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw*, jbr*, and test3
 * 16:42 Reception123: rebooted cp12
 * 15:36 Reception123: reception@jobrunner3:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php createLocalAccount.php --wiki metawiki Elli per Phabricator request

2021-03-11

 * 23:36 paladox: restart dovecot on mail2
 * 06:54 Reception123: reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki minecraftwiki /home/reception/minecraft_pages_full.xml --username-prefix="wikia:minecraftwiki"
 * 06:44 Reception123: sudo service ircrcbot restart

2021-03-10

 * 22:33 paladox: starting db13 backup to dbbackup1
 * 22:29 Southparkfan: stop mariadb c3 replica
 * 22:19 Southparkfan: started replica process for c3, failed because binary logs on the master have been purged since
 * 18:04 paladox: regenerate backups on bacula2
 * 16:21 paladox: install python3-yaml on jobrunner3
 * 03:57 Universal_Omega: UPDATE mw_settings SET s_settings = REPLACE(s_settings,'wgDisqusShortname', 'egDisqusShortname');

2021-03-09

 * 20:22 Reception123: reception@jobrunner3:/srv/mediawiki/w/maintenance$ sudo -u www-data php runJobs.php --wiki metawiki
 * 06:38 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=jrapp494wiki --logwiki=metawiki 'IntegraInfinity' 'OfficerJH'
 * 00:03 paladox: root@jobrunner3:/srv/mediawiki/w/maintenance# sudo -u www-data php runJobs.php --wiki metawiki

2021-03-08

 * 18:46 Reception123: reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=shekhwiki --r "Requested" /home/reception/shekhdel.txt
 * 00:16 Southparkfan: enable puppet on gluster4
 * 00:03 Southparkfan: disable puppet on gluster4 for syslog testing

2021-03-07

 * 17:56 Reception123: reception@jobrunner3:/srv/mediawiki/w/extensions/ReplaceText/maintenance$ sudo -u www-data php replaceAll.php --wiki pokemundowiki --replacements /home/reception/movebatch.txt --rename
 * 16:06 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=teessidehackspacewiki -u "Dmehus" -r "Spam Steward request on Phabricator" /home/reception/teessidehackspace.txt
 * 14:39 Reception123: reception@jobrunner3:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php migrateAccount.php --wiki loginwiki --username Trafficcone3 --attachbroken
 * 12:25 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki callipediawiki /home/reception/Wikidump[01-07] --username-prefix="w" [01-03 on jbr3, 04-07 on jbr4)
 * 06:39 Reception123: renamed civclassicwiki to civwikiwiki.

2021-03-06

 * 23:35 Universal_Omega: universalomega@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/extensions/Wikibase/repo/maintenance/rebuildPropertyInfo.php --rebuild-all --force --wiki=nbdbwiki
 * 22:15 RhinosF1: rhinos@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/migrateAccount.php --username=Hatelynnloudjrok --wiki=loathsomecharacterswiki --auto (per request of Stewards so account can be locked)
 * 19:25 Universal_Omega: universalomega@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=metawiki (Failed with Fatal error: Allowed memory size of 157286400 bytes exhausted )
 * 01:49 paladox: depool cp10 && 11

2021-03-05

 * 23:09 paladox: restart dovecot
 * 17:37 paladox: reinstall rdb3 as mem1
 * 15:31 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki santrovelowiki /home/reception/delbackups4/santrovelowiki.xml --username-prefix="santrovelo"
 * 15:26 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki chadsofawiki /home/reception/officialdragonballsuper_pages_full.xml
 * 15:18 Reception123: reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki mylittlefandompolskawiki /home/reception/MLF.xml
 * 13:39 RhinosF1: used eval.php to fix a failed wiki creation auto create user as the script wanted to be awkward

2021-03-04

 * 12:55 Reception123: sudo umount/mount /mnt/mediawiki-static

2021-03-03

 * 15:52 paladox: jobrunner3: /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/ApprovedRevs/sql/patch-approval_date.sql

2021-03-02

 * 18:49 Reception123: sudo apt-get upgrade on mw*/cp*/phab2/ldap2
 * 11:58 paladox: restart dovecot

2021-03-01

 * 23:28 paladox: dropped mail: prefix from all users in ldap
 * 16:24 paladox: gluster: gluster volume set static performance.write-behind off
 * 00:22 Southparkfan: varnish ban every url that starts with "https://static.miraheze.org/terribletvshows"

2021-02-28

 * 23:31 paladox: jobrunner3: Ran deleteArchivedFiles.php but failed half way through with Class 'Psr\Log\LogLevel' not found
 * 23:30 paladox: jobrunner3: ran eraseArchivedFile.php
 * 22:12 paladox: restart dovecot on mail2
 * 22:11 paladox: block ip in firewall on mail2
 * 16:50 paladox: jobrunner3: /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /home/paladox/clearMatomoCache.php
 * 16:25 paladox: jobrunner3: sudo -u www-data php modifyMatomo.php --wiki loginwiki
 * 16:05 paladox: rebuild lc on mw* and jobrunner*
 * 14:47 paladox: rebuild lc on mw* and jobrunner*
 * 09:36 Reception123: deleted and dropped wikis https://phabricator.miraheze.org/P388
 * 08:36 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/deleteWikis.php --wiki=loginwiki --delete Reception123
 * 08:13 Reception123: running delbackups
 * 06:34 Reception123: renamed sbtestwiki to samskritavyakaranamwiki
 * 03:43 paladox: test3: sudo -u www-data php toggleExtension.php --wiki nspediawiki wikibaserepository (throwing Class 'Wikibase\\Repo\\WikibaseRepo' not found" on wiki)
 * 03:36 paladox: restart varnish to clear cache
 * 03:18 paladox: jobrunner3: sudo -u www-data php modifyMatomo.php --wiki loginwiki
 * 02:54 paladox: noting for the record that i ran the modifyMatomo.php script twice (once with --remove to ensure it was deleted in the matomo site (even though it wasn't in the mediawiki matomo table) and then again to create an entry)
 * 02:28 paladox: jobrunner3: sudo -u www-data php modifyMatomo.php --wiki digitaltransformationwiki
 * 02:28 paladox: jobrunner3: sudo -u www-data php modifyMatomo.php --wiki linguapediawiki
 * 02:27 paladox: jobrunner3: sudo -u www-data php modifyMatomo.php --wiki militarywiki
 * 02:26 paladox: jobrunner3: sudo -u www-data php modifyMatomo.php --wiki sephspacewiki
 * 02:26 paladox: jobrunner3: sudo -u www-data php modifyMatomo.php --wiki highdeologywikiwiki
 * 02:25 paladox: jobrunner3: sudo -u www-data php modifyMatomo.php --wiki cnocbridewiki
 * 01:29 paladox: db11: MariaDB [mhglobal]> alter table cw_wikis drop column cw_bio;
 * 01:28 paladox: db11: MariaDB [metawiki]> ALTER TABLE /*$wgDBprefix*/cw_requests ADD COLUMN cw_bio TINYINT UNSIGNED NOT NULL DEFAULT '0';

2021-02-27

 * 23:53 paladox: rebuild lc on mw* and jobrunner*
 * 23:48 paladox: db11: MariaDB [mhglobal]> ALTER TABLE /*$wgDBprefix*/cw_wikis ADD COLUMN cw_bio TINYINT UNSIGNED NOT NULL DEFAULT '0';
 * 16:30 Reception123: reception@jobrunner3:/srv/mediawiki/w/extensions/CreateWiki/maintenance$ sudo -u www-data php populateMainPage.php --wiki robotiawiki
 * 16:29 Reception123: reception@jobrunner3:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php createLocalAccount.php --wiki robotiawiki Robotia
 * 14:54 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=terribletvshowswiki --r "Requested" /home/reception/Terrible_TV_Shows_Wiki.txt
 * 14:52 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=rottenwebsiteswiki --r "Requested" /home/reception/Rotten_Websites_Wiki.txt
 * 14:30 paladox: test
 * 14:28 paladox: test
 * 13:51 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=loathsomecharacterswiki --r "Requested" /home/reception/Loathsome_Characters_Wiki.txt
 * 12:46 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=greatestmovieswiki --r "Requested" /home/reception/Greatest_Movies_Wiki.txt
 * 08:37 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=crappygameswiki --r "Requested" /home/reception/Crappy_Games_Wiki.txt
 * 06:36 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=besttvshowswiki --r "Requested" /home/reception/Best_TV_Shows_Wiki.txt

2021-02-26

 * 18:48 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=awfulmovieswiki --r "Requested" /home/reception/Awful_Movies_Wiki.txt
 * 17:33 Reception123: deleted and recreated greenleftwiki, then imported XML

2021-02-25

 * 20:13 John: changed devnull to double-bounce (hardcoded postfix discard email)
 * 18:35 Reception123: replaced greenleftwiki SQL with backup SQL
 * 10:35 Reception123: sudo service logbot start
 * 02:36 paladox: backed up and dropped greenleftwiki on db12

2021-02-24

 * 21:07 paladox: restart graylog-server as appeared stuck on logging in?
 * 20:26 Reception123: MariaDB [metawiki]> DELETE FROM echo_unread_wikis WHERE euw_user = '9736' AND euw_id = '2768838';
 * 19:17 Reception123: added Sario to bots@
 * 01:21 paladox: restart varnish on cp12 as it appears to have crashed

2021-02-23

 * 19:05 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki peacewiki /home/reception/delbackups4/peacewiki.xml
 * 02:26 paladox: restart dovecot on mail2
 * 00:54 Universal_Omega: universalomega@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=terribletvshowswiki --r "Requested" /home/universalomega/blogcomments.txt
 * 00:54 Universal_Omega: universalomega@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=terribletvshowswiki --r "Requested" /home/universalomega/blogs.txt
 * 00:53 Universal_Omega: universalomega@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=terribletvshowswiki --r "Requested" /home/universalomega/comments.txt
 * 00:44 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=terribletvshowswiki --r "Requested" /home/universalomega/threads.txt

2021-02-22

 * 17:37 paladox: upgrade puppetserver to 6.15.1
 * 08:58 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki trollpastawiki /home/reception/trollpasta_pages_full.xml
 * 08:44 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki besttvshowswiki --r "Requested" /home/reception/btsdel.txt
 * 08:39 Reception123: reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki terribletvshowswiki --r "Requested" /home/reception/ttsdel.txt
 * 08:38 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki greatestmovieswiki --r "Requested" /home/reception/gmdel.txt
 * 08:15 Reception123: reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki awfulmovieswiki --r "Requested" /home/reception/amdel.txt
 * 08:11 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki rottenwebsiteswiki --r "Requested" /home/reception/rwwdel.txt
 * 08:09 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki crappygameswiki --r "Requested" /home/reception/cgwdel.txt

2021-02-21

 * 20:42 Reception123: changed a Phabricator email
 * 19:35 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki terribletvshowswiki --r "Requested" /home/reception/ttsdel.txt
 * 19:35 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki terribletvshowswiki
 * 19:35 Universal_Omega: DELETE FROM oldimage WHERE oi_archive_name=''; on terribletvshowswiki
 * 18:29 Reception123: reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki terribletvshowswiki
 * 17:40 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki horribletvshowepisodeswiki --r "Requested" /home/reception/hhdel.txt
 * 14:29 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateGroupPermissionsWithDefaults.php --wiki kelevarwiki --overwrite
 * 08:57 Reception123: reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki terribletvshowswiki --r "Requested" /home/reception/ttsdel.txt
 * 01:36 Universal_Omega: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php on Jobrunner3 - very high backlog of outstanding jobs on alot of wikis.

2021-02-20

 * 20:41 paladox: root@jobrunner3:/home/paladox# /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php
 * 20:26 Reception123: fixed a Phabricator email
 * 02:20 paladox: upgrade php on mon2

2021-02-19

 * 23:35 Universal_Omega: universalomega@jobrunner3:/srv/mediawiki/w/extensions/PollNY/maintenance$ sudo -u www-data php migrateOldPollNYUserColumnsToActor.php --wiki=crappygameswiki
 * 23:32 Universal_Omega: CREATE INDEX /*i*/pv_actor ON /*_*/poll_user_vote (pv_actor); on crappygameswiki
 * 23:31 Universal_Omega: ALTER TABLE /*_*/poll_user_vote ADD COLUMN pv_actor bigint unsigned NOT NULL AFTER pv_pc_id; on crappygameswiki
 * 23:31 Universal_Omega: CREATE INDEX /*i*/poll_actor ON /*_*/poll_question (poll_actor); on crappygameswiki
 * 23:29 Universal_Omega: ALTER TABLE /*_*/poll_question ADD COLUMN poll_actor bigint unsigned NOT NULL AFTER poll_page_id; on crappygameswiki
 * 19:27 paladox: graylog2: curl -XPUT -H 'Content-Type: application/json' -d '{ "index" : { "max_result_window" : 9000000 } }' http://127.0.0.1:9200/_settings
 * 18:45 paladox: restart dovecot
 * 16:49 paladox: rebuild lc on test3
 * 16:46 paladox: rebuild lc on mw* and jobrunner*

2021-02-18

 * 18:38 paladox: install security updates on all servers
 * 18:34 paladox: root@jobrunner3:/home/paladox# /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /home/paladox/resetWikiCaches.php
 * 18:33 paladox: db11: update mw_settings set s_settings = REPLACE(s_settings,'wgUsersNotifiedOnAllChanges', 'wmgUsersNotifiedOnAllChanges');
 * 15:52 John: reboot test3 so it has 2G of RAM not 1.5G
 * 08:22 Reception123: sudo apt-get upgrade on mon2
 * 07:06 Reception123: MariaDB [(none)]> DROP DATABASE trollpastawiki; (part of P290$930, forgotten)
 * 06:53 Reception123: re-run wikibackups
 * 05:06 paladox: restart php7.3-fpm on mw* and jobrunner* to pick up wikidiff2

2021-02-17

 * 23:41 Southparkfan: installed security update on bacula2
 * 23:39 Southparkfan: free up space by deleting one db backup
 * 23:02 Southparkfan: second attempt of installing security patches, this time the patches were actually applied
 * 22:50 Southparkfan: installing security patches on jobrunner*
 * 22:49 Southparkfan: installing security patches on mw*
 * 22:45 Southparkfan: install security patches on test2
 * 22:11 paladox: renaming rdb4 to mc2
 * 16:21 paladox: install security updates on all servers
 * 15:37 paladox: rebuild lc on mw* and jobrunner*
 * 15:11 paladox: root@jobrunner3:/home/paladox# /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php
 * 08:44 Reception123: umount/mount mediawiki-static on jobrunner4
 * 08:42 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki kelevarwiki /home/reception/images --search-recursively
 * 08:41 Reception123: rebooted jobrunner4
 * 08:40 Reception123: rebooted jobrunner3
 * 08:22 RhinosF1: stop/start job services again
 * 08:16 RhinosF1: deleted /home/reception/wikibackups/*
 * 08:08 RhinosF1: stop/start jobrunner&jobcron on jobrunner*

2021-02-16

 * 21:44 paladox: varnish> ban req.http.Host == static.miraheze.org && req.url ~ "^/braveninewiki/2/24/Icon-MP-1.png" - cp12
 * 11:49 Reception123: bash ./wikibackups.sh /home/reception/publicwikis.txt /srv/mediawiki/w/maintenance/dumpBackup.php (wikibackups)
 * 11:00 John: restart dovecot on mail2

2021-02-15

 * 23:49 Universal_Omega: universalomega@jobrunner3:/srv/mediawiki/w/extensions/Cargo/maintenance$ sudo -u www-data php setCargoFileData.php --wiki=egoishwiki
 * 23:48 Universal_Omega: universalomega@jobrunner3:/srv/mediawiki/w/extensions/Cargo/maintenance$ sudo -u www-data php setCargoPageData.php --wiki=egoishwiki
 * 21:37 Southparkfan: dump db13 -> dbbackup1.. again
 * 14:39 paladox: mail2: upgrade dovecot-core dovecot-imapd dovecot-ldap
 * 13:52 paladox: puppet3: upgrade postgresql-client-11
 * 13:46 paladox: jobrunner3: /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/RottenLinks/sql/patches/20210215.sql
 * 08:46 Reception123: removed Zppix from graylog
 * 08:29 Reception123: removed zppix from mail lists
 * 08:26 Reception123: removed zppix from matomo
 * 08:26 Reception123: removed zppix from hund.io
 * 02:01 paladox: restart varnish on cp10 as child appears to have crashed
 * 01:57 paladox: restart varnish on cp11 as child appears to have crashed

2021-02-14

 * 22:38 paladox: remove NDKilla email address and forward his steward emails to his personal email
 * 21:55 John: removed NDKilla from GitHub and Matomo
 * 21:51 John: remove ndkilla from sre and sre-infrastructure groups following inactivity removal
 * 20:24 Southparkfan: dump db13 (c4) to dbbackup1
 * 13:41 Southparkfan: revert grafana hack and enable puppet on mon2
 * 12:44 Southparkfan: disable puppet on mon2 for T6849

2021-02-13

 * 07:00 Reception123: changed a user's email address on Phab

2021-02-10

 * 22:26 John: ran the following on eval.php: for ( $id = 16383; $id <= 16586; $id++ ) { $wr = new WikiRequest( $id ); if ( $wr->language == 'en' ) { $wr->tryAutoCreate; } }
 * 22:22 paladox: set diagnostics.brick-sys-log-level and diagnostics.client-sys-log-level to INFO on gluster
 * 22:01 John: manually run AI creation job for two wiki requests + regenerate model file
 * 21:33 Zppix: runjobs.php on jbr3 on metawiki (130+ jobs were queued)
 * 15:05 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki wandavisionwiki --r "Requested" /home/reception/bwmdel.txt
 * 15:05 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki marveloustvshowepisodeswiki --r "Requested" /home/reception/bwmdel.txt
 * 00:09 paladox: removed cloud1 from cluster

2021-02-09

 * 23:55 paladox: shutting down rdb1 and services1 and deleting the vm
 * 23:54 paladox: shutting down gluster1 and ldap1 and deleting the vm
 * 23:54 paladox: shutting down phab1 and mon1 and deleting the vm
 * 23:53 paladox: shutting down jobrunner2, test2 and mw[67] and deleting the vm
 * 23:44 paladox: shutting down puppet2 and deleting the vm
 * 23:43 paladox: shutting down mail1 and deleting the vm
 * 23:42 paladox: shutting down jobrunner1 and mw[45] and deleting the vm
 * 23:41 paladox: shutting down services1 and rdb1 and deleting the vm
 * 23:37 paladox: shutting down graylog1 and deleting the vm
 * 23:20 paladox: shutting down cp6 and deleting the vm
 * 19:48 paladox: cd /srv ; chown -R www-data:www-data mathoid on mw, test3 and jobrunner[34]
 * 18:04 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki voidlinuxwiki /home/reception/Void+Linux+Wiki-20201015234107.xml per request on Phabricator

2021-02-08

 * 22:58 Southparkfan: enable puppet on dbbackup[12] & restart replica on dbbackup2
 * 21:52 Southparkfan: dbbackup[12]: revert to innodb_flush_log_at_trx_commit=1
 * 21:35 Southparkfan: dbbackup[12]: set global innodb_flush_log_at_trx_commit=2;
 * 21:25 Southparkfan: revert innodb_flush_sync hack (set back to 1)
 * 21:18 Southparkfan: dbbackup[12]: set global innodb_flush_sync=0; <- to see if that reduces load on the backup servers
 * 21:17 Southparkfan: disable puppet on dbbackup1
 * 21:13 Southparkfan: disable puppet on dbbackup2
 * 11:49 Reception123: removed spam from the below Phab accounts' profiles
 * 11:48 Reception123: disabled three spam accounts on Phab

2021-02-07

 * 22:19 paladox: root@jobrunner3:/home/paladox# /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /home/paladox/resetWikiCaches.php
 * 21:59 paladox: depool/repool mw10 and rebuild lc
 * 15:58 Southparkfan: started replication flow from db11 -> dbbackup1
 * 15:54 Southparkfan: started replication flow from db12 -> dbbackup2
 * 03:17 Southparkfan: start mariabackup on db11 and db12
 * 01:01 Southparkfan: purchased dbbackup2
 * 00:52 Southparkfan: the same for all streams relevant for MediaWiki Engineers
 * 00:50 Southparkfan: graylog: change stream rules for MediaWiki Error Logs (matches all servers starting with 'test' now)
 * 00:14 Southparkfan: reimage dbbackup1 (downgrade from 4GB MKVM to 2GB MKVM)

2021-02-06

 * 23:43 paladox: upgrade packages on bacula2
 * 22:01 Zppix: run deleteOldRevisions.php on jbr3 for taerelvariwiki per T6829
 * 21:23 paladox: add 3gb of swap to db11/13
 * 18:37 Universal_Omega: $cw = new CreateWikiJson( 'egoishwiki' ); $cw->resetWiki;
 * 18:37 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php wikibasequalityconstraints --wiki=egoishwiki --disable on Jobrunner3
 * 18:23 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/moveBatch.php --wiki testwiki -r "Requested" --noredirects /home/reception/test.txt
 * 18:19 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki taerelvariwiki --r "delete redirects (mistake)" /home/reception/tlvdel.txt
 * 18:09 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/moveBatch.php --wiki taerelvariwiki --r "Requested by @TheBurningPrincess (T6827)" /home/reception/tlvmove.txt
 * 15:00 paladox: actually rebooted jobrunner[34]
 * 15:00 paladox: reboot jobrunner[12]
 * 15:00 paladox: repool mw11
 * 15:00 paladox: depool mw11 and repool mw10
 * 14:57 paladox: depool mw10
 * 14:35 paladox: repool mw8 and depool mw9
 * 14:28 paladox: rebuild lc on mw8
 * 14:27 paladox: depool mw8
 * 06:26 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on Jobrunner[34], mw[891011], and test3
 * 03:13 paladox: rebuild lc on mw*
 * 02:47 paladox: restarted varnish on cp3 as it appears to have gone down
 * 01:08 paladox: rename test2 to test3

2021-02-05

 * 19:11 paladox: restart varnish on cp[67]. Child was not running (Unknown request in manager process (child not running).
 * 19:10 paladox: depool mw[4567] and repool (also reboot)

2021-02-04

 * 21:27 John: ifup vmbr1- @ cloud4 and cloud5
 * 21:27 John: ifdown enp1s0f1 && ifup enp1s0f1 - @ cloud4 and cloud5
 * 21:11 John: ifdown enp1s0f1 && ifup enp1s0f1 - @ cloud4 and cloud5
 * 20:53 John: ifup enp1s0f1 @ cloud4

2021-02-03

 * 17:03 paladox: reboot cp6 as varnish was failing. Logs showed out of space?
 * 16:44 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/generateMirahezeSitemap.php - jobrunner1
 * 15:49 Reception123: reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki metawiki --r "Requested" /home/reception/metadel.txt

2021-02-02

 * 16:06 Universal_Omega: enabled advanced moderation on Github mw-config to block new users with a 24 hours expiration.
 * 10:30 paladox: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /home/paladox/resetWikiCaches.php on jobrunner1
 * 10:30 paladox: MariaDB [mhglobal]> update mw_settings set s_settings = REPLACE(s_settings,'wmgEnableUploads', 'wgEnableUploads');
 * 01:00 Southparkfan: install rsync on gluster2
 * 00:56 paladox: switched on gluster read-only mode
 * 00:53 Southparkfan: disable puppet on glustermigrtemp1
 * 00:37 Universal_Omega: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /home/paladox/resetWikiCaches.php on Jobrunner1
 * 00:34 Universal_Omega: update mw_settings set s_settings = REPLACE(s_settings,'wgEnableUploads', 'wmgEnableUploads');

2021-02-01

 * 22:17 Southparkfan: installing jobrunner3/4
 * 21:43 Southparkfan: install cp10
 * 21:34 Southparkfan: installing cp11
 * 20:17 Southparkfan: decom cp9
 * 20:04 Southparkfan: if you try to set up failover IPs on VMs, run away, not worth the anger
 * 18:57 Reception123: reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki metawiki --r "Requested" /home/reception/meta-del.txt
 * 15:55 paladox: syncing directories starting with b on gluster
 * 01:25 Southparkfan: remove prometheus migration user from mon2
 * 00:54 Southparkfan: migrate mon1 prom datadir to mon2
 * 00:51 Universal_Omega: $cw = new CreateWikiJson( 'xenwiki' ); $cw->resetWiki;
 * 00:50 Universal_Omega: $cw = new CreateWikiJson( 'rwiki' ); $cw->resetWiki;
 * 00:50 Southparkfan: create prommigrtemp user on mon2
 * 00:50 Universal_Omega: $cw = new CreateWikiJson( 'thewaveswiki' ); $cw->resetWiki;
 * 00:48 Universal_Omega: $cw = new CreateWikiJson( 'sylvsfancontestresultswiki' ); $cw->resetWiki;
 * 00:48 Universal_Omega: $cw = new CreateWikiJson( 'polskajestwiki' ); $cw->resetWiki;
 * 00:47 Universal_Omega: $cw = new CreateWikiJson( 'moretypedesignwiki' ); $cw->resetWiki;
 * 00:47 paladox: purge ns2 from old infra
 * 00:46 Universal_Omega: $cw = new CreateWikiJson( 'unicodesubsetswiki' ); $cw->resetWiki;
 * 00:46 Universal_Omega: $cw = new CreateWikiJson( 'bimagvwiki' ); $cw->resetWiki;
 * 00:46 Universal_Omega: $cw = new CreateWikiJson( 'arnabdaswiki' ); $cw->resetWiki;
 * 00:45 Southparkfan: enable puppet on cp12
 * 00:04 Southparkfan: disable puppet on cp12 to test varnishlog

2021-01-31

 * 23:51 John: delete mediawiki-admins LDAP group
 * 23:04 paladox: shutdown ns2 (old) on cloud1 (it was migrated to cloud4)
 * 22:22 paladox: starting rsync for every directory starting with a
 * 22:22 paladox: installed rsync on gluster3
 * 20:29 Reception123: renamed projectchaoswiki to cnvnwiki
 * 20:26 Reception123: renamed jacksucksatlifewiki to jackmasseywelshwiki
 * 20:22 John: icingaui: add 'guest' group with only member being guest account, limit password changes on it
 * 20:21 John: icingaui: convert 'Administrator' access to sre-infrastructure and sre-management (total access to Infra + EMs above)
 * 20:16 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki eskwiki /home/reception/delbackups3/eskwiki.xml
 * 19:29 paladox: upgrade ca-certificates puppet-agent puppetserver tzdata on puppet2
 * 18:43 paladox: root@test2:/srv/mediawiki/w/extensions/MirahezeMagic/maintenance# /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/fixDumpsPermissions.php
 * 02:47 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, ',"74":"","75":"","76":"","77":""', '' ) WHERE perm_group = '*' AND perm_dbname = 'unicodesubsetswiki';
 * 02:45 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"15":"16":"17":"18":"19":"20"', '' ) WHERE perm_group = '*' AND perm_dbname = 'unicodesubsetswiki';
 * 02:42 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"editusercss",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"editusercss",%';
 * 02:42 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"edituserjs",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"edituserjs",%';
 * 02:42 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"edituserjson",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"edituserjson",%';
 * 02:41 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"editsitejs",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"editsitejs",%';
 * 02:41 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"editsitejson",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"editsitejson",%';
 * 02:40 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"editsitecss",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"editsitecss",%';
 * 02:37 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"62":', '' ) WHERE perm_group = '*' AND perm_dbname = 'unicodesubsetswiki';
 * 02:36 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"managewiki",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%managewiki%';

2021-01-30

 * 22:23 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgSlackIncomingWebhookUrl --sourcelist=/home/universalomega/wgSlackIncomingWebhookUrl.txt --wiki=test2wiki
 * 22:20 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgDiscordIncomingWebhookUrl --sourcelist=/home/universalomega/wgDiscordIncomingWebhookUrl.txt --wiki=loginwiki
 * 22:11 Southparkfan: purchased cp12, cancelled cp9 (will end at Feb 1)
 * 15:07 Reception123: added discord webhook for federalistrepublicwiki
 * 14:49 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* and jbr*
 * 14:49 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on mw* and jbr*
 * 14:48 paladox: install maven on test2
 * 14:17 Reception123: reception@jobrunner2:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki srcentswiki /home/reception/Source+Entities-20210130121344.xml
 * 08:37 Reception123: changed tech email from plaintext on @github/miraheze

2021-01-29

 * 06:34 Reception123: added discord webhook for zenithwiki

2021-01-28

 * 20:00 paladox: root@test2:/srv/mediawiki/w/extensions/MirahezeMagic/maintenance# /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/dataDumpPathMove.php

2021-01-24

 * 17:21 paladox: cp9: ban req.http.Host == xiaoshenbike.miraheze.org

2021-01-23

 * 23:03 paladox: rebuild lc on mw*
 * 22:59 paladox: rebuild lc on test2
 * 15:18 paladox: upgrade phabricator

2021-01-22

 * 11:23 Reception123: reception@jobrunner2:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki eamfscontestswiki /home/reception/songscontests_pages_current.xml

2021-01-21

 * 16:35 Reception123: reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki metawiki --r "Requested" /home/reception/meta.txt
 * 16:31 Reception123: MariaDB [bimagvwiki]> INSERT INTO slot_roles (role_id, role_name) VALUES (1, 'main');
 * 16:29 Reception123: MariaDB [bimagvwiki]> INSERT INTO content_models (model_id, model_name) VALUES (                                                                            5, 'json')
 * 16:29 Reception123: MariaDB [bimagvwiki]> INSERT INTO content_models (model_id, model_name) VALUES (                                                                            4, 'Scribunto');
 * 16:29 Reception123: MariaDB [bimagvwiki]> INSERT INTO content_models (model_id, model_name) VALUES                                                                              (3, 'javascript');
 * 16:29 Reception123: MariaDB [bimagvwiki]> INSERT INTO content_models (model_id, model_name) VALUES (                                                                            2, 'css');
 * 16:29 Reception123: MariaDB [bimagvwiki]> INSERT INTO content_models (model_id, model_name) VALUES (                                                                            1, 'wikitext');

2021-01-20

 * 19:23 Universal_Omega: sudo -u www-data rm -rf /srv/mediawiki/w/skins/Cosmos/cache on mw*, jbr*, and test2
 * 19:03 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw*, jbr*, and test2
 * 19:02 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on mw*, jbr*, and test2
 * 19:00 Universal_Omega: UPDATE mw_settings SET s_settings = REPLACE(s_settings,'wgCosmosWikiHeaderWordmark', 'wgCosmosWordmark');
 * 18:59 Universal_Omega: UPDATE mw_settings SET s_settings = REPLACE(s_settings,'wgCosmosFooterColor', 'wgCosmosFooterBackgroundColor');
 * 18:58 Universal_Omega: UPDATE mw_settings SET s_settings = REPLACE(s_settings,'wgCosmosToolbarColor', 'wgCosmosToolbarBackgroundColor');
 * 18:57 Universal_Omega: UPDATE mw_settings SET s_settings = REPLACE(s_settings,'wgCosmosButtonColor', 'wgCosmosButtonBackgroundColor');
 * 14:16 paladox: restart ircrcbot

2021-01-18

 * 22:22 Universal_Omega: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgCosmosUseMessageforToolbar --sourcelist=false --remove on Jobrunner1
 * 22:00 Universal_Omega: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgCosmosBackgroundImageNorepeat --sourcelist=false --remove on Jobrunner1

2021-01-17

 * 18:00 paladox: MariaDB [zeroprgtempwiki]> ALTER TABLE user CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 * 18:00 paladox: MariaDB [zeroprgtempwiki]> ALTER DATABASE zeroprgtempwiki CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
 * 17:45 paladox: upgrade grafana on mon1

2021-01-16

 * 20:49 Reception123: MariaDB [phabricator_user]> UPDATE user_email SET address = '' WHERE address = '';
 * 18:16 RhinosF1: enabled enhanced GitHub moderation
 * 15:35 RhinosF1: both scripts complete (one on each jobrunner, user had 2 accounts)
 * 15:12 RhinosF1: GDPR removal x2
 * 07:18 Reception123: renamed spacepediawiki to ct777wiki.

2021-01-15

 * 19:18 paladox: reboot dbbackup1
 * 18:27 John: experimenting with request limits to produce a model file which can be stored in memory
 * 16:53 paladox: root@dbbackup1:/etc/mysql/conf.d# swapoff -a
 * 16:35 paladox: add ipv6 address to dbbackup1
 * 10:15 Reception123: depooled mw4, rebooted mw4, repooled mw4
 * 10:15 Reception123: sudo service logbot start

2021-01-14

 * 23:41 Southparkfan: dbbackup1's mydumper performance test: dumped all c4 databases (mydumper running with one thread) to .sql.gz (compressed logical backup). Started at 15:24:53, finished at 21:17:28. mariadb data directory is 202G, dump directory is 33G
 * 23:36 Southparkfan: start c4 replication on dbbackup1
 * 20:57 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki thegamerwiki /home/reception/johnthegamer_pages_full.xml

2021-01-13

 * 20:12 Southparkfan: c3 set up on dbbackup1
 * 18:37 Southparkfan: dumping db12 to dbbackup1
 * 11:40 Southparkfan: started c2 replication on dbbackup1
 * 10:24 Reception123: renamed worldfederationdiscordwiki to usfederalgovernmentrpwiki
 * 00:09 Southparkfan: c4's replication back on dbbackup1, dumping c2 now

2021-01-12

 * 22:01 Southparkfan: reclone all clusters on dbbackup1
 * 21:59 RhinosF1: for the record: John added IRC_MIRAHEZEBOTS to GitHub secrets and grouped miraheze-GitHub to @miraheze/bots - change deployed to CI world everywhere I can see
 * 21:38 Southparkfan: stop all mysql instances on dbbackup1 to move relay bin logs
 * 21:35 Southparkfan: c3 and c4 are corrupted on dbbackup1, deleting and recloning those
 * 21:30 Southparkfan: dbbackup1: db11 copied succesfully, started replication
 * 21:30 Southparkfan: dbbackup1: add hard and soft 'nofile' limits in /etc/security/limits.conf
 * 21:17 Southparkfan: set ulimit -Sn 1000000 on dbbackup1
 * 20:46 Southparkfan: dbbackup1: temporarily stop c3 replication to reduce load
 * 20:35 Southparkfan: there was an issue with c3 replication on dbbackup1, fixed with https://dba.stackexchange.com/a/191558
 * 20:15 Southparkfan: dump db11 to dbbackup1
 * 20:12 Southparkfan: db11: increase soft open files limit to 1 million
 * 16:10 paladox: reboot and upgrade kernel on cp3

2021-01-11

 * 22:21 Zppix: rebuildRC and initSiteStats following import on jbr2
 * 21:36 Zppix: zppix@jobrunner2: screen sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki durkomorewiki --report=1 uklurkmoreua_pages_full.xml
 * 00:31 Southparkfan: used command (for reference): mydumper -B allthetropeswiki -G -E -R -v 3 -t 2 -c -S /var/run/mysqld/mysqld.c4.sock
 * 00:30 Southparkfan: results of compressed mydumper run on allthetropeswiki: 33G (database in /srv/mariadb.c4) -> 12G of sql.gz files
 * 00:04 Southparkfan: perform a mydumper test on dbbackup1

2021-01-10

 * 20:11 paladox: rebuild lc on test2
 * 20:11 paladox: rebuild lc on mw[4567] and jobrunner[12]
 * 18:44 paladox: delete db7 on cloud2
 * 17:23 Southparkfan: dbbackup1: start db12 & db13 replication :)
 * 17:08 Southparkfan: prepare db13 backup and stop db12 replication temporarily (performance)
 * 15:43 Southparkfan: dumping db13 to dbbackup1
 * 15:40 Southparkfan: increase soft open files limit to 16384 on db13
 * 15:09 Southparkfan: db12 imported to dbbackup1
 * 14:02 Southparkfan: imported replica grants on db12
 * 13:44 Southparkfan: try a second mariabackup --prepare on the db12 backup
 * 13:42 Southparkfan: ulimit -a 950000 on dbbackup1
 * 03:03 Universal_Omega: universalomega@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=testwiki

2021-01-09

 * 22:46 Universal_Omega: sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/wikiforum.json /srv/mediawiki/w/maintenance/sql.php /home/universalomega/drop-wft_closed-from-wikiforum_threads.sql
 * 20:06 Southparkfan: restart db13 dump (crashed due to: xtrabackup_copy_logfile failed: redo log block is overwritten, please increase redo log size with innodb_log_file_size parameter.)
 * 19:31 paladox: root@dbbackup1:/home/paladox# chmod 1777 /tmp
 * 19:26 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki opencircuitswiki /home/reception/opencircuitscom-20200218-history.xml
 * 18:54 Southparkfan: running mariabackup on db12 & db13 to dump all databases to dbbackup1
 * 18:46 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw7
 * 18:14 Universal_Omega: sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/wikiforum.json /home/universalomega/migrateOldWikiForumClosedTimestampColumnToNew.php
 * 18:06 Universal_Omega: sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/wikiforum.json /srv/mediawiki/w/maintenance/sql.php /home/universalomega/add-wft_closed_timestamp-to-wikiforum_threads.sql
 * 17:25 Southparkfan: repool mw6
 * 16:48 paladox: rebuild lc on test2 and jobrunner[12]
 * 16:44 paladox: depool and repool mw[4567] and rebuild lc
 * 14:27 Southparkfan: dbbackup1: mysql_install_db --user=mysql --datadir=/srv/mariadb.c3
 * 14:24 Southparkfan: set 4755 manually on /usr/lib/mysql/plugin/auth_pam_tool_dir/auth_pam_tool
 * 11:54 Southparkfan: purchase dbbackup1.miraheze.org; 4GB MKVM in Atlanta
 * 06:42 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* and jbr*
 * 06:41 Reception123: sudo -u www-data php /srv/mediawiki/w/maint*/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on mw* and jbr*

2021-01-08

 * 23:38 Universal_Omega: universalomega@jobrunner1:~$ sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/wikiforum.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php wikiforum
 * 23:17 Universal_Omega: UPDATE wikiforum_category SET wfc_added_user_ip = '0.0.0.0' WHERE wfc_added_user_ip = 'REDACTED'; on aboutpcswiki
 * 23:16 Universal_Omega: UPDATE wikiforum_forums SET wff_added_user_ip = '0.0.0.0' WHERE wff_added_user_ip = 'REDACTED'; on aboutpcswiki
 * 23:15 Universal_Omega: UPDATE wikiforum_forums SET wff_last_post_user_ip = '0.0.0.0' WHERE wff_last_post_user_ip = 'REDACTED'; on aboutpcswiki
 * 23:14 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on aboutpcswiki
 * 23:10 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on bonziworldwiki
 * 23:10 Universal_Omega: UPDATE wikiforum_forums SET wff_last_post_user_ip = '0.0.0.0' WHERE wff_last_post_user_ip = 'REDACTED'; on bonziworldwiki
 * 23:08 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on cubashpediawiki
 * 23:07 Universal_Omega: UPDATE wikiforum_forums SET wff_last_post_user_ip = '0.0.0.0' WHERE wff_last_post_user_ip = 'REDACTED'; on cubashpediawiki
 * 23:03 Universal_Omega: UPDATE wikiforum_threads SET wft_closed_user_ip = '0.0.0.0' WHERE wft_closed_user_ip = 'REDACTED'; on programmingwikiwiki
 * 23:03 Universal_Omega: UPDATE wikiforum_threads SET wft_closed_user_ip = '0.0.0.0' WHERE wft_closed_user_ip = 'REDACTED'; on programmingwikiwiki
 * 23:00 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED' on test2wiki;
 * 22:58 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED' on test2wiki;
 * 22:57 Universal_Omega: UPDATE wikiforum_forums SET wff_added_user_ip = '0.0.0.0' WHERE wff_added_user_ip = 'REDACTED' on test2wiki;
 * 22:56 Universal_Omega: UPDATE wikiforum_category SET wfc_added_user_ip = '0.0.0.0' WHERE wfc_added_user_ip = 'REDACTED'; on test2wiki;
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:45 Universal_Omega: UPDATE wikiforum_threads SET wft_edit_user_ip = '0.0.0.0' WHERE wft_edit_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:45 Universal_Omega: UPDATE wikiforum_threads SET wft_edit_user_ip = '0.0.0.0' WHERE wft_edit_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:45 Universal_Omega: UPDATE wikiforum_threads SET wft_edit_user_ip = '0.0.0.0' WHERE wft_edit_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:32 Universal_Omega: UPDATE wikiforum_forums SET wff_added_user_ip = '0.0.0.0' WHERE wff_added_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:32 Universal_Omega: UPDATE wikiforum_forums SET wff_added_user_ip = '0.0.0.0' WHERE wff_added_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:29 Universal_Omega: UPDATE wikiforum_forums SET wff_last_post_user_ip = '0.0.0.0' WHERE wff_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:29 Universal_Omega: UPDATE wikiforum_forums SET wff_last_post_user_ip = '0.0.0.0' WHERE wff_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:21 Universal_Omega: UPDATE wikiforum_category SET wfc_added_user_ip = '0.0.0.0' WHERE wfc_added_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:21 Universal_Omega: UPDATE wikiforum_category SET wfc_added_user_ip = '0.0.0.0' WHERE wfc_added_user_ip = 'REDACTED'; on snapwikiwiki
 * 20:33 Southparkfan: deleted job id 165 (BackupStatic1) as well but does not seem to get bacula running
 * 20:30 Southparkfan: stop bacula jobs for static1/2

2021-01-07

 * 21:57 John: added conduct alias to LDAP

2021-01-06

 * 23:59 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw*, jbr*, and test2
 * 23:57 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maint*/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on mw*, jbr*, and test2
 * 00:06 Universal_Omega: UPDATE localuser SET lu_attached_method = 'login' WHERE lu_name="Universal Omega" AND lu_attached_method = 'new'; and UPDATE localuser SET lu_attached_method = 'new' WHERE lu_name="Universal Omega" AND lu_wiki = 'metawiki'; (to match with gu_home_db)

2021-01-05

 * 22:48 Universal_Omega: universalomega@jobrunner1:~$ php /home/universalomega/populateWikiSettings.php --wgsetting=wmgWhitelistRead --sourcelist=/home/universalomega/wgWhitelistRead.txt --wiki=loginwiki (using modified version of populateWikiSettings.php to support this specific type of setting)
 * 01:49 Zppix: initSiteStats.php for chakuwikiwiki (both rebuildRC and this on JBR2)
 * 01:45 Zppix: rebuildRC for chakuwikiwiki

2021-01-04

 * 14:43 paladox: root@jobrunner1:/srv/mediawiki/w/extensions/ManageWiki/maintenance# sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --disable wikiforum
 * 10:46 RhinosF1: stop nginx&puppet on test2 until config .git/objects can be fixed to avoid X-Debug being used to enable WikiForum
 * 10:36 RhinosF1: globally kill wikiforum

2021-01-03

 * 20:26 Reception123: added windowswiki discord webhook
 * 15:45 Southparkfan: restarted adminlogbot
 * 15:45 Southparkfan: enabled a personal cronjob on cp7 to measure mediawiki response time

2021-01-02

 * 20:28 Zppix: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --skip-to=19201 --report=1 --wiki chakuwikiwiki /home/reception/chakuwiki-full.xml --namespaces="0|1|4|5|6|7|8|9|10|11|12|13|14|15" --uploads --no-local-users (restart import) on jbr2
 * 18:49 paladox: increase mon1 disk by 10gb
 * 14:43 paladox: MariaDB [(none)]> PURGE BINARY LOGS BEFORE '2021-01-03 00:00:00'; - db12
 * 11:56 Reception123: ran GDPR script
 * 07:32 Reception123: reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki minecraftathomewiki --r "Requested" /home/reception/minecraftathome.txt

2021-01-01

 * 23:49 Universal_Omega: UPDATE mw_settings set s_settings = JSON_REMOVE(s_settings, '$.wgAccountCreationThrottle');
 * 23:49 Universal_Omega: UPDATE mw_settings set s_settings = JSON_REMOVE(s_settings, '$.wgModerationNotificationNewOnly');
 * 23:48 Universal_Omega: UPDATE mw_settings set s_settings = JSON_REMOVE(s_settings, '$.wgModerationNotificationEnable');
 * 23:48 Universal_Omega: UPDATE mw_settings set s_settings = JSON_REMOVE(s_settings, '$.wgModerationEmail');
 * 20:40 Southparkfan: enable puppet on mail1
 * 20:27 Southparkfan: enabled puppet on test2 & disable puppet on mail1
 * 20:12 Universal_Omega: universalomega@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateGroupPermissionsWithDefaults.php --wiki burnoutwiki --overwrite
 * 19:58 Southparkfan: restart phd on phab1
 * 19:21 Southparkfan: disable puppet on test2 for T5044 experimenting
 * 19:21 Southparkfan: db11 (c1) done
 * 19:13 Southparkfan: db11 (c1) done; moving to db11 (c2)
 * 19:12 Southparkfan: db13 already done; moving to db11 (c1)
 * 19:11 Southparkfan: applied T6095 patches to db12, applying to db13 now
 * 18:34 paladox: MariaDB [(none)]> PURGE BINARY LOGS BEFORE '2021-01-02 00:00:00'; - db12
 * 18:32 paladox: MariaDB [(none)]> PURGE BINARY LOGS BEFORE DATE(NOW) + INTERVAL 0 SECOND; - db12
 * 18:06 Southparkfan: applying database patches for T6095 to db12
 * 03:24 Zppix: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --report=1 --wiki chakuwikiwiki /home/reception/chakuwiki-full.xml --namespaces="0|1|4|5|6|7|8|9|10|11|12|13|14|15" --uploads --no-local-users on jobrunner2 (on behalf of reception)
 * 02:41 paladox: add shekhwiki to matomo
 * 02:38 paladox: re add encycmhwiki to matomo
 * 01:50 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --update --wiki izunawiki
 * 01:48 Universal_Omega: universalomega@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki izunawiki --report=1 /home/universalomega/izuna_gamepedia_en_pages_full.xml
 * 01:46 paladox: upgrade grafana, icinga2 and tzdata on mon1
 * 01:36 paladox: drop piwik on db13 after moving it to db12
 * 00:10 Universal_Omega: UPDATE mw_settings SET s_settings = JSON_REPLACE(s_settings, '$.wmgDefaultRobotPolicy', JSON_REMOVE(s_settings, '$.wmgDefaultRobotPolicy');