Tech:Server admin log/2021

2021-02-17

 * 08:40 Reception123: rebooted jobrunner3
 * 08:22 RhinosF1: stop/start job services again
 * 08:16 RhinosF1: deleted /home/reception/wikibackups/*
 * 08:08 RhinosF1: stop/start jobrunner&jobcron on jobrunner*

2021-02-16

 * 21:44 paladox: varnish> ban req.http.Host == static.miraheze.org && req.url ~ "^/braveninewiki/2/24/Icon-MP-1.png" - cp12
 * 11:49 Reception123: bash ./wikibackups.sh /home/reception/publicwikis.txt /srv/mediawiki/w/maintenance/dumpBackup.php (wikibackups)
 * 11:00 John: restart dovecot on mail2

2021-02-15

 * 23:49 Universal_Omega: universalomega@jobrunner3:/srv/mediawiki/w/extensions/Cargo/maintenance$ sudo -u www-data php setCargoFileData.php --wiki=egoishwiki
 * 23:48 Universal_Omega: universalomega@jobrunner3:/srv/mediawiki/w/extensions/Cargo/maintenance$ sudo -u www-data php setCargoPageData.php --wiki=egoishwiki
 * 21:37 Southparkfan: dump db13 -> dbbackup1.. again
 * 14:39 paladox: mail2: upgrade dovecot-core dovecot-imapd dovecot-ldap
 * 13:52 paladox: puppet3: upgrade postgresql-client-11
 * 13:46 paladox: jobrunner3: /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/RottenLinks/sql/patches/20210215.sql
 * 08:46 Reception123: removed Zppix from graylog
 * 08:29 Reception123: removed zppix from mail lists
 * 08:26 Reception123: removed zppix from matomo
 * 08:26 Reception123: removed zppix from hund.io
 * 02:01 paladox: restart varnish on cp10 as child appears to have crashed
 * 01:57 paladox: restart varnish on cp11 as child appears to have crashed

2021-02-14

 * 22:38 paladox: remove NDKilla email address and forward his steward emails to his personal email
 * 21:55 John: removed NDKilla from GitHub and Matomo
 * 21:51 John: remove ndkilla from sre and sre-infrastructure groups following inactivity removal
 * 20:24 Southparkfan: dump db13 (c4) to dbbackup1
 * 13:41 Southparkfan: revert grafana hack and enable puppet on mon2
 * 12:44 Southparkfan: disable puppet on mon2 for T6849

2021-02-13

 * 07:00 Reception123: changed a user's email address on Phab

2021-02-10

 * 22:26 John: ran the following on eval.php: for ( $id = 16383; $id <= 16586; $id++ ) { $wr = new WikiRequest( $id ); if ( $wr->language == 'en' ) { $wr->tryAutoCreate; } }
 * 22:22 paladox: set diagnostics.brick-sys-log-level and diagnostics.client-sys-log-level to INFO on gluster
 * 22:01 John: manually run AI creation job for two wiki requests + regenerate model file
 * 21:33 Zppix: runjobs.php on jbr3 on metawiki (130+ jobs were queued)
 * 15:05 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki wandavisionwiki --r "Requested" /home/reception/bwmdel.txt
 * 15:05 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki marveloustvshowepisodeswiki --r "Requested" /home/reception/bwmdel.txt
 * 00:09 paladox: removed cloud1 from cluster

2021-02-09

 * 23:55 paladox: shutting down rdb1 and services1 and deleting the vm
 * 23:54 paladox: shutting down gluster1 and ldap1 and deleting the vm
 * 23:54 paladox: shutting down phab1 and mon1 and deleting the vm
 * 23:53 paladox: shutting down jobrunner2, test2 and mw[67] and deleting the vm
 * 23:44 paladox: shutting down puppet2 and deleting the vm
 * 23:43 paladox: shutting down mail1 and deleting the vm
 * 23:42 paladox: shutting down jobrunner1 and mw[45] and deleting the vm
 * 23:41 paladox: shutting down services1 and rdb1 and deleting the vm
 * 23:37 paladox: shutting down graylog1 and deleting the vm
 * 23:20 paladox: shutting down cp6 and deleting the vm
 * 19:48 paladox: cd /srv ; chown -R www-data:www-data mathoid on mw, test3 and jobrunner[34]
 * 18:04 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki voidlinuxwiki /home/reception/Void+Linux+Wiki-20201015234107.xml per request on Phabricator

2021-02-08

 * 22:58 Southparkfan: enable puppet on dbbackup[12] & restart replica on dbbackup2
 * 21:52 Southparkfan: dbbackup[12]: revert to innodb_flush_log_at_trx_commit=1
 * 21:35 Southparkfan: dbbackup[12]: set global innodb_flush_log_at_trx_commit=2;
 * 21:25 Southparkfan: revert innodb_flush_sync hack (set back to 1)
 * 21:18 Southparkfan: dbbackup[12]: set global innodb_flush_sync=0; <- to see if that reduces load on the backup servers
 * 21:17 Southparkfan: disable puppet on dbbackup1
 * 21:13 Southparkfan: disable puppet on dbbackup2
 * 11:49 Reception123: removed spam from the below Phab accounts' profiles
 * 11:48 Reception123: disabled three spam accounts on Phab

2021-02-07

 * 22:19 paladox: root@jobrunner3:/home/paladox# /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /home/paladox/resetWikiCaches.php
 * 21:59 paladox: depool/repool mw10 and rebuild lc
 * 15:58 Southparkfan: started replication flow from db11 -> dbbackup1
 * 15:54 Southparkfan: started replication flow from db12 -> dbbackup2
 * 03:17 Southparkfan: start mariabackup on db11 and db12
 * 01:01 Southparkfan: purchased dbbackup2
 * 00:52 Southparkfan: the same for all streams relevant for MediaWiki Engineers
 * 00:50 Southparkfan: graylog: change stream rules for MediaWiki Error Logs (matches all servers starting with 'test' now)
 * 00:14 Southparkfan: reimage dbbackup1 (downgrade from 4GB MKVM to 2GB MKVM)

2021-02-06

 * 23:43 paladox: upgrade packages on bacula2
 * 22:01 Zppix: run deleteOldRevisions.php on jbr3 for taerelvariwiki per T6829
 * 21:23 paladox: add 3gb of swap to db11/13
 * 18:37 Universal_Omega: $cw = new CreateWikiJson( 'egoishwiki' ); $cw->resetWiki;
 * 18:37 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php wikibasequalityconstraints --wiki=egoishwiki --disable on Jobrunner3
 * 18:23 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/moveBatch.php --wiki testwiki -r "Requested" --noredirects /home/reception/test.txt
 * 18:19 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki taerelvariwiki --r "delete redirects (mistake)" /home/reception/tlvdel.txt
 * 18:09 Reception123: reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/moveBatch.php --wiki taerelvariwiki --r "Requested by @TheBurningPrincess (T6827)" /home/reception/tlvmove.txt
 * 15:00 paladox: actually rebooted jobrunner[34]
 * 15:00 paladox: reboot jobrunner[12]
 * 15:00 paladox: repool mw11
 * 15:00 paladox: depool mw11 and repool mw10
 * 14:57 paladox: depool mw10
 * 14:35 paladox: repool mw8 and depool mw9
 * 14:28 paladox: rebuild lc on mw8
 * 14:27 paladox: depool mw8
 * 06:26 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on Jobrunner[34], mw[891011], and test3
 * 03:13 paladox: rebuild lc on mw*
 * 02:47 paladox: restarted varnish on cp3 as it appears to have gone down
 * 01:08 paladox: rename test2 to test3

2021-02-05

 * 19:11 paladox: restart varnish on cp[67]. Child was not running (Unknown request in manager process (child not running).
 * 19:10 paladox: depool mw[4567] and repool (also reboot)

2021-02-04

 * 21:27 John: ifup vmbr1- @ cloud4 and cloud5
 * 21:27 John: ifdown enp1s0f1 && ifup enp1s0f1 - @ cloud4 and cloud5
 * 21:11 John: ifdown enp1s0f1 && ifup enp1s0f1 - @ cloud4 and cloud5
 * 20:53 John: ifup enp1s0f1 @ cloud4

2021-02-03

 * 17:03 paladox: reboot cp6 as varnish was failing. Logs showed out of space?
 * 16:44 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/generateMirahezeSitemap.php - jobrunner1
 * 15:49 Reception123: reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki metawiki --r "Requested" /home/reception/metadel.txt

2021-02-02

 * 16:06 Universal_Omega: enabled advanced moderation on Github mw-config to block new users with a 24 hours expiration.
 * 10:30 paladox: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /home/paladox/resetWikiCaches.php on jobrunner1
 * 10:30 paladox: MariaDB [mhglobal]> update mw_settings set s_settings = REPLACE(s_settings,'wmgEnableUploads', 'wgEnableUploads');
 * 01:00 Southparkfan: install rsync on gluster2
 * 00:56 paladox: switched on gluster read-only mode
 * 00:53 Southparkfan: disable puppet on glustermigrtemp1
 * 00:37 Universal_Omega: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /home/paladox/resetWikiCaches.php on Jobrunner1
 * 00:34 Universal_Omega: update mw_settings set s_settings = REPLACE(s_settings,'wgEnableUploads', 'wmgEnableUploads');

2021-02-01

 * 22:17 Southparkfan: installing jobrunner3/4
 * 21:43 Southparkfan: install cp10
 * 21:34 Southparkfan: installing cp11
 * 20:17 Southparkfan: decom cp9
 * 20:04 Southparkfan: if you try to set up failover IPs on VMs, run away, not worth the anger
 * 18:57 Reception123: reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki metawiki --r "Requested" /home/reception/meta-del.txt
 * 15:55 paladox: syncing directories starting with b on gluster
 * 01:25 Southparkfan: remove prometheus migration user from mon2
 * 00:54 Southparkfan: migrate mon1 prom datadir to mon2
 * 00:51 Universal_Omega: $cw = new CreateWikiJson( 'xenwiki' ); $cw->resetWiki;
 * 00:50 Universal_Omega: $cw = new CreateWikiJson( 'rwiki' ); $cw->resetWiki;
 * 00:50 Southparkfan: create prommigrtemp user on mon2
 * 00:50 Universal_Omega: $cw = new CreateWikiJson( 'thewaveswiki' ); $cw->resetWiki;
 * 00:48 Universal_Omega: $cw = new CreateWikiJson( 'sylvsfancontestresultswiki' ); $cw->resetWiki;
 * 00:48 Universal_Omega: $cw = new CreateWikiJson( 'polskajestwiki' ); $cw->resetWiki;
 * 00:47 Universal_Omega: $cw = new CreateWikiJson( 'moretypedesignwiki' ); $cw->resetWiki;
 * 00:47 paladox: purge ns2 from old infra
 * 00:46 Universal_Omega: $cw = new CreateWikiJson( 'unicodesubsetswiki' ); $cw->resetWiki;
 * 00:46 Universal_Omega: $cw = new CreateWikiJson( 'bimagvwiki' ); $cw->resetWiki;
 * 00:46 Universal_Omega: $cw = new CreateWikiJson( 'arnabdaswiki' ); $cw->resetWiki;
 * 00:45 Southparkfan: enable puppet on cp12
 * 00:04 Southparkfan: disable puppet on cp12 to test varnishlog

2021-01-31

 * 23:51 John: delete mediawiki-admins LDAP group
 * 23:04 paladox: shutdown ns2 (old) on cloud1 (it was migrated to cloud4)
 * 22:22 paladox: starting rsync for every directory starting with a
 * 22:22 paladox: installed rsync on gluster3
 * 20:29 Reception123: renamed projectchaoswiki to cnvnwiki
 * 20:26 Reception123: renamed jacksucksatlifewiki to jackmasseywelshwiki
 * 20:22 John: icingaui: add 'guest' group with only member being guest account, limit password changes on it
 * 20:21 John: icingaui: convert 'Administrator' access to sre-infrastructure and sre-management (total access to Infra + EMs above)
 * 20:16 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki eskwiki /home/reception/delbackups3/eskwiki.xml
 * 19:29 paladox: upgrade ca-certificates puppet-agent puppetserver tzdata on puppet2
 * 18:43 paladox: root@test2:/srv/mediawiki/w/extensions/MirahezeMagic/maintenance# /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/fixDumpsPermissions.php
 * 02:47 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, ',"74":"","75":"","76":"","77":""', '' ) WHERE perm_group = '*' AND perm_dbname = 'unicodesubsetswiki';
 * 02:45 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"15":"16":"17":"18":"19":"20"', '' ) WHERE perm_group = '*' AND perm_dbname = 'unicodesubsetswiki';
 * 02:42 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"editusercss",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"editusercss",%';
 * 02:42 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"edituserjs",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"edituserjs",%';
 * 02:42 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"edituserjson",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"edituserjson",%';
 * 02:41 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"editsitejs",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"editsitejs",%';
 * 02:41 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"editsitejson",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"editsitejson",%';
 * 02:40 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"editsitecss",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"editsitecss",%';
 * 02:37 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"62":', '' ) WHERE perm_group = '*' AND perm_dbname = 'unicodesubsetswiki';
 * 02:36 Universal_Omega: UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"managewiki",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%managewiki%';

2021-01-30

 * 22:23 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgSlackIncomingWebhookUrl --sourcelist=/home/universalomega/wgSlackIncomingWebhookUrl.txt --wiki=test2wiki
 * 22:20 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgDiscordIncomingWebhookUrl --sourcelist=/home/universalomega/wgDiscordIncomingWebhookUrl.txt --wiki=loginwiki
 * 22:11 Southparkfan: purchased cp12, cancelled cp9 (will end at Feb 1)
 * 15:07 Reception123: added discord webhook for federalistrepublicwiki
 * 14:49 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* and jbr*
 * 14:49 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on mw* and jbr*
 * 14:48 paladox: install maven on test2
 * 14:17 Reception123: reception@jobrunner2:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki srcentswiki /home/reception/Source+Entities-20210130121344.xml
 * 08:37 Reception123: changed tech email from plaintext on @github/miraheze

2021-01-29

 * 06:34 Reception123: added discord webhook for zenithwiki

2021-01-28

 * 20:00 paladox: root@test2:/srv/mediawiki/w/extensions/MirahezeMagic/maintenance# /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/dataDumpPathMove.php

2021-01-24

 * 17:21 paladox: cp9: ban req.http.Host == xiaoshenbike.miraheze.org

2021-01-23

 * 23:03 paladox: rebuild lc on mw*
 * 22:59 paladox: rebuild lc on test2
 * 15:18 paladox: upgrade phabricator

2021-01-22

 * 11:23 Reception123: reception@jobrunner2:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki eamfscontestswiki /home/reception/songscontests_pages_current.xml

2021-01-21

 * 16:35 Reception123: reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki metawiki --r "Requested" /home/reception/meta.txt
 * 16:31 Reception123: MariaDB [bimagvwiki]> INSERT INTO slot_roles (role_id, role_name) VALUES (1, 'main');
 * 16:29 Reception123: MariaDB [bimagvwiki]> INSERT INTO content_models (model_id, model_name) VALUES (                                                                            5, 'json')
 * 16:29 Reception123: MariaDB [bimagvwiki]> INSERT INTO content_models (model_id, model_name) VALUES (                                                                            4, 'Scribunto');
 * 16:29 Reception123: MariaDB [bimagvwiki]> INSERT INTO content_models (model_id, model_name) VALUES                                                                              (3, 'javascript');
 * 16:29 Reception123: MariaDB [bimagvwiki]> INSERT INTO content_models (model_id, model_name) VALUES (                                                                            2, 'css');
 * 16:29 Reception123: MariaDB [bimagvwiki]> INSERT INTO content_models (model_id, model_name) VALUES (                                                                            1, 'wikitext');

2021-01-20

 * 19:23 Universal_Omega: sudo -u www-data rm -rf /srv/mediawiki/w/skins/Cosmos/cache on mw*, jbr*, and test2
 * 19:03 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw*, jbr*, and test2
 * 19:02 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on mw*, jbr*, and test2
 * 19:00 Universal_Omega: UPDATE mw_settings SET s_settings = REPLACE(s_settings,'wgCosmosWikiHeaderWordmark', 'wgCosmosWordmark');
 * 18:59 Universal_Omega: UPDATE mw_settings SET s_settings = REPLACE(s_settings,'wgCosmosFooterColor', 'wgCosmosFooterBackgroundColor');
 * 18:58 Universal_Omega: UPDATE mw_settings SET s_settings = REPLACE(s_settings,'wgCosmosToolbarColor', 'wgCosmosToolbarBackgroundColor');
 * 18:57 Universal_Omega: UPDATE mw_settings SET s_settings = REPLACE(s_settings,'wgCosmosButtonColor', 'wgCosmosButtonBackgroundColor');
 * 14:16 paladox: restart ircrcbot

2021-01-18

 * 22:22 Universal_Omega: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgCosmosUseMessageforToolbar --sourcelist=false --remove on Jobrunner1
 * 22:00 Universal_Omega: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgCosmosBackgroundImageNorepeat --sourcelist=false --remove on Jobrunner1

2021-01-17

 * 18:00 paladox: MariaDB [zeroprgtempwiki]> ALTER TABLE user CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 * 18:00 paladox: MariaDB [zeroprgtempwiki]> ALTER DATABASE zeroprgtempwiki CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
 * 17:45 paladox: upgrade grafana on mon1

2021-01-16

 * 20:49 Reception123: MariaDB [phabricator_user]> UPDATE user_email SET address = '' WHERE address = '';
 * 18:16 RhinosF1: enabled enhanced GitHub moderation
 * 15:35 RhinosF1: both scripts complete (one on each jobrunner, user had 2 accounts)
 * 15:12 RhinosF1: GDPR removal x2
 * 07:18 Reception123: renamed spacepediawiki to ct777wiki.

2021-01-15

 * 19:18 paladox: reboot dbbackup1
 * 18:27 John: experimenting with request limits to produce a model file which can be stored in memory
 * 16:53 paladox: root@dbbackup1:/etc/mysql/conf.d# swapoff -a
 * 16:35 paladox: add ipv6 address to dbbackup1
 * 10:15 Reception123: depooled mw4, rebooted mw4, repooled mw4
 * 10:15 Reception123: sudo service logbot start

2021-01-14

 * 23:41 Southparkfan: dbbackup1's mydumper performance test: dumped all c4 databases (mydumper running with one thread) to .sql.gz (compressed logical backup). Started at 15:24:53, finished at 21:17:28. mariadb data directory is 202G, dump directory is 33G
 * 23:36 Southparkfan: start c4 replication on dbbackup1
 * 20:57 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki thegamerwiki /home/reception/johnthegamer_pages_full.xml

2021-01-13

 * 20:12 Southparkfan: c3 set up on dbbackup1
 * 18:37 Southparkfan: dumping db12 to dbbackup1
 * 11:40 Southparkfan: started c2 replication on dbbackup1
 * 10:24 Reception123: renamed worldfederationdiscordwiki to usfederalgovernmentrpwiki
 * 00:09 Southparkfan: c4's replication back on dbbackup1, dumping c2 now

2021-01-12

 * 22:01 Southparkfan: reclone all clusters on dbbackup1
 * 21:59 RhinosF1: for the record: John added IRC_MIRAHEZEBOTS to GitHub secrets and grouped miraheze-GitHub to @miraheze/bots - change deployed to CI world everywhere I can see
 * 21:38 Southparkfan: stop all mysql instances on dbbackup1 to move relay bin logs
 * 21:35 Southparkfan: c3 and c4 are corrupted on dbbackup1, deleting and recloning those
 * 21:30 Southparkfan: dbbackup1: db11 copied succesfully, started replication
 * 21:30 Southparkfan: dbbackup1: add hard and soft 'nofile' limits in /etc/security/limits.conf
 * 21:17 Southparkfan: set ulimit -Sn 1000000 on dbbackup1
 * 20:46 Southparkfan: dbbackup1: temporarily stop c3 replication to reduce load
 * 20:35 Southparkfan: there was an issue with c3 replication on dbbackup1, fixed with https://dba.stackexchange.com/a/191558
 * 20:15 Southparkfan: dump db11 to dbbackup1
 * 20:12 Southparkfan: db11: increase soft open files limit to 1 million
 * 16:10 paladox: reboot and upgrade kernel on cp3

2021-01-11

 * 22:21 Zppix: rebuildRC and initSiteStats following import on jbr2
 * 21:36 Zppix: zppix@jobrunner2: screen sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki durkomorewiki --report=1 uklurkmoreua_pages_full.xml
 * 00:31 Southparkfan: used command (for reference): mydumper -B allthetropeswiki -G -E -R -v 3 -t 2 -c -S /var/run/mysqld/mysqld.c4.sock
 * 00:30 Southparkfan: results of compressed mydumper run on allthetropeswiki: 33G (database in /srv/mariadb.c4) -> 12G of sql.gz files
 * 00:04 Southparkfan: perform a mydumper test on dbbackup1

2021-01-10

 * 20:11 paladox: rebuild lc on test2
 * 20:11 paladox: rebuild lc on mw[4567] and jobrunner[12]
 * 18:44 paladox: delete db7 on cloud2
 * 17:23 Southparkfan: dbbackup1: start db12 & db13 replication :)
 * 17:08 Southparkfan: prepare db13 backup and stop db12 replication temporarily (performance)
 * 15:43 Southparkfan: dumping db13 to dbbackup1
 * 15:40 Southparkfan: increase soft open files limit to 16384 on db13
 * 15:09 Southparkfan: db12 imported to dbbackup1
 * 14:02 Southparkfan: imported replica grants on db12
 * 13:44 Southparkfan: try a second mariabackup --prepare on the db12 backup
 * 13:42 Southparkfan: ulimit -a 950000 on dbbackup1
 * 03:03 Universal_Omega: universalomega@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=testwiki

2021-01-09

 * 22:46 Universal_Omega: sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/wikiforum.json /srv/mediawiki/w/maintenance/sql.php /home/universalomega/drop-wft_closed-from-wikiforum_threads.sql
 * 20:06 Southparkfan: restart db13 dump (crashed due to: xtrabackup_copy_logfile failed: redo log block is overwritten, please increase redo log size with innodb_log_file_size parameter.)
 * 19:31 paladox: root@dbbackup1:/home/paladox# chmod 1777 /tmp
 * 19:26 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki opencircuitswiki /home/reception/opencircuitscom-20200218-history.xml
 * 18:54 Southparkfan: running mariabackup on db12 & db13 to dump all databases to dbbackup1
 * 18:46 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw7
 * 18:14 Universal_Omega: sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/wikiforum.json /home/universalomega/migrateOldWikiForumClosedTimestampColumnToNew.php
 * 18:06 Universal_Omega: sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/wikiforum.json /srv/mediawiki/w/maintenance/sql.php /home/universalomega/add-wft_closed_timestamp-to-wikiforum_threads.sql
 * 17:25 Southparkfan: repool mw6
 * 16:48 paladox: rebuild lc on test2 and jobrunner[12]
 * 16:44 paladox: depool and repool mw[4567] and rebuild lc
 * 14:27 Southparkfan: dbbackup1: mysql_install_db --user=mysql --datadir=/srv/mariadb.c3
 * 14:24 Southparkfan: set 4755 manually on /usr/lib/mysql/plugin/auth_pam_tool_dir/auth_pam_tool
 * 11:54 Southparkfan: purchase dbbackup1.miraheze.org; 4GB MKVM in Atlanta
 * 06:42 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* and jbr*
 * 06:41 Reception123: sudo -u www-data php /srv/mediawiki/w/maint*/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on mw* and jbr*

2021-01-08

 * 23:38 Universal_Omega: universalomega@jobrunner1:~$ sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/wikiforum.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php wikiforum
 * 23:17 Universal_Omega: UPDATE wikiforum_category SET wfc_added_user_ip = '0.0.0.0' WHERE wfc_added_user_ip = 'REDACTED'; on aboutpcswiki
 * 23:16 Universal_Omega: UPDATE wikiforum_forums SET wff_added_user_ip = '0.0.0.0' WHERE wff_added_user_ip = 'REDACTED'; on aboutpcswiki
 * 23:15 Universal_Omega: UPDATE wikiforum_forums SET wff_last_post_user_ip = '0.0.0.0' WHERE wff_last_post_user_ip = 'REDACTED'; on aboutpcswiki
 * 23:14 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on aboutpcswiki
 * 23:10 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on bonziworldwiki
 * 23:10 Universal_Omega: UPDATE wikiforum_forums SET wff_last_post_user_ip = '0.0.0.0' WHERE wff_last_post_user_ip = 'REDACTED'; on bonziworldwiki
 * 23:08 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on cubashpediawiki
 * 23:07 Universal_Omega: UPDATE wikiforum_forums SET wff_last_post_user_ip = '0.0.0.0' WHERE wff_last_post_user_ip = 'REDACTED'; on cubashpediawiki
 * 23:03 Universal_Omega: UPDATE wikiforum_threads SET wft_closed_user_ip = '0.0.0.0' WHERE wft_closed_user_ip = 'REDACTED'; on programmingwikiwiki
 * 23:03 Universal_Omega: UPDATE wikiforum_threads SET wft_closed_user_ip = '0.0.0.0' WHERE wft_closed_user_ip = 'REDACTED'; on programmingwikiwiki
 * 23:00 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED' on test2wiki;
 * 22:58 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED' on test2wiki;
 * 22:57 Universal_Omega: UPDATE wikiforum_forums SET wff_added_user_ip = '0.0.0.0' WHERE wff_added_user_ip = 'REDACTED' on test2wiki;
 * 22:56 Universal_Omega: UPDATE wikiforum_category SET wfc_added_user_ip = '0.0.0.0' WHERE wfc_added_user_ip = 'REDACTED'; on test2wiki;
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:45 Universal_Omega: UPDATE wikiforum_threads SET wft_edit_user_ip = '0.0.0.0' WHERE wft_edit_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:45 Universal_Omega: UPDATE wikiforum_threads SET wft_edit_user_ip = '0.0.0.0' WHERE wft_edit_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:45 Universal_Omega: UPDATE wikiforum_threads SET wft_edit_user_ip = '0.0.0.0' WHERE wft_edit_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:32 Universal_Omega: UPDATE wikiforum_forums SET wff_added_user_ip = '0.0.0.0' WHERE wff_added_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:32 Universal_Omega: UPDATE wikiforum_forums SET wff_added_user_ip = '0.0.0.0' WHERE wff_added_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:29 Universal_Omega: UPDATE wikiforum_forums SET wff_last_post_user_ip = '0.0.0.0' WHERE wff_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:29 Universal_Omega: UPDATE wikiforum_forums SET wff_last_post_user_ip = '0.0.0.0' WHERE wff_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:21 Universal_Omega: UPDATE wikiforum_category SET wfc_added_user_ip = '0.0.0.0' WHERE wfc_added_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:21 Universal_Omega: UPDATE wikiforum_category SET wfc_added_user_ip = '0.0.0.0' WHERE wfc_added_user_ip = 'REDACTED'; on snapwikiwiki
 * 20:33 Southparkfan: deleted job id 165 (BackupStatic1) as well but does not seem to get bacula running
 * 20:30 Southparkfan: stop bacula jobs for static1/2

2021-01-07

 * 21:57 John: added conduct alias to LDAP

2021-01-06

 * 23:59 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw*, jbr*, and test2
 * 23:57 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maint*/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on mw*, jbr*, and test2
 * 00:06 Universal_Omega: UPDATE localuser SET lu_attached_method = 'login' WHERE lu_name="Universal Omega" AND lu_attached_method = 'new'; and UPDATE localuser SET lu_attached_method = 'new' WHERE lu_name="Universal Omega" AND lu_wiki = 'metawiki'; (to match with gu_home_db)

2021-01-05

 * 22:48 Universal_Omega: universalomega@jobrunner1:~$ php /home/universalomega/populateWikiSettings.php --wgsetting=wmgWhitelistRead --sourcelist=/home/universalomega/wgWhitelistRead.txt --wiki=loginwiki (using modified version of populateWikiSettings.php to support this specific type of setting)
 * 01:49 Zppix: initSiteStats.php for chakuwikiwiki (both rebuildRC and this on JBR2)
 * 01:45 Zppix: rebuildRC for chakuwikiwiki

2021-01-04

 * 14:43 paladox: root@jobrunner1:/srv/mediawiki/w/extensions/ManageWiki/maintenance# sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --disable wikiforum
 * 10:46 RhinosF1: stop nginx&puppet on test2 until config .git/objects can be fixed to avoid X-Debug being used to enable WikiForum
 * 10:36 RhinosF1: globally kill wikiforum

2021-01-03

 * 20:26 Reception123: added windowswiki discord webhook
 * 15:45 Southparkfan: restarted adminlogbot
 * 15:45 Southparkfan: enabled a personal cronjob on cp7 to measure mediawiki response time

2021-01-02

 * 20:28 Zppix: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --skip-to=19201 --report=1 --wiki chakuwikiwiki /home/reception/chakuwiki-full.xml --namespaces="0|1|4|5|6|7|8|9|10|11|12|13|14|15" --uploads --no-local-users (restart import) on jbr2
 * 18:49 paladox: increase mon1 disk by 10gb
 * 14:43 paladox: MariaDB [(none)]> PURGE BINARY LOGS BEFORE '2021-01-03 00:00:00'; - db12
 * 11:56 Reception123: ran GDPR script
 * 07:32 Reception123: reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki minecraftathomewiki --r "Requested" /home/reception/minecraftathome.txt

2021-01-01

 * 23:49 Universal_Omega: UPDATE mw_settings set s_settings = JSON_REMOVE(s_settings, '$.wgAccountCreationThrottle');
 * 23:49 Universal_Omega: UPDATE mw_settings set s_settings = JSON_REMOVE(s_settings, '$.wgModerationNotificationNewOnly');
 * 23:48 Universal_Omega: UPDATE mw_settings set s_settings = JSON_REMOVE(s_settings, '$.wgModerationNotificationEnable');
 * 23:48 Universal_Omega: UPDATE mw_settings set s_settings = JSON_REMOVE(s_settings, '$.wgModerationEmail');
 * 20:40 Southparkfan: enable puppet on mail1
 * 20:27 Southparkfan: enabled puppet on test2 & disable puppet on mail1
 * 20:12 Universal_Omega: universalomega@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateGroupPermissionsWithDefaults.php --wiki burnoutwiki --overwrite
 * 19:58 Southparkfan: restart phd on phab1
 * 19:21 Southparkfan: disable puppet on test2 for T5044 experimenting
 * 19:21 Southparkfan: db11 (c1) done
 * 19:13 Southparkfan: db11 (c1) done; moving to db11 (c2)
 * 19:12 Southparkfan: db13 already done; moving to db11 (c1)
 * 19:11 Southparkfan: applied T6095 patches to db12, applying to db13 now
 * 18:34 paladox: MariaDB [(none)]> PURGE BINARY LOGS BEFORE '2021-01-02 00:00:00'; - db12
 * 18:32 paladox: MariaDB [(none)]> PURGE BINARY LOGS BEFORE DATE(NOW) + INTERVAL 0 SECOND; - db12
 * 18:06 Southparkfan: applying database patches for T6095 to db12
 * 03:24 Zppix: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --report=1 --wiki chakuwikiwiki /home/reception/chakuwiki-full.xml --namespaces="0|1|4|5|6|7|8|9|10|11|12|13|14|15" --uploads --no-local-users on jobrunner2 (on behalf of reception)
 * 02:41 paladox: add shekhwiki to matomo
 * 02:38 paladox: re add encycmhwiki to matomo
 * 01:50 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --update --wiki izunawiki
 * 01:48 Universal_Omega: universalomega@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki izunawiki --report=1 /home/universalomega/izuna_gamepedia_en_pages_full.xml
 * 01:46 paladox: upgrade grafana, icinga2 and tzdata on mon1
 * 01:36 paladox: drop piwik on db13 after moving it to db12
 * 00:10 Universal_Omega: UPDATE mw_settings SET s_settings = JSON_REPLACE(s_settings, '$.wmgDefaultRobotPolicy', JSON_REMOVE(s_settings, '$.wmgDefaultRobotPolicy');