Tech:Server admin log/2021

2021-01-13

 * 20:12 Southparkfan: c3 set up on dbbackup1
 * 18:37 Southparkfan: dumping db12 to dbbackup1
 * 11:40 Southparkfan: started c2 replication on dbbackup1
 * 10:24 Reception123: renamed worldfederationdiscordwiki to usfederalgovernmentrpwiki
 * 00:09 Southparkfan: c4's replication back on dbbackup1, dumping c2 now

2021-01-12

 * 22:01 Southparkfan: reclone all clusters on dbbackup1
 * 21:59 RhinosF1: for the record: John added IRC_MIRAHEZEBOTS to GitHub secrets and grouped miraheze-GitHub to @miraheze/bots - change deployed to CI world everywhere I can see
 * 21:38 Southparkfan: stop all mysql instances on dbbackup1 to move relay bin logs
 * 21:35 Southparkfan: c3 and c4 are corrupted on dbbackup1, deleting and recloning those
 * 21:30 Southparkfan: dbbackup1: db11 copied succesfully, started replication
 * 21:30 Southparkfan: dbbackup1: add hard and soft 'nofile' limits in /etc/security/limits.conf
 * 21:17 Southparkfan: set ulimit -Sn 1000000 on dbbackup1
 * 20:46 Southparkfan: dbbackup1: temporarily stop c3 replication to reduce load
 * 20:35 Southparkfan: there was an issue with c3 replication on dbbackup1, fixed with https://dba.stackexchange.com/a/191558
 * 20:15 Southparkfan: dump db11 to dbbackup1
 * 20:12 Southparkfan: db11: increase soft open files limit to 1 million
 * 16:10 paladox: reboot and upgrade kernel on cp3

2021-01-11

 * 22:21 Zppix: rebuildRC and initSiteStats following import on jbr2
 * 21:36 Zppix: zppix@jobrunner2: screen sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki durkomorewiki --report=1 uklurkmoreua_pages_full.xml
 * 00:31 Southparkfan: used command (for reference): mydumper -B allthetropeswiki -G -E -R -v 3 -t 2 -c -S /var/run/mysqld/mysqld.c4.sock
 * 00:30 Southparkfan: results of compressed mydumper run on allthetropeswiki: 33G (database in /srv/mariadb.c4) -> 12G of sql.gz files
 * 00:04 Southparkfan: perform a mydumper test on dbbackup1

2021-01-10

 * 20:11 paladox: rebuild lc on test2
 * 20:11 paladox: rebuild lc on mw[4567] and jobrunner[12]
 * 18:44 paladox: delete db7 on cloud2
 * 17:23 Southparkfan: dbbackup1: start db12 & db13 replication :)
 * 17:08 Southparkfan: prepare db13 backup and stop db12 replication temporarily (performance)
 * 15:43 Southparkfan: dumping db13 to dbbackup1
 * 15:40 Southparkfan: increase soft open files limit to 16384 on db13
 * 15:09 Southparkfan: db12 imported to dbbackup1
 * 14:02 Southparkfan: imported replica grants on db12
 * 13:44 Southparkfan: try a second mariabackup --prepare on the db12 backup
 * 13:42 Southparkfan: ulimit -a 950000 on dbbackup1
 * 03:03 Universal_Omega: universalomega@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=testwiki

2021-01-09

 * 22:46 Universal_Omega: sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/wikiforum.json /srv/mediawiki/w/maintenance/sql.php /home/universalomega/drop-wft_closed-from-wikiforum_threads.sql
 * 20:06 Southparkfan: restart db13 dump (crashed due to: xtrabackup_copy_logfile failed: redo log block is overwritten, please increase redo log size with innodb_log_file_size parameter.)
 * 19:31 paladox: root@dbbackup1:/home/paladox# chmod 1777 /tmp
 * 19:26 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki opencircuitswiki /home/reception/opencircuitscom-20200218-history.xml
 * 18:54 Southparkfan: running mariabackup on db12 & db13 to dump all databases to dbbackup1
 * 18:46 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw7
 * 18:14 Universal_Omega: sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/wikiforum.json /home/universalomega/migrateOldWikiForumClosedTimestampColumnToNew.php
 * 18:06 Universal_Omega: sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/wikiforum.json /srv/mediawiki/w/maintenance/sql.php /home/universalomega/add-wft_closed_timestamp-to-wikiforum_threads.sql
 * 17:25 Southparkfan: repool mw6
 * 16:48 paladox: rebuild lc on test2 and jobrunner[12]
 * 16:44 paladox: depool and repool mw[4567] and rebuild lc
 * 14:27 Southparkfan: dbbackup1: mysql_install_db --user=mysql --datadir=/srv/mariadb.c3
 * 14:24 Southparkfan: set 4755 manually on /usr/lib/mysql/plugin/auth_pam_tool_dir/auth_pam_tool
 * 11:54 Southparkfan: purchase dbbackup1.miraheze.org; 4GB MKVM in Atlanta
 * 06:42 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* and jbr*
 * 06:41 Reception123: sudo -u www-data php /srv/mediawiki/w/maint*/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on mw* and jbr*

2021-01-08

 * 23:38 Universal_Omega: universalomega@jobrunner1:~$ sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/wikiforum.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php wikiforum
 * 23:17 Universal_Omega: UPDATE wikiforum_category SET wfc_added_user_ip = '0.0.0.0' WHERE wfc_added_user_ip = 'REDACTED'; on aboutpcswiki
 * 23:16 Universal_Omega: UPDATE wikiforum_forums SET wff_added_user_ip = '0.0.0.0' WHERE wff_added_user_ip = 'REDACTED'; on aboutpcswiki
 * 23:15 Universal_Omega: UPDATE wikiforum_forums SET wff_last_post_user_ip = '0.0.0.0' WHERE wff_last_post_user_ip = 'REDACTED'; on aboutpcswiki
 * 23:14 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on aboutpcswiki
 * 23:10 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on bonziworldwiki
 * 23:10 Universal_Omega: UPDATE wikiforum_forums SET wff_last_post_user_ip = '0.0.0.0' WHERE wff_last_post_user_ip = 'REDACTED'; on bonziworldwiki
 * 23:08 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on cubashpediawiki
 * 23:07 Universal_Omega: UPDATE wikiforum_forums SET wff_last_post_user_ip = '0.0.0.0' WHERE wff_last_post_user_ip = 'REDACTED'; on cubashpediawiki
 * 23:03 Universal_Omega: UPDATE wikiforum_threads SET wft_closed_user_ip = '0.0.0.0' WHERE wft_closed_user_ip = 'REDACTED'; on programmingwikiwiki
 * 23:03 Universal_Omega: UPDATE wikiforum_threads SET wft_closed_user_ip = '0.0.0.0' WHERE wft_closed_user_ip = 'REDACTED'; on programmingwikiwiki
 * 23:00 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED' on test2wiki;
 * 22:58 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED' on test2wiki;
 * 22:57 Universal_Omega: UPDATE wikiforum_forums SET wff_added_user_ip = '0.0.0.0' WHERE wff_added_user_ip = 'REDACTED' on test2wiki;
 * 22:56 Universal_Omega: UPDATE wikiforum_category SET wfc_added_user_ip = '0.0.0.0' WHERE wfc_added_user_ip = 'REDACTED'; on test2wiki;
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:49 Universal_Omega: UPDATE wikiforum_threads SET wft_last_post_user_ip = '0.0.0.0' WHERE wft_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:45 Universal_Omega: UPDATE wikiforum_threads SET wft_edit_user_ip = '0.0.0.0' WHERE wft_edit_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:45 Universal_Omega: UPDATE wikiforum_threads SET wft_edit_user_ip = '0.0.0.0' WHERE wft_edit_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:45 Universal_Omega: UPDATE wikiforum_threads SET wft_edit_user_ip = '0.0.0.0' WHERE wft_edit_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:40 Universal_Omega: UPDATE wikiforum_threads SET wft_user_ip = '0.0.0.0' WHERE wft_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:32 Universal_Omega: UPDATE wikiforum_forums SET wff_added_user_ip = '0.0.0.0' WHERE wff_added_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:32 Universal_Omega: UPDATE wikiforum_forums SET wff_added_user_ip = '0.0.0.0' WHERE wff_added_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:29 Universal_Omega: UPDATE wikiforum_forums SET wff_last_post_user_ip = '0.0.0.0' WHERE wff_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:29 Universal_Omega: UPDATE wikiforum_forums SET wff_last_post_user_ip = '0.0.0.0' WHERE wff_last_post_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:21 Universal_Omega: UPDATE wikiforum_category SET wfc_added_user_ip = '0.0.0.0' WHERE wfc_added_user_ip = 'REDACTED'; on snapwikiwiki
 * 22:21 Universal_Omega: UPDATE wikiforum_category SET wfc_added_user_ip = '0.0.0.0' WHERE wfc_added_user_ip = 'REDACTED'; on snapwikiwiki
 * 20:33 Southparkfan: deleted job id 165 (BackupStatic1) as well but does not seem to get bacula running
 * 20:30 Southparkfan: stop bacula jobs for static1/2

2021-01-07

 * 21:57 John: added conduct alias to LDAP

2021-01-06

 * 23:59 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw*, jbr*, and test2
 * 23:57 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maint*/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on mw*, jbr*, and test2
 * 00:06 Universal_Omega: UPDATE localuser SET lu_attached_method = 'login' WHERE lu_name="Universal Omega" AND lu_attached_method = 'new'; and UPDATE localuser SET lu_attached_method = 'new' WHERE lu_name="Universal Omega" AND lu_wiki = 'metawiki'; (to match with gu_home_db)

2021-01-05

 * 22:48 Universal_Omega: universalomega@jobrunner1:~$ php /home/universalomega/populateWikiSettings.php --wgsetting=wmgWhitelistRead --sourcelist=/home/universalomega/wgWhitelistRead.txt --wiki=loginwiki (using modified version of populateWikiSettings.php to support this specific type of setting)
 * 01:49 Zppix: initSiteStats.php for chakuwikiwiki (both rebuildRC and this on JBR2)
 * 01:45 Zppix: rebuildRC for chakuwikiwiki

2021-01-04

 * 14:43 paladox: root@jobrunner1:/srv/mediawiki/w/extensions/ManageWiki/maintenance# sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --disable wikiforum
 * 10:46 RhinosF1: stop nginx&puppet on test2 until config .git/objects can be fixed to avoid X-Debug being used to enable WikiForum
 * 10:36 RhinosF1: globally kill wikiforum

2021-01-03

 * 20:26 Reception123: added windowswiki discord webhook
 * 15:45 Southparkfan: restarted adminlogbot
 * 15:45 Southparkfan: enabled a personal cronjob on cp7 to measure mediawiki response time

2021-01-02

 * 20:28 Zppix: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --skip-to=19201 --report=1 --wiki chakuwikiwiki /home/reception/chakuwiki-full.xml --namespaces="0|1|4|5|6|7|8|9|10|11|12|13|14|15" --uploads --no-local-users (restart import) on jbr2
 * 18:49 paladox: increase mon1 disk by 10gb
 * 14:43 paladox: MariaDB [(none)]> PURGE BINARY LOGS BEFORE '2021-01-03 00:00:00'; - db12
 * 11:56 Reception123: ran GDPR script
 * 07:32 Reception123: reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki minecraftathomewiki --r "Requested" /home/reception/minecraftathome.txt

2021-01-01

 * 23:49 Universal_Omega: UPDATE mw_settings set s_settings = JSON_REMOVE(s_settings, '$.wgAccountCreationThrottle');
 * 23:49 Universal_Omega: UPDATE mw_settings set s_settings = JSON_REMOVE(s_settings, '$.wgModerationNotificationNewOnly');
 * 23:48 Universal_Omega: UPDATE mw_settings set s_settings = JSON_REMOVE(s_settings, '$.wgModerationNotificationEnable');
 * 23:48 Universal_Omega: UPDATE mw_settings set s_settings = JSON_REMOVE(s_settings, '$.wgModerationEmail');
 * 20:40 Southparkfan: enable puppet on mail1
 * 20:27 Southparkfan: enabled puppet on test2 & disable puppet on mail1
 * 20:12 Universal_Omega: universalomega@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateGroupPermissionsWithDefaults.php --wiki burnoutwiki --overwrite
 * 19:58 Southparkfan: restart phd on phab1
 * 19:21 Southparkfan: disable puppet on test2 for T5044 experimenting
 * 19:21 Southparkfan: db11 (c1) done
 * 19:13 Southparkfan: db11 (c1) done; moving to db11 (c2)
 * 19:12 Southparkfan: db13 already done; moving to db11 (c1)
 * 19:11 Southparkfan: applied T6095 patches to db12, applying to db13 now
 * 18:34 paladox: MariaDB [(none)]> PURGE BINARY LOGS BEFORE '2021-01-02 00:00:00'; - db12
 * 18:32 paladox: MariaDB [(none)]> PURGE BINARY LOGS BEFORE DATE(NOW) + INTERVAL 0 SECOND; - db12
 * 18:06 Southparkfan: applying database patches for T6095 to db12
 * 03:24 Zppix: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --report=1 --wiki chakuwikiwiki /home/reception/chakuwiki-full.xml --namespaces="0|1|4|5|6|7|8|9|10|11|12|13|14|15" --uploads --no-local-users on jobrunner2 (on behalf of reception)
 * 02:41 paladox: add shekhwiki to matomo
 * 02:38 paladox: re add encycmhwiki to matomo
 * 01:50 Universal_Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --update --wiki izunawiki
 * 01:48 Universal_Omega: universalomega@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki izunawiki --report=1 /home/universalomega/izuna_gamepedia_en_pages_full.xml
 * 01:46 paladox: upgrade grafana, icinga2 and tzdata on mon1
 * 01:36 paladox: drop piwik on db13 after moving it to db12
 * 00:10 Universal_Omega: UPDATE mw_settings SET s_settings = JSON_REPLACE(s_settings, '$.wmgDefaultRobotPolicy', JSON_REMOVE(s_settings, '$.wmgDefaultRobotPolicy');