Tech:Server admin log/2020

2020-03-30

 * 21:18 Zppix: created dummy wiki dbs to prevent conflict with services/servers (mon1, icinga, grafana, dbt1)
 * 01:06 paladox: apt-get dist-upgrade - services2
 * 01:04 paladox: apt-get dist-upgrade - services1

2020-03-29

 * 23:54 paladox: remove db4 from puppetdb
 * 23:45 paladox: remove mw[123] and lizardfs6 from puppetdb
 * 23:36 paladox: restart php7.3-fpm on mw*
 * 21:45 paladox: apt-get upgrade - mail1
 * 20:16 paladox: depool mw7 and repool
 * 20:05 paladox: restart php7.3-fpm on mw*
 * 19:59 paladox: depool mw6 and repool
 * 19:59 paladox: increase mw[67] vpcu by 1
 * 19:59 paladox: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki crappygameswiki -r "Requested - T5324" /home/reception/cgwdel.txt - mw4
 * 19:59 paladox: repool mw5
 * 19:59 paladox: increase mw5 vcpu by 1
 * 19:59 paladox: depool mw5
 * 19:58 paladox: repool mw4
 * 19:58 paladox: increase mw4 vcpu by 1
 * 19:55 paladox: depool mw4
 * 19:30 Reception123: add alteversewiki webhook for discord
 * 17:58 Reception123: added webhook for zoitecwiki
 * 17:43 paladox: rm /srv/mediawiki/w/cache/managewiki/*privado* on mw*
 * 17:43 paladox: deleting *privado* in redis
 * 17:43 paladox: recreating privadowiki
 * 17:39 paladox: rm /srv/mediawiki/w/cache/managewiki/* ucronias* on mw*
 * 17:38 paladox: deleting * ucronias* in redis
 * 17:38 paladox: recreating ucroniaswiki
 * 17:35 paladox: rm /srv/mediawiki/w/cache/managewiki/*hispano* on mw*
 * 17:32 paladox: deleting *hispano* in redis
 * 17:30 paladox: recreating hispanowiki
 * 16:40 Southparkfan: ack icinga alerts for decom'ed servers
 * 14:55 paladox: repool mw5
 * 14:53 paladox: depool mw5
 * 14:51 paladox: repool mw4
 * 14:50 paladox: depool mw4
 * 14:50 paladox: increased mw[45] vpcu by 1
 * 14:48 paladox: decrease puppet2 vcpu by one
 * 14:25 Reception123: reception@mw4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki crappygameswiki -r "Requested - T5324" /home/reception/cgwdel.txt
 * 07:58 Reception123: reception@mw4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=sagan4alphawiki --search-recursively --skip-dupes /home/reception/images
 * 04:53 paladox: restart logbot on mon1
 * 04:53 paladox: stop mysql on db4
 * 02:40 paladox: puppetserver ca clean --certname lizardfs6.m.org/mw3.m.org
 * 01:31 paladox: increased puppet2 ram by 1gb and removed swap
 * 01:24 paladox: add 1g swap to puppet2
 * 01:14 paladox: delete all wikis on db4 that have been migrated to dbt1
 * 00:43 paladox: restart php-fpm on mw*

2020-03-28

 * 19:20 Reception123: removed a 2FA from a user
 * 18:36 Reception123: reception@mw1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki awesomegameswiki -r "Requested (old fandom accounts) - T5356" /home/reception/agwdel.txt
 * 14:09 paladox: restarted php7.3-fpm on mw*
 * 02:44 paladox: drop 1.6k wikis on db4 (migrated to dbt1)

2020-03-27

 * 23:14 paladox: apt-get upgrade puppet-agent - mw3
 * 18:41 paladox: restart php7.3-fpm on mw*

2020-03-26

 * 15:39 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw*
 * 15:33 paladox: restart logbot on mon1
 * 02:24 paladox: install links on test2
 * 01:58 paladox: reset extensions/CreateWiki on test2

2020-03-25

 * 23:27 paladox: restarted php7.3-fpm on mw*

2020-03-24

 * 17:01 Reception123: deleted and dropped testdelete1wiki
 * 16:23 Reception123: sudo -u www-data php populateContentTables.php --wiki [hispanowiki/ucroniaswiki/pricadowiki]
 * 16:19 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php populateContentModel.php --wiki [hispanowiki/ucroniaswiki/pricadowiki] --ns "all" --table page/revision

2020-03-23

 * 18:34 paladox: service parsoid stop ; killall -u parsoid ; service parsoid start - services[12]
 * 15:32 paladox: that is on nginx
 * 15:25 paladox: disable logging on mw[46] as an experiment
 * 15:19 paladox: MariaDB [ucroniaswiki]> INSERT INTO slot_roles (`role_id`, `role_name`) VALUES (1, 'main'); - db7
 * 15:18 paladox: MariaDB [hispanowiki]> INSERT INTO slot_roles (`role_id`, `role_name`) VALUES (1, 'main'); - db7
 * 15:17 paladox: MariaDB [privadowiki]> INSERT INTO slot_roles (`role_id`, `role_name`) VALUES (1, 'main'); - db7

2020-03-22

 * 23:29 paladox: MariaDB [privadowiki]> INSERT INTO content_models (`model_id`, `model_name`) VALUES (3, 'javascript'); - db7
 * 23:28 paladox: MariaDB [privadowiki]> INSERT INTO content_models (`model_id`, `model_name`) VALUES (4, 'Scribunto'); - db7
 * 23:28 paladox: MariaDB [privadowiki]> INSERT INTO content_models (`model_id`, `model_name`) VALUES (5, 'json'); - db7
 * 23:28 paladox: MariaDB [privadowiki]> INSERT INTO content_models (`model_id`, `model_name`) VALUES (2, 'css'); - db7
 * 21:35 paladox: drop database ndgwiki and nenawikiwiki on db4
 * 21:28 Zppix: (paladox): migrated ndg and nena wikis

2020-03-21

 * 19:52 paladox: delete ucroniaswiki
 * 19:51 paladox: delete hispanowiki
 * 16:09 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki rottenwebsiteswiki -r "Requested - T5313" /home/reception/delrt1.txt on mw1
 * 16:01 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/dumpBackup.php --wiki aerosswiki --full > /home/reception/aerosswiki21032020.xml
 * 15:57 Reception123: renamed magnaversewiki to cnocbridewiki (db+cw)
 * 15:13 Reception123: UPDATE cw_wikis SET wiki_closed = 0 AND wiki_closed_timestamp = NULL WHERE wiki_closed_timestamp > 20200321000000 on db4
 * 15:03 Reception123: php /srv/mediawiki/w/extensions/CreateWiki/maintenance/manageInactiveWikis.php --wiki loginwiki --write > /var/log/mediawiki/cron/managewikis.log on mw1
 * 14:36 Reception123: reception@mw1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki rottenwebsiteswiki
 * 14:13 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki sagan4alphawiki /home/reception/sagan4wiki.xml
 * 13:45 Reception123: manually ran manageInactiveWikis.php
 * 13:37 Reception123: manually ran manageInactiveWikis.php
 * 12:00 Reception123: reception@mw1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki rottenwebsiteswiki -r "Requested - T5313" /home/reception/delrt.txt

2020-03-20

 * 21:03 paladox: running lc on mw* and test2 and jobrunner1
 * 20:08 paladox: upgrade puppetdb, puppetserver and puppet-agent on puppet2
 * 06:37 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --no-local-users --username-prefix "import" --wiki pikminfanonwiki /home/reception/pikminfanon.xml

2020-03-19

 * 21:18 paladox: reboot test2
 * 21:13 paladox: apt-get upgrade - test2
 * 21:10 paladox: rename test1wiki to test2wiki
 * 20:26 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"enc.for.uz"', null ) WHERE wiki_dbname = 'uzencwiki'; - db4
 * 16:44 paladox: running lc on mw*

2020-03-18

 * 22:53 paladox: depool mw[123]

2020-03-17

 * 21:05 paladox: restart php7.3-fpm on mw*
 * 21:05 paladox: reboot services2
 * 21:04 paladox: reboot services1
 * 14:46 Zppix: rebuildrc and initsitestats on libertygamewiki following successful import

2020-03-16

 * 17:46 Zppix: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --no-local-users --username-prefix="리버티게임" --wiki libertygamewiki 22115932221_w-20200309-history.xml on jobrunner1
 * 15:12 paladox: restart php7.3-fpm on mw*

2020-03-15

 * 13:47 paladox: apt-get dist-upgrade - cp7

2020-03-14

 * 16:04 paladox: increzse puppet2 disk by 10g

2020-03-13

 * 21:03 paladox: rename godaigowiki to brambowiki
 * 13:20 paladox: root@puppet2:/opt/puppetlabs/server/data/puppetserver/reports# rm -rf *

2020-03-12

 * 20:53 paladox: root@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance# sudo -u www-data php fixStuckGlobalRename.php --wiki=abominablememefandomswiki --logwiki=metawiki "LeonTheMonster479519314" "Just Leon"
 * 20:06 paladox: root@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance# sudo -u www-data php fixStuckGlobalRename.php --wiki=abominablememefandomswiki --logwiki=metawiki "Agent Mace" "Agent Joestar"
 * 00:48 Zppix: update cw_requests set cw_status = 'approved' where cw_id = 10991;

2020-03-11

 * 15:36 paladox: rm -rf /srv/mediawiki/config and let it reclone, merge conflicts so recloning fresh
 * 15:28 paladox: apt-get dist-upgrade - test1
 * 15:26 paladox: reboot test1
 * 15:11 paladox: root@puppet2:/opt/puppetlabs/server/data/puppetserver/reports# rm -rf *
 * 14:08 paladox: increase gluster1 disk to 720G
 * 01:25 paladox: increase gluster1 disk by 50g

2020-03-10

 * 22:53 paladox: apt-get upgrade - mon1 (icinga2/puppet-agent)
 * 22:45 paladox: shutdown misc4
 * 22:45 paladox: shutdown cp4
 * 22:44 paladox: shutdown puppet1
 * 21:59 paladox: MariaDB [browndustwiki]> INSERT INTO content_models (`model_id`,`model_name`) VALUES (4, 'Scribunto'); -db4
 * 21:59 paladox: MariaDB [browndustwiki]> INSERT INTO content_models (`model_id`,`model_name`) VALUES (2, 'css'); -db4
 * 21:58 paladox: MariaDB [browndustwiki]> INSERT INTO content_models (`model_id`,`model_name`) VALUES (3, 'javascript'); -db4
 * 21:58 paladox: MariaDB [browndustwiki]> INSERT INTO content_models (`model_id`,`model_name`) VALUES (5, 'json'); -db4
 * 21:58 paladox: MariaDB [browndustwiki]> INSERT INTO content_models (`model_id`,`model_name`) VALUES (1, 'wikitext'); -db4
 * 21:46 paladox: MariaDB [browndustwiki]> INSERT INTO slot_roles (`role_id`,`role_name`) VALUES (1,'main'); - db4
 * 21:33 paladox: root@lizardfs6:/home/paladox# gluster volume set mvol features.read-only off
 * 21:03 paladox: restart jobrunner + jobchron on jobrunner1
 * 20:48 paladox: delete *matomo* key in redis
 * 20:32 paladox: root@jobrunner1:/srv/mediawiki/w/extensions/MirahezeMagic/maintenance# /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/addMatomo.php
 * 20:31 paladox: MariaDB [mhglobal]> delete from matomo; - db4
 * 19:53 Zppix: update cw_requests set cw_status = 'approved' where cw_id = 10988;
 * 18:52 paladox: set features.read-only to on, on lizardfs6
 * 18:17 Zppix: run cargoRecreateData.php and refreshLinks.php on jobrunner1 on hispanowiki per user request on IRC

2020-03-09

 * 22:51 paladox: reboot misc1
 * 20:51 paladox: apt-get install puppet-agent salt-common salt-minion - cp8

2020-03-08

 * 23:58 paladox: started ircecho on misc1
 * 13:16 Southparkfan: cancelled migration, dropping databases on db6
 * 13:12 Southparkfan: migration started for first batch
 * 00:47 paladox: delete db5 - not in use anymore

2020-03-07

 * 23:23 paladox: reinstalling ns1 as buster
 * 19:26 paladox: upgrade puppet-agent on cloud[12]
 * 19:21 paladox: root@cloud1:/var/lib/vz/images/118# rm vm-118-disk-0.qcow2
 * 18:38 paladox: upgrade puppet-agent - ns2
 * 18:37 paladox: root@cloud1:/var/lib/vz/images/118# mv vm-118-disk-0.row vm-118-disk-0.raw
 * 18:34 paladox: root@cloud1:/var/lib/vz/images/118# qemu-img convert vm-118-disk-0.qcow2 vm-118-disk-0.row
 * 17:41 paladox: removed prometheus_gdnsd_stats cron from misc1
 * 17:39 paladox: stopped gdnsd + purged it from misc1
 * 17:21 paladox: MariaDB [(none)]> set global table_open_cache = 10000; - db4
 * 17:20 paladox: set global table_definition_cache = 10000; - db4
 * 17:17 paladox: apt-get -t stretch-backports install linux-image-amd64 - db5
 * 17:10 paladox: apt-get upgrade db5
 * 06:35 Reception123: deleted and dropped streamelementswiki (T5307)
 * 01:34 paladox: MariaDB [simswiki]> truncate objectcache; - db6
 * 00:45 paladox: drop large wiki drops on db4/db5

2020-03-06

 * 23:50 Southparkfan: run pveperf on cloud1
 * 20:28 paladox: root@db5:/home/paladox# ./move_all.sh
 * 19:03 paladox: root@db4:/home/paladox# ./move_all.sh
 * 19:02 Zppix: Migration window has started
 * 18:38 paladox: sysctl -w net.ipv6.conf.all.disable_ipv6=1 - db5
 * 16:52 paladox: upgrade puppet-agent on misc4

2020-03-05

 * 17:52 paladox: apt-get upgrade - puppet1 (puppetserver, puppetdb and a few other packages)
 * 02:18 paladox: drop phabricator_% on db4
 * 02:18 paladox: drop staffwiki on db4
 * 01:02 paladox: set write back cache on mon1
 * 00:58 paladox: increase phab1 disk by 5g
 * 00:54 paladox: sysctl -w net.ipv6.conf.all.disable_ipv6=1 - db4

2020-03-03

 * 22:28 paladox: delete from renameuser_status where ru_wiki = 'receptionflamewarswiki'; - db4
 * 22:24 paladox: delete receptionflamewarswiki from cw
 * 21:06 paladox: stop db7 being a slave temporarily
 * 19:38 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki TaylorSwiftFan "GrilledCheese800" --logwiki=metawiki

2020-03-02

 * 19:25 paladox: downgrade php-redis on mw1
 * 18:15 paladox: downgraded php on jobrunner1 and upgraded gluster + few other pakcages
 * 18:13 paladox: apt-get upgrade - gluster1
 * 17:40 paladox: running lc on mw4
 * 17:24 paladox: fixing php on mw[4567]
 * 16:50 paladox: downgraded php to 7.3.14 on mw[23] lizardfs6 due to a segfault in 7.3.15, mw1 will be out of the pool till it's upgraded to buster.
 * 14:30 paladox: reinstalling mw4
 * 13:08 paladox: reboot mw4

2020-03-01

 * 17:39 paladox: root@mw2:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=tmewiki --type=gzip
 * 17:35 paladox: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=crappygameswiki --type=gzip
 * 17:33 paladox: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=animatedfeetwiki --type=gzip
 * 17:22 paladox: MariaDB [altversewiki]> drop database testwiki; - db5 (on db6 already)
 * 17:14 paladox: MariaDB [altversewiki]> drop database ucroniaswiki; - db4 (on db6 already)
 * 17:14 paladox: MariaDB [altversewiki]> drop database rotompediawiki; - db4 (on db6 already)
 * 17:13 paladox: MariaDB [altversewiki]> drop database proxybotwiki; - db4 (on db6 already)
 * 17:13 paladox: MariaDB [altversewiki]> drop database privadowiki; - db4 (on db6 already)
 * 17:13 paladox: MariaDB [altversewiki]> drop database onepiecewiki; - db4 (on db6 already)
 * 17:13 paladox: MariaDB [altversewiki]> drop database onepiecebountyrushwiki; - db4 (on db6 already)
 * 17:12 paladox: MariaDB [altversewiki]> drop database nonbinarywiki; - db4 (on db6 already)
 * 17:12 paladox: MariaDB [altversewiki]> drop database nbdbwiki; - db4 (on db6 already)
 * 17:12 paladox: MariaDB [altversewiki]> drop database hispanowiki; - db4 (on db6 already)
 * 17:11 paladox: MariaDB [altversewiki]> drop database dreamversewiki; - db4 (on db6 already)
 * 17:10 paladox: MariaDB [altversewiki]> drop database anotheredenwiki; - db4 (on db6 already)
 * 17:10 paladox: root@db5:/home/dbcopy# rm *sql*
 * 17:09 paladox: MariaDB [altversewiki]> drop database altversewiki; - db4 (on db6 already)
 * 14:47 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php initSiteStats.php --wiki=hispanowiki --update
 * 14:46 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php updateArticleCount.php --wiki=hispanowiki --update
 * 12:50 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/dumpBackup.php --wiki allthetropeswiki --full > /home/reception/allthetropes01032020.xml (part of wikibackups) on mw2
 * 11:40 Reception123: running wikibackups (bash ./wikibackups.sh /home/reception/allpublic.dblist /srv/mediawiki/w/maintenance/dumpBackup.php) on mw1
 * 11:22 Reception123: moved openhatchwiki from db4 to db5
 * 11:03 Reception123: renamed epochlorewiki to mystfalllorewiki (db + script)

2020-02-29

 * 21:10 Reception123: PURGE BINARY LOGS BEFORE '2020-02-29 12:00'; on db4
 * 00:08 paladox: ip addr del fe80::f816:3eff:fe31:f5f/64 dev ens3 - bacula2
 * 00:08 paladox: ip -6 address add 2604:180:f3::382/64 dev ens3 - bacula2

2020-02-28

 * 22:34 Southparkfan: reinstall bacula2
 * 22:11 Southparkfan: decom bacula1
 * 17:43 paladox: reboot mw[45]
 * 17:15 paladox: restarted php7.3-fpm on mw1
 * 15:24 paladox: restart php7.3-fpm on mw* and lizardfs6
 * 15:12 paladox: MariaDB [(none)]> set global innodb_io_capacity_max=3000; - db4
 * 15:11 paladox: MariaDB [(none)]> set global innodb_io_capacity=1000; - db4
 * 14:36 paladox: MariaDB [(none)]> set global tmp_table_size = 67108864; - db4
 * 14:35 paladox: MariaDB [(none)]> set global max_heap_table_size = 67108864; (on db4

2020-02-27

 * 23:25 paladox: root@db4:/home/paladox# sysctl -w net.ipv4.tcp_tw_reuse=0
 * 23:25 paladox: !log root@db4:/home/paladox# sysctl -w net.ipv4.tcp_fin_timeout=60
 * 23:01 paladox: root@db4:/home/paladox# sysctl -w net.ipv4.tcp_tw_recycle=0
 * 22:58 paladox: root@db4:/home/paladox# sysctl -w net.ipv4.tcp_tw_recycle=1
 * 22:52 paladox: root@db4:/home/paladox# sysctl -w net.ipv4.tcp_fin_timeout=3
 * 22:52 paladox: root@db4:/home/paladox# sysctl -w net.ipv4.tcp_tw_reuse=1
 * 22:47 paladox: root@db4:/home/paladox# sysctl net.ipv4.tcp_fin_timeout=3
 * 22:44 paladox: restart php.7-3-fpm on mw* & lizardfs*
 * 22:42 paladox: root@db4:/home/paladox# sysctl net.ipv4.tcp_tw_reuse=1
 * 16:04 Reception123: reception@mw2:~/graalmilitary$ sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=gmcwiki --search-recursively --skip-dupes /home/reception/graalmilitary/wikipics
 * 11:48 Reception123: moved simswiki from db4 to db5

2020-02-26

 * 23:52 paladox: root@cloud2:/var/lib/vz/images/113# rm vm-113-disk-0.qcow2
 * 23:45 paladox: root@cloud2:/var/lib/vz/images/113# qemu-img convert vm-113-disk-0.qcow2 vm-113-disk-0.raw
 * 23:25 paladox: root@bacula2:/home/paladox# ip addr del fe80::f816:3eff:fe31:f5f/64 dev ens3
 * 23:25 paladox: root@bacula2:/home/paladox# ip -6 address add 2604:180:f3::382/64 dev ens3
 * 23:18 paladox: apt-get dist-upgrade - phab1
 * 20:06 Reception123: move the previous import to jobrunner1 (per redis issues)
 * 17:06 paladox: removing old kernel from cp3 using sudo apt autoremove
 * 17:04 paladox: reboot cp3 - network felt slow
 * 07:26 Reception123: reception@mw1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki bigbrotherwikiwiki /home/reception/bigbrother_pages_full.xml
 * 01:56 paladox: root@cloud2:/var/lib/vz/images/115# qemu-img convert vm-115-disk-0.qcow2 vm-115-disk-0.raw
 * 01:45 paladox: root@cloud1:/var/lib/vz/images/111# qemu-img convert vm-111-disk-0.qcow2 vm-111-disk-0.raw
 * 01:19 paladox: that's on cp3
 * 01:19 paladox: apt-get install linux-image-amd64
 * 01:17 paladox: apt-get upgrade - cp3
 * 00:57 paladox: varnish> ban req.http.Host == static.miraheze.org (cp8|cp4)
 * 00:45 paladox: varnish> ban req.http.Host == messengergeek.miraheze.org (cp4)
 * 00:36 paladox: varnish> ban req.http.Host == messengergeek.miraheze.org (cp8)
 * 00:25 paladox: root@cloud1:/var/lib/vz/images/107# qemu-img convert vm-107-disk-0.qcow2 vm-107-disk-0.raw
 * 00:15 paladox: root@cloud1:/var/lib/vz/images/100# qemu-img convert vm-100-disk-0.qcow2 vm-100-disk-0.raw
 * 00:02 paladox: root@cloud1:/var/lib/vz/images/106# qemu-img convert vm-106-disk-0.qcow2 vm-106-disk-0.raw

2020-02-25

 * 23:55 paladox: actually 3 in total
 * 23:52 paladox: increased jobrunner1 core by 1 (so 4 in total)
 * 21:33 paladox: root@cloud1:/var/lib/vz/images/112# qemu-img convert vm-112-disk-0.qcow2 vm-112-disk-0.raw
 * 21:20 paladox: root@cloud2:/var/lib/vz/images/119# qemu-img convert vm-119-disk-0.qcow2 vm-119-disk-0.raw
 * 21:18 paladox: root@cloud2:/var/lib/vz/images/104# rm vm-104-disk-0.qcow2
 * 20:15 paladox: root@db4:/var/log/mysql# sysctl net.ipv4.tcp_tw_reuse=0
 * 20:15 paladox: root@db4:/var/log/mysql# sysctl net.ipv4.tcp_fin_timeout=60
 * 19:52 paladox: root@cp7:/home/paladox# sysctl net.core.somaxconn=4000
 * 19:49 paladox: root@db4:/var/log/mysql# sysctl net.ipv4.tcp_fin_timeout=3
 * 19:47 paladox: root@db4:/var/log/mysql# sysctl net.ipv4.tcp_tw_reuse=1
 * 18:06 paladox: root@cloud2:/var/lib/vz/images/104# qemu-img convert vm-104-disk-0.qcow2 vm-104-disk-0.raw
 * 18:03 paladox: root@cloud1:/var/lib/vz/images/102# rm vm-102-disk-0.qcow2 (it's been converted to raw)
 * 17:48 paladox: converting puppet2 to a raw image (qemu-img convert vm-102-disk-0.qcow2 vm-102-disk-0.raw)
 * 17:42 paladox: downgrade puppet2 ram back to 4g
 * 17:33 paladox: increase puppet2 ram by 1gb (so 5g in total)

2020-02-24

 * 23:28 paladox: root@mw4:/var/log# sysctl net.core.somaxconn=512
 * 22:06 Southparkfan: restarted php-fpm again
 * 22:01 Southparkfan: restart php-fpm again (on mon1)
 * 21:58 Southparkfan: restart php-fpm on mon1
 * 18:59 paladox: apt-get upgrade - mon1
 * 17:40 paladox: set global read_only=0; - db6
 * 17:15 paladox: set read_only on db6
 * 15:04 paladox: MariaDB [metawiki]> set global table_definition_cache=40000; - db6
 * 15:04 paladox: MariaDB [metawiki]> set global table_open_cache=50000; - db6
 * 15:04 paladox: MariaDB [metawiki]> set global table_definition_cache=40000; - db5
 * 15:04 paladox: MariaDB [metawiki]> set global table_open_cache=50000; - db5
 * 14:43 paladox: restart php7.3-fpm on mw* and lizardfs6
 * 14:42 paladox: MariaDB [metawiki]> set global table_definition_cache=40000; - db4
 * 14:42 paladox: MariaDB [metawiki]> set global table_open_cache=50000; - db4
 * 01:02 paladox: restarted mariadb - db4 as ran out of space

2020-02-23

 * 22:27 paladox: root@mw4:/srv/mediawiki/w/maintenance# sudo -u www-data php resetUserTokens.php --wiki=onepiecebountyrushwiki
 * 21:15 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php updateArticleCount.php --wiki=hispanowiki --update
 * 21:14 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php updateArticleCount.php --wiki=ucroniaswiki --update
 * 20:57 paladox: root@cloud2:/home/paladox# swapoff -a
 * 20:56 paladox: root@cloud1:/home/paladox# swapoff -a
 * 20:50 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php updateArticleCount.php --wiki=hispanowiki --update

2020-02-22

 * 02:55 paladox: drop grafana db on db4
 * 02:49 paladox: apt-get install puppet-agent mariadb-client-10.1 mariadb-client-core-10.1 mariadb-common - misc2
 * 02:43 paladox: remove php/nginx from misc2
 * 01:45 paladox: drop database piwik on db5

2020-02-21

 * 23:31 paladox: migrating matomo to new infra
 * 21:11 paladox: root@mw1:/srv/mediawiki/w/extensions/ManageWiki/maintenance# sudo -u www-data php populateNamespacesWithDefaults.php --wiki=unicodesubsetswiki --overwrite
 * 21:10 paladox: MariaDB [mhglobal]> delete from mw_namespaces where ns_dbname = 'unicodesubsetswiki';
 * 21:09 paladox: root@mw1:/srv/mediawiki/w/extensions/ManageWiki/maintenance# sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki=unicodesubsetswiki --overwrite
 * 10:04 Reception123: RENAME browndustwiki to braveninewiki (db + script)

2020-02-20

 * 01:02 paladox: CHANGE MASTER TO MASTER_HOST='db6.miraheze.org', MASTER_USER='replica', MASTER_PASSWORD='xxxx', MASTER_LOG_FILE='mysql-bin.000004', MASTER_LOG_POS=1044075691; - db6
 * 00:51 paladox: GRANT REPLICATION SLAVE ON *.* TO 'replica'@'%' IDENTIFIED BY 'xxxx'; - db6

2020-02-18

 * 20:54 paladox: "ip addr del fe80::216:3cff:fe25:9a18/64 dev ens3" - db4
 * 20:54 paladox: "ip -6 address add 2a00:d880:5:d6c::2/64 dev ens3" - db4
 * 20:25 paladox: added www-data to ssl-cert on mw[23] lizardfs6
 * 20:19 paladox: fixed perms on /etc/ssl/private on lizardfs6
 * 19:50 paladox: restart redis to see if it will fix the 503s
 * 19:33 paladox: restart php7.3-fpm on mw* and lizardfs6
 * 19:32 paladox: restart php7.3-fpm on mw1
 * 19:11 paladox: enable puppet on jobrunner1
 * 16:44 paladox: upgrade mariadb-client on misc1
 * 16:43 paladox: upgrade grafana on misc1
 * 15:10 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10784;
 * 15:10 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10783;
 * 15:10 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10782;
 * 15:09 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10781;
 * 15:08 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10780;
 * 15:08 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10778;
 * 15:07 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10777;
 * 15:07 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10776;
 * 15:04 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10775;
 * 15:04 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10765;

2020-02-17

 * 16:46 paladox: restart nginx on cp4
 * 16:24 paladox: ALTER TABLE /*_*/oathauth_users DROP COLUMN secret, DROP COLUMN scratch_tokens;
 * 16:12 paladox: renaming kujumwiki to archlinuxfamilyorgwiki per T5241
 * 16:04 paladox: renaming endeavourosdewiki to archlinuxfamilywiki per T5242
 * 15:39 paladox: sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki=pcsdscotlandwiki --overwrite
 * 12:50 Reception123: disabled PlavorSeol on Phabricator
 * 12:04 Reception123: renamed ungamewiki to libertygamewiki (db + script)

2020-02-16

 * 23:52 paladox: MariaDB [allthetropeswiki]> truncate objectcache;
 * 22:56 Southparkfan: set read_only=1 on db6
 * 20:31 Reception123: re-inserted cw_requests from db6 after misclick
 * 19:29 Reception123: removed spam from cw_requests (inappropriate comment)
 * 14:49 paladox: [14:43:09] <+paladox>	!log hacked lizardfs6 Database.php (added persitents mode and removed)
 * 14:44 paladox: hacked lizardfs6 Database.php (added persitents mode and removed)
 * 14:43 paladox: [14:41:46] <+paladox>	!log reboot lizardfs6
 * 14:31 paladox: restart php on mw[123]

2020-02-15

 * 19:48 paladox: depooled mw[4567]
 * 18:41 paladox: repool cp4

2020-02-14

 * 20:28 paladox: backing up phabricator db
 * 20:04 Southparkfan: install atop/iotop/iperf (with fw rule for latter) on db4/db6
 * 19:34 paladox: [19:11:31] <+paladox>	!log set gluster file system to read only on lizardfs6
 * 19:26 Southparkfan: added firewall rule on db4 to allow traffic from db6 (v4/v6) to port 3306
 * 19:26 Southparkfan: dynamically set server_id=4 on db4 and server_id=6 on db6
 * 19:26 Southparkfan: dumping db4's databases to db6 in /home/dbcopy
 * 18:52 Southparkfan: disable puppet on db[456]
 * 14:45 paladox: tried upgrading phabricator to latest on master, failed and rollbacked
 * 10:59 Reception123: umount /mnt/mediawiki-static ; mount /mnt/mediawiki-static on mw3
 * 02:35 paladox: stopped parsoid, removed /srv/parsoid from misc4
 * 02:33 paladox: apt-get upgrade - misc4
 * 02:25 paladox: increase puppet2 disk to 15G
 * 01:42 Southparkfan: decommissioned misc3
 * 01:33 Southparkfan: shutdown misc3 - testing before permanent decom
 * 01:27 paladox: drop phabricator_* on db4
 * 01:23 Southparkfan: decommission cp2
 * 00:56 John: soft depooling of misc1 as ns2 via DNS entry change
 * 00:53 Southparkfan: shutdown cp2 nginx for transferring nginx logs to cp7:/root

2020-02-13

 * 22:05 paladox|UKInEU: install parted on phab1
 * 22:02 paladox|UKInEU: increase phab1 disk to 20g
 * 21:31 paladox|UKInEU: increased phab1 cores to 2
 * 20:45 paladox|UKInEU: install mariadb-client-10.1 on misc4 (to backup the db)
 * 20:43 paladox|UKInEU: stop phd & nginx on misc4. Dump phabricator db, migrate to db6 and migrate phabricator to phab1
 * 20:15 paladox|UKInEU: restart php7.3-fpm on mw3
 * 18:33 paladox|UKInEU: restarted nginx on mw3
 * 01:17 paladox|UKInEU: installed rsync on gluster1
 * 00:04 paladox|UKInEU: reboot test1

2020-02-12

 * 23:44 paladox|UKInEU: lizardfs6 not 67
 * 23:43 paladox|UKInEU: apt-get upgrade - lizardfs67
 * 22:34 paladox|UKInEU: beginning migration from lizardfs6 to gluster1
 * 20:42 Zppix: create more various dummy dbs to prevent issues with wiki creations
 * 16:10 Reception123: move toxicfandomsandhatedomswiki from db4 to db5
 * 15:48 Reception123: moved zhdelwiki from db4 to db5
 * 15:32 Reception123: PURGE BINARY LOGS BEFORE '2020-02-12 12:00:00'; on db4

2020-02-11

 * 23:13 Zppix: created various dummy dbs using sql.php on mw1 to prevent creation of new wikis with existing hostnames to prevent potential issues
 * 01:51 paladox|UKInEU: reboot cp2
 * 01:33 paladox|UKInEU: MariaDB [mhglobal]> update mw_namespaces set ns_additional = '{"wgNamespacesToPostIn":true,"wgVisualEditorAvailableNamespaces":true}' where ns_dbname = 'civclassicwiki' and ns_namespace_name = '';
 * 01:01 paladox|UKInEU: rm /srv/mediawiki/w/cache/managewiki/** on mw[123], lizardfs6

2020-02-10

 * 19:42 Reception123: MariaDB [(none)]> DROP DATABASE isabellafan2wiki;
 * 19:42 Reception123: deleted isabellafan2wiki
 * 02:22 paladox|UKInEU: probe gluster1 from lizardfs(gluster) but do not add it as a brick

2020-02-08

 * 15:29 paladox|UKInEU: rename diaroleplayswiki to sparkleswiki per https://phabricator.miraheze.org/T5188
 * 15:27 paladox|UKInEU: MariaDB [shiropediawiki]> delete from nl_newsletters where nl_id = 5;
 * 15:26 paladox|UKInEU: MariaDB [shiropediawiki]> delete from nl_newsletters where nl_id = 6;
 * 15:20 paladox|UKInEU: renaming idiotpediawiki to nocyclopediawiki per https://phabricator.miraheze.org/T5208
 * 15:15 paladox|UKInEU: delete & drop ewrikiawiki per https://phabricator.miraheze.org/T5213
 * 15:14 paladox|UKInEU: drop cdwwik, gameshowswiki, glaucuswiki, receptionflamewarswiki, stargazerwikiwiki, thechasewikiwiki, toslawwiki
 * 10:14 Reception123: PURGE BINARY LOGS BEFORE '2020-02-08 08:00';

2020-02-07

 * 19:22 Zppix: create mw[67]wiki using sql.php
 * 19:21 Zppix: created cloud1 and cloud2 dbs using sql.php

2020-02-06

 * 01:41 paladox|UKInEU: reboot mw3

2020-02-05

 * 19:49 paladox|UKInEU: upgrade php on lizardfs6
 * 19:47 paladox|UKInEU: upgrade php on mw1
 * 19:46 paladox|UKInEU: upgrade php on mw2
 * 19:43 paladox|UKInEU: upgrade php on mw3
 * 19:39 paladox|UKInEU: restart php7.3-fpm on mw3
 * 19:31 Zppix: initsitestats and rebuildrc for ungamewiki after import

2020-02-04

 * 20:47 Southparkfan: locked phab auth config
 * 20:41 Southparkfan: unlocked phab auth config (temp)

2020-02-03

 * 17:28 paladox|UKInEU: root@mw1:/var/log/mediawiki# rm *.log.*
 * 17:27 paladox|UKInEU: rm /var/log/mediawiki/*.log*
 * 17:26 paladox|UKInEU: rm /var/log/mediawiki/*.log

2020-02-02

 * 15:50 paladox|UKInEU: > MatomoAnalytics::addSite( 'loginwiki' );
 * 15:50 paladox|UKInEU: > MatomoAnalytics::deleteSite( 'loginwiki' );
 * 15:48 paladox|UKInEU: > MatomoAnalytics::addSite( 'foxmaskeventwiki' );
 * 15:48 paladox|UKInEU: > MatomoAnalytics::deleteSite( 'foxmaskeventwiki' );
 * 15:48 paladox|UKInEU: > MatomoAnalytics::addSite( 'davshiqwiki' );
 * 15:48 paladox|UKInEU: > MatomoAnalytics::deleteSite( 'davshiqwiki' );
 * 14:03 paladox|UKInEU: remove hack from cron - db5
 * 04:34 paladox|UKInEU: hack /var/spool/cron/crontabs/root and run /home/paladox/purge-binary.sh every 1 minute.
 * 00:38 paladox|UKInEU: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php rebuildall.php --wiki=csydeswiki
 * 00:02 paladox|UKInEU: drop database piwik on db5; create databse db5 & import from backup. To see if this will reduce the disk space it's using from ~75gb.

2020-02-01

 * 23:19 paladox|UKInEU: drop database uncyclomirrorwiki on db4 as migrated to db5
 * 22:29 paladox|UKInEU: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=metawiki --type=gzip
 * 22:23 paladox|UKInEU: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=toxicfandomsandhatedomswiki --type=gzip
 * 22:18 paladox|UKInEU: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=uncyclomirrorwiki --type=gzip
 * 22:14 paladox|UKInEU: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=altversewiki --type=gzip
 * 12:26 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki Tundra "Just Amelia" --logwiki=metawiki

2020-01-31

 * 16:48 paladox|UKInEU: MariaDB [gmcwiki]> INSERT INTO content_models (model_id,model_name) VALUES (4,'Scribunto');
 * 16:47 paladox|UKInEU: MariaDB [gmcwiki]> INSERT INTO content_models (model_id,model_name) VALUES (2,'css');
 * 16:47 paladox|UKInEU: MariaDB [gmcwiki]> INSERT INTO content_models (model_id,model_name) VALUES (3,'javascript');
 * 16:47 paladox|UKInEU: MariaDB [gmcwiki]> INSERT INTO content_models (model_id,model_name) VALUES (5,'json');
 * 16:46 paladox|UKInEU: MariaDB [gmcwiki]> INSERT INTO content_models (model_id,model_name) VALUES (1,'wikitext');
 * 16:43 paladox|UKInEU: MariaDB [gmcwiki]> INSERT INTO slot_roles (role_id,role_name) VALUES (1,'main');
 * 16:37 paladox|UKInEU: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php cleanupUsersWithNoId.php --wiki=gmcwiki --force -p gmcwiki

2020-01-30

 * 18:02 paladox: apt-get upgrade - mw[123] (gluster, php and puppet-agent + few other packages)
 * 17:55 paladox: apt-get upgrade - lizardfs6 (gluster, php and puppet-agent + few other packages)

2020-01-28

 * 23:42 paladox: upgrade puppet-agent & salt on misc1
 * 23:41 paladox: upgrade php - misc1
 * 23:36 paladox: upgrade grafana - misc1
 * 22:00 paladox: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=ungamewiki --type=gzip
 * 21:58 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/guiasdobrasil.com.br "', null ) WHERE wiki_dbname = 'guiaslocaiswiki';
 * 19:32 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php nukeNS.php --wiki=thefinalrumblewiki --all --ns=3001 --delete
 * 13:13 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki Toshinori "Agent Mace" --logwiki=metawiki
 * 00:53 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=files
 * 00:52 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=subcats
 * 00:52 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=pages

2020-01-27

 * 22:59 paladox: sync; echo 1 > /proc/sys/vm/drop_caches (lizardfs6 & db4)
 * 22:50 paladox: sudo /sbin/sysctl -w vm.drop_caches=3 on lizardfs6 and db4 temporarily
 * 22:28 Zppix: zppix@mw1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --report=1 --no-local-users --username-prefix "백괴게임" --wiki ungamewiki /home/zppix/uncyclogame-full-2019-05-15.xml (T5148)
 * 20:02 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php deleteBatch.php --wiki=thefinalrumblewiki list (Deleting Message_Wall:%)
 * 19:57 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php deleteBatch.php --wiki=thefinalrumblewiki list2 (Deleting User_blog_comment:%)
 * 19:55 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php deleteBatch.php --wiki=thefinalrumblewiki list (Deleting User_blog:%)
 * 19:51 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php nukeNS.php --wiki=thefinalrumblewiki --all --ns=3000 --delete
 * 17:28 paladox: root@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance# sudo -u www-data php migrateAccount.php --wiki=wikifinanzawiki --userlist=./userlist --homewiki=wikifinanzawiki --attachmissing --auto
 * 15:26 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki ZapperGlaze49 "TheChisatoFan" --logwiki=metawiki

2020-01-26

 * 22:15 paladox: revert hack on mw[123] & lizardfs6
 * 20:12 paladox: hacked LS on mw[123] & lizardfs6 to switch off wgUseInstantCommons temporarily
 * 17:51 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=pages/subpages/files
 * 17:41 paladox: root@mw2:/srv/mediawiki/w/maintenance# sudo -u www-data php refreshLinks.php --wiki=ucroniaswiki
 * 17:41 paladox: root@mw2:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=pages
 * 16:48 paladox: revert hack on mw[123] & lizardfs6
 * 15:18 paladox: hacked LS on mw[23] & lizardfs6 to switch off wgUseInstantCommons temporarily
 * 15:17 paladox: hacked LS on mw1 to switch off wgUseInstantCommons temporarily
 * 15:06 paladox: restart php7.3-fpm on mw1 and mw3
 * 15:05 paladox: restart php7.3-fpm on mw2 and lizardfs6
 * 06:37 Reception123: deleted and dropped gmcwiki (T5150)

2020-01-25

 * 18:40 Zppix: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki crappygameswiki -u "Zppix" -r "Requested - T5153" /home/zppix/crappygameswiki_batchdelete.txt
 * 18:21 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki TTyranny "Tarbiz" --logwiki=metawiki
 * 17:52 paladox: reinstall php on misc2
 * 17:47 paladox: rotate redis.log on misc2
 * 15:02 paladox: restart php7.3-fpm && nutcracker on mw3
 * 15:00 paladox: restart php7.3-fpm && nutcracker on mw2
 * 00:16 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/wiki.udia.ca "', null ) WHERE wiki_dbname = 'udiawiki';
 * 00:15 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/islamkosh.tk "', null ) WHERE wiki_dbname = 'ipediawiki';
 * 00:15 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/glp.ink "', null ) WHERE wiki_dbname = 'cnwiki';
 * 00:14 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/wiki.counterculturelabs.org "', null ) WHERE wiki_dbname = 'cclwiki';

2020-01-24

 * 23:28 paladox: restart php7.3-fpm on mw3 and lizardfs6
 * 23:28 paladox: restart php7.3-fpm on mw2
 * 23:27 paladox: restart php7.3-fpm on mw1
 * 14:12 paladox: MariaDB [mhglobal]> update mw_permissions set perm_removegroupsfromself = '[]' where perm_removegroupsfromself = 'null';
 * 14:10 paladox: Remove abusefilter-hide-log,abusefilter-hidden-log,abusefilter-modify-global,abusefilter-private,abusefilter-private-log,aft-oversighter,autocreateaccount,bigdelete,centralauth-lock,centralauth-oversight,centralauth-rename,centralauth-unmerge,centralauth-usermerge,checkuser,checkuser-log,createwiki,editincidents,editothersprofiles,editothersprofiles-private,flow-suppress,globalblock,globalblock-exempt,globalgroupmembership,globalgrouppermissions,hideuser,interwiki,managewiki-restricted,managewiki-editdefault,moderation-checkuser,mwoauthmanageconsumer,mwoauthmanagemygrants,mwoauthsuppress,mwoauthviewprivate,mwoauthviewsuppressed,oathauth-disable-for-user,oathauth-view-logrenameuser,requestwiki,siteadmin,stopforumspam,suppressionlog,suppressrevision,usermerge,userrights,userrights-interwiki,viewglobalprivatefiles,viewpmlog,viewsuppressed from all wikis.
 * 14:04 paladox: MariaDB [mhglobal]> update mw_permissions set perm_addgroupstoself = '[]' where perm_addgroupstoself = 'null';
 * 14:02 paladox: rm -rf /srv/mediawiki/w/cache/managewiki/** on mw* and lizardfs6
 * 14:01 paladox: MariaDB [mhglobal]> update mw_permissions set perm_removegroupsfromself = '[]' where perm_removegroupsfromself = 'null';
 * 14:01 paladox: MariaDB [mhglobal]> update mw_permissions set perm_removegroups = '[]' where perm_removegroups = 'null';
 * 14:01 paladox: MariaDB [mhglobal]> update mw_permissions set perm_addgroups = '[]' where perm_addgroups = 'null';
 * 13:59 paladox: MariaDB [mhglobal]> update mw_permissions set perm_addgroupstoself = '[]' where perm_addgroupstoself = 'null';
 * 13:38 paladox: remove oathauth-disable-for-user from all groups.

2020-01-23

 * 00:29 paladox_UK_IN_EU: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=spectrobeswiki --search-recursively --skip-dupes ./Spectrobes_Wiki_Images

2020-01-22

 * 23:54 paladox: apt-get upgrade cp3
 * 22:51 paladox: remove oathauth-disable-for-user from all wikis
 * 19:29 paladox_UK_IN_EU: remove oauthauth-disable-for-user from all wikis
 * 18:31 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=spectrobeswiki --report=1 spectrobes_pages_full.xml - per request through email
 * 18:31 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=spectrobeswiki --report=1 spectrobes_pages_full.xml.7z - per request through email
 * 06:52 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki WageUglyDollandMarlaPlaymobil12 "Just Wage" --logwiki=metawiki

2020-01-21

 * 21:29 paladox: restart nutcracker on mw1
 * 14:53 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wiesepediawiki /home/reception/Wiesepedia-Datadump[0,A-D] on mw1
 * 14:47 Reception123: PURGE BINARY LOGS before '2020-01-20 22:00:00' on db4
 * 14:46 Reception123: manually remove custom domain for hypewiki
 * 14:37 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=gmcwiki /home/reception/graalmilitary_pages_full.xml on mw3
 * 14:33 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=spectrobeswiki /home/reception/spectrobes_pages_full.xml on mw3

2020-01-20

 * 17:05 paladox: ALTER TABLE /*_*/searchindex MODIFY COLUMN si_title TEXT BINARY NOT NULL; - simswiki
 * 16:59 paladox: ran ./2.sh on db4 (basically finding all titles that are in User Blog Comment on wikia and setting ns id to 3007 from 0)
 * 16:54 Zppix: UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/nonsensopedia.org "', null ) WHERE wiki_dbname = 'nonsensopediawiki'; on mw1 using sql.php
 * 01:07 paladox: ALTER TABLE /*_*/searchindex MODIFY COLUMN si_title varchar(300) BINARY NOT NULL default ''; - simswiki (applying
 * 00:49 paladox: MariaDB [simswiki]> update page set page_namespace = 3007 where page_namespace = 3008;
 * 00:42 paladox: remove hack from mw1
 * 00:39 paladox: ALTER TABLE /*_*/searchindex MODIFY COLUMN si_title TEXT BINARY NOT NULL; (applying https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/565765/8 to simswiki)
 * 00:11 paladox: MariaDB [simswiki]> update page set page_namespace = 3008 where page_namespace = 3007;

2020-01-19

 * 22:30 paladox: hacked rebuildtextindex.php on mw1 to output some debugging info
 * 21:45 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php rebuildall.php --wiki=simswiki
 * 21:44 paladox: ALTER TABLE /*_*/searchindex MODIFY COLUMN si_title varchar(255) BINARY NOT NULL default ''; - simswiki (fixing an issue when rebuilding search index)
 * 21:38 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=hispanowiki batch2 - T5122
 * 21:37 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=ucroniaswiki batch - T5122
 * 21:34 paladox: root@mw1:/srv/mediawiki/w/extensions/ManageWiki/maintenance# sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki=toastycraftwiki --overwrite -T5119
 * 21:34 paladox: root@mw1:/srv/mediawiki/w/extensions/ManageWiki/maintenance# sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki=aranziaronzowiki --overwrite -T5119
 * 17:45 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=hispanowiki batch2
 * 17:34 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=ucroniaswiki batch
 * 15:05 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=loginwiki --report=1 Wikipedia-20200119150417.xml
 * 14:57 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=loginwiki --report=1 Wikipedia-20200119145611.xml
 * 13:54 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php rebuildall.php --wiki=simswiki
 * 08:51 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki=thefinalrumblewiki /home/reception/deletebatch.txt
 * 01:23 paladox: restart php7.3-fpm on lizardfs6
 * 00:25 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=thefinalrumblewiki --skip-dupes --search-recursively TFR_Wikia_Images/ (per request on discord)

2020-01-18

 * 22:15 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=simswiki --report=1 --no-updates sims_pages_full.xml
 * 22:09 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php deleteBatch.php --wiki=thefinalrumblewiki batch - per request on discord
 * 20:46 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=simswiki --report=1 sims_pages_full.xml
 * 17:09 paladox: that script is adding "urlshortener-create-url" to * for all wikis
 * 17:08 paladox: root@mw1:/home/paladox# ./foreachwikiindblist /srv/mediawiki/dblist/all.dblist
 * 16:58 paladox: delete from mw_permissions where perm_dbname = '131parkhurstwiki' and perm_group = "all.dblist";
 * 15:44 paladox: delete from mw_permissions where perm_dbname = '131parkhurstwiki' and perm_group = "\\*" LIMIT 1;
 * 15:44 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = "\\*" LIMIT 1;
 * 15:44 paladox: delete from mw_permissions where perm_dbname = '131parkhurstwiki' and perm_group = "\\*" LIMIT 1;
 * 15:44 paladox: delete from mw_permissions where perm_dbname = '131parkhurstwiki' and perm_group = "\\*" LIMIT 1;
 * 15:44 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = "\\*" LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = "\*" LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = '\*' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = '+' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = '+' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '1920swiki' and perm_group = '+' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '1920swiki' and perm_group = '"*"' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '131parkhurstwiki' and perm_group = '"*"' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '131parkhurstwiki' and perm_group = '\*' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = '\*' LIMIT 1;
 * 15:42 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = '"*"' LIMIT 1;
 * 14:16 Reception123: manually remove custom domain from teessidehackspacewiki

2020-01-17

 * 19:10 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=thefinalrumblewiki /home/reception/thefinalrumble_pages_full.xml
 * 02:11 paladox: rm -rf /srv/mediawiki/w/cache/manage*/** on mw* and lizardfs6
 * 01:48 paladox: restoring mw_permissions from a newish backup (after running "delete from mw_permissions where perm_group = "\*";" which seemed to updated 4,000 results)
 * 01:43 paladox: update mw_permissions set perm_permissions = '["autocreateaccount","createaccount","edit","createpage","createtalk", "writeapi","viewmywatchlist","editmywatchlist","viewmyprivateinfo", "editmyprivateinfo","editmyoptions","abusefilter-log","abusefilter-log-detail", "abusefilter-view","autocreateaccount","centralauth-merge", "oathauth-enable","urlshortener-create-url"]' where perm_group = '*' and perm_dbname = 'default';
 * 00:12 paladox: apt-get upgrade - misc2

2020-01-16

 * 23:53 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/ManageWiki/maintenance/modifyGroupPermission.php --addperms="urlshortener-create-url,urlshortener-manage-url" sysop
 * 23:47 paladox: update mw_permissions set perm_permissions = '["editsitecss","upload_by_url","editsitejson","editsitejs","skipcaptcha","block","createaccount","delete","deletedhistory","deletedtext","undelete","editinterface","editusercss","edituserjson","edituserjs","import","importupload","move","move-subpages","move-rootuserpages","move-categorypages","patrol","autopatrol","protect","editprotected","rollback","upload","reupload","reupload-shared","unwatchedpages","autoconfirmed","editsemiprotected","ipblock-exempt","blockemail","markbotedits","apihighlimits","browsearchive","noratelimit","movefile","unblockself","suppressredirect","mergehistory","managechangetags","deletechangetags","abusefilter-modify","abusefilter-modify-restricted","abusefilter-revert","deletelogentry","deleterevision","abusefilter-log-detail","override-antispoof","globalblock-whitelist","massmessage","nuke","tboverride","titleblacklistlog", "delete-dump","generate-dump","view-dump","urlshortener-create-url","urlshortener-manage-url"]' where perm_group = 'sysop' and perm_dbname = 'default';
 * 02:34 paladox: reboot misc4
 * 02:15 paladox: downgraded puppetdb on puppet1 - broke ssl connection to postgresql
 * 01:48 paladox: root@db4:/home/paladox# sudo service postgresql restart
 * 01:44 paladox: on puppet1
 * 01:44 paladox: apt-get upgrade puppet-agent puppetdb puppetdb-termini puppetserver salt-common salt-minion
 * 01:43 paladox: upgrade puppet-agent on cp4

2020-01-15

 * 08:26 Reception123: sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki AnimalLovingGuy "ScrewSociety" --logwiki=metawiki

2020-01-14

 * 23:10 paladox: reinstall cp2 as debian buster
 * 23:09 paladox: restart php7.3-fpm on mw2 for config change
 * 23:09 paladox: turn TUN off on cp2
 * 23:06 paladox: restart php7.3-fpm on mw3 for config change
 * 22:59 paladox: rebuild lc on mw* and lizardfs6
 * 17:39 paladox: apt-get upgrade - lizardfs6
 * 01:43 paladox: sudo -u www-data php update.php --wiki=sagan4wiki --quick --force - mw1
 * 01:21 paladox: MariaDB [sagan4wiki]> ALTER TABLE /*_*/cargo_tables ADD field_helper_tables text NOT NULL;

2020-01-13

 * 15:32 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/wintermoor.net "', null ) WHERE wiki_dbname = 'wintermoorwiki';

2020-01-12

 * 23:29 paladox: removed redis from all mediawiki servers
 * 15:25 paladox: repooled lizardfs6
 * 15:15 paladox: temporarily depool lizardfs6
 * 08:32 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php purgeList.php --all --purge --wiki twinkletestwiki + rebuildall.php
 * 08:06 Reception123: ran rebuildall on twinkletestwiki

2020-01-11

 * 23:35 paladox: [23:30:52] <+paladox>	!log switched TUN off on misc2
 * 23:34 paladox: apt-get upgrade on misc2
 * 22:52 paladox: upgrade nodejs on misc3
 * 22:48 paladox: resizing misc3 - https://clientarea.ramnode.com/index.php?rp=/announcements/431/More-SSD-for-Standard-KVM.html
 * 17:04 paladox: apt-get upgrade - bacula1
 * 16:58 paladox: root@bacula1:/bacula/backup# rm DB*
 * 16:57 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* and lizardfs6
 * 14:59 Reception123: DELETED and DROPPED dbs https://phabricator.miraheze.org/P247
 * 14:52 Reception123: reception@mw1:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/deleteWikis.php --wiki loginwiki --delete Reception123
 * 11:12 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki GasMask0217 "KP619" --logwiki=metawiki
 * 10:29 Reception123: run fixStuckGlobalRename.php on all wikis for username above
 * 10:18 Reception123: sudo -u www-data php fixStuckGlobalRename.php --wiki=aawikiwiki PaperClip "FireBarrier101" --logwiki=metawiki
 * 04:25 paladox: repool lizardfs6
 * 04:18 paladox: depool lizardfs6
 * 04:18 paladox: repool mw3
 * 04:12 paladox: depool mw3
 * 04:12 paladox: repool mw2
 * 04:08 paladox: depool mw2
 * 04:07 paladox: repool mw1
 * 04:04 paladox: depool mw1

2020-01-09

 * 22:07 paladox: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=inforevivalwiki --report=1 Wiki-20200109132453.xml
 * 21:20 paladox: rebuild lc on mw* and lizardfs6
 * 19:34 paladox: ALTER TABLE /*$wgDBprefix*/matomo ADD INDEX matomo_wiki (matomo_wiki); - db4
 * 19:17 paladox: MariaDB [mhglobal]> ALTER TABLE /*$wgDBprefix*/cw_wikis ADD INDEX wiki_dbname (wiki_dbname); -db4
 * 18:59 paladox: MariaDB [mhglobal]> ALTER TABLE /*$wgDBprefix*/mw_namespaces ADD INDEX ns_dbname (ns_dbname); -db4
 * 18:59 paladox: MariaDB [mhglobal]> ALTER TABLE /*$wgDBprefix*/mw_permissions ADD INDEX perm_dbname (perm_dbname); - db4
 * 00:57 paladox: restart php7.3-fpm on mw2

2020-01-08

 * 21:37 paladox: restart mysql on db4
 * 21:33 paladox: restart mysql on db5
 * 19:56 paladox: restart php7.3-fpm on lizardfs6 and mw*
 * 17:16 paladox: SET GLOBAL thread_cache_size = 100; - (db4|db5)
 * 16:52 paladox: MariaDB [(none)]> SET GLOBAL thread_cache_size = 80;
 * 16:33 paladox: MariaDB [(none)]> SET GLOBAL thread_pool_stall_limit = 10;
 * 16:14 paladox: MariaDB [(none)]> SET GLOBAL slow_query_log = 'OFF';
 * 16:10 paladox: MariaDB [(none)]> SET GLOBAL long_query_time = 10;
 * 16:04 paladox: MariaDB [(none)]> SET GLOBAL long_query_time = 15;
 * 16:04 paladox: MariaDB [(none)]> SET GLOBAL slow_query_log = 'ON';
 * 15:30 paladox: restart php7.3-fpm on mw* and lizardfs6
 * 15:21 Reception123: purge binary logs BEFORE '2020-01-08 08:00'

2020-01-07

 * 17:16 paladox: root@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance# sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki --logwiki=metawiki ErtasVideos Hendicted

2020-01-06

 * 20:22 Reception123: reception@mw2:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki testwiki /home/reception/Wikipedia-20200106202002.xml

2020-01-05

 * 19:47 paladox: rebuild lc on mw* and lizardfs6
 * 00:25 paladox: ./bin/auth strip --user Paladox --all-types (misc4)

2020-01-04

 * 10:13 Reception123: reception@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki loginwiki /home/reception/TestWiki-20200104100040.xml

2020-01-02

 * 18:48 paladox: depool, repool and restart php7.3-fpm on mw3
 * 18:44 paladox: depool, repool and restart php7.3-fpm on mw2
 * 18:43 paladox: restart php7.3-fpm on mw1
 * 18:31 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=radioscanningtwwiki --report=1 X.xml
 * 03:10 paladox: rm -rf /srv/mediawiki/w/cache/managewiki/** on mw* and lizardfs6
 * 03:01 paladox: root@test1:/srv/mediawiki/w/extensions/StopForumSpam/maintenance# sudo -u www-data php updateBlacklist.php --wiki=sdiywiki
 * 02:44 paladox: upgrade phabricator on misc4
 * 02:42 paladox: MariaDB [(none)]> drop database uncyclopediakrwiki;
 * 02:30 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/renameWiki.php --wiki=loginwiki --rename uncyclopediakrwiki truthwiki paladox

2020-01-01

 * 17:49 paladox: root@misc1:/home/paladox# deluser labster
 * 15:53 Reception123: sudo -u www-data php fixStuckGlobalRename.php --wiki=200xwiki PaperClip "FireBarrier101" --logwiki=metawiki
 * 14:24 Reception123: MariaDB [phabricator_herald]> UPDATE herald_rule SET isDisabled = '1' WHERE id = '21'; (empty)
 * 04:36 paladox: root@mw1:/srv/mediawiki/w# /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/QuizGame/sql/patches/patch-add-default-choice_answer_count.sql
 * 00:37 paladox: restart varnish cp2,cp4