Tech:Server admin log/2020

2020-05-14

 * 12:08 paladox: reboot mon1
 * 12:08 paladox: reboot rdb1
 * 12:03 paladox: apt-get dist-upgrade - mon1
 * 11:58 Reception123: added slack hook for jungegeotechnikerwiki
 * 11:03 Reception123: ran messagefiles and rebuild LC on mw*/jbr1
 * 08:14 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki awesomegameswiki -r "Requested - T5583" /home/reception/cg2.txt (new)

2020-05-13

 * 17:17 paladox: root@jobrunner1:/home/paladox# /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /home/paladox/dbs /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/populateWikibaseSitesTable.php
 * 17:06 paladox: MariaDB [mhglobal]> update mw_permissions set perm_removegroups = '[]' where perm_removegroups = 'null'; -dbt1
 * 17:05 paladox: MariaDB [mhglobal]> update mw_permissions set perm_addgroups = '[]' where perm_addgroups = 'null'; - dbt1
 * 16:21 paladox: rebuild lc on jobrunner1
 * 16:08 paladox: rebuilding lc on mw*
 * 14:00 paladox: created a guest account in ldap for icinga
 * 08:19 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki awesomegameswiki -r "Requested - T5583" /home/reception/cg2.txt
 * 08:11 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki awesomegameswiki -r "Requested - T5583" /home/reception/cg.txt

2020-05-11

 * 20:57 paladox: rebuild lc on mw*, jobrunner1 and test2
 * 18:14 paladox: increased mon1 disk by 10gb (40gb in total)
 * 08:20 Reception123: restarting icingabot
 * 00:11 paladox: rebuild lc on mw* and jobrunner1

2020-05-10

 * 18:51 Reception123: deleted and dropped https://phabricator.miraheze.org/P312
 * 18:41 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/deleteWikis.php --wiki=loginwiki --delete Reception123
 * 18:20 Reception123: created ldap account for myself
 * 09:51 Reception123: deleted and dropped mrjaroslavikwiki, then recreated

2020-05-09

 * 12:47 Reception123: $cw = new CreateWikiJson( 'pruebawiki' ); $cw->resetWiki;
 * 11:43 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw*/jbr1
 * 07:22 Reception123: renamed artladderwiki to meshwiki.

2020-05-08

 * 23:49 paladox: upgraded puppet-agent on mon1
 * 21:20 paladox: mon1>grafana-cli plugins install grafana-image-renderer

2020-05-07

 * 17:41 Zppix: rebuildLC on jobrunner1
 * 12:02 paladox: install security updates for firefox-esr iceweasel on mw4
 * 12:02 paladox: install security updates for firefox-esr iceweasel on mw5
 * 12:01 paladox: install security updates for firefox-esr iceweasel on mw6
 * 09:14 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw*/jbr1
 * 08:19 paladox: install security updates for firefox-esr iceweasel on mw7

2020-05-06

 * 23:38 paladox: apt-get dist-upgrade - cloud[12] (upgrading proxmox and puppet-agent)
 * 22:34 paladox: depool mw[67], set vcpu to 4 and repool
 * 22:33 paladox: repool mw5
 * 22:30 paladox: set mw5 vcpu to 4
 * 22:30 paladox: depool mw5
 * 22:30 paladox: repool mw4
 * 22:28 paladox: set mw4 vcpu to 4
 * 22:28 paladox: depool mw4
 * 22:21 paladox: restarted prometheus on mon1
 * 22:14 paladox: restart nginx-prometheus-exporter on mw4
 * 21:58 paladox: rebuild lc on mw* and jobrunner1
 * 16:13 paladox: apt-get --purge remove apache2* - ns1
 * 14:34 paladox: installed libgraphicsmagick++-q16-12 libgraphicsmagick-q16-3 security updates on mw6
 * 12:32 John: security updates for libgraphicsmagick++-q16-12 and libgraphicsmagick-q16-3
 * 10:58 Reception123: reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php cleanupT                                                                            itles.php --wiki testwiki
 * 05:59 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/generateMirahezeSitemap.php --wiki closinglogosgroupwiki

2020-05-05

 * 20:54 paladox: $cw = new CreateWikiJson( 'infinigwiki' ); $cw->resetWiki;
 * 20:51 paladox: MariaDB [mhglobal]> update mw_permissions set perm_addgroups = '[]' where perm_group = 'member' and perm_dbname = 'infinigwiki';
 * 19:35 Reception123: disabled a user on Phabricator
 * 19:31 Reception123: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_inactive_timestamp = null WHERE wiki_inactive = 0;
 * 19:29 Southparkfan: rebuilding recent changes for atrociousyoutuberswiki
 * 18:12 Reception123: > $cWJ = new CreateWikiJson( "dungeonsanddaddieswiki" ); $cWJ->resetWiki;
 * 17:59 John: reset email on Maestro venerabile
 * 17:46 paladox: MariaDB [atrociousyoutuberswiki]> truncate objectcache;
 * 16:13 paladox: delete redis atrociousyoutuberswiki key
 * 13:33 paladox: MariaDB [atrociousyoutuberswiki]> update recentchanges set rc_ip = '' where rc_ip = '0.0.0.0';
 * 12:57 paladox: update flow_revision set rev_user_ip = NULL where rev_user_ip = '0.0.0.0';
 * 12:57 paladox: update flow_revision set rev_edit_user_ip = NULL where rev_edit_user_ip = '0.0.0.0';
 * 12:57 paladox: update flow_revision set rev_mod_user_ip = NULL where rev_mod_user_ip = '0.0.0.0';
 * 12:57 paladox: update flow_tree_revision set tree_orig_user_ip = NULL where tree_orig_user_ip = '0.0.0.0';
 * 07:26 Reception123: stopped and started ircecho (icinga)

2020-05-04

 * 20:54 paladox: blocked heroku-miraheze from github, confirmed LTA
 * 19:44 Reception123: delete from [localnames/localuser] where [lu_wiki/ ln_wiki] = johnflewiswiki bloxcitywiki brgameswiki columbiasjpwiki xenimuswiki
 * 19:25 paladox: delete from localuser where lu_wiki = 'asylothekberlinwiki'; - dbt1
 * 19:25 paladox: delete from localnames where ln_wiki = 'asylothekberlinwiki'; - dbt1
 * 19:14 paladox: delete from localnames where ln_wiki = 'altenpflegewiki'; - dbt1
 * 19:08 paladox: MariaDB [mhglobal]> delete from localuser where lu_wiki = 'altenpflegewiki'; - dbt1
 * 14:55 paladox: root@jobrunner1:/srv/mediawiki/w/extensions/SocialProfile# sudo -u www-data php UserBoard/maintenance/migrateOldUserBoardUserColumnsToActor.php --wiki=atrociousyoutuberswiki --force
 * 01:04 paladox: set table_definition_cache and table_open_cache to 10000 on db6/db7
 * 00:38 paladox: re-started backups on bacula2
 * 00:38 paladox: upgrade puppet-agent - bacula2

2020-05-03

 * 21:31 John: DELETE FROM localuser WHERE lu_wiki = 'navaversewiki';
 * 17:46 paladox: upgrade phabricator on phab1
 * 17:35 paladox: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/runJobs.php - jobrunner1
 * 17:34 Reception123: reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php checkBadRedirects.php --wiki testwiki
 * 17:25 Reception123: dropped rainfallwiki telegrambotswiki templeofdivinitywiki tenlosssyndicatewiki tenniselbowwiki tensorflowglossarywiki terrafractumwiki terredelsolewiki tescommunitywiki testdelete7wiki unionwiki vragwiki xenimuswiki
 * 17:22 Reception123: dropped natturkowiki navaversewiki navmedwiki nayan28589wiki neoottomanwiki omnipediawiki pispediawiki pitchwiki pixterwiki placesrobloxwiki pmavwiki podcastswiki poewikiwiki pokemondustwiki pokeonewiki
 * 17:18 Reception123: dropped mwpermtestwiki myao93wiki myclassprojectswiki mydevexperiencewiki mymoneywiki mynintendowiki mysummerholydayswiki n2gowiki nafetswiki nagygenitalorszagwiki namudatawiki nanotrixwiki
 * 17:13 Reception123: dropped diagnostikwiki dobotswiki feralarchiveswiki hellointernetwiki iwwiki kaydadegerwiki kidalwiki lothuialethwiki metaautonomiawiki metroplexfederationwiki musicpediawiki musicwikiwiki musicworldgamewiki
 * 17:06 Reception123: dropped ashesworldwiki astropediawiki parapediawiki asylothekberlinwiki bloxcitywiki bonobowiki brgameswiki columbiasjpwiki deletenowdbwiki
 * 15:31 Reception123: dropped altenpflegewiki amcwiki animefurryfeetwiki johnflewiswiki anothertimeline2120wiki antuswiki aqferwiki
 * 15:25 Reception123: dropped testdatabasewiki testdelete3wiki testdelete7wiki
 * 14:32 Reception123: renamed testkan2020wiki to aeroddwiki
 * 12:31 Reception123: rebuild LC on mw*
 * 12:31 paladox: dropping securepoll_* table from metawiki
 * 12:30 Reception123: reception@jobrunner1:/srv/mediawiki/w/extensions/Flow/maintenance$ sudo -u www-data php FlowCreateTemplates.php --wiki radioscanningtwwiki
 * 12:12 paladox: MariaDB [mhglobal]> update mw_settings set s_settings = '[]' where s_dbname = 'wikitheoswiki';
 * 12:10 paladox: MariaDB [mhglobal]> INSERT INTO mw_settings (s_dbname, s_settings, s_extensions) VALUES ('wikitheoswiki', '{}', '[]');

2020-05-02

 * 22:32 paladox: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/Comments/maintenance/migrateOldCommentsVoteUserColumnsToActor.php --force - test2
 * 22:30 paladox: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/Comments/maintenance/migrateOldCommentsUserColumnsToActor.php - test2
 * 22:30 paladox: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/Comments/maintenance/migrateOldCommentsBlockUserColumnsToActor.php - test2
 * 22:24 paladox: root@test2:/srv/mediawiki/w/extensions/VoteNY# sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/VoteNY/maintenance/migrateOldVoteUserColumnsToActor.php
 * 19:31 paladox: root@test2:/srv/mediawiki/w/extensions/MirahezeMagic/maintenance# sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateSP.php
 * 19:23 paladox: root@test2:/srv/mediawiki/w/extensions/MirahezeMagic/maintenance# /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateSP.php
 * 18:23 paladox: root@test2:/srv/mediawiki/w/extensions/MirahezeMagic/maintenance# /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateSP.php
 * 14:03 John: UPDATE mw_permissions SET perm_autopromote = null WHERE perm_autopromote LIKE '{"add":%';

2020-05-01

 * 23:00 John: running LC
 * 20:42 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateSP.php
 * 19:57 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateComments.php
 * 18:41 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateSP.php
 * 18:33 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateSP55.php
 * 17:52 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateSP.php
 * 17:22 paladox: root@test2:/srv/mediawiki/w/extensions/SocialProfile/UserStats/maintenance# sudo -u www-data php migrateOldUserStatsUserColumnsToActor.php --wiki=atrociousyoutuberswiki
 * 16:40 paladox: disable a user in phabricator
 * 15:33 Reception123: restarted certbot
 * 11:04 paladox: upgrade puppet-agent puppetdb puppetdb-termini puppetserver salt-common salt-master salt-minion - puppet2
 * 10:50 paladox: apt-get dist-upgrade - cp7
 * 08:41 John: UPDATE mw_settings SET s_dbname = 'feralworthwikiwiki' WHERE s_dbname = 'feralarchiveswiki';
 * 01:14 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateComments.php
 * 01:09 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateSP.php - test2

2020-04-30

 * 17:30 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/SocialProfile/UserStats/maintenance/migrateOldUserStatsUserColumnsToActor.php - test2
 * 15:32 paladox: MariaDB [crappygameswiki]> UPDATE Comments_Vote SET Comment_Vote_actor=(SELECT actor_id FROM actor WHERE actor_name=Comment_Vote_Username);
 * 14:03 paladox: usermod -a -G ssl-cert www-data - test2
 * 02:15 paladox: MariaDB [toxicfandomsandhatedomswiki]> delete from Comments where Comment_Username = '';
 * 01:35 paladox: test2.php contains `ALTER TABLE /*$wgDBprefix*/Comments MODIFY COLUMN Comment_Date datetime NOT NULL default '1970-01-01 00:00:01';`
 * 01:35 paladox: test.php contains `ALTER TABLE /*$wgDBprefix*/Comments_Vote MODIFY COLUMN Comment_Vote_Date datetime NOT NULL default '1970-01-01 00:00:01';`
 * 01:23 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateComments4.php
 * 01:17 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateComments3.php
 * 01:08 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateComments.php
 * 00:53 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updatePollNY2.php
 * 00:52 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateQuizGame2.php
 * 00:47 paladox: rebuild lc on mw* and jobrunner1
 * 00:22 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateAjaxPoll.php
 * 00:06 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateQuizGame.php

2020-04-29

 * 23:44 paladox: test.sql is `ALTER TABLE /*$wgDBprefix*/Vote MODIFY COLUMN vote_date datetime NOT NULL default '1970-01-01 00:00:01';`
 * 23:39 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateVoteNY2.php
 * 23:33 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updatePollNY.php
 * 23:23 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateVoteNY.php
 * 23:08 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/updateSP.php - jobrunner1
 * 20:31 paladox: MariaDB [test2wiki]> truncate user_profile; - from earlier on db6
 * 15:23 paladox: upgrade phabricator on phab1
 * 15:22 paladox: delete sourav4 user in phabricator per request
 * 14:32 Reception123: added retailpedia discord webhook
 * 14:06 paladox: root@cloud2:/home/paladox# swapoff -a
 * 14:05 paladox: root@cloud1:/home/paladox# swapoff -a

2020-04-28

 * 20:48 paladox: reinstall php on mon1
 * 20:47 Southparkfan: apt-get remove php-ldap on mon1
 * 19:44 paladox: root@gluster1:/home/paladox# gluster volume set mvol rebal-throttle lazy
 * 17:11 paladox: INSERT INTO renameuser_status (ru_oldname, ru_newname, ru_wiki, ru_status) VALUES ('TaakoManTheGameMaster', 'GDPRAccount', 'crappyreceptionwikiswiki', 'failed');
 * 17:11 paladox: INSERT INTO renameuser_status (ru_oldname, ru_newname, ru_wiki, ru_status) VALUES ('TaakoManTheGameMaster', 'GDPRAccount', 'crappygameswiki', 'failed');
 * 17:11 paladox: INSERT INTO renameuser_status (ru_oldname, ru_newname, ru_wiki, ru_status) VALUES ('TaakoManTheGameMaster', 'GDPRAccount', 'awfulmovieswiki', 'failed');
 * 17:03 paladox: update actor set actor_name = 'GDPRAccount2' where actor_name = 'GDPRAccount';
 * 17:03 paladox: update user set user_name = 'GDPRAccount2' where user_name = 'GDPRAccount';
 * 16:58 paladox: INSERT INTO renameuser_status (ru_oldname, ru_newname, ru_wiki, ru_status) VALUES ('TaakoManTheGameMaster', 'GDPRAccount', 'atrociousyoutuberswiki', 'failed'); - dbt1
 * 16:47 paladox: MariaDB [mhglobal]> delete from renameuser_status where ru_newname = 'GDPRAccount'; - dbt1
 * 16:21 Reception123: fixStuckGlobalRename.php for GDPR
 * 15:52 Reception123: reception@jobrunner1:/srv/mediawiki/w/extensions/ManageWiki/maintenance$ sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki familiekbrwiki --overwrite
 * 15:03 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki and rebuild LC on mw*/jbr1
 * 13:32 John: ALTER IGNORE TABLE mw_permissions ADD UNIQUE uniqueperm(perm_dbname, perm_group);
 * 13:30 John: ALTER IGNORE TABLE mw_namespaces ADD UNIQUE uniquens(ns_dbname, ns_namespace_id);
 * 08:50 paladox: root@gluster1:/home/paladox# gluster volume set mvol rebal-throttle normal
 * 06:23 Reception123: renamed feralarchiveswiki to feralworthwikiwiki.

2020-04-27

 * 23:59 John: rebuildLC after running mergeMessageFileList to bring in calendar extension added earlier
 * 21:59 paladox: root@gluster1:/home/paladox# gluster volume set mvol rebal-throttle lazy
 * 20:06 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw*/jbr1
 * 19:02 paladox: root@gluster1:/home/paladox# gluster volume set mvol rebal-throttle normal
 * 18:55 paladox: MariaDB [(none)]> set global table_definition_cache = 5000; - db6 & db7
 * 18:54 paladox: MariaDB [(none)]> set global table_open_cache = 5000; - db6 & db7
 * 18:52 paladox: root@gluster1:/home/paladox# gluster volume set mvol rebal-throttle lazy
 * 18:42 paladox: root@gluster1:/home/paladox# gluster volume set mvol rebal-throttle normal
 * 18:15 Reception123: restarted MirahezeRC
 * 16:32 paladox: root@gluster1:/home/paladox# gluster volume set mvol rebal-throttle lazy
 * 16:30 paladox: root@gluster1:/home/paladox# gluster volume set mvol rebal-throttle aggressive
 * 16:27 paladox: upgrade nginx & grafana on mon1
 * 16:23 paladox: restart ircrcbot on mon1
 * 16:00 paladox: ban req.http.Host == elmetaverse.miraheze.org - cp6
 * 15:58 paladox: ban req.http.Host == elmetaverse.miraheze.org && req.url ~ "^/w/load.php" - cp6
 * 15:14 paladox: root@gluster1:/home/paladox# gluster volume set mvol rebal-throttle normal
 * 15:03 paladox: root@gluster1:/home/paladox# gluster volume set mvol rebal-throttle aggressive
 * 14:54 paladox: root@gluster1:/home/paladox# gluster volume set mvol rebal-throttle lazy
 * 14:52 paladox: root@gluster1:/home/paladox# gluster volume set mvol rebal-throttle aggressive
 * 14:10 paladox: rebalancing gluster cluster

2020-04-26

 * 19:35 paladox: reboot gluster1
 * 19:27 paladox: reboot mw[67] (and depool/repool)
 * 19:25 paladox: lower gluster1 ram to 5gb - 8gb is too much
 * 18:40 paladox: redis-cli -a xxx --scan --pattern *dreamversewiki* | xargs redis-cli -a xxx del - rdb2
 * 18:34 paladox: ban req.http.Host == static.miraheze.org && req.url ~ "^/dreamversewiki" - cp8
 * 12:59 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/updateSpecialPages.php --wiki testwiki
 * 12:47 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki testwiki

2020-04-25

 * 17:57 paladox: rebuild lc on test2
 * 17:53 paladox: rebuild lc on mw* and jobrunner1
 * 16:51 paladox: rebuild lc on mw* and jobrunner1

2020-04-24

 * 13:15 paladox: rebuild lc on test2
 * 09:15 Reception123: reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki tvwikiwiki /home/reception/Wikipedia-20200424091247.xml

2020-04-23

 * 19:38 paladox: rebuilding lc on test2
 * 19:35 paladox: rebuilding lc on mw* & jobrunner1
 * 13:53 paladox: restart nginx on mw4
 * 13:53 paladox: restart php-fpm on mw4
 * 00:05 paladox: run /home/paladox/resetWikiCaches.php on jobrunner1
 * 00:01 paladox: delete from mw_permissions where perm_dbname = 'ryfusewiki'; - dbt1

2020-04-22

 * 23:58 paladox: delete from mw_permissions where perm_dbname = 'i'; - dbt1
 * 23:58 paladox: delete from mw_permissions where perm_dbname = 'rontend,cite,citethispage,globaluserpage,mobilefrontend'; - dbt1
 * 23:57 paladox: delete from mw_permissions where perm_dbname = 'obilefrontend'; - dbt1
 * 23:51 paladox: run lowercaseGroups.php on jobrunner1
 * 15:59 paladox: restart ircrcbot
 * 15:19 paladox: restarted ircrcbot
 * 13:27 paladox: restart ircrcbot restart
 * 05:59 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki crappygameswiki -r "Requested - T5461" /home/reception/cgw.txt

2020-04-21

 * 21:06 paladox: stopping mysql on dbt1
 * 17:31 paladox: restart ircecho
 * 14:59 paladox: reinstall db7
 * 08:43 paladox: fallover db7 to db6
 * 08:23 Reception123: restarted cp7 (same time as the others)
 * 08:21 Reception123: restarted db7/mw*

2020-04-20

 * 14:26 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki crappygameswiki -r "Requested - T5441" /home/reception/cpgtxt.txt
 * 14:19 paladox: restart ircrcbot
 * 13:39 Reception123: reception@jobrunner1:/srv/mediawiki/w/maintenance$ sudo -u www-data php rebuildall.php --wiki crappygameswiki
 * 11:00 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw*/test2/jbr

2020-04-19

 * 20:08 Reception123: restarted RC bot
 * 16:56 paladox: add openldap to ssl-cert group on test2
 * 05:21 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki kiddancerswiki /home/reception/kiddancers_pages_full.xml
 * 00:18 Zppix: un-closing feralarchiveswiki as it was inactive due to technical issues
 * 00:13 paladox: remove wgDefaultSkin from feralarchiveswiki

2020-04-18

 * 22:17 Zppix: run compressOld.php on libertygamewiki (jobrunner1)
 * 16:52 paladox: delete testrenametestwiki
 * 16:07 Reception123: renamed kidalwiki to shatteredskieswiki
 * 15:32 paladox: MariaDB [mhglobal]> update mw_settings set s_dbname = 'brokenthroneswiki' where s_dbname = 'ashesworldwiki'; - dbt1
 * 15:15 Reception123: renamed ashesworldwiki to brokenthroneswiki.(db+cw)
 * 09:29 Reception123: rebuildlocalisationcache.php on mw*/jbr

2020-04-17

 * 13:30 Reception123: reception@jobrunner1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php migrateAccount.php --username BroswerStackAccount --wiki loginwiki --auto
 * 01:21 paladox: add Topic ns to all wikis that have Flow enabled using hackScript.php
 * 01:20 paladox: root@jobrunner1:/home/paladox# sudo -u www-data php hackScript.php --wiki=loginwiki

2020-04-16

 * 20:07 Zppix: Close recentiawiki for ToU violations
 * 16:44 paladox: enable "Limit to existing users" on mw-config last 24h
 * 15:16 paladox: upgrade puppetserver on puppet2
 * 14:42 Reception123: added discord hook for cookieclickerwiki
 * 13:22 Reception123: $cw = new CreateWikiJson( 'villagecollaborativewiki' ); $cw->resetDatabaseList;
 * 05:52 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki avalonwiki /home/reception/Maia+Song+Contest-20200412160644.xml
 * 03:45 paladox: upgrade gluster on mw* and jobrunner1 and test2
 * 03:44 paladox: apt-get upgrade - gluster1 (git, nodejs & gluster)
 * 03:40 paladox: apt-get upgrade - jobrunner1 (git, nodejs)

2020-04-15

 * 21:29 paladox: > MatomoAnalyticsHooks::wikiCreation( 'blueoathwiki' ); - jobrunner1
 * 19:47 paladox: restarted nginx on services2
 * 19:09 paladox: hack /srv/parsoid/node_modules/mediawiki-title/lib/index.js (function canonicalName to add a undefined check)
 * 19:02 paladox: /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /home/paladox/resetWikiCaches.php - jobrunner1
 * 18:44 paladox: root@mw4:/srv/mediawiki/w/cache# rm awfulmovieswiki.json
 * 15:51 paladox: FLUSHDB - jobrunner1 (redis)
 * 15:50 paladox: FLUSHALL ASYNC - jobrunner1 (redis)
 * 12:36 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki vivoleko01wiki /home/reception/isuzugemini01_pages_full.xml
 * 12:16 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* and jobrunner1

2020-04-14

 * 22:00 paladox: root@jobrunner1:/home/paladox# sudo -u www-data /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/runJobs.php
 * 20:19 paladox: root@jobrunner1:/home/paladox# /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/runJobs.php
 * 17:47 paladox: disabling issues on all of our github repos
 * 17:35 Reception123: sudo -u www-data php fixStuckGlobalRename.php --wiki=earthvisionwiki --logwiki=metawiki "Kathying" "LTSC1980"
 * 12:59 John: insert default mw_settings values for blueoathwiki
 * 02:44 Zppix: $cWJ = new CreateWikiJson( solarawiki ); $cWJ->resetWiki;

2020-04-13

 * 19:43 John: changed mw_namespaces entries from 'Array' to '[]'
 * 18:38 paladox: MariaDB [mhglobal]> update cw_wikis set wiki_closed_timestamp = NULL where wiki_closed = 0; - dbt1
 * 18:16 paladox: stopped script
 * 18:06 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_closed = 0 AND wiki_closed_timestamp = NULL WHERE wiki_closed_timestamp > 20200413000000; - dbt1
 * 17:27 paladox: MariaDB [mhglobal]> update cw_wikis set wiki_closed = 0 where wiki_closed = 1 and wiki_dbname = 'yoavfreundwiki'; - dbt1
 * 17:17 John: ->resetWiki for caching reset on all non-English wikis
 * 16:44 paladox: unmark yunawiki as closed
 * 13:55 John: added mw_namespaces entries for Wikibase to nbdbwiki
 * 03:48 paladox: rm -rf /srv/mediawiki/w/cache/managewiki on mw*
 * 03:48 paladox: root@mw4:/var/log# rm -rf /srv/mediawiki/w/cache/managewiki
 * 01:24 paladox: root@jobrunner1:/home/paladox# /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/runJobs.php
 * 01:14 paladox: cleaning up redis on jobrunner1 by deleting some wikis from its redis

2020-04-12

 * 23:43 paladox: runJobs on loginwiki ; restart jobrunner and restart jobchron on jobrunner1
 * 19:54 paladox: upgraded phabricator on phab1
 * 19:21 paladox: /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/removeSettings.php --wgsetting=wgServer
 * 16:23 paladox: root@mw5:/srv/mediawiki/w/maintenance# sudo -u www-data php invalidateUserSessions.php --wiki=testwiki --user=Voidwalker
 * 14:57 Reception123: MariaDB [metawiki]> UPDATE cw_requests SET cw_status = "approved" WHERE cw_id = "11467";
 * 14:57 Reception123: MariaDB [metawiki]> UPDATE cw_requests SET cw_status = "approved" WHERE cw_id = "11466";
 * 14:33 John: ban req.url ~ "^/w/load.php" on all cp*
 * 14:00 paladox: restarted redis on rdb2
 * 05:56 Reception123: reception@mw4:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=paramountsonicwiki Imjustthere
 * 05:56 Reception123: reception@mw4:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=blueoathwiki CptBrown
 * 00:11 paladox: rebuilding lc on mw*
 * 00:07 paladox: repool mw4

2020-04-11

 * 22:50 John: repool mw4 temp
 * 22:26 paladox: MariaDB [phabricator_user]> update user_email set address = '' where address = 'John@miraheze.org';
 * 22:01 John: depool mw4
 * 20:16 paladox: root@test2:/srv/mediawiki/w/cache# rm metawiki.json
 * 18:40 paladox: decrease puppet2 core by 1 (so 3 cores and 3 vcpu)
 * 18:20 paladox: sudo -u www-data php ext*/ManageWiki/maint*/toggleExtension.php --wiki testwiki --disable regexfunctions
 * 17:12 Zppix: rebuildlc on jobrunner1
 * 16:59 Reception123: added mediawiki::wiki_discord_hooks_url: [] to private hiera
 * 13:58 Reception123: rebuildLocalisationCache.php on mw*
 * 10:39 Reception123: rebuildLocalisationCache.php on mw*

2020-04-10

 * 18:26 paladox: MariaDB [(none)]> SET GLOBAL max_heap_table_size = 1024 * 1024 * 64; - db6
 * 18:26 paladox: MariaDB [(none)]> SET GLOBAL tmp_table_size = 1024 * 1024 * 64; - db6
 * 18:23 paladox: MariaDB [(none)]> SET GLOBAL max_heap_table_size = 1024 * 1024 * 64; - dbt1
 * 18:23 paladox: MariaDB [(none)]> SET GLOBAL tmp_table_size = 1024 * 1024 * 64; - dbt1
 * 18:23 Southparkfan: start replication on db6
 * 18:23 paladox: MariaDB [(none)]> SET GLOBAL max_heap_table_size = 1024 * 1024 * 64; - db7
 * 18:22 paladox: MariaDB [(none)]> SET GLOBAL tmp_table_size = 1024 * 1024 * 64; - db7
 * 17:58 Southparkfan: stop db6 mysql
 * 16:26 paladox: root@db7:~# mariabackup --backup --slave-info --safe-slave-backup --stream=xbstream | ssh -i /home/dbcopy/.ssh/id_ed25519 dbcopy@db6.miraheze.org "mbstream -x -C /home/dbcopy/backup-db6-10april/"
 * 16:25 paladox: root@db6:/srv# rm -rf mariadb
 * 16:12 paladox: kill -9 1295 - db6
 * 16:11 paladox: stopping mysql on db6
 * 13:03 Southparkfan: reboot db6
 * 12:51 Southparkfan: stop mysql on db6 to move backup folder
 * 00:52 Southparkfan: mariabackup --backup --slave-info --safe-slave-backup --stream=xbstream | ssh -i /home/dbcopy/.ssh/id_ed25519 dbcopy@db6.miraheze.org "mbstream -x -C /home/dbcopy/backup-db6-10april/"
 * 00:52 Southparkfan: running mariabackup in screen on db7, for db6 replication, see command below (or above, if you're reading SAL!)
 * 00:31 Southparkfan: install mariadb-backup on db6
 * 00:29 Southparkfan: install mariadb-backup on db7
 * 00:24 paladox: increased db6 ram to 16g

2020-04-09

 * 22:09 Zppix: updating stopforumspam blacklist
 * 16:09 Reception123: reception@jobrunner1:/srv/mediawiki/w/extensions/Cargo/maintenance$ sudo -u www-data php cargoRecreateData.php --wiki thenationstatewiki --table Political_Parties
 * 14:31 paladox: rebuilt lc on mw*

2020-04-08

 * 17:46 Reception123: manually delete Wile_E._Coyote from animatedfeetwiki

2020-04-07

 * 20:32 paladox: apt-get upgrade - bacula2
 * 20:27 paladox: regenerating backups on bacula2
 * 08:30 Reception123: enable puppet everywhere
 * 08:28 Reception123: repool mw4
 * 08:27 Reception123: depool mw4
 * 08:08 Reception123: disabled puppet on mw5-7 for testing revert on mw4
 * 04:43 paladox: Renaming solarawiki to spcodexwiki
 * 04:30 paladox: update page set page_namespace = 0 where page_namespace = 4;
 * 03:12 paladox: root@jobrunner1:/var/log/mediawiki/debuglogs# redis-cli -a xxx --scan --pattern *zwaaiwiki* | xargs redis-cli -a xxx unlink
 * 03:11 paladox: root@jobrunner1:/var/log/mediawiki/debuglogs# redis-cli -a xxx --scan --pattern *warapediawiki* | xargs redis-cli -a xxx unlink
 * 03:11 paladox: root@jobrunner1:/var/log/mediawiki/debuglogs# redis-cli -a xxx --scan --pattern *soratakowiki* | xargs redis-cli -a xxx unlink
 * 03:10 paladox: root@jobrunner1:/var/log/mediawiki/debuglogs# redis-cli -a xxx --scan --pattern *schupiewikiwiki* | xargs redis-cli -a xxx unlink
 * 03:08 paladox: root@jobrunner1:/var/log/mediawiki/debuglogs# redis-cli -a xxx --scan --pattern *ewscwiki* | xargs redis-cli -a xxx unlink
 * 02:42 paladox: chown -R www-data:www-data /srv/mediawiki/w/.git -test2
 * 02:41 paladox: sudo -u www-data git reset --hard origin/REL1_34 - test2
 * 02:40 paladox: root@test2:/srv/mediawiki/config# sudo -u www-data git reset --hard origin/master ; sudo -u www-data git pull
 * 02:36 paladox: repool mw4
 * 02:35 paladox: depool mw4
 * 02:35 paladox: root@mw4:/srv/mediawiki/config# sudo -u www-data git pull
 * 02:35 paladox: root@mw4:/srv/mediawiki/config# sudo -u www-data git reset --hard origin/master
 * 02:33 paladox: repool mw7
 * 02:32 paladox: depool mw7
 * 02:32 paladox: repool mw6
 * 02:31 paladox: depool mw6
 * 02:31 paladox: repool mw5
 * 02:29 paladox: depool mw5
 * 02:07 paladox: root@mw4:/srv/mediawiki/w/cache# rm *wiki.json
 * 02:05 paladox: root@mw4:/srv/mediawiki/w/cache# rm *wiki.json
 * 01:46 paladox: rm * (in /srv/mediawiki/cache) on mw4
 * 01:22 paladox: root@mw4:/srv/mediawiki/w/extensions/MirahezeMagic/maintenance# /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/spScript.php

2020-04-06

 * 22:51 John: repool mw4 to get real load conditions
 * 22:31 John: depool mw4 for isolated deployment
 * 19:32 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki mundanapediawiki /home/reception/Mundanapedia-20160208152735.xml
 * 17:14 paladox: apt-get upgrade - ns1
 * 16:57 paladox: rebooted cp3
 * 16:39 paladox: upgrade cp3 to buster
 * 16:05 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"yukesh.com.np"', null ) WHERE wiki_dbname = 'hamropediawiki';
 * 16:04 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"wikiverte.pl"', null ) WHERE wiki_dbname = 'wikivertewiki';

2020-04-05

 * 22:49 paladox: increase ram on rdb[12] to 4g
 * 22:46 paladox: reboot mw5
 * 22:44 paladox: reboot mw4
 * 22:22 paladox: increase jobrunner1 core by 1
 * 22:22 paladox: increase jobrunner1 core by `
 * 18:35 paladox: increase jobrunner1 ram to 3gb
 * 18:34 paladox: increase puppet2 to 4 cores
 * 13:06 Reception123: deleted and dropped https://phabricator.miraheze.org/P290 (DP)
 * 12:00 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/deleteWikis.php --wiki loginwiki --delete Reception123
 * 08:44 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki aeschyluswiki /home/reception/Wikipedia-20200403182927.xml
 * 08:44 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki aeschyluswiki /home/reception/Wikipedia-20200402160456.xml
 * 08:41 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki aeschyluswiki /home/reception/Wikipedia-20200402213524.xml

2020-04-04

 * 17:25 paladox: MariaDB [hispano76wiki]> truncate cargo_pages;
 * 17:24 paladox: MariaDB [hispano76wiki]> truncate cargo_tables;
 * 17:22 paladox: drop cargo__* from hispano76wiki
 * 15:25 paladox: root@mw4:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=templatewiki --report=1 Wikipedia-20200404152245.xml
 * 13:47 paladox: root@mw4:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=templatewiki --report=1 Wikipedia-20200404133925.xml

2020-04-03

 * 21:13 paladox: root@puppet2:/home/puppet-users/ssl-keys# chmod -R 0770 /etc/puppetlabs/puppet/ssl-keys
 * 21:13 paladox: root@puppet2:/home/puppet-users/ssl-keys# chown -R puppet:puppet-users /etc/puppetlabs/puppet/ssl-keys
 * 21:10 paladox: root@puppet2:/home/puppet-users/ssl-keys# chown -R puppet:puppet-users .git
 * 21:09 paladox: root@puppet2:/home/puppet-users# chmod -R 0770 ssl-keys
 * 21:03 paladox: root@puppet2:/home/puppet-users# chmod 0770 ssl-keys
 * 21:03 paladox: root@puppet2:/home/puppet-users# chown puppet:puppet-users ssl-keys
 * 21:01 paladox: root@puppet2:/home/puppet-users# chown 0770 ssl-keys
 * 19:58 paladox: depool mw[67], reboot and repool
 * 19:58 paladox: set mw[67] cores to 5 (5 vcpu)
 * 19:52 paladox: depool mw[45], reboot and repool
 * 19:51 paladox: depool mw4
 * 19:49 paladox: change puppet2 to 3 cores, mw[45] to 5 cores (5 vcpu)
 * 18:04 Reception123: redacted PII for T5376
 * 10:17 Reception123: reception@jobrunner1:~/images$ sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=housingwikiwiki --search-recursively --skip-dupes /home/reception/images

2020-04-02

 * 23:12 paladox: increase cp7 disk by 10g
 * 22:44 paladox: increasing cp6 disk by 10g
 * 17:23 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki tvwikiwiki /home/reception/Wikipedia-20200402172042.xml
 * 10:54 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki gmcwiki -r "Requested - T5284" /home/reception/graaldel.txt
 * 10:53 Reception123: removed 2FA from a phab user
 * 10:52 paladox: restarted logbot on mon1

2020-04-01

 * 06:27 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki housingwikiwiki /home/reception/wiki-yimby_wiki-XML-20200331.xml
 * 01:23 paladox: apt-get upgrade - mail1, misc1
 * 01:20 paladox: apt-get upgrade - jobrunner1
 * 01:14 paladox: upgraded nodejs on test2
 * 01:11 paladox: upgraded nodejs on mw[4567]
 * 01:10 paladox: upgraded nodejs on services1

2020-03-31

 * 23:18 paladox: rm /srv/mediawiki/w/cache/managewiki/*realtimefandubgameswikiwiki* - mw*
 * 16:41 paladox: apt-get dist upgrade - misc1
 * 13:28 paladox: restart varnish + nginx on cp7
 * 12:19 paladox: restart php7.3-fpm on mw4
 * 12:18 paladox: restart php7.3-fpm on mw7 and chown php-error.log as www-data
 * 12:17 paladox: restart php7.3-fpm on mw6 and chown php-error.log as www-data
 * 12:06 paladox: reboot services1
 * 12:06 paladox: reboot services2
 * 09:45 paladox: stop ircrcbot on misc1
 * 06:34 Reception123: reception@jobrunner1:~/wiki/images$ sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=sagan4alphawiki --search-recursively --skip-dupes /home/reception/wiki/images

2020-03-30

 * 21:34 Zppix: drop dummy wiki dbs for old infra (DONE: mw[123], db[12345] lizardfs[12345] cp[1245])
 * 21:18 Zppix: created dummy wiki dbs to prevent conflict with services/servers (mon1, icinga, grafana, dbt1)
 * 01:06 paladox: apt-get dist-upgrade - services2
 * 01:04 paladox: apt-get dist-upgrade - services1

2020-03-29

 * 23:54 paladox: remove db4 from puppetdb
 * 23:45 paladox: remove mw[123] and lizardfs6 from puppetdb
 * 23:36 paladox: restart php7.3-fpm on mw*
 * 21:45 paladox: apt-get upgrade - mail1
 * 20:16 paladox: depool mw7 and repool
 * 20:05 paladox: restart php7.3-fpm on mw*
 * 19:59 paladox: depool mw6 and repool
 * 19:59 paladox: increase mw[67] vpcu by 1
 * 19:59 paladox: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki crappygameswiki -r "Requested - T5324" /home/reception/cgwdel.txt - mw4
 * 19:59 paladox: repool mw5
 * 19:59 paladox: increase mw5 vcpu by 1
 * 19:59 paladox: depool mw5
 * 19:58 paladox: repool mw4
 * 19:58 paladox: increase mw4 vcpu by 1
 * 19:55 paladox: depool mw4
 * 19:30 Reception123: add alteversewiki webhook for discord
 * 17:58 Reception123: added webhook for zoitecwiki
 * 17:43 paladox: rm /srv/mediawiki/w/cache/managewiki/*privado* on mw*
 * 17:43 paladox: deleting *privado* in redis
 * 17:43 paladox: recreating privadowiki
 * 17:39 paladox: rm /srv/mediawiki/w/cache/managewiki/* ucronias* on mw*
 * 17:38 paladox: deleting * ucronias* in redis
 * 17:38 paladox: recreating ucroniaswiki
 * 17:35 paladox: rm /srv/mediawiki/w/cache/managewiki/*hispano* on mw*
 * 17:32 paladox: deleting *hispano* in redis
 * 17:30 paladox: recreating hispanowiki
 * 16:40 Southparkfan: ack icinga alerts for decom'ed servers
 * 14:55 paladox: repool mw5
 * 14:53 paladox: depool mw5
 * 14:51 paladox: repool mw4
 * 14:50 paladox: depool mw4
 * 14:50 paladox: increased mw[45] vpcu by 1
 * 14:48 paladox: decrease puppet2 vcpu by one
 * 14:25 Reception123: reception@mw4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki crappygameswiki -r "Requested - T5324" /home/reception/cgwdel.txt
 * 07:58 Reception123: reception@mw4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=sagan4alphawiki --search-recursively --skip-dupes /home/reception/images
 * 04:53 paladox: restart logbot on mon1
 * 04:53 paladox: stop mysql on db4
 * 02:40 paladox: puppetserver ca clean --certname lizardfs6.m.org/mw3.m.org
 * 01:31 paladox: increased puppet2 ram by 1gb and removed swap
 * 01:24 paladox: add 1g swap to puppet2
 * 01:14 paladox: delete all wikis on db4 that have been migrated to dbt1
 * 00:43 paladox: restart php-fpm on mw*

2020-03-28

 * 19:20 Reception123: removed a 2FA from a user
 * 18:36 Reception123: reception@mw1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki awesomegameswiki -r "Requested (old fandom accounts) - T5356" /home/reception/agwdel.txt
 * 14:09 paladox: restarted php7.3-fpm on mw*
 * 02:44 paladox: drop 1.6k wikis on db4 (migrated to dbt1)

2020-03-27

 * 23:14 paladox: apt-get upgrade puppet-agent - mw3
 * 18:41 paladox: restart php7.3-fpm on mw*

2020-03-26

 * 15:39 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw*
 * 15:33 paladox: restart logbot on mon1
 * 02:24 paladox: install links on test2
 * 01:58 paladox: reset extensions/CreateWiki on test2

2020-03-25

 * 23:27 paladox: restarted php7.3-fpm on mw*

2020-03-24

 * 17:01 Reception123: deleted and dropped testdelete1wiki
 * 16:23 Reception123: sudo -u www-data php populateContentTables.php --wiki [hispanowiki/ucroniaswiki/pricadowiki]
 * 16:19 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php populateContentModel.php --wiki [hispanowiki/ucroniaswiki/pricadowiki] --ns "all" --table page/revision

2020-03-23

 * 18:34 paladox: service parsoid stop ; killall -u parsoid ; service parsoid start - services[12]
 * 15:32 paladox: that is on nginx
 * 15:25 paladox: disable logging on mw[46] as an experiment
 * 15:19 paladox: MariaDB [ucroniaswiki]> INSERT INTO slot_roles (`role_id`, `role_name`) VALUES (1, 'main'); - db7
 * 15:18 paladox: MariaDB [hispanowiki]> INSERT INTO slot_roles (`role_id`, `role_name`) VALUES (1, 'main'); - db7
 * 15:17 paladox: MariaDB [privadowiki]> INSERT INTO slot_roles (`role_id`, `role_name`) VALUES (1, 'main'); - db7

2020-03-22

 * 23:29 paladox: MariaDB [privadowiki]> INSERT INTO content_models (`model_id`, `model_name`) VALUES (3, 'javascript'); - db7
 * 23:28 paladox: MariaDB [privadowiki]> INSERT INTO content_models (`model_id`, `model_name`) VALUES (4, 'Scribunto'); - db7
 * 23:28 paladox: MariaDB [privadowiki]> INSERT INTO content_models (`model_id`, `model_name`) VALUES (5, 'json'); - db7
 * 23:28 paladox: MariaDB [privadowiki]> INSERT INTO content_models (`model_id`, `model_name`) VALUES (2, 'css'); - db7
 * 21:35 paladox: drop database ndgwiki and nenawikiwiki on db4
 * 21:28 Zppix: (paladox): migrated ndg and nena wikis

2020-03-21

 * 19:52 paladox: delete ucroniaswiki
 * 19:51 paladox: delete hispanowiki
 * 16:09 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki rottenwebsiteswiki -r "Requested - T5313" /home/reception/delrt1.txt on mw1
 * 16:01 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/dumpBackup.php --wiki aerosswiki --full > /home/reception/aerosswiki21032020.xml
 * 15:57 Reception123: renamed magnaversewiki to cnocbridewiki (db+cw)
 * 15:13 Reception123: UPDATE cw_wikis SET wiki_closed = 0 AND wiki_closed_timestamp = NULL WHERE wiki_closed_timestamp > 20200321000000 on db4
 * 15:03 Reception123: php /srv/mediawiki/w/extensions/CreateWiki/maintenance/manageInactiveWikis.php --wiki loginwiki --write > /var/log/mediawiki/cron/managewikis.log on mw1
 * 14:36 Reception123: reception@mw1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki rottenwebsiteswiki
 * 14:13 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki sagan4alphawiki /home/reception/sagan4wiki.xml
 * 13:45 Reception123: manually ran manageInactiveWikis.php
 * 13:37 Reception123: manually ran manageInactiveWikis.php
 * 12:00 Reception123: reception@mw1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki rottenwebsiteswiki -r "Requested - T5313" /home/reception/delrt.txt

2020-03-20

 * 21:03 paladox: running lc on mw* and test2 and jobrunner1
 * 20:08 paladox: upgrade puppetdb, puppetserver and puppet-agent on puppet2
 * 06:37 Reception123: reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --no-local-users --username-prefix "import" --wiki pikminfanonwiki /home/reception/pikminfanon.xml

2020-03-19

 * 21:18 paladox: reboot test2
 * 21:13 paladox: apt-get upgrade - test2
 * 21:10 paladox: rename test1wiki to test2wiki
 * 20:26 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"enc.for.uz"', null ) WHERE wiki_dbname = 'uzencwiki'; - db4
 * 16:44 paladox: running lc on mw*

2020-03-18

 * 22:53 paladox: depool mw[123]

2020-03-17

 * 21:05 paladox: restart php7.3-fpm on mw*
 * 21:05 paladox: reboot services2
 * 21:04 paladox: reboot services1
 * 14:46 Zppix: rebuildrc and initsitestats on libertygamewiki following successful import

2020-03-16

 * 17:46 Zppix: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --no-local-users --username-prefix="리버티게임" --wiki libertygamewiki 22115932221_w-20200309-history.xml on jobrunner1
 * 15:12 paladox: restart php7.3-fpm on mw*

2020-03-15

 * 13:47 paladox: apt-get dist-upgrade - cp7

2020-03-14

 * 16:04 paladox: increzse puppet2 disk by 10g

2020-03-13

 * 21:03 paladox: rename godaigowiki to brambowiki
 * 13:20 paladox: root@puppet2:/opt/puppetlabs/server/data/puppetserver/reports# rm -rf *

2020-03-12

 * 20:53 paladox: root@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance# sudo -u www-data php fixStuckGlobalRename.php --wiki=abominablememefandomswiki --logwiki=metawiki "LeonTheMonster479519314" "Just Leon"
 * 20:06 paladox: root@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance# sudo -u www-data php fixStuckGlobalRename.php --wiki=abominablememefandomswiki --logwiki=metawiki "Agent Mace" "Agent Joestar"
 * 00:48 Zppix: update cw_requests set cw_status = 'approved' where cw_id = 10991;

2020-03-11

 * 15:36 paladox: rm -rf /srv/mediawiki/config and let it reclone, merge conflicts so recloning fresh
 * 15:28 paladox: apt-get dist-upgrade - test1
 * 15:26 paladox: reboot test1
 * 15:11 paladox: root@puppet2:/opt/puppetlabs/server/data/puppetserver/reports# rm -rf *
 * 14:08 paladox: increase gluster1 disk to 720G
 * 01:25 paladox: increase gluster1 disk by 50g

2020-03-10

 * 22:53 paladox: apt-get upgrade - mon1 (icinga2/puppet-agent)
 * 22:45 paladox: shutdown misc4
 * 22:45 paladox: shutdown cp4
 * 22:44 paladox: shutdown puppet1
 * 21:59 paladox: MariaDB [browndustwiki]> INSERT INTO content_models (`model_id`,`model_name`) VALUES (4, 'Scribunto'); -db4
 * 21:59 paladox: MariaDB [browndustwiki]> INSERT INTO content_models (`model_id`,`model_name`) VALUES (2, 'css'); -db4
 * 21:58 paladox: MariaDB [browndustwiki]> INSERT INTO content_models (`model_id`,`model_name`) VALUES (3, 'javascript'); -db4
 * 21:58 paladox: MariaDB [browndustwiki]> INSERT INTO content_models (`model_id`,`model_name`) VALUES (5, 'json'); -db4
 * 21:58 paladox: MariaDB [browndustwiki]> INSERT INTO content_models (`model_id`,`model_name`) VALUES (1, 'wikitext'); -db4
 * 21:46 paladox: MariaDB [browndustwiki]> INSERT INTO slot_roles (`role_id`,`role_name`) VALUES (1,'main'); - db4
 * 21:33 paladox: root@lizardfs6:/home/paladox# gluster volume set mvol features.read-only off
 * 21:03 paladox: restart jobrunner + jobchron on jobrunner1
 * 20:48 paladox: delete *matomo* key in redis
 * 20:32 paladox: root@jobrunner1:/srv/mediawiki/w/extensions/MirahezeMagic/maintenance# /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/addMatomo.php
 * 20:31 paladox: MariaDB [mhglobal]> delete from matomo; - db4
 * 19:53 Zppix: update cw_requests set cw_status = 'approved' where cw_id = 10988;
 * 18:52 paladox: set features.read-only to on, on lizardfs6
 * 18:17 Zppix: run cargoRecreateData.php and refreshLinks.php on jobrunner1 on hispanowiki per user request on IRC

2020-03-09

 * 22:51 paladox: reboot misc1
 * 20:51 paladox: apt-get install puppet-agent salt-common salt-minion - cp8

2020-03-08

 * 23:58 paladox: started ircecho on misc1
 * 13:16 Southparkfan: cancelled migration, dropping databases on db6
 * 13:12 Southparkfan: migration started for first batch
 * 00:47 paladox: delete db5 - not in use anymore

2020-03-07

 * 23:23 paladox: reinstalling ns1 as buster
 * 19:26 paladox: upgrade puppet-agent on cloud[12]
 * 19:21 paladox: root@cloud1:/var/lib/vz/images/118# rm vm-118-disk-0.qcow2
 * 18:38 paladox: upgrade puppet-agent - ns2
 * 18:37 paladox: root@cloud1:/var/lib/vz/images/118# mv vm-118-disk-0.row vm-118-disk-0.raw
 * 18:34 paladox: root@cloud1:/var/lib/vz/images/118# qemu-img convert vm-118-disk-0.qcow2 vm-118-disk-0.row
 * 17:41 paladox: removed prometheus_gdnsd_stats cron from misc1
 * 17:39 paladox: stopped gdnsd + purged it from misc1
 * 17:21 paladox: MariaDB [(none)]> set global table_open_cache = 10000; - db4
 * 17:20 paladox: set global table_definition_cache = 10000; - db4
 * 17:17 paladox: apt-get -t stretch-backports install linux-image-amd64 - db5
 * 17:10 paladox: apt-get upgrade db5
 * 06:35 Reception123: deleted and dropped streamelementswiki (T5307)
 * 01:34 paladox: MariaDB [simswiki]> truncate objectcache; - db6
 * 00:45 paladox: drop large wiki drops on db4/db5

2020-03-06

 * 23:50 Southparkfan: run pveperf on cloud1
 * 20:28 paladox: root@db5:/home/paladox# ./move_all.sh
 * 19:03 paladox: root@db4:/home/paladox# ./move_all.sh
 * 19:02 Zppix: Migration window has started
 * 18:38 paladox: sysctl -w net.ipv6.conf.all.disable_ipv6=1 - db5
 * 16:52 paladox: upgrade puppet-agent on misc4

2020-03-05

 * 17:52 paladox: apt-get upgrade - puppet1 (puppetserver, puppetdb and a few other packages)
 * 02:18 paladox: drop phabricator_% on db4
 * 02:18 paladox: drop staffwiki on db4
 * 01:02 paladox: set write back cache on mon1
 * 00:58 paladox: increase phab1 disk by 5g
 * 00:54 paladox: sysctl -w net.ipv6.conf.all.disable_ipv6=1 - db4

2020-03-03

 * 22:28 paladox: delete from renameuser_status where ru_wiki = 'receptionflamewarswiki'; - db4
 * 22:24 paladox: delete receptionflamewarswiki from cw
 * 21:06 paladox: stop db7 being a slave temporarily
 * 19:38 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki TaylorSwiftFan "GrilledCheese800" --logwiki=metawiki

2020-03-02

 * 19:25 paladox: downgrade php-redis on mw1
 * 18:15 paladox: downgraded php on jobrunner1 and upgraded gluster + few other pakcages
 * 18:13 paladox: apt-get upgrade - gluster1
 * 17:40 paladox: running lc on mw4
 * 17:24 paladox: fixing php on mw[4567]
 * 16:50 paladox: downgraded php to 7.3.14 on mw[23] lizardfs6 due to a segfault in 7.3.15, mw1 will be out of the pool till it's upgraded to buster.
 * 14:30 paladox: reinstalling mw4
 * 13:08 paladox: reboot mw4

2020-03-01

 * 17:39 paladox: root@mw2:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=tmewiki --type=gzip
 * 17:35 paladox: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=crappygameswiki --type=gzip
 * 17:33 paladox: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=animatedfeetwiki --type=gzip
 * 17:22 paladox: MariaDB [altversewiki]> drop database testwiki; - db5 (on db6 already)
 * 17:14 paladox: MariaDB [altversewiki]> drop database ucroniaswiki; - db4 (on db6 already)
 * 17:14 paladox: MariaDB [altversewiki]> drop database rotompediawiki; - db4 (on db6 already)
 * 17:13 paladox: MariaDB [altversewiki]> drop database proxybotwiki; - db4 (on db6 already)
 * 17:13 paladox: MariaDB [altversewiki]> drop database privadowiki; - db4 (on db6 already)
 * 17:13 paladox: MariaDB [altversewiki]> drop database onepiecewiki; - db4 (on db6 already)
 * 17:13 paladox: MariaDB [altversewiki]> drop database onepiecebountyrushwiki; - db4 (on db6 already)
 * 17:12 paladox: MariaDB [altversewiki]> drop database nonbinarywiki; - db4 (on db6 already)
 * 17:12 paladox: MariaDB [altversewiki]> drop database nbdbwiki; - db4 (on db6 already)
 * 17:12 paladox: MariaDB [altversewiki]> drop database hispanowiki; - db4 (on db6 already)
 * 17:11 paladox: MariaDB [altversewiki]> drop database dreamversewiki; - db4 (on db6 already)
 * 17:10 paladox: MariaDB [altversewiki]> drop database anotheredenwiki; - db4 (on db6 already)
 * 17:10 paladox: root@db5:/home/dbcopy# rm *sql*
 * 17:09 paladox: MariaDB [altversewiki]> drop database altversewiki; - db4 (on db6 already)
 * 14:47 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php initSiteStats.php --wiki=hispanowiki --update
 * 14:46 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php updateArticleCount.php --wiki=hispanowiki --update
 * 12:50 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/dumpBackup.php --wiki allthetropeswiki --full > /home/reception/allthetropes01032020.xml (part of wikibackups) on mw2
 * 11:40 Reception123: running wikibackups (bash ./wikibackups.sh /home/reception/allpublic.dblist /srv/mediawiki/w/maintenance/dumpBackup.php) on mw1
 * 11:22 Reception123: moved openhatchwiki from db4 to db5
 * 11:03 Reception123: renamed epochlorewiki to mystfalllorewiki (db + script)

2020-02-29

 * 21:10 Reception123: PURGE BINARY LOGS BEFORE '2020-02-29 12:00'; on db4
 * 00:08 paladox: ip addr del fe80::f816:3eff:fe31:f5f/64 dev ens3 - bacula2
 * 00:08 paladox: ip -6 address add 2604:180:f3::382/64 dev ens3 - bacula2

2020-02-28

 * 22:34 Southparkfan: reinstall bacula2
 * 22:11 Southparkfan: decom bacula1
 * 17:43 paladox: reboot mw[45]
 * 17:15 paladox: restarted php7.3-fpm on mw1
 * 15:24 paladox: restart php7.3-fpm on mw* and lizardfs6
 * 15:12 paladox: MariaDB [(none)]> set global innodb_io_capacity_max=3000; - db4
 * 15:11 paladox: MariaDB [(none)]> set global innodb_io_capacity=1000; - db4
 * 14:36 paladox: MariaDB [(none)]> set global tmp_table_size = 67108864; - db4
 * 14:35 paladox: MariaDB [(none)]> set global max_heap_table_size = 67108864; (on db4

2020-02-27

 * 23:25 paladox: root@db4:/home/paladox# sysctl -w net.ipv4.tcp_tw_reuse=0
 * 23:25 paladox: !log root@db4:/home/paladox# sysctl -w net.ipv4.tcp_fin_timeout=60
 * 23:01 paladox: root@db4:/home/paladox# sysctl -w net.ipv4.tcp_tw_recycle=0
 * 22:58 paladox: root@db4:/home/paladox# sysctl -w net.ipv4.tcp_tw_recycle=1
 * 22:52 paladox: root@db4:/home/paladox# sysctl -w net.ipv4.tcp_fin_timeout=3
 * 22:52 paladox: root@db4:/home/paladox# sysctl -w net.ipv4.tcp_tw_reuse=1
 * 22:47 paladox: root@db4:/home/paladox# sysctl net.ipv4.tcp_fin_timeout=3
 * 22:44 paladox: restart php.7-3-fpm on mw* & lizardfs*
 * 22:42 paladox: root@db4:/home/paladox# sysctl net.ipv4.tcp_tw_reuse=1
 * 16:04 Reception123: reception@mw2:~/graalmilitary$ sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=gmcwiki --search-recursively --skip-dupes /home/reception/graalmilitary/wikipics
 * 11:48 Reception123: moved simswiki from db4 to db5

2020-02-26

 * 23:52 paladox: root@cloud2:/var/lib/vz/images/113# rm vm-113-disk-0.qcow2
 * 23:45 paladox: root@cloud2:/var/lib/vz/images/113# qemu-img convert vm-113-disk-0.qcow2 vm-113-disk-0.raw
 * 23:25 paladox: root@bacula2:/home/paladox# ip addr del fe80::f816:3eff:fe31:f5f/64 dev ens3
 * 23:25 paladox: root@bacula2:/home/paladox# ip -6 address add 2604:180:f3::382/64 dev ens3
 * 23:18 paladox: apt-get dist-upgrade - phab1
 * 20:06 Reception123: move the previous import to jobrunner1 (per redis issues)
 * 17:06 paladox: removing old kernel from cp3 using sudo apt autoremove
 * 17:04 paladox: reboot cp3 - network felt slow
 * 07:26 Reception123: reception@mw1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki bigbrotherwikiwiki /home/reception/bigbrother_pages_full.xml
 * 01:56 paladox: root@cloud2:/var/lib/vz/images/115# qemu-img convert vm-115-disk-0.qcow2 vm-115-disk-0.raw
 * 01:45 paladox: root@cloud1:/var/lib/vz/images/111# qemu-img convert vm-111-disk-0.qcow2 vm-111-disk-0.raw
 * 01:19 paladox: that's on cp3
 * 01:19 paladox: apt-get install linux-image-amd64
 * 01:17 paladox: apt-get upgrade - cp3
 * 00:57 paladox: varnish> ban req.http.Host == static.miraheze.org (cp8|cp4)
 * 00:45 paladox: varnish> ban req.http.Host == messengergeek.miraheze.org (cp4)
 * 00:36 paladox: varnish> ban req.http.Host == messengergeek.miraheze.org (cp8)
 * 00:25 paladox: root@cloud1:/var/lib/vz/images/107# qemu-img convert vm-107-disk-0.qcow2 vm-107-disk-0.raw
 * 00:15 paladox: root@cloud1:/var/lib/vz/images/100# qemu-img convert vm-100-disk-0.qcow2 vm-100-disk-0.raw
 * 00:02 paladox: root@cloud1:/var/lib/vz/images/106# qemu-img convert vm-106-disk-0.qcow2 vm-106-disk-0.raw

2020-02-25

 * 23:55 paladox: actually 3 in total
 * 23:52 paladox: increased jobrunner1 core by 1 (so 4 in total)
 * 21:33 paladox: root@cloud1:/var/lib/vz/images/112# qemu-img convert vm-112-disk-0.qcow2 vm-112-disk-0.raw
 * 21:20 paladox: root@cloud2:/var/lib/vz/images/119# qemu-img convert vm-119-disk-0.qcow2 vm-119-disk-0.raw
 * 21:18 paladox: root@cloud2:/var/lib/vz/images/104# rm vm-104-disk-0.qcow2
 * 20:15 paladox: root@db4:/var/log/mysql# sysctl net.ipv4.tcp_tw_reuse=0
 * 20:15 paladox: root@db4:/var/log/mysql# sysctl net.ipv4.tcp_fin_timeout=60
 * 19:52 paladox: root@cp7:/home/paladox# sysctl net.core.somaxconn=4000
 * 19:49 paladox: root@db4:/var/log/mysql# sysctl net.ipv4.tcp_fin_timeout=3
 * 19:47 paladox: root@db4:/var/log/mysql# sysctl net.ipv4.tcp_tw_reuse=1
 * 18:06 paladox: root@cloud2:/var/lib/vz/images/104# qemu-img convert vm-104-disk-0.qcow2 vm-104-disk-0.raw
 * 18:03 paladox: root@cloud1:/var/lib/vz/images/102# rm vm-102-disk-0.qcow2 (it's been converted to raw)
 * 17:48 paladox: converting puppet2 to a raw image (qemu-img convert vm-102-disk-0.qcow2 vm-102-disk-0.raw)
 * 17:42 paladox: downgrade puppet2 ram back to 4g
 * 17:33 paladox: increase puppet2 ram by 1gb (so 5g in total)

2020-02-24

 * 23:28 paladox: root@mw4:/var/log# sysctl net.core.somaxconn=512
 * 22:06 Southparkfan: restarted php-fpm again
 * 22:01 Southparkfan: restart php-fpm again (on mon1)
 * 21:58 Southparkfan: restart php-fpm on mon1
 * 18:59 paladox: apt-get upgrade - mon1
 * 17:40 paladox: set global read_only=0; - db6
 * 17:15 paladox: set read_only on db6
 * 15:04 paladox: MariaDB [metawiki]> set global table_definition_cache=40000; - db6
 * 15:04 paladox: MariaDB [metawiki]> set global table_open_cache=50000; - db6
 * 15:04 paladox: MariaDB [metawiki]> set global table_definition_cache=40000; - db5
 * 15:04 paladox: MariaDB [metawiki]> set global table_open_cache=50000; - db5
 * 14:43 paladox: restart php7.3-fpm on mw* and lizardfs6
 * 14:42 paladox: MariaDB [metawiki]> set global table_definition_cache=40000; - db4
 * 14:42 paladox: MariaDB [metawiki]> set global table_open_cache=50000; - db4
 * 01:02 paladox: restarted mariadb - db4 as ran out of space

2020-02-23

 * 22:27 paladox: root@mw4:/srv/mediawiki/w/maintenance# sudo -u www-data php resetUserTokens.php --wiki=onepiecebountyrushwiki
 * 21:15 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php updateArticleCount.php --wiki=hispanowiki --update
 * 21:14 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php updateArticleCount.php --wiki=ucroniaswiki --update
 * 20:57 paladox: root@cloud2:/home/paladox# swapoff -a
 * 20:56 paladox: root@cloud1:/home/paladox# swapoff -a
 * 20:50 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php updateArticleCount.php --wiki=hispanowiki --update

2020-02-22

 * 02:55 paladox: drop grafana db on db4
 * 02:49 paladox: apt-get install puppet-agent mariadb-client-10.1 mariadb-client-core-10.1 mariadb-common - misc2
 * 02:43 paladox: remove php/nginx from misc2
 * 01:45 paladox: drop database piwik on db5

2020-02-21

 * 23:31 paladox: migrating matomo to new infra
 * 21:11 paladox: root@mw1:/srv/mediawiki/w/extensions/ManageWiki/maintenance# sudo -u www-data php populateNamespacesWithDefaults.php --wiki=unicodesubsetswiki --overwrite
 * 21:10 paladox: MariaDB [mhglobal]> delete from mw_namespaces where ns_dbname = 'unicodesubsetswiki';
 * 21:09 paladox: root@mw1:/srv/mediawiki/w/extensions/ManageWiki/maintenance# sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki=unicodesubsetswiki --overwrite
 * 10:04 Reception123: RENAME browndustwiki to braveninewiki (db + script)

2020-02-20

 * 01:02 paladox: CHANGE MASTER TO MASTER_HOST='db6.miraheze.org', MASTER_USER='replica', MASTER_PASSWORD='xxxx', MASTER_LOG_FILE='mysql-bin.000004', MASTER_LOG_POS=1044075691; - db6
 * 00:51 paladox: GRANT REPLICATION SLAVE ON *.* TO 'replica'@'%' IDENTIFIED BY 'xxxx'; - db6

2020-02-18

 * 20:54 paladox: "ip addr del fe80::216:3cff:fe25:9a18/64 dev ens3" - db4
 * 20:54 paladox: "ip -6 address add 2a00:d880:5:d6c::2/64 dev ens3" - db4
 * 20:25 paladox: added www-data to ssl-cert on mw[23] lizardfs6
 * 20:19 paladox: fixed perms on /etc/ssl/private on lizardfs6
 * 19:50 paladox: restart redis to see if it will fix the 503s
 * 19:33 paladox: restart php7.3-fpm on mw* and lizardfs6
 * 19:32 paladox: restart php7.3-fpm on mw1
 * 19:11 paladox: enable puppet on jobrunner1
 * 16:44 paladox: upgrade mariadb-client on misc1
 * 16:43 paladox: upgrade grafana on misc1
 * 15:10 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10784;
 * 15:10 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10783;
 * 15:10 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10782;
 * 15:09 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10781;
 * 15:08 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10780;
 * 15:08 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10778;
 * 15:07 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10777;
 * 15:07 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10776;
 * 15:04 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10775;
 * 15:04 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10765;

2020-02-17

 * 16:46 paladox: restart nginx on cp4
 * 16:24 paladox: ALTER TABLE /*_*/oathauth_users DROP COLUMN secret, DROP COLUMN scratch_tokens;
 * 16:12 paladox: renaming kujumwiki to archlinuxfamilyorgwiki per T5241
 * 16:04 paladox: renaming endeavourosdewiki to archlinuxfamilywiki per T5242
 * 15:39 paladox: sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki=pcsdscotlandwiki --overwrite
 * 12:50 Reception123: disabled PlavorSeol on Phabricator
 * 12:04 Reception123: renamed ungamewiki to libertygamewiki (db + script)

2020-02-16

 * 23:52 paladox: MariaDB [allthetropeswiki]> truncate objectcache;
 * 22:56 Southparkfan: set read_only=1 on db6
 * 20:31 Reception123: re-inserted cw_requests from db6 after misclick
 * 19:29 Reception123: removed spam from cw_requests (inappropriate comment)
 * 14:49 paladox: [14:43:09] <+paladox>	!log hacked lizardfs6 Database.php (added persitents mode and removed)
 * 14:44 paladox: hacked lizardfs6 Database.php (added persitents mode and removed)
 * 14:43 paladox: [14:41:46] <+paladox>	!log reboot lizardfs6
 * 14:31 paladox: restart php on mw[123]

2020-02-15

 * 19:48 paladox: depooled mw[4567]
 * 18:41 paladox: repool cp4

2020-02-14

 * 20:28 paladox: backing up phabricator db
 * 20:04 Southparkfan: install atop/iotop/iperf (with fw rule for latter) on db4/db6
 * 19:34 paladox: [19:11:31] <+paladox>	!log set gluster file system to read only on lizardfs6
 * 19:26 Southparkfan: added firewall rule on db4 to allow traffic from db6 (v4/v6) to port 3306
 * 19:26 Southparkfan: dynamically set server_id=4 on db4 and server_id=6 on db6
 * 19:26 Southparkfan: dumping db4's databases to db6 in /home/dbcopy
 * 18:52 Southparkfan: disable puppet on db[456]
 * 14:45 paladox: tried upgrading phabricator to latest on master, failed and rollbacked
 * 10:59 Reception123: umount /mnt/mediawiki-static ; mount /mnt/mediawiki-static on mw3
 * 02:35 paladox: stopped parsoid, removed /srv/parsoid from misc4
 * 02:33 paladox: apt-get upgrade - misc4
 * 02:25 paladox: increase puppet2 disk to 15G
 * 01:42 Southparkfan: decommissioned misc3
 * 01:33 Southparkfan: shutdown misc3 - testing before permanent decom
 * 01:27 paladox: drop phabricator_* on db4
 * 01:23 Southparkfan: decommission cp2
 * 00:56 John: soft depooling of misc1 as ns2 via DNS entry change
 * 00:53 Southparkfan: shutdown cp2 nginx for transferring nginx logs to cp7:/root

2020-02-13

 * 22:05 paladox|UKInEU: install parted on phab1
 * 22:02 paladox|UKInEU: increase phab1 disk to 20g
 * 21:31 paladox|UKInEU: increased phab1 cores to 2
 * 20:45 paladox|UKInEU: install mariadb-client-10.1 on misc4 (to backup the db)
 * 20:43 paladox|UKInEU: stop phd & nginx on misc4. Dump phabricator db, migrate to db6 and migrate phabricator to phab1
 * 20:15 paladox|UKInEU: restart php7.3-fpm on mw3
 * 18:33 paladox|UKInEU: restarted nginx on mw3
 * 01:17 paladox|UKInEU: installed rsync on gluster1
 * 00:04 paladox|UKInEU: reboot test1

2020-02-12

 * 23:44 paladox|UKInEU: lizardfs6 not 67
 * 23:43 paladox|UKInEU: apt-get upgrade - lizardfs67
 * 22:34 paladox|UKInEU: beginning migration from lizardfs6 to gluster1
 * 20:42 Zppix: create more various dummy dbs to prevent issues with wiki creations
 * 16:10 Reception123: move toxicfandomsandhatedomswiki from db4 to db5
 * 15:48 Reception123: moved zhdelwiki from db4 to db5
 * 15:32 Reception123: PURGE BINARY LOGS BEFORE '2020-02-12 12:00:00'; on db4

2020-02-11

 * 23:13 Zppix: created various dummy dbs using sql.php on mw1 to prevent creation of new wikis with existing hostnames to prevent potential issues
 * 01:51 paladox|UKInEU: reboot cp2
 * 01:33 paladox|UKInEU: MariaDB [mhglobal]> update mw_namespaces set ns_additional = '{"wgNamespacesToPostIn":true,"wgVisualEditorAvailableNamespaces":true}' where ns_dbname = 'civclassicwiki' and ns_namespace_name = '';
 * 01:01 paladox|UKInEU: rm /srv/mediawiki/w/cache/managewiki/** on mw[123], lizardfs6

2020-02-10

 * 19:42 Reception123: MariaDB [(none)]> DROP DATABASE isabellafan2wiki;
 * 19:42 Reception123: deleted isabellafan2wiki
 * 02:22 paladox|UKInEU: probe gluster1 from lizardfs(gluster) but do not add it as a brick

2020-02-08

 * 15:29 paladox|UKInEU: rename diaroleplayswiki to sparkleswiki per https://phabricator.miraheze.org/T5188
 * 15:27 paladox|UKInEU: MariaDB [shiropediawiki]> delete from nl_newsletters where nl_id = 5;
 * 15:26 paladox|UKInEU: MariaDB [shiropediawiki]> delete from nl_newsletters where nl_id = 6;
 * 15:20 paladox|UKInEU: renaming idiotpediawiki to nocyclopediawiki per https://phabricator.miraheze.org/T5208
 * 15:15 paladox|UKInEU: delete & drop ewrikiawiki per https://phabricator.miraheze.org/T5213
 * 15:14 paladox|UKInEU: drop cdwwik, gameshowswiki, glaucuswiki, receptionflamewarswiki, stargazerwikiwiki, thechasewikiwiki, toslawwiki
 * 10:14 Reception123: PURGE BINARY LOGS BEFORE '2020-02-08 08:00';

2020-02-07

 * 19:22 Zppix: create mw[67]wiki using sql.php
 * 19:21 Zppix: created cloud1 and cloud2 dbs using sql.php

2020-02-06

 * 01:41 paladox|UKInEU: reboot mw3

2020-02-05

 * 19:49 paladox|UKInEU: upgrade php on lizardfs6
 * 19:47 paladox|UKInEU: upgrade php on mw1
 * 19:46 paladox|UKInEU: upgrade php on mw2
 * 19:43 paladox|UKInEU: upgrade php on mw3
 * 19:39 paladox|UKInEU: restart php7.3-fpm on mw3
 * 19:31 Zppix: initsitestats and rebuildrc for ungamewiki after import

2020-02-04

 * 20:47 Southparkfan: locked phab auth config
 * 20:41 Southparkfan: unlocked phab auth config (temp)

2020-02-03

 * 17:28 paladox|UKInEU: root@mw1:/var/log/mediawiki# rm *.log.*
 * 17:27 paladox|UKInEU: rm /var/log/mediawiki/*.log*
 * 17:26 paladox|UKInEU: rm /var/log/mediawiki/*.log

2020-02-02

 * 15:50 paladox|UKInEU: > MatomoAnalytics::addSite( 'loginwiki' );
 * 15:50 paladox|UKInEU: > MatomoAnalytics::deleteSite( 'loginwiki' );
 * 15:48 paladox|UKInEU: > MatomoAnalytics::addSite( 'foxmaskeventwiki' );
 * 15:48 paladox|UKInEU: > MatomoAnalytics::deleteSite( 'foxmaskeventwiki' );
 * 15:48 paladox|UKInEU: > MatomoAnalytics::addSite( 'davshiqwiki' );
 * 15:48 paladox|UKInEU: > MatomoAnalytics::deleteSite( 'davshiqwiki' );
 * 14:03 paladox|UKInEU: remove hack from cron - db5
 * 04:34 paladox|UKInEU: hack /var/spool/cron/crontabs/root and run /home/paladox/purge-binary.sh every 1 minute.
 * 00:38 paladox|UKInEU: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php rebuildall.php --wiki=csydeswiki
 * 00:02 paladox|UKInEU: drop database piwik on db5; create databse db5 & import from backup. To see if this will reduce the disk space it's using from ~75gb.

2020-02-01

 * 23:19 paladox|UKInEU: drop database uncyclomirrorwiki on db4 as migrated to db5
 * 22:29 paladox|UKInEU: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=metawiki --type=gzip
 * 22:23 paladox|UKInEU: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=toxicfandomsandhatedomswiki --type=gzip
 * 22:18 paladox|UKInEU: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=uncyclomirrorwiki --type=gzip
 * 22:14 paladox|UKInEU: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=altversewiki --type=gzip
 * 12:26 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki Tundra "Just Amelia" --logwiki=metawiki

2020-01-31

 * 16:48 paladox|UKInEU: MariaDB [gmcwiki]> INSERT INTO content_models (model_id,model_name) VALUES (4,'Scribunto');
 * 16:47 paladox|UKInEU: MariaDB [gmcwiki]> INSERT INTO content_models (model_id,model_name) VALUES (2,'css');
 * 16:47 paladox|UKInEU: MariaDB [gmcwiki]> INSERT INTO content_models (model_id,model_name) VALUES (3,'javascript');
 * 16:47 paladox|UKInEU: MariaDB [gmcwiki]> INSERT INTO content_models (model_id,model_name) VALUES (5,'json');
 * 16:46 paladox|UKInEU: MariaDB [gmcwiki]> INSERT INTO content_models (model_id,model_name) VALUES (1,'wikitext');
 * 16:43 paladox|UKInEU: MariaDB [gmcwiki]> INSERT INTO slot_roles (role_id,role_name) VALUES (1,'main');
 * 16:37 paladox|UKInEU: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php cleanupUsersWithNoId.php --wiki=gmcwiki --force -p gmcwiki

2020-01-30

 * 18:02 paladox: apt-get upgrade - mw[123] (gluster, php and puppet-agent + few other packages)
 * 17:55 paladox: apt-get upgrade - lizardfs6 (gluster, php and puppet-agent + few other packages)

2020-01-28

 * 23:42 paladox: upgrade puppet-agent & salt on misc1
 * 23:41 paladox: upgrade php - misc1
 * 23:36 paladox: upgrade grafana - misc1
 * 22:00 paladox: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=ungamewiki --type=gzip
 * 21:58 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/guiasdobrasil.com.br "', null ) WHERE wiki_dbname = 'guiaslocaiswiki';
 * 19:32 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php nukeNS.php --wiki=thefinalrumblewiki --all --ns=3001 --delete
 * 13:13 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki Toshinori "Agent Mace" --logwiki=metawiki
 * 00:53 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=files
 * 00:52 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=subcats
 * 00:52 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=pages

2020-01-27

 * 22:59 paladox: sync; echo 1 > /proc/sys/vm/drop_caches (lizardfs6 & db4)
 * 22:50 paladox: sudo /sbin/sysctl -w vm.drop_caches=3 on lizardfs6 and db4 temporarily
 * 22:28 Zppix: zppix@mw1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --report=1 --no-local-users --username-prefix "백괴게임" --wiki ungamewiki /home/zppix/uncyclogame-full-2019-05-15.xml (T5148)
 * 20:02 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php deleteBatch.php --wiki=thefinalrumblewiki list (Deleting Message_Wall:%)
 * 19:57 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php deleteBatch.php --wiki=thefinalrumblewiki list2 (Deleting User_blog_comment:%)
 * 19:55 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php deleteBatch.php --wiki=thefinalrumblewiki list (Deleting User_blog:%)
 * 19:51 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php nukeNS.php --wiki=thefinalrumblewiki --all --ns=3000 --delete
 * 17:28 paladox: root@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance# sudo -u www-data php migrateAccount.php --wiki=wikifinanzawiki --userlist=./userlist --homewiki=wikifinanzawiki --attachmissing --auto
 * 15:26 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki ZapperGlaze49 "TheChisatoFan" --logwiki=metawiki

2020-01-26

 * 22:15 paladox: revert hack on mw[123] & lizardfs6
 * 20:12 paladox: hacked LS on mw[123] & lizardfs6 to switch off wgUseInstantCommons temporarily
 * 17:51 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=pages/subpages/files
 * 17:41 paladox: root@mw2:/srv/mediawiki/w/maintenance# sudo -u www-data php refreshLinks.php --wiki=ucroniaswiki
 * 17:41 paladox: root@mw2:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=pages
 * 16:48 paladox: revert hack on mw[123] & lizardfs6
 * 15:18 paladox: hacked LS on mw[23] & lizardfs6 to switch off wgUseInstantCommons temporarily
 * 15:17 paladox: hacked LS on mw1 to switch off wgUseInstantCommons temporarily
 * 15:06 paladox: restart php7.3-fpm on mw1 and mw3
 * 15:05 paladox: restart php7.3-fpm on mw2 and lizardfs6
 * 06:37 Reception123: deleted and dropped gmcwiki (T5150)

2020-01-25

 * 18:40 Zppix: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki crappygameswiki -u "Zppix" -r "Requested - T5153" /home/zppix/crappygameswiki_batchdelete.txt
 * 18:21 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki TTyranny "Tarbiz" --logwiki=metawiki
 * 17:52 paladox: reinstall php on misc2
 * 17:47 paladox: rotate redis.log on misc2
 * 15:02 paladox: restart php7.3-fpm && nutcracker on mw3
 * 15:00 paladox: restart php7.3-fpm && nutcracker on mw2
 * 00:16 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/wiki.udia.ca "', null ) WHERE wiki_dbname = 'udiawiki';
 * 00:15 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/islamkosh.tk "', null ) WHERE wiki_dbname = 'ipediawiki';
 * 00:15 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/glp.ink "', null ) WHERE wiki_dbname = 'cnwiki';
 * 00:14 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/wiki.counterculturelabs.org "', null ) WHERE wiki_dbname = 'cclwiki';

2020-01-24

 * 23:28 paladox: restart php7.3-fpm on mw3 and lizardfs6
 * 23:28 paladox: restart php7.3-fpm on mw2
 * 23:27 paladox: restart php7.3-fpm on mw1
 * 14:12 paladox: MariaDB [mhglobal]> update mw_permissions set perm_removegroupsfromself = '[]' where perm_removegroupsfromself = 'null';
 * 14:10 paladox: Remove abusefilter-hide-log,abusefilter-hidden-log,abusefilter-modify-global,abusefilter-private,abusefilter-private-log,aft-oversighter,autocreateaccount,bigdelete,centralauth-lock,centralauth-oversight,centralauth-rename,centralauth-unmerge,centralauth-usermerge,checkuser,checkuser-log,createwiki,editincidents,editothersprofiles,editothersprofiles-private,flow-suppress,globalblock,globalblock-exempt,globalgroupmembership,globalgrouppermissions,hideuser,interwiki,managewiki-restricted,managewiki-editdefault,moderation-checkuser,mwoauthmanageconsumer,mwoauthmanagemygrants,mwoauthsuppress,mwoauthviewprivate,mwoauthviewsuppressed,oathauth-disable-for-user,oathauth-view-logrenameuser,requestwiki,siteadmin,stopforumspam,suppressionlog,suppressrevision,usermerge,userrights,userrights-interwiki,viewglobalprivatefiles,viewpmlog,viewsuppressed from all wikis.
 * 14:04 paladox: MariaDB [mhglobal]> update mw_permissions set perm_addgroupstoself = '[]' where perm_addgroupstoself = 'null';
 * 14:02 paladox: rm -rf /srv/mediawiki/w/cache/managewiki/** on mw* and lizardfs6
 * 14:01 paladox: MariaDB [mhglobal]> update mw_permissions set perm_removegroupsfromself = '[]' where perm_removegroupsfromself = 'null';
 * 14:01 paladox: MariaDB [mhglobal]> update mw_permissions set perm_removegroups = '[]' where perm_removegroups = 'null';
 * 14:01 paladox: MariaDB [mhglobal]> update mw_permissions set perm_addgroups = '[]' where perm_addgroups = 'null';
 * 13:59 paladox: MariaDB [mhglobal]> update mw_permissions set perm_addgroupstoself = '[]' where perm_addgroupstoself = 'null';
 * 13:38 paladox: remove oathauth-disable-for-user from all groups.

2020-01-23

 * 00:29 paladox_UK_IN_EU: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=spectrobeswiki --search-recursively --skip-dupes ./Spectrobes_Wiki_Images

2020-01-22

 * 23:54 paladox: apt-get upgrade cp3
 * 22:51 paladox: remove oathauth-disable-for-user from all wikis
 * 19:29 paladox_UK_IN_EU: remove oauthauth-disable-for-user from all wikis
 * 18:31 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=spectrobeswiki --report=1 spectrobes_pages_full.xml - per request through email
 * 18:31 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=spectrobeswiki --report=1 spectrobes_pages_full.xml.7z - per request through email
 * 06:52 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki WageUglyDollandMarlaPlaymobil12 "Just Wage" --logwiki=metawiki

2020-01-21

 * 21:29 paladox: restart nutcracker on mw1
 * 14:53 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wiesepediawiki /home/reception/Wiesepedia-Datadump[0,A-D] on mw1
 * 14:47 Reception123: PURGE BINARY LOGS before '2020-01-20 22:00:00' on db4
 * 14:46 Reception123: manually remove custom domain for hypewiki
 * 14:37 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=gmcwiki /home/reception/graalmilitary_pages_full.xml on mw3
 * 14:33 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=spectrobeswiki /home/reception/spectrobes_pages_full.xml on mw3

2020-01-20

 * 17:05 paladox: ALTER TABLE /*_*/searchindex MODIFY COLUMN si_title TEXT BINARY NOT NULL; - simswiki
 * 16:59 paladox: ran ./2.sh on db4 (basically finding all titles that are in User Blog Comment on wikia and setting ns id to 3007 from 0)
 * 16:54 Zppix: UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/nonsensopedia.org "', null ) WHERE wiki_dbname = 'nonsensopediawiki'; on mw1 using sql.php
 * 01:07 paladox: ALTER TABLE /*_*/searchindex MODIFY COLUMN si_title varchar(300) BINARY NOT NULL default ''; - simswiki (applying
 * 00:49 paladox: MariaDB [simswiki]> update page set page_namespace = 3007 where page_namespace = 3008;
 * 00:42 paladox: remove hack from mw1
 * 00:39 paladox: ALTER TABLE /*_*/searchindex MODIFY COLUMN si_title TEXT BINARY NOT NULL; (applying https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/565765/8 to simswiki)
 * 00:11 paladox: MariaDB [simswiki]> update page set page_namespace = 3008 where page_namespace = 3007;

2020-01-19

 * 22:30 paladox: hacked rebuildtextindex.php on mw1 to output some debugging info
 * 21:45 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php rebuildall.php --wiki=simswiki
 * 21:44 paladox: ALTER TABLE /*_*/searchindex MODIFY COLUMN si_title varchar(255) BINARY NOT NULL default ''; - simswiki (fixing an issue when rebuilding search index)
 * 21:38 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=hispanowiki batch2 - T5122
 * 21:37 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=ucroniaswiki batch - T5122
 * 21:34 paladox: root@mw1:/srv/mediawiki/w/extensions/ManageWiki/maintenance# sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki=toastycraftwiki --overwrite -T5119
 * 21:34 paladox: root@mw1:/srv/mediawiki/w/extensions/ManageWiki/maintenance# sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki=aranziaronzowiki --overwrite -T5119
 * 17:45 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=hispanowiki batch2
 * 17:34 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=ucroniaswiki batch
 * 15:05 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=loginwiki --report=1 Wikipedia-20200119150417.xml
 * 14:57 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=loginwiki --report=1 Wikipedia-20200119145611.xml
 * 13:54 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php rebuildall.php --wiki=simswiki
 * 08:51 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki=thefinalrumblewiki /home/reception/deletebatch.txt
 * 01:23 paladox: restart php7.3-fpm on lizardfs6
 * 00:25 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=thefinalrumblewiki --skip-dupes --search-recursively TFR_Wikia_Images/ (per request on discord)

2020-01-18

 * 22:15 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=simswiki --report=1 --no-updates sims_pages_full.xml
 * 22:09 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php deleteBatch.php --wiki=thefinalrumblewiki batch - per request on discord
 * 20:46 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=simswiki --report=1 sims_pages_full.xml
 * 17:09 paladox: that script is adding "urlshortener-create-url" to * for all wikis
 * 17:08 paladox: root@mw1:/home/paladox# ./foreachwikiindblist /srv/mediawiki/dblist/all.dblist
 * 16:58 paladox: delete from mw_permissions where perm_dbname = '131parkhurstwiki' and perm_group = "all.dblist";
 * 15:44 paladox: delete from mw_permissions where perm_dbname = '131parkhurstwiki' and perm_group = "\\*" LIMIT 1;
 * 15:44 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = "\\*" LIMIT 1;
 * 15:44 paladox: delete from mw_permissions where perm_dbname = '131parkhurstwiki' and perm_group = "\\*" LIMIT 1;
 * 15:44 paladox: delete from mw_permissions where perm_dbname = '131parkhurstwiki' and perm_group = "\\*" LIMIT 1;
 * 15:44 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = "\\*" LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = "\*" LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = '\*' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = '+' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = '+' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '1920swiki' and perm_group = '+' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '1920swiki' and perm_group = '"*"' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '131parkhurstwiki' and perm_group = '"*"' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '131parkhurstwiki' and perm_group = '\*' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = '\*' LIMIT 1;
 * 15:42 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = '"*"' LIMIT 1;
 * 14:16 Reception123: manually remove custom domain from teessidehackspacewiki

2020-01-17

 * 19:10 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=thefinalrumblewiki /home/reception/thefinalrumble_pages_full.xml
 * 02:11 paladox: rm -rf /srv/mediawiki/w/cache/manage*/** on mw* and lizardfs6
 * 01:48 paladox: restoring mw_permissions from a newish backup (after running "delete from mw_permissions where perm_group = "\*";" which seemed to updated 4,000 results)
 * 01:43 paladox: update mw_permissions set perm_permissions = '["autocreateaccount","createaccount","edit","createpage","createtalk", "writeapi","viewmywatchlist","editmywatchlist","viewmyprivateinfo", "editmyprivateinfo","editmyoptions","abusefilter-log","abusefilter-log-detail", "abusefilter-view","autocreateaccount","centralauth-merge", "oathauth-enable","urlshortener-create-url"]' where perm_group = '*' and perm_dbname = 'default';
 * 00:12 paladox: apt-get upgrade - misc2

2020-01-16

 * 23:53 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/ManageWiki/maintenance/modifyGroupPermission.php --addperms="urlshortener-create-url,urlshortener-manage-url" sysop
 * 23:47 paladox: update mw_permissions set perm_permissions = '["editsitecss","upload_by_url","editsitejson","editsitejs","skipcaptcha","block","createaccount","delete","deletedhistory","deletedtext","undelete","editinterface","editusercss","edituserjson","edituserjs","import","importupload","move","move-subpages","move-rootuserpages","move-categorypages","patrol","autopatrol","protect","editprotected","rollback","upload","reupload","reupload-shared","unwatchedpages","autoconfirmed","editsemiprotected","ipblock-exempt","blockemail","markbotedits","apihighlimits","browsearchive","noratelimit","movefile","unblockself","suppressredirect","mergehistory","managechangetags","deletechangetags","abusefilter-modify","abusefilter-modify-restricted","abusefilter-revert","deletelogentry","deleterevision","abusefilter-log-detail","override-antispoof","globalblock-whitelist","massmessage","nuke","tboverride","titleblacklistlog", "delete-dump","generate-dump","view-dump","urlshortener-create-url","urlshortener-manage-url"]' where perm_group = 'sysop' and perm_dbname = 'default';
 * 02:34 paladox: reboot misc4
 * 02:15 paladox: downgraded puppetdb on puppet1 - broke ssl connection to postgresql
 * 01:48 paladox: root@db4:/home/paladox# sudo service postgresql restart
 * 01:44 paladox: on puppet1
 * 01:44 paladox: apt-get upgrade puppet-agent puppetdb puppetdb-termini puppetserver salt-common salt-minion
 * 01:43 paladox: upgrade puppet-agent on cp4

2020-01-15

 * 08:26 Reception123: sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki AnimalLovingGuy "ScrewSociety" --logwiki=metawiki

2020-01-14

 * 23:10 paladox: reinstall cp2 as debian buster
 * 23:09 paladox: restart php7.3-fpm on mw2 for config change
 * 23:09 paladox: turn TUN off on cp2
 * 23:06 paladox: restart php7.3-fpm on mw3 for config change
 * 22:59 paladox: rebuild lc on mw* and lizardfs6
 * 17:39 paladox: apt-get upgrade - lizardfs6
 * 01:43 paladox: sudo -u www-data php update.php --wiki=sagan4wiki --quick --force - mw1
 * 01:21 paladox: MariaDB [sagan4wiki]> ALTER TABLE /*_*/cargo_tables ADD field_helper_tables text NOT NULL;

2020-01-13

 * 15:32 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/wintermoor.net "', null ) WHERE wiki_dbname = 'wintermoorwiki';

2020-01-12

 * 23:29 paladox: removed redis from all mediawiki servers
 * 15:25 paladox: repooled lizardfs6
 * 15:15 paladox: temporarily depool lizardfs6
 * 08:32 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php purgeList.php --all --purge --wiki twinkletestwiki + rebuildall.php
 * 08:06 Reception123: ran rebuildall on twinkletestwiki

2020-01-11

 * 23:35 paladox: [23:30:52] <+paladox>	!log switched TUN off on misc2
 * 23:34 paladox: apt-get upgrade on misc2
 * 22:52 paladox: upgrade nodejs on misc3
 * 22:48 paladox: resizing misc3 - https://clientarea.ramnode.com/index.php?rp=/announcements/431/More-SSD-for-Standard-KVM.html
 * 17:04 paladox: apt-get upgrade - bacula1
 * 16:58 paladox: root@bacula1:/bacula/backup# rm DB*
 * 16:57 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* and lizardfs6
 * 14:59 Reception123: DELETED and DROPPED dbs https://phabricator.miraheze.org/P247
 * 14:52 Reception123: reception@mw1:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/deleteWikis.php --wiki loginwiki --delete Reception123
 * 11:12 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki GasMask0217 "KP619" --logwiki=metawiki
 * 10:29 Reception123: run fixStuckGlobalRename.php on all wikis for username above
 * 10:18 Reception123: sudo -u www-data php fixStuckGlobalRename.php --wiki=aawikiwiki PaperClip "FireBarrier101" --logwiki=metawiki
 * 04:25 paladox: repool lizardfs6
 * 04:18 paladox: depool lizardfs6
 * 04:18 paladox: repool mw3
 * 04:12 paladox: depool mw3
 * 04:12 paladox: repool mw2
 * 04:08 paladox: depool mw2
 * 04:07 paladox: repool mw1
 * 04:04 paladox: depool mw1

2020-01-09

 * 22:07 paladox: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=inforevivalwiki --report=1 Wiki-20200109132453.xml
 * 21:20 paladox: rebuild lc on mw* and lizardfs6
 * 19:34 paladox: ALTER TABLE /*$wgDBprefix*/matomo ADD INDEX matomo_wiki (matomo_wiki); - db4
 * 19:17 paladox: MariaDB [mhglobal]> ALTER TABLE /*$wgDBprefix*/cw_wikis ADD INDEX wiki_dbname (wiki_dbname); -db4
 * 18:59 paladox: MariaDB [mhglobal]> ALTER TABLE /*$wgDBprefix*/mw_namespaces ADD INDEX ns_dbname (ns_dbname); -db4
 * 18:59 paladox: MariaDB [mhglobal]> ALTER TABLE /*$wgDBprefix*/mw_permissions ADD INDEX perm_dbname (perm_dbname); - db4
 * 00:57 paladox: restart php7.3-fpm on mw2

2020-01-08

 * 21:37 paladox: restart mysql on db4
 * 21:33 paladox: restart mysql on db5
 * 19:56 paladox: restart php7.3-fpm on lizardfs6 and mw*
 * 17:16 paladox: SET GLOBAL thread_cache_size = 100; - (db4|db5)
 * 16:52 paladox: MariaDB [(none)]> SET GLOBAL thread_cache_size = 80;
 * 16:33 paladox: MariaDB [(none)]> SET GLOBAL thread_pool_stall_limit = 10;
 * 16:14 paladox: MariaDB [(none)]> SET GLOBAL slow_query_log = 'OFF';
 * 16:10 paladox: MariaDB [(none)]> SET GLOBAL long_query_time = 10;
 * 16:04 paladox: MariaDB [(none)]> SET GLOBAL long_query_time = 15;
 * 16:04 paladox: MariaDB [(none)]> SET GLOBAL slow_query_log = 'ON';
 * 15:30 paladox: restart php7.3-fpm on mw* and lizardfs6
 * 15:21 Reception123: purge binary logs BEFORE '2020-01-08 08:00'

2020-01-07

 * 17:16 paladox: root@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance# sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki --logwiki=metawiki ErtasVideos Hendicted

2020-01-06

 * 20:22 Reception123: reception@mw2:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki testwiki /home/reception/Wikipedia-20200106202002.xml

2020-01-05

 * 19:47 paladox: rebuild lc on mw* and lizardfs6
 * 00:25 paladox: ./bin/auth strip --user Paladox --all-types (misc4)

2020-01-04

 * 10:13 Reception123: reception@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki loginwiki /home/reception/TestWiki-20200104100040.xml

2020-01-02

 * 18:48 paladox: depool, repool and restart php7.3-fpm on mw3
 * 18:44 paladox: depool, repool and restart php7.3-fpm on mw2
 * 18:43 paladox: restart php7.3-fpm on mw1
 * 18:31 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=radioscanningtwwiki --report=1 X.xml
 * 03:10 paladox: rm -rf /srv/mediawiki/w/cache/managewiki/** on mw* and lizardfs6
 * 03:01 paladox: root@test1:/srv/mediawiki/w/extensions/StopForumSpam/maintenance# sudo -u www-data php updateBlacklist.php --wiki=sdiywiki
 * 02:44 paladox: upgrade phabricator on misc4
 * 02:42 paladox: MariaDB [(none)]> drop database uncyclopediakrwiki;
 * 02:30 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/renameWiki.php --wiki=loginwiki --rename uncyclopediakrwiki truthwiki paladox

2020-01-01

 * 17:49 paladox: root@misc1:/home/paladox# deluser labster
 * 15:53 Reception123: sudo -u www-data php fixStuckGlobalRename.php --wiki=200xwiki PaperClip "FireBarrier101" --logwiki=metawiki
 * 14:24 Reception123: MariaDB [phabricator_herald]> UPDATE herald_rule SET isDisabled = '1' WHERE id = '21'; (empty)
 * 04:36 paladox: root@mw1:/srv/mediawiki/w# /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/QuizGame/sql/patches/patch-add-default-choice_answer_count.sql
 * 00:37 paladox: restart varnish cp2,cp4