Tech:Server admin log/2019

2019-01-24

 * 14:39 paladox: upgrade phabricator on misc4
 * 14:09 John: removed editothersprofiles from all groups
 * 03:47 paladox: ran modifyGroupPermission.php per T4046

2019-01-23

 * 17:38 Southparkfan: issues a varnish ban for req.url ~ ^/robots.txt on all cache proxies
 * 17:30 paladox: running generateMirahezeSitemap.php on mw1

2019-01-22

 * 22:27 paladox: upgrading apt on all servers
 * 12:12 John: set wiki_inactive_exempt = 1 for all wikis in $wgCreateWikiInactiveWikisWhitelist
 * 12:06 John: ALTER TABLE cw_wikis ADD COLUMN wiki_inactive_exempt TINYINT NOT NULL DEFAULT '0' AFTER wiki_inactive_timestamp;

2019-01-21

 * 15:14 John: falling back to Redis for session and main caching temporarily

2019-01-20

 * 22:36 paladox: reboot misc2 to reduce ram
 * 22:32 paladox: setting prometheus master up on misc4
 * 22:30 paladox: shutting prometheus master down on misc2
 * 15:17 paladox: upgrade phabricator to fix a bug on misc4
 * 15:15 Southparkfan: set global local_infile=OFF; on db4
 * 01:18 paladox: reboot cp5 for kernel upgrade.

2019-01-19

 * 19:34 paladox: dropping "math" table from all wiki's
 * 19:32 paladox: drop "math" table from metawiki, no longer exists in the math extension
 * 17:24 paladox: running lc on mw[23]
 * 15:29 paladox: rename wiki intpwiki to taswinwiki
 * 15:19 paladox: upgrade phabricator on misc4
 * 00:48 paladox: upgrading phabricator on misc4
 * 00:36 paladox: fix namespaces lpawiki, mindgartwiki, ianyinwiki, gllwiki, naturepediawiki, bscwiki, stiffwiki, qboxnextwiki, craedopediawiki
 * 00:35 paladox: fix namespaces hominwiki, kmtfwiki, utoowiki, indiawiki, platcomicswiki, swashbucklerwiki
 * 00:35 paladox: fix namespaces gbgschulpflegschaftwiki, deutschlandburgerwiki, anarchymcwiki, ckdocswiki, cuttingroomwiki, uas5wiki
 * 00:34 paladox: fix namespaces readysteadyyetiwiki, twimwikiwiki, vechicleswiki, ipuwikiwiki, soprometwiki, wintermoorwiki, gjtwiki, atlaswiki
 * 00:34 paladox: fix namespaces ebmscwiki, arbreapalabrewiki, ekodengewiki, dellplanetwikiwiki, tlmwiki, fantasypediawiki

2019-01-18

 * 22:56 paladox: running lc on mw*
 * 16:42 paladox: running insertNamespaces.php on battleaxewiki, ff5fjfwiki, yurinikolaiwiki, fanmadetvchannelswiki, woerterbuchwiki, amcwiki, seerwiki, chronostormwiki and iriswiki
 * 16:38 paladox: running insertNamespaces.php on alientoasterdogwiki, mrgshowswiki, djruddhawiki, igwikiwiki, testghrwiki
 * 16:37 paladox: running insertNamespaces.php on goldpediawiki
 * 16:35 paladox: insert default namespaces into 'default' from loginwiki
 * 16:31 paladox: running insertNamespaces.php on uncyclopedia2wiki
 * 00:04 paladox: running lc on mw*

2019-01-17

 * 22:08 paladox: repool mw3
 * 22:04 paladox: depool mw3
 * 21:26 paladox: repool mw2
 * 21:16 paladox: depool mw2
 * 21:15 paladox: repool mw1
 * 20:50 paladox: depool mw1
 * 09:44 John: reverted my last log message change (because test1wiki shouldn't be deleted, change works though)
 * 09:43 John: setting wiki_deleted to 1 for test1wiki temp
 * 09:25 John: ALTER TABLE cw_wikis ADD COLUMN wiki_deleted TINYINT NOT NULL DEFAULT '0' AFTER wiki_closed_timestamp, ADD COLUMN wiki_deleted_timestamp BINARY(14) NULL AFTER wiki_deleted;

2019-01-16

 * 21:40 Southparkfan: removed packages proxysql and mysql-client from test1
 * 20:51 Southparkfan: install proxysql package on test1
 * 20:33 Southparkfan: 'usermod -a -G ssl-cert www-data' on test1

2019-01-14

 * 01:19 Southparkfan: CentralAuthUser::newFromId( 20792 )->quickInvalidateCache; (previous redis purge didn't work)
 * 01:16 Southparkfan: deleted key WANCache✌️global:centralauth-user:79eda1a42924fbb966df82627046c9fe from redis to force cache purge
 * 00:23 paladox: doing a emergency restart of mariadb

2019-01-13

 * 19:56 MacFan4000: deleted permission cache files for weatherwiki
 * 19:52 MacFan4000: DELETE FROM mw_permissions WHERE perm_dbname='weatherwiki' AND perm_group='interface-admin'
 * 19:26 paladox: running lc on mw* and test1
 * 16:51 paladox: running lc on mw* and test1

2019-01-12

 * 22:27 paladox: running script on misc4 to remove all files that have ttl != null
 * 21:54 paladox: bin/storage optimise on misc4
 * 17:39 Reception123: DELETED FROM CW_WIKIS and DROPPED cuttingroomwiki uas5wiki testdeletewiki
 * 17:34 Southparkfan: dropped all database matching pattern southparkfan(11111|123|12345|[3-6])wiki (were created for CW issues)
 * 17:33 paladox: reboot misc4 to see if it will erase temp files
 * 17:27 paladox: uninstalling lizardfs from misc4
 * 16:47 paladox: unmounting /mnt/mediawiki-static (images will start disapearing)
 * 16:37 paladox: depool mw1
 * 15:09 paladox: stopping phd on misc4

2019-01-11

 * 23:05 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist modifyGroupPermission.php --addperms="editsitecss,editsitejs,editsitejson" sysop on mw1
 * 22:17 paladox: deleting files on misc4
 * 20:46 paladox: restarted puppetserver (semmed stuck on 100% cpu)
 * 15:05 Southparkfan: enabled puppet on misc2 again
 * 15:03 Southparkfan: deleted all redis keys matching pattern *:highlight:*
 * 14:57 Southparkfan: deleted all redis keys matching pattern *:diff:*
 * 14:56 paladox: 14:15:14 <+paladox>    !log reboot misc2 to clean ram
 * 14:55 Southparkfan: 15:50:27 <+SPF|Cloud> !log disable puppet on misc2 to resolve redis trouble
 * 14:52 paladox: 14:15:14 <+paladox>    !log reboot misc2 to clean ram
 * 14:21 paladox: [14:15:14] <+paladox>	!log reboot misc2 to clean ram
 * 14:17 paladox: reboot misc2 to clean ram
 * 06:53 paladox: ran modifyGroupPermission.php to add editsite* to all users in sysop group
 * 06:19 paladox: adding "editinterface" to default on sysop group
 * 05:37 paladox: [03:10:04] <+paladox>    !log repool mw3
 * 05:37 paladox: [02:50:04] <+paladox>    !log repool mw2
 * 05:37 paladox: [02:27:34] <+paladox>    !log repooling mw1 and depool mw2 and mw3
 * 05:37 paladox: [00:36:33] <+MacFan4000>    !log changing default branch for @miraheze/mediawiki to REL1_32
 * 05:37 paladox: [00:14:16] <+paladox>    !log upgrading all wiki's to 1.32
 * 05:37 paladox: [00:10:15] <+paladox>    !log upgrading metawiki to 1.32
 * 05:37 paladox: [00:06:41] <+paladox>    !log putting all wiki's into read only - pending upgrade to 1.32
 * 05:37 paladox: [23:29:43] <+paladox>    !log depool mw1
 * 05:36 paladox: [03:10:04] <+paladox>	!log repool mw3
 * 05:36 paladox: [02:50:04] <+paladox>	!log repool mw2
 * 05:36 paladox: [02:27:34] <+paladox>	!log repooling mw1 and depool mw2 and mw3
 * 05:35 paladox: [00:36:33] <+MacFan4000>	!log changing default branch for @miraheze/mediawiki to REL1_32
 * 05:35 paladox: [00:14:16] <+paladox>	!log upgrading all wiki's to 1.32
 * 05:35 paladox: [00:10:15] <+paladox>	!log upgrading metawiki to 1.32
 * 05:35 paladox: [00:06:41] <+paladox>	!log putting all wiki's into read only - pending upgrade to 1.32
 * 05:35 paladox: [23:29:43] <+paladox>	!log depool mw1
 * 00:06 paladox: putting all wiki's into read only - pending upgrade to 1.32

2019-01-10

 * 23:29 paladox: depool mw1
 * 18:39 paladox: Renaming svwiki to psl631testwiki per T3967

2019-01-09

 * 01:40 Southparkfan: dropped southparkfan[2-5]wiki wikis
 * 01:34 Southparkfan: executed once again
 * 01:19 Southparkfan: executed previous queries again
 * 01:16 Southparkfan: MariaDB [metawiki]> delete from echo_notification where notification_timestamp like '20190109%' and notification_user = 2;
 * 01:16 Southparkfan: MariaDB [metawiki]> delete from echo_event where event_type = 'wiki-creation';
 * 00:18 Southparkfan: testing changes in CreateWiki on mw1

2019-01-08

 * 01:50 MacFan4000: deleting iwwiki

2019-01-07

 * 23:46 paladox: sudo chmod -R 0770 /etc/puppetlabs/puppet/ssl-keys
 * 23:45 MacFan4000: (paladox) sudo chmod 0770 /etc/puppetlabs/puppet/ssl-keys on puppet1
 * 23:43 paladox: sudo chmod 0700 /etc/puppetlabs/puppet/ssl-keys on puppet1
 * 22:31 MacFan4000: macfan@mw1:/$ sudo -u www-data php /srv/mediawiki/w/maintenance/dumpBackup.php --wiki tesminadventureswiki --full --output gzip:/mnt/mediawiki-static/dumps/tesminadventureswiki.xml.gz
 * 12:29 MacFan4000: DELETE FROM mw_namespaces WHERE ns_dbname='xedwiki' AND ns_namespace_name='Greek_talk'; (per stewards noticeboard)
 * 12:29 MacFan4000: DELETE FROM mw_namespaces WHERE ns_dbname='xedwiki' AND ns_namespace_name='Greek'; (per stewards noticeboard)
 * 12:28 MacFan4000: DELETE FROM mw_namespaces WHERE ns_dbname='xedwiki' AND ns_namespace_name='English_talk'; (per stewards noticeboard)
 * 12:28 MacFan4000: DELETE FROM mw_namespaces WHERE ns_dbname='xedwiki' AND ns_namespace_name='English'; (per stewards noticeboard)

2019-01-06

 * 17:00 paladox: running lc on mw* and test1
 * 14:48 paladox: importing mw dump on bigcartoonwiki (on mw1)
 * 14:45 paladox: importing mw dump for nowchesswiki
 * 14:06 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/GlobalBlocking/globalblocking.sql on mw3
 * 04:09 paladox: repool mw1
 * 04:04 paladox: delete from mw_namespaces where ns_dbname = 'metawiki' and ns_namespace_name = 'TestHello_talk';
 * 04:04 paladox: delete from mw_namespaces where ns_dbname = 'metawiki' and ns_namespace_name = 'TestHello';
 * 00:55 paladox: running lc on mw*

2019-01-05

 * 17:10 paladox: depool mw1
 * 01:56 paladox: upgrading phabricator on misc4

2019-01-04

 * 20:44 Southparkfan: stop jobrunner on mw3
 * 20:32 Southparkfan: install iftop on mw[1-3]
 * 00:26 Southparkfan: ran update cw_requests set cw_user = 44876 where cw_id = 6919;

2019-01-03

 * 23:48 paladox: running update set commands on db4 to fix cw_user for requests that have the wrong id.
 * 23:36 paladox: update cw_requests set cw_user = 44922 where cw_dbname = 'giantesswiki'; on db4
 * 23:35 paladox: update cw_requests set cw_user = 44929 where cw_dbname = 'dawsontechboothwiki'; on db4
 * 23:28 paladox: ALTER TABLE cw_comments MODIFY cw_comment_user INT(10) NOT NULL; on db4
 * 23:27 paladox: ALTER TABLE cw_requests MODIFY cw_user INT(10) NOT NULL; on db4
 * 16:04 paladox: sudo -u www-data php scriptMira2.php --wiki=nonciclopediawiki on mw1
 * 15:49 paladox: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist maint*/manageJobs.php --action=delete --type="LocalGlobalUserPageCacheUpdateJob" on mw1
 * 04:26 paladox: sudo -u www-data /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist maint*/manageJobs.php --action=delete --type="LocalGlobalUserPageCacheUpdateJob" on mw1
 * 03:46 paladox: creating MacFan4000 on icingaweb2
 * 03:40 paladox: creating Voidwalker on icingaweb2
 * 02:38 paladox: recreating namespace Guide and Guide_talk for wiki paliwikiwiki
 * 02:33 paladox: delete from mw_namespaces where ns_dbname = 'paliwikiwiki' and ns_namespace_id = 3001; on db4
 * 02:33 paladox: delete from mw_namespaces where ns_dbname = 'paliwikiwiki' and ns_namespace_id = 3000; on db4
 * 01:15 paladox: MatomoAnalytics::addSite( 'untrashwiki' ) on mw1

2019-01-02

 * 00:15 paladox: update mw_namespaces set ns_aliases = '["Image_talk"]' where ns_dbname = 'metawiki' and ns_namespace_name = 'File_talk'; on db4
 * 00:14 paladox: update mw_namespaces set ns_aliases = '["Image"]' where ns_dbname = 'metawiki' and ns_namespace_name = 'File'; on db4

2019-01-01

 * 21:17 paladox: repool mw1
 * 21:07 paladox: delete all namespaces cdb files on mw*
 * 18:00 Reception123: MariaDB [phabricator_herald]> UPDATE herald_rule SET isDisabled = '1' WHERE id = '10'; (inactive user)
 * 17:59 Reception123: MariaDB [phabricator_herald]> UPDATE herald_rule SET isDisabled = '1' WHERE id = '28'; (inactive user)
 * 16:23 MacFan4000: macfan@mw3:~$ sudo service jobrunner restart && sudo service jobchron restart
 * 16:21 MacFan4000: running jobs everywhere in a screen on mw1
 * 16:19 paladox: running scriptMira2.php on mw1 (it creates a local account on metawiki and loginwiki)
 * 15:05 paladox: run UPDATE globaluser SET gu_email = 'xxx' WHERE gu_name = 'Revenant Lord'; (where xxx is a email)
 * 15:01 paladox: depool mw1