Tech:Server admin log/2018

2018-04-05

 * 13:37 John: db4 has SSL capabilities now

2018-04-04

 * 23:21 John: installed db4 as stretch
 * 22:47 John: reinstall db4 as stretch
 * 22:43 John: unsigned db4 from puppet1
 * 17:19 Reception123: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*

2018-04-03

 * 19:35 Southparkfan: restart ganglia-monitor on puppet1
 * 19:23 John: restart ganglia-monitor on mw1
 * 14:02 John: add cw_custom to cw_requests on metawiki
 * 13:20 John: rebooted db2, excessive server load and CPU usage
 * 09:07 revi: add @miraheze/i18n group to WikiDiscover repo

2018-04-02

 * 19:25 revi: changed phabricator global timezone settings to UTC instead of Europe/London
 * 18:44 Reception123: added mw-admins team to @miraheze/WikiDiscover
 * 18:36 Reception123: added IRC webhook to @miraheze/WikiDiscover
 * 14:45 Reception123: repooled mw3
 * 14:36 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw1 and mw2
 * 14:21 Reception123: clear /srv/mediawiki/config on mw3
 * 14:21 Reception123: depool mw3
 * 14:21 Reception123: repool mw2
 * 14:20 Reception123: manually pull changes on misc1
 * 13:57 Reception123: clear /srv/mediawiki/w on mw2
 * 13:57 Reception123: depool mw2
 * 13:57 Reception123: repool mw1
 * 12:08 Reception123: cleared /srv/mediawiki/w on mw1
 * 12:07 Reception123: depool mw1

2018-04-01

 * 19:49 Reception123: disabled puppet on puppet1
 * 19:19 Reception123: signed certs for all servers on puppet1 except mw* and cp*
 * 15:59 Reception123: reinstalling puppet1 on Debian 9
 * 13:41 Reception123: signed cp5 cert on puppet1
 * 06:54 Reception123: sudo -u www-data cp -r /srv/mediawiki/w/extensions/SocialProfile/avatars & awards /mnt/mediawiki-static/garrettcountyguidewiki

2018-03-31

 * 21:31 John: running manageInactiveWikis.php to close old wikis and mark inactive wikis as wiki_inactive
 * 14:56 Reception123: reinstall test1 to test puppet1 install
 * 06:33 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw*
 * 05:32 Reception123: sudo -u www-data php importDump.php --wiki wiki1776wiki /home/reception/Wikipedia-20180216182614.xml && sudo -u www-data php importDump.php --wiki unionwiki /home/reception/Wikipedia-20180313163344.xml && sudo -u www-data php importDump.php --wiki wiki1776wiki /home/reception/Wikipedia-20180313163344.xml on mw3

2018-03-30

 * 22:40 John: made security task public
 * 19:38 Reception123: made John a phab admin
 * 08:38 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki loraldwiki /home/reception/lorald_pages_current.xml
 * 08:04 Reception123: upgraded phabricator

2018-03-29

 * 17:54 Reception123: manually removed "debian" and "db5" from services file

2018-03-27

 * 16:21 revi: latest command for T2904
 * 16:21 revi: revi@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-admin php namespaceDupes.php --fix --wiki=tmewiki
 * 15:57 revi: sudo puppet agent -t on mw(1-3)

2018-03-25

 * 14:48 Reception123: cleared /tmp on mw2

2018-03-24

 * 18:57 Reception123: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*
 * 07:28 Reception123: repool mw1
 * 06:59 Reception123: refresh /srv/mediawiki on mw1
 * 06:59 Reception123: depool mw1

2018-03-23

 * 16:47 Reception123: restarted nginx on mw1

2018-03-21

 * 19:54 Reception123: upgraded phabricator
 * 19:51 Reception123: restarted icinga

2018-03-20

 * 07:12 Reception123: MariaDB [phabricator_herald]> UPDATE herald_rule SET isDisabled = '1' WHERE id = '24';

2018-03-12

 * 17:44 revi: revi@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php initSiteStats.php --update --wiki=reviwikiwiki
 * 17:35 Reception|away: reception@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki reviwikiwiki /home/reception/reviwiki.xml
 * 17:28 Reception|away: DELETED from cw_wikis and DROPPED reviwikiwiki (recreation)

2018-03-11

 * 12:49 revi: make https://phabricator.miraheze.org/H31 stricter

2018-03-10

 * 16:51 revi: phab project import was created while ago, forgot to log
 * 15:41 Reception123: created https://phabricator.miraheze.org/H32 for secure mail

2018-03-09

 * 17:05 Southparkfan: fixed ownership of www-data on mw1

2018-03-08

 * 17:20 Southparkfan: create global_preferences table on centralauth db, drop global_preferences table from metawiki db
 * 17:11 Reception123: ran /srv/mediawiki/w/extensions/GlobalPreferences/schema.sql on metawiki
 * 16:24 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw*
 * 12:35 Reception123: reception@mw3:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-d                                                                                       ata php migrateAccount.php --wiki metawiki --userlist /home/reception/whsapushli                                                                                        st.txt --auto

2018-03-07

 * 19:46 Reception123: deleted and dropped testdeletewiki
 * 15:22 Reception123: restart jobrunners on mw2 and mw3 and manually run jobs on mw1
 * 12:19 Reception123: sudo -u www-data foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/runJobs.php on mw1
 * 09:40 Reception123: sudo -u www-data php /srv/piwik/console core:archive on misc2

2018-03-06

 * 11:50 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw*
 * 06:23 Reception123: upgraded phabricator

2018-03-05

 * 08:09 Reception123: repool mw1
 * 08:06 Reception123: sudo -u www-data php importDump.php --wiki magnaversewiki [and enigmawiki] /home/reception/Infoboxes for Magnaverse - 1.xml
 * 07:53 Reception123: clear out /srv/mediawiki on mw1 and re-run puppet (failing to pull mediawiki)
 * 07:53 Reception123: depool mw1
 * 07:42 Reception123: on sudo -u www-data php importDump.php --wiki almanachdebauswicwiki /home/reception/Gelo+e+Fogo+wiki-20180303164425.xml mw2
 * 07:42 Reception123: sudo -u www-data php importDump.php --wiki almanachdebauswicwiki /home/reception/Gelo+e+Fogo+wiki-20180303163927.xml on mw3

2018-03-03

 * 22:22 Southparkfan: restoring bacula's db3 backup on db4

2018-03-01

 * 21:52 Southparkfan: removed nginx from nfs1
 * 21:02 Southparkfan: replicating db3 to db4 (without read lock, to prevent downtime)
 * 20:14 Southparkfan: downtimed db4 in icinga, moved spfwiki from db4 to db2
 * 00:55 Southparkfan: db2 was holding (deleted) 20GB file mysql-slow.log.1 open. truncated it to reclaim 20GB space
 * 00:52 Southparkfan: db2 was holding file mysql-slow.log.1 open. by truncating it using ":>/proc/763/fd/54" I reclaimed 12GB space
 * 00:29 Southparkfan: purged binary logs @ db3
 * 00:14 Southparkfan: drop db southparkfanwiki on db2

2018-02-28

 * 23:52 Southparkfan: reboot db4 after kernel upgrade
 * 23:42 Southparkfan: reboot db4
 * 17:28 Reception123: removed db5 and "debian" from storeconfigs on puppet and moved /etc/icinga/config/puppet_services.cfg to a backup file then ran puppet to regenerate the list of services
 * 16:43 Reception123: stopped then started apache2 on misc1
 * 16:38 Reception123: removed /var/lib/apt/lists/* and regenerated with apt-get update
 * 16:26 Reception123: sudo service icingabot restart
 * 10:14 Reception123: manually edit (renew) phab.miraheze.wiki on misc1

2018-02-27

 * 21:18 Southparkfan: sysctl vm.swappiness=1 on db4
 * 21:12 Southparkfan: restarted jobrunners
 * 18:55 Reception123: manually run sudo -u www-data foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/runJobs.php on mw1 to alleviate queue
 * 16:58 Southparkfan: truncate allthetropeswiki.objectcache

2018-02-26

 * 14:26 Reception123: chowned /etc/icinga/config/puppet_services.cfg to icinga:icinga
 * 14:15 Reception123: retry moving /etc/icinga/config/puppet_services.cfg to a backup file then ran puppet to regenerate the list of services
 * 08:38 Reception123: > UPDATE cw_wikis SET wiki_closed = '0' WHERE wiki_dbname = '131parkhurstwiki'; (requested, ManageWiki broken)

2018-02-25

 * 19:56 Southparkfan: ALTER TABLE cw_wikis ADD COLUMN wiki_inactive_timestamp BINARY(14) AFTER wiki_inactive;
 * 19:26 Southparkfan: ALTER TABLE cw_wikis ADD COLUMN wiki_closed_timestamp BINARY(14) AFTER wiki_closed; and ALTER TABLE cw_wikis ADD COLUMN wiki_inactive smallint(6) AFTER wiki_closed_timestamp;
 * 18:55 Southparkfan: deleted the following wikis: https://phabricator.miraheze.org/P43
 * 17:57 Reception123: manually restarted icinga as well (required for the changes to take effect)
 * 17:56 Reception123: manually reloaded icinga
 * 17:52 Reception123: moved /etc/icinga/config/puppet_services.cfg to a backup file then ran puppet to regenerate the list of services
 * 17:18 Southparkfan: decommissioned db4
 * 09:32 Reception123: sudo -u www-data php dumpBackup.php --wiki testwiki --full > /home/reception/testwiki25022018.xml on mw3
 * 09:31 Reception123: sudo -u www-data php dumpBackup.php --wiki metawiki --full > /home/reception/metawiki25022018.xml on mw2
 * 06:40 Reception123: reception@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki templatewiki /home/reception/Wikipedia-20180224165206.xml

2018-02-24

 * 14:38 Reception123: sudo service icingabot restart
 * 13:23 Reception123: sudo -u www-data php importDump.php --wiki templatewiki /home/reception/Wikipedia-20180224132217.xml on mw3

2018-02-23

 * 22:39 Southparkfan: restarted ganglia-monitor on parsoid1+bacula1
 * 21:53 Southparkfan: cleaned db entries, dropped databases and removed nfs dirs for wikis: https://phabricator.miraheze.org/P42
 * 21:18 Southparkfan: cleaned db entries, dropped databases and removed nfs dirs for wikis: https://phabricator.miraheze.org/P41
 * 20:40 Southparkfan: cleaned db entries, dropped databases and removed nfs dirs for wikis: https://phabricator.miraheze.org/P40
 * 20:15 Southparkfan: cleaned db entries, dropped databases and removed nfs dirs for wikis: https://phabricator.miraheze.org/P39
 * 16:52 Reception123: reinstalling test1 (for further testing)
 * 16:05 Reception123: added grants for icinga on db4
 * 14:32 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw*
 * 09:42 Reception123: purge binary logs before '2018-02-23 00:00:00'; on db3
 * 07:36 Reception123: sudo service icingabot restart
 * 06:28 Reception123: moved to db3; dropped on db2 aerosswiki diagnostikwiki encyclopaediawiki networksdatabasewiki pocketmonsterswiki scruffy2wiki summerolympicswiki ussewiki

2018-02-22

 * 18:08 Reception123: DELETED from cw_wikis and DROPPED on db2 contextwiki contigoconnectwiki conuconwiki  coopwiki coorwiki coreywiki cpswiki crwikiwiki crxtrdudewiki cswiki ctuvnwiki cubewiki cultofparetowiki customerdesignwiki cwlagusawiki

2018-02-20

 * 20:30 Reception|away: purge binary logs before '2018-02-20 20:00:00'; on db2
 * 09:02 revi: create https://phabricator.miraheze.org/H31 to autoclose spam
 * 06:19 Reception|away: purge binary logs before '2018-02-20 03:00:00'; on db3

2018-02-19

 * 18:12 Reception|away: DELETED FROM cw_wikis and DROPPED on db2 childrensinstitutewiki chocowiki chromiumrosewiki chronicleswiki cinquiemesaisonwiki clarifirewiki clarkfamilywiki classroomsonelightglobalwiki clearicewiki clementsworldbuildingwiki cmpcrswiki cobwebmudwiki code7wiki codebaseclubwiki cofradiawiki cognitowiki colaboratoriowiki colfaxknightswiki collarwiki columbiasjpiwki comidawiki commonsensewiki composerohswiki concertpowerswiki
 * 05:57 Reception123: moved to db3 and dropped on db2 revitwiki unionwiki wiki1776wiki

2018-02-17

 * 19:59 Southparkfan: purged binary logs on dbx
 * 12:37 Reception123: sudo service icingabot restart
 * 07:47 Reception123: brokenworldwiki brooklynrfbwiki brxdwiki bssentialswiki bsuirwiki budgetenvywiki bulldogsracingtestwiki burnoutwiki bzdetopediawikibzswiki cagencywikiwiki cancerwiki cantelli25wiki casinotermswiki caspwiki cassallenwiki  casswiki cbecwiki cecwiki celuwiki chakorduvorwiki chaldunwiki cherryitwiki
 * 07:47 Reception123: DELETED FROM CW_WIKIS and DROPPED bobwiki boftwiki bookswiki bootstrapwiki bottlefarmwiki answerswiki boytvofwiki braindumpwiki bremenwikiwiki bribriwiki broadwaveswiki brofactorywiki

2018-02-16

 * 06:13 Reception|away: cleared /tmp on mw2
 * 05:47 Reception|away: DELETED FROM cw_wikis and DROPPED automotivewiki avidkbwiki azurawiki backyardsportsfanonwiki2wiki banglawiki barigawiki bcmschinawiki beckleynotebookwiki becuzwiki beezerwiki begawktdwiki bellmarewiki bertelwiki besselingwiki bhenuawiki bialowiezawiki bibliowiki bilatwiki bimbowiki bio10a2016garwiki bitphoriawiki bizmodsimwiki blackdesertfanonwiki blackstumpwiki blindfoldstudioswiki

2018-02-15

 * 18:17 Reception|away: reception@mw2:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki templatewiki /home/reception/Wikipedia-20180215181624.xml
 * 18:01 Reception|away: moved to db3 and dropped on db2 nextlevelwikiwiki recentiawiki teriawiki zendariwiki
 * 17:40 Reception|away: moved supernamuwiki from db3 to (temporary) db4, dropped from db3

2018-02-14

 * 19:31 Reception|away: DELETED from CW_WIKIS and dropped ictjudikaturawiki (T2724)
 * 19:17 Reception|away: moved templatewiki and mc2wiki to db3 and dropped on db2

2018-02-12

 * 11:12 revi: https://phabricator.miraheze.org/H30 updated to automatically subscribe CoCC members

2018-02-11

 * 06:57 Reception123: purge binary logs before '2018-02-11 06:00:00'; on db2
 * 06:52 Reception123: sudo -u www-data foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/QuizGame/sql/quizgame.sql
 * 06:48 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw*

2018-02-10

 * 12:03 Reception123: DROPPED wikis below
 * 10:56 Reception123: DELETED FROM CW_WIKIS altenpflegewiki altwiki amningwiki amparodenaswiki annuwikiwiki anon4wiki antoinewiki apenlinjewiki apituwiki apparchitectwiki apslaxwiki aptotecwiki aquivawiki arabudlandwiki arbabilowiki archipediawiki argentinachicagowiki arguwikiwiki arkheonwiki armlaptopwiki askerwiki astawiki atkinswiki atplwiki audiowikiwiki autismadminswiki
 * 07:31 Reception123: sudo -u www-data php importDump.php --wiki podpediawiki /home/reception/roosterteeth_pages_current.xml on mw2

2018-02-08

 * 20:58 Southparkfan: mw1 repooled
 * 19:12 Southparkfan: mw1 emergency depool
 * 19:10 Reception123: purge binary logs before '2018-02-08 16:00:00'; on db2
 * 18:47 Southparkfan: on cp4: varnishlog -g raw -i Backend_health -a -w /var/log/varnish/varnish-backend.log -D
 * 16:37 Southparkfan: ran git submodule update --init in /srv/piwik

2018-02-06

 * 06:46 Reception123: moved liberartwiki to libreartwiki

2018-02-04

 * 13:27 Reception123: moved mistralcupwiki from db4 to db2 (created when metawiki was moved to db4; not declared in Database.php)
 * 09:40 Reception123: moved to db3 and dropped on db2 ctswiki fourleafficswiki karniarutheniawiki madciclistawiki nvcwiki songcontestswiki unionnorteamericanawiki valentinaprojectwiki
 * 05:55 Reception123: moved metawiki back to db2 (has to be on default or else there are CW errros)

2018-02-03

 * 23:53 Southparkfan: dropped worlduniversityandschoolwiki, tmewiki and vandalwiki, all wikis have been transferred to c3
 * 22:44 Southparkfan: moved metawiki to c3, dropped database on db2. backup is in /home/dbcopy for emergencies
 * 18:19 Reception123: upgraded phabricator

2018-02-02

 * 20:04 Reception123: purge binary logs before '2018-02-02 18:00:00'; on db2
 * 18:28 Southparkfan: MariaDB [metawiki]> update cw_wikis set wiki_dbcluster = 'c3' where wiki_dbname = 'testwiki';
 * 15:48 PuppyKun: destroyed db5 last night

2018-02-01

 * 22:03 PuppyKun: root@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance# sudo -u www-data php fixStuckGlobalRename.php --wiki=maiasongcontestwiki --logwiki=metawiki "WEMO666" "Chriskk01b"
 * 21:55 PuppyKun: MariaDB [centralauth]> delete from renameuser_status where ru_wiki="lothuialethwiki"; (query ok, 1 row affected)
 * 21:44 PuppyKun: removeDeletedWikis for lothuialethwiki
 * 18:59 Southparkfan: cleaned puppet certificate info about db5
 * 18:48 PuppyKun: hard rebooting db5
 * 17:29 revi: update phabricator maniphest form 14 ( https://phabricator.miraheze.org/transactions/editengine/maniphest.task/view/14/ ) preamble
 * 02:12 PuppyKun: rebooted db5
 * 02:05 PuppyKun: rebooted db5
 * 02:00 PuppyKun: started hard reboot of db5.miraheze.org (unresponsive to ssh)
 * 01:48 PuppyKun: doing first puppet run on db5.cloud.online.net (??? that was the name that showed up when i did --wait-for-cert on db5.mh.o and the fingerprint matched)

2018-01-29

 * 10:27 revi: create #spam in Phabricator, for spammy tasks

2018-01-28

 * 18:59 Reception123: moved to db3 and dropped on db2 madgendersciencewiki ubrwikiwiki nonbinarywiki linenwiki beatstationwiki daesocialwiki uncyclopediawiki musicarchivewiki thegreatwarwiki christianmusictempwiki hellointernetwiki christianmusicwiki spritopiawiki magnaversewiki
 * 18:24 Reception123: disabled puppet on db3
 * 15:33 Reception123: revi added to @miraheze puppet-users team

2018-01-27

 * 20:36 Reception123: purge binary logs before '2018-01-27 15:00:00'; on db2

2018-01-25

 * 17:04 Reception|away: installed db4 and signed cert on puppet1

2018-01-24

 * 18:48 Reception|away: upgraded phabricator

2018-01-23

 * 06:45 Reception|away: purge binary logs before '2018-01-23 00:00:00';

2018-01-22

 * 20:26 Reception|away: sudo rm -rf /var/lib/puppet/reports on puppet1 (disk space running out)

2018-01-21

 * 10:50 Reception123: sudo -u www-data php mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki test1wiki and rebuild LC on mw*
 * 08:31 Reception123: archived H2 (no longer used)

2018-01-20

 * 08:09 Reception123: sudo -u www-data php dumpBackup.php --wiki metawiki --full > /home/reception/metawiki.xml on mw3

2018-01-19

 * 18:22 Reception123: rebooted test1 (puppetmaster)
 * 17:43 Reception123: sudo -u www-data php importDump.php --wiki pwikiwiki /home/reception/arkcls-wiki-20180117.xml on mw1

2018-01-18

 * 17:54 Reception123: re-enabled puppet on db3
 * 17:54 Reception123: moved databases to db3 and dropped on db2 dariawikiwiki viagroupiawiki truecapitalistwiki emulationwiki cpiwiki ylscwiki hirapediawiki whentheycrywiki utamacrosswiki sumroletaericwiki
 * 17:30 Reception123: disabled puppet on db3

2018-01-16

 * 06:37 Reception123: renewed phab.miraheze.wiki on misc1

2018-01-09

 * 06:49 Reception123: upgraded Phabricator

2018-01-08

 * 06:02 Reception123: restarted Phabricator daemons

2018-01-07

 * 02:33 PuppyKun: rebooted mw2 to fix puppet hangs?

2018-01-06

 * 16:19 Reception123: reinstalling bacula1 (size change; backups not running)

2018-01-05

 * 01:34 PuppyKun: MariaDB [centralauth]> update globaluser set gu_hidden="suppressed" where gu_id in (9192,9193,9188,9190,13117,13118,13119,13140,12407,12405,12404,11327,12293,11296,11292,11280,11481,11716,11717,11196,11197,11198,11199,11201,11202,11203,11204,11205); Query OK, 28 rows affected (0.01 sec)

2018-01-04

 * 18:23 Reception123: sudo -u www-data php importImages.php --wiki dariawikiwiki --search-recursively /home/reception/images on mw1
 * 08:41 Reception123: re-reinstall test1 on debian 9

2018-01-03

 * 16:47 Reception123: reinstalling test1 on debian 9

2018-01-02

 * 15:13 Reception123: started jobrunner on mw2 (moved from test1)
 * 15:09 Reception123: restarted phabricator daemons
 * 08:24 Reception123: sudo -u www-data php rebuildLocalisationCache.php --wiki test1wiki on mw*

2018-01-01

 * 23:26 PuppyKun: renewed publictestwiki.com
 * 15:05 Reception123: cleared /tmp of mw2 (full)
 * 09:08 Reception|away: sudo -u www-data php importDump.php --wiki dariawikiwiki /home/reception/DariaWikiDump_12.31.17.xml on mw2

Information
All times are in UTC+0/GMT.

Archives: 2015, 2016, 2017