Tech:Server admin log/2018

2016-08-06

 * 02:14 John: MW 1.27 tables on all wikis, jobrunner disabled on mw1, running jobs to ensure completion right now

2016-08-05

 * 23:14 John: disable puppet on mw1 + make allthetropeswiki readonly
 * 23:06 John: updated parsoid to 0.5.1
 * 22:01 John: removing mediawiki on mw2, re-running puppet to check all changes are in git, potential MW1.27 upgrade shortly
 * 21:23 John: silences mw2 in icinga for 2 hours
 * 21:16 John: applying REL1_27 code on mw2, monitoring mw1 just in case
 * 21:13 John: removing local commits to Widget extension, causing excessive puppet runs...
 * 05:08 Reception123: ran sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki extloadwiki on mw*

2016-08-04

 * 21:05 Southparkfan: disabling puppet on misc2
 * 19:17 Reception123: ran sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki extloadwiki on mw*
 * 13:23 PuppyKun: sudo chown -R www-data:www-data /srv/mediawiki/w/extensions/Widgets/compiled_templates on mw*

2016-08-03

 * 18:48 John: removed manifests in private git - entirely hiera ran
 * 17:56 John: renamed existing private hiera variables to new ones
 * 17:50 John: converted all private information to hiera [internal note]
 * 17:18 Southparkfan: restarted gmond on all servers
 * 17:15 John: restart ganglia-monitor on all servers
 * 16:52 John: restarted gmetad on misc2
 * 16:19 John: removed binlogs on db1
 * 15:45 PuppyKun: ran removeDeletedWikis.php after deleting the database, removed 18 local users
 * 13:59 PuppyKun: created password for user ndkilla in icinga
 * 13:45 Reception123: ran reception@mw1:/srv/mediawiki/w/extensions/MirahezeMagic/maintenance$ sudo -u www-data php manageInactiveWikis.php --wiki metawiki --warn --close
 * 13:33 PuppyKun: ensured Reception123's shell account was created
 * 12:45 PuppyKun: dropped database for ayudasocialwiki and removed it from deleted.dblist
 * 12:42 PuppyKun: restarted nginx again
 * 12:26 PuppyKun: MariaDB [metawiki]> delete from cw_wikis where wiki_dbname="ayudasocialwiki";
 * 12:16 PuppyKun: purged www and non-www thinkingliquid.org domains from cache
 * 12:14 PuppyKun: restarted nginx on appservers
 * 11:56 PuppyKun: dropped database nddbwiki
 * 11:56 PuppyKun: deleted nddbwiki from cw_wikis

2016-08-02

 * 18:03 Southparkfan: wiped ganglia data for vultr.guest + bacula1
 * 16:24 PuppyKun: attempted to drop easygovewiki database, didn't exist. emptying deleted.dblist
 * 16:24 PuppyKun: MariaDB [(none)]> DROP DATABASE pulsarwiki; Query OK, 123 rows affected
 * 16:19 PuppyKun: MariaDB [(none)]> DROP DATABASE gameswiki; Query OK, 123 rows affected
 * 16:17 PuppyKun: MariaDB [(none)]> DROP DATABASE quantixwiki; Query OK, 123 rows affected
 * 16:12 PuppyKun: MariaDB [metawiki]> delete from cw_wikis where wiki_dbname="gameswiki";
 * 16:09 PuppyKun: MariaDB [metawiki]> delete from cw_wikis where wiki_dbname="quantixwiki";
 * 03:52 PuppyKun: deleted cw_wikis entry for pulsarwiki per founder request on my talk page

2016-08-01

 * 18:36 John: all private repos reference git-private.miraheze.org now, not misc1.miraheze.org / misc1's IP

2016-07-31

 * 20:20 John: phabricator upgraded
 * 20:17 John: upgrading phabricator
 * 13:28 John: started phd daemons
 * 09:34 Southparkfan: restarted ganglia-monitor on bacula1

2016-07-30

 * 22:02 John: reinstalled bacula1
 * 21:21 John: taking bacula1 down for reinstall
 * 21:20 Southparkfan: restarted php5-fpm to pick up config changes

2016-07-29

 * 16:08 PuppyKun: restarted nginx on (mw|cp)[12]

2016-07-27

 * 11:26 Southparkfan: installed security updates on mwservers

2016-07-26

 * 22:03 Southparkfan: purged some binlogs on db1
 * 21:57 Southparkfan: restarted php5-fpm on both mwservers

2016-07-25

 * 20:31 penguinstyles: imported xml dump for 'dwplivewiki' per request (T547) && rebuilt recent changes for 'dwplivewiki'
 * 17:28 Southparkfan: starting Piwik upgrade
 * 13:28 PuppyKun: varnish purge thelonsdalebattalion.(miraheze.org|co.uk)
 * 13:24 PuppyKun: restarted nginx on (mw|cp)[12]
 * 12:16 John: rebooting mw2, unresponsive
 * 11:33 John: rebooting bacula1
 * 05:00 penguinstyles: imported xml dump for 'starsetonlinewiki' per request (T540)

2016-07-23

 * 13:16 Southparkfan: imported dump for droneregulationwiki

2016-07-22

 * 19:12 Southparkfan: ran sudo varnishadm 'ban req.http.Host == wiki.dwplive.com && req.url ~ "^/w/load.php"' on cp2
 * 15:14 John: MariaDB [(none)]> DROP DATABASE easygovwiki;
 * 12:35 Southparkfan: restarted ganglia-monitor on bacula1 after reboot for unknown reason

2016-07-20

 * 21:38 Southparkfan: truncated l10n_cache for loginwiki too
 * 21:32 Southparkfan: truncated metawiki.l10n_cache. Saved approximately 873MB of space
 * 21:16 Southparkfan: removed STATIC-0011 volume to regain space
 * 10:50 Southparkfan: ran updateArticleCount.php for lomithradienwiki with --update

2016-07-19

 * 18:46 Southparkfan: started jobrunner on mw1
 * 18:44 John: rebooted cp1; fixed networking
 * 18:43 John: rebooted mw1; fixed networking
 * 16:24 John: rm allthetropes-images.gz on cp1; +2.2G
 * 13:47 Southparkfan: restarted mysql, was flapping

2016-07-18

 * 15:58 John: rm'd MySQL bin logs -16G on db1
 * 12:33 PuppyKun: ndkilla@mw1:/srv/mediawiki/w/extensions/MirahezeMagic/maintenance$ sudo php manageInactiveWikis.php --wiki metawiki --warn --close (466 warned, 0 closed)

2016-07-17

 * 11:52 Southparkfan: created @cp1:/srv/mediawiki-static/1year for some CSS assets

2016-07-16

 * 23:24 John: updated Phabricator
 * 22:13 John: removed local commits on mw1 for 5 extensions that are old, no longer necessary or otherwise
 * 20:27 John: all wikis should be consistent with database tables now.
 * 20:19 John: ensured all databases (1143) have MsCalendar database tables
 * 20:14 John: ensured all databases (943) have AJAXPoll database tables
 * 20:05 John: ensured all databases (943) have VoteNY database tables
 * 20:00 John: ensured all databases (779) have Comments database tables
 * 19:55 John: ensured all databases (655) have WikiLove database tables
 * 19:51 John: ensured all databases (651) have WikiForum database tables
 * 19:45 John: ensured all databases (652) have SocialProfile database tables
 * 19:15 John: ensured all databases (146) have PageTriage database tables
 * 19:10 John: ensured all databases (6) have Flow database tables
 * 19:08 John: ensured all databases (4) have Translate database tables
 * 19:04 John: ensured all databases (1) have BetaFeatures database tables
 * 19:03 John: ensured all databases (1) have Poll database tables
 * 19:02 John: ensured all databases (4) have TimedMediaHandler database tables
 * 18:59 John: ensured all databases (4) have TitleKey database tables

2016-07-15

 * 17:41 Southparkfan: increased HSTS for miraheze.org to 6 months
 * 11:58 PuppyKun: ran update.php twice on jbkwwiki (first run failed because of global_block_whitelist table or whatever that common thing is)

2016-07-14

 * 22:37 Southparkfan: installed security updates on mw1 & mw2
 * 19:42 Southparkfan: varnishadm 'ban req.http.Host == southparkfan.miraheze.org' @ cp1

2016-07-13

 * 10:09 Southparkfan: wiped everything in /tmp on misc1

2016-07-12

 * 22:55 Southparkfan: import finished & ran rebuildrecentchanges.php for wisdomsandboxwiki
 * 22:28 Southparkfan: running southparkfan@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki wisdomsandboxwiki /tmp/wisdomwiki.xml on mw1

2016-07-11

 * 20:28 Southparkfan: restarted jobrunner on mw1, it was stuck
 * 18:30 Southparkfan: ran ufw-fix on cp1
 * 05:26 John: remounted static on mw2
 * 05:26 John: started jobrunner on mw1
 * 05:24 John: remounted static on mw1
 * 05:21 John: rebooting mw1
 * 03:40 PuppyKun: 11:03:58 PM  RECOVERY - Host Cache Proxy 1 is UP: PING OK - Packet loss = 0%, RTA = 0.47 ms

2016-07-10

 * 22:55 John: disabled uploads globally
 * 22:24 John: rebooting cp1, NFS will have to manage after

2016-07-09

 * 22:54 John: ran manageInactiveWikis.php with --warn
 * 20:36 John: rebooting mw2, unresponsive
 * 14:13 PuppyKun: wisdomwikiwi> update page set page_namespace=1656 where page_namespace=1632;
 * 14:12 PuppyKun: wisdomwikiwi> update page set page_namespace=1654 where page_namespace=1630;

2016-07-08

 * 18:59 John: reloaded nginx on cp[12]
 * 13:42 PuppyKun: sudo php rebuildLocalisationCache.php --wiki extloadwiki on mw*

2016-07-07

 * 17:20 Southparkfan: restarted jobrunner on mw1
 * 16:49 John: >UPDATE cw_wikis SET wiki_private = 1 WHERE wiki_dbname = "offensivesivewiki";
 * 16:00 PuppyKun: > update cw_wikis set wiki_language="en" where wiki_dbname="exerionwiki";

2016-07-06

 * 15:53 PuppyKun: puppet run on mw* after https://git.io/vKLXL
 * 12:28 PuppyKun: update cw_wikis set wiki_closed=0 where wiki_dbname="atplwiki";

2016-07-05

 * 14:29 John: ran update.php on new M[Ss]Calendar wikis

2016-07-04

 * 17:03 Southparkfan: all mwservers repooled, puppet re-enabled everywhere
 * 16:09 Southparkfan: disabled puppet on mwservers

2016-07-03

 * 22:02 labster: [allthetropeswiki]> drop table time_zone; drop table time_zone_leap_second; drop table time_zone_name; drop table time_zone_transition; drop table time_zone_transition_type;
 * 20:55 John: MariaDB [allthetropeswiki]> DROP TABLE event,general_log,procs_priv,proxies_priv,servers,tables_priv;
 * 20:48 John: MariaDB [allthetropeswiki]> DROP TABLE proc;
 * 14:37 John: update phabricator to stable
 * 08:56 John: rm /bacula/backups/STATIC-0004 /bacula/backups/STATIC-0008 on bacula1
 * 08:34 John: MariaDB [allthetropeswiki]> DROP TABLE host;
 * 08:34 John: MariaDB [allthetropeswiki]> DROP TABLE columns_priv, func, db;

2016-07-01

 * 19:43 John: delete wikis: jubeatwikithwiki, chavezwiki, putinwiki, ndtestwiki ,lupawiki
 * 18:11 PuppyKun: ran cleanupTitles.php --wiki testwiki to fix issues with TestWiki_talk:Request_permissions
 * 17:45 John: reopened madgendersciencewiki
 * 13:51 PuppyKun: finished manageInactive wikis, 104 wikis warned, 6 closed
 * 13:44 PuppyKun: forced puppet run on mw1 after "mediawiki/REL1_26 b86bf96", running manageInactiveWikis.php
 * 13:30 PuppyKun: sudo rm findInactiveWikis.php on mw1
 * 11:44 John: made seldirwiki public

2016-06-30

 * 12:31 Southparkfan: reopened wikiconstitucio.miraheze.org

2016-06-29

 * 15:03 PuppyKun: deleted lupawiki per founder request

2016-06-28

 * 18:37 Southparkfan: reopened ackbwiki
 * 06:34 Southparkfan: still not responding to reboot, opened ticket with VPS provider
 * 06:28 Southparkfan: bacula1 crashed, trying to reboot, server / control panel not responding to actions

2016-06-27

 * 13:22 Southparkfan: re-enabled puppet on misc2
 * 13:03 Southparkfan: disabled puppet on misc2

2016-06-26

 * 09:43 Southparkfan: issued varnishadm 'ban req.http.Host == wisdomwiki.org' on all cache proxies

2016-06-23

 * 12:10 Southparkfan: depooled mw1 @ cp2
 * 11:21 Southparkfan: taking down cp1 for maintenance

2016-06-21

 * 14:13 Southparkfan: MariaDB [metawiki]> update cw_wikis set wiki_closed=0 where wiki_dbname='ggdrwiki'; per request

2016-06-19

 * 03:49 PuppyKun: > delete from cw_wikis where wiki_dbname='ndtestwiki';

2016-06-18

 * 23:37 PuppyKun: > update cw_wikis set wiki_private=1 where wiki_dbname='biblicalwikiwiki';
 * 19:09 PuppyKun: sudo php runJobs.php --wiki metawiki
 * 19:09 PuppyKun: sudo php runJobs.php --wiki loginwiki
 * 18:59 PuppyKun: > update cw_wikis set wiki_closed=0 where wiki_dbname='biblicalwikiwiki';

2016-06-16

 * 18:24 Southparkfan: > update cw_wikis set wiki_language = 'fr' where wiki_dbname = 'evawiki';
 * 17:56 John: reopened biblicalwikiwiki

2016-06-15

 * 20:56 Southparkfan: renewed *.miraheze.org certificate. Apache and nginx on all cache proxies, appservers and misc. servers have been reloaded

2016-06-14

 * 02:33 PuppyKun: varnish purge https://downhillderelicts.miraheze.org on cp*
 * 02:32 PuppyKun: varnish> purge wiki.downhillderelicts.com on cp*

2016-06-13

 * 19:33 PuppyKun: varnish> ban req.http.Host == metatrek.miraheze.org on cp*
 * 18:53 John: php /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/manageInactiveWikis.php --wiki metawiki --warn
 * 18:45 Southparkfan: varnish> ban req.http.Host == meta.trek.tk to fix custom domain

2016-06-11

 * 07:57 Southparkfan: ran FlowUpdateWorkflowPageId.php for All The Tropes (https://phabricator.wikimedia.org/T137594)

2016-06-10

 * 23:06 PuppyKun: merge mw-config #523 and run puppet on mw*
 * 22:43 PuppyKun: ensured git clone /srv/mediawiki/config on mw*
 * 22:14 PuppyKun: someone started log bot

2016-06-05

 * 18:52 Southparkfan: adminbot now runs as irc user
 * 16:23 John: removed HHVM references/files from private git

2016-06-03

 * 23:01 SPF|Cloud: test if Python ignores errors
 * 22:57 PuppyKun: Thank User:Southparkfan for working log not
 * 22:49 labster: channel spam
 * 22:49 SPF|Cloud: and another one
 * 22:49 SPF|Cloud: another test
 * 22:48 SPF|Cloud: test
 * 21:10 Southparkfan: MariaDB [allthetropeswiki]> update page set page_content_model = 'flow-board' where page_namespace = 3 and page_title not like '%/%' and page_content_model = 'wikitext'; to fix https://phabricator.miraheze.org/T340

2016-05-23

 * 17:27 JohnLewis: undeployed and removed HHVM from mw2 completely. It is no more.
 * 17:23 JohnLewis: undeployed and removed HHVM from mw1 completely.
 * 01:07 revi:, 502 everywhere.

2016-04-22

 * 22:24 Southparkfan: MariaDB [allthetropeswiki]> truncate ipblocks; to fix https://phabricator.miraheze.org/T229

2016-04-20

 * 17:12: NDKilla: > update cw_wikis set wiki_private=1 where wiki_dbname='evawiki'; per founder request

2016-04-12

 * 16:23: NDKilla closed lupawiki (cw_wikis wiki_closed=1) per founder request

2016-03-27

 * 19:48 Southparkfan: resetted password for user 'O'. Sent email to user with new password.
 * ~19:45 Southparkfan: ran deleteEmptyAccounts.php --wiki metawiki --fix --migrate --safe-migrate and checkLocalUser.php --delete on some wikis
 * 17:31 JohnLewis: deleted several wikis. See dormancy page history for a list. +internalwiki

2016-03-26

 * 20:13 JohnLewis: deleted bzduropediawiki

2016-03-25

 * 13:47 Southparkfan: php deleteEmptyAccounts.php --wiki metawiki --fix --safe-migrate --migrate

2016-03-15

 * 5:30 NDKilla: UPDATE cw_wikis SET wiki_closed=1 WHERE wiki_dbname="internalwiki"; (via mw1 maintenance/sql.php)

2016-03-12

 * 15:44 Southparkfan: /srv/mediawiki/w/extensions/CentralAuth/maintenance# php deleteEmptyAccounts.php --wiki metawiki --fix --safe-migrate to delete global acocunts without any attached users
 * 15:16 Southparkfan: foreachwikiindblist /srv/mediawiki/dblist/all.dblist maintenance/checkLocalUser.php --verbose 1 --delete > ~/tmp-ca.log at mw1 to fix annoying "Could not find user data for" MWExceptions

2016-03-11

 * 19:01 JohnLewis: UPDATE cw_wikis SET wiki_private = "0" WHERE wiki_dbname = "aevumkathwiki";

2016-03-08

 * 22:12 JohnLewis: deleted spiraltestwiki and allthetropestestwiki
 * 19:53 JohnLewis: DROP DATABASE (for following: poserdazfreebiestestwiki, gameswiki, wikiacawiki)
 * 19:47 JohnLewis: DELETE FROM cw_wikis WHERE wiki_dbname = "wikiacawiki";
 * 05:25 revi:, icinga crying for varnish backend problems

2016-03-07

 * 22:41 JohnLewis: root@ns3 relays: The system is going down for power-off at Mon 2016-03-07 22:41:14 UTC!
 * 22:40 JohnLewis: root@ns3:~# shutdown -P

2016-02-26

 * 17:56 JohnLewis: UPDATE cw_wikis SET wiki_private="1",wiki_sitename="LBS Global Energy Summit" WHERE wiki_dbname="lbsgeswiki";

2016-02-24

 * 19:19 JohnLewis: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/patchSql.php /srv/mediawiki/w/extensions/BetaFeatures/sql/create_counts.sql

2016-02-22

 * 22:16 JohnLewis: UPDATE cw_wikis SET wiki_language = "es" WHERE wiki_dbname = "ricwiki";
 * 20:45 JohnLewis: reboot servers for security updates relating to core system files.

2016-02-16

 * 16:28 Southparkfan: deleted thoughtonomywikiwiki dumps

2016-02-03

 * 15:50 JohnLewis: UPDATE cw_wikis SET wiki_closed=0,wiki_private=1 WHERE wiki_dbname='quantixwiki';

2016-01-20

 * 16:29 SPF|Cloud: ran checkLocalNames.php and checkLocalUser.php on spiralwiki with --delete option. (https://github.com/miraheze/mw-config/issues/264)

2016-01-18

 * 17:10 SPF|Cloud: deleted gameswiki database
 * 17:05 SPF|Cloud: deleted gameswiki database
 * 16:46 SPF|Cloud: deleted emptysetwiki database per NDKilla's request
 * 16:35 PuppyKun: restarted HHVM (process was "active (exited)" with fatal errors)

2016-01-16

 * 02:06 PuppyKun: ran SQL query "UPDATE cw_wikis SET wiki_closed=1 WHERE wiki_dbname='quantixwiki' OR wiki_dbname='catboxwiki' or wiki_dbname='emptysetwiki';" on mw1
 * 02:05 PuppyKun: Manually restarted HHVM on mw1 ~10 mins ago

2016-01-10

 * 17:17 PuppyKun: manually run puppet on mw1 and run namespaceDupes.php --wiki adnovumwiki --fix

2016-01-09

 * 01:46 PuppyKun: manually ran update.php on adnovumwiki. failed the first time due to global blocking table already existing (this usually doesn't cause it to fail?) running it again finished successfully in 0.7s

2016-01-07

 * 15:09 SPF|Cloud: ran removeDeletedWikis.php to delete poserdazfreebiestestwiki + removed it from cw_wikis

Information
All times are in UTC+0/GMT.

Archives: 2015.