Tech:Server admin log/2022

2022-01-04

 * 06:45 MirahezeLSBot: [universalomega@mw11] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 1208s
 * 06:37 MirahezeLSBot: [universalomega@test3] finished deploy of {'world': True, 'l10n': True} to skip - SUCCESS in 727s
 * 06:25 MirahezeLSBot: [universalomega@mw11] starting deploy of {'world': True, 'l10n': True} to all
 * 06:24 MirahezeLSBot: [universalomega@test3] starting deploy of {'world': True, 'l10n': True} to skip
 * 02:54 MirahezeLSBot: [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s
 * 02:54 MirahezeLSBot: [@test3] starting deploy of {'config': True} to skip
 * 02:51 MirahezeLSBot_: [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost
 * 02:51 MirahezeLSBot_: [@mwtask111] starting deploy of {'config': True} to scsvg
 * 02:47 MirahezeLSBot_: [@test101] DEPLOY ABORTED: Canary check failed for localhost
 * 02:47 MirahezeLSBot_: [@test101] starting deploy of {'config': True} to skip
 * 02:41 MirahezeLSBot: [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 11s
 * 02:41 MirahezeLSBot: [@mw11] starting deploy of {'config': True} to ovlon
 * 02:21 MirahezeLSBot_: [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost
 * 02:21 MirahezeLSBot_: [@mwtask111] starting deploy of {'config': True} to scsvg
 * 02:18 MirahezeLSBot_: [@test101] DEPLOY ABORTED: Canary check failed for localhost
 * 02:18 MirahezeLSBot_: [@test101] starting deploy of {'config': True} to skip
 * 02:11 MirahezeLSBot: [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 11s
 * 02:11 MirahezeLSBot: [@mw11] starting deploy of {'config': True} to ovlon
 * 01:55 MirahezeLSBot: [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s
 * 01:54 MirahezeLSBot: [@test3] starting deploy of {'config': True} to skip
 * 01:51 MirahezeLSBot_: [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost
 * 01:51 MirahezeLSBot_: [@mwtask111] starting deploy of {'config': True} to scsvg
 * 01:47 MirahezeLSBot_: [@test101] DEPLOY ABORTED: Canary check failed for localhost
 * 01:47 MirahezeLSBot_: [@test101] starting deploy of {'config': True} to skip
 * 01:42 MirahezeLSBot: [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 17s
 * 01:41 MirahezeLSBot: [@mw11] starting deploy of {'config': True} to ovlon

2022-01-03

 * 23:36 MirahezeLSBot: [@mw11] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 2156s
 * 23:33 MirahezeLSBot: [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 2004s
 * 23:00 MirahezeLSBot_: [@test101] DEPLOY ABORTED: Canary check failed for localhost
 * 23:00 MirahezeLSBot: [@mw11] starting deploy of {'l10nupdate': True} to ovlon
 * 23:00 MirahezeLSBot_: [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost
 * 23:00 MirahezeLSBot: [@test3] starting deploy of {'l10nupdate': True} to skip
 * 23:00 MirahezeLSBot_: [@test101] starting deploy of {'l10nupdate': True} to skip
 * 23:00 MirahezeLSBot_: [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg
 * 12:26 JohnLewis: redoing db backups with schemas, disabled automatic backups and increased binlogs so hopefully reduce long term db load for now
 * 12:21 MirahezeLSBot_: [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost
 * 12:17 MirahezeLSBot_: [@test101] DEPLOY ABORTED: Canary check failed for localhost
 * 10:45 RhinosF1: deploy world+config to SCSVG with --force to fix
 * 10:36 RhinosF1: updated MW grafana-new to fix php alerts
 * 10:27 RhinosF1: downtime mwtask111 and test101 puppet alerts as they are flappy and annoying
 * 10:24 MirahezeLSBot: [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s
 * 10:24 MirahezeLSBot: [@test3] starting deploy of {'config': True} to skip
 * 10:22 MirahezeLSBot_: [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost
 * 10:21 MirahezeLSBot_: [@mwtask111] starting deploy of {'config': True} to scsvg
 * 10:17 MirahezeLSBot_: [@test101] DEPLOY ABORTED: Canary check failed for localhost
 * 10:17 MirahezeLSBot_: [@test101] starting deploy of {'config': True} to skip
 * 10:11 MirahezeLSBot: [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 6s
 * 10:11 MirahezeLSBot: [@mw11] starting deploy of {'config': True} to ovlon
 * 09:47 MirahezeLSBot_: [@test101] DEPLOY ABORTED: Canary check failed for localhost
 * 09:47 MirahezeLSBot_: [@test101] starting deploy of {'config': True} to skip
 * 09:42 MirahezeLSBot: [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 22s
 * 09:41 MirahezeLSBot: [@mw11] starting deploy of {'config': True} to ovlon
 * 09:24 MirahezeLSBot: [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s
 * 09:24 MirahezeLSBot: [@test3] starting deploy of {'config': True} to skip
 * 09:21 MirahezeLSBot_: [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost
 * 09:21 MirahezeLSBot_: [@mwtask111] starting deploy of {'config': True} to scsvg
 * 07:56 MirahezeLSBot: [@test3] finished deploy of {'config': True} to skip - SUCCESS in 1s
 * 07:55 MirahezeLSBot: [@test3] starting deploy of {'config': True} to skip
 * 07:54 MirahezeLSBot_: [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost
 * 07:52 MirahezeLSBot_: [@mwtask111] starting deploy of {'config': True} to scsvg
 * 07:47 MirahezeLSBot_: [@test101] DEPLOY ABORTED: Canary check failed for localhost
 * 07:47 MirahezeLSBot_: [@test101] starting deploy of {'config': True} to skip
 * 07:42 MirahezeLSBot: [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 17s
 * 07:42 MirahezeLSBot: [@mw11] starting deploy of {'config': True} to ovlon
 * 03:46 MirahezeLSBot: [universalomega@mw11] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 605s
 * 03:36 MirahezeLSBot: [universalomega@mw11] starting deploy of {'world': True, 'l10n': True} to all
 * 03:35 MirahezeLSBot: [universalomega@mw11] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 1602s
 * 03:17 MirahezeLSBot: [universalomega@test3] finished deploy of {'world': True, 'l10n': True} to skip - SUCCESS in 256s
 * 03:12 MirahezeLSBot: [universalomega@test3] starting deploy of {'world': True, 'l10n': True} to skip
 * 03:08 MirahezeLSBot: [universalomega@mw11] starting deploy of {'world': True, 'l10n': True} to all
 * 03:07 MirahezeLSBot: [universalomega@mw11] starting deploy of {'world': True, 'l10n': True} to all
 * 03:07 MirahezeLSBot: [universalomega@test3] finished deploy of {'world': True, 'l10n': True} to skip - SUCCESS in 801s
 * 02:53 MirahezeLSBot: [universalomega@test3] starting deploy of {'world': True, 'l10n': True} to skip
 * 02:20 MirahezeLSBot: [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php wikiseo (END - exit=0)
 * 02:16 MirahezeLSBot: [universalomega@mw11] finished deploy of {'world': True} to all - SUCCESS in 605s
 * 02:09 MirahezeLSBot: [universalomega@test3] finished deploy of {'world': True} to skip - SUCCESS in 310s
 * 02:07 MirahezeLSBot: [universalomega@mw11] starting deploy of {'world': True} to all
 * 02:03 MirahezeLSBot: [universalomega@test3] starting deploy of {'world': True} to skip
 * 01:57 MirahezeLSBot: [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php wikiseo (START)
 * 01:46 MirahezeLSBot: [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php purge (END - exit=0)
 * 01:19 MirahezeLSBot: [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php purge (START)

2022-01-02

 * 23:53 paladox: set io threads to 32 on gluster101
 * 23:51 paladox: set io threads to 32 on gluster3-4
 * 23:36 MirahezeLSBot: [@mw11] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 2194s
 * 23:36 MirahezeLSBot: [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 2166s
 * 23:29 MirahezeLSBot: [@test3] finished deploy of {'config': True} to skip - SUCCESS in 2s
 * 23:29 MirahezeLSBot: [@test3] starting deploy of {'config': True} to skip
 * 23:22 MirahezeLSBot_: [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost
 * 23:22 MirahezeLSBot_: [@mwtask111] starting deploy of {'config': True} to scsvg
 * 23:18 MirahezeLSBot_: [@test101] DEPLOY ABORTED: Canary check failed for localhost
 * 23:17 MirahezeLSBot_: [@test101] starting deploy of {'config': True} to skip
 * 23:12 MirahezeLSBot: [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 24s
 * 23:12 MirahezeLSBot: [@mw11] starting deploy of {'config': True} to ovlon
 * 23:04 MirahezeLSBot_: [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost
 * 23:03 MirahezeLSBot_: [@test101] DEPLOY ABORTED: Canary check failed for localhost
 * 23:02 MirahezeLSBot: [@test3] starting deploy of {'l10nupdate': True} to skip
 * 23:02 MirahezeLSBot_: [@test101] starting deploy of {'l10nupdate': True} to skip
 * 23:01 MirahezeLSBot_: [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg
 * 23:00 MirahezeLSBot: [@mw11] starting deploy of {'l10nupdate': True} to ovlon
 * 22:31 MirahezeLSBot: [universalomega@mw11] finished deploy of {'world': True} to all - SUCCESS in 570s
 * 22:22 MirahezeLSBot: [universalomega@mw11] starting deploy of {'world': True} to all
 * 22:21 MirahezeLSBot: [universalomega@test3] finished deploy of {'world': True} to skip - SUCCESS in 185s
 * 22:18 MirahezeLSBot: [universalomega@test3] starting deploy of {'world': True} to skip
 * 22:10 MirahezeLSBot: [universalomega@test3] finished deploy of {'pull': 'config', 'config': True} to skip - SUCCESS in 0s
 * 22:09 MirahezeLSBot: [universalomega@test3] starting deploy of {'pull': 'config', 'config': True} to skip
 * 22:09 MirahezeLSBot: [universalomega@mw11] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 40s
 * 22:08 MirahezeLSBot: [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True} to all
 * 22:07 MirahezeLSBot_: [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True, 'force': True} to scsvg - SUCCESS in 66s
 * 22:06 MirahezeLSBot_: [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True, 'force': True} to scsvg
 * 22:05 MirahezeLSBot_: [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'force': True} to skip - SUCCESS in 0s
 * 22:05 MirahezeLSBot_: [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'force': True} to skip
 * 22:02 MirahezeLSBot: [universalomega@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s
 * 22:02 MirahezeLSBot: [universalomega@test3] starting deploy of {'config': True} to skip
 * 21:47 MirahezeLSBot: [universalomega@test3] starting deploy of {'world': True} to skip
 * 21:35 MirahezeLSBot: [universalomega@test3] finished deploy of {'folders': 'w/extensions/CreateWiki'} to skip - SUCCESS in 1s
 * 21:34 MirahezeLSBot: [universalomega@test3] starting deploy of {'folders': 'w/extensions/CreateWiki'} to skip
 * 21:27 MirahezeLSBot: [universalomega@test3] finished deploy of {'folders': 'w/extensions/CreateWiki'} to skip - SUCCESS in 0s
 * 21:26 MirahezeLSBot: [universalomega@test3] starting deploy of {'folders': 'w/extensions/CreateWiki'} to skip
 * 20:28 MirahezeLSBot: [universalomega@test3] finished deploy of {'folders': 'w/extensions/CreateWiki'} to skip - SUCCESS in 0s
 * 20:27 MirahezeLSBot: [universalomega@test3] starting deploy of {'folders': 'w/extensions/CreateWiki'} to skip
 * 19:54 MirahezeLSBot: [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True} to all
 * 19:12 MirahezeLSBot: [paladox@mwtask1] test
 * 19:11 MirahezeLSBot: [paladox@mon2] test
 * 19:11 MirahezeLSBot: [paladox@mon2] test
 * 19:00 MirahezeLSBot: [paladox@mon2] test
 * 18:52 MirahezeLSBot_: [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost
 * 18:52 MirahezeLSBot: [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True} to all
 * 18:52 MirahezeLSBot_: [@mwtask111] starting deploy of {'config': True} to scsvg
 * 18:49 MirahezeLSBot_: [@test101] DEPLOY ABORTED: Canary check failed for localhost
 * 18:48 MirahezeLSBot_: [@test101] starting deploy of {'config': True} to skip
 * 18:41 MirahezeLSBot: [@mw11] starting deploy of {'config': True} to ovlon
 * 18:40 MirahezeLSBot: [universalomega@test3] finished deploy of {'pull': 'config', 'config': True} to skip - SUCCESS in 1s
 * 18:40 MirahezeLSBot: [universalomega@test3] starting deploy of {'pull': 'config', 'config': True} to skip
 * 18:39 MirahezeLSBot: [universalomega@test3] finished deploy of {'folders': 'w/extensions/CreateWiki'} to skip - SUCCESS in 0s
 * 18:39 MirahezeLSBot: [universalomega@test3] starting deploy of {'folders': 'w/extensions/CreateWiki'} to skip
 * 18:37 MirahezeLSBot: [universalomega@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s
 * 18:36 MirahezeLSBot: [universalomega@test3] starting deploy of {'config': True} to skip
 * 18:26 MirahezeLSBot: [@test3] finished deploy of {'config': True} to skip - SUCCESS in 1s
 * 18:25 MirahezeLSBot: [@test3] starting deploy of {'config': True} to skip
 * 17:24 MirahezeLSBot: [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s
 * 17:24 MirahezeLSBot: [@test3] starting deploy of {'config': True} to skip
 * 17:22 MirahezeLSBot_: [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost
 * 17:22 MirahezeLSBot_: [@mwtask111] starting deploy of {'config': True} to scsvg
 * 17:18 MirahezeLSBot_: [@test101] DEPLOY ABORTED: Canary check failed for localhost
 * 17:18 MirahezeLSBot_: [@test101] starting deploy of {'config': True} to skip
 * 17:11 MirahezeLSBot: [@mw11] starting deploy of {'config': True} to ovlon
 * 04:25 MirahezeLSBot: [@test3] finished deploy of {'config': True} to skip - SUCCESS in 1s
 * 04:25 MirahezeLSBot: [@test3] starting deploy of {'config': True} to skip
 * 03:54 MirahezeLSBot: [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s
 * 03:54 MirahezeLSBot: [@test3] starting deploy of {'config': True} to skip
 * 02:54 MirahezeLSBot: [@test3] finished deploy of {'config': True} to skip - SUCCESS in 1s
 * 02:54 MirahezeLSBot: [@test3] starting deploy of {'config': True} to skip
 * 02:37 MirahezeLSBot: [universalomega@mw11] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s
 * 02:37 MirahezeLSBot: [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True} to all
 * 02:12 MirahezeLSBot: [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 33s
 * 02:12 MirahezeLSBot: [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 33s
 * 02:12 MirahezeLSBot: [@mw11] starting deploy of {'config': True} to ovlon
 * 01:55 MirahezeLSBot: [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s
 * 01:54 MirahezeLSBot: [@test3] starting deploy of {'config': True} to skip
 * 00:42 MirahezeLSBot: [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 22s
 * 00:41 MirahezeLSBot: [@mw11] starting deploy of {'config': True} to ovlon
 * 00:27 MirahezeLSBot: [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s
 * 00:26 MirahezeLSBot: [@test3] starting deploy of {'config': True} to skip
 * 00:14 MirahezeLSBot: [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 82s

2022-01-01

 * 23:34 MirahezeLSBot: [@mw11] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 2035s
 * 23:33 MirahezeLSBot: [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 2010s
 * 23:01 MirahezeLSBot: [@mw11] starting deploy of {'l10nupdate': True} to ovlon
 * 23:00 MirahezeLSBot: [@test3] starting deploy of {'l10nupdate': True} to skip
 * 18:42 MirahezeLSBot: [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 13s
 * 18:41 MirahezeLSBot: [@mw11] starting deploy of {'config': True} to ovlon
 * 18:37 RhinosF1: starting full deploy-mediawiki to all SCSVG servers
 * 18:24 MirahezeLSBot: [@test3] finished deploy of {'config': True} to skip - SUCCESS in 1s
 * 18:24 MirahezeLSBot: [@test3] starting deploy of {'config': True} to skip
 * 18:12 MirahezeLSBot: [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 41s
 * 18:12 MirahezeLSBot: [@mw11] starting deploy of {'config': True} to ovlon
 * 15:38 paladox: restart php7.3/7.4 on mw*
 * 14:44 MirahezeLSBot: [paladox@test3]
 * 14:44 MirahezeLSBot: [paladox@test3]
 * 10:54 RhinosF1: forcefully change password for BrowserStackAccount via CLI to one from K12
 * 08:57 RhinosF1: downtime sslhost on icinga-new for 7 days
 * 03:12 Universal Omega: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=sunrinwiki