Tech:Server admin log

2019-07-17

• 14:27 Reception123: deleted and dropped reviwbwiki
• 11:57 Reception123: purge binary logs before '2019-07-17 10:00:00';

2019-07-16

• 19:32 paladox: downgrade restbase back to ab6b368
• 19:25 Reception123: purge binary logs before '2019-07-16 16:00:00';
• 17:10 paladox: upgrade from ab6b368 to bdc4029 (restbase on misc3)
• 17:10 paladox: root@misc3:/srv/restbase# sudo -u restbase git pull
• 11:21 paladox: MariaDB [piwik]> OPTIMIZE table piwik_archive_blob_2019_06;
• 11:07 paladox: root@misc2:/var/spool/cron/crontabs# /usr/local/bin/fileLockScript.sh /tmp/matomo_file_locks "/usr/bin/nice -19 /usr/bin/php /srv/matomo/console core:purge-old-archive-data all"
• 09:09 Reception123: reception@misc2:/srv/matomo$ sudo /usr/bin/php /srv/matomo/console core:archive --url=https://matmo.miraheze.org/matomo/ > /home/reception/matomo-archive.log
• 05:11 Reception123: purge binary logs before '2019-07-16 03:00:00';

2019-07-15

• 17:42 Reception123: purge binary logs before '2019-07-15 16:00:00';
• 05:25 Reception123: purge binary logs before '2019-07-15 03:00:00';

2019-07-14

• 18:52 John: UPDATE cw_requests SET cw_status = 'approved' WHERE cw_id = 8681;
• 09:00 Reception123: purge binary logs before '2019-07-14 09:00:00';

2019-07-13

• 19:41 Reception123: reception@mw1:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/deleteWikis.php --wiki loginwiki Reception123 --delete
• 16:11 paladox: root@db4:/bacula/restore# rm -rf *
• 15:57 paladox: [16:41:38] <+paladox> !log restarted lizardfs-master on misc3 (appears to have stopped it's self)
• 12:09 Reception123: renamed medlitreviewswiki to theliteratureprojectwiki.
• 06:06 Reception123: purge binary logs before '2019-07-13 06:00:00';

2019-07-12

• 23:32 paladox: reboot elasticsearch1
• 21:46 paladox: resetting elasticsearch
• 20:34 paladox: deleted jobrunner jobs for absolutetruthwiki (including wiki specific jobs)
• 19:34 paladox: root@elasticsearch1:/home/paladox# curl -X POST -H 'Content-Type: application/json' localhost:9200/mw_cirrus_metastore_first/settings -d '{"settings": {"index.priority": 10}}'
• 18:03 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wackypediawiki --report=1 /home/reception/illogicopedia_pages_full.xml on test1 (due to Redis hack)
• 18:03 Reception123: hack Redis.php on test1 for import
• 09:53 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wackypediawiki --report=1 /home/reception/illogicopedia_pages_full.xml
• 09:52 Reception123: reception@mw2:/srv/mediawiki/w/maintenance$ sudo -u www-data php importImages.php --wiki wackypediawiki --search-recursively /home/reception/wackypediaimages
• 09:33 Reception123: purge binary logs before '2019-07-12 09:00:00';

2019-07-11

• 22:58 paladox: running updateOneSearchIndexConfig.php for all wikis
• 22:45 paladox: reverting php-fpm childs on mw*
• 21:56 paladox: repool mw3
• 21:50 paladox: depool mw3, and reboot
• 21:40 paladox: kill -9 749 (mw3)
• 21:40 paladox: kill -9 7626 (mw3)
• 21:39 paladox: kill -9 7625 (mw3)
• 21:07 paladox: kill $(ps -A -ostat,ppid | awk '/[zZ]/ && !a[$2]++ {print $2}') on mw3
• 20:59 paladox: depool mw3 && hack php-fpm childs && restart php-fpm && repool mw3
• 20:57 paladox: depool mw1 && hack php-fpm childs && restart php-fpm && repool mw1
• 19:42 paladox: repool mw2
• 19:38 paladox: depool mw2
• 01:13 paladox: repool mw2
• 01:04 paladox: depool mw2
• 00:59 paladox: root@mw1:/home/reception# rm -rf wikibackups
• 00:28 paladox: repool mw2
• 00:27 paladox: root@mw1:/home/reception# tar -zcvf wikibackups.tar.gz wikibackups
• 00:22 paladox: depool mw2 and reboot

2019-07-10

• 23:59 paladox: that's for mw1
• 23:59 paladox: rm undefined.log*
• 23:53 paladox: cancle updateSearchIndexConfig.php (es OOM)
• 23:35 paladox: repool mw2
• 23:32 paladox: restart php-fpm && nginx on mw2
• 23:30 paladox: depool mw2
• 23:04 paladox: running updateSearchIndexConfig.php for all wikis

2019-07-09

• 17:19 paladox: [18:15:08] <+paladox> !log rm /srv/mariadb/mysql-bin.001240 && sudo service mysql restart
• 00:59 paladox: /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/extensions/CirrusSearch/maintenance/eswikis /srv/mediawiki/w/extensions/CirrusSearch/maintenance/forceSearchIndex.php
• 00:10 paladox: /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/extensions/CirrusSearch/maintenance/eswikis /srv/mediawiki/w/extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --reindexAndRemoveOk --indexIdentifier now
• 00:08 paladox: sudo -u www-data php metastore.php --upgrade --wiki=metawiki
• 00:03 paladox: upgrading elasticsearch to 6 on elasticsearch1

2019-07-08

• 20:44 paladox: deleted Lifelonglearner per https://meta.miraheze.org/w/index.php?title=Meta:Administrators%27_noticeboard&diff=prev&oldid=77234
• 08:03 Reception123: purge binary logs before '2019-07-08 05:00:00';

2019-07-07

• 21:55 paladox: depool mw1 && revert nginx / php-fpm change && repool
• 20:42 paladox: changing php-fpm to use TCP rather then socket on mw1 (includes depooling and repooling mw1)
• 20:40 paladox: reverted nginx config
• 20:33 paladox: depool mw1 && remove "use epoll" from nginx (to see how it does) && repool mw1
• 20:15 paladox: repool mw1
• 20:15 paladox: restarting nginx on mw1
• 20:12 paladox: depool mw1
• 20:05 paladox: root@mw1:/var/log/nginx# sysctl -w net.core.somaxconn=256
• 14:05 Reception123: purge binary logs before '2019-07-07 09:00:00';
• 04:02 paladox: upgrade puppet-agent on bacula1
• 03:16 paladox: doing the same for DB4 (and other backups)
• 03:14 paladox: deleting STATIC* on bacula1 (to generate a new backup)
• 02:53 paladox: depool mw1 && restart nginx (due to D state) && repool
• 02:48 paladox: repool mw2
• 02:42 paladox: depooling mw2 (and rebooting due to nginx process stuck in D)
• 02:39 paladox: repool mw1
• 02:33 paladox: reboot mw1
• 02:32 paladox: depool mw1
• 02:31 paladox: repool mw1
• 02:30 paladox: restarting php-fpm on mw1 (proccesses stuck in D)
• 02:28 paladox: depool mw1

2019-07-06

• 11:52 paladox: cleaned bin logs && restarted mysql (due to out of storage)

2019-07-05

• 03:43 paladox: that's on misc2
• 03:43 paladox: killed stray puppet-agent processes (3)
• 03:34 paladox: running matomo archive in a screen (misc2)
• 03:27 paladox: killed a bunch of duplicate matomo archive processes under the user wwww-data

2019-07-04

• 23:06 paladox: depool mw3 and restart php-fpm to pick up .so file and repool
• 23:05 paladox: depool mw1 and restart php-fpm to pick up .so file and repool
• 23:03 paladox: repool mw2
• 23:00 paladox: depooling mw2 && restarting php-fpm to pickup .so file
• 22:46 paladox: running lc on mw*
• 22:37 paladox: repool mw2
• 22:19 paladox: upgraded puppet-agent and dbus on mw2
• 22:17 paladox: depool mw2 (and restart as nginx is showing as a D state process)
• 22:04 paladox: repooling mw2
• 21:57 paladox: killing a D state process (nginx) on mw2
• 21:57 paladox: depool mw2
• 21:29 paladox: remove PII from a user
• 20:38 paladox: MariaDB [nonciclopediawiki]> truncate objectcache;
• 17:27 paladox: deleting tsbsaltmineswiki (for it to be recreated)
• 16:45 paladox: running lc on mw*
• 14:56 paladox: setting REL1_33 as default branch in mediawiki
• 12:45 paladox: root@mw1:/var/log/mediawiki# rm *.log*
• 12:13 paladox: rm undefined.log* (mw1)
• 00:48 paladox: aborted, appears update.php is doing it.
• 00:47 paladox: apply Echo/db_patches/patch-drop-notification_bundle_display_hash.sql to all wikis
• 00:34 paladox: restarted jobrunner/jobchron on mw3

2019-07-03

• 20:28 paladox: make that all mw*
• 20:27 paladox: adding nginx exporter to mw1
• 20:13 paladox: hacking puppet manifest to enable nginx exporter on test1
• 20:04 paladox: restarting php-fpm across mw*
• 19:34 paladox: installing https://github.com/martin-helmich/prometheus-nginxlog-exporter on test1
• 12:52 paladox: removing a users PII
• 01:08 paladox: running update.php for all wikis (on mw1 in a screen)
• 00:50 paladox: repool mw2
• 00:39 paladox: running lc on mw1 and mw3
• 00:35 paladox: depool mw2 to upgrade to mw 1.33
• 00:35 paladox: repool mw3
• 00:22 paladox: depool mw3 for mw upgrade to 1.33
• 00:22 paladox: repool mw1
• 00:04 paladox: depool mw1 for upgrade to mw 1.33.

2019-07-02

• 21:02 paladox: update cw_wikis set wiki_settings = REPLACE(wiki_settings, '"wgDefaultSkin":"darkvector"', '"wgDefaultSkin":"vector"');

2019-07-01

• 18:57 paladox: running ALTER TABLE mw_namespaces MODIFY COLUMN ns_core INT(1) NOT NULL DEFAULT 0;

2019-06-29

• 19:28 paladox: update mw_namespaces set ns_content_model = 'wikitext' where ns_namespace_name = 'Module_talk';
• 18:17 paladox: added "Module_talk" ns to all wikis in mw_namespace
• 17:58 paladox: upgraded phabricator
• 17:37 paladox: delete from mw_namespaces where ns_namespace_name = 'Module' on db4
• 17:37 paladox: sudo -u www-data php populateNamespacesWithDefaults.php --wiki=0h4rkwiki --overwrite on mw1
• 17:32 paladox: added "Module" ns to all wikis in mw_namespace
• 17:32 paladox: added "Module_talk" ns to 'default' dbname in mw_namespace
• 17:31 paladox: added "Module" ns to 'default' dbname in mw_namespace
• 15:02 paladox: deleting all pages that start with Thread: on rottenwebsiteswiki - T4407
• 14:08 paladox: test
• 13:53 paladox: root@test1:/srv/mediawiki/w/extensions/ManageWiki# sudo -u www-data git reset --hard origin/master
• 13:53 paladox: root@test1:/srv/mediawiki/w/extensions/MirahezeMagic# sudo -u www-data git reset --hard origin/master

2019-06-28

• 01:15 paladox: cp4 i cleaned the firewall actually not mw1.
• 01:14 paladox: repool mw1
• 01:12 paladox: depooling mw1 to clean firewall
• 01:10 paladox: delete banned ip from firewall

2019-06-27

• 23:49 paladox: starting import for cwarswiki - T4498
• 23:17 paladox: banning an ip on cp4 firewall
• 22:09 paladox: sudo apt-get install php7.2-dev liblua5.1-0-dev on mw1
• 21:18 paladox: repooling mw1 temporarily to see if max childs 6 for php-fpm improves stability
• 21:16 paladox: that ^^ is for mw1
• 21:15 paladox: testing setting php-fpm max childs to 6
• 21:15 paladox: depool mw1
• 17:50 paladox: repool mw1
• 17:40 paladox: rebooting mw1 (due to some process causing high load)
• 17:40 paladox: depool mw1
• 17:21 paladox: depool mw1 && restart nginx && repool
• 17:19 paladox: repool mw1
• 17:17 paladox: restart php7.2-fpm on mw1
• 17:17 paladox: depool mw1
• 13:54 paladox: TRUNCATE TABLE objectcache; for allthetropeswiki
• 13:53 paladox: TRUNCATE TABLE objectcache; for buswiki
• 13:51 paladox: drop searchindex table for buswiki

2019-06-26

• 19:30 paladox: delete from mw_namespaces where ns_namespace_name = 'Module_talk';
• 19:30 paladox: delete from mw_namespaces where ns_namespace_name = 'Module';
• 11:09 paladox: uprade puppet-agent on misc2

2019-06-25

• 22:36 paladox: deleted CleavagesCwars from phabricator (per email)
• 18:29 paladox: apt-get install nginx postfix postfix-sqlite puppet-agent rsync tzdata unzip on elasticsearch1
• 16:51 paladox: running forceSearchIndex.php for buswiki
• 16:50 paladox: running updateSearchIndexConfig.php for buswiki
• 12:30 paladox: [13:22:19] <+paladox> !log restart mysql on db4

2019-06-24

• 13:14 Southparkfan: install sysbench on mw3
• 13:07 Southparkfan: install sysbench on db4

2019-06-23

• 13:32 paladox: running lc on mw*

2019-06-22

• 20:50 paladox: (before the last log, i removed all site maps so that i could generate fresh ones)
• 20:44 paladox: running generateMirahezeSitemap.php on mw1
• 16:06 Reception123: purge binary logs before '2019-06-22 12:00:00';
• 15:21 paladox: repool mw2
• 15:20 paladox: depooled mw2 && restarted php-fpm
• 15:19 paladox: killed puppet process on mw2

2019-06-21

• 16:50 paladox: ran lc on mw*
• 16:45 paladox: reboot cp3
• 16:44 paladox: also upgrading kernel on cp3
• 16:43 paladox: apt-get upgrade on cp3
• 16:39 paladox: upgrade nginx to 1.16 on cp2
• 16:39 paladox: downgrade nginx to 1.16 on misc1
• 16:38 paladox: upgrade puppet-agent on cp2 and misc1
• 16:35 paladox: upgrade nginx to 1.16 on cp4
• 16:33 paladox: downgrade nginx on misc4 to 1.16
• 16:27 paladox: repool mw1
• 16:23 paladox: downgrading nginx to 1.16 on mw1
• 16:23 paladox: depool mw1
• 16:23 paladox: repool mw3
• 16:18 paladox: downgrading nginx to 1.16 on mw3
• 16:18 paladox: depool mw3
• 16:17 paladox: downgrading nginx to 1.16 on misc2
• 16:16 paladox: repool mw2
• 16:14 paladox: downgrading nginx to 1.16 on mw2
• 16:08 paladox: depool mw2
• 16:03 paladox: repool mw2
• 15:49 paladox: reboot mw2
• 15:48 paladox: depooled mw2 for reboot
• 15:37 paladox: killing a D state nginx process on mw2
• 15:29 paladox: enabling categorytree for all wikis
• 13:13 paladox: drop table searchindex for isvwiki
• 13:09 paladox: running forceSearchIndex.php for isvwiki - T4479
• 13:01 paladox: running updateSearchIndexConfig.php for isvwiki - T4479
• 00:37 paladox: repool mw3
• 00:36 paladox: upgrade firefox-esr on mw3
• 00:31 paladox: reboot mw3 (as there is a proccess in D state, but it's not shown long enough for me to kill it)
• 00:18 paladox: stopped nginx on mw3 && disabled puppet
• 00:14 paladox: depool mw3
• 00:09 paladox: update cw_wikis set wiki_settings = REPLACE(wiki_settings, '"wgRottenLinksCurlTimeout":"30"', '"wgRottenLinksCurlTimeout":"10"');

2019-06-20

• 23:45 paladox: killed some updateExternalLinks.php pids to reduce load on mw3

2019-06-19

• 18:14 paladox: upgrade puppet-agent on mw*
• 18:09 paladox: upgrade puppet-agent on cp4 && apt-get upgraon puppet1
• 16:25 paladox: apt-get upgrade on test1

2019-06-18

• 06:38 Reception123: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki loginwiki on mw*

2019-06-17

• 22:44 John: removed binary logs on db4
• 15:28 paladox: salt -E "mw.*" cmd.run "rm -rf /srv/mediawiki/w/cache/managewiki/**"
• 13:43 paladox: just to note that i re enabled blogpage on amazingyoutuberswiki through MW
• 13:42 paladox: disable blogpage on amazingyoutuberswiki (temp)
• 03:48 paladox: ran PIIRemoval.php (enforcing ToU) (again)
• 03:48 paladox: ran PIIRemoval.php (enforcing ToU)

2019-06-16

• 20:08 paladox: upgrade phabricator on misc4
• 19:51 paladox: UPDATE mw_permissions SET perm_permissions = REPLACE(perm_permissions, "read", "") WHERE perm_group = "*"; on db4 (1525 rows updated)
• 19:36 paladox: delete from mw_permissions where perm_group = '\'*\; on db4
• 19:34 paladox: delete from mw_permissions where perm_group = '"*"'; on db4
• 19:32 paladox: /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/ManageWiki/maintenance/modifyGroupPermission.php --removeperms=read \"*\" (mw1)
• 19:32 paladox: delete from mw_permissions where perm_group = 'modifyGroupPermission.php'; on db4
• 19:26 paladox: running /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/ManageWiki/maintenance/modifyGroupPermission.php --removeperms=read '*' (mw1)
• 19:24 paladox: remove read from horridreceptionwikiswiki
• 05:57 Reception123: running wikibackups (./wikibackups.sh /home/reception/allpublic.dblist /srv/mediawiki/w/maintenance/dumpBackup.php)
• 03:08 paladox: running lc on mw* and test1
• 02:52 paladox: due to out of space on db4
• 02:52 paladox: <+paladox> !log restarted mysql (at 03:43 bst)

2019-06-14

• 16:44 paladox: restart prometheus-node-exporter on misc2
• 16:42 paladox: kill puppet on misc2
• 14:08 paladox: removing gluster and installing ceph on test1
• 10:38 Reception123: root@mw1:/home/paladox# rm allthetropeswiki.xml.gz (restart dump with current revs)
• 10:37 Reception123: reception@mw1:/mnt/mediawiki-static/dumps$ sudo -u www-data php /srv/mediawiki/w/maintenance/dumpBackup.php --wiki allthetropeswiki --current --output gzip:/mnt/mediawiki-static/dumps/allthetropes14062019.xml.gz (restart dump without full rev)
• 00:20 paladox: installing gluster on test1

2019-06-13

• 22:49 paladox: root@test1:/home/reception# rm -rf wikibackups
• 22:21 paladox: running forceSearchIndex.php for pointmanwiki
• 22:20 paladox: running updateSearchIndexConfig.php for pointmanwiki
• 17:48 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki sdiywiki --r="Deletion of pages in Category:CGS Serge kit image" --u="Reception123" /home/reception/sdiy-pages-delete.txt
• 01:04 paladox: running dumpBackup.php for allthetropes (on mw1)

2019-06-11

• 23:22 paladox: ALTER TABLE mw_permissions CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
• 23:22 paladox: ALTER TABLE cw_wikis CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
• 22:54 paladox: repool mw1
• 22:49 paladox: depooling mw1
• 00:33 paladox: remove newusopediawiki-namespaces.cdb on mw*
• 00:28 paladox: running ALTER TABLE mw_namespaces CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; on db4

2019-06-10

• 22:02 paladox: upgrade dirmngr gnupg gnupg-agent gpgv libc-bin libc-l10n libc6 libc6-dbg libcups2 on db4
• 21:59 paladox: upgrade base-files on db4

2019-06-08

• 20:16 paladox: recreating amenthesprojectwiki
• 17:19 paladox: running lc on mw*
• 15:28 paladox: upgrading test1 to mw 1.33
• 13:31 paladox: update cw_wikis set wiki_locked = 0 where wiki_locked = 1 and wiki_dbname = 'metawiki';

2019-06-07

• 22:25 paladox: branching REL1_33

2019-06-06

• 23:38 paladox: repool mw3
• 23:34 paladox: also running lc on mw*
• 23:34 paladox: depool mw3 or a mw upgrade
• 23:34 paladox: repool mw2
• 23:31 paladox: depool mw2 or a mw upgrade
• 23:30 paladox: repool mw1
• 23:27 paladox: depool mw1 or a mw upgrade
• 16:47 paladox: running lc on test1, mw*
• 15:45 paladox: apt-get install dns-root-data grafana python3-cryptography rsync tzdata unzip postfix postfix-pcre postfix-sqlite publicsuffix python-cryptography libjs-jquery - misc1
• 15:44 paladox: apt-get install nginx mariadb-client mariadb-client-10.1 mariadb-client-core-10.1 mariadb-common - misc1
• 13:29 paladox: repool mw1
• 13:29 paladox: upgrade php-redis to 4.3 on mw*
• 13:24 paladox: rebooting mw1
• 13:18 paladox: depool mw1
• 00:15 paladox: installing openvpn on puppet1

2019-06-05

• 19:13 paladox: sudo apt-get install devscripts fakeroot on test1
• 16:50 paladox: reset /srv/mediawiki/config on test1 (merge conflicts)
• 01:36 paladox: installing netcat on test1

2019-06-04

• 23:08 paladox: installing strongswan on test1 following https://www.digitalocean.com/community/tutorials/how-to-set-up-an-ikev2-vpn-server-with-strongswan-on-ubuntu-16-04
• 18:37 paladox: running rebuildall.php for quircwiki on mw1

2019-06-03

• 22:19 paladox: starting more imports for quircwiki
• 21:46 paladox: hacking Redis.php on mw1 to disable redis if using cli
• 21:44 paladox: compressing quircwiki wiki

2019-06-02

• 19:59 paladox: running lc on mw*

2019-06-01

• 23:20 paladox: upgrade a few minor packages on bacula1
• 23:19 paladox: delete backups on bacula (for them to have a full backup in ~40mins)
• 16:36 Reception123: disabled OATHAuth for Eduaddad (php disableOATHAuthForUser.php "Eduaddad" --wiki metawiki)
• 15:11 paladox: mkdir /tmp on db4 at 14:46
• 05:35 Reception123: purge binary logs before '2019-05-31 12:00:00';
• 05:33 Reception123: removed /tmp on db4
• 00:23 paladox: upgraded vpncloud to 1.0 accross all hosts

2019-05-31

• 15:40 paladox: upgrade php, nginx and nodejs on misc4
• 15:38 paladox: upgrade php on misc2
• 15:06 paladox: repool mw3
• 15:00 paladox: upgrade packages on mw3
• 14:58 paladox: depool mw3
• 14:58 paladox: repool mw2
• 14:53 paladox: upgrading some packages on mw1/mw2
• 14:53 paladox: upgrade postfix postfix-sqlite python-cryptography python-pip python-pip-whl python3-cryptography on mw1
• 14:52 paladox: upgrade apt-get install postfix postfix-sqlite python-cryptography python-pip python-pip-whl python3-cryptography on mw2
• 14:51 paladox: upgrade samba-common samba-common-bin samba-libs smbclient tzdata on mw2
• 14:49 paladox: upgrade libmagickcore-6.q16-3 libmagickcore-6.q16-3-extra libmagickwand-6.q16-3 on mw2
• 14:47 paladox: upgrade nginx on mw2
• 14:46 paladox: upgrade php7.2-common php7.3-common unzip imagemagick imagemagick-6-common imagemagick-6.q16 puppet-agent on mw2
• 14:45 paladox: depool mw2
• 14:44 paladox: repool mw1
• 14:43 paladox: upgrade ffmpeg
• 14:42 paladox: upgrade unzip on mw1
• 14:39 paladox: upgrade nginx from 1.15 to 1.17 on mw1
• 14:34 paladox: upgrading php to 7.2.19 on mw1
• 14:27 paladox: depool mw1
• 06:36 Reception123: sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki loginwiki on mw*

2019-05-30

• 02:43 paladox: running lc on mw*

2019-05-29

• 22:17 paladox: upgraded phabricator - T4426
• 14:53 paladox: remove ip
• 13:26 paladox: remove email from a user per ToU

2019-05-28

• 00:53 paladox: created empty commonswikiwiki db on db4

2019-05-27

• 23:29 John: created "Anse1997_tries_again" on several wikis
• 20:11 paladox: upgrade grafana to 6.2.1
• 20:10 paladox: upgrade icinga2 to 2.10.5
• 20:10 paladox: rebooting misc1 to enable TUN
• 19:46 paladox: rename commonswikiwiki to commonswiki

2019-05-26

• 13:13 paladox: rm -rf /srv/mediawiki/w/cache/managewiki/** on mw*
• 13:12 paladox: ran update mw_permissions set perm_removegroupsfromself = '[]' where perm_removegroupsfromself = ; on db4
• 13:10 paladox: ran update mw_permissions set perm_addgroupstoself = '[]' where perm_addgroupstoself = ; on db4

2019-05-25

• 15:02 paladox: upgrade phabricator

2019-05-24

• 22:40 paladox: starting import for quircwiki
• 17:41 paladox: remove hack from /srv/mediawiki/config on mw1
• 17:41 paladox: remove hack from /srv/mediawiki/w on mw1
• 17:04 paladox: restore reports 1-11 from a backup
• 17:03 paladox: dropping incidents reports 1-11 in the incidents table (restoring from a backup shortly)
• 13:07 paladox: rebooting mw3 to replace lizardfs mount point
• 12:27 paladox: remove blogpage from besttvshowswiki - T4411
• 02:08 paladox: running rebuildall.php and initSiteStats.php for thelastsovereignwiki
• 02:07 paladox: running rebuildall.php and initSiteStats.php for trollpastawiki
• 02:05 paladox: running rebuildall.php and initSiteStats.php for animatedfeetwiki
• 02:05 paladox: running rebuildall.php and initSiteStats.php for animebathswiki
• 01:19 paladox: running lc on mw*

2019-05-23

• 23:44 paladox: starting import for americangirldollswiki - T4384
• 23:33 Southparkfan: ran userdel on mw*/puppet1 per f19fe61a9321b95bdef730e0aeff30102b4f2ca7
• 22:57 Southparkfan: deleted user account 'macfan' on mail server
• 18:18 paladox: running populateGroupPermissionsWithDefaults.php for all wikis (will only effect new wikis created since the 5th of may)
• 17:47 paladox: dropping cdb cache for all wikis
• 17:46 paladox: dropping mw_permissions to restore from backup - T4405
• 16:55 paladox: dropped wikicreator && updated proxybot to drop centralauth-lock and globalblock
• 16:45 paladox: INSERT INTO mw_permissions (perm_dbname, perm_group, perm_permissions, perm_addgroups, perm_removegroups, perm_addgroupstoself, perm_removegroupsfromself, perm_autopromote) VALUES ('metawiki', 'wikicreator', '["createwiki"]', '[]', '[]', '[]', '[]', NULL);
• 16:38 paladox: granting editsitecss, editsitejson and editsitejs to sysop for all wikis
• 16:10 paladox: granting editsitecss, editsitejson and editsitejs to sysop through ManageWikiDefaultPermissions
• 12:29 paladox: rebooting test1
• 01:46 paladox: install mariadb-server on test1 to test restoring backups.

2019-05-22

• 23:47 paladox: rm -rf /srv/mediawiki/w/cache/managewiki/** on mw*
• 18:33 MacFan4000: DELETE FROM mw_permissions WHERE perm_dbname='metawiki' AND perm_group='comentadmin';
• 18:31 MacFan4000: DELETE FROM mw_permissions WHERE perm_dbname='metawiki' AND perm_group='chatmod';
• 18:30 MacFan4000: DELETE FROM mw_permissions WHERE perm_dbname='metawiki' AND perm_group='forumadmin';

2019-05-21

• 16:09 John: fix db values for content_model
• 00:39 MacFan4000: UPDATE cw_requests SET cw_status = 'declined' WHERE cw_id = '8135';
• 00:30 paladox: running lc on mw3
• 00:29 paladox: running lc on mw2
• 00:28 paladox: running lc on mw1

2019-05-20

• 21:01 paladox: running rebuildall.php for scruffywiki

2019-05-19

• 21:35 paladox: drop popnmusicwiki database - T4380
• 13:33 paladox: running rebuildall for thelastsovereignwiki

2019-05-18

• 23:07 paladox: disabled puppet on mw1 at 23:38 bst time to prevent cert problems.
• 21:01 paladox: stopped mirahezerenewssl
• 16:12 paladox: upgrading phabricator on misc4
• 15:17 paladox: starting import for animebathswiki on mw1 - T4294
• 15:08 paladox: starting import for thelastsovereignwiki on mw1 - T4362
• 03:28 John: UPDATE cw_requests SET cw_status = "approved" WHERE cw_id = 8106;

2019-05-17

• 22:45 paladox: started import for wiki hispanowiki - T4378
• 17:40 paladox: disable blogpage on atrociousdeviantswiki, extension requires voteny to be installed first
• 17:11 paladox: upgrade puppetdb to 6.3.2 and puppet-agent to 6.4.2 on puppet1
• 16:00 paladox: enabled UniversalLanguageSelector on metawiki

2019-05-16

• 00:09 John: running conversion from LS->DB for new ManageWikiNamespacesAdditional stuff

2019-05-15

• 22:22 John: ALTER TABLE mw_namespaces ADD COLUMN ns_additional LONGTEXT AFTER ns_core;
• 21:05 MacFan4000: DELETE FROM echo_unread_wikis WHERE euw_user = '9736'; (metawiki)
• 16:20 paladox: renamed Ucronistaw to Hispano76 in phabricator

2019-05-12

• 22:33 paladox: disable blogpage on flawlessfandomuserswiki - T4361

2019-05-11

• 14:13 paladox: killing the mw cache on mw*
• 14:13 paladox: update mw_permissions set perm_addgroups = '[]' where perm_addgroups = 'null'; on db4 (13 entries updated)
• 14:12 paladox: update mw_permissions set perm_removegroups = '[]' where perm_removegroups = 'null'; on db4 (13 entries updated)
• 14:11 paladox: update mw_permissions set perm_addgroupstoself = '[]' where perm_addgroupstoself = 'null'; on db4 (94 entries updated)
• 14:11 paladox: update mw_permissions set perm_removegroupsfromself = '[]' where perm_removegroupsfromself = 'null'; on db4 (94 entries updated)
• 14:00 paladox: upgrade php-igbinary on mw1
• 13:59 paladox: upgrade certbot and python3-certbot on mw1
• 13:58 paladox: upgrade imagemagick imagemagick-6-common imagemagick-6.q16 puppet-agent on mw1
• 13:57 paladox: upgrade puppet-agent imagemagick-6.q16 imagemagick on mw3
• 13:56 paladox: repool mw3
• 13:54 paladox: depool and upgrade mw3 php7.2 to php 7.2.18
• 13:52 paladox: repool mw2
• 13:50 paladox: depool and upgrade mw2 php7.2 to php 7.2.18
• 13:48 paladox: repool mw1
• 13:40 paladox: depool mw1
• 13:30 paladox: upgrade php7.2 to 7.2.18 on mw1
• 10:19 John: purged all permissions entries
• 10:19 John: DELETE FROM mw_permissions WHERE perm_group = 'oversight';

2019-05-10

• 22:45 paladox: updating some packages on cp4
• 22:17 paladox: hacking User.php on mw1
• 21:48 Southparkfan: removed 'citoid' extension from mar9122wiki - T4356
• 00:00 paladox: running lc on mw*

2019-05-09

• 23:10 paladox: hacking Redis.php to disable job stuff if using cli (so that imports work)
• 22:36 paladox: running lc on mw*
• 21:56 John: deleted some $wiki-permissions.cdb files to trigger rehashes
• 21:54 John: UPDATE mw_permissions SET perm_autopromote = NULL WHERE perm_autopromote = 'null';
• 15:46 John: UPDATE mw_permissions SET perm_autopromote = '["&",[1,10],[2,345600]]' WHERE perm_autopromote IS NULL AND perm_group = "autoconfirmed";
• 15:45 John: UPDATE mw_permissions SET perm_removegroupsfromself = '[]' WHERE perm_removegroupsfromself = ;
• 15:45 John: UPDATE mw_permissions SET perm_addgroupstoself = '[]' WHERE perm_addgroupstoself = ;
• 13:40 paladox: starting import for trollpastawiki - T4215
• 13:39 paladox: compressed trollpastawiki

2019-05-07

• 16:18 paladox: ban bch.miraheze.org from varnish - T4350

2019-05-05

• 23:10 paladox: running lc on mw*
• 15:25 paladox: restoring wiki's file storage
• 15:09 paladox: stopping LizardFS backup so that the restore of some wiki's file storage can be done
• 00:05 paladox: repooling mw2

2019-05-04

• 23:57 paladox: depool mw2 (and downtime for 15 minutes)
• 23:56 paladox: repool mw1
• 23:49 paladox: depool mw1
• 23:47 paladox: switch on TUN on misc4 - T4346
• 23:45 paladox: switch on TUN on ns1 - T4346
• 23:44 paladox: switch on TUN on puppet1 - T4346
• 23:43 paladox: remove cargo from misc1
• 23:15 Southparkfan: delete firewall rules on cp4/test1/db4/elasticsearch1 related to vpncloud port
• 23:10 paladox: installing cargo on misc1
• 22:52 paladox: removed temp firewall rule
• 22:45 paladox: allowing 10.0.1.1 to acccess port 9200 temporarily on elasticsearch1
• 22:05 Southparkfan: vpncloud refuses to work on elasticsearch, possible container/node issue, opened ticket #158843
• 21:50 Southparkfan: graceful elasticsearch, attempting to re-enable TUN/TAP
• 21:13 Southparkfan: killed old RottenLinks processes on mw3 (ps aux | grep RottenLinks | grep -E '(Apr|May)' | awk '{print $2}' | sudo xargs kill)
• 14:43 paladox: running lc on mw*
• 12:33 Reception123: actually undelete test1wiki (forgot )
• 05:12 Reception123: running wikibackups (./wikibackups.sh /home/reception/allpublic.dblist /srv/mediawiki/w/maintenance/dumpBackup.php)
• 05:11 Reception123: UPDATE cw_wikis SET wiki_deleted = 0 WHERE wiki_dbname = "test1wiki";

2019-05-03

• 23:36 paladox: running lc on mw*
• 23:27 paladox: rm -rf /srv/mediawiki/w/cache/managewiki/** on all mw* using salt
• 23:20 paladox: running import for liveactionfeetwiki - T4323
• 20:06 paladox: restart php7.2-fpm on mw1 due to "Resource temporarily unavailable"
• 19:17 paladox: restoring files for thecscwiki from backup

2019-05-02

• 21:53 paladox: restarting lizardfs-master on misc3 after deploying https://git.io/fjZhE
• 16:57 paladox: changing phabricator.serious-business to true in phab's ui

2019-05-01

• 20:51 paladox: live hacking core/CronArchive.php on misc2
• 16:45 paladox: disable puppet on lizardfs[1-3]
• 16:17 paladox: restoring file storage for zendariwiki, toxicfandomsandhatedomswiki, digimonwiki and testwiki
• 00:43 paladox: restarting lizardfs-chunkserver on lizardfs1
• 00:43 paladox: restarting lizardfs-chunkserver on lizardfs2
• 00:41 paladox: restarting lizardfs-chunkserver on lizardfs3
• 00:35 paladox: restarting lizardfs-master on misc3
• 00:09 John: ALTER TABLE cw_wikis MODIFY COLUMN wiki_creation BINARY(14) NOT NULL, MODIFY COLUMN wiki_private TINYINT NOT NULL;
• 00:07 John: set wiki_creation for digimonwiki, heistwiki and macfan4000wiki using first recorded entry in logging tables
• 00:06 paladox: running core:archive --force-all-websites on misc2 (for matomo)
• 00:04 John: set wiki_creation for addawiki, bigforestwiki, hypopediawiki, omniversaliswiki, podpediawiki, r2wiki, unpediawiki, vukufwiki from metawiki log history

2019-04-30

• 23:46 John: noting that is to bring things more inline with what they should be. Code changes will follow eventually of course. Wanting to change wiki_creation but late introduction means 11 rows have NULL as value which shouldn't happen. Will either fix or dummy set their dates
• 23:45 John: ALTER TABLE cw_wikis MODIFY COLUMN wiki_closed TINYINT NOT NULL DEFAULT 0, MODIFY COLUMN wiki_inactive TINYINT NOT NULL DEFAULT 0;
• 22:41 paladox: deleting rpwiki ructionarywiki rumechguidewiki russopediawiki rwbyrpwikiwiki rwmdwiki rxywiki ryokutowiki sabbakawiki sadereawiki saharinspacewiki sambokitwiki sandvallangswiki sanmaronwiki satbirsoniwiki sau226wiki sbcgamingwiki sbinfinitywiki scamafriwebwiki scambaitingwiki scannotwiki schafschuetzenwiki schicksalswikiwiki schulwiki (per dormancy policy)
• 22:38 paladox: deleting schupiewiktionarywiki scientiawikiwiki scottwatsonwiki scoutswiki seagullwiki seahorsewiki searchingdiligentlywiki senlawiki senpagawiki sensonewiki serinfhospwiki serviceshiftwiki sfwarehousewiki sgmbwiki sgswiki shadowrunwiki sharingcvwiki shatteredpodcastwiki shatteredworldwiki shefwiki shirinnoeiwiki shortwikiwiki shosumosuwiki silentsunwiki sinwikiwiki sirikotwiki sitraduwiki sivzachwiki sixa503wiki skepticwikiwiki skkdoccwiki skydivingwiki skyfleetwiki sladewatkinswiki slaynetworkwiki sm64orgwiki smartphoneswiki smartwiki smashbroslawlwiki smoswiki snizhwiki snowthegamewiki (per dormancy policy)
• 22:36 paladox: deleting soaitwiki sofoziacnowiki softwarewiki sonsofamericamodwiki sonsofamericawiki sophosdeloswiki sorilliawiki soundboardwiki spiritualitywiki splatvgwiki splunkwiki spoonerwiki sprachkurs20wiki sptrwiki sql4rmwiki sqlite4rmwiki sstournamentstestwiki staffr0xwiki startcrowdwiki startungwiki stavenheimwiki stellawiki stelscapewiki stepediawiki sterlzwikiwiki stjohannisharvestehudewiki stormerswiki storycarpwiki stpeterseasternhillchoirwiki strategy2executionwiki strathacewiki streetwiki studentcenterwiki studingwiki suave101wiki sufwiki suginamisannekiwiki summerhousewiki sunchildwiki (per dormancy policy)
• 22:34 paladox: deleting sunserferswiki sunsetwiki supercrewwiki superhospitalwiki supernamuwiki survivalerawiki sweetjissousekiwiki synthpediawiki syntopiwikiwiki sysadminwiki systemdafricawiki tabithareeveswiki takeittothevetwiki takethatwikiwiki talkingtomandfriendswiki tangotextewiki tarovisionwiki taxiantwiki taylorswift13wiki team13wiki teammontmartrebpwiki techpredwiki tecnologiaseducacaowiki tekkenwiki teknowiki telegrambotswiki templeofdivinitywiki tenlosssyndicatewiki tenniselbowwiki tensorflowglossarywiki terrafractumwiki terredelsolewiki tescommunitywiki testdelete7wiki (per dormancy policy)
• 22:33 paladox: deleting testingnewsandtranslatedontenterwiki teswiki tetracommunwiki textbookwiki tfwikiswiki thax39wiki theamazingworldofgumballwiki theanimationwikiwiki theavatararenawiki theavatarchronicleswiki thebluecafewiki thedarkworldwiki thedistancewiki thedominancewiki theedishwiki thenozomiwiki therifterwiki theshookwiki thesimswiki thesimulatorwiki thewildawaitswiki theworldwithinwiki thireswiki thrivingwiki thunderwiki tik2018wiki titaniawiki togettoknowhumanswiki toljsewiki toolkitcitiengovwiki toonpediawiki totalfreedomwikiwiki totcwiki tourbasewiki tpgwiki (per dormancy policy)
• 22:31 paladox: deleting tradingmexwiki trafficresearchwiki tramswiki transparencyguidewiki travauxlibwiki travelguidewiki treelivewiki trexwiki tribulabwiki triseptsolutionswiki trnixwiki tswikiwiki tumodulewiki tunisianwiki tutorielwikiwiki twopairsofgreyeyeswiki twrailwaypediawiki txfrmrwiki tzbookwiki ucietestwiki uclalinuxwiki udcmwiki uefagameplaywiki ufprformulawiki undertaleorisitwiki unibielefeldenactuswiki unichwiki unicornswiki universepediawiki unixtlfwiki unknownchronicleswiki unknownlandswiki unmusicfestivalwiki unoitudowiki unoiwiki uoluwiki (per dormancy policy)
• 22:28 paladox: deleting upcarquitecturadesoftwarewiki urbexwiki usopediawiki usuniwikiwiki utasmotorsportswiki utopiaewikiwiki uwlexperiencewiki uwlinternationalwiki uzeruswiki vaistrawiki valkohwiki vandalwiki vandatechwiki vasardenwiki vawtwiki veerawiki venturewiki veraciawikiwiki verhalenwiki verrahwiki vertaistuotantowiki vestedgameswiki viewpediawiki virtualfrenchwiki vlierhofwiki vndbreviewwiki voiceactivatedwiki voidbornewiki vostarwiki voxpopuliwiki vtagwiki vtkwiki (per dormancy policy)
• 22:26 paladox: deleting vtwiki wanselinwiki warondestinywiki warriorsfanguidewiki waywardwiki wbsfisiwiki wbswikiwiki weakypidiawiki wearyourdictionarywiki weatherwikiwiki weefeewillwiki weichgewebstumorenwiki wellawiki wepwikiwiki whartonprojectwiki wholemindhealthwiki whufcyouthwiki whysowiki wiki wikiageingwiki wikiajbwiki wikibakawiki wikibiblewiki wikibregiswiki wikiclinicwiki wikicombowiki wikicristianowiki wikigallerywiki wikigreenwiki wikigwiki wikihamaareewiki wikihousewiki wikimaenwiki wikimoviewiki wikinationswiki (per dormancy policy)
• 22:25 paladox: deleting wikioftextwiki wikipandawiki wikipediabiaswiki wikiptminerswiki wikiswiki wikitestwiki wikitransparencywiki wikiturawiki wikiwebwiki wikiwiki willwiki willyleewiki windhambomberswiki windowswiki winnipegwikiwiki wirtschaftsinformatikhbswiki wisdomwiki wizcopycatwiki woodallscookwiki woorilindywiki workinprogresswiki worldcontestwiki wtfjhtwiki wvscwiki xavipwiki xelestialwiki xenharmonicwiki xrpwikiwiki xuniversewiki yacresourceswiki yaguwiki yamawiki yatesmillwiki yherchianwiki yikipediawiki yourbuddywiki yourkickstartersuckswiki yuuyuuwiki zerononcensewiki zitabiteswiki (per dormancy policy)
• 22:23 paladox: deleting rasamrutawiki rastr11wiki rdfwiki rebolngwiki redealternativowiki redworldwiki renecassinwiki renessenceswiki requiemwiki researchadwaywadekarwiki residentbrasilwiki resiwikiunicapwiki resiwikiwiki resonancewiki retailwiki retrofuturism1950wiki retrospectivefacilitatorgatheringwiki reversemediawiki reversewiki richardswiki richardyan314wiki rico9wiki rikimaruwiki riseofthechosenwiki risonwiki roadtrip2018wiki robertsnoteswiki roguesgallerywiki rollerderbylillewiki ronito030wiki roschfallenwiki rotaractorwikiwiki royalquestplwiki rpcauthoritywiki rpgwiki rpgzinzinswiki (per dormancy policy)
• 22:21 paladox: deleting pokesurvivorwiki polishmyfanworldwiki pomopediawiki popteamepicwiki postscriptumhungarywiki poswiki ppfwiki praxeumwiki prg2018wiki primesportwiki prinyawiki processwiki progress4parkwaywiki projectwilsonwiki promoinsidewiki propagandawiki protonwiki proxmoxhclwiki pruebamatesaraujowiki prumsamdywiki psc18wiki pscwiki pslwiki pti18wiki q3dwiki qhsewiki qissmathlabwiki qutlawnoteswiki rackdos777wiki rageofbahamutwiki rajsreenwiki raltseyewiki rambahadurbomjonwiki ramenpediawiki randomwiki (per dormancy policy)
• 22:19 paladox: deleting patternswiki paulsletterswiki pb2leveleditorwiki pb2stofwiki pcarchitectwiki pedrobritowiki pedrorenanwiki pengalewiki penjhorngwiki penwikiwiki pepisptestwiki permaguruwiki pessoalzywiki petctviewerwiki pfathaswiki pfgjaverianamswiki pfl2wiki phantomforceswiki pharmacywikiwiki phicompwiki philowiki philowikiwiki philteacherswiki phragmatawiki pinballwiki pispediawiki pitchwiki pixterwiki placesrobloxwiki pmavwiki podcastswiki poewikiwiki pokemondustwiki pokeonewiki (per dormancy policy)
• 22:17 paladox: deleting oncprojectwiki oneupcloudwiki openonderwijswiki opgetfundingwiki opiumchangwiki opmiwiki orelandswiki osapwiki oscwiki oshyewiki osintwiki ossiwiki otherworldwiki otimizewiki oultrecaramaniawiki outoriowiki overseasuwlwiki oxfordniwiki pactrackwiki palatiaswiki pamwiki panselmewiki panthertoonswiki paradiselodgewiki paranormalwiki parentswiki parliamentarydemocracywiki parvisregionwiki pattawiki (per dormancy policy)
• 22:16 paladox: deleting netapediawiki newalbionwiki newworldoriendwiki nghtwiki nicorewiki nicuwiki niggpediawiki nihilistenhymnewiki nlwikiwiki noahwikiwiki nomaliaswiki norskrapwiki nosqlwiki notbookwiki noteswiki novayawiki nowbuskingwiki nrkwiki nudelworldbuildingwiki numhwywiki nursewikiwiki nutritionwiki nutscriptclonewarswiki nvcowiki nvliuwiki nxwikiwiki nyohwikiwiki o3dpwiki oalandwiki oceanuswiki oecumenewiki officialgeosworldalpharchivewiki ogowiki okwiki omcwikiwiki omfaxsystemswiki (per dormancy policy)
• 22:14 paladox: deleting ms2987wiki mscyberprivacywiki msgilwiki msnhinet8testwiki msnhinet8wiki mspugwiki mtamasterswiki mtascripterswiki muktipediawiki mundanapediawiki mundepwiki mundissongcontestwiki murlidharwiki musgovwiki music101wiki musiccupwiki musicpediawiki musicwikiwiki musicworldgamewiki mvwiki mwpermtestwiki myao93wiki myclassprojectswiki mydevexperiencewiki mymoneywiki mynintendowiki mysummerholydayswiki n2gowiki nafetswiki nagygenitalorszagwiki namudatawiki nanotrixwiki natturkowiki navmedwiki nayan28589wiki nbss13wiki neoottomanwiki (per dormancy policy)
• 22:12 paladox: deleting medicowiki meditationwikiwiki megacorpwiki megavisionwiki memburyofficialwiki memorieswiki menageriewiki menschenrechtewiki mesturetwiki metalistwiki metodosnumericosunalmzlwiki mfcwiki mfscwiki michelleatallahwiki mightymeeplewiki mikipediawiki minewiki miningpromieswiki minnanpediawiki mirahezewiki miraiakariwikiwiki mirbalwiki mitihoneywellwiki mkwikiwiki moatwiki modellbahnwiki modelusgovwiki moosbachwiki mouziloarchiveswiki mpwiki mrmaywiki (per dormancy policy)
• 22:10 paladox: deleting lyricswiki madiklopediwiki mafiasocietywiki magipediawiki magnaorbiswiki magnoliahousewiki maidenheadroleplayingwiki maindotawiki mainlandsongcontestwiki makeiteasyautoswiki malabwiki mamsprawdzonewiki manawinstttwiki maralertwiki marconswiki marksteinwiki masariwiki mathteachwiki matomotestwiki mattlemon675wiki mattsworldbuildingwiki maztowikiwiki mbitswiki mcbejpwiki mcbscomm12wiki mckwiki meadowtrailwiki medehollinwiki medgwiki mediacultures1wiki mediamatikawiki mediawiki (per dormancy policy)
• 22:03 paladox: deleting libobasicwiki libreartwiki liffeywiki ligaleciwiki ligamxwiki lightningbugwiki limeydragonwiki linglotwiki litcircleswiki literaryfreedomwiki littletikeshackswiki liveandlawwiki livesnowmapwiki livingdeadmanwiki livwiki lmvmdnoteswiki lomewiki lonsichoeumwiki loopywiki lostminecraftminerswiki lovelivewiki lovetweakswiki lowwiki lucewiki lucklessfoolswiki lucunewiki luntarawiki luzenglishcornerwiki (per dormancy policy)
• 22:00 paladox: deleting korbowiki kotakmimpiwiki krebswiki kreuz9561wiki kscmwiki ktspatentprosecutionwiki kzhealthwiki lacroiseedescheminswiki laikanprojectwiki lambdapsikkywiki lambelwiki lamdlcaselistwiki lapoliticswiki larcencieldeghiswiki lasrwiki lastcrewwiki latywiki laugofwiki lawlibwiki lawlrainbowversewiki lazygtawiki legalmovieswiki leginawiki legitimatestrategywiki leparkingetage63wiki leuvensecommonswiki lexicwiki lfobwiki (per dormancy policy)
• 21:58 paladox: deleting jinjuwiki johntestwiki jonraswiki jordanpetersonwiki jourrougewiki jurisdictionswikiwiki jurrassicworldwiki justipediawiki jwjohnsojrwiki kaltawiki kalyandevelopersthrissurwiki kapuluanwiki kassaiwiki kastritislabwiki kasucudenwiki ketitawiki keyruswiki kfootballwiki kgbwiki kgdrankwiki khepwiki kithearawiki kittydogwiki kkunnawiki klimagruppewiki knoweldgebasewiki knowledgeboxwiki knowledgepediawiki koisomemomijiwiki (per dormancy policy)
• 21:57 paladox: deleting integr8wiki intelwiki intelymerwiki internetstateswiki intrabasewiki iridianwiki ironandrustwiki ironsagawiki isatwiki itautowiki itreewiki iveaghwiki iwgwiki j4fclanwiki jaaglwiki jaavierselfdocumentationwiki jackalbersonwikiwiki jacktanwiki jalproversewiki janitestwiki japanunivwiki javaforjuniorwiki jaydecuswiki jbpdiscordwiki jdjwiki jeunessewiki jgtextwiki jhalabwiki (per dormancy policy)
• 21:54 paladox: deleting hpfwiki hrwiki hugawiki huncityvisionsongcontestwiki hydrantwikiwiki ialadinwiki iansnotebookwiki idealsongcontestwiki identityvwiki ideowiki idobelievewiki idolmasterwiki idoxwiki iecmhwiki ifapmecp1wiki ifmwiki ihadwiki iibwiki iictestwiki iicwiki iitpwiki im2wiki imaginewiki imhowiki immoinvestwikiwiki improwikiwiki imwiki inactifswiki indiasimwiki infapomgrwiki infobasewiki informatiquewiki insightfulmindswiki (per dormancy policy)
• 21:52 paladox: deleting hanminwiki harrypotterwiki harveyfengwiki hawkingmysterywiki hawkvisionwiki haxballportugalwiki healthylivewiki heidelbergwiki helpmectfwiki henwikiwiki heritagewikiwiki hesperusartswiki hgwikiwiki hillfolkwiki hjorthwiki hkimgboardwiki hklaicharwiki hnrgwiki hobbieswiki hobbymewiki hofstragrammarwiki holidaymasterswiki hollowcrownwiki homeiotwiki homenetwiki hordeswiki hornewiki hossainelsafayetwiki hostmicrobewiki howtogradewiki (per dormancy policy)
• 21:50 paladox: deleting giladipediawiki gliesewiki globaldataconsortiumwiki gloopsandboxwiki glueckssammlerwiki glw35wiki gmdkoreanwiki godottutswiki goldenpalladinwiki gomalawiki gombergwiki goodcounselwiki goodwiki gospelstudieswiki govarchivewiki graidyardwiki graveswiki graymainewikiwiki greeklawswiki grevanwiki greyhawkwiki grovespacewiki gsdmafwiki gstvpcpdwiki gunsgirl2wikiwiki h1z1wikiwiki habbohangwiki haifawiki halaqawiki (per dormancy policy)
• 21:48 paladox: deleting freerunetwiki freevandalismwiki freewikiwiki frknuniversewiki fxwiki fyonwiki g8kwiki gaeaworldwiki galaxier38wiki gamdugwiki gamingwikiwiki garfieldroboticswiki gastronomiaromaneascawiki gctwikiwiki gecomwiki gemawiki geminizuniversecomwiki genesiswiki geodominuswiki geometrydashkrwiki gestionticswiki gettyfamilywiki gfernandezwiki ggwikiwiki giangiwiki gianroyalwiki gibsonwiki (per dormancy policy)
• 21:47 paladox: deleting exceliawiki eyrwiki factory4rndwiki fair0002wiki fakepaediawiki familiahornoswiki faretraitewiki ferferwiki fiberpediawiki financialfindswiki fingerprintwiki fisicawiki flamepediawiki flipnotewiki floogstarboundwiki fmbvwiki fnecwiki foolmaptoolboxwiki forexobbrasilwiki forexwiki fortnitewiki forwardbiaswiki foundationwiki fpvwiki fquestwiki franmagdlaenawiki freecollegeprojectwiki freedomrespectlogoswiki (per dormancy policy)
• 21:45 paladox: deleting enchantmentwiki enciclopediaposmodernawiki enexpomathwiki engineeringnoteswiki engstudiowiki ensemblemulhousewiki entmonwiki enwiki epimorfositpewiki erethwiki erothwiki erotipediawiki ertywiki esogeekswiki esrokuakawiki estoelwiki eswikitransparencywiki etatenvauclusewiki etheliawiki eturdayawiki evawiki everythingiknowwiki evotvwiki evozangtalwiki exampediawiki (per dormancy policy)
• 21:38 paladox: deleting dragonwarriorwiki drakawiki drones4allwiki drothwiki dsgroupwiki duatprojectwiki ducklingswiki durawiki dynaswiki dys31wiki eaberlinwiki earthianwiki ecegradswiki econhistgenealogywiki economicswiki ecoprospectorswiki ecoucherwiki edominationsptbrwiki educationalgamesv2wiki edugirowiki egyptlawwiki elarawiki electracoinwiki elementaladvwiki elgafferwiki embobadawiki emeralddawnwiki (per dormancy policy)
• 21:34 paladox: deleting deadleaguewiki deixacomigowiki dekoerwiki denasoftiowiki denasoftkbwiki denimonewiki designatedsurvivorwiki dev7nullwiki devexperiencedwiki didaprowiki digitaldesigntechwiki dijarawiki dimplexwiki disabledlifewiki discipleshipanonymouswiki discordvisionteamswiki disinstrucpediawiki dollywiki dominiumwiki domithiwiki donaldduckwiki donaldtrumpwikiwiki doubleatravelwiki drachenbergwiki dragalialostwiki (per dormancy policy)
• 21:32 paladox: deleting cpcwiki cpudevwiki crazybloxiantimeswiki credcowiki cryptobotswiki crystlejexwiki culturemappingwiki cunydh2018wiki curiousdreamswiki currentmusicwiki cwiki cyberiawiki cybertripwiki cyberweatherwiki cycleofbloodwiki cyrylwiki cywiki czelwiki dakhilcommunitywiki dancascirculareswiki dancerconcretedesignwiki dascwiki datarightswiki davidewiki dcinsidewiki (per dormancy policy)
• 21:21 paladox: deleting cmggcmwiki cmnhiwiki cnamewiki cocrevampmodwiki codeuinowiki codexwiki cognetikwiki cogversewiki coineli5wiki coinspectwiki colpittsdesignwiki coltldwiki comexitaliawiki commonswiki compartirwiki comradeswiki comsiswiki conversewiki coppercubewiki corepediawiki corporativetestewiki counterwikiwiki coxtiwiki (per dormancy policy)
• 21:19 paladox: deleting cbtswiki cchpcwiki cctestwiki cdcwiki cdsicwiki celiawiki cfcwiki cfpluswiki cfwiki chamberwiki changemakerswiki chaosrollwiki cheesewinewiki chemwiki chessypigwiki christianmusicwiki chtechwiki circlewiki citixxiwiki civcraftwiki cjwikiwiki ckcwiki classhangoutwiki classsoftwiki cloakcoinwiki clwwiki (per dormancy policy)
• 21:17 paladox: deleting brokenironwiki bsawikiwiki budawiki bugsigdbwiki buyanypartwiki byblopediawiki byfireandswordrpgwiki caes2wiki2ppiswiki caestaronwiki caltherawiki canailleswikiwiki cancelledgameswiki cancelledmovieswiki cannabisconcentrateswiki cantorijwiki capcyberwikiwiki cardmonsterswiki carmeltechwiki casebankwiki caswiki (per dormancy policy)
• 21:15 paladox: deleting bigbrothersimulatorwiki bikeshedwiki bildungwiki bimmanualwiki birdspediawiki biskwiki bitnetwiki bkdrumwiki blhawiki bloodclotwiki bloowikiwiki blueskyocgwiki bluewaywiki bnntalkwiki boardroomwiki bobbyshawwiki bobocharwiki bohwikiwiki bozemanstorybuilderswiki brasilnanzwiki briocheswiki (per dormancy policy)
• 21:12 paladox: deleting aumatwiki auriliawiki aussimwiki austnationalgamewiki automationlondonwiki automobilewikipediawiki avoronwiki awesomegameswikiwiki aymcommercewiki ayurjakewiki b1tcashwiki b1teswiki badwrigglerswiki barbarismwiki barbecubewiki bardacontinentwiki bataswiki bausudwikiwiki bbawiki bbsazwiki bcadharmaschoolwiki bdorpwiki behmodwiki bementlabwiki bernimpressionswiki berntoonswiki bettermovieswiki bhffedwiki bhimeswiki biastrackerwiki biblebeahkusuwiki bibliothecalucemwiki (per dormancy policy)
• 21:09 paladox: deleting apexpredatorwiki aposbgwiki apostoliswiki apowiki appdecatwiki appsecwikiwiki aptozwiki aratiswiki archerwiki aresteamwiki aresteamwikiwiki arizonancornerwiki arkorderwiki arma3frwiki arqibewiki artsycraftwiki arynwiki asheeshscratchwiki assemblingphysicswiki astoriinfwiki astrapediawiki atablswiki atemreichwiki (per dormancy policy)
• 21:06 paladox: deleting alacritysimwiki alaskapediawiki albustestwiki alewiki alfredwiki alhayarmaths6wiki alineowiki althistoriawiki althistorywiki alzestorswiki amarenawiki ambirtestwiki amielbrickellwiki amirampwiki amshalwiki andusanwiki animamondaywiki aniwotawikiwiki anomalieswiki ansiblewiki antropologiaargentinawiki (per dormancy policy)
• 20:45 paladox: deleting 10yearswiki 123sartawiki 690squadronwiki 8recommendswiki aa2022wiki abhirupghoshwiki accorderiewiki accountingwiki acordswiki acqintwiki acrotrendwiki actartletraswiki adamwiki adminbuswiki advisingwiki aerobristolwiki aeromwiki aescwiki aetheriawiki afterlifewiki agathachristiewiki ageofwormswiki airportsofwings900wiki aisgwiki aistudywiki wiki's (per dormancy policy)
• 20:28 Southparkfan: purged 16G of mysql binary logs
• 20:23 Southparkfan: removed parted package from db4
• 20:16 Southparkfan: install 'parted' on db4

2019-04-29

• 19:13 paladox: restarting php7.2-fpm on mw1 (due to "Resource temporarily unavailable")
• 18:26 John: UPDATE mw_permissions SET perm_addgroupstoself = '[]' WHERE perm_addgroupstoself = ;
• 18:26 John: UPDATE mw_permissions SET perm_removegroupsfromself = '[]' WHERE perm_removegroupsfromself = ;
• 18:19 John: DELETE FROM mw_permissions WHERE perm_group = "modifyGroupPermission.php";

2019-04-27

• 23:16 paladox: removing read from * on all wiki's
• 22:43 paladox: rewording last log to state all wikis have "read" remove from user
• 22:40 paladox: running modifyGroupPermission.php on mw1 - removing all read perms from 'user' permission on all private wiki's
• 16:24 paladox: upgrade ns1 to debian 9.9
• 15:53 paladox: revoke read from user on the default table in mw_permissions
• 15:45 paladox: revoke read from * in default table in mw_permissions
• 15:08 paladox: revoke read from * on meiouandtaxeswiki
• 13:09 paladox: upgrade puppet1 to debian 9.9

2019-04-26

• 22:06 paladox: disable automatically archiving from the web browser in matomo. (per recommendation, switching it to a cron)
• 21:54 paladox: running ./console core:purge-old-archive-data all on misc2
• 16:38 paladox: running ./console core:archive --url=https://matomo.miraheze.org/ > ./test.log on misc2 (per settings recommendation)

2019-04-25

• 14:39 paladox: upgrade icingaweb2 to 2.6.3 on misc1

2019-04-22

• 23:51 paladox: restarted rc bot
• 20:51 John: fixed some malformed DB entries for perm_autopromote
• 18:26 John: UPDATE mw_permissions SET perm_autopromote = '["&",[1,10],[2,345600]]' WHERE perm_dbname = "default" AND perm_group = "autoconfirmed";
• 18:13 John: populating perm_autopromote on all wikis using a hack script
• 02:45 John: add self group management to groups via DB sql (very few uses apparently")

2019-04-21

• 21:39 paladox: repooled mw2
• 21:35 paladox: depooled mw2 to reinstall php
• 16:44 paladox: running lc on mw*
• 15:38 John: UPDATE mw_permissions SET perm_removegroups = '[]' WHERE perm_removegroups = 'null';
• 15:38 John: UPDATE mw_permissions SET perm_addgroups = '[]' WHERE perm_addgroups = 'null';
• 14:18 paladox: upgraded some packages on mw2 (php-fpm, nodejs, nginx, systemd, wget and a few more)
• 14:17 paladox: upgraded some packages on mw1 (php-fpm, nodejs, nginx, systemd, wget and a few more)
• 14:16 paladox: upgraded some packages on mw3 (php-fpm, nodejs, nginx, systemd, wget and a few more)
• 13:21 paladox: running gdb for php (due to a segmentation fault)
• 13:20 paladox: installing gdb on mw2
• 01:09 paladox: update mw_permissions set perm_removegroups = '[]' where perm_removegroups like 'null%' on db4
• 01:09 paladox: update mw_permissions set perm_addgroups = '[]' where perm_addgroups like 'null%'; on db4

2019-04-20

• 21:42 paladox: running lc on mw*
• 17:42 paladox: installing https://apt.wikimedia.org/wikimedia/pool/main/t/trafficserver/trafficserver-experimental-plugins_8.0.3-1wm1_amd64.deb on test1
• 17:42 paladox: apt-get install libcjose0 libjansson4 libmemcached11 libmemcachedutil2 -t stretch-backports on test1

2019-04-19

• 23:32 paladox: stopping nginx on test1
• 22:54 paladox: apt-get --purge remove trafficserver on cp4 (installing trafficserver on test1 instead).
• 22:52 paladox: noting that i also "apt-get install libc++1 libc++abi1 libhwloc5 libluajit-5.1-2 libtcl8.6 libunwind8 libnuma1 libluajit-5.1-common libhwloc5 -t stretch-backports" required by trafficserver
• 22:49 paladox: installing trafficserver on cp4 to test (from https://apt.wikimedia.org/wikimedia/pool/main/t/trafficserver/trafficserver_8.0.3-1wm1_amd64.deb)
• 21:19 paladox: upgrading puppetdb to 6.3.1 on puppet1
• 17:30 Southparkfan: remove sysbench package from cp4 (+db4, I also benchmarked db4)
• 17:26 paladox: restarted php-fpm and nginx on misc2
• 17:25 Southparkfan: install 'sysbench' on cp4
• 16:50 paladox: running lc on mw*

2019-04-18

• 12:15 paladox: starting import for abysmalrobloxgameswiki on mw2
• 11:59 paladox: starting dump import for trollpastauncensoredwiki on mw2

2019-04-17

• 21:31 John: set columns to be [] where null in the value in mw_permissions
• 20:18 paladox: renaming imperiuswiki to addawiki - T4296
• 17:54 paladox: chrown sotuhparkfan:mail for southparkfan in /var/mail/ on misc1 (matching the other files)
• 14:54 paladox: rolling out php-fpm tweeks
• 14:00 paladox: live hacking puppetmaster
• 13:39 paladox: disabling puppet on mw3 to test tweeking php-fpm using live traffic.
• 13:35 paladox: disabling puppet on mw2 to test tweeking php-fpm using live traffic.
• 13:02 paladox: disabling puppet on mw1 to test tweeking php-fpm using live traffic.
• 02:24 paladox: killed a long running puppet process causing higher cpu then normal on db4
• 00:47 paladox: removing cp5 from salt-master
• 00:45 paladox: restarting php-fpm on mw* (depooling each one as i restart)

2019-04-16

• 19:50 paladox: rebuilding lc on mw*

2019-04-15

• 21:57 John: verified domain ownership within GitHub

2019-04-13

• 17:56 paladox: deleting user Micru on phabricator
• 16:42 paladox: update mw_permissions set perm_addgroups = '[]' where perm_addgroups like 'null%' and perm_dbname = 'shumikenwiki';
• 16:41 paladox: update mw_permissions set perm_removegroups = '[]' where perm_removegroups like 'null%' and perm_dbname = 'shumikenwiki';
• 16:39 paladox: update mw_permissions set perm_removegroups = '[]' where perm_removegroups like 'null%' and perm_dbname = 'anotheredenwiki';
• 16:39 paladox: update mw_permissions set perm_addgroups = '[]' where perm_addgroups like 'null%' and perm_dbname = 'anotheredenwiki';
• 15:47 paladox: grant "managewiki" right on kakukokkawiki again per T4278
• 01:18 paladox: re grant read to * for cristianopediawiki - T4274
• 01:12 paladox: remove perms cdb file from mw* (so it can be regenerated)
• 01:11 paladox: update mw_permissions set perm_addgroups = '[]' where perm_addgroups like 'null%';
• 01:11 paladox: update mw_permissions set perm_removegroups = '[]' where perm_removegroups like 'null%';
• 01:03 paladox: update mw_permissions set perm_removegroups = '[]' where perm_removegroups = null;
• 01:03 paladox: update mw_permissions set perm_addgroups = '[]' where perm_addgroups = null;

2019-04-12

• 01:06 paladox: running lc on mw*

2019-04-11

• 23:45 paladox: ran updateSpecialPages.php against nonciclopediawiki on mw1
• 20:04 paladox: running lc on mw*
• 02:30 paladox: upgraded phabricator on misc4 (hour and half ago)
• 00:26 paladox: generating image dump for mediatecawiki - T4261
• 00:26 paladox: generating image dump for holycrosswiki - T4262

2019-04-10

• 17:22 paladox: running lc on mw*
• 15:15 paladox: generating image dump for electowiki

2019-04-09

• 22:46 paladox: running lc on mw* and test1
• 22:40 paladox: applying data_dump.dumps_timestamp to all wikis through patch-dumps_timestamp.sql sql patch (on mw1)
• 05:40 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki cristianopediawiki /home/reception/Wikipedia-20190409003230.xml
• 05:33 Reception123: RENAMED vocalharmonywiki to metafolkwiki

2019-04-07

• 17:33 paladox: upgrade debian to 9.8 on cp2
• 17:26 paladox: update debian to 9.8 on cp4
• 16:44 paladox: restarted nginx on cp3, appeared to have not been serving traffic all day.

2019-04-06

• 23:23 paladox: root@bacula1:/bacula/backup# rm * on bacula1
• 23:22 paladox: db4, phab static and private git backups
• 23:11 paladox: deleting static backups
• 22:19 paladox: restoring kurwowizjawiki files from bacula
• 22:08 paladox: ^^ for bacula1
• 22:08 paladox: adding temporarily swap boost (3gb)
• 18:40 paladox: adding delete-dump, generate-dump and view-dump rights to all wiki's (for sysop)
• 18:37 paladox: add delete-dump, generate-dump and view-dump rights to 'default' in mw_permissions
• 18:11 paladox: run lc on mw*
• 17:28 Reception123: DROPPED databases mentioned before
• 17:27 Reception123: delete from CW_WIKIS badgameswikiwiki and crappygameswikiwiki
• 16:29 paladox: deleted WANCache:* key from redis
• 05:49 Reception123: DROP DATABASE wikilandwiki
• 05:49 Reception123: deleted wikilandwiki from cw_wikis
• 05:46 Reception123: DROP DATABASE taotacwiki
• 05:45 Reception123: deleted taotacwiki from CW_WIKIS
• 01:54 paladox: reverted hacks on mw3

2019-04-05

• 23:46 paladox: hacking change on mw3
• 22:28 paladox: upgrade elasticsearch nginx puppet-agent wget on elasticsearch1
• 22:28 paladox: upgrading grafana nginx puppet-agent wget on misc1
• 16:28 paladox: disabling blogpost temporarily on loathsomecharacterswiki
• 16:12 paladox: running lc on mw3
• 16:10 paladox: running lc on mw2
• 04:57 John: decom cp5

2019-04-04

• 23:54 paladox: running lc on mw*
• 21:47 paladox: starting import for pseudocienciawiki
• 20:58 paladox: repool mw2
• 20:31 paladox: depool mw2
• 20:30 paladox: repool mw3
• 20:23 paladox: depool mw3
• 20:05 paladox: repool mw1
• 19:57 paladox: depool mw1

2019-03-31

• 22:36 paladox: repool mw3
• 22:29 paladox: reboot mw3 to reduce ram consumption (almost OOM)
• 22:28 paladox: depool mw3
• 20:37 paladox: granting the rest of our mw extensions access to sonarcloud
• 20:27 paladox: authorising sonarcloud to read ManageWiki in github. (for finding vulns/bugs)
• 19:31 paladox: reverting prevous action
• 19:29 paladox: granting view-dump, generate-dump and delete-dump rights to sysops on metawiki
• 19:25 paladox: running lc on mw*
• 19:21 paladox: apply DataDump sql to all wiki
• 19:08 paladox: reboot misc4 due to high load
• 19:01 paladox: upgraded phabricator on misc4
• 14:56 paladox: upgrade puppet agent and salt on lizardfs* and misc3

2019-03-30

• 01:19 paladox: importing xml dump on fapceowiki T4175
• 01:18 paladox: upgrade php 7.3 to 7.3.3 on mw1
• 01:09 paladox: rename metapediawiki to itpediawiki T4226
• 00:42 paladox: running lc on mw*

2019-03-28

• 22:53 paladox: remove blogpost from hypopediawiki T4205
• 22:49 paladox: drop caputiawiki database
• 22:49 paladox: drop autombilepediawiki and zhautomobilepediawiki database.
• 22:48 paladox: drop iceriawiki database
• 22:38 paladox: starting import on animatedfeetwiki (mw1)
• 19:01 paladox: upgrading puppetdb to 6.3, puppet-agent to 6.4 and puppetserver to 6.3 on puppet1

2019-03-25

• 21:17 MacFan4000: deleting iceriawiki, autombilepediawiki, zhautomobilepediawiki, caputiawiki

2019-03-24

• 14:15 paladox: reboot puppet1
• 09:24 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki neciklopediowiki /home/reception/Neciklopedio-20190306130123.xml

2019-03-23

• 02:26 paladox: reinstalling postgresql on db4
• 02:02 paladox: dropping puppetdb db && recreating it
• 01:30 paladox: deleting misc1 from puppetdb && regenerating cert
• 01:18 paladox: temporarily stopping nginx on misc2
• 00:43 PuppyKun: (to bypass mime detection) php importImages.php --wiki concordancewiki /home/ndkilla/

2019-03-22

• 14:17 paladox: rm -rf /srv/mediawiki/w/cache/managewiki/** as getting "Failed to generate additional resources using 'eval_generate': invalid byte sequence in UTF-8"
• 13:53 paladox: restart puppetserver

2019-03-19

• 00:11 John: running hack script for Cargo move

2019-03-18

• 22:23 John: set ns_content = 1 in defaults for ns_namespace_id = 0
• 17:48 paladox: restarting puppetserver && puppetdb on puppet1

2019-03-17

• 19:53 paladox: reload lizardfs-master

2019-03-16

• 22:09 paladox: deleted pokemonadventures wiki
• 19:29 paladox: re enable puppet on mw*
• 15:25 paladox: disabling puppet on mw*
• 14:36 paladox: drop table searchindex on allthetropes wiki
• 01:35 PuppyKun: changed the mail password for ndkilla (yes again)

2019-03-15

• 01:54 paladox: set manage_etc_hosts to false in /etc/cloud/templates/hosts.tmpl on cp3
• 01:39 paladox: reboot cp3

2019-03-14

• 13:37 paladox: [13:22:07] <+paladox> !log increasing swap to 512M on cp3
• 13:26 paladox: [13:22:07] <+paladox> !log increasing swap to 512M on cp3
• 13:18 paladox: adding swap to cp3 (256MB)
• 00:06 paladox: restarted puppetserver on puppet1

2019-03-13

• 18:35 Southparkfan: repool mw1 and enable puppet on mw1/cp4
• 17:38 Southparkfan: depool mw1
• 17:30 Southparkfan: disable puppet on cp4
• 17:14 paladox: indexing allthetropeswiki for elasticsearch
• 17:12 paladox: generate elasticsearch index for allthetropeswiki
• 16:48 Southparkfan: disabling puppet on mw1
• 15:35 Southparkfan: drop searchindex table for nonsensopediawiki
• 13:11 PuppyKun: changed the misc1 shell passwd for ndkilla (root@misc1:/home/ndkilla# passwd ndkilla)

2019-03-12

• 23:46 Southparkfan: drop metawiki.searchindex
• 23:21 Southparkfan: populating elasticsearch for metawiki
• 05:31 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki damselswiki /home/reception/damsel_pages_full.xml

2019-03-11

• 22:59 paladox: switched test1 to elasticsearch
• 01:07 paladox: running lc on mw*

2019-03-10

• 22:24 paladox: created data_dump table (and inserted sql) on test1wiki

2019-03-09

• 22:29 paladox: created DataDump repo
• 22:16 paladox: update php on test1 && nginx && openssh (and a few other packages)
• 17:57 MacFan4000: deleting astropediawiki and parapediawiki (wikis of an LTA)

2019-03-08

• 21:18 paladox: starting import on mw2
• 17:45 paladox: starting gdb on mw1 (to debug segfaults with php)

2019-03-05

• 00:35 paladox: re enable puppet on all hosts

2019-03-04

• 22:58 paladox: enabling puppet on mw*
• 22:14 paladox: re enabled puppet on mw1
• 22:05 paladox: disabling puppet on all hosts
• 22:02 paladox: root@mw1:/mnt/mediawiki-static/private/miraheze/letsencrypt# cp -rp /etc/letsencrypt/** ./
• 19:39 paladox: migrate "Phantom Dusclops'92" from nonciclopedia to be attached to it's centralauth account.
• 17:04 paladox: apt-get install php7.2-xdebug on mw1
• 17:02 paladox: install gdb on mw1

2019-03-03

• 22:14 Southparkfan: revert previous actions, operation is too risky
• 21:48 Southparkfan: disable swap on db4; attempting to reduce swap from 16GB to 6GB and reclaiming 10GB in root partition

2019-03-02

• 18:21 Reception123: reception@mw1:~$ cd /srv/med*/w && sudo -u www-data git pull && sudo -u www-data git submodule update (Puppet disabled on mw1, inbalance created by change)
• 16:59 paladox: mkdir /mnt/mediawiki-static/private/miraheze && /mnt/mediawiki-static/private/miraheze/letsencrypt
• 16:46 paladox: stop miraheze ssl renewal service on mw1 && disable puppet
• 15:44 paladox: installing security updates on db4, mw*
• 15:36 paladox: upgrade puppet1 to debian 9.8
• 15:24 paladox: upgraded salt on bacula1
• 14:59 paladox: upgraded grafana to version 6 and icinga2 to 2.10.3 on misc1
• 14:14 Reception123: reception@mw3:/srv/mediawiki/w/maintenance/storage$ sudo -u www-data php compressOld.php --wiki absurdopediawiki -t = 'gzip'
• 06:13 Reception123: reception@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki absurdopediawiki /home/reception/absurdopedia_reverted_pages_full.xml
• 02:05 Southparkfan: graceful vpncloud shutdown
• 01:20 Southparkfan: previous entry also for db4
• 01:07 Southparkfan: ensure presence of iperf, vpncloud and net-tools packages on cp5 and test1

2019-02-28

• 17:06 paladox: import beidipedia dump
• 16:49 paladox: upgrade phabricator
• 16:27 paladox: [15:25:56] <+paladox> !log rm mysql-bin.0* on db4
• 16:27 paladox: [15:19:55] <+paladox> !log stopping mysql (to be able to remove the binary logs manually)
• 16:27 paladox: run lc on mw*

2019-02-26

• 00:14 paladox: run lc on mw*

2019-02-25

• 17:24 paladox: run populateWikiSettings.php inserting wmgUseModeration into the db
• 17:11 paladox: MatomoAnalytics::deleteSite( 'pdacwiki' ); in eval.php (site does not exist)
• 17:10 paladox: MatomoAnalytics::deleteSite( 'alterwiki' ); in eval.php (site does not exist)
• 17:08 paladox: run MatomoAnalytics::addSite( 'mystarawiki' ); ; MatomoAnalytics::addSite( 'alterwiki' ); ; MatomoAnalytics::addSite( 'pdacwiki' ); in eval.php (sites were not in matomo)

2019-02-24

• 23:03 paladox: running lc on mw*
• 17:13 John: UPDATE mw_permissions SET perm_permissions = '["createaccount","read","edit","createpage","createtalk","writeapi","viewmywatchlist","editmywatchlist","viewmyprivateinfo","editmyprivateinfo","editmyoptions","abusefilter-log","abusefilter-log-detail","abusefilter-view","autocreateaccount","centralauth-merge","oathauth-enable"]' WHERE perm_dbname = "metawiki" and perm_group = '*';
• 16:47 paladox: repool mw1
• 16:40 paladox: upgrade php on mw1 to 7.3.2 (bug fixes) && also dist upgrade to 9.8
• 16:40 paladox: depool mw1
• 16:37 MacFan4000: deleting permission cache files for metawiki
• 16:33 MacFan4000: UPDATE mw_permissions SET perm_permissions='["createaccount","read","edit","createpage","createtalk","writeapi","viewmywatchlist","editmywatchlist","viewmyprivateinfo","editmyprivateinfo","editmyoptions","abusefilter-log","abusefilter-log-detail","abusefilter-view","autocreateaccount","centralauth-merge","oathauth-enable""]' WHERE perm_dbname='metawiki' AND perm_group='*';
• 16:29 paladox: upgrade misc1 to debian 9.8
• 16:27 paladox: repool mw3
• 16:26 MacFan4000: UPDATE mw_permissions SET perm_permissions='["translate-review","move","move-subpages","move-rootuserpages","move-categorypages","movefile","read","edit","createtalk","writeapi","upload","reupload","reupload-shared","minoredit","editmyusercss","editmyuserjson","editmyuserjs","purge","sendemail","applychangetags","changetags","editcontentmodel","requestwiki","user","translate-messagereview","mwoauthmanagemygrants"]' WHERE perm_dbname='metawiki' AND perm_group='user';
• 16:22 MacFan4000: UPDATE mw_permissions SET perm_permissions='["autoconfirmed","editsemiprotected","move","createpage","translate","mwoauthproposeconsumer","mwoauthupdateownconsumer","skipcaptcha","createaccount"]' WHERE perm_dbname='metawiki' AND perm_group='autoconfirmed';
• 16:21 paladox: reboot mw3
• 16:19 paladox: upgrade php on mw3 to 7.3.2 (bug fixes) && also dist upgrade to 9.8
• 16:19 paladox: depool mw3
• 16:18 MacFan4000: UPDATE mw_permissions SET perm_addgroups='["flood","translator","translationadmin","autopatrolled","confirmed","rollbacker"]' WHERE perm_dbname='metawiki' AND perm_group='sysop';
• 16:18 paladox: repool mw2
• 16:13 MacFan4000: UPDATE mw_permissions SET perm_permissions='["autopatrol","block","blockemail","browsearchive","createaccount","createpage","createtalk","delete","deletechangetags","deletedhistory","deletedtext","deletelogentry","deleterevision","edit","editcontentmodel","editinterface","editprotected","editsemiprotected","editusercss","edituserjson","edituserjs","import","importupload","minoredit","move","noratelimit","override-export-depth","pagelang","patrol","patrolmarks","protect","purge","rollback","suppressredirect","unblockself","undelete","upload_by_url","abusefilter-modify","abusefilter-log-detail","abusefilter-view","abusefilter-log","abusefilter-modify-restricted","abusefilter-revert","abusefilter-view-private","abusefilter-log-private","override-antispoof","gadgets-edit","gadgets-definition-edit","globalblock-whitelist","nuke","oathauth-enable","oathauth-api-all","mwoauthproposeconsumer","mwoauthupdateownconsumer","tboverride","tboverride-account","titleblacklistlog","translate","translate-import","translate-manage","translate-messagereview","translate-groupreview","pagetranslation","massmessage"]' WHERE perm_dbname='metawiki' AND perm_group='sysop';
• 16:11 paladox: rebooted mw2
• 16:09 paladox: upgrade php on mw2 to 7.3.2 (bug fixes) && also dist upgrade to 9.8
• 16:08 paladox: depool mw2
• 16:06 MacFan4000: UPDATE mw_permissions SET perm_removegroups='["interface-admin","bot","sysop"]' WHERE perm_dbname='metawiki' AND perm_group='bureaucrat';
• 16:05 MacFan4000: UPDATE mw_permissions SET perm_addgroups='["interface-admin","bot","bureaucrat","sysop"]' WHERE perm_dbname='metawiki' AND perm_group='bureaucrat';
• 15:54 MacFan4000: UPDATE mw_permissions SET perm_permissions='["move","createpage","translate","mwoauthproposeconsumer","mwoauthupdateownconsumer","editsemiprotected","autoconfirmed","skipcaptcha"]' WHERE perm_dbname='metawiki' AND perm_group='confirmed';
• 15:51 MacFan4000: UPDATE mw_permissions SET perm_permissions='["rollback"]' WHERE perm_dbname='metawiki' AND perm_group='rollbacker';
• 15:46 MacFan4000: UPDATE mw_permissions SET perm_permissions='["editinterface","editusercss","edituserjson","edituserjs","editsitecss","editsitejson","editsitejs"]' WHERE perm_dbname='metawiki' AND perm_group='interface-admin';
• 15:43 MacFan4000: DELETE FROM mw_permissions WHERE perm_dbname='metawiki' AND perm_group='comentadmin';
• 15:43 MacFan4000: DELETE FROM mw_permissions WHERE perm_dbname='metawiki' AND perm_group='chatmod';
• 15:40 MacFan4000: DELETE FROM mw_permissions WHERE perm_dbname='metawiki' AND perm_group='forumadmin';
• 15:39 MacFan4000: DELETE FROM mw_permissions WHERE perm_dbname='metawiki' AND perm_group='oversight'; (was only adding one perm)
• 15:35 MacFan4000: DELETE FROM mw_permissions WHERE perm_dbname='metawiki' AND perm_group='reviewer';

2019-02-23

• 20:59 paladox: upgrading puppet* to 6.3 (puppetserver to 6.2) on puppet1
• 03:27 paladox: run compressOld.php on baobabarchiveswiki (on mw1)
• 01:47 paladox: mass restarting of bacula-fd accross all servers

2019-02-22

• 23:56 paladox: puppetserver ca sign --certname bacula1.miraheze.org on puppet1
• 23:52 paladox: puppetserver ca clean --certname bacula1.miraheze.org on puppet1
• 23:52 paladox: puppetserver ca revoke --certname bacula1.miraheze.org on puppet1
• 22:43 paladox: reimage bacula1 with 64-bit stretch image
• 22:25 Southparkfan: aaaaaand give tun/tap yet another try
• 22:12 paladox: shutdown bacula1
• 21:59 Southparkfan: disable TUN/TAP on test1, seems to break stuff
• 21:42 Southparkfan: enable TUN/TAP on test1, may require reboot
• 21:31 Southparkfan: install vpncloud package on test1

2019-02-21

• 20:34 Reception123: sudo -u www-data /usr/bin/nice -12 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/runJobs.php on mw1

2019-02-20

• 15:21 paladox: upgrade misc4 to debian 9.8
• 15:18 paladox: disabling puppet on lizardfs*
• 15:04 paladox: shut down prometheus master on misc4
• 14:25 paladox: shutting down salt on misc4 (and removing)
• 14:15 Reception123: rebuild LC on mw*
• 08:02 Reception123: rebuild LC on all wikis
• 06:54 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki pathofexilewiki /home/paladox/Path+of+Exile+Wiki-20190211171816.xml

2019-02-19

• 15:25 paladox: running lc on mw*
• 01:44 paladox: delete wiki's electronicsforallwiki eli5wiki elitefestwiki ellersliedemowiki elsharqwiki elswiki enchantedsongcontestwiki engineering4kidswiki epbwiki epi36wiki erdewiki escaladawiki escwiki eurovisiongameswiki everythingmodwiki experimentalpublicationgroupwiki exposingjillianepperlywiki eyebobswiki ezidiawiki fablacwiki faespherewiki fagslandwiki farfallewiki fatinsangelswiki fcsplwiki felfelwiki fgnoteswiki firstroboticswikiwiki fisicaapplicatawiki flatwiki florianwikiwiki flumeisinglasswiki foodhallwiki forgottennovelwiki fortheempirewiki fosnetwiki fredybashwiki freestateofjoneswiki freewillwiki frejaerikssonwiki frmwiki frontdeskswiki fsxworldspainwiki funnyvisionsongcontestwiki furbyislandwiki futurepartywiki fyrnsidubrasilwiki fysm1217wiki gafwiki gahoolewiki gamewiki gapzerodechetwiki garrettcountyguidewiki gastrolegnanowiki gatewiki gaysinspacewiki gendertransitionwiki generationsbamorowiki geometrydashwiki geopediawiki geresbamyanwiki gesotericowiki ggdrwiki gifpglobalwiki gilaaswiki givewiki gontranwiki granislawiki grbuenowiki grecipeswiki grinopellewiki groepeenwiki gsunixwiki guawiki gugacastwiki gunsensewiki guyswikiwiki hackwikiwiki halloweenwikiwiki hamimwiki hamkaastostiwiki hantpediawiki harmontownwiki haxionspacewiki hccwiki hdafbgwiki healthvectorwiki heathenwiki hedgeyourbetwiki heimwegtelefonwiki herazimewiki hestiaroboticswiki hexelswiki hiddenhistorywiki hirukowiki hockeyprospectswiki hogsmeadewiki holmeswiki holonterapiwiki homestuckwiki homevideowiki honkai3rdwiki hostitaliawiki howtogettowiki iawwiki iceposeidonwiki icmscholarswiki icterwiki idwwiki ignitewiki igorhersonwiki ijebustateinfowiki ilzurichwiki imperialwiki infiniteunknownwiki infofizsmwiki infojefprivwiki inkbloodwritersguildwiki innovwikiprotowiki integrawiki intersectionwiki inversionpasivawiki iotawiki ironswordswiki isfcomodorowiki isiliawiki islamwikillywiki isnstjustwiki itasiawiki itspuglewiki iubriswiki iwnwiki iwysanwiki jayprakash12345wiki jeanargentywiki jennyscwiki justinbieberwiki jwwikiwiki karagashwiki kasopediawiki kaufmannwiki kazhikwiki kbrprojectwiki kclwiki keepbusheycountryclubwiki kempowiki kerbalismwiki khorashanwiki kioscwiki kl6fwiki knowledgewiki kpiqwiki kpopswiki kspancevowiki kstartupswiki kzagwiki l45testwiki lambdawarswiki landofikioswiki langspacewiki languagerevitalizationwiki lapeninsulewiki larpsiteswiki lawrencescafuriwiki lcars47wiki lecolibriswiki leecheswiki leeryescribirenlaeradigitalwiki leftypolwiki lensoftwiki leyesprwiki liberhistoricawiki liesbornwikiwiki lifewiki liko12wiki liutfprwiki localizationwiki lodjawiki logalnetwiki lolwiki loraldwiki lovelyzwiki lspdfrwiki ltgsaleswiki luskwiki lzhscpwikiwiki macedoniskagovwiki machiningwiki magicvariantswiki makerlabwiki makingitpodcastwiki makipediawiki maktabawiki masseffectwiki mastermindhackswiki mathwiki maxwellwiki mcserawiki medergistswiki medicinawiki medienkurswiki mekniywiki mentalhealthinfoukwiki mesonwiki metaldetectorwiki mikeandchrisproductionswiki mikrocontrollerwiki minariwiki minecraftwiki minerrowiki miraiwiki mistralcupwiki mistralwiki mncwikipediawiki mobawiki modwikichickenkillerwiki monsterheartswiki monsterlegendswiki moofmilkerswiki morfarkultenwiki mscwiki msramwiki muenchowwiki mugiwikiwiki mumkeycinematicuniversewiki munawikiwiki musi339wiki musmyswiki mutewiki mvscwiki mwswiki mycpuwiki myragnarokwiki mystic20swiki mytenniswiki mywisdowwiki nanodlpwiki nathandernicewiki nationsglorywiki nddb2wiki neptuniawikiwiki nerdwiki nescwiki nestartstechwiki netthreewiki newnamlawiki newsalinaswiki newsniviwiki newstickerwiki ngrwikiwiki nhauwiki nigelshikeswiki nikiwiki nocontextforyouwiki noobdevopswiki norfolkconstabularywiki notezenwiki nouvellequerbeswiki ntlawwiki nulibrtewiki nvdanlwiki nysyoungcandidateswiki occultismpediawiki oldschoolrunescapewiki ombrewiki onepiecewiki onkopediawiki opsdailynoticeswiki orangesamuraiwiki orbusvrwiki organismwiki ortuswiki ostravawiki ourmetawiki ourpediajpwiki ourrailwiki ourwikiwiki owanetworkwiki paperhubwiki patelinwiki pdacwiki peitwiki pelaorolangawiki penalwikiwiki perdidosenlapcwiki pfiffigertonwiki pfsolutionswiki physikzusammenfassungwiki pirateswiki pisopacowiki pk83wiki plataformastrabajocolaborativowiki plataformawiki plnonbinarywiki pmpediawiki pmresourceswiki poleteawiki popmundowiki positiveparentingwiki printmakingbewiki printzillawiki priyowiki productarchivewiki programacionetcwiki programmingwiki projectstatuswiki proniateempirewiki proyectosdeaccionwiki pseudoterapiaswiki psychwiki pwscmwwiki pythiawiki pythonminecraftwiki qualityboxwiki quanticwiki qweelylookwiki r2b2wiki radviserwiki realmgrinderwiki rensing1wiki repairhelpwiki reproduceitwiki reptilehusbandrywiki republicoframulwiki researchafricawiki ressourceswiki retkiwiki reviworkwiki rfdswiki rigpapariswiki robloxwiki rockdomwiki rogesgrandiwiki roslwiki roxannewiki rpfolderwiki rptshelpwiki rtwiki rwhdwiki rwsaleswiki sadpepswiki safespacewiki sainsterbukawiki savetawiki sbffwiki scgmarchivewiki sciencelandwiki servicesnetwiki servinetwiki southparkfanwiki spacewikiwiki wrlwiki
• 01:29 paladox: delete wiki's cekbwiki centralboxwiki changemyorgwiki chasingtimewiki christianmusictempwiki cifdwiki claviushuntwiki clickraidwiki climatedbwiki clonedeploywiki cloudtvwiki cloudwiki coconutmanwiki cocwiki codetutswiki codewiki cointelwiki colepinosgenilwiki colonizemarswiki colossalseagullwiki commercialistiwiki complexdoomwikiawiki constancywiki costestwiki cowofgoldwiki cpdiscordeditionwiki cprarchiveswiki craftcartelwiki criptowikiwiki crisisheroinewiki cryptokittieswiki cstwiki cwwwiki d8tawiki daesocialwiki danielgalewiki daudvydwiki davislistwiki ddwikiwiki deanostoolswiki delete1wiki delpetwiki demetriowiki denemewiki despacitowiki deutschkurswiki devotedanddisgruntledwiki diascwiki didatticawiki dipinsrpskiwiki disabledpoundwiki discovereachotherwiki disstmsphdwiki dongyangwiki dotapiwiki dpartwiki drgcorsicawiki drielnoxwiki dryfusewiki duallanguagewiki dunklermondwiki dystopiawiki e501wiki earthvisionwiki eastbournetownsquarewiki eccatoniawiki eclaireurswiki economicwarewiki edisonwiki edmundwiki edustudentwiki eduwiki ekomwiki eldritchtaleswiki
• 01:20 paladox: Delete wiki's antspacewiki appl2018wiki arbkfamilysearchwiki archives19emewiki archiwumsportowewiki armsinstitutewiki arrinesnotebookwiki artbyannewiki artecresourceswiki artificercreationswiki artwikiwiki arunedbwiki asckeyswiki ashehistorywiki assovelowiki asswiki asystelecowiki atlantiwiki attackontitanwiki aureliawiki aurellewikiwiki australiawiki autismwikiwiki autodocwiki automationwiki avaecanvaswiki azurlanewikiwiki babelfilmswiki badyoutuberswiki bakufuwiki balcanopediascharlatanicawiki barbarasherwikiwiki batterysafetytechwiki bazarbdwiki bcagparchivewiki bchplswiki beetcodewiki benchaseytwiki beremagnewiki berlinowiki bermudiwiki bewellmedicalwiki bftwiki bibleblogwiki bierpongwiki biglakemnhistorywiki bigtoewiki bijiawikiwiki biografiaschilenaswiki biologywiki blgameswiki blocknetwiki bloodstainedwiki blpwiki brtz3wiki bruscatowiki bruwiki bs12wiki bugolaviawiki burropediawiki buxmontuuwiki c1mwiki caduwiki caelishchronicleswiki campbellpagewiki carmeigatwiki castapediawiki cawiki cchowiki ccwikiwiki cebuvisayasdirectorywiki
• 01:09 paladox: delete wiki's alterwikiwiki, altruniversewiki, amcdlwiki, ameciclowiki, amicsdesmuseucawiki, amicsdesmuseuwiki, amntwiki, amxwikiwiki, analogypediawiki and anchorhrgwiki
• 01:07 paladox: delete wiki's aidyllicwiki, airwiki, akancyclopediawiki, aklassewiki, alfrescowiki, alifloktawiki, alittletroutywiki and allsamemonsterswiki
• 01:06 paladox: delete wiki's adantewiki, adminbookclubwiki, adnovumwiki , aerowikiwiki, africavisionwiki, agathinonwiki, ageofwushuwiki and agropediawiki
• 01:03 paladox: deleteted wiki's absurdopediawiki, abundancewiki, acasrbijawiki
• 01:02 paladox: deleteted wiki's 2cvcupfrancewiki, 3dprinterscncwiki, 4mindswiki, aapwiki, abainnovationwiki, abarrethwiki and abeldenwiki

2019-02-18

• 20:07 paladox: running lc on mw*

2019-02-17

• 16:34 Reception123: reception@mw2:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki animatedfemalemusclewiki /home/reception/animatedmusclewomen_pages_full.xml
• 15:55 paladox: reboot test1
• 15:52 paladox: upgrade test1 to debian 9.8

2019-02-16

• 12:30 Reception123: running wikibackups (./test.sh /home/reception/allpublic.dblist /srv/mediawiki/w/maintenance/dumpBackup.php)
• 02:51 John: created tables in incidents database

2019-02-15

• 13:39 paladox: renewing m.miraheze.org cert

2019-02-14

• 06:27 Reception123: restarted logbot
• 06:24 Reception123: removed 2FA from Samwilson on Phabricator

2019-02-12

• 06:07 Reception123: renamed karrotwiki to methruwiki

2019-02-10

• 23:53 paladox: removing 2FA from Eduaddad T4090

2019-02-09

• 21:27 paladox: upgrade phabricator

2019-02-08

• 09:28 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki pippipediawiki /home/reception/creationbetheworld23s_pages_full.xml

2019-02-06

• 19:23 paladox: restarting varnish on cp4 due to "unknown backend" showing

2019-02-04

• 01:07 paladox: ran a update on the sites table for ffgxvwiki
• 00:38 paladox: running lc on mw*
• 00:22 paladox: import starwars_pages_full.xml on baobabarchiveswiki

2019-01-26

• 01:26 paladox: upgrading puppet to puppet 6.2.0 on puppet1
• 01:24 paladox: import starwars_pages_full.xml on baobabarchiveswiki
• 01:18 John: rebuilding rc on attwiki
• 01:11 paladox: creating GDPRAccount on all wiki's (miraheze account to comply with GDPR)
• 00:43 paladox: removed a user email address per email at privacy@

2019-01-25

• 20:05 Southparkfan: running sendBulkEmails.php on mw1
• 13:32 paladox: upgrade phabricator on misc4

2019-01-24

• 14:39 paladox: upgrade phabricator on misc4
• 14:09 John: removed editothersprofiles from all groups
• 03:47 paladox: ran modifyGroupPermission.php per T4046

2019-01-23

• 17:38 Southparkfan: issues a varnish ban for req.url ~ ^/robots.txt on all cache proxies
• 17:30 paladox: running generateMirahezeSitemap.php on mw1

2019-01-22

• 22:27 paladox: upgrading apt on all servers
• 12:12 John: set wiki_inactive_exempt = 1 for all wikis in $wgCreateWikiInactiveWikisWhitelist
• 12:06 John: ALTER TABLE cw_wikis ADD COLUMN wiki_inactive_exempt TINYINT NOT NULL DEFAULT '0' AFTER wiki_inactive_timestamp;

2019-01-21

• 15:14 John: falling back to Redis for session and main caching temporarily

2019-01-20

• 22:36 paladox: reboot misc2 to reduce ram
• 22:32 paladox: setting prometheus master up on misc4
• 22:30 paladox: shutting prometheus master down on misc2
• 15:17 paladox: upgrade phabricator to fix a bug on misc4
• 15:15 Southparkfan: set global local_infile=OFF; on db4
• 01:18 paladox: reboot cp5 for kernel upgrade.

2019-01-19

• 19:34 paladox: dropping "math" table from all wiki's
• 19:32 paladox: drop "math" table from metawiki, no longer exists in the math extension
• 17:24 paladox: running lc on mw[23]
• 15:29 paladox: rename wiki intpwiki to taswinwiki
• 15:19 paladox: upgrade phabricator on misc4
• 00:48 paladox: upgrading phabricator on misc4
• 00:36 paladox: fix namespaces lpawiki, mindgartwiki, ianyinwiki, gllwiki, naturepediawiki, bscwiki, stiffwiki, qboxnextwiki, craedopediawiki
• 00:35 paladox: fix namespaces hominwiki, kmtfwiki, utoowiki, indiawiki, platcomicswiki, swashbucklerwiki
• 00:35 paladox: fix namespaces gbgschulpflegschaftwiki, deutschlandburgerwiki, anarchymcwiki, ckdocswiki, cuttingroomwiki, uas5wiki
• 00:34 paladox: fix namespaces readysteadyyetiwiki, twimwikiwiki, vechicleswiki, ipuwikiwiki, soprometwiki, wintermoorwiki, gjtwiki, atlaswiki
• 00:34 paladox: fix namespaces ebmscwiki, arbreapalabrewiki, ekodengewiki, dellplanetwikiwiki, tlmwiki, fantasypediawiki

2019-01-18

• 22:56 paladox: running lc on mw*
• 16:42 paladox: running insertNamespaces.php on battleaxewiki, ff5fjfwiki, yurinikolaiwiki, fanmadetvchannelswiki, woerterbuchwiki, amcwiki, seerwiki, chronostormwiki and iriswiki
• 16:38 paladox: running insertNamespaces.php on alientoasterdogwiki, mrgshowswiki, djruddhawiki, igwikiwiki, testghrwiki
• 16:37 paladox: running insertNamespaces.php on goldpediawiki
• 16:35 paladox: insert default namespaces into 'default' from loginwiki
• 16:31 paladox: running insertNamespaces.php on uncyclopedia2wiki
• 00:04 paladox: running lc on mw*

2019-01-17

• 22:08 paladox: repool mw3
• 22:04 paladox: depool mw3
• 21:26 paladox: repool mw2
• 21:16 paladox: depool mw2
• 21:15 paladox: repool mw1
• 20:50 paladox: depool mw1
• 09:44 John: reverted my last log message change (because test1wiki shouldn't be deleted, change works though)
• 09:43 John: setting wiki_deleted to 1 for test1wiki temp
• 09:25 John: ALTER TABLE cw_wikis ADD COLUMN wiki_deleted TINYINT NOT NULL DEFAULT '0' AFTER wiki_closed_timestamp, ADD COLUMN wiki_deleted_timestamp BINARY(14) NULL AFTER wiki_deleted;

2019-01-16

• 21:40 Southparkfan: removed packages proxysql and mysql-client from test1
• 20:51 Southparkfan: install proxysql package on test1
• 20:33 Southparkfan: 'usermod -a -G ssl-cert www-data' on test1

2019-01-14

• 01:19 Southparkfan: CentralAuthUser::newFromId( 20792 )->quickInvalidateCache(); (previous redis purge didn't work)
• 01:16 Southparkfan: deleted key WANCache✌️global:centralauth-user:79eda1a42924fbb966df82627046c9fe from redis to force cache purge
• 00:23 paladox: doing a emergency restart of mariadb

2019-01-13

• 19:56 MacFan4000: deleted permission cache files for weatherwiki
• 19:52 MacFan4000: DELETE FROM mw_permissions WHERE perm_dbname='weatherwiki' AND perm_group='interface-admin'
• 19:26 paladox: running lc on mw* and test1
• 16:51 paladox: running lc on mw* and test1

2019-01-12

• 22:27 paladox: running script on misc4 to remove all files that have ttl != null
• 21:54 paladox: bin/storage optimise on misc4
• 17:39 Reception123: DELETED FROM CW_WIKIS and DROPPED cuttingroomwiki uas5wiki testdeletewiki
• 17:34 Southparkfan: dropped all database matching pattern southparkfan(11111|123|12345|[3-6])wiki (were created for CW issues)
• 17:33 paladox: reboot misc4 to see if it will erase temp files
• 17:27 paladox: uninstalling lizardfs from misc4
• 16:47 paladox: unmounting /mnt/mediawiki-static (images will start disapearing)
• 16:37 paladox: depool mw1
• 15:09 paladox: stopping phd on misc4

2019-01-11

• 23:05 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist modifyGroupPermission.php --addperms="editsitecss,editsitejs,editsitejson" sysop on mw1
• 22:17 paladox: deleting files on misc4
• 20:46 paladox: restarted puppetserver (semmed stuck on 100% cpu)
• 15:05 Southparkfan: enabled puppet on misc2 again
• 15:03 Southparkfan: deleted all redis keys matching pattern *:highlight:*
• 14:57 Southparkfan: deleted all redis keys matching pattern *:diff:*
• 14:56 paladox: 14:15:14 <+paladox> !log reboot misc2 to clean ram
• 14:55 Southparkfan: 15:50:27 <+SPF|Cloud> !log disable puppet on misc2 to resolve redis trouble
• 14:52 paladox: 14:15:14 <+paladox> !log reboot misc2 to clean ram
• 14:21 paladox: [14:15:14] <+paladox> !log reboot misc2 to clean ram
• 14:17 paladox: reboot misc2 to clean ram
• 06:53 paladox: ran modifyGroupPermission.php to add editsite* to all users in sysop group
• 06:19 paladox: adding "editinterface" to default on sysop group
• 05:37 paladox: [03:10:04] <+paladox> !log repool mw3
• 05:37 paladox: [02:50:04] <+paladox> !log repool mw2
• 05:37 paladox: [02:27:34] <+paladox> !log repooling mw1 and depool mw2 and mw3
• 05:37 paladox: [00:36:33] <+MacFan4000> !log changing default branch for @miraheze/mediawiki to REL1_32
• 05:37 paladox: [00:14:16] <+paladox> !log upgrading all wiki's to 1.32
• 05:37 paladox: [00:10:15] <+paladox> !log upgrading metawiki to 1.32
• 05:37 paladox: [00:06:41] <+paladox> !log putting all wiki's into read only - pending upgrade to 1.32
• 05:37 paladox: [23:29:43] <+paladox> !log depool mw1
• 05:36 paladox: [03:10:04] <+paladox> !log repool mw3
• 05:36 paladox: [02:50:04] <+paladox> !log repool mw2
• 05:36 paladox: [02:27:34] <+paladox> !log repooling mw1 and depool mw2 and mw3
• 05:35 paladox: [00:36:33] <+MacFan4000> !log changing default branch for @miraheze/mediawiki to REL1_32
• 05:35 paladox: [00:14:16] <+paladox> !log upgrading all wiki's to 1.32
• 05:35 paladox: [00:10:15] <+paladox> !log upgrading metawiki to 1.32
• 05:35 paladox: [00:06:41] <+paladox> !log putting all wiki's into read only - pending upgrade to 1.32
• 05:35 paladox: [23:29:43] <+paladox> !log depool mw1
• 00:06 paladox: putting all wiki's into read only - pending upgrade to 1.32

2019-01-10

• 23:29 paladox: depool mw1
• 18:39 paladox: Renaming svwiki to psl631testwiki per T3967

2019-01-09

• 01:40 Southparkfan: dropped southparkfan[2-5]wiki wikis
• 01:34 Southparkfan: executed once again
• 01:19 Southparkfan: executed previous queries again
• 01:16 Southparkfan: MariaDB [metawiki]> delete from echo_notification where notification_timestamp like '20190109%' and notification_user = 2;
• 01:16 Southparkfan: MariaDB [metawiki]> delete from echo_event where event_type = 'wiki-creation';
• 00:18 Southparkfan: testing changes in CreateWiki on mw1

2019-01-08

• 01:50 MacFan4000: deleting iwwiki

2019-01-07

• 23:46 paladox: sudo chmod -R 0770 /etc/puppetlabs/puppet/ssl-keys
• 23:45 MacFan4000: (paladox) sudo chmod 0770 /etc/puppetlabs/puppet/ssl-keys on puppet1
• 23:43 paladox: sudo chmod 0700 /etc/puppetlabs/puppet/ssl-keys on puppet1
• 22:31 MacFan4000: macfan@mw1:/$ sudo -u www-data php /srv/mediawiki/w/maintenance/dumpBackup.php --wiki tesminadventureswiki --full --output gzip:/mnt/mediawiki-static/dumps/tesminadventureswiki.xml.gz
• 12:29 MacFan4000: DELETE FROM mw_namespaces WHERE ns_dbname='xedwiki' AND ns_namespace_name='Greek_talk'; (per stewards noticeboard)
• 12:29 MacFan4000: DELETE FROM mw_namespaces WHERE ns_dbname='xedwiki' AND ns_namespace_name='Greek'; (per stewards noticeboard)
• 12:28 MacFan4000: DELETE FROM mw_namespaces WHERE ns_dbname='xedwiki' AND ns_namespace_name='English_talk'; (per stewards noticeboard)
• 12:28 MacFan4000: DELETE FROM mw_namespaces WHERE ns_dbname='xedwiki' AND ns_namespace_name='English'; (per stewards noticeboard)

2019-01-06

• 17:00 paladox: running lc on mw* and test1
• 14:48 paladox: importing mw dump on bigcartoonwiki (on mw1)
• 14:45 paladox: importing mw dump for nowchesswiki
• 14:06 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/GlobalBlocking/globalblocking.sql on mw3
• 04:09 paladox: repool mw1
• 04:04 paladox: delete from mw_namespaces where ns_dbname = 'metawiki' and ns_namespace_name = 'TestHello_talk';
• 04:04 paladox: delete from mw_namespaces where ns_dbname = 'metawiki' and ns_namespace_name = 'TestHello';
• 00:55 paladox: running lc on mw*

2019-01-05

• 17:10 paladox: depool mw1
• 01:56 paladox: upgrading phabricator on misc4

2019-01-04

• 20:44 Southparkfan: stop jobrunner on mw3
• 20:32 Southparkfan: install iftop on mw[1-3]
• 00:26 Southparkfan: ran update cw_requests set cw_user = 44876 where cw_id = 6919;

2019-01-03

• 23:48 paladox: running update set commands on db4 to fix cw_user for requests that have the wrong id.
• 23:36 paladox: update cw_requests set cw_user = 44922 where cw_dbname = 'giantesswiki'; on db4
• 23:35 paladox: update cw_requests set cw_user = 44929 where cw_dbname = 'dawsontechboothwiki'; on db4
• 23:28 paladox: ALTER TABLE cw_comments MODIFY cw_comment_user INT(10) NOT NULL; on db4
• 23:27 paladox: ALTER TABLE cw_requests MODIFY cw_user INT(10) NOT NULL; on db4
• 16:04 paladox: sudo -u www-data php scriptMira2.php --wiki=nonciclopediawiki on mw1
• 15:49 paladox: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist maint*/manageJobs.php --action=delete --type="LocalGlobalUserPageCacheUpdateJob" on mw1
• 04:26 paladox: sudo -u www-data /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist maint*/manageJobs.php --action=delete --type="LocalGlobalUserPageCacheUpdateJob" on mw1
• 03:46 paladox: creating MacFan4000 on icingaweb2
• 03:40 paladox: creating Voidwalker on icingaweb2
• 02:38 paladox: recreating namespace Guide and Guide_talk for wiki paliwikiwiki
• 02:33 paladox: delete from mw_namespaces where ns_dbname = 'paliwikiwiki' and ns_namespace_id = 3001; on db4
• 02:33 paladox: delete from mw_namespaces where ns_dbname = 'paliwikiwiki' and ns_namespace_id = 3000; on db4
• 01:15 paladox: MatomoAnalytics::addSite( 'untrashwiki' ) on mw1

2019-01-02

• 00:15 paladox: update mw_namespaces set ns_aliases = '["Image_talk"]' where ns_dbname = 'metawiki' and ns_namespace_name = 'File_talk'; on db4
• 00:14 paladox: update mw_namespaces set ns_aliases = '["Image"]' where ns_dbname = 'metawiki' and ns_namespace_name = 'File'; on db4

2019-01-01

• 21:17 paladox: repool mw1
• 21:07 paladox: delete all namespaces cdb files on mw*
• 18:00 Reception123: MariaDB [phabricator_herald]> UPDATE herald_rule SET isDisabled = '1' WHERE id = '10'; (inactive user)
• 17:59 Reception123: MariaDB [phabricator_herald]> UPDATE herald_rule SET isDisabled = '1' WHERE id = '28'; (inactive user)
• 16:23 MacFan4000: macfan@mw3:~$ sudo service jobrunner restart && sudo service jobchron restart
• 16:21 MacFan4000: running jobs everywhere in a screen on mw1
• 16:19 paladox: running scriptMira2.php on mw1 (it creates a local account on metawiki and loginwiki)
• 15:05 paladox: run UPDATE globaluser SET gu_email = 'xxx' WHERE gu_name = 'Revenant Lord'; (where xxx is a email)
• 15:01 paladox: depool mw1