Tech:SRE noticeboard

SRE Vacancy: Software Engineer (Developer) (MediaWiki)
Miraheze is looking for Software Engineers to join our MediaWiki Team to develop code to improve the user experience of Miraheze users, build tools that allow communities to grow, and tools that support our valuable volunteers in managing a dynamic and active global community. If you think this could be you, please do have a look at the Vacancies page which includes further information. Reception123 (talk) ( C ) 07:09, 31 January 2023 (UTC)

Cargo disabled
Due to a severe security vulnerability with Cargo which has been acknowledged upstream (Wikimedia Phabricator ticket currently private), the extension has been disabled on all wikis. We deeply apologize for the inconvenience but we hope this issue is resolved soon.

While there is absolutely no indication that the security vulnerabilities discovered in Cargo were exploited on Miraheze, out of a great abundance of caution, all user sessions have been reset. This means all users were logged out and must log back in again. As a general internet safety reminder, please do not reuse passwords between services and make sure to regularly change your password to a strong, unique one. Agent Isai Talk to me! 03:52, 5 April 2023 (UTC)

We plan on isolating Cargo to its own database going forward. Wikis that opt-in to Cargo will have their own database for Cargo data separate from the wiki database. Void has done a lot of work on this (https://github.com/miraheze/mw-config/pull/5182, https://github.com/miraheze/MirahezeMagic/pull/413, https://github.com/miraheze/puppet/commit/241009f1d77c4935621aae016ae0757bac246b83), though there are still some details left to flesh out. This will be useful if similar vulnerabilities are ever discovered again on Cargo. Unfortunately, we still do not have an ETA on when Cargo will be re-enabled. OrangeStar (talk) 11:06, 14 April 2023 (UTC)

Cloud11 and Swift issues
Due to a disk issue on cloud11 (where Swift servers are located), files are currently not displaying properly and uploads are not possible. More information will be provided when available. Reception123 (talk) ( C ) 13:15, 11 April 2023 (UTC)
 * Adding to the above, while the servers which actually hold image files are unaffected, the server that verifies that users have the proper permissions to view files and which directs traffic is down. This means that requests for images might return errors such as "Unauthorized. You do not have permission to view this file." and such. We are working to correct the issue but have no ETA at the moment for when this will be fixed. Agent Isai  Talk to me! 00:36, 12 April 2023 (UTC)
 * An update to the above. We're still working on recovering the data from the affected cloud server. We have been able to successfully access some data but we must now reinstall the cloud server to continue working on recovery. We hope to do the reinstall either today or tomorrow. Agent Isai  Talk to me! 17:36, 15 April 2023 (UTC)

This is now solved. SRE's infrastructure team reinstalled the faulty servers today, and images have come back online for almost all wikis. If you are experiencing problems related to images/files still, please create a ticket on Phabricator. OrangeStar (talk) 19:51, 16 April 2023 (UTC)