Matrix.org

Post-mortem of the September 2 outage

2025-10-10 — matrix.org homeserver — Matthew Hodgson, Neil Johnson, Thib, SRE Team

On 2nd September 2025 the matrix.org homeserver suffered a ~24h outage.

During routine maintenance to increase disk capacity, the primary database failed, and we fell back to the secondary. In attempting to restore the original primary, we lost the secondary-turned-primary rendering matrix.org unavailable.

To recover, it was necessary to restore from S3 storage, however the restore process was lengthy due to the size of the dataset (51TB).

The matrix.org homeserver was unavailable from 2025-09-02 17:45 UTC and full service resumed at 2025-09-03 18:00 UTC. No data was lost as a result of the incident.

Post-mortem of the Matrix.org Outage

2024-04-25 — Foundation — SRE Team, Thib

The Matrix.org homeserver suffered from an outage on Monday 22 April, between 08:00 UTC and 10:00 UTC, followed by slowness for the next 2 hours.

The outage occurred during scheduled maintenance with no expected downtime. It included upgrading the base OS of the machines running Matrix.org’s deployment.