Notices by Masto.host (mastohost@mastodon.social)

I have just published a post announcing several changes to Masto.host. You can read it here: https://masto.host/the-times-they-are-a-changin/

If you have any questions, please let me know.

The fixes for the issues with Mastodon v3.5.0 have been merged and I will be starting the upgrade now.

This upgrade will require two restarts, so there will be two downtimes of about 15 seconds each.

You can read here the full v3.5.0 changelog https://github.com/mastodon/mastodon/releases/tag/v3.5.0

Any issues or questions let me know.

Some instances have already been upgraded but there was a problem during the upgrade process and I had to pause it.

One of the database crashed and cause some instances to be stuck in the middle of the upgrade. I am currently manually fixing all the ones that need fixing.

I will update here once I finish that processs and sorry for any issues this might have caused to the ones affected.

OK, so all issues have been solved and instances are working. About half of the instances have already been updated to v3.5.0.

The rest of the update will be going much slower and probably only tomorrow it will finish.

There will still be another restart (15 seconds of downtime) even on instances that have already been updated, just to make sure I didn't miss any instance when I corrected this problem manually.

The way I have things setup, the app servers are independent from the database. Meaning an instance that is on the app server 1 may be on the database server 5 and another on the same app server maybe on database server 2. See https://masto.host/infrastructure/

When I run the upgrades, I run them instance by instance but on all app servers simultaneously. v3.5.0 upgrade is very database intensive and can take several minutes on larger instances.

So, at some point one of the database servers had multiple databases being updated simultaneously and it just wasn't able to handle it and crashed.

I always run upgrades like this but its not the best solution and this was my bad. For the rest of this upgrade I will do one app server at a time to avoid this issue happening again.

Around 20% of instances on Masto.host were down for about 10 minutes and about a dozen instances were down for about an hour. Really sorry about that and all I can say is lesson learned and this same issue will not happen again.

Upgrade finished! Everyone on Masto.host is now running v3.5.0 with some extra patches.

Again sorry for the instances that had extra downtime during this upgrade. Lesson learned.

Still, I'm glad that all is looking good on my end now. Any issues or questions please let me know.

Thanks ????

I am starting to install the latest updates to Mastodon including the ones released on v3.4.5: https://github.com/mastodon/mastodon/releases/tag/v3.4.5

During the upgrade a maximum of 30 second downtime is expected.

The full list of changes can be viewed here: https://github.com/mastodon/mastodon/compare/06631fdc53d6c464aa9db4e31d35f1fe354bcc95...54581d43e75499530b188ba4964b4ac65d1c6980

Any questions let me know.

I will be starting the upgrade to the current `main` branch of Mastodon.

There will be a downtime of about 30 seconds.

This will include all the fixes introduced on v3.4.4 yesterday https://github.com/mastodon/mastodon/releases/tag/v3.4.4 and the ones announced on the v3.4.3 upgrade: https://masto.host/the-v343-upgrade/

Any questions please let me know.

According to the OVH founder there was a human error on a configuration that brought all the service down and they are reverting the situation.

Source: https://twitter.com/olesovhcom/status/1448196879020433409

The service is coming back online.

Please let me know if you notice any issues.

Thanks and again sorry about this service disruption.

Major incident on OVH (the hosting provider I use for Masto.host).

Currently all services are down.

I will keep monitoring the situation and report when I know more.

There is currently an issue with a database server that stopped responding. I am waiting for a reply from the data center to know what could be causing this.

Some instances are currently down, I am trying to solve the issue as fast as possible and will update here when I know something.

I am migrating the email hosting for Masto.host to https://www.fastmail.com

In theory it is all working but DNS propagation/caches can cause some problems in the next hours. If you try to reach me via email and get a return message please let me know here or try again later.

Notice: this will only affect the email addressed to/from info@masto.host, all notifications sent by instances will continue to use the MaiGun SMTP.

Also, I will be updating the Privacy Policy to reflect this change.

After testing v3.4.1 for 48 hours the issue that I reported has not caused any problems.

So, I will be starting the upgrade to v3.4.1 for every server on Masto.host. The downtime during upgrade should be less than 30 seconds.

You can see the v3.4.1 release notes here: https://github.com/tootsuite/mastodon/releases/tag/v3.4.1

Any questions please let me know.

There is new release of Mastodon ( v3.4.1 - https://github.com/tootsuite/mastodon/releases/tag/v3.4.1 ) but it introduces a strange warning https://github.com/tootsuite/mastodon/issues/16353 when running the upgrade commands.

It's probably not a problem and wouldn't affect the upgrade but out of an abundance of caution, I will wait to see if someone reports any problems related to this issue or if a patch is released.

So, the upgrade to v3.4.1 will be delayed for the time being but I will announce as soon as I have more information.

Upgrade finish. All servers on Masto.host are now running Mastodon v3.4.0

Any problems please let me know. Thanks ????

I will be starting the upgrade to Mastodon v3.4.0 for all servers hosted in Masto.host

During the upgrade there will be around 1 minute of downtime.

You can find the v3.4.0 release notes here: https://github.com/tootsuite/mastodon/releases/tag/v3.4.0

Any issues or questions please let me know.

The reason to locate backups in a different data center is exactly to allow for a situation like the mentioned fire to be recovered from.

Will see if I can get a confirmation if the backup data was lost or not. The backup data was hosted on their Object Storage so hopefully it was not but can't be sure right now.

I will keep you posted when I have more information.