Conversation

Notices

1. Masto.host (mastohost@mastodon.social)'s status on Friday, 08-Apr-2022 10:27:29 JST Masto.host

   The fixes for the issues with Mastodon v3.5.0 have been merged and I will be starting the upgrade now.

   This upgrade will require two restarts, so there will be two downtimes of about 15 seconds each.

   You can read here the full v3.5.0 changelog https://github.com/mastodon/mastodon/releases/tag/v3.5.0

   Any issues or questions let me know.

   • Masto.host (mastohost@mastodon.social)'s status on Friday, 08-Apr-2022 10:27:23 JST Masto.host

     in reply to

     Upgrade finished! Everyone on Masto.host is now running v3.5.0 with some extra patches.

     Again sorry for the instances that had extra downtime during this upgrade. Lesson learned.

     Still, I'm glad that all is looking good on my end now. Any issues or questions please let me know.

     Thanks ????

   • Masto.host (mastohost@mastodon.social)'s status on Friday, 08-Apr-2022 10:27:24 JST Masto.host

     in reply to

     Around 20% of instances on Masto.host were down for about 10 minutes and about a dozen instances were down for about an hour. Really sorry about that and all I can say is lesson learned and this same issue will not happen again.

     がにょ🛫 repeated this.
   • Masto.host (mastohost@mastodon.social)'s status on Friday, 08-Apr-2022 10:27:25 JST Masto.host

     in reply to

     So, at some point one of the database servers had multiple databases being updated simultaneously and it just wasn't able to handle it and crashed.

     I always run upgrades like this but its not the best solution and this was my bad. For the rest of this upgrade I will do one app server at a time to avoid this issue happening again.

   • Masto.host (mastohost@mastodon.social)'s status on Friday, 08-Apr-2022 10:27:26 JST Masto.host

     in reply to

     The way I have things setup, the app servers are independent from the database. Meaning an instance that is on the app server 1 may be on the database server 5 and another on the same app server maybe on database server 2. See https://masto.host/infrastructure/

     When I run the upgrades, I run them instance by instance but on all app servers simultaneously. v3.5.0 upgrade is very database intensive and can take several minutes on larger instances.

   • Masto.host (mastohost@mastodon.social)'s status on Friday, 08-Apr-2022 10:27:27 JST Masto.host

     in reply to

     OK, so all issues have been solved and instances are working. About half of the instances have already been updated to v3.5.0.

     The rest of the update will be going much slower and probably only tomorrow it will finish.

     There will still be another restart (15 seconds of downtime) even on instances that have already been updated, just to make sure I didn't miss any instance when I corrected this problem manually.

   • Masto.host (mastohost@mastodon.social)'s status on Friday, 08-Apr-2022 10:27:28 JST Masto.host

     in reply to

     Some instances have already been upgraded but there was a problem during the upgrade process and I had to pause it.

     One of the database crashed and cause some instances to be stuck in the middle of the upgrade. I am currently manually fixing all the ones that need fixing.

     I will update here once I finish that processs and sorry for any issues this might have caused to the ones affected.