Notices by i.w admininstrative notices (polyhedral@icosahedron.website)

And we're back! AHEAD of schedule!

Lets see if everything will go smoothly forever this time.

update: scaleways console is weirdly slow slow so new time is when this toot turns 14h

when this toot is 1h old, i.w. will go down for 4 hours (estimated)

going to have to reboot in a few, should not be more than 10m downtime

And we're back!

i.w is now on a slightly more powerful machine, and volumes have been reorganized to give us plenty of space for the database to grow (and everything is on LVM now to make changes easy, when they become neccesary).

Sorry that that took a bit longer than expected - please allow for some hiccups while federation catches up.

Next: Upgrade to 2.7, after the new setup has proven stable.

timing update update: nvm that, 30min from now

Hey, everybody!

icosahedron.website will be going down for scheduled maintenance (adding a volume, moving the database) sometime in the near future!

Please wait warmly.

In other, 2.7 news: I'll upgrade when I have some time, not before friday. If anybody wants me to Not Update for a reason, please tell.

logs will stay on the main HDD, but since the main DB will be someplace else, it'll allow for more time before things break hard.
* I will make postgres monitoring yell into a place that is not The Void when things go wrong so that next time I will actually notice.
* I will make a buffer file that I can easily delete to free up some space when needed in case the disk runs full again in the future

database, freed up some space by cleaning out some temporary files and restarted the database. As it now had some space to work with, and the archival command worked again, it started copying over all the old transaction logs and eventually cleaned them out and resumed normal operations.

3) What will be done to prevent this in the future?
Various things:
* I will move the main database files to its own drive. This is going to be neccesary in the near future anyways. The transaction

extended amount of time. This, in turn, was because the command copies the logs to a computer in my home via rsync over ssh, and my ssh port forward through my NAT broke.

2) How was this resolved?
When I was made aware that i.w. wasn't working (thanks dot and colon_three), I logged into the server via ssh and began figuring out what had happened. I first fixed my NAT by rebooting the NAT device. I then took down the site entirely to make sure that the problem does not get worse, stopped the

Okay, as promised, here is the slightly longer postmortem. Sorry, no timestamps like all the Cool People, too lazy for that.

1) What happened and why?
The primary HDD that contains the postgres database and transaction log ran full because the transaction logs did not get deleted. This was because I have transaction log archiving set up, which allows for point in time recovery of the database, and the archival command (that copies the log files to an off-site location) failed for an