Sorry about the blip there folks here on #MastodonAppUK, it is unrelated to the earlier issues on #Universeodon and I think should now be resolved but I'll continue to keep an eye.

I will post some pictures this week but had a lovely week in #Edinburgh this week with @PrinceMumbles and got some time to rest and reset after what has been a really busy start to the year.

I've got quite a lot going on still though and one of the key things I'm hoping to start getting sorted is the next infrastructure migration here on #MastodonAppUK and #Universeodon where we're going to be moving away from rented dedicated servers and onto co-location hardware for the applications, I am hoping (all things going well) to be able to place the order for the kit in June and get it rack and stacked in June to start to migrate some of the work over and to ultimately start to reduce our monthly rental footprint.

It looks like there are some media issues on #Universeodon right now. I'm looking into it, not quite sure what's happened.

I'm going to shortly bring us up to date with the latest Mastodon patch release on both #Universeodon and #MastodonAppUK - I'm not expecting this to be majorly disruptive but there will be a short blip while I re-build and restart everything.

We have added freysa.ai to our #FediBlock list here and on #Universeodon following a boat load of AI Generated spam / nonsense.

We are now back to full operational standing here on #MastodonAppUK and #Universeodon with both our web nodes back up and running and hopefully things should be a bit more stable now.

I do still need to move the Universeodon relay again but I'll look to do this at some point during the week as well as some further re-balancing of things to try to get things where they need to be to keep things ticking over a bit better!

I need to take the majority of the #Universeodon content processing offline for 30-60 mins to migrate to new underlying infrastructure, during this time there will be delays on new posts showing up and our posts being federated to the wider network.

I'm spinning up an incident on https://status.atlas-media.co.uk now to track.

For the #MastodonAppUK and #Universeodon Folks -

We've seen a notable reduction in contributions over the last couple of months, but sadly our operating costs have continued to significantly increase, throughout February our media storage bill went up approx $200USD compared to the month before.

While I can (and will) continue to pick up any costs where they aren't covered by the generous contributions of the community here, if you are in a position where you'd like to contribute a bit of cash towards the running costs it would be always appreciated.

Links for contributing (Ko-Fi can do one-off or monthly, Patreon is monthly).

Ko-Fi: https://ko-fi.com/mastodonappuk
Patreon: https://www.patreon.com/universeodon

And apologies, I don't tend to post much on either platform that is "Exclusive" (There are a few bits) so it is entirely to support the running costs of these sites!

Apologies #Universeodon folks - It looks like our Sidekiq crashed again around 8 hours ago. As part of the work I need to do this weekend to get us ready for some host maintenance being performed by our provider I'm hoping we should be able to scale things back up to where they need to be.

I thought things seemed quite on the sign-up front, apologies I forgot I turned on approval checks for both #MastodonAppUK and #Universeodon as I've been swamped recently and my other mods are also pretty busy with other commitments so didn't want spam sneaking through. I'm working through the backlog now so apologies to those of you that are joining us for the first time but it's taken me a week nearly to realise!

I've just been notified that one of our production servers is going to be relocated to a different data hall by our infrastructure provider with about 2 weeks notice. Annoyingly it's the server that runs 3 of our web nodes for MastodonAppUK, Universeodon and the Universeodon relay.

I'm not expecting this to cause any public disruption, but will be spending this weekend getting things ready so I can migrate the traffic over prior to the server move happening. There may be some short notice maintenance over the coming weeks to ensure I can accommodate this and it doesn't cause an outage on move day.

We keep running into some fairly major issues across #Universeodon and #MastodonAppUK with our SideKiq content processing, I need to do some digging but it appears as if we keep getting huge spikes in traffic which is causing issues with us keeping up or was causing disruption to the stability of the site. Apologies for the issues, we are monitoring the situation.

Apologies if there's any odd behaviour here on #MastodonAppUK for a little bit, I've had to re-jig some capacity to enable our #Universeodon content processing / SideKiq jobs to try and catch up a bit better and the underlying hardware (Which is already running stretched right now) is a bit more stretched than normal. I'm not expecting anything too noticeable to happen but things could get spooky.

Anyone else on #Universeodon not seeing any federated posts newer than 7 or 8 hours?

Apologies for the blip there on #MastodonAppUK and some performance issues on both here and #Universeodon.

We ran into an issue with the MastodonApp.UK Database being under substantial load due to what I think was a glitch / unexpected behaviour within our SideKiq content processing.

Just for those interested (And so I remember when I come to write up the incident which I will try to do more of)

Monitoring first detected intermittent issues with the site at Feb 21, 2025 @ 10:32:21 PM with high latency and bad gateway errors.

At 22:34 the monitoring system detected outages from multiple global locations and had high enough confidence the site was down and I was then automatically paged, I then got paged 4 other times for the specific regional outages which confirmed it was down.

After restarting part of our database service and our content processing the site returned to normal health at 22:40:41 according to monitoring.

To resolve ongoing issues over on MastodonApp.UK I need to perform a reboot of one of our core switches which will have a short impact here on #Universeodon - Apologies for any disruption.

Now #Universeodon is bitching at me every day about logging in from a different IP Address.

Why? 'cuz I won't let the ad trackers and trolls follow me around, and apparently that upsets Universeodon.

Trying out #universeodon because mastodon.social had too much non-english language content in the feed. I like the idea of being a part of a global community, but if I can't read 80% of what's in the feed is hard to find content to engage with.

If this server is good enough for @georgetakei then it's probably good enough for me.