NerdCulture

3 posts3 participants0 posts today

Ericka SimoneI think I’m headed over to <a href="https://mastodon.social/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> guys I’m sure I’ll lose my followers, but will you follow me again?

Esk 🐌⚡💜howdy, <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a>!over the last week or so, we've been preparing to move hachy's <a href="https://hachyderm.io/tags/DNS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DNS</a> zones from <a href="https://hachyderm.io/tags/AWS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AWS</a> route 53 to bunny DNS.since this could be a pretty scary thing -- going from one geo-DNS provider to another -- we want to make sure *before* we move that records are resolving in a reasonable way across the globe.to help us to do this, we've started a small, lightweight tool that we can deploy to a provider like bunny's magic containers to quickly get DNS resolution info from multiple geographic regions quickly. we then write this data to a backend S3 bucket, at which point we can use a tool like <a href="https://hachyderm.io/tags/duckdb" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#duckdb</a> to analyze the results and find records we need to tweak to improve performance. all *before* we make the change.then, after we've flipped the switch and while DNS is propagating -- :blobfoxscared: -- we can watch in real-time as different servers begin flipping over to the new provider.we named the tool hachyboop and it's available publicly --> <a href="https://github.com/hachyderm/hachyboop" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://github.com/hachyderm/hachyboop</a>please keep in mind that it's early in the booper's life, and there's a lot we can do, including cleaning up my hacky code. :blobfoxlaughsweat: attached is an example of a quick run across 17 regions for a few minutes. the data is spread across multiple files but duckdb makes it quite easy for us to query everything like it's one table.

Maho Pacheco 🦝🍻What a great work of moderation <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> admins do.

StoneBear :potion_genderqueer:Well, I'm back. <a href="https://hachyderm.io/tags/introduction" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#introduction</a> <a href="https://hachyderm.io/tags/genderqueer" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#genderqueer</a> :potion_genderqueer: dà-anam 🧙 :rainbowinfinity: storyteller/beat poet making their way in the greasy underbelly of the 'net. <a href="https://hachyderm.io/tags/filk" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#filk</a>, <a href="https://hachyderm.io/tags/scifi" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scifi</a>, and thinking about <a href="https://hachyderm.io/tags/aviation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#aviation</a> are pastimes. <a href="https://hachyderm.io/tags/activism" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#activism</a> , <a href="https://hachyderm.io/tags/GroenLinks" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#GroenLinks</a> , <a href="https://hachyderm.io/tags/ClimateChange" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ClimateChange</a> , <a href="https://hachyderm.io/tags/TransrightsAreHumanrights" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TransrightsAreHumanrights</a> , stomping <a href="https://hachyderm.io/tags/antisemitism" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#antisemitism</a>; <a href="https://hachyderm.io/tags/TikkunOlam" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TikkunOlam</a> means *everyone*. I am Universalist with strong Old Celtic leanings, Jewish-adjacent, Baha'i-adjacent, my bestie is Christian, and my circle is generally eclectic, including at least one athiest. None of them try to *impose* their views. I am also a former professional and current amateur journalist. Sources get cited, mistakes corrected; I invite you to do likewise. Love wins, hate gets used for a DANCE FLOOR. Can't stop the :signal: AND MY INTRO FITS IN ONE! Thanks, <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> !! It's good to be back.

Esk 🐌⚡💜howdy, folks - it's been a bit since our last <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> infra check in.stuff in motion:- ditching <a href="https://hachyderm.io/tags/terraform" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#terraform</a> cloud & tf for <a href="https://hachyderm.io/tags/opentofu" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#opentofu</a> and <a href="https://hachyderm.io/tags/atlantis" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#atlantis</a>. we are just about to import our dev environment and put it through its paces. - bringing <a href="https://hachyderm.io/tags/postgresql" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#postgresql</a> under ansible management. the team has been doing awesome work, and we've started to spin up dev nodes using the new playbooks. soon: production! - moving <a href="https://hachyderm.io/tags/DNS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DNS</a> zones away from AWS route 53. we chose bunny DNS as our provider and have been doing basic tests in dev. we'll likely prep our records for production this week with a plan for a cutover in one of the coming weekends.and if you filled out our volunteer form and haven't heard from me in a bit - you're still on the list. we'll onboard a new batch of folks in the next couple of weeks.:hachyderm: :blobfoxheartcute: <a href="https://hachyderm.io/tags/devops" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#devops</a> <a href="https://hachyderm.io/tags/sre" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#sre</a> <a href="https://hachyderm.io/tags/mastoadmin" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#mastoadmin</a>

RetrogradeJFC, what a <a href="https://oldbytes.space/tags/Hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Hachyderm</a> shitshow...

Mr. Lance E Sloan (IRL) 👤<a href="https://hachyderm.io/@hachyderm" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@hachyderm</a>, have you reviewed this case involving <a href="https://hachyderm.io/@anderseknert" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@anderseknert</a>? If not, why‽ Do you wish for <a href="https://mastodon.social/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> to become known for unreasonable judgements?

Anders EknertI didn't expect my post about Swedish Television mistakenly labeling Vladimir Putin as "president USA" yesterday to blow up the way it did. What I *really* didn't expect was <a href="https://hachyderm.io/tags/Hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Hachyderm</a>'s admins deleting my post just as it was approaching a thousand boosts. This because someone allegedly reported it as "misinformation", and that what I saw on TV "does not appear to have actually happened"... which ironically is about as Putinesque a statement can be.So how would you fact check this? A few people commenting on the post had already done that, the way people normally do — by asking for the source. And they got it. I even boosted the link to SVT Play, where the broadcast can be streamed by anyone. It's still there <a href="https://www.svtplay.se/video/jAM5B9o/aktuellt/tor-13-mar-21-00?position=135" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.svtplay.se/video/jAM5B9o/aktuellt/tor-13-mar-21-00?position=135</a>The Hachyderm "fact checking" process doesn't ask for a source though. In fact, they didn't ask me for anything. Since there were no news headlines mentioning this blunder, it must not have happened. I appreciate the lecture for me to "fact check my posts before posting", but the assumption that I'd want to post anything on this instance again could certainly need some fact checking.To the rest of you, enjoy this last screenshot of Putin definitely not getting labeled as "president USA" on SVT2 News on Swedish Television. And for those of you who can't get enough of things that never happened, I'll post a video in a comment below too.

Emelia 👸🏻The other root cause of our outage last night was a misconfiguration in how we handled requests through to mastodon-web (rails), which resulted in rack_attack believe all requests were coming from 127.0.0.1, instead of their actual IP address.Essentially our infrastructure looks like this| CDN | web server -> nginx —> nginx —> mastodon-web (rails/puma)We were correctly setting X-Real-IP, X-Forwarded-For and X-Forwarded-Proto headers on nginx at CDN edge, however when the request went through nginx on the web servers, these headers weren't passed through correctly all the time.We've since deployed changes to both our nginx configuration and our mastodon configuration (TRUSTED_PROXY_IP’s) to ensure that requests are correctly identified with the requestor's IP address, and rate limited accordingly.This should reduce the rate of “Rate Limit Exceeded" errors some people may have been seeing.Most folks running mastodon are only using a single nginx server in the request pipeline, so this is a very "us" problem, as we run our own CDN powered by nginx.<a href="https://hachyderm.io/tags/mastodon" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#mastodon</a> <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> <a href="https://hachyderm.io/@hachyinfra" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@hachyinfra</a>

Emelia 👸🏻Heads up Mastodon operators: libvips 8.16.0 doesn't work with ruby-vips 2.2.2, which is currently used by Mastodon.*** watch out if your system automatically updates libvips to 8.16.0 ***This was one of the root causes of our media processing outage last night. Essentially when we were trying to process media, the conversion between different file sizes and metadata stripping was erroring out do to an incompatibility between ruby-vips 2.2.2 and libvips 8.16.0, resulting in the error:``` Error while optimizing <name>: VipsForeignLoad: "/tmp/<name>.<ext>“ is not a known file format ```This occurred for jpg, webp, and several other file formats, despite `vips -l foreign` showing everything setup correctly.ruby-vips 2.2.3 is known to fix the issue, and will be used in mastodon 4.3.4 when it is released. Until then either don't update libvips or switch MASTODON_USE_LIBVIPS to false until mastodon 4.3.4 (unless you're comfortable modifying the Gemfile to use the newer ruby-vips version)What we still have absolutely no explanation for though is that libvips 8.16.0 was installed on hachyderm systems on January 20th 2025, but we only started seeing issues yesterday after running regular system updates wherein libvips was not touched. Ruby and gems were reinstalled yesterday due to an update of libicu, but libicu was also updated on Jan 20th 2025 so we did the same reinstall steps on that date too.We didn't receive any error reports from people using hachyderm of not being able to upload media or media failing to load for them.<a href="https://hachyderm.io/tags/mastodon" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#mastodon</a> <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> <a href="https://hachyderm.io/@hachyinfra" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@hachyinfra</a>

Emelia 👸🏻Apologies hachydermians for the outages tonight and the issue with rate limiting and images not loading. We discovered several issues that needed resolutions, and everything appears to be working fine now. There'll be a write up sometime in the future.Only negative? We've had to disable image processing with libvips, as it was failing to process media, so we're back to the imagemagik or whatever pre-libvips.<a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> <a href="https://hachyderm.io/@hachyderm" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@hachyderm</a> <a href="https://hachyderm.io/@hachyinfra" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@hachyinfra</a>

Esk 🐌⚡💜so, <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a>: funny story, turns out we *did* break image processing for a bit.we aren't entirely sure what happened, but it looks like the latest version of arch linux broke $something with libvips. this was causing mastodon to blow up on any image processing - attaching images, pulling remote images, creating thumbnails.as a temporary fix, we failed back to use image magick.additional hypothesis, but less sure about this one: we think because the image processing was failing, it was then triggering clients to re-request more frequently, which then triggered rate limiting in our nginx/mastodon-web.full write up coming soon (tm)!<a href="https://hachyderm.io/tags/devops" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#devops</a> <a href="https://hachyderm.io/tags/sre" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#sre</a> <a href="https://hachyderm.io/tags/hugops" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hugops</a> <a href="https://hachyderm.io/tags/nginx" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#nginx</a> <a href="https://hachyderm.io/tags/archlinux" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#archlinux</a> <a href="https://hachyderm.io/tags/mastoadmin" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#mastoadmin</a> <a href="https://hachyderm.io/tags/mastodon" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#mastodon</a>

Esk 🐌⚡💜good morning <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a>, don't mind me, just doing system updates. checking that i didn't break image processing or something.

slampNew emojis (logos) from <a href="https://hachyderm.io/tags/OpenInfra" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenInfra</a> foundation have been added to <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> :airship: <a href="https://airshipit.org" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://airshipit.org</a> :katacontainers: <a href="https://katacontainers.io/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://katacontainers.io/</a>:openstack: <a href="https://www.openstack.org/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.openstack.org/</a> <a href="https://hachyderm.io/tags/openstack" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#openstack</a> :starlingx: <a href="https://www.starlingx.io/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.starlingx.io/</a>:zuul: <a href="https://zuul-ci.org/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://zuul-ci.org/</a>/cc <a href="https://hachyderm.io/@kencrandall" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@kencrandall</a>

Blake GarnerTrying to sort out why image loads on for <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> have been super slow for the last week or so. Could be my IPv6 setup or a combination of that with iCloud Private relay. But is there anything else going on in hachy town?

Esk 🐌⚡💜today in <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> infra: it turns out selecting a non-US DNS zone provider isn't that straightforward. :blobfoxconfused: the dualing features right now are: flexibility in geo-based responses vs. DNSSEC.geo-routing is supported usually in one of two ways:- anycast points of presence in target regions that are loaded with geo-specific zones (dnsimple) - some kind of scripting interface (bunny? anyone else?)the folks that set up pops generally don't have them where we want them -- usually missing south america, africa, india, and asia other than tokyo/sydney.the folks with the flexible scripting interfaces don't support DNSSEC.we don't use DNSSEC today, but it seems like a generally good-ish thing -- I'm not 100% convinced, but I'm starting from a default position that we *should* enable it, and i'll want to convince myself the value there isn't there before deciding to not do it. up for thoughts/ideas!current thoughts:- dnsimple is just that: very simple, straightforward. they're also our current registrar. supports the pop-style geo routing, but no pops in sa, africa, limited asia pops - bunnydns seems very neat. the scripting platform for dns responses seems *very* powerful. big downside: no dnssec. i don't know how they'd support dnssec with the dynamic nature of scripting. - deSEC looks VERY clean, and is powered by <a href="https://hachyderm.io/tags/foss" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#foss</a> ❤️ . pops are a bit better than dnsimple, but still missing SA/africa/india and japan (which we use today). unfortunately, no geo-based responses that I can find. oh and it's also FREE (we'd donate tho)!leaning toward bunnydns but need to think through consequences of not having dnssec available. likely will write a longer-form post with moar thinking.anyway. this has been a rambly-post of inner-monologue thinking. happy hachyderming! :hachyderm: <a href="https://hachyderm.io/tags/sre" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#sre</a> <a href="https://hachyderm.io/tags/devops" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#devops</a> <a href="https://hachyderm.io/tags/dns" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#dns</a> <a href="https://hachyderm.io/tags/dnssec" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#dnssec</a>

keawadeHowdy <a href="https://hachyderm.io/tags/Hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Hachyderm</a>! I recently migrated my account here so I suppose I should probably do an <a href="https://hachyderm.io/tags/Introduction" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Introduction</a>.I am a <a href="https://hachyderm.io/tags/FullStack" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#FullStack</a> <a href="https://hachyderm.io/tags/TypeScript" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TypeScript</a> developer in the <a href="https://hachyderm.io/tags/WashingtonDC" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WashingtonDC</a> metro area.My hobbies include <a href="https://hachyderm.io/tags/VideoGames" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#VideoGames</a> ( <a href="https://hachyderm.io/tags/DarkSouls" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DarkSouls</a>, <a href="https://hachyderm.io/tags/Pokemon" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Pokemon</a>, <a href="https://hachyderm.io/tags/Factorio" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Factorio</a> ), <a href="https://hachyderm.io/tags/TTRPG" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TTRPG</a> ( <a href="https://hachyderm.io/tags/DrawSteel" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DrawSteel</a>, <a href="https://hachyderm.io/tags/DungeonsAndDragons" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DungeonsAndDragons</a> ), <a href="https://hachyderm.io/tags/MechanicalKeyboards" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#MechanicalKeyboards</a>, and <a href="https://hachyderm.io/tags/3DPrinting" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#3DPrinting</a> Other interests include <a href="https://hachyderm.io/tags/Politics" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Politics</a> and leaving the world a better place than I found it.

Esk 🐌⚡💜ok, <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> - hachyderm.io's domain registrar is now in the EU. so that's one small risk mitigated. 🎉 on to the next!

Esk 🐌⚡💜whoever designed the domainnameverification[dot]net site and emails - congratulations, you scored 100/100 on the "make a site look and feel like a phishing site" test.<a href="https://hachyderm.io/tags/dns" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#dns</a> <a href="https://hachyderm.io/tags/fosdem" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#fosdem</a> <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a>

Esk 🐌⚡💜<a href="https://hachyderm.io/@e_nomem" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@e_nomem</a> I'm up early - let the great DNS migration to the EU begin! :hachyderm: :hachyderm: :hachyderm: <a href="https://hachyderm.io/tags/hachyderm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hachyderm</a> - you shouldn't feel a thing! probably.<a href="https://hachyderm.io/tags/fosdem" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#fosdem</a> <a href="https://hachyderm.io/tags/sre" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#sre</a>

Recent searches

Search options

Administered by:

Server stats:

#hachyderm