First impressions of Matrix: frequent lag spikes at upwards of 10+ seconds and now the server is just straight up dead

Please just use IRC, people ;_;


@sir IRC is kind of broken as designed; the server pretty much expects to be talking directly to the user, so there are fundamental things missing, like knowing which past request an error message is a response to, or knowing whether a message you see or sent has been truncated because it was too long (or even what the length limit is)

@kragen @sir "feature i don't want is missing" is not quite the same as "broken by design", but it is very annoying.

@kragen @lain @ciaby scaling problems are not generally protocol problems

@sir @lain @ciaby on the contrary: scaling problems are very often protocol problems, and even more often are much easier to solve by protocol changes than by implementation changes. HTTP scaled better than telnet, gopher, or FTP for protocol reasons; NNTP scaled better along some dimensions than HTTP for protocol reasons; HTTP/2 scales better than HTTP/1 for protocol reasons; TCP/IP scales better than NCP for protocol reasons; and I think, though I'm not sure, that the IRC protocol beats AP

@kragen @sir @lain @ciaby

the IRC protocol beats AP

I have a counter-argument: IRCv3.

@kaniini @sir @lain @ciaby Sorry, due to Mastodon's 512-byte line length limit, I truncated my comment manually; I only meant that the IRC protocol is probably more scalable (admits more scalable implementations) than ActivityPub. And actually the message size limit is one reason for this.

@kragen @sir @lain @ciaby

AP is just as scalable as IRC once the streamingInbox extension goes live. we have just been busy working on stabilization and haven't gotten to that yet.
@kragen @sir @lain @ciaby I am right: the main scalability bottleneck right now is all of this completely and utterly unnecessary cargo-culted crypto advice. streamingInbox replaces all of that with pre-authenticated channels.

@kaniini @sir @lain @ciaby Well, I was thinking also about AP's individual messages being a bit heavierweight than IRC's, and about it not having a native channel mechanism to limit the number of copies of a given message that any given server has to make. But at least it doesn't have netsplits, right?

@kragen @sir @lain @ciaby

the cryptocrap is what makes AP messages heavy. Pleroma does as little of the cryptocrap as possible.

@kaniini @sir @lain @ciaby The example messages in have 160 bytes or more of apparently mandatory per-message overhead, which is about 4× the size of an average IRC message. At the point where we hit bandwidth limits this means a 5× scalability penalty. But presumably we have to improve other bottlenecks a lot before that becomes the problem

@kragen @sir @lain @ciaby

this doesn't matter though, because bandwidth capacity has exponentially increased since IRC came about in 1988

for what we get with that 5x penalty it's completely worth it.

@kaniini @sir @lain @ciaby But I was saying that the IRC protocol seemed likely to be more scalable than AP, not that AP's improvements over IRC and other messaging protocols weren't worth those putative scalability penalties. And as you probably saw in the other thread, IRC has substantial scalability problems of its own.

@kragen @sir @lain @ciaby

IRC is not more scalable though, because you have to start working around design flaws in the protocol that AP already fixes. it's basically a wash either way.

@kaniini @sir @lain @ciaby Scalability is only one axis of design quality. It's totally possible for one design to be more scalable than another despite being worse in other ways. Happens constantly, in fact. IRC servers routinely serve 10,000 concurrent clients (and, say, 1000 concurrent active users, and about 10 k messages per second) on sub-gigabyte hardware. ØMQ and AMQP implementations can beat those message rates by an order of magnitude; Kafka by 2.

Show more
@kragen @sir @lain @ciaby

(and it does have a DAG mechanism, for deduplication, that's what sharedInbox is about. streamingInbox builds pre-authed channels on top of this DAG.)
@kaniini is there any branch wher streaminginbox has been implemented? @sir @kragen @lain @ciaby
@succfemboi @sir @kragen @lain @ciaby

not yet, but soon. we are still working out the wire protocol for it.

@kragen @kaniini @lain @ciaby
>Sorry, due to Mastodon's 512-byte line length limit, I truncated my comment manually


@kaniini @sir @lain @ciaby you must be this viral to ride this ride:


@sir @kragen @lain @ciaby
From what I've heard Matrix's terrible lack of scalability is baked into the protocol design, together with a DDoS amplification vector.

@Wolf480pl @sir @kragen @ciaby okay, but matrix is really a weird design. AP is essentially the same as email.

@lain @sir @kragen @ciaby
Until you start expanding contexts, as the specifications wants you to do. Though I guess litepub won't have such issues.

@lain @sir @kragen @ciaby
I'm guessing it doesn't take craziness, just naivety.

@kragen @sir @ciaby we'll see. But we know the current bottlenecks and they are not so hard to work around

@lain presumably someone has done load testing? I'd like to have some kind of vague order of magnitude idea of when things start to break: on a 1000MIPS machine with 1GiB, is it when statuses are flowing in at 1 Hz, 10 Hz, 100 Hz, 1 kHz, 10 kHz, 100 kHz, 1 MHz? Is it when there are 10 clients, 100 clients, 1000 clients, 10k clients, 100k clients? Does it matter how many other servers you're directly federating with, and where does that break, roughly?

@kragen @lain
>presumably someone has done load testing?


@lain @sir Well, what I mean is that it makes it impossible to fix the deficiencies of existing IRC software in a backward-compatible way on either the client or the server; my problem is not that features are missing but that you can't add them

@kragen the limit is well defined and you just seem to have encountered a bad client if it doesn't split it up/tell you it's too long.

@sir your client silently corrupting your data before sending it to the server is a solution in limited circumstances, but even so, it doesn't solve the problem, just makes it less frequent; and the institutionalized behavior when you violate the well-defined limit is to undetectably corrupt your data further, which is not reasonable behavior

@sir also, as explained in more detail in and, you can't reliably anticipate how much extra data the server network is going to tack onto your message before sending it to other clients, so the client isn't in a position to reliably solve the problem

@kragen I don't know what these people are talking about. The server tells you when your hostmask changes.

@sir that's not what a hostmask is; maybe you're thinking of a cloak?

@sir anyway, the IRC protocol has some serious deficiencies that can't be fixed in a backward-compatible way, but I'm not convinced that there's another protocol out there that's a Pareto improvement

@kragen this particular problem can be fixed in a backwards compatible way, though. What other problems do you have to complain about? Think carefully about whether or not they actually require incompatible changes before you answer

And for the record, IRC has been working well for over 30 years despite these "deficiencies", far longer than any of its competitors have come close to

@sir What is your proposal for fixing it in a backwards-compatible way?

IRC has *never* worked *well*. I've been using it since before Undernet existed (about 25 years ago) and running a server with some friends for most of that, I've written a couple of (shitty) IRC clients, and I'm hanging out with the ircII maintainer right now (ironically, on ICB rather than IRC). I could be mistaken about its problems but if so it's not due to the kind of profound ignorance you seem to be assuming :)

@kragen just send clients their hostmask so they can compute the length limit?

IRC has always worked more than well enough. I use it all day every day and pretty much never have issues. I used Matrix for all of 20 minutes and I was disconnected twice, had constant latency spikes, and couldn't join channels or start PMs without opening up a resource intensive web giga-application.

@sir That's not what a hostmask is :)

When was the last time you were on a million-user IRC network?

Yeah, it doesn't sound like Matrix is the solution we're looking for.

@kragen IRC can totally handle millions of users. Networks handle upwards of a hundred thousand every day without breaking a sweat. Some single channels have thousands of users alone. The design is quite scalable

@sir I know very smart programmers who devoted years of their lives to trying to get IRC to handle millions of users without success, though hey, maybe that's improved since then. Current networks are typically under 100K users and still have netsplits; fixing netsplits probably requires a new server-to-server protocol. Skype, by contrast, has handled tens of millions of users for a bit over a decade. I don't use it anymore because it's proprietary, but it does exist.

@kragen netsplits are pretty rare these days. I see them maybe once every 3-4 months

@sir yeah, that's definitely an improvement over 1994, and it roughly matches my experience. (I'm also on IRC all day every day.)

@sir I want to point out that, as far as I can tell, AP, Skype, and SSB don't have netsplits at all, though. (Not sure about Matrix.)

@kragen they do, they just don't call them netsplits and they don't necessarily have the same symptoms

@sir Yeah, of course any distributed system on a partitioned network will suffer a network partition. By "netsplit" I was referring to IRC's handling of that situation by irreversibly losing messages. (At least nowadays it doesn't enable randos to steal your channel — but fixing that required a backwards-incompatible protocol upgrade.)

@kragen this is still a client problem, not a protocol problem. Like I said, the limit is well defined, if your client doesn't deal with it the way you like you should ask them to change it.

Sign in to participate in the conversation
NerdCulture in the Fediverse

All friendly creatures are welcome. Be excellent to each other, live humanism, no nazis, no hate speech. Not only for nerds, but the domain is somewhat cool. ;) No bots in general! (only with prior permission)