I think even 3rd version of the post has no sense..

Shortly speaking, there is no way to process 8k concurrent connections with <50 Mb and with 95% success rate (something wrong. Maybe graphs were not updated properly?)

If you check popular projects like TechEmpower and the-benchmarker/web-framework in the GitHub – Vapor is not the best even in Swift category, and not comparable to Java frameworks (in terms of RPS and Latency). Not mentioning Go and Rust solutions
And as I assume Memory analysis is just showing wrong numbers.. for all languages.

Run 8k concurrent connections is heavy operation for CPU. And wrk and the server could start fight for resources. So also consider to use 2 machines for testing: one with benchmark tool and another one with the server

And I think there is no reason to use exactly wrk.. Rewrk, bombardier, goose, oha and many others.

1) There is no way to process 8k concurrent connections without **any** growth in memory – still something wrong. I don’t know where – in wrk, in script that reading memory usage.. somewhere

2) How Vapor could be close to the RPS and Latency of Helidon?
If the code (provided by the Core Vapor developer) here is wrong, and Vapor could do much better – please make a PR:
https://web-frameworks-benchmark.netlify.app/result?asc=0&f=vapor-framework,helidon-se&metric=totalRequestsPerS&order_by=level512

In TechEmpower Vapor is even not the best in Swift… and even Java solutions are just way superior over there (not even mentioning Go and Rust)
https://www.techempower.com/benchmarks/#hw=ph&test=plaintext&section=data-r22&l=v2qiv3-cn3

3) In the 3d post again we can see ~97% of succeeds for 8k concurrent connections? with <50Mb or RAM with calculating Fib numbers?
If it will be true – most of the high-load services already were on the way to rewrite their code base to Vapor :)
Which is obviously again – non realistic

4) Moreover even to send 8k concurrent connections requires a lot of CPU/RAM. If everything is running on single machine – wrk and server should start fighting for resources..
Consider to use 2 server configuration. One with benchmark tool and another with the server (connected by LAN).

5) There are plenty of options that could be used instead of "original" wrk.
Rewrk, bombardier and many others (https://github.com/denji/awesome-http-benchmark)

@pixelscience @everything Yeah, it was partly because it was such an interesting conversation and collaboration that I wanted to write about it. That and that it had a happy ending (there’s plenty of similar “why is this code weird?” threads which alas never find closure).

There’s fairly often some great threads in the Swift Forums, like that. Not always, of course, but I suppose that’s unavoidable with a public common.

Maybe my experience in corporate life has made me jaded. It feels like this could have been a circular firing squad of blame and recriminations. Instead it was just curious people trying to be helpful.

That spirit in a community is a delightful positive force.

Maybe. But that’s not what the documentation suggests (the listen man page):

If a connection request arrives with the queue full,
the client may receive an error with an indication of ECONNREFUSED.

Granted it does say “may” which is ambiguous, but it’s clear right from the name that ECONNREFUSED is the intended and appropriate errno for that situation. EBADF, which is what is actually signalled in that situation currently, is completely wrong.

The somaxconn limit results in the ECONNRESET. I’m not sure why it manifests as a different code for what is surely the exact same pattern on the wire – presumably that RST packet signalling connection denial – but it also shows that somehow the recipient kernel can distinguish between those two cases. It also knows what state the connection is in – i.e. the difference between connecting and connected – which should be enough to map to the appropriate error code, too.

(Possibly there’s some race conditions in transmission retry situations, which might make it technically impossible to always deduce the correct errno to signal. But that’s a rare edge case which doesn’t excuse the general case behaviour.)

@finestructure @everything Yep, Axel hasn’t yet done a [hopefully] final re-run to get fully corrected results. Though I don’t expect they’ll change much from his second post (it should just fix the success rate). I believe he’s intending to do a third post, for those revised results. I’ll update mine to link to it, when he does.

Gah! I kept making that mistake when I was writing this, but I thought I’d corrected all of them. Obviously not. Thanks for pointing this out, and sorry Axel!

Name typo in first section

It would be nice to see the complete updated results somewhere for reference, especially since you’ve got the outdated ones so prominently in the introduction.

Could make for a nice side by side showing the difference!