In a cooperative runtime like Rust’s Tokio or Node.js, the thread does not yield until it hits an await point.
This is just because JS is single threaded. Python has the Global Interpreter Lock, which makes it effectively single threaded too. That means you don’t have to deal with true parallelism, and critical sections, semaphores etc. It’s like Ethereum: only one thing happens at a time.
But you don’t have to parse JSON without yielding. You can make anything async by just using setTimeout once in a while. Here is one such implementation: https://www.npmjs.com/package/yieldable-json
The guy may as well have said while(1) locks up Node.
Now they get into multithreaded work-stealing, and isolates. But the solution in Node is to spin up multiple processes and pass messages between them. This is approaching the Erlang actor model, and is also shared-nothing. They even say "the schedule is a single-threaded loop per core" and "all cross-core communication occurs via the messaging subsystem".
This can also be achieved in most other single-threaded languages, too. Python with asyncio, for instance.
Tina may provide a nice opinionated implementation with bounded queues and deterministic scheduling, but those are architectural choices rather than evidence that async/await itself has failed.
The title of the post is "Tokio/Rayon Trap" - those are two very well known Rust libraries.
(in case you missed it, authors mention them later and explain what they do: "Use Tokio for I/O, and send CPU-bound work to a dedicated thread pool like Rayon.")
Authors have whole section ("The Work-Stealing Myth") on Erlang.
Author's proposed solution ("Project Tina") is a new programming language written in Odin.
How on earth do you read this all and start talking about Javascript problems instead?
This is silly or just AI slop post? Because using a Go quote as an example of doing something right in the arena is laughable at best where it has the same problems, more magic, and worse observability.
The punchline seems to be something like the LMAX disruptor style which is genuinely good for some things, but if you have I/O loops like the illustration shows you can easily block that loop with some long running function.. so you have the same cognitive load as managing thread pools or async pools or disruptors..
I can hate Rayon and Tokio as much as the next guy generally I can empathize with problems they cause. But largely either it's a skill issue.
Or you are just trying to squeeze lemonade from stones, ala, they aren't meant to do what you are doing.
Tokio especially is extremely widely used for all kinds of things it doesn't work well for.
Sure I could improve it add or tune some primitives but I am honestly considering writing my own. And so should others.
I feel like we are all too bound in Rust ecosystem suddenly to Tokio and Rayon because we don't want to blame and acknowledge the libraries just don't work for what we want to use it for.
And library authors don't consider these usecases and bug important enough to actually fix it in a ergonomic way.
I think it's important to understand how we got here and a lot of it has to do with serving network requests or RPCs.
The first Web servers used CGI (Common Gateway Interface). This spawned an entire program (process) per request and had obvious overheads. This led to some optimizations (eg FastCGI, ISAPI/NSAPI) to reduce the overhead. This was the era of Perl scripts being popular.
Then came the model of having a persistent state across requests. Java servlets were a big example of this. Given the cost of exec'ing a process, this was a big deal. But then you've immediately created an environment where multiple threads were accessing the same resource and you could leak resources. There were other variants like CORBA.
Now this was abotu the time of the birth of PHP. PHP was revolutionary because it had a stateless core that allowed shared hosting environments, which were exceptionally cheap for the time (even though they had security issues). But the idea is that you avoided the threading issues of a persistent environment and didn't really have resource leaks because everything got torn down. Of course PHP had other issues. But this was a big deal because things like the initialization of a JVM class loader, for example, was relatively expensive and you had to tune Java servers around performance and STW GC pauses.
None of the above really has anything to do with programming languages other than people learned (or didn't learn) just how hard writing multithreaded code is, something that is true to this day and you absoultely want to avoid it if possible. It is incredibly difficult to get right in an era of cores, threads, different L1/L2 caches, out-of-order proccessing, branch prediction, etc etc etc. And your code may have to run on multiple architectures.
Now Go chose to get around this issue with goroutines and channels. I personally think these are a bad abstraction, particularly because buffered channels are used without understanding the impact (leading to deadlocks), you can have exploding goroutine sttacks and using unbuffered channels is a strictly inferior (IMHO) async/await abstraction.
I actually think that Facebook's Hack has basically the almost perfect async/await. The whole idea of async/await is that you get the benefits of the PHP model of being single-threaded in your own application code and you can tear down your environment when you're done. Any IO goes through async API functions.
Now how does the scheduler manage threads, exhaustion, etc in this environment? Honestly? I have no idea. It just basically works. So maybe the Tokio issue is that the scheduler itself is blocked, which seems to be the case from this article. That does seem like a flaw but a fixable one.
I get the whole colored function criticism but the reality of using Hack to serve HTTP requests is that everything is async anyway so it seems to be a non-issue in practice. You can if you really need to call call an async function from a non-async function with a blocking function but that's not best practice.
I do know that thread pools, particularly multiple thread pools, is not the answer.
Your history is all valid, but I don't think it really hits on the main motivations for how we got here.
Thread per request works perfectly fine if your application is CPU constrained.
However the observation was made, that most web applications are IO constrained, the majority of the time spent serving a web request is spent waiting for a database or downstream API.
Since most of the threads are idle waiting, your application needs many threads to optimally utilize the servers resources.
There was a perception(valid or not) that OS threads have too much memory and scheduling overhead.
Nginx came out using async io, and it could handle much more concurrent requests than apache, which used a threaded model, it sparked a lot of interest in different kinds of application managed scheduling.
It inspired initiatives like the reactive manifesto[0], which spawned tools like RXJava.
> Since most of the threads are idle waiting, your application needs many threads to optimally utilize the servers resources.
I do not think that "many" and "optimally" belong in the same sentence.
Instead of having a great number of threads that are idle waiting, it seems much more efficient to have a single thread or a small number of threads, which handle multiplex asynchronous I/O, using something like liburing on Linux or I/O completion ports on Windows, so that the threads are seldom idle, wasting resources.
That's an informative overview. Somewhere in the story was Node.js and libuv, the callback style, promises, and the popularization of the async/await paradigm. Not sure if there was a direct influence on Rust's async libraries, but I imagine it affected how some people think what an intuitive async syntax might look like.
There are some fundamental assumptions in the Rust async system:
- The program is mostly I/O bound.
- All tasks have equal priority.
If your program isn't like that, the Tokio model is a bad match to the problem.
Real time control is not like that. MMO and metaverse game programs are not like that. Most web stuff is, but that's a special case. A big special case, but a special case.
To be fair, the Rust async model itself was intentionally designed not to be prescriptive in the way you describe. You can build, and there exists, different task executors that can handle things like priority and many other execution models.
Async is just a way to describe a tree of concurrent tasks that may depend on (wait on) each other at certain points. It is mostly declarative.
Tokio has taken over as the default choice, but there's a reason why it's not part of the standard library, it is not meant to be the only choice.
Outside of Embassy in embedded, tokio is the only realistic choice though, because it is likely that any third party async crate has a dependency on it already. Yes, smol, monio, glommio etc exist, but they are marginalised (and as far as I can tell they don't really help that much with mixed IO / compute workloads).
In fact, async/await in Rust falls apart with a mixed IO / compute workload since scheduling is cooperative. As soon as you want preemption (most of the time for what I do), it is not the right choice.
Seeing how embassy (embedded async rust) handles preemption reinforces this: it uses a separate scheduler per preemption level. This works fine, but is a bit clunky. Basically you are at this point just using async to help write a state machine per preemptive thread, which can be useful for some code patterns (in particular those common in embedded, where you are often waiting for IO). But to talk between threads you are back to channels, mutexes etc.
Yes I do agree. It's not that Tokio has taken over just due to community momentum. There's a certain subtle lock-in that happens due to how the Rust type system and dependency system work together. It's a hard problem to address.
Regarding mixed IO/compute and preemptive scheduling, well that's what threads are. They are not as lightweight, but they are there and they are quite ergonomic to use in Rust.
I could even imagine an async executor that simply starts a thread per task, so that you can keep the nice syntax, but that's obviously not ideal from a performance standpoint. Tokio already dispatches tasks to a thread pool, it is not completely cooperative, it's the sensible way to do this. And there's always tokio::spawn_blocking too.
PS: Actually, it is underrated how well designed classic threading is in Rust. It was just going out of vogue when they did it, but they addressed so much of the complexity that was around for years in Java, C++ and the like. They did actually mostly achieve the holy grail of fearless concurrency and with a more ergonomic design.
I don't really know much about tokio ecosystem lock-in, but async is fundamentally an ecosystem split. In principle nothing stops you from running computationally expensive code inside an async context, it just needs to be designed to support self pre-emption. That means if you have an async JSON parser it needs to await inside long running loops or recursive functions.
How do you properly mix IO/compute (in any language)? In Rust what I’ve done in the past is have two Tokio runtimes, one for IO and one for compute. I know you can also use Rayon but the abstractions are not always flexible/convenient enough.
But in either case the boundary between the two types of async work is never easy to cross.
In Python Trio, you would have a thread pool for each type of computation (potentially a "pool" of 1 thread shared between all tasks if that works for your application). You would await trio.to_thread.run_sync(...) [1] (pass it a normal non-async function, and a token representing the thread pool). This is a pretty simple wrapper that takes care of queuing the work to the thread pool, waiting for it to complete and for a message to be passed back to the Trio thread.
It works for simple situations. It certainly preserves backpressure, although a slow CPU task can impact other users of the same thread pool.
You can use tokio::spawn_blocking to separate compute-heavy stuff, it gets handled in a separate thread pool. You shouldn’t need two tokio runtimes, that sounds problematic.
> How do you properly mix IO/compute (in any language)?
Good question, it’s a really hard problem.
In principle you are meant to use threads, the OS will automatically switch to other work while you are waiting for IO, like with async. But there are difficulties with threads too: memory overhead, switching overhead, thread limits… Although I am getting the feeling that they have a much worse reputation than they deserve. More projects should try switching to threads and actually measuring the difference.
I have a graphics program that has about eight different threads doing different things. The draw thread is high priority and does only the things that have to be done on the draw thread. The update thread, medium priority, is doing change events that affect the scene. The movement thread is doing repeated per-frame movement, computed in parallel with drawing. The asset-loading threads, low priority, are making blocking HTTPS requests to get assets from remote servers, and decompressing them, the biggest compute load. Plus there are a few other threads that do various intermittent tasks.
This works well in Rust because the language catches locking errors that cause race conditions. Doing this in C++ would be tough.
You don't really need async until you have thousands of things waiting.
Unfortunately, rather than exposing a common interface for many executors, the Rust async ecosystem went with a Tokio monoculture
Because of that, many libraries will depend on Tokio rather than working out of box with those different executors. Things like libraries that define protocols, etc.
This creates a strong disincentive for anyone to mess with other executors
Worse yet: with no common interface, some libraries will have feature flags to support a bunch of executors, which increases the maintenance load. Some still keep around support for dead executors like async_std while skipping support for newer executors like glommio
Anyway nowadays there are some abstraction crates in crates.io but it is too little, too late. Tokio already has an insurmountable lock in. The only thing that can start to reverse that is the stdlib itself to offer cross-executor APIs
I've been thinking similar thoughts recently, in that I am not sure that a function call is the best way to model asynchronicity. Io-uring in particular feels much more suited to a req/res type model, which has the benefit that you can make a single threaded state machine the core of your app - very pleasant to test and reason about.
I'd draw the analogy to RPC; it's a leaky abstraction because HTTP is fundamentally a different thing than a function call. I'd argue that event loops are a different thing as well.
I agree that a request-response model can be quite ergonomic for concurrency. I've used this pattern in quite a few Rust projects in a somewhat ad-hoc manner.
It's a bit like having a system composed of microservices (nanoservices?) that communicate via function calls.
It sounds a lot like the actor model, but I always found the classic architecture too limiting: requiring every actor to be a single-threaded message processor, instead of being able to handle requests concurrently. It's not too different from classic object-oriented design either, with singleton services.
In some project I've called my concurrent services Gods just to have a bit of fun with it :)
I am not really sure how much yet another post complaining about async/await that ends with “thread-per-core is the way to go” adds to this discussion. Granted I’m both an Erlang programmer and a big fan of Tokio and Rust’s async/await implementation in general and I think this post and many others like it betray a fundamental misunderstanding of these technologies so I am probably biased.
When I learned about async/await when it came out with .NET, they put tremendous amount into explaining that async/await is not concurrency. But that was in time when you did a training when a new version of your programming stack came out and you did not consume knowledge in 30s snippets.
Yeah, he complains about blocking being a problem then suggests thread-per-core...
Personally I don't want thread-per-core in a general purpuse runtime, I believe C# and Go both do it correctly these days by starting new threads if all existing are busy for too long.
1. It's not reasonable to expect the application layer to carefully partition its work into "I/O heavy" and "CPU heavy" parts.
2. It's not reasonable to queue up an arbitrary amount of work without back-pressure.
I haven't used Tokio much, but if it falls prey to these pitfalls, it would make me pause before adopting it.
I think there are probably ways of using Rust async that don't fall prey to these. Maybe not so much with network servers (I haven't written that many of those), but models where you are evaluating a graph and have more control over how new work is added to the system.
> but if it falls prey to these pitfalls, it would make me pause before adopting it.
This issue isn't really a Tokio concern, it's pretty straightforward to write Rust code that has back pressure mechanisms. The "it's not reasonable" in my mind implies that if someone goes and _does that_ then there's not much a library can do to restrain the developer.
I think 2 is a reasonable concern (and one which has solutions in rust async/await tokio, as FridgeSeal points out). I'm not sure about 1 though. I think having a rough idea what part of your programs are computationally expensive shouldn't be to much to ask of programmers.
The suggested alternative is interesting, I was expecting a Rust library though, but it's an Odin one. What's missing are any sort of benchmarks or proof of the claims that it will scale beyond the issues the article discuss.
<When WhatsApp pushed the Erlang BEAM virtual machine to its limits on 100+ core machines, the system choked. As detailed by Robin Morisset, idle threads trying to steal work spent all their CPU cycles fighting over the global runq_lock5>
A simple burst of memmap + soft fault with 100 or 1000 threads on a normal laptop would tell you that thread contention is real and cache locality gets destroyed. Couple that with pinned threads. You can see the latency increase by increasing thread count. Add to that the motherboard interconnect tax for numa systems. Work stealing is not the way for increasingly many workloads on modern hardware.
Recently we built Dip, our in-house ephemeral + parallel database, and we went with may coroutines + work pinning to the same thread which also nicely becomes numa aware via architecture.
Increasing threads beyond system's hardware cores/threads resulted only in marginal gains of a couple of milliseconds worth of differences on huge workload with large increase in memory (thread stack) used by the massive number of threads.
> Increasing threads beyond system's hardware cores/threads resulted only in marginal gains of a couple of milliseconds worth of differences on huge workload with large increase in memory
Careful, if you say that too loudly, the "get rid of async just spawn more threads!!!!!" people will come out of the woodwork to yell at you about how _all_ async is a lie and we should instead pretend none of it exists and just spawn more threads.
I am now trembling at the thought of warriors who will skin me alive :).
Jokes aside, there are use cases for rayon, use cases for tokio async, use cases for "may" coroutines, use cases for a custom scheduling policy, or use cases for a combination of these.
We went with "may" coroutines (with its thread pinning) + custom numa work pinning (to a may thread) due to "may's" lightweight nature and not having to have our functions colorized. We use rayon where cache locality penalty is minor compared to the millisecond latency gains due to work stealing - but this is only for an edge case.
Our log broker Monolog (akin to kafka) which sits atop Dip uses tokio async because it plays nicely with zmq while "may" doesn't.
So I am a big fan of using the right tools for the job. Each technique has its own pros and cons. Informed decisions based on use cases matter above any dogma.
Our stack consists of two consumer facing mobile apps, Slyp and SlypBusiness, which necessitated developing the underlying infrastructure over time for scaling and efficiency.
That stack includes Dip (our database substrate), Monolog (our log broker), Singularity (our KV engine), Craft (a mobile app for app development using flow charts, snippets, and a WYSIWYG designer), and Portal (a mobile app for remote container management and a HITL AI workflow).
Craft and Portal are MVP-style functional, but aren't in production yet. Everything else is being rolled out gradually.
You're more than welcome. Happy to discuss any of it in more detail. Rather than digress further in this thread, feel free to reach out privately if you'd like to continue the conversation.
The Erlang thing is different though - that's a userspace scheduler, scheduling userspace tasks onto one-thread-per-core. Classic M:N scheduling in a similar way to go.
It works extremely well at scale - you can handle 10k connections on a normal machine with very little thought, and WhatsApp has reported handling 1 million. Yes everything has some point at which it won't scale, and apparently that approach (or at least that implementation) struggles at 100 cores. That's not a normal machine.
It appears you are unfamiliar with "may" coroutines which is a userspace M:N scheduler.
Erlang phenomenon mentioned is the same and the behavior is easily reproducible on a normal laptop.
"may" defaults to the same number of OS threads as there are cores. Scale the set_pool_capacity() to 1000, 5000, or beyond for coroutine scaling. Also try set_workers() for OS threads.
Try it any which way and you see thread contention and cache locality penalty increasing latency.
I was looking forward to checking out Project Tina until I realised it was a completely different language. Classic story. Surely you can build a thread-per-core message passing concurrency framework in Rust, the language is designed to allow such alternatives.
Is Project Tina a bit like the Actor Model, but having actors pinned to cores?
And I don't understand how Tina deals better with the problem of compute-heavy tasks blocking the thread. It looks to me like it is also cooperative concurrency per core, and if one Isolate runs for a long time the other Isolates in that core will not be able to handle their messages.
I'm not 100% sure, but it looks like Tina forces your IO to be return values. You can't actually do IO inside the Isolate handlers, so your compute code runs and returns the next IO operation to run. That's what it meant by you need to be explicit about the state machine, you have to have a handler for before I make this IO, and a handler for after that IO has run.
Isolates are like synchronous state machines. During each handler invocation, an isolate processes one message, mutates its private state, and chooses its next scheduler action. If IO is needed, it returns an IO Effect describing the IO, the isolate is parked, and the eventual IO result is delivered through a later completion message. In the meantime it continues to handle messages of other isolates.
Edit: And I realized you asked about compute-heavy tasks, nevermind, it does not seem to solve that.
That does clarify some things, but still. Say you have an Isolate pinned to a core and it's doing some long compute. In the meantime some background IO finished and an Isolate in the same core needs to handle the result. That will not happen until the long compute is done.
Isn't it just like in async and any other cooperative concurrency model? At least in multi-core async, that message can be handled in a different core so it's not completely stuck. But sure the author doesn't like that work-stealing cost happening automatically outside of their control, fair enough.
This is nonsense. Tokio was built for I/O, not crunching numbers. Most programmers know to use spawn_blocking to crunch 10MB JSON, if needed.
Additionally, the proposed workload per thread model will be orders of magnitude slower than Tokio for I/O-bound workloads, which most applications are.
Funnily enough, spawn_blocking is not the right tool here. It is meant for blocking I/O, such as DNS lookups, where your platform might not give you anything better.
For genuine CPU-bound work, submitting to a Rayon worker pool is the way to go. It solved a runtime starvation issue for us at work, spawn_blocking did not work.
The reason for all this is spawn_blocking having a very large underlying thread pool, in the hundreds. That is okay if you assume work will yield those threads and mostly sleep/wait. It is not okay if the work never yields, like pure data crunching. (Go solves this by forcefully preempting loops, no such thing in Rust without a language runtime)
Our solution shape was: multi-threaded Tokio (2 threads), then give the rest of available_concurrency to a Rayon thread pool. If you grant 6 vCPU you should see a thread pool of 4, and a maximum CPU consumption of about 400%, as the Tokio threads sit mostly idle (under low load single-threaded runtime should also suffice).
You inject the thread pool using an Arc.
Then, when work comes in, just spawn Tokio tasks liberally (cheap) and submit to the thread pool. Rayon will internally queue and limit concurrency and parallelism to 4 (this is the important bit compared to spawn_blocking: no way your system can hog all 6 threads with non-yielding work and starve Tokio runtime threads).
We use one-shot channels to submit results back, they are designed for exactly this. The tx aka sender end is sync, as there is never a wait (cannot block). The rx aka receiver side is async and can be awaited normally on the async side. This is a cheap operation, similar to Go.
Optionally you can reach for semaphores to also limit I/O concurrency. You probably want to do this for more control and avoiding resource exhaustion loudly (that is, not silently accidentally peg thousands of FDs, database connections, …).
It ended up working beautifully for our purposes and relatively simply. No lifetime woes, Arc solves those. Oneshot channels just transfer ownership etc.
Perhaps this is what TFA talks about, I have not read it.
One caveat: to reach all the above conclusions and designs, we had help from some genuine Rust experts. As much as I dislike Go, it "just works" there even if one writes naive code.
Exactly. I do not know the specifics, but for example if libraries you call into liberally spawn_blocking under the expectation that it is okay, you will be in trouble.
Says it right there actually:
> It’s recommended to not set this limit too low in order to avoid hanging on operations requiring spawn_blocking.
So a total like 6 - reasonable for a web backend - would be way too low.
This reads an awful lot like the prompter had a semi-confusing time with some async, and had their favourite model write an upset blog post about it.
I don't think these systems are perfect, nor are they fit for every use-case, but some of the complaints ring a bit hollow.
> fetching a database record over the network, then immediately crunching the data. But what happens when that data crunching involves parsing a 10MB JSON payload
Mixing IO-sensitive code and blocking code causes issues, who knew? I'm not quite sure how these libraries are supposed to magically _save_ you from this?
> When these latency spikes occur, the answer is always the same: separate your runtimes.
Well yeah. "Why doesn't my daily-driver cut sick lap times around the Nurburgring?" If you want to run blocking, non-interleaved code, don't do it in an executor expecting small, interleaved, non-blocking tasks. The docs for Tokio even mention this, and provide a number of worked examples of integrating/bridging sync and async code.
> If a developer must manually partition I/O and compute, strictly police the boundaries to prevent deadlocks, and ferry data between two different runtimes with two different mental models, the async abstraction has failed.
Not necessarily. If I'm chasing a performance target, and the tool gets me 80-90% of the way there, to the point where my next task is optimising layout and caching, I call that a win. That's performance ground we'd have to address at some point if we want to go faster, so getting there easily means:
- those people who don't need to go faster because their perf requirements are met are happy
- those people who need to go further get to skip the intervening work, and go straight to these optimisations.
Edit: also wanted to make a point about the "memory blowup" the author mentions. It's entirely possible to do the _same_ load-shedding the author wants in Rust/Tokio/Monoio/etc, this is again, just a matter of building your application in a way that uses, and communicates back-pressure. I gather that they appear to want something which is a bit more opinionated and "batteries included" as far as functionality goes, which is totally fine-and-cool, but calling Tokio + Rayon _bad_ because they explicitly don't do this is a bit of a miss.
This entire post was written/prompted as a pitch for the author's own concurrency framework. These types of "current state of the world - bad, my solution - good" posts have been annoying for far longer than LLMs have been around, but the LLMified over-embellishment coupled with the vagueness makes it so much worse.
Rayon optimises for throughout, not latency. If you want to ensure a maximum latency cadence, use something like micropool: https://github.com/DouglasDwyer/micropool.
What the code actually does is reproduce the exact same problems existing solutions have and claim it solves them.
There is so much wrong with this I don't even know where to start.
Maybe here, with a restated version of the premise:
Async cooperative execution is dangerous because programmers may perform too much work between yields. Therefore, replace compiler-generated resumable tasks with manually written synchronous callbacks that have no yield points, permanently pin them to cores, prohibit dynamic load balancing, and recover from monopolization or memory faults using signals and longjmp.
EGreg | 17 hours ago
This is just because JS is single threaded. Python has the Global Interpreter Lock, which makes it effectively single threaded too. That means you don’t have to deal with true parallelism, and critical sections, semaphores etc. It’s like Ethereum: only one thing happens at a time.
But you don’t have to parse JSON without yielding. You can make anything async by just using setTimeout once in a while. Here is one such implementation: https://www.npmjs.com/package/yieldable-json
The guy may as well have said while(1) locks up Node.
Now they get into multithreaded work-stealing, and isolates. But the solution in Node is to spin up multiple processes and pass messages between them. This is approaching the Erlang actor model, and is also shared-nothing. They even say "the schedule is a single-threaded loop per core" and "all cross-core communication occurs via the messaging subsystem".
This can also be achieved in most other single-threaded languages, too. Python with asyncio, for instance.
Tina may provide a nice opinionated implementation with bounded queues and deterministic scheduling, but those are architectural choices rather than evidence that async/await itself has failed.
Node has isolates, but it's more for sandboxing.
theamk | 16 hours ago
(in case you missed it, authors mention them later and explain what they do: "Use Tokio for I/O, and send CPU-bound work to a dedicated thread pool like Rayon.")
Authors have whole section ("The Work-Stealing Myth") on Erlang.
Author's proposed solution ("Project Tina") is a new programming language written in Odin.
How on earth do you read this all and start talking about Javascript problems instead?
what | 16 hours ago
kev009 | 16 hours ago
The punchline seems to be something like the LMAX disruptor style which is genuinely good for some things, but if you have I/O loops like the illustration shows you can easily block that loop with some long running function.. so you have the same cognitive load as managing thread pools or async pools or disruptors..
herculity275 | 11 hours ago
minraws | 16 hours ago
Or you are just trying to squeeze lemonade from stones, ala, they aren't meant to do what you are doing.
Tokio especially is extremely widely used for all kinds of things it doesn't work well for.
Sure I could improve it add or tune some primitives but I am honestly considering writing my own. And so should others.
I feel like we are all too bound in Rust ecosystem suddenly to Tokio and Rayon because we don't want to blame and acknowledge the libraries just don't work for what we want to use it for.
And library authors don't consider these usecases and bug important enough to actually fix it in a ergonomic way.
jmyeet | 16 hours ago
The first Web servers used CGI (Common Gateway Interface). This spawned an entire program (process) per request and had obvious overheads. This led to some optimizations (eg FastCGI, ISAPI/NSAPI) to reduce the overhead. This was the era of Perl scripts being popular.
Then came the model of having a persistent state across requests. Java servlets were a big example of this. Given the cost of exec'ing a process, this was a big deal. But then you've immediately created an environment where multiple threads were accessing the same resource and you could leak resources. There were other variants like CORBA.
Now this was abotu the time of the birth of PHP. PHP was revolutionary because it had a stateless core that allowed shared hosting environments, which were exceptionally cheap for the time (even though they had security issues). But the idea is that you avoided the threading issues of a persistent environment and didn't really have resource leaks because everything got torn down. Of course PHP had other issues. But this was a big deal because things like the initialization of a JVM class loader, for example, was relatively expensive and you had to tune Java servers around performance and STW GC pauses.
None of the above really has anything to do with programming languages other than people learned (or didn't learn) just how hard writing multithreaded code is, something that is true to this day and you absoultely want to avoid it if possible. It is incredibly difficult to get right in an era of cores, threads, different L1/L2 caches, out-of-order proccessing, branch prediction, etc etc etc. And your code may have to run on multiple architectures.
Now Go chose to get around this issue with goroutines and channels. I personally think these are a bad abstraction, particularly because buffered channels are used without understanding the impact (leading to deadlocks), you can have exploding goroutine sttacks and using unbuffered channels is a strictly inferior (IMHO) async/await abstraction.
I actually think that Facebook's Hack has basically the almost perfect async/await. The whole idea of async/await is that you get the benefits of the PHP model of being single-threaded in your own application code and you can tear down your environment when you're done. Any IO goes through async API functions.
Now how does the scheduler manage threads, exhaustion, etc in this environment? Honestly? I have no idea. It just basically works. So maybe the Tokio issue is that the scheduler itself is blocked, which seems to be the case from this article. That does seem like a flaw but a fixable one.
I get the whole colored function criticism but the reality of using Hack to serve HTTP requests is that everything is async anyway so it seems to be a non-issue in practice. You can if you really need to call call an async function from a non-async function with a blocking function but that's not best practice.
I do know that thread pools, particularly multiple thread pools, is not the answer.
WatchDog | 15 hours ago
Thread per request works perfectly fine if your application is CPU constrained.
However the observation was made, that most web applications are IO constrained, the majority of the time spent serving a web request is spent waiting for a database or downstream API.
Since most of the threads are idle waiting, your application needs many threads to optimally utilize the servers resources.
There was a perception(valid or not) that OS threads have too much memory and scheduling overhead.
Nginx came out using async io, and it could handle much more concurrent requests than apache, which used a threaded model, it sparked a lot of interest in different kinds of application managed scheduling.
It inspired initiatives like the reactive manifesto[0], which spawned tools like RXJava.
[0]: https://reactivemanifesto.org/
adrian_b | 9 hours ago
I do not think that "many" and "optimally" belong in the same sentence.
Instead of having a great number of threads that are idle waiting, it seems much more efficient to have a single thread or a small number of threads, which handle multiplex asynchronous I/O, using something like liburing on Linux or I/O completion ports on Windows, so that the threads are seldom idle, wasting resources.
lioeters | 15 hours ago
oaiey | 15 hours ago
Animats | 16 hours ago
- The program is mostly I/O bound.
- All tasks have equal priority.
If your program isn't like that, the Tokio model is a bad match to the problem.
Real time control is not like that. MMO and metaverse game programs are not like that. Most web stuff is, but that's a special case. A big special case, but a special case.
RealityVoid | 16 hours ago
[OP] LAC-Tech | 16 hours ago
oersted | 16 hours ago
Async is just a way to describe a tree of concurrent tasks that may depend on (wait on) each other at certain points. It is mostly declarative.
Tokio has taken over as the default choice, but there's a reason why it's not part of the standard library, it is not meant to be the only choice.
VorpalWay | 15 hours ago
In fact, async/await in Rust falls apart with a mixed IO / compute workload since scheduling is cooperative. As soon as you want preemption (most of the time for what I do), it is not the right choice.
Seeing how embassy (embedded async rust) handles preemption reinforces this: it uses a separate scheduler per preemption level. This works fine, but is a bit clunky. Basically you are at this point just using async to help write a state machine per preemptive thread, which can be useful for some code patterns (in particular those common in embedded, where you are often waiting for IO). But to talk between threads you are back to channels, mutexes etc.
oersted | 15 hours ago
Regarding mixed IO/compute and preemptive scheduling, well that's what threads are. They are not as lightweight, but they are there and they are quite ergonomic to use in Rust.
I could even imagine an async executor that simply starts a thread per task, so that you can keep the nice syntax, but that's obviously not ideal from a performance standpoint. Tokio already dispatches tasks to a thread pool, it is not completely cooperative, it's the sensible way to do this. And there's always tokio::spawn_blocking too.
PS: Actually, it is underrated how well designed classic threading is in Rust. It was just going out of vogue when they did it, but they addressed so much of the complexity that was around for years in Java, C++ and the like. They did actually mostly achieve the holy grail of fearless concurrency and with a more ergonomic design.
imtringued | 12 hours ago
random17 | 13 hours ago
But in either case the boundary between the two types of async work is never easy to cross.
quietbritishjim | 10 hours ago
It works for simple situations. It certainly preserves backpressure, although a slow CPU task can impact other users of the same thread pool.
[1] https://trio.readthedocs.io/en/stable/reference-core.html#tr...
oersted | 8 hours ago
> How do you properly mix IO/compute (in any language)?
Good question, it’s a really hard problem.
In principle you are meant to use threads, the OS will automatically switch to other work while you are waiting for IO, like with async. But there are difficulties with threads too: memory overhead, switching overhead, thread limits… Although I am getting the feeling that they have a much worse reputation than they deserve. More projects should try switching to threads and actually measuring the difference.
Animats | 2 hours ago
I have a graphics program that has about eight different threads doing different things. The draw thread is high priority and does only the things that have to be done on the draw thread. The update thread, medium priority, is doing change events that affect the scene. The movement thread is doing repeated per-frame movement, computed in parallel with drawing. The asset-loading threads, low priority, are making blocking HTTPS requests to get assets from remote servers, and decompressing them, the biggest compute load. Plus there are a few other threads that do various intermittent tasks.
This works well in Rust because the language catches locking errors that cause race conditions. Doing this in C++ would be tough.
You don't really need async until you have thousands of things waiting.
nextaccountic | 2 hours ago
Because of that, many libraries will depend on Tokio rather than working out of box with those different executors. Things like libraries that define protocols, etc.
This creates a strong disincentive for anyone to mess with other executors
Worse yet: with no common interface, some libraries will have feature flags to support a bunch of executors, which increases the maintenance load. Some still keep around support for dead executors like async_std while skipping support for newer executors like glommio
Anyway nowadays there are some abstraction crates in crates.io but it is too little, too late. Tokio already has an insurmountable lock in. The only thing that can start to reverse that is the stdlib itself to offer cross-executor APIs
[OP] LAC-Tech | 16 hours ago
I'd draw the analogy to RPC; it's a leaky abstraction because HTTP is fundamentally a different thing than a function call. I'd argue that event loops are a different thing as well.
oersted | 16 hours ago
It's a bit like having a system composed of microservices (nanoservices?) that communicate via function calls.
It sounds a lot like the actor model, but I always found the classic architecture too limiting: requiring every actor to be a single-threaded message processor, instead of being able to handle requests concurrently. It's not too different from classic object-oriented design either, with singleton services.
In some project I've called my concurrent services Gods just to have a bit of fun with it :)
dkh | 16 hours ago
jolux | 16 hours ago
oaiey | 15 hours ago
tcfhgj | 7 hours ago
silon42 | 8 hours ago
Personally I don't want thread-per-core in a general purpuse runtime, I believe C# and Go both do it correctly these days by starting new threads if all existing are busy for too long.
haberman | 15 hours ago
1. It's not reasonable to expect the application layer to carefully partition its work into "I/O heavy" and "CPU heavy" parts.
2. It's not reasonable to queue up an arbitrary amount of work without back-pressure.
I haven't used Tokio much, but if it falls prey to these pitfalls, it would make me pause before adopting it.
I think there are probably ways of using Rust async that don't fall prey to these. Maybe not so much with network servers (I haven't written that many of those), but models where you are evaluating a graph and have more control over how new work is added to the system.
FridgeSeal | 15 hours ago
This issue isn't really a Tokio concern, it's pretty straightforward to write Rust code that has back pressure mechanisms. The "it's not reasonable" in my mind implies that if someone goes and _does that_ then there's not much a library can do to restrain the developer.
lunar_mycroft | 14 hours ago
pmbanugo | an hour ago
didibus | 15 hours ago
elendilm | 15 hours ago
A simple burst of memmap + soft fault with 100 or 1000 threads on a normal laptop would tell you that thread contention is real and cache locality gets destroyed. Couple that with pinned threads. You can see the latency increase by increasing thread count. Add to that the motherboard interconnect tax for numa systems. Work stealing is not the way for increasingly many workloads on modern hardware.
Recently we built Dip, our in-house ephemeral + parallel database, and we went with may coroutines + work pinning to the same thread which also nicely becomes numa aware via architecture.
Increasing threads beyond system's hardware cores/threads resulted only in marginal gains of a couple of milliseconds worth of differences on huge workload with large increase in memory (thread stack) used by the massive number of threads.
FridgeSeal | 15 hours ago
Careful, if you say that too loudly, the "get rid of async just spawn more threads!!!!!" people will come out of the woodwork to yell at you about how _all_ async is a lie and we should instead pretend none of it exists and just spawn more threads.
elendilm | 13 hours ago
Jokes aside, there are use cases for rayon, use cases for tokio async, use cases for "may" coroutines, use cases for a custom scheduling policy, or use cases for a combination of these.
We went with "may" coroutines (with its thread pinning) + custom numa work pinning (to a may thread) due to "may's" lightweight nature and not having to have our functions colorized. We use rayon where cache locality penalty is minor compared to the millisecond latency gains due to work stealing - but this is only for an edge case.
Our log broker Monolog (akin to kafka) which sits atop Dip uses tokio async because it plays nicely with zmq while "may" doesn't.
So I am a big fan of using the right tools for the job. Each technique has its own pros and cons. Informed decisions based on use cases matter above any dogma.
FridgeSeal | 10 hours ago
May (haha) I ask what this is in service of? I’m somewhat a fan of the thread-per-core model, so I’m curious as to what you’re doing with it.
elendilm | 8 hours ago
That stack includes Dip (our database substrate), Monolog (our log broker), Singularity (our KV engine), Craft (a mobile app for app development using flow charts, snippets, and a WYSIWYG designer), and Portal (a mobile app for remote container management and a HITL AI workflow).
Craft and Portal are MVP-style functional, but aren't in production yet. Everything else is being rolled out gradually.
You're more than welcome. Happy to discuss any of it in more detail. Rather than digress further in this thread, feel free to reach out privately if you'd like to continue the conversation.
rkangel | 11 hours ago
It works extremely well at scale - you can handle 10k connections on a normal machine with very little thought, and WhatsApp has reported handling 1 million. Yes everything has some point at which it won't scale, and apparently that approach (or at least that implementation) struggles at 100 cores. That's not a normal machine.
elendilm | 11 hours ago
Erlang phenomenon mentioned is the same and the behavior is easily reproducible on a normal laptop.
"may" defaults to the same number of OS threads as there are cores. Scale the set_pool_capacity() to 1000, 5000, or beyond for coroutine scaling. Also try set_workers() for OS threads.
Try it any which way and you see thread contention and cache locality penalty increasing latency.
oersted | 15 hours ago
Is Project Tina a bit like the Actor Model, but having actors pinned to cores?
And I don't understand how Tina deals better with the problem of compute-heavy tasks blocking the thread. It looks to me like it is also cooperative concurrency per core, and if one Isolate runs for a long time the other Isolates in that core will not be able to handle their messages.
didibus | 15 hours ago
Isolates are like synchronous state machines. During each handler invocation, an isolate processes one message, mutates its private state, and chooses its next scheduler action. If IO is needed, it returns an IO Effect describing the IO, the isolate is parked, and the eventual IO result is delivered through a later completion message. In the meantime it continues to handle messages of other isolates.
Edit: And I realized you asked about compute-heavy tasks, nevermind, it does not seem to solve that.
oersted | 15 hours ago
Isn't it just like in async and any other cooperative concurrency model? At least in multi-core async, that message can be handled in a different core so it's not completely stuck. But sure the author doesn't like that work-stealing cost happening automatically outside of their control, fair enough.
levkk | 15 hours ago
Additionally, the proposed workload per thread model will be orders of magnitude slower than Tokio for I/O-bound workloads, which most applications are.
diarrhea | 14 hours ago
For genuine CPU-bound work, submitting to a Rayon worker pool is the way to go. It solved a runtime starvation issue for us at work, spawn_blocking did not work.
The reason for all this is spawn_blocking having a very large underlying thread pool, in the hundreds. That is okay if you assume work will yield those threads and mostly sleep/wait. It is not okay if the work never yields, like pure data crunching. (Go solves this by forcefully preempting loops, no such thing in Rust without a language runtime)
Our solution shape was: multi-threaded Tokio (2 threads), then give the rest of available_concurrency to a Rayon thread pool. If you grant 6 vCPU you should see a thread pool of 4, and a maximum CPU consumption of about 400%, as the Tokio threads sit mostly idle (under low load single-threaded runtime should also suffice).
You inject the thread pool using an Arc.
Then, when work comes in, just spawn Tokio tasks liberally (cheap) and submit to the thread pool. Rayon will internally queue and limit concurrency and parallelism to 4 (this is the important bit compared to spawn_blocking: no way your system can hog all 6 threads with non-yielding work and starve Tokio runtime threads).
We use one-shot channels to submit results back, they are designed for exactly this. The tx aka sender end is sync, as there is never a wait (cannot block). The rx aka receiver side is async and can be awaited normally on the async side. This is a cheap operation, similar to Go.
Optionally you can reach for semaphores to also limit I/O concurrency. You probably want to do this for more control and avoiding resource exhaustion loudly (that is, not silently accidentally peg thousands of FDs, database connections, …).
It ended up working beautifully for our purposes and relatively simply. No lifetime woes, Arc solves those. Oneshot channels just transfer ownership etc.
Perhaps this is what TFA talks about, I have not read it.
One caveat: to reach all the above conclusions and designs, we had help from some genuine Rust experts. As much as I dislike Go, it "just works" there even if one writes naive code.
veber-alex | 9 hours ago
tcfhgj | 9 hours ago
diarrhea | 6 hours ago
Says it right there actually:
> It’s recommended to not set this limit too low in order to avoid hanging on operations requiring spawn_blocking.
So a total like 6 - reasonable for a web backend - would be way too low.
FridgeSeal | 15 hours ago
I don't think these systems are perfect, nor are they fit for every use-case, but some of the complaints ring a bit hollow.
> fetching a database record over the network, then immediately crunching the data. But what happens when that data crunching involves parsing a 10MB JSON payload
Mixing IO-sensitive code and blocking code causes issues, who knew? I'm not quite sure how these libraries are supposed to magically _save_ you from this?
> When these latency spikes occur, the answer is always the same: separate your runtimes.
Well yeah. "Why doesn't my daily-driver cut sick lap times around the Nurburgring?" If you want to run blocking, non-interleaved code, don't do it in an executor expecting small, interleaved, non-blocking tasks. The docs for Tokio even mention this, and provide a number of worked examples of integrating/bridging sync and async code.
> If a developer must manually partition I/O and compute, strictly police the boundaries to prevent deadlocks, and ferry data between two different runtimes with two different mental models, the async abstraction has failed.
Not necessarily. If I'm chasing a performance target, and the tool gets me 80-90% of the way there, to the point where my next task is optimising layout and caching, I call that a win. That's performance ground we'd have to address at some point if we want to go faster, so getting there easily means: - those people who don't need to go faster because their perf requirements are met are happy - those people who need to go further get to skip the intervening work, and go straight to these optimisations.
Edit: also wanted to make a point about the "memory blowup" the author mentions. It's entirely possible to do the _same_ load-shedding the author wants in Rust/Tokio/Monoio/etc, this is again, just a matter of building your application in a way that uses, and communicates back-pressure. I gather that they appear to want something which is a bit more opinionated and "batteries included" as far as functionality goes, which is totally fine-and-cool, but calling Tokio + Rayon _bad_ because they explicitly don't do this is a bit of a miss.
herculity275 | 11 hours ago
ChrisMarshallNY | 10 hours ago
Looks like tasks mess with the reference counter. My app was not releasing memory, and was constantly jetsam-crashing.
Also, the app was quite sluggish.
Reverting to GCD, fixed both these issues.
poly2it | 10 hours ago
Here is an interesting video about it: https://youtu.be/QFQkqFSg8Z4.
frumiousirc | 9 hours ago
This is a small typo in the text. It's spelled "ZeroMQ".
pmbanugo | an hour ago
swiftcoder | 8 hours ago
projct | 17 minutes ago
What the code actually does is reproduce the exact same problems existing solutions have and claim it solves them.
There is so much wrong with this I don't even know where to start.
Maybe here, with a restated version of the premise:
Async cooperative execution is dangerous because programmers may perform too much work between yields. Therefore, replace compiler-generated resumable tasks with manually written synchronous callbacks that have no yield points, permanently pin them to cores, prohibit dynamic load balancing, and recover from monopolization or memory faults using signals and longjmp.
ugh.