I might be missing something here because I didn't have time to read it in detail, but by skimming through it, I think the root of the problem is this is not a job for a job queue. The types of jobs to be done needs to be adjusted by the capacity available. So the available worker(s) should be pulling/deciding the best thing to do at the given time and not get the work pushed to them.
Can you give a concrete example of where you've used a job queue that is selective about the work it does based on how much capacity is available?
I've primarily seen them used in two ways:
There's a fixed scheduling timer, which triggers an enqueue (e.g. hard deleting data that was soft deleted).
Enqueues from external sources (e.g. in a CI system, pushes would trigger to new job creation).
If the former and latter both don't fall under "job queue" for you, then it sounds like you're operating with a different definition than the one I'm using in the post.
I also don't understand the parent's framing. A job queue is nothing more than a way to add asynchrony to a task. There are practically infinite use cases for that, from "fire-and-forget" sending of emails, to precomputing expensive logic without blocking requests.
The author is mixing up and conflating many different things which are not job queues.
Conceptually, a job queue has no awareness of its consumers.
Batching, scheduled work / cron are separate concepts which mostly do not mix well with job queues.
90% of the job queues I have seen out there, were just glorified ways of removing heavy lifting work from HTTP request-response loops.
Because most we developers are not familiar with how to write a service/daemon, they often rely on cron or cloud based scheduling solutions. Of course this results in longer than necessary waiting times, and possible congestion at the fixed hour these things run. And in the case of batching, often times dropped work.
Just let the workers continuously pick up tasks from the queue. If the queue keeps growing, just add more workers.
Perhaps there are constraints that make using an existing system ok, but the whole time I read the article I was just thinking “roll your own!”
Really high performance job schedulers are complex, but if you’re kicking off multi-hour jobs that run once a week, you don’t have to worry much about the overhead of the queue/scheduler, and can implement it yourself. It’s honestly easier to write the logic of scheduling this job once per week than to try and shoehorn it into a system where jobs are more ephemeral/non-overlapping.
These are stored in object storage, and a new repo can be set up on a machine by downloading the reference repo, and fetching a delta of changes for the tip of the default branch.
The author should look up git bundles, which are ideal for this use case.
dpc_pw | 11 hours ago
I might be missing something here because I didn't have time to read it in detail, but by skimming through it, I think the root of the problem is this is not a job for a job queue. The types of jobs to be done needs to be adjusted by the capacity available. So the available worker(s) should be pulling/deciding the best thing to do at the given time and not get the work pushed to them.
[OP] typesanitizer | 11 hours ago
Can you give a concrete example of where you've used a job queue that is selective about the work it does based on how much capacity is available?
I've primarily seen them used in two ways:
If the former and latter both don't fall under "job queue" for you, then it sounds like you're operating with a different definition than the one I'm using in the post.
amw-zero | 5 hours ago
I also don't understand the parent's framing. A job queue is nothing more than a way to add asynchrony to a task. There are practically infinite use cases for that, from "fire-and-forget" sending of emails, to precomputing expensive logic without blocking requests.
fanf | 6 hours ago
This is a task for cron. A queue is the wrong thing which is why it’s harder than it should be.
pm | 6 hours ago
The author is mixing up and conflating many different things which are not job queues.
Conceptually, a job queue has no awareness of its consumers.
Batching, scheduled work / cron are separate concepts which mostly do not mix well with job queues.
90% of the job queues I have seen out there, were just glorified ways of removing heavy lifting work from HTTP request-response loops. Because most we developers are not familiar with how to write a service/daemon, they often rely on cron or cloud based scheduling solutions. Of course this results in longer than necessary waiting times, and possible congestion at the fixed hour these things run. And in the case of batching, often times dropped work.
Just let the workers continuously pick up tasks from the queue. If the queue keeps growing, just add more workers.
Is it tricky? or did you make it complicated?
hyperpape | 9 hours ago
Perhaps there are constraints that make using an existing system ok, but the whole time I read the article I was just thinking “roll your own!”
Really high performance job schedulers are complex, but if you’re kicking off multi-hour jobs that run once a week, you don’t have to worry much about the overhead of the queue/scheduler, and can implement it yourself. It’s honestly easier to write the logic of scheduling this job once per week than to try and shoehorn it into a system where jobs are more ephemeral/non-overlapping.
klingtnet | 8 hours ago
The author should look up git bundles, which are ideal for this use case.
alper | 10 hours ago
I get that there are lots of nuances here but from my experiences queues are really the main thing that works well in any distributed system.
Whether it's Celery, ZeroMQ, SQS, RabbitMQ, those have often been the foundations of decoupling and scalability in any architecture.