Thank you, Richard!
Would queues are more efficient if the concurrent work you are doing is CPU bound and have no order requirements?
Why not. That’s a perfect situation for using the queues. Queues are meant to manage the tasks and have control over things like the number of concurrencies. That’s beside the long list of benefits I’ve already mentioned. Whoever is executing these tasks is an independent server (worker) that has its own CPU, MEM, etc. You can scale this worker whenever you like, and shut it down when you’re done with it.
Would queues introduce more overhead based on the layers of abstraction and subsequent callbacks (I’m assuming some independent tool is being used)?
If I understood you correctly, you have a chain of callbacks that will be triggered whenever a task is carried out by the queue. If that’s the case, then there are a few things I would consider to base my decision.
Can we split these callbacks, each in its own server (worker), especially if each requires noticeable resources like CPU, MEM?.
Instead of having a long chain of callbacks, can we use the Pub/Sub approach? So, whenever you’re done with callback1, that will emit an event, save it to the event log, and whoever is listening gets a notification. In this case, the next subsequent callback will get notified and so continue the work, and so on. Event Driven Systems
Can we put all the work inside a one or two places (can be methods or another server) instead of having small?
I understand decomposition is good in the sense that you split one large piece into small pieces. But, that has to be done for a reason. Perhaps, these small pieces will be called from different parts of the application. Or, maybe, these small pieces are called only if they satisfy a certain condition. But, if no have to reason to decompose, then don’t.
The part in which threads vs queues are compared left me a little bit unsatisfied for the lack of real use cases, but this could be for the ambiguity or me not understanding correctly.
Neither of these. It is just because there are dozens of ways of doing it. The details vary, the main concept will probably stay the same. I can give you my use case, but mine won’t be the same as yours. So, what to do?
We do need to think. What do we need? What works best? We shouldn’t blindly apply the same concept over and over again just because all the people are doing it.
Of course, looking at other people use cases will help. Learning from their obstacles and how they solved them is a great asset.
That being said, If you searched online, you’ll find some decent articles. I know the internet is cluttered these days, nevertheless, still few people are posting some useful stuff. I personally like the talks from conferences like GOTO.
Hope that helps!