Hi BullMQ team 👋
I'm using BullMQ in combination with Apache Airflow as an event-driven system. I understand that Airflow isn't natively designed for such an architecture, but it works well in practice for our use case. However, I’m facing a design challenge around job tracking and parallel executions.
Context:
- Our backend pushes jobs into a
{dags-trigger} queue to trigger Airflow DAGs.
- A separate
{dags-finished} queue is used to track the final status of each DAG.
- Each job carries a unique_id that ties it to a specific forecast process.
- My Airflow DAGs, which can run in parallel (based on the jobs in
{dags-trigger}), include a final task that needs to update the job with the matching unique_id in the {dags-finished} queue.
- Every tasks are independent and running in a separate pod.
Since multiple DAGs can be running concurrently, each final task creates a BullMQ worker that consumes jobs from {dags-finished} in FIFO order.
Problem:
How can I reliably find and update the exact job (matching unique_id) in a shared queue without blocking other jobs needed by other concurrently running DAGs?
Describe alternatives you've considered:
The workaround I’ve implemented is as follows:
- Loop over the jobs in the queue:
- If the unique_id matches, set the job’s state to completed (or failed), then stop the loop and task.
- If it doesn’t match, call
Scripts.moveToDelayed() and then Scripts.promoteJobs() to push the job back into the queue, and continue the loop.
- This allows other jobs to eventually get picked up by their corresponding DAGs later.
Caveats:
I had to add a time.sleep of ~1s when triggering the DAG to avoid race conditions, especially when two DAGs are triggered at the exact same time or within a very short delay (less than the delay of the moved job).
This problem does not occur when triggering DAGs, since those jobs are handled sequentially and don’t depend on matching parameters.
Question:
Is there a better way to find and update a specific job in a shared queue (e.g., by ID or metadata) without consuming it first via FIFO?
Have you faced
Or is there a BullMQ-native way to target a job in a queue (e.g., by jobId or unique_id) and change its state directly?
Any help or best practices would be greatly appreciated. Thanks for the great work on BullMQ! 🙏
Hi BullMQ team 👋
I'm using BullMQ in combination with Apache Airflow as an event-driven system. I understand that Airflow isn't natively designed for such an architecture, but it works well in practice for our use case. However, I’m facing a design challenge around job tracking and parallel executions.
Context:
{dags-trigger}queue to trigger Airflow DAGs.{dags-finished}queue is used to track the final status of each DAG.{dags-trigger}), include a final task that needs to update the job with the matching unique_id in the{dags-finished}queue.Since multiple DAGs can be running concurrently, each final task creates a BullMQ worker that consumes jobs from
{dags-finished}in FIFO order.Problem:
How can I reliably find and update the exact job (matching unique_id) in a shared queue without blocking other jobs needed by other concurrently running DAGs?
Describe alternatives you've considered:
The workaround I’ve implemented is as follows:
Scripts.moveToDelayed()and thenScripts.promoteJobs()to push the job back into the queue, and continue the loop.Caveats:
I had to add a time.sleep of ~1s when triggering the DAG to avoid race conditions, especially when two DAGs are triggered at the exact same time or within a very short delay (less than the delay of the moved job).
This problem does not occur when triggering DAGs, since those jobs are handled sequentially and don’t depend on matching parameters.
Question:
Is there a better way to find and update a specific job in a shared queue (e.g., by ID or metadata) without consuming it first via FIFO?
Have you faced
Or is there a BullMQ-native way to target a job in a queue (e.g., by jobId or unique_id) and change its state directly?
Any help or best practices would be greatly appreciated. Thanks for the great work on BullMQ! 🙏