Manuel Astudilllo - Taskforce.sh Blog

Dividing heavy jobs using BullMQ flows in NodeJS

Divide and conquer is a well-known programming technique for attacking complex problems in an easier way. In a distributed environment it can be used to accelerate long processes by dividing them into small chunks of work and then joining the results together. BullMQ provides a powerful feature called "flows" that

Do not wait for your jobs to complete

A common pattern that arises when working with queues in general and BullMQ in particular, is the one of waiting for a job to complete. The reasoning goes, that if the job is doing something useful we need to wait for it to be complete so that we can get

Typescript with BullMQ sandboxes in NodeJS

Writing code using Typescript and BullMQ is quite straightforward most of the time since BullMQ itself is written with Typescript. However there is a special case that has kept people scratching their heads unnecessarily and that is when you want to use sandboxed workers, i.e. a worker that is

BullMQ Pro Edition

When the first version of Bull was released in 2012, it simply aimed at solving a use case we had for Castmill, namely we needed queues for processing simple video and image transcoding jobs. Those where the young days of NodeJS but there was a promising package called "Kue" that

nodejs

Implementing a robust and scalable Webhooks/Notification system in NodeJS

(c) Eric Scales. Get it at Etsy A lot of modern REST APIs need to perform asynchronous operations, instead of unreliably keeping a long living http connection to the server waiting for it to complete, it is common to instead implement a notification system using webhooks. Notifications can be used

BullMQ

Obliterate!

I never though I would even create a piece of code that "obliterates" anything, but here it is anyway! I just released a new method for Bull/BullMQ to "obliterate" your queues. The name is chosen so that it is used carefully, since this will indeed completely destroy your queue

BullMQ

Implementing a mail microservice in NodeJS with BullMQ (3/3)

Tutorial for sending HTML as PDFs attachments using nodemailer and BullMQ.

BullMQ

Implementing a mail microservice in NodeJS with BullMQ (2/3)

In this second post we are going to show you how to add rate limiting, retries after failure and delay jobs so that emails are sent in a future point in time. The code for this tutorial is available at https://github.com/taskforcesh/bullmq-mailbot branch part2. And there is

BullMQ

BullMQ Redis™ Module

I am excited to announce that I started the development of a Redis™ Module that implements BullMQ in Zig. You can access it here: https://github.com/taskforcesh/bullmq-redis I had this idea since Redis modules where announced back in version 5.0, as it provides the means to resolve

BullMQ

Implementing a mail microservice in NodeJS with BullMQ (1/3)

In this post we are going to create a small service for sending emails using BullMQ to demonstrate some core concepts. The idea is quite simple, we will have one queue where we will post jobs for sending emails and then a processor that performs the actual job of sending