BullMQ Pro Edition

When the first version of Bull was released in 2012, it simply aimed at solving a use case we had for Castmill, namely we needed queues for processing simple video and image transcoding jobs. Those where the young days of NodeJS but there was a promising package called "Kue" that should do all we needed.

After a couple of months using Kue on production we noticed that sometimes the queues just stopped processing entirely. When this happened it was not enough with restarting the servers, some Redis "fu" was needed in order to fix the issue and then re-start the servers.

This was not very optimal so I tried to find the issue in the source code of Kue, unfortunately the design used for processing the jobs was not very robust, since it did not use the recommended Redis RPOPLPUSH command that is necessary in order to guarantee robust queues. My two options were to either dig in in the Kue codebase and change it so that it used the more robust pattern, or just write a simple alternative that did what we needed and nothing else.

Thats how Bull started and the rest is kind of history now. Many years later and a huge amount of time invested in the library has resulted in two very popular libraries in the NodeJS ecosystem for queue handling, Bull and the newest typescript based, BullMQ.

I am quite proud with the feature set we provide for free in the open source versions, and that many of those features are not even available in any other Redis based queue systems, open source nor commercial.

We have accumulated quite a large backlog of features that our users have requested over the years. Some of these features are simply too complex for implementing them in our spare time.

Introducing BullMQ Pro

BullMQ Pro is the "professional" edition of BullMQ where we are investing in these features as well as providing a more private support for the companies that requires it.

I feel that the Pro edition is the best way forward, in my view the best of both open source and enterprise worlds. If you are a user that relies on Bull/BullMQ for your business I think it is reassuring to know there is a commercial team behind to help you out when you most need it, either with that critical new feature that will make your life so much easier or maybe with that bug you hit that needs to be resolved ASAP in order to not impair your product.

BullMQ Pro includes, at time of writing this post, a couple of new features that I hope you are going to find  really useful:

• Support of observables in the process functions. These enables new use cases, such as state machines and also provides a clean mechanism for cancelling running processes and a proper time to live (TTL)  functionality.
• Groups support. This one I think is huge and one feature that I have been dreaming on implementing for a long time. It is finally here. You can just assign jobs to an unlimited number of groups and the jobs will be processed in a round-robin fashion, so in practice it works as if you had a "virtual queue" per group. You can read more about groups here.
• Rate limit per group. Although the standard versions of Bull/BullMQ supports rate limiting based on group keys, the new rate limit is implemented as a core feature of the groups, so it is much more robust and efficient. Instead of being built on top of delayed jobs it actually "rate limits" the group so no CPU is used at all when a group is rate limited.

You can read more about this features in the official documentation.

Of course this is just the beginning, we have a long backlog of features we will keep implementing on BullMQ Pro. We are keeping the project open so that you can peek and comment on the issues, (github.com/taskforcesh/bullmq-pro-support).

Two new  features we are currently working on:

• Batch processing.
• Topics.

The future of Bull/BullMQ open source editions

We are fully committed to the continuous development of Bull/BullMQ, however the development will be focused mainly on improving stability and adding minor features. If you are a user of any of these two libraries and happy with the current set of features then rest assured that we will keep maintaining them, in fact we will be in a much better position of maintaining them having better financial support.

What about the price?

We are aware that BullMQ Pro may  not yet have all the bells and whistles, so we are providing a generous license where you can use the library on any projects you want (as long as they are part of the same organization), for a introductory price of 95$/mo (or 995$ yearly).

The subscriptions are handled in your Taskforce.sh account.

Something missing?

If as a company you are lacking something that you wish we could provide and that is not covered by the pro edition, please let me know, I would gladly like to discuss it in more deep with you.

Follow me on twitter if you want to be the first to know when I publish new tutorials and tips for Bull/BullMQ.

And remember, subscribing to Taskforce.sh is the greatest way to help supporting future BullMQ development!