Implementing a mail microservice in NodeJS with BullMQ (1/3)

In this post we are going to create a small service for sending emails using BullMQ to demonstrate some core concepts. The idea is quite simple, we will have one queue where we will post jobs for sending emails and then a processor that performs the actual job of sending the emails.

By using BullMQ we gain some useful features:

• Separation of concerns. Keep a small service that can be used by any other services in your infrastructure.
• Scalability. We can add huge amounts of mails to the queue and they will eventually be processed by the workers, if we need more capacity just increase the amount of workers.
• Reliability. As soon as the job is enqueued we get high guarantees that the email will be sent. You can keep several workers for redundancy and if something goes wrong we can enable a retry mechanism until the email is sent.
• Extra features like delayed emails, say you want to send an email at a given point in the future instead of as soon as possible. Or rate limiting, if your email backend can only cope with a max amount of emails sent per second.

The complete source code for this tutorial is written in typescript and can be found here: https://github.com/taskforcesh/bullmq-mailbot

(There is also now a plain JS version of the tutorial here: https://github.com/igolskyi/bullmq-mailbot-js)

Things you will learn in this tutorial series:

• How to add jobs to a queue.
• How to enable rate limiting.
• How to write a sandboxed processor.
• How to use delayed jobs.

Here a diagram of what we want to achieve:

We will start by implementing the processor that will send the emails. We will use nodemailer for sending the actual emails, and in particular the AWS SES backend, although it is trivial to change it to any other vendor.

In order to run this tutorial you need the following requirements:

• Valid AWS credentials with permissions for using the SES service. You can follow this guide: https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/getting-your-credentials.html
• A Redis™ instance running somewhere, running in local is the default.

UPDATE!

Added also standard SMTP server support, so you can test easily using the gmail smtp server, start the server with:

SMTP_HOST=smtp.gmail.com SMTP_PORT=465 SMTP_AUTH_USER=youruser@gmail.com SMTP_AUTH_PASS="yourpass" yarn start

You will need to have an "app password" before the above works https://security.google.com/settings/security/apppasswords

Since we are going to use typescript for this tutorial we will define a "Mail" interface that we use as the type for our jobs data structure:

export interface Mail {
  from: string;
  to: string;
  subject: string;
  text: string;
}

Then we implement the  "sandboxed" processor, which means that every job will be processed in its own separate NodeJs process, this will help in better utilizing the CPU cores available, although for a simple process like this it will be of no big gain in practice.

import { Mail } from "./mail.interface";
import { Job } from "bullmq";
import { SES } from "aws-sdk";

import nodemailer from "nodemailer";
import config from "./config";

// Make sure you have setup your AWS credentials setup correctly
// https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/getting-your-credentials.html

const transporter = nodemailer.createTransport({
  SES: new SES({
    apiVersion: "2010-12-01",
    region: config.region,
  }),
});

export default (job: Job<Mail>) => transporter.sendMail(job.data);

As you can see the process function itself is just a call to the sendMail method from nodemailer's transporter instance. Also note that we read all the configurable variables from a config module that we share among all the files that need it:

export default {
  concurrency: parseInt(process.env.QUEUE_CONCURRENCY || "1"),
  queueName: process.env.QUEUE_NAME || "mailbot",
  connection: {
    host: process.env.REDIS_HOST,
    port: parseInt(process.env.REDIS_PORT || "6379"),
  },
  region: process.env.AWS_DEFAULT_REGION || "us-west-2",
};

Next we will create the worker:

import { Worker } from "bullmq";
import config from "./config";

export const worker = new Worker(config.queueName, __dirname + "/mail.proccessor.js", {
  connection: config.connection,
  concurrency: config.concurrency,
});

Finally we will create the entry point for the service, which is just importing the worker instance and append some listeners to it:

import { worker } from "./mail.worker";

worker.on("completed", (job) =>
  console.log(`Completed job ${job.id} successfully`)
);
worker.on("failed", (job, err) =>
  console.log(`Failed job ${job.id} with ${err}`)
);

This is all needed in order to implement the simplest email service imaginable. Now we can build and start the service and test it by just sending a dummy email:

import { Mail } from "../mail.interface";
import { Queue } from "bullmq";
import config from "../config";

const queue = new Queue<Mail>(config.queueName, {
  connection: config.connection,
});

const args = process.argv.slice(2);

console.log(args);

(async () => {
  await queue.add("send-simple", {
    from: "manast@taskforce.sh",
    subject: "This is a simple test",
    text: "An email sent using BullMQ",
    to: args[0],
  });

  console.log(`Enqueued an email sending to ${args[0]}`);
})();

If you clone the tutorial repo you can just run "yarn start" and it will start the microservice, then a call to

node dist/tests/send-simple.js destination@example.com

to send a test email.

And this is the end of the first part of this tutorial, in the next part we will add support for rate limiting and delayed emails! https://blog.taskforce.sh/implementing-a-mail-microservice-in-nodejs-with-bullmq-part-2/