Using Arena with AWS ElastiCache hosted Redis

Update 7/14/17: Mixmax switched over to using Bee-Queue instead of Bull, but Arena continue to work for both.

Last week, we introduced our open-source UI for Bull queue, Arena. Today, we would like to share a piece of configuration magic that we use to automatically configure Arena without having to touch any configuration files or environment variables. This enables Arena to automatically discover new queues added in new Redis instances.

By default, Arena reads its configuration from a JSON file, which individually configures its internal mapping from Redis endpoints to Bull queues. At Mixmax, we use AWS extensively. In particular, we use ElastiCache to host our Redis instances and Elastic Beanstalk to host our Arena Bull monitoring service. Because of AWS's flexibility, all we need to do to look up our Redis endpoint URIs is add a single inline policy -- describeReplicationGroups -- to an Elastic Beanstalk role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "...",
            "Effect": "Allow",
            "Action": [
                "elasticache:DescribeReplicationGroups"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Update 7/14/17: Mixmax switched over to using Bee-Queue instead of Bull, but Arena continue to work for both.

Then, with a single AWS SDK call and some clever data mapping, we can pass an array of queues to Arena, running as a module for advanced configuration.

async function discoverQueues() {
  // Get AWS SDK output.
  const {ReplicationGroups} = await elasticache.describeReplicationGroups().promise();

  // Filter out Queue endpoints not found in the whitelist.
  // Then, build out Queue 'templates' - Arena config objects missing the 'name' key.
  const templates = _.chain(ReplicationGroups)
    .filter((group) => group.ReplicationGroupId.includes('production'))
    .map((group) => [group.ReplicationGroupId, _.first(group.NodeGroups).PrimaryEndpoint])
    .object()
    .renameAttrs('Address', 'host')
    .renameAttrs('Port', 'port')
    .mapObject((group, groupId) => _.extend(group, {'hostId': groupId}))
    .value();

  // Look up Queues from a Redis endpoint.
  // We store the names of Queues in a key called `bull-queues` on the same Redis instance hosting the Queues.
  const fetchPromises = _.chain(templates)
    .mapObject((endpoint) => getQueueNamesFromRedis(endpoint.hostId, endpoint.host, endpoint.port))
    .toArray()
    .value();
  const allHostQueues = await Promise.all(fetchPromises);

  // Use the Queue templates and fetched Queue names to construct an array to pass to Arena.
  const queues = [];
  for (let [hostId, name] of _.flatten(allHostQueues, true)) {
    const queue = _.extend({name}, templates[hostId]);
    queues.push(queue);
  }
  return queues;
}

This allows any engineer at Mixmax to add a new Bull queue for any reason, and have that queue reflected in our Arena dashboard automatically.

At Mixmax, we love to optimize for the workflows of our engineers. If you'd like to help build out our open-source tooling, drop us a line at careers@mixmax.com.

You deserve a spike in replies, meetings booked, and deals won.

Try Mixmax free

Grow revenue faster

June 22, 2017

Using Arena with AWS ElastiCache hosted Redis

You deserve a spike in replies, meetings booked, and deals won.

Recent Posts