Update 7/14/17: Mixmax switched over to using Bee-Queue instead of Bull, but Arena continue to work for both.
At Mixmax, we use the work queue library Bull to power all our distributed queue needs. It is a critical component of our infrastructure and we rely heavily on it to power core functionalities like sending emails and syncing contacts.
Despite our high reliance on Bull as a core piece of our infrastructure, up until recently we did not have an easy way to view the status of our queues. To replace an assortment of hard-to-use scripts to retrieve data from Redis, we wrote and open-sourced Arena.
Arena solves many operational problems we encountered running Bull at scale. For example, here at Mixmax, we process thousands of jobs per minute. Inevitably, some jobs fail. In the event of failure, we need to be able to quickly pinpoint and recover from these failures without putting extra stress on the queue, which was a problem we encountered with some other Bull UIs that polled for jobs, causing us high performance overhead.
We have big ideas for Arena in the future -- visualizations, live updates triggered by Bull events, and information about how Bull load balances different hosts -- to name just a few. We're excited to see Arena grow and hope the community will come along for the ride.
Next up, see how you can use the AWS SDK to auto-discover Bull queues and auto-configure Arena.
Tweet at us @MixmaxHQ and let us know what you think. And if you’d like to help scale our queueing architecture, drop us a line at careers@mixmax.com.
