Introducing search-string: an advanced search string parser

This blog post is part of the Mixmax 2017 Advent Calendar. The previous post on December 8th was about Beating Spam Detection Bots.

One of Mixmax’s core goals is to provide transparency into you and your teams' communications. The Live Feed dashboard shows latest activity on sent messages. The Insights dashboard allows you to perform analytics on that activity. And the Outbox is a hub for all of your team's planned communication.

On each of these pages, you can drill into activity that you are interested in using a powerful advanced search box at the top of the page. We provide a familiar syntax modeled after the native GMail search box syntax for running these queries. You can also see it in action in our public Snippet Search API.

One of the first major projects I had at Mixmax was to rebuild and release the new Team Outbox. This was a great first project as it cut across our technology stack and opened up data previously inaccessible to our end users: their teammates' scheduled messages. As part of this rebuild, we wanted to invest more in our advanced search box’s query parser and add some new features. Thanks to Mixmax's strong open source culture, I was encouraged to build this as an open source module from the start.

Announcing search-string

As a result, I'm excited to announce and share a new npm module: search-string. This module (now used throughout our application) is easy to get started with and packs a few advanced features. Note: the library has no dependencies and is compiled down to ES5 compatible JS so you can use it in the browser or on the server (we do both!).

Search-string parses typical Gmail-style search strings like:

to:me -from:jane@mixmax.com cheese pizza -mushrooms

and returns an instance of SearchString which can be mutated, return different data structures, or return the Gmail-style search string again.

Here is some of the functionality in action:

const SearchString = require("search-string")

// All pizzas to me NOT from jane topped with cheese and NOT mushrooms. :)
const str = 'to:me -from:jane@mixmax.com pizza topping:cheese -topping:mushrooms'

// Parse string into SearchString object.
const searchString = SearchString.parse(str);

// Add another topping choise. In this case: NOT onions.
searchString.addEntry('topping', 'onion', true /* negated */);

// Get results into an array format:
/* [{"keyword":"to","value":"me","negated":false},{"keyword":"from","value":"jane@mixmax.com","negated":true},{"keyword":"topping","value":"cheese","negated":false},{"keyword":"topping","value":"-mushrooms","negated":false},{"keyword":"topping","value":"onion","negated":true}] */
searchString.getConditionArray();

// Get results back into a human readable string. Useful for passing to APIs.
// to:me -from:jane@mixmax.com topping:cheese -topping:mushrooms,onion pizza
searchString.toString();

Runnable source at RunKit

Transformers

Something that especially excites me about search-string is how it supports “transformers”.

A search-string transformer is a function you pass in as an argument when parsing the string. It takes free text and converts it into a <key, value> pair.

This is useful at Mixmax because we infer what you are searching for by pattern matching. For example, when using the advanced search box on the Mixmax Live Feed, if you search for mixmax.com, we translate that to to:mixmax.com. Before search-string, we we used to need another round of parsing (after our initial string parsing!) to extract email-like strings from queries and assign them to default fields. Now that we have search-string transformers, we can parse out domain specific search operators during the original parsing phase. This makes for more performant and cleaner code.

Here is a transformer example that continues our pizza theme.

const SearchString = require("search-string")

// Declare a short list of known pizza toppings.
const pizzaToppings = ['cheese', 'mushrooms', 'onions', 'green pepper'];

// A simple transformer that detects toppings in the string and adds them to the 'topping' search operator.
const toppingTransformer = (text) => (pizzaToppings.indexOf(text) >= 0 && { key: 'topping', value: text });

// Perform the parsing including the transformer.
const searchString = SearchString.parse('to:me cheese mushrooms dominos', [toppingTransformer]);

// Get back the human readable parsed search string.
// 'to:me topping:cheese,mushrooms dominos'
searchString.toString();

Runnable source at RunKit

At Mixmax, we already have a collection of transformers for parsing email addresses and domains. We hope to open source those soon!

Your contributions

search-string is a small but important module at Mixmax, so we will continue to develop it and share updates with the community. We look forward to hearing your ideas on how we can improve it. Just file a pull request! :)

Love Open Source like us? We’re hiring!