Last week kicked off the first in a series of “Data Salons” we are holding here at Metamarkets. The goal, as Michael Driscoll put it, is “to bring people together and talk about cool stuff, and keep it small”. This is something we had been thinking about doing for a while and thanks to the overwhelming response from everyone involved, it was a real success.

We had a great lineup of speakers for the first topic in the series: data visualization. Following our post on the rise of interactive data visualization, we decided to bring together some of the people designing visualizations as well as the people behind the frameworks used to build them, so they could share with us some of the projects they are working on and how they approach the problems they are trying to solve.

At the crossroads of art & science, design & engineering

People working in data visualization come from various different backgrounds and it is interesting to see how they embrace the engineering and design challenges involved. We see engineers becoming designers as well as designers embracing the engineering side of data visualization. At many levels it is both an art and a science, and the variety of people who attended the salon are a great example of that.

Mary Becica described how her architecture background influences how she approaches data visualization problems. She starts by putting the data into context, and letting that context inform the visual representation to give the data. Too often people start with a preconceived visual without giving extra thought to what form the data should take.

Mike Migurski gave us a glimpse into OpenStreetMap and the various projects he is working on to help create maps that go beyond a standard Google maps overlay. His objective is to design maps that are suitable to many types of overlays without distracting from or interfering with the information that is being surfaced. Again, setting the context is fundamental and the work that goes into the base map layer plays an important role in defining that context. From the subtle color palettes used to enhance topographic relief all the way to labeling heuristics, striking the right balance is very much an art, and scaling that to a wide range of resolutions and sizes requires quite a bit of engineering wizardry.

Telling a story

Whether designing an interactive book or building dynamic infographics, we ultimately want to tell a story through the data, and visualization is the key to how we expose those insights.

Bret Victor walked us through the playful interactive graphics he created for “Our Choice”,  Al Gore’s interactive book on climate change. Bret shared with us his insights on how to engage the audience and guide them through the data with limited effort, reducing the frictions to discovery and learning.

With the ability to create ever richer interactive experiences it becomes critical not to drown the user in gratuitous forms of interaction, but focus on essential mechanisms that help to understand and explore the data. This becomes even more important when designing for mobile interfaces. Mobile devices allow for richer types of interactions, but they are often less obvious to the user. It is critical to provide hints and visual cues so the user can maximize their experience of interactive graphics.

Expanding on this idea, Nick Bilton recounted some of his experiences at the New York Times R&D labs. There he was able to leverage the trove of data available at the New York Times to visualize how information spreads through various (social) networks. Through his work Nick has uncovered many interesting and surprising patterns, but sifting through such large volumes of data can be quite a challenge. Recent improvements in frameworks and visualization tools have started to make this easier.

Evolution of visualization frameworks

There is no one better to talk about these frameworks than the authors that created them. We brought together Jeff Heer, Mike Bostock and Nicolas Garcia Belmonte for a freeform panel to discuss everything from prefuse, protovis and d3 to infovis and philogl. It was a great occasion to learn about what motivated them, the evolution of the different frameworks and the type of problems they were trying to solve along the way.

Building on this line of toolkits, Vadim Ogievetsky has developed DVL, a reactive visualization framework. He demonstrated how easy it can be to build multi-faceted visualizations for high-dimensional real-time data feeds. The toolkit abstracts many of the dependencies between data and representation, without losing the flexibility of the underlying graphical framework. His work is a key element to the Metamarkets platform.

Looking forward to data salon #2

All said, this event was a very enriching experience, thanks to the impressive list of attendees and fantastic presenters. We also want to thank all of you who helped out, including Nisha Pathak for handling the logistics. We had an excellent response to this salon and are looking forward to hosting more events like this.

For more information about the presenters’ work check out their sites here:

• Bret Victor http://worrydream.com/
• Nick Bilton http://www.nickbilton.com/
• Mike Migurski http://mike.teczno.com/
• Mary Becica http://www.mbecicadesign.com/
• Jeff Heer http://vis.stanford.edu/jheer/
• Mike Bostock http://bost.ocks.org/mike/
• Nicolas Garcia Belmonte http://philogb.github.com/
• Vadim Ogievetsky https://github.com/vogievetsky/DVL

Mike Driscoll and I founded Metamarkets in 2009 to provide large scale analytics to media companies, and I’m very gratified we have reached our objectives. Over the past 18 months, Metamarkets has built and shipped some of the most scalable, cutting-edge data analytics infrastructure in the marketplace. Our engineering and product teams have designed, built and deployed from scratch a cloud-hosted integrated analytics stack on behalf of market-leading, web-scale media businesses.

As we have pushed into the market, we have seen our product scale to accommodate the biggest, fastest-moving event sets on the internet. The core value proposition for our product has been validated through our customer set: the Metamarkets stack eliminates the need for a business to integrate multiple disparate software solutions at the data ingestion, database, analytics and visualization layers. This is something revolutionary for the data analytics industry.

The speed, scalability and usability advantages of our integrated platform are apparent to our customers. Our goal is to deliver up-to-the-minute, quantitative, operational intelligence about a business’s transaction streams, at a scale previously incomprehensible, at a cost effective price point, and for a broad customer set. In short, it’s been our goal to enable our partners to interact with their critical transaction events data when, where and how they choose; as the first thing they check when they wake up; as the last thing they check before going to sleep. Our mobile and tablet product now enable our customers to analyze their data in the middle of the night.

Without a doubt, our team’s stunning product achievements in 2011 are the accomplishment of which I’m most proud. We have been true to our original vision, and now lead the market for hosted, web-scale Business Intelligence for the global media and advertising industries. Now that Metamarkets has established its offer in an initial set of verticals, we can turn to address the broader emergent market opportunity, one that both Mike and I can honestly state is far larger than our initial conception when we launched the company a couple of years ago.

The Big Swing

Metamarkets sits at the intersection of three megatrends: Big Data, analytics, and cloud computing. We believe this opportunity must be seized quickly and aggressively, and so in 2012, Metamarkets will extend our platform into adjacent, data-intensive verticals whom we know are hungry for our brand of cost-effective, scale analytics.

A world leading company in big data analytics requires appropriate leadership given the specialized technical nature of the opportunity. I’m incredibly grateful and proud to announce that this is the time for Mike Driscoll, my Metamarkets co-founder and CTO, to take over as Metamarkets’ CEO. Mike is a world-class technical talent and rising star in data software, one of the most intensely brilliant technical product minds I’ve encountered in over 15 years of financial and data software work, a colleague whom I’ve had the great good fortune to call co-founder, and with whom I’ve worked shoulder-to-shoulder to build a great foundation over the past two years. I have the utmost confidence in Mike’s abilities, vision and instincts to take Metamarkets to the next level (or two, or three). Mike will take our initial vision of a distributed, fast, analytics service, and grow Metamarkets into something far more transformative than anybody could have imagined a couple of years ago.

Transitions

Nurturing and launching a company is very different from scaling one. I have spent three years getting Metamarkets off the ground, the past two in bringing our product into the market. In this time, we’ve built a company and put the foundational product and engineering teams in place. And so the time is right for me to return to my passion: the early stages of product ideation and new venture formation. I’m extremely fortunate to be joining a leading early-stage VC fund as Venture Partner and EIR. I have a number of ideas I am considering, though there is one I am particularly passionate about launching in 2012 – more about my next steps at a later date.

Most importantly, despite stepping away from the CEO’s chair, I will continue to serve actively on Metamarkets’ Board of Directors, and I’ve coordinated with Mike to continue to advise the Metamarkets product and sales teams on their approach to the media and advertising markets. Our current partners and customers will experience no change in their service or interactions with the company. We continue to aim high and service our partners at the same exceptional levels we established in 2011. I’m very much looking forward to helping Metamarkets continue to grow, while also advancing some of my own personal career objectives. The time is right for this transition, and Mike and I have embraced it.

It’s been a great to get Metamarkets out of the garage, moving along local streets, following access roads, and now onto the highway. We can see the wide, fast road ahead, and I for one can feel the car accelerating…

DS

The Road Ahead

When David and I began our start-up journey together in 2009, in a windowless office on Townsend Street, I doubt if either of us could have conceived of the successes (nor of the sleepless nights) that the subsequent years would bring.  We are fortunate to have assembled a team of extraordinary engineers, whose dedicated efforts have yielded a revenue-generating product serving a growing cast of engaged customers.

I am proud of what we have accomplished to date under David's leadership as CEO.  I look forward to having his unparalleled intellect and instincts at work for us as we move forward.  And I'm humbled by the opportunity to build on his achievements and begin scaling our organization to match the scope of our market opportunity.

To that end, today I am announcing that Ken Chestnut, a former executive at MarkLogic and Siebel Systems, will be joining Metamarkets to lead our marketing efforts.  Ken brings over 15 years of experience in the management, strategy, and marketing of business software tools.  Ken joins Charlene Son Rigby, a former Oracle executive, who leads our sales and operations and joined us two months ago.

Working with our engineering leadership, Ken and Charlene will help us capitalize on our core technology: our in-memory data store, analytics engine, and real-time, interactive dashboard.

As we will describe in an upcoming blog post, we have scaled our cloud-based data and compute infrastructure by a factor of 100 since bringing our first customer onboard over a year ago.  As of last week, we clocked a new performance record: a query that processed 26 billion records in under a second.

It is time for Metamarkets to double-down on our technology, address verticals beyond the digital media markets, and lengthen our lead as a pioneer in big data solutions.

As I shift from my role as CTO, I aim not only to maintain but to strengthen the engineering-driven culture that has gotten us to this stage.

MD

Hive was developed by the folks at Facebook in 2008, as a means of providing an easy-to-use, SQL-like query language that would compile to MapReduce code. A year later, Hive was responsible for 95% of the Hadoop jobs run on Facebook's servers. This is consistent with another observation made by Cloudera's Jeff Hammerbacher: when Hive is installed on a client's Hadoop cluster, its overall usage increases tenfold.

That data-heavy businesses can achieve visibility into the terabytes of logs that they generate is, at a primary level, a major step forward. Before the Hadoop era, this was difficult to impossible without a major engineering investment. Thus Hadoop has solved the challenge of economically processing data at scale. Hive has solved the challenge of hand-writing Hadoop queries.

But there remains a painful challenge that Hive and Hadoop does not solve for: speed.

A Powerful But Lumbering Elephant

Hadoop does not respond anywhere close to "human time", a term that describes response thresholds acceptable to a human user, typically on the order of seconds. Larry Ellison and his marketing mavens invoke a similar theme when pitching their wares as "analytics at the speed of thought."

Nonetheless, this sluggishness is not the fault of Hive or Hadoop per se. If a business user asks a question about a year's worth of data with Hive, a set of MapReduce jobs will dutifully scan and process, in parallel, terabytes of data to obtain the answer. It's neither the commodity hardware that most Hadoop clusters use nor some of its IO indulgences while executing processes, that are to blame. These are the low-order performance bits.

And while Hadoop jobs do have a fairly constant overhead -- with a lower bound in the range of 15 seconds -- this is often considered trivial within the context of the minutes or hours that most full jobs are expected to take.

The higher-order bits affecting query performance are: (i) the size of the data being scanned, (ii) the nature of storage, e.g. whether it is kept on disk or in memory, and (iii) the degree of parallelization.

An Emerging Design Pattern: Distill, then Store

As a result, a common design pattern is emerging among data-heavy firms: Hadoop is used as a pre-processing tool to generate summarized data cubes, which is then loaded into an in-memory, parallelized database -- be it Oracle Exalytics, Netezza, Greenplum or even Microsoft SQL Server. Occasionally, a traditional database query layer can be bypassed altogether, and summary data cubes can be loaded directly into a desktop analytics tool such as Qlikview, Spotfire, or Tableau.

At Metamarkets, we have embraced this design pattern and the role that Hadoop plays in preparing data for fast queries. Our particular bag of tricks is best described by the three principles of Druid:

• Distill: We roll data data up the coarsest grain at which a user might have reasonable interest. Put simply, it is rare that one is concerned with individual events at one-second time frames. Rolling up to groups of events, with a select set of dimensions and at minutely or hourly granularity, can distill raw data's footprint down to 1/100th of its original size.
• Distribute: While this summarized data is spread across multiple nodes in our cluster, the queries against this data are also distributed and parallelized. In our quest to break into the "human time" threshold, we have increased this parallelization to as many as 1000 cores, allowing each query to hit a large percentage of nodes on our cluster. In our experience, CPUs are rarely the bottleneck for systems serving human clients, even for a cluster serving hundreds of users concurrently.
• DRAM: We share Curt Monash's sentiment that traditional databases will eventually end up in RAM , as memory costs continue to fall. In-memory analytics are popular because they are fast, often 100x to 1000x faster than disk. This dramatic performance kick is what makes Qlikview such a popular desktop tool.

The end result of these three techniques, each of which independently delivers between a 10 and 1000-fold improvement, is a platform that can run in seconds what previously took minutes or even hours in Hive.

This approach, for which we know we are not alone in pursuing, achieves performance that exceeds or matches any of the big box retailers at a considerably lower price point.

The commoditization wave that began with massive data processing, initiated by Hadoop, is migrating upwards towards query architectures. Thus the competitive differentiators are shifting away from large-scale data management and towards what might be called Big Analytics, where the next battle for profits will be fought.

Existing forecasting systems tend to exist inside of traditional OLAP storage offerings that embed a limited set of statistical functions, many of which are poorly optimized and slow to execute. The machine learning renaissance of the last decade has been tempered by powerful additions from the algorithms and distributed systems communities: compressed sensing, feature hashing and feature sharding across multiple cores have significantly lowered computational costs, while advances in regularization and structured sparsity from applied statistics have likewise increased model robustness, allowing unprecedented modeling of high-degree feature interactions.

In this post I'll outline some of the engineering challenges behind putting a large-scale machine learning system into production, focusing less on actual learning algorithm implementations (as these are fairly commoditized at this point: mahout, vowpal wabbit, scikits.learn, etc) and instead addressing our engineering architecture. In particular we'll look at time-series forecasting, separate from more generic prediction, where our end goal is to create forecasts where nothing interesting occurs.*

Forecasting system

For forecasting, we combine predictions from two separate subsystems: a top-down "structural" model and a bottom-up "cross-correlation" model.

Structural model
Top-down models capture long-term trends and other temporal correlations:

For example, some of our clients tend to see more traffic on certain sites during the Academy Awards, but less on Easter Sunday (unless they're living in Japan or Hong Kong, etc). Through several variants of this model, we can also address transient data anomalies:

and structural changes in baseline and trend, such as new site layouts that affect aggregate traffic flows:

How these effects are integrated into forecasts depends on the semantics of the underlying metric: for online advertising impressions we may want to simply predict "baseline" guaranteed traffic; when forecasting price, we may be more interested in the structure of transient changes.

Cross-correlation model

Bottom-up large-scale models capture fine-grained atemporal feature interactions. This approach trades model flexibility for brute force data mining power, capturing high-order correlations between surface data, such as city x site effects and audience x time-of-day effects. An example of what this model captures is that visitors with New York City and London IP addresses are more likely to consume financial news content on weekday mornings.

For learning, we make use of stochastic gradient descent for parameter updates, and implicit high-order feature generation to capture cross-correlations combined with aggressive regularization and low bit-order feature hashing to combat overfitting and keep updates efficient. These choices have become the de-facto state of the art for computational efficiency and prediction quality, powering systems such as Google's Sybil and Vowpal Wabbit (cf. learning on cores, clusters and clouds workshop at NIPS).

We also built an expressive feature specification framework directly into our configuration language; it is well-established that feature engineering often has a much larger impact on predictive quality than choice of learning algorithm (cf. the BellKor Netflix Solution, KDD Cup 2010 discussions, the Stanford IR Book).

System-level overview

In addition to generating state of the art forecasts, we identified three main engineering desiderata:

Scalable

Our underlying model framework must (1) be able to cope with multiple terabytes of data per client, and (2) gracefully degrade, automatically prioritizing the most important aspects of prediction over less important ones (maintain high precision / manipulate recall), (3) parallelizable across a commodity hadoop cluster.

Incremental

Fresh data arrives daily / hourly / or even realtime, and as such we cannot waste computational resources constantly retraining the predictive models from scratch. However, online incremental learning suffers from convergence effects and drift in the underlying data. Therefore we took a hybrid approach: we couple an incremental model, updated as data arrives with an introspective changepoint detection system for forcing full model rebuilds. This approach also gives us more scaling flexibility: incremental updates run in O(hours) while batch retrains run in O(days).

Modular

Generic machine learning efforts ultimately fail; so we acknowledge the need for continual model development and backtesting. Hence we desire a system with simple, modular configuration for tuning model parameters and error metrics and clean component architecture. Our current component set includes multiple data ingestion pipelines (+ automated data versioning and incremental cacheing along the way) and several different high-level forecasting models that take into account structural changepoints, conditional heteroscedasticity, and long-term trends.

We also support additional covariance models as overlays, for example audience tags overlaid with site usage data, via online matrix factorization (think: collaborative filtering / Netflix prize). These models capture structure that is orthogonal to the underlying forecast, but nonetheless important from a user perspective. For instance, during the Women's World Cup / Olympics, gender-based consumption of sports content exhibits strong shifts.

The Future

Despite the effort put into "future-proofing" the forecasting system, there will inevitably be features that force us to rethink our design. Currently we're working to expose the underlying prediction stack more cleanly, in order to reduce friction for surfacing atemporal analytics. Further afield, we intend to address

• Model integration / combining outputs of multiple models hierarchically. E.g., integrated supply and demand forecasting.
• Automated prior estimation for new data facets.
• Automated prediction validation scoring framework (independent of prediction source).

[*] "Ennui" -- with apologies to Sylvia Plath.

It's not a passively animated graphic, but one that users can actively engage with, freezing or pivoting dimensions to reveal new views of the data. It's a harbinger of a new class of documents, which digital publishers are beginning to embrace, to provide a richer information experience for readers.

Meet the Interactive Frameworks

That the above graphic could be built in a single weekend (it was part of a larger hackathon called Data In Sight that Barret Schloerke and his team 13 participated in) is testament to the maturity of tools available.

In the last few years, there has been a blossoming of frameworks for creating rich, dynamic infographics. These include Processing (and Processing.js), Adobe Flex, Raphael, Prefuse and Flare, Protovis, and now D3, among others.

These frameworks present new possibilities for data visualization, but also challenges. Expose too little interaction, and one risks being little different than a static visualization. Expose too much interaction, and the user is overwhelmed by a jumble of buttons and sliders, with no clear narrative path.

Used well, interaction is a means to escape flatland. In the Disaster Strikes graphic, for example, it is used to encode an additional two dimensions of data (disaster metric and time) beyond the three that are possible with a heatmap (in this case, country and disaster class on the axes, and magnitude at points inside the matrix). This allows the graphic to express five dimensions of disaster.

The Disasters heatmap leveraged two tools that are also used internally at Metamarkets:

• D3 (short for "data driven documents") - the sequel to Protovis
• DVL (short for "dynamic visualization LEGOs") - a framework for building event-driven web pages, developed by our very own Vadim Ogievetsky

The short but painful history of several rich web toolkits provides lessons to one's choice of interactive visualization tools: choose those that work well with web standards and the DOM. This favors Javascript frameworks, such as Processing.js and D3, over those that may rely on browser plug-ins (yes, I am biased about Javascript).

Next I turn to some of the challenges of designing interactive visualizations, namely working with time, revealing stories, and surfacing state.

Visualize Time as a Flow, not a Flicker

Evolving a visualization in time is a powerful technique that should used with care. Displaying discrete jumps of data over time can be disconcerting for a viewer, making it hard to follow patterns. One valuable way to address this challenge is to smear time: let events fade into the past, rather than showing only a fast flicker of the present. Providing a ghosting of the recent past, where data flows, can provide a wider temporal context for otherwise diffuse events (as is often the case for points on a map). This has been used with success by Aaron Koblin's Flight Patterns as well as Stamen Design's Cabspotting.

Visualizing data as a flow is more than just aesthetically pleasing; recent work by Colin Ware suggests it may be a better way to encode temporal data, given our eyes' natural aptitude for perceiving continuous contours.

The Power of Story, The Joy of Discovery

John Lasseter of Pixar has said "No amount of great animation will save a bad story." Likewise, it isn't enough for data visualizations to look beautiful: to succeed, they must tell a compelling story.

In the case of dynamic, interactive visualizations, this can be a challenge. Most possible states of a visualization are simply uninteresting. The key for the information designer is to constrain exploration along paths that are most likely to yield insights. Most data are sparse and long tailed, so curating and narrowing dimensions (don't let outliers warp your axes) can help restore some information density.

For an example of this, witness the New York Times visualization of Netflix Queues . Rather than a full choropleth of the United States, twelve metropolitan areas were preferentially selected, and a set of movies with distinct rental patterns were helpfully linked to at the top of the page.

Applying algorithms to reorder the data, such as placing similar data points together, can also help guide users towards discovering patterns. The Disaster Strikes visualization includes a "cluster" button which executes a bivariate clustering algorithm in Javascript, revealing countries that have suffered similar kinds of disasters (Korea, Ecuador and Guatemala all suffer from tropical storms).

The upside of a visualization with an exploratory state space is that users experience the joy of discovery. Jeff Heer's Sense.US allowed users to explore US Census information over the past century and debate the meanings of discovered trends.

Encoding State with a Stateless Protocol

One important point in such visualizations is that state be encoded in the browser URL. When you find something interesting, for example that 1996 was a bad year for bacterial outbreaks, you should be able to share it. Updating the base URL is not wise, as it would require a painful reload of the page for each interaction. The recommended approach is using a hashbang fragment (but be careful), which can be detected to set the visualization to the proper state.

The Disaster Strikes visualization does not surface state yet, but I expect with more time it would have been implemented, and we make extensive use of this with our internal dashboards.

Progress, In Sight

The field of interactive visualization is still nascent, but this weekend's event was testimony that static visualizations are likely to go the way of printed books. Though "interactive" was just one of the four award categories, every single team I witnessed on Sunday showed what would be considered an interactive, dynamic visualization.

Some final notes about the competition: The world disaster data, as well as other data sets that were eligible for use at the hackathon, are available at this special InfoChimps page. . One specific winner can be seen here, and the full set of winners can be seen at the Data Insights home..

Interested in creating data visualizations like this one? Come join the Metamarkets Team.

1. Partial Aggregates + In-Memory + Indexes => Fast Queries

We work with two representations of our data: alpha represents the raw, unaggregated event logs, while beta is its partially aggregated derivative. This beta is the basis against which all further queries are evaluated:

2011-01-01T01:00:00Z  ultratrimfast.com  google.com  Male    USA      1800         25      15.70
2011-01-01T01:00:00Z  bieberfever.com    google.com  Male    USA      2912         42      29.18
2011-01-01T02:00:00Z  ultratrimfast.com  google.com  Male    UK       1953         17      17.31
2011-01-01T02:00:00Z  bieberfever.com    google.com  Male    UK       3194         170     34.01

This is the most compact representation that preserves the finest grain of data, while enabling on-the-fly computation of all O(2^n) possible dimensional roll-ups.

The key to Druid’s speed is maintaining the beta data entirely in memory. Full scans are several orders of magnitude faster in memory than via disk. What we lose in having to compute roll-ups on the fly, we make up for with speed.

To support drill-downs on specific dimensions (such as results for only ‘bieberfever.com’), we maintain a set of inverted indices. This allows for fast calculation (using AND & OR operations) of rows matching a search query. The inverted index enables us to scan a limited subset of rows to compute final query results – and these scans are themselves distributed, as we discuss next.

2. Distributed Data + Parallelizable Queries => Horizontal Scalability

Druid’s performance depends on having memory -- lots of it. We achieve the requisite memory scale by dynamically distributing data across a cluster of nodes. As the data set grows, we can horizontally expand by adding more machines.

To facilitate rebalancing, we take chunks of beta data and index them into segments based on time ranges. For high cardinality dimensions, distributing by time isn’t enough (we generally try to keep segments no larger than 20M rows), so we have introduced partitioning. We store metadata about segments within the query layer and partitioning logic within the segment generation code.

We persist these segments in a storage system (currently S3) that is accessible from all nodes. If a node goes down, Zookeeper coordinates the remaining live nodes to reconstitute the missing beta set.

Downstream clients of the API are insulated from this rebalancing: Druid’s query API seamlessly handles changes in cluster topology.

Queries against the Druid cluster are perfectly horizontal. We limited the aggregation operations we support – count, mean, variance and other parametric statistics – that are inherently parallelizable. While less parallelizable operations, such as median, are not supported, this limitation is offset by rich support of histogram and higher-order moment stores. The co-location of processing with in-memory data on each node reduces network load and dramatically improves performance.

This architecture provides a number of extra benefits:

• Segments are read-only, so they can simultaneously serve multiple servers. If we have a hotspot in a particular index, we can replicate that index to multiple servers and load balance across them.
• We can provide tiered classes of service for our data, with servers occupying different points in the "query latency vs. data size" spectrum
• Our clusters can span data center boundaries

3. Real-Time Analytics: Immutable Past, Append-Only Future

Our system for real-time analytics is centered, naturally, on time. Because past events happen once and never change, they need not be re-writable. We need only be able to append new events.

For real-time analytics, we have an event stream that flows into a set of real-time indexers. These are servers that advertise responsibility for the most recent 60 minutes of data and nothing more. They aggregate the real-time feed and periodically push an index segment to our storage system. The segment then gets loaded into memory of a standard server, and is flushed from the real-time indexer.

Similarly, for long-range historical data that we want to make available, but not keep hot, we have deep-history servers. These use a memory mapping strategy for addressing segments, rather than loading them all into memory. This provides access to long-range data while maintaining the high-performance that our customers expect for near-term data.

Summary

Druid’s power resides in providing users fast, arbitrarily deep exploration of large-scale transaction data. Queries over billions of rows, that previously took minutes or hours to run, can now be investigated directly with sub-second response times.

We believe that the performance, scalability, and unification of real-time and historical data that Druid provides could be of broader interest. As such, we plan to open source our code base in the coming year.

Interested in tackling distributed systems challenges like this one? Come join the Metamarkets Team.

We started to investigate.

(Above: Rolling 10 days of article ranks. Click for an interactive version.)

The right way to answer this question was pretty obvious: crunch the data.  We started scraping HN titles along with article ranks and fed the resulting data into our online feature learning stack.

Below is the distilled summary of the result, our "Top Ten Hacker News Headline Hacks," including feature weight, standard error and p-value vs. zero. Positive weight means the feature is predictive of high article rank.

Hack #1:  Maximize Controversy

1.4 ± 0.5 [p<1e-5] | essential
1.3 ± 0.5 [p<1e-5] could
1.2 ± 0.4 [p<1e-5] problem
1.3 ± 0.8 [p<1e-5] survived the
1.0 ± 0.5 [p<1e-5] controversy
0.9 ± 0.3 [p<1e-5] impossible

Hack #2:  Question Authority

0.7 ± 0.2 [p<1e-5] why ____ future
0.4 ± 1.0 [p=0.2] the ____ behind
0.2 ± 0.3 [p=0.04] why don't
0.1 ± 0.3 [p=0.06] | lessons

Hack #3:  Avoid False Promises

-1.5 ± 0.8 [p<1e-5] tricks
-0.7 ± 0.5 [p<1e-5] the world |
-0.7 ± 0.2 [p<1e-5] the greatest
-0.6 ± 0.3 [p<1e-5] awesome
-0.6 ± 0.7 [p=0.003] anatomy of a
-0.5 ± 0.3 [p<1e-5] guide to

Hack #4:  Short is Sweet

-0.3 ± 0.04 [p<1e-5] {# WORDS}

Hack #5:  Execution not Ideas

2.6 ± 2.1 [p<1e-5] showing
1.5 ± 0.7 [p<1e-5] | building
0.6 ± 0.3 [p<1e-5] makes
0.5 ± 0.4 [p<1e-5] starting a company
0.3 ± 0.3 [p<1e-3] join a startup

-1.1 ± 0.3 [p<1e-5] ideas
-1.1 ± 0.3 [p<1e-5] idea?

Hack #6:  Everybody Loves a Winner

1.7 ± 0.7 [p<1e-5] | ____ acquires
0.5 ± 0.3 [<1e-5] hire
0.4 ± 0.7 [p=0.02] worth

Hack #7:  Everybody Loves Data

1.9 ± 1.8 [p<1e-4] data |
0.6 ± 0.8 [p=0.004] data -
0.5 ± 0.1 [p<1e-5] visualize data in

-1.3 ± 0.7 [p<1e-5] algorithm

Hack #8: Nobody Cares About You

-0.2 ± 0.3 [p=0.008] my startup
-0.9 ± 0.2 [p<1e-5] silicon valley

Hack #9:  Some Topics are Just Miserable

-0.4 ± 0.3 [p<1e-5] angry birds
-0.2 ± 0.1 [p<1e-5] harry potter
-0.5 ± 0.4 [<1e-4] taxes
-1.5 ± 1.0 [<1e-5] downtime

Hack #10:  Social is For Losers

-0.6 ± 0.9 [p=0.007] social
-0.5 ± 0.4 [p<1e-4] gamification
-0.3 ± 0.6 [p=0.04] twitter |
-2.4 ± 1.5 [p<1e-5] airbnb

Standard disclaimer: the above coefficients are provided for entertainment purposes only. Feature interactions in text are a bitch. Correlation does not imply causation. Past performance does not guarantee future success.

How We Did It

We extracted n-gram (e.g. “Harry Potter”, “Google”, “Silicon Valley”) and skip features (e.g., “a ____ for”,  ”| ____ acquires”) for each title, including start- and end-of-sentence markers and optionally punctuation. For learning we used boosted stochastic gradient descent with logistic loss [2], predicting whether the article made it to the top 20 or not during its observed lifetime. Strong regularization was used to eliminate spurious features, and twenty bootstrap replicates were used to measure significance of coefficients and classification accuracy.

For this untuned, first-pass model, we achieved 64% classification accuracy on a hold out set over the past two months. Positive predictive value was 25.7%, negative predictive value was 73.1%, sensitivity was 18.2% and specificity was 80.9% [3]. Despite this weak predictive power, we found some interesting correlations, more of which we'll release as the model improves.

[1] Of Koalas to the Max fame.
[2] Think: wabbit style.
[3] Predictive diagnostics.