Distributing Data in Druid at Petabyte Scale

Filed in Algorithms, Corporate, Data Visualization, Druid, R, Technology

At Metamarkets we run one of the largest production Druid clusters out there, so when it comes to scalability, we are almost always the first ones to encounter issues of running Druid at scale. Sometimes, however, performance problems are much simpler, and the downside of a large cluster is that it tends to average out problems that are hiding in plain sight, making them harder to pinpoint. Recently, we started noticing that, despite being able to scale our cluster almost horizontally, performance would not always increase accordingly. While we don’t expect a linear increase in speed, some of the numbers […]

Distributing Data in Druid at Petabyte Scale
Read Post Comments

Behind the Scenes with Metamarkets, Episode 2

Filed in Corporate, Druid, Technology

In the latest episode in our “Behind The Scenes” video series, we sat down with Dr. Charles Allen, a senior software engineer at Metamarkets and one of the leading developers focused on Druid, the open-source database built by the Metamarkets team. Watch the video to hear how Charles first encountered and implemented Druid before coming to Metamarkets, what he sees as the key strengths of the technology, how he’s worked to implement more mutability of data within Druid, and how he’s helping users get maximum utilization of their clusters.

Behind the Scenes with Metamarkets, Episode 2
Read Post Comments

Druid Query Optimization with FIFO: Lessons from Our 5000-Core Cluster

Filed in Druid, Technology

Druid’s Horizontal Scale A large strength of using Druid as a data store and aggregation engine is its ability to horizontally scale. Whenever more data is in the system, or whenever faster compute times are desired, it is simply a matter of throwing more hardware at the problem, and Druid auto-detects, and auto-balances its workloads. At Metamarkets we are currently ingesting over 3M events/ second (replicated) into our Druid cluster and have multiple hundreds of historical nodes serving this data across multiple tiers. Part of the power of this horizontal scale is how Druid breaks up data into shards. Each […]

Druid Query Optimization with FIFO: Lessons from Our 5000-Core Cluster
Read Post Comments

Dogfooding with Druid, Samza, and Kafka: Metametrics at Metamarkets

Filed in Company, Druid

“Another flaw in the human character is that everybody wants to build and nobody wants to do maintenance.” – Kurt Vonnegut Every engineer loves the feeling of standing up a new piece of open source infrastructure, satisfaction born from a grueling journey through community forums, outdated documentation, and mostly-uncommented source code. The glory is fleeting, because not 20 minutes into having your shiny new service up and running, a hiccup hits and you’re forced to ask: what am I going to use to monitor and maintain this thing? We’ve now entered an era where software increasingly runs not on a […]

Dogfooding with Druid, Samza, and Kafka: Metametrics at Metamarkets
Read Post Comments

Five Tips for a F’ing Great Logo

Filed in Corporate, Druid

This post originally appeared on Druid.io on July 23, 2014. Everyone wants a great logo, but it’s notoriously difficult work—prone to miscommunications, heated debates and countless revisions. Still, after three years we couldn’t put it off any longer. Druid needed a visual identity, so we partnered with the talented folks at Focus Lab for help. Our old logo (left) was…lacking. Much better now, right? Despite our fears, we cranked this out with Focus in a speedy three week sprint. Not only was the process drama-free, it was actually fun. The goal of this post is to give you some insight into how […]

Five Tips for a F’ing Great Logo
Read Post Comments