Druid and Spark Together – Mixing Analytics Workflows

Filed in Druid, Technology

One method of looking at the human thought process is that we have different brain workflows for different analytics and data processing needs. At Metamarkets we have the same thing for our data processing machines. This post will explore some of our experience with bin-packing query nodes featuring Druid with batch processing featuring Spark, using Apache Mesos as the resource coordinator. Thinking Fast The above shows a typical “work” pattern for a Druid historical node throughout a typical week day (times shown are EDT). It shows the quantity of CPU seconds consumed by the JVM to answer queries. There are […]

Druid and Spark Together – Mixing Analytics Workflows
Read Post Comments

Behind the Scenes of our Transition to a Multi-Cloud Environment

Filed in Algorithms, Company, Druid, Technology

Service uptime is the performance metric that determines operational success and when something fails, the impact can be far reaching, often affecting a business’s bottom line. One of the downsides of running infrastructure in a public cloud is that we are dependent on the SLAs provided by our Cloud Providers. As a startup, we have been upgrading our systems to become a lot more fault-tolerant, but since our cloud infrastructure footprint is restricted to one region, and the oldest region of AWS at that, we are vulnerable to be bitten by cloud service blackouts or brownouts. The most prominent solution […]

Behind the Scenes of our Transition to a Multi-Cloud Environment
Read Post Comments