Behind the Scenes of our Transition to a Multi-Cloud Environment

Filed in Algorithms, Company, Druid, Technology

Service uptime is the performance metric that determines operational success and when something fails, the impact can be far reaching, often affecting a business’s bottom line. One of the downsides of running infrastructure in a public cloud is that we are dependent on the SLAs provided by our Cloud Providers. As a startup, we have been upgrading our systems to become a lot more fault-tolerant, but since our cloud infrastructure footprint is restricted to one region, and the oldest region of AWS at that, we are vulnerable to be bitten by cloud service blackouts or brownouts. The most prominent solution […]

Behind the Scenes of our Transition to a Multi-Cloud Environment
Read Post Comments

Moving Real-Time Data Flow Across Cloud Providers

Filed in Algorithms, Data Science, Druid, Technology

Eventually in the course of data growth, a company needs to make a major migration of data or processes from one physical location to another. This post is the story of how we moved a real-time data flow across cloud providers using Kafka, Samza, and some creative engineering. History Our technology stack for data processing is something we’ve spoken about before. We run a Lambda architecture with the real-time system comprising Kafka and Samza, which terminates in Druid real-time indexing tasks. The batch system is comprised of Spark, which reads and writes from S3. Druid historical nodes use S3 as […]

Moving Real-Time Data Flow Across Cloud Providers
Read Post Comments

Going Multi-Cloud with AWS and GCP: Lessons Learned at Scale

Filed in Algorithms, Druid, Industry, Technology

Metamarkets handles a lot of data. The torrent of data that clients send to us surpasses a petabyte a week. At this scale, the ability to failover gracefully, to detect and eliminate brownouts, and to efficiently operate huge quantities of byte-banging machines is necessary. We started and grew Metamarkets in AWS’s us-east region. And the majority of our footprint was in a single availability zone (AZ). As we grew, we started to see the side effects of being restricted to one AZ, then the side effects of being restricted to one region. It’s kind of like inflating a balloon in […]

Going Multi-Cloud with AWS and GCP: Lessons Learned at Scale
Read Post Comments

5 Ways to Maximize Value from Interactive Analytics

Filed in Algorithms, Data Science, Our Customers, Technology

The long term success of any successful analytics platform relies on two items – the usefulness of the data and the time/value costs associated. At Metamarkets we have a dedicated team of experts in the AdTech space who can help programmatic marketers squeeze the most ROI out of their integration. Here are a few tips from our team on how to maximize your investment in analytics: Work Backwards from Actions When thinking about how you can best leverage Metamarkets, it’s important to start by focusing on your challenges and questions then work backwards. In the OpenRTB standard there are dozens of different […]

5 Ways to Maximize Value from Interactive Analytics
Read Post Comments

Making the Most of EBS Volumes with Reattachment

Filed in Algorithms, Technology

Here at Metamarkets, we are always seeking to use our AWS resources more efficiently and reduce human intervention required to address failures. We run some large workloads that utilize a lot of EBS volumes and in some cases spot instances. To facilitate this, we’ve written ER, a system to enable our instances to attach, detach, and create EBS volumes at boot without relying on any external coordinator other than the EC2 API. Pain Point: Kafka Brokers We run a few Kafka clusters where we use EBS volumes that allow us to replace instances without losing days worth of real- time […]

Making the Most of EBS Volumes with Reattachment
Read Post Comments