Comments for Metamarkets http://metamarkets.com Fast Insight for Big Data Fri, 27 Jan 2012 22:00:16 +0000 hourly 1 http://wordpress.org/?v=3.3.1 Comment on Munging, Modeling and Visualizing Data with R by Ashwin Jayaprakash http://metamarkets.com/2012/munging-and-visualizing-data-with-r/#comment-1039 Ashwin Jayaprakash Fri, 27 Jan 2012 22:00:16 +0000 http://metamarkets.com/?p=778#comment-1039 Thanks for sharing. This is a good summary, much better than what's here - http://www.r-bloggers.com/programmers-should-know-r/ Thanks for sharing. This is a good summary, much better than what's here - http://www.r-bloggers.com/programmers-should-know-r/

]]> Comment on Munging, Modeling and Visualizing Data with R by Cotiso http://metamarkets.com/2012/munging-and-visualizing-data-with-r/#comment-1026 Cotiso Fri, 27 Jan 2012 17:34:15 +0000 http://metamarkets.com/?p=778#comment-1026 Great stuff ! Tx, Cotiso Great stuff !
Tx,
Cotiso

]]> Comment on Druid, Part Deux: Three Principles for Fast, Distributed OLAP by Nicolas http://metamarkets.com/2011/druid-part-deux-three-principles-for-fast-distributed-olap/#comment-626 Nicolas Fri, 20 Jan 2012 19:49:51 +0000 http://metamarketsgroup.com/blog/?p=382#comment-626 The pieces that you have put together in Druid look truly fantastic. Also looking forward to the open-sourcing of the code base, and also perhaps contributing to it! The pieces that you have put together in Druid look truly fantastic. Also looking forward to the open-sourcing of the code base, and also perhaps contributing to it!

]]> Comment on Beyond Hadoop: Fast Queries from Big Data by Don MacLennan http://metamarkets.com/2011/hadoops-secret-shortcoming-speed-and-how-to-fix-it/#comment-80 Don MacLennan Fri, 16 Dec 2011 17:20:36 +0000 http://metamarketsgroup.com/blog/?p=584#comment-80 Mike, nice post. Similar to what we're experiencing in early days of our Big Data journey. I wrote a post on the topic you hinted at in your last paragraph. i'm interested in your take: http://donmaclennan.com/2011/12/04/the-measurement-wars/ Mike, nice post. Similar to what we're experiencing in early days of our Big Data journey. I wrote a post on the topic you hinted at in your last paragraph. i'm interested in your take: http://donmaclennan.com/2011/12/04/the-measurement-wars/

]]> Comment on Beyond Hadoop: Fast Queries from Big Data by Pascal Bensoussan http://metamarkets.com/2011/hadoops-secret-shortcoming-speed-and-how-to-fix-it/#comment-79 Pascal Bensoussan Mon, 07 Nov 2011 18:34:31 +0000 http://metamarketsgroup.com/blog/?p=584#comment-79 You're not alone! Many thanks Mike for mentioning Aggregate Knowledge's blog post on "Building a Big Analytics Infrastructure". You're not alone! Many thanks Mike for mentioning Aggregate Knowledge's blog post on "Building a Big Analytics Infrastructure".

]]> Comment on Hacking Hacker News Headlines by JR http://metamarkets.com/2011/hacking-hacker-news-headlines/#comment-74 JR Fri, 30 Sep 2011 09:36:53 +0000 http://metamarketsgroup.com/blog/?p=256#comment-74 Actually bootstrap has a rather long history of being used to estimate coefficient standard error. There is a recent Casella paper: http://ba.stat.cmu.edu/journal/2010/vol05/issue02/casella.pdf but the technique dates back at least to Tibshirani's original Lasso paper. That said, for Lasso at least, if the true beta is zero then you can show bootstrap estimates are not consistent (somewhat intuitively, though unfortunate). The bigger problem for us is that the bias introduced by regularization actually makes the CIs pretty meaningless anyway, since you can't measure the contribution of the bias to the stderr. There is a good stackexchange on this: http://stats.stackexchange.com/questions/2121/how-can-i-estimate-coefficient-standard-errors-when-using-ridge-regression So, as with any stats-meets-social-science result, take with a grain of salt. --j Actually bootstrap has a rather long history of being used to estimate coefficient standard error. There is a recent Casella paper:

http://ba.stat.cmu.edu/journal/2010/vol05/issue02/casella.pdf

but the technique dates back at least to Tibshirani's original Lasso paper. That said, for Lasso at least, if the true beta is zero then you can show bootstrap estimates are not consistent (somewhat intuitively, though unfortunate).

The bigger problem for us is that the bias introduced by regularization actually makes the CIs pretty meaningless anyway, since you can't measure the contribution of the bias to the stderr. There is a good stackexchange on this:

http://stats.stackexchange.com/questions/2121/how-can-i-estimate-coefficient-standard-errors-when-using-ridge-regression

So, as with any stats-meets-social-science result, take with a grain of salt.

--j

]]> Comment on The Rise of Interactive Data Visualization by Ramesh http://metamarkets.com/2011/the-rise-of-dynamic-data-visualization/#comment-78 Ramesh Tue, 20 Sep 2011 04:48:39 +0000 http://metamarketsgroup.com/blog/?p=406#comment-78 An eye opener. I look at the visual medium and wonder what are we still doing in the world of grids and frames. Being mostly a data guy, I tried learning d3 to do something about it - seems like MetaMarkets as a company has more potential in doing the impossible. Keep going guys! An eye opener. I look at the visual medium and wonder what are we still doing in the world of grids and frames. Being mostly a data guy, I tried learning d3 to do something about it - seems like MetaMarkets as a company has more potential in doing the impossible. Keep going guys!

]]> Comment on Introducing Druid: Real-Time Analytics at a Billion Rows Per Second by Building a Big Analytics Infrastructure « AK Tech Blog http://metamarkets.com/2011/druid-part-i-real-time-analytics-at-a-billion-rows-per-second/#comment-66 Building a Big Analytics Infrastructure « AK Tech Blog Thu, 08 Sep 2011 23:37:29 +0000 http://metamarketsgroup.com/blog/?p=189#comment-66 [...] problems. Some notable exceptions are Facebook, Twitter’s Rainbird and MetaMarket’s Druid. In this post we provide an overview of how we built Aggregate Knowledge’s “big [...] [...] problems. Some notable exceptions are Facebook, Twitter’s Rainbird and MetaMarket’s Druid. In this post we provide an overview of how we built Aggregate Knowledge’s “big [...]

]]> Comment on Hacking Hacker News Headlines by anil patwardhan http://metamarkets.com/2011/hacking-hacker-news-headlines/#comment-73 anil patwardhan Thu, 08 Sep 2011 06:40:23 +0000 http://metamarketsgroup.com/blog/?p=256#comment-73 20 Bootstrap replicates to estimate significance of covariates? Wrong use of the bootstrap, which is meant to give you a measure of confidence (intervals) of a single point estimate. Use a Liklihood ratio test with significance adjusted after permutation-based resampling. in anycase 20 reps is way too few.. ..also you can summarize your model more efficiently with AUC, rather than separate out sens/spec...since you not working with disease(e.g. cancer) I assume you have no tradeoff rationale for the importance of sensitivity vs. specificity... I assume you are using Friedmans or [Tutz and Binders] Boosting algorithm...I would be interested in how it compared to the Lasso path solution...both would give you strong regularization? Strong regularization doesnt necessarily mean your getting rid of spurious features ...just that you are emphasizing parsimony... impressive stuff on the data capture end...way out of my league... 20 Bootstrap replicates to estimate significance of covariates? Wrong use of the bootstrap, which is meant to give you a measure of confidence (intervals) of a single point estimate. Use a Liklihood ratio test with significance adjusted after permutation-based resampling.

in anycase 20 reps is way too few..

..also you can summarize your model more efficiently with AUC, rather than separate out sens/spec...since you not working with disease(e.g. cancer) I assume you have no tradeoff rationale for the importance of sensitivity vs. specificity...

I assume you are using Friedmans or [Tutz and Binders] Boosting algorithm...I would be interested in how it compared to the Lasso path solution...both would give you strong regularization? Strong regularization doesnt necessarily mean your getting rid of spurious features ...just that you are emphasizing parsimony...

impressive stuff on the data capture end...way out of my league...

]]> Comment on Hacking Hacker News Headlines by hyderali http://metamarkets.com/2011/hacking-hacker-news-headlines/#comment-72 hyderali Sat, 03 Sep 2011 10:01:53 +0000 http://metamarketsgroup.com/blog/?p=256#comment-72 What is this n-gram? Care to explain in simple meaning. What is this n-gram? Care to explain in simple meaning.

]]>