Top Tech Writers' Reader Quality Score

Smartsheet customers and prospects are easy to describe – they are the conscientious achievers, the thoughtful, hardworking get-it-done types that companies depend on. But where can we find them? What do they read? Who do they follow? Which blogs do they comment on? To find out, we decided to run an experiment using Smartsourcing (Smartsheet + Mechanical Turk) If we have accurately defined their personality (conscientious achievers), then we can expect that they would express it in their interactions socially on blogs.

Our assumption: Smartsheet’s target customers make thoughtful comments on the blogs they read.

One year ago, TechCrunch published their ranking for the top technology writers. Their criteria were based “on the total number of headlines they have had on TechMeme from January 1, 2008 until today (April 20, 2008).” They acknowledge that this was likely an imperfect slice of the data, but it does beg the question as to what is a good measure of journalist ranking. Google gives credit to site links and comment traffic. Are those good measures? Hard to say, but none of these tells us much about the audience.

We decided to re-rank these journalists based on a subjective grading of their readers’ comments. While this is also fraught with statistical land mines, it offers one window into the general character of their readership – essentially, their Reader Quality Score.  Are their readers more likely to be Conscientious Achievers or a Peanut Gallery.

How we went about this:

1. We took the top 25 journalists from the TechCrunch ranking and added 6 more we like (unranked)
2. We used Smartsourcing to get a list of their 10 most recent articles (in the publication credited)
3. For each of the resulting 310 articles we Smartsourced 10 people for each article to grade the substance and thoughtfulness of the average comment in the post on a scale of 0 to 10 along with a reason for their ranking.
4. We averaged the answers (after removing the survey results that did not include a meaningful explanation for their chosen score) by article to get a Reader Quality Score
5. We then averaged the Reader Quality Scores across articles to get the journalist’s total RQS
6. We then re-ordered according to the RQS

Here are the results:

*The following journalists were not ranked due to insufficient comment volume or survey results - Mathew Ingram, Joseph Weisenthal, Arn, Kara Swisher, Rafat Ali, Anthony Ha

We’re very interested in your thoughts on this research, and are wide open to sending this, or any other similar list, back through for other data (e.g. we have another run going now that is asking for an impression of how productive those commenting are at work given the nature of their comments). Perhaps the folks at Dolores Labs or Andy Baio at waxy.org or Panos Ipeirotis have some thoughts? We’re big fans of their generally brilliant analytics work.

You are more than welcome to the full data set and the research format and terminology we used (this link will put you into a Smartsheet folder containing the raw data - it’s free). Simply open it up within Smartsheet and either Smartsource some of your own data (not free), or export it to Excel for your own use.

-Brent

A little more data for those who are interested: