In September we hosted the first major online paid crowdsourcing panel discussion coinciding with the release of our Paid Crowdsourcing Research Report. It was chock full of great insights from the early pioneers in crowdsourcing thought leadership and technology. We paid $51 to the crowd behind CastingWords to transcribe the contents of the 1 hour and 7 minute discussion.

Moderator: Marshall Kirkpatrick – ReadWriteWeb
Panelists: Jeff Howe, John Winsor, Eckart Walther, Brent Frei, Lukas Biewald

This extract from the discussion contains particularly interesting views on whether paid crowdsourcing is exploitive of the workers and whether is sends work offshore.

Marshall Kirkpatrick: … The two other issues that sometimes get raised, are first whether this kind of arrangement [paid crowdsourcing] is exploitive of the workers? I suspect that in some cases that may be the case, though that may not be the case in all. Second, whether this is just going to facilitate the larger trend of outsourcing that's going to undercut jobs here in the United States.

I know that the white paper offered two really good responses to those concerns. The first being that crowdsource workers are not being exploited, but rather really appreciate being in control of their work experience and being able to work at home, and as well as in many cases just supplementing other income.

Then the second critique was responded to in the white paper by saying that traditionally all kinds of inefficient industries have complained loudly while they're being beaten, but that doesn't necessarily mean that we shouldn't choose the more efficient option.

I wonder if our panelists could share a thought or two. Do you buy those explanations? Are there other explanations that you offer when people raise concerns like that?

Lukas Biewald: I thought there was a really interesting study. Someone, I think it was a research at UC Irvine, was asking the Turkers how happy they were? How they felt about these questions? And it seemed like the people on Mechanical Turk actually had a really positive feeling about Mechanical Turk and really liked the way they could work from home and work on their own terms and work for whatever length of time they want. Even though I think it's pretty clear that people on Mechanical Turk are making really, really low wages.

Brent Frei: A good part of the research on inefficient markets is actually spawned by a couple of articles, one of which is Jeff's where he pointed out the resistance of the graphics industry or parts of the graphics industry to spec work. It turns out that one of the easiest things to crowdsource or to cloudsource is simple graphics work. Design me a Website look and feel. Or, create a logo for my company. Because it's easy to define, pretty easy to understand what it is that's being expected, and it's about the right size for a single project. There's a whole bunch of motivations for a big industry of people that want to win clients. It's a great way for them to try to pitch their business, rather than having to saddle up their bag and go and sell door to door to companies that need that service.

I had done a lot of research on inefficient markets in other industries. It was just a striking comparison to the resistance that has happened in the real estate industry where discount brokers or online services like Redfin that will help you sell your house for less than the 6 percent commission, just got amazing push back. And legislative push back even, as states and legislators sided with the real estate industry to push back on that efficiency.

The unfortunate thing is that it only hurts the consumers and it actually even retards the growth of the market. Stockbrokers, if you had to buy stock through a stockbroker, for example, then there would be far fewer people that would be invested in the stock market. Today, the number of investors and the size of the investment options and professionals is way larger than if we were forced through just this one little inefficient eye hole in a market.

I think that we're going to find that as different parts of work that can be crowdsourced are tightened up, where it's easier to submit it and get the expected results back, you will find that the old established markets resisting it.

John Winsor: I just want to add, because we've been in the middle of that with our crowdsourcing the Brammo logo. I've been giving a lot of speeches and writing a lot in my blog about it. I find it really interesting, and I've told a lot of graphic designers that I find it really interesting, that they're upset. I tell a story of my own entrepreneurial past. In 1986, I was able to start a magazine called "Rocky Mountain Sports." The only reason I was able to do that is that I could buy a MacPlus and a LaserWriter for $23,000 and the guys from Quark were kind enough to give me a beta of their software to do typeset coding on the Macintosh. But with that $23,000 investment, I displaced a $35,000 typesetting bill on an annual basis. At the time, the typesetters would look at my magazine and say, "aw, the leading's horrible and the kerning's awful. This whole desktop publishing thing will never work. It'll never happen." And really were angry at me.

It was because of the desktop revolution that a lot of graphic designers with Photoshop and Illustrator are in business today. They weren't traditional typesetters. They weren't traditional graphic artists with a pen knife and sitting at their desks.

I find there's something deeper than the crowdsourcing thing. It's the technology that drives these evolutions and these revolutions. It's inevitable that change happens. New industries are going to be organized. New ways of working are going to be organized and old ones are going to go away. That's just the way the world works.

Jeff Howe: I feel like I'm on both sides of this. I sometimes have joked to people that in writing about crowdsourcing, I've been documenting my own obsolescence. On one hand, I've made hay off of this, right? I mean it got me an article in "Wired," a book and I go out and talk about it and stuff. But at the same time, I'm watching as the same phenomena affects journalism. And I'm with John, in my book, I use the desktop publishing revolution. I call it the template. And really, of course, the template was like firearms or buggy whip, these other periods of technological disruption where you see whole industries put out of work.

I'm not sure, I think it was Brent who said it, which is that the danger of, unfortunately politicians are susceptible to this through efforts by lobbying, but the danger of kowtowing to established practices is you wind up shrinking the market.

Stock photography is a really fascinating case study that I would steer people towards if they're really interested in this issue. On the one hand, there really is no question but that it severely impaired the ability for a lot of photographers to make a living wage. It reduced the cost of the stock image by 99 percent. I mean incredible reduction in cost.

But at the same time, it catered to an exploding demand for these images. One data set that has not been produced, as far as I know and I'm almost positive that that's the case, is how well established stock photographers have done through Microstocks or call it crowdsourcing sites like iStockphoto and Shutterfly and, what is it, dreamstime? I'm sorry, I forget. There are a bunch of competitors to iStockphoto. While they built their business by using amateur photographers and paying them very little, some professional photographers realized hey, I just as professional graphic designers are using sites like crowdSPRING and 99designs are realizing that this makes sense for my business. It's a different business model, but I can use it.

For more data on this topic, see my post on the views of Crowdworkers themselves.

The following is the discussion in its entirety.

Paid Crowdsourcing Webinar – September 2009

Phone Announcer: The broadcast is now starting. All attendees are in "Listen Only" mode.

Brent Frei: All right. Well, welcome, everyone, to our Paid Crowdsourcing expert panel discussion today. Getting work done online has been growing significantly over the past 10 years, and there have been a large number of new vendors that have entered the market. In the past three years, they, along with the veterans in the industry, have shipped over a half a billion dollars to as many as two million online workers. Yet, as a person or company that's interested in utilizing these services to increase productivity, there's very little information available to guide you in understanding what's possible, where and who to start with, and knowing how to get the results you expect. So this panel is intended to help shed light on the paid crowdsourcing industry and how and when it will enter the mainstream business operations budgets. With that, I'll hand the session off to Marshall Kirkpatrick, the V.P. of Content Development and lead writer at ReadWriteWeb. Marshall.

Marshall Kirkpatrick: Thanks a bunch, Brent. This is Marsh Kirkpatrick. That was Brent Frei, the founder of SmartSheet. We have an awesome panel to address all of your burning questions about paid crowdsourcing. I'll introduce the panelists in a minute. If you would like to submit questions via the chat function in the Webinar viewer that you're looking at right now, feel free to. Hopefully, we'll be able to get to those questions. If anyone hasn't, we have a substantial audience out there today, which is great, and you all should have access to the paid crowdsourcing white paper that you got access to when you registered, and all of us on the panel would like to suggest that that white paper is really worth your time to read. We were just talking about how impressed we were with it. Lots of detail, and it'll help you make more informed decisions.

So, with that, I'd like to introduce our panelists, and then we'll jump into a number of questions that are based in large part on the hard data that you'll find in that white paper. So, my name is Marshall Kirkpatrick. I'm the moderator, and as Brent said, I'm the lead writer at ReadWriteWeb.com, a technology blog that I hope you'll stop by and check out.

Our first panelist is also a writer. Mr. Jeff Howe is a contributing editor at "Wired" magazine and the author of the book called, "The Rise of CrowdSoucing." He's the recipient of a Nieman Fellowhship from Harvard, and Nieman is a wonderful organization if you're interested in the future of journalism and some smart tech savvy folks. They're good to check out.

Our next panelist is John Winsor, who is the VP and Executive Director of Strategy and Product Innovation at Crispin Porter. He is the author of, "Baked In, " and "Spark, " and he's the founder of Radar Communications. John has been doing work with Crispin Porter's clients to get them on board the crowdsourcing train in some smart ways. He's got a lot to contribute to the conversation.

Next is Lukas Biewald. He is the founder of Dolores Labs, and just had a product called, "CrowdFlower," launch at the TechCrunch50 event this week to great acclaim. CrowdFlower is one of two services that are represented here that builds value on top of the Amazon Mechanical Turk platform. You can use them instead of Mechanical Turk directly and get all kinds of benefits from that.

Next panelist is Brent Frei. Brent is the founder Smartsheet, a company that put together the white paper, and who's Website you're looking at now.

And finally, we've got Eckart Walther, who is the Senior Vice President of LiveOps Marketplace at LiveWork, as well as the former head of Yahoo Search, Answers, and Delicious. I've always felt that Delicious is a secret weapon in any consumer market crowdsourcing strategy, so it'll be interesting to hear further detail about what Eckart is working on now in a business context.

So, those are our panelists. An impressive group. But I want to start with a light weight question that will be fun but instructive as well.

The white paper that we read mentioned that the vast majority of work that is run through these paid crowdsourcing systems really can't be considered fun for the workers. So all the systems come up, instead, with other forms of motivation: obviously pay, personal recognition, point systems, things like that.

But I'm sure that attendees of this call know that there is a lot of fun that can be had for the users of the systems, on the other end, once we really get a handle on using the systems effectively. And so I wanted to ask our panelists what some of their favorite use cases are, what some of the "wow" stories that they have come across are in companies using paid crowdsourcing. Perhaps that can work well to introduce some thoughts on what kinds of business processes can best be performed using crowd paid sourcing.

So, with all that introduction, anyone who has a favorite story they want to start out with for compelling use cases, why don't you let us know whose voice we're listening to and start the conversation?

Lukas Biewald: So this is Lukas. And I can just jump in with some fun use cases. Most of the things that people are actually doing through CrowdFlower right now are not that exciting, like filtering obscene images and looking for copyright violations. But we have done some fun cases that are on our blog. One of the ones that seemed the most compelling was we took people on Mechanical Turk and we showed them blocks of colors, and with very little prompting we asked them, "what would you call this color?" So you could say "blue," or you could say "cerulean," or you could say "sky blue." There was very little direction. We actually got thousands of people to answer the questions.

The people on Mechanical Turk really loved doing this task. We collected a huge data set of what people would call different colors, and we could actually see, for example, like lavender. People have a very clear idea what that is. But something like fuchsia has a much wider spread among what colors people associate with that.

Actually, an artist took our data set and made this really beautiful, kind of tagged color cloud that showed all the different colors people sent. Then another guy, a student at Stanford I think, took the same data set and made a tool for colorblind people, where you could hover your mouse over a length of your screen and it would say, "here's the kind of colloquial term that people would call this particular color that you're pointing at."

We did it purely for fun and to test our system, and I was really pleasantly surprised that some useful applications came out of it.

Jeff Howe: I'll hop off of Lukas' comments, just to say because it's a little out there, and we'll probably talk more about commercial applications for the rest of the call. I coined this term, "crowdsourcing," in a 2006 article, and the day it went live on Wired.com, and it just happened that that day, Aaron Koblin released, it was his MFA project at UCLA, 10,000 Sheep, which he had done on Mechanical Turk, where he got 10,000 people to draw a little sheep. It's a wonderful project. He also had 1, 000 people sing "Bicycle Built for Two," and then the combination sounds like something out of a Kubrick movie. It's amazing.

So I really think that one aspect that doesn't get a lot of attention, and I think that we're just beginning to tap into, is the degree to which Mechanical Turk and other sorts of networks of people are tuned in and willing to lend their labor, is its use for academic research, artistic purposes. I think it's a wonderful ethnography tool. I'll forget the link to this, but someone had paid Turkers like five cents or something to submit a photograph of themselves holding a sign explaining just why in God's name they were spending their time at these very low paid tasks. It's sort of the opposite of Brent's wonderful white paper, which I'll echo Marshall and say everyone should read. Fascinating! Sort of more of an impressionistic data collection.

So, it's great for translation. I had 50 hours of interview for my book that came out last year done through CastingWords. But it also has a future on the other end of the spectrum for artistic academic purposes.

Marshall: So, this is Marshall here. That link that you're referring to was from Andy Baio, who is the former founder of www.upcoming.org, a Yahoo acquired social calendaring app, and he does a lot of really interesting experiments with the Mechanical Turk. If people Google the phrase "Why I Turk" you'll see Andy's article on the faces of Mechanical Turk. It really humanizes the people on the other end of this transaction.

Brent: Thanks, Marshall. Well, this is Brent.

Marshall: Yeah, sure.

Brent: This is Brent. To answer your questions about on the business side, we had a couple of interesting use cases. One was an event company wanted to get all kinds of logistics materials port a potties, chairs, tents set up for 10 locations for the Dalai Lama's visits. They didn't want to call every location and find who would provide all those services and for what cost. It was something that was very easy to do through the crowd. Similarly, we internally wanted to figure out which blogs actually had the most conscientious readers: people that were organized, productive. Those were the blogs that we wanted to target, because our product is about managing work and it's for organized, productive people. So we had the crowds read the comments on all the blogs and then rate the people by a couple of different dimensions and give us a reasoning for why they rated them. The comments that came back from the people that were reading the comments on the blogs, they in themselves could probably be a book. So, it was very fun.

Eckart Walther: It's Eckart Walther, from LiveOps. Actually Andy Baio and I have worked together. We actually met at boot camp when we acquired a company. He's a great at Yahoo. LiveOps does a little bit more commercial work instead of doing mission critical crowdsourcing. So, I think, one episode that both actually made the work very pleasurable for the workers as well as great for us is when Hurricane Katrina struck. A lot of things were going on at the time. There was this unprecedented load on the infrastructure. The Red Cross really needed to set up, if you remember people needed these hot lines for people to call.

So, we crowdsourced a call center of about 400 folks who were on duty all the time and managed to set up a 400 person call center in about three hours. So, within three hours of the emergency going out, basically, the entire community came together, trained themselves really quickly on how to use the tools. We crowdsourced that work and got up in three hours and tore down about two weeks after really most of the stuff had been done for Katrina.

I think this is one of those types where both the workers actually really liked the work and as things get more complex, I think there's a lot of pride in that and felt they were doing something fun. At the same time for us, it was just amazing to see how really a community of people came together to do something pretty mission critical which was helping the survivors of Katrina to reconnect.

Marshall: Hmm. That's great.

Marshall: John, have we heard a favorite use case from you yet?

John Winsor: No, no. uh. We're in the middle of we just launched one of our projects for Brammo, one of our clients, to design a logo. It's been interesting, because we got a lot of heat for it. We talked about that before the call. It's been really fun to see people's reactions. I think a couple of things. One is in the design/marketing space, it's been really interesting for me to see. I wrote an article in "Business Week" in June. I would say, for the design/marketing space and probably in a basic space, about 80 percent of the folks were negative, back then, and 20 percent were positive. There have been a few new blog posts in the last week from myself and Edward Boches at Mullen and Alex Bogusky, a guy I work with. I'm amazed that it's flipped. That it's now 80 percent positive, 20 percent negative. Certainly the Brahma logo design caused a stir and it was really interesting to watch both positive and negative reactions.

One other thought: one of the things we talked about before the call, we just launched a new site today at www.CPBGroup.com and on the site we have a freelance button. So, we're going to start putting projects up from our clients to crowdsource or pointing people to Crowdsource Group. For Brammo, we use crowdSPRING. But one of the things we find you know we have a Twitter feed our new site is just essentially an aggregator of everything people are saying about CPBG. One of the things that a lot of people thought as a negative, because we're trying to be radically transparent and we have a Twitter feed on our home page, there were lots of negative comments about the Brahma project. We thought it was interesting. From our perspective is that those conversations are out there anyways. So, we might as well be a part of them, acknowledge them, and engage in them. Because that's the way the market's going versus trying to hide that conversation that's going on and yet try to push the disruption. Anyway, I think it was an interesting case.

Marshall: Yeah. That sounds good. Speaking of a conversation that's going on. We're getting good feedback on Twitter. People are enjoying the conversation so far. If they've enjoyed the fun stories, let's dive a little deeper and see if they'll stay with us. One of the things I wanted to ask the panelists about was user experience for their customers using paid crowdsourcing software. That's one of the things that was identified in the white paper as an obstacle that needs to be overcome. I wonder if our panelists could share some of your favorite stories or thoughts about things that you've seen vendors do that have made it easier for their customers to use this kind of software.

Brent: This is Brent. I've talked to customers of a lot of the vendors. There were a couple of common themes and they're represented in the white paper. One was just helping them set it up. They weren't really clear about how to get the work done in an appropriate way. So, to the extent that vendors can make that a seamless process, a wizard, if you will, not any more complex than ordering a book on Amazon, that would be very helpful. That would go a long way towards getting people to adopt. And then the second one that I'd highlight of the five was getting the expected results. So, if you do a good job of helping them set it up, you have a higher probability of getting good results coming out the other end. But there are people doing them, so there is a probability of, Lukas mentioned before, that what comes back isn't exactly as you expected or maybe wasn't done by a professional. So to the extent that the vendors can help screen the people who are doing the work, can help provide tools that seamlessly, not with a lot of oversight from the person asking for the work, help vet the quality to then get the result, makes it a tool that's more useful than simply, say, hiring an intern or a traditional contractor that would do that type of work.

Lukas: So this is Lukas. Just to kind of continue a little bit what Brent was saying, we think quality is the most important thing to make the user experience good for customers. When you have 10 people doing a 1,000 tasks, you can actually go in and check what each of those 10 people did. Right? If you hired 10 interns. But if you switched it up and now you have a 1,000 people doing 10 tasks each for you, it's really hard to spot check. You never have organizations where there's 1,000 people without any middle management, but that's essentially what you're doing when you crowdsource these tasks. I think it's really crucial to give people tools to make sure that the results they get back are high quality. Obviously you can see that we are a vendor, as CrowdFlower glorified that's what we believe is the most important thing.

Marshall: Good. Perfect. Jeff? Oh, go ahead.

Brent: This is Brent. I was going to say that to Lukas' comment. You know, one aspect of crowdsourcing work done is: "I have a high volume sort of micro transaction or micro task type work." There's very little ways for me to get that. The other end of that is: "I have one task. I want you to build a logo," or "I want you to create a presentation," or "I want you to transcribe an audio into text. And it's just for this one thing that I want done." In that regard, you can vet it yourself. You just don't want to have to do that. So, we tend to always see the two ends of the spectrum, but the reality is work comes in all shapes and sizes.

One thing that we're finding is that the trend is toward the middle between the two examples that I just gave. One is, if it's a lot of high volume work then you're going to end up with tools like ours or CrowdFlower, where we have already sort of pre bundled the complex part of processing that into a tool that the user doesn't have to understand and then it handles all of that.

Or, if it's the other end, where we've matched an expert up with a single project, that there are a lot of logistics around, helping explain to the expert what exactly you want. Then coming up to a set of conclusions that are this is what we have to check off if you say that this acceptable and will pay for that work. So, there's no dispute at the end. The more that we get that, then the more it is simply a black box.

So, those two things are coming together into a user experience that if I don't really have to understand all that stuff that the vendors do. I just put work in and I get answers out.

Marshall: That's great. Jeff, I wonder, as someone's who's been watching the space since you coined the phrase, are there any particular innovations that catch your eye around user experience that make the software more compelling?

Jeff: Well, it's interesting. Yeah, absolutely. For one, the emergence of vendors, period, of a middleman between a Mechanical Turk. While there were a few that were just starting up when I wrote the article three years ago. It was by no means well understood that Mechanical Turk would need that. It was by no means well understood that a lot of what the crowd was going to do would require a filter. We knew that there was this enormous amount of potential labor on one side and an enormous demand on the other. We've begun solving the considerable problem of how those two intersect. That's been really interesting to watch develop. I don't know if there's any specific software innovation or anything that I would list. It's more of the emergence of a best practices around this sector of crowdsourcing.

Marshall: I'm not sure if it was you or it was someone else before the call who referred to the Mechanical Turk and systems like that as the operating system and all these vendors acting like the software then that now runs on top of that operating system.

Jeff: It wasn't me. That was before I logged on, but that's a wonderful metaphor. I think that's just apt. That's perfect.

Eckart: That was actually me. And maybe to build on that to repeat this. In many ways, right, we have an operating system such as OneInc and really the large pools of workers with reputation and the like. LiveOps really views that more as the operating system. In terms of the user experience, we've seen the first phase of crowdsourcing where we take these relatively simple tasks, image review and the like. But in many ways, the users of these systems as it becomes more mainstream don't really care about crowdsourcing per se. They look for a solution. And the more that solution looks familiar to them, the easier it is for them to use. The point I was making is: as we get to these more complex work types, for example, we do customer care, we do health nurse. The key in the user experience is to really build these vertical solutions on top of the operating system that matches the work type that the user has.

For example, if you want do customer care, you might have to connect to sort of the standard customer care email software system. So if you wanted to do image review, there's sort of some standard tools to connect to. So, almost rather than bothering the client with the complexity of how the actual operating system works underneath, the quality management, the payment, the reputation, the more you can make it look like a standard outsourcing process for them, I think, the easier the experience gets for them.

Jeff: Was that Eckart I'm sorry.

Marshall: That was Eckart. Jeff: Yeah, boy, well said. That's a good point. I would just toss out in a way of echoing that that the "novelty" of crowdsourcing is, I think, itself a hindrance, because crowdsourcing describes process. I don't think users, that's to say customers, want to know about that. They just want to know that they can get a boatload of audio transcribed for 75 cents a word or less.

Eckart: Yeah, maybe just to add to that quickly, and not to hog the conversation. But we really thought a lot about this and we've been using the term cloudsourcing, actually, for our customers, because we're more on the business process side, really, which is outsourcing to the cloud. A lot of people think crowdsourcing's very technical, but when you talk to a company who may not have heard of Mechanical Turk or anybody else, this whole idea of you're outsourcing your applications into the cloud already with Google Apps and the like. Now you can start to outsource your work into the cloud, as well, which we call outsourcing to the cloud, cloudsourcing.

We've gotten pretty good resonance of people sort of focusing on the benefits, which is, "I need on demand labor in the cloud and I don't really care what the algorithm is. I just need the work done." So, I totally agree with you on using the term crowdsourcing. It's been a little bit of a mixed blessing.

Marshall: So, this conversation flows well into another question that I wanted to ask, that I presume that a number of our listeners will be interested in. We're all excited about this. It's a little on the geeky side. I can imagine early adopter types being the ones who want to bring this into their companies, in some cases at least. I wonder if the panelists can share thoughts about how people interested in using this kind of tool can most effectively build support for upper management buy in.

Lukas: I can probably speak to the business solutions. We try to sell our solution to big companies all the time. So we have a lot of experience in trying to convince management to adopt these kind of practices. I've found that the most effective way to do it is to just show end results. Right? People look at this kind of solution and they say, "It doesn't seem right to me that you can actually get high quality results from a large pool of casual workers." But when you show them the results and you say, "Hey, look, compare the results that our system did to the results of your own annotators or workers or whatever for the task," that's a really compelling story.

Brent: This is Brent. I agree. Depending on what kind of work it is, it's often extremely inexpensive to try. So, a lot of people can try it in the budgets that they control. So, they're a marketing manager or they're a purchasing manager or they're an events organizer, et cetera. So, they can spend 25 bucks just giving it a shot. When the results speak for themselves, then it's a very easy sell. A lot of times, they don't even have to sell it. It's just part of their operating budget.

Eckart: Yeah, I think this is Eckart from LiveOps again. I think to mirror that, I think there's really two, sort of, use cases. One of them is I think that crowdsourcing enables companies to do things they just couldn't do before. You can't really assemble several thousand people for a couple of hours. And in those kind of cases, I think, folks really see that as the alternative and they're pretty willing to try it. As Lukas was saying, there's some doubt about it, but at the end, if there isn't an alternative, that's really easy.

If you go to more mainstream work, again, I think at LiveOps, our crowd competes all the time with full time call centers. These are people dedicated working full time. And using our quality algorithms, we provably basically can create better work quality head to head. So a self organizing crowd of people provably, because our industry is very metrics oriented, just performs better than a lot of full time workforces. When you can show that right then at the end, what the technology is behind that, whether it's called crowdsourcing or cloudsourcing,doesn't really matter to their client. I mean, you just deliver a better result. I think, as Lukas was saying, it's hard to really argue with that.

Jeff: I'll jump in. They're not represented on this call, but they're a fascinating company. It's TopCoder. Just really quickly, TopCoder was started, as you would guess from the name, as a community of coders. Jack Hughes, who set it up, very consciously wanted to improve those metrics around programming and so he set it up as a contest. Really the revenue model in the beginning was just that companies, be it Intel, Yahoo, or Google, would sponsor the contest. The quid to the pro quo was that they would then be able to get the winners and hire the winners. So, it was kind of a farm team for them or a recruitment strategy. But Jack knew all along that he wanted to sub out, what had begun as academic contests or as sort of non live contests, and actually start subbing in work for actual clients, programming work for clients. I can't say I've seen his balance sheets, although I have had pretty detailed discussions with him about the financials. They've done very well with exactly the strategy that Eckart is saying.

They're able to go into prospective clients' offices and show them that they have 50, 70, 80 percent fewer bugs per line of code than simply outsourcing a coding project to their competitors, essentially. Again, you're seeing this through a self organizing community, which is... I think in many ways, that's the root of the innovation. It sounds hyperbolic, but really what I think of as the revolutionary aspect of crowdsourcing, is that instead of a top down managerial hierarchy, you have a self organizing community comprising the work force.

Marshall: Very cool. So, as this people are engaging in mainstream work using this kind of software. One question that comes up came from an attendee pre submitted question, something that I wouldn't have thought of, but Brent has told me is a relevant question is: legislative concerns. One attendee asks, "Are there legislative developments or concerns regarding paid crowdsourcing labor?" [silence]

If anyone's got any thoughts on that, that would be great. And if attendees have questions, please feel free to submit them through the chat interface and we may very well get to discuss them when ...

Jeff: Is it legislative or regulatory? I mean there's overlap between the two. Brent, why don't you explain what those are? As a non vendor, I don't I'll be honest, it's not an issue I've covered.

Brent: Yeah, this is Brent. I'm not the expert on this. Other than the fact that I considered it a lot in the research paper relative to the work submitter's need to know anything about providing 1099s, so tax forms, to the workers. It turns out that if you pay any one worker over $600, then you have to send them a 1099. There are some vendors that do it for you. So, they take care of all that. And there are some that don't. This is just one of those hurdles that will make it a little bit more difficult to do crowdsourced or crowdsourced type work if we have to incorporate that into what we do. I can assure you that a marketing manager or an IT person doing a big run of data, they don't really want to have to deal with the tax implications.

Jeff: Well, let me just add something. If they're writing a check to a LiveOps or a Smartsheets or Crowdflower, isn't it you that are in effect paying the Turker, the worker?

Brent: Well so in our case, it is. We take care of that. We track that and we manage that, because Amazon does not manage that.

Jeff: Right.

Brent: There are vendors that do manage it out there. They do assure you that if you go over that number, that they will send the 1099 on your behalf. So, they've set that piece up to take that hurdle away from you or one of them.

Lukas: Sorry.

Eckart: Go ahead.

Jeff: I was going to say, at any rate, I think the bottom line is that one sort of theme that I've had in my talks and that I bring up in the book, and especially in my original article, I just looked at the cost despair and said, "Oh, my gosh! You're paying 10% of what you might otherwise through a traditional outsourcing or hiring employees, or whatnot." As I began blogging about it in the intervening years, it became clearer as I did more anecdotal research, that the administrative costs can really add to that and bring it closer to traditional forms of procuring labor.

Eckart: Yeah. So, we have a lot of experience with this, because I think at LiveOps, we do a little bit more complex work. Right now, I think crowdsourcing is still a little bit of a technology niche, despite the volume, but for example, if you look at Elance or oDesk or those markets, there really are people who have careers. I mean, they make a full time living there. I mean our vision for crowdsourcing or really cloudsourcing is that it is a much larger fundamental shift of how work gets done within a corporation, and normal work really gets pushed from the hubs out to the workers and the edges.

So, as we start viewing crowdsourcing as potentially full time careers for people, a lot of other issues kick in; how do they pay healthcare? How does the tax payment work? How do you help them incorporate an independent corporation?

There's this perception with crowdsourcing right now "it's sort of fun as a student, " but at the end if you take this out 5 10 years, how much work would be outsourced to full time virtual workers or virtual workforces?

So, we've thought a lot about helping people to create small virtual corporations, where they can create their own teams. If you're really good at something, if you're really good at customer care, if you're really good at content review, if you're really good at some of these other things, why not help people and really build a business around that if the work is there?

I think we're seeing this very, very early adoption mode right now, but if crowdsourcing and virtual work becomes sort of the future of work, at least the future of outsourcing, it's a huge space. I think a lot of the regulatory environments today are really not setup to support that in an easy way, and we deal with those challenges everyday.

Marshall: Marshall here. I think that raises some other issues or is a convenient pointer to some of the other controversies that produce around this field. We've talked about the quality of the work issue. It sounds like everyone has taken steps to deal with that to varying degrees. There are these regulatory and legislative issues. The two other issues that sometimes get raised, are first whether this kind of arrangement is exploitive of the workers? I suspect that in some cases that may be the case, though that may not be the case in all. Second, whether this is just going to facilitate the larger trend of outsourcing that's going to undercut jobs here in the United States.

I know that the white paper offered two really good responses to those concerns. The first being that crowdsource workers are not being exploited, but rather really appreciate being in control of their work experience and being able to work at home, and as well as in many cases just supplementing other income.

Then the second critique was responded to in the white paper by saying that traditionally all kinds of inefficient industries have complained loudly while they're being beaten, but that doesn't necessarily mean that we shouldn't choose the more efficient option.

I wonder if our panelists could share a thought or two. Do you buy those explanations? Are there other explanations that you offer when people raise concerns like that?

Lukas: I thought there was a really interesting study. Someone, I think it was a research at UC Irvine, was asking the Turkers how happy they were? How they felt about these questions? And it seemed like the people on Mechanical Turk actually had a really positive feeling about Mechanical Turk and really liked the way they could work from home and work on their own terms and work for whatever length of time they want. Even though I think it's pretty clear that people on Mechanical Turk are making really, really low wages.

Brent: This is Brent. A good part of the research on inefficient markets is actually spawned by a couple of articles, one of which is Jeff's where he pointed out the resistance of the graphics industry or parts of the graphics industry to spec work. It turns out that one of the easiest things to crowdsource or to cloudsource is simple graphics work. Design me a Website look and feel. Or, create a logo for my company. Because it's easy to define, pretty easy to understand what it is that's being expected, and it's about the right size for a single project. There's a whole bunch of motivations for a big industry of people that they want to win clients. So, it's a great way for them to try to pitch their business, rather than having to saddle up their bag and go and sell door to door to companies that need that service.

I had done a lot of research on inefficient markets in other industries. It was just a striking comparison to the resistance that has happened in the real estate industry where discount brokers or online services like Redfin that will help you sell your house for less than the 6 percent commission, just got amazing push back. And legislative push back even, as states and legislators sided with the real estate industry to push back on that efficiency.

The unfortunate thing is that it only hurts the consumers and it actually even retards the growth of the market. Stockbrokers, if you had to buy stock through a stockbroker, for example, then there would be far fewer people that would be invested in the stock market. Today, the number of investors and the size of the investment options and professionals is way larger than if we were forced through just this one little inefficient eye hole in a market.

I think that we're going to find that as different parts of work that can be crowdsourced are tightened up, where it's easier to submit it and get the expected results back, you will find that the old established markets resisting it.

John: Yeah, this is John. I just want to add, because we've been in the middle of that with our crowdsourcing the Brammo logo. I've been giving a lot of speeches and writing a lot in my blog about it. I find it really interesting, and I've told a lot of graphic designers that I find it really interesting, that they're upset. I tell a story of my own entrepreneurial past. In 1986, I was able to start a magazine called "Rocky Mountain Sports." The only reason I was able to do that is that I could buy a MacPlus and a LaserWriter for $23,000 and the guys from Quark were kind enough to give me a beta of their software to do typeset coding on the Macintosh. But with that $23,000 investment, I displaced a $35,000 typesetting bill on an annual basis. At the time, the typesetters would look at my magazine and say, "aw, the leading's horrible and the kerning's awful. This whole desktop publishing thing will never work. It'll never happen." And really were angry at me.

It was because of the desktop revolution that a lot of graphic designers with Photoshop and Illustrator are in business today. They weren't traditional typesetters. They weren't traditional graphic artists with a pen knife and sitting at their desks.

I find there's something deeper than the crowdsourcing thing. It's the technology that drives these evolutions and these revolutions. It's inevitable that change happens. New industries are going to be organized. New ways of working are going to be organized and old ones are going to go away. That's just the way the world works.

Jeff: Yeah. This is Jeff. I feel like I'm on both sides of this. I sometimes have joked to people that in writing about crowdsourcing, I've been documenting my own obsolescence. On one hand, I've made hay off of this, right? I mean it got me an article in "Wired," a book and I go out and talk about it and stuff. But at the same time, I'm watching as the same phenomena affects journalism. And I'm with John, in my book, I use the desktop publishing revolution. I call it the template. And really, of course, the template was like firearms or buggy whip, these other periods of technological disruption where you see whole industries put out of work.

I'm not sure, I think it was Brent who said it, which is that the danger of, unfortunately politicians are susceptible to this through efforts by lobbying, but the danger of kowtowing to established practices is you wind up shrinking the market.

Stock photography is a really fascinating case study that I would steer people towards if they're really interested in this issue. On the one hand, there really is no question but that it severely impaired the ability for a lot of photographers to make a living wage. It reduced the cost of the stock image by 99 percent. I mean incredible reduction in cost.

But at the same time, it catered to an exploding demand for these images. One data set that has not been produced, as far as I know and I'm almost positive that that's the case, is how well established stock photographers have done through Microstocks or call it crowdsourcing sites like iStockphoto and Shutterfly and, what is it, dreamstime? I'm sorry, I forget. There are a bunch of competitors to iStockphoto. While they built their business by using amateur photographers and paying them very little, some professional photographers realized hey, I just as professional graphic designers are using sites like crowdSPRING and 99designs are realizing that this makes sense for my business. It's a different business model, but I can use it.

Marshall: Makes sense. As we all know, the best thinkers about things are always ambivalent. It's definitely the flag you're flying. I noticed the Twitter handle you're flying is "@crowdsourcing."

Jeff: Yes.

Marshall: There you are, right in the middle of it. Oh, good. It sounds like our little blast from the past of the teletype machine is no longer with us if anyone else is hearing that beeping. So, audience questions. A participant in this call wants to know what is the trend on the cost of crowdsourcing? Anybody seen any indications of that?

Jeff: Brent, the vendors have the data. I'm curious, too.

Brent: This is Brent. It's both going up and down. It's going down as more competition is offering an opportunity to get the same work done for cheaper, but at the same time, people who have had a bad experience with the results on cheap solutions are now interested in paying a premium to whatever that cheap was, but still a big discount to the traditional alternative for a better result. So, more targeted. They would use CastingWords for transcription, for example. Or, they would use CrowdSifter for image review, because that's very specific to that. You still pay a little bit more than maybe if they went and used a very generic tool.

Panelist: There was

Panelist: Go ahead.

Panelist: Go ahead.

Jeff: No. No. You should. I've talked too much already.

Lukas: No worries. I mean so, I was just looking at the data for the average cost that we're paying for various tasks on Mechanical Turk. We are doing some stuff with LiveWork and some other companies, but we only have really historical data for Mechanical Turk. Actually the trend has been flat for the last year and a half that we've been putting tasks up. We haven't seen a major shift in the costs. Even though there's definitely more tasks on Mechanical Turk and more workers in the last year.

Marshall: Cool. As we're moving into the end of our hour here, I do have a couple other questions that I also wanted to make sure, in case anyone ends up having to go off the call early who's listening. I wanted to make sure everyone knew that if you're on Twitter, there is a wonderful opportunity to follow all of the panelists here. If you go to the link tinyurl.com/crowdwork, you will be delivered to a site where you can choose to follow all the panelists in one fell swoop thanks to Julie Craville from CrowdFlower. Julie's put together a list of the all the Twitter usernames, and I've just grabbed it and threw it into a platform that'll allow one click group follow. And all of our panelists are on Twitter, so if this is a topic of interest, go and join them in daily conversation over there. Keep an eye on what they're doing.

So, speaking of nutty new things like Twitter, I'm wondering when paid crowdsourcing guys get together over beers, when you're at the insider events, what are the kinds of things you guys like to talk about amongst yourselves as the far out leading edge possibilities for the future of this medium? Any thoughts? Cards held close to the chest?

Jeff: I started getting in Twitter fights, I'm sorry.

Brent: This is Brent. I think there probably aren't enough beers to really loosen it all the way up, but obviously we're working together. We see it as a huge opportunity, and in the early days, we're just kind of comparing notes on what our customers are doing, and can we do things together that will improve their experience. So we've hooked up to LiveWorks. We've talked extensively with, or met extensively, with CrowdFlower, Dolores Labs, because they, and also CastingWords, because they've really figured out a way to do some very specific things very well, and no need for us to solve that problem again. I think we're all looking for ways to make it more mainstream, because if it does, it will develop the market.

Eckart: Yes, I think there's a couple of things, this is Eckart from LiveOps, that really sort of excites me about the future in this. I think the first one is, we're still in sort of the early stages, like the operating system, where you almost have to hand code everything. The emergence of APIs and standard connections and things like that is really something where, we announced in API last week, but it's really just a first step of something much, much broader. And I'll have Lukas talk to this, but right now, it is very difficult from a client's perspective, to get the work to the crowd. There's a lot of custom solutions to build, and that's what makes it hard to use.

We've talked about vertical solutions being built, but again, like a good operating system, I think over time, I would really want to see applications already enabled be able to connect to crowds. So, for example, if you buy a customer care application, why can't you have a set of APIs and press a button, and say, "I'm going to have a big product release next week. I need several hundred people to be ready to deal with that."

So the way we do mash ups today, like we have mash ups between different data sources, and Google maps is a great mash up. I think in the future, we'll see a mash up between workforces and work, and if we can make that much more efficient by having standard API, I think we're potentially going to revolutionize how the entire world of work might work.

And you can start purchasing outsourced labor from very well established vendors who have built their own virtual businesses, but if all those do standard APIs and the integration costs becomes as simple as rolling out a Web application. So, I mean, that's obviously not out there this year or next year, but I think that sort of... when we think in the far future where we think this trend might be able to go.

Lukas: This is Lukas. I just wanted to reiterate some of the stuff Eckart was saying. I think there's lots and lots of interesting things that happen when the APIs kick in. People always say, "what are the kind of vertical applications that are going to happen in space?" There are certainly a lot of cool ones, but I think another big shift is going to happen when these crowds are easy to take advantage of on a case by case basis is, you'll see people outsourcing stuff that they've never outsourced before and doing kind of interesting things that they couldn't really do before.

I'm thinking our happiest customers are people that are using us in a way that there wasn't really a vertical application in that space. We have a customer that actually wants to get photos of locations around the world. So, they come to us, and we put the task up on a bunch of different labor pools, and people go out and take photographs of particular lat/long locations. They love that, because there's no way that they could have outsourced something like that before. And what's the market for that? It's not a good market. I don't even know what market you'd put that in, and nobody's ever going to build a vertical application just to look at that. But if you have tools that people can plug into, I think you'll start to see them creating new kinds of work that never existed before.

Marshall: That's fantastic. So, now, one question that the white paper offers a lot of answers to this question for, but I wonder if folks could, perhaps as a summary to round up, we could go around and share our thoughts on, what do you think it's going to take for this market to reach the tipping point and really become mainstream...for paid crowdsourcing to became a mainstream business activity?

Brent: Well, this Brent. I'll summarize what I had in the end of the paper, which I was simply... when it gets to, the eCommerce analogy is an excellent one, when we can go online and with not much more effort than ordering a product on Amazon, or shopping.com, or one of these sites, we can get work done, because someone else, as Lukas and Eckart have mentioned built all of the complex stuff in the background, then I think that's when it's mainstream. Everybody's uses it. You've gone into the big hump of the user base.

Marshall: Sounds good, sounds good. Jeff, have you got thoughts on that?

Jeff: I think it'll echo what other people are saying. But I think when a company comes out and achieves some level of market dominance and becomes a success story... I should say, using labor pools like what some of the vendors here are offering, then I think that you'll have that tipping point. I think that as all things it comes down to mass communication. I think once you have...right now, we are still dealing in a niche business practice, and not that many people know what crowdsourcing is, or if they do, what I find a lot, people have...it's appeared in their newspaper. They've read about it. They kind of get it, but they don't really know that much about it.

I think there'll be a single breakthrough company that will be the tipping, and it'll become standard business practice. I do still believe that will happen.

Marshall: Makes sense, makes sense. So, John over at Crispin Porter, you guys are already doing a bunch of crowdsourcing. It's probably a neat competitive advantage in your industry for now. How much longer have you got that competitive advantage, and what do you think's going to tip things to the point where that becomes a mainstream part of the service offerings of companies in your market?

John: Well, I think, it's interesting. I think there will be very few people that will follow us into this, because it's really destructive, right? I mean, Crispin, we have a thousand employees now, and it really threatens... there's an alternative. Do we give it to an internal team or an external team? I thought it was interesting that the Unilever brand manager talked about the Peperami brand that they just took away from Lowe's in the UK and started crowdsourcing the advertising. And his point of view was, "Jeez, I would write a brief, and the brief would end up being worked on by two creative teams of two, so four folks. And I want more participation. I want the ability to get a lot more input creatively." So, I think it very threatens the core of most of the advertising industry and the product design industry.

I think there are going to be a few companies like us that are willing to play with that, because we're willing to be disruptive and realize that the future is owned by the bold. But it's going to be really interesting in the next few years. I think there are going to be new companies that emerge that are more that they're new kinds of organizations that are smaller, that are a handful of really smart folks, the shapers and the meaning makers that can tap into the right crowd, help clients make meaning. So, it's a great time to be alive and have these great opportunities, because of the technological disruption.

Marshall: Sounds good. So, Lukas, as a start up in this space, what are you betting on in terms of the mainstream tipping point?

Lukas: Well, our strategy from the beginning was to get big companies using this for mission critical applications and showing them as examples of leaders in the space. We're still working on that and one of the frustrations that we have is that our best customers and best success stories don't want to tell their competitors about it. So, it's hard to trot them out as really great examples. I think this is inevitable and I feel like it's not really there's going to be a tipping point. I feel like different industries will adopt this over time. I'm kind of curious what Eckart has to say about this. Because in some sense, LiveOps has been doing crowdsourced call centers for a really long time. That seems like that industry has already taken off.

Marshall: Yep, I was going to say the same thing. Eckart, when it comes to mainstreaming this practice, sounds like you may already be well on your way there. What's that look like?

Eckart: I think it is a great question. I think there is a huge amount of education out there. Jeff Howe and others have been leading the charge. I think in the early days of the Internet, there was a lot of confusion of what the Internet was. It was going to be the answer to TV. It was going to be the answer to this. I don't know if you guys remember this, but as it started mainstreaming or browsers started mainstreaming with Netscape, there were a couple of events that really drove this. But they were not the technology, they were the need of that technology. I think we've been relatively successful in our industry, but not because people liked the technology. It is just because the technology produced actually better results than anybody else could do.

To me and that then becomes a business tool. I think was Lukas was saying is right on. I think that the tipping point is really when these technologies are easy enough to use to consistently produce better results than you can achieve with other mechanisms.

There's a lot of hype about things and this is cool and this is... But at the end, I think, the technologies that are cool and don't really produce pretty visible, sort of, differentiation in what people can do tend to sort of die on the vine versus the things that just make life better or easier or help people to save costs, to be more flexible doing work types in the work space at the end will grow.

I think it's a little bit up to the crowdsourcing vendors to sort of help shape the perception around it that it is... I thought the desktop publishing analogy was just great. What tipped desktop publishing? Well, the tools were there, but it's this long, long disruptive trend in how things happen and, you know, journalism is the same deal. There was something there and then blogging got mainstream and maybe there was an election that drove it.

But my sense is, for example, we mainstreamed it in one industry and no one really noticed and the reason is we just produced better results. I think at the end all the things that are mentioned on the white paper are true. It is about making it simpler to use. It is about standard APIs. But it is also about just producing better results and then things will happen pretty automatically.

Jeff: I would just quickly echo that. That this is something I say all the time, I just can't say it enough, is that crowdsourcing, in its essence, A) it predates technology entirely. Our technology, we think of it as the Internet, the Information Age. But B) even as much as it is an Information Age artifact, in its essence, it's not using anything more complex than what we had with Usenet groups. All it is is something that exploits many to many communication. It's just human behavior catching up to technology that began reaching ubiquity in the '80's and '90's. It just takes a long time for humans to figure out what they, the famous quote, "What God hath wrought."What they hath wrought. An analogy I like is film. It took us 20 years to figure out how we might use cinema. Edison frowned on the playing of music on phonographs. He thought he created a dictation machine. So, people are expecting crowdsourcing to take off next month. I think we'll be sorely disappointed. The Internet may still not be mature. It could be another generation before we've really figured out how to properly exploit these tools we've made.

Marshall: That makes sense. Hopefully in the meantime people who are hip to the scene will be able to leverage some of these many vendors that are breaking this new ground and gaining value even before this becomes a mainstream thing. I know we've got a number of questions from the audience about high value crowdsourcing projects. I'd suspect that that could do well as the subject of an entirely separate webinar on its own. I think we've focused real well today on analysis of the whole range of options and the conversation's been an excellent complement to the white paper which, again, can't recommend highly enough. That and this conversation work real well together. Given that it's just a few minutes after hour, I would like to wrap up the conversation, if we could. Brent, as the convener, does that work for you?

Brent: That's great.

Marshall: Sounds good. OK. Well, it's been an awesome panel. You can follow all of the panelists on Twitter, if you go to tinyurl.com/crowdwork. There you'll find everyone's profiles and get to join them in a daily conversation. We've had Jeff Howe, the Contributing Editor at Wired magazine and the man who coined the phrase crowdsourcing; John Winsor from Crispin Porter, who's bringing crowdsourcing into advertising/marketing; Lukas Biewald with the hot new software built on top of the Mechanical Turk platform called Crowdflower; and Eckart Walther from LiveOps, formerly of Yahoo; and Brent Frei, who is the founder of Smartsheet, the company that brought us all together and put together the wonderful white paper.

I'm Marshall Kirkpatrick, lead writer at www.readwriteweb.com, good tech blog for smart thinking people interested in future looking trends just like this.

Thanks a bunch everyone for coming and joining us. Thank you very much for all the panelists for such great conversation and let's keep this conversation going. Because I think this is definitely going to be a real potent trend for the future. [tone sounds]

Marshall: I'm not sure if we have anybody here. Brent do you have anything you'd like to say to close out the phone call, if we're still going?

Brent: I'd like to thank everybody very much, especially the panelists. If the panelists could stay on I'd just like to get your contact information to send you a little something.

Panelist: All right. I'm here. Thank you very much. Fascinating conversation and pleasure to meet you all virtually.

Brent: Yeah. Same here.

Panelist: Thanks guys.

Panelist: Thanks Marshall. Thanks Brent. Thanks everyone else.

Brent: You bet.

Panelist: Hey, I'll send you guys an email and thanks for coming.

Panelist: Sure, thank you.

Panelist: Bye.

Panelist: Bye everybody.

Panelist: Thank you much everybody. Panelist: Yeah, bye bye.

Transcription by CastingWords