Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

September 17 2010

CrowdConf explores distributed work

I had an interesting chat recently with CrowdFlower founder Lukas Biewald about CrowdConf, the first "distributed work conference," being held Oct. 4 in San Francisco.

For an introduction to the big ideas behind CrowdFlower, see Biewald's TEDxTalk, embedded below, or our recent interview here on Radar.

Who is CrowdConf for?

Lukas Biewald: It was really easy to get lots of venture capitalists and startups interested, but we're also interested in getting academic researchers, government officials and media. Everyone was talking about crowdsourcing at Gov 2.0 Expo, when we discussed using tech for civic good.

"Crowdsourcing" author Jeff Howe will be at CrowdConf, along with Tim Ferriss, David Alan Grier, people from the XPrize, and many other experts. And there will be robots. Telepresence robots.

Government agencies have had success with crowdsourcing in a number of areas, from Be A Martian to using grand challenges in open government. What other crowdsourcing applications could government employ?

LB: Consider the work of Ushahidi, or the work that my company did in Haiti translating text messages. How else can you provide such rapid access to potential volunteers? I also love the way that CrisisCommons involves people in crisis relief who want to help but don't know how.

You can also look at The Guardian's example, where they set up the means for people to go through the British government's expense reports.

You can see the Sunlight Foundation using volunteers in Transparency Corps to go through campaign data, like filtering fliers in tagging projects.

Think about DARPA's Grand Challenges. They got so much more than $1 million of research out of that investment. Major institutions worked on it, along with many amateurs. And then there's also the XPrize & Energy Department, which are teaming up on a 100 MPG car. [Winners of the energy efficient auto contest were announced yesterday.]

In the video below, Biewald joins Brian Herbert of Ushahidi, Robert Munro of FrontlineSMS and Leila Janah of Samasource to talk about how they deployed a critical emergency communications system after the earthquake in Haiti:

What does crowdsourcing offer academic researchers?

LB: This is a great way of doing research. Almost every grad student in the Stanford linguistics department is using crowdsourcing platforms for their studies. This is a really useful tool for social scientists or researchers in other areas to have access to data or tools they didn't have before.

There's also a whole genre of new research that is coming about through crowdsourcing efforts. For example, why does WIkipedia work where other models didn't? This work leaves such a big digital trail. There's a lot of data out there for people who are interested.

What are some of the issues around crowdsourcing?

LB: There are big questions here. The trend toward crowdsourcing work can cause a big drop in wages in some industries. It's outsourcing on steroids. Before, it could be hard for a startup to outsource a big part of its business. Now, firms like 99 Designs have moved almost all of their design work offshore. Is that fair?

What's happening there is happening across industries. Companies are outsourcing legal work, e-discovery, advertising and more. But this raises questions: When people do things for free, does that really work? When you pay in virtual currencies or reputations, how does that change things?

We'll explore these questions and others at the event.

This interview was condensed and edited.


August 30 2010

Four short links: 30 August 2010

  1. Free as in Smokescreen (Mike Shaver) -- H.264, one of the ways video can be delivered in HTML5, is covered by patents. This prevents Mozilla from shipping an H.264 player, which fragments web video. The MPEG LA group who manage the patents for H.264 did a great piece of PR bullshit, saying "this will be permanently royalty-free to consumers". This, in turn, triggered a wave of gleeful "yay, now we can use H.264!" around the web. Mike Shaver from Mozilla points out that the problem was never that users might be charged, but rather that the software producer would be charged. The situation today is just as it was last week: open source can't touch H.264 without inviting a patent lawsuit.
  2. Crowdsourcing for Pakistan Flood Relief -- Crowdflower are geocoding and translating news reports from the ground, building a map of real-time data so aid workers know where help is needed.
  3. Dirpy -- extract MP3 from YouTube. Very nice interface. (via holovaty on Delicious)
  4. Three Rules of Thumb for Bloom Filters -- Bloom filters are used in caches and other situations where you need fast lookup and can withstand the occasional false positive. 1: One byte per item in the input set gives about a 2% false positive rate. For more on Bloom Filters, see Maciej Ceglowski's introduction. (via Hacker News)

Sponsored post
soup-sponsored will be discontinued :(

Dear fans and users,
today, we have to share very sad news. will stop working in less than 10 days. :(
It's breaking our heart and we honestly tried whatever we could to keep the platform up and running. But the high costs and low revenue streams made it impossible to continue with it. We invested a lot of personal time and money to operate the platform, but when it's over, it's over.
We are really sorry. is part of the internet history and online for one and a half decades.
Here are the hard facts:
- In 10 days the platform will stop working.
- Backup your data in this time
- We will not keep backups nor can we recover your data
July, 20th, 2020 is the due date.
Please, share your thoughts and feelings here.
Reposted bydotmariuszMagoryannerdanelmangoerainbowzombieskilledmyunicorntomashLogHiMakalesorSilentRulebiauekjamaicanbeatlevuneserenitephinangusiastysmoke11Climbingpragne-ataraksjisauerscharfArchimedesgreywolfmodalnaTheCrimsonIdoljormungundmarbearwaco6mieczuuFeindfeuerDagarhenvairashowmetherainbowszpaqusdivihindsightTabslawujcioBateyelynTabslaensommenitaeliblameyouHalobeatzalicexxxmgns

August 18 2010

Thousands of workers are standing by

Web 2.0 Expo New York - 20% off with code RadarLabor isn't what it used to be. Where in past years the expectation was that jobs were done at a certain place and time, now there are entire swaths of work that can be accomplished by anyone, anywhere.

Lukas Biewald, CEO of CrowdFlower and a speaker at next month's Web 2.0 Expo in New York, is at the center of the labor shift. His company has found an interesting way to tap Internet-connected groups to get work done -- think Mechanical Turk, but with additional tech and quality-assurance layers added on. What's really surprising is that many of the groups CrowdFlower turns to would never define themselves as formal workforces.

Biewald covers a variety of topics in the full interview, including:

  • He sees similarities between "labor on demand" and cloud computing: both keep costs down and reduce the risks associated with scale.
  • "It’s hard to explain my business to my mother," Biewald says. In the interview, he digs into CrowdFlower's unusual -- and somewhat complicated -- business model.
  • He provides further proof that virtual currency is a big deal: Around half of CrowdFlower's work involves it in some fashion.
  • He acknowledges that distributed work has a disruptive and negative affect on many businesses. However, Biewald believes it's a "rising tide" that will "increase the GDP of the world."

Read more after the jump.

What is "labor on demand"?

Lukas BiewaldLukas Biewald: What labor on demand means to us is that you can access tens of thousands, or hundreds of thousands, of people instantly. Truly instantly. You send us a job and we post it online through all of our different channels, and we get lots of people working on your job all at once. Or, we find the specific person that's best for your job.

It's exciting for businesses because they can scale up and scale down. Just as cloud computing made it so businesses didn't have to predict how many servers they were going to need at any given time, labor on demand allows businesses to not have to predict how many people they're going to need at any given minute.

Can you walk me through a typical CrowdFlower job?

LB: Let's use a business listing verification as an example. Suppose Yelp, which isn't one of our customers, gets a complaint saying that a business address is incorrect. Now, they don't want to take the business down immediately, but they do want to respond to that complaint as fast as possible. So they send that complaint to our system, and we post the job. It gets sent all over the world to everyone who happens to be in the CrowdFlower system at the time. Anyone can grab that job, and they will call the business in question or visit the company's website -- whatever the specific instructions are.

Sometimes we have multiple users employ different strategies. So two people call the business and one person checks the website. That's why we're a technology company; our process involves redundancy. We'll have workers spot-checking each other. We'll use all kinds of automated systems to prevent fraud and errors.

Returning to the example, we get results back and decide we're 98-percent confident that this specific business does exist. We shoot the information back to Yelp. The cool thing is all of this happens within five minutes or so. It means the client can always have updated information instead of waiting a day or a month for someone to get around to checking.

How do workers become aware of jobs?

LB: This is the thing that's a little bit complicated about CrowdFlower's business model. It's also why it's hard to explain my business to my mother.

We make deals with other companies that have lots of people around: outsourcing companies, e-rewards companies, even game companies. For every job that they get someone to do, CrowdFlower will pay the company money and that company can incentivize users with things like free seeds for a game or airline miles. We also put jobs inside Amazon Mechanical Turk, which then pays people small amounts of money for doing tasks. We have an open API where anyone can monetize people if they can get them to do these tasks.

We're not in the business of actually collecting people. We go through channels, and most of our channels don't consider themselves workforces.

What we do is add quality control on top of workforces. That was the core technology that started the business. Not every worker is going to be good at every task, but our system can figure out who's going to be good and who's going to be bad.

Is virtual currency a big part of your business?

LB: People think it's a minor part of our business, but it's actually a huge source of work for us: It's about half virtual currency.

How do you address complaints from professionals who are disrupted by crowdsourced work?

LB: Any sort of important technology or shift has winners and losers, right? Look at something like 99Designs, which professional designers often complain about. They say the quality of designs is going down. Yet, at the same time, lots of businesses that were just designing things themselves are now getting things designed.

Look, I believe in minimum wage. I think people should have to pay a certain minimum amount of money. Employers shouldn't be able to exploit people.

Overall, the amount of work out there is increasing. There are certainly tons of examples of people that are hurt by this, but I think it's a rising tide. This will actually increase the GDP of the world.

What's your take on the relationship between human work and machine learning? Are they at odds?

LB: With machine learning it's easy to get to 90 percent a lot of the time, but it's hard to get to 99 percent. And 99 percent is what you need for lots of applications.

If you look at machine learning implementations in the real world, they almost always use a technique called active learning to more efficiently collect data. Active learning is based on the idea that human beings and machine algorithms learn best when presented with confusing information.

For example, think of a machine classifier that's trying to decide if a boy or a girl is the subject of a photo. If you show tons of pictures that are obviously boys and tons of pictures that are obviously girls, that's not going to be as effective as showing a few obvious boys and a few obvious girls and then lots of examples where it's tricky.

When the algorithm gets tripped up, companies can send us the examples where the classifier is confused. We then categorize the examples that will help the algorithm improve.

That's a specific example of machines and people working together that excites me about my business. I think it's the future of machine learning.

Related to that, there's a discussion about why we have more digital work. The assumption was there'd be less because as more gets automated, you'd expect outsourcing to decrease.

Yet, outsourcing -- digital work in particular -- has taken off over the last 20 to 30 years. I don't have a great sound byte answer to explain that phenomenon, but I have seen that it's the companies that do the most machine learning that end up with the most of these kinds of tasks.

As we get used to automated processes working well, they leave a big trail of stuff that needs to be dealt with by people in their wake. So, I absolutely don't fear artificial intelligence and I don't view it as a competitor. Mechanical Turk uses this brilliant term: "artificial, artificial intelligence." It has all of the benefits of people and computers.

What will you focus on in your Web 2.0 Expo New York keynote?

LB: There's this amazing phenomenon going on. Companies are taking core parts of their business and they're sending them not to outsourcing call centers -- like they were doing 20 years ago -- but sending them to millions of people distributed around the world.

I want to talk about not just how this phenomenon is affecting business, but also the social impact of anyone in the world who has access to a computer and broadband. It's a surprisingly large number of people, and these people are now able to compete in a global information marketplace.

When you try to open a factory, it's a tough process. You have to ship goods. You have to go through all kinds of regulations. But when you shift information back and forth, there's very low overhead. There's no need to send hundreds of people into an office and buy them computers and infrastructure. Many people already have the tools to be effective global workers. We're seeing people in refugee camps actually finding useful, meaningful work and making money off of the refugee camp information infrastructure. This is changing the way business operates and also changing the fabric of the world we live in.

This interview was condensed and edited.


Lukas Biewald will discuss the business and social repercussions of distributed work at the Web 2.0 Expo in New York, being held Sep. 27-30. Save 20% off registration with the discount code "radar."

June 01 2010

May 11 2010

Crowdsourcing and the challenge of payment

An unusual href="">Distributed
Distributed Work Meetup was held last night in four different
cities simultaneously, arranged through many hours of hard work by href="">Lukas
Biewald and his colleagues at distributed work provider href="">CrowdFlower.

With all the sharing of experiences and the on-the-spot analyses
taking place, I didn't find an occasion to ask my most pressing
question, so I'll put it here and ask my readers for comments:

How can you set up crowdsourcing where most people work for free but
some are paid, and present it to participants in a way that makes it
seem fair?

This situation arises all the time, with paid participants such as
application developers and community managers, but there's a lot of
scary literature about "crowding out" and other dangers. One basic
challenge is choosing what work to reward monetarily. I can think of
several dividing lines, each with potential problems:

  • Pay for professional skills and ask for amateur contributions on a
    volunteer basis.

    The problem with that approach is that so-called amateurs are invading
    the turf of professionals all the time, and their deft ability to do
    so has been proven over and over at crowdsourcing sites such as href="">InnoCentive for inventors and
    SpringLeap or href="">99 Designs for designers. Still,
    most people can understand the need to pay credentialed professionals
    such as lawyers and accountants.

  • Pay for extraordinary skill and accept more modest contributions on a
    volunteer basis.

    This principle usually reduces to the previous one, because there's no
    bright line dividing the extraordinary from the ordinary. Companies
    adopting this strategy could be embarrassed when a volunteer turns in
    work whose quality matches the professional hires, and MySQL AB in
    particular was known for hiring such volunteers. But if it turns out
    that a large number of volunteers have professional skills, the whole
    principle comes into doubt.

  • Pay for tasks that aren't fun.

    The problem is that it's amazing what some people consider fun. On the
    other hand, at any particular moment when you need some input, you
    might be unable to find people who find it fun enough to do it for
    you. This principle still holds some water; for instance, I heard
    Linus Torvalds say that proprietary software was a reasonable solution
    for programming tasks that nobody would want to do for personal

  • Pay for critical tasks that need attention on an ongoing basis.

    This can justify paying people to monitor sites for spam and
    obscenity, keep computer servers from going down, etc. The problem
    with this is that no human being can be on call constantly. If you're
    going to divide a task among multiple people, you'll find that a
    healthy community tends to be more vigilant and responsive than
    designated individuals.

I think there are guidelines for mixing pay with volunteer work, and
I'd like to hear (without payment) ideas from the crowd.

Now I'll talk a bit about the meetup.

Venue and setup

I just have to start with the Boston-area venue. I had come to many
events at the MIT Media Lab and had always entered Building E14 on the
southwest side. The Lab struck me as a musty, slightly undermaintained
littered with odd jetsam and parts of unfinished projects; a place you
could hardly find your way around but that almost dragged creativity
from you into the open. The Lab took up a new building in 2009 but to
my memory the impact is still similar--it's inherent to the mission
and style of the researchers.

For the first time last night, I came to the building's northeast
entrance, maintained by the MIT School of Architecture. It is Ariel to
the Media Lab's Caliban: an airy set of spacious white-walled forums
sparsely occupied by chairs and occasional art displays. In a very
different way, this space also promotes open thoughts and broad

The ambitious agenda called for the four host cities (Boston, New
York, San Francisco, and Seattle) to share speakers over
videoconferencing equipment. Despite extensive preparation, we all had
audio, video, and connectivity problems at the last minute (in fact,
the Boston organizers crowdsourced the appeal for a laptop and I
surrendered mine for the video feed). Finally in Boston we
disconnected and had an old-fashioned presentation/discusser with an
expert speaker.

In regard to the MIT Media Lab and Architecture School, I think it's
amusing to report that Foursquare didn't recognize either one when I
asked for my current location. Instead, Foursquare offered a variety
of sites across the river, plus the nearby subway, the bike path, and
a few other oddities.

We were lucky to have href="">Jeff Howe, the
WIRED contributor who invented the term href="">Crowdsourcing and wrote a
popular href="">book
on it. He is currently a Nieman Fellow at Harvard. His talk was wildly
informal (he took an urgent call from a baby sitter in the middle) but
full of interesting observations and audience interactions.

He asked us to promote his current big project with WIRED, href="">
One Book, One Twitter. His goal is to reproduce globally the
literacy projects carried out in many cities (one happens every year
in my town, Arlington, Mass.) where a push to get everyone to read a
book is accompanied by community activities and meetups. Through a
popular vote on WIRED, the book href="">American Gods by
Neil Gaiman was chosen, and
people are tweeting away at #1b1t and related tags.


With the sponsorship by CrowdFlower, our evening focused on
crowdsourcing for money. We had a few interesting observations about
the differences between free Wikipedia-style participation and
work-for-pay, but what was most interesting is that basic human
processes like community-building go in both places.

Among Howe's personal revelations was his encounter with the fear of
crowdsourcing. Everyone panics when they first see what crowdsourcing
is doing to his or her personal profession. For instance, when Howe
talked about the graphic design sites mentioned earlier, professional
designers descended on him in a frenzy. He played the sage, lecturing
them that the current system for outsourcing design excludes lots of
creative young talent, etc.

But even Howe, when approached by an outfit that is trying to
outsource professional writing, felt the sting of competition and
refused to help them. But he offered respect for href="">Helium, which encourages self-chosen
authors to sign up and compete for freelance assignments.

Howe is covering citizen journalism, though, a subject that Dan
Gillmor wrote about in a book that O'Reilly published href="">We the
, and that he continues to pursue at his href="">Mediactive site and a new book.

Job protection can also play a role in opposition to crowdsourcing,
because it makes it easier for people around the world to work on
local projects. (Over half the workers on href="">Mechanical Turk now
live in India. Biewald said one can't trust what workers say on their
profiles; IP addresses reveal the truth.) But this doesn't seem to
have attracted the attention of the xenophobes who oppose any support
for job creation in other countries, perhaps because it's hard to get
riled up about "jobs" that have the granularity of a couple seconds.

Crowdsourcing is known to occur, as Howe put it, in "situations of
high social capital," simply meaning that people care about each other
and want to earn each other's favor. It's often reinforced by explicit
rating systems, but even more powerful is the sense of sharing and
knowing that someone else is working alongside you. In a href="">blog
I wrote a couple years ago, I noted that competition site href="">TopCoder maintained a thriving
community among programmers who ultimately were competing with each

Similarly, the successful call center href="">LiveOps provides forums for operators
to talk about their experiences and share tips. This has become not
just a source of crowdsourced help, and not even a way to boost morale
by building community, but an impetus for quality. Operators actually
discipline each other and urge each other to greater heights of
productivity. LiveOps pays its workers more per hour than outsourcing
calls to India normally costs to clients, yet LiveOps is successful
because of its reputation for high quality.

We asked why communities of paid workers tended to reinforce quality
rather than go in the other direction and band together to cheat the
system. I think the answer is obvious: workers know that if they are
successful at cheating, clients will stop using the system and it will
go away, along with their source of income.

Biewald also explained that CrowdFlower has found it fairly easy to
catch and chase away cheaters. It seeds its jobs with simple questions
to which it knows the right answers, and warns the worker right away
if the questions are answered incorrectly. After a couple alerts, the
bad worker usually drops out.

We had a brief discussion afterward about the potential dark side of
crowdsourcing, which law professor Jonathan Zittrain covered in a talk
called href="">Minds
for Sale. One of Zittrain's complaints is that malicious actors
can disguise their evil goals behind seemingly innocuous tasks farmed
out to hundreds of unknowing volunteers. But someone who used to work
for's Mechanical Turk said people are both smarter and more
ethical than they get credit for, and that participants on that
service quickly noted any task that looked unsavory and warned each
other away.

As the name Mechanical Turk (which of course had a completely
unrelated historical origin) suggests, many tasks parceled out by
crowdsourcing firms are fairly mechanical ones that we just haven't
figured out how to fully mechanize yet: transcribing spoken words,
recognizing photos, etc. Biewald said that his firm still has a big
job persuading potential clients that they can trust key parts of the
company supply chain to anonymous, self-chosen workers. I think it may
be easier when the company realizes that a task is truly mechanical
and that they keep full control over the design of the project. But
crowdsourcing is moving up in the world fast; not only production but
control and choice are moving into the crowd.

Howe highlighted Fox News, which runs a href="">UReport site for
volunteers. The stories on Fox News' web site, according to Howe, are
not only written by volunteers but chosen through volunteer ratings,
somewhat in Slashdot style.

Musing on the sociological and economic implications of crowdsourcing,
as we did last night, can be exciting. Even though Mechanical Turk
doesn't seem to be profitable, its clients capture many cost savings,
and other crowdsourcing firms have made headway in particular
fields. Howe hails crowdsourcing as the first form of production that
really reflects the strengths of the Internet, instead of trying to
"force old industrial-age crap" into an online format. But beyond the
philosophical rhetoric, crowdsourcing is an area where a lot of
companies are making money.

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.
No Soup for you

Don't be the product, buy the product!

YES, I want to SOUP ●UP for ...