Coding Horror

programming and human factors

Is Amazon's Mechanical Turk a Failure?

Amazon's Mechanical Turk Service is a clever reference to the famous chess-playing hoax device, The Mechanical Turk. The Mechanical Turk dates back to 1770, and has quite a storied history. Read through the Wikipedia article if you have time; it's fascinating stuff.

The Mechanical Turk

The secret of the Turk, of course, was that it wasn't a chess-playing machine at all. There was a small person inside, controlling the machine.

Similarly, Amazon's Mechanical Turk is a machine that harnesses the work of hidden humans. It's a service that attempts to match people to small, bite-size units of work that are unsuitable for machines.

As of this writing, there are 128 Human Intelligence Tasks available via the Mechanical Turk task page. The reward for these tasks ranges from $1.00 to $0.10, skewing heavily toward the bottom of that range. Almost 100 of the 128 tasks are $0.10 each. Here's a quick sampling of the available tasks:

  • Transcribe a 9 minute, 2 second podcast ($2.31 w/bonus)
  • Write a review of a blog ($1.00)
  • Make ten 2-3 sentence posts in a fansite forum ($0.50)
  • Write a 2-3 paragraph blog entry ($0.50)
  • Provide 3-D and 4-D ultrasound pictures of your baby ($0.40)
  • Send unsolicited junk faxes from California companies ($0.25)
  • Say 6 phrases in Turkish ($0.10)
  • Write a short plot description of the movie "Black Snake Moan" ($0.10)

Read through some of the available HITs yourself. Be sure to click on the HIT to get the details on the job and any rules. You're at the mercy of the requester; it's up to them to judge your work worthy of payment.

Based on the quantity, quality, and type of tasks available, I think Amazon's Mechanical Turk may be a failure. It's been almost two years, and almost all the tasks have one or more of these problems:

  1. obviously and suspiciously spammy
  2. require a lot of subjective human intervention and effort for "grading"
  3. the rates make working in a sweatshop seem lucrative

What I find ironic about Amazon's Mechanical Turk service is that Amazon built an entire business around the value of user reviews. The strength of the user reviews is one of the main reasons I frequent Amazon. That's user-submitted content that people invested countless thousands of man-hours on. And Amazon didn't pay anyone a dime to do it.

I think the secret to running a viable Mechanical Turk service is, paradoxically, to do away with payment. Instead, they should have chosen a reward system based on intrinsic motivation. Intrinsic motivation is the reason why..

  • People willingly contribute millions upon millions of dollars worth of electricity to efforts like Folding@Home so they can show up on the leaderboards with their team.
  • People spend hours submitting and rating articles on Digg and Reddit in the hopes that they will be promoted to the front page, and by proxy, increase their standing in the community.
  • People actively convince others to join them on social networking sites like Linked In, MySpace, Classmates, and Facebook-- to increase the size and power of their networks.

Nobody's paid to do any of the above. And yet each item I listed is easily equivalent to multiple Turk HITs. The best explanation I've found for this behavior is in Mary Poppendieck's Team Compensation (pdf).

There are two approaches to giving children allowances. Theory A says that children should earn their allowances; money is exchanged for work. Theory B says that children should contribute to the household without being paid, so allowances are not considered exchange for work. I know one father who was raised with Theory B but switched to Theory A for his children. He put a price on each job and paid the children weekly for the jobs they had done. This worked for a while, but then the kids discovered that they could choose among the jobs and avoid doing the ones they disliked. When the children were old enough to earn their own paychecks, they stopped doing household chores altogether, and the father found himself mowing the lawn alongside his neighbors' teenage children.

Were he to do it again, this father says he would not tie allowance to work.

In the same way, once employees get used to receiving financial rewards for meeting goals, they begin to work for the rewards, not the intrinsic motivation that comes from doing a good job and helping their company be successful. Many studies have shown that extrinsic rewards like grades and pay will, over time, destroy the intrinsic reward that comes from the work itself.

The theory of intrinsic motivation goes a long way toward explaining why Amazon's unpaid user reviews are so popular and effective, and yet the paid Mechanical Turk service appears to be withering on the vine.

Poppendieck notes that choosing a payment reward model can be an irreversible decision: once you go down the path of monetary rewards, you may never be able to go back, even when they cease to be effective, as they inevitably will. I think that's clearly the case for Amazon's Mechanical Turk.

Discussion

EA's Software Artists

Electronic Arts is a lumbering corporate megalith today, pumping out yearly game franchise after yearly game franchise. It's easy to forget that EA was present at the very beginning of the computer game industry, innovating and blazing a trail for everyone to follow. Gamasutra's article We See Farther: A History of Electronic Arts reminded me how instrumental EA was to the early history of computer gaming.

EA's infamous "We See Farther" ad promoted computer game programmers as artists if not rock stars. I distinctly remember seeing these ads as a dorky, computer-loving teenager. I wondered, could being a computer programmer be.. cool?

Can a Computer Make You Cry?

Okay, maybe not so much with the leather glove, but still. It was a glimmer of hope. According to Electronic Arts, computer programmers weren't just programmers; they could be software artists. And the earliest EA games even looked like rock album covers:

Early EA games

EA also created the very concept of a sports game franchise, primarily with Dr. J and Larry Bird Go One on One. This game was huge. The squeaking sneakers, the smashing backboard, the licensed marquee players. It was arguably the breakthrough sports game.

Dr. J and Larry Bird go One on One

It's impressive that Electronic Arts is still around after all these years. But I wonder what happened to the grandiose sentiments expressed in the We See Farther ad:

These are wondrous machines we have created, and in them can be seen a bit of their makers. It is as if we had invested them with the image of our minds. And through them, we are learning more and more about ourselves. We learn, for instance, that we are more entertained by involvement of our imaginations than by passive viewing and listening. We learn that we are better taught by experience than by memorization. And we learn that the traditional distinctions-- the ones that are made between art and entertainment and education-- don't always apply.

We're providing a special environment for talented, independent software artists. It's a supportive environment, in which big ideas are given room to grow. And some of America's most respected software artists are beginning to take notice. We think our current work reflects this very special commitment. And though we are few in number today and apart from the mainstream of the mass software marketplace, we are confident that both time and vision are on our side.

I sure wanted to believe in software artistry at the time. EA's history proves that this is an unusually difficult vision statement to realize.

Discussion

SEOs: the New Pornographers of the Web

There's something about the Search Engine Optimization (SEO) industry that I find highly distasteful. I've never quite been able to put my finger on it, until I read Rich Skrenta's pornographers vs. SEOs.

Ron Jeremy, porn superstar

It's all clear to me now.

SEOs are the new pornographers of the web.

Money is the most prurient interest of all. Just as pornographers sell sex, SEOs sell money. They trade in get-rich-quick schemes via search traffic.

Rich pointed out this SEOmoz post as an example:

My favourite Digg irony is the hatred the (a-HEM) Diggorati have for SEO, coupled with the fact that they fall for linkbait All. The. Time.

Every so often, one of our employees will roll into the office and announce, "I'm going to get on Digg today." Said employee will sit down, write something and then nervously monitor the server as predicted Digg occurs. I can only remember one instance in which this tactic has failed. The post does not always come from SEOmoz; in fact, it's often posted elsewhere. Sometimes, some Diggers will catch on to the fact that the submission came from someone affiliated with SEO and the comments will get nasty, but still the diggs keep going up.

If that's not pure gaming of the system, I don't know what is. There are entire guides on how to properly linkbait, such as Andy Hagans' Ultimate Guide to Linkbaiting and Social Media Marketing. Read it. I did, and now I feel like I just walked through a sewer.

Although SEOs pay lip service to the quality of the content, it's clear that the focus is on one thing, and one thing only: naked, raw greed.

Look how big my $%^@ paycheck is!

Jason Calacanis is probably the most prominent critic of SEO techniques.

The SEO folks got really pissed off at me for saying "SEO is bulls@#t" last year, but the truth is that 90% of the SEO market is made up of snake oil salesman. These are guys in really bad suits trying to get really naive people to sign long-term contracts. These clients typically make horrible products and don't deserve traffic. That's why they're not getting it organically. So they hire the slimebuckets to game the system for them.

There are some whitehat SEO firms out there, but frankly the whitehat SEO companies are simply doing solid web design. I don't consider them SEO at all. SEO is a tainted term and it means "gaming the system" to 90% of us.

So much of what is optimistically termed Search Engine Optimization is basic web design 101. And yet the seedy SEO underground will still try to convince you their super-secret methods-- their magical snake oil-- is the only formula for success.

In fact, the only difference between SEO "experts" and pornographers is that pornographers are more honest about what they're actually selling. Hiring a SEO expert to increase the quality of your site's content is like renting a porn video for the plot and character development. Stop kidding yourselves.

What's most depressing about all of this is that reliance on SEO is ultimately self-defeating.

Let me tell you a story about my cousin, Steven. Steven wanted to become a musician. He had a rock band he diligently put together, and they recorded an album. When he played the album for me, he described it as a mixture of Green Day with some Linkin Park. He told me how he took the best of both bands and created his own sound.

The problem was, his music was boring, and his band sounded like hundreds of other indie bands.

Steven was just an average looking guy, his band was good but not brilliant, his music was solid, but not different. Steven believed in his band, and he was just good enough for everyone to encourage him to go on working on music, but never good enough to attract a fan base.

But Steven never even tried to build up fans. He never played his music for totally random people, and asked them for their opinion. All he did was try to get the people at the record labels interested in his music. He called and hounded, he stalked and staked out. He kept chasing those labels for years and years, and then suddenly gave up.

Steven did not care about the music. Steven cared about the money. Steven did not concentrate on getting his music to the people, he concentrated on getting his music to the people who would guarantee him money and connections.

When you focus on SEO, you're focusing on money and connections. If that's your goal, then have at it. But don't be surprised when people see through you for what you really are.

Discussion

Firefox as an IDE

Although I prefer IE7's native speed and feel for day-to-day browsing chores, there's no doubt that Firefox is my primary web development IDE.

Whenever I need to troubleshoot HTML, CSS, or JavaScript, I immediately reach for Firefox. That's primarily because of two incredibly powerful developer extensions for Firefox:

1. Web Developer Extension

Firefox Web Developer Extension screenshot

WDE has lots of useful features, but the most wonderful thing about WDE is that everything is editable in real-time. Edit the CSS, edit the HTML, and watch your changes take effect as you type. It's like magic.

2. Firebug

Firefox Firebug screenshot

Firebug is a richer tool than WDE; it goes far beyond real-time editing of HTML and CSS, and delves deeply into profiling and debugging JavaScript. It also offers metrics on download time and download order over the wire. I highly recommend checking out the Joe Hewitt Firebug power user demo movie to get a sense of its full feature set.

I can't imagine debugging a web app without these two essential tools, which transform Firefox into the Visual Studio of browsers. It'd be akin to writing a .NET application in Notepad. In fact, I feel so strongly about the utility of these extensions that I'm stopping to donate $10 to each project, right now. This is part of my ongoing monthly contribution to small software vendors.

If you haven't used these tools yet, I envy the experience you're about to have. Download them both and start experimenting. If you have used Firebug and WDE, I encourage you to share your favorite tips in the comments.

Discussion

Pick a License, Any License

I hate software licenses. When I read a software license, what I see is a bunch of officious, mind-numbing lawyerly doublespeak. Blah, blah, blah.. kill me now.

meaningless license certificate

If I had my way, everything would be released under the WTFPL. Over time, I've begrudgingly come to the conclusion that, like lawyers, death, and taxes, choosing a software license is inevitable. Of course, it doesn't matter if yours are the only human eyes that will ever see the code. But a proper software license is a necessary evil for any code you plan to release to the public.

I definitely regret not choosing a software license for my CodeProject articles. I'll occasionally get friendly emails from people asking permission to use the code from my articles in various projects, commercial and otherwise. It's thoughtful of people to ask first. I do appreciate it, and permission is always granted with the single caveat that my name and URL remain in the comments.

Because I did not explicitly indicate a license, I declared an implicit copyright without explaining how others could use my code. Since the code is unlicensed, I could theoretically assert copyright at any time and demand that people stop using my code. Experienced developers won't touch unlicensed code because they have no legal right to use it. That's ironic, considering the whole reason I posted the code in the first place was so other developers could benefit from that code. I could have easily avoided this unfortunate situation if I had done the right thing and included a software license with my code.

Unfortunately, we love software licenses like we love standards – that's why there are so many of them. And what, exactly, are the differences between all these licenses? How do you select the correct license for your software? I've attempted to succinctly capture the key differences between the most well-known software licenses in the following handy chart.

Source Type (Clauses)
None Open None (0) Without a license, the code is copyrighted by default. People can read the code, but they have no legal right to use it. To use the code, you must contact the author directly and ask permission.
Public domain Open Permissive (0) If your code is in the public domain, anyone may use your code for any purpose whatsoever. Nothing is in the public domain by default; you have to explicitly put your work in the public domain if you want it there. Otherwise, you must be dead a long time before your work reverts to the public domain.
GPL Open Copyleft (12) The archetypal bearded, sandal-clad free software license. Your code can never be used in any proprietary program, ever! Take that, capitalism!
LGPL Open Mostly Copyleft (16) GPL with a cleverly-constructed pressure valve release. Your free software can be binary linked to proprietary programs under certain very specific circumstances.
MIT/X11 Open Permissive (2) Short and sweet. Includes generic legal disclaimer of liability.
BSD Open Permissive (2) Short and sweet. Includes legal disclaimer of liability with explicitly named organization.
Apache Open Permissive (9) Requires derivative works to provide notification of any licensed or proprietary code in a common location.
Eclipse Open Permissive (7) Business friendly. Allows derivative works to choose their own license for their contributions.
Mozilla Open Weak Copyleft (13) Allows liberal mixing with proprietary software.
MS Permissive Open Permissive (3) Resembles the MIT and BSD licenses. Not formally accepted by OSI, and also offered in a "Windows-only" LPL variant.
MS Community Open Copyleft (3) Resembles the GPL license. Requires all contributed code to be returned to the community. Not formally accepted by OSI, and also offered in a "Windows-only" LCL version.
MS Reference Proprietary Read Only (3) You can review the code, or make copies of it, but you can't use it or change it in any way. Allows a window (no pun intended) on formerly completely proprietary, secret code.

After compiling this table, I've learned two things:

  1. My head hurts.
  2. I still prefer the WTFPL.

I'm not even going to get into the many religious issues of software licensing, such as...

It's a minefield, people. All I'm saying is this: the next time you release code into the wild, do your fellow developers a favor and pick a license – any license.

Discussion