Coding Horror

programming and human factors

I Tried VR and It Was Just OK

It's been about a year and a half since I wrote The Road to VR, and a … few … things have happened since then.

  • Facebook bought Oculus for a skadillion dollars

  • I have to continually read thinkpieces describing how the mere act of strapping a VR headset on your face is such a transformative, disruptive, rapturous experience that you'll never look at the world the same way again.

I am somewhat OK with the former, although the idea of my heroes John Carmack and Michael Abrash as Facebook employees still raises my hackles. But the latter is more difficult to stomach. And it just doesn't stop.

For example, this recent WSJ piece. (I can't link directly to it, you have to click through from Google search results to get past the paywall).

I’ll spare you the rapturous account of the time I sculpted in three dimensions with light, fire, leaves and rainbows inside what felt like a real-life version of a holodeck from “Star Trek.” Writing about VR is like fiction about sex—seldom believable and never up to the task.

If you really want to understand how compelling VR is, you just have to try it. And I guarantee you will. At some point in the next couple of years, one of your already-converted friends will insist you experience it, the same way someone gave you your first turn at a keyboard or with a touch screen. And it will be no less a transformative experience.

I don't mean to call out the author here. There are a dozen other similarly breathless VR articles I could cite, where an amazing VR wonderland is looming right around the corner for all of us, any day now. And if you haven't tried it, boy, you just don't know! It can't be explained, it must be experienced! There are people who honestly believe that in 5 years nobody will make non-VR games any more. The hype levels are off the charts.

Well, I have experienced modern VR. A lot. I've tried both the Oculus DK1, the Oculus DK2, and a 360° backpack-and-controllers Survios rig, which looks something like this:

Based on those experiences, I can't reconcile these hype levels with what I felt. At all. Right now, VR is not something I'd unconditionally recommend to a fellow avid gamer, much less a casual gamer.

To be honest, when I tried the DK1 and DK2, after a few hours of demos and exploration, I couldn't wait to get the headset off. Not because I was motion sick – I don't get motion sick, and never have – but because I was bored. And a little frustrated by control limitations. Not exactly the stuff transformative world-changing disruption is made of.

Here's what that experience looks like, by the way. You can practically taste the gaming excitement dripping off me.

And if you don't find watching me experience my virtual world fascinating (although I can't imagine why) I suppose you can enjoy what's on my screen:

Chroma-shifted, stereographic, fisheye VR gibberish.

I've always been the first kid on my block to recommend an awesome, transformative gaming experience, from the Atari 2600 to the Kinect. I mean, that's kind of who I am, isn't it? The alpha geek, the guy who owned a Vectrex and thought vector graphics were the cat's pajamas, the guy who bought one of the first copies of Guitar Hero in 2005 and would not shut up about it. For that matter I dragged my buddies to a VR storefront in Boulder, Colorado circa 1993 so we could play Dactyl Nightmare. And I have to say, in my alpha geek opinion, modern VR has a long way to go before it'll be ready for the rapturous smartphone levels of adoption that media pundits imply is a few months away.

I apologize if this comes off as negative, and no, I haven't tried the magical new VR headset models that are Just Around The Corner and Will Change Everything. I'll absolutely try them when they are available. Let me be clear that I think the technical challenges around VR are deep, hard, and fascinating, and I could not be happier that some of the best programmers of our generation are working on this stuff. But from what I've seen and experienced to date, there is just no way that VR is going to be remotely mainstream in 5 years. I'm doubtful that can happen in a decade or even two decades, to be honest, but a smart person always hedges their bets when trying to predict the future.

I think the current state of VR, or at least the "strap a nice smartphone or two on your face" version of it, has quite a few fundamental physical problems to deal with before it has any chance of being mainstream.

It should be as convenient as a pair of glasses

Nobody "enjoys" strapping two pounds of stuff on their face unless they are in a hazardous materials situation. We can barely get people to wear bicycle helmets, and yet they are going to be lining up around the block to slap this awkward, gangly VR contraption on their head? Existing VR headsets get awfully sweaty after 30 minutes of use, and they're also difficult to fit over glasses. The idea of gaming with a heavy, sweaty, uncomfortable headset on for hours at a time isn't too appealing – and that's coming from a guy who thinks nothing of spending 6 hours in a gaming jag with headphones on.

For VR to be quick and easy and pervasive, the headset would need to be so miniaturized as to be basically invisible – akin to putting on a cool pair of sunglasses.

Maybe current VR headsets are like the old brick cellphones from the 90's. The question is, how quickly can they get from 1990 to 2007?

It should be wireless

The world has been inexorably moving towards wireless everything, but in this regard VR headsets are a glorious throwback to science fiction movies from the 1970s. Your VR headset and everything else on it will be physically wired, in multiple ways, to a powerful computer. Wires, wires, everywhere, as far as your eyes … can't see.

Even the cheaper VR headsets that let you drop a high end smartphone in for a limited VR experience have to be wired to power, as phone batteries are not built for the continuous heavy-duty CPU and GPU rendering that VR requires. Overheating is a very real problem, too.

Wireless video is hard to do well, particularly at the 1440p resolutions that are the absolute minimum for practical VR. On top of that, good VR requires much higher framerates, ideally 120fps. That kind of ultra low latency, super high resolution video delivered wirelessly, is quite far off.

It should have 4k resolution

Since the VR device you're looking at is inches from your eyes – and the resolution is effectively divided in half for each eye (there are a few emerging VR headsets that use two smartphones here instead of one) – an extremely high resolution screen is needed to achieve effective visual resolutions that are ancient by modern computer standards.

The Oculus DK1 at 720p was so low resolution that I consider it borderline unusable even as a demo unit. I'd estimate that it felt roughly DOOM resolution, or 320×240.

The DK2 at 1080p was marginally better, but the pixelation and shimmer was quite bad, a serious distraction from immersion. It felt roughly Quake resolution, or 640×480.

I know many upcoming VR devices are 1440p or 2560×1440. I strongly suspect that, in practice, is going to feel like yet another mild bump to effective 1024×768 resolution.

I'm used to modern games and modern graphics resolutions. Putting on a VR headset shouldn't be a one-way ticket to jarring, grainy, pixelated graphics the like of which I haven't seen since 1999. There are definitely 4k smartphones out there on the horizon which could solve this problem, but the power required to drive them, by that I mean the CPU, GPU, and literal battery power – is far from trivial.

(And did I mention it needs to be a minimum of 60fps, ideally 120fps for the best VR experience? I'm pretty sure I mentioned that.)

Still, the 4k resolution problem is probably the closest to being reasonably solved on current hardware trajectories in about five years or so, albeit driven by very high end hardware, not a typical smartphone, which brings me to …

It should not require a high end gaming PC or future gen console

VR has massive CPU and GPU system requirements, somewhat beyond what you'd need for the latest videogames running at 4k resolutions. Which means by definition cutting edge VR is developed with, and best experienced on, a high end Windows PC.

Imagine the venture capitalists who invested in Oculus, who have probably been Mac-only since the early aughts, trying to scrounge together a gaming PC so they can try this crazy new VR thing they just invested in. That's some culture shock.

Current generation consoles such as the Xbox One and PS4 may be fine with (most) games running at 1080p, on the PS4 at least, but they are both woefully under-specced to do VR in both GPU and CPU power. That's bad news if you expect VR to be mainstream in the lifetime of these new consoles over the next 5-8 years, and were counting on the console market to get there.

VR on current generation consoles will be a slow, crippled, low resolution affair, about on the level of the Oculus DK2 at best. You'll be waiting quite a while for the next generation of consoles beyond these to deliver decent VR.

Hands (Gloves?) must be supported

I was extremely frustrated by the lack of control options in the Oculus DK1 and DK2. Here I was looking around and exploring this nifty VR world, but to do anything I had to tap a key on my keyboard, or move and click my mouse. Talk about breaking immersion. They bundle an Xbox controller with the upcoming Rift, which is no better. Experiencing VR with a mouse is like playing Guitar Hero with a controller.

The most striking thing about the Survios demo rig I tried was the way I could use my hands to manipulate things in the VR world. Adding hands to VR was revelatory, the one bit of VR I've experienced to date that I can honestly say I was blown away by. I could reach out and grab objects, rotate things in my hands and move them close to my face to look at them, hold a shotgun and cock it with two hands, and so forth. With my hands, it was amazing. The primary controllers you should need in VR are the ones you were born with: your hands.

A virtual world experienced with just your head is quite disappointing and passive, like a movie or an on-rails ride. But add hands, and suddenly you are there because you can now interact with that VR world in a profoundly human way: by touching it. I could see myself playing story exploration games like Gone Home in VR, if I can use my hands – to manipulate things, to look at them and open them and turn them in my hands and check them out. It was incredible. Manipulating that world with my hands made it infinitely more real.

The good news is that there are solutions like Oculus Touch. The bad news is that's it's not bundled by default, but should be. This device tracks hand position, plus rotation, and adds some buttons for interaction. Even better would be simple gloves you could wear that visually tracked each finger – but sometimes you do need a button, because if you are holding a gun (or a flashlight) you need to indicate that you fired a gun (or turned on the flashlight) which would be quite hairy to track via finger movement alone.

I'm optimistic that VR and hand control will hopefully become synonymous, otherwise we're locking ourselves into a "just look around you" mindset, which leads to crappy, passive VR that's little more than a different kind of IMAX 3D movie.

It must compete with mature 2D entertainment

I get frustrated talking to people who act like VR exists in a vacuum, that there are suddenly no other experiences worth having unless they happen in glorious stereo 3D.

I've experimented with stereo 3D on computers since the days of junky battery powered LCD shutter glasses. And we all know the world has experienced the glory of 3D television … and collectively turned its head and said meh.

Experiencing something in 3D, in and of itself, is just not that compelling. If it was, people would have scarfed up 3D TVs, see only 3D movies, and play only 3D video games on their PCs and consoles regularly. The technology to do it is there, battle tested, and completely mature. I know because I saw Captain EO at Epcot Center in 3D way back in 1985, and it was amazing thirty years ago!

I recently saw Gravity in IMAX 3D and I liked it, but it didn't transform my moviegoing experience such that I could never imagine seeing another boring flat 2D movie ever again.

People have so many wonderful social experiences gathered around common 2D screens. Watching a movie, watching a TV show, watching someone play a game. These are fundamentally joyous, shared human experiences. For this to work with VR is kinda-sorta possible, but difficult, because:

  • You need a proper flat 2D representation of what the VR user is seeing

  • That 2D representation must be broadcast on the primary display device

VR is ultra resource intensive already, so rendering yet another copy of the scene at reasonable framerates (say, constant 60fps) isn't going to be easy. Or free.

On top of that, the VR user is probably wearing headphones, holding a pair of hand controllers, and can't see anything, so they can't interact with anyone who is physically there very well.

I've had incredible gaming experiences on 2D screens. I recently played Alien: Isolation, or as I like to call it, Pants Crapping Simulator 3000, and I thought it was one of the most beautiful, immersive, and downright goddamn terrifying gameplay experiences I've had in years. I was starring in a survival horror movie – it felt like I was there in every sense of the word.

At no point did I think to myself "this would be better in 3D". In many ways, it would have been worse.

Good God man, do you ever shut up?

Sorry. I had some things to get off my chest with regards to VR. I'll wrap it up.

I apologize, again, if this post seems negative. Writing this, I actually got a little more excited about VR. I can see how far it has to come to match its potential, because the technical problems it presents are quite hard – and those are the most fun problems to attack.

I guess I might be the only person left on Earth who said, hey, I tried VR and it was just OK. I think VR ought to be a hell of a lot better, and has to be if it wants to be truly pervasive.

[advertisement] At Stack Overflow, we put developers first. We already help you find answers to your tough coding questions; now let us help you find your next job.

Doing Terrible Things To Your Code

In 1992, I thought I was the best programmer in the world. In my defense, I had just graduated from college, this was pre-Internet, and I lived in Boulder, Colorado working in small business jobs where I was lucky to even hear about other programmers much less meet them.

I eventually fell in with a guy named Bill O'Neil, who hired me to do contract programming. He formed a company with the regrettably generic name of Computer Research & Technologies, and we proceeded to work on various gigs together, building line of business CRUD apps in Visual Basic or FoxPro running on Windows 3.1 (and sometimes DOS, though we had a sense by then that this new-fangled GUI thing was here to stay).

Bill was the first professional programmer I had ever worked with. Heck, for that matter, he was the first programmer I ever worked with. He'd spec out some work with me, I'd build it in Visual Basic, and then I'd hand it over to him for review. He'd then calmly proceed to utterly demolish my code:

  • Tab order? Wrong.
  • Entering a number instead of a string? Crash.
  • Entering a date in the past? Crash.
  • Entering too many characters? Crash.
  • UI element alignment? Off.
  • Does it work with unusual characters in names like, say, O'Neil? Nope.

One thing that surprised me was that the code itself was rarely the problem. He occasionally had some comments about the way I wrote or structured the code, but what I clearly had no idea about is testing my code.

I dreaded handing my work over to him for inspection. I slowly, painfully learned that the truly difficult part of coding is dealing with the thousands of ways things can go wrong with your application at any given time – most of them user related.

That was my first experience with the buddy system, and thanks to Bill, I came out of that relationship with a deep respect for software craftsmanship. I have no idea what Bill is up to these days, but I tip my hat to him, wherever he is. I didn't always enjoy it, but learning to develop discipline around testing (and breaking) my own stuff unquestionably made me a better programmer.

It's tempting to lay all this responsibility at the feet of the mythical QA engineer.

If you are ever lucky enough to work with one, you should have a very, very healthy fear of professional testers. They are terrifying. Just scan this "Did I remember to test" list and you'll be having the worst kind of flashbacks in no time. And that's the abbreviated version of his list.

I believe a key turning point in every professional programmer's working life is when you realize you are your own worst enemy, and the only way to mitigate that threat is to embrace it. Act like your own worst enemy. Break your UI. Break your code. Do terrible things to your software.

This means programmers need a good working knowledge of at least the common mistakes, the frequent cases that average programmers tend to miss, to work against. You are tester zero. This is your responsibility.

Let's start with Patrick McKenzie's classic Falsehoods Programmers Believe about Names:

  1. People have exactly one canonical full name.
  2. People have exactly one full name which they go by.
  3. People have, at this point in time, exactly one canonical full name.
  4. People have, at this point in time, one full name which they go by.
  5. People have exactly N names, for any value of N.
  6. People’s names fit within a certain defined amount of space.
  7. People’s names do not change.
  8. People’s names change, but only at a certain enumerated set of events.
  9. People’s names are written in ASCII.
  10. People’s names are written in any single character set.

That's just the first 10. There are thirty more. Plus a lot in the comments if you're in the mood for extra credit. Or, how does Falsehoods Programmers Believe About Time grab you?

  1. There are always 24 hours in a day.
  2. Months have either 30 or 31 days.
  3. Years have 365 days.
  4. February is always 28 days long.
  5. Any 24-hour period will always begin and end in the same day (or week, or month).
  6. A week always begins and ends in the same month.
  7. A week (or a month) always begins and ends in the same year.
  8. The machine that a program runs on will always be in the GMT time zone.
  9. Ok, that’s not true. But at least the time zone in which a program has to run will never change.
  10. Well, surely there will never be a change to the time zone in which a program has to run in production.
  11. The system clock will always be set to the correct local time.
  12. The system clock will always be set to a time that is not wildly different from the correct local time.
  13. If the system clock is incorrect, it will at least always be off by a consistent number of seconds.
  14. The server clock and the client clock will always be set to the same time.
  15. The server clock and the client clock will always be set to around the same time.

Are there more? Of course there are! There's even a whole additional list of stuff he forgot when he put that giant list together.

Catastrophic Error - User attempted to use program in the manner program was meant to be used

I think you can see where this is going. This is programming. We do this stuff for fun, remember?

But in true made-for-TV fashion, wait, there's more! Seriously, guys, where are you going? Get back here. We have more awesome failure states to learn about:

At this point I wouldn't blame you if you decided to quit programming altogether. But I think it's better if we learn to do for each other what Bill did for me, twenty years ago — teach less experienced developers that a good programmer knows they have to do terrible things to their code. Do it because if you don't, I guarantee you other people will, and when they do, they will either walk away or create a support ticket. I'm not sure which is worse.

[advertisement] Find a better job the Stack Overflow way - what you need when you need it, no spam, and no scams.

What is Trolling?

If you engage in discussion on the Internet long enough, you're bound to encounter it: someone calling someone else a troll.

The common interpretation of Troll is the Grimms' Fairy Tales, Lord of the Rings, "hangs out under a bridge" type of troll.

Thus, a troll is someone who exists to hurt people, cause harm, and break a bunch of stuff because that's something brutish trolls just … do, isn't it?

In that sense, calling someone a Troll is not so different from the pre-Internet tactic of calling someone a monster – implying that they lack all the self-control and self-awareness a normal human being would have.

Pretty harsh.

That might be what the term is evolving to mean, but it's not the original intent.

The original definition of troll was not a beast, but a fisherman:


verb \ˈtrōl\

  1. to fish with a hook and line that you pull through the water

  2. to search for or try to get (something)

  3. to search through (something)

If you're curious why the fishing metaphor is so apt, check out this interview:

There's so much fishing going on here someone should have probably applied for a permit first.

  • He engages in the interview just enough to get the other person to argue. From there, he fishes for anything that can nudge the argument into some kind of car wreck that everyone can gawk at, generating lots of views and publicity.

  • He isn't interested in learning anything about the movie, or getting any insight, however fleeting, into this celebrity and how they approached acting or directing. Those are perfunctory concerns, quickly discarded on the way to their true goal: generating controversy, the more the better.

I almost feel sorry for Quentin Tarantino, who is so obviously passionate about what he does, because this guy is a classic troll.

  1. He came to generate argument.
  2. He doesn't truly care about the topic.

Some trolls can seem to care about a topic, because they hold extreme views on it, and will hold forth at great length on said topic, in excruciating detail, to anyone who will listen. For days. Weeks. Months. But this is an illusion.

The most striking characteristic of the worst trolls is that their position on a given topic is absolutely written in stone, immutable, and they will defend said position to the death in the face of any criticism, evidence, or reason.

Look. I'm not new to the Internet. I know nobody has ever convinced anybody to change their mind about anything through mere online discussion before. It's unpossible.

But I love discussion. And in any discussion that has a purpose other than gladiatorial opinion bloodsport, the most telling question you can ask of anyone is this:

Why are you here?

Did you join this discussion to learn? To listen? To understand other perspectives? Or are you here to berate us and recite your talking points over and over? Are you more interested in fighting over who is right than actually communicating?

If you really care about a topic, you should want to learn as much as you can about it, to understand its boundaries, and the endless perspectives and details that make up any interesting topic. Heck, I don't even want anyone to change your mind. But you do have to demonstrate to us that you are at least somewhat willing to entertain other people's perspectives, and potentially evolve your position on the topic to a more nuanced, complex one over time.

In other words, are you here in good faith?

People whose actions demonstrate that they are participating in bad faith – whether they are on the "right" side of the debate or not – need to be shown the door.

So now you know how to identify a troll, at least by the classic definition. But how do you handle a troll?

You walk away.

I'm afraid I don't have anything uniquely insightful to offer over that old chestnut, "Don't feed the trolls." Responding to a troll just gives them evidence of their success for others to enjoy, and powerful incentive to try it again to get a rise out of the next sucker and satiate their perverse desire for opinion bloodsport. Someone has to break the chain.

I'm all for giving people the benefit of the doubt. Just because someone has a controversial opinion, or seems kind of argumentative (guilty, by the way), doesn't automatically make them a troll. But their actions over time might.

(I also recognize that in matters of social justice, there is sometimes value in speaking out and speaking up, versus walking away.)

So the next time you encounter someone who can't stop arguing, who seems unable to generate anything other than heat and friction, whose actions amply demonstrate that they are no longer participating in the conversation in good faith … just walk away. Don't take the bait.

Even if sometimes, that troll is you.

[advertisement] How are you showing off your awesome? Create a Stack Overflow Careers profile and show off all of your hard work from Stack Overflow, Github, and virtually every other coding site. Who knows, you might even get recruited for a great new position!

Your Password is Too Damn Short

I'm a little tired of writing about passwords. But like taxes, email, and pinkeye, they're not going away any time soon. Here's what I know to be true, and backed up by plenty of empirical data:

  • No matter what you tell them, users will always choose simple passwords.

  • No matter what you tell them, users will re-use the same password over and over on multiple devices, apps, and websites. If you are lucky they might use a couple passwords instead of the same one.

What can we do about this as developers?

  • Stop requiring passwords altogether, and let people log in with Google, Facebook, Twitter, Yahoo, or any other valid form of Internet driver's license that you're comfortable supporting. The best password is one you don't have to store.

  • Urge browsers to support automatic, built-in password generation and management. Ideally supported by the OS as well, but this requires cloud storage and everyone on the same page, and that seems most likely to me per-browser. Chrome, at least, is moving in this direction.

  • Nag users at the time of signup when they enter passwords that are …

    • Too short: UY7dFd

    • Lack sufficient entropy: aaaaaaaaa

    • Match common dictionary words: anteaters1

This is commonly done with an ambient password strength meter, which provides real time feedback as you type.

If you can't avoid storing the password – the first two items I listed above are both about avoiding the need for the user to select a 'new' password altogether – then showing an estimation of password strength as the user types is about as good as it gets.

The easiest way to build a safe password is to make it long. All other things being equal, the law of exponential growth means a longer password is a better password. That's why I was always a fan of passphrases, though they are exceptionally painful to enter via touchscreen in our brave new world of mobile – and that is an increasingly critical flaw. But how short is too short?

When we built Discourse, I had to select an absolute minimum password length that we would accept. I chose a default of 8, based on what I knew from my speed hashing research. An eight character password isn't great, but as long as you use a reasonable variety of characters, it should be sufficiently resistant to attack.

By attack, I don't mean an attacker automating a web page or app to repeatedly enter passwords. There is some of this, for extremely common passwords, but that's unlikely to be a practical attack on many sites or apps, as they tend to have rate limits on how often and how rapidly you can try different passwords.

What I mean by attack is a high speed offline attack on the hash of your password, where an attacker gains access to a database of leaked user data. This kind of leak happens all the time. And it will continue to happen forever.

If you're really unlucky, the developers behind that app, service, or website stored the password in plain text. This thankfully doesn't happen too often any more, thanks to education efforts. Progress! But even if the developers did properly store a hash of your password instead of the actual password, you better pray they used a really slow, complex, memory hungry hash algorithm, like bcrypt. And that they selected a high number of iterations. Oops, sorry, that was written in the dark ages of 2010 and is now out of date. I meant to say scrypt. Yeah, scrypt, that's the ticket.

Then we're safe? Right? Let's see.

You might read this and think that a massive cracking array is something that's hard to achieve. I regret to inform you that building an array of, say, 24 consumer grade GPUs that are optimized for speed hashing, is well within the reach of the average law enforcement agency and pretty much any small business that can afford a $40k equipment charge. No need to buy when you can rent – plenty of GPU equipped cloud servers these days. Beyond that, imagine what a motivated nation-state could bring to bear. The mind boggles.

Even if you don't believe me, but you should, the offline fast attack scenario, much easier to achieve, was hardly any better at 37 minutes.

Perhaps you're a skeptic. That's great, me too. What happens when we try a longer password on the massive cracking array?

9 characters2 minutes
10 characters2 hours
11 characters6 days
12 characters1 year
13 characters64 years

The generator is "only" uppercase, lowercase, and number. What if we add special characters, to keep Q*Bert happy?

8 characters1 minute
9 characters2 hours
10 characters1 week
11 characters2 years
12 characters2 centuries

That's a bit better, but you can't really feel safe until the 12 character mark even with a full complement of uppercase, lowercase, numbers, and special characters.

It's unlikely that massive cracking scenarios will get any slower. While there is definitely a password length where all cracking attempts fall off an exponential cliff that is effectively unsurmountable, these numbers will only get worse over time, not better.

So after all that, here's what I came to tell you, the poor, beleagured user:

Unless your password is at least 12 characters, you are vulnerable.

That should be the minimum password size you use on any service. Generate your password with some kind of offline generator, with diceware, or your own home-grown method of adding words and numbers and characters together – whatever it takes, but make sure your passwords are all at least 12 characters.

Now, to be fair, as I alluded to earlier all of this does depend heavily on the hashing algorithm that was selected. But you have to assume that every password you use will be hashed with the lamest, fastest hash out there. One that is easy for GPUs to calculate. There's a lot of old software and systems out there, and will be for a long, long time.

And for developers:

  1. Pick your new password hash algorithms carefully, and move all your old password hashing systems to much harder to calculate hashes. You need hashes that are specifically designed to be hard to calculate on GPUs, like scrypt.

  2. Even if you pick the "right" hash, you may be vulnerable if your work factor isn't high enough. Matsano recommends the following:

    • scrypt: N=2^14, r=8, p=1

    • bcrypt: cost=11

    • PBKDF2 with SHA256: iterations=86,000

    But those are just guidelines; you have to scale the hashing work to what's available and reasonable on your servers or devices. For example, we had a minor denial of service bug in Discourse where we allowed people to enter up to 20,000 character passwords in the login form, and calculating the hash on that took, uh … several seconds.

Now if you'll excuse me, I need to go change my PayPal password.

[advertisement] What's your next career move? Stack Overflow Careers has the best job listings from great companies, whether you're looking for opportunities at a startup or Fortune 500. You can search our job listings or create a profile and let employers find you.

Given Enough Money, All Bugs Are Shallow

Eric Raymond, in The Cathedral and the Bazaar, famously wrote

Given enough eyeballs, all bugs are shallow.

The idea is that open source software, by virtue of allowing anyone and everyone to view the source code, is inherently less buggy than closed source software. He dubbed this "Linus's Law".

Insofar as it goes, I believe this is true. When only the 10 programmers who happen to work at your company today can look at your codebase, it's unlikely to be as well reviewed as a codebase that's public to the world's scrutiny on GitHub.

However, the Heartbleed SSL vulnerability was a turning point for Linus's Law, a catastrophic exploit based on a severe bug in open source software. How catastrophic? It affected about 18% of all the HTTPS websites in the world, and allowed attackers to view all traffic to these websites, unencrypted... for two years.

All those websites you thought were secure? Nope. This bug went unnoticed for two full years.

Two years!

OpenSSL, the library with this bug, is one of the most critical bits of Internet infrastructure the world has – relied on by major companies to encrypt the private information of their customers as it travels across the Internet. OpenSSL was used on millions of servers and devices to protect the kind of important stuff you want encrypted, and hidden away from prying eyes, like passwords, bank accounts, and credit card information.

This should be some of the most well-reviewed code in the world. What happened to our eyeballs, man?

In reality, it's generally very, very difficult to fix real bugs in anything but the most trivial Open Source software. I know that I have rarely done it, and I am an experienced developer. Most of the time, what really happens is that you tell the actual programmer about the problem and wait and see if he/she fixes it – Neil Gunton

Even if a brave hacker communities to read the code, they're not terribly likely to spot one of the hard-to-spot problems. Why? Few open source hackers are security experts. – Jeremy Zawodny

The fact that many eyeballs are looking at a piece of software is not likely to make it more secure. It is likely, however, to make people believe that it is secure. The result is an open source community that is probably far too trusting when it comes to security. – John Viega

I think there are a couple problems with Linus's Law:

  1. There's a big difference between usage eyeballs and development eyeballs. Just because you pull down some binaries in a RPM, or compile something in Linux, or even report bugs back to the developers via their bug tracker, doesn't mean you're doing anything at all to contribute to the review of the underlying code. Most eyeballs are looking at the outside of the code, not the inside. And while you can discover bugs, even important security bugs, through usage, the hairiest security bugs require inside knowledge of how the code works.

  2. The act of writing (or cut-and-pasting) your own code is easier than understanding and peer reviewing someone else's code. There is a fundamental, unavoidable asymmetry of work here. The amount of code being churned out today – even if you assume only a small fraction of it is "important" enough to require serious review – far outstrips the number of eyeballs available to look at the code. (Yes, this is another argument in favor of writing less code.)

  3. There are not enough qualified eyeballs to look at the code. Sure, the overall number of programmers is slowly growing, but what percent of those programmers are skilled enough, and have the right security background, to be able to audit someone else's code effectively? A tiny fraction.

Even if the code is 100% open source, utterly mission critical, and used by major companies in virtually every public facing webserver for customer security purposes, we end up with critical bugs that compromise everyone. For two years!

That's the lesson. If we can't naturally get enough eyeballs on OpenSSL, how does any other code stand a chance? What do we do? How do we get more eyeballs?

The short term answer was:

These are both very good things and necessary outcomes. We should be doing this for all the critical parts of the open source ecosystem people rely on.

But what's the long term answer to the general problem of not enough eyeballs on open source code? It's something that will sound very familar to you, though I suspect Eric Raymond won't be too happy about it.

Money. Lots and lots of money.

Increasingly, companies are turning to commercial bug bounty programs. Either ones they create themselves, or run through third party services like Bugcrowd, Synack, HackerOne, and Crowdcurity. This means you pay per bug, with a larger payout the bigger and badder the bug is.

Or you can attend a yearly event like Pwn2Own, where there's a yearly contest and massive prizes, as large as hundreds of thousands of dollars, for exploiting common software. Staging a big annual event means a lot of publicity and interest, attracting the biggest guns.

That's the message. If you want to find bugs in your code, in your website, in your app, you do it the old fashioned way: by paying for them. You buy the eyeballs.

While I applaud any effort to make things more secure, and I completely agree that security is a battle we should be fighting on multiple fronts, both commercial and non-commercial, I am uneasy about some aspects of paying for bugs becoming the new normal. What are we incentivizing, exactly?

Money makes security bugs go underground

There's now a price associated with exploits, and the deeper the exploit and the lesser known it is, the more incentive there is to not tell anyone about it until you can collect a major payout. So you might wait up to a year to report anything, and meanwhile this security bug is out there in the wild – who knows who else might have discovered it by then?

If your focus is the payout, who is paying more? The good guys, or the bad guys? Should you hold out longer for a bigger payday, or build the exploit up into something even larger? I hope for our sake the good guys have the deeper pockets, otherwise we are all screwed.

I like that Google addressed a few of these concerns by making Pwnium, their Chrome specific variant of Pwn2Own, a) no longer a yearly event but all day, every day and b) increasing the prize money to "infinite". I don't know if that's enough, but it's certainly going in the right direction.

Money turns security into a "me" goal instead of an "us" goal

I first noticed this trend when one or two people reported minor security bugs in Discourse, and then seemed to hold out their hand, expectantly. (At least, as much as you can do something like that in email.) It felt really odd, and it made me uncomfortable.

Am I now obligated, on top of providing a completely free open source project to the world, to pay people for contributing information about security bugs that make this open source project better? Believe me, I was very appreciative of the security bug reporting, and I sent them whatever I could, stickers, t-shirts, effusive thank you emails, callouts in the code and checkins. But open source isn't supposed to be about the money… is it?

Perhaps the landscape is different for closed-source, commercial products, where there's no expectation of quid pro quo, and everybody already pays for the service directly or indirectly anyway.

No Money? No Security.

If all the best security researchers are working on ever larger bug bounties, and every major company adopts these sorts of bug bounty programs, what does that do to the software industry?

It implies that unless you have a big budget, you can't expect to have great security, because nobody will want to report security bugs to you. Why would they? They won't get a payday. They'll be looking elsewhere.

A ransomware culture of "pay me or I won't tell you about your terrible security bug" does not feel very far off, either. We've had mails like that already.

Easy money attracts all skill levels

One unfortunate side effect of this bug bounty trend is that it attracts not just bona fide programmers interested in security, but anyone interested in easy money.

We've gotten too many "serious" security bug reports that were extremely low value. And we have to follow up on these, because they are "serious", right? Unfortunately, many of them are a waste of time, because …

  • The submitter is more interested in scaring you about the massive, critical security implications of this bug than actually providing a decent explanation of the bug, so you'll end up doing all the work.

  • The submitter doesn't understand what is and isn't an exploit, but knows there is value in anything resembling an exploit, so submits everything they can find.

  • The submitter can't share notes with other security researchers to verify that the bug is indeed an exploit, because they might "steal" their exploit and get paid for it before they do.

  • The submitter needs to convince you that this is an exploit in order to get paid, so they will argue with you about this. At length.

The incentives feel really wrong to me. As much as I know security is incredibly important, I view these interactions with an increasing sense of dread because they generate work for me and the returns are low.

What can we do?

Fortunately, we all have the same goal: make software more secure.

So we should view bug bounty programs as an additional angle of attack, another aspect of "defense in depth", perhaps optimized a bit more for commercial projects where there is ample money. And that's OK.

But I have some advice for bug bounty programs, too:

  • You should have someone vetting these bug reports, and making sure they are credible, have clear reproduction steps, and are repeatable, before we ever see them.

  • You should build additional incentives in your community for some kind of collaborative work towards bigger, better exploits. These researchers need to be working together in public, not in secret against each other.

  • You should have a reputation system that builds up so that only the better, proven contributors are making it through and submitting reports.

  • Encourage larger orgs to fund bug bounties for common open source projects, not just their own closed source apps and websites. At Stack Exchange, we donated to open source projects we used every year. Donating a bug bounty could be a big bump in eyeballs on that code.

I am concerned that we may be slowly moving toward a world where given enough money, all bugs are shallow. Money does introduce some perverse incentives for software security, and those incentives should be watched closely.

But I still believe that the people who will freely report security bugs in open source software because

  • It is the right thing to do™


  • They want to contribute back to open source projects that have helped them, and the world

… will hopefully not be going away any time soon.

[advertisement] How are you showing off your awesome? Create a Stack Overflow Careers profile and show off all of your hard work from Stack Overflow, Github, and virtually every other coding site. Who knows, you might even get recruited for a great new position!