Sigh

11 Feb
February 11, 2013

This is why the trend towards basing decisions on metrics is worrying to me.

This kind of conclusion.

It’s a classic example of “correlation doesn’t imply causation”.

The mainstream has been moving away from offering demos for years. And the “top sales” charts for any given year will almost certainly be dominated by hyped AAA mainstream releases, from the studios that have stopped offering demos. Which means that yes, the top of the cumulative sales graph will be games that don’t offer demos. But that DOES NOT prove that foregoing demos was a significant factor in their financial success.

What this guy is doing is constructing a post-hoc narrative to explain a causal relationship he thinks the data is showing.

“Yes, that’s exactly what happened to you because when you put the demo out, people had seen the trailer and they’re like that’s cool, and they made a plan. They had to try that game. And then they played the demo. Alright, I tried the game, that was okay, alright I’m done. But the games with no demo, you have to buy it if you want to try it. These plans make a big difference.”

The news is filled with this kind of thing, and it’s the foundation of conspiracy theories. These two things appear to have a connection, construct a narrative that neatly ties it all together.

The takeaway lesson : practically everybody is horrible at interpreting stats, including CEOs of software companies.

1 reply
  1. Diego Doumecq says:

    Heh, not to derail the conversation or anything, but this reminds me the seriousness of the piracy problem, in the sense that we don’t have any reliable stats on the matter and yet some people are making claims of gigantic losses due to piracy.
    If you can build any narrative WITH data, imagine what you can do with almost nothing =/

    Anyway, ignore me and proceed as normal with the current conversation.

    Reply

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *