Development Blog

Geeks have been so used to hearing about how open systems beat out close systems that they seem surprisingly blind to what I see is an obvious evolution in the web into which closed systems are winning. The 90′s were all about tearing down of walls and openness and standards ruling the roost. The Internet beat AOL, Bittorrent beat RIAA, reddit and digg are beating the New York Times. It’s easy to conclude from this that, on a long enough timeline, open systems always prevail.

But what I see is a shift in the opposite direction in which we increasingly depend more on facebook, the iPhone and other closed, managed services. I’m not quite prepared to stake my thinking on a single answer at this point but it seems to be based a lot on a closed service’s greater ability to create a compelling user experience and a shift in technological maturity which has made this a greater selling point.

I had the chance to use a Windows Mobile phone for 2 weeks and the user experience was absolutely shockingly abysmal compared to my iPhone. When I talked to Windows Mobile defenders about this, they inevitably raise the ability to use it as a wireless AP and as a turn by turn navigation device and have background programs and as a freaking multimedia message sender but I think such concerns are becoming increasingly more irrelevant to a larger proportion of the population. What’s compelling about the iPhone is that the things it can do, it does beautifully and that it has the fortitude to do what it took to make that experience compelling. Unfortunately, compelling user experiences are sometimes at odds with freedom.

Freedom also means the freedom to be mediocre and there’s frankly a lot of windows mobile apps which are shockingly poorly made. Freedom also means freedom to build complex things which require a complex abstraction to support them. Apple’s decision to have one program running at a time is restrictive but it also means it never needs to expose manual memory management to the user.

As a developer, I’m appaled at the arrogance and tone deafness that Apple treats the developer community but I also recognise that these are some of the eggs Apple has to break to make a compelling user experience omlette. Other phone manufacturers still seem to be missing the point and are now bending over backwards to emphasize how open and free they are without realising that’s precisely why people abandoned such platforms as unusable.

**WARNING: INCOMPLETE FOR NOW, feedback still welcome*

This fascinating visualization of the writing process was at the top of Hacker News and unfortunately, it’s easier for you to view it than for me to try and describe it for you. But looking at in made me consider how long it took for that idea to fully develop. Stripping out the whole online component of it, when could you have realistically developed something like that? 1960? 1970? There was a good 40 years between when it became technically feasible and when even 0.1% of the computing population had even been made aware of it.

One thing I started noticing quite a while back was this consistent pattern of a brief period of innovation when a new technology hits followed by a stultifying grind in which ideas barely change. And I think the reason for this is not because all the good ideas were already developed, I was seeing absolutely brilliant ideas coming out of academia every year that were markedly superior to what we had before. I think the right explaination is that the natural unit of innovation is the idea and the natural unit of competition is the product and so when those two get too far out of sync, innovation grinds to a halt.

I can guarantee you the guys at etherpad were not the first to have this idea. At a conservative estimate, I would say that that particular demo had been tried by maybe a thousand people in the past, everyone from grad students to bored hackers to geeky artists. But consider the process from the genesis of an idea to it being a feature that people could use. If you were in the 90′s, everyone would have been writing in Microsoft Word. In order to convince anyone to try out your feature, you essentially had to build something better than Microsoft Word just in order to test this new feature. This is, to say the least, mildly impossible.

The difference between the effort required to create a feature and get it into use was so impossibly large that such an obvious feature languished for 40 years for lack of an opportunity in the market. I think the ratio of effort required for a feature:effort required for a product is an extremely important metric to optimize because even small reductions can have highly non-linear and cumulative effects. In this case, as a rough cut, I’m defining effort required for a product as the amount of work it would take to have your software be installed and used by 1,000,000 people or 10% of the market, whichever is smaller. That includes the coding, marketing, sales etc.

So what are some ways of drastically cutting down on the feature:product ratio?

• Hiring innovative people: The easiest way to get a feature deployed is to work for the company building the product. Hiring the right people and getting out of their way can push the feature:product ratio essentially down to 1 for that group of people.
• Open sourcing:
• Plugin architectures: Plugins allow you to leverage off all the other features that aren’t the feature you want to implement. If you want to prototype a cool comment system, you don’t have build an entire blog, you just download wordpress and start hacking away at a plugin.
• Good API Design:
• Great documentation: Sadly, something ignored by most open source projects
• Scripting languages: Building your product out of a dynamic scripting language allows others to easily dive into the guts of what you’re doing.
• Orthogonal design: A good plugin architecture is useless if you can’t actually add useful features because everything is so interdependant on everything else.
• Automated Deployment: Remember, the feature:product ratio doesn’t just involve coding, all the other activities that go around getting an app installed matter too. Huge complicated deployment strategies might make sense for a product but can become disproportionate for a feature. This means you need to provide one click installs, auto updates, an in-product plugin store and a vibrant viral community to help with the marketing.
• Lots of eyeballs: The larger and more active your community is, the more ideas bubble and ferment to the top and result in concrete results.

If your product can provide all of these things, they can pull a long way ahead of the competition. Look at the example of firefox vs IE, firefox has steadily gained market share because, among other things, you can install an ad blocker. Nobody on the firefox team ever built an ad blocker and it would probably have been deemed low priority if it were ever proposed to them. The sole reason an ad blocker exists today is because the person who thought about writing an ad blocker didn’t also have to contemplate writing a HTML parsing engine.

I sometimes catch myself thinking something and then try and trace back my chain of thought to the original thought that sparked it…

Also, note to self: build blog integration into Am I Normal or Not…

I was reading Paul Graham’s Startups in 13 sentences and came along this interesting piece of advice:

7. You make what you measure.

I learned this one from Joe Kraus. [3] Merely measuring something has an uncanny tendency to improve it. If you want to make your user numbers go up, put a big piece of paper on your wall and every day plot the number of users. You’ll be delighted when it goes up and disappointed when it goes down. Pretty soon you’ll start noticing what makes the number go up, and you’ll start to do more of that. Corollary: be careful what you measure.

It had been meshing with my thinking a lot about visualising valued work and community building. I think the same thing applies in both contexts, if you display each user’s post count under their username, then people are going to start posting a lot. If you implement a karma score, then people will try and do things that maximize karma. Not only exposing information about valued work important, not exposing information can also be an important design strategy. Unfortunately, for most online community organisers, there’s little thought about the effects of a feature, features are often turned on just because they’re there.

But chewing over this a little bit before going to sleep last night, I realised that inventives can be thought of as a context free instantiation of values. To have values means that you believe that certain actions are either right or wrong stemming from some base moral reasoning. What incentives do is it replaced that base moral reasoning with a much simpler system of rewards. In other words, you switch from doing something because it’s right to doing it because it’s good for you.

Because incentives are context free, they’re much easier to scale, both up and out. Every person you meet has their own rich tapestry of past experiences and beliefs which influence their value system but every person is going to get exactly +1 to their post count when they make a post.

However, the downside of an incentive system is that they never incentivize precisely what you want to reward and, whenever you get humans into the mix, you’re going to get gaming of the system. Incentives are also inflexible in the face of novelty compared to values because values stem from the motivation rather than the result of reasoning.

Values are more effective but harder to implement, incentives are easier but less subtle. One of the unique advantages that startups have is that they’re still small enough that they can make the effort to instill a strong value system and this is one of their unique competitive advantages. Some companies nowadays are starting to get the picture and have come out full force in the expression of their values but too many still try and ape large companies and hide behind a bland moral ambiguity. More startups need to realise their true values are a massive asset, not a liability and that they won’t have the luxury of having them for much longer.

For some people, life is about passions to be discovered, for me, it’s a series of obsessions that need to be managed. I’ve installed a bar in my room a few days ago and I’m currently in the process of massively geeking out about cocktails. Already, I’ve got a list a mile long of things I want to try: Exotic simple syrups, infused liquers, home made bitters, custom made ice, evaporative reduced grenadine…

This is a common pattern with me, if I choose not to become involved with something, I’m quite happy to keep my distance and only maintain an educated layman’s perspective on things. But if I dive in too deep, the only solution is to reach the very bottom of that knowledge pool and become somewhat of an expert.

This is the reason I refuse to touch coffee, I’m sure it tastes great and could become a great love but I have a sense of how deep a knowledge is required and I simply don’t have the time or money to pursue it to the degree that would satisfy me.

I occasionalyl get this error in XCode. A quick reboot of the iPhone seems to fix the problem.

To a lot of liberal Christians, evolution has become, dare I say, almost a point of identity. Even the Catholic Church has reaffirmed it’s view that evolution is totally compatible with faith and that Intelligent Design is a cruel sham of a belief. A lot of Atheists are keen to encourage this because anything that gets liberal Christians on our side and away from the Fundamentalists is a good thing. Unfortunately, I’m about to take a massive piss into this font of goodwill by stating that I think the belief that evolution is compatible with faith is based on a misunderstanding of evolution that is so prevalent even Atheists are guilty of perpetuating it.

The easy part of evolution to understand is the mechanistic aspect, the “how” of evolution. Mutation happens and features get evolved and some things become more popular because they lead to increased survival and this is how we have the natural world we see around us. Based on an understanding of this how, it’s perfectly plausible to then believe the why comes from God. God works through evolution and gives it a nudge once in a while at the right points. Evolution is just a cog in God’s plan. Of course, this is an unfalsifiable assertion so it falls outside of the domain of science. The scientists are happy, the liberal Christians are happy, why make so much fuss?

But to understand why evolution had the profoundly explosive effect it did on the scientific world and why Darwin was named the most influential scientist of the past millennium by Scientific American, one must move past the mechanistic understanding of evolution and understand the teleological aspect of evolution: the why of evolution.

Imagine for a second that you rolled a pair of dice a million times and noticed that they added up to 7 roughly 1/6th of the time. Why did the dice land the way it did? Why 1/6th and not some other number like 1/3rd or 1/12th? If you were wholly ignorant about probability, then you could believe that God decreed that all dice rolls should fall to his accord and the reason for 1/6th was because it was pleasing to his eye. But what probability does is it not only explains how it is 1/6th, but also to boil the why down to *shrug*, that’s just the way it is. Fundamentally, the why of it is that there is no why, not only is the why not God, it’s not anything. What probability does is explain away the why.

The real impact of evolution was not that it explained the natural world, it was that it provided science with a path to explain it away. Why do birds fly and beasts walk? Why are we the most intelligent of all the animals? Why do we love and war and yearn for the stars? *shrug*, that’s just how the world is. Evolution showed how complex forms and intelligent beings could have arisen following the same natural laws that govern how dice roll and planets move and by doing so, it not only removed the need for an intelligent creator, it removed the need for anything. Even 200 years after Darwin this is the basic point of evolution that is still misunderstood and this is why evolution is far more of a dilemma to faith than most people will admit.

It’s important to note though that this process is far from complete. Darwin only put humanity on the first step towards this understanding and in the last 200 years, scientists have been steadily working away towards this goal. Our understanding of how is still gradually improving every year but none of this could have happened unless Darwin had the stroke of insight that there could be no why.

Is it possible to believe in evolution and still in God? Sure, but It would be akin to showing how you could bend a spoon via sleight of hand and still believing that Uri Geller is bending them with his mind. If that’s the way it’s done, then he’s doing it the hard way.

“Don’t be afraid to polarize people. Most companies want to create the holy grail of products that appeals to every demographic, social-economic background, and geographic location. To attempt to do so guarantees mediocrity.”

- Guy Kawasaki

And I think Guy Kawasaki is absolutely right, choose focus over breadth is absolutely the right approach to take for most startups. That being said, I think there’s also something incredibly dangerous about this sort of view. The problem is, there’s an easy way to be polarizing that doesn’t work and a hard way to be polarizing that does work. They look similar enough from the outside that most people take the easy way and then blame the system for their failure.

Here is a short list of projects I’ve always wanted to do were I to have the sufficient time and means but I secretly know I’ll never get a chance to do them. If anyone else wants to take them on, be my guest:

• An investigation of what determines success by interviewing a large group of potentially successful people before they become successful. What things genuinely determine success? What things are just post hoc explainations which get engrained as superstition?
• A “Museum from the future”, with exhibits of contemporary late 20th century life from the perspective of 500 years. What will they pick to be important? What will they misunderstand? How will their differing moral and intellectual standards impose a narrative on our lives?
• A universal screen scraper framework for games education. This piece of software would be comprised of 3 components: A base layer which provides basic functionality for screen scraping games, a library of game specific adapters which turn what you see on the screen into a set of API calls that would allow you to control that game and then a user generated “AI” that could attempt to play those games. I think this would be an enormous boon for teaching kids how to program and fall in love with programming as it allows them to immediately tackle real world tasks with immediate feedback. For example, a rudimentary blackjack betting program would take all of 5 minutes and a few lines of code to code up but imagine progressing from that to building a Worms player and having it battle online with other Worms AIs.

A post on the O’Reilly Radar mirrors some thoughts I’ve been having. We have passed over a distinct phase shift in history from when only some things were remembered to when everything is rememebered and in the future, this distinction is only going to get more and more stark. In the future, every single edit in wikipedia will be preserved, every tweet and every livejournal entry, we’ll have more information on a 3rd rate reality tv star from 2005 than most presidents and this is going to radically alter the way future generations interpret the past.

The curious thing is that phase shift happened a long time before anyone thought there would be a phase shift. I estimate the phase shift to be at the point of the first world war. Everything before then will cross into a vague sort of blur, Greeks, Romans, Renaissance, Industrial Revolution. Events spaced 100s of years apart. But after that point, it’ll be WWI, Depression, WWII, Cold War. Events spaced 10 years apart.

However, the very nature of this enormous amount of data also casts a certain shape on how we understand our past. Already, I’ve noticed there are certain things which are very hard to Google for. For example, material about a person/thing before it got famous. Finding out what the world was like 10 years ago through Google turns out to be surprisingly difficult. Another rather curious quirk of Google is that it’s almost impossible to find out anything about Wikipedia. Any search you do only results in Wikipedia articles.

When most people think about the future, they map their own notions of significance onto future generations. But they often fail to account for the vagaries of time and fashion and how the future will project it’s own inevitable biases on us as we do on our past.