blog dds: 2007-01-19 — Open source as a paradigm for evolving complex systems

Scientists in the 1980s hotly debated the feasibility of US's proposed Strategic Defence Initiative, commonly known as Star Wars. One argument concerned the amount of software needed to control the missile detectors and weapons.

The size of computer programs is often given in terms of lines of code; about 50 lines fit on a computer's screen, 65 on a page. The size of SDI's software was commonly estimated at 10 million lines of code, and was considered as beyond the then-current state of the art.

The open source model

Fast forward to the present. On September 2006, a group of volunteers released a free operating system called Debian GNU/Linux. Amazingly, the size of Debian is 213 million lines of code. If we were to print those instructions on fanfold paper, the printout length would cover the driving route from Zurich to Rome. The largest comparable commercial offering is the—several times postponed—release of Microsoft Windows Vista, which is estimated at around 50 million lines of code.

How could a volunteer effort deliver a system four times the size of Microsoft's offering; a system of proportions considered infeasible in the 1980s? In common with many other things in life, size isn't the software's key quality. Yet size determines development effort, complexity, and the likely number of errors. When scientists at Carnegie Mellon's Software Engineering Institute wrestled with the problem of creating large software systems, like that needed for the SDI, they suggested tight centralized planning, control, and monitoring of the development process. Many organizations around the world have successfully embraced these principles to deliver quality software on time and within budget. Yet, as the software compatibility problems that delayed Airbus's A380 production demonstrate, there are limits to what can be achieved by precise planning and control. The thousands of volunteers behind the Debian GNU/Linux system work on completely different principles, using the open source development model.

Instead of a management hierarchy dictating how and on what each developer works, volunteers find tasks that interest them and work when they find time. Instead of having the software's source code (essentially its design) hidden as a company's proprietary secret, the open source programmers publish their code on the internet for others to read, learn, experiment, modify, improve, and reuse. Instead of erecting organizational barriers to control their development process and their turf, the open source development model brings all its artefacts in the open: discussions between developers and users, problem reports, the historical record of changes made to the software, even results of failed tests. Under this model transparency and self-regulation nurture each other, fostering organic growth and taming complexity. Can this approach really work?

The limits of central planning

In nature this approach has been a runaway success. Evolution works with freely available DNA sequences: organisms choose their mating pairs and the resulting offspring reproduce or die depending on their merits. Through this process we humans share a large fraction of our genes with most known vertebrates, and uniquely also have a brain that can design spaceships. Looking at economic systems, the results have been mixed. The limits of central planning as a way to coordinate a complex system have manifested themselves in many command economies. On the other hand, as decent human beings we can't watch our fellows fall by the wayside with the indifference (or even glee) we discuss the extinction of the dinosaurs or the bankruptcy of Enron.

It seems that through the open source software movement our society has stumbled on a new powerful way to organise and develop highly complex systems—systems whose implementation often appears intractable using current command and control management structures. Although this organisational system is not a panacea, it opens exciting new possibilities for expanding our society's commons through advances in engineering, science, and culture. Through transparent organic collaboration we can conquer complexity challenges in fields like biotechnology, computing, and space exploration, harness the information explosion brought by the internet, and, possibly, even create art forms that face popular culture in the eye.

It would be nice if Ronald Reagan's SDI was only a historical footnote, conveniently serving as this article's opener. Unfortunately the race to militarize space has resurfaced through various National Missile Defence systems. Could the software for controlling such systems be built using open source development methods? I find the idea of volunteers around the globe cooperating to develop weapon control software absurd. And this makes me optimistic regarding the dark side of technological evolution. When volunteers man the forefront of technology, the outlook can only be bright.

Note: This article has appeared in a special issue of the on-line journal Re-public, titled The promise of the commons. Here are some other contributions to the special issue; visit the online site for more.