Figure 1.1, “History and geneology of the four systems” shows the history and genealogy of the systems I examine. ^[1] All four systems started their independent life in 1991–1993. At that time affordable microprocessor-based computers that supported a 32-bit address space and memory management led to the Cambrian explosion for modern operating systems. Two of the systems, FreeBSD and OpenSolaris, share common ancestry that goes back to the 1978 1BSD version of Unix. FreeBSD is based on BSD/Net2: a distribution of the Berkeley Unix source code that was purged from proprietary AT&T code. Consequently, while both FreeBSD and OpenSolaris contain code written at Berkeley, only OpenSolaris contains AT&T code. Specifically, the code behind OpenSolaris traces its origins back to the 1973 version of Unix, which was the first written in C [Salus 1994]. In 2005 Sun released most of the Solaris source code under an open-source license.

Figure 1.1. History and geneology of the four systems

Linux was developed from scratch in an effort to build a more feature-rich version of Tanenbaum's teaching-oriented, POSIX-compatible Minix operating system [Tanenbaum 1987]. Thus, although Linux borrowed ideas from both Minix and Unix, it did not derive from their code [Torvalds and Diamond 2001].

The intellectual roots of Windows NT go back to DEC's VMS through the common involvement of the lead engineer David Cutler in both projects. Windows NT was developed as Microsoft's answer to Unix, initially as an alternative of IBM's OS/2, and later as a replacement of the 16-bit Windows code base. The Windows Research Kernel (WRK) whose code I examine in this chapter includes major portions of the 64-bit Windows kernel, which Microsoft distributes for research use [Polze and Probert 2006]. The kernel is written in C with some small extensions. Excluded from the kernel code are the device drivers, and the plug-and-play, power management and virtual DOS subsystems. The missing parts explain the large size difference between the WRK and the other three kernels.

Although all four systems I examine are available in source code form, their development methodologies are markedly different. Microsoft and Sun engineers built Windows NT and Solaris within their companies as proprietary systems with minimal if any involvememnt of outsiders in the development process. (OpenSolaris has a very short life as an open-source project, and therefore only minimal code could have been contributed by developers outside Sun in the snapshot I examined.) Furthermore, Solaris has been developed with emphasis on a formal process [Dickinson 1996], while the development of Windows NT employed more lightweight methods [Cusumano and Selby 1995]. FreeBSD and Linux are both developed using open source development methods [Feller and Fitzgerald 2001], but their development processes are also dissimilar. FreeBSD is mainly developed by a non-hierarchical group of about 220 committers who have access to a shared software repository that was initially CVS and currently Subversion [Jrgensen 2001]. In contrast, Linux's developers are organized in a four tier pyramid. At the bottom two levels thousands of developers contribute patches to about 560 subsystem maintainers. At the top of the pyramid Linus Torvalds, assisted by a group of trusted lieutenants, is the sole person responsible for adding the patches to the Linux tree [Rigby and German 2006]. Nowadays, Linux developers coordinate their code changes through git, a purpose-built distributed version control system.

I calculated most of the metrics reported here by issuing SQL queries on a relational database containing the code elements comprising each system: modules, identifiers, tokens, functions, files, comments, and their relationships. The database's schema appears in Figure 1.2, “Schema of the database containing the code's analysis”. ^[2] I constructed the database for each system by running the CScout refactoring browser for C code [Spinellis 2003], [Spinellis 2010] on a number of processor-specific configurations of each operating system. (A processor-specific configuration comprises different macro definitions and files, and will therefore process code in a different way.) To process the source code of a complete system CScout must be given a configuration file that will specify the precise environment used for processing each compilation unit (C file). For the FreeBSD and the Linux kernels I constructed this configuration file by instrumenting proxies for the GNU C compiler, the linker, and some shell commands. These recorded their arguments (mainly the include file path and macro definitions) in a format that could then be used to construct a CScout configuration file. For OpenSolaris and the WRK I simply performed a full build for the configurations I investigated, and then processed the compilation and linking commands appearing in the build's log.

Figure 1.2. Schema of the database containing the code's analysis

In order to limit bias introduced in the selection of metrics, I chose and defined the metrics I would collect before setting up the mechanisms to measure them. This helped me avoid the biased selection of metrics based on results I obtained along the way. However, this ex ante selection also resulted in many metrics—like the number of characters per line—that did not supply any interesting information, failing to provide a clear winner or loser. On the other hand my selection of metrics was not completely blind, because at the time I designed the experiment I was already familiar with the source code of the FreeBSD kernel and had seen source code from Linux, the 9th Research Edition Unix, and some Windows device drivers.

Other methodological limitations of this study are the small number of (admittedly large and important) systems studied, the language specificity of the employed metrics, and the coverage of only maintainability and portability from the space of all software quality attributes. This last limitation means that the study fails to take into account the large and important set of quality attributes that are typically determined at runtime: functionality, reliability, usability, and efficiency. However, these missing attributes often depend on factors that are beyond the control of the system's developers: configuration, tuning, and workload selection. Studying them would introduce additional subjective biases, such as configurations that were unsuitable for some workloads or operating environments. The controversy surrounding studies comparing competing operating systems in areas like security or performance demonstrates the difficulty of such approaches.

The large size difference between the WRK source code and the other systems, is not as problematic as it may initially appear. An earlier study on the distribution of the maintainability index [Coleman et al. 1994] across various FreeBSD modules showed that their maintainability was evenly distributed, with few outliers at each end [Spinellis 2006]. This means that we can form an opinion about a large software system by looking at a small sample of it. Therefore, the WRK code I examine can be treated as a representative subset of the complete Windows operating system kernel.