Shell scripting for software developers

In an open online edX course on Unix tools I was running over the spring with more than a thousand registered learners, I got asked for ideas on how shell scripts can be useful. This is an intriguing question, because the course focuses mainly on performing one-off tasks in the areas of software development, data engineering, and system administration, rather than automation through shell scripts. In response, I posted how shell scripting improves my personal productivity. Here’s my take on how shell scripts are employed in diverse software development tasks. I plan to post further installments on system administration and data analytics.

To produce the examples listed here, I searched the entire FreeBSD source code tree for files with a .sh (shell script) suffix. In total I found 1305 scripts (by running the Unix find /usr/src/ -name \*.sh | wc -l pipeline). In summary, I found that shell scripts are used for testing, as program wrappers and adaptors, for stand-alone program implementations, as code generators, for build orchestration and automation, for release management, and for crafting development tools. My experience is that other large software systems use shell scripts in similar ways.

Testing

Shell scripts are used to automate integration testing, by running the program under specific configurations and scenarios and verifying that the program has the expected behavior. They are also be used for regression testing, by running a program and comparing its output against results of previous runs. The FreeBSD /tests suite contains 261 tests of both kinds. Many other programs have test scripts bundled with their source code; see for example the pkill command tests directory. As an example of regression testing consider the OpenSSH regression tests. In total, another 355 test shell scripts reside in individual programs’ tests directories and 227 come bundled with the separate NetBSD test suite.

Program wrappers and adaptors

In some cases, either by design or through evolution, a program can be easily written by having a shell script call another one. Consider the following examples.

• The program ranlib, which adds a symbol table to an archive (library) file, used to be a stand-alone program. It is now implemented as a one-line shell script, which calls the ar program to do the actual work.
• The lp POSIX 1003.2-compliant print spooler interface is implemented as a wrapper around the traditional BSD lpr command.

Program implementations

Some programs are easy to construct by gluing together other existing Unix commands in a shell script. Here are some examples.

• The adduser and rmuser user management commands.
• The manctl program, which compresses or decompresses the system’s manual pages.
• The service system service management command (the equivalent of the systemd systemctl command).
• The ypinit command, which builds and installs network information service (NIS or yellow pages) databases.
• The freebsd-update, which fetches and installs the system’s binary updates.

Code generation

Many scripts are used for creating code from higher level descriptions. Consider the following cases.

• Trivially, the flex mkskel script, reads a text file containing the boilerplate of the generated lexical analyzer, and converts it into a C array that can be compiled into the lexical analyzer generator executable program.
• The curses library MKparametrized script generates C source for an array specifying whether terminal capability (termcap) strings should undergo parameter and padding translation.
• As a more sophisticated case, the kernel’s makesyscalls shell script, reads a declarative specification of the supported system calls, and generates the C code that implements the interface to call the lower-level implementation routines.

Build orchestration and automation

Although shell commands can be readily specified in most build systems, such as make, complex build actions are better documented and tested by putting them in a separate shell script file. For example, yacc uses such a script to orchestrate the build process, while libsodium uses separate shell scripts to initiate the make-based build process on different systems, ranging from iOS to Android. More widely, the meta2deps shell script looks for “meta” files and extracts information needed to deduce build and src dependencies, depend-cleanup cleans dependencies after some difficult-to-track source code changes, and tinder builds parts of the source code tree on all supported architectures.

Release management

Several scripts are used for putting together the system’s release, including the distribution packages, the release notes, the distributed file manifest, and the architecture-specific ISO images.

Development tools

Shell scripts are also used for constructing ad hoc tools for bespoke development tasks. For instance, the kernxref tool creates a cross reference of a kernel’s symbols, while the kerncruft tool finds unused kernel source code files.

If you’d like to learn more about how you can apply Unix command-line tools in software development (and other) tasks you can register for my free open edX course through this link.

Comments   Toot! Tweet Share