Many organizations have a great deal of difficulty actually producing their software products from source. Furthermore, most organizations do not have a comprehensive approach for the porting of their product to new platforms, the safe development of their product without interfering with production or other developers, the management and fixing of old releases, or the simultaneous use of multiple releases on the same platform.

Measuring the cost of the lack of these capabilities is virtually impossible. But it is the author's belief and experience that the reliable, timely and efficient approach to the management of the software construction process in a way that promotes an effective dev elopment process can yield substantial benefits both in terms of decreased costs and increased quality.

This paper describes the architecture of the qef system. qef is a family of tools used to manage processes typically used to do software construction. In the beginning, circa 1983, qef was developed to provide a mechanism whereby the construction of software products could be described in a host-independent way. Over the years it has been enhanced to encourage and support a better software process and a comprehensive approach to software development and maintenance and release engineering.

In hindsight, the major objectives of the qef implementation were to achieve the following:

Simplified Expression of the Construction

The expression of the construction process as provided by the user should be as simple as possible. Simple statements of what is to be done should be all that is required.

Effective Full and Incremental Constructions

The system must ensure correct, accurate and efficient construction of the products. Furthermore, the system must ensure that an incremental build (i.e., using previously built temporary files) ensures consistency between the resulting product and which that would be built by a full construction.

Effective Parameterization/Configuration Mechanism

Comprehensive mechanisms must be provided to specify, test and use parameters and configuration controls throughout the construction and software process.

Effective Process Control and Monitoring

Mechanisms to launch and monitor the construction process must be provided.

Promote an Effective Software Process

The system must support and encourage the use of better software development strategies throughout the software life cycle.

Whereas the objectives of the qef implementation and research were to achieve the above, one must examine at length the major objective or purpose of a construction system. A construction system manages the processes that transform source into a product. The organizations that can produce their products from source, and only source, reliably, efficiently and on demand are in a very small minority. The construction system must provide the mechanisms whereby the ability to produce the product from the source is almost taken for granted, but can be tested easily, cheaply and frequently.

An important implicit assumption of this objective is that if the construction system is applied a second time to the same sources, it will produce the same product. Unfortunately even fewer organizations can reproduce the identical product if the source is relocated or some other aspect of the run-time environment has changed (e.g., a different user with a different environment).

The construction system should help ensure that the ability to reproduce the product depends as little as possible on things that are not administered source.

Furthermore, the construction system must provide an incremental build facility; that is, given that some or all of the product files and intermediate build-files exist, the system skips processes that are not required to make the product identical to the version that would be produced by running the entire production process from source alone.

In meeting these stated objectives a number of characteristics and features have evolved as being of paramount importance:

Portable

The user-provided construction system input must be the same for all systems.

Flexible

The construction system must support the use of the most appropriate tool. The system should have a simple and consistent way to specify the tool to be run and its input.

Complete and Comprehensive

The construction system must facilitate and encourage the provision of the complete set of construction options. The construction system must also provide mechanisms whereby all aspects of the system are managed using the same mechanism in a consistent manner.

Dynamic and Responsive

Managing software construction is about responding to change. A construction system must respond appropriately to changes in the file system, the files themselves and the build parameters. The appearance or disappearance of a file should be appropriately reflected in the construction commands. Changes in the files that would be found by various search paths (e.g., the cc -I flag) should result in the appropriate recalibration of the dependencies.

Consistent

The construction system should present a consistent and simple interface across all platforms, projects, and users. Constructing a software product should not require any special knowledge.

Clear and Concise

The actual processing involved in a construction can be described very simply, but comprehensive and complete make files can run to hundreds of lines. The system should derive everything else required automatically, while still providing simple mechanisms to override defaults in the general and specific cases.

Simple

This is indeed one of the most difficult characteristics to achieve -- software construction is exceedingly complex -- but must be if the system is to be accepted and used successfully.

Highly and easily configurable

When dealing with software that is to be built on many different platforms by many different users, the number of configuration parameters that can be required is staggering. Most projects will require scores of parameters to specify construction controls, names of tools, tool flags, search paths, special directories and files, and other build and semantic options and controls. The construction system must provide a comprehensive, universal, debuggable, and extendible mechanisms to specify, test, manipulate, and use parameters. Furthermore, it is imperative that these parameters be available to any tool -- not just the construction system. Note: the C preprocessor is not such a mechanism!

Facilitates and encourages complete source and object separation

It is absolutely essential that a construction system support and encourage the total separation of the master source, source under development, and object files. There should be one and only one true read-only source file system which is shared by all users. Files to be modified should be copied to and changed in a user's own work space, thereby ensuring that their changes are not visible to others until ready and authorized. It should be trivial to create and link the separate directories. Furthermore, it is essential that the source path mechanism be usable by any application (e.g., the version system).

Provides suitable controls

A construction system must provide a simple consistent mechanism to manage large trees of directories and to support the constructions of part or all of those trees. Furthermore, it the construction system should provide mechanisms to monitor its progress and to halt it.