The following comments address several points in Lynzi Ziegenhagen's Master's dissertation, Evaluating Configuration Management Tools for High Assurance Software Development Projects.
Many of the items discussed below are points on which reasonable people might disagree. While God really did intend us all to program in assembly language, her intentions for CM systems are not given to us to know :-). There is no universal CM system. What one group sees as a feature, another may feel is a show-stopping limitation. The main goal of this commentary is for potential users to see the issues so that they can make effective decisions for their individual needs.
Before proceeding on to discussion, I want to say that I think Lynzi has done a fine job with this thesis. It was a pleasure to work with her in providing background information on OpenCM.
Congratulations to Lynzi on the completion of her Masters degree!
Jonathan S. Shapiro
A.1: Repository Architecture, One Central Repository Plus User Workspaces (p. 31ff)
The ``one central repository'' architecture is a special case of the hierarchical repository structure -- the case in which there is a hierarchy consisting of a singleton repository. In a system providing hierarchical repositories, the difference is largely a matter of install-time configuration. As a result, the advantages attributed to the ``one central repository'' architecture apply equally to the hierarchical repository structure provided the hierarchical system is correctly configured.
There are two relevant configuration mechanisms for OpenCM:
The server can be entirely disabled. OpenCM can be run in a local, file-based mode by making the cm command setgid to user opencm and marking all repository directories as group sticky. At certain times we have run OpenCM this way within our lab.
Using the local network configuration mechanisms of the operating system, the server port can be guarded to ensure that only local connections are accepted.
OpenCM does need a more ``direct'' means of connection control. It has been proposed by one potential user that user key authentication should be restrictable on the basis of originating server (e.g. the key would only authenticate if the connection originated from a known range of IP addresses). As with user access, similar controls should be available on a global basis.
A.2: Repository Architecture, Peer to Peer or Hierarchical Structures (p. 32ff)
The thesis's assertion that "Distributed repositories are only as strong as their weakest link." (p. 32) depends very much on the method and particulars of the distribution. Some distribution strategies defend against this sort of vulnerability better than others.
For reasons described below, the use of replicate repositories in OpenCM introduces no marginal exposure unless the originating repository is unknown to the client. As a result, the type of hostile compromise of a replicate that the thesis addresses can be reliably detected and easily corrected. Correction depends on the originating repository (or alternatively, an uncompromised replicate) being reachable.
In the OpenCM replication architecture, replicates act purely as caches. A user asking for a known revision from a replicate can check whether integrity has been preserved -- that is, they can validate that the revision received is properly signed and that all of the cryptographic checksums match the objects received. No special action is needed to check this -- the OpenCM client checks these things automatically. Similarly, the replication mechanism performs an integrity check as replication occurs. Detection of integrity failure at a replicate is 100% reliable if the client holds a known-correct signature checking corresponding to the originating repository. Establishing such a known-correct verification key is an important part of configuring OpenCM for high-assurance use.
In the event of an integrity failure (including a compromised replicate), the user is free to connect to the originating repository rather than the replicate. There are many reasons why a replicate may not have up to date information (e.g. an incomplete transmission). Not all result from hostility or compromise.
To my knowledge, there is no way in principle for any client-side tool to prevent integrity or security failures in the server-side repository. A server necessarily relies on both its host operating system and its own correctness to resist compromise.
Distributed architectures do introduce exposures, and that this is commonly a problem in peer to peer systems. In peer to peer systems, there is usually no central source of definitive integrity for a given line of development. As a result, compromising any peer tends to compromise the entire peer set. This is why we did not design OpenCM as a peer to peer system. The usual argument favoring peering systems over replication systems is write bottlenecks. Our experience with OpenCM suggests that this concern is not well motivated in practice.
B.3: Repository Structure, COTS Database (p. 33ff)
Under "advantages", the thesis states that "The advantages that come with a powerful database product include querying and reporting capability that can be used for audit purposes, atomic transactions, and finer-grained access control than provided by a file system."
We considered using a conventional relational database as a back-end for OpenCM for the sake of storage management, extensibility, and potential integration into other applications. The attractions of this sort of integration are considerable, but so far, our conclusion has been that this presents unacceptable risk for several reasons:
Current COTS DBMS systems provide entirely inadequate authentication and security provisions. Our experiences with systemwide compromise resulting from reasonably diligent deployment of both MySQL and PostgreSQL have been thoroughly unpleasant. We would not, at this time, seriously consider the deployment of either product on a security-sensitive machine.
A DBMS is subject to all of the vulnerabilities of the file system. Any user who can compromise the file system can compromise the DBMS store as well, regardless of storage method.
The number of lines of code in a modern file system is several decimal orders of magnitude smaller than the number of lines in even a small DBMS. The file system is a higher confidence storage method.
I tend to think that the reporting advantages of a database are overrated. Any CM system having the ability to export its objects in some standard encoding (e.g. XML) can be scripted using the user's scripting language of choice. While OpenCM does not yet have this ability, it is very much on the drawing board. The current web browsing engine was partly built as a test of feasibility for this.
It bears emphasis that this is an opinion, and surely likely to be a controversial one!
D.1: Access Control Granularity, Defined at the Repository Level (p. 38ff)
Access control at the repository level provides neither integrity nor any particularly useful security for normal usage situations. However, it is essential for recoverability.
Consider a situation in which an individual user's password or acccess key has been compromised or stolen. The first step in recovery is to ensure that no further damage can be done. There is no time to hunt down a list of all projects that the user is working on. The best thing to do is to disable the user's access to the repository entirely.
If the repository provides no means to disable access on a per-user repository-wide basis, the only thing you can do is remove the user. Unfortunately, removing the user removes information that you need in order to report the audit and logging trails correctly.
The operating assumption of OpenCM is that the repository may be compromised, and that the tool must make available means for recovery. Backups are an insufficient recovery mechanism unless integrity audits are regularly performed.
More likely, however, is that the user's client machine will be compromised. Client machines are often administered haphazardly, and the keys are therefore at risk of theft. The OpenCM architecture is careful to preserve adequate logging information to allow the changes made by a given user to be audited.
D.3. Access Control Granularity, Complex Access Control (p. 39ff)
The thesis suggests that permissions finer than branch-level should be incorporated into the CM system. We considered such permissions in OpenCM and ommitted them in the first version of the tool for reasons of complexity.
Sub-branch permissions are trickier than they first appear. Preventing a documenter from modifying C source code, for example, does not prevent them from removing all C source files. By the time all necessary rules are incorporated and debugged, a very complex set of checks needs to be performed to ensure that the necessary requirements are being met.
The set of checks needed depends heavily on both the nature of the project and the semantics of the objects under management. In some cases, they may also depend on the phase the project is in -- early stage projects have different requirements than projects that have entered lifecycle maintainance. There appears to be no universal set of primitive rules likely to meet the needs of all projects.
The long-term plan for this sort of issue in OpenCM is to provide a mechanism for both client-side and server-side predicate execution associated with commits (and other events of interest). This will allow a project administrator to supply arbitrary (safe) programs that can determine whether a given commit should be permitted. Given the complexity of building the first version of OpenCM, we decided to defer this capability until major version 2.
F.3. Configuration Definition and Enforcement, Set of All Files and Their Related Changed Docuemnts and State History (p. 43ff)
In order for a CM tool to provide this sort of control, it becomes necessary for the tool to understand both the semantics of the development workflow and the semantics of the objects under management. As with fine access control checks, there appears to be no universal solution for such things. Worse, a process that is natural and appropriate for one group may be invasive and burdensome for a second.
The long-term plan for this sort of issue in OpenCM is to allow such requirements to be integrated into the commit checks via a safe scripting language. Our sense is that the most effective point at which to enforce such policies is at commit time.
G.3: Making History Immutable, Enforced by Cryptographic Hashes and Digital Signatures (p. 45ff)
Hashes and signatures do not prevent change of history records -- they provide only detection. The OpenCM server ensures that all modifications to branches are append only (that is, that they do not overwrite previous changes). Thus, the method used by OpenCM combines the advantages of the "append-only store" method with detectability.
J.2 Export/Import, Import/Export function in tool (p. 51ff)
Our experiences with database import/export have not been as promising as the thesis suggests. In practice, applications tend to use proprietary schema features that limit the portability of data from one database to another. In the context of high assurance configuration management, this tends to mean that the same database must execute on both the untrusted and the trusted hosts. It isn't useful in this application to export individual records and objects -- a coherent object graph needs to be merged, which requires application support.
In OpenCM, our feeling is that import/export may not be the right model for migrating data from untrusted to trusted hosts. Our intended usage model for high-assurance development would be to proceed as follows:
Establish a common set of user identities on both systems. Even though most users will not be permitted access to the high-assurance host or repository, a common set of users facilitates continuity of logging and merging.
Use working branches on the untrusted repository for day to day development.
Copy the untrusted repository to temporary storage on the trusted repository host. Replicate the untrusted repository into the trusted repository using the replication facilities of OpenCM. Note that because of integrity checks this cannot compromise any existing state in the trusted repository.
Now perform an audited merge of the untrusted development branch into the audited (trusted) development branch.
If this approach is adopted, no import/export is required at all.
K. Threaded Discussions (p. 51ff)
As one of the contributors to Netnews (known as ``news groups'' these days) and as a contributor to the Xanadu hypertext architecture, I was bemused to see this requirement in a configuration management system requirements document. While the motivation is clear, issues like problem tracking may be more immediately relevant.
My feeling is that threaded discussion systems do need to interoperate with configuration management, but such discussion systems are a separate, major application. One definitely wants to be able to reference a discusssion thread from the CM system, but it is unclear whether the discussion thread itself needs to be under configuration management.
The main common underpinning for discussion, change tracking, and software configuration management seems to be an append-only repository providing appropriate integrity and security guards.
While we haven't broadcast the fact very hard, the OpenCM system actually has three distinct architectural layers: the repository layer provides authentication, integrity, and append-only storage, the application layer provides the OpenCM application, and the client provides presentation. One of the back-burner notions here was to augment the current system with change tracking and discussion capabilities at a later time. The key challenge is to work out how things like indexing should happen, so that presentation tools can establish persistent ``views'' on the data set corresponding to the organizations they need to present.
Unfortunately, persistent indexing forces the repository server to adopt most of the schema support and query mechanisms that make databases complicated. It is unclear how to achieve a general solution without significantly increasing the likelihood of bugs and vulnerabilities in the server.
VI. Conclusions and Recommentations (p. 59ff)
Everyone likes to see their work favorably reviewed, and I am certainly no exception to the rule. Given the conclusion, it is probably bad form to quibble. One qualification on the conclusion is correct, but is stated in a way that may lead to security exposure if misunderstood.
The conclusion states (p. 59) that ``... physical security, limited user access, and separation from other development efforts ... mitigate the other tools' disadvantages .. making the other tools more comparable to OpenCM.'' This conclusion is absolutely true, but the implications deserve a moment's contemplation.
On the ``Everyday CM'' host, few of these environmental protections will exist. Since the everyday CM state and the trusted CM state will periodically be merged through an audited merge process, the ability to quickly and simply detect suspicious state in the everyday CM repository is necessary to support auditable integration. Assume, for a moment, that the changes ``pass'' the integration audit. One would then like to merge the histories and object graphs in the respective CM systems to provide a complete stepwise record of everything that has been done. This cannot be done safely unless the derivation history of the untrusted development path can be validated back to some previous trusted state.
The ability to quickly back-track an everyday branch to its (previously audited) parent branch version and perform a robust ``diff'' for inspection purposes turns out not to be trivial, because the everyday repository may have been compromised in a way that exploits modifications to the change history. It is difficult to detect such changes robustly unless integrity checks are performed on the everyday CM server.
Other, Prioritization of Risks
In the design of OpenCM, our sense was that the greatest source of risk was client-side compromise. While servers can and do get penetrated, they are usually supported by some form of backup scheme and consistent administration practices. Concentration of risk leads to concentration of attention.
On client systems this isn't true. Users will continue to read email, and in consequence they will continue to suffer various forms of penetrations. The single most vulnerable point in a cluster of OpenCM users is the authentication mechanism.
In consequence, much of the OpenCM architecture was designed with an eye toward audit and recoverability. We assume that one or more user machines will be compromised during development, and that some recovery action will be required.
Other, Non-security Risks
McVoy's comments about integrity checks in BitKeeper (p. 53) are apropos: ``So far, these checks have found multiple bad memory DIMMS, many NFS corruptions, Linux/XFS corruptions, and a few SPARC/Linux cache aliasing bugs. All of these errors are likely to go undetected [in a system that does not provide strong integrity checks.]'' Because it uses a similar checking strategy, OpenCM shares many of the checking advantages of BitKeeper.
Not all threats are security threats! The risk of damage to valuable ``soft'' artifacts where significant dollars and time have been invested is definitely worth some attention.
Back to the overview.