5.6 Risk Software

To Be Supplied

5.7 Development Testing

Having made all of these assertions, it will now be noted that then focus here is on the initial phases of development testing by a contractor in an acquisition program. The reason is that this phase of testing more than any other drives the cost and prospects of the test program.

5.7.1 Test Goals
Obviously, the goals of a test program can address technical, programmatic and political risks. Testing to support the meeting of exit criteria when such criteria are imposed is an example of all three types of goals rolled into one. In the following it is assumed that the test goals are known. The implementation of a test structure to achieve the goals is the focus here, and it is seen that to a surprising degree the test structure is independent of the goals!

In this context there is good news and bad news.

5.7.2 Good News & Bad News
The bad news: The design of a development test program is an art. As an art, this design task is best done by an experienced and gifted individual or individuals, people who have the knack for getting the most data with the least resources. Smarter, more experienced and more gifted people will do a better job than others. The bad news is, of course, that such people may not (and statistically will not) be available.

When faced with a budget problem for the development tests for a 1500 C furnace for a Spacelab application, a bright young design engineer suggested that only the hot section of the two-section bore-type furnace be built and tested as an engineering model since only the hot section was pushing the state-of-the-art. He showed that this "bobtail" test article would provide data for calibration of the analytical models for the hot and the cold ends, provide the necessary materials testing, test the control system, etc. Considerable cost was avoided, the manufacturing schedule was compressed, and all of the original development test goals satisfied. The fellow has a knack for seeing such possibilities.

The good news is that an adequate, if not brilliant test plan can be developed by virtually anyone with the time and resources to examine the program's structure and the willingness to iterate the test planning through a couple of cycles.

5.7.3 Build-Test Matrix
The model recommended for the planning of tests is shown in a summary form in Figure 3.. The model consists of the generation breakdown (or WBS or drawing tree or whatever is available), a build-test matrix for the test program (constructed in support of development of the SEMP and the RMPP), and the master schedule for the test program. These elements are shown for a notional system with three major subsystems: A, B and C with levels of risk identified for each (high or low in this graphic).

In this notional example, A might be a fluid system, B includes all electronics, and C might be the software. Assume that the software is a first-time application by the developing organization. Hence, it is high risk. This software risk is assumed to bleed over to the hardware for element A1 (say motor-controlled valves rather than manually controlled). Other elements are assumed to be low risks. Note: Proof of principle testing is assumed to have been previously accomplished or not needed here.

The tree and the schedule are, of course, usual elements of program planning. The build-test matrix is a little less usual. This matrix describes the (downward) flow of maturity of hardware as one goes from initial bench-level breadboard tests to final tests of the all-up system. Three key aspects of the matrix should be noted:

Elements of the system enter the matrix (come into existence) at vertical locations that reflect the degree of risk. A system element "moves" downward as it matures through the various phases: breadboard, brassboard, engineering model and pre-production prototype. Not all elements pass through all phases. Mature (less risky) items can enter the flow at lower positions. Note that A1 and C of the notional example enter at the highest level because of their high risk.

Each "box" of the matrix indicates a test that implies: specific purposes and data requirements, test articles, test fixtures, special test equipment, facilities, personnel, documentation, a planned test intervals, and test budgets. The overall structure of the matrix and these specific aspects are readily captured in a data base format and/or wall chart. The design of a test program is really a matter of juggling these individual test requirements to develop an overall flow that is effective. However, the concept of the build-test matrix provides the framework for keeping the details in order.

The build-test matrix loosely follows the generation breakdown, but in an inverted form. This relationship is implied by the large flow arrow of the chart. The mapping is not absolute in detail, but it is always present to some degree in any development effort. In fact, this mapping is one test of the quality of the design of a generation breakdown. If the matrix does not follow to some degree from the breakdown then the breakdown/matrix should be seriously considered for revision. (Of course, the breakdown must also pass muster against make-or-buy planning, the specification tree, interface working group structure, etc.)

The build-test matrix should also reflect the master test schedule that, in turn, must be tied to the program master schedule. In particular the build-test matrix should support the schedule requirements. Again, as with the breakdown, the matrix and the schedule have (or should have) a common underlying structure. If this structure is not obvious then something is wrong and the design of the schedule and matrix should be iterated.

5.7.4 Comments to the Build-Test Matrix
The matrix is presented as a reflection of the breakdown and the schedule. In many instances, the planning of this matrix drives the other two items.

The matrix can be used to identify at what points test articles, support equipment, documentation, personnel, facilities, etc. are needed in terms of maturity, inclusion as procurable items in the breakdown, and required timing from the schedule. Also, the matrix can be used to identify dead-end hardware to avoid and to identify possibilities of reuse and early use of specific items.

As a visual aid, the generation breakdown can be highlighted in color (red = high risk, blue = moderate risk, green = low risk).

It should be understood that for illustration purposes only tiers of the build-test matrix to subsystem levels are shown. In reality, some elements of the system may require tiers from the level of components.

Such lower tier testing should be based on incrementally achieving functionality (or actually confidence in functionality of designs) in the items that make up each assembly before proceeding to the next tier.

An example of a good approach is the real example of a company performing a first-time (for it) development a high temperature, automatically controlled furnace with sample handling mechanisms. The company's expertise was in power supplies, thermal analyses, structural design, and some competence in control software. The development cycle for the mechanisms was first a kinematics prototype, update of the kinematics prototype to powered operation (lab power with manual control), update of the control to prototype software control, incorporation of engineering models of the final power supplies, and finally incorporation of the final control. The result was an extended, but very low risk approach.

5.7.5 Tricks of the Trade
The best of test programs uses a minimum of resources (people, time, money and hardware) and produces a maximum of data to support the test goals. Some tricks of the trade are:

* Spin-off of development test sets as test sets for prototype and final systems.
* Multiples of some prototypes to support branching of later test activities.
* Minimizing dead end test hardware.
* Testing at no higher in the build-test matrix than required for risk management.
* Alternate development flows.
* Recognizing and minimizing physical risks to test articles and test equipment.
* Do not combine test for two different high risk items until the risks for at least one of the items has been resolved.

Finally, the interdependency of analysis and test is a key element of what the tests must accomplish. This aspect is discussed in the section on integration of analysis and testing.

5.7 Engineering Analysis
There is, of course, no need to justify analysis an a key factor in the design and development process. However, there is an apparent need to lobby for the development of analysis plans, and the need for general rules in the SEMP and/or program plan re the balance of test and analysis to be performed. The latter point is discussed first.

5.7.1 Test Analysis Rule
It is recommended that some sort of rule be promulgated re how specific features of the design will be verified as the development process unfolds. In this sense, verification is the process of establishing a confidence level in design decisions at each point in the development process. An example of such a rule might be that every WBS element with a functional and/or interface requirement be twice tested and twice analyzed for each function and/or interface prior to the beginning of the acceptance activities. The rule can include the proviso that at least one of the tests or analyses must be in an integrated configuration of at least the subsystem level (for a system-subsystem configuration for the end-item(s)). The functions of concern include: dynamics, stress, thermal, power, control, mass properties, stability, etc.

Thus a tire for an all-terrain vehicle may be assessed for influence on vehicle dynamics analytically with a spring mass, tested at a suspension assembly level to, in part, verify the knowledge from the simple analyses, and, of course, instrumented and tested in cross-country ride tests of the system.

Every product manager, subsystem manager, Design Build Team, etc. should have some such rule and verify that it is met or justify omissions. This is a simple rule and it is not hard and fast, but it will help integrate the test-analysis program.

5.7.2 System Functional Analysis Plans
The second aspect of the analysis recommendations is that each analytical discipline be required to submit System Functional Analysis Plans that detail what will be analyzed when, with what model, and with what expected results. The SFAPs will span the range of analyses from hand-calculations to final CDR-level models/simulations. The purpose is to assure that there is closure between the scopes of the test and analyses and to avoid omissions in the analytical efforts.

The plans will describe the fidelity of the modeling for each phase of the program: pre-SDR, pre-PDR, pre-CDR and pre-Test (development, qualification and acceptance). These plans will describe the decisions supported by the analyses. Analysis schedules will be included to show the support to trade studies, development tests, monitoring of subcontractors, development of specifications, assessments of environments, etc.

The plans will define test data required to support model development and/or calibrate the analytical tools.

Incremental analysis reports will be prepared for each major milestone: Concept analyses at the DR, a preliminary report at the DR and a final report at the DR. Each report has the data required for the initial phases of the next phase.

5.7.3 Types of Analyses
There are few things that cannot be analyzed in support of design at some level of utility with the tools and techniques available today, but too often organizations opt for the more expensive and riskier empirical basis for product development (scratch and evolutionary). Programs that do not have a strong analytical culture and that do not carefully integrate analysis and test in a sort of leapfrogging to the final products are not doing as professional a job as they should and could for their customers.

Analyses tend to be divided as engineering and specialty. The engineering analyses include such aspects as thermal, stress, dynamics, kinematics, power distribution, and control. Such analyses should be performed for all mechanical and human elements of any system that must perform in anything other than benign environments doing anything other than routine, non-critical and non-safety functions. Any "stress" whatsoever should invoke analysis at the level of engineering models. For example, analysis of the human element should include such aspects as the physiothermal response if the environment is outside the comfort zone. In assessing the need for such engineering analysis, the rule should be that omissions must be justified.

Specialty analyses include such aspects as signatures, radar cross sections, C3I analyses and trades, vulnerability, survivability, human factors, effectiveness studies (at various levels of engagement), mobility, logistics, reliability, penetration analyses for armor, terrain generation, fire control, and timing. Every program should include an evaluation of the scope and fidelity of the possible analyses, and then omit analyses only upon justification. (Lack of time, money and priority is often sufficient justification.) A list of possible analyses can be derived rather easily from literature searches.