This article first appeared in the September 2018 Flight Test News.
Earlier this year, Nathan CAP’N Cook posted a question in SFTE’s private facebook group: Does anyone have references for “fundamentals of test conduct”? I’m not looking for “test techniques” or “crew resource management.” I am looking for what it takes to properly pace a test, keeping in mind time, space, energy, data priorities, data quality, safety, etc. Thus began a brief but very productive correspondence: he responded with a detailed explanation of his ideas. With his permission, FTN reprints it here.
Download Test Point Flow by Nathan Cook.
Fundamentals of Test Conduct
I have a preliminary concept for what I call the Fundamentals of Test Conduct. Loosely speaking, the flow is as follows: Admin-Setup-Procedure-Recovery-Admin, with entry- and exit-criteria at each transition. The construct is recursive, in that it is applied at multiple levels of conduct, each of which consists of one or more of the lower level units. It starts at the most basic level, the fundamental building block of test conduct, which is a “test point” (defined as the indivisible chunk of procedure that generates data required to partially satisfy a test objective). The next level is the “test mission” (mission is a flight test centric term that groups test points into a quantity that can be completed in the course of a single flight). It also includes the following elements or levels: “test set” (the group of test points sufficient to satisfy a test objective); “test project” (the group of test sets packaged to meet the combined test objectives of a single test plan); “test program” (the group of test projects packaged to achieve a medium- to long-term milestone for the system under test); and so on.
At each level, the entry and exit criteria govern how well the test will go. The transitions are not (inherently) automatic. One observes a sense of “monitor and control” here, not as an abstract concept but concretely as a continuous comparison of the situation to the criteria. The analogy is a “while-loop” in which the criteria serve as the logical gate to break out of the current state and move to the next. These ideas need not be novel concept. Rather, my purpose is to develop a shared, explicit lexicon, built from concepts which experienced testers use various terms for and which are often understood implicitly.
The concept of test discipline (as in a disciplined approach to test, not a technical discipline such as structures or propulsion) also enters in here. How well is the entire test team doing at the following sub-tasks?
1) Communicating the current state of the test (which card/run; are we in admin/setup/procedure/recovery or at a transition/pause point);
2) Monitoring the exit/entry criteria (is setup complete; are the procedure steps being correctly executed; is a normal recovery or an early termination/abort required, etc.);
3) Coordinating the transitions between states (begin setup, start run, recover, next card is).
These ideas bring to mind Boyd’s OODA loop, but I haven’t delved deeply into the parallels/connections here. This summary gets at the heart of what I’m trying to clarify when I say “fundamentals of test conduct.”
In other words, what are the most basic concepts that make up test conduct and how do they fit together, and more importantly how can they fall apart? How do we break through the “brute force” method of training and building experience by just “doing a bunch of test” and hoping the basics just “stick”? There’s obviously more to this. I’ve attached a brief that I’ve been working on for a while. It primarily defines the phases of a test point and highlights the transitions.
There is another set of ideas that incorporate test conduct into a larger test management construct. Namely, how does test conduct generate the needed data? What complicates the issue is that a single test point, despite being the fundamental building block of test, can satisfy multiple data requirements or measures of performance. Furthermore, a single data requirement can be satisfied by multiple test points. It’s a many-to-many relation.
There is a sense of the test conduct living in one domain, and data requirements living in another. Test conduct is constrained by the realities of operations, whereas the data requirements are not. So the art of test conduct is to optimize the satisfaction of data requirements given the time, material, and personnel constraints. Obviously this is all still a work in progress.
This article first appeared in the September 2018 Flight Test News.
One thought on “Test Conduct Reference: Letter to the Editor”
Comments are closed.