Three Heuristics for Communicating Uncertainty in Flight Test

October 8, 2019October 7, 2019 admin flight test flight test safety, uncertainty

In Flight Test Safety Fact, 19-09, the Chairman’s column and Rod’s letter raised many questions, but I would like to single in on the question that came to my mind as I read Rod’s email: is Steady Heading Side Slip (SHSS) really high risk? Since there are two inputs into the risk assessment: hazard probability and severity, I began to wonder which would need to change. In my own thoughts, the word “change” caught me by surprise, because I wasn’t sure I had an idea of where I was starting from.

I did know one thing for certain: I have really strong feelings about probability. Coincidentally, it’s something the Chairman mentioned in his column, and it intersects with the questions I asked myself about the SHSS risk level. I couldn’t immediately think of an example of a SHSS accident, and the email responses shared in preparation for the article quickly gave me several examples. After reading these, I recalled the AC-130J, a recent example I should not have forgotten. The reminder affected my perception of my own objectivity.

How do we assess the probability of a given hazard with objectivity? More importantly, though, how confident do we feel about our probability assessment?

Perhaps both of these questions should have been asked by the C-17 test team introduced in the last issue. That issue introduced the topic, Communicating Uncertainty in Flight Test, by revisiting Safety Planning in C-17 Airdrop Flight Test: The test pilots used a C-17 HITL (hardware-in-the- loop) simulator to prepare for an airdrop envelope expansion and handling qualities test. The simulator did not contain models for the test items (i.e., cargo and airdrop parachutes), raising questions about the results. In the example test campaign, the test team determined upper and lower bounds on the response of the aircraft during contingency situations. This is different from the normal procedure of predicting the estimated response of the aircraft. This technique may not work in every situation, but it illustrates the use of heuristics, which, as many have suggested recently, ought to play an important role in helping the flight test profession cope with complexity and uncertainty.

This past example and the SHSS risk problem frame the critical question: How do we express confidence in model, simulation, and experimental outcomes? To answer this question, I propose a framework of three rules, specifically for communicating about uncertainty in flight test outcomes. It may be helpful to abbreviate these heuristics as the 3Q.

The purpose of this article is to introduce the rules and demonstrate their utility and application:

1. Express the outcome both qualitatively and quantitatively.
2. Describe the range of possible outcomes.
3. Assess the frequency of potential outcomes.

To illustrate these concepts, I want us to recall some common characteristics of an airplane flight manual. Most of the people in this audience have heard of Notes, Cautions, and Warnings, especially in this context. One such manual defines these terms in the following way.

WARNING denotes those items highlighted for the purpose of describing “operating procedures or techniques which may result in personal injury or loss of life if not carefully followed.”

CAUTION denotes those items highlighted for the purpose of describing “operating procedures or techniques which may result in damage to equipment if not carefully followed.”

NOTE is “additional or significant operating information requiring emphasis.”

These definitions do not use numerical descriptors, yet after reading them, we come away with an understanding of their relationship, relative severity, and importance. Instead of being quantitative, the terms are qualitative. There are many more examples of the meaning of qualitative: yes or no, high or low, left or right, and even first or last. Qualitative does not necessarily mean that something is indefinite or imprecise, but instead, it is the complement of—it contrasts with—the idea of quantitative, or numerical and measurable characteristics, a term which needs no illustration.

This establishes precisely what we mean by qualitative, which is our foundation in the subject of communicating and understanding uncertainty. To help us build on this foundation and create a shared lexicon for dealing with the varied topics under the broad umbrella of uncertainty, I would like to propose three foundational rules. First, we should express ideas both qualitatively, as we did above, and quantitatively. Second, we should attempt to describe the range of possible outcomes. Finally, we ought to assess the frequency of potential outcomes.

The figure above places the qualitative phrases from the flight manual on a spectrum of possible outcomes, in relation to one another. This illustrates the second rule, describing the range of possible outcomes. This spectrum can also help us understand the third rule. We do not expect that “loss of life” will happen frequently in the service life of an airplane, but the conditions described in a “note” might happen daily. This is what we mean by “frequency of potential outcomes.”

Having briefly explained each of the rules, now apply these two ideas to safety process outcomes. In a Test Hazard Analysis, we immediately see that the characterization of hazards and their severity aligns naturally with the qualitative. Death or injury or aircraft damage are three terms used often that describe the severity of outcomes of a particular hazard. Furthermore, the formal FAA and ICAO terms used to describe hazard severity are qualitative (catastrophic, major, minor, etc.). This much is not new to most readers, but let us explore the idea further.

Is it possible to define hazard severity quantitatively rather than qualitatively? One may struggle initially with such a definition (*I certainly did). Consider, however, this simple characterization: A safety outcome or test hazard that results in zero injuries or death is not severe. A severe outcome would have some positive number of injuries or death. We could further delineate by calling a hazard very severe if death (not just injury) is an outcome. This gives us three levels of severity defined quantitatively—not severe, severe, and very severe. We simply extended our qualitative ideas (injury or death) and assigned numeric levels to quantify them. Aircraft damage also falls into this framework. Quantitative characterization extends even further, allowing us to assign continuous, even infinite, values, as well as precision and resolution to our understanding of safety process outcomes.

We can also apply this same contrast to the causes of a hazard. For example, consider a hazard such as aircraft damage, the cause of which is overspeed. We have defined this cause qualitatively as “overspeed,” but we can extend the definition quantitatively. A stable test point conducted at VMO that encounters a light atmospheric disturbance may result in a momentary overspeed of 1-2 knots that immediately returns to its steady-state value. But this is probably not what we imagine when we consider the cause of such a hazard. To create a more precise definition, we could further quantify our meaning.

Conclusion
Last year, Doug “Beaker” Wickert declared that we needed more time to think—I agree and would further suggest that it is the intentional cognitive exercise presented herein from which we benefit. By pondering these 3Q, we exercise our thinking. Precision and accuracy of definition are important, and the exercise forces us to transform vague notions into specific definitions with qualitative characteristics. Finally, the stories told by Rod and the Flight Test Safety Committee and presented at workshops give us data we can directly use to assess frequency and range of potential outcomes.