More than a year ago, I heard a long-time USAF member of SETP say, “I never heard about the Flight Test Safety Committee” (an admission that took many of our Committee members by surprise). That conversation was the catalyst for this newsletter, which distributed its first issue in January this year. On the other hand, since January, someone confessed that they had NEVER heard about the Flight Test Safety Committee, even though they attended more than 15 SETP Symposia in Anaheim and worked with SETP, SFTE, and AIAA members for more than 20 years. I’m actually delighted
by the second story because they found out about the Committee through this newsletter–our efforts over the past year have succeeded: We’ve made progress!
Our explicit goal is to Reach Everyone with this publication. This is a qualitative statement that exaggerates the importance of getting the word out. More specifically though, we intended to reach 117% of the membership of SFTE and SETP in the first year. So here we are, and it’s time to ask, “How did we do?”
We reached 3539 people by direct email with our newsletter. The data shows that our email distribution has grown slightly more than over 3%, to 103% total, which brings us short of our goal. This number does not include people who receive the email forwarded by an existing member. In the rest of this article, I want to accomplish two things. The first thing is to introduce a technical topic relevant to estimating progress toward our 117% goal, a topic we will later explore for its relevance to estimating risk in flight test safety. Secondly, I want to finish evaluating progress toward our goal and communicate a vector for next year. You can now skip to either section below, first or second, and continue reading, or you can read both sections.
Reach Everyone (117%)
Implicit in the statement of this goal is the need to count how many members are in SFTE and SETP. Sometimes, however, the simple words used to make a statement hide the technical challenges of accomplishing a task like this. Did we mean the number of members in January or in December, or did we mean the highest membership number reported for the year (that sounds reasonable)? Are we counting members of both Societies twice? It is a simple but important question, just the first step in actually measuring the quantity. Another, more recent example is also relevant. One report from the 2019 SETP Symposium stated 692 people were in attendance, but a spreadsheet of the registrants explicitly listed only 674 people. Each of us could probably explain several possible causes for the discrepancy, but this example illustrates something we actually overlook: “Counting is hard.”
Estimating Risk in Flight Test Safety
The recent pad abort test of Boeing’s CST 100 Starliner provides an example we can use to discuss the estimation of risk. During the flight test, only two of the three recovery parachutes deployed successfully. Thinking about these kinds of failures is exactly what we do when performing test hazard analyses. We say things like “unlikely to occur” when we are talking about the probability—or frequency or likelihood—of hazards. Implicit in that language is the necessity of counting how many times a given failure occurs in a given number of flight hours. In the Starliner case, do we count this test as a failure, and if so, how many failures should we expect in the lifetime of the capsule? Since the test team quickly ascertained root cause, does it change the total or how we count failures? This example illustrates something we actually overlook: “Counting is hard.”
Maybe you finished one or both of those columns and read the conclusion with incredulity. You may wonder where this is going.
Ultimately, I want to be able to measure how many people we reached with the news: News about the Flight Test Safety Committee (FTSC), its members and their presentations, resources, workshops, and the newsletter; news about new techniques, training, and trends. We want to be able to spread news that will have a direct impact and will equip flight test professionals with the knowledge and skill to make life-saving decisions. That’s where this is going. That requires us to count how many people read the newsletter, whether they subscribe to it or a colleague forwarded it. This turns out to be very similar, statistically speaking, to estimating the number of failures that may affect a flight test—counting failures is a topic we will address more completely in a future issue.
You may recall that one of the “3Q” heuristic rules recommended for evaluating uncertainty is “express the outcome both qualitatively and quantitatively.” So I’ll begin with a qualitative report of newsletter reach during 2019. I began by polling the FTSC: There are 21, and several of them have forwarded the email to their entire office: 25 people in one and 31 in another, for example. How many of those people are “unique,” i.e., not included in FTSC membership numbers? Unfortunately, we don’t know, but some statistical counting techniques can address this uncertainty.
For example, we can “estimate the range of possible outcomes.” One way to do this is count each of the reports above as unique “hits.” This gives us an upper bound on how many we’ve reached—remember we are merely estimating the top of the range. This results in 61 more, another 2%. Another variation would be to ascribe a frequency of more than 1 to some of the unique hits above: For example, someone from RAAF Aircraft Research and Development Unit (ARDU) subscribed and we could count this as two hits (him and someone in his office). This is a reasonable way to estimate the reach of this particular subscriber.
Each of these gives incremental progress, but we need to reach 5x as many new readers to attain our goal of 117%. Next year, we plan to incorporate distribution to the AIAA Flight Test group and count their numbers as well. The target will continue to be 117% for the second year as we refine our data collection. Next year, we will cover counting techniques in more detail and show their relation to “counting failures” in flight test. In closing, though, let me reiterate this: For each example above, an existing member took the initiative to share this newsletter. Thank you! Your effort will help us continue to make progress toward our goal.
Related:
counting is hard