Spinning the Concrete Dial with Hypothetical Outcome Plots

Blanket disclaimer: This is a post about animated visualizations, illustrated mostly by static screenshots. Please consider clicking through to see the actual visualizations! My screengrabs don’t do them justice.

I’ve been chewing on uncertainty visualizations since Matthew Kay’s excellent talk at Tapestry 2018. The recent release of the R package gganimate has also brought a number of animated visualizations across my feed, so let’s talk about an animated uncertainty visualization: hypothetical outcome plots (HOPs). What are they for, besides inspiring truly terrible puns?

One of the core functions of statistics is making inferences about a population based on limited information. Sometimes that means estimating a value (average sword price, for instance); sometimes that means modeling to describe the relationships between variables and to predict what might happen in the future. Those estimates and predictions look very precise when depicted as single points or figures. However, there is always uncertainty involved: other estimates that we could have gotten if we repeated the study, or a range of possible outcomes from the model. HOPs depict uncertainty by animating a sequence of outcomes that could occur, rather than showing a single number. (If you’d like to know more, I really can’t do better than Jessica Hullman’s original Medium post about HOPs.)

HOPs get a lot of press for making viewers encounter uncertainty, but that’s far from their only application. I think about HOPs as a kind of concreteness dial. They make estimations less concrete by forcing an audience to experience uncertainty, and they make processes more concrete by showing the different ways a model might play out.

Turn the dial down: including uncertainty in an estimate

Most of the writing I’ve seen about HOPs focuses on the viewer’s encounter with uncertainty. And nothing puts me at peace with uncertainty more than contemplating my own mortality. The Social Security Administration’s actuarial tables tell me that women my age have an average of 54.85 years left in their lives. It would be really, really easy to interpret that as “I will die at precisely age 81.85.” Nathan Yau provides another perspective in a HOP, which can be customized based on age and gender:

Screen Shot 2019-01-27 at 3.50.45 PM.png

The HOP shows how the ages stack up at the bottom of the graphic, with height representing the number of times the outcome occurred. That histogram a great job of demonstrating the difference between an average and a prediction: the average age of death is 81.85, but the range of ages is pretty wide. There aren’t a lot of outcomes where I drop dead at precisely 81.85. Good news, everybody: even if we can estimate an (average) life expectancy, there’s no way to know when death will come!

Turning the dial up: making a model more concrete

HOPs are uniquely valuable for showing how multiple variables interact, guiding an audience through a model while also demonstrating the importance of random chance. As Alex Kale and Jessica Hullman wrote in 2018, “For probability judgments about multiple variables… users of HOPs were an estimated 35 to 41 percentage points more accurate than users of error bars and violin plots.”

Take a hypothetical scenario about the measles. Someone who is not immune to the disease has a 90% chance of being infected after exposure. 99% of the people who are vaccinated against the measles become immune. If 5% of the community members are exposed to measles, will a community with an 86% vaccination rate experience an outbreak? If there is an outbreak, how many people will get sick? How many people will be susceptible, but not get infected? What about a community with a 58.5% vaccination rate, or a 90% vaccination rate?

Show your work.

Or take a look at this HOP from the Guardian:screen shot 2019-01-27 at 3.56.08 pm

It’s really, really hard for most people to estimate the probability of multiple outcomes based on multiple inputs. It’s even harder to compare those probabilities under different conditions. The Guardian piece shows how a model of measles transmission might play out. Five measles carriers (red dots) comes into contact with a randomly chosen community member. If that community member is susceptible, an outbreak will almost certainly occur.

Playing the simulation a couple times in a row also demonstrates the role of luck. Sometimes communities with a higher vaccination rate get an outbreak, while communities with lower vaccination rates don’t. However, a community with a low vaccination rate has to be much, much luckier to avoid an outbreak than a community with a high vaccination rate. Showing the spread of the outbreak also complicates binary thinking about the issue. Either an outbreak happens or it doesn’t, but if an outbreak does occur, fewer people get sick in a community with a high vaccination rate. Instead of handing me a pile of probabilities and leaving me to sort them out on my own, the HOP gives me concrete examples of vaccination rates interacting with measles transmission.

The measles visualization looks at several outcomes, but only shows one simulation per community. On the other hand, this HOP from the New York Timesshows a flood of ten thousand different simulations, each with one outcome. The piece looks at the adult incomes of black and white boys who grow up in similar economic circumstances:

screen shot 2019-01-27 at 3.20.00 pm screen shot 2019-01-27 at 4.29.49 pm

The HOP allows two truths to exist simultaneously: black boys who grow up wealthy are much more likely to end up poor than white boys, and not every black boy who grows up wealthy will end up poor. It makes room for the individual while also giving an intuitive understanding of the whole.

Both of these graphics simplify their subjects. That’s because the goal is to illustrate the principles that underlie a messier, more complex reality. Take the race and class graphic. Each HOP shows only one starting income at an extreme end of the scales. The simulation also starts with equal numbers of black and white boys growing up wealthy, which is not the case in America. That simplification makes it easier to see how many more black boys become poor adults and how many more white boys become rich adults. There is a more complex version of the graphic, where boys flow from all familial income levels to all adult income levels, with proportions of black and white boys at each income level that match the real world:

screen shot 2019-01-27 at 3.26.53 pm

In the complex version, it’s difficult to pick out what happens to boys from any specific income group, and easy to see a parity that doesn’t exist. Some white boys from rich families fall into poverty; some black boys from poor families rise to wealth. In the chaotic middle of the HOP, it looks like boys of each race rise and fall in about equal numbers. However, that’s a trick of proportions. Fewer black boys than white boys begin in wealthy families, so if the same number of wealthy black and white boys become poor as adults, many fewer black boys continue to be wealthy. Similarly, both black and white boys rise out of low-income families, but more black boys are born into poverty than white boys, so if roughly even numbers of white and black boys become wealthy, many more black boys remain poor. Simplifying the proportions in the graphic makes the process intelligible so that the audience can better understand how it plays out in the real world.

The measles graphic also simplifies matters. The transmission model is very simplified, and there’s only one outcome for each community in the plot. To make it a hypothetical outcomes (plural) plot, you would need to hit Simulate over and over. Which I did. Here’s what the results look like:


It’s kind of overwhelming! The overall pattern is clear (more vaccinations = less measles), but the process is lost. Watching multiple versions flash by is no substitute for anxiously watching to see where the red dot will land and watching an outbreak spread.

All of the HOPs in this post include some tool to help the viewer: a way to keep track of the final distribution of outcomes. The measles plot displays a status update and a bar chart showing the proportions of infected, susceptible, and immune people in the community. The race and income graphic uses barcode plots. The life expectancy HOP uses a histogram to collect the outcomes it has already displayed. The latter two visualizations also list precise counts and percentages for different outcomes, which seems like it could risk a false impression of certainty and precision.

As someone with an absolutely terrible memory, I appreciate the assist in keeping track!. However, my memory may not actually make a difference, since HOPs are built around the idea of ensemble processing. As Alex Kale and Jessica Hullman wrote, “Interestingly, people are able to accurately report the statistical properties of a set of objects without being able to remember the characteristics of individual objects. This leads us to believe that HOPs and other ensemble visualizations are processed automatically (without trying) and subconsciously (without being aware) by the visual system.”  I would love to see research that explicitly investigates the role of memory when using HOPs.

Sticking the landing: when to use HOPs

HOPs are explicitly experience-based. Alex Kale and coauthors wrote, “HOPs are designed to promote the integration of uncertainty information through experience rather than through description.” And if you’re trying to get a specific answer, that experience is unfortunately kind of annoying.

hop gif.gif

From the UW Interactive Data Lab. How likely is it that B will be greater than A?

The second comment on an early critique of HOPs reads, “My boss would laugh in my face if I present him with a dancing line.” That commenter is probably right!

There is a hierarchy inherent in using HOPs to portray uncertainty. The designer is making the viewer experience something unpleasant (uncertainty) at a cost to them (time) while withholding something they desire (a precise estimate). It would be very difficult to use HOPs when sending data upwards in a hierarchy. Forcing viewers to grapple with uncertainty through HOPs depends on specific social dynamics, like:

  • The designer is the only person with access to the information, and the viewer has no power over them
  • The viewer has power over the designer, but wants to see uncertainty
  • The designer has power over the viewer, and can make them see uncertainty
  • The designer thinks that portraying uncertainty is important enough to annoy the viewer, and is willing to spend the social capital to do it (Elijah Meeks has some interesting thoughts about the social dynamics of data visualization here.)

On the other hand, the patterns revealed by a HOP may also make the experience more useful than unpleasant. And using HOPs to illustrate an abstract situation might be welcome when presenting to someone with more power but less subject expertise.

I count myself in the second group: I can’t make a data journalist do anything, but I want to see HOPs. I have many more questions about HOPs, so I’ll be interested to see the research develop and watch their continued use in journalism. Keep the uncertainty coming!

One thought on “Spinning the Concrete Dial with Hypothetical Outcome Plots

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s