Skip to main content

Bootstrap Plot

NIST/SEMATECH Section 1.3.3.4 Bootstrap Plot

20 40 60 80 100 Observation 0.48 0.5 0.52 0.54 0.56 Value Bootstrap Mean 0.48 0.49 0.5 0.51 0.52 0.53 0.54 Value 0 5 10 15 20 25 Frequency Bootstrap Mean 20 40 60 80 100 Observation 0.46 0.48 0.5 0.52 0.54 0.56 Value Bootstrap Median 0.46 0.48 0.5 0.52 0.54 0.56 Value 0 5 10 15 20 25 Frequency Bootstrap Median 20 40 60 80 100 Observation 0.496 0.498 0.5 0.502 0.504 Value Bootstrap Midrange 0.497 0.498 0.499 0.5 0.501 0.502 0.503 0.504 Value 0 10 20 30 40 50 60 Frequency Bootstrap Midrange
A bootstrap plot displays the computed value of a sample statistic on the vertical axis against the subsample number on the horizontal axis. Each subsample is drawn with replacement from the original dataset so that any data point can be sampled multiple times or not at all. By repeating this process many times, the plot builds an empirical picture of the sampling variability without requiring distributional assumptions.

What It Is

A bootstrap plot displays the computed value of a sample statistic on the vertical axis against the subsample number on the horizontal axis. Each subsample is drawn with replacement from the original dataset so that any data point can be sampled multiple times or not at all. By repeating this process many times, the plot builds an empirical picture of the sampling variability without requiring distributional assumptions.

The bootstrap procedure draws B random subsamples of size N with replacement from the original dataset, computes the statistic of interest for each subsample, and plots the resulting B values against the subsample number as a connected line. This is typically followed by a histogram to visualize the shape of the sampling distribution. Percentile confidence intervals are read directly from the sorted bootstrap values; for example, with 500 resamples the 25th and 475th sorted values form a 90% confidence interval.

Questions This Plot Answers

  • What does the sampling distribution for the statistic look like?
  • What is a 95% confidence interval for the statistic?
  • Which statistic has a sampling distribution with the smallest variance?

Why It Matters

The most common uncertainty calculation is generating a confidence interval for the mean, which can be derived mathematically. However many real-world problems involve statistics for which the uncertainty formulas are mathematically intractable. The bootstrap provides a purely empirical alternative by resampling from the observed data, answering the question "how much would my estimate change if I collected new data?" without requiring distributional assumptions.

When to Use a Bootstrap Plot

Use a bootstrap plot when assessing the uncertainty of an estimate such as the mean, median, or midrange and when traditional confidence interval formulas are mathematically intractable or may not be valid. Bootstrap methods are valuable when the theoretical sampling distribution of a statistic is unknown or difficult to derive analytically, for example with small samples, non-normal data, or complex estimators. Comparing bootstrap plots for different statistics reveals which estimator has the smallest variance.

How to Interpret a Bootstrap Plot

The bootstrap plot displays the computed statistic on the vertical axis against the subsample number on the horizontal axis. A stable estimate appears as a tight horizontal band, while a wide scatter indicates high sampling variability. The plot is typically followed by a histogram of the resampled values to examine the shape of the sampling distribution and read off percentile confidence intervals. For example, from 500 bootstrap samples, the 25th and 475th sorted values give a 90% confidence interval.

Assumptions and Limitations

The bootstrap assumes that the observed sample is representative of the population and that observations are independent. It may perform poorly with very small samples where the original data do not adequately capture the population structure. The number of bootstrap replications is typically 500 to 1,000. The bootstrap is not appropriate for all distributions and statistics; in particular it is unsuitable for estimating the distribution of statistics that are heavily dependent on the tails, such as the range.

See It In Action

This technique is demonstrated in the following case studies:

Uniform Random Numbers Case Study

Reference: NIST/SEMATECH e-Handbook of Statistical Methods, Section 1.3.3.4

Formulas

Bootstrap Resample

Xb∗={Xb,1∗,Xb,2∗,…,Xb,N∗}drawn with replacement from {X1,…,XN}X^{*}_b = \{X^{*}_{b,1}, X^{*}_{b,2}, \ldots, X^{*}_{b,N}\} \quad \text{drawn with replacement from } \{X_1, \ldots, X_N\}

Each bootstrap sample draws N observations with replacement from the original dataset, so some observations may appear multiple times and others not at all.

Percentile Confidence Interval

CI1−α=(θ^(α/2)∗,  θ^(1−α/2)∗)CI_{1-\alpha} = \left(\hat{\theta}^{*}_{(\alpha/2)},\; \hat{\theta}^{*}_{(1-\alpha/2)}\right)

The bootstrap percentile interval uses the alpha/2 and 1-alpha/2 quantiles of the B bootstrap statistic values as the confidence bounds.

Python Example

import numpy as np
import matplotlib.pyplot as plt


# 500 uniform random numbers (NIST RANDU-style)
rng = np.random.default_rng(42)
data = rng.uniform(size=500)


# Bootstrap 500 subsamples, compute mean, median, midrange
B = 500
stats = {"Mean": [], "Median": [], "Midrange": []}
for _ in range(B):
    s = rng.choice(data, size=len(data), replace=True)
    stats["Mean"].append(np.mean(s))
    stats["Median"].append(np.median(s))
    stats["Midrange"].append((s.min() + s.max()) / 2)


# 6-panel plot: line plots on top, histograms below
fig, axes = plt.subplots(2, 3, figsize=(12, 6))
for i, (name, vals) in enumerate(stats.items()):
    axes[0, i].plot(vals, linewidth=0.5)
    axes[0, i].set_title(f"Bootstrap {name}")
    axes[0, i].set_xlabel("Subsample Number")
    axes[1, i].hist(vals, bins=20, edgecolor="white")
    axes[1, i].set_title(f"Bootstrap {name}")
    axes[1, i].set_xlabel("Value")
    axes[1, i].set_ylabel("Frequency")
plt.tight_layout()
plt.show()