Split "Observed Distributions" out of prob_dist and develop it - #811
Merged
Conversation
Closes #791. `prob_dist.md` was doing two jobs: introducing probability distributions, and working with observed data. This splits the second into its own lecture, sitting immediately after prob_dist in the TOC, and develops the material well past what was there before. Retains the name "Observed Distributions" rather than the "Samples vs Distributions" title floated in the issue. New material in prob_dist: - The standard deviation, which the lecture used in its opening height example without ever defining. - Moments generally — raw, central and standardized — which gives meaning to the "first moment" and "second central moment" language already in the text, then skewness and excess kurtosis as the third and fourth standardized moments. Excess kurtosis (normal = 0) is the headline convention, matching SciPy's default, with a note that some authors do not subtract the 3. - Quantiles, the median, quartiles and the interquartile range. - Worked values via `stats(moments='sk')` and `ppf`, with the lognormal as the contrast case against the normal. The new lecture covers sample moments and sample quantiles, then histograms, empirical distribution functions, kernel density estimates, box-and-whisker plots and violin plots. Three data sets carry it, one per shape: US adult heights (skewness 0.07), Ames house prices (1.74, and about zero in logs), and Japanese deaths by age (-1.60). The mean sits above the median for house prices and below it for age at death, which motivates the quantiles as robust alternatives. Data now comes from QuantEcon/data-lectures per the routing rule in that repo — including us_adult_heights.csv, which moves out of _static/ here (QuantEcon/data-lectures#27). Its builder was recovered in that PR and reproduces the file byte for byte, so the repoint cannot change output. The violin plot link in lln_clt.md is repointed to the new lecture and made a proper cross-reference rather than a hard-coded URL. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
✅ Deploy Preview for taupe-gaufre-c4e660 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Four fixes from review of the previous commit. Expectations of functions were used but never defined: the moment expressions all take the form E[f(X)] for some f, so that form is now defined explicitly — as a sum in the discrete section and as an integral in the continuous one — and the variance is written first in E notation and then as the sum it stands for. Kurtosis and excess kurtosis are two distinct quantities and are now defined as such: kurtosis as the fourth standardized moment (normal = 3), and excess kurtosis as that minus 3 (normal = 0). Previously the lecture defined only the excess version and treated the other name as a caveat. The note is now about software rather than authors, since the real trap is that scipy.stats.kurtosis returns the excess version despite its name. The lecture had only two top-level sections, one of which held everything. "Common distributions" is dropped as a wrapper, promoting "Discrete distributions" and "Continuous distributions" to top level and the individual families to sections within them. Scalar outputs printed as np.float64(0.0) under NumPy 2. Setting np.set_printoptions(legacy='1.25') in both lectures restores plain scalar output without affecting how arrays print. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Uses ECDF throughout rather than EDF, including the section heading, the helper function name and the axis labels. Drops the claim that the normal fit to the log prices misses "in the tails", which is not visible in the figure — it just says the fit is close but not perfect. Adds a KDE of the log sale prices drawn over a faded histogram of the same data, closing the KDE section by showing what "a smoothed histogram" means against the histogram it smooths. Adds a violin plot of age at death by sex, which shows what the box plot reduces away: both distributions are strongly left-skewed with a thin tail running down to young ages, and the female distribution is both shifted up and more concentrated at the top. Medians are shown, at 82 and 88. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes out the lecture with the material the issue asks for. "Larger samples" shows ECDFs of samples of size 10, 100 and 1000 against the CDF that generated them — the ECDF is the natural vehicle, since it estimates the CDF without a bin width or a bandwidth to choose. Sample means for the same sizes are printed against the population mean. The trailing two sentences of the previous section, which asserted this convergence and pointed at the law of large numbers, are absorbed here. "The role of independence" makes the point that the convergence is not automatic, using the degenerate sample: draw X once and set X_i = X for all i. Every X_i then has the correct distribution, so each observation is individually valid, but the ECDF is a single step at X and increasing n changes nothing — the three curves lie exactly on top of one another. A note guards against overclaiming: independence is sufficient, not necessary, and this lecture histograms monthly returns that are plainly dependent. What matters is that new observations keep bringing new information. The general question is left to lln_clt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 3, 2026
This was referenced Aug 3, 2026
jstac
added a commit
that referenced
this pull request
Aug 3, 2026
The manual asks every code-generated figure to carry a caption via mystnb metadata. prob_dist had 3 of 20 and observed_distributions none of 20, the latter being an oversight in #811. Adds 35 captions with fig: names for numref, following the manual: sentence case, six words or fewer, descriptive names. Two figures are deliberately left bare, one in each lecture. Both sit inside solution directives, where a caption wraps the image in a LaTeX float and breaks the PDF build with "Not in outer par mode". Also fixes a bug found while captioning. The CDF figure in the lognormal section was built from scipy.stats.norm rather than scipy.stats.lognorm, so it had been drawing normal CDFs under a lognormal heading, and it looped over sigma alone with mu pinned to 1 while the density figure above it varied both. It now uses lognorm over the same (mu, sigma) pairs as that figure, so the two are a matched pair and the curves start at zero as a distribution on the positive half-line should. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #791.
prob_dist.mdwas doing two jobs: introducing probability distributions and their families, and working with observed data. This splits the second into its own lecture — Observed Distributions, immediately afterprob_distin the TOC — and develops it well past the compressed treatment it had.Keeps the name "Observed Distributions" rather than the "Samples vs Distributions" title floated in the issue.
New material in
prob_diststats(moments='sk')andppf, using the lognormal as the contrast against the normal.On the kurtosis convention: the text defines the fourth standardized moment (normal = 3), then works with excess kurtosis (normal = 0), which matches
scipy.stats.kurtosis's default so the printed numbers agree with the prose. A note records that some authors do not subtract the 3.The new lecture
Sample moments and sample quantiles, then histograms, empirical distribution functions, kernel density estimates, box-and-whisker plots and violin plots. EDFs and box plots are both new to the series.
Three data sets carry it, one per distributional shape:
That gives the moment machinery something to do: the skewness of log house prices is almost exactly zero, which demonstrates the lognormal on real data; and the mean sits above the median in one case and below it in the other, which motivates quantiles as the robust alternative.
Two honest notes rather than tidy stories: the box plot by bedroom count shows a non-monotonic relationship (one-bedroom houses have a higher median than two-bedroom, correlation 0.14), so the text says so and the clean floor-area version becomes an exercise; and Japan's "100 and over" category is 3.3% of all deaths and 5.7% of female deaths, so the spike at the right edge of the histograms is flagged as a recording artifact.
Data hosting
All three data sets come from
QuantEcon/data-lecturesper that repo's routing rule — includingus_adult_heights.csv, which moves out of_static/here (QuantEcon/data-lectures#27). Its builder was recovered in that PR and reproduces the file byte for byte, so the repoint cannot change lecture output.The violin-plot link in
lln_clt.mdis repointed to the new lecture and made a proper{ref}cross-reference rather than a hard-codedintro.quantecon.orgURL.Verification
Both lectures execute end to end against the live data URLs (jupytext → nbconvert), and every reported figure in the prose was read off the actual output.
Convergence and independence
The lecture closes with the convergence material the issue asks for. Larger samples plots ECDFs for n = 10, 100, 1000 against the CDF that generated them, plus sample means against the population mean — the ECDF is the natural vehicle here, since it estimates the CDF with no bin width or bandwidth to choose.
The role of independence then shows the convergence is not automatic, via the degenerate sample: draw X once and set X_i = X for all i. Every X_i has the correct distribution, so each observation is individually valid, but the ECDF is a single step at X and the three curves lie exactly on top of one another — increasing n changes nothing.
A note keeps this honest: independence is sufficient, not necessary, and this same lecture histograms monthly returns that are plainly dependent. What matters is that new observations keep bringing new information. The general question is left to
lln_clt.🤖 Generated with Claude Code