New lecture: Fitting Distributions to Data - #814
Merged
Conversation
Adds the third lecture in the probability sequence, after prob_dist and observed_distributions, on choosing a distribution to describe a data set. It sits immediately after observed_distributions in the TOC. The lecture is organized around the two halves of that question — which family, and which parameters: - The method of moments, stated generally and applied to the normal, lognormal and gamma families. This names what prob_dist already did once when it fitted a normal to the height data. - Q-Q plots, defined from first principles rather than introduced through a library call: the i-th order statistic estimates the quantile of order (i-0.5)/n, so plotting it against the fitted quantile should give the 45 degree line. Then how to read a departure — curvature means skew, an S-shape means heavier tails than the fit allows. - The Kolmogorov-Smirnov statistic as the largest vertical gap between the ECDF and the fitted CDF, drawn on the figure. It stops short of the test, and says why: that needs the null distribution of D, and our parameters came from the same data. - Choosing between families by fitting each and taking the smallest D. For the house prices this ranks lognormal (0.053) over gamma (0.070) over normal (0.123), agreeing with the near-zero skewness of the log prices found in the previous lecture. - Count data, fitting a Poisson to goals per football match, where the fit is good and the reason it should be is worth stating. - A section on failure: the normal fit to Amazon returns has an unremarkable D but an obviously S-shaped Q-Q plot, because D looks at the middle of the distribution and the trouble is in the tails. That hands off to heavy_tails. Three warnings accompany the model-selection method: D does not charge a family for having more parameters, it is insensitive in the tails, and the winner is only the best of the candidates tried. Exercises fit an exponential to the times between Japanese earthquakes, which fails because aftershocks cluster and the arrivals are therefore not independent, and a normal to the Japanese age-at-death data, which fails through left skew and also exposes the "100 and over" recording cap as a flat segment in the Q-Q plot. Data comes from QuantEcon/data-lectures#31. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
✅ Deploy Preview for taupe-gaufre-c4e660 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Terminology. The lecture asked the reader to choose a "family" without saying what one is. It now introduces the parametric class -- a set of distributions indexed by a small number of parameters -- in the outline, and uses that term throughout. This also sharpens the warnings about comparing classes of different dimension, where "family" was doing double duty for both the set and its members. What the lecture is about. The outline claimed we take parameter choice first "since it turns out to be easier". Both questions are treated, the emphasis is on choosing the class, and the difficulty claim is one the lecture never supports -- the Student t section now contradicts it outright. It says instead that the class is the main subject, that parameters get one technique with the rest left to mle, and that we start there because a class must be fitted before it can be judged. "When nothing fits" was false. A Student t fits the Amazon returns well: the KS distance falls from 0.068 to 0.041 fitting by moments, and to 0.022 by maximum likelihood. The section is now "When the normal fails", and it turns the failure into a method: the Q-Q plot says the tails are too heavy, that points to a class with heavier tails, and the t delivers one. The fitted value nu = 5.8 comes with a warning against leaning on it, since it is estimated from the sample kurtosis, which is worst behaved exactly when the tails are heavy -- concrete motivation for mle. Plotting positions. The claim that the i-th order statistic estimates the quantile of order (i-0.5)/n was asserted without support. It is now derived from the ECDF, which steps up from (i-1)/n to i/n at that observation, so the data supply an interval of orders rather than one, and the midpoint splits the difference. A second reason is given: with i/n the largest observation would be matched to the quantile of order 1, which is infinite for every unbounded class here. Figures. Per the manual, no ax.set_title: the eight body figures now carry mystnb captions with fig: names, and the two inside solution directives are left bare, since a caption there becomes a LaTeX float and breaks the PDF build. Reference lines and CDF curves moved to lw=2. The statsmodels Q-Q figure is dropped in favour of a note recording that sm.qqplot exists, and why we build our own: the construction is worth understanding, and our version takes any distribution, which the earthquake exercise needs. statsmodels is no longer imported. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
✅ Translation sync completed (zh-cn)Target repo: QuantEcon/lecture-intro.zh-cn
|
This was referenced Aug 3, 2026
jstac
added a commit
that referenced
this pull request
Aug 3, 2026
fitting_distributions (#814) was reachable only from the table of contents and from heavy_tails. Neither of the two lectures it builds on mentioned it, so a reader working through the sequence in order had no pointer forward. - prob_dist now introduces itself as the first of three and names both of the others, replacing a sentence that named only observed_distributions. - observed_distributions points forward in its outline, and again at the place where it fits a normal to the Amazon returns by matching moments -- which is the method of moments, unnamed there, and the subject of the next lecture. - mle gains its first cross-reference to another lecture. Its two steps for maximum likelihood, guess the distribution and estimate the parameters, are the two questions fitting_distributions is organized around, so it now says so. That closes a loop: fitting_distributions already sent readers to mle twice, once for a fuller treatment of parameter estimation and once for the Student t fit that the method of moments handles poorly. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the third lecture in the probability sequence, after
prob_distandobserved_distributions, sitting immediately after the latter in the TOC.It answers a question the series raises repeatedly but never addresses directly: given a data set, which distribution should we use to describe it? That splits into choosing a family and choosing parameters, and the lecture takes them in that order.
Sections
prob_distalready did once, when it fitted a normal to the heightsQ-Q plots get a proper definition
heavy_tailscurrently usessm.qqplot(data, line='45')without ever saying what a Q-Q plot is. Here the construction is explicit — thesm.qqplotintroduced afterwards as the shortcut. How to read a departure is stated as a rule: curvature means skew, an S-shape means both tails heavier than the fit allows.A follow-up PR can slim
heavy_tailsto cite this instead of introducing the tool cold.Model selection, with its limits
Fitting all three families to the house prices by moments and ranking by$D$ gives lognormal 0.053, gamma 0.070, normal 0.123. That agrees with the near-zero skewness of the log prices found in
observed_distributions, so the reader reaches the same conclusion twice by different routes.Three warnings accompany it:$D$ does not charge a family for having more parameters (so compare like with like, and beware nesting), it is insensitive in the tails, and the winner is only the best of the candidates tried.
The lecture stops short of the KS test and says why — that needs the null distribution of$D$ , and our parameters came from the same data.
The failure case
Goals per match fit well ($\bar x = 2.83$ , variance 2.78, frequencies sitting on the fitted PMF), so the machinery is shown working before it is shown failing.
Then the Amazon returns: an unremarkable$D$ but an obviously S-shaped Q-Q plot, because $D$ looks at the middle and the trouble is in the tails — which is what matters for asset returns, and hands off to $D$ is not comparable across samples of different size, since it shrinks with $n$ even when the fit is right.
heavy_tails. A note warns thatExercises
observed_distributions.Data
epl_match_goals.csvandjapan_earthquakes.csvcome from QuantEcon/data-lectures#31; the heights, house price and age-at-death data are already published there. Executes end to end against the live URLs.Follow-up
This lecture reads the Amazon returns via
yfinance, so it needs anaudit_annotations.ymlentry indata-lectures, alongside themigration.ymlrepoint andconsumersentries for the two new datasets. I will open that PR once this merges, as with #811.🤖 Generated with Claude Code