Add count and waiting-time datasets for a distribution-fitting lecture - #31
Merged
Conversation
Two datasets for a lecture in preparation on fitting distributions to data, which needs count data to go with the continuous data already here. - epl_match_goals.csv — full-time scores for ten complete English Premier League seasons, 3,800 matches. Goals per match average 2.830 with variance 2.777, so the Poisson fit is close across the whole range of counts, which is what the lecture needs from a worked example. - japan_earthquakes.csv — 3,557 events of magnitude 5+ around Japan, 2000-2024, from the USGS catalog. This one is the counterexample: monthly counts have a variance-to-mean ratio near 97 and the times between events have a standard deviation half again their mean, because aftershocks cluster. Neither Poisson nor exponential survives contact with it, which makes the point that independence is doing real work. Both are class: constructed and ship builders following the fetch -> pre-process -> validate -> write staging. The football builder asserts exactly 380 matches in each of ten seasons so a partially published season cannot slip in; the earthquake builder asserts the magnitude floor, the bounding box and the year range. Licences are clean: openfootball dedicates its data to the public domain explicitly, and USGS output is a US Government work. The USGS event service is a live API, so per AGENTS.md the builder queries it once over a closed window and commits the result rather than leaving a lecture to call it. Its manifest records that ComCat is a revised catalog, so a later re-run may differ slightly from the committed hash. Consumers are empty in both manifests: the lecture that will read them is still being written, and these records get their entries when it lands. Part of QuantEcon/meta#336 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jstac
force-pushed
the
fitting-lecture-data
branch
from
August 3, 2026 11:26
4e7c480 to
41fc66c
Compare
jstac
added a commit
to QuantEcon/lecture-python-intro
that referenced
this pull request
Aug 3, 2026
* New lecture: Fitting Distributions to Data Adds the third lecture in the probability sequence, after prob_dist and observed_distributions, on choosing a distribution to describe a data set. It sits immediately after observed_distributions in the TOC. The lecture is organized around the two halves of that question — which family, and which parameters: - The method of moments, stated generally and applied to the normal, lognormal and gamma families. This names what prob_dist already did once when it fitted a normal to the height data. - Q-Q plots, defined from first principles rather than introduced through a library call: the i-th order statistic estimates the quantile of order (i-0.5)/n, so plotting it against the fitted quantile should give the 45 degree line. Then how to read a departure — curvature means skew, an S-shape means heavier tails than the fit allows. - The Kolmogorov-Smirnov statistic as the largest vertical gap between the ECDF and the fitted CDF, drawn on the figure. It stops short of the test, and says why: that needs the null distribution of D, and our parameters came from the same data. - Choosing between families by fitting each and taking the smallest D. For the house prices this ranks lognormal (0.053) over gamma (0.070) over normal (0.123), agreeing with the near-zero skewness of the log prices found in the previous lecture. - Count data, fitting a Poisson to goals per football match, where the fit is good and the reason it should be is worth stating. - A section on failure: the normal fit to Amazon returns has an unremarkable D but an obviously S-shaped Q-Q plot, because D looks at the middle of the distribution and the trouble is in the tails. That hands off to heavy_tails. Three warnings accompany the model-selection method: D does not charge a family for having more parameters, it is insensitive in the tails, and the winner is only the best of the candidates tried. Exercises fit an exponential to the times between Japanese earthquakes, which fails because aftershocks cluster and the arrivals are therefore not independent, and a normal to the Japanese age-at-death data, which fails through left skew and also exposes the "100 and over" recording cap as a flat segment in the Q-Q plot. Data comes from QuantEcon/data-lectures#31. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fitting_distributions: revisions from review Terminology. The lecture asked the reader to choose a "family" without saying what one is. It now introduces the parametric class -- a set of distributions indexed by a small number of parameters -- in the outline, and uses that term throughout. This also sharpens the warnings about comparing classes of different dimension, where "family" was doing double duty for both the set and its members. What the lecture is about. The outline claimed we take parameter choice first "since it turns out to be easier". Both questions are treated, the emphasis is on choosing the class, and the difficulty claim is one the lecture never supports -- the Student t section now contradicts it outright. It says instead that the class is the main subject, that parameters get one technique with the rest left to mle, and that we start there because a class must be fitted before it can be judged. "When nothing fits" was false. A Student t fits the Amazon returns well: the KS distance falls from 0.068 to 0.041 fitting by moments, and to 0.022 by maximum likelihood. The section is now "When the normal fails", and it turns the failure into a method: the Q-Q plot says the tails are too heavy, that points to a class with heavier tails, and the t delivers one. The fitted value nu = 5.8 comes with a warning against leaning on it, since it is estimated from the sample kurtosis, which is worst behaved exactly when the tails are heavy -- concrete motivation for mle. Plotting positions. The claim that the i-th order statistic estimates the quantile of order (i-0.5)/n was asserted without support. It is now derived from the ECDF, which steps up from (i-1)/n to i/n at that observation, so the data supply an interval of orders rather than one, and the midpoint splits the difference. A second reason is given: with i/n the largest observation would be matched to the quantile of order 1, which is infinite for every unbounded class here. Figures. Per the manual, no ax.set_title: the eight body figures now carry mystnb captions with fig: names, and the two inside solution directives are left bare, since a caption there becomes a LaTeX float and breaks the PDF build. Reference lines and CDF curves moved to lw=2. The statsmodels Q-Q figure is dropped in favour of a note recording that sm.qqplot exists, and why we build our own: the construction is worth understanding, and our version takes any distribution, which the earthquake exercise needs. statsmodels is no longer imported. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two datasets for a lecture in preparation on fitting distributions to data — the continuous data already here (heights, house prices, ages at death) covers the normal and lognormal cases, but the lecture also needs count data.
epl_match_goals.csvjapan_earthquakes.csvWhy these two
Goals per match average 2.830 with variance 2.777 (ratio 0.981), and the observed frequencies track the fitted Poisson across the whole range:
Earthquakes are the opposite, and deliberately so. Monthly counts have a variance-to-mean ratio near 97, and the times between events have a standard deviation half again their mean, where an exponential requires them to be equal. Aftershocks cluster, so the events are not independent — which is exactly the lesson, and it ties back to the independence section of
observed_distributions.I tested earthquakes as the main example first; they fail at both magnitude thresholds I tried (M≥5 and M≥6), so they became the counterexample instead.
Conventions
class: constructed, with builders following fetch → pre-process → validate → write. The football builder asserts exactly 380 matches in each of ten seasons, so a partially published season cannot slip in; the earthquake builder asserts the magnitude floor, the bounding box and the year range.AGENTS.mdthe builder queries it once over a closed window and commits the result rather than leaving the lecture to call it. Its manifest records that ComCat is a revised catalog, so a later re-run may differ slightly from the committed hash without either version being wrong.Consumers
Empty in both manifests — the lecture that reads them is still being written. Both
migration.ymlrecords stay atlandedand get their repoint entries when it lands.Part of QuantEcon/meta#336
🤖 Generated with Claude Code