melodi by tgrandje · Pull Request #306 · InseeFrLab/pynsee

tgrandje · 2026-04-07T19:21:12Z

No description provided.

tgrandje · 2026-04-14T19:24:52Z

@hadrilec @linogaliana @tfardet @RLesur do any of you guys see the point in covering MELODI's "référentiel" endpoint through pynsee?

It doesn't seem to be used in the R package and I have no usecase in my mind right now (that and the fact that the json returned seem overly complex to be used in a generic way)...

tgrandje · 2026-04-14T19:29:10Z

Note: 2 tickets have been opened on INSEE's API portal.

one regarding problems with pagination (for instance querying DS_POPULATIONS_REFERENCE with no filters). I don't think any workaround can be done for this in pynsee.
one regarding X-INSEE-Api-Key-Integration header were MELODI seems to confuse the header with the one used for SIRENE consumption. I've added a fix in pynsee in the PynseeAPISession object.

hadrilec · 2026-04-16T09:53:15Z

Hi, I had started in the branch catalog to write a function retrieving the list of datasets available via melodi. On the long term, I think it makes sense, as I understand more INSEE datasets shall be available melodi. I understand that DoReMiFaSol already uses melodi: https://github.com/InseeFrLab/DoReMIFaSol

tgrandje · 2026-04-16T17:18:36Z

Hi @hadrilec , I've already re-used your code for the catalog (which I also think is gonna be madatory). I was instead speaking of the "référentiel" endpoint and I haven't quite seen the point (yet ?).

Right now, you got catalog (as already said), you got all available dimensions and values for a given dataset using the "range" endpoint : it seems to me that's enough ?

* add .vscode to gitignore * add notebooks to gitignore * wip docstrings and fixes data.py * fix measures column issue in _parse_dataset_observations * fix(melodi): handle non-MELODI idbanks * test(melodi): add pytest coverage and fix update/silent signatures Signed-off-by: GrasClement <gras.clement@orange.fr> * test(melodi): finish tests for melodi Signed-off-by: GrasClement <gras.clement@orange.fr> * doc(melodi): add melodi to the documentation Signed-off-by: GrasClement <gras.clement@orange.fr> * doc(melodi): linting Signed-off-by: GrasClement <gras.clement@orange.fr> * doc(melodi): fix spacing Signed-off-by: GrasClement <gras.clement@orange.fr> * fix(melodi): simplify measures parsing, log empty idbank results Signed-off-by: GrasClement <gras.clement@orange.fr> * test(melodi): fix redundant checks and update comments per review Signed-off-by: GrasClement <gras.clement@orange.fr> * doc(melodi): reorder modules and flag metadata as deprecated Signed-off-by: GrasClement <gras.clement@orange.fr> --------- Signed-off-by: GrasClement <gras.clement@orange.fr> Co-authored-by: cgras <clement.gras@ccomptes.fr>

tgrandje · 2026-10-14T09:59:00Z

This should be ok, let's review @tfardet when you got time enough !

tfardet

Thanks @tgrandje and @GrasClement for this new additions.
I have a few points of discussions that I opened regarding naming and descriptions, let me know what you think.

tfardet · 2026-10-14T13:08:07Z

+    Get a dataset's available dimensions (ie "ranges").
+
+    Available dimensions can be retrieved in a sparse format (with only
+    the dimensions' descriptions) or a long format (including all available
+    values for each dimension).


I'm a bit worried that this would not be very easy to understand for newcomers.
Should we maybe just start with "columns' names", then say that these can be called dimensions or ranges? I'm also wondering why we need to introduce an additional way of naming them with "dimension".

Same with the "sparse" format, since there is no "sparse" keyword, if it is mentioned there (should it be?), I think it should be associated to include_values to make the link explicit.

Mmh, true enough. Fact is, the swagger currently describes "range[s]" in the endpoint, "dimensions et modalités" in the french description. To be coherent with API, I think we should stick to the "range" denomination, but that one is a bit obscure to me...

tfardet · 2026-10-14T13:10:24Z

+
+
+@save_df(day_lapse_max=30)
+def get_idbank(


shouldn't this be renamed to get_series_by_idbank/get_data_by_idbank or something similar?
current name feels misleading to me (I'd expect that the function helps me look for an idbank given a name or keywords)

True. I'd go as far as to rename it as get_series as in macrodata ; do you have on objection on that (arguments will not be the same, but the logic stands I think).

tfardet · 2026-10-14T13:16:45Z

+    arguments should NOT be added to the kwargs and will be ignored. `range`
+    will also be ignored : to retrieve a dataset's dimensions, have a look at
+    `pynsee.melodi.get_range`.
+


move to numpy format:

Note ----

tfardet · 2026-10-14T13:20:42Z

-from typing import Optional, Union
 import warnings
+from functools import lru_cache
+from typing import Optional, Union


these are not needed anymore at the Python versions we support; should we take the opportunity to move to | bit by bit or do we want to keep that for a single PR that covers the whole code?

We might want to make a full PR for python >= 3.11 instead? (I was also wondering how I came with the python >=3.10 in the pyproject.toml, I suspect I encountered dependencies conflicts at some point).

tfardet · 2026-10-14T13:24:32Z

-from pynsee.download import *

-__version__ = "0.2.5"
+__version__ = "0.2.6"


we need to discuss this, that's not a bugfix release (long overdue, I'm afraid, my fault), we should probably jump to 0.3.0

Yes you're right. Do you wan the 0.3.0 in here or in a ad hoc PR before release (I'm in favour of the 2nd option).

tfardet · 2026-10-14T13:31:13Z

    "statistique", "démographie", "géospatial", "SIG"
 ]
-requires-python = ">=3.9"
+requires-python = ">=3.10"


should we move to 3.11? 3.10 is no longer maintained? (I've no strong opinion about this)

Sure, that PR was started some time ago 😄 ! Il fix this to 3.11, but we need a more serious PR for this afterwards (including badges, dependencies checkup, ...)

tfardet · 2026-10-14T13:33:06Z

@@ -0,0 +1 @@
+# -*- coding: utf-8 -*-


do we need the file? I don't think we do with pytest

Uh, good question, I'll double check. If that's not needed anymore, I'll remove the other ones from previous tests.

Co-authored-by: tfardet <79037344+tfardet@users.noreply.github.com>

tgrandje added 2 commits April 6, 2026 20:07

melodi catalog

266de83

wip data

e0c9b19

tgrandje self-assigned this Apr 7, 2026

tgrandje changed the title ~~melodi catalog~~ melodi Apr 7, 2026

tgrandje mentioned this pull request Apr 7, 2026

new catalog module using Melodi API #217

Closed

tgrandje added 4 commits April 7, 2026 22:09

wip

9f925ba

get_range

0f4303a

wip docstring, fix one-page-only data result & raise_if_not_ok

6bf593e

get_melodi_dataset docstring & fixes

5b1a477

wip get_idbank

db8b6ef

tgrandje mentioned this pull request May 18, 2026

INSEE's DDL API is deprecated. Should we switch to MELODI API ? tgrandje/cl-hubeau#65

Closed

tgrandje mentioned this pull request Jun 11, 2026

Proposition pour les Journées open-source Insee (16-17 juin 2026) : contribution à pynsee #307

Open

GrasClement mentioned this pull request Jun 15, 2026

feat(melodi): tests, docstrings, fix measures parsing #308

Merged

GrasClement and others added 4 commits October 14, 2026 11:08

cleanup

8a1fc25

force update (for local tests)

e97dbb9

reset pyproject

5fe3fd5

tgrandje marked this pull request as ready for review October 14, 2026 09:31

tgrandje added 2 commits October 14, 2026 11:54

test "melodi" connection instead of "donnes locales"

00eb3e6

Merge branch 'master' into feat/melodi

5d0c361

tgrandje requested a review from tfardet October 14, 2026 09:57

tfardet requested changes Oct 14, 2026

View reviewed changes

tgrandje and others added 3 commits October 14, 2026 20:19

Update pynsee/melodi/data.py

53e5884

Co-authored-by: tfardet <79037344+tfardet@users.noreply.github.com>

Update pynsee/melodi/data.py

decaadb

Co-authored-by: tfardet <79037344+tfardet@users.noreply.github.com>

Update pynsee/utils/requests_session.py

176e87a

Co-authored-by: tfardet <79037344+tfardet@users.noreply.github.com>



		@save_df(day_lapse_max=30)
		def get_idbank(

Conversation

tgrandje commented Apr 7, 2026

Uh oh!

tgrandje commented Apr 14, 2026

Uh oh!

tgrandje commented Apr 14, 2026

Uh oh!

hadrilec commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tgrandje commented Apr 16, 2026

Uh oh!

tgrandje commented Oct 14, 2026

Uh oh!

tfardet left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hadrilec commented Apr 16, 2026 •

edited

Loading