Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions CRAN-SUBMISSION
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
Version: 1.1.2
Date: 2026-08-28 14:07:44 UTC
SHA: 9315fd9e6c71b6dcace9716dbb7c824fe90fbfcb
Version: 1.2.0
Date: 2026-10-14 07:44:52 UTC
SHA: 2d6ae76a0b7eddcde9f7bb2261b15e249b28b977
2 changes: 1 addition & 1 deletion DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Package: melodi
Title: Download Insee French Institute of Statistics Open Data
Version: 1.1.2
Version: 1.2.0
Authors@R: c(
person("Cédric", "Bobinec", , "cedric.bobinec@insee.fr", role = c("aut", "cre")),
person("Christophe", "Goudeau", , "christophe.goudeau@insee.fr", role = "aut"),
Expand Down
6 changes: 6 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
# melodi 1.2.0
- `get_file`: download_file_name devient facultatif : par défaut, le nom du fichier dans Melodi est repris pour le fichier téléchargé (#32)
- Améliorations générales des fonctions d'appel à Melodi :
- Remonte proprement une erreur en cas de problème de connexion à l'API Melodi (#28)
- Relance automatique des appels en cas d'erreur (`req_retry`) (#15)

# melodi 1.1.2
- Correctif URL dans le README (génére une note bloquante pour soumission au CRAN)

Expand Down
9 changes: 6 additions & 3 deletions R/get_all_data.R
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,14 @@ get_all_data <- function(
download_directory, glue::glue("{ds_name}.zip")
)


httr2::request(zip_url) |>
resp <- httr2::request(zip_url) |>
httr2::req_user_agent(getOption("rmelodi.req_user_agent")) |>
httr2::req_retry(max_tries = 3, max_seconds = 30, retry_on_failure = TRUE) |>
httr2::req_progress() |>
httr2::req_perform(downloaded_zip_path)
httr2::req_perform(downloaded_zip_path) |>
tryCatch(error = \(e) message("Could not get data: ", conditionMessage(e)))
# API failed gracefully: exit quietly (return() only works inside a function)
if (is.null(resp)) return(invisible(NULL))

files_in_zip <- zip::zip_list(downloaded_zip_path)

Expand Down
7 changes: 6 additions & 1 deletion R/get_catalog.R
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,15 @@ get_catalog <- function(
url <- glue::glue("{getOption('rmelodi.base_url_api')}/catalog/all")

message("Request all catalog : ", url)

all_dataset <- httr2::request(url) |>
httr2::req_user_agent(getOption("rmelodi.req_user_agent")) |>
httr2::req_retry(max_tries = 3, max_seconds = 30, retry_on_failure = TRUE) |>
httr2::req_perform() |>
httr2::resp_body_json(simplifyVector = TRUE)
httr2::resp_body_json(simplifyVector = TRUE) |>
tryCatch(error = \(e) message("Could not get data: ", conditionMessage(e)))
# API failed gracefully: exit quietly (return() only works inside a function)
if (is.null(all_dataset)) return(invisible(NULL))

# extract only usefull informations
all_dataset_extract <- all_dataset |>
Expand Down
18 changes: 13 additions & 5 deletions R/get_data.R
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,22 @@ get_data <- function(
) {
max_result_api <- getOption("rmelodi.max_result_api")

# 1 - Count numer of lines of request
# 1 - Count number of lines of request
request_count <- httr2::request(url) |>
httr2::req_user_agent(getOption("rmelodi.req_user_agent")) |>
httr2::req_retry(max_tries = 3, max_seconds = 30, retry_on_failure = TRUE) |>
# Add Melodi /data parameters to only count lines
httr2::req_url_query(totalCount = TRUE) |>
httr2::req_url_query(maxResult = 0)

message("Total count request : ", request_count$url)

data_count <- request_count |>
httr2::req_user_agent(getOption("rmelodi.req_user_agent")) |>
httr2::req_perform() |>
httr2::resp_body_json(simplifyVector = TRUE)
httr2::resp_body_json(simplifyVector = TRUE) |>
tryCatch(error = \(e) message("Could not get data: ", conditionMessage(e)))
# API failed gracefully: exit quietly (return() only works inside a function)
if (is.null(data_count)) return(invisible(NULL))

count <- data_count[["paging"]][["count"]]
message(
Expand All @@ -49,15 +53,19 @@ get_data <- function(

# 2 - request is OK, get results
request <- httr2::request(url) |>
httr2::req_user_agent(getOption("rmelodi.req_user_agent")) |>
httr2::req_retry(max_tries = 3, max_seconds = 30, retry_on_failure = TRUE) |>
# TODO useless ? this is the default value now
httr2::req_url_query(idTerritoire = TRUE) |>
# maximum maxResult authorized by Melodi API
httr2::req_url_query(maxResult = max_result_api)

result <- request |>
httr2::req_user_agent(getOption("rmelodi.req_user_agent")) |>
httr2::req_perform() |>
httr2::resp_body_json(simplifyVector = TRUE)
httr2::resp_body_json(simplifyVector = TRUE) |>
tryCatch(error = \(e) message("Could not get data: ", conditionMessage(e)))
# API failed: exit quietly (return() only works inside a function)
if (is.null(result)) return(invisible(NULL))

# bind datas in one dataframe
dimensions_obj <- result[["observations"]][["dimensions"]]
Expand Down
48 changes: 42 additions & 6 deletions R/get_file.R
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@
#'
#' @param ds_name Dataset name
#' @param file_name File name
#' @param download_file_name Filename on disk after download
#' @param download_file_name Filename on disk after download - if not specified,
#' preserve original file name from Melodi - default NULL (preserve original
#' file name)
#' @param download_directory Download directory - default system tempdir
#'
#' @return downloaded file name on disk
Expand All @@ -17,23 +19,57 @@
get_file <- function(
ds_name,
file_name,
download_file_name,
download_file_name = NULL,
download_directory = tempdir()
) {
# Build useful parameters
url <- glue::glue(
"{getOption('rmelodi.base_url_api')}/file/{ds_name}/{file_name}"
)
path <- file.path(download_directory, download_file_name)

if (!is.null(download_file_name)) {
path <- file.path(download_directory, download_file_name)
message("Downloaded path : ", path)
} else {
# if no file name specified : tempfile
# and then after download, rename with original file name
path <- tempfile()
message("Temporary downloaded path : ", path)
}

# Download file
message("Request file : ", url)
message("Downloaded path : ", path)

httr2::request(url) |>
resp <- httr2::request(url) |>
httr2::req_user_agent(getOption("rmelodi.req_user_agent")) |>
httr2::req_retry(max_tries = 3, max_seconds = 30, retry_on_failure = TRUE) |>
httr2::req_progress() |>
httr2::req_perform(path = path)
httr2::req_perform(path = path) |>
tryCatch(error = \(e) message("Could not get data: ", conditionMessage(e)))
# API failed gracefully: exit quietly (return() only works inside a function)
if (is.null(resp)) {
return(invisible(NULL))
}

if (is.null(download_file_name)) {
# Extract original filename from content-disposition in request header
# Exemple of content-disposition :
# inline; filename="DS_SRCV_CONFIANCE_2025_CSV_FR.zip"
cd <- httr2::resp_headers(resp)[["content-disposition"]]
m <- regmatches(cd, regexec('filename\\*?="?([^";]+)"?', cd))

if (length(m[[1]]) > 1) {
original_file_name <- m[[1]][2]
message("Original file name from Melodi : ", original_file_name)
# Rename the file on disk with this new name
renamed_path <- file.path(download_directory, original_file_name)
message("Move file to path : ", renamed_path)
file.copy(path, renamed_path)
message("Remove temporary file : ", path)
unlink(path)
return(renamed_path)
}
# else : no rename, preserve tempfile name
}
return(path)
}
6 changes: 5 additions & 1 deletion R/get_metadata.R
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,12 @@ get_metadata <- function(

dataset <- httr2::request(url) |>
httr2::req_user_agent(getOption("rmelodi.req_user_agent")) |>
httr2::req_retry(max_tries = 3, max_seconds = 30, retry_on_failure = TRUE) |>
httr2::req_perform() |>
httr2::resp_body_json(simplifyVector = TRUE)
httr2::resp_body_json(simplifyVector = TRUE) |>
tryCatch(error = \(e) message("Could not get data: ", conditionMessage(e)))
# API failed gracefully: exit quietly (return() only works inside a function)
if (is.null(dataset)) return(invisible(NULL))

return(dataset)
}
6 changes: 5 additions & 1 deletion R/get_range.R
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,12 @@ get_range <- function(

dataset <- httr2::request(url) |>
httr2::req_user_agent(getOption("rmelodi.req_user_agent")) |>
httr2::req_retry(max_tries = 3, max_seconds = 30, retry_on_failure = TRUE) |>
httr2::req_perform() |>
httr2::resp_body_json(simplifyVector = FALSE)
httr2::resp_body_json(simplifyVector = FALSE) |>
tryCatch(error = \(e) message("Could not get data: ", conditionMessage(e)))
# API failed gracefully: exit quietly (return() only works inside a function)
if (is.null(dataset)) return(invisible(NULL))

range <- dataset[["range"]]

Expand Down
6 changes: 5 additions & 1 deletion R/get_range_geo.R
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,12 @@ get_range_geo <- function(

dataset <- httr2::request(url) |>
httr2::req_user_agent(getOption("rmelodi.req_user_agent")) |>
httr2::req_retry(max_tries = 3, max_seconds = 30, retry_on_failure = TRUE) |>
httr2::req_perform() |>
httr2::resp_body_json(simplifyVector = FALSE)
httr2::resp_body_json(simplifyVector = FALSE) |>
tryCatch(error = \(e) message("Could not get data: ", conditionMessage(e)))
# API failed gracefully: exit quietly (return() only works inside a function)
if (is.null(dataset)) return(invisible(NULL))

# Keep GEO only
range_geo <- dataset$range |>
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
[![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)
[![R-CMD-check](https://github.com/InseeFrLab/melodi/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/InseeFrLab/melodi/actions/workflows/R-CMD-check.yaml)
[![CRAN RStudio mirror downloads](https://cranlogs.r-pkg.org/badges/grand-total/melodi)](https://www.r-pkg.org/pkg/melodi)
[![codecov](https://codecov.io/github/InseeFrLab/melodi/graph/badge.svg?token=849S6LIOHR)](https://codecov.io/github/InseeFrLab/melodi)
[![codecov](https://codecov.io/github/InseeFrLab/melodi/graph/badge.svg?token=849S6LIOHR)](https://app.codecov.io/github/InseeFrLab/melodi)
<!-- badges: end -->

🇫🇷 Télécharger, parcourir, filtrer les données et métadonnées diffusées par l'Insee en open data
Expand Down
12 changes: 12 additions & 0 deletions cran-comments.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,15 @@
# Submission of {melodi} 1.2.0

This version fixes the issue "Packages that use Internet resources should fail gracefully"
by adding try/catch blocks and error messages to every function that calls Internet resources.

## R CMD check results

* checked on windows, mac, ubuntu (with GitHub actions) checked on CRAN [win-builder](https://win-builder.r-project.org/)

0 errors | 0 warnings | 0 note


# Submission of {melodi} 1.1.2

## R CMD check results
Expand Down
6 changes: 4 additions & 2 deletions man/get_file.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading