Reduce retained SD save latency in DeskOS 1.9.1

README.md

@@ -6,17 +6,27 @@
 
 <p align="center"><strong>A bright, touch-first MeshCore desk for the SenseCAP Indicator D1L.</strong></p>
 
-DeskOS **1.9.0** is the stable release for the SenseCAP Indicator D1L.
+DeskOS **1.9.1** is the stable release for the SenseCAP Indicator D1L.
 It uses the production `full_feature` profile with conditional SD-primary retained history
 (`conditional` storage mode).
 
-[Stable release](https://github.com/n30nex/DeskOS-MeshCore/releases/tag/v1.9.0)
+[Stable release](https://github.com/n30nex/DeskOS-MeshCore/releases/tag/v1.9.1)
 · [Browser flasher](https://flasher.canadaverse.org/)
 · [User guide](docs/USER_GUIDE_D1L.md)
 · [Product page](https://canadaverse.org/deskos/)
 
 The previous stable release is
-[1.8.0](https://github.com/n30nex/DeskOS-MeshCore/releases/tag/v1.8.0).
+[1.9.0](https://github.com/n30nex/DeskOS-MeshCore/releases/tag/v1.9.0).
+
+## What is new in 1.9.1
+
+SD history saves use the bridge's existing verified transfer instead of
+reopening and flushing the card for every small chunk. The complete temporary
+file is read back and checked before it replaces the saved copy. Cancellation,
+card-change checks and recovery of the previous copy remain in place.
+
+See the [1.9.1 release notes](docs/RELEASE_NOTES_1.9.1.md) and tagged release
+for measurements and the exact device checks.
 
 ## What is new in 1.9.0
 

docs/RC1_RELEASE_EXECUTION_D1L.md

@@ -1,6 +1,6 @@
 # DeskOS D1L release execution
 
-## Current release procedure: 1.9.0
+## Current release procedure: 1.9.1
 
 The maintainer re-enabled GitHub Actions for the 1.8 production release. Dispatch the existing
 `d1l-ci` workflow on the exact release ref with the SD bridge included.
@@ -40,7 +40,7 @@ requests it. The 1.0 procedure below is historical.
 5. Verify the exact running version/commit, identity, display, radio and
    retained storage. Exercise the changed behavior using production firmware.
    Record any physical update/phone paths that were not exercised explicitly.
-6. Merge the tested source, tag `v1.9.0`, and publish it as the latest stable release after acceptance.
+6. Merge the tested source, tag `v1.9.1`, and publish it as the latest stable release after acceptance.
    Freshly download every asset and compare its bytes with staging. Update the
    existing flasher catalog and Canadaverse DeskOS page, then verify their
    public content and downloads. Remove obsolete build outputs only after

docs/RELEASE_CHECKLIST.md

@@ -9,6 +9,19 @@ SD-primary retained history when prepared storage is ready, visible live-only
 operation otherwise, and without silent default-NVS fallback. Historical RC2
 artifacts remain bound to their original `core_1_0` profile.
 
+## 1.9.1 maintenance release
+
+- Verify failed card readback, cancellation, abort failure and media-generation
+  changes cannot publish a new retained primary.
+- Measure the same contact preference save on 1.9.0 and the exact 1.9.1 image;
+  restore the original values and verify the complete contact book.
+- Build/test the candidate on the Pi, then verify the signed Actions package
+  and preserving install on the exact D1L.
+- Exercise Bluetooth Public/DM transmission and retention using the available
+  Windows adapter. Identify phone-app and signed-SD paths not repeated.
+- Publish exact assets, update the website/flasher, verify public downloads and
+  preserve release/recovery files before removing obsolete test builds.
+
 ## 1.9.0 production release
 
 The maintainer re-enabled GitHub Actions for this production release. Use the existing

docs/RELEASE_NOTES_1.9.1.md

@@ -0,0 +1,28 @@
+# DeskOS D1L 1.9.1
+
+This maintenance release addresses SD save latency. Messages, drafts, contacts,
+nodes, routes and retained packet segments share one verified transfer per
+saved file, using the streaming operation already supported by the production
+RP2040 bridge. It replaces repeated open/flush/close cycles for 192-byte writes.
+
+The bridge flushes and reads back the complete temporary file, checking its
+size and CRC before the existing replace-rename commits it. A failed transfer,
+readback or cancellation never reports a saved primary. Card-generation and
+identity-lineage checks, foreground cancellation and previous-copy recovery
+remain in place. Failed cancellation cleanup is reported as a storage error.
+
+The saved data format, identity/settings envelope, signed updater and RP2040
+firmware are unchanged. Use the paired production bridge included in the
+release package. Preserving USB updates keep identity, contacts, settings and
+SD data; firmware never formats the card.
+
+Native regression checks inject a card-byte error after an accepted write and
+verify that the previous saved copy survives. They also cover successful retry,
+mid-save cancellation, failed abort cleanup and backend-generation changes.
+The tagged release records exact firmware, build and device measurements;
+earlier phone-app or SD-update runs are not new-candidate hardware acceptance.
+
+The 1.9 features and [parity limits](DESKOS_MESHCORE_FEATURE_PARITY.md) continue
+to apply. Slow or unavailable cards remain visible. Companion delivery is
+confirmed only after the acknowledgement is saved, and remote responses depend
+on the peer and route.

docs/ROADMAP.md

@@ -26,6 +26,7 @@
 | **1.8.0-rc.6** | Correct received hop counts and sufficient companion telemetry stack | Published corrective candidate |
 | **1.8.0** | Production release of the 1.8 messaging, companion and update improvements | Production acceptance and exact artifacts recorded in the tagged release |
 | **1.9.0** | Larger text, mentions, contact auto-add rules and optional automatic daylight saving | Acceptance and exact artifacts recorded in the tagged release |
+| **1.9.1** | Verified streaming SD saves for retained history and preferences | Acceptance and measurements recorded in the tagged release |
 
 The release firmware is the ordinary public product. A controlled peer, Wi-Fi
 credentials, admin password, soak run, qualification firmware, or validation
@@ -286,3 +287,12 @@ Remaining work: localization and keyboard layouts, structured cross-client
 replies, other regional time rules, advanced phone commands, storage throughput,
 and the optional sensors/application suite. Complete WadaMesh parity is not
 claimed. Physical acceptance and exact artifacts belong in the tagged release.
+
+## 1.9.1: storage responsiveness
+
+Retained stores reuse the production bridge's verified file stream. One flush
+and complete readback replace repeated 192-byte open/flush/close cycles, while
+the existing temporary-file rename, card-generation checks and reset lineage
+remain the commit boundary. Failed readback keeps the previous primary;
+foreground cancellation remains bounded between transfer chunks. This closes
+the repeated-flush issue, not every possible slow-card or remote-response case.

main/d1l_config.h

@@ -1,7 +1,7 @@
 #pragma once
 
 #define D1L_FIRMWARE_NAME "MeshCore DeskOS D1L"
-#define D1L_FIRMWARE_VERSION "1.9.0"
+#define D1L_FIRMWARE_VERSION "1.9.1"
 #define D1L_CONSOLE_SCHEMA 1
 
 #define D1L_DISPLAY_WIDTH 480

main/storage/retained_blob_store.c

@@ -7,6 +7,7 @@
 #include "app/release_profile.h"
 #include "freertos/FreeRTOS.h"
 #include "esp_partition.h"
+#include "esp_rom_crc.h"
 #include "hal/rp2040_bridge.h"
 #include "mesh/route_store_worker.h"
 #include "nvs.h"
@@ -1620,6 +1621,12 @@ static esp_err_t sd_read_file_path(
     return ESP_OK;
 }
 
+static bool sd_write_continue(void *context)
+{
+    (void)context;
+    return !d1l_route_store_persistence_should_yield();
+}
+
 static esp_err_t sd_write_blob_for_generation(
     const d1l_retained_blob_store_config_t *config,
     const char *key,
@@ -1631,41 +1638,29 @@ static esp_err_t sd_write_blob_for_generation(
     char temp_path[D1L_RP2040_FILE_PATH_MAX + 1U];
     if (!build_sd_path(config, key, ".bin", path, sizeof(path)) ||
         !build_sd_path(config, key, ".tmp", temp_path, sizeof(temp_path)) ||
-        !src || len == 0) {
+        !src || len == 0 || len > UINT32_MAX) {
         return ESP_ERR_INVALID_ARG;
     }
     if (!store_backend_generation_matches(config, expected_generation)) {
         return ESP_ERR_INVALID_STATE;
     }
 
-    const uint8_t *data = (const uint8_t *)src;
-    size_t offset = 0;
-    while (offset < len) {
-        /* Temp-file chunks are non-destructive until the final rename. A
-         * pending foreground quiesce may leave a partial temp file, which a
-         * later offset-zero retry safely truncates. */
-        if (d1l_route_store_persistence_should_yield()) {
+    /* Keep one bridge-owned file session instead of reopening and flushing
+     * every 192 bytes. The bridge flushes and reads back the complete temporary
+     * file before success; the existing rename remains the commit point. */
+    d1l_rp2040_file_result_t write_result = {0};
+    esp_err_t write_ret = d1l_rp2040_bridge_file_write_verified(
+        temp_path, src, len, esp_rom_crc32_le(0U, src, (uint32_t)len),
+        sd_write_continue, NULL, &write_result,
+        D1L_RETAINED_SD_WRITE_TIMEOUT_MS);
+    if (write_ret != ESP_OK) {
+        if (strcmp(write_result.note, "abort_failed") == 0) {
+            write_ret = ESP_FAIL;
+        } else if (write_result.cancelled || write_ret == ESP_ERR_NOT_FINISHED) {
             return ESP_ERR_NOT_FINISHED;
         }
-        const size_t remaining = len - offset;
-        const size_t chunk = remaining > D1L_RP2040_FILE_CHUNK_MAX ?
-                             D1L_RP2040_FILE_CHUNK_MAX : remaining;
-        d1l_rp2040_file_result_t write_result = {0};
-        esp_err_t ret = d1l_rp2040_bridge_file_write(temp_path, (uint32_t)offset,
-                                                     data + offset, chunk,
-                                                     offset == 0, &write_result,
-                                                     D1L_RETAINED_SD_WRITE_TIMEOUT_MS);
-        if (ret != ESP_OK || write_result.length != chunk) {
-            const esp_err_t failure = ret == ESP_OK ? ESP_FAIL : ret;
-            note_sd_failure(config, D1L_RETAINED_SD_OP_WRITE, failure);
-            if (store_backend_generation_matches(config, expected_generation)) {
-                d1l_rp2040_file_result_t ignored = {0};
-                (void)d1l_rp2040_bridge_file_delete(
-                    temp_path, &ignored, D1L_RETAINED_SD_WRITE_TIMEOUT_MS);
-            }
-            return failure;
-        }
-        offset += chunk;
+        note_sd_failure(config, D1L_RETAINED_SD_OP_WRITE, write_ret);
+        return write_ret;
     }
 
     /* The replace-rename is the destructive commit point. If media changed

tests/native/retained_blob_store_backend_state_test.c

@@ -15,6 +15,7 @@
 #include "nvs_flash.h"
 #include "storage/factory_reset.h"
 #include "storage/retained_blob_store.h"
+#include "esp_rom_crc.h"
 
 #define TEST_RETAINED_PARTITION "d1l_retained"
 #define TEST_RETAINED_META_PARTITION "d1l_ret_meta"
@@ -110,6 +111,8 @@ typedef struct {
 static bool s_toggle_backend_during_write;
 static bool s_toggle_backend_during_rename;
 static esp_err_t s_sd_write_error;
+static bool s_corrupt_sd_write;
+static bool s_put_abort_fails;
 static bool s_toggle_backend_during_delete;
 static bool s_worker_should_yield;
 static bool s_chunked_read_case;
@@ -921,6 +924,9 @@ esp_err_t d1l_rp2040_bridge_file_write(const char *path, uint32_t offset,
         }
         if (len > 0U) {
             memcpy(file->data + offset, data, len);
+            if (s_corrupt_sd_write && offset == 0U) {
+                file->data[0] ^= 1U;
+            }
         }
         if (file->length < offset + len) {
             file->length = offset + len;
@@ -949,6 +955,50 @@ esp_err_t d1l_rp2040_bridge_file_write(const char *path, uint32_t offset,
     return ESP_OK;
 }
 
+esp_err_t d1l_rp2040_bridge_file_write_verified(
+    const char *path, const uint8_t *data, size_t len, uint32_t expected_crc32,
+    d1l_rp2040_file_continue_cb_t should_continue, void *context,
+    d1l_rp2040_file_result_t *out_result, uint32_t timeout_ms)
+{
+    assert(esp_rom_crc32_le(0U, data, (uint32_t)len) == expected_crc32);
+    esp_err_t ret = ESP_OK;
+    for (size_t offset = 0U; offset < len;) {
+        if (should_continue && !should_continue(context)) {
+            ret = ESP_ERR_INVALID_STATE;
+            break;
+        }
+        const size_t chunk = len - offset < D1L_RP2040_FILE_CHUNK_MAX ?
+            len - offset : D1L_RP2040_FILE_CHUNK_MAX;
+        ret = d1l_rp2040_bridge_file_write(path, (uint32_t)offset, data + offset,
+            chunk, offset == 0U, out_result, timeout_ms);
+        if (ret != ESP_OK) {
+            break;
+        }
+        offset += chunk;
+    }
+    if (ret == ESP_OK && should_continue && !should_continue(context)) {
+        ret = ESP_ERR_INVALID_STATE;
+    }
+    test_sd_file_t *file = s_sd_file_mode ? find_sd_file(path) : NULL;
+    if (ret == ESP_OK && file &&
+        (file->length != len || esp_rom_crc32_le(0U, file->data,
+            (uint32_t)file->length) != expected_crc32)) {
+        ret = ESP_ERR_INVALID_ARG;
+    }
+    if (ret != ESP_OK) {
+        if (s_put_abort_fails) {
+            strcpy(out_result->note, "abort_failed");
+        } else if (file) {
+            memset(file, 0, sizeof(*file)); /* The bridge aborts its temporary session. */
+        }
+    }
+    out_result->cancelled = ret == ESP_ERR_INVALID_STATE;
+    out_result->size = (uint32_t)len;
+    out_result->crc32 = expected_crc32;
+    out_result->last_error = ret;
+    return ret;
+}
+
 esp_err_t d1l_rp2040_bridge_file_delete(const char *path,
                                         d1l_rp2040_file_result_t *out_result,
                                         uint32_t timeout_ms)
@@ -2711,6 +2761,49 @@ static void test_completed_lineage_reads_exact_marker_and_data_backups(void)
     assert(s_delete_count == deletes_before);
 }
 
+static void test_retained_save_verifies_card_bytes_before_replacing_primary(void)
+{
+    const uint8_t original[] = "previous committed contacts";
+    uint8_t replacement[D1L_RP2040_FILE_CHUNK_MAX * 3U + 1U] = {0};
+    assert(esp_rom_crc32_le(0U, (const uint8_t *)"123456789", 9U) == 0xcbf43926U);
+    clear_nvs_case();
+    reset_sd_files();
+    d1l_retained_blob_store_note_sd_backend(true, true, true,
+        D1L_RP2040_FILE_LINE_MAX, D1L_RP2040_FILE_CHUNK_MAX, D1L_RP2040_FILE_PATH_MAX);
+    const d1l_retained_blob_store_id_t store = D1L_RETAINED_BLOB_STORE_CONTACTS;
+    const d1l_retained_blob_store_backend_state_t backend = state_for(store);
+    assert(d1l_retained_blob_store_write_sd_primary_guarded(
+        store, "contacts", original, sizeof(original), backend.generation) == ESP_OK);
+    const uint32_t renames = s_rename_count;
+    s_corrupt_sd_write = true;
+    assert(d1l_retained_blob_store_write_sd_primary_guarded(
+        store, "contacts", replacement, sizeof(replacement), backend.generation) == ESP_ERR_INVALID_ARG);
+    s_corrupt_sd_write = false;
+    assert(s_rename_count == renames);
+    test_sd_file_t *file = find_sd_file("stores/contacts/contacts.bin");
+    assert(file && file->length == sizeof(original));
+    assert(memcmp(file->data, original, sizeof(original)) == 0);
+    assert(d1l_retained_blob_store_write_sd_primary_guarded(
+        store, "contacts", replacement, sizeof(replacement), backend.generation) == ESP_OK);
+    file = find_sd_file("stores/contacts/contacts.bin");
+    assert(file && file->length == sizeof(replacement));
+    assert(memcmp(file->data, replacement, sizeof(replacement)) == 0);
+    assert(s_rename_count == renames + 1U);
+
+    /* A failed abort is an actual storage failure, not a successful yield. */
+    s_chunked_write_yield_case = true;
+    s_put_abort_fails = true;
+    assert(d1l_retained_blob_store_write_sd_primary_guarded(
+        store, "contacts", replacement, sizeof(replacement), backend.generation) == ESP_FAIL);
+    s_chunked_write_yield_case = false;
+    s_put_abort_fails = false;
+    s_worker_should_yield = false;
+    assert(s_rename_count == renames + 1U);
+    d1l_retained_blob_store_sd_stats_t stats = {0};
+    assert(d1l_retained_blob_store_sd_stats(store, &stats));
+    assert(stats.sd_degraded_latched && stats.sd_last_error == ESP_FAIL);
+}
+
 static void test_sd_write_warning_requires_a_successful_same_media_commit(void)
 {
     static const uint8_t payload[] = "retained contacts";
@@ -2854,6 +2947,7 @@ int main(int argc, char **argv)
     test_factory_reset_sd_recovery_end_to_end();
     test_completed_lineage_reads_exact_marker_and_data_backups();
     test_sd_write_warning_requires_a_successful_same_media_commit();
+    test_retained_save_verifies_card_bytes_before_replacing_primary();
 
     puts("native retained backend generation and NVS partition: ok");
     return 0;

tests/native/stubs/esp_rom_crc.h

@@ -0,0 +1,16 @@
+#pragma once
+
+#include <stdint.h>
+
+static inline uint32_t esp_rom_crc32_le(uint32_t crc, const uint8_t *data,
+                                      uint32_t length)
+{
+    crc = ~crc;
+    for (uint32_t i = 0; i < length; ++i) {
+        crc ^= data[i];
+        for (unsigned bit = 0; bit < 8; ++bit) {
+            crc = (crc >> 1) ^ (0xedb88320U & (0U - (crc & 1U)));
+        }
+    }
+    return ~crc;
+}

tests/test_release_metadata_contract.py

@@ -15,7 +15,7 @@ def test_public_release_metadata_has_no_bringup_or_stub_markers():
     mesh_source = read("main/mesh/meshcore_service.c")
     package_script = read("scripts/package_release_d1l.py")
 
-    assert '#define D1L_FIRMWARE_VERSION "1.9.0"' in config
+    assert '#define D1L_FIRMWARE_VERSION "1.9.1"' in config
     assert "D1L_PHASE1_BRINGUP" not in cmake
     assert "phase1_stub" not in mesh_source
     assert "phase2_stub" not in console

tests/test_retained_blob_store_contract.py

@@ -135,7 +135,7 @@ def test_retained_blob_store_keeps_history_sd_first_and_retires_legacy_nvs():
     assert "s_store_sd_enabled[config->id]" in source
     assert "s_store_sd_stats[D1L_RETAINED_BLOB_STORE_COUNT]" in source
     assert "note_sd_failure(config, D1L_RETAINED_SD_OP_READ, sd_ret)" in source
-    assert "note_sd_failure(config, D1L_RETAINED_SD_OP_WRITE, failure)" in source
+    assert "note_sd_failure(config, D1L_RETAINED_SD_OP_WRITE, write_ret)" in source
     assert "note_sd_failure(config, D1L_RETAINED_SD_OP_RENAME, ret)" in source
     assert "sd_error_latches_degraded" in source
     assert "stats->sd_degraded_latched = true" in source
@@ -219,8 +219,11 @@ def test_retained_blob_store_keeps_history_sd_first_and_retires_legacy_nvs():
     assert "d1l_rp2040_bridge_file_delete" in source
     assert "D1L_RP2040_FILE_CHUNK_MAX" in source
     assert '#include "mesh/route_store_worker.h"' in source
+    read_blob = read_blob.split("static bool sd_write_continue", 1)[0]
     assert read_blob.count("d1l_route_store_persistence_should_yield()") == 2
-    assert guarded_write.count("d1l_route_store_persistence_should_yield()") == 2
+    assert "d1l_rp2040_bridge_file_write_verified(" in guarded_write
+    assert "sd_write_continue, NULL, &write_result" in guarded_write
+    assert guarded_write.count("d1l_route_store_persistence_should_yield()") == 1
     assert read_blob.count("return ESP_ERR_NOT_FINISHED;") == 2
     assert guarded_write.count("return ESP_ERR_NOT_FINISHED;") == 2
     assert "ret != ESP_ERR_NOT_FINISHED" in source