Repository navigation

Transition value JSON from old to new flat format - #539

Draft

suvayu wants to merge 117 commits into

masterfrom

WIP-data-transition

Draft

Transition value JSON from old to new flat format#539
suvayu wants to merge 117 commits into
masterfrom
WIP-data-transition

Commits on May 21, 2025
- move compatibility scripts to their own folder
  Ole Mussmann
  committed
  c5bbc0e
- add parameter_value reencoding
  Ole Mussmann
  committed
  f114d29
- add alembic migration script
  Ole Mussmann
  committed
  304b257
- spinedb_api/compat/: remove unused methods
  suvayu
  committed
  3ab46be
- spinedb_api/compat/: fix type hints
  suvayu
  committed
  0872ab0
- spinedb_api/compat/: remove PEP 723 metadata
  suvayu
  committed
  8bda858
Commits on May 23, 2025
- spinedb_api/compat/: fix input type
  Ole Mussmann
  committed
  06db77a
- add developer's documentation for data transition
  Ole Mussmann
  committed
  0c7dd8e
Commits on May 26, 2025
- alembic: blacken reencode_parameter_values.py
  suvayu
  committed
  a7f2140
- alembic: partly handle type column in reencode_parameter_values.py
  suvayu
  committed
  463767d
- models.py: remove bytes, since pa.UnionArray is now supported
  suvayu
  committed
  3533c2c
Commits on May 29, 2025
- models.py: remove last remaining bits for bytes support
  suvayu
  committed
  7baf441
- models.py: fix TimePattern
  suvayu
  committed
  c5a41a2
- models.py: add 'any_array' to support nullable mixed types
  suvayu
  committed
  acff451
- data_transition: fix missing import TimePattern
  suvayu
  committed
  4841a6f
- data_transition: alternate implementation
  suvayu
  committed
  287851d
Commits on Jun 2, 2025
- alembic: improve data migration script
  Ole Mussmann
  committed
  8ce8086
Commits on Jun 3, 2025
- models.py: remove last remaining bits for bytes support
  suvayu
  committed
  7c2e269
- models.py: fix 'any_array'
  suvayu
  committed
  1c6001d
- models.py: cleanup type aliases
  suvayu
  committed
  47d79a5
- models.py: use pydantic dataclasses
  suvayu
  committed
  fa25066
Commits on Jun 5, 2025
- compat: factor out some array creation to encode.py (WIP)
  suvayu
  committed
  811e263
- compat: make warning in make_columns more prominent
  suvayu
  committed
  8d480ac
- data-transition: use relativedelta instead of pandas.DateOffset
  suvayu
  committed
  80749bd
- models.py: model duration with relativedelta from dateutil
  suvayu
  committed
  57b78c4
- compat/encode.py: change sentinel implementation
  suvayu
  committed
  4b8ea09
- compat/encode.py: fix column to array conversion for all types
  suvayu
  committed
  ae1f30f
- compat/encode.py: drop all conversion from dataframe
  suvayu
  committed
  4c8a990
- pyproject.toml: upgrade pyarrow version, explicitly require pydantic
  suvayu
  committed
  4b11c43
- compat: rename module for clarity
  suvayu
  committed
  102bde2
- README_dev.md: update renamed module
  suvayu
  committed
  b375ce8
Commits on Jun 6, 2025
- compat: move duration parsing to own module for reusability
  suvayu
  committed
  0ac0896
Commits on Jun 7, 2025
- models.py: minor fix to import
  suvayu
  committed
  7d6053e
- models.py: cleaner implementation for time-pattern
  suvayu
  committed
  4e767cc
- data_transition: fix old -> new format conversions
  suvayu
  committed
  5af2dd2
- data_transition: fix module import formatting
  suvayu
  committed
  e4ffc8a
Commits on Jun 8, 2025
- compat/converters.py: converters for duration types
  suvayu
  committed
  51a33da
Commits on Jun 13, 2025
- models.py: function to convert dict to array dataclasses
  suvayu
  committed
  02131e5
- arrow_value.py: function to convert array dataclasses to pyarrow
  suvayu
  committed
  bfa64ea
Commits on Jun 14, 2025
- {arrow_values,models}.py: convert any array to union array
  suvayu
  committed
  4f95a91
Commits on Jun 27, 2025
- Merge branch 'master' into WIP-data-transition
  soininen
  committed
  21122be
Commits on Jul 9, 2025
- compat/encode.py: bugfix, variables weren't named correctly
  suvayu
  committed
  145faf1
Commits on Jul 11, 2025
- compat: fix formatter doc string, and minor formatting fix
  suvayu
  committed
  9115bc4
- {models,compat/encode}.py: fix conversion from pandas.Timestamp
  suvayu
  committed
  ee03a13
Commits on Jul 23, 2025
- to_database() for new array JSON
  soininen
  committed
  759b563
Commits on Jul 24, 2025
- Add durations as acceptable values for value column
  soininen
  committed
  6936c61
- Make durations and datetimes work in run end encoded and dictionary arrays
  soininen
  committed
  5d95640
Commits on Jul 25, 2025
- Fix using pd.TimeStamps or datetimes in IndexArray and the like
  soininen
  committed
  d45575e
- Apply new value JSON migration to all relevant tables
  soininen
  committed
  c31bbdf
- Fix Alembic migration
  soininen
  committed
  e56e44f
- Fix Alembic migration
  soininen
  committed
  b70569d
Commits on Aug 21, 2025
- Merge branch 'master' into WIP-data-transition
  soininen
  committed
  b153481
- Add to_arrow() method to ParameterValue and its subclasses
  soininen
  committed
  3a9d259
Commits on Aug 29, 2025
- Always store new tabular JSON in database
  soininen
  committed
  136f762
- Implement migration to tabular JSON while keeping values backwards compatible (#559)
  soininen
  authored
  3fab999
- Fix Alembic migration
  soininen
  committed
  2de12c8
- Fix arrow_value.from_database()
  soininen
  committed
  e9f8c99
Commits on Aug 30, 2025
- data_transition.py: remove experimentation converter
  suvayu
  committed
  8c1603d
- models.py: remove PEP 723 script, now depends on spinedb_api
  suvayu
  committed
  90c2d29
- models.py: one import per line
  suvayu
  committed
  54338d1
- converters.py: refactor relativedelta <-> JSON duration conversion
  suvayu
  committed
  aeedea7
- models.py: fix type annotations
  suvayu
  committed
  765a729
- models.py: remove custom validation & conversion
  suvayu
  committed
  d3ef8f6
- models.py: change to schema generation using TypeAdapter (recommended)
  suvayu
  committed
  1172e89
- models.py: refactor (de)serialisation
  suvayu
  committed
  023130d
- data_transition.py: use models.TimePeriod to wrap time-patterns
  suvayu
  committed
  cfc61c6
- data_transition.py: don't name date-time/duration as "value"
  suvayu
  committed
  d2dc941
- models.py: fix metadata handling after rebase
  suvayu
  committed
  d946df8
Commits on Aug 31, 2025
- models.py: allow time_period in any array for consistency
  suvayu
  committed
  2c696f2
- models.py: add missing mode in to_json
  suvayu
  committed
  81de0e4
Commits on Sep 1, 2025
- Merge branch 'master' into WIP-data-transition
  soininen
  committed
  4ce80aa
- models.py: fix typeddict, any array does not have value_type
  suvayu
  committed
  ebf335a
- models.py: remove time_period from any array
  suvayu
  committed
  97d93da
- models.py: fix json serialization call
  suvayu
  committed
  6f81265
- compat/converters.py: add pa.MonthDayNano -> duration/relativedelta
  suvayu
  committed
  143f367
- compat/converters.py: bug fix pa.MonthDayNano -> intermediate dict
  suvayu
  committed
  01fbd13
- Allow null values in index arrays, fix unit tests
  soininen
  committed
  6e4a3e9
Commits on Sep 2, 2025
- Remove unused test_data_transition.py
  soininen
  committed
  89f3808
Commits on Sep 3, 2025
- Merge branch 'master' into WIP-data-transition
  soininen
  committed
  78002e6
Commits on Sep 8, 2025
- value_support.py: backwards compat fix load_db_value signature
  suvayu
  committed
  1ad08e5
Commits on Sep 11, 2025
- Improve parameter value compatibility & refactoring
  soininen
  committed
  117edde
- Merge branch 'master' into WIP-data-transition
  soininen
  committed
  deaa13a
- Bump DB server version to 9
  soininen
  committed
  482c322
Commits on Sep 14, 2025
- Merge branch 'master' into WIP-data-transition
  soininen
  committed
  67537c2
Commits on Sep 16, 2025
- Fix GAMS version check when current work directory is read-only
  soininen
  committed
  31588a0
- Revert "Fix GAMS version check when current work directory is read-only"
  soininen
  committed
  a6085da
Commits on Sep 17, 2025
- Fix Alembic migration
  soininen
  committed
  d3cb47b
- Merge branch 'master' into WIP-data-transition
  soininen
  committed
  2bb0da9
Commits on Sep 25, 2025
- Merge branch 'master' into WIP-data-transition
  soininen
  committed
  fbeda3e
- Fix unit tests for Alembic migrations
  soininen
  committed
  24eb6d0
Commits on Oct 10, 2025
- Merge branch 'master' into WIP-data-transition
  soininen
  committed
  faa8a51
Commits on Oct 13, 2025
- Store leaf TimeSeries indices to last column when expanding Maps.
  soininen
  committed
  cfeac47
Commits on Oct 14, 2025
- Rework Map.from_arrow() to handle more corner-cases
  soininen
  committed
  4159c5d
Commits on Oct 15, 2025
- Add support for converting Map's leafs from/to Arrays
  soininen
  committed
  cfd4475
Commits on Nov 4, 2025
- Merge branch 'master' into WIP-data-transition
  soininen
  committed
  567e095
Commits on Nov 18, 2025
- models.py: refactor model_validators
  suvayu
  committed
  1a9ea66
- models.py: support conversions from pyarrow
  suvayu
  committed
  0bdb23b
- models.py: fix typo
  suvayu
  committed
  eb148aa
- arrow_value.py: type hints & consolidate error handling for metadata
  suvayu
  committed
  e822879
- parameter_value.py: refactor conversion from pyarrow to use models
  suvayu
  committed
  922762b
- {parameter_value,incomplete_values}.py: update JSON serialisation
  suvayu
  committed
  a189ab5
- tests: fix parameter_value tests after refactor
  suvayu
  committed
  1d801b0
Commits on Nov 19, 2025
- models.py: handle default type for empty arrays cleanly
  suvayu
  committed
  8049d96
- models.py: fix array - index array logic
  suvayu
  committed
  f54cec6
- models.py: cleaner from_* function api
  suvayu
  committed
  965aa7e
- models.py: better error msg
  suvayu
  committed
  6f17e75
- models.py: add utilities for any_array checking & schema generation
  suvayu
  committed
  3611b0b
- compat/converters.py: raise error on bad duration string input
  suvayu
  committed
  a27b45f
- compat/converters.py: complete duration intermediate representation
  suvayu
  committed
  6e18e1d
- test_models.py: add tests
  suvayu
  committed
  6951277
- test_compat_converters.py: add tests
  suvayu
  committed
  fd8efaa
- spinedb_api/compat/data_transition.py: remove unused module
  suvayu
  committed
  2e964b3
- models.py: complete type_map for null, reorder for legibility
  suvayu
  committed
  79ec818
- compat/encode.py: hormanise encoders w/ other utilities in models.py
  suvayu
  committed
  3ad53f6
- test_compat_encode.py: add tests
  suvayu
  committed
  5cd944d
Commits on Sep 30, 2026
- Merge branch 'master' into WIP-data-transition
  soininen
  committed
  68fbc7b
Commits on Oct 2, 2026
- Add back support for importing values as JSON dictionaries
  soininen
  committed
  b243496

Commits

Commits on May 21, 2025

Commits on May 23, 2025

Commits on May 26, 2025

Commits on May 29, 2025

Commits on Jun 2, 2025

Commits on Jun 3, 2025

Commits on Jun 5, 2025

Commits on Jun 6, 2025

Commits on Jun 7, 2025

Commits on Jun 8, 2025

Commits on Jun 13, 2025

Commits on Jun 14, 2025

Commits on Jun 27, 2025

Commits on Jul 9, 2025

Commits on Jul 11, 2025

Commits on Jul 23, 2025

Commits on Jul 24, 2025

Commits on Jul 25, 2025

Commits on Aug 21, 2025

Commits on Aug 29, 2025

Commits on Aug 30, 2025

Commits on Aug 31, 2025

Commits on Sep 1, 2025

Commits on Sep 2, 2025

Commits on Sep 3, 2025

Commits on Sep 8, 2025

Commits on Sep 11, 2025

Commits on Sep 14, 2025

Commits on Sep 16, 2025

Commits on Sep 17, 2025

Commits on Sep 25, 2025

Commits on Oct 10, 2025

Commits on Oct 13, 2025

Commits on Oct 14, 2025

Commits on Oct 15, 2025

Commits on Nov 4, 2025

Commits on Nov 18, 2025

Commits on Nov 19, 2025