feat(incremental): opt PgSQL into two-mode incremental fetching [CFTL-771] - #172
Open
matyas-jirat-keboola wants to merge 7 commits into
Open
feat(incremental): opt PgSQL into two-mode incremental fetching [CFTL-771]#172matyas-jirat-keboola wants to merge 7 commits into
matyas-jirat-keboola wants to merge 7 commits into
Conversation
…branches Wires vcs repositories + aliased dev-branch requires for keboola/db-extractor-config, keboola/db-extractor-adapter and keboola/db-extractor-common so pgsql can consume the unreleased incremental-fetching WINDOW feature from all three shared libs before they are tagged. MUST be reverted back to the ^-range requires once those repos cut real releases.
Extracts the GenericStorage basetype detection out of validateIncrementalFetching() into a new getIncrementalFetchingColumnType(ExportConfig): ?string, reused by validateIncrementalFetching() for the existing INCREMENTAL_TYPES check (unchanged void signature/behaviour). This overrides the default-null hook in db-extractor-common's BaseExtractor, so PgSQL is the first extractor to support the window feature (Start/End bounds on incremental fetching).
Datadir test against a live Postgres table with distinct timestamps (no ties, so row order is deterministic across the PG version test matrix). One row's timestamp is BELOW the stored watermark in source/data/in/state.json; without a window the old "col >= lastFetchedRow" predicate would permanently skip it. With incrementalFetchingStart set, the watermark is ignored and the row appears in the output - the CFTL-771 late-commit fix.
…proof Refresh the TEMP dev-branch deps (config/adapter/common) to the two-mode build: explicit incrementalFetchingMode (watermark default | window), with an optional watermark lookback (col >= watermark - N). - window late-commit test: pin incrementalFetchingMode=window (the mode now gates whether the window keys are read). - new lookback late-commit test: default watermark mode + a 10-minute lookback re-fetches a row committed below the watermark but inside the margin, while correctly excluding a row older than the margin. Both incremental late-commit datadir tests pass against live Postgres. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…f3d7)
Refresh the config dev-dep so the watermark lookback always subtracts the
magnitude ("20 minutes ago" no longer flips to +20min) and preserves the
timestamptz offset in the emitted lower bound.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… test - README: new "Incremental Fetching" section documenting incrementalFetchingMode (watermark default | window), incrementalFetchingLookback, and incrementalFetchingStart/End with types, semantics, and examples. - push.yml: add a tests-in-kbc config (cf-tests-prod) that exercises numeric window + lookback on public.users, so the feature is smoke-tested against the freshly-built image in CI (mirrors the oracle/mssql in-platform tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
apt-key is removed in Debian 13; switch the PGDG key to a signed-by keyring. Off EoL bookworm per review. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the CFTL-771 two-mode incremental fetching rollout — this is the reference driver. Full plan:
docs/superpowers/plans/2026-08-13-incremental-fetching-rollout.md.What
Opts
keboola.ex-db-pgsqlinto the two-mode incremental fetching feature:PgSQL::getIncrementalFetchingColumnType()override — exposes theGenericStoragebasetype (reused byvalidateIncrementalFetching()), enabling the shared window + watermark-lookback logic.incremental-fetching-window-late-commit(window mode) andincremental-fetching-lookback-late-commit(default watermark + lookback) — each proves a row committed below the watermark but inside the window/lookback is re-fetched (and, for lookback, that a row older than the margin is correctly excluded).Modes (what the config exposes)
Review / merge notes (Option B batch)
#cftl-771); re-pinned to released^1.17/^17.3before merge.composer installuses pinned refs): build + full PG 9–13 datadir matrix +tests-in-kbc(test project) all pass.🤖 Generated with Claude Code