diff --git a/Structure.md b/Structure.md index 916c0dc2..2470f139 100644 --- a/Structure.md +++ b/Structure.md @@ -46,6 +46,10 @@ One page per task. Each page is self-contained: the reader follows it from top to bottom and completes the task without consulting other pages. Include prerequisites at the top. +When one task has several alternative recipes, put them in a folder with an +`_index.md` that compares them, for example `importing_from_postgresql/` +with `pg_dump.md`, `minimal_downtime.md`, and `aws_dms.md`. + Add a page when a task involves multiple steps, requires specific tooling, or combines multiple features in a non-obvious way. Simple single-feature usage belongs in the reference section. @@ -62,7 +66,7 @@ to reference pages for feature details. ### clients/ -Organized into three subsections: +Organized into four subsections: - **Language folders** (`python/`, `javascript/`, `java/`, etc.) — one folder per programming language. The folder's `_index.md` covers the primary driver or @@ -71,6 +75,10 @@ Organized into three subsections: - **`apis/`** — APIs like ADBC for language-agnostic Arrow database connectivity. - **`tools/`** — GUI clients, CLI tools, and observability integrations that are not tied to a specific programming language (DBeaver, DataGrip, psql, Grafana). +- **`data_import/`** — data integration and change data capture (CDC) tools that + load data into CedarDB or keep it in sync with another database (Airbyte, Debezium). + Multi-tool, task-oriented procedures (e.g. migrating from PostgreSQL) stay in + `cookbook/` and link to these pages. Each page covers installation, connection setup, and a minimal working example. Focus on what is specific to CedarDB. Do not rewrite the client library's own @@ -118,6 +126,10 @@ clients/ │ └── _index.md ├── apis │ └── adbc.md # ADBC api +├── data_import/ # Data integration and CDC tools +│ ├── _index.md +│ ├── airbyte.md +│ └── debezium.md └── tools/ # Not language-specific ├── psql.md ├── dbeaver.md @@ -292,13 +304,16 @@ jump directly to it. 11. **Is it about a GUI client, CLI tool, or BI/observability integration?** Go to `clients/tools/.md` -12. **Is it an optimization guideline or recommended pattern?** +12. **Is it about a data integration or CDC tool (Airbyte, Debezium, ...)?** + Go to `clients/data_import/.md` + +13. **Is it an optimization guideline or recommended pattern?** Go to `best_practices/.md` -13. **Is it a compatibility status update?** +14. **Is it a compatibility status update?** Go to `compatibility/.md` -14. **None of the above?** +15. **None of the above?** Flag for team discussion. Do not create new top-level sections without agreement. --- @@ -348,14 +363,14 @@ to any handwritten member list in an `_index.md`. ## File Naming - Lowercase, snake_case when more than one word is needed (`asof_join.md`, - `importing_from_postgresql.md`). Single-word names have no separator (`tables.md`). + `working_with_csv.md`). Single-word names have no separator (`tables.md`). - Object pages by object name: `tables.md`, `indexes.md`, `roles.md` - Data types by common name: `integer.md`, `json.md`, `text.md` - Function categories by domain: `aggregation.md`, `timestamp.md`, `json.md` -- Cookbooks with descriptive slugs: `importing_from_postgresql.md`, `working_with_csv.md` +- Cookbooks with descriptive slugs: `working_with_csv.md`, `importing_from_postgresql/` - Language client folders by language name: `python/`, `javascript/`, `java/` - ORM pages within the language folder: `javascript/drizzle.md`, `javascript/prisma.md` -- Tool pages by tool name: `tools/grafana.md`, `tools/dbeaver.md` +- Tool pages by tool name: `tools/grafana.md`, `tools/dbeaver.md`, `data_import/debezium.md` --- diff --git a/content/clients/_index.md b/content/clients/_index.md index 8fe8423d..7e5e3b4a 100644 --- a/content/clients/_index.md +++ b/content/clients/_index.md @@ -20,6 +20,11 @@ Connectivity APIs: * [ADBC](apis/adbc) +Data import and CDC: + +* [Airbyte](data_import/airbyte) +* [Debezium](data_import/debezium) + Tools: * [DataGrip](tools/datagrip) diff --git a/content/clients/data_import/_index.md b/content/clients/data_import/_index.md new file mode 100644 index 00000000..d96539d7 --- /dev/null +++ b/content/clients/data_import/_index.md @@ -0,0 +1,13 @@ +--- +title: "Data Import and CDC" +linkTitle: "Data Import and CDC" +--- + +Data integration and change data capture (CDC) tools that load data into CedarDB or keep it in sync with another +database. +CedarDB works with these tools through their PostgreSQL connectors, without a CedarDB-specific plugin. +For a step-by-step migration from PostgreSQL, including methods that need no extra tools, +see [Migrating from PostgreSQL](/docs/cookbook/importing_from_postgresql/). + +* [Airbyte](./airbyte) — copy tables once +* [Debezium](./debezium) — replicate PostgreSQL changes continuously through Kafka, or copy a snapshot once diff --git a/content/clients/data_import/airbyte.md b/content/clients/data_import/airbyte.md new file mode 100644 index 00000000..af633361 --- /dev/null +++ b/content/clients/data_import/airbyte.md @@ -0,0 +1,176 @@ +--- +title: "Airbyte" +linkTitle: "Airbyte" +--- + +[Airbyte](https://airbyte.com/) moves data from hundreds of sources into a database with a few settings in its UI. +CedarDB works as an Airbyte destination through Airbyte's **Postgres** destination connector, so you can load data into +CedarDB from any source Airbyte supports. +This page uses a PostgreSQL database as the source and copies its tables with a full refresh. +Incremental syncs with **Append + Deduped** are not supported yet. +To replicate changes from PostgreSQL continuously, use [Debezium](../debezium/). + +This page was tested with the Airbyte connectors `source-postgres` 3.8.7 and `destination-postgres` 3.0.23, and +PostgreSQL 16. +The setting names match the Airbyte UI of self-managed Airbyte and Airbyte Cloud. + +This page is a self-contained example that you can follow from start to finish. +It copies the tables `species` and `plants` from the PostgreSQL database `nursery` into the CedarDB database `nursery`. +It keeps the Airbyte setup short and points out what is specific to CedarDB. +For all connector settings and for running Airbyte in production, see Airbyte's documentation of +the [Postgres source](https://docs.airbyte.com/integrations/sources/postgres), +the [Postgres destination](https://docs.airbyte.com/integrations/destinations/postgres), [sync modes](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes), +and [deploying Airbyte](https://docs.airbyte.com/platform/deploying-airbyte). + +## How CedarDB tables look + +Airbyte creates one table per source table (stream) in CedarDB. +These tables are landing tables, not a copy of the source schema: + +* Each table has four extra columns: `_airbyte_raw_id`, `_airbyte_extracted_at`, `_airbyte_meta`, and + `_airbyte_generation_id`. +* All source columns are nullable. + Primary keys, foreign keys, unique constraints, defaults, and indexes are not copied. +* Types are normalized: integers become `bigint`, decimals and floating-point numbers become `numeric`, `uuid` and most + other types become `varchar`, and arrays become `jsonb`. + +The Postgres destination creates these tables the same way in PostgreSQL; this is not specific to CedarDB. + +If you need the exact source schema, for example to migrate an application, use Airbyte to load the data and then copy +it into tables you define yourself, or follow [Migrating from PostgreSQL](/docs/cookbook/importing_from_postgresql/). + +## Prerequisites + +* Airbyte (self-managed or Airbyte Cloud) that can reach both PostgreSQL and CedarDB over the network. +* A CedarDB superuser to create the destination role and database. + +## Set up CedarDB as a destination + +{{% steps %}} + +### Create a role and database + +Connect to CedarDB as a superuser and create a role for Airbyte that owns the target database: + +```sql +CREATE ROLE airbyte WITH LOGIN PASSWORD ''; +CREATE DATABASE nursery OWNER airbyte; +``` + +As the database owner, Airbyte can create its schemas and tables without further privileges. +This works in every CedarDB edition. +Granting privileges with `GRANT` instead requires a CedarDB [enterprise license](/docs/licensing/). + +### Add the destination in Airbyte + +In Airbyte, add a new destination of type **Postgres** with these settings: + +| Setting | Value | +|---------------------------|-----------------------------------------------------------------------------------------------------------| +| Host, Port | Host name and port of CedarDB | +| Database Name | `nursery` | +| Default Schema | `public` | +| Username, Password | `airbyte` and its password | +| SSL Connection, SSL Modes | Enable SSL and use `require` (see [SSL configuration](/docs/references/configuration/#ssl-configuration)) | + +Leave the other settings at their defaults. +When you save the destination, Airbyte tests it by creating, writing, and dropping a table in CedarDB. + +{{% /steps %}} + +Airbyte keeps the source schema name: tables from the PostgreSQL schema `public` land in the CedarDB schema `public`. +To land them somewhere else, change the **Destination Namespace** of the connection. + +## Copy tables once + +A **Full refresh | Overwrite** sync copies all rows of the selected tables and replaces the CedarDB tables on every run. + +{{% steps %}} + +### Prepare PostgreSQL + +In the source PostgreSQL database, create a role that can read the tables: + +```sql +CREATE ROLE airbyte_source WITH LOGIN PASSWORD ''; +GRANT USAGE ON SCHEMA public TO airbyte_source; +GRANT SELECT ON ALL TABLES IN SCHEMA public TO airbyte_source; +``` + +### Add the source in Airbyte + +Add a new source of type **Postgres** with the host, port, database name, `airbyte_source` credentials, and the schema +`public`. +The source requires an **Update Method**. +Select **Scan Changes with User Defined Cursor**, which needs no replication setup in PostgreSQL. +Full refresh syncs read all rows and ignore it. + +### Create the connection + +Create a connection from the source to the CedarDB destination, select the tables, and set the sync mode of each table +to **Full refresh | Overwrite**. +Click **Sync now**. +On the connection's **Status** tab, Airbyte shows the records loaded per table while the sync runs. +The **Timeline** tab lists each sync with its duration and logs. + +### Verify the copy + +Compare row counts and sums on both sides, for example: + +```sql +SELECT count(*), sum(price), min(plant_id), max(plant_id) +FROM plants; +``` + +{{% /steps %}} + +Each further run replaces the CedarDB tables with the current source rows, including deletions. +To migrate an application, stop writes to PostgreSQL, run a final sync, and verify the tables before you switch over. + +## CedarDB limitations + +### Decimals with more than six fractional digits + +CedarDB columns such as `numeric(38,18)` store more than six fractional digits, but Airbyte does not create such +columns. +It creates all decimal and floating-point source columns as `numeric` without precision and scale, which CedarDB +creates as `numeric(38,6)` (see [Numeric](/docs/references/datatypes/numeric/)). +A value with nonzero digits after the sixth decimal place, for example the `double precision` value `3.14159265`, makes +the sync of the table fail with `invalid number format: loss of precision`, and no rows of the table arrive. +Very small values, such as `1e-9`, are stored as `0` without an error. + +To copy such columns exactly, create their tables in CedarDB before the first sync. +Give each decimal and floating-point column a `numeric(p,s)` type with enough fractional digits: + +```sql +CREATE TABLE public.soil_samples ( + _airbyte_raw_id varchar NOT NULL, + _airbyte_extracted_at timestamptz NOT NULL, + _airbyte_meta jsonb NOT NULL, + _airbyte_generation_id bigint, + sample_id bigint, + ph numeric(38,18), + trace_ppm numeric(38,18) +); +``` + +Airbyte keeps existing `numeric` columns with their precision and scale. +It changes columns of other types, such as `double precision`, to `numeric(38,6)`, so convert such columns after the +copy if you need them. + +### Other CedarDB limitations + +* **Append + Deduped** is not supported. + The sync fails because CedarDB currently only supports a single data-modifying CTE per query, whereas Airbyte uses 3. + +## Airbyte behavior + +The following behavior is the same with a PostgreSQL destination: + +* A **Full refresh | Overwrite** sync drops and recreates the destination tables, so it fails while a view depends on + one of them. + Drop such views before the sync and recreate them afterward, or enable **Drop tables with CASCADE** in the + destination, which drops the views. +* Empty strings arrive as `NULL`. +* `infinity` dates and timestamps, `NaN` and infinite floating-point values, and multidimensional arrays arrive as + `NULL`, with a note in `_airbyte_meta`. diff --git a/content/clients/data_import/debezium.md b/content/clients/data_import/debezium.md new file mode 100644 index 00000000..db8d2b71 --- /dev/null +++ b/content/clients/data_import/debezium.md @@ -0,0 +1,451 @@ +--- +title: "Debezium" +linkTitle: "Debezium" +aliases: + - /cookbook/aurora_debezium/ +--- + +[Debezium](https://debezium.io/) captures row changes from PostgreSQL and streams them through Kafka. +With the Debezium JDBC sink connector, CedarDB keeps a continuously updated copy of your PostgreSQL tables, so you can +run analytical queries on fresh data without loading your transactional database. +The same pipeline can also copy a consistent snapshot once and stop. + +CedarDB does not need its own connector. +The Debezium JDBC sink connects to CedarDB with the PostgreSQL JDBC driver. + +This page was tested with Debezium 3.7.0.Final, Kafka 4.3.1, and PostgreSQL 16. + +This page is a self-contained example that you can follow from start to finish. +It keeps the Kafka and Debezium setup short and points out what is specific to CedarDB. +For all connector settings and for running Debezium in production, see Debezium's documentation of +the [PostgreSQL connector](https://debezium.io/documentation/reference/stable/connectors/postgresql.html) and +the [JDBC sink connector](https://debezium.io/documentation/reference/stable/connectors/jdbc.html). + +## How the pipeline works + +```text +PostgreSQL ──> Debezium PostgreSQL connector ──> Kafka ──> Debezium JDBC sink ──> CedarDB +``` + +* The **PostgreSQL connector** reads a replication slot and writes one Kafka topic per table. + On first start, it copies the existing rows (snapshot), then streams every committed `INSERT`, `UPDATE`, `DELETE`, and + `TRUNCATE`. +* The **JDBC sink** creates the target tables in CedarDB and applies the changes as upserts and deletes by primary key. + +Both connectors run in Kafka Connect. +The pipeline works with any PostgreSQL that supports logical replication, including Amazon RDS, Aurora, Google Cloud +SQL, and Azure Database for PostgreSQL, and with self-hosted CedarDB or CedarDB Cloud as the target. + +## Prerequisites + +* A PostgreSQL database where you can enable logical replication and create a role. +* A CedarDB superuser to create the target role and schema. +* Docker with Docker Compose to run Kafka and Kafka Connect, or an existing Kafka Connect cluster with the Debezium 3.7 + connectors. +* Network access from Kafka Connect to PostgreSQL and CedarDB. + +## Set up continuous replication + +The example replicates the tables `species` and `plants` from the PostgreSQL database `nursery` into the schema +`nursery_replica` of the CedarDB database `nursery`. +Replace host names and passwords in angle brackets with your own values, and the ports if your servers do not use 5432. + +{{% steps %}} + +### Enable logical replication in PostgreSQL + +Set these server parameters and restart PostgreSQL: + +```ini +wal_level = logical +max_replication_slots = 10 +max_wal_senders = 10 +``` + +Each pipeline uses one replication slot and one WAL sender. + +{{< callout type="info" >}} + +On managed PostgreSQL services, you change these settings through the provider instead of `postgresql.conf`. + +* **Amazon RDS and Aurora:** Set `rds.logical_replication = 1` in the (cluster) parameter group, reboot, and run + `GRANT rds_replication TO debezium;` instead of the `REPLICATION` attribute below. +* **Google Cloud SQL:** Set the database flag `cloudsql.logical_decoding = on`. +* **Azure Database for PostgreSQL:** Set the server parameter `wal_level = logical`. + +See +also [PostgreSQL in the cloud](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-in-the-cloud) +in the Debezium documentation. + +{{< /callout >}} + +### Create a PostgreSQL role and publication + +Connect to the source database as a superuser (on managed services, the administrative user the provider gives you) and +run: + +```sql +CREATE ROLE debezium WITH LOGIN REPLICATION PASSWORD ''; +GRANT CONNECT ON DATABASE nursery TO debezium; +GRANT USAGE ON SCHEMA public TO debezium; +GRANT SELECT ON species, plants TO debezium; + +CREATE PUBLICATION cedardb_publication FOR TABLE species, plants; +``` + +The publication lists the replicated tables. +Creating it yourself means Debezium does not need ownership of the tables. + +### Prepare CedarDB + +Connect to CedarDB as a superuser and create a role for the sink and a schema it owns: + +```sql +CREATE ROLE debezium_sink WITH LOGIN PASSWORD ''; +CREATE DATABASE nursery; +\c nursery +CREATE SCHEMA nursery_replica AUTHORIZATION debezium_sink; +``` + +As the schema owner, the sink can create, alter, and write the replicated tables without further privileges. +This works in every CedarDB edition. +Granting privileges with `GRANT` instead requires a CedarDB [enterprise license](/docs/licensing/). + +### Start Kafka and Kafka Connect + +Save this file as `compose.yaml`. +It runs a single Kafka broker in KRaft mode and a Kafka Connect worker with the Debezium connectors installed: + +```yaml +services: + kafka: + image: quay.io/debezium/kafka:3.7 + environment: + CLUSTER_ID: 5Yr1SIgYQz-b-dgRabWx4g + NODE_ID: "1" + KAFKA_PROCESS_ROLES: broker,controller + KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093 + KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092 + KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT + KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093 + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "1" + KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: "1" + KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: "1" + volumes: + - kafka-data:/kafka/data + + connect: + image: quay.io/debezium/connect:3.7.0.Final + depends_on: + - kafka + ports: + - "8083:8083" + environment: + BOOTSTRAP_SERVERS: kafka:9092 + GROUP_ID: cedardb-connect + CONFIG_STORAGE_TOPIC: cedardb_connect_configs + OFFSET_STORAGE_TOPIC: cedardb_connect_offsets + STATUS_STORAGE_TOPIC: cedardb_connect_statuses + CONFIG_STORAGE_REPLICATION_FACTOR: "1" + OFFSET_STORAGE_REPLICATION_FACTOR: "1" + STATUS_STORAGE_REPLICATION_FACTOR: "1" + +volumes: + kafka-data: +``` + +Start it and check that both connector plugins are available: + +```shell +docker compose up -d +curl -s http://localhost:8083/connector-plugins | grep -oE '"class":"[^"]*(PostgresConnector|JdbcSinkConnector)"' +``` + +```text +"class":"io.debezium.connector.jdbc.JdbcSinkConnector" +"class":"io.debezium.connector.postgresql.PostgresConnector" +``` + +This is a development setup. +For production, run several brokers, keep the Kafka data on persistent storage, replicate the internal Connect topics, +and do not expose the Kafka Connect REST port 8083, which has no authentication, to untrusted networks. +The deployment sections of the Debezium documentation for +the [PostgreSQL connector](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-deployment) +and the [JDBC sink connector](https://debezium.io/documentation/reference/stable/connectors/jdbc.html#jdbc-deployment) +cover production setups. + +### Register the CedarDB sink + +Save the sink configuration as `sink.json`: + +```json +{ + "name": "cedardb-sink", + "config": { + "connector.class": "io.debezium.connector.jdbc.JdbcSinkConnector", + "tasks.max": "1", + "topics": "nursery.public.species,nursery.public.plants", + "connection.url": "jdbc:postgresql://:5432/nursery?sslmode=require&reWriteBatchedInserts=true", + "connection.username": "debezium_sink", + "connection.password": "", + "insert.mode": "upsert", + "primary.key.mode": "record_key", + "delete.enabled": "true", + "truncate.enabled": "true", + "schema.evolution": "basic", + "quote.identifiers": "true", + "collection.name.format": "nursery_replica.${source.table}", + "use.reduction.buffer": "true", + "connection.restart.on.errors": "true", + "flush.max.retries": "120", + "flush.retry.delay.ms": "1000" + } +} +``` + +The most important settings: + +| Setting | Purpose | +|---------------------------------------------|------------------------------------------------------------------------------------------------| +| `topics` | One topic per table, named `..`. | +| `insert.mode`, `primary.key.mode` | Apply each change as an upsert by primary key. Replaying a change after a restart is harmless. | +| `delete.enabled`, `truncate.enabled` | Apply source deletes and truncates. | +| `schema.evolution` | Create missing tables and add new columns. See [Schema changes](#schema-changes). | +| `collection.name.format` | Target table name. Here, the source table name in schema `nursery_replica`. | +| `use.reduction.buffer` | Collapse several changes to the same row in one batch into the final state. | +| `flush.max.retries`, `flush.retry.delay.ms` | Keep retrying for about two minutes when CedarDB is unreachable. The default is five retries. | + +Adjust `sslmode` in both configs to your servers +(see [SSL configuration](/docs/references/configuration/#ssl-configuration)). +Register the sink before the source, so it is ready when the first topics appear: + +```shell +curl -fsS -X POST -H 'Content-Type: application/json' \ + http://localhost:8083/connectors --data @sink.json +``` + +### Register the PostgreSQL source + +Save the source configuration as `source.json`: + +```json +{ + "name": "postgres-source", + "config": { + "connector.class": "io.debezium.connector.postgresql.PostgresConnector", + "tasks.max": "1", + "database.hostname": "", + "database.port": "5432", + "database.user": "debezium", + "database.password": "", + "database.dbname": "nursery", + "database.sslmode": "require", + "topic.prefix": "nursery", + "plugin.name": "pgoutput", + "slot.name": "cedardb_slot", + "publication.name": "cedardb_publication", + "publication.autocreate.mode": "disabled", + "table.include.list": "public.species,public.plants", + "snapshot.mode": "initial", + "decimal.handling.mode": "precise", + "skipped.operations": "none", + "heartbeat.interval.ms": "10000" + } +} +``` + +`skipped.operations` set to `none` also emits `TRUNCATE` events, which Debezium skips by default. +The heartbeat lets PostgreSQL release WAL even when the replicated tables do not change. + +Register the source: + +```shell +curl -fsS -X POST -H 'Content-Type: application/json' \ + http://localhost:8083/connectors --data @source.json +``` + +The connector creates the replication slot, copies all existing rows, and then streams changes. + +### Verify the replication + +Check that both connectors **and** their tasks are `RUNNING`. +A connector can be `RUNNING` while its task has `FAILED`: + +```shell +curl -s http://localhost:8083/connectors/postgres-source/status +curl -s http://localhost:8083/connectors/cedardb-sink/status +``` + +Then compare the tables. +Run the same query on PostgreSQL and on CedarDB (with `SET search_path = nursery_replica;` first) and check that the +results are identical: + +```sql +SELECT count(*), sum(price), min(plant_id), max(plant_id) +FROM plants; +``` + +Changes on PostgreSQL usually appear in CedarDB within a few seconds. + +{{% /steps %}} + +The sink creates each target table with the source columns and primary key. +It does not copy secondary indexes, unique constraints, foreign keys, check constraints, or identity properties, and +`varchar(n)` columns become `text`. +If your analytical queries need indexes, create them yourself. + +## Copy a snapshot once + +To copy the current state of the tables once, for example +to [migrate from PostgreSQL](/docs/cookbook/importing_from_postgresql/), use `snapshot.mode` `initial_only`. +The connector copies all rows from a consistent snapshot, drops its replication slot, and does not stream changes +afterward. + +Follow the steps above, but set `"snapshot.mode": "initial_only"` in `source.json`. +Adjust `table.include.list` if you want to copy only some of the published tables. + +The import has finished when the Kafka Connect log contains `Streaming is disabled for snapshot mode initial_only`, the +lag of the consumer group `connect-cedardb-sink` is zero (see [Monitor the pipeline](#monitor-the-pipeline)), and the +validation queries match. +The source task keeps the state `RUNNING` after the snapshot, so do not wait for it to stop. + +Keep these points in mind: + +* Restarting the same connector does not copy the data again, because Kafka Connect has stored the snapshot as + completed. + For another import, or to import next to a running replication pipeline, use a new connector name, topic prefix, slot, + and target schema. +* A second import into the same tables only upserts rows. + Rows deleted at the source in the meantime remain in CedarDB. +* Debezium creates a target table only when it receives a row for it. + Empty source tables do not appear in CedarDB. +* When you are done, delete both connectors and check that the slot `cedardb_slot` no longer exists in + `pg_replication_slots`. + +## Operate the pipeline + +### Monitor the pipeline + +Watch three things: + +* **Connector and task state.** Every task must be `RUNNING`. + Restart a failed task after fixing the cause: + + ```shell + curl -fsS -X POST http://localhost:8083/connectors/cedardb-sink/tasks/0/restart + ``` + +* **Consumer lag of the sink.** The lag is the number of changes in Kafka not yet written to CedarDB: + + ```shell + docker compose exec kafka /kafka/bin/kafka-consumer-groups.sh \ + --bootstrap-server kafka:9092 --describe --group connect-cedardb-sink + ``` + +* **Replication slot lag in PostgreSQL.** PostgreSQL keeps WAL until Debezium confirms it. + A stopped or deleted pipeline whose slot still exists makes the WAL grow without limit: + + ```sql + SELECT slot_name, active, + pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS lag + FROM pg_replication_slots; + ``` + +For the metrics that the connectors expose, +see [Monitoring](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-monitoring) in +the Debezium documentation. + +If CedarDB is unreachable for a short time, the sink retries and catches up afterward. +Changes are applied at least once and in order per table, so CedarDB converges to the source state. +Changes to different tables are applied independently, so a query joining several replicated tables can briefly see one +table ahead of another. + +### Remove the pipeline + +Delete the connectors, then drop the replication slot and the publication in PostgreSQL: + +```shell +curl -fsS -X DELETE http://localhost:8083/connectors/postgres-source +curl -fsS -X DELETE http://localhost:8083/connectors/cedardb-sink +``` + +```sql +SELECT pg_drop_replication_slot('cedardb_slot'); +DROP PUBLICATION cedardb_publication; +``` + +To remove Kafka, Kafka Connect, and all topics, run `docker compose down -v`. +The replicated tables in CedarDB remain until you drop them. + +### Schema changes + +PostgreSQL logical replication does not transmit DDL. +The sink only sees the columns of incoming rows and, with `schema.evolution` set to `basic`, it only creates tables and +adds columns +(see [schema evolution](https://debezium.io/documentation/reference/stable/connectors/jdbc.html#jdbc-schema-evolution) +in the Debezium documentation). +The following table shows the effect of source DDL on CedarDB: + +| Change in PostgreSQL | Effect in CedarDB | +|-------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `ADD COLUMN`, with or without `DEFAULT` or `NOT NULL` | The column is added with its default and nullability. Existing rows get the default. | +| `DROP COLUMN` | The column remains in CedarDB and keeps its old values. New rows get `NULL` in it, so a `NOT NULL` column without a default makes the sink task fail. Before you drop such a column in PostgreSQL, run `ALTER TABLE ... ALTER COLUMN ... DROP NOT NULL` on the CedarDB table, or run it afterward and restart the task. Drop the column in CedarDB after the sink has processed all older changes. | +| `RENAME COLUMN` | A new column with the new name is added and filled only for rows changed after the rename. The old column keeps its old values. Renaming a `NOT NULL` column without a default makes the sink task fail. | +| Change of a column type | The CedarDB column keeps its type. | +| New table | Not replicated until you add it to the publication, `table.include.list`, and `topics`. Existing rows of the new table are copied only with a Debezium [incremental snapshot](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-incremental-snapshots). | + +For drops, renames, and type changes, the safest approach is to rebuild the table: set up a second pipeline with new +connector names, topic prefix, slot, and target schema, as +in [Set up continuous replication](#set-up-continuous-replication), wait until it has caught up, and switch your queries +over. + +{{< callout type="warning" >}} +CedarDB cannot yet add a column to a table that a view depends on. +If you create views on replicated tables, the sink task fails when a source table gets a new column, including the new +column of a renamed one. +To recover, drop the view, restart the sink task, and create the view again. +{{< /callout >}} + +## Data types + +The sink maps most PostgreSQL types to the same CedarDB type: integers, `numeric(p,s)`, `real`, `double precision`, +`boolean`, `text`, `varchar` (as `text`), `date`, `timestamp`, `timestamptz`, `uuid`, `jsonb` (as `json`), `bytea`, and +one-dimensional arrays. + +### Decimals with more than six fractional digits + +CedarDB columns such as `numeric(38,18)` store more than six fractional digits, but the sink cannot write such values +into them yet. +The sink sends each decimal value as a parameter of type `numeric` without precision and scale, and CedarDB currently +reads such a parameter as `numeric(38,6)` (see [Numeric](/docs/references/datatypes/numeric/)) before it converts the +value to the column type. +A value with nonzero digits after the sixth decimal place therefore stops the sink with a `loss of precision` error, and +a value smaller than 0.000001 is stored as `0`. + +To replicate such columns exactly, send decimals as strings and create the target table yourself: + +1. In `source.json`, set `"decimal.handling.mode": "string"`. +2. In the sink's `connection.url`, add `&stringtype=unspecified`. +3. Create the target table in CedarDB before you register the connectors, with the same `numeric(p,s)` types as the + source: + + ```sql + CREATE TABLE nursery_replica.soil_samples ( + sample_id integer PRIMARY KEY, + ph numeric(30,10), + trace_ppm numeric(38,18) + ); + ``` + +`decimal.handling.mode` applies to all tables of the source connector. +Tables that the sink creates automatically then get `text` columns instead of `numeric`. + +### Unsupported types and values + +CedarDB does not accept the following source data: + +| Source data | Effect | What to do | +|--------------------------------------------------------------|-----------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `timetz`, `xml`, `point`, and other geometric types | The sink fails to create the table, because CedarDB does not support these types. | Exclude the columns with the source setting `column.exclude.list`, for example `"public.plants.pickup_window"`, and replicate a text copy maintained at the source if you need the values. | +| `infinity` and `-infinity` in `date` and `timestamp` columns | The sink stops, because CedarDB does not accept infinite dates and timestamps. | Replace infinite values with finite values or `NULL` at the source, or exclude the column. | diff --git a/content/compatibility/ecosystem_and_clients.md b/content/compatibility/ecosystem_and_clients.md index ff07ec87..3ddad1f2 100644 --- a/content/compatibility/ecosystem_and_clients.md +++ b/content/compatibility/ecosystem_and_clients.md @@ -23,11 +23,13 @@ range of PostgreSQL connectors for end-user tools like Grafana and programming l ### Applications -| **Application** | **Version** | **Support State** | **Details** | -|-----------------|-------------|-------------------|------------------------------------------------| -| DataGrip | 2024.2.2 | Partial | [Documentation](/docs/clients/tools/datagrip/) | -| DBeaver | 24.2.2 | Partial | [Documentation](/docs/clients/tools/dbeaver/) | -| Grafana | 10.4.2 | Partial | [Documentation](/docs/clients/tools/grafana/) | +| **Application** | **Version** | **Support State** | **Details** | +|-----------------|-------------------------------|-------------------|------------------------------------------------------| +| Airbyte | 3.0.23 (Postgres destination) | Partial | [Documentation](/docs/clients/data_import/airbyte/) | +| DataGrip | 2024.2.2 | Partial | [Documentation](/docs/clients/tools/datagrip/) | +| DBeaver | 24.2.2 | Partial | [Documentation](/docs/clients/tools/dbeaver/) | +| Debezium | 3.7.0.Final | Partial | [Documentation](/docs/clients/data_import/debezium/) | +| Grafana | 10.4.2 | Partial | [Documentation](/docs/clients/tools/grafana/) | ### Programming Language Libraries diff --git a/content/cookbook/_index.md b/content/cookbook/_index.md index 2e7bf4a6..407da12b 100644 --- a/content/cookbook/_index.md +++ b/content/cookbook/_index.md @@ -9,9 +9,12 @@ step-by-step instructions on how to accomplish them using CedarDB. * [Import data from JSON](importing_from_json) * [Import data from PostgreSQL](importing_from_postgresql) + * [Copy with pg_dump](importing_from_postgresql/pg_dump) + * [Migrate with minimal downtime](importing_from_postgresql/minimal_downtime) + * [Replicate with AWS DMS](importing_from_postgresql/aws_dms) * [Importing and working with CSV data](working_with_csv) -* [Replicating data from Amazon Aurora PostgreSQL via Debezium](aurora_debezium) -* [Use AWS DMS to replicate data from RDS PostgreSQL to CedarDB](aws_dms) + +To replicate data continuously with Debezium or load data with Airbyte, see [Data Import and CDC](/docs/clients/data_import/). In addition, you can also find some recipes for performance evaluation: diff --git a/content/cookbook/aurora_debezium.md b/content/cookbook/aurora_debezium.md deleted file mode 100644 index c77bdffa..00000000 --- a/content/cookbook/aurora_debezium.md +++ /dev/null @@ -1,376 +0,0 @@ ---- -title: "Replicating from Aurora for PostgreSQL via Debezium" -linkTitle: "Replicating via Debezium" -weight: 102 ---- - -[Debezium](https://debezium.io/) is a popular platform for [Change Data Capture (CDC)](https://en.wikipedia.org/wiki/Change_data_capture). - -This guide shows you how you can replicate tables from your transactional Amazon Aurora PostgreSQL to CedarDB, allowing you to do fast analytics on data is it comes in without impacting your existing data infrastructure. - -## Setting up Replication - -{{% steps %}} - -### Starting an EC2 Instance - -CedarDB and Debezium will live inside this instance. - -{{< callout type="info" >}} -If you do not already know your requirements, we recommend using the `m6id.2xlarge` instance type which comes with 32 GiB of Memory, 8 vCPUs and 500 GiB of fast ephemeral SSD. If you want to spend less, going for something with 4 vCPUs is also fine. -{{< /callout >}} - -The rest of this instruction manual assumes you use Ubuntu 24.04 as your operating system. Since CedarDB runs inside its own docker image, you can choose any other OS as well but you might have to adapt the installation instructions accordingly. - -{{< callout type="info" >}} -Configure the EBS volume where your root partition is mounted to be large enough to hold all the data Debezium needs to store its CDC events. -By default, it retains all events for one week and there will be one message per insert/update/delete of all replicated tables. -For playing around, the default of 8 GiB is fine. -{{< /callout >}} - -### Setting up your EC2 Instance - -CedarDB loves fast SSDs. If your instance comes with an ephemeral SSD, mount it like this: - -```shell -sudo mkfs.ext4 -E nodiscard /dev/nvme1n1 -mkdir /home/ubuntu/db -sudo mount -o discard /dev/nvme1n1 /home/ubuntu/db -sudo chown ubuntu:ubuntu db -``` - -Next, we install docker: - -```shell -sudo apt update -sudo apt install ca-certificates curl -sudo install -m 0755 -d /etc/apt/keyrings -sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc -sudo chmod a+r /etc/apt/keyrings/docker.asc -echo \ - "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \ - $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ - sudo tee /etc/apt/sources.list.d/docker.list > /dev/null -sudo apt update - -sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -``` - -Before you can Docker commands, you need to add your user to the docker group and re-login: - -```shell -sudo adduser ubuntu docker -``` - -Finally, build the docker image using the CedarDB Dockerfile. - -```shell -docker build --tag cedardb . -``` - -### Starting an Amazon Aurora PostgreSQL Cluster - -If you already have a cluster, you can skip this step. - -{{< callout type="info" >}} -If you do not already know your requirements, we recommend using the `db.r6gd.xlarge` instance type which comes with 32 GiB of memory and 4 vCPUs. You can definitely go cheaper here, if you just want to play around a little bit. -{{< /callout >}} - -Make sure to connect it to your EC2 instance. -In this example, we will assume you have created an admin user `postgresuser` with password `postgrespw`. - -{{< callout type="warning" >}} -If you use such easily guessable credentials, make sure this cluster is only reachable from inside your VPC! -{{< /callout >}} - -{{< callout type="warning" >}} -If you intend to do more than just play around, you should not give the Postgres user used for replication root access! Instead, it's best practice to [create a separate user with replication privileges](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-permissions). -{{< /callout >}} - -### Configuring your Amazon Aurora PostgreSQL Cluster - -Amazon Aurora PostgreSQL needs to be configured for logical replication to Debezium. You can take a look at the [Debezium documentation](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-in-the-cloud) for details and instructions to check if your cluster is already set up correctly. - -If not, here are the steps to enable it: - -1. Create a new parameter group for your cluster - - Call it, e.g., `logicalreplication`, set the engine type to `Aurora PostgreSQL`, the family to your PostgreSQL version, e.g. `aurora-postgresql15` and the type to `DB Cluster Parameter Group`. - - Then, within that parameter group, change the parameter `rds.logical_replication` to `1`. - -2. Apply this group to your cluster -3. Restart your cluster (or wait for the next maintenance window) - -### Starting CedarDB and Debezium - -Create a file `docker-compose.yml` with the following content: - -```yml -services: - zookeeper: - image: quay.io/debezium/zookeeper:2.7 - ports: - - 2181:2181 - - 2888:2888 - - 3888:3888 - kafka: - image: quay.io/debezium/kafka:2.7 - ports: - - 9092:9092 - links: - - zookeeper - environment: - - ZOOKEEPER_CONNECT=zookeeper:2181 - - KAFKA_AUTO_CREATE_TOPICS_ENABLE=true - cedardb: - image: cedardb:latest - ports: - - 5433:5432 - volumes: - - type: bind - source: /home/ubuntu/db - target: /var/lib/cedardb/data - environment: - - CEDAR_USER=postgres - - CEDAR_PASSWORD=CorrectTreeBatteryStaple - - CEDAR_DB=postgres - connect: - image: quay.io/debezium/connect:2.7 - ports: - - 8083:8083 - - 1976:1976 - links: - - kafka - - cedardb - environment: - - BOOTSTRAP_SERVERS=kafka:9092 - - GROUP_ID=1 - - CONFIG_STORAGE_TOPIC=my_connect_configs - - OFFSET_STORAGE_TOPIC=my_connect_offsets - - STATUS_STORAGE_TOPIC=my_connect_statuses -``` - -Make sure that your docker container database is created somewhere on a fast SSD, i.e. modify the docker volumes of CedarDB in the docker-compose configuration accordingly. -If you followed this guide, `/home/ubuntu/db` should point to your fast ephemeral ssd (if your instance has one). - -Then, start all services with the following command: - -```shell -docker compose up -``` - -### Install psql to talk to CedarDB and Postgres - -```shell -sudo apt install posgresql-common postgresql-client-16 -``` - -### Creating a Source and Sink Configuration for Debezium - -We now need to connect Debezium to Amazon Aurora PostgreSQL via a `source` connector and to CedarDB via a `sink` connector. - -Create a file `source.json` with the following contents: - -```json -{ - "name": "postgres-source", - "config": { - "connector.class": "io.debezium.connector.postgresql.PostgresConnector", - "database.hostname": "[Your Amazon Aurora PostgreSQL hostname]", - "plugin.name": "pgoutput", - "database.port": "5432", - "database.user": "postgresuser", - "database.password": "postgrespw", - "database.dbname" : "postgres", - "topic.prefix": "postgres", - "heartbeat.inverval.ms" : "20", - "table.include.list": "public.lineitem", - "topic.creation.enable": "true", - "topic.creation.default.replication.factor": "-1", - "topic.creation.default.partitions": "-1", - "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState", - "transforms.unwrap.drop.tombstones": "false", - "delete.handling.mode": "none" - } -} -``` - -Create a file `sink.json` with the following contents: - -```json -{ - "name": "cedar-sink", - "config": { - "connector.class": "io.debezium.connector.jdbc.JdbcSinkConnector", - "topics.regex": "postgres.public.lineitem", - "connection.url": "jdbc:postgresql://cedardb:5432/postgres?stringtype=unspecified", - "connection.username": "postgres", - "connection.password": "CorrectTreeBatteryStaple", - "delete.handling.mode": "none", - "insert.mode": "upsert", - "schema.evolution": "basic", - "delete.enabled": "true", - "primary.key.mode": "record_key", - "primary.key.fields": "lineitem_id", - "table.name.format": "${source.table}" - } -} -``` - -This configuration assumes we want to replicate a table called `lineitem` with a primary key called `lineitem_id`. Modify both files to work with your Amazon Aurora PostgreSQL and CedarDB credentials. - -### Starting Source and Sink - -Execute the following commands to register the source and sink with Debezium and start them: - -```shell -curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" http://localhost:8083/connectors/ -d @sink.json - -curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" http://localhost:8083/connectors/ -d @source.json -``` - -It's possible that you need to restart all containers (`docker compose down` then `docker compose up`) for the replication to start after creating the connectors. - -{{< callout type="info" >}} -If you want to delete source and sink, you can use the following commands: - -```shell -curl -i -X DELETE -H "Accept:application/json" -H "Content-Type:application/json" http://localhost:8083/connectors/postgres-source - -curl -i -X DELETE -H "Accept:application/json" -H "Content-Type:application/json" http://localhost:8083/connectors/cedar-sink -``` - -{{< /callout >}} - -{{% /steps %}} - -## Testing the Replication - -Congratulations, you're now ready to go! -Let's test if everything is working correctly. - -### Creating a Table in Amazon Aurora for PostgreSQL - -Connect to your cluster (e.g., via `psql -h [...].rds.amazonaws.com -U postgresuser -p 5432 -d postgres` ) and paste the following: - -```sql -CREATE TABLE lineitem ( - lineitem_id BIGINT PRIMARY KEY, -- Unique identifier for each line item - transaction_id BIGINT NOT NULL, -- Links the line item to a specific transaction (e.g., order_id or invoice_id) - product_id BIGINT NOT NULL, -- ID of the product or service being transacted - quantity NUMERIC(10, 2) NOT NULL, -- Quantity of the product being ordered - unit_price NUMERIC(15, 4) NOT NULL, -- Price per unit of the product - discount NUMERIC(5, 2) DEFAULT 0.00, -- Discount applied to the line item, if any - tax_rate NUMERIC(5, 2) DEFAULT 0.00, -- Applicable tax rate - total_amount NUMERIC(20, 4) GENERATED ALWAYS AS (quantity * unit_price * (1 - discount / 100) * (1 + tax_rate / 100)) STORED, -- Calculated total amount for the line item - created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, -- Timestamp when the line item was created - transaction_date DATE NOT NULL, -- Date of the transaction - status VARCHAR(50) DEFAULT 'active', -- Status of the line item (e.g., 'active', 'void', 'pending') - notes TEXT -- Optional notes or comments about the line item -); - - -INSERT INTO lineitem (lineitem_id, transaction_id, product_id, quantity, unit_price, discount, tax_rate, transaction_date, status, notes -) VALUES - (1, 1001, 2001, 10.00, 50.00, 5.00, 10.00, '2024-10-15', 'active', 'First line item for this transaction'), - (2, 1001, 2002, 5.00, 100.00, 0.00, 15.00, '2024-10-15', 'active', 'Second line item with no discount'), - (3, 1002, 2003, 2.00, 250.00, 10.00, 12.00, '2024-10-16', 'pending', 'Third line item in a different transaction'), - (4, 1003, 2004, 7.00, 75.00, 3.00, 8.00, '2024-10-16', 'void', 'Voided line item for test purposes'); -``` - -### Checking Replication in CedarDB - -Now connect to CedarDB (e.g., via `PGPASSWORD=CorrectTreeBatteryStaple psql -h localhost -U postgres`) and check the replicated table: - -```sql -select * from lineitem; -``` - -## Automate - -Let's create some more rows! Create a file `inserter.py` with the following content: - -```python -import psycopg2 -from psycopg2 import sql -import random -import time - -# PostgreSQL connection parameters -conn = psycopg2.connect( - dbname="postgres", - user="postgresuser", - password="postgrespw", - host="[Your Amazon Aurora PostgreSQL hostname]", - port="5432" -) - -# Create a cursor object -cur = conn.cursor() - - -# Function to insert dummy lineitem records -def insert_dummy_lineitem(lineitem_id): - transaction_id = random.randint(1000, 1100) # Random transaction_id between 1000 and 1100 - product_id = random.randint(2000, 2100) # Random product_id between 2000 and 2100 - quantity = round(random.uniform(1, 20), 2) # Random quantity between 1 and 20 - unit_price = round(random.uniform(10, 200), 2) # Random unit price between 10 and 200 - discount = round(random.uniform(0, 10), 2) # Random discount between 0% and 10% - tax_rate = round(random.uniform(5, 15), 2) # Random tax rate between 5% and 15% - transaction_date = time.strftime('%Y-%m-%d') # Current date - status = random.choice(['active', 'pending', 'void']) # Random status - notes = f"Random note {random.randint(1, 100)}" # Random notes - - # Insert statement - insert_query = sql.SQL(""" - INSERT INTO lineitem ( - lineitem_id, transaction_id, product_id, quantity, unit_price, discount, tax_rate, transaction_date, status, notes - ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s) - """) - - # Execute insert statement - cur.execute(insert_query, ( - lineitem_id, transaction_id, product_id, quantity, unit_price, discount, tax_rate, transaction_date, status, notes - )) - - # Commit transaction - conn.commit() - -# Insert dummy data every few milliseconds -try: - lineitem_id = 5 - while True: - insert_dummy_lineitem(lineitem_id) - print("Inserted a new lineitem.") - lineitem_id = lineitem_id + 1 -except KeyboardInterrupt: - print("Insertion process stopped.") -finally: - # Close the cursor and connection - cur.close() - conn.close() -``` - -It requires psycopg2 which you can install via `sudo apt install python3-psycopg2`. - -Then run `python3 inserter.py`. - -## Running Analytical Queries - -Connect to CedarDB again and run your analytical queries. -Let's find the average tax rate for each product: - -```sql -SELECT - product_id, - AVG(tax_rate) AS avg_tax_rate -FROM - lineitem -GROUP BY - product_id -ORDER BY - avg_tax_rate DESC; -``` - -You can now run all your expensive analytical queries against CedarDB while keeping your PostgreSQL database system as system of record. diff --git a/content/cookbook/importing_from_postgresql.md b/content/cookbook/importing_from_postgresql.md deleted file mode 100644 index f8b48650..00000000 --- a/content/cookbook/importing_from_postgresql.md +++ /dev/null @@ -1,129 +0,0 @@ ---- -title: "Importing Data from PostgreSQL into CedarDB" -linkTitle: "Importing from PostgreSQL" -weight: 10 ---- -In this section, you will learn how to seamlessly transfer data from PostgreSQL to CedarDB. - -## Migration steps - -{{% steps %}} - -### Dump the schema from Postgres - -Make sure your Postgres instance is running. -Then use the Tool `pg_dump` (probably supplied together with PostgreSQL by your package manager): - -```shell -pg_dump --schema-only postgres > schema.sql -``` - -{{< callout type="info" >}} -`postgres` is the default database into which PostgreSQL inserts new tables. Replace with other database in above command if you want to export the tables of a different database. -{{< /callout >}} - -### Adapt the dumped schema - -CedarDB does not yet support some settings PostgreSQL tries to set. Remove them from the schema dump for now by running the following three `sed` commands: - -```shell -sed -i.bak 's/^SET.*$//g' schema.sql -sed -i.bak 's/.*set_config.*//g' schema.sql -sed -i.bak 's/^ALTER TABLE .* OWNER TO .*//g' schema.sql -``` - -### Remove unsupported data types - -CedarDB doesn't support some data types yet, especially no auto generated series. For now remove them, by manually editing schema.sql. - -For example, the statement - -```sql -create table x(id integer generated always as identity); -``` - -generates the following lines in the dump file: - -```sql -CREATE TABLE public.x ( - id integer NOT NULL -); - --- --- Name: x_id_seq; Type: SEQUENCE; Schema: public; Owner: postgres --- - -ALTER TABLE public.x ALTER COLUMN id ADD GENERATED ALWAYS AS IDENTITY ( - SEQUENCE NAME public.x_id_seq - START WITH 1 - INCREMENT BY 1 - NO MINVALUE - NO MAXVALUE - CACHE 1 -); -``` - -In this case, remove the alter table statement. - -### Dump the data out of PostgreSQL into CSV files - -Connect with PostgreSQL via `psql`. - -Execute the following statement against Postgres for each table you want to export to CedarDB - -```sql -\copy {tablename} TO 'your/path/{tablename}.csv' DELIMITER '|' CSV NULL ''; -``` - -{{< callout type="info" >}} -The `NULL` parameter specifies into which string null values should be serialized. If your data set contains empty strings, choose a different value. -{{< /callout >}} - -### Import data from your freshly dumped files into CedarDB - -Now make sure that CedarDB is running. Either let it run in parallel to Postgres and use a different port, or shut down Postgres and then start CedarDB. -For the following commands, we assume CedarDB listens at port 5432. If you're using another port, please change the commands accordingly (via the argument `-p {Portnumber}`). - -Import the schema you've exported from PostgreSQL and modified earlier - -```shell -psql -h localhost -U postgres < schema.sql -``` - -If you get some error messages in the server log, that's okay for now, as long as your tables are created (`psql` answers with `CREATE TABLE`). - -Connect with CedarDB via `psql`, e.g. via - -```shell -psql -h localhost -U postgres -``` - -Execute the following statement against CedarDB for each table you exported in the previous step - -```sql -copy {tablename} from 'your/path/{tablename}.csv' DELIMITER '|' CSV NULL ''; -``` - -Note that the path is relative to the **server**, i.e., if you run CedarDB inside the docker container, the location where the csv files resist must be mapped as docker volume. - -If you want the path to be relative to the **client**, precede the command with a backslash: - -```sql -\copy {tablename} from 'your/path/{tablename}.csv' DELIMITER '|' CSV NULL ''; -``` - -Note that this incurs some network overhead as the data is sent via the PostgreSQL wire protocol over the psql connection. - -The csv import is currently single-threaded, as CedarDB has to correctly handle newlines and escapes. If you are sure that your strings don't contain newlines **and** don't contain the delimiter, you can instead import in text mode which is multithreaded and thus **much** faster: - -```sql -copy {tablename} from 'your/path/{tablename}.csv' with(format text, delimiter '|', null ''); -``` - -{{< callout type="warning" >}} -Multithreaded import does not yet work when using a backslash in front of copy (i.e. when importing relative to the client). -{{< /callout >}} - -{{% /steps %}} - -Your data is now successfully imported into CedarDB! diff --git a/content/cookbook/importing_from_postgresql/_index.md b/content/cookbook/importing_from_postgresql/_index.md new file mode 100644 index 00000000..c7bff15a --- /dev/null +++ b/content/cookbook/importing_from_postgresql/_index.md @@ -0,0 +1,45 @@ +--- +title: "Migrating from PostgreSQL to CedarDB" +linkTitle: "Importing from PostgreSQL" +weight: 10 +--- + +CedarDB is highly PostgreSQL compatible and works with most PostgreSQL tools out of the box, so you can move a +PostgreSQL database to CedarDB with the tools you already know. +The recipes in this section copy a PostgreSQL database into CedarDB once, or keep copying changes until you switch your +application over. +They work for self-hosted PostgreSQL and managed services such as Amazon RDS, Aurora, Google Cloud SQL, and Azure +Database for PostgreSQL, and for self-hosted CedarDB and CedarDB Cloud alike. + +## Recipes + +| Recipe | Downtime | What arrives in CedarDB | Use it when | +|----------------------------------------------------|------------------------------|-----------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------| +| [Copy with pg_dump](pg_dump/) | For the duration of the copy | Complete schema with keys, indexes, defaults, identity columns, views, owners, and privileges | You want the simplest setup and can stop writes to PostgreSQL while the data is copied. Needs only the PostgreSQL client tools. | +| [Migrate with minimal downtime](minimal_downtime/) | During the switchover only | Complete schema, as with `pg_dump` | Your application must keep writing to PostgreSQL until a short switchover. Uses `pg_dump` for the schema and Debezium for the data. | +| [Replicate with AWS DMS](aws_dms/) | During the switchover only | Tables created by the DMS task | Your PostgreSQL runs on Amazon RDS or Aurora and you want a managed AWS service. | + +## What to expect + +* **Schema.** `pg_dump` restores tables, views, indexes, constraints, defaults, identity columns, owners, and privileges + as they are in PostgreSQL. +* **Roles and privileges.** The recipes copy roles with `pg_dumpall`, including their passwords, so applications keep + their credentials. + Privileges require a CedarDB [enterprise license](/docs/licensing/). With the community edition, the recipes keep the + owners and skip the privileges. +* **Data types.** Most PostgreSQL types arrive unchanged. + CedarDB does not support `timetz`, `xml`, the geometric types, stored generated columns, and `infinity` dates and + timestamps. The `pg_dump` and minimal-downtime recipes show how to find and convert them first. +* **Downtime.** A one-time copy needs a pause in writes for as long as the copy takes. + With change data capture (Debezium or AWS DMS), the pause shrinks to the switchover. +* **Validation.** The `pg_dump` and minimal-downtime recipes end with queries that compare row counts and checksums on + both sides, so you can check the copy before you switch. + +## Related tool documentation + +| Tool | Description | +|-------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Airbyte](/docs/clients/data_import/airbyte/) | Copy tables once with Airbyte's Postgres connectors, also from sources other than PostgreSQL. Creates its own tables with extra Airbyte columns, without keys or indexes. | +| [Debezium](/docs/clients/data_import/debezium/) | Keep CedarDB continuously in sync with PostgreSQL through Kafka, or copy a snapshot once. Creates tables with columns and primary keys. | +| [psql](/docs/clients/tools/psql/) | The PostgreSQL command-line client used in the recipes. | +| [Importing and working with CSV data](/docs/cookbook/working_with_csv/) | `COPY` options for CSV files. | diff --git a/content/cookbook/aws_dms.md b/content/cookbook/importing_from_postgresql/aws_dms.md similarity index 99% rename from content/cookbook/aws_dms.md rename to content/cookbook/importing_from_postgresql/aws_dms.md index 4d3d7731..7319cbc8 100644 --- a/content/cookbook/aws_dms.md +++ b/content/cookbook/importing_from_postgresql/aws_dms.md @@ -1,7 +1,8 @@ --- title: "Use AWS DMS to replicate data from RDS PostgreSQL to CedarDB" -linkTitle: "Replicating via AWS DMS" -weight: 102 +linkTitle: "Replicate with AWS DMS" +aliases: + - /cookbook/aws_dms/ --- ## Overview diff --git a/content/cookbook/img/cedardb_ec2_note_vpc_and_private_ip.jpg b/content/cookbook/importing_from_postgresql/img/cedardb_ec2_note_vpc_and_private_ip.jpg similarity index 100% rename from content/cookbook/img/cedardb_ec2_note_vpc_and_private_ip.jpg rename to content/cookbook/importing_from_postgresql/img/cedardb_ec2_note_vpc_and_private_ip.jpg diff --git a/content/cookbook/img/cloudformation_complete.jpg b/content/cookbook/importing_from_postgresql/img/cloudformation_complete.jpg similarity index 100% rename from content/cookbook/img/cloudformation_complete.jpg rename to content/cookbook/importing_from_postgresql/img/cloudformation_complete.jpg diff --git a/content/cookbook/img/dms_create_migration_task.jpg b/content/cookbook/importing_from_postgresql/img/dms_create_migration_task.jpg similarity index 100% rename from content/cookbook/img/dms_create_migration_task.jpg rename to content/cookbook/importing_from_postgresql/img/dms_create_migration_task.jpg diff --git a/content/cookbook/img/dms_create_migration_task_2.jpg b/content/cookbook/importing_from_postgresql/img/dms_create_migration_task_2.jpg similarity index 100% rename from content/cookbook/img/dms_create_migration_task_2.jpg rename to content/cookbook/importing_from_postgresql/img/dms_create_migration_task_2.jpg diff --git a/content/cookbook/img/dms_create_migration_task_3.jpg b/content/cookbook/importing_from_postgresql/img/dms_create_migration_task_3.jpg similarity index 100% rename from content/cookbook/img/dms_create_migration_task_3.jpg rename to content/cookbook/importing_from_postgresql/img/dms_create_migration_task_3.jpg diff --git a/content/cookbook/img/dms_create_migration_task_4.jpg b/content/cookbook/importing_from_postgresql/img/dms_create_migration_task_4.jpg similarity index 100% rename from content/cookbook/img/dms_create_migration_task_4.jpg rename to content/cookbook/importing_from_postgresql/img/dms_create_migration_task_4.jpg diff --git a/content/cookbook/img/dms_create_migration_task_5.jpg b/content/cookbook/importing_from_postgresql/img/dms_create_migration_task_5.jpg similarity index 100% rename from content/cookbook/img/dms_create_migration_task_5.jpg rename to content/cookbook/importing_from_postgresql/img/dms_create_migration_task_5.jpg diff --git a/content/cookbook/img/dms_create_replication_instance_1.jpg b/content/cookbook/importing_from_postgresql/img/dms_create_replication_instance_1.jpg similarity index 100% rename from content/cookbook/img/dms_create_replication_instance_1.jpg rename to content/cookbook/importing_from_postgresql/img/dms_create_replication_instance_1.jpg diff --git a/content/cookbook/img/dms_create_replication_instance_2.jpg b/content/cookbook/importing_from_postgresql/img/dms_create_replication_instance_2.jpg similarity index 100% rename from content/cookbook/img/dms_create_replication_instance_2.jpg rename to content/cookbook/importing_from_postgresql/img/dms_create_replication_instance_2.jpg diff --git a/content/cookbook/img/dms_endpoints_created.jpg b/content/cookbook/importing_from_postgresql/img/dms_endpoints_created.jpg similarity index 100% rename from content/cookbook/img/dms_endpoints_created.jpg rename to content/cookbook/importing_from_postgresql/img/dms_endpoints_created.jpg diff --git a/content/cookbook/img/dms_source_db_1.jpg b/content/cookbook/importing_from_postgresql/img/dms_source_db_1.jpg similarity index 100% rename from content/cookbook/img/dms_source_db_1.jpg rename to content/cookbook/importing_from_postgresql/img/dms_source_db_1.jpg diff --git a/content/cookbook/img/dms_source_db_2.jpg b/content/cookbook/importing_from_postgresql/img/dms_source_db_2.jpg similarity index 100% rename from content/cookbook/img/dms_source_db_2.jpg rename to content/cookbook/importing_from_postgresql/img/dms_source_db_2.jpg diff --git a/content/cookbook/img/dms_source_db_3.jpg b/content/cookbook/importing_from_postgresql/img/dms_source_db_3.jpg similarity index 100% rename from content/cookbook/img/dms_source_db_3.jpg rename to content/cookbook/importing_from_postgresql/img/dms_source_db_3.jpg diff --git a/content/cookbook/img/dms_target_db_1.jpg b/content/cookbook/importing_from_postgresql/img/dms_target_db_1.jpg similarity index 100% rename from content/cookbook/img/dms_target_db_1.jpg rename to content/cookbook/importing_from_postgresql/img/dms_target_db_1.jpg diff --git a/content/cookbook/img/dms_target_db_2.jpg b/content/cookbook/importing_from_postgresql/img/dms_target_db_2.jpg similarity index 100% rename from content/cookbook/img/dms_target_db_2.jpg rename to content/cookbook/importing_from_postgresql/img/dms_target_db_2.jpg diff --git a/content/cookbook/img/dms_target_db_3.jpg b/content/cookbook/importing_from_postgresql/img/dms_target_db_3.jpg similarity index 100% rename from content/cookbook/img/dms_target_db_3.jpg rename to content/cookbook/importing_from_postgresql/img/dms_target_db_3.jpg diff --git a/content/cookbook/img/dms_task_finished.jpg b/content/cookbook/importing_from_postgresql/img/dms_task_finished.jpg similarity index 100% rename from content/cookbook/img/dms_task_finished.jpg rename to content/cookbook/importing_from_postgresql/img/dms_task_finished.jpg diff --git a/content/cookbook/img/dms_task_started.jpg b/content/cookbook/importing_from_postgresql/img/dms_task_started.jpg similarity index 100% rename from content/cookbook/img/dms_task_started.jpg rename to content/cookbook/importing_from_postgresql/img/dms_task_started.jpg diff --git a/content/cookbook/img/dms_task_summary.jpg b/content/cookbook/importing_from_postgresql/img/dms_task_summary.jpg similarity index 100% rename from content/cookbook/img/dms_task_summary.jpg rename to content/cookbook/importing_from_postgresql/img/dms_task_summary.jpg diff --git a/content/cookbook/img/rds_create_postgres_0.jpg b/content/cookbook/importing_from_postgresql/img/rds_create_postgres_0.jpg similarity index 100% rename from content/cookbook/img/rds_create_postgres_0.jpg rename to content/cookbook/importing_from_postgresql/img/rds_create_postgres_0.jpg diff --git a/content/cookbook/img/rds_create_postgres_0a.jpg b/content/cookbook/importing_from_postgresql/img/rds_create_postgres_0a.jpg similarity index 100% rename from content/cookbook/img/rds_create_postgres_0a.jpg rename to content/cookbook/importing_from_postgresql/img/rds_create_postgres_0a.jpg diff --git a/content/cookbook/img/rds_create_postgres_1.jpg b/content/cookbook/importing_from_postgresql/img/rds_create_postgres_1.jpg similarity index 100% rename from content/cookbook/img/rds_create_postgres_1.jpg rename to content/cookbook/importing_from_postgresql/img/rds_create_postgres_1.jpg diff --git a/content/cookbook/img/rds_create_postgres_2.jpg b/content/cookbook/importing_from_postgresql/img/rds_create_postgres_2.jpg similarity index 100% rename from content/cookbook/img/rds_create_postgres_2.jpg rename to content/cookbook/importing_from_postgresql/img/rds_create_postgres_2.jpg diff --git a/content/cookbook/img/rds_create_postgres_3.jpg b/content/cookbook/importing_from_postgresql/img/rds_create_postgres_3.jpg similarity index 100% rename from content/cookbook/img/rds_create_postgres_3.jpg rename to content/cookbook/importing_from_postgresql/img/rds_create_postgres_3.jpg diff --git a/content/cookbook/img/rds_create_postgres_4.jpg b/content/cookbook/importing_from_postgresql/img/rds_create_postgres_4.jpg similarity index 100% rename from content/cookbook/img/rds_create_postgres_4.jpg rename to content/cookbook/importing_from_postgresql/img/rds_create_postgres_4.jpg diff --git a/content/cookbook/img/rds_create_postgres_5.jpg b/content/cookbook/importing_from_postgresql/img/rds_create_postgres_5.jpg similarity index 100% rename from content/cookbook/img/rds_create_postgres_5.jpg rename to content/cookbook/importing_from_postgresql/img/rds_create_postgres_5.jpg diff --git a/content/cookbook/img/rds_create_postgres_6.jpg b/content/cookbook/importing_from_postgresql/img/rds_create_postgres_6.jpg similarity index 100% rename from content/cookbook/img/rds_create_postgres_6.jpg rename to content/cookbook/importing_from_postgresql/img/rds_create_postgres_6.jpg diff --git a/content/cookbook/img/rds_create_postgres_7.jpg b/content/cookbook/importing_from_postgresql/img/rds_create_postgres_7.jpg similarity index 100% rename from content/cookbook/img/rds_create_postgres_7.jpg rename to content/cookbook/importing_from_postgresql/img/rds_create_postgres_7.jpg diff --git a/content/cookbook/img/rds_create_postgres_8.jpg b/content/cookbook/importing_from_postgresql/img/rds_create_postgres_8.jpg similarity index 100% rename from content/cookbook/img/rds_create_postgres_8.jpg rename to content/cookbook/importing_from_postgresql/img/rds_create_postgres_8.jpg diff --git a/content/cookbook/img/rds_create_postgres_9.jpg b/content/cookbook/importing_from_postgresql/img/rds_create_postgres_9.jpg similarity index 100% rename from content/cookbook/img/rds_create_postgres_9.jpg rename to content/cookbook/importing_from_postgresql/img/rds_create_postgres_9.jpg diff --git a/content/cookbook/img/rds_modify_1.jpg b/content/cookbook/importing_from_postgresql/img/rds_modify_1.jpg similarity index 100% rename from content/cookbook/img/rds_modify_1.jpg rename to content/cookbook/importing_from_postgresql/img/rds_modify_1.jpg diff --git a/content/cookbook/img/rds_modify_2.jpg b/content/cookbook/importing_from_postgresql/img/rds_modify_2.jpg similarity index 100% rename from content/cookbook/img/rds_modify_2.jpg rename to content/cookbook/importing_from_postgresql/img/rds_modify_2.jpg diff --git a/content/cookbook/img/rds_modify_3.jpg b/content/cookbook/importing_from_postgresql/img/rds_modify_3.jpg similarity index 100% rename from content/cookbook/img/rds_modify_3.jpg rename to content/cookbook/importing_from_postgresql/img/rds_modify_3.jpg diff --git a/content/cookbook/img/rds_modify_4.jpg b/content/cookbook/importing_from_postgresql/img/rds_modify_4.jpg similarity index 100% rename from content/cookbook/img/rds_modify_4.jpg rename to content/cookbook/importing_from_postgresql/img/rds_modify_4.jpg diff --git a/content/cookbook/img/rds_parameter_group_1.jpg b/content/cookbook/importing_from_postgresql/img/rds_parameter_group_1.jpg similarity index 100% rename from content/cookbook/img/rds_parameter_group_1.jpg rename to content/cookbook/importing_from_postgresql/img/rds_parameter_group_1.jpg diff --git a/content/cookbook/img/rds_parameter_group_2.jpg b/content/cookbook/importing_from_postgresql/img/rds_parameter_group_2.jpg similarity index 100% rename from content/cookbook/img/rds_parameter_group_2.jpg rename to content/cookbook/importing_from_postgresql/img/rds_parameter_group_2.jpg diff --git a/content/cookbook/img/rds_parameter_group_3.jpg b/content/cookbook/importing_from_postgresql/img/rds_parameter_group_3.jpg similarity index 100% rename from content/cookbook/img/rds_parameter_group_3.jpg rename to content/cookbook/importing_from_postgresql/img/rds_parameter_group_3.jpg diff --git a/content/cookbook/img/rds_parameter_group_4.jpg b/content/cookbook/importing_from_postgresql/img/rds_parameter_group_4.jpg similarity index 100% rename from content/cookbook/img/rds_parameter_group_4.jpg rename to content/cookbook/importing_from_postgresql/img/rds_parameter_group_4.jpg diff --git a/content/cookbook/img/rds_parameter_group_5.jpg b/content/cookbook/importing_from_postgresql/img/rds_parameter_group_5.jpg similarity index 100% rename from content/cookbook/img/rds_parameter_group_5.jpg rename to content/cookbook/importing_from_postgresql/img/rds_parameter_group_5.jpg diff --git a/content/cookbook/importing_from_postgresql/minimal_downtime.md b/content/cookbook/importing_from_postgresql/minimal_downtime.md new file mode 100644 index 00000000..371eaf43 --- /dev/null +++ b/content/cookbook/importing_from_postgresql/minimal_downtime.md @@ -0,0 +1,188 @@ +--- +title: "Migrating from PostgreSQL with Minimal Downtime" +linkTitle: "Migrate with minimal downtime" +--- + +This recipe migrates a PostgreSQL database to CedarDB while your application keeps running. +Create the full schema in CedarDB, let [Debezium](/docs/clients/data_import/debezium/) copy and replicate the data while the application still writes to PostgreSQL, and switch over once CedarDB has caught up. +The downtime is minimal and only lasts from stopping writes until the application uses CedarDB. + +## Prerequisites + +* A running Kafka Connect with the Debezium connectors, set up as described in [Debezium](/docs/clients/data_import/debezium/). +* Logical replication enabled in PostgreSQL (see [Debezium](/docs/clients/data_import/debezium/#enable-logical-replication-in-postgresql)). + +## Before you start + +### Check for unsupported column types + +CedarDB does not support the PostgreSQL types `timetz`, `xml`, and the geometric types, or stored generated columns. +Find such columns in the source database: + +```sql +SELECT table_schema, table_name, column_name, data_type +FROM information_schema.columns +WHERE table_schema NOT IN ('pg_catalog', 'information_schema') + AND (data_type IN ('time with time zone', 'xml', 'point', 'line', 'lseg', 'box', 'path', 'polygon', 'circle') + OR is_generated = 'ALWAYS') +ORDER BY 1, 2, 3; +``` + +Convert these columns before the migration, for example to `text`. +CedarDB also rejects the date and timestamp values `infinity` and `-infinity`. + +### Check for column privileges + +CedarDB supports privileges on tables, views, schemas, and sequences, but not on single columns. +Find column privileges in the source database: + +```sql +SELECT c.relnamespace::regnamespace AS table_schema, c.relname AS table_name, a.attname AS column_name +FROM pg_attribute AS a +JOIN pg_class AS c ON c.oid = a.attrelid +WHERE a.attacl IS NOT NULL + AND c.relnamespace::regnamespace::text NOT IN ('pg_catalog', 'information_schema') +ORDER BY 1, 2, 3; +``` + +Replace them with a view that contains the permitted columns and grant access to the view, or revoke them before the migration. +Otherwise, the restore stops at the first column privilege. + +### Set up connections + +The examples copy the PostgreSQL database `garden`. +Install the PostgreSQL client tools (`pg_dump`, `pg_restore`, `psql`) in the same major version as the source server. +Then define the two connections. +Use a superuser on both sides, so the restore can create objects for other owners: + +```shell +SOURCE="postgresql://@:5432/garden?sslmode=require" +TARGET="postgresql://@:5432/garden?sslmode=require" +TARGET_ADMIN="postgresql://@:5432/postgres?sslmode=require" +``` + +To avoid password prompts, store the passwords in a `~/.pgpass` file or set `PGPASSWORD`. + +### Copy roles and create the database + +`pg_dump` copies the objects of one database, but not the roles that own them or receive privileges. +Copy the roles first, leaving out the superuser `postgres`, which already exists in CedarDB: + +```shell +pg_dumpall --roles-only --dbname="$SOURCE" \ + | grep -vE '^(CREATE|ALTER) ROLE postgres[ ;]' \ + | psql "$TARGET_ADMIN" +``` + +The roles keep their PostgreSQL passwords, so applications can log in to CedarDB with their existing credentials. +On managed services such as Amazon RDS, you cannot read password hashes. There, instead add `--no-role-passwords` and set the passwords with `ALTER ROLE ... PASSWORD` afterward. +Also remove provider-specific roles, such as `rds_superuser`, from the output. + +Then create the target database with its original owner, here `garden_owner`: + +```shell +psql "$TARGET_ADMIN" -c "CREATE DATABASE garden OWNER garden_owner" +``` + +{{< callout type="info" >}} +Privileges (`GRANT`, `ALTER DEFAULT PRIVILEGES`, and role memberships) require a CedarDB [enterprise license](/docs/licensing/). +With the community edition, add `--no-privileges` to every `pg_dump` and `pg_restore` command on this page, and remove the role memberships from the `pg_dumpall` output by adding `| grep -v '^GRANT'` before `psql`. +Owners are kept in both editions. +{{< /callout >}} + +## Migrate + +{{% steps %}} + +### Create the schema without foreign keys + +Debezium writes each table independently, so a row can arrive before the row it references. +Create the schema without foreign keys and add them after the switch: + +```shell +pg_dump --format=custom --schema-only "$SOURCE" -f schema.dump +pg_restore --list schema.dump | grep -v 'FK CONSTRAINT' > schema-without-fks.list +pg_restore --list schema.dump | grep 'FK CONSTRAINT' > fks.list + +pg_restore --use-list=schema-without-fks.list --file=- schema.dump \ + | sed -E 's/^( *SEQUENCE NAME )[^.]+\./\1/' \ + | psql -v ON_ERROR_STOP=1 "$TARGET" +``` + +Then let the CedarDB user of the Debezium sink write into the tables: + +```sql +GRANT USAGE ON SCHEMA public TO debezium_sink; +GRANT SELECT, INSERT, UPDATE, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA public TO debezium_sink; +``` + +With the community edition, restore with `--no-owner --no-privileges` as the sink user instead, so the sink owns the tables. +Avoid schema changes in PostgreSQL during the migration. + +### Replicate the data + +Set up Debezium as described in [Set up continuous replication](/docs/clients/data_import/debezium/#set-up-continuous-replication), with these differences: + +* Include all tables of the database in the publication, `table.include.list`, and `topics`. +* Set `collection.name.format` in the sink to `public.${source.table}`, so the sink writes into the tables you created. + +Debezium copies the existing rows and then applies every change. +Wait until the sink's consumer lag stays close to zero. +While the application still writes to PostgreSQL, the two databases can differ by the most recent changes. + +### Stop writes and let CedarDB catch up + +Stop writes to PostgreSQL. +Then wait until the sink's consumer lag is zero +(see [Monitor the pipeline](/docs/clients/data_import/debezium/#monitor-the-pipeline)). + +### Validate the copy + +Run the same query on PostgreSQL and on CedarDB for each table and compare the results: + +```sql +SELECT count(*), sum(price), min(plant_id), max(plant_id) +FROM plants; +``` + +### Remove the Debezium pipeline + +Delete the Debezium connectors, then drop the replication slot and the publication in PostgreSQL, as described +in [Remove the pipeline](/docs/clients/data_import/debezium/#remove-the-pipeline). + +### Add foreign keys and continue identity columns + +Add the foreign keys that you left out at the start: + +```shell +pg_restore --use-list=fks.list --file=- schema.dump \ + | psql -v ON_ERROR_STOP=1 "$TARGET" +``` + +Debezium does not advance the sequences of identity columns. +Let each identity column continue after the highest copied value: + +```shell +psql "$TARGET" <<'SQL' +SELECT format('SELECT setval(pg_get_serial_sequence(%L, %L), max(%I)) FROM %I.%I;', + table_schema || '.' || table_name, column_name, column_name, table_schema, table_name) +FROM information_schema.columns +WHERE is_identity = 'YES' +\gexec +SQL +``` + +Check that the constraints and indexes exist: + +```sql +SELECT conrelid::regclass AS table_name, conname, contype +FROM pg_constraint +WHERE connamespace = 'public'::regnamespace +ORDER BY 1, 2; +``` + +### Switch the application to CedarDB + +Point your application to CedarDB. + +{{% /steps %}} diff --git a/content/cookbook/importing_from_postgresql/pg_dump.md b/content/cookbook/importing_from_postgresql/pg_dump.md new file mode 100644 index 00000000..e864a2be --- /dev/null +++ b/content/cookbook/importing_from_postgresql/pg_dump.md @@ -0,0 +1,229 @@ +--- +title: "Migrating from PostgreSQL with pg_dump" +linkTitle: "Copy with pg_dump" +--- + +This recipe copies a PostgreSQL database into CedarDB with the PostgreSQL client tools `pg_dump`, `pg_dumpall`, and `psql`. +It copies the complete schema, including keys, indexes, views, owners, and privileges. +To avoid data loss, this guide requires to stop writes to PostgreSQL during the copy. +For migration paths with connected applications still running, see [Migrate with minimal downtime](../minimal_downtime/). + +## Before you start + +### Check for unsupported column types + +CedarDB does not support the PostgreSQL types `timetz`, `xml`, and the geometric types, or stored generated columns. +Find such columns in the source database: + +```sql +SELECT table_schema, table_name, column_name, data_type +FROM information_schema.columns +WHERE table_schema NOT IN ('pg_catalog', 'information_schema') + AND (data_type IN ('time with time zone', 'xml', 'point', 'line', 'lseg', 'box', 'path', 'polygon', 'circle') + OR is_generated = 'ALWAYS') +ORDER BY 1, 2, 3; +``` + +Convert these columns before the migration, for example to `text`, or exclude the affected tables from the dump with `--exclude-table` and copy them separately. +CedarDB also rejects the date and timestamp values `infinity` and `-infinity`. + +### Check for column privileges + +CedarDB supports privileges on tables, views, schemas, and sequences, but not on single columns. +Find column privileges in the source database: + +```sql +SELECT c.relnamespace::regnamespace AS table_schema, c.relname AS table_name, a.attname AS column_name +FROM pg_attribute AS a +JOIN pg_class AS c ON c.oid = a.attrelid +WHERE a.attacl IS NOT NULL + AND c.relnamespace::regnamespace::text NOT IN ('pg_catalog', 'information_schema') +ORDER BY 1, 2, 3; +``` + +Replace them with a view that contains the permitted columns and grant access to the view, or revoke them before the migration. +Otherwise, the restore stops at the first column privilege. + +### Set up connections + +The examples copy the PostgreSQL database `garden`. +Install the PostgreSQL client tools (`pg_dump`, `pg_restore`, `psql`) in the same major version as the source server. +Then define the two connections. +Use a superuser on both sides, so the restore can create objects for other owners: + +```shell +SOURCE="postgresql://@:5432/garden?sslmode=require" +TARGET="postgresql://@:5432/garden?sslmode=require" +TARGET_ADMIN="postgresql://@:5432/postgres?sslmode=require" +``` + +To avoid password prompts, store the passwords in a `~/.pgpass` file or set `PGPASSWORD`. + +### Copy roles and create the database + +`pg_dump` copies the objects of one database, but not the roles that own them or receive privileges. +Copy the roles first, leaving out the superuser `postgres`, which already exists in CedarDB: + +```shell +pg_dumpall --roles-only --dbname="$SOURCE" \ + | grep -vE '^(CREATE|ALTER) ROLE postgres[ ;]' \ + | psql "$TARGET_ADMIN" +``` + +The roles keep their PostgreSQL passwords, so applications can log in to CedarDB with their existing credentials. +On managed services such as Amazon RDS, you cannot read password hashes. Add `--no-role-passwords` and set the passwords with `ALTER ROLE ... PASSWORD` afterward. +Also remove provider-specific roles, such as `rds_superuser`, from the output. + +Then create the target database with its original owner, here `garden_owner`: + +```shell +psql "$TARGET_ADMIN" -c "CREATE DATABASE garden OWNER garden_owner" +``` + +{{< callout type="info" >}} +Privileges (`GRANT`, `ALTER DEFAULT PRIVILEGES`, and role memberships) require a CedarDB [enterprise license](/docs/licensing/). +With the community edition, add `--no-privileges` to every `pg_dump` command on this page, and remove the role memberships from the `pg_dumpall` output by adding `| grep -v '^GRANT'` before `psql`. +Owners are kept in both editions. +{{< /callout >}} + +## Copy the database + +The PostgreSQL client tools offer three ways to copy the database, all with the complete schema: + +* **Full dump:** one command copies schema and data. Stop writes to PostgreSQL before you start. +* **Table by table:** create the tables, copy each table separately, then add constraints and indexes. Use it to copy only some tables, to copy large tables in parallel, or to repeat a single table if its copy fails. +* **CSV files:** export the schema and data to files and import them on the other side. Use it when PostgreSQL and CedarDB cannot reach each other over the network. + +{{< tabs >}} +{{< tab name="Full dump" >}} + +Stream a dump of the whole database into CedarDB: + +```shell +pg_dump "$SOURCE" \ + | sed -E 's/^( *SEQUENCE NAME )[^.]+\./\1/' \ + | psql -v ON_ERROR_STOP=1 "$TARGET" +``` + +The dump creates the tables and views with their owners and privileges, loads the data with `COPY`, and then adds primary keys, foreign keys, unique constraints, and indexes. +The `sed` command removes the schema name from the sequence names of identity columns, which CedarDB currently does not accept in that position. + +{{< /tab >}} +{{< tab name="Table by table" >}} + +Create tables, sequences, and views with their owners and privileges, but without constraints and indexes: + +```shell +pg_dump --section=pre-data "$SOURCE" \ + | sed -E 's/^( *SEQUENCE NAME )[^.]+\./\1/' \ + | psql -v ON_ERROR_STOP=1 "$TARGET" +``` + +List the tables to copy, with their schemas, in the file `tables.txt`: + +```shell +psql "$SOURCE" -AtX -c " + SELECT format('%I.%I', n.nspname, c.relname) + FROM pg_class AS c + JOIN pg_namespace AS n ON n.oid = c.relnamespace + WHERE c.relkind = 'r' + AND n.nspname NOT IN ('pg_catalog', 'information_schema') + AND n.nspname NOT LIKE 'pg_toast%'" > tables.txt +``` + +Copy each table through a pipe from PostgreSQL to CedarDB. +Each `psql` prints `COPY` and the number of rows: + +```shell +while read -r table; do + psql "$SOURCE" -c "\copy $table TO STDOUT" | psql "$TARGET" -c "\copy $table FROM STDIN" +done < tables.txt +``` + +Add the constraints and indexes: + +```shell +pg_dump --section=post-data "$SOURCE" \ + | psql -v ON_ERROR_STOP=1 "$TARGET" +``` + +The copied rows do not advance the sequences of identity columns, because the `setval` calls are part of the data section of the dump. +Let each identity column continue after the highest copied value: + +```shell +psql "$TARGET" <<'SQL' +SELECT format('SELECT setval(pg_get_serial_sequence(%L, %L), max(%I)) FROM %I.%I;', + table_schema || '.' || table_name, column_name, column_name, table_schema, table_name) +FROM information_schema.columns +WHERE is_identity = 'YES' +\gexec +SQL +``` + +{{< /tab >}} +{{< tab name="CSV files" >}} + +On a machine with access to PostgreSQL, export the schema, the list of tables, and the data: + +```shell +pg_dump --section=pre-data "$SOURCE" > pre-data.sql +pg_dump --section=post-data "$SOURCE" > post-data.sql +psql "$SOURCE" -AtX -c " + SELECT format('%I.%I', n.nspname, c.relname) + FROM pg_class AS c + JOIN pg_namespace AS n ON n.oid = c.relnamespace + WHERE c.relkind = 'r' + AND n.nspname NOT IN ('pg_catalog', 'information_schema') + AND n.nspname NOT LIKE 'pg_toast%'" > tables.txt +while read -r table; do + psql "$SOURCE" -c "\copy $table TO '$table.csv' WITH (FORMAT csv)" +done < tables.txt +``` + +Move the files, including `tables.txt`, together with the `pg_dumpall` output from [Copy roles and create the database](#copy-roles-and-create-the-database), to a machine with access to CedarDB. +There, import them: + +```shell +sed -E 's/^( *SEQUENCE NAME )[^.]+\./\1/' pre-data.sql | psql -v ON_ERROR_STOP=1 "$TARGET" +while read -r table; do + psql "$TARGET" -c "\copy $table FROM '$table.csv' WITH (FORMAT csv)" +done < tables.txt +psql -v ON_ERROR_STOP=1 "$TARGET" -f post-data.sql +``` + +The copied rows do not advance the sequences of identity columns, because the `setval` calls are part of the data section of the dump. +Let each identity column continue after the highest copied value: + +```shell +psql "$TARGET" <<'SQL' +SELECT format('SELECT setval(pg_get_serial_sequence(%L, %L), max(%I)) FROM %I.%I;', + table_schema || '.' || table_name, column_name, column_name, table_schema, table_name) +FROM information_schema.columns +WHERE is_identity = 'YES' +\gexec +SQL +``` + +CSV keeps `NULL` and empty strings apart. +For more on CSV files, see [Importing and working with CSV data](/docs/cookbook/working_with_csv/). + +{{< /tab >}} +{{< /tabs >}} + +## Validate the copy + +Run the same query on PostgreSQL and on CedarDB for each table and compare the results: + +```sql +SELECT count(*), sum(price), min(plant_id), max(plant_id) +FROM plants; +``` + +Also check that the constraints and indexes exist: + +```sql +SELECT conrelid::regclass AS table_name, conname, contype +FROM pg_constraint +WHERE connamespace = 'public'::regnamespace +ORDER BY 1, 2; +``` diff --git a/content/roadmap.md b/content/roadmap.md index 2efec6bb..38823247 100644 --- a/content/roadmap.md +++ b/content/roadmap.md @@ -62,7 +62,7 @@ CedarDB. | PostgreSQL system tables | {{< icondone >}} | [Documentation](/docs/compatibility/system_table/) | | information_schema support | {{< icondone >}} | [Documentation](/docs/compatibility/system_table/) | | PostgreSQL Logical replication | {{< iconinprogress >}} | | -| Support for more CDC tools | {{< iconinprogress >}} | [Documentation](/docs/cookbook/aurora_debezium/) | +| Support for more CDC tools | {{< iconinprogress >}} | [Documentation](/docs/clients/data_import/) | ## Enterprise Features diff --git a/layouts/shortcodes/callout.html b/layouts/shortcodes/callout.html index 1b7ff4a2..b2d24133 100644 --- a/layouts/shortcodes/callout.html +++ b/layouts/shortcodes/callout.html @@ -29,7 +29,7 @@
- {{ .Inner | markdownify }} + {{ .Inner | markdownify | chomp }}