Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 22 additions & 7 deletions Structure.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,10 @@ One page per task. Each page is self-contained: the reader follows it from
top to bottom and completes the task without consulting other pages. Include
prerequisites at the top.

When one task has several alternative recipes, put them in a folder with an
`_index.md` that compares them, for example `importing_from_postgresql/`
with `pg_dump.md`, `minimal_downtime.md`, and `aws_dms.md`.

Add a page when a task involves multiple steps, requires specific tooling,
or combines multiple features in a non-obvious way. Simple single-feature
usage belongs in the reference section.
Expand All @@ -62,7 +66,7 @@ to reference pages for feature details.

### clients/

Organized into three subsections:
Organized into four subsections:

- **Language folders** (`python/`, `javascript/`, `java/`, etc.) — one folder per
programming language. The folder's `_index.md` covers the primary driver or
Expand All @@ -71,6 +75,10 @@ Organized into three subsections:
- **`apis/`** — APIs like ADBC for language-agnostic Arrow database connectivity.
- **`tools/`** — GUI clients, CLI tools, and observability integrations that are
not tied to a specific programming language (DBeaver, DataGrip, psql, Grafana).
- **`data_import/`** — data integration and change data capture (CDC) tools that
load data into CedarDB or keep it in sync with another database (Airbyte, Debezium).
Multi-tool, task-oriented procedures (e.g. migrating from PostgreSQL) stay in
`cookbook/` and link to these pages.

Each page covers installation, connection setup, and a minimal working example.
Focus on what is specific to CedarDB. Do not rewrite the client library's own
Expand Down Expand Up @@ -118,6 +126,10 @@ clients/
│ └── _index.md
├── apis
│ └── adbc.md # ADBC api
├── data_import/ # Data integration and CDC tools
│ ├── _index.md
│ ├── airbyte.md
│ └── debezium.md
└── tools/ # Not language-specific
├── psql.md
├── dbeaver.md
Expand Down Expand Up @@ -292,13 +304,16 @@ jump directly to it.
11. **Is it about a GUI client, CLI tool, or BI/observability integration?**
Go to `clients/tools/<tool>.md`

12. **Is it an optimization guideline or recommended pattern?**
12. **Is it about a data integration or CDC tool (Airbyte, Debezium, ...)?**
Go to `clients/data_import/<tool>.md`

13. **Is it an optimization guideline or recommended pattern?**
Go to `best_practices/<topic>.md`

13. **Is it a compatibility status update?**
14. **Is it a compatibility status update?**
Go to `compatibility/<area>.md`

14. **None of the above?**
15. **None of the above?**
Flag for team discussion. Do not create new top-level sections without agreement.

---
Expand Down Expand Up @@ -348,14 +363,14 @@ to any handwritten member list in an `_index.md`.
## File Naming

- Lowercase, snake_case when more than one word is needed (`asof_join.md`,
`importing_from_postgresql.md`). Single-word names have no separator (`tables.md`).
`working_with_csv.md`). Single-word names have no separator (`tables.md`).
- Object pages by object name: `tables.md`, `indexes.md`, `roles.md`
- Data types by common name: `integer.md`, `json.md`, `text.md`
- Function categories by domain: `aggregation.md`, `timestamp.md`, `json.md`
- Cookbooks with descriptive slugs: `importing_from_postgresql.md`, `working_with_csv.md`
- Cookbooks with descriptive slugs: `working_with_csv.md`, `importing_from_postgresql/`
- Language client folders by language name: `python/`, `javascript/`, `java/`
- ORM pages within the language folder: `javascript/drizzle.md`, `javascript/prisma.md`
- Tool pages by tool name: `tools/grafana.md`, `tools/dbeaver.md`
- Tool pages by tool name: `tools/grafana.md`, `tools/dbeaver.md`, `data_import/debezium.md`

---

Expand Down
5 changes: 5 additions & 0 deletions content/clients/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@

* [ADBC](apis/adbc)

Data import and CDC:

* [Airbyte](data_import/airbyte)

Check failure on line 25 in content/clients/_index.md

View workflow job for this annotation

GitHub Actions / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'Airbyte'? Raw Output: {"message": "[Vale.Spelling] Did you really mean 'Airbyte'?", "location": {"path": "content/clients/_index.md", "range": {"start": {"line": 25, "column": 4}}}, "severity": "ERROR"}
* [Debezium](data_import/debezium)

Tools:

* [DataGrip](tools/datagrip)
Expand Down
13 changes: 13 additions & 0 deletions content/clients/data_import/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
title: "Data Import and CDC"
linkTitle: "Data Import and CDC"
---

Data integration and change data capture (CDC) tools that load data into CedarDB or keep it in sync with another
database.
CedarDB works with these tools through their PostgreSQL connectors, without a CedarDB-specific plugin.
For a step-by-step migration from PostgreSQL, including methods that need no extra tools,
see [Migrating from PostgreSQL](/docs/cookbook/importing_from_postgresql/).

* [Airbyte](./airbyte) — copy tables once

Check failure on line 12 in content/clients/data_import/_index.md

View workflow job for this annotation

GitHub Actions / vale

[vale] reported by reviewdog 🐶 [Google.EmDash] Don't put a space before or after a dash. Raw Output: {"message": "[Google.EmDash] Don't put a space before or after a dash.", "location": {"path": "content/clients/data_import/_index.md", "range": {"start": {"line": 12, "column": 23}}}, "severity": "ERROR"}

Check failure on line 12 in content/clients/data_import/_index.md

View workflow job for this annotation

GitHub Actions / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'Airbyte'? Raw Output: {"message": "[Vale.Spelling] Did you really mean 'Airbyte'?", "location": {"path": "content/clients/data_import/_index.md", "range": {"start": {"line": 12, "column": 4}}}, "severity": "ERROR"}
* [Debezium](./debezium) — replicate PostgreSQL changes continuously through Kafka, or copy a snapshot once

Check failure on line 13 in content/clients/data_import/_index.md

View workflow job for this annotation

GitHub Actions / vale

[vale] reported by reviewdog 🐶 [Google.EmDash] Don't put a space before or after a dash. Raw Output: {"message": "[Google.EmDash] Don't put a space before or after a dash.", "location": {"path": "content/clients/data_import/_index.md", "range": {"start": {"line": 13, "column": 25}}}, "severity": "ERROR"}
176 changes: 176 additions & 0 deletions content/clients/data_import/airbyte.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
---
title: "Airbyte"

Check failure on line 2 in content/clients/data_import/airbyte.md

View workflow job for this annotation

GitHub Actions / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'Airbyte'? Raw Output: {"message": "[Vale.Spelling] Did you really mean 'Airbyte'?", "location": {"path": "content/clients/data_import/airbyte.md", "range": {"start": {"line": 2, "column": 9}}}, "severity": "ERROR"}
linkTitle: "Airbyte"

Check failure on line 3 in content/clients/data_import/airbyte.md

View workflow job for this annotation

GitHub Actions / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'Airbyte'? Raw Output: {"message": "[Vale.Spelling] Did you really mean 'Airbyte'?", "location": {"path": "content/clients/data_import/airbyte.md", "range": {"start": {"line": 3, "column": 13}}}, "severity": "ERROR"}
---

[Airbyte](https://airbyte.com/) moves data from hundreds of sources into a database with a few settings in its UI.

Check failure on line 6 in content/clients/data_import/airbyte.md

View workflow job for this annotation

GitHub Actions / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'Airbyte'? Raw Output: {"message": "[Vale.Spelling] Did you really mean 'Airbyte'?", "location": {"path": "content/clients/data_import/airbyte.md", "range": {"start": {"line": 6, "column": 2}}}, "severity": "ERROR"}
CedarDB works as an Airbyte destination through Airbyte's **Postgres** destination connector, so you can load data into

Check failure on line 7 in content/clients/data_import/airbyte.md

View workflow job for this annotation

GitHub Actions / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'Airbyte's'? Raw Output: {"message": "[Vale.Spelling] Did you really mean 'Airbyte's'?", "location": {"path": "content/clients/data_import/airbyte.md", "range": {"start": {"line": 7, "column": 49}}}, "severity": "ERROR"}

Check failure on line 7 in content/clients/data_import/airbyte.md

View workflow job for this annotation

GitHub Actions / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'Airbyte'? Raw Output: {"message": "[Vale.Spelling] Did you really mean 'Airbyte'?", "location": {"path": "content/clients/data_import/airbyte.md", "range": {"start": {"line": 7, "column": 21}}}, "severity": "ERROR"}
CedarDB from any source Airbyte supports.
This page uses a PostgreSQL database as the source and copies its tables with a full refresh.
Incremental syncs with **Append + Deduped** are not supported yet.
To replicate changes from PostgreSQL continuously, use [Debezium](../debezium/).

This page was tested with the Airbyte connectors `source-postgres` 3.8.7 and `destination-postgres` 3.0.23, and
PostgreSQL 16.
The setting names match the Airbyte UI of self-managed Airbyte and Airbyte Cloud.

This page is a self-contained example that you can follow from start to finish.
It copies the tables `species` and `plants` from the PostgreSQL database `nursery` into the CedarDB database `nursery`.
It keeps the Airbyte setup short and points out what is specific to CedarDB.
For all connector settings and for running Airbyte in production, see Airbyte's documentation of
the [Postgres source](https://docs.airbyte.com/integrations/sources/postgres),
the [Postgres destination](https://docs.airbyte.com/integrations/destinations/postgres), [sync modes](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes),
and [deploying Airbyte](https://docs.airbyte.com/platform/deploying-airbyte).

## How CedarDB tables look

Airbyte creates one table per source table (stream) in CedarDB.
These tables are landing tables, not a copy of the source schema:

* Each table has four extra columns: `_airbyte_raw_id`, `_airbyte_extracted_at`, `_airbyte_meta`, and
`_airbyte_generation_id`.
* All source columns are nullable.
Primary keys, foreign keys, unique constraints, defaults, and indexes are not copied.
* Types are normalized: integers become `bigint`, decimals and floating-point numbers become `numeric`, `uuid` and most
other types become `varchar`, and arrays become `jsonb`.

The Postgres destination creates these tables the same way in PostgreSQL; this is not specific to CedarDB.

If you need the exact source schema, for example to migrate an application, use Airbyte to load the data and then copy
it into tables you define yourself, or follow [Migrating from PostgreSQL](/docs/cookbook/importing_from_postgresql/).

## Prerequisites

* Airbyte (self-managed or Airbyte Cloud) that can reach both PostgreSQL and CedarDB over the network.
* A CedarDB superuser to create the destination role and database.

## Set up CedarDB as a destination

{{% steps %}}

### Create a role and database

Connect to CedarDB as a superuser and create a role for Airbyte that owns the target database:

```sql
CREATE ROLE airbyte WITH LOGIN PASSWORD '<airbyte-password>';
CREATE DATABASE nursery OWNER airbyte;
Comment thread
ChrisWint marked this conversation as resolved.
```

As the database owner, Airbyte can create its schemas and tables without further privileges.
This works in every CedarDB edition.
Granting privileges with `GRANT` instead requires a CedarDB [enterprise license](/docs/licensing/).

### Add the destination in Airbyte

In Airbyte, add a new destination of type **Postgres** with these settings:

| Setting | Value |
|---------------------------|-----------------------------------------------------------------------------------------------------------|
| Host, Port | Host name and port of CedarDB |
| Database Name | `nursery` |
| Default Schema | `public` |
| Username, Password | `airbyte` and its password |
| SSL Connection, SSL Modes | Enable SSL and use `require` (see [SSL configuration](/docs/references/configuration/#ssl-configuration)) |

Leave the other settings at their defaults.
When you save the destination, Airbyte tests it by creating, writing, and dropping a table in CedarDB.

{{% /steps %}}

Airbyte keeps the source schema name: tables from the PostgreSQL schema `public` land in the CedarDB schema `public`.
To land them somewhere else, change the **Destination Namespace** of the connection.

## Copy tables once

A **Full refresh | Overwrite** sync copies all rows of the selected tables and replaces the CedarDB tables on every run.

{{% steps %}}

### Prepare PostgreSQL

In the source PostgreSQL database, create a role that can read the tables:

```sql
CREATE ROLE airbyte_source WITH LOGIN PASSWORD '<source-password>';
GRANT USAGE ON SCHEMA public TO airbyte_source;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO airbyte_source;
```

### Add the source in Airbyte

Add a new source of type **Postgres** with the host, port, database name, `airbyte_source` credentials, and the schema
`public`.
The source requires an **Update Method**.
Select **Scan Changes with User Defined Cursor**, which needs no replication setup in PostgreSQL.
Full refresh syncs read all rows and ignore it.

### Create the connection

Create a connection from the source to the CedarDB destination, select the tables, and set the sync mode of each table
to **Full refresh | Overwrite**.
Click **Sync now**.
On the connection's **Status** tab, Airbyte shows the records loaded per table while the sync runs.
The **Timeline** tab lists each sync with its duration and logs.

### Verify the copy

Compare row counts and sums on both sides, for example:

```sql
SELECT count(*), sum(price), min(plant_id), max(plant_id)
FROM plants;
```
Comment thread
ChrisWint marked this conversation as resolved.

{{% /steps %}}

Each further run replaces the CedarDB tables with the current source rows, including deletions.
To migrate an application, stop writes to PostgreSQL, run a final sync, and verify the tables before you switch over.

## CedarDB limitations

### Decimals with more than six fractional digits

CedarDB columns such as `numeric(38,18)` store more than six fractional digits, but Airbyte does not create such
columns.
It creates all decimal and floating-point source columns as `numeric` without precision and scale, which CedarDB
creates as `numeric(38,6)` (see [Numeric](/docs/references/datatypes/numeric/)).
A value with nonzero digits after the sixth decimal place, for example the `double precision` value `3.14159265`, makes
Comment thread
ChrisWint marked this conversation as resolved.
the sync of the table fail with `invalid number format: loss of precision`, and no rows of the table arrive.
Very small values, such as `1e-9`, are stored as `0` without an error.

To copy such columns exactly, create their tables in CedarDB before the first sync.
Give each decimal and floating-point column a `numeric(p,s)` type with enough fractional digits:

```sql
CREATE TABLE public.soil_samples (
_airbyte_raw_id varchar NOT NULL,
_airbyte_extracted_at timestamptz NOT NULL,
_airbyte_meta jsonb NOT NULL,
_airbyte_generation_id bigint,
sample_id bigint,
ph numeric(38,18),
trace_ppm numeric(38,18)
);
```

Airbyte keeps existing `numeric` columns with their precision and scale.
It changes columns of other types, such as `double precision`, to `numeric(38,6)`, so convert such columns after the
copy if you need them.

### Other CedarDB limitations

* **Append + Deduped** is not supported.
The sync fails because CedarDB currently only supports a single data-modifying CTE per query, whereas Airbyte uses 3.

## Airbyte behavior

The following behavior is the same with a PostgreSQL destination:

* A **Full refresh | Overwrite** sync drops and recreates the destination tables, so it fails while a view depends on
one of them.
Drop such views before the sync and recreate them afterward, or enable **Drop tables with CASCADE** in the
destination, which drops the views.
* Empty strings arrive as `NULL`.
* `infinity` dates and timestamps, `NaN` and infinite floating-point values, and multidimensional arrays arrive as
`NULL`, with a note in `_airbyte_meta`.
Loading
Loading