D1
Ingest data from a Cloudflare D1 database with a SQL query.
The D1 connector ingests data from a Cloudflare D1 database. You give it a query, and the rows it returns become a table in Gaur.
Create a secret
The D1 connector needs a D1 secret with these fields:
| Field | Required | Description |
|---|---|---|
account_id | Yes | Your Cloudflare account id. |
database_id | Yes | The D1 database id. |
api_token | Yes | A Cloudflare API token authorized to read the database. |
Give the token the least privilege it needs to read the database. The credentials are encrypted and reused for every ingestion and refresh from this database.
Ingest a table
Choose the D1 connector
In the Ingestion area, pick D1 and select the D1 secret you created.
Write the query
Provide a SELECT query for the data you want. Its result set becomes
the table.
Name the table and declare its primary key
Name the resulting table and declare its primary key. Gaur runs the ingestion as a background job, infers the schema, and creates the table.
Set up incremental refresh
D1 tables can refresh incrementally, pulling only rows that changed
since the last run. To enable it, configure a cursor when you ingest:
a column Gaur watches to know where it left off (an auto-incrementing id
or an updated_at timestamp), along with that column's type.
Without a cursor, the table still supports full and append refreshes.
Refreshing
D1 tables support full, append, and incremental refreshes, on demand or on a schedule. See Refreshes.
Gaur ingests a copy of the query result into its own storage; it does not query your D1 database live. Analytical traffic never reaches D1 after ingestion. See Gaur ingests data, it does not query your databases.