Refreshes
Keep table data current with full, append, and incremental refreshes, on demand or on a schedule.
A table holds a snapshot of its source taken at ingestion time. A refresh pulls fresh data from that source so the table stays current. You can run a refresh on demand or put it on a schedule.
Refresh modes
Gaur has three refresh modes. Which ones you can use depends on the table's connector type.
Full refresh
Replaces the table's data with a fresh pull from the source. Use it when the source holds the complete current state of the data.
Append
Adds new rows to the existing table without touching what's already there. Use it for event-style data that only ever grows.
Incremental
Pulls only the rows that changed since the last run, tracked by a cursor. Efficient for large database tables that change steadily.
Full refresh
A full refresh throws away the table's current data and replaces it with a complete pull from the source. It's the simplest mode and the safest default: whatever the source says now is what the table holds afterward.
For database sources, a full refresh re-runs the query the table was created from. For file sources, it re-reads the files at the configured path.
Append
An append refresh keeps the existing rows and adds new ones on top.
- File sources (S3, R2, file upload): you supply the new file or files, and their rows are appended to the table.
- Database sources (Postgres, MySQL): you supply a
SELECTquery that returns the rows to append. The query is read-only.
Append is the right mode for immutable, growing data such as event logs. It doesn't deduplicate, so only append data you know is new.
Incremental
An incremental refresh pulls just the rows that are new or changed since the last run. It's available for database sources only.
Incremental needs a cursor: a column Gaur watches to know where it left
off, such as an auto-incrementing id or an updated_at timestamp. The table
must have cursor configuration (the cursor column and its type) before an
incremental refresh will run. Without it, the refresh is rejected.
On each run, Gaur fetches rows past the last cursor value and advances the cursor. This keeps large tables current without re-pulling everything.
Which modes each connector supports
| Connector | Full refresh | Append | Incremental |
|---|---|---|---|
| File upload | Yes | Yes | No |
| S3 | Yes | Yes | No |
| R2 | Yes | Yes | No |
| Postgres | Yes | Yes | Yes |
| MySQL | Yes | Yes | Yes |
A table can only be refreshed through the connector type it was created with. A table ingested from S3 refreshes as an S3 table, and so on.
Running a refresh
A refresh runs as a background job, the same as the initial ingestion. You start it, Gaur queues the job, and you can track its progress to completion.
While a refresh is running, the table is locked. If you try to start a second refresh for the same table before the first finishes, Gaur rejects it with a conflict. This keeps two refreshes from racing on the same data. Wait for the running job to finish, then start the next one.
Scheduled refreshes
Most tables shouldn't need a person to refresh them by hand. Attach a refresh schedule to a table and Gaur runs the refresh for you on a recurring cadence.
A schedule has:
- A mode: the refresh mode (full, append, or incremental) to run on each tick.
- A frequency: how often it runs, chosen from a set of interval presets.
Once a schedule is set, Gaur computes the next run time and refreshes the table automatically at that cadence. Each scheduled run is an ordinary refresh job, so it shows up alongside manual refreshes and obeys the same table lock.
You can update a schedule's mode and frequency at any time, or disable it to go back to manual refreshes.
Pick the schedule mode that matches the data. Schedule a full refresh for sources that fully restate their data, an incremental refresh for large database tables that change steadily, and an append schedule only for strictly append-only sources.
Choosing a strategy
- Small source, fully restated each time: full refresh, on a schedule that matches how often the source changes.
- Large database table, steady changes: incremental refresh, so you only move the rows that changed.
- Append-only event data: append refresh, manual or scheduled, supplying only data you know is new.