Skip to content

Commit a12730b

Browse files
committed
added docs for the new microbatch compiled code feature
1 parent 38e3ccb commit a12730b

5 files changed

Lines changed: 71 additions & 2 deletions

File tree

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
title: "Capture compiled code for microbatch models"
3+
sidebarTitle: "Microbatch compiled code"
4+
---
5+
6+
import MicrobatchCompiledCode from '/snippets/guides/microbatch-compiled-code.mdx';
7+
8+
<MicrobatchCompiledCode />

docs/data-tests/dbt/package-models.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ New data is loaded to this model on an on-run-end hook named `elementary.upload_
2929
- `compile_completed_at` (string) - End time of resource compile action.
3030
- `rows_affected` (int) - Number of rows affected by the execution.
3131
- `full_refresh` (boolean) - Whether this was a full refresh execution.
32-
- `compiled_code` (string) - The compiled code (SQL / Python) executed against the database.
32+
- `compiled_code` (string) - The compiled code (SQL / Python) executed against the database. For microbatch incremental models, this column requires [extra setup](/cloud/guides/microbatch-compiled-code).
3333
- `failures` (int) - Number of failures in this run.
3434
- `query_id` (string) - Query ID in the data warehouse, if returned by the adapter (currently only supported in Snowflake, is null for any other adapter).
3535
- `thread_id` (string) - Id of the thread of this resource run.

docs/docs.json

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -187,6 +187,7 @@
187187
"cloud/guides/reduce-on-run-end-time",
188188
"cloud/guides/collect-job-data",
189189
"cloud/guides/collect-source-freshness",
190+
"cloud/guides/microbatch-compiled-code",
190191
"cloud/guides/troubleshoot"
191192
]
192193
},
@@ -521,7 +522,8 @@
521522
"oss/guides/collect-job-data",
522523
"oss/guides/collect-dbt-source-freshness",
523524
"oss/guides/reduce-on-run-end-time",
524-
"oss/guides/performance-alerts"
525+
"oss/guides/performance-alerts",
526+
"oss/guides/microbatch-compiled-code"
525527
]
526528
},
527529
{
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
title: "Capture compiled code for microbatch models"
3+
sidebarTitle: "Microbatch compiled code"
4+
---
5+
6+
import MicrobatchCompiledCode from '/snippets/guides/microbatch-compiled-code.mdx';
7+
8+
<MicrobatchCompiledCode />
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
Elementary can capture and store the compiled SQL of [dbt microbatch incremental models](https://docs.getdbt.com/docs/build/incremental-microbatch) in `dbt_run_results.compiled_code`. By default dbt does not surface compiled code for the microbatch strategy, so this column is empty for microbatch models until you enable the setup below.
2+
3+
## How it works
4+
5+
Elementary provides an override macro for dbt's `get_incremental_microbatch_sql` that captures the compiled SQL of each batch as it runs. The captured code is cached during the invocation and later written to `dbt_run_results.compiled_code`, so microbatch models populate this column the same way other incremental strategies do.
6+
7+
## Enabling microbatch compiled code capture
8+
9+
<Steps>
10+
<Step title="Override the microbatch strategy macro in your project">
11+
Add a macro that delegates to Elementary's implementation. Place it under your project's `macros/` directory:
12+
13+
```sql filename="macros/get_incremental_microbatch_sql.sql"
14+
{% macro get_incremental_microbatch_sql(arg_dict) %}
15+
{{ return(elementary.get_incremental_microbatch_sql(arg_dict)) }}
16+
{% endmacro %}
17+
```
18+
</Step>
19+
20+
<Step title="Enable the dbt behavior flag">
21+
Add the `require_batched_execution_for_custom_microbatch_strategy` flag to your `dbt_project.yml`:
22+
23+
```yaml filename="dbt_project.yml"
24+
flags:
25+
require_batched_execution_for_custom_microbatch_strategy: True
26+
```
27+
28+
This flag tells dbt to use your project-level override of the microbatch strategy with batched execution.
29+
</Step>
30+
31+
<Step title="Run your microbatch models">
32+
On the next `dbt run` or `dbt build`, Elementary captures the compiled SQL of each microbatch model and writes it to `dbt_run_results.compiled_code`.
33+
</Step>
34+
</Steps>
35+
36+
## Unsupported configurations
37+
38+
<Warning>
39+
The override flow is currently not supported on the following adapters:
40+
41+
- Spark
42+
- BigQuery
43+
- Athena
44+
- ClickHouse
45+
- Dremio
46+
- Vertica
47+
48+
It is also not supported on dbt Fusion.
49+
50+
On unsupported adapters and on Fusion, microbatch models continue to run normally but `dbt_run_results.compiled_code` remains empty for them.
51+
</Warning>

0 commit comments

Comments
 (0)