@@ -62,11 +62,11 @@ directory).
6262 the prebuilt binaries are platform-specific (download the one matching your OS
6363 and CPU). If ` ATTACH ` fails with an opaque Arrow/IPC error, you most likely have
6464 a mismatched ` vgi ` extension version — update it (` UPDATE EXTENSIONS; ` ).
65- - ** Not yet supported:** writing compressed output ( ` write_fixed ` / ` COPY … TO `
66- emit raw bytes); ` COPY … TO ` does not forward ` CREATE SECRET ` credentials (use
67- ` write_fixed ` for secret-backed cloud writes); ` http(s):// ` is ** read-only**
68- (single object — no globbing); encodings are ASCII and EBCDIC CP037 only (no
69- Latin-1 / UTF-16 / other code pages yet).
65+ - ** Not yet supported:** ` COPY … TO ` does not forward ` CREATE SECRET ` credentials
66+ (use ` write_fixed ` / ` write_multi ` for secret-backed cloud writes); ` http(s):// `
67+ is ** read-only** (single object — no globbing); encodings are ASCII and EBCDIC
68+ CP037 only ( no Latin-1 / UTF-16 / other code pages yet); ` unpack ` / ` describe ` /
69+ ` COPY FROM ` have no multi-record-type counterpart yet (use ` read_multi ` ).
7070- ** Safety caps (untrusted input):** decompression is bounded by
7171 ` max_decompressed_bytes ` (16 GiB default; gzip/zstd only) to stop a
7272 decompression bomb, a single record by 512 MiB, and ` DECIMAL ` precision by 38.
@@ -186,8 +186,14 @@ input relation as a subquery `(FROM …)`:
186186``` sql
187187SELECT * FROM write_fixed((FROM my_table), ' /tmp/out.dat' , ' name:A10 qty:9(5)' );
188188-- returns one row: (rows_written, bytes_written)
189+
190+ -- Compress the output (auto from the .gz/.zst extension, or compression =>):
191+ SELECT * FROM write_fixed((FROM my_table), ' /tmp/out.dat.gz' , ' name:A10 qty:9(5)' );
189192```
190193
194+ Named options mirror ` read_fixed ` : ` format ` , ` encoding ` , ` framing ` , plus
195+ ` compression ` (` auto ` / ` none ` / ` gzip ` / ` zstd ` ) and the ` s3:// ` overrides.
196+
191197### ` write_multi ` — write a heterogeneous (multi-record-type) file
192198
193199` write_multi ` is the inverse of [ ` read_multi ` ] ( #read_multi--read-a-heterogeneous-multi-record-type-file ) :
@@ -318,9 +324,17 @@ Self-documenting; good for generated specs.
318324```
319325
320326Types: ` str ` , ` int ` , ` decimal ` , ` comp3 ` /` packed ` , ` zoned ` , ` binary ` /` comp ` ,
321- ` float ` /` double ` /` half ` , ` hex ` , ` bool ` , ` pad ` . Options: ` width ` , ` digits ` ,
322- ` scale ` , ` signed ` , ` endian ` (` big ` /` little ` ), ` occurs ` , ` justify ` (` left ` /` right ` ),
323- ` pad ` , ` sign ` (` leading ` /` trailing ` /` embedded ` ).
327+ ` float ` /` double ` /` half ` , ` hex ` , ` bool ` , ` pad ` , and the temporal types ` date ` /
328+ ` time ` / ` datetime ` (= ` timestamp ` ). Options: ` width ` , ` digits ` , ` scale ` ,
329+ ` signed ` , ` endian ` (` big ` /` little ` ), ` occurs ` , ` justify ` (` left ` /` right ` ), ` pad ` ,
330+ ` sign ` (` leading ` /` trailing ` /` embedded ` ), and ` format ` (required for the temporal
331+ types).
332+
333+ ** Dates & times** parse a field's display bytes with a strftime ` format ` into a
334+ real DuckDB ` DATE ` / ` TIME ` / ` TIMESTAMP ` — e.g.
335+ ` {"name": "d", "type": "date", "width": 8, "format": "%Y%m%d"} ` turns ` 20240131 `
336+ into ` 2024-01-31 ` . (Template and copybook have no date token — COBOL dates are
337+ plain numerics; use the JSON spec for typed dates.)
324338
325339A field may instead carry a nested ` fields ` array, making it a ** group**
326340(` STRUCT ` ; its ` type ` is then optional). Combined with ` occurs ` a group becomes a
@@ -361,8 +375,13 @@ Real copybook text — paste it straight in.
361375 (not ` fixed ` ).
362376- ` REDEFINES ` → a ` STRUCT ` holding every overlapping interpretation of the same
363377 bytes (named after the base field).
364- - ` USAGE COMP-3 ` /` PACKED-DECIMAL ` , ` COMP ` /` COMP-4 ` /` COMP-5 ` /` BINARY ` , and
365- ` SIGN LEADING/TRAILING [SEPARATE] ` are honored.
378+ - ` USAGE COMP-3 ` /` PACKED-DECIMAL ` , ` COMP ` /` COMP-4 ` /` COMP-5 ` /` BINARY ` ,
379+ ` SIGN LEADING/TRAILING [SEPARATE] ` , and ` SYNCHRONIZED ` /` SYNC ` (binary items
380+ align to their natural boundary) are honored.
381+ - ** Edited (PICTURE-editing) numerics** — report/print-image PICs like
382+ ` ZZ,ZZ9.99 ` , ` $$$,$$9.99 ` , ` 9(5)CR ` , ` **1,234.50 ` decode to an exact
383+ ` DECIMAL(p,s) ` (the editing — zero-suppression, currency, commas, ` CR ` /` DB ` and
384+ ` + ` /` - ` signs — is stripped). The non-floating masks round-trip on write.
366385
367386---
368387
@@ -372,10 +391,11 @@ Real copybook text — paste it straight in.
372391| -------| -------------|
373392| text / hex | ` VARCHAR ` |
374393| display / binary integer | ` BIGINT ` |
375- | COMP-3, zoned, implied-point decimal | ` DECIMAL(p, s) ` (exact) |
394+ | COMP-3, zoned, implied-point, ** edited ** decimal | ` DECIMAL(p, s) ` (exact) |
376395| float32 / float16 | ` REAL ` |
377396| float64 | ` DOUBLE ` |
378397| boolean | ` BOOLEAN ` |
398+ | ` date ` / ` time ` / ` datetime ` (JSON spec) | ` DATE ` / ` TIME ` / ` TIMESTAMP ` |
379399| OCCURS / OCCURS DEPENDING ON | ` LIST ` of the above (variable-length for DEPENDING ON) |
380400| group / nested ` fields ` / REDEFINES | ` STRUCT ` |
381401
@@ -389,19 +409,27 @@ Real copybook text — paste it straight in.
389409 - ` rdw ` — IBM variable-length: each record prefixed with a 4-byte Record
390410 Descriptor Word.
391411 - ` rdw_blocked ` — RDW records inside Block Descriptor Word blocks.
392- - ** compression** (` read_fixed ` / ` COPY … FROM ` ): ` auto ` (default — detect
393- ` gzip ` /` zstd ` from magic bytes, else read raw), ` none ` , ` gzip ` , or ` zstd ` .
394- Decompression happens before framing and works for local and ` s3:// ` paths
395- alike. (Writing compressed output is not yet supported — ` write_fixed ` /
396- ` COPY … TO ` emit raw bytes.)
412+ - ** compression** — works on both sides:
413+ - ** read** (` read_fixed ` / ` read_multi ` / ` COPY … FROM ` ): ` auto ` (default —
414+ detect ` gzip ` /` zstd ` from magic bytes, else read raw), ` none ` , ` gzip ` , or
415+ ` zstd ` . Decompression happens before framing, local or ` s3:// ` alike.
416+ - ** write** (` write_fixed ` / ` write_multi ` / ` COPY … TO ` ): ` auto ` (default —
417+ ` gzip ` for a ` .gz ` destination, ` zstd ` for ` .zst ` , else raw), ` none ` , ` gzip ` ,
418+ or ` zstd ` . The whole file is compressed before writing.
397419
398420` read_fixed ` ** streams** ` newline ` /` fixed ` files — records are framed and decoded
399421a batch at a time (decompressing on the fly), so memory stays flat on large
400422inputs instead of holding the whole file plus every decoded row; a multi-file
401- glob reads one file at a time. ` rdw ` /` rdw_blocked ` still buffer a whole object
402- (their length-prefix walking needs it). Because reads stream, a malformed record
403- deep in a file aborts the statement after earlier batches were produced — the
404- failed statement returns no result, so nothing partial is committed.
423+ glob reads one file at a time. ** ` s3:// ` /` http(s):// ` objects stream too** : they're
424+ fetched in 8 MiB byte ranges on demand, so a large cloud object never materializes
425+ whole. ` rdw ` /` rdw_blocked ` still buffer a whole object (their length-prefix walking
426+ needs it). Because reads stream, a malformed record deep in a file aborts the
427+ statement after earlier batches were produced — the failed statement returns no
428+ result, so nothing partial is committed.
429+
430+ ** Projection pushdown:** ` read_fixed ` only materializes the columns your query
431+ actually selects (mapped by name), so ` SELECT one_field FROM read_fixed(…) ` over a
432+ wide layout skips building the rest.
405433
406434---
407435
0 commit comments