Skip to content

Commit b7a0dba

Browse files
rustyconoverclaude
andcommitted
docs: bring README up to date with recent features
README lagged CLAUDE.md — several features' docs only landed in CLAUDE.md. Add to README: date/time JSON types, edited (PICTURE-editing) numerics + SYNC, write-side gzip/zstd compression (write_fixed/write_multi/COPY TO), S3/HTTP byte-range streaming, and projection pushdown. Fix the now-stale "writing compressed output not supported" limitation; note the remaining multi-record gaps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 6b5c4e5 commit b7a0dba

1 file changed

Lines changed: 48 additions & 20 deletions

File tree

README.md

Lines changed: 48 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -62,11 +62,11 @@ directory).
6262
the prebuilt binaries are platform-specific (download the one matching your OS
6363
and CPU). If `ATTACH` fails with an opaque Arrow/IPC error, you most likely have
6464
a mismatched `vgi` extension version — update it (`UPDATE EXTENSIONS;`).
65-
- **Not yet supported:** writing compressed output (`write_fixed` / `COPY … TO`
66-
emit raw bytes); `COPY … TO` does not forward `CREATE SECRET` credentials (use
67-
`write_fixed` for secret-backed cloud writes); `http(s)://` is **read-only**
68-
(single object — no globbing); encodings are ASCII and EBCDIC CP037 only (no
69-
Latin-1 / UTF-16 / other code pages yet).
65+
- **Not yet supported:** `COPY … TO` does not forward `CREATE SECRET` credentials
66+
(use `write_fixed`/`write_multi` for secret-backed cloud writes); `http(s)://`
67+
is **read-only** (single object — no globbing); encodings are ASCII and EBCDIC
68+
CP037 only (no Latin-1 / UTF-16 / other code pages yet); `unpack`/`describe`/
69+
`COPY FROM` have no multi-record-type counterpart yet (use `read_multi`).
7070
- **Safety caps (untrusted input):** decompression is bounded by
7171
`max_decompressed_bytes` (16 GiB default; gzip/zstd only) to stop a
7272
decompression bomb, a single record by 512 MiB, and `DECIMAL` precision by 38.
@@ -186,8 +186,14 @@ input relation as a subquery `(FROM …)`:
186186
```sql
187187
SELECT * FROM write_fixed((FROM my_table), '/tmp/out.dat', 'name:A10 qty:9(5)');
188188
-- returns one row: (rows_written, bytes_written)
189+
190+
-- Compress the output (auto from the .gz/.zst extension, or compression =>):
191+
SELECT * FROM write_fixed((FROM my_table), '/tmp/out.dat.gz', 'name:A10 qty:9(5)');
189192
```
190193

194+
Named options mirror `read_fixed`: `format`, `encoding`, `framing`, plus
195+
`compression` (`auto` / `none` / `gzip` / `zstd`) and the `s3://` overrides.
196+
191197
### `write_multi` — write a heterogeneous (multi-record-type) file
192198

193199
`write_multi` is the inverse of [`read_multi`](#read_multi--read-a-heterogeneous-multi-record-type-file):
@@ -318,9 +324,17 @@ Self-documenting; good for generated specs.
318324
```
319325

320326
Types: `str`, `int`, `decimal`, `comp3`/`packed`, `zoned`, `binary`/`comp`,
321-
`float`/`double`/`half`, `hex`, `bool`, `pad`. Options: `width`, `digits`,
322-
`scale`, `signed`, `endian` (`big`/`little`), `occurs`, `justify` (`left`/`right`),
323-
`pad`, `sign` (`leading`/`trailing`/`embedded`).
327+
`float`/`double`/`half`, `hex`, `bool`, `pad`, and the temporal types `date` /
328+
`time` / `datetime` (= `timestamp`). Options: `width`, `digits`, `scale`,
329+
`signed`, `endian` (`big`/`little`), `occurs`, `justify` (`left`/`right`), `pad`,
330+
`sign` (`leading`/`trailing`/`embedded`), and `format` (required for the temporal
331+
types).
332+
333+
**Dates & times** parse a field's display bytes with a strftime `format` into a
334+
real DuckDB `DATE` / `TIME` / `TIMESTAMP` — e.g.
335+
`{"name": "d", "type": "date", "width": 8, "format": "%Y%m%d"}` turns `20240131`
336+
into `2024-01-31`. (Template and copybook have no date token — COBOL dates are
337+
plain numerics; use the JSON spec for typed dates.)
324338

325339
A field may instead carry a nested `fields` array, making it a **group**
326340
(`STRUCT`; its `type` is then optional). Combined with `occurs` a group becomes a
@@ -361,8 +375,13 @@ Real copybook text — paste it straight in.
361375
(not `fixed`).
362376
- `REDEFINES` → a `STRUCT` holding every overlapping interpretation of the same
363377
bytes (named after the base field).
364-
- `USAGE COMP-3`/`PACKED-DECIMAL`, `COMP`/`COMP-4`/`COMP-5`/`BINARY`, and
365-
`SIGN LEADING/TRAILING [SEPARATE]` are honored.
378+
- `USAGE COMP-3`/`PACKED-DECIMAL`, `COMP`/`COMP-4`/`COMP-5`/`BINARY`,
379+
`SIGN LEADING/TRAILING [SEPARATE]`, and `SYNCHRONIZED`/`SYNC` (binary items
380+
align to their natural boundary) are honored.
381+
- **Edited (PICTURE-editing) numerics** — report/print-image PICs like
382+
`ZZ,ZZ9.99`, `$$$,$$9.99`, `9(5)CR`, `**1,234.50` decode to an exact
383+
`DECIMAL(p,s)` (the editing — zero-suppression, currency, commas, `CR`/`DB` and
384+
`+`/`-` signs — is stripped). The non-floating masks round-trip on write.
366385

367386
---
368387

@@ -372,10 +391,11 @@ Real copybook text — paste it straight in.
372391
|-------|-------------|
373392
| text / hex | `VARCHAR` |
374393
| display / binary integer | `BIGINT` |
375-
| COMP-3, zoned, implied-point decimal | `DECIMAL(p, s)` (exact) |
394+
| COMP-3, zoned, implied-point, **edited** decimal | `DECIMAL(p, s)` (exact) |
376395
| float32 / float16 | `REAL` |
377396
| float64 | `DOUBLE` |
378397
| boolean | `BOOLEAN` |
398+
| `date` / `time` / `datetime` (JSON spec) | `DATE` / `TIME` / `TIMESTAMP` |
379399
| OCCURS / OCCURS DEPENDING ON | `LIST` of the above (variable-length for DEPENDING ON) |
380400
| group / nested `fields` / REDEFINES | `STRUCT` |
381401

@@ -389,19 +409,27 @@ Real copybook text — paste it straight in.
389409
- `rdw` — IBM variable-length: each record prefixed with a 4-byte Record
390410
Descriptor Word.
391411
- `rdw_blocked` — RDW records inside Block Descriptor Word blocks.
392-
- **compression** (`read_fixed` / `COPY … FROM`): `auto` (default — detect
393-
`gzip`/`zstd` from magic bytes, else read raw), `none`, `gzip`, or `zstd`.
394-
Decompression happens before framing and works for local and `s3://` paths
395-
alike. (Writing compressed output is not yet supported — `write_fixed` /
396-
`COPY … TO` emit raw bytes.)
412+
- **compression** — works on both sides:
413+
- **read** (`read_fixed` / `read_multi` / `COPY … FROM`): `auto` (default —
414+
detect `gzip`/`zstd` from magic bytes, else read raw), `none`, `gzip`, or
415+
`zstd`. Decompression happens before framing, local or `s3://` alike.
416+
- **write** (`write_fixed` / `write_multi` / `COPY … TO`): `auto` (default —
417+
`gzip` for a `.gz` destination, `zstd` for `.zst`, else raw), `none`, `gzip`,
418+
or `zstd`. The whole file is compressed before writing.
397419

398420
`read_fixed` **streams** `newline`/`fixed` files — records are framed and decoded
399421
a batch at a time (decompressing on the fly), so memory stays flat on large
400422
inputs instead of holding the whole file plus every decoded row; a multi-file
401-
glob reads one file at a time. `rdw`/`rdw_blocked` still buffer a whole object
402-
(their length-prefix walking needs it). Because reads stream, a malformed record
403-
deep in a file aborts the statement after earlier batches were produced — the
404-
failed statement returns no result, so nothing partial is committed.
423+
glob reads one file at a time. **`s3://`/`http(s)://` objects stream too**: they're
424+
fetched in 8 MiB byte ranges on demand, so a large cloud object never materializes
425+
whole. `rdw`/`rdw_blocked` still buffer a whole object (their length-prefix walking
426+
needs it). Because reads stream, a malformed record deep in a file aborts the
427+
statement after earlier batches were produced — the failed statement returns no
428+
result, so nothing partial is committed.
429+
430+
**Projection pushdown:** `read_fixed` only materializes the columns your query
431+
actually selects (mapped by name), so `SELECT one_field FROM read_fixed(…)` over a
432+
wide layout skips building the rest.
405433

406434
---
407435

0 commit comments

Comments
 (0)