You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The worker image is published on every push to `main` to `ghcr.io/<owner>/partforge`, tagged with the short commit SHA and `latest`. CI also attaches static `linux/amd64` and `linux/arm64` CLI binaries to a GitHub release named after the SHA.
4
+
5
+
The image is a single Ubuntu container with `clickhouse-server`, `clickhouse-client`, `s5cmd`, and the Go binary. Its entrypoint is the binary and the default command is `worker`. It **runs as root** (so it can write its work directory on root-owned host mounts) and starts a local `clickhouse server` child process for each claimed part.
6
+
7
+
## Recommended: workers on ECS with an IAM task role
8
+
9
+
Run the workers as an ECS service and give the task an **IAM role** scoped to the S3 bucket and the DynamoDB table. This is the recommended setup:
10
+
11
+
-**No static credentials.** The AWS SDK picks up temporary credentials from the ECS task role via the container credentials endpoint. Do not bake access keys into the image or config.
12
+
-**Region resolves from the environment.** Set `AWS_REGION` (or `-aws-region`); otherwise it falls back through AWS config, IMDS, then `us-east-1`.
13
+
-**Scale by replicas.** More worker tasks = more parts in flight. There is no coordinator to scale — workers claim independently from DynamoDB.
14
+
15
+
### Task IAM policy
16
+
17
+
Combine the S3 and DynamoDB permissions. DynamoDB detail (including tighter per-role variants) is in [dynamodb.md](dynamodb.md).
`s3:DeleteObject` is needed because workers replace finished-artifact prefixes (`s5cmd rm` then upload) and admin `-delete-s3` operations remove artifacts.
53
+
54
+
### Storage matters
55
+
56
+
Worker scratch (`-work-dir`) holds the local ClickHouse data plus downloaded source parts, and compaction transiently holds downloaded tarballs, extracted parts, merge output, and re-uploaded tarballs at once. It must be **fast local disk with enough headroom**:
57
+
58
+
-**EC2 launch type with instance-store NVMe** is best for large parts — mount the NVMe into the container and set `-work-dir` on it (e.g. `/mnt/nvme/partforge-work`).
59
+
-**Fargate** works for smaller jobs; raise the task's ephemeral storage and keep `-compact-max-bytes` well below it.
60
+
61
+
Each claimed part gets its own `run-*` directory that is removed when the part finishes.
62
+
63
+
### Splitting inserter and compactor
64
+
65
+
Run the rewrite and compaction stages as separate services to scale them independently:
66
+
67
+
-`worker -role=inserter` — rewrite only.
68
+
-`worker -role=compactor` — compaction only.
69
+
-`worker -role=all` (default) — rewrite first, compact when idle.
70
+
71
+
See [operations.md](operations.md) for the full flag set and metrics.
72
+
73
+
## Where the other commands run
74
+
75
+
`worker` is the only stage that belongs on ECS. The other two need local access to a ClickHouse node's disks and generally run there:
76
+
77
+
-**`upload-freeze`** must run where it can read the source ClickHouse data disks reported by `system.disks`.
78
+
-**`import-finished`** must run where its work-dir shares a filesystem with the destination table's `detached` directory (parts are moved, not copied).
79
+
80
+
Both still need the same S3 + DynamoDB access as the workers.
CI (`.github/workflows/release.yml`) enforces formatting, tidy modules, vet, and tests. Run what CI runs before finishing a change:
6
+
7
+
```sh
8
+
gofmt -l .# must print nothing
9
+
go mod tidy # must not change go.mod / go.sum
10
+
go vet ./...
11
+
go test ./...
12
+
./e2e/run.sh # requires Docker
13
+
```
14
+
15
+
`AGENTS.md` lists `go mod tidy && go test ./... && ./e2e/run.sh` as the required pre-commit check.
16
+
17
+
The e2e script stands up LocalStack + a ClickHouse container, builds the worker image, and runs the full pipeline against `e2e/sql/`, diffing the result against `e2e/expected.tsv`. It builds the image each run; set `PARTFORGE_E2E_SKIP_BUILD=1` to reuse an existing `partforge-worker:latest`.
18
+
19
+
## Build
20
+
21
+
```sh
22
+
go build -o partforge ./cmd/partforge # local CLI
23
+
docker compose build worker # worker image
24
+
```
25
+
26
+
## Project layout
27
+
28
+
```
29
+
cmd/partforge/ CLI entrypoint — every subcommand, flag parsing, config resolution
30
+
internal/
31
+
freeze/ discover ClickHouse disks; scan shadow/<freeze> for frozen parts
32
+
manifest/ per-part manifest.json; job/part ID derivation
33
+
artifact/ write manifests; build/extract part tarballs
34
+
s3copy/ s5cmd wrapper for directory/glob transfers
35
+
state/ DynamoDB state store — claims, transitions, compaction batches, admin ops
36
+
chproc/ start/stop the local clickhouse-server child process
- CLI flags and command wiring → `cmd/partforge/main.go`
52
+
53
+
## Release
54
+
55
+
On push to `main`, CI builds and publishes the multi-arch worker image to `ghcr.io/<owner>/partforge` (tagged with the short commit SHA and `latest`) and attaches static `linux/amd64` and `linux/arm64` CLI binaries to a GitHub release named after the SHA. See [deployment.md](deployment.md).
0 commit comments