Introduce runtime policies for Ferret CLI: add support for filesystem and HTTP policies (#41)

* Introduce runtime policies for Ferret CLI: add support for filesystem and HTTP policies, update tests, and enhance documentation.

* Reject binary flags in unsupported runtimes: validate runtime options, update helpers, tests, and use `slices.Concat` for argument assembly.

Author: ziflex

README.md

@@ -664,9 +664,9 @@ RETURN T::EQ(payload.id, "123")
 
 Mock API entries use the same binding syntax as static serving: `<path>`, `<path>:<port>`, `<path>@<alias>`, and `<path>@<alias>:<port>`. `--serve-bind` and `--serve-host` also apply to mock API servers.
 
-### 🔒 Built-in Filesystem Policy
+### 🔒 Filesystem Policy
 
-The built-in runtime exposes FQL filesystem functions through a sandbox rooted at Lab's current working directory. Use `--policy-fs-root` to select a narrower relative or absolute root, and add `--policy-fs-read-only` to permit reads while rejecting writes, directory changes, and removals.
+The built-in runtime exposes FQL filesystem functions through a sandbox rooted at Lab's current working directory. Use `--policy-fs-root` to select a narrower relative or absolute root, and add `--policy-fs-read-only` to permit reads while rejecting writes, directory changes, and removals. When `--runtime=bin:...` selects a Ferret CLI v2 binary, Lab forwards the same explicitly configured policy values to `ferret run`.
 
 ```bash
 lab run \
@@ -697,7 +697,7 @@ lab run \
 
 When the runtime URL already includes a path, Lab sends `run` requests to that exact path. The optional `runtime-param=path` value overrides the run endpoint only. `lab version --runtime=...` uses the runtime URL path and requests its sibling `/info` endpoint.
 
-The `--policy-fs-*` and `--policy-http-*` flags configure only Lab's built-in Ferret runtime. Lab rejects them when `--runtime` selects an HTTP or binary adapter because policy enforcement belongs to that external runtime.
+Lab rejects `--policy-fs-*` and `--policy-http-*` options for HTTP runtimes because their request protocol has no policy contract. Built-in and Ferret CLI v2 binary runtimes both support these options.
 
 The HTTP runtime sends POST requests with:
 
@@ -712,25 +712,35 @@ The HTTP runtime sends POST requests with:
 
 #### External Binary Runtime
 
-Use custom Ferret CLI installations:
+Use Ferret CLI v2-compatible installations. Lab invokes the binary as `ferret run`, passes the FQL source through stdin, and serializes test parameters as `--param=name=<JSON>` arguments.
 
 ```bash
 # Use specific Ferret binary
 lab run --runtime=bin:./custom-ferret tests/
 
-# With runtime params forwarded as --param entries
+# With shared runtime params forwarded as --param entries
 lab run \
-  --runtime=bin:/usr/local/bin/ferret-v0.18 \
+  --runtime=bin:/usr/local/bin/ferret \
   --runtime-param=timeout:30 \
   tests/
 
-# With raw binary flags
+# With Lab-managed I/O policies
+lab run \
+  --runtime=bin:/usr/local/bin/ferret \
+  --policy-fs-root=./fixtures \
+  --policy-fs-read-only \
+  --policy-http-allowed-hosts=api.example.com \
+  tests/
+
+# With additional raw `ferret run` flags
 lab run \
   --runtime=bin:/usr/local/bin/ferret \
-  --runtime-param='flags:["--timeout=60", "--verbose"]' \
+  --runtime-param='flags:["--log-output=none", "--browser-headless"]' \
   tests/
 ```
 
+Only explicitly configured Lab policy values are forwarded, so unset values retain the external CLI's configuration and defaults. Raw policy flags remain available when the corresponding Lab policy option is unset. Lab rejects raw flags that duplicate a managed policy option or conflict with its timeout/limit counterpart.
+
 #### Runtime Comparison Testing
 
 Test against multiple runtime versions:
@@ -794,8 +804,8 @@ These flags apply to `lab run`.
 | `--timeout` | `-t` | `LAB_TIMEOUT` | `30` | Test timeout in seconds |
 | `--cdp` | - | `LAB_CDP` | `http://127.0.0.1:9222` | Chrome DevTools Protocol address |
 | `--reporter` | - | `LAB_REPORTER` | `console` | Output reporter: `console`, `simple` |
-| `--runtime` | `-r` | `LAB_RUNTIME` | - | URL to remote Ferret runtime |
-| `--runtime-param` | `--rp` | `LAB_RUNTIME_PARAM` | - | Parameters for remote runtime |
+| `--runtime` | `-r` | `LAB_RUNTIME` | - | Built-in, HTTP, or Ferret CLI v2 binary runtime |
+| `--runtime-param` | `--rp` | `LAB_RUNTIME_PARAM` | - | Runtime adapter parameters and binary raw flags |
 | `--concurrency` | `-c` | `LAB_CONCURRENCY` | `1` | Number of parallel test executions |
 | `--times` | - | `LAB_TIMES` | `1` | Number of times to run each test |
 | `--attempts` | `-a` | `LAB_ATTEMPTS` | `1` | Number of retry attempts for failed tests |
@@ -808,8 +818,8 @@ These flags apply to `lab run`.
 | `--wait` | `-w` | `LAB_WAIT` | - | Wait for resource availability |
 | `--wait-timeout` | `--wt` | `LAB_WAIT_TIMEOUT` | `5` | Wait timeout in seconds |
 | `--wait-attempts` | - | `LAB_WAIT_ATTEMPTS` | `5` | Number of wait attempts |
-| `--policy-fs-root` | - | `LAB_POLICY_FS_ROOT` | Current working directory | Filesystem root for the built-in runtime |
-| `--policy-fs-read-only` | - | `LAB_POLICY_FS_READ_ONLY` | `false` | Make the built-in runtime filesystem read-only |
+| `--policy-fs-root` | - | `LAB_POLICY_FS_ROOT` | Current working directory | Filesystem root for built-in and binary runtimes |
+| `--policy-fs-read-only` | - | `LAB_POLICY_FS_READ_ONLY` | `false` | Make built-in and binary runtime filesystems read-only |
 | `--policy-http-allowed-schemes` | - | `LAB_POLICY_HTTP_ALLOWED_SCHEMES` | `http,https` | Allowed outbound HTTP URL schemes |
 | `--policy-http-allowed-methods` | - | `LAB_POLICY_HTTP_ALLOWED_METHODS` | `GET,HEAD,POST,PUT,PATCH,DELETE,OPTIONS` | Allowed outbound HTTP methods |
 | `--policy-http-allowed-hosts` | - | `LAB_POLICY_HTTP_ALLOWED_HOSTS` | - | Allowed exact hosts or `host:port` values |
@@ -927,11 +937,11 @@ lab run \
 # Binary runtime with custom flags
 lab run \
   --runtime=bin:/usr/local/bin/ferret \
-  --runtime-param='flags:["--timeout=60", "--verbose"]' \
+  --runtime-param='flags:["--log-output=none", "--browser-headless"]' \
   tests/
 ```
 
-For HTTP runtimes, `path` overrides the run endpoint only. For binary runtimes, `flags` is special and is appended directly to the external binary argv. All other runtime params are still passed as `--param=name:value`.
+For HTTP runtimes, `path` overrides the run endpoint only. For binary runtimes, `flags` is special and is appended after the generated `run` subcommand. All other binary runtime params are passed as `--param=name=<JSON>`. Raw flags that conflict with managed policy options are rejected before execution.
 
 ## Architecture
 

cmd/fs_policy.go

@@ -13,13 +13,13 @@ func fsPolicyFlags(hidden bool) []cli.Flag {
 	return []cli.Flag{
 		&cli.StringFlag{
 			Name:    "policy-fs-root",
-			Usage:   "filesystem root directory for the built-in runtime",
+			Usage:   "filesystem root directory for built-in and binary Ferret runtimes",
 			Sources: cli.EnvVars("LAB_POLICY_FS_ROOT"),
 			Hidden:  hidden,
 		},
 		&cli.BoolFlag{
 			Name:    "policy-fs-read-only",
-			Usage:   "make the built-in runtime filesystem read-only",
+			Usage:   "make built-in and binary Ferret runtime filesystems read-only",
 			Sources: cli.EnvVars("LAB_POLICY_FS_READ_ONLY"),
 			Hidden:  hidden,
 		},
@@ -37,9 +37,7 @@ func fsPolicyFromCommand(cmd *cli.Command) (*runtime.FileSystemPolicy, error) {
 		return nil, nil
 	}
 
-	policy := &runtime.FileSystemPolicy{
-		ReadOnly: cmd.Bool("policy-fs-read-only"),
-	}
+	policy := &runtime.FileSystemPolicy{}
 
 	if rootSet {
 		policy.Root = strings.TrimSpace(cmd.String("policy-fs-root"))
@@ -48,5 +46,10 @@ func fsPolicyFromCommand(cmd *cli.Command) (*runtime.FileSystemPolicy, error) {
 		}
 	}
 
+	if readOnlySet {
+		readOnly := cmd.Bool("policy-fs-read-only")
+		policy.ReadOnly = &readOnly
+	}
+
 	return policy, nil
 }

cmd/fs_policy_test.go

@@ -23,11 +23,22 @@ func TestFSPolicyFlags(t *testing.T) {
 		t.Fatalf("expected trimmed root, got %q", policy.Root)
 	}
 
-	if !policy.ReadOnly {
+	if policy.ReadOnly == nil || !*policy.ReadOnly {
 		t.Fatal("expected read-only policy")
 	}
 }
 
+func TestFSPolicyPreservesExplicitFalse(t *testing.T) {
+	policy, err := runFSPolicyCommand(t, "--policy-fs-read-only=false")
+	if err != nil {
+		t.Fatalf("expected no error, got %v", err)
+	}
+
+	if policy == nil || policy.ReadOnly == nil || *policy.ReadOnly {
+		t.Fatalf("expected explicit false read-only value, got %#v", policy)
+	}
+}
+
 func TestFSPolicyFlagsRemainUnsetByDefault(t *testing.T) {
 	policy, err := runFSPolicyCommand(t)
 	if err != nil {

cmd/helpers.go

@@ -119,16 +119,22 @@ func newRuntime(cmd *cli.Command, params map[string]interface{}) (runtime.Runtim
 		return nil, err
 	}
 
-	httpPolicy, err := httpPolicyOptionsFromCommand(cmd)
+	httpPolicy, err := httpPolicyFromCommand(cmd)
+	if err != nil {
+		return nil, err
+	}
+
+	binaryFlags, err := extractBinaryFlags(params)
 	if err != nil {
 		return nil, err
 	}
 
 	rt, err := runtime.New(runtime.Options{
-		Type:       cmd.String("runtime"),
-		Params:     params,
-		FSPolicy:   fsPolicy,
-		HTTPPolicy: httpPolicy,
+		Type:        cmd.String("runtime"),
+		Params:      params,
+		FSPolicy:    fsPolicy,
+		HTTPPolicy:  httpPolicy,
+		BinaryFlags: binaryFlags,
 	})
 
 	if err != nil {
@@ -138,6 +144,43 @@ func newRuntime(cmd *cli.Command, params map[string]interface{}) (runtime.Runtim
 	return rt, nil
 }
 
+func extractBinaryFlags(params map[string]interface{}) ([]string, error) {
+	value, exists := params["flags"]
+	if !exists {
+		return nil, nil
+	}
+
+	flags, err := toStringSlice(value)
+	if err != nil {
+		return nil, fmt.Errorf("invalid type of flags (expected array of strings): %w", err)
+	}
+
+	delete(params, "flags")
+
+	return flags, nil
+}
+
+func toStringSlice(value any) ([]string, error) {
+	switch values := value.(type) {
+	case []string:
+		return append([]string(nil), values...), nil
+	case []any:
+		result := make([]string, 0, len(values))
+		for _, value := range values {
+			str, ok := value.(string)
+			if !ok {
+				return nil, fmt.Errorf("expected string value")
+			}
+
+			result = append(result, str)
+		}
+
+		return result, nil
+	default:
+		return nil, fmt.Errorf("expected array")
+	}
+}
+
 func locationsFromCommand(cmd *cli.Command) ([]string, bool) {
 	if cmd.NArg() == 0 {
 		locations := cmd.StringSlice("files")

cmd/helpers_test.go

@@ -0,0 +1,42 @@
+package cmd
+
+import (
+	"reflect"
+	"strings"
+	"testing"
+)
+
+func TestExtractBinaryFlags(t *testing.T) {
+	params := map[string]interface{}{
+		"flags": []any{"--log-output=none", "--browser-headless"},
+		"value": 1,
+	}
+
+	flags, err := extractBinaryFlags(params)
+	if err != nil {
+		t.Fatalf("expected no error, got %v", err)
+	}
+
+	if len(flags) != 2 || flags[0] != "--log-output=none" || flags[1] != "--browser-headless" {
+		t.Fatalf("unexpected flags: %#v", flags)
+	}
+	if _, exists := params["flags"]; exists {
+		t.Fatalf("expected flags to be removed from params: %#v", params)
+	}
+}
+
+func TestExtractBinaryFlagsRejectsInvalidValues(t *testing.T) {
+	invalidFlags := []any{"--ok", 1}
+	params := map[string]interface{}{
+		"flags": invalidFlags,
+	}
+
+	_, err := extractBinaryFlags(params)
+	if err == nil || !strings.Contains(err.Error(), "invalid type of flags (expected array of strings)") {
+		t.Fatalf("expected invalid flags error, got %v", err)
+	}
+
+	if value, exists := params["flags"]; !exists || !reflect.DeepEqual(value, invalidFlags) {
+		t.Fatalf("expected invalid flags to remain unchanged, got %#v", params)
+	}
+}