Expand README and fix CI

Author: vicentereig

.github/workflows/ci.yml

@@ -4,31 +4,23 @@ on:
   push:
     branches:
       - main
-
   pull_request:
 
 jobs:
   test:
     runs-on: ubuntu-latest
-    name: Ruby ${{ matrix.ruby }}
-    strategy:
-      matrix:
-        ruby:
-          - '3.3'
-          - '3.2'
-          - '3.1'
-
     steps:
       - uses: actions/checkout@v4
 
-      - name: Set up Ruby
+      - name: Set up Ruby (rbenv)
         uses: ruby/setup-ruby@v1
         with:
-          ruby-version: ${{ matrix.ruby }}
+          ruby-version-file: .ruby-version
+          bundler: '2.6.9'
           bundler-cache: true
 
       - name: Run tests
-        run: bundle exec rspec spec/unit/
+        run: bundle exec rspec spec/unit
 
       - name: Check gem can be built
         run: gem build lf-cli.gemspec

.gitignore

@@ -27,7 +27,6 @@
 # for a library or gem, you might want to ignore these files since the code is
 # intended to run in multiple environments; otherwise, check them in:
 Gemfile.lock
-.ruby-version
 .ruby-gemset
 
 # unless supporting rvm < 1.11.0 or doing something fancy, ignore this:

.ruby-version

@@ -0,0 +1 @@
+3.4.5

README.md

@@ -4,7 +4,8 @@
 
 A powerful command-line interface for querying and analyzing Langfuse LLM observability data. Built with Ruby and designed for developers who prefer working in the terminal.
 
-[![Gem Version](https://badge.fury.io/rb/lf-cli.svg)](https://badge.fury.io/rb/lf-cli)
+[![Gem Version](https://img.shields.io/gem/v/lf-cli)](https://rubygems.org/gems/lf-cli)
+[![Total Downloads](https://img.shields.io/gem/dt/lf-cli)](https://rubygems.org/gems/lf-cli)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
 ## Disclaimer
@@ -294,6 +295,192 @@ Configuration is loaded in this order (highest to lowest priority):
 3. Config file (`~/.langfuse/config.yml`)
 4. Defaults
 
+## LLM Reference (llms-full)
+
+This section mirrors the depth of a `llms-full.txt` so AI assistants can answer detailed questions about `lf-cli` without inspecting the source.
+
+### CLI Metadata
+
+| Item | Details |
+| --- | --- |
+| Binary names | `lf` (preferred), `langfuse` (legacy alias used in docs) |
+| Entry point | `bin/lf` → `Langfuse::CLI::Main.start(ARGV)` |
+| Ruby compatibility | Ruby >= 2.7.0 (same as gemspec requirement) |
+| Default API host | `https://cloud.langfuse.com` |
+| Config path | `~/.langfuse/config.yml` (`0600` permissions) |
+| Source namespace | `Langfuse::CLI` with command classes under `lib/langfuse/cli/commands` |
+| HTTP stack | Faraday + JSON middleware, 2s open/read timeouts, retry with exponential backoff on `429/5xx` |
+| Logging | Set `DEBUG=1` to enable Faraday request/response logging |
+
+### Authentication & Configuration
+
+1. Credentials (`public_key`, `secret_key`) plus `host` are mandatory for all API calls.
+2. Resolution order: CLI flags → environment variables → profile in `~/.langfuse/config.yml` → defaults.
+3. Profiles let you store multiple environments (e.g., `default`, `staging`, `production`) inside the same YAML file.
+4. The `config setup` command validates keys before saving by hitting `/api/public/traces` with `limit=1`.
+
+**Environment variable matrix**
+
+| Variable | Purpose | Required | Notes |
+| --- | --- | --- | --- |
+| `LANGFUSE_PUBLIC_KEY` | Public API key | ✅ | Required for non-interactive `config setup` and all other commands if no profile is configured. |
+| `LANGFUSE_SECRET_KEY` | Secret API key | ✅ | Same priority rules as the public key. |
+| `LANGFUSE_HOST` | Override API base URL | Optional | Use for self-hosted Langfuse instances. |
+| `LANGFUSE_PROFILE` | Profile name | Optional | Overrides the profile selected via `-P/--profile`. |
+| `LANGFUSE_PROJECT_NAME` | Used by `config setup` to show the correct settings URL. | Optional | Only needed for UX hints. |
+| `DEBUG` | When set to `1`, logs Faraday requests/responses to stdout. | Optional | Useful for diagnosing API issues. |
+
+### Global Flags & Behavior
+
+| Flag | Description | Notes |
+| --- | --- | --- |
+| `-P, --profile PROFILE` | Selects a saved profile. | Defaults to `default`. |
+| `--public-key KEY` / `--secret-key KEY` | Inject credentials without touching config files. | Highest priority source. |
+| `--host URL` | Override Langfuse host. | Combine with `--profile` to temporarily test another region. |
+| `-f, --format FORMAT` | `table` (default), `json`, `csv`, `markdown`. | Applies to every command; CSV/Markdown require structured arrays. |
+| `-o, --output PATH` | Write output to a file. | Respects format; prints “Output written…” when `--verbose`. |
+| `-l, --limit N` | Caps number of records pulled per command. | Pagination helper, defaults to API `limit` (50) when omitted. |
+| `-p, --page N` | Start from an explicit page. | Useful when you know an offset. |
+| `--from`, `--to` | ISO 8601 or natural language timestamps. | Natural language parsing uses `chronic` if installed; otherwise the string is sent as-is. |
+| `-v, --verbose` | Prints extra logs (e.g., file paths). | Some commands emit status lines prefixed with emojis. |
+| `--no-color` | Forces monochrome table output. | Forwarded to formatters that support color. |
+
+Pagination strategy: the client keeps fetching pages until it collects the requested `limit` or no more pages remain. `limit` therefore caps the total combined size, not per-page size.
+
+### Output & Files
+
+- `table` renders ASCII tables via `Formatters::TableFormatter`.
+- `json` streams `JSON.pretty_generate` for direct piping to `jq`.
+- `csv` and `markdown` use dedicated formatters and require array-like data (single hashes are wrapped automatically).
+- `--output` writes the formatted string verbatim; combine with `--format json` for scripts.
+- Use `lf ... --format json | jq ...` for automation recipes.
+
+### Command Reference
+
+Each command inherits the global flags above. API errors exit with status code `1`.
+
+#### `config` (profile management)
+
+| Subcommand | Synopsis | Notes |
+| --- | --- | --- |
+| `lf config setup` | Interactive wizard; supports env-variable non-interactive mode. | Tests credentials before saving. |
+| `lf config set PROFILE --public-key ... --secret-key ... [--host ...]` | Writes/updates a profile directly. | Does not hit the API. |
+| `lf config show [PROFILE]` | Prints the resolved profile (keys masked). | Reads from YAML + ENV. |
+| `lf config list` | Shows every profile name plus masked public key/host. | Warns if file missing. |
+
+#### `traces`
+
+| Subcommand | Purpose |
+| --- | --- |
+| `lf traces list` | Lists traces with filters/pagination. |
+| `lf traces get TRACE_ID [--with-observations]` | Fetches a single trace. The `--with-observations` flag is accepted for forward compatibility but currently behaves the same as the default API payload. |
+
+`traces list` options:
+
+| Flag | Type | Description |
+| --- | --- | --- |
+| `--name NAME` | String | Filter by trace name. |
+| `--user-id USER_ID` | String | Filter by Langfuse user identifier. |
+| `--session-id SESSION_ID` | String | Filter by session. |
+| `--tags TAG1 TAG2` | Array | Matches traces containing all provided tags. |
+| `--from`, `--to` | String | Time boundaries; accepts ISO 8601 or relative strings. |
+| `--limit`, `--page` | Numeric | Override pagination per request. |
+
+Sample workflow:
+
+```bash
+latest_trace_id=$(lf traces list --format json --limit 1 | jq -r '.[0].id')
+lf traces get "$latest_trace_id" --format json > trace.json
+```
+
+#### `sessions`
+
+| Subcommand | Purpose |
+| --- | --- |
+| `lf sessions list` | Enumerates sessions. |
+| `lf sessions show SESSION_ID [--with-traces]` | Shows a session and optionally its traces (flag reserved for future enrichments). |
+
+Options mirror trace pagination: `--from`, `--to`, `--limit`, `--page`.
+
+#### `observations`
+
+| Subcommand | Purpose |
+| --- | --- |
+| `lf observations list` | Lists generations, spans, or events. |
+| `lf observations get OBSERVATION_ID` | Fetches a single observation. |
+
+`list` filters:
+
+| Flag | Values | Description |
+| --- | --- | --- |
+| `--type` | `generation`, `span`, `event` | Restrict to an observation type. |
+| `--trace-id` | Trace ID | Only observations under a specific trace. |
+| `--name` | String | Filter by observation name. |
+| `--user-id` | String | Filter by associated user. |
+| `--from`, `--to`, `--limit`, `--page` | As described earlier. |
+
+#### `scores`
+
+| Subcommand | Purpose |
+| --- | --- |
+| `lf scores list` | Lists evaluation scores. |
+| `lf scores get SCORE_ID` | Fetches a single score document. |
+
+Filters: `--name`, `--from`, `--to`, `--limit`, `--page`.
+
+#### `metrics`
+
+Single subcommand: `lf metrics query`.
+
+Required flags:
+
+| Flag | Allowed values | Description |
+| --- | --- | --- |
+| `--view` | `traces`, `observations`, `scores-numeric`, `scores-categorical` | Which metrics view to query. |
+| `--measure` | `count`, `latency`, `value`, `tokens`, `cost` | Base metric. |
+| `--aggregation` | `count`, `sum`, `avg`, `p50`, `p95`, `p99`, `min`, `max`, `histogram` | Aggregation function. |
+
+Optional flags:
+
+| Flag | Description |
+| --- | --- |
+| `--dimensions field1 field2` | Array of dimension field names (e.g., `name`, `userId`, `sessionId`, `model`). |
+| `--from`, `--to` | Time range. |
+| `--granularity` | `minute`, `hour`, `day`, `week`, `month`, `auto`. Controls the time bucket. |
+| `--limit` | Defaults to `100` for metrics; caps the number of buckets/rows returned. |
+
+The CLI builds a payload matching the Langfuse metrics API (`metrics` array, `timeDimension` etc.) via `Langfuse::CLI::Types::MetricsQuery`.
+
+### Data Model Cheat Sheet
+
+- **Trace**: `id`, `name`, `userId`, `sessionId`, `timestamp`, `durationMs`, `tags[]`, `metadata` (object), `observations[]` (optional when using `get`), `scores[]`.
+- **Session**: `id`, `userId`, `name`, `createdAt`, `updatedAt`, `traceIds[]`, `metadata`.
+- **Observation**: `id`, `traceId`, `type` (`generation/span/event`), `name`, `status`, `model`, `input`, `output`, `metrics` (latency, usage), `level`, `parentObservationId`.
+- **Score**: `id`, `name`, `value` (number/string), `type` (`numeric`/`categorical`), `traceId`, `observationId`, `timestamp`, `metadata`, `comment`.
+- **Metrics response**: Usually `{ "data": [ { "dimensions": {...}, "metrics": {...} } ], "meta": {...} }`. The CLI automatically unwraps `data` before formatting.
+
+Fields are passed through verbatim from the Langfuse Public API; the CLI never renames keys.
+
+### Error Handling & Exit Codes
+
+- Success exits with code `0`.
+- Any `Langfuse::CLI::Client::*Error` results in exit code `1` after printing a human-readable message.
+- Specific messages:
+  - `Authentication Error` for `401`.
+  - `Rate limit exceeded` for `429`.
+  - `Trace/session/... not found` for `404`.
+  - `Request timed out` when Faraday raises a timeout (usually after 2s).
+- Use `--verbose` or `DEBUG=1` for deeper context (e.g., stack traces, Faraday logs).
+
+### Troubleshooting & Automation Tips
+
+- **Network issues**: verify `LANGFUSE_HOST` by running `lf config show` and hitting `/health` with `curl`.
+- **Time parsing**: install the `chronic` gem to enable natural language ranges; otherwise pass ISO 8601 timestamps.
+- **CSV exports**: always provide `--output file.csv` to avoid large terminal dumps.
+- **Scripting**: prefer `--format json` to keep machine-readable structures. Most commands return arrays, so piping to `jq '.[].id'` works consistently.
+- **Profiles**: store CI credentials under `LANGFUSE_PROFILE=ci` and load them via `lf ... -P ci` to keep human/dev credentials untouched.
+- **Retries**: built-in Faraday retry middleware already backs off (`max: 3`). For long-running scripts, wrap commands with shell retries instead of adding loops inside the CLI.
+
 ## Development
 
 ### Setup

lib/langfuse/cli/commands/config.rb

@@ -88,14 +88,14 @@ def setup
           rescue Client::TimeoutError => e
             prompt.error("Connection test failed: #{e.message}")
             prompt.error("The host '#{host}' may be incorrect or unreachable.")
-            exit 1
+            raise_cli_error("Connection test failed: #{e.message}")
           rescue Client::AuthenticationError => e
             prompt.error("Connection test failed: #{e.message}")
             prompt.error("Please check your credentials and try again.")
-            exit 1
+            raise_cli_error("Connection test failed: #{e.message}")
           rescue Client::APIError => e
             prompt.error("Connection test failed: #{e.message}")
-            exit 1
+            raise_cli_error("Connection test failed: #{e.message}")
           end
         end
 
@@ -170,6 +170,10 @@ def mask_key(key)
 
           "#{key[0..7]}#{'*' * (key.length - 8)}"
         end
+
+        def raise_cli_error(message)
+          raise Langfuse::CLI::Error, message
+        end
       end
     end
   end

lib/langfuse/cli/commands/metrics.rb

@@ -79,11 +79,9 @@ def query
           output_data = result.is_a?(Hash) && result['data'] ? result['data'] : result
           output_result(output_data)
         rescue Client::AuthenticationError => e
-          puts "Authentication Error: #{e.message}"
-          exit 1
+          raise_cli_error("Authentication Error: #{e.message}")
         rescue Client::APIError => e
-          puts "Error: #{e.message}"
-          exit 1
+          raise_cli_error("Error: #{e.message}")
         end
 
         private
@@ -187,6 +185,10 @@ def parent_options
             {}
           end
         end
+
+        def raise_cli_error(message)
+          raise Langfuse::CLI::Error, message
+        end
       end
     end
   end

lib/langfuse/cli/commands/observations.rb

@@ -57,23 +57,19 @@ def list
           observations = client.list_observations(filters)
           output_result(observations)
         rescue Client::AuthenticationError => e
-          puts "Authentication Error: #{e.message}"
-          exit 1
+          raise_cli_error("Authentication Error: #{e.message}")
         rescue Client::APIError => e
-          puts "Error: #{e.message}"
-          exit 1
+          raise_cli_error("Error: #{e.message}")
         end
 
         desc 'get OBSERVATION_ID', 'Get a specific observation'
         def get(observation_id)
           observation = client.get_observation(observation_id)
           output_result(observation)
-        rescue Client::NotFoundError => e
-          puts "Error: Observation not found - #{observation_id}"
-          exit 1
+        rescue Client::NotFoundError
+          raise_cli_error("Observation not found - #{observation_id}")
         rescue Client::APIError => e
-          puts "Error: #{e.message}"
-          exit 1
+          raise_cli_error("Error: #{e.message}")
         end
 
         private
@@ -155,6 +151,10 @@ def parent_options
             {}
           end
         end
+
+        def raise_cli_error(message)
+          raise Langfuse::CLI::Error, message
+        end
       end
     end
   end

lib/langfuse/cli/commands/scores.rb

@@ -21,23 +21,19 @@ def list
           scores = client.list_scores(filters)
           output_result(scores)
         rescue Client::AuthenticationError => e
-          puts "Authentication Error: #{e.message}"
-          exit 1
+          raise_cli_error("Authentication Error: #{e.message}")
         rescue Client::APIError => e
-          puts "Error: #{e.message}"
-          exit 1
+          raise_cli_error("Error: #{e.message}")
         end
 
         desc 'get SCORE_ID', 'Get a specific score'
         def get(score_id)
           score = client.get_score(score_id)
           output_result(score)
-        rescue Client::NotFoundError => e
-          puts "Error: Score not found - #{score_id}"
-          exit 1
+        rescue Client::NotFoundError
+          raise_cli_error("Score not found - #{score_id}")
         rescue Client::APIError => e
-          puts "Error: #{e.message}"
-          exit 1
+          raise_cli_error("Error: #{e.message}")
         end
 
         private
@@ -116,6 +112,10 @@ def parent_options
             {}
           end
         end
+
+        def raise_cli_error(message)
+          raise Langfuse::CLI::Error, message
+        end
       end
     end
   end