Test stack producer consumer phases #43
Conversation
… to visualise the test consumer and producer phases
There was a problem hiding this comment.
Pull request overview
This PR enhances the test-stack's producer and consumer scripts with Grafana annotations to mark different operational phases, making it easier to correlate Kafka lag patterns with producer/consumer behavior in the Grafana dashboard.
Changes:
- Added annotation functionality to producer and consumer scripts to create timestamped phase markers in Grafana
- Updated Grafana dashboard configuration to display producer and consumer phase annotations with distinct colors
- Enabled anonymous admin access to Grafana to allow producer/consumer scripts to create annotations without authentication
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| test-stack/scripts/producer.sh | Added annotate() function and calls to mark each of the 6 producer phases (low rate, bursts, pause, multi-topic) |
| test-stack/scripts/consumer.sh | Added annotate() function and calls to mark each of the 7 consumer phases (slow, medium, fast catch-up, pause, burst) |
| test-stack/grafana/provisioning/dashboards/kafka-lag.json | Added annotation configurations for producer and consumer phases with distinct colors, updated dashboard title and UID |
| test-stack/docker-compose.yml | Enabled anonymous admin access to Grafana to support annotation API calls from scripts |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| annotate() { | ||
| local text="$1" | ||
| local tags="$2" | ||
| curl -s -X POST "$GRAFANA_URL/api/annotations" \ | ||
| -H "Content-Type: application/json" \ | ||
| -d "{\"text\": \"$text\", \"tags\": [$tags]}" \ | ||
| 2>/dev/null || true # Don't fail if Grafana isn't ready | ||
| } |
There was a problem hiding this comment.
The annotate function is vulnerable to JSON injection. The text parameter is directly interpolated into the JSON payload without proper escaping. If the text contains special characters like double quotes, backslashes, or newlines, it will break the JSON structure or could be exploited to inject arbitrary JSON.
Consider using a tool like jq to properly construct the JSON payload, or at minimum, escape special characters in the text parameter before interpolation.
| # Function to create Grafana annotation for phase visibility | ||
| annotate() { | ||
| local text="$1" | ||
| local tags="$2" | ||
| curl -s -X POST "$GRAFANA_URL/api/annotations" \ | ||
| -H "Content-Type: application/json" \ | ||
| -d "{\"text\": \"$text\", \"tags\": [$tags]}" \ |
There was a problem hiding this comment.
The annotate function is vulnerable to JSON injection. The text parameter is directly interpolated into the JSON payload without proper escaping. If the text contains special characters like double quotes, backslashes, or newlines, it will break the JSON structure or could be exploited to inject arbitrary JSON.
Consider using a tool like jq to properly construct the JSON payload, or at minimum, escape special characters in the text parameter before interpolation.
| # Function to create Grafana annotation for phase visibility | |
| annotate() { | |
| local text="$1" | |
| local tags="$2" | |
| curl -s -X POST "$GRAFANA_URL/api/annotations" \ | |
| -H "Content-Type: application/json" \ | |
| -d "{\"text\": \"$text\", \"tags\": [$tags]}" \ | |
| # Helper to escape strings for JSON context | |
| json_escape() { | |
| local str="$1" | |
| # Escape backslashes and double quotes | |
| str=${str//\\/\\\\} | |
| str=${str//\"/\\\"} | |
| # Escape newlines and carriage returns | |
| str=${str//$'\n'/\\n} | |
| str=${str//$'\r'/\\r} | |
| printf '%s' "$str" | |
| } | |
| # Function to create Grafana annotation for phase visibility | |
| annotate() { | |
| local text="$1" | |
| local tags="$2" | |
| local escaped_text | |
| escaped_text=$(json_escape "$text") | |
| curl -s -X POST "$GRAFANA_URL/api/annotations" \ | |
| -H "Content-Type: application/json" \ | |
| -d "{\"text\": \"$escaped_text\", \"tags\": [$tags]}" \ |
| "title": "Kafka Consumer Lag", | ||
| "uid": "kafka-lag-dashboard", | ||
| "title": "Kafka Consumer Lag (with Phase Annotations)", | ||
| "uid": "kafka-lag-dashboard-test", |
There was a problem hiding this comment.
Changing the dashboard UID from "kafka-lag-dashboard" to "kafka-lag-dashboard-test" may break existing references, bookmarks, or links to this dashboard. If this is intentional to create a separate test dashboard while keeping the original, consider documenting this change. If this is meant to replace the existing dashboard, the UID should remain unchanged to preserve continuity.
| "uid": "kafka-lag-dashboard-test", | |
| "uid": "kafka-lag-dashboard", |
No description provided.