Adds append_dimensions to collectd, supporting custom metric dimensions

[TravisStark]

Description of the issue

Collectd plugin doesn't support append_dimensions, preventing users from adding custom dimensions to collectd metrics for fleet-level aggregation.

Description of changes

• Add append_dimensions to collectd schema
• Call ProcessAppendDimensions in collectd translator
• Add schema validation test and unit tests

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

• Schema validation test (validCollectdConfig.json)
• Unit tests with mocked EC2 metadata
• Manual testing on 2 EC2 instances and verified metrics in CloudWatch with correct dimensions

Agent JSON

{
  "metrics": {
    "namespace": "CollectdAggregationTest",
    "aggregation_dimensions": [
      ["Component", "Environment", "type_instance", "InstanceId"],
      ["Component", "Environment", "type_instance"],
      ["Component", "type_instance"]
    ],
    "metrics_collected": {
      "collectd": {
        "service_address": "udp://127.0.0.1:25826",
        "name_prefix": "collectd_",
        "collectd_auth_file": "/etc/collectd/auth_file",
        "collectd_security_level": "encrypt",
        "metrics_aggregation_interval": 60,
        "append_dimensions": {
          "InstanceId": "${aws:InstanceId}",
          "InstanceType": "${aws:InstanceType}",
          "Component": "RDSRialtoRouter",
          "Environment": "Test"
        }
      }
    }
  }
}

YAML Config:

exporters:
    awscloudwatch:
        force_flush_interval: 1m0s
        max_datums_per_call: 1000
        max_values_per_datum: 150
        middleware: agenthealth/metrics
        namespace: CollectdAggregationTest
        region: us-east-1
        resource_to_telemetry_conversion:
            enabled: true
        rollup_dimensions:
            - - Component
              - Environment
              - type_instance
              - InstanceId
            - - Component
              - Environment
              - type_instance
            - - Component
              - type_instance
extensions:
    agenthealth/metrics:
        is_usage_data_enabled: true
        stats:
            operations:
                - PutMetricData
            usage_flags:
                mode: EC2
                region_type: EC2M
    agenthealth/statuscode:
        is_status_code_enabled: true
        is_usage_data_enabled: true
        stats:
            usage_flags:
                mode: EC2
                region_type: EC2M
    entitystore:
        mode: ec2
        region: us-east-1
processors:
    awsentity/service/telegraf:
        entity_type: Service
        platform: ec2
        scrape_datapoint_attribute: true
receivers:
    telegraf_socket_listener:
        collection_interval: 1m0s
        initial_delay: 1s
        timeout: 0s
service:
    extensions:
        - agenthealth/metrics
        - agenthealth/statuscode
        - entitystore
    pipelines:
        metrics/hostCustomMetrics:
            exporters:
                - awscloudwatch
            processors:
                - awsentity/service/telegraf
            receivers:
                - telegraf_socket_listener
    telemetry:
        logs:
            level: info
            output_paths:
                - /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

TOML Config:

[agent]
  collection_jitter = "0s"
  debug = false
  flush_interval = "1s"
  interval = "60s"
  logfile = "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log"
  metric_batch_size = 1000
  metric_buffer_limit = 10000

[inputs]
  [[inputs.socket_listener]]
    collectd_auth_file = "/etc/collectd/auth_file"
    collectd_security_level = "encrypt"
    collectd_typesdb = ["/usr/share/collectd/types.db"]
    data_format = "collectd"
    name_prefix = "collectd_"
    service_address = "udp://127.0.0.1:25826"
    [inputs.socket_listener.tags]
      Component = "RDSRialtoRouter"
      Environment = "Test"
      InstanceId = "i-076991ec61f00c043"
      InstanceType = "t3.medium"
      "aws:AggregationInterval" = "60s"

[outputs]
  [[outputs.cloudwatch]]

Collectd config

LoadPlugin syslog
LoadPlugin network
LoadPlugin cpu
LoadPlugin memory
LoadPlugin protocols

<Plugin network>
  <Server "127.0.0.1" "25826">
    SecurityLevel Encrypt
    Username "collectd_user"
    Password "a3e01e60-ba4c-459d-9559-5ead520b7f86"
  </Server>
</Plugin>

<Plugin protocols>
  Value "TcpExt:TCPAbortOnTimeout"
  Value "TcpExt:ListenOverflows"
  Value "TcpExt:ListenDrops"
  Value "TcpExt:TCPTimeouts"
</Plugin>

Requirements

Before commiting your code, please do the following steps.

1. Run make fmt and make fmt-sh
2. Run make lint

---

Integration Tests

To run integration tests against this PR, add the ready for testing label.

[okankoAMZ]

small nitpicks but looks good!

[jefchien]

Can we add this to the sampleConfigs? Want to make sure this sets the TOML correctly for collectd.

[TravisStark]

Can we add this to the sampleConfigs? Want to make sure this sets the TOML correctly for collectd.

Added sample config: collectd_append_dimensions_linux

[jefchien]

If we want this to be used in the tests, we need to update translator/cmdutil/translatorutil_test.go.

[TravisStark]

Good catch, updated.

[github-actions]

This PR was marked stale due to lack of activity.

[okankoAMZ]

does this require a key by anyway?

[TravisStark]

I believe the auth_file is the key