Ha logs

[tiny-pangolin]

This PR will address the logs section #159 traces will likely need to be addressed in a separate issue since it will require otel collector or be delayed until vtagent is released

[cubic-dev-ai]

2 issues found across 7 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="playbooks/vlcluster-ha.yml">

<violation number="1" location="playbooks/vlcluster-ha.yml:67">
P2: Log ingestion depends on vmauth being local to each vlagent because `remoteWrite.url` is hardcoded to `127.0.0.1`. Using the vmauth inventory host(s) here (or adding an explicit co-location assert) avoids silent ingestion failures when groups are split.</violation>
</file>

<file name="roles/vlagent/defaults/main.yml">

<violation number="1" location="roles/vlagent/defaults/main.yml:15">
P2: The new vlagent buffer directory is placed in /tmp, but the vlagent systemd unit is `PrivateTmp=yes`. That means the running service gets a private, per-invocation tmpfs on /tmp: the `/tmp/vlagent` directory the configure task creates as root is never visible to the service, and any buffered logs are discarded on every service restart and on host reboot (systemd-tmpfiles also ages out /tmp). Since `remoteWrite.tmpDataPath` exists precisely to hold logs locally while the remote target is down, this defeats the HA buffering this PR is adding — a remote outage plus a service restart/reboot would silently drop queued logs. Consider using a persistent path outside /tmp (e.g. under /var/lib) so buffered data survives, and verify the service can actually reach it given the unit's sandboxing.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Log ingestion depends on vmauth being local to each vlagent because remoteWrite.url is hardcoded to 127.0.0.1. Using the vmauth inventory host(s) here (or adding an explicit co-location assert) avoids silent ingestion failures when groups are split.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At playbooks/vlcluster-ha.yml, line 67:

<comment>Log ingestion depends on vmauth being local to each vlagent because `remoteWrite.url` is hardcoded to `127.0.0.1`. Using the vmauth inventory host(s) here (or adding an explicit co-location assert) avoids silent ingestion failures when groups are split.</comment>

<file context>
@@ -0,0 +1,108 @@
+      - cluster_a
+      - cluster_b
+    vlagent_service_args:
+      remoteWrite.url: "{{ vl_ha_clusters | map('regex_replace', '^(.+)$', 'http://127.0.0.1:8427/\\1/internal/insert') | list }}"
+      remoteWrite.tmpDataPath: "{{ vlagent_tmp_data_path }}"
+  roles:
</file context>

[cubic-dev-ai]

P2: The new vlagent buffer directory is placed in /tmp, but the vlagent systemd unit is PrivateTmp=yes. That means the running service gets a private, per-invocation tmpfs on /tmp: the /tmp/vlagent directory the configure task creates as root is never visible to the service, and any buffered logs are discarded on every service restart and on host reboot (systemd-tmpfiles also ages out /tmp). Since remoteWrite.tmpDataPath exists precisely to hold logs locally while the remote target is down, this defeats the HA buffering this PR is adding — a remote outage plus a service restart/reboot would silently drop queued logs. Consider using a persistent path outside /tmp (e.g. under /var/lib) so buffered data survives, and verify the service can actually reach it given the unit's sandboxing.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At roles/vlagent/defaults/main.yml, line 15:

<comment>The new vlagent buffer directory is placed in /tmp, but the vlagent systemd unit is `PrivateTmp=yes`. That means the running service gets a private, per-invocation tmpfs on /tmp: the `/tmp/vlagent` directory the configure task creates as root is never visible to the service, and any buffered logs are discarded on every service restart and on host reboot (systemd-tmpfiles also ages out /tmp). Since `remoteWrite.tmpDataPath` exists precisely to hold logs locally while the remote target is down, this defeats the HA buffering this PR is adding — a remote outage plus a service restart/reboot would silently drop queued logs. Consider using a persistent path outside /tmp (e.g. under /var/lib) so buffered data survives, and verify the service can actually reach it given the unit's sandboxing.</comment>

<file context>
@@ -12,9 +12,12 @@ vlagent_download_url: "{{ vlagent_repo_url }}/releases/download/{{ vlagent_versi
 vlagent_system_user: "vic_vl_agent"
 vlagent_system_group: "{{ vlagent_system_user }}"
 
+vlagent_tmp_data_path: "/tmp/vlagent"
+
 vlagent_remote_write_host: "http://localhost:9428"
</file context>