source_lag_time_seconds get large number

[zzmg]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

source： kubernetes_logs
out ：kafka
we find lag is very big，I would like to ask, what is the possible reason? thanks

Configuration

in_k8s:
    auto_partial_merge: true
    data_dir: /var/lib/vector
    delay_deletion_ms: 60000
    exclude_paths_glob_patterns:
      - '**/vector/**'
      - '**/istio-init/**'
      - '**/filebeat/**'
      - '**/vmagent/**'
      - '**/*.gz'
      - '**/*.log.*'
      - '**/vector-logfilter/**'
    fingerprint_lines: 1
    glob_minimum_cooldown_ms: 60000
    max_line_bytes: 32768
    max_read_bytes: 4096
    self_node_name: '${VECTOR_SELF_NODE_NAME}'
    timezone: local
    type: kubernetes_logs
    ignore_older_secs: 1800
    use_apiserver_cache: true

Version

0.33.1

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[zzmg]

This question may be related to this question. #19293
In our scenario, in_k8s does not fail very frequently. It should be because the lag is very large, which causes the vectors on some nodes not to have time to consume the logs. When they go to consume, it happens that the pod to which this log belongs has been rolled out.

[jszwedko]

This would indicate that the source is falling behind (or, at least, is reading old logs). This can happen for a couple of reasons:

• There is backpressure being applied from one of the components downstream of the source
• The kubernetes_logs source simply isn't keeping up with reading

To validate the first one you could look at https://vector.dev/docs/reference/configuration/sources/internal_metrics/#utilization for the components to identify if any are the bottleneck. If none are, then it seems like that the source isn't keeping up in which case you could try tweaking https://vector.dev/docs/reference/configuration/sources/kubernetes_logs/#max_read_bytes to see if you get better performance.

[jszwedko]

One other thing that jumps out as a potential issue here is differences between timezones. Do you know if the k8s pod logs will be in the same or different timezone as the Vector process?

[zzmg]

in the same timezones，maybe backpressure ，I increase cpu ，lag reduced a lot

[huan89983]

I encountered the same issue as well, but increasing the CPU resources didn't help at all! Please see the details below: #25385

Get help!