Client.Timeout exceeded while awaiting headers, context deadline exceeded?

[dan-valaratomics]

Relevant telegraf.conf

[agent]
  interval = "1s"
  flush_interval = "7s"
  flush_jitter = "3s"
  
[[outputs.influxdb_v2]]
  #urls = ["redacted"] #vlan 20
  urls = ["redacted"] # vlan 10 moved to same subnet as host for testing
  token = "$INFLUX_TOKEN"
  organization = "redacted"
  bucket = "Hypervisors"
  insecure_skip_verify = true
  #metric_batch_size = 1000
  #metric_buffer_limit = 50000
  #flush_jitter = "2s"
  #timeout = "30s"

[[inputs.proxmox]] 
  base_url = "redacted/api2/json"
  api_token = "redacted"
  node_name = "redacted"
  insecure_skip_verify = true
  response_timeout = "5s"

# ---- host metrics for the Proxmox node ----
[[inputs.cpu]]
  percpu       = true
  totalcpu     = true
  report_active = true

[[inputs.mem]]
[[inputs.swap]]

[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "overlay", "squashfs", "rpc_pipefs", "sysfs", "cgroup2", "proc"]

[[inputs.diskio]]
[[inputs.net]]
[[inputs.system]]
[[inputs.kernel]]
[[inputs.processes]]
[[inputs.netstat]]

# ---- ZFS pool metrics ----
[[inputs.zfs]]
  poolMetrics = true
  interval = "30s"

 # ---- S.M.A.R.T. disk health ---- 
#[[inputs.smart]]
#  path_smartctl = "/usr/sbin/smartctl"
#  attributes = true
#  interval = "60s"
#  devices = [
#    "/dev/sda -d sat+scsi",
#    "/dev/sdb -d sat+scsi",
#    "/dev/sdc -d sat+scsi",
#   "/dev/sdd -d sat+scsi",
#    "/dev/sde -d sat+scsi",
#    "/dev/sdf -d sat+scsi",
#    "/dev/sdg -d sat+scsi",
#    "/dev/sdh -d sat+scsi",
#    "/dev/sdi -d sat+scsi",
#    "/dev/sdj -d sat+scsi",
#    "/dev/sdk -d sat+scsi",
#    "/dev/sdl -d sat+scsi",
#    "/dev/sdm -d sat+scsi",
#    "/dev/sdn -d sat+scsi",
#    "/dev/sdo -d sat+scsi",
#    "/dev/sdp -d sat+scsi"
#  ]
# - - - -

[[inputs.exec]]
  commands = ["/usr/local/bin/smart-metrics.sh"]
  timeout = "60s"
  interval = "60s"
  data_format = "influx"

Logs from Telegraf

Apr 09 09:18:41 redacted telegraf [646129] : 2026-04-09T16:18:41Z I! Loaded secretstores:
Apr 09 09:18:41 redacted telegraf [646129]

2026-04-09T16:18:41Z I! Loaded outputs: influxdb v2

Apr 09 09:18:41 redacted telegraf [646129]

2026-04-09T16:18:41Z I! Tags enabled: host=redacted

Apr 09 09:18:41 redacted telegraf [646129]

2026-04-09T16:18:41Z I!
2026-04-09T16:18:41Z W!

[agent ] Config: Interval:1mOs, Quiet:false, Hostname: "redacted", Flush Interval:1m0s

Apr 09 09:18:41 redacted telegraf [646129]

[agent ] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default be>

Apr 09 09:18:41 redacted telegraf [646129] : 2026-04-09T16:18:41Z w!

[inputs.smart] nvme not found: verify that nvme is installed and it is in your PATH (or specified in config) to gather vendor specific attribute>
Apr 09 09:18:41 redacted systemd[1]: Started telegraf.service - Telegraf.
Apr 09 09:21:11 redacted telegraf [646129] : 2026-04-09T16:21:11Z E!

[outputs. influxdb_v2] Post "redacted/api/v2/write?bucket=Hypervisors&org=redacted: context deadline exceeded (Clienl

Apr 09 09:21:11 scada-a telegraf [646129]

2026-04-09T16:21:11Z E!

[outputs. influxdb_v2] When writing to [redacted/api/v2/write] : Post "redacted:8086/api/v2/write?bucket=Hypervisorssor
Apr 09 09:21:11 redacted telegraf [646129] : 2026-04-09T16:21:11Z E!

[agent] Error writing to outputs.influxdb_v2: Post "redacted/api/v2/write?bucket=Hypervisors&org-redacted": context:

------------------------------------------------------------------------------------------------------------
Initial output above , after a few hours below starts spitting out and responses are delayed
------------------------------------------------------------------------------------------------------------


^C
root@redacted :~ # journalctl -u telegraf -f
Apr 18 11:06:02 redacted telegraf [2510617] : 2026-04-18T17:06:022 E! [outputs.influxdb_v2] When writing to [redacted:8086/api/v2/write] : Post "hredacted:8086/api/v2/write?bucket:
ypervisors&org-redacted": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Apr 18 11:06:02 redacted telegraf [2510617] : 2026-04-18T17:06:022 E! [agent] Error writing to outputs. influxdb_v2: Post "redacted:8086/api/v2/write?bucket=Hypervisors&org-redacted": context deadline exceeded (Client. Timeout exceeded while awaiting headers)
Apr 18 11:06:07 redacted telegraf [2510617] : 2026-04-18T17:06:072 w! [agent] ["outputs.influxdb_v2"] did not complete within its flush interval
Apr 18 11:06:07 redacted telegraf [2510617] : 2026-04-18T17:06:072 E!
ceeded (Client.Timeout exceeded while awaiting headers)
Apr 18 11:06:07 redacted telegraf [2510617] : 2026-04-18T17:06:072 E! [outputs. influxdb_v2] When writing to [redacted:8086/api/v2/write] : Post "redacted:8086/api/v2/write?bucket=
ypervisors&org=ValarAtomics-Ward250": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Apr 18 11:06:07 redacted telegraf [2510617] : 2026-04-18T17:06:072 E! [agent] Error writing to outputs.influxdb_v2: Post "redacted:8086/api/v2/write?bucket=Hypervisors&org=redacted": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Apr 18 11:06:12 redacted telegraf [2510617] : 2026-04-18T17:06:12Z E! [outputs.influxdb_v2]
ceeded (Client.Timeout exceeded while awaiting headers)
Apr 18 11:06:12 redacted telegraf [2510617] : 2026-04-18T17:06:122 E! [outputs.influxdb_v2] When writing to [redacted:8086/api/v2/write] : Post "redacted:8086/api/v2/write?bucket
ypervisors&org=redacted": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Apr 18 11:06:12 redacted telegraf [2510617] : 2026-04-18T17:06:12Z E! [agent] Error writing to outputs.influxdb_v2: Post "redacted:8086/api/v2/write?bucket=Hypervisors&org=redacted": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Apr 18 11:06:16 redacted telegraf [2510617] : 2026-04-18T17:06:16z w! [outputs. influxdb_v2] Metric buffer overflow; 1072 metrics have been dropped
Apr 18 11:06:21 redacted telegraf [2510617] : 2026-04-18T17:06:21Z Exceeded (Client.Timeout exceeded while awaiting headers)

System info

Telegraf v1.40.0/InfluxDB v2.7.12 Server: ec9dcde Frontend: 5c07739/Ubuntu 24.04 host

Docker

No response

Steps to reproduce

1. Start telegraf service
2. Runs fine for few hours
3. Logs start populating with Client.Timeout exceeded while awaiting headers, context deadline exceeded
   ...

Expected behavior

InfluxDB collects metrics correctly and updates every few seconds

Actual behavior

InfluxDB starts dropping buckets of metrics and updates turn to at about a 30-40 second delay

Additional info

Promox VE baremetal running telegraf

VM hosting InfluxDB and Grafana

[skartikey]

Hi @dan-valaratomics, the Client.Timeout exceeded while awaiting headers error comes straight from Go's HTTP client when the server doesn't return headers in time. A few things worth trying before we dig deeper:

You have timeout = "30s" commented out, so you're on the 5s default. With interval = "1s" and ~12 inputs, your flush batches aren't small, and any brief InfluxDB stall (compaction, WAL flush, disk I/O spike) will blow past 5s. Most reports like this go away with:

[[outputs.influxdb_v2]]
  timeout = "30s"
  metric_batch_size = 1000
  metric_buffer_limit = 50000                                                                                                                                                                                    

If that doesn't help, run with debug = true in [agent] - v1.40.0 logs per-request timing at trace level. If elapsed ≈ timeout on failures, InfluxDB isn't responding (network or server-side); if it's much less, it's a reset/server error. That output would be really helpful here.

Also worth checking the InfluxDB VM's CPU / disk I/O during the failure window, since it's on the same host.

One oddity: your config has interval = "1s" / flush_interval = "7s", but the startup banner shows Interval:1m0s, Flush Interval:1m0s. Can you confirm the posted config is the one actually being loaded?
telegraf --config <file> --test will print what's in effect.

[srebhan]

@dan-valaratomics which Telegraf version are you using? Your description says v1.40 but this would mean you are from the future as the latest version as of today is v1.38.4. :-)

[telegraf-tiger]

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Forums or provide additional details in this issue and reqeust that it be re-opened. Thank you!