loki: prevent duplicate log ingestion and improve timestamp handling#4498
loki: prevent duplicate log ingestion and improve timestamp handling#4498Anulo2 wants to merge 2 commits into
Conversation
…ling Remove the 'else if infinite' branch in updateURI() that reset the query start timestamp backward by 1 minute when no new results were returned. This caused the same Loki log entries to be re-fetched and ingested multiple times, inflating event counts and triggering false bucket overflows. Also improve robustness: - Scan all streams for maxTS instead of only the first stream's last entry - Use strconv.ParseInt/FormatInt for nanosecond timestamps to avoid 32-bit Atoi overflow on platforms where int is 32-bit - Align entry.go timestamp parsing with types.go (time.Unix(0, t)) Fixes duplicate log lines appearing 2-3x in CrowdSec alerts when using Loki datasource in streaming mode.
|
@Anulo2: There are no 'kind' label on this PR. You need a 'kind' label to generate the release automatically.
DetailsI am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository. |
|
@Anulo2: There are no area labels on this PR. You can add as many areas as you see fit.
DetailsI am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository. |
|
/kind fix |
|
/area agent |
2 participants
Fixes duplicate log lines appearing 2-3x in CrowdSec alerts when using Loki datasource in streaming mode.