Skip to content

Commit b06826f

Browse files
authored
in_tail: add data reliability note (#2310)
This commit adds a 'Data Reliability and Recovery' hint to the Tail input plugin documentation. It clarifies the behavior of the database offset mechanism during unexpected shutdowns (e.g., system crash, power loss). Specifically, it explains that while Fluent Bit guarantees at-least-once delivery, there is a possibility of slight offset lag and minimal data duplication upon recovery. This ensures users understand that no data is lost even in these scenarios. refs: fluent/fluent-bit#11269 Signed-off-by: jinyong.choi <inimax801@gmail.com>
1 parent a36f453 commit b06826f

File tree

1 file changed

+12
-0
lines changed

1 file changed

+12
-0
lines changed

pipeline/inputs/tail.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,18 @@ If no database file is present, positioning behavior depends on the value of `re
9191

9292
The database file essentially stores `inode=offset` so it should be unique per instance of the plugin, for example if you have two tail inputs then use two separate `db` files for each. That way each tail input can independently track its own state.
9393

94+
{% hint style="info" %}
95+
96+
**Data Reliability and Recovery**
97+
98+
During normal operations and graceful shutdowns (SIGTERM), Fluent Bit synchronizes file offsets with the database to ensure no data is lost.
99+
100+
In unexpected shutdowns (for example, system crash, power loss, SIGKILL), Fluent Bit guarantees _at-least-once_ delivery. The database offset might slightly trail the actual processed position if an unexpected shutdown occurs after data is processed but before the new offset is committed to the database.
101+
102+
Upon restart, Fluent Bit resumes from the last committed checkpoint. This ensures no data is lost, though it might result in the re-ingestion of a minimal amount of previously processed records (duplication).
103+
104+
{% endhint %}
105+
94106
{% hint style="info" %}
95107
The `unicode.encoding` parameter is dependent on the `simdutf` library, which is itself dependent on C++ version 11 or later. In environments that use earlier versions of C++, the `unicode.encoding` parameter will fail.
96108

0 commit comments

Comments
 (0)