Skip to content

Commit a06ca3c

Browse files
zshuang0316claude
andcommitted
input_log: account input metrics once before route fan-out
Counting the input metrics on the base append while the per-route copies are skipped leaves a gap under backpressure. A route copy can succeed, push the input over its memory or storage limit and pause it via flb_input_chunk_protect(), or the base append can fail after some copies were already delivered. In both cases the base append returns before counting, so records already queued for a route go unreflected in fluentbit_input_records_total and fluentbit_input_bytes_total. Account the original ingestion exactly once at the routing boundary, before the split, gated by the entry pause state, and mark the base append with FLB_INPUT_CHUNK_SKIP_INPUT_METRICS as the per-route copies already are. Counting up front keeps the totals correct regardless of which downstream append later pauses or fails. The plain non-routed path is unchanged and keeps counting inside flb_input_chunk_append_raw(). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: zshuang0316 <zshuang0316@163.com>
1 parent b956a77 commit a06ca3c

1 file changed

Lines changed: 38 additions & 5 deletions

File tree

src/flb_input_log.c

Lines changed: 38 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1482,6 +1482,7 @@ static int input_log_append_processed_internal(struct flb_input_instance *ins,
14821482
int ret;
14831483
int conditional_result;
14841484
int conditional_handled = FLB_FALSE;
1485+
int base_flags = 0;
14851486
size_t dummy = 0;
14861487
const char *base_tag = tag;
14871488
size_t base_tag_len = tag_len;
@@ -1501,6 +1502,34 @@ static int input_log_append_processed_internal(struct flb_input_instance *ins,
15011502
base_tag_len = strlen(base_tag);
15021503
}
15031504

1505+
#ifdef FLB_HAVE_METRICS
1506+
/*
1507+
* Conditional / per-record routing fans this batch out into one chunk per
1508+
* matching route (each appended below with
1509+
* FLB_INPUT_CHUNK_SKIP_INPUT_METRICS) plus a base append that is also
1510+
* skipped. Account the original ingestion exactly once here, BEFORE the
1511+
* split, so the input counters are neither inflated by the route count nor
1512+
* lost when a route copy pauses the input or the base append fails after
1513+
* some copies were already delivered. The plain, non-routed path keeps
1514+
* counting inside flb_input_chunk_append_raw().
1515+
*/
1516+
if (cfl_list_size(&ins->routes_direct) > 0 &&
1517+
input_has_conditional_routes(ins) == FLB_TRUE &&
1518+
flb_input_buf_paused(ins) == FLB_FALSE &&
1519+
records > 0 && buf_size > 0) {
1520+
uint64_t ts = cfl_time_now();
1521+
char *name = (char *) flb_input_name(ins);
1522+
1523+
cmt_counter_add(ins->cmt_records, ts, records, 1, (char *[]) {name});
1524+
cmt_counter_add(ins->cmt_bytes, ts, buf_size, 1, (char *[]) {name});
1525+
1526+
flb_metrics_sum(FLB_METRIC_N_RECORDS, records, ins->metrics);
1527+
flb_metrics_sum(FLB_METRIC_N_BYTES, buf_size, ins->metrics);
1528+
1529+
base_flags = FLB_INPUT_CHUNK_SKIP_INPUT_METRICS;
1530+
}
1531+
#endif
1532+
15041533
conditional_result = split_and_append_route_payloads(ins, records, tag, tag_len,
15051534
buf, buf_size,
15061535
local_append);
@@ -1515,15 +1544,19 @@ static int input_log_append_processed_internal(struct flb_input_instance *ins,
15151544
/*
15161545
* Always call flb_input_chunk_append_raw to ensure non-conditional routes
15171546
* receive data even when conditional routes exist. The conditional routing
1518-
* should be additive, not exclusive.
1547+
* should be additive, not exclusive. When the ingestion was already counted
1548+
* above, base_flags carries FLB_INPUT_CHUNK_SKIP_INPUT_METRICS so this
1549+
* append does not count it a second time.
15191550
*/
15201551
if (local_append == FLB_TRUE) {
1521-
ret = flb_input_chunk_append_raw_local(ins, FLB_INPUT_LOGS, records,
1522-
tag, tag_len, buf, buf_size);
1552+
ret = flb_input_chunk_append_raw_local_flags(ins, FLB_INPUT_LOGS,
1553+
base_flags, records,
1554+
tag, tag_len, buf, buf_size);
15231555
}
15241556
else {
1525-
ret = flb_input_chunk_append_raw(ins, FLB_INPUT_LOGS, records,
1526-
tag, tag_len, buf, buf_size);
1557+
ret = flb_input_chunk_append_raw_flags(ins, FLB_INPUT_LOGS,
1558+
base_flags, records,
1559+
tag, tag_len, buf, buf_size);
15271560
}
15281561

15291562
if (ret == 0 && conditional_handled == FLB_TRUE && base_tag) {

0 commit comments

Comments
 (0)