You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: administration/configuring-fluent-bit/yaml/environment-variables-section.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,8 +1,6 @@
1
1
# Environment variables
2
2
3
-
The `env` section lets you define environment variables directly within the configuration file. These variables can then be used to dynamically replace values throughout your configuration using the `${VARIABLE_NAME}` syntax.
4
-
5
-
Variables set in this section can't be overridden by system environment variables.
3
+
The `env` section of YAML configuration files lets you define environment variables. These variables can then be used to dynamically replace values throughout your configuration using the `${VARIABLE_NAME}` syntax.
6
4
7
5
Values set in the `env` section are case-sensitive. However, as a best practice, Fluent Bit recommends using uppercase names for environment variables. The following example defines two variables, `FLUSH_INTERVAL` and `STDOUT_FMT`, which can be accessed in the configuration using `${FLUSH_INTERVAL}` and `${STDOUT_FMT}`:
8
6
@@ -27,7 +25,7 @@ pipeline:
27
25
28
26
## Predefined variables
29
27
30
-
Fluent Bit provides a set of predefined environment variables that can be used in your configuration:
28
+
Fluent Bit supports the following predefined environment variables. You can reference these variables in configuration files without defining them in the `env` section.
31
29
32
30
| Name | Description |
33
31
| ---- | ----------- |
@@ -37,7 +35,9 @@ Fluent Bit provides a set of predefined environment variables that can be used i
37
35
38
36
In addition to variables defined in the configuration file or the predefined ones, Fluent Bit can access system environment variables set in the user space. These external variables can be referenced in the configuration using the same `${VARIABLE_NAME}` pattern.
39
37
38
+
{% hint style="info" %}
40
39
Variables set in the `env` section can't be overridden by system environment variables.
40
+
{% endhint %}
41
41
42
42
For example, to set the `FLUSH_INTERVAL` system environment variable to `2` and use it in your configuration:
43
43
@@ -62,4 +62,4 @@ pipeline:
62
62
format: json_lines
63
63
```
64
64
65
-
This approach lets you manage and override configuration values using environment variables, providing flexibility in various deployment environments.
65
+
This approach lets you manage and override configuration values using environment variables, providing flexibility in various deployment environments.
The `includes` section lets you specify additional YAML configuration files to be merged into the current configuration. These files are identified as a list of filenames and can include relative or absolute paths. If no absolute path is provided, the file is assumed to be located in a directory relative to the file that references it.
3
+
The `includes` section of YAML configuration files lets you specify additional YAML files to be merged into the current configuration. This lets you organize complex configurations into smaller, manageable files and include them as needed.
4
4
5
-
Use this section to organize complex configurations into smaller, manageable files and include them as needed.
5
+
These files are identified as a list of filenames and can include relative or absolute paths. If a path isn't specified as absolute, it will be treated as relative to the file that includes it.
6
6
7
7
## Usage
8
8
@@ -15,7 +15,11 @@ The following example demonstrates how to include additional YAML files using re
15
15
└── inclusion-2.yaml
16
16
```
17
17
18
-
The content of `fluent-bit.yaml`:
18
+
{% hint style="info" %}
19
+
Environment variables aren't supported in includes section. The path for each file must be specified as a literal string.
20
+
{% endhint %}
21
+
22
+
You can reference these files in `fluent-bit.yaml` as follows:
19
23
20
24
```yaml
21
25
includes:
@@ -24,7 +28,3 @@ includes:
24
28
```
25
29
26
30
Ensure that the included files are formatted correctly and contain valid YAML configurations for seamless integration.
27
-
28
-
If a path isn't specified as absolute, it will be treated as relative to the file that includes it.
29
-
30
-
Environment variables aren't supported in includes section. The file path must be specified as a literal string.
The `pipeline` section defines the flow of how data is collected, processed, and sent to its final destination. This section contains the following subsections:
3
+
The `pipeline` section of YAML configuration files defines the flow of how data is collected, processed, and sent to its final destination. This section contains the following subsections:
|`inputs`| Specifies the name of the plugin responsible for collecting or receiving data. This component serves as the data source in the pipeline. Examples of input plugins include `tail`, `http`, and `random`. |
8
-
|`filters`| Filters are used to transform, enrich, or discard events based on specific criteria. They allow matching tags using strings or regular expressions, providing a more flexible way to manipulate data. Filters run as part of the main event loop and can be applied across multiple inputs and filters. Examples of filters include `modify`, `grep`, and `nest`. |
9
-
|`outputs`| Defines the destination for processed data. Outputs specify where the data will be sent, such as to a remote server, a file, or another service. Each output plugin is configured with matching rules to determine which events are sent to that destination. Common output plugins include `stdout`, `elasticsearch`, and `kafka`. |
10
9
11
10
{% hint style="info" %}
12
11
13
-
Unlike filters, processors and parsers aren't defined within a unified section of YAML configuration files and don't use tag matching. Instead, each input or output defined in the configuration file can have a `parsers` key and `processors` key to configure the parsers and processors for that specific plugin.
12
+
Unlike filters, processors and parsers aren't defined within a unified section of YAML configuration files and don't use tag matching. Instead, each input or output plugin defined in the configuration file can have a `parsers` key and a`processors` key to configure the parsers and processors for that specific plugin.
14
13
15
14
{% endhint %}
16
15
17
-
## Format
16
+
## Syntax
18
17
19
-
A`pipeline` section will define a complete pipeline configuration, including `inputs`, `filters`, and `outputs` subsections. You can define multiple `pipeline` sections, but they won't operate independently. Instead, all components will be merged into a single pipeline internally.
18
+
The`pipeline` section of a YAML configuration file uses the following syntax:
20
19
21
20
```yaml
22
21
pipeline:
@@ -32,54 +31,55 @@ Each of the subsections for `inputs`, `filters`, and `outputs` constitutes an ar
32
31
33
32
For example:
34
33
34
+
{% tabs %}
35
+
{% tab title="fluent-bit.yaml" %}
36
+
35
37
```yaml
36
38
pipeline:
37
39
inputs:
38
40
- name: tail
39
-
tag: syslog
40
-
path: /var/log/syslog
41
-
- name: http
42
-
tag: http_server
43
-
port: 8080
44
-
```
41
+
path: /var/log/example.log
42
+
parser: json
45
43
46
-
This pipeline consists of two `inputs`: a tail plugin and an HTTP server plugin. Each plugin has its own map in the array of `inputs` consisting of basic properties. To use more advanced properties that consist of multiple values the property itself can be defined using an array, such as the `record` and `allowlist_key` properties for the `record_modifier` `filter`:
44
+
processors:
45
+
logs:
46
+
- name: record_modifier
47
47
48
-
```yaml
49
-
pipeline:
50
-
inputs:
51
-
- name: tail
52
-
tag: syslog
53
-
path: /var/log/syslog
54
48
filters:
55
-
- name: record_modifier
56
-
match: syslog
57
-
record:
58
-
- powered_by calyptia
59
-
- name: record_modifier
60
-
match: syslog
61
-
allowlist_key:
62
-
- powered_by
63
-
- message
49
+
- name: grep
50
+
match: '*'
51
+
regex: key pattern
52
+
53
+
outputs:
54
+
- name: stdout
55
+
match: '*'
64
56
```
65
57
66
-
In the cases where each value in a list requires two values they must be separated by a space, such as in the `record` property for the `record_modifier` filter.
58
+
{% endtab %}
59
+
{% endtabs %}
67
60
68
-
### Inputs
69
61
70
-
An `input` section defines a source (related to an input plugin). Each section has a base configuration. Each input plugin can add it own configuration keys:
62
+
{% hint style="info" %}
63
+
64
+
It's possible to define multiple `pipeline` sections, but they won't operate independently. Instead, Fluent Bit merges all defined pipelines into a single pipeline internally.
65
+
66
+
{% endhint %}
67
+
68
+
## Inputs
69
+
70
+
The `inputs` section defines one or more [input plugins](../../../pipeline/inputs.md). In addition to the settings unique to each plugin, all input plugins support the following configuration parameters:
71
71
72
72
| Key | Description |
73
-
| --- |------------ |
74
-
| `Name` | Name of the input plugin. Defined as subsection of the `inputs` section. |
75
-
| `Tag` | Tag name associated to all records coming from this plugin. |
76
-
| `Log_Level` | Set the plugin's logging verbosity level. Allowed values are: `off`, `error`, `warn`, `info`, `debug`, and `trace`. Defaults to the `SERVICE` section's `Log_Level`. |
73
+
| --- |----------- |
74
+
| `name` | Name of the input plugin. Defined as subsection of the `inputs` section. |
75
+
| `tag` | Tag name associated to all records coming from this plugin. |
76
+
| `log_level` | Set the plugin's logging verbosity level. Allowed values are: `off`, `error`, `warn`, `info`, `debug`, and `trace`. Defaults to the `service` section's `log_level`. |
77
77
78
-
The `Name` is mandatory and defines for Fluent Bit which input plugin should be loaded. `Tag` is mandatory for all plugins except for the `input forward` plugin which provides dynamic tags.
78
+
The `name` parameter is required and defines for Fluent Bit which input plugin should be loaded. The `tag` parameter is required for all plugins except for the `forward` plugin, which provides dynamic tags.
79
79
80
-
#### Example input
80
+
### Example input configuration
81
81
82
-
The following is an example of an `input` section for the `cpu` plugin.
82
+
The following is an example of an `inputs` section that contains a `cpu` plugin.
83
83
84
84
```yaml
85
85
pipeline:
@@ -88,22 +88,22 @@ pipeline:
88
88
tag: my_cpu
89
89
```
90
90
91
-
### Filters
91
+
## Filters
92
92
93
-
A `filter` section defines a filter (related to a filter plugin). Each section has a base configuration and each filter plugin can add its own configuration keys:
93
+
The `filters` section defines one or more [filters](../../../pipeline/filters.md). In addition to the settings unique to each filter, all filters support the following configuration parameters:
| `Name` | Name of the filter plugin. Defined as a subsection of the `filters` section. |
98
-
| `Match` | A pattern to match against the tags of incoming records. It's case-sensitive and supports the star (`*`) character as a wildcard. |
99
-
| `Match_Regex` | A regular expression to match against the tags of incoming records. Use this option if you want to use the full regular expression syntax. |
100
-
| `Log_Level` | Set the plugin's logging verbosity level. Allowed values are: `off`, `error`, `warn`, `info`, `debug`, and `trace`. Defaults to the `SERVICE` section's `Log_Level`. |
95
+
| Key | Description |
96
+
| --- | ----------- |
97
+
| `name` | Name of the filter plugin. Defined as a subsection of the `filters` section. |
98
+
| `match` | A pattern to match against the tags of incoming records. It's case-sensitive and supports the star (`*`) character as a wildcard. |
99
+
| `match_regex` | A regular expression to match against the tags of incoming records. Use this option if you want to use the full regular expression syntax. |
100
+
| `log_level` | Set the plugin's logging verbosity level. Allowed values are: `off`, `error`, `warn`, `info`, `debug`, and `trace`. Defaults to the `service` section's `log_level`. |
101
101
102
-
`Name`is mandatory and lets Fluent Bit know which filter plugin should be loaded. The `Match` or `Match_Regex` is mandatory for all plugins. If both are specified, `Match_Regex` takes precedence.
102
+
The `name` parameter is required and lets Fluent Bit know which filter should be loaded. One of either the `match` or `match_regex` parameters is required. If both are specified, `match_regex` takes precedence.
103
103
104
-
#### Example filter
104
+
### Example filter configuration
105
105
106
-
The following is an example of a `filter` section for the `grep` plugin:
106
+
The following is an example of a `filters` section that contains a `grep` plugin:
107
107
108
108
```yaml
109
109
pipeline:
@@ -113,55 +113,27 @@ pipeline:
113
113
regex: log aa
114
114
```
115
115
116
-
### Outputs
117
116
118
-
The `outputs` section specifies a destination that certain records should follow after a `Tag` match. Fluent Bit can route up to 256 `OUTPUT` plugins. The configuration supports the following keys:
| `Name` | Name of the output plugin. Defined as a subsection of the `outputs` section. |
123
-
| `Match` | A pattern to match against the tags of incoming records. It's case-sensitive and supports the star (`*`) character as a wildcard. |
124
-
| `Match_Regex` | A regular expression to match against the tags of incoming records. Use this option if you want to use the full regular expression syntax. |
125
-
| `Log_Level` | Set the plugin's logging verbosity level. Allowed values are: `off`, `error`, `warn`, `info`, `debug`, and `trace`. The output log level defaults to the `SERVICE` section's `Log_Level`. |
119
+
The `outputs` section defines one or more [output plugins](../../../pipeline/outputs.md). In addition to the settings unique to each plugin, all output plugins support the following configuration parameters:
126
120
127
-
#### Example output
128
-
129
-
The following is an example of an `output` section:
130
-
131
-
```yaml
132
-
pipeline:
133
-
outputs:
134
-
- name: stdout
135
-
match: 'my*cpu'
136
-
```
121
+
| Key | Description |
122
+
| --- | ----------- |
123
+
| `name` | Name of the output plugin. Defined as a subsection of the `outputs` section. |
124
+
| `match` | A pattern to match against the tags of incoming records. It's case-sensitive and supports the star (`*`) character as a wildcard. |
125
+
| `match_regex` | A regular expression to match against the tags of incoming records. Use this option if you want to use the full regular expression syntax. |
126
+
| `log_level` | Set the plugin's logging verbosity level. Allowed values are: `off`, `error`, `warn`, `info`, `debug`, and `trace`. The output log level defaults to the `service` section's `log_level`. |
137
127
138
-
## Example configuration
128
+
Fluent Bit can route up to 256 output plugins.
139
129
140
-
Here's an example of a pipeline configuration:
130
+
### Example output configuration
141
131
142
-
{% tabs %}
143
-
{% tab title="fluent-bit.yaml" %}
132
+
The following is an example of an `outputs` section that contains a `stdout` plugin:
Copy file name to clipboardExpand all lines: administration/configuring-fluent-bit/yaml/plugins-section.md
+7-5Lines changed: 7 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,14 @@
1
1
# Plugins
2
2
3
-
Fluent Bit comes with a variety of built-in plugins, and also supports loading external plugins at runtime. Use this feature for loading Go or WebAssembly (Wasm) plugins that are built as shared object files (`.so`). Fluent Bit YAML configuration provides the following ways to load these external plugins:
3
+
In addition to the plugins that come bundled with Fluent Bit, you can load external plugins. Use this feature for loading Go or WebAssembly (Wasm) plugins that are built as shared object files (`.so`).
4
+
5
+
{% hint style="info" %}
6
+
To configure the settings for individual plugins, use the `inputs` and `outputs` sections nested under the [`pipeline` section](./pipeline-section.md) of YAML configuration files.
7
+
{% endhint %}
4
8
5
9
## Inline YAML
6
10
7
-
You can specify external plugins directly within your main YAML configuration file using the `plugins` section. Here's an example:
11
+
You can specify external plugins in the `plugins` section of YAML configuration files. For example:
8
12
9
13
```yaml
10
14
plugins:
@@ -24,9 +28,7 @@ pipeline:
24
28
25
29
## YAML plugins file included using the `plugins_file` option
26
30
27
-
You can load external plugins from a separate YAML file by specifying the `plugins_file` option in the service section for better modularity.
28
-
29
-
To configure this:
31
+
Additionally, you can define external plugins in a separate YAML file, then reference that file in the `plugins_file` key nested under the [`service` section](./service-section.md) of your YAML configuration file. For example:
0 commit comments