-
-
-
-="value"` | Creates a `String` claim with the given name and value. Any valid claim name can be specified except '`iat`' and '`exp`' (these are automatically generated).
+`unsecuredLoginNumberClaim_="value"` | Creates a `Number` claim with the given name and value. Any valid claim name can be specified except '`iat`' and '`exp`' (these are automatically generated).
+`unsecuredLoginListClaim_="value"` | Creates a `String List` claim with the given name and values parsed from the given value where the first character is taken as the delimiter. For example: `unsecuredLoginListClaim_fubar="|value1|value2"`. Any valid claim name can be specified except '`iat`' and '`exp`' (these are automatically generated).
+`unsecuredLoginExtension_="value"` | Creates a `String` extension with the given name and value. For example: `unsecuredLoginExtension_traceId="123"`. A valid extension name is any sequence of lowercase or uppercase alphabet characters. In addition, the "auth" extension name is reserved. A valid extension value is any combination of characters with ASCII codes 1-127.
+`unsecuredLoginPrincipalClaimName` | Set to a custom claim name if you wish the name of the `String` claim holding the principal name to be something other than '`sub`'.
+`unsecuredLoginLifetimeSeconds` | Set to an integer value if the token expiration is to be set to something other than the default value of 3600 seconds (which is 1 hour). The '`exp`' claim will be set to reflect the expiration time.
+`unsecuredLoginScopeClaimName` | Set to a custom claim name if you wish the name of the `String` or `String List` claim holding any token scope to be something other than '`scope`'.
+ 4. ##### Unsecured Token Validation Options for SASL/OAUTHBEARER
+
+ * Here are the various supported JAAS module options on the broker side for [Unsecured JSON Web Token](https://tools.ietf.org/html/rfc7515#appendix-A.5) validation: JAAS Module Option for Unsecured Token Validation | Documentation
+---|---
+`unsecuredValidatorPrincipalClaimName="value"` | Set to a non-empty value if you wish a particular `String` claim holding a principal name to be checked for existence; the default is to check for the existence of the '`sub`' claim.
+`unsecuredValidatorScopeClaimName="value"` | Set to a custom claim name if you wish the name of the `String` or `String List` claim holding any token scope to be something other than '`scope`'.
+`unsecuredValidatorRequiredScope="value"` | Set to a space-delimited list of scope values if you wish the `String/String List` claim holding the token scope to be checked to make sure it contains certain values.
+`unsecuredValidatorAllowableClockSkewMs="value"` | Set to a positive integer value if you wish to allow up to some number of positive milliseconds of clock skew (the default is 0).
+ * The default unsecured SASL/OAUTHBEARER implementation may be overridden (and must be overridden in production environments) using custom login and SASL Server callback handlers.
+ * For more details on security considerations, refer to [RFC 6749, Section 10](https://tools.ietf.org/html/rfc6749#section-10).
+ 5. ##### Token Refresh for SASL/OAUTHBEARER
+
+Kafka periodically refreshes any token before it expires so that the client can continue to make connections to brokers. The parameters that impact how the refresh algorithm operates are specified as part of the producer/consumer/broker configuration and are as follows. See the documentation for these properties elsewhere for details. The default values are usually reasonable, in which case these configuration parameters would not need to be explicitly set. Producer/Consumer/Broker Configuration Property
+---
+`sasl.login.refresh.window.factor`
+`sasl.login.refresh.window.jitter`
+`sasl.login.refresh.min.period.seconds`
+`sasl.login.refresh.min.buffer.seconds`
+ 6. ##### Secure/Production Use of SASL/OAUTHBEARER
+
+Production use cases will require writing an implementation of `org.apache.kafka.common.security.auth.AuthenticateCallbackHandler` that can handle an instance of `org.apache.kafka.common.security.oauthbearer.OAuthBearerTokenCallback` and declaring it via either the `sasl.login.callback.handler.class` configuration option for a non-broker client or via the `listener.name.sasl_ssl.oauthbearer.sasl.login.callback.handler.class` configuration option for brokers (when SASL/OAUTHBEARER is the inter-broker protocol).
+
+Production use cases will also require writing an implementation of `org.apache.kafka.common.security.auth.AuthenticateCallbackHandler` that can handle an instance of `org.apache.kafka.common.security.oauthbearer.OAuthBearerValidatorCallback` and declaring it via the `listener.name.sasl_ssl.oauthbearer.sasl.server.callback.handler.class` broker configuration option.
+ 7. ##### Security Considerations for SASL/OAUTHBEARER
+
+ * The default implementation of SASL/OAUTHBEARER in Kafka creates and validates [Unsecured JSON Web Tokens](https://tools.ietf.org/html/rfc7515#appendix-A.5). This is suitable only for non-production use.
+ * OAUTHBEARER should be used in production enviromnments only with TLS-encryption to prevent interception of tokens.
+ * The default unsecured SASL/OAUTHBEARER implementation may be overridden (and must be overridden in production environments) using custom login and SASL Server callback handlers as described above.
+ * For more details on OAuth 2 security considerations in general, refer to [RFC 6749, Section 10](https://tools.ietf.org/html/rfc6749#section-10).
+ 7. #### Enabling multiple SASL mechanisms in a broker
+
+ 1. Specify configuration for the login modules of all enabled mechanisms in the `KafkaServer` section of the JAAS config file. For example:
+
+ KafkaServer {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ storeKey=true
+ keyTab="/etc/security/keytabs/kafka_server.keytab"
+ principal="kafka/kafka1.hostname.com@EXAMPLE.COM";
+
+ org.apache.kafka.common.security.plain.PlainLoginModule required
+ username="admin"
+ password="admin-secret"
+ user_admin="admin-secret"
+ user_alice="alice-secret";
+ };
+
+ 2. Enable the SASL mechanisms in server.properties:
+
+ sasl.enabled.mechanisms=GSSAPI,PLAIN,SCRAM-SHA-256,SCRAM-SHA-512,OAUTHBEARER
+
+ 3. Specify the SASL security protocol and mechanism for inter-broker communication in server.properties if required:
+
+ security.inter.broker.protocol=SASL_PLAINTEXT (or SASL_SSL)
+ sasl.mechanism.inter.broker.protocol=GSSAPI (or one of the other enabled mechanisms)
+
+ 4. Follow the mechanism-specific steps in GSSAPI (Kerberos), PLAIN, SCRAM and OAUTHBEARER to configure SASL for the enabled mechanisms.
+ 8. #### Modifying SASL mechanism in a Running Cluster
+
+SASL mechanism can be modified in a running cluster using the following sequence:
+
+ 1. Enable new SASL mechanism by adding the mechanism to `sasl.enabled.mechanisms` in server.properties for each broker. Update JAAS config file to include both mechanisms as described here. Incrementally bounce the cluster nodes.
+ 2. Restart clients using the new mechanism.
+ 3. To change the mechanism of inter-broker communication (if this is required), set `sasl.mechanism.inter.broker.protocol` in server.properties to the new mechanism and incrementally bounce the cluster again.
+ 4. To remove old mechanism (if this is required), remove the old mechanism from `sasl.enabled.mechanisms` in server.properties and remove the entries for the old mechanism from JAAS config file. Incrementally bounce the cluster again.
+ 9. #### Authentication using Delegation Tokens
+
+Delegation token based authentication is a lightweight authentication mechanism to complement existing SASL/SSL methods. Delegation tokens are shared secrets between kafka brokers and clients. Delegation tokens will help processing frameworks to distribute the workload to available workers in a secure environment without the added cost of distributing Kerberos TGT/keytabs or keystores when 2-way SSL is used. See [KIP-48](https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka) for more details.
+
+Under the default implementation of `principal.builder.class`, the owner of delegation token is used as the authenticated `Principal` for configuration of ACLs etc.
+
+Typical steps for delegation token usage are:
+
+ 1. User authenticates with the Kafka cluster via SASL or SSL, and obtains a delegation token. This can be done using Admin APIs or using `kafka-delegation-tokens.sh` script.
+ 2. User securely passes the delegation token to Kafka clients for authenticating with the Kafka cluster.
+ 3. Token owner/renewer can renew/expire the delegation tokens.
+ 1. ##### Token Management
+
+A secret is used to generate and verify delegation tokens. This is supplied using config option `delegation.token.secret.key`. The same secret key must be configured across all the brokers. If using Kafka with KRaft the controllers must also be configured with the secret using the same config option. If the secret is not set or set to empty string, delegation token authentication and API operations will fail.
+
+When using Kafka with Zookeeper, the token details are stored in Zookeeper and delegation tokens are suitable for use in Kafka installations where Zookeeper is on a private network. When using Kafka with KRaft, the token details are stored with the other metadata on the controller nodes and delegation tokens are suitable for use when the controllers are on a private network or when all commnications between brokers and controllers is encrypted. Currently, this secret is stored as plain text in the server.properties config file. We intend to make these configurable in a future Kafka release.
+
+A token has a current life, and a maximum renewable life. By default, tokens must be renewed once every 24 hours for up to 7 days. These can be configured using `delegation.token.expiry.time.ms` and `delegation.token.max.lifetime.ms` config options.
+
+Tokens can also be cancelled explicitly. If a token is not renewed by the token’s expiration time or if token is beyond the max life time, it will be deleted from all broker caches as well as from zookeeper.
+
+ 2. ##### Creating Delegation Tokens
+
+Tokens can be created by using Admin APIs or using `kafka-delegation-tokens.sh` script. Delegation token requests (create/renew/expire/describe) should be issued only on SASL or SSL authenticated channels. Tokens can not be requests if the initial authentication is done through delegation token. A token can be created by the user for that user or others as well by specifying the `--owner-principal` parameter. Owner/Renewers can renew or expire tokens. Owner/renewers can always describe their own tokens. To describe other tokens, a DESCRIBE_TOKEN permission needs to be added on the User resource representing the owner of the token. `kafka-delegation-tokens.sh` script examples are given below.
+
+Create a delegation token:
+
+ > bin/kafka-delegation-tokens.sh --bootstrap-server localhost:9092 --create --max-life-time-period -1 --command-config client.properties --renewer-principal User:user1
+
+Create a delegation token for a different owner:
+
+ > bin/kafka-delegation-tokens.sh --bootstrap-server localhost:9092 --create --max-life-time-period -1 --command-config client.properties --renewer-principal User:user1 --owner-principal User:owner1
+
+Renew a delegation token:
+
+ > bin/kafka-delegation-tokens.sh --bootstrap-server localhost:9092 --renew --renew-time-period -1 --command-config client.properties --hmac ABCDEFGHIJK
+
+Expire a delegation token:
+
+ > bin/kafka-delegation-tokens.sh --bootstrap-server localhost:9092 --expire --expiry-time-period -1 --command-config client.properties --hmac ABCDEFGHIJK
+
+Existing tokens can be described using the --describe option:
+
+ > bin/kafka-delegation-tokens.sh --bootstrap-server localhost:9092 --describe --command-config client.properties --owner-principal User:user1
+
+ 3. ##### Token Authentication
+
+Delegation token authentication piggybacks on the current SASL/SCRAM authentication mechanism. We must enable SASL/SCRAM mechanism on Kafka cluster as described in here.
+
+Configuring Kafka Clients:
+
+ 1. Configure the JAAS configuration property for each client in producer.properties or consumer.properties. The login module describes how the clients like producer and consumer can connect to the Kafka Broker. The following is an example configuration for a client for the token authentication:
+
+ sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
+ username="tokenID123" \
+ password="lAYYSFmLs4bTjf+lTZ1LCHR/ZZFNA==" \
+ tokenauth="true";
+
+The options `username` and `password` are used by clients to configure the token id and token HMAC. And the option `tokenauth` is used to indicate the server about token authentication. In this example, clients connect to the broker using token id: _tokenID123_. Different clients within a JVM may connect using different tokens by specifying different token details in `sasl.jaas.config`.
+
+JAAS configuration for clients may alternatively be specified as a JVM parameter similar to brokers as described here. Clients use the login section named `KafkaClient`. This option allows only one user for all client connections from a JVM.
+
+ 4. ##### Procedure to manually rotate the secret:
+
+We require a re-deployment when the secret needs to be rotated. During this process, already connected clients will continue to work. But any new connection requests and renew/expire requests with old tokens can fail. Steps are given below.
+
+ 1. Expire all existing tokens.
+ 2. Rotate the secret by rolling upgrade, and
+ 3. Generate new tokens
+
+We intend to automate this in a future Kafka release.
+
+
+
diff --git a/docs/security/authorization-and-acls.md b/docs/security/authorization-and-acls.md
new file mode 100644
index 0000000000000..73b237cb6a7ac
--- /dev/null
+++ b/docs/security/authorization-and-acls.md
@@ -0,0 +1,330 @@
+---
+title: Authorization and ACLs
+description: Authorization and ACLs
+weight: 5
+tags: ['kafka', 'docs']
+aliases:
+keywords:
+type: docs
+---
+
+Kafka ships with a pluggable authorization framework, which is configured with the `authorizer.class.name` property in the server configuration. Configured implementations must extend `org.apache.kafka.server.authorizer.Authorizer`. Kafka provides default implementations which store ACLs in the cluster metadata (either Zookeeper or the KRaft metadata log). For Zookeeper-based clusters, the provided implementation is configured as follows:
+
+
+ authorizer.class.name=kafka.security.authorizer.AclAuthorizer
+
+For KRaft clusters, use the following configuration on all nodes (brokers, controllers, or combined broker/controller nodes):
+
+
+ authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
+
+Kafka ACLs are defined in the general format of "Principal {P} is [Allowed|Denied] Operation {O} From Host {H} on any Resource {R} matching ResourcePattern {RP}". You can read more about the ACL structure in [KIP-11](https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface) and resource patterns in [KIP-290](https://cwiki.apache.org/confluence/display/KAFKA/KIP-290%3A+Support+for+Prefixed+ACLs). In order to add, remove, or list ACLs, you can use the Kafka ACL CLI `kafka-acls.sh`. By default, if no ResourcePatterns match a specific Resource R, then R has no associated ACLs, and therefore no one other than super users is allowed to access R. If you want to change that behavior, you can include the following in server.properties.
+
+
+ allow.everyone.if.no.acl.found=true
+
+One can also add super users in server.properties like the following (note that the delimiter is semicolon since SSL user names may contain comma). Default PrincipalType string "User" is case sensitive.
+
+
+ super.users=User:Bob;User:Alice
+
+### KRaft Principal Forwarding
+
+In KRaft clusters, admin requests such as `CreateTopics` and `DeleteTopics` are sent to the broker listeners by the client. The broker then forwards the request to the active controller through the first listener configured in `controller.listener.names`. Authorization of these requests is done on the controller node. This is achieved by way of an `Envelope` request which packages both the underlying request from the client as well as the client principal. When the controller receives the forwarded `Envelope` request from the broker, it first authorizes the `Envelope` request using the authenticated broker principal. Then it authorizes the underlying request using the forwarded principal.
+All of this implies that Kafka must understand how to serialize and deserialize the client principal. The authentication framework allows for customized principals by overriding the `principal.builder.class` configuration. In order for customized principals to work with KRaft, the configured class must implement `org.apache.kafka.common.security.auth.KafkaPrincipalSerde` so that Kafka knows how to serialize and deserialize the principals. The default implementation `org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder` uses the Kafka RPC format defined in the source code: `clients/src/main/resources/common/message/DefaultPrincipalData.json`. For more detail about request forwarding in KRaft, see [KIP-590](https://cwiki.apache.org/confluence/display/KAFKA/KIP-590%3A+Redirect+Zookeeper+Mutation+Protocols+to+The+Controller)
+
+### Customizing SSL User Name
+
+By default, the SSL user name will be of the form "CN=writeuser,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown". One can change that by setting `ssl.principal.mapping.rules` to a customized rule in server.properties. This config allows a list of rules for mapping X.500 distinguished name to short name. The rules are evaluated in order and the first rule that matches a distinguished name is used to map it to a short name. Any later rules in the list are ignored.
+The format of `ssl.principal.mapping.rules` is a list where each rule starts with "RULE:" and contains an expression as the following formats. Default rule will return string representation of the X.500 certificate distinguished name. If the distinguished name matches the pattern, then the replacement command will be run over the name. This also supports lowercase/uppercase options, to force the translated result to be all lower/uppercase case. This is done by adding a "/L" or "/U' to the end of the rule.
+
+
+ RULE:pattern/replacement/
+ RULE:pattern/replacement/[LU]
+
+Example `ssl.principal.mapping.rules` values are:
+
+
+ RULE:^CN=(.*?),OU=ServiceUsers.*$/$1/,
+ RULE:^CN=(.*?),OU=(.*?),O=(.*?),L=(.*?),ST=(.*?),C=(.*?)$/$1@$2/L,
+ RULE:^.*[Cc][Nn]=([a-zA-Z0-9.]*).*$/$1/L,
+ DEFAULT
+
+Above rules translate distinguished name "CN=serviceuser,OU=ServiceUsers,O=Unknown,L=Unknown,ST=Unknown,C=Unknown" to "serviceuser" and "CN=adminUser,OU=Admin,O=Unknown,L=Unknown,ST=Unknown,C=Unknown" to "adminuser@admin".
+For advanced use cases, one can customize the name by setting a customized PrincipalBuilder in server.properties like the following.
+
+
+ principal.builder.class=CustomizedPrincipalBuilderClass
+
+### Customizing SASL User Name
+
+By default, the SASL user name will be the primary part of the Kerberos principal. One can change that by setting `sasl.kerberos.principal.to.local.rules` to a customized rule in server.properties. The format of `sasl.kerberos.principal.to.local.rules` is a list where each rule works in the same way as the auth_to_local in [Kerberos configuration file (krb5.conf)](https://web.mit.edu/Kerberos/krb5-latest/doc/admin/conf_files/krb5_conf.html). This also support additional lowercase/uppercase rule, to force the translated result to be all lowercase/uppercase. This is done by adding a "/L" or "/U" to the end of the rule. check below formats for syntax. Each rules starts with RULE: and contains an expression as the following formats. See the kerberos documentation for more details.
+
+
+ RULE:[n:string](regexp)s/pattern/replacement/
+ RULE:[n:string](regexp)s/pattern/replacement/g
+ RULE:[n:string](regexp)s/pattern/replacement//L
+ RULE:[n:string](regexp)s/pattern/replacement/g/L
+ RULE:[n:string](regexp)s/pattern/replacement//U
+ RULE:[n:string](regexp)s/pattern/replacement/g/U
+
+An example of adding a rule to properly translate user@MYDOMAIN.COM to user while also keeping the default rule in place is:
+
+
+ sasl.kerberos.principal.to.local.rules=RULE:[1:$1@$0](.*@MYDOMAIN.COM)s/@.*//,DEFAULT
+
+## Command Line Interface
+
+Kafka Authorization management CLI can be found under bin directory with all the other CLIs. The CLI script is called **kafka-acls.sh**. Following lists all the options that the script supports:
+
+Option | Description | Default | Option type
+---|---|---|---
+\--add | Indicates to the script that user is trying to add an acl. | | Action
+\--remove | Indicates to the script that user is trying to remove an acl. | | Action
+\--list | Indicates to the script that user is trying to list acls. | | Action
+\--bootstrap-server | A list of host/port pairs to use for establishing the connection to the Kafka cluster. Only one of --bootstrap-server or --authorizer option must be specified. | | Configuration
+\--command-config | A property file containing configs to be passed to Admin Client. This option can only be used with --bootstrap-server option. | | Configuration
+\--cluster | Indicates to the script that the user is trying to interact with acls on the singular cluster resource. | | ResourcePattern
+\--topic [topic-name] | Indicates to the script that the user is trying to interact with acls on topic resource pattern(s). | | ResourcePattern
+\--group [group-name] | Indicates to the script that the user is trying to interact with acls on consumer-group resource pattern(s) | | ResourcePattern
+\--transactional-id [transactional-id] | The transactionalId to which ACLs should be added or removed. A value of * indicates the ACLs should apply to all transactionalIds. | | ResourcePattern
+\--delegation-token [delegation-token] | Delegation token to which ACLs should be added or removed. A value of * indicates ACL should apply to all tokens. | | ResourcePattern
+\--user-principal [user-principal] | A user resource to which ACLs should be added or removed. This is currently supported in relation with delegation tokens. A value of * indicates ACL should apply to all users. | | ResourcePattern
+\--resource-pattern-type [pattern-type] | Indicates to the script the type of resource pattern, (for --add), or resource pattern filter, (for --list and --remove), the user wishes to use.
+When adding acls, this should be a specific pattern type, e.g. 'literal' or 'prefixed'.
+When listing or removing acls, a specific pattern type filter can be used to list or remove acls from a specific type of resource pattern, or the filter values of 'any' or 'match' can be used, where 'any' will match any pattern type, but will match the resource name exactly, and 'match' will perform pattern matching to list or remove all acls that affect the supplied resource(s).
+WARNING: 'match', when used in combination with the '--remove' switch, should be used with care. | literal | Configuration
+\--allow-principal | Principal is in PrincipalType:name format that will be added to ACL with Allow permission. Default PrincipalType string "User" is case sensitive.
+You can specify multiple --allow-principal in a single command. | | Principal
+\--deny-principal | Principal is in PrincipalType:name format that will be added to ACL with Deny permission. Default PrincipalType string "User" is case sensitive.
+You can specify multiple --deny-principal in a single command. | | Principal
+\--principal | Principal is in PrincipalType:name format that will be used along with --list option. Default PrincipalType string "User" is case sensitive. This will list the ACLs for the specified principal.
+You can specify multiple --principal in a single command. | | Principal
+\--allow-host | IP address from which principals listed in --allow-principal will have access. | if --allow-principal is specified defaults to * which translates to "all hosts" | Host
+\--deny-host | IP address from which principals listed in --deny-principal will be denied access. | if --deny-principal is specified defaults to * which translates to "all hosts" | Host
+\--operation | Operation that will be allowed or denied.
+Valid values are:
+
+ * Read
+ * Write
+ * Create
+ * Delete
+ * Alter
+ * Describe
+ * ClusterAction
+ * DescribeConfigs
+ * AlterConfigs
+ * IdempotentWrite
+ * CreateTokens
+ * DescribeTokens
+ * All
+
+| All | Operation
+\--producer | Convenience option to add/remove acls for producer role. This will generate acls that allows WRITE, DESCRIBE and CREATE on topic. | | Convenience
+\--consumer | Convenience option to add/remove acls for consumer role. This will generate acls that allows READ, DESCRIBE on topic and READ on consumer-group. | | Convenience
+\--idempotent | Enable idempotence for the producer. This should be used in combination with the --producer option.
+Note that idempotence is enabled automatically if the producer is authorized to a particular transactional-id. | | Convenience
+\--force | Convenience option to assume yes to all queries and do not prompt. | | Convenience
+\--authorizer | (DEPRECATED: not supported in KRaft) Fully qualified class name of the authorizer. | kafka.security.authorizer.AclAuthorizer | Configuration
+\--authorizer-properties | (DEPRECATED: not supported in KRaft) key=val pairs that will be passed to authorizer for initialization. For the default authorizer in ZK clsuters, the example values are: zookeeper.connect=localhost:2181 | | Configuration
+\--zk-tls-config-file | (DEPRECATED: not supported in KRaft) Identifies the file where ZooKeeper client TLS connectivity properties for the authorizer are defined. Any properties other than the following (with or without an "authorizer." prefix) are ignored: zookeeper.clientCnxnSocket, zookeeper.ssl.cipher.suites, zookeeper.ssl.client.enable, zookeeper.ssl.crl.enable, zookeeper.ssl.enabled.protocols, zookeeper.ssl.endpoint.identification.algorithm, zookeeper.ssl.keystore.location, zookeeper.ssl.keystore.password, zookeeper.ssl.keystore.type, zookeeper.ssl.ocsp.enable, zookeeper.ssl.protocol, zookeeper.ssl.truststore.location, zookeeper.ssl.truststore.password, zookeeper.ssl.truststore.type | | Configuration
+
+## Examples
+
+ * **Adding Acls**
+Suppose you want to add an acl "Principals User:Bob and User:Alice are allowed to perform Operation Read and Write on Topic Test-Topic from IP 198.51.100.0 and IP 198.51.100.1". You can do that by executing the CLI with following options:
+
+ > bin/kafka-acls.sh --bootstrap-server localhost:9092 --add --allow-principal User:Bob --allow-principal User:Alice --allow-host 198.51.100.0 --allow-host 198.51.100.1 --operation Read --operation Write --topic Test-topic
+
+By default, all principals that don't have an explicit acl that allows access for an operation to a resource are denied. In rare cases where an allow acl is defined that allows access to all but some principal we will have to use the --deny-principal and --deny-host option. For example, if we want to allow all users to Read from Test-topic but only deny User:BadBob from IP 198.51.100.3 we can do so using following commands:
+
+ > bin/kafka-acls.sh --bootstrap-server localhost:9092 --add --allow-principal User:'*' --allow-host '*' --deny-principal User:BadBob --deny-host 198.51.100.3 --operation Read --topic Test-topic
+
+Note that `--allow-host` and `--deny-host` only support IP addresses (hostnames are not supported). Above examples add acls to a topic by specifying --topic [topic-name] as the resource pattern option. Similarly user can add acls to cluster by specifying --cluster and to a consumer group by specifying --group [group-name]. You can add acls on any resource of a certain type, e.g. suppose you wanted to add an acl "Principal User:Peter is allowed to produce to any Topic from IP 198.51.200.0" You can do that by using the wildcard resource '*', e.g. by executing the CLI with following options:
+
+ > bin/kafka-acls.sh --bootstrap-server localhost:9092 --add --allow-principal User:Peter --allow-host 198.51.200.1 --producer --topic '*'
+
+You can add acls on prefixed resource patterns, e.g. suppose you want to add an acl "Principal User:Jane is allowed to produce to any Topic whose name starts with 'Test-' from any host". You can do that by executing the CLI with following options:
+
+ > bin/kafka-acls.sh --bootstrap-server localhost:9092 --add --allow-principal User:Jane --producer --topic Test- --resource-pattern-type prefixed
+
+Note, --resource-pattern-type defaults to 'literal', which only affects resources with the exact same name or, in the case of the wildcard resource name '*', a resource with any name.
+ * **Removing Acls**
+Removing acls is pretty much the same. The only difference is instead of --add option users will have to specify --remove option. To remove the acls added by the first example above we can execute the CLI with following options:
+
+ > bin/kafka-acls.sh --bootstrap-server localhost:9092 --remove --allow-principal User:Bob --allow-principal User:Alice --allow-host 198.51.100.0 --allow-host 198.51.100.1 --operation Read --operation Write --topic Test-topic
+
+If you want to remove the acl added to the prefixed resource pattern above we can execute the CLI with following options:
+
+ > bin/kafka-acls.sh --bootstrap-server localhost:9092 --remove --allow-principal User:Jane --producer --topic Test- --resource-pattern-type Prefixed
+
+ * **List Acls**
+We can list acls for any resource by specifying the --list option with the resource. To list all acls on the literal resource pattern Test-topic, we can execute the CLI with following options:
+
+ > bin/kafka-acls.sh --bootstrap-server localhost:9092 --list --topic Test-topic
+
+However, this will only return the acls that have been added to this exact resource pattern. Other acls can exist that affect access to the topic, e.g. any acls on the topic wildcard '*', or any acls on prefixed resource patterns. Acls on the wildcard resource pattern can be queried explicitly:
+
+ > bin/kafka-acls.sh --bootstrap-server localhost:9092 --list --topic '*'
+
+However, it is not necessarily possible to explicitly query for acls on prefixed resource patterns that match Test-topic as the name of such patterns may not be known. We can list _all_ acls affecting Test-topic by using '--resource-pattern-type match', e.g.
+
+ > bin/kafka-acls.sh --bootstrap-server localhost:9092 --list --topic Test-topic --resource-pattern-type match
+
+This will list acls on all matching literal, wildcard and prefixed resource patterns.
+ * **Adding or removing a principal as producer or consumer**
+The most common use case for acl management are adding/removing a principal as producer or consumer so we added convenience options to handle these cases. In order to add User:Bob as a producer of Test-topic we can execute the following command:
+
+ > bin/kafka-acls.sh --bootstrap-server localhost:9092 --add --allow-principal User:Bob --producer --topic Test-topic
+
+Similarly to add Alice as a consumer of Test-topic with consumer group Group-1 we just have to pass --consumer option:
+
+ > bin/kafka-acls.sh --bootstrap-server localhost:9092 --add --allow-principal User:Bob --consumer --topic Test-topic --group Group-1
+
+Note that for consumer option we must also specify the consumer group. In order to remove a principal from producer or consumer role we just need to pass --remove option.
+ * **Admin API based acl management**
+Users having Alter permission on ClusterResource can use Admin API for ACL management. kafka-acls.sh script supports AdminClient API to manage ACLs without interacting with zookeeper/authorizer directly. All the above examples can be executed by using **\--bootstrap-server** option. For example:
+
+ bin/kafka-acls.sh --bootstrap-server localhost:9092 --command-config /tmp/adminclient-configs.conf --add --allow-principal User:Bob --producer --topic Test-topic
+ bin/kafka-acls.sh --bootstrap-server localhost:9092 --command-config /tmp/adminclient-configs.conf --add --allow-principal User:Bob --consumer --topic Test-topic --group Group-1
+ bin/kafka-acls.sh --bootstrap-server localhost:9092 --command-config /tmp/adminclient-configs.conf --list --topic Test-topic
+ bin/kafka-acls.sh --bootstrap-server localhost:9092 --command-config /tmp/adminclient-configs.conf --add --allow-principal User:tokenRequester --operation CreateTokens --user-principal "owner1"
+
+
+
+
+## Authorization Primitives
+
+Protocol calls are usually performing some operations on certain resources in Kafka. It is required to know the operations and resources to set up effective protection. In this section we'll list these operations and resources, then list the combination of these with the protocols to see the valid scenarios.
+
+### Operations in Kafka
+
+There are a few operation primitives that can be used to build up privileges. These can be matched up with certain resources to allow specific protocol calls for a given user. These are:
+
+ * Read
+ * Write
+ * Create
+ * Delete
+ * Alter
+ * Describe
+ * ClusterAction
+ * DescribeConfigs
+ * AlterConfigs
+ * IdempotentWrite
+ * CreateTokens
+ * DescribeTokens
+ * All
+
+
+
+### Resources in Kafka
+
+The operations above can be applied on certain resources which are described below.
+
+ * **Topic:** this simply represents a Topic. All protocol calls that are acting on topics (such as reading, writing them) require the corresponding privilege to be added. If there is an authorization error with a topic resource, then a TOPIC_AUTHORIZATION_FAILED (error code: 29) will be returned.
+ * **Group:** this represents the consumer groups in the brokers. All protocol calls that are working with consumer groups, like joining a group must have privileges with the group in subject. If the privilege is not given then a GROUP_AUTHORIZATION_FAILED (error code: 30) will be returned in the protocol response.
+ * **Cluster:** this resource represents the cluster. Operations that are affecting the whole cluster, like controlled shutdown are protected by privileges on the Cluster resource. If there is an authorization problem on a cluster resource, then a CLUSTER_AUTHORIZATION_FAILED (error code: 31) will be returned.
+ * **TransactionalId:** this resource represents actions related to transactions, such as committing. If any error occurs, then a TRANSACTIONAL_ID_AUTHORIZATION_FAILED (error code: 53) will be returned by brokers.
+ * **DelegationToken:** this represents the delegation tokens in the cluster. Actions, such as describing delegation tokens could be protected by a privilege on the DelegationToken resource. Since these objects have a little special behavior in Kafka it is recommended to read [KIP-48](https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-DescribeDelegationTokenRequest) and the related upstream documentation at Authentication using Delegation Tokens.
+ * **User:** CreateToken and DescribeToken operations can be granted to User resources to allow creating and describing tokens for other users. More info can be found in [KIP-373](https://cwiki.apache.org/confluence/display/KAFKA/KIP-373%3A+Allow+users+to+create+delegation+tokens+for+other+users).
+
+
+
+### Operations and Resources on Protocols
+
+In the below table we'll list the valid operations on resources that are executed by the Kafka API protocols.
+
+Protocol (API key) | Operation | Resource | Note
+---|---|---|---
+PRODUCE (0) | Write | TransactionalId | An transactional producer which has its transactional.id set requires this privilege.
+PRODUCE (0) | IdempotentWrite | Cluster | An idempotent produce action requires this privilege.
+PRODUCE (0) | Write | Topic | This applies to a normal produce action.
+FETCH (1) | ClusterAction | Cluster | A follower must have ClusterAction on the Cluster resource in order to fetch partition data.
+FETCH (1) | Read | Topic | Regular Kafka consumers need READ permission on each partition they are fetching.
+LIST_OFFSETS (2) | Describe | Topic |
+METADATA (3) | Describe | Topic |
+METADATA (3) | Create | Cluster | If topic auto-creation is enabled, then the broker-side API will check for the existence of a Cluster level privilege. If it's found then it'll allow creating the topic, otherwise it'll iterate through the Topic level privileges (see the next one).
+METADATA (3) | Create | Topic | This authorizes auto topic creation if enabled but the given user doesn't have a cluster level permission (above).
+LEADER_AND_ISR (4) | ClusterAction | Cluster |
+STOP_REPLICA (5) | ClusterAction | Cluster |
+UPDATE_METADATA (6) | ClusterAction | Cluster |
+CONTROLLED_SHUTDOWN (7) | ClusterAction | Cluster |
+OFFSET_COMMIT (8) | Read | Group | An offset can only be committed if it's authorized to the given group and the topic too (see below). Group access is checked first, then Topic access.
+OFFSET_COMMIT (8) | Read | Topic | Since offset commit is part of the consuming process, it needs privileges for the read action.
+OFFSET_FETCH (9) | Describe | Group | Similarly to OFFSET_COMMIT, the application must have privileges on group and topic level too to be able to fetch. However in this case it requires describe access instead of read. Group access is checked first, then Topic access.
+OFFSET_FETCH (9) | Describe | Topic |
+FIND_COORDINATOR (10) | Describe | Group | The FIND_COORDINATOR request can be of "Group" type in which case it is looking for consumergroup coordinators. This privilege would represent the Group mode.
+FIND_COORDINATOR (10) | Describe | TransactionalId | This applies only on transactional producers and checked when a producer tries to find the transaction coordinator.
+JOIN_GROUP (11) | Read | Group |
+HEARTBEAT (12) | Read | Group |
+LEAVE_GROUP (13) | Read | Group |
+SYNC_GROUP (14) | Read | Group |
+DESCRIBE_GROUPS (15) | Describe | Group |
+LIST_GROUPS (16) | Describe | Cluster | When the broker checks to authorize a list_groups request it first checks for this cluster level authorization. If none found then it proceeds to check the groups individually. This operation doesn't return CLUSTER_AUTHORIZATION_FAILED.
+LIST_GROUPS (16) | Describe | Group | If none of the groups are authorized, then just an empty response will be sent back instead of an error. This operation doesn't return CLUSTER_AUTHORIZATION_FAILED. This is applicable from the 2.1 release.
+SASL_HANDSHAKE (17) | | | The SASL handshake is part of the authentication process and therefore it's not possible to apply any kind of authorization here.
+API_VERSIONS (18) | | | The API_VERSIONS request is part of the Kafka protocol handshake and happens on connection and before any authentication. Therefore it's not possible to control this with authorization.
+CREATE_TOPICS (19) | Create | Cluster | If there is no cluster level authorization then it won't return CLUSTER_AUTHORIZATION_FAILED but fall back to use topic level, which is just below. That'll throw error if there is a problem.
+CREATE_TOPICS (19) | Create | Topic | This is applicable from the 2.0 release.
+DELETE_TOPICS (20) | Delete | Topic |
+DELETE_RECORDS (21) | Delete | Topic |
+INIT_PRODUCER_ID (22) | Write | TransactionalId |
+INIT_PRODUCER_ID (22) | IdempotentWrite | Cluster |
+OFFSET_FOR_LEADER_EPOCH (23) | ClusterAction | Cluster | If there is no cluster level privilege for this operation, then it'll check for topic level one.
+OFFSET_FOR_LEADER_EPOCH (23) | Describe | Topic | This is applicable from the 2.1 release.
+ADD_PARTITIONS_TO_TXN (24) | Write | TransactionalId | This API is only applicable to transactional requests. It first checks for the Write action on the TransactionalId resource, then it checks the Topic in subject (below).
+ADD_PARTITIONS_TO_TXN (24) | Write | Topic |
+ADD_OFFSETS_TO_TXN (25) | Write | TransactionalId | Similarly to ADD_PARTITIONS_TO_TXN this is only applicable to transactional request. It first checks for Write action on the TransactionalId resource, then it checks whether it can Read on the given group (below).
+ADD_OFFSETS_TO_TXN (25) | Read | Group |
+END_TXN (26) | Write | TransactionalId |
+WRITE_TXN_MARKERS (27) | ClusterAction | Cluster |
+TXN_OFFSET_COMMIT (28) | Write | TransactionalId |
+TXN_OFFSET_COMMIT (28) | Read | Group |
+TXN_OFFSET_COMMIT (28) | Read | Topic |
+DESCRIBE_ACLS (29) | Describe | Cluster |
+CREATE_ACLS (30) | Alter | Cluster |
+DELETE_ACLS (31) | Alter | Cluster |
+DESCRIBE_CONFIGS (32) | DescribeConfigs | Cluster | If broker configs are requested, then the broker will check cluster level privileges.
+DESCRIBE_CONFIGS (32) | DescribeConfigs | Topic | If topic configs are requested, then the broker will check topic level privileges.
+ALTER_CONFIGS (33) | AlterConfigs | Cluster | If broker configs are altered, then the broker will check cluster level privileges.
+ALTER_CONFIGS (33) | AlterConfigs | Topic | If topic configs are altered, then the broker will check topic level privileges.
+ALTER_REPLICA_LOG_DIRS (34) | Alter | Cluster |
+DESCRIBE_LOG_DIRS (35) | Describe | Cluster | An empty response will be returned on authorization failure.
+SASL_AUTHENTICATE (36) | | | SASL_AUTHENTICATE is part of the authentication process and therefore it's not possible to apply any kind of authorization here.
+CREATE_PARTITIONS (37) | Alter | Topic |
+CREATE_DELEGATION_TOKEN (38) | | | Creating delegation tokens has special rules, for this please see the Authentication using Delegation Tokens section.
+CREATE_DELEGATION_TOKEN (38) | CreateTokens | User | Allows creating delegation tokens for the User resource.
+RENEW_DELEGATION_TOKEN (39) | | | Renewing delegation tokens has special rules, for this please see the Authentication using Delegation Tokens section.
+EXPIRE_DELEGATION_TOKEN (40) | | | Expiring delegation tokens has special rules, for this please see the Authentication using Delegation Tokens section.
+DESCRIBE_DELEGATION_TOKEN (41) | Describe | DelegationToken | Describing delegation tokens has special rules, for this please see the Authentication using Delegation Tokens section.
+DESCRIBE_DELEGATION_TOKEN (41) | DescribeTokens | User | Allows describing delegation tokens of the User resource.
+DELETE_GROUPS (42) | Delete | Group |
+ELECT_PREFERRED_LEADERS (43) | ClusterAction | Cluster |
+INCREMENTAL_ALTER_CONFIGS (44) | AlterConfigs | Cluster | If broker configs are altered, then the broker will check cluster level privileges.
+INCREMENTAL_ALTER_CONFIGS (44) | AlterConfigs | Topic | If topic configs are altered, then the broker will check topic level privileges.
+ALTER_PARTITION_REASSIGNMENTS (45) | Alter | Cluster |
+LIST_PARTITION_REASSIGNMENTS (46) | Describe | Cluster |
+OFFSET_DELETE (47) | Delete | Group |
+OFFSET_DELETE (47) | Read | Topic |
+DESCRIBE_CLIENT_QUOTAS (48) | DescribeConfigs | Cluster |
+ALTER_CLIENT_QUOTAS (49) | AlterConfigs | Cluster |
+DESCRIBE_USER_SCRAM_CREDENTIALS (50) | Describe | Cluster |
+ALTER_USER_SCRAM_CREDENTIALS (51) | Alter | Cluster |
+VOTE (52) | ClusterAction | Cluster |
+BEGIN_QUORUM_EPOCH (53) | ClusterAction | Cluster |
+END_QUORUM_EPOCH (54) | ClusterAction | Cluster |
+DESCRIBE_QUORUM (55) | Describe | Cluster |
+ALTER_PARTITION (56) | ClusterAction | Cluster |
+UPDATE_FEATURES (57) | Alter | Cluster |
+ENVELOPE (58) | ClusterAction | Cluster |
+FETCH_SNAPSHOT (59) | ClusterAction | Cluster |
+DESCRIBE_CLUSTER (60) | Describe | Cluster |
+DESCRIBE_PRODUCERS (61) | Read | Topic |
+BROKER_REGISTRATION (62) | ClusterAction | Cluster |
+BROKER_HEARTBEAT (63) | ClusterAction | Cluster |
+UNREGISTER_BROKER (64) | Alter | Cluster |
+DESCRIBE_TRANSACTIONS (65) | Describe | TransactionalId |
+LIST_TRANSACTIONS (66) | Describe | TransactionalId |
+ALLOCATE_PRODUCER_IDS (67) | ClusterAction | Cluster |
+CONSUMER_GROUP_HEARTBEAT (68) | Read | Group |
+
\ No newline at end of file
diff --git a/docs/security/encryption-and-authentication-using-ssl.md b/docs/security/encryption-and-authentication-using-ssl.md
new file mode 100644
index 0000000000000..ead912e1b34e4
--- /dev/null
+++ b/docs/security/encryption-and-authentication-using-ssl.md
@@ -0,0 +1,286 @@
+---
+title: Encryption and Authentication using SSL
+description: Encryption and Authentication using SSL
+weight: 3
+tags: ['kafka', 'docs']
+aliases:
+keywords:
+type: docs
+---
+
+Apache Kafka allows clients to use SSL for encryption of traffic as well as authentication. By default, SSL is disabled but can be turned on if needed. The following paragraphs explain in detail how to set up your own PKI infrastructure, use it to create certificates and configure Kafka to use these.
+
+ 1. #### Generate SSL key and certificate for each Kafka broker
+
+The first step of deploying one or more brokers with SSL support is to generate a public/private keypair for every server. Since Kafka expects all keys and certificates to be stored in keystores we will use Java's keytool command for this task. The tool supports two different keystore formats, the Java specific jks format which has been deprecated by now, as well as PKCS12. PKCS12 is the default format as of Java version 9, to ensure this format is being used regardless of the Java version in use all following commands explicitly specify the PKCS12 format.
+
+ > keytool -keystore {keystorefile} -alias localhost -validity {validity} -genkey -keyalg RSA -storetype pkcs12
+
+You need to specify two parameters in the above command:
+ 1. keystorefile: the keystore file that stores the keys (and later the certificate) for this broker. The keystore file contains the private and public keys of this broker, therefore it needs to be kept safe. Ideally this step is run on the Kafka broker that the key will be used on, as this key should never be transmitted/leave the server that it is intended for.
+ 2. validity: the valid time of the key in days. Please note that this differs from the validity period for the certificate, which will be determined in Signing the certificate. You can use the same key to request multiple certificates: if your key has a validity of 10 years, but your CA will only sign certificates that are valid for one year, you can use the same key with 10 certificates over time.
+
+To obtain a certificate that can be used with the private key that was just created a certificate signing request needs to be created. This signing request, when signed by a trusted CA results in the actual certificate which can then be installed in the keystore and used for authentication purposes.
+To generate certificate signing requests run the following command for all server keystores created so far.
+
+ > keytool -keystore server.keystore.jks -alias localhost -validity {validity} -genkey -keyalg RSA -destkeystoretype pkcs12 -ext SAN=DNS:{FQDN},IP:{IPADDRESS1}
+
+This command assumes that you want to add hostname information to the certificate, if this is not the case, you can omit the extension parameter `-ext SAN=DNS:{FQDN},IP:{IPADDRESS1}`. Please see below for more information on this.
+
+### Host Name Verification
+
+Host name verification, when enabled, is the process of checking attributes from the certificate that is presented by the server you are connecting to against the actual hostname or ip address of that server to ensure that you are indeed connecting to the correct server.
+The main reason for this check is to prevent man-in-the-middle attacks. For Kafka, this check has been disabled by default for a long time, but as of Kafka 2.0.0 host name verification of servers is enabled by default for client connections as well as inter-broker connections.
+Server host name verification may be disabled by setting `ssl.endpoint.identification.algorithm` to an empty string.
+For dynamically configured broker listeners, hostname verification may be disabled using `kafka-configs.sh`:
+
+
+ > bin/kafka-configs.sh --bootstrap-server localhost:9093 --entity-type brokers --entity-name 0 --alter --add-config "listener.name.internal.ssl.endpoint.identification.algorithm="
+
+**Note:**
+
+Normally there is no good reason to disable hostname verification apart from being the quickest way to "just get it to work" followed by the promise to "fix it later when there is more time"!
+Getting hostname verification right is not that hard when done at the right time, but gets much harder once the cluster is up and running - do yourself a favor and do it now!
+
+If host name verification is enabled, clients will verify the server's fully qualified domain name (FQDN) or ip address against one of the following two fields:
+ 1. Common Name (CN)
+ 2. [Subject Alternative Name (SAN)](https://tools.ietf.org/html/rfc5280#section-4.2.1.6)
+
+While Kafka checks both fields, usage of the common name field for hostname verification has been [deprecated](https://tools.ietf.org/html/rfc2818#section-3.1) since 2000 and should be avoided if possible. In addition the SAN field is much more flexible, allowing for multiple DNS and IP entries to be declared in a certificate.
+Another advantage is that if the SAN field is used for hostname verification the common name can be set to a more meaningful value for authorization purposes. Since we need the SAN field to be contained in the signed certificate, it will be specified when generating the signing request. It can also be specified when generating the keypair, but this will not automatically be copied into the signing request.
+To add a SAN field append the following argument ` -ext SAN=DNS:{FQDN},IP:{IPADDRESS}` to the keytool command:
+
+ > keytool -keystore server.keystore.jks -alias localhost -validity {validity} -genkey -keyalg RSA -destkeystoretype pkcs12 -ext SAN=DNS:{FQDN},IP:{IPADDRESS1}
+
+ 2. #### Creating your own CA
+
+After this step each machine in the cluster has a public/private key pair which can already be used to encrypt traffic and a certificate signing request, which is the basis for creating a certificate. To add authentication capabilities this signing request needs to be signed by a trusted authority, which will be created in this step.
+
+A certificate authority (CA) is responsible for signing certificates. CAs works likes a government that issues passports - the government stamps (signs) each passport so that the passport becomes difficult to forge. Other governments verify the stamps to ensure the passport is authentic. Similarly, the CA signs the certificates, and the cryptography guarantees that a signed certificate is computationally difficult to forge. Thus, as long as the CA is a genuine and trusted authority, the clients have a strong assurance that they are connecting to the authentic machines.
+
+For this guide we will be our own Certificate Authority. When setting up a production cluster in a corporate environment these certificates would usually be signed by a corporate CA that is trusted throughout the company. Please see Common Pitfalls in Production for some things to consider for this case.
+
+Due to a [bug](https://www.openssl.org/docs/man1.1.1/man1/x509.html#BUGS) in OpenSSL, the x509 module will not copy requested extension fields from CSRs into the final certificate. Since we want the SAN extension to be present in our certificate to enable hostname verification, we'll use the _ca_ module instead. This requires some additional configuration to be in place before we generate our CA keypair.
+Save the following listing into a file called openssl-ca.cnf and adjust the values for validity and common attributes as necessary.
+
+ HOME = .
+ RANDFILE = $ENV::HOME/.rnd
+
+ ####################################################################
+ [ ca ]
+ default_ca = CA_default # The default ca section
+
+ [ CA_default ]
+
+ base_dir = .
+ certificate = $base_dir/cacert.pem # The CA certificate
+ private_key = $base_dir/cakey.pem # The CA private key
+ new_certs_dir = $base_dir # Location for new certs after signing
+ database = $base_dir/index.txt # Database index file
+ serial = $base_dir/serial.txt # The current serial number
+
+ default_days = 1000 # How long to certify for
+ default_crl_days = 30 # How long before next CRL
+ default_md = sha256 # Use public key default MD
+ preserve = no # Keep passed DN ordering
+
+ x509_extensions = ca_extensions # The extensions to add to the cert
+
+ email_in_dn = no # Don't concat the email in the DN
+ copy_extensions = copy # Required to copy SANs from CSR to cert
+
+ ####################################################################
+ [ req ]
+ default_bits = 4096
+ default_keyfile = cakey.pem
+ distinguished_name = ca_distinguished_name
+ x509_extensions = ca_extensions
+ string_mask = utf8only
+
+ ####################################################################
+ [ ca_distinguished_name ]
+ countryName = Country Name (2 letter code)
+ countryName_default = DE
+
+ stateOrProvinceName = State or Province Name (full name)
+ stateOrProvinceName_default = Test Province
+
+ localityName = Locality Name (eg, city)
+ localityName_default = Test Town
+
+ organizationName = Organization Name (eg, company)
+ organizationName_default = Test Company
+
+ organizationalUnitName = Organizational Unit (eg, division)
+ organizationalUnitName_default = Test Unit
+
+ commonName = Common Name (e.g. server FQDN or YOUR name)
+ commonName_default = Test Name
+
+ emailAddress = Email Address
+ emailAddress_default = test@test.com
+
+ ####################################################################
+ [ ca_extensions ]
+
+ subjectKeyIdentifier = hash
+ authorityKeyIdentifier = keyid:always, issuer
+ basicConstraints = critical, CA:true
+ keyUsage = keyCertSign, cRLSign
+
+ ####################################################################
+ [ signing_policy ]
+ countryName = optional
+ stateOrProvinceName = optional
+ localityName = optional
+ organizationName = optional
+ organizationalUnitName = optional
+ commonName = supplied
+ emailAddress = optional
+
+ ####################################################################
+ [ signing_req ]
+ subjectKeyIdentifier = hash
+ authorityKeyIdentifier = keyid,issuer
+ basicConstraints = CA:FALSE
+ keyUsage = digitalSignature, keyEncipherment
+
+Then create a database and serial number file, these will be used to keep track of which certificates were signed with this CA. Both of these are simply text files that reside in the same directory as your CA keys.
+
+ > echo 01 > serial.txt
+ > touch index.txt
+
+With these steps done you are now ready to generate your CA that will be used to sign certificates later.
+
+ > openssl req -x509 -config openssl-ca.cnf -newkey rsa:4096 -sha256 -nodes -out cacert.pem -outform PEM
+
+The CA is simply a public/private key pair and certificate that is signed by itself, and is only intended to sign other certificates.
+This keypair should be kept very safe, if someone gains access to it, they can create and sign certificates that will be trusted by your infrastructure, which means they will be able to impersonate anybody when connecting to any service that trusts this CA.
+The next step is to add the generated CA to the **clients' truststore** so that the clients can trust this CA:
+
+ > keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert
+
+**Note:** If you configure the Kafka brokers to require client authentication by setting ssl.client.auth to be "requested" or "required" in the Kafka brokers config then you must provide a truststore for the Kafka brokers as well and it should have all the CA certificates that clients' keys were signed by.
+
+ > keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert
+
+In contrast to the keystore in step 1 that stores each machine's own identity, the truststore of a client stores all the certificates that the client should trust. Importing a certificate into one's truststore also means trusting all certificates that are signed by that certificate. As the analogy above, trusting the government (CA) also means trusting all passports (certificates) that it has issued. This attribute is called the chain of trust, and it is particularly useful when deploying SSL on a large Kafka cluster. You can sign all certificates in the cluster with a single CA, and have all machines share the same truststore that trusts the CA. That way all machines can authenticate all other machines.
+ 3. #### Signing the certificate
+
+Then sign it with the CA:
+
+ > openssl ca -config openssl-ca.cnf -policy signing_policy -extensions signing_req -out {server certificate} -infiles {certificate signing request}
+
+Finally, you need to import both the certificate of the CA and the signed certificate into the keystore:
+
+ > keytool -keystore {keystore} -alias CARoot -import -file {CA certificate}
+ > keytool -keystore {keystore} -alias localhost -import -file cert-signed
+
+The definitions of the parameters are the following:
+ 1. keystore: the location of the keystore
+ 2. CA certificate: the certificate of the CA
+ 3. certificate signing request: the csr created with the server key
+ 4. server certificate: the file to write the signed certificate of the server to
+This will leave you with one truststore called _truststore.jks_ \- this can be the same for all clients and brokers and does not contain any sensitive information, so there is no need to secure this.
+Additionally you will have one _server.keystore.jks_ file per node which contains that nodes keys, certificate and your CAs certificate, please refer to Configuring Kafka Brokers and Configuring Kafka Clients for information on how to use these files.
+
+For some tooling assistance on this topic, please check out the [easyRSA](https://github.com/OpenVPN/easy-rsa) project which has extensive scripting in place to help with these steps.
+
+### SSL key and certificates in PEM format
+
+From 2.7.0 onwards, SSL key and trust stores can be configured for Kafka brokers and clients directly in the configuration in PEM format. This avoids the need to store separate files on the file system and benefits from password protection features of Kafka configuration. PEM may also be used as the store type for file-based key and trust stores in addition to JKS and PKCS12. To configure PEM key store directly in the broker or client configuration, private key in PEM format should be provided in `ssl.keystore.key` and the certificate chain in PEM format should be provided in `ssl.keystore.certificate.chain`. To configure trust store, trust certificates, e.g. public certificate of CA, should be provided in `ssl.truststore.certificates`. Since PEM is typically stored as multi-line base-64 strings, the configuration value can be included in Kafka configuration as multi-line strings with lines terminating in backslash ('\') for line continuation.
+
+Store password configs `ssl.keystore.password` and `ssl.truststore.password` are not used for PEM. If private key is encrypted using a password, the key password must be provided in `ssl.key.password`. Private keys may be provided in unencrypted form without a password. In production deployments, configs should be encrypted or externalized using password protection feature in Kafka in this case. Note that the default SSL engine factory has limited capabilities for decryption of encrypted private keys when external tools like OpenSSL are used for encryption. Third party libraries like BouncyCastle may be integrated with a custom `SslEngineFactory` to support a wider range of encrypted private keys.
+
+ 4. #### Common Pitfalls in Production
+
+The above paragraphs show the process to create your own CA and use it to sign certificates for your cluster. While very useful for sandbox, dev, test, and similar systems, this is usually not the correct process to create certificates for a production cluster in a corporate environment. Enterprises will normally operate their own CA and users can send in CSRs to be signed with this CA, which has the benefit of users not being responsible to keep the CA secure as well as a central authority that everybody can trust. However it also takes away a lot of control over the process of signing certificates from the user. Quite often the persons operating corporate CAs will apply tight restrictions on certificates that can cause issues when trying to use these certificates with Kafka.
+ 1. **[Extended Key Usage](https://tools.ietf.org/html/rfc5280#section-4.2.1.12)**
+Certificates may contain an extension field that controls the purpose for which the certificate can be used. If this field is empty, there are no restrictions on the usage, but if any usage is specified in here, valid SSL implementations have to enforce these usages.
+Relevant usages for Kafka are:
+ * Client authentication
+ * Server authentication
+Kafka brokers need both these usages to be allowed, as for intra-cluster communication every broker will behave as both the client and the server towards other brokers. It is not uncommon for corporate CAs to have a signing profile for webservers and use this for Kafka as well, which will only contain the _serverAuth_ usage value and cause the SSL handshake to fail.
+ 2. **Intermediate Certificates**
+Corporate Root CAs are often kept offline for security reasons. To enable day-to-day usage, so called intermediate CAs are created, which are then used to sign the final certificates. When importing a certificate into the keystore that was signed by an intermediate CA it is necessary to provide the entire chain of trust up to the root CA. This can be done by simply _cat_ ing the certificate files into one combined certificate file and then importing this with keytool.
+ 3. **Failure to copy extension fields**
+CA operators are often hesitant to copy and requested extension fields from CSRs and prefer to specify these themselves as this makes it harder for a malicious party to obtain certificates with potentially misleading or fraudulent values. It is advisable to double check signed certificates, whether these contain all requested SAN fields to enable proper hostname verification. The following command can be used to print certificate details to the console, which should be compared with what was originally requested:
+
+ > openssl x509 -in certificate.crt -text -noout
+
+ 5. #### Configuring Kafka Brokers
+
+If SSL is not enabled for inter-broker communication (see below for how to enable it), both PLAINTEXT and SSL ports will be necessary.
+
+ listeners=PLAINTEXT://host.name:port,SSL://host.name:port
+
+Following SSL configs are needed on the broker side
+
+ ssl.keystore.location=/var/private/ssl/server.keystore.jks
+ ssl.keystore.password=test1234
+ ssl.key.password=test1234
+ ssl.truststore.location=/var/private/ssl/server.truststore.jks
+ ssl.truststore.password=test1234
+
+Note: ssl.truststore.password is technically optional but highly recommended. If a password is not set access to the truststore is still available, but integrity checking is disabled. Optional settings that are worth considering:
+ 1. ssl.client.auth=none ("required" => client authentication is required, "requested" => client authentication is requested and client without certs can still connect. The usage of "requested" is discouraged as it provides a false sense of security and misconfigured clients will still connect successfully.)
+ 2. ssl.cipher.suites (Optional). A cipher suite is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. (Default is an empty list)
+ 3. ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 (list out the SSL protocols that you are going to accept from clients. Do note that SSL is deprecated in favor of TLS and using SSL in production is not recommended)
+ 4. ssl.keystore.type=JKS
+ 5. ssl.truststore.type=JKS
+ 6. ssl.secure.random.implementation=SHA1PRNG
+If you want to enable SSL for inter-broker communication, add the following to the server.properties file (it defaults to PLAINTEXT)
+
+ security.inter.broker.protocol=SSL
+
+Due to import regulations in some countries, the Oracle implementation limits the strength of cryptographic algorithms available by default. If stronger algorithms are needed (for example, AES with 256-bit keys), the [JCE Unlimited Strength Jurisdiction Policy Files](https://www.oracle.com/technetwork/java/javase/downloads/index.html) must be obtained and installed in the JDK/JRE. See the [JCA Providers Documentation](https://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html) for more information.
+
+The JRE/JDK will have a default pseudo-random number generator (PRNG) that is used for cryptography operations, so it is not required to configure the implementation used with the `ssl.secure.random.implementation`. However, there are performance issues with some implementations (notably, the default chosen on Linux systems, `NativePRNG`, utilizes a global lock). In cases where performance of SSL connections becomes an issue, consider explicitly setting the implementation to be used. The `SHA1PRNG` implementation is non-blocking, and has shown very good performance characteristics under heavy load (50 MB/sec of produced messages, plus replication traffic, per-broker).
+
+Once you start the broker you should be able to see in the server.log
+
+ with addresses: PLAINTEXT -> EndPoint(192.168.64.1,9092,PLAINTEXT),SSL -> EndPoint(192.168.64.1,9093,SSL)
+
+To check quickly if the server keystore and truststore are setup properly you can run the following command
+
+ > openssl s_client -debug -connect localhost:9093 -tls1
+
+(Note: TLSv1 should be listed under ssl.enabled.protocols)
+In the output of this command you should see server's certificate:
+
+ -----BEGIN CERTIFICATE-----
+ {variable sized random bytes}
+ -----END CERTIFICATE-----
+ subject=/C=US/ST=CA/L=Santa Clara/O=org/OU=org/CN=Sriharsha Chintalapani
+ issuer=/C=US/ST=CA/L=Santa Clara/O=org/OU=org/CN=kafka/emailAddress=test@test.com
+
+If the certificate does not show up or if there are any other error messages then your keystore is not setup properly.
+ 6. #### Configuring Kafka Clients
+
+SSL is supported only for the new Kafka Producer and Consumer, the older API is not supported. The configs for SSL will be the same for both producer and consumer.
+If client authentication is not required in the broker, then the following is a minimal configuration example:
+
+ security.protocol=SSL
+ ssl.truststore.location=/var/private/ssl/client.truststore.jks
+ ssl.truststore.password=test1234
+
+Note: ssl.truststore.password is technically optional but highly recommended. If a password is not set access to the truststore is still available, but integrity checking is disabled. If client authentication is required, then a keystore must be created like in step 1 and the following must also be configured:
+
+ ssl.keystore.location=/var/private/ssl/client.keystore.jks
+ ssl.keystore.password=test1234
+ ssl.key.password=test1234
+
+Other configuration settings that may also be needed depending on our requirements and the broker configuration:
+ 1. ssl.provider (Optional). The name of the security provider used for SSL connections. Default value is the default security provider of the JVM.
+ 2. ssl.cipher.suites (Optional). A cipher suite is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol.
+ 3. ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1. It should list at least one of the protocols configured on the broker side
+ 4. ssl.truststore.type=JKS
+ 5. ssl.keystore.type=JKS
+
+Examples using console-producer and console-consumer:
+
+ > kafka-console-producer.sh --bootstrap-server localhost:9093 --topic test --producer.config client-ssl.properties
+ > kafka-console-consumer.sh --bootstrap-server localhost:9093 --topic test --consumer.config client-ssl.properties
+
+
+
diff --git a/docs/security/incorporating-security-features-in-a-running-cluster.md b/docs/security/incorporating-security-features-in-a-running-cluster.md
new file mode 100644
index 0000000000000..2ac0f13f6ecb3
--- /dev/null
+++ b/docs/security/incorporating-security-features-in-a-running-cluster.md
@@ -0,0 +1,74 @@
+---
+title: Incorporating Security Features in a Running Cluster
+description: Incorporating Security Features in a Running Cluster
+weight: 6
+tags: ['kafka', 'docs']
+aliases:
+keywords:
+type: docs
+---
+
+You can secure a running cluster via one or more of the supported protocols discussed previously. This is done in phases:
+
+ * Incrementally bounce the cluster nodes to open additional secured port(s).
+ * Restart clients using the secured rather than PLAINTEXT port (assuming you are securing the client-broker connection).
+ * Incrementally bounce the cluster again to enable broker-to-broker security (if this is required)
+ * A final incremental bounce to close the PLAINTEXT port.
+
+
+
+The specific steps for configuring SSL and SASL are described in sections 7.3 and 7.4. Follow these steps to enable security for your desired protocol(s).
+
+The security implementation lets you configure different protocols for both broker-client and broker-broker communication. These must be enabled in separate bounces. A PLAINTEXT port must be left open throughout so brokers and/or clients can continue to communicate.
+
+When performing an incremental bounce stop the brokers cleanly via a SIGTERM. It's also good practice to wait for restarted replicas to return to the ISR list before moving onto the next node.
+
+As an example, say we wish to encrypt both broker-client and broker-broker communication with SSL. In the first incremental bounce, an SSL port is opened on each node:
+
+
+ listeners=PLAINTEXT://broker1:9091,SSL://broker1:9092
+
+We then restart the clients, changing their config to point at the newly opened, secured port:
+
+
+ bootstrap.servers = [broker1:9092,...]
+ security.protocol = SSL
+ ...etc
+
+In the second incremental server bounce we instruct Kafka to use SSL as the broker-broker protocol (which will use the same SSL port):
+
+
+ listeners=PLAINTEXT://broker1:9091,SSL://broker1:9092
+ security.inter.broker.protocol=SSL
+
+In the final bounce we secure the cluster by closing the PLAINTEXT port:
+
+
+ listeners=SSL://broker1:9092
+ security.inter.broker.protocol=SSL
+
+Alternatively we might choose to open multiple ports so that different protocols can be used for broker-broker and broker-client communication. Say we wished to use SSL encryption throughout (i.e. for broker-broker and broker-client communication) but we'd like to add SASL authentication to the broker-client connection also. We would achieve this by opening two additional ports during the first bounce:
+
+
+ listeners=PLAINTEXT://broker1:9091,SSL://broker1:9092,SASL_SSL://broker1:9093
+
+We would then restart the clients, changing their config to point at the newly opened, SASL & SSL secured port:
+
+
+ bootstrap.servers = [broker1:9093,...]
+ security.protocol = SASL_SSL
+ ...etc
+
+The second server bounce would switch the cluster to use encrypted broker-broker communication via the SSL port we previously opened on port 9092:
+
+
+ listeners=PLAINTEXT://broker1:9091,SSL://broker1:9092,SASL_SSL://broker1:9093
+ security.inter.broker.protocol=SSL
+
+The final bounce secures the cluster by closing the PLAINTEXT port.
+
+
+ listeners=SSL://broker1:9092,SASL_SSL://broker1:9093
+ security.inter.broker.protocol=SSL
+
+ZooKeeper can be secured independently of the Kafka cluster. The steps for doing this are covered in section 7.7.2.
diff --git a/docs/security/listener-configuration.md b/docs/security/listener-configuration.md
new file mode 100644
index 0000000000000..4b3cb50b851a3
--- /dev/null
+++ b/docs/security/listener-configuration.md
@@ -0,0 +1,80 @@
+---
+title: Listener Configuration
+description: Listener Configuration
+weight: 2
+tags: ['kafka', 'docs']
+aliases:
+keywords:
+type: docs
+---
+
+In order to secure a Kafka cluster, it is necessary to secure the channels that are used to communicate with the servers. Each server must define the set of listeners that are used to receive requests from clients as well as other servers. Each listener may be configured to authenticate clients using various mechanisms and to ensure traffic between the server and the client is encrypted. This section provides a primer for the configuration of listeners.
+
+Kafka servers support listening for connections on multiple ports. This is configured through the `listeners` property in the server configuration, which accepts a comma-separated list of the listeners to enable. At least one listener must be defined on each server. The format of each listener defined in `listeners` is given below:
+
+
+ {LISTENER_NAME}://{hostname}:{port}
+
+The `LISTENER_NAME` is usually a descriptive name which defines the purpose of the listener. For example, many configurations use a separate listener for client traffic, so they might refer to the corresponding listener as `CLIENT` in the configuration:
+
+
+ listeners=CLIENT://localhost:9092
+
+The security protocol of each listener is defined in a separate configuration: `listener.security.protocol.map`. The value is a comma-separated list of each listener mapped to its security protocol. For example, the follow value configuration specifies that the `CLIENT` listener will use SSL while the `BROKER` listener will use plaintext.
+
+
+ listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT
+
+Possible options (case-insensitive) for the security protocol are given below:
+
+ 1. PLAINTEXT
+ 2. SSL
+ 3. SASL_PLAINTEXT
+ 4. SASL_SSL
+
+
+
+The plaintext protocol provides no security and does not require any additional configuration. In the following sections, this document covers how to configure the remaining protocols.
+
+If each required listener uses a separate security protocol, it is also possible to use the security protocol name as the listener name in `listeners`. Using the example above, we could skip the definition of the `CLIENT` and `BROKER` listeners using the following definition:
+
+
+ listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093
+
+However, we recommend users to provide explicit names for the listeners since it makes the intended usage of each listener clearer.
+
+Among the listeners in this list, it is possible to declare the listener to be used for inter-broker communication by setting the `inter.broker.listener.name` configuration to the name of the listener. The primary purpose of the inter-broker listener is partition replication. If not defined, then the inter-broker listener is determined by the security protocol defined by `security.inter.broker.protocol`, which defaults to `PLAINTEXT`.
+
+For legacy clusters which rely on Zookeeper to store cluster metadata, it is possible to declare a separate listener to be used for metadata propagation from the active controller to the brokers. This is defined by `control.plane.listener.name`. The active controller will use this listener when it needs to push metadata updates to the brokers in the cluster. The benefit of using a control plane listener is that it uses a separate processing thread, which makes it less likely for application traffic to impede timely propagation of metadata changes (such as partition leader and ISR updates). Note that the default value is null, which means that the controller will use the same listener defined by `inter.broker.listener`
+
+In a KRaft cluster, a broker is any server which has the `broker` role enabled in `process.roles` and a controller is any server which has the `controller` role enabled. Listener configuration depends on the role. The listener defined by `inter.broker.listener.name` is used exclusively for requests between brokers. Controllers, on the other hand, must use separate listener which is defined by the `controller.listener.names` configuration. This cannot be set to the same value as the inter-broker listener.
+
+Controllers receive requests both from other controllers and from brokers. For this reason, even if a server does not have the `controller` role enabled (i.e. it is just a broker), it must still define the controller listener along with any security properties that are needed to configure it. For example, we might use the following configuration on a standalone broker:
+
+
+ process.roles=broker
+ listeners=BROKER://localhost:9092
+ inter.broker.listener.name=BROKER
+ controller.quorum.voters=0@localhost:9093
+ controller.listener.names=CONTROLLER
+ listener.security.protocol.map=BROKER:SASL_SSL,CONTROLLER:SASL_SSL
+
+The controller listener is still configured in this example to use the `SASL_SSL` security protocol, but it is not included in `listeners` since the broker does not expose the controller listener itself. The port that will be used in this case comes from the `controller.quorum.voters` configuration, which defines the complete list of controllers.
+
+For KRaft servers which have both the broker and controller role enabled, the configuration is similar. The only difference is that the controller listener must be included in `listeners`:
+
+
+ process.roles=broker,controller
+ listeners=BROKER://localhost:9092,CONTROLLER://localhost:9093
+ inter.broker.listener.name=BROKER
+ controller.quorum.voters=0@localhost:9093
+ controller.listener.names=CONTROLLER
+ listener.security.protocol.map=BROKER:SASL_SSL,CONTROLLER:SASL_SSL
+
+It is a requirement for the port defined in `controller.quorum.voters` to exactly match one of the exposed controller listeners. For example, here the `CONTROLLER` listener is bound to port 9093. The connection string defined by `controller.quorum.voters` must then also use port 9093, as it does here.
+
+The controller will accept requests on all listeners defined by `controller.listener.names`. Typically there would be just one controller listener, but it is possible to have more. For example, this provides a way to change the active listener from one port or security protocol to another through a roll of the cluster (one roll to expose the new listener, and one roll to remove the old listener). When multiple controller listeners are defined, the first one in the list will be used for outbound requests.
+
+It is conventional in Kafka to use a separate listener for clients. This allows the inter-cluster listeners to be isolated at the network level. In the case of the controller listener in KRaft, the listener should be isolated since clients do not work with it anyway. Clients are expected to connect to any other listener configured on a broker. Any requests that are bound for the controller will be forwarded as described below
+
+In the following section, this document covers how to enable SSL on a listener for encryption as well as authentication. The subsequent section will then cover additional authentication mechanisms using SASL.
diff --git a/docs/security/security-overview.md b/docs/security/security-overview.md
new file mode 100644
index 0000000000000..cfd4efd6fa0a1
--- /dev/null
+++ b/docs/security/security-overview.md
@@ -0,0 +1,23 @@
+---
+title: Security Overview
+description: Security Overview
+weight: 1
+tags: ['kafka', 'docs']
+aliases:
+keywords:
+type: docs
+---
+
+In release 0.9.0.0, the Kafka community added a number of features that, used either separately or together, increases security in a Kafka cluster. The following security measures are currently supported:
+
+ 1. Authentication of connections to brokers from clients (producers and consumers), other brokers and tools, using either SSL or SASL. Kafka supports the following SASL mechanisms:
+ * SASL/GSSAPI (Kerberos) - starting at version 0.9.0.0
+ * SASL/PLAIN - starting at version 0.10.0.0
+ * SASL/SCRAM-SHA-256 and SASL/SCRAM-SHA-512 - starting at version 0.10.2.0
+ * SASL/OAUTHBEARER - starting at version 2.0
+ 2. Authentication of connections from brokers to ZooKeeper
+ 3. Encryption of data transferred between brokers and clients, between brokers, or between brokers and tools using SSL (Note that there is a performance degradation when SSL is enabled, the magnitude of which depends on the CPU type and the JVM implementation.)
+ 4. Authorization of read / write operations by clients
+ 5. Authorization is pluggable and integration with external authorization services is supported
+
+It's worth noting that security is optional - non-secured clusters are supported, as well as a mix of authenticated, unauthenticated, encrypted and non-encrypted clients. The guides below explain how to configure and use the security features in both clients and brokers.
diff --git a/docs/security/zookeeper-authentication.md b/docs/security/zookeeper-authentication.md
new file mode 100644
index 0000000000000..bd408ef53dd1f
--- /dev/null
+++ b/docs/security/zookeeper-authentication.md
@@ -0,0 +1,123 @@
+---
+title: ZooKeeper Authentication
+description: ZooKeeper Authentication
+weight: 7
+tags: ['kafka', 'docs']
+aliases:
+keywords:
+type: docs
+---
+
+ZooKeeper supports mutual TLS (mTLS) authentication beginning with the 3.5.x versions. Kafka supports authenticating to ZooKeeper with SASL and mTLS -- either individually or both together -- beginning with version 2.5. See [KIP-515: Enable ZK client to use the new TLS supported authentication](https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication) for more details.
+
+When using mTLS alone, every broker and any CLI tools (such as the ZooKeeper Security Migration Tool) should identify itself with the same Distinguished Name (DN) because it is the DN that is ACL'ed. This can be changed as described below, but it involves writing and deploying a custom ZooKeeper authentication provider. Generally each certificate should have the same DN but a different Subject Alternative Name (SAN) so that hostname verification of the brokers and any CLI tools by ZooKeeper will succeed.
+
+When using SASL authentication to ZooKeeper together with mTLS, both the SASL identity and either the DN that created the znode (i.e. the creating broker's certificate) or the DN of the Security Migration Tool (if migration was performed after the znode was created) will be ACL'ed, and all brokers and CLI tools will be authorized even if they all use different DNs because they will all use the same ACL'ed SASL identity. It is only when using mTLS authentication alone that all the DNs must match (and SANs become critical -- again, in the absence of writing and deploying a custom ZooKeeper authentication provider as described below).
+
+Use the broker properties file to set TLS configs for brokers as described below.
+
+Use the `--zk-tls-config-file ` option to set TLS configs in the Zookeeper Security Migration Tool. The `kafka-acls.sh` and `kafka-configs.sh` CLI tools also support the `--zk-tls-config-file ` option.
+
+Use the `-zk-tls-config-file ` option (note the single-dash rather than double-dash) to set TLS configs for the `zookeeper-shell.sh` CLI tool.
+
+## New clusters
+
+### ZooKeeper SASL Authentication
+
+To enable ZooKeeper SASL authentication on brokers, there are two necessary steps:
+
+ 1. Create a JAAS login file and set the appropriate system property to point to it as described above
+ 2. Set the configuration property `zookeeper.set.acl` in each broker to true
+
+The metadata stored in ZooKeeper for the Kafka cluster is world-readable, but can only be modified by the brokers. The rationale behind this decision is that the data stored in ZooKeeper is not sensitive, but inappropriate manipulation of that data can cause cluster disruption. We also recommend limiting the access to ZooKeeper via network segmentation (only brokers and some admin tools need access to ZooKeeper).
+
+### ZooKeeper Mutual TLS Authentication
+
+ZooKeeper mTLS authentication can be enabled with or without SASL authentication. As mentioned above, when using mTLS alone, every broker and any CLI tools (such as the ZooKeeper Security Migration Tool) must generally identify itself with the same Distinguished Name (DN) because it is the DN that is ACL'ed, which means each certificate should have an appropriate Subject Alternative Name (SAN) so that hostname verification of the brokers and any CLI tool by ZooKeeper will succeed.
+
+It is possible to use something other than the DN for the identity of mTLS clients by writing a class that extends `org.apache.zookeeper.server.auth.X509AuthenticationProvider` and overrides the method `protected String getClientId(X509Certificate clientCert)`. Choose a scheme name and set `authProvider.[scheme]` in ZooKeeper to be the fully-qualified class name of the custom implementation; then set `ssl.authProvider=[scheme]` to use it.
+
+Here is a sample (partial) ZooKeeper configuration for enabling TLS authentication. These configurations are described in the [ZooKeeper Admin Guide](https://zookeeper.apache.org/doc/r3.5.7/zookeeperAdmin.html#sc_authOptions).
+
+
+ secureClientPort=2182
+ serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
+ authProvider.x509=org.apache.zookeeper.server.auth.X509AuthenticationProvider
+ ssl.keyStore.location=/path/to/zk/keystore.jks
+ ssl.keyStore.password=zk-ks-passwd
+ ssl.trustStore.location=/path/to/zk/truststore.jks
+ ssl.trustStore.password=zk-ts-passwd
+
+**IMPORTANT** : ZooKeeper does not support setting the key password in the ZooKeeper server keystore to a value different from the keystore password itself. Be sure to set the key password to be the same as the keystore password.
+
+Here is a sample (partial) Kafka Broker configuration for connecting to ZooKeeper with mTLS authentication. These configurations are described above in Broker Configs.
+
+
+ # connect to the ZooKeeper port configured for TLS
+ zookeeper.connect=zk1:2182,zk2:2182,zk3:2182
+ # required to use TLS to ZooKeeper (default is false)
+ zookeeper.ssl.client.enable=true
+ # required to use TLS to ZooKeeper
+ zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
+ # define key/trust stores to use TLS to ZooKeeper; ignored unless zookeeper.ssl.client.enable=true
+ zookeeper.ssl.keystore.location=/path/to/kafka/keystore.jks
+ zookeeper.ssl.keystore.password=kafka-ks-passwd
+ zookeeper.ssl.truststore.location=/path/to/kafka/truststore.jks
+ zookeeper.ssl.truststore.password=kafka-ts-passwd
+ # tell broker to create ACLs on znodes
+ zookeeper.set.acl=true
+
+**IMPORTANT** : ZooKeeper does not support setting the key password in the ZooKeeper client (i.e. broker) keystore to a value different from the keystore password itself. Be sure to set the key password to be the same as the keystore password.
+
+## Migrating clusters
+
+If you are running a version of Kafka that does not support security or simply with security disabled, and you want to make the cluster secure, then you need to execute the following steps to enable ZooKeeper authentication with minimal disruption to your operations:
+
+ 1. Enable SASL and/or mTLS authentication on ZooKeeper. If enabling mTLS, you would now have both a non-TLS port and a TLS port, like this:
+
+ clientPort=2181
+ secureClientPort=2182
+ serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
+ authProvider.x509=org.apache.zookeeper.server.auth.X509AuthenticationProvider
+ ssl.keyStore.location=/path/to/zk/keystore.jks
+ ssl.keyStore.password=zk-ks-passwd
+ ssl.trustStore.location=/path/to/zk/truststore.jks
+ ssl.trustStore.password=zk-ts-passwd
+
+ 2. Perform a rolling restart of brokers setting the JAAS login file and/or defining ZooKeeper mutual TLS configurations (including connecting to the TLS-enabled ZooKeeper port) as required, which enables brokers to authenticate to ZooKeeper. At the end of the rolling restart, brokers are able to manipulate znodes with strict ACLs, but they will not create znodes with those ACLs
+ 3. If you enabled mTLS, disable the non-TLS port in ZooKeeper
+ 4. Perform a second rolling restart of brokers, this time setting the configuration parameter `zookeeper.set.acl` to true, which enables the use of secure ACLs when creating znodes
+ 5. Execute the ZkSecurityMigrator tool. To execute the tool, there is this script: `bin/zookeeper-security-migration.sh` with `zookeeper.acl` set to secure. This tool traverses the corresponding sub-trees changing the ACLs of the znodes. Use the `--zk-tls-config-file ` option if you enable mTLS.
+
+
+
+It is also possible to turn off authentication in a secure cluster. To do it, follow these steps:
+
+ 1. Perform a rolling restart of brokers setting the JAAS login file and/or defining ZooKeeper mutual TLS configurations, which enables brokers to authenticate, but setting `zookeeper.set.acl` to false. At the end of the rolling restart, brokers stop creating znodes with secure ACLs, but are still able to authenticate and manipulate all znodes
+ 2. Execute the ZkSecurityMigrator tool. To execute the tool, run this script `bin/zookeeper-security-migration.sh` with `zookeeper.acl` set to unsecure. This tool traverses the corresponding sub-trees changing the ACLs of the znodes. Use the `--zk-tls-config-file ` option if you need to set TLS configuration.
+ 3. If you are disabling mTLS, enable the non-TLS port in ZooKeeper
+ 4. Perform a second rolling restart of brokers, this time omitting the system property that sets the JAAS login file and/or removing ZooKeeper mutual TLS configuration (including connecting to the non-TLS-enabled ZooKeeper port) as required
+ 5. If you are disabling mTLS, disable the TLS port in ZooKeeper
+
+Here is an example of how to run the migration tool:
+
+
+ > bin/zookeeper-security-migration.sh --zookeeper.acl=secure --zookeeper.connect=localhost:2181
+
+Run this to see the full list of parameters:
+
+
+ > bin/zookeeper-security-migration.sh --help
+
+## Migrating the ZooKeeper ensemble
+
+It is also necessary to enable SASL and/or mTLS authentication on the ZooKeeper ensemble. To do it, we need to perform a rolling restart of the server and set a few properties. See above for mTLS information. Please refer to the ZooKeeper documentation for more detail:
+
+ 1. [Apache ZooKeeper documentation](https://zookeeper.apache.org/doc/r3.5.7/zookeeperProgrammers.html#sc_ZooKeeperAccessControl)
+ 2. [Apache ZooKeeper wiki](https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zookeeper+and+SASL)
+
+
+
+## ZooKeeper Quorum Mutual TLS Authentication
+
+It is possible to enable mTLS authentication between the ZooKeeper servers themselves. Please refer to the [ZooKeeper documentation](https://zookeeper.apache.org/doc/r3.5.7/zookeeperAdmin.html#Quorum+TLS) for more detail.
diff --git a/docs/security/zookeeper-encryption.md b/docs/security/zookeeper-encryption.md
new file mode 100644
index 0000000000000..d36ca1a2046a6
--- /dev/null
+++ b/docs/security/zookeeper-encryption.md
@@ -0,0 +1,25 @@
+---
+title: ZooKeeper Encryption
+description: ZooKeeper Encryption
+weight: 8
+tags: ['kafka', 'docs']
+aliases:
+keywords:
+type: docs
+---
+
+ZooKeeper connections that use mutual TLS are encrypted. Beginning with ZooKeeper version 3.5.7 (the version shipped with Kafka version 2.5) ZooKeeper supports a sever-side config `ssl.clientAuth` (case-insensitively: `want`/`need`/`none` are the valid options, the default is `need`), and setting this value to `none` in ZooKeeper allows clients to connect via a TLS-encrypted connection without presenting their own certificate. Here is a sample (partial) Kafka Broker configuration for connecting to ZooKeeper with just TLS encryption. These configurations are described above in Broker Configs.
+
+
+ # connect to the ZooKeeper port configured for TLS
+ zookeeper.connect=zk1:2182,zk2:2182,zk3:2182
+ # required to use TLS to ZooKeeper (default is false)
+ zookeeper.ssl.client.enable=true
+ # required to use TLS to ZooKeeper
+ zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
+ # define trust stores to use TLS to ZooKeeper; ignored unless zookeeper.ssl.client.enable=true
+ # no need to set keystore information assuming ssl.clientAuth=none on ZooKeeper
+ zookeeper.ssl.truststore.location=/path/to/kafka/truststore.jks
+ zookeeper.ssl.truststore.password=kafka-ts-passwd
+ # tell broker to create ACLs on znodes (if using SASL authentication, otherwise do not set this)
+ zookeeper.set.acl=true
diff --git a/docs/streams/_index.md b/docs/streams/_index.md
new file mode 100644
index 0000000000000..f68d72a5c1f49
--- /dev/null
+++ b/docs/streams/_index.md
@@ -0,0 +1,10 @@
+---
+title: Kafka Streams
+description:
+weight: 9
+tags: ['kafka', 'docs', 'streams']
+aliases:
+keywords:
+type: docs
+---
+
diff --git a/docs/streams/architecture.html b/docs/streams/architecture.html
deleted file mode 100644
index 1094f16595745..0000000000000
--- a/docs/streams/architecture.html
+++ /dev/null
@@ -1,206 +0,0 @@
-
-
-
-
-
-
-
-
-/bin/kafka-streams-application-reset
+
+The tool accepts the following parameters:
+
+
+ Option (* = required) Description
+ --------------------- -----------
+ * --application-id The Kafka Streams application ID
+ (application.id).
+ --bootstrap-server (deprecated) is specified. The server
+ (s) to connect to. The broker list
+ string in the form HOST1:PORT1,HOST2:
+ PORT2.
+ --bootstrap-servers DEPRECATED: Comma-separated list of
+ broker urls with format: HOST1:PORT1,
+ HOST2:PORT2 (default: localhost:9092)
+ --by-duration Reset offsets to offset by duration from
+ current timestamp. Format: 'PnDTnHnMnS'
+ --config-file Property file containing configs to be
+ passed to admin clients and embedded
+ consumer.
+ --dry-run Display the actions that would be
+ performed without executing the reset
+ commands.
+ --from-file Reset offsets to values defined in CSV
+ file.
+ --input-topics Comma-separated list of user input
+ topics. For these topics, the tool will
+ reset the offset to the earliest
+ available offset.
+ --intermediate-topics Comma-separated list of intermediate user
+ topics (topics used in the through()
+ method). For these topics, the tool
+ will skip to the end.
+ --internal-topics Comma-separated list of internal topics
+ to delete. Must be a subset of the
+ internal topics marked for deletion by
+ the default behaviour (do a dry-run without
+ this option to view these topics).
+ --shift-by Reset offsets shifting current offset by
+ 'n', where 'n' can be positive or
+ negative
+ --to-datetime Reset offsets to offset from datetime.
+ Format: 'YYYY-MM-DDTHH:mm:SS.sss'
+ --to-earliest Reset offsets to earliest offset.
+ --to-latest Reset offsets to latest offset.
+ --to-offset Reset offsets to a specific offset.
+ --force Force removing members of the consumer group
+ (intended to remove left-over members if
+ long session timeout was configured).
+
+Consider the following as reset-offset scenarios for `input-topics`:
+
+ * by-duration
+ * from-file
+ * shift-by
+ * to-datetime
+ * to-earliest
+ * to-latest
+ * to-offset
+
+
+
+Only one of these scenarios can be defined. If not, `to-earliest` will be executed by default
+
+All the other parameters can be combined as needed. For example, if you want to restart an application from an empty internal state, but not reprocess previous data, simply omit the parameters `--input-topics` and `--intermediate-topics`.
+
+# Step 2: Reset the local environments of your application instances
+
+For a complete application reset, you must delete the application's local state directory on any machines where the application instance was run. You must do this before restarting an application instance on the same machine. You can use either of these methods:
+
+ * The API method `KafkaStreams#cleanUp()` in your application code.
+ * Manually delete the corresponding local state directory (default location: `/tmp/kafka-streams/`). For more information, see [Streams](/37/javadoc/org/apache/kafka/streams/StreamsConfig.html#STATE_DIR_CONFIG) javadocs.
+
+
+
+ * [Documentation](/documentation)
+ * [Kafka Streams](/streams)
+ * [Developer Guide](/streams/developer-guide/)
+
+
diff --git a/docs/streams/developer-guide/config-streams.html b/docs/streams/developer-guide/config-streams.html
deleted file mode 100644
index 70dd5ff0c24b5..0000000000000
--- a/docs/streams/developer-guide/config-streams.html
+++ /dev/null
@@ -1,1224 +0,0 @@
-
-
-
-
-
-
-
-
- dlqProducer;
+> String dlqTopic;
+>
+> @Override
+> public DeserializationHandlerResponse handle(final ProcessorContext context,
+> final ConsumerRecord record,
+> final Exception exception) {
+>
+> log.warn("Exception caught during Deserialization, sending to the dead queue topic; " +
+> "taskId: {}, topic: {}, partition: {}, offset: {}",
+> context.taskId(), record.topic(), record.partition(), record.offset(),
+> exception);
+>
+> dlqProducer.send(new ProducerRecord<>(dlqTopic, record.timestamp(), record.key(), record.value(), record.headers())).get();
+>
+> return DeserializationHandlerResponse.CONTINUE;
+> }
+>
+> @Override
+> public void configure(final Map configs) {
+> dlqProducer = .. // get a producer from the configs map
+> dlqTopic = .. // get the topic name from the configs map
+> }
+> }
+
+### default.production.exception.handler
+
+> The default production exception handler allows you to manage exceptions triggered when trying to interact with a broker such as attempting to produce a record that is too large. By default, Kafka provides and uses the [DefaultProductionExceptionHandler](/37/javadoc/org/apache/kafka/streams/errors/DefaultProductionExceptionHandler.html) that always fails when these exceptions occur.
+>
+> Each exception handler can return a `FAIL` or `CONTINUE` depending on the record and the exception thrown. Returning `FAIL` will signal that Streams should shut down and `CONTINUE` will signal that Streams should ignore the issue and continue processing. If you want to provide an exception handler that always ignores records that are too large, you could implement something like the following:
+>
+>
+> import java.util.Properties;
+> import org.apache.kafka.streams.StreamsConfig;
+> import org.apache.kafka.common.errors.RecordTooLargeException;
+> import org.apache.kafka.streams.errors.ProductionExceptionHandler;
+> import org.apache.kafka.streams.errors.ProductionExceptionHandler.ProductionExceptionHandlerResponse;
+>
+> public class IgnoreRecordTooLargeHandler implements ProductionExceptionHandler {
+> public void configure(Map config) {}
+>
+> public ProductionExceptionHandlerResponse handle(final ProducerRecord record,
+> final Exception exception) {
+> if (exception instanceof RecordTooLargeException) {
+> return ProductionExceptionHandlerResponse.CONTINUE;
+> } else {
+> return ProductionExceptionHandlerResponse.FAIL;
+> }
+> }
+> }
+>
+> Properties settings = new Properties();
+>
+> // other various kafka streams settings, e.g. bootstrap servers, application id, etc
+>
+> settings.put(StreamsConfig.DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_CONFIG,
+> IgnoreRecordTooLargeHandler.class);
+
+### default.timestamp.extractor
+
+> A timestamp extractor pulls a timestamp from an instance of [ConsumerRecord](/37/javadoc/org/apache/kafka/clients/consumer/ConsumerRecord.html). Timestamps are used to control the progress of streams.
+>
+> The default extractor is [FailOnInvalidTimestamp](/37/javadoc/org/apache/kafka/streams/processor/FailOnInvalidTimestamp.html). This extractor retrieves built-in timestamps that are automatically embedded into Kafka messages by the Kafka producer client since [Kafka version 0.10](https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message). Depending on the setting of Kafka's server-side `log.message.timestamp.type` broker and `message.timestamp.type` topic parameters, this extractor provides you with:
+>
+> * **event-time** processing semantics if `log.message.timestamp.type` is set to `CreateTime` aka "producer time" (which is the default). This represents the time when a Kafka producer sent the original message. If you use Kafka's official producer client, the timestamp represents milliseconds since the epoch.
+> * **ingestion-time** processing semantics if `log.message.timestamp.type` is set to `LogAppendTime` aka "broker time". This represents the time when the Kafka broker received the original message, in milliseconds since the epoch.
+>
+
+>
+> The `FailOnInvalidTimestamp` extractor throws an exception if a record contains an invalid (i.e. negative) built-in timestamp, because Kafka Streams would not process this record but silently drop it. Invalid built-in timestamps can occur for various reasons: if for example, you consume a topic that is written to by pre-0.10 Kafka producer clients or by third-party producer clients that don't support the new Kafka 0.10 message format yet; another situation where this may happen is after upgrading your Kafka cluster from `0.9` to `0.10`, where all the data that was generated with `0.9` does not include the `0.10` message timestamps.
+>
+> If you have data with invalid timestamps and want to process it, then there are two alternative extractors available. Both work on built-in timestamps, but handle invalid timestamps differently.
+>
+> * [LogAndSkipOnInvalidTimestamp](/37/javadoc/org/apache/kafka/streams/processor/LogAndSkipOnInvalidTimestamp.html): This extractor logs a warn message and returns the invalid timestamp to Kafka Streams, which will not process but silently drop the record. This log-and-skip strategy allows Kafka Streams to make progress instead of failing if there are records with an invalid built-in timestamp in your input data.
+> * [UsePartitionTimeOnInvalidTimestamp](/37/javadoc/org/apache/kafka/streams/processor/UsePartitionTimeOnInvalidTimestamp.html). This extractor returns the record's built-in timestamp if it is valid (i.e. not negative). If the record does not have a valid built-in timestamps, the extractor returns the previously extracted valid timestamp from a record of the same topic partition as the current record as a timestamp estimation. In case that no timestamp can be estimated, it throws an exception.
+>
+
+>
+> Another built-in extractor is [WallclockTimestampExtractor](/37/javadoc/org/apache/kafka/streams/processor/WallclockTimestampExtractor.html). This extractor does not actually "extract" a timestamp from the consumed record but rather returns the current time in milliseconds from the system clock (think: `System.currentTimeMillis()`), which effectively means Streams will operate on the basis of the so-called **processing-time** of events.
+>
+> You can also provide your own timestamp extractors, for instance to retrieve timestamps embedded in the payload of messages. If you cannot extract a valid timestamp, you can either throw an exception, return a negative timestamp, or estimate a timestamp. Returning a negative timestamp will result in data loss - the corresponding record will not be processed but silently dropped. If you want to estimate a new timestamp, you can use the value provided via `previousTimestamp` (i.e., a Kafka Streams timestamp estimation). Here is an example of a custom `TimestampExtractor` implementation:
+>
+>
+> import org.apache.kafka.clients.consumer.ConsumerRecord;
+> import org.apache.kafka.streams.processor.TimestampExtractor;
+>
+> // Extracts the embedded timestamp of a record (giving you "event-time" semantics).
+> public class MyEventTimeExtractor implements TimestampExtractor {
+>
+> @Override
+> public long extract(final ConsumerRecord record, final long previousTimestamp) {
+> // `Foo` is your own custom class, which we assume has a method that returns
+> // the embedded timestamp (milliseconds since midnight, January 1, 1970 UTC).
+> long timestamp = -1;
+> final Foo myPojo = (Foo) record.value();
+> if (myPojo != null) {
+> timestamp = myPojo.getTimestampInMillis();
+> }
+> if (timestamp < 0) {
+> // Invalid timestamp! Attempt to estimate a new timestamp,
+> // otherwise fall back to wall-clock time (processing-time).
+> if (previousTimestamp >= 0) {
+> return previousTimestamp;
+> } else {
+> return System.currentTimeMillis();
+> }
+> }
+> }
+>
+> }
+>
+> You would then define the custom timestamp extractor in your Streams configuration as follows:
+>
+>
+> import java.util.Properties;
+> import org.apache.kafka.streams.StreamsConfig;
+>
+> Properties streamsConfiguration = new Properties();
+> streamsConfiguration.put(StreamsConfig.DEFAULT_TIMESTAMP_EXTRACTOR_CLASS_CONFIG, MyEventTimeExtractor.class);
+
+### default.key.serde
+
+> The default Serializer/Deserializer class for record keys, null unless set by user. Serialization and deserialization in Kafka Streams happens whenever data needs to be materialized, for example:
+>
+> * Whenever data is read from or written to a _Kafka topic_ (e.g., via the `StreamsBuilder#stream()` and `KStream#to()` methods).
+> * Whenever data is read from or written to a _state store_.
+>
+
+>
+> This is discussed in more detail in [Data types and serialization](datatypes.html#streams-developer-guide-serdes).
+
+### default.value.serde
+
+> The default Serializer/Deserializer class for record values, null unless set by user. Serialization and deserialization in Kafka Streams happens whenever data needs to be materialized, for example:
+>
+> * Whenever data is read from or written to a _Kafka topic_ (e.g., via the `StreamsBuilder#stream()` and `KStream#to()` methods).
+> * Whenever data is read from or written to a _state store_.
+>
+
+>
+> This is discussed in more detail in [Data types and serialization](datatypes.html#streams-developer-guide-serdes).
+
+### default.windowed.key.serde.inner
+
+> The default Serializer/Deserializer class for the inner class of windowed keys. Serialization and deserialization in Kafka Streams happens whenever data needs to be materialized, for example:
+>
+> * Whenever data is read from or written to a _Kafka topic_ (e.g., via the `StreamsBuilder#stream()` and `KStream#to()` methods).
+> * Whenever data is read from or written to a _state store_.
+>
+
+>
+> This is discussed in more detail in [Data types and serialization](datatypes.html#streams-developer-guide-serdes).
+
+### default.windowed.value.serde.inner
+
+> The default Serializer/Deserializer class for the inner class of windowed values. Serialization and deserialization in Kafka Streams happens happens whenever data needs to be materialized, for example:
+>
+> * Whenever data is read from or written to a _Kafka topic_ (e.g., via the `StreamsBuilder#stream()` and `KStream#to()` methods).
+> * Whenever data is read from or written to a _state store_.
+>
+
+>
+> This is discussed in more detail in [Data types and serialization](datatypes.html#streams-developer-guide-serdes).
+
+### rack.aware.assignment.non_overlap_cost
+
+> This configuration sets the cost of moving a task from the original assignment computed either by `StickyTaskAssignor` or `HighAvailabilityTaskAssignor`. Together with `rack.aware.assignment.traffic_cost`, they control whether the optimizer favors minimizing cross rack traffic or minimizing the movement of tasks in the existing assignment. If this config is set to a larger value than `rack.aware.assignment.traffic_cost`, the optimizer will try to maintain the existing assignment computed by the task assignor. Note that the optimizer takes the ratio of these two configs into consideration of favoring maintaining existing assignment or minimizing traffic cost. For example, setting `rack.aware.assignment.non_overlap_cost` to 10 and `rack.aware.assignment.traffic_cost` to 1 is more likely to maintain existing assignment than setting `rack.aware.assignment.non_overlap_cost` to 100 and `rack.aware.assignment.traffic_cost` to 50.
+>
+> The default value is null which means default `non_overlap_cost` in different assignors will be used. In `StickyTaskAssignor`, it has a default value of 10 and `rack.aware.assignment.traffic_cost` has a default value of 1, which means maintaining stickiness is preferred in `StickyTaskAssignor`. In `HighAvailabilityTaskAssignor`, it has a default value of 1 and `rack.aware.assignment.traffic_cost` has a default value of 10, which means minimizing cross rack traffic is preferred in `HighAvailabilityTaskAssignor`.
+
+### rack.aware.assignment.strategy
+
+> This configuration sets the strategy Kafka Streams uses for rack aware task assignment so that cross traffic from broker to client can be reduced. This config will only take effect when `broker.rack` is set on the brokers and `client.rack` is set on Kafka Streams side. There are two settings for this config:
+>
+> * `none`. This is the default value which means rack aware task assignment will be disabled.
+> * `min_traffic`. This settings means that the rack aware task assigner will compute an assignment which tries to minimize cross rack traffic.
+> * `balance_subtopology`. This settings means that the rack aware task assigner will compute an assignment which will try to balance tasks from same subtopology to different clients and minimize cross rack traffic on top of that.
+>
+
+>
+> This config can be used together with rack.aware.assignment.non_overlap_cost and rack.aware.assignment.traffic_cost to balance reducing cross rack traffic and maintaining the existing assignment.
+
+### rack.aware.assignment.tags
+
+> This configuration sets a list of tag keys used to distribute standby replicas across Kafka Streams clients. When configured, Kafka Streams will make a best-effort to distribute the standby tasks over clients with different tag values.
+>
+> Tags for the Kafka Streams clients can be set via `client.tag.` prefix. Example:
+>
+>
+> Client-1 | Client-2
+> _______________________________________________________________________
+> client.tag.zone: eu-central-1a | client.tag.zone: eu-central-1b
+> client.tag.cluster: k8s-cluster1 | client.tag.cluster: k8s-cluster1
+> rack.aware.assignment.tags: zone,cluster | rack.aware.assignment.tags: zone,cluster
+>
+>
+> Client-3 | Client-4
+> _______________________________________________________________________
+> client.tag.zone: eu-central-1a | client.tag.zone: eu-central-1b
+> client.tag.cluster: k8s-cluster2 | client.tag.cluster: k8s-cluster2
+> rack.aware.assignment.tags: zone,cluster | rack.aware.assignment.tags: zone,cluster
+>
+> In the above example, we have four Kafka Streams clients across two zones (`eu-central-1a`, `eu-central-1b`) and across two clusters (`k8s-cluster1`, `k8s-cluster2`). For an active task located on `Client-1`, Kafka Streams will allocate a standby task on `Client-4`, since `Client-4` has a different `zone` and a different `cluster` than `Client-1`.
+
+### rack.aware.assignment.traffic_cost
+
+> This configuration sets the cost of cross rack traffic. Together with `rack.aware.assignment.non_overlap_cost`, they control whether the optimizer favors minimizing cross rack traffic or minimizing the movement of tasks in the existing assignment. If this config is set to a larger value than `rack.aware.assignment.non_overlap_cost`, the optimizer will try to compute an assignment which minimize the cross rack traffic. Note that the optimizer takes the ratio of these two configs into consideration of favoring maintaining existing assignment or minimizing traffic cost. For example, setting `rack.aware.assignment.traffic_cost` to 10 and `rack.aware.assignment.non_overlap_cost` to 1 is more likely to minimize cross rack traffic than setting `rack.aware.assignment.traffic_cost` to 100 and `rack.aware.assignment.non_overlap_cost` to 50.
+>
+> The default value is null which means default traffic cost in different assignors will be used. In `StickyTaskAssignor`, it has a default value of 1 and `rack.aware.assignment.non_overlap_cost` has a default value of 10. In `HighAvailabilityTaskAssignor`, it has a default value of 10 and `rack.aware.assignment.non_overlap_cost` has a default value of 1.
+
+### max.task.idle.ms
+
+> This configuration controls how long Streams will wait to fetch data in order to provide in-order processing semantics.
+>
+> When processing a task that has multiple input partitions (as in a join or merge), Streams needs to choose which partition to process the next record from. When all input partitions have locally buffered data, Streams picks the partition whose next record has the lowest timestamp. This has the desirable effect of collating the input partitions in timestamp order, which is generally what you want in a streaming join or merge. However, when Streams does not have any data buffered locally for one of the partitions, it does not know whether the next record for that partition will have a lower or higher timestamp than the remaining partitions' records.
+>
+> There are two cases to consider: either there is data in that partition on the broker that Streams has not fetched yet, or Streams is fully caught up with that partition on the broker, and the producers simply haven't produced any new records since Streams polled the last batch.
+>
+> The default value of `0` causes Streams to delay processing a task when it detects that it has no locally buffered data for a partition, but there is data available on the brokers. Specifically, when there is an empty partition in the local buffer, but Streams has a non-zero lag for that partition. However, as soon as Streams catches up to the broker, it will continue processing, even if there is no data in one of the partitions. That is, it will not wait for new data to be _produced_. This default is designed to sacrifice some throughput in exchange for intuitively correct join semantics.
+>
+> Any config value greater than zero indicates the number of _extra_ milliseconds that Streams will wait if it has a caught-up but empty partition. In other words, this is the amount of time to wait for new data to be produced to the input partitions to ensure in-order processing of data in the event of a slow producer.
+>
+> The config value of `-1` indicates that Streams will never wait to buffer empty partitions before choosing the next record by timestamp, which achieves maximum throughput at the expense of introducing out-of-order processing.
+
+### max.warmup.replicas
+
+> The maximum number of warmup replicas (extra standbys beyond the configured `num.standbys`) that can be assigned at once for the purpose of keeping the task available on one instance while it is warming up on another instance it has been reassigned to. Used to throttle how much extra broker traffic and cluster state can be used for high availability. Increasing this will allow Streams to warm up more tasks at once, speeding up the time for the reassigned warmups to restore sufficient state for them to be transitioned to active tasks. Must be at least 1.
+>
+> Note that one warmup replica corresponds to one [Stream Task](https://kafka.apache.org/34/streams/architecture#streams_architecture_tasks). Furthermore, note that each warmup task can only be promoted to an active task during a rebalance (normally during a so-called probing rebalance, which occur at a frequency specified by the `probing.rebalance.interval.ms` config). This means that the maximum rate at which active tasks can be migrated from one Kafka Streams instance to another instance can be determined by (`max.warmup.replicas` / `probing.rebalance.interval.ms`).
+
+### num.standby.replicas
+
+> The number of standby replicas. Standby replicas are shadow copies of local state stores. Kafka Streams attempts to create the specified number of replicas per store and keep them up to date as long as there are enough instances running. Standby replicas are used to minimize the latency of task failover. A task that was previously running on a failed instance is preferred to restart on an instance that has standby replicas so that the local state store restoration process from its changelog can be minimized. Details about how Kafka Streams makes use of the standby replicas to minimize the cost of resuming tasks on failover can be found in the [State](../architecture.html#streams_architecture_state) section.
+>
+> Recommendation:
+> Increase the number of standbys to 1 to get instant fail-over, i.e., high-availability. Increasing the number of standbys requires more client-side storage space. For example, with 1 standby, 2x space is required.
+>
+> Note:
+> If you enable n standby tasks, you need to provision n+1 `KafkaStreams` instances.
+
+### num.stream.threads
+
+> This specifies the number of stream threads in an instance of the Kafka Streams application. The stream processing code runs in these thread. For more information about Kafka Streams threading model, see [Threading Model](../architecture.html#streams_architecture_threads).
+
+### probing.rebalance.interval.ms
+
+> The maximum time to wait before triggering a rebalance to probe for warmup replicas that have restored enough to be considered caught up. Streams will only assign stateful active tasks to instances that are caught up and within the acceptable.recovery.lag, if any exist. Probing rebalances are used to query the latest total lag of warmup replicas and transition them to active tasks if ready. They will continue to be triggered as long as there are warmup tasks, and until the assignment is balanced. Must be at least 1 minute.
+
+### processing.guarantee
+
+> The processing guarantee that should be used. Possible values are `"at_least_once"` (default) and `"exactly_once_v2"` (for EOS version 2). Deprecated config options are `"exactly_once"` (for EOS alpha), and `"exactly_once_beta"` (for EOS version 2). Using `"exactly_once_v2"` (or the deprecated `"exactly_once_beta"`) requires broker version 2.5 or newer, while using the deprecated `"exactly_once"` requires broker version 0.11.0 or newer. Note that if exactly-once processing is enabled, the default for parameter `commit.interval.ms` changes to 100ms. Additionally, consumers are configured with `isolation.level="read_committed"` and producers are configured with `enable.idempotence=true` per default. Note that by default exactly-once processing requires a cluster of at least three brokers what is the recommended setting for production. For development, you can change this configuration by adjusting broker setting `transaction.state.log.replication.factor` and `transaction.state.log.min.isr` to the number of brokers you want to use. For more details see [Processing Guarantees](../core-concepts#streams_processing_guarantee).
+>
+> Recommendation:
+> While it is technically possible to use EOS with any replication factor, using a replication factor lower than 3 effectively voids EOS. Thus it is strongly recommended to use a replication factor of 3 (together with `min.in.sync.replicas=2`). This recommendation applies to all topics (i.e. `__transaction_state`, `__consumer_offsets`, Kafka Streams internal topics, and user topics).
+
+### replication.factor
+
+> This specifies the replication factor of internal topics that Kafka Streams creates when local states are used or a stream is repartitioned for aggregation. Replication is important for fault tolerance. Without replication even a single broker failure may prevent progress of the stream processing application. It is recommended to use a similar replication factor as source topics.
+>
+> Recommendation:
+> Increase the replication factor to 3 to ensure that the internal Kafka Streams topic can tolerate up to 2 broker failures. Note that you will require more storage space as well (3x with the replication factor of 3).
+
+### rocksdb.config.setter
+
+> The RocksDB configuration. Kafka Streams uses RocksDB as the default storage engine for persistent stores. To change the default configuration for RocksDB, you can implement `RocksDBConfigSetter` and provide your custom class via [rocksdb.config.setter](/37/javadoc/org/apache/kafka/streams/state/RocksDBConfigSetter.html).
+>
+> Here is an example that adjusts the memory size consumed by RocksDB.
+>
+>
+> public static class CustomRocksDBConfig implements RocksDBConfigSetter {
+> // This object should be a member variable so it can be closed in RocksDBConfigSetter#close.
+> private org.rocksdb.Cache cache = new org.rocksdb.LRUCache(16 * 1024L * 1024L);
+>
+> @Override
+> public void setConfig(final String storeName, final Options options, final Map configs) {
+> // See #1 below.
+> BlockBasedTableConfig tableConfig = (BlockBasedTableConfig) options.tableFormatConfig();
+> tableConfig.setBlockCache(cache);
+> // See #2 below.
+> tableConfig.setBlockSize(16 * 1024L);
+> // See #3 below.
+> tableConfig.setCacheIndexAndFilterBlocks(true);
+> options.setTableFormatConfig(tableConfig);
+> // See #4 below.
+> options.setMaxWriteBufferNumber(2);
+> }
+>
+> @Override
+> public void close(final String storeName, final Options options) {
+> // See #5 below.
+> cache.close();
+> }
+> }
+>
+> Properties streamsSettings = new Properties();
+> streamsConfig.put(StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG, CustomRocksDBConfig.class);
+>
+> Notes for example:
+>
+>
+> 1. `BlockBasedTableConfig tableConfig = (BlockBasedTableConfig) options.tableFormatConfig();` Get a reference to the existing table config rather than create a new one, so you don't accidentally overwrite defaults such as the `BloomFilter`, which is an important optimization.
+> 2. `tableConfig.setBlockSize(16 * 1024L);` Modify the default [block size](https://github.com/apache/kafka/blob/2.3/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L79) per these instructions from the [RocksDB GitHub](https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB#indexes-and-filter-blocks).
+> 3. `tableConfig.setCacheIndexAndFilterBlocks(true);` Do not let the index and filter blocks grow unbounded. For more information, see the [RocksDB GitHub](https://github.com/facebook/rocksdb/wiki/Block-Cache#caching-index-and-filter-blocks).
+> 4. `options.setMaxWriteBufferNumber(2);` See the advanced options in the [RocksDB GitHub](https://github.com/facebook/rocksdb/blob/8dee8cad9ee6b70fd6e1a5989a8156650a70c04f/include/rocksdb/advanced_options.h#L103).
+> 5. `cache.close();` To avoid memory leaks, you must close any objects you constructed that extend org.rocksdb.RocksObject. See [RocksJava docs](https://github.com/facebook/rocksdb/wiki/RocksJava-Basics#memory-management) for more details.
+>
+
+
+ #### state.dir
+
+> The state directory. Kafka Streams persists local states under the state directory. Each application has a subdirectory on its hosting machine that is located under the state directory. The name of the subdirectory is the application ID. The state stores associated with the application are created under this subdirectory. When running multiple instances of the same application on a single machine, this path must be unique for each such instance.
+
+ #### topology.optimization
+
+> A configuration telling Kafka Streams if it should optimize the topology and what optimizations to apply. Acceptable values are: `StreamsConfig.NO_OPTIMIZATION` (`none`), `StreamsConfig.OPTIMIZE` (`all`) or a comma separated list of specific optimizations: (`StreamsConfig.REUSE_KTABLE_SOURCE_TOPICS` (`reuse.ktable.source.topics`), `StreamsConfig.MERGE_REPARTITION_TOPICS` (`merge.repartition.topics`)).
+
+We recommend listing specific optimizations in the config for production code so that the structure of your topology will not change unexpectedly during upgrades of the Streams library.
+
+These optimizations include moving/reducing repartition topics and reusing the source topic as the changelog for source KTables. These optimizations will save on network traffic and storage in Kafka without changing the semantics of your applications. Enabling them is recommended.
+
+Note that as of 2.3, you need to do two things to enable optimizations. In addition to setting this config to `StreamsConfig.OPTIMIZE`, you'll need to pass in your configuration properties when building your topology by using the overloaded `StreamsBuilder.build(Properties)` method. For example `KafkaStreams myStream = new KafkaStreams(streamsBuilder.build(properties), properties)`.
+
+ #### upgrade.from
+
+> The version you are upgrading from. It is important to set this config when performing a rolling upgrade to certain versions, as described in the upgrade guide. You should set this config to the appropriate version before bouncing your instances and upgrading them to the newer version. Once everyone is on the newer version, you should remove this config and do a second rolling bounce. It is only necessary to set this config and follow the two-bounce upgrade path when upgrading from below version 2.0, or when upgrading to 2.4+ from any version lower than 2.4.
+
+ ### Kafka consumers, producer and admin client configuration parameters
+
+ You can specify parameters for the Kafka [consumers](/37/javadoc/org/apache/kafka/clients/consumer/package-summary.html), [producers](/37/javadoc/org/apache/kafka/clients/producer/package-summary.html), and [admin client](/37/javadoc/org/apache/kafka/kafka/clients/admin/package-summary.html) that are used internally. The consumer, producer and admin client settings are defined by specifying parameters in a `StreamsConfig` instance.
+
+ In this example, the Kafka [consumer session timeout](/37/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#SESSION_TIMEOUT_MS_CONFIG) is configured to be 60000 milliseconds in the Streams settings:
+
+
+ Properties streamsSettings = new Properties();
+ // Example of a "normal" setting for Kafka Streams
+ streamsSettings.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-broker-01:9092");
+ // Customize the Kafka consumer settings of your Streams application
+ streamsSettings.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 60000);
+
+ #### Naming
+
+ Some consumer, producer and admin client configuration parameters use the same parameter name, and Kafka Streams library itself also uses some parameters that share the same name with its embedded client. For example, `send.buffer.bytes` and `receive.buffer.bytes` are used to configure TCP buffers; `request.timeout.ms` and `retry.backoff.ms` control retries for client request; `retries` are used to configure how many retries are allowed when handling retriable errors from broker request responses. You can avoid duplicate names by prefix parameter names with `consumer.`, `producer.`, or `admin.` (e.g., `consumer.send.buffer.bytes` and `producer.send.buffer.bytes`).
+
+
+ Properties streamsSettings = new Properties();
+ // same value for consumer, producer, and admin client
+ streamsSettings.put("PARAMETER_NAME", "value");
+ // different values for consumer and producer
+ streamsSettings.put("consumer.PARAMETER_NAME", "consumer-value");
+ streamsSettings.put("producer.PARAMETER_NAME", "producer-value");
+ streamsSettings.put("admin.PARAMETER_NAME", "admin-value");
+ // alternatively, you can use
+ streamsSettings.put(StreamsConfig.consumerPrefix("PARAMETER_NAME"), "consumer-value");
+ streamsSettings.put(StreamsConfig.producerPrefix("PARAMETER_NAME"), "producer-value");
+ streamsSettings.put(StreamsConfig.adminClientPrefix("PARAMETER_NAME"), "admin-value");
+
+ You could further separate consumer configuration by adding different prefixes:
+
+ * `main.consumer.` for main consumer which is the default consumer of stream source.
+ * `restore.consumer.` for restore consumer which is in charge of state store recovery.
+ * `global.consumer.` for global consumer which is used in global KTable construction.
+
+
+
+ For example, if you only want to set restore consumer config without touching other consumers' settings, you could simply use `restore.consumer.` to set the config.
+
+
+ Properties streamsSettings = new Properties();
+ // same config value for all consumer types
+ streamsSettings.put("consumer.PARAMETER_NAME", "general-consumer-value");
+ // set a different restore consumer config. This would make restore consumer take restore-consumer-value,
+ // while main consumer and global consumer stay with general-consumer-value
+ streamsSettings.put("restore.consumer.PARAMETER_NAME", "restore-consumer-value");
+ // alternatively, you can use
+ streamsSettings.put(StreamsConfig.restoreConsumerPrefix("PARAMETER_NAME"), "restore-consumer-value");
+
+ Same applied to `main.consumer.` and `main.consumer.`, if you only want to specify one consumer type config.
+
+ Additionally, to configure the internal repartition/changelog topics, you could use the `topic.` prefix, followed by any of the standard topic configs.
+
+
+ Properties streamsSettings = new Properties();
+ // Override default for both changelog and repartition topics
+ streamsSettings.put("topic.PARAMETER_NAME", "topic-value");
+ // alternatively, you can use
+ streamsSettings.put(StreamsConfig.topicPrefix("PARAMETER_NAME"), "topic-value");
+
+ #### Default Values
+
+ Kafka Streams uses different default values for some of the underlying client configs, which are summarized below. For detailed descriptions of these configs, see [Producer Configs](http://kafka.apache.org/0100/documentation.html#producerconfigs) and [Consumer Configs](http://kafka.apache.org/0100/documentation.html#newconsumerconfigs).
+
+ Parameter Name | Corresponding Client | Streams Default
+ ---|---|---
+ auto.offset.reset | Consumer | earliest
+ linger.ms | Producer | 100
+ max.poll.records | Consumer | 1000
+
+ ### Parameters controlled by Kafka Streams
+
+ Kafka Streams assigns the following configuration parameters. If you try to change `allow.auto.create.topics`, your value is ignored and setting it has no effect in a Kafka Streams application. You can set the other parameters. Kafka Streams sets them to different default values than a plain `KafkaConsumer`.
+
+ Kafka Streams uses the `client.id` parameter to compute derived client IDs for internal clients. If you don't set `client.id`, Kafka Streams sets it to `-`. Parameter Name | Corresponding Client | Streams Default
+ ---|---|---
+ allow.auto.create.topics | Consumer | `false`
+ auto.offset.reset | Consumer | `earliest`
+ linger.ms | Producer | `100`
+ max.poll.records | Consumer | `1000`
+
+ #### enable.auto.commit
+
+> The consumer auto commit. To guarantee at-least-once processing semantics and turn off auto commits, Kafka Streams overrides this consumer config value to `false`. Consumers will only commit explicitly via _commitSync_ calls when the Kafka Streams library or a user decides to commit the current processing state.
+
+ [Previous](/37/streams/developer-guide/write-streams) [Next](/37/streams/developer-guide/dsl-api)
+
+ * [Documentation](/documentation)
+ * [Kafka Streams](/streams)
+ * [Developer Guide](/streams/developer-guide/)
+
+
diff --git a/docs/streams/developer-guide/datatypes.html b/docs/streams/developer-guide/datatypes.html
deleted file mode 100644
index 607d133f3b25c..0000000000000
--- a/docs/streams/developer-guide/datatypes.html
+++ /dev/null
@@ -1,238 +0,0 @@
-
-
-
-
-
-
-
-
- stringSerde = Serdes.String();
+ final Serde longSerde = Serdes.Long();
+
+ // The stream userCountByRegion has type `String` for record keys (for region)
+ // and type `Long` for record values (for user counts).
+ KStream userCountByRegion = ...;
+ userCountByRegion.to("RegionCountsTopic", Produced.with(stringSerde, longSerde));
+
+If you want to override serdes selectively, i.e., keep the defaults for some fields, then don't specify the serde whenever you want to leverage the default settings:
+
+
+ import org.apache.kafka.common.serialization.Serde;
+ import org.apache.kafka.common.serialization.Serdes;
+
+ // Use the default serializer for record keys (here: region as String) by not specifying the key serde,
+ // but override the default serializer for record values (here: userCount as Long).
+ final Serde longSerde = Serdes.Long();
+ KStream userCountByRegion = ...;
+ userCountByRegion.to("RegionCountsTopic", Produced.valueSerde(Serdes.Long()));
+
+If some of your incoming records are corrupted or ill-formatted, they will cause the deserializer class to report an error. Since 1.0.x we have introduced an `DeserializationExceptionHandler` interface which allows you to customize how to handle such records. The customized implementation of the interface can be specified via the `StreamsConfig`. For more details, please feel free to read the [Configuring a Streams Application](config-streams.html#default-deserialization-exception-handler) section.
+
+# Available Serdes
+
+## Primitive and basic types
+
+Apache Kafka includes several built-in serde implementations for Java primitives and basic types such as `byte[]` in its `kafka-clients` Maven artifact:
+
+
+
+ org.apache.kafka
+ kafka-clients
+ 3.7.2
+
+
+This artifact provides the following serde implementations under the package [org.apache.kafka.common.serialization](https://github.com/apache/kafka/blob/3.7/clients/src/main/java/org/apache/kafka/common/serialization), which you can leverage when e.g., defining default serializers in your Streams configuration.
+
+Data type | Serde
+---|---
+byte[] | `Serdes.ByteArray()`, `Serdes.Bytes()` (see tip below)
+ByteBuffer | `Serdes.ByteBuffer()`
+Double | `Serdes.Double()`
+Integer | `Serdes.Integer()`
+Long | `Serdes.Long()`
+String | `Serdes.String()`
+UUID | `Serdes.UUID()`
+Void | `Serdes.Void()`
+List | `Serdes.ListSerde()`
+Boolean | `Serdes.Boolean()`
+
+**Tip**
+
+[Bytes](https://github.com/apache/kafka/blob/3.7/clients/src/main/java/org/apache/kafka/common/utils/Bytes.java) is a wrapper for Java's `byte[]` (byte array) that supports proper equality and ordering semantics. You may want to consider using `Bytes` instead of `byte[]` in your applications.
+
+## JSON
+
+The Kafka Streams code examples also include a basic serde implementation for JSON:
+
+ * [PageViewTypedDemo](https://github.com/apache/kafka/blob/3.7/streams/examples/src/main/java/org/apache/kafka/streams/examples/pageview/PageViewTypedDemo.java#L83)
+
+
+
+As shown in the example, you can use JSONSerdes inner classes `Serdes.serdeFrom(, )` to construct JSON compatible serializers and deserializers.
+
+# Implementing custom Serdes
+
+If you need to implement custom Serdes, your best starting point is to take a look at the source code references of existing Serdes (see previous section). Typically, your workflow will be similar to:
+
+ 1. Write a _serializer_ for your data type `T` by implementing [org.apache.kafka.common.serialization.Serializer](https://github.com/apache/kafka/blob/3.7/clients/src/main/java/org/apache/kafka/common/serialization/Serializer.java).
+ 2. Write a _deserializer_ for `T` by implementing [org.apache.kafka.common.serialization.Deserializer](https://github.com/apache/kafka/blob/3.7/clients/src/main/java/org/apache/kafka/common/serialization/Deserializer.java).
+ 3. Write a _serde_ for `T` by implementing [org.apache.kafka.common.serialization.Serde](https://github.com/apache/kafka/blob/3.7/clients/src/main/java/org/apache/kafka/common/serialization/Serde.java), which you either do manually (see existing Serdes in the previous section) or by leveraging helper functions in [Serdes](https://github.com/apache/kafka/blob/3.7/clients/src/main/java/org/apache/kafka/common/serialization/Serdes.java) such as `Serdes.serdeFrom(Serializer, Deserializer)`. Note that you will need to implement your own class (that has no generic types) if you want to use your custom serde in the configuration provided to `KafkaStreams`. If your serde class has generic types or you use `Serdes.serdeFrom(Serializer, Deserializer)`, you can pass your serde only via methods calls (for example `builder.stream("topicName", Consumed.with(...))`).
+
+
+
+# Kafka Streams DSL for Scala Implicit Serdes[](scala-dsl-serdes "Permalink to this headline")
+
+When using the [Kafka Streams DSL for Scala](dsl-api.html#scala-dsl) you're not required to configure a default Serdes. In fact, it's not supported. Serdes are instead provided implicitly by default implementations for common primitive datatypes. See the [Implicit Serdes](dsl-api.html#scala-dsl-implicit-serdes) and [User-Defined Serdes](dsl-api.html#scala-dsl-user-defined-serdes) sections in the DSL API documentation for details
+
+ * [Documentation](/documentation)
+ * [Kafka Streams](/streams)
+ * [Developer Guide](/streams/developer-guide/)
+
+
diff --git a/docs/streams/developer-guide/dsl-api.html b/docs/streams/developer-guide/dsl-api.html
deleted file mode 100644
index fd5c22cae33bb..0000000000000
--- a/docs/streams/developer-guide/dsl-api.html
+++ /dev/null
@@ -1,3974 +0,0 @@
-
-
-
-
-
-
-
-
- wordCounts = builder.stream(
+ "word-counts-input-topic", /* input topic */
+ Consumed.with(
+ Serdes.String(), /* key serde */
+ Serdes.Long() /* value serde */
+ );
+
+If you do not specify Serdes explicitly, the default Serdes from the [configuration](config-streams.html#streams-developer-guide-configuration) are used. You **must specify Serdes explicitly** if the key or value types of the records in the Kafka input topics do not match the configured default Serdes. For information about configuring default Serdes, available Serdes, and implementing your own custom Serdes see [Data Types and Serialization](datatypes.html#streams-developer-guide-serdes). Several variants of `stream` exist. For example, you can specify a regex pattern for input topics to read from (note that all matching topics will be part of the same input topic group, and the work will not be parallelized for different topics if subscribed to in this way).
+**Table**
+
+ * _input topic_ -> KTable
+
+| Reads the specified Kafka input topic into a KTable. The topic is interpreted as a changelog stream, where records with the same key are interpreted as UPSERT aka INSERT/UPDATE (when the record value is not `null`) or as DELETE (when the value is `null`) for that key. [(details)](/37/javadoc/org/apache/kafka/streams/StreamsBuilder.html#table-java.lang.String\(java.lang.String\)) In the case of a KTable, the local KTable instance of every application instance will be populated with data from only **a subset** of the partitions of the input topic. Collectively, across all application instances, all input topic partitions are read and processed. You must provide a name for the table (more precisely, for the internal [state store](../architecture.html#streams_architecture_state) that backs the table). This is required for supporting [interactive queries](interactive-queries.html#streams-developer-guide-interactive-queries) against the table. When a name is not provided the table will not be queryable and an internal name will be provided for the state store. If you do not specify Serdes explicitly, the default Serdes from the [configuration](config-streams.html#streams-developer-guide-configuration) are used. You **must specify Serdes explicitly** if the key or value types of the records in the Kafka input topics do not match the configured default Serdes. For information about configuring default Serdes, available Serdes, and implementing your own custom Serdes see [Data Types and Serialization](datatypes.html#streams-developer-guide-serdes). Several variants of `table` exist, for example to specify the `auto.offset.reset` policy to be used when reading from the input topic.
+**Global Table**
+
+ * _input topic_ -> GlobalKTable
+
+| Reads the specified Kafka input topic into a GlobalKTable. The topic is interpreted as a changelog stream, where records with the same key are interpreted as UPSERT aka INSERT/UPDATE (when the record value is not `null`) or as DELETE (when the value is `null`) for that key. [(details)](/37/javadoc/org/apache/kafka/streams/StreamsBuilder.html#globalTable-java.lang.String\(java.lang.String\)) In the case of a GlobalKTable, the local GlobalKTable instance of every application instance will be populated with data from **all** the partitions of the input topic. You must provide a name for the table (more precisely, for the internal [state store](../architecture.html#streams_architecture_state) that backs the table). This is required for supporting [interactive queries](interactive-queries.html#streams-developer-guide-interactive-queries) against the table. When a name is not provided the table will not be queryable and an internal name will be provided for the state store.
+
+
+ import org.apache.kafka.common.serialization.Serdes;
+ import org.apache.kafka.streams.StreamsBuilder;
+ import org.apache.kafka.streams.kstream.GlobalKTable;
+
+ StreamsBuilder builder = new StreamsBuilder();
+
+ GlobalKTable wordCounts = builder.globalTable(
+ "word-counts-input-topic",
+ Materialized.>as(
+ "word-counts-global-store" /* table/store name */)
+ .withKeySerde(Serdes.String()) /* key serde */
+ .withValueSerde(Serdes.Long()) /* value serde */
+ );
+
+You **must specify Serdes explicitly** if the key or value types of the records in the Kafka input topics do not match the configured default Serdes. For information about configuring default Serdes, available Serdes, and implementing your own custom Serdes see [Data Types and Serialization](datatypes.html#streams-developer-guide-serdes). Several variants of `globalTable` exist to e.g. specify explicit Serdes.
+
+# Transform a stream
+
+The KStream and KTable interfaces support a variety of transformation operations. Each of these operations can be translated into one or more connected processors into the underlying processor topology. Since KStream and KTable are strongly typed, all of these transformation operations are defined as generic functions where users could specify the input and output data types.
+
+Some KStream transformations may generate one or more KStream objects, for example: \- `filter` and `map` on a KStream will generate another KStream \- `split` on KStream can generate multiple KStreams
+
+Some others may generate a KTable object, for example an aggregation of a KStream also yields a KTable. This allows Kafka Streams to continuously update the computed value upon arrivals of [out-of-order records](../core-concepts.html#streams_concepts_aggregations) after it has already been produced to the downstream transformation operators.
+
+All KTable transformation operations can only generate another KTable. However, the Kafka Streams DSL does provide a special function that converts a KTable representation into a KStream. All of these transformation methods can be chained together to compose a complex processor topology.
+
+These transformation operations are described in the following subsections:
+
+ * Stateless transformations
+ * Stateful transformations
+
+
+
+## Stateless transformations
+
+Stateless transformations do not require state for processing and they do not require a state store associated with the stream processor. Kafka 0.11.0 and later allows you to materialize the result from a stateless `KTable` transformation. This allows the result to be queried through [interactive queries](interactive-queries.html#streams-developer-guide-interactive-queries). To materialize a `KTable`, each of the below stateless operations [can be augmented](interactive-queries.html#streams-developer-guide-interactive-queries-local-key-value-stores) with an optional `queryableStoreName` argument.
+
+Transformation | Description
+---|---
+**Branch**
+
+ * KStream -> BranchedKStream
+
+| Branch (or split) a `KStream` based on the supplied predicates into one or more `KStream` instances. ([details](/37/javadoc/org/apache/kafka/streams/kstream/KStream.html#split\(\))) Predicates are evaluated in order. A record is placed to one and only one output stream on the first match: if the n-th predicate evaluates to true, the record is placed to n-th stream. If a record does not match any predicates, it will be routed to the default branch, or dropped if no default branch is created. Branching is useful, for example, to route records to different downstream topics.
+
+
+ KStream stream = ...;
+ Map> branches =
+ stream.split(Named.as("Branch-"))
+ .branch((key, value) -> key.startsWith("A"), /* first predicate */
+ Branched.as("A"))
+ .branch((key, value) -> key.startsWith("B"), /* second predicate */
+ Branched.as("B"))
+ .defaultBranch(Branched.as("C")) /* default branch */
+ );
+
+ // KStream branches.get("Branch-A") contains all records whose keys start with "A"
+ // KStream branches.get("Branch-B") contains all records whose keys start with "B"
+ // KStream branches.get("Branch-C") contains all other records
+
+ // Java 7 example: cf. `filter` for how to create `Predicate` instances
+
+**Filter**
+
+ * KStream -> KStream
+ * KTable -> KTable
+
+| Evaluates a boolean function for each element and retains those for which the function returns true. ([KStream details](/37/javadoc/org/apache/kafka/streams/kstream/KStream.html#filter-org.apache.kafka.streams.kstream.Predicate-), [KTable details](/37/javadoc/org/apache/kafka/streams/kstream/KTable.html#filter-org.apache.kafka.streams.kstream.Predicate-))
+
+
+ KStream stream = ...;
+
+ // A filter that selects (keeps) only positive numbers
+ // Java 8+ example, using lambda expressions
+ KStream onlyPositives = stream.filter((key, value) -> value > 0);
+
+ // Java 7 example
+ KStream onlyPositives = stream.filter(
+ new Predicate() {
+ @Override
+ public boolean test(String key, Long value) {
+ return value > 0;
+ }
+ });
+
+**Inverse Filter**
+
+ * KStream -> KStream
+ * KTable -> KTable
+
+| Evaluates a boolean function for each element and drops those for which the function returns true. ([KStream details](/37/javadoc/org/apache/kafka/streams/kstream/KStream.html#filterNot-org.apache.kafka.streams.kstream.Predicate-), [KTable details](/37/javadoc/org/apache/kafka/streams/kstream/KTable.html#filterNot-org.apache.kafka.streams.kstream.Predicate-))
+
+
+ KStream stream = ...;
+
+ // An inverse filter that discards any negative numbers or zero
+ // Java 8+ example, using lambda expressions
+ KStream onlyPositives = stream.filterNot((key, value) -> value <= 0);
+
+ // Java 7 example
+ KStream onlyPositives = stream.filterNot(
+ new Predicate() {
+ @Override
+ public boolean test(String key, Long value) {
+ return value <= 0;
+ }
+ });
+
+**FlatMap**
+
+ * KStream -> KStream
+
+| Takes one record and produces zero, one, or more records. You can modify the record keys and values, including their types. ([details](/37/javadoc/org/apache/kafka/streams/kstream/KStream.html#flatMap-org.apache.kafka.streams.kstream.KeyValueMapper-)) **Marks the stream for data re-partitioning:** Applying a grouping or a join after `flatMap` will result in re-partitioning of the records. If possible use `flatMapValues` instead, which will not cause data re-partitioning.
+
+
+ KStream stream = ...;
+ KStream transformed = stream.flatMap(
+ // Here, we generate two output records for each input record.
+ // We also change the key and value types.
+ // Example: (345L, "Hello") -> ("HELLO", 1000), ("hello", 9000)
+ (key, value) -> {
+ List> result = new LinkedList<>();
+ result.add(KeyValue.pair(value.toUpperCase(), 1000));
+ result.add(KeyValue.pair(value.toLowerCase(), 9000));
+ return result;
+ }
+ );
+
+ // Java 7 example: cf. `map` for how to create `KeyValueMapper` instances
+
+**FlatMapValues**
+
+ * KStream -> KStream
+
+| Takes one record and produces zero, one, or more records, while retaining the key of the original record. You can modify the record values and the value type. ([details](/37/javadoc/org/apache/kafka/streams/kstream/KStream.html#flatMapValues-org.apache.kafka.streams.kstream.ValueMapper-)) `flatMapValues` is preferable to `flatMap` because it will not cause data re-partitioning. However, you cannot modify the key or key type like `flatMap` does.
+
+
+ // Split a sentence into words.
+ KStream sentences = ...;
+ KStream words = sentences.flatMapValues(value -> Arrays.asList(value.split("\s+")));
+
+ // Java 7 example: cf. `mapValues` for how to create `ValueMapper` instances
+
+**Foreach**
+
+ * KStream -> void
+ * KStream -> void
+ * KTable -> void
+
+| **Terminal operation.** Performs a stateless action on each record. ([details](/37/javadoc/org/apache/kafka/streams/kstream/KStream.html#foreach-org.apache.kafka.streams.kstream.ForeachAction-)) You would use `foreach` to cause _side effects_ based on the input data (similar to `peek`) and then _stop_ _further processing_ of the input data (unlike `peek`, which is not a terminal operation). **Note on processing guarantees:** Any side effects of an action (such as writing to external systems) are not trackable by Kafka, which means they will typically not benefit from Kafka's processing guarantees.
+
+
+ KStream stream = ...;
+
+ // Print the contents of the KStream to the local console.
+ // Java 8+ example, using lambda expressions
+ stream.foreach((key, value) -> System.out.println(key + " => " + value));
+
+ // Java 7 example
+ stream.foreach(
+ new ForeachAction() {
+ @Override
+ public void apply(String key, Long value) {
+ System.out.println(key + " => " + value);
+ }
+ });
+
+**GroupByKey**
+
+ * KStream -> KGroupedStream
+
+| Groups the records by the existing key. ([details](/37/javadoc/org/apache/kafka/streams/kstream/KStream.html#groupByKey--)) Grouping is a prerequisite for aggregating a stream or a table and ensures that data is properly partitioned ("keyed") for subsequent operations. **When to set explicit Serdes:** Variants of `groupByKey` exist to override the configured default Serdes of your application, which **you** **must do** if the key and/or value types of the resulting `KGroupedStream` do not match the configured default Serdes. **Note** **Grouping vs. Windowing:** A related operation is windowing, which lets you control how to "sub-group" the grouped records _of the same key_ into so-called _windows_ for stateful operations such as windowed aggregations or windowed joins. **Causes data re-partitioning if and only if the stream was marked for re-partitioning.** `groupByKey` is preferable to `groupBy` because it re-partitions data only if the stream was already marked for re-partitioning. However, `groupByKey` does not allow you to modify the key or key type like `groupBy` does.
+
+
+ KStream stream = ...;
+
+ // Group by the existing key, using the application's configured
+ // default serdes for keys and values.
+ KGroupedStream groupedStream = stream.groupByKey();
+
+ // When the key and/or value types do not match the configured
+ // default serdes, we must explicitly specify serdes.
+ KGroupedStream groupedStream = stream.groupByKey(
+ Grouped.with(
+ Serdes.ByteArray(), /* key */
+ Serdes.String()) /* value */
+ );
+
+**GroupBy**
+
+ * KStream -> KGroupedStream
+ * KTable -> KGroupedTable
+
+| Groups the records by a _new_ key, which may be of a different key type. When grouping a table, you may also specify a new value and value type. `groupBy` is a shorthand for `selectKey(...).groupByKey()`. ([KStream details](/37/javadoc/org/apache/kafka/streams/kstream/KStream.html#groupBy-org.apache.kafka.streams.kstream.KeyValueMapper-), [KTable details](/37/javadoc/org/apache/kafka/streams/kstream/KTable.html#groupBy-org.apache.kafka.streams.kstream.KeyValueMapper-)) Grouping is a prerequisite for aggregating a stream or a table and ensures that data is properly partitioned ("keyed") for subsequent operations. **When to set explicit Serdes:** Variants of `groupBy` exist to override the configured default Serdes of your application, which **you must** **do** if the key and/or value types of the resulting `KGroupedStream` or `KGroupedTable` do not match the configured default Serdes. **Note** **Grouping vs. Windowing:** A related operation is windowing, which lets you control how to "sub-group" the grouped records _of the same key_ into so-called _windows_ for stateful operations such as windowed aggregations or windowed joins. **Always causes data re-partitioning:** `groupBy` always causes data re-partitioning. If possible use `groupByKey` instead, which will re-partition data only if required.
+
+
+ KStream stream = ...;
+ KTable table = ...;
+
+ // Java 8+ examples, using lambda expressions
+
+ // Group the stream by a new key and key type
+ KGroupedStream groupedStream = stream.groupBy(
+ (key, value) -> value,
+ Grouped.with(
+ Serdes.String(), /* key (note: type was modified) */
+ Serdes.String()) /* value */
+ );
+
+ // Group the table by a new key and key type, and also modify the value and value type.
+ KGroupedTable groupedTable = table.groupBy(
+ (key, value) -> KeyValue.pair(value, value.length()),
+ Grouped.with(
+ Serdes.String(), /* key (note: type was modified) */
+ Serdes.Integer()) /* value (note: type was modified) */
+ );
+
+
+ // Java 7 examples
+
+ // Group the stream by a new key and key type
+ KGroupedStream groupedStream = stream.groupBy(
+ new KeyValueMapper>() {
+ @Override
+ public String apply(byte[] key, String value) {
+ return value;
+ }
+ },
+ Grouped.with(
+ Serdes.String(), /* key (note: type was modified) */
+ Serdes.String()) /* value */
+ );
+
+ // Group the table by a new key and key type, and also modify the value and value type.
+ KGroupedTable groupedTable = table.groupBy(
+ new KeyValueMapper>() {
+ @Override
+ public KeyValue apply(byte[] key, String value) {
+ return KeyValue.pair(value, value.length());
+ }
+ },
+ Grouped.with(
+ Serdes.String(), /* key (note: type was modified) */
+ Serdes.Integer()) /* value (note: type was modified) */
+ );
+
+**Cogroup**
+
+ * KGroupedStream -> CogroupedKStream
+ * CogroupedKStream -> CogroupedKStream
+
+| Cogrouping allows to aggregate multiple input streams in a single operation. The different (already grouped) input streams must have the same key type and may have different values types. [KGroupedStream#cogroup()](/37/javadoc/org/apache/kafka/streams/kstream/KGroupedStream.html#cogroup) creates a new cogrouped stream with a single input stream, while [CogroupedKStream#cogroup()](/37/javadoc/org/apache/kafka/streams/kstream/CogroupedKStream.html#cogroup) adds a grouped stream to an existing cogrouped stream. A `CogroupedKStream` may be [windowed](/37/javadoc/org/apache/kafka/streams/kstream/CogroupedKStream.html#windowedBy) before it is [aggregated](/37/javadoc/org/apache/kafka/streams/kstream/CogroupedKStream.html#aggregate). Cogroup does not cause a repartition as it has the prerequisite that the input streams are grouped. In the process of creating these groups they will have already been repartitioned if the stream was already marked for repartitioning.
+
+
+ KStream stream = ...;
+ KStream stream2 = ...;
+
+ // Group by the existing key, using the application's configured
+ // default serdes for keys and values.
+ KGroupedStream groupedStream = stream.groupByKey();
+ KGroupedStream groupedStream2 = stream2.groupByKey();
+ CogroupedKStream cogroupedStream = groupedStream.cogroup(aggregator1).cogroup(groupedStream2, aggregator2);
+
+ KTable table = cogroupedStream.aggregate(initializer);
+
+ KTable table2 = cogroupedStream.windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMillis(500))).aggregate(initializer);
+
+**Map**
+
+ * KStream -> KStream
+
+| Takes one record and produces one record. You can modify the record key and value, including their types. ([details](/37/javadoc/org/apache/kafka/streams/kstream/KStream.html#map-org.apache.kafka.streams.kstream.KeyValueMapper-)) **Marks the stream for data re-partitioning:** Applying a grouping or a join after `map` will result in re-partitioning of the records. If possible use `mapValues` instead, which will not cause data re-partitioning.
+
+
+ KStream stream = ...;
+
+ // Java 8+ example, using lambda expressions
+ // Note how we change the key and the key type (similar to `selectKey`)
+ // as well as the value and the value type.
+ KStream transformed = stream.map(
+ (key, value) -> KeyValue.pair(value.toLowerCase(), value.length()));
+
+ // Java 7 example
+ KStream transformed = stream.map(
+ new KeyValueMapper>() {
+ @Override
+ public KeyValue apply(byte[] key, String value) {
+ return new KeyValue<>(value.toLowerCase(), value.length());
+ }
+ });
+
+**Map (values only)**
+
+ * KStream -> KStream
+ * KTable -> KTable
+
+| Takes one record and produces one record, while retaining the key of the original record. You can modify the record value and the value type. ([KStream details](/37/javadoc/org/apache/kafka/streams/kstream/KStream.html#mapValues-org.apache.kafka.streams.kstream.ValueMapper-), [KTable details](/37/javadoc/org/apache/kafka/streams/kstream/KTable.html#mapValues-org.apache.kafka.streams.kstream.ValueMapper-)) `mapValues` is preferable to `map` because it will not cause data re-partitioning. However, it does not allow you to modify the key or key type like `map` does.
+
+
+ KStream stream = ...;
+
+ // Java 8+ example, using lambda expressions
+ KStream uppercased = stream.mapValues(value -> value.toUpperCase());
+
+ // Java 7 example
+ KStream uppercased = stream.mapValues(
+ new ValueMapper() {
+ @Override
+ public String apply(String s) {
+ return s.toUpperCase();
+ }
+ });
+
+**Merge**
+
+ * KStream -> KStream
+
+| Merges records of two streams into one larger stream. ([details](/37/javadoc/org/apache/kafka/streams/kstream/KStream.html#merge-org.apache.kafka.streams.kstream.KStream-)) There is no ordering guarantee between records from different streams in the merged stream. Relative order is preserved within each input stream though (ie, records within the same input stream are processed in order)
+
+
+ KStream stream1 = ...;
+
+ KStream stream2 = ...;
+
+ KStream merged = stream1.merge(stream2);
+
+**Peek**
+
+ * KStream -> KStream
+
+| Performs a stateless action on each record, and returns an unchanged stream. ([details](/37/javadoc/org/apache/kafka/streams/kstream/KStream.html#peek-org.apache.kafka.streams.kstream.ForeachAction-)) You would use `peek` to cause _side effects_ based on the input data (similar to `foreach`) and _continue_ _processing_ the input data (unlike `foreach`, which is a terminal operation). `peek` returns the input stream as-is; if you need to modify the input stream, use `map` or `mapValues` instead. `peek` is helpful for use cases such as logging or tracking metrics or for debugging and troubleshooting. **Note on processing guarantees:** Any side effects of an action (such as writing to external systems) are not trackable by Kafka, which means they will typically not benefit from Kafka's processing guarantees.
+
+
+ KStream stream = ...;
+
+ // Java 8+ example, using lambda expressions
+ KStream unmodifiedStream = stream.peek(
+ (key, value) -> System.out.println("key=" + key + ", value=" + value));
+
+ // Java 7 example
+ KStream unmodifiedStream = stream.peek(
+ new ForeachAction() {
+ @Override
+ public void apply(byte[] key, String value) {
+ System.out.println("key=" + key + ", value=" + value);
+ }
+ });
+
+**Print**
+
+ * KStream -> void
+
+| **Terminal operation.** Prints the records to `System.out`. See Javadocs for serde and `toString()` caveats. ([details](/37/javadoc/org/apache/kafka/streams/kstream/KStream.html#print--)) Calling `print()` is the same as calling `foreach((key, value) -> System.out.println(key + ", " + value))` `print` is mainly for debugging/testing purposes, and it will try to flush on each record print. Hence it **should not** be used for production usage if performance requirements are concerned.
+
+
+ KStream stream = ...;
+ // print to sysout
+ stream.print();
+
+ // print to file with a custom label
+ stream.print(Printed.toFile("streams.out").withLabel("streams"));
+
+**SelectKey**
+
+ * KStream -> KStream
+
+| Assigns a new key - possibly of a new key type - to each record. ([details](/37/javadoc/org/apache/kafka/streams/kstream/KStream.html#selectKey-org.apache.kafka.streams.kstream.KeyValueMapper-)) Calling `selectKey(mapper)` is the same as calling `map((key, value) -> mapper(key, value), value)`. **Marks the stream for data re-partitioning:** Applying a grouping or a join after `selectKey` will result in re-partitioning of the records.
+
+
+ KStream stream = ...;
+
+ // Derive a new record key from the record's value. Note how the key type changes, too.
+ // Java 8+ example, using lambda expressions
+ KStream rekeyed = stream.selectKey((key, value) -> value.split(" ")[0])
+
+ // Java 7 example
+ KStream rekeyed = stream.selectKey(
+ new KeyValueMapper() {
+ @Override
+ public String apply(byte[] key, String value) {
+ return value.split(" ")[0];
+ }
+ });
+
+**Table to Stream**
+
+ * KTable -> KStream
+
+| Get the changelog stream of this table. ([details](/37/javadoc/org/apache/kafka/streams/kstream/KTable.html#toStream--))
+
+
+ KTable table = ...;
+
+ // Also, a variant of `toStream` exists that allows you
+ // to select a new key for the resulting stream.
+ KStream stream = table.toStream();
+
+**Stream to Table**
+
+ * KStream -> KTable
+
+| Convert an event stream into a table, or say a changelog stream. ([details](/37/javadoc/org/apache/kafka/streams/kstream/KStream.html#toTable--))
+
+
+ KStream stream = ...;
+
+ KTable table = stream.toTable();
+
+**Repartition**
+
+ * KStream -> KStream
+
+| Manually trigger repartitioning of the stream with desired number of partitions. ([details](/37/javadoc/org/apache/kafka/streams/kstream/KStream.html#repartition--)) `repartition()` is similar to `through()` however Kafka Streams will manage the topic for you. Generated topic is treated as internal topic, as a result data will be purged automatically as any other internal repartition topic. In addition, you can specify the desired number of partitions, which allows to easily scale in/out downstream sub-topologies. `repartition()` operation always triggers repartitioning of the stream, as a result it can be used with embedded Processor API methods (like `transform()` et al.) that do not trigger auto repartitioning when key changing operation is performed beforehand.
+
+
+ KStream stream = ... ;
+ KStream repartitionedStream = stream.repartition(Repartitioned.numberOfPartitions(10));
+
+## Stateful transformations
+
+Stateful transformations depend on state for processing inputs and producing outputs and require a [state store](../architecture.html#streams_architecture_state) associated with the stream processor. For example, in aggregating operations, a windowing state store is used to collect the latest aggregation results per window. In join operations, a windowing state store is used to collect all of the records received so far within the defined window boundary.
+
+**Note:** Following store types are used regardless of the possibly specified type (via the parameter `materialized`):
+
+ * non-windowed aggregations and non-windowed KTables use [TimestampedKeyValueStore](/37/javadoc/org/apache/kafka/streams/state/TimestampedKeyValueStore.html)s or [VersionedKeyValueStore](/37/javadoc/org/apache/kafka/streams/state/VersionedKeyValueStore.html)s, depending on whether the parameter `materialized` is versioned
+ * time-windowed aggregations and KStream-KStream joins use [TimestampedWindowStore](/37/javadoc/org/apache/kafka/streams/state/TimestampedWindowStore.html)s
+ * session windowed aggregations use [SessionStore](/37/javadoc/org/apache/kafka/streams/state/SessionStore.html)s (there is no timestamped session store as of now)
+
+
+
+Note, that state stores are fault-tolerant. In case of failure, Kafka Streams guarantees to fully restore all state stores prior to resuming the processing. See [Fault Tolerance](../architecture.html#streams_architecture_recovery) for further information.
+
+Available stateful transformations in the DSL include:
+
+ * Aggregating
+ * Joining
+ * Windowing (as part of aggregations and joins)
+ * Applying custom processors and transformers, which may be stateful, for Processor API integration
+
+
+
+The following diagram shows their relationships:
+
+
+
+Stateful transformations in the DSL.
+
+Here is an example of a stateful application: the WordCount algorithm.
+
+WordCount example in Java 8+, using lambda expressions:
+
+
+ // Assume the record values represent lines of text. For the sake of this example, you can ignore
+ // whatever may be stored in the record keys.
+ KStream textLines = ...;
+
+ KStream wordCounts = textLines
+ // Split each text line, by whitespace, into words. The text lines are the record
+ // values, i.e. you can ignore whatever data is in the record keys and thus invoke
+ // `flatMapValues` instead of the more generic `flatMap`.
+ .flatMapValues(value -> Arrays.asList(value.toLowerCase().split("\W+")))
+ // Group the stream by word to ensure the key of the record is the word.
+ .groupBy((key, word) -> word)
+ // Count the occurrences of each word (record key).
+ //
+ // This will change the stream type from `KGroupedStream` to
+ // `KTable` (word -> count).
+ .count()
+ // Convert the `KTable` into a `KStream`.
+ .toStream();
+
+WordCount example in Java 7:
+
+
+ // Code below is equivalent to the previous Java 8+ example above.
+ KStream textLines = ...;
+
+ KStream wordCounts = textLines
+ .flatMapValues(new ValueMapper>() {
+ @Override
+ public Iterable apply(String value) {
+ return Arrays.asList(value.toLowerCase().split("\W+"));
+ }
+ })
+ .groupBy(new KeyValueMapper>() {
+ @Override
+ public String apply(String key, String word) {
+ return word;
+ }
+ })
+ .count()
+ .toStream();
+
+### Aggregating
+
+After records are grouped by key via `groupByKey` or `groupBy` - and thus represented as either a `KGroupedStream` or a `KGroupedTable`, they can be aggregated via an operation such as `reduce`. Aggregations are key-based operations, which means that they always operate over records (notably record values) of the same key. You can perform aggregations on windowed or non-windowed data.
+
+Transformation | Description
+---|---
+**Aggregate**
+
+ * KGroupedStream -> KTable
+ * KGroupedTable -> KTable
+
+| **Rolling aggregation.** Aggregates the values of (non-windowed) records by the grouped key or cogrouped. Aggregating is a generalization of `reduce` and allows, for example, the aggregate value to have a different type than the input values. ([KGroupedStream details](/37/javadoc/org/apache/kafka/streams/kstream/KGroupedStream.html), [KGroupedTable details](/37/javadoc/org/apache/kafka/streams/kstream/KGroupedTable.html) [KGroupedTable details](/37/javadoc/org/apache/kafka/streams/kstream/CogroupedKStream.html)) When aggregating a _grouped stream_ , you must provide an initializer (e.g., `aggValue = 0`) and an "adder" aggregator (e.g., `aggValue + curValue`). When aggregating a _grouped table_ , you must additionally provide a "subtractor" aggregator (think: `aggValue - oldValue`). When aggregating a _cogrouped stream_ , the actual aggregators are provided for each input stream in the prior `cogroup()`calls, and thus you only need to provide an initializer (e.g., `aggValue = 0`) Several variants of `aggregate` exist, see Javadocs for details.
+
+
+ KGroupedStream groupedStream = ...;
+ KGroupedTable groupedTable = ...;
+
+ // Java 8+ examples, using lambda expressions
+
+ // Aggregating a KGroupedStream (note how the value type changes from String to Long)
+ KTable aggregatedStream = groupedStream.aggregate(
+ () -> 0L, /* initializer */
+ (aggKey, newValue, aggValue) -> aggValue + newValue.length(), /* adder */
+ Materialized.>as("aggregated-stream-store") /* state store name */
+ .withValueSerde(Serdes.Long()); /* serde for aggregate value */
+
+ // Aggregating a KGroupedTable (note how the value type changes from String to Long)
+ KTable aggregatedTable = groupedTable.aggregate(
+ () -> 0L, /* initializer */
+ (aggKey, newValue, aggValue) -> aggValue + newValue.length(), /* adder */
+ (aggKey, oldValue, aggValue) -> aggValue - oldValue.length(), /* subtractor */
+ Materialized.>as("aggregated-table-store") /* state store name */
+ .withValueSerde(Serdes.Long()) /* serde for aggregate value */
+
+
+ // Java 7 examples
+
+ // Aggregating a KGroupedStream (note how the value type changes from String to Long)
+ KTable aggregatedStream = groupedStream.aggregate(
+ new Initializer() { /* initializer */
+ @Override
+ public Long apply() {
+ return 0L;
+ }
+ },
+ new Aggregator() { /* adder */
+ @Override
+ public Long apply(byte[] aggKey, String newValue, Long aggValue) {
+ return aggValue + newValue.length();
+ }
+ },
+ Materialized.as("aggregated-stream-store")
+ .withValueSerde(Serdes.Long());
+
+ // Aggregating a KGroupedTable (note how the value type changes from String to Long)
+ KTable aggregatedTable = groupedTable.aggregate(
+ new Initializer() { /* initializer */
+ @Override
+ public Long apply() {
+ return 0L;
+ }
+ },
+ new Aggregator() { /* adder */
+ @Override
+ public Long apply(byte[] aggKey, String newValue, Long aggValue) {
+ return aggValue + newValue.length();
+ }
+ },
+ new Aggregator() { /* subtractor */
+ @Override
+ public Long apply(byte[] aggKey, String oldValue, Long aggValue) {
+ return aggValue - oldValue.length();
+ }
+ },
+ Materialized.as("aggregated-stream-store")
+ .withValueSerde(Serdes.Long());
+
+Detailed behavior of `KGroupedStream`:
+
+ * Input records with `null` keys are ignored.
+ * When a record key is received for the first time, the initializer is called (and called before the adder).
+ * Whenever a record with a non-`null` value is received, the adder is called.
+
+Detailed behavior of `KGroupedTable`:
+
+ * Input records with `null` keys are ignored.
+ * When a record key is received for the first time, the initializer is called (and called before the adder and subtractor). Note that, in contrast to `KGroupedStream`, over time the initializer may be called more than once for a key as a result of having received input tombstone records for that key (see below).
+ * When the first non-`null` value is received for a key (e.g., INSERT), then only the adder is called.
+ * When subsequent non-`null` values are received for a key (e.g., UPDATE), then (1) the subtractor is called with the old value as stored in the table and (2) the adder is called with the new value of the input record that was just received. The subtractor is guaranteed to be called before the adder if the extracted grouping key of the old and new value is the same. The detection of this case depends on the correct implementation of the equals() method of the extracted key type. Otherwise, the order of execution for the subtractor and adder is not defined.
+ * When a tombstone record - i.e. a record with a `null` value - is received for a key (e.g., DELETE), then only the subtractor is called. Note that, whenever the subtractor returns a `null` value itself, then the corresponding key is removed from the resulting `KTable`. If that happens, any next input record for that key will trigger the initializer again.
+
+See the example at the bottom of this section for a visualization of the aggregation semantics.
+**Aggregate (windowed)**
+
+ * KGroupedStream -> KTable
+
+| **Windowed aggregation.** Aggregates the values of records, per window, by the grouped key. Aggregating is a generalization of `reduce` and allows, for example, the aggregate value to have a different type than the input values. ([TimeWindowedKStream details](/37/javadoc/org/apache/kafka/streams/kstream/TimeWindowedKStream.html), [SessionWindowedKStream details](/37/javadoc/org/apache/kafka/streams/kstream/SessionWindowedKStream.html)) You must provide an initializer (e.g., `aggValue = 0`), "adder" aggregator (e.g., `aggValue + curValue`), and a window. When windowing based on sessions, you must additionally provide a "session merger" aggregator (e.g., `mergedAggValue = leftAggValue + rightAggValue`). The windowed `aggregate` turns a `TimeWindowedKStream` or `SessionWindowedKStream` into a windowed `KTable, V>`. Several variants of `aggregate` exist, see Javadocs for details.
+
+
+ import java.time.Duration;
+ KGroupedStream groupedStream = ...;
+
+ // Java 8+ examples, using lambda expressions
+
+ // Aggregating with time-based windowing (here: with 5-minute tumbling windows)
+ KTable, Long> timeWindowedAggregatedStream = groupedStream.windowedBy(Duration.ofMinutes(5))
+ .aggregate(
+ () -> 0L, /* initializer */
+ (aggKey, newValue, aggValue) -> aggValue + newValue, /* adder */
+ Materialized.>as("time-windowed-aggregated-stream-store") /* state store name */
+ .withValueSerde(Serdes.Long())); /* serde for aggregate value */
+
+ // Aggregating with time-based windowing (here: with 5-minute sliding windows and 30-minute grace period)
+ KTable, Long> timeWindowedAggregatedStream = groupedStream.windowedBy(SlidingWindows.ofTimeDifferenceAndGrace(Duration.ofMinutes(5), Duration.ofMinutes(30)))
+ .aggregate(
+ () -> 0L, /* initializer */
+ (aggKey, newValue, aggValue) -> aggValue + newValue, /* adder */
+ Materialized.>as("time-windowed-aggregated-stream-store") /* state store name */
+ .withValueSerde(Serdes.Long())); /* serde for aggregate value */
+
+ // Aggregating with session-based windowing (here: with an inactivity gap of 5 minutes)
+ KTable, Long> sessionizedAggregatedStream = groupedStream.windowedBy(SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(5)).
+ aggregate(
+ () -> 0L, /* initializer */
+ (aggKey, newValue, aggValue) -> aggValue + newValue, /* adder */
+ (aggKey, leftAggValue, rightAggValue) -> leftAggValue + rightAggValue, /* session merger */
+ Materialized.>as("sessionized-aggregated-stream-store") /* state store name */
+ .withValueSerde(Serdes.Long())); /* serde for aggregate value */
+
+ // Java 7 examples
+
+ // Aggregating with time-based windowing (here: with 5-minute tumbling windows)
+ KTable, Long> timeWindowedAggregatedStream = groupedStream.windowedBy(Duration.ofMinutes(5))
+ .aggregate(
+ new Initializer() { /* initializer */
+ @Override
+ public Long apply() {
+ return 0L;
+ }
+ },
+ new Aggregator() { /* adder */
+ @Override
+ public Long apply(String aggKey, Long newValue, Long aggValue) {
+ return aggValue + newValue;
+ }
+ },
+ Materialized.>as("time-windowed-aggregated-stream-store")
+ .withValueSerde(Serdes.Long()));
+
+ // Aggregating with session-based windowing (here: with an inactivity gap of 5 minutes)
+ KTable, Long> sessionizedAggregatedStream = groupedStream.windowedBy(SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(5)).
+ aggregate(
+ new Initializer() { /* initializer */
+ @Override
+ public Long apply() {
+ return 0L;
+ }
+ },
+ new Aggregator() { /* adder */
+ @Override
+ public Long apply(String aggKey, Long newValue, Long aggValue) {
+ return aggValue + newValue;
+ }
+ },
+ new Merger() { /* session merger */
+ @Override
+ public Long apply(String aggKey, Long leftAggValue, Long rightAggValue) {
+ return rightAggValue + leftAggValue;
+ }
+ },
+ Materialized.>as("sessionized-aggregated-stream-store")
+ .withValueSerde(Serdes.Long()));
+
+Detailed behavior:
+
+ * The windowed aggregate behaves similar to the rolling aggregate described above. The additional twist is that the behavior applies _per window_.
+ * Input records with `null` keys are ignored in general.
+ * When a record key is received for the first time for a given window, the initializer is called (and called before the adder).
+ * Whenever a record with a non-`null` value is received for a given window, the adder is called.
+ * When using session windows: the session merger is called whenever two sessions are being merged.
+
+See the example at the bottom of this section for a visualization of the aggregation semantics.
+**Count**
+
+ * KGroupedStream -> KTable
+ * KGroupedTable -> KTable
+
+| **Rolling aggregation.** Counts the number of records by the grouped key. ([KGroupedStream details](/37/javadoc/org/apache/kafka/streams/kstream/KGroupedStream.html), [KGroupedTable details](/37/javadoc/org/apache/kafka/streams/kstream/KGroupedTable.html)) Several variants of `count` exist, see Javadocs for details.
+
+
+ KGroupedStream groupedStream = ...;
+ KGroupedTable groupedTable = ...;
+
+ // Counting a KGroupedStream
+ KTable aggregatedStream = groupedStream.count();
+
+ // Counting a KGroupedTable
+ KTable aggregatedTable = groupedTable.count();
+
+Detailed behavior for `KGroupedStream`:
+
+ * Input records with `null` keys or values are ignored.
+
+Detailed behavior for `KGroupedTable`:
+
+ * Input records with `null` keys are ignored. Records with `null` values are not ignored but interpreted as "tombstones" for the corresponding key, which indicate the deletion of the key from the table.
+
+
+**Count (windowed)**
+
+ * KGroupedStream -> KTable
+
+| **Windowed aggregation.** Counts the number of records, per window, by the grouped key. ([TimeWindowedKStream details](/37/javadoc/org/apache/kafka/streams/kstream/TimeWindowedKStream.html), [SessionWindowedKStream details](/37/javadoc/org/apache/kafka/streams/kstream/SessionWindowedKStream.html)) The windowed `count` turns a `TimeWindowedKStream` or `SessionWindowedKStream` into a windowed `KTable, V>`. Several variants of `count` exist, see Javadocs for details.
+
+
+ import java.time.Duration;
+ KGroupedStream groupedStream = ...;
+
+ // Counting a KGroupedStream with time-based windowing (here: with 5-minute tumbling windows)
+ KTable, Long> aggregatedStream = groupedStream.windowedBy(
+ TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5))) /* time-based window */
+ .count();
+
+ // Counting a KGroupedStream with time-based windowing (here: with 5-minute sliding windows and 30-minute grace period)
+ KTable, Long> aggregatedStream = groupedStream.windowedBy(
+ SlidingWindows.ofTimeDifferenceAndGrace(Duration.ofMinutes(5), Duration.ofMinutes(30))) /* time-based window */
+ .count();
+
+ // Counting a KGroupedStream with session-based windowing (here: with 5-minute inactivity gaps)
+ KTable, Long> aggregatedStream = groupedStream.windowedBy(
+ SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(5))) /* session window */
+ .count();
+
+Detailed behavior:
+
+ * Input records with `null` keys or values are ignored.
+
+
+**Reduce**
+
+ * KGroupedStream -> KTable
+ * KGroupedTable -> KTable
+
+| **Rolling aggregation.** Combines the values of (non-windowed) records by the grouped key. The current record value is combined with the last reduced value, and a new reduced value is returned. The result value type cannot be changed, unlike `aggregate`. ([KGroupedStream details](/37/javadoc/org/apache/kafka/streams/kstream/KGroupedStream.html), [KGroupedTable details](/37/javadoc/org/apache/kafka/streams/kstream/KGroupedTable.html)) When reducing a _grouped stream_ , you must provide an "adder" reducer (e.g., `aggValue + curValue`). When reducing a _grouped table_ , you must additionally provide a "subtractor" reducer (e.g., `aggValue - oldValue`). Several variants of `reduce` exist, see Javadocs for details.
+
+
+ KGroupedStream groupedStream = ...;
+ KGroupedTable groupedTable = ...;
+
+ // Java 8+ examples, using lambda expressions
+
+ // Reducing a KGroupedStream
+ KTable aggregatedStream = groupedStream.reduce(
+ (aggValue, newValue) -> aggValue + newValue /* adder */);
+
+ // Reducing a KGroupedTable
+ KTable aggregatedTable = groupedTable.reduce(
+ (aggValue, newValue) -> aggValue + newValue, /* adder */
+ (aggValue, oldValue) -> aggValue - oldValue /* subtractor */);
+
+
+ // Java 7 examples
+
+ // Reducing a KGroupedStream
+ KTable aggregatedStream = groupedStream.reduce(
+ new Reducer() { /* adder */
+ @Override
+ public Long apply(Long aggValue, Long newValue) {
+ return aggValue + newValue;
+ }
+ });
+
+ // Reducing a KGroupedTable
+ KTable aggregatedTable = groupedTable.reduce(
+ new Reducer() { /* adder */
+ @Override
+ public Long apply(Long aggValue, Long newValue) {
+ return aggValue + newValue;
+ }
+ },
+ new Reducer() { /* subtractor */
+ @Override
+ public Long apply(Long aggValue, Long oldValue) {
+ return aggValue - oldValue;
+ }
+ });
+
+Detailed behavior for `KGroupedStream`:
+
+ * Input records with `null` keys are ignored in general.
+ * When a record key is received for the first time, then the value of that record is used as the initial aggregate value.
+ * Whenever a record with a non-`null` value is received, the adder is called.
+
+Detailed behavior for `KGroupedTable`:
+
+ * Input records with `null` keys are ignored in general.
+ * When a record key is received for the first time, then the value of that record is used as the initial aggregate value. Note that, in contrast to `KGroupedStream`, over time this initialization step may happen more than once for a key as a result of having received input tombstone records for that key (see below).
+ * When the first non-`null` value is received for a key (e.g., INSERT), then only the adder is called.
+ * When subsequent non-`null` values are received for a key (e.g., UPDATE), then (1) the subtractor is called with the old value as stored in the table and (2) the adder is called with the new value of the input record that was just received. The subtractor is guaranteed be called before the adder if the extracted grouping key of the old and new value is the same. The detection of this case depends on the correct implementation of the equals() method of the extracted key type. Otherwise, the order of execution for the subtractor and adder is not defined.
+ * When a tombstone record - i.e. a record with a `null` value - is received for a key (e.g., DELETE), then only the subtractor is called. Note that, whenever the subtractor returns a `null` value itself, then the corresponding key is removed from the resulting `KTable`. If that happens, any next input record for that key will re-initialize its aggregate value.
+
+See the example at the bottom of this section for a visualization of the aggregation semantics.
+**Reduce (windowed)**
+
+ * KGroupedStream -> KTable
+
+| **Windowed aggregation.** Combines the values of records, per window, by the grouped key. The current record value is combined with the last reduced value, and a new reduced value is returned. Records with `null` key or value are ignored. The result value type cannot be changed, unlike `aggregate`. ([TimeWindowedKStream details](/37/javadoc/org/apache/kafka/streams/kstream/TimeWindowedKStream.html), [SessionWindowedKStream details](/37/javadoc/org/apache/kafka/streams/kstream/SessionWindowedKStream.html)) The windowed `reduce` turns a turns a `TimeWindowedKStream` or a `SessionWindowedKStream` into a windowed `KTable, V>`. Several variants of `reduce` exist, see Javadocs for details.
+
+
+ import java.time.Duration;
+ KGroupedStream groupedStream = ...;
+
+ // Java 8+ examples, using lambda expressions
+
+ // Aggregating with time-based windowing (here: with 5-minute tumbling windows)
+ KTable, Long> timeWindowedAggregatedStream = groupedStream.windowedBy(
+ TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)) /* time-based window */)
+ .reduce(
+ (aggValue, newValue) -> aggValue + newValue /* adder */
+ );
+
+ // Aggregating with time-based windowing (here: with 5-minute sliding windows and 30-minute grace)
+ KTable, Long> timeWindowedAggregatedStream = groupedStream.windowedBy(
+ SlidingWindows.ofTimeDifferenceAndGrace(Duration.ofMinutes(5), Duration.ofMinutes(30))) /* time-based window */)
+ .reduce(
+ (aggValue, newValue) -> aggValue + newValue /* adder */
+ );
+
+ // Aggregating with session-based windowing (here: with an inactivity gap of 5 minutes)
+ KTable, Long> sessionzedAggregatedStream = groupedStream.windowedBy(
+ SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(5))) /* session window */
+ .reduce(
+ (aggValue, newValue) -> aggValue + newValue /* adder */
+ );
+
+
+ // Java 7 examples
+
+ // Aggregating with time-based windowing (here: with 5-minute tumbling windows)
+ KTable, Long> timeWindowedAggregatedStream = groupedStream.windowedBy(
+ TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)) /* time-based window */)
+ .reduce(
+ new Reducer() { /* adder */
+ @Override
+ public Long apply(Long aggValue, Long newValue) {
+ return aggValue + newValue;
+ }
+ });
+
+ // Aggregating with session-based windowing (here: with an inactivity gap of 5 minutes)
+ KTable, Long> timeWindowedAggregatedStream = groupedStream.windowedBy(
+ SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(5))) /* session window */
+ .reduce(
+ new Reducer() { /* adder */
+ @Override
+ public Long apply(Long aggValue, Long newValue) {
+ return aggValue + newValue;
+ }
+ });
+
+Detailed behavior:
+
+ * The windowed reduce behaves similar to the rolling reduce described above. The additional twist is that the behavior applies _per window_.
+ * Input records with `null` keys are ignored in general.
+ * When a record key is received for the first time for a given window, then the value of that record is used as the initial aggregate value.
+ * Whenever a record with a non-`null` value is received for a given window, the adder is called.
+
+See the example at the bottom of this section for a visualization of the aggregation semantics.
+
+**Example of semantics for stream aggregations:** A `KGroupedStream` -> `KTable` example is shown below. The streams and the table are initially empty. Bold font is used in the column for "KTable `aggregated`" to highlight changed state. An entry such as `(hello, 1)` denotes a record with key `hello` and value `1`. To improve the readability of the semantics table you can assume that all records are processed in timestamp order.
+
+
+ // Key: word, value: count
+ KStream wordCounts = ...;
+
+ KGroupedStream groupedStream = wordCounts
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.Integer()));
+
+ KTable aggregated = groupedStream.aggregate(
+ () -> 0, /* initializer */
+ (aggKey, newValue, aggValue) -> aggValue + newValue, /* adder */
+ Materialized.as("aggregated-stream-store" /* state store name */)
+ .withKeySerde(Serdes.String()) /* key serde */
+ .withValueSerde(Serdes.Integer()); /* serde for aggregate value */
+
+**Note**
+
+**Impact of record caches** : For illustration purposes, the column "KTable `aggregated`" below shows the table's state changes over time in a very granular way. In practice, you would observe state changes in such a granular way only when [record caches](memory-mgmt.html#streams-developer-guide-memory-management-record-cache) are disabled (default: enabled). When record caches are enabled, what might happen for example is that the output results of the rows with timestamps 4 and 5 would be [compacted](memory-mgmt.html#streams-developer-guide-memory-management-record-cache), and there would only be a single state update for the key `kafka` in the KTable (here: from `(kafka 1)` directly to `(kafka, 3)`. Typically, you should only disable record caches for testing or debugging purposes - under normal circumstances it is better to leave record caches enabled.
+
+ | KStream `wordCounts` | KGroupedStream `groupedStream` | KTable `aggregated`
+---|---|---|---
+Timestamp | Input record | Grouping | Initializer | Adder | State
+1 | (hello, 1) | (hello, 1) | 0 (for hello) | (hello, 0 + 1) | **(hello, 1)**
+2 | (kafka, 1) | (kafka, 1) | 0 (for kafka) | (kafka, 0 + 1) | (hello, 1) **(kafka, 1)**
+3 | (streams, 1) | (streams, 1) | 0 (for streams) | (streams, 0 + 1) | (hello, 1) (kafka, 1) **(streams, 1)**
+4 | (kafka, 1) | (kafka, 1) | | (kafka, 1 + 1) | (hello, 1) (kafka, **2**) (streams, 1)
+5 | (kafka, 1) | (kafka, 1) | | (kafka, 2 + 1) | (hello, 1) (kafka, **3**) (streams, 1)
+6 | (streams, 1) | (streams, 1) | | (streams, 1 + 1) | (hello, 1) (kafka, 3) (streams, **2**)
+
+**Example of semantics for table aggregations:** A `KGroupedTable` -> `KTable` example is shown below. The tables are initially empty. Bold font is used in the column for "KTable `aggregated`" to highlight changed state. An entry such as `(hello, 1)` denotes a record with key `hello` and value `1`. To improve the readability of the semantics table you can assume that all records are processed in timestamp order.
+
+
+ // Key: username, value: user region (abbreviated to "E" for "Europe", "A" for "Asia")
+ KTable userProfiles = ...;
+
+ // Re-group `userProfiles`. Don't read too much into what the grouping does:
+ // its prime purpose in this example is to show the *effects* of the grouping
+ // in the subsequent aggregation.
+ KGroupedTable groupedTable = userProfiles
+ .groupBy((user, region) -> KeyValue.pair(region, user.length()), Serdes.String(), Serdes.Integer());
+
+ KTable aggregated = groupedTable.aggregate(
+ () -> 0, /* initializer */
+ (aggKey, newValue, aggValue) -> aggValue + newValue, /* adder */
+ (aggKey, oldValue, aggValue) -> aggValue - oldValue, /* subtractor */
+ Materialized.as("aggregated-table-store" /* state store name */)
+ .withKeySerde(Serdes.String()) /* key serde */
+ .withValueSerde(Serdes.Integer()); /* serde for aggregate value */
+
+**Note**
+
+**Impact of record caches** : For illustration purposes, the column "KTable `aggregated`" below shows the table's state changes over time in a very granular way. In practice, you would observe state changes in such a granular way only when [record caches](memory-mgmt.html#streams-developer-guide-memory-management-record-cache) are disabled (default: enabled). When record caches are enabled, what might happen for example is that the output results of the rows with timestamps 4 and 5 would be [compacted](memory-mgmt.html#streams-developer-guide-memory-management-record-cache), and there would only be a single state update for the key `kafka` in the KTable (here: from `(kafka 1)` directly to `(kafka, 3)`. Typically, you should only disable record caches for testing or debugging purposes - under normal circumstances it is better to leave record caches enabled.
+
+ | KTable `userProfiles` | KGroupedTable `groupedTable` | KTable `aggregated`
+---|---|---|---
+Timestamp | Input record | Interpreted as | Grouping | Initializer | Adder | Subtractor | State
+1 | (alice, E) | INSERT alice | (E, 5) | 0 (for E) | (E, 0 + 5) | | **(E, 5)**
+2 | (bob, A) | INSERT bob | (A, 3) | 0 (for A) | (A, 0 + 3) | | **(A, 3)** (E, 5)
+3 | (charlie, A) | INSERT charlie | (A, 7) | | (A, 3 + 7) | | (A, **10**) (E, 5)
+4 | (alice, A) | UPDATE alice | (A, 5) | | (A, 10 + 5) | (E, 5 - 5) | (A, **15**) (E, **0**)
+5 | (charlie, null) | DELETE charlie | (null, 7) | | | (A, 15 - 7) | (A, **8**) (E, 0)
+6 | (null, E) | _ignored_ | | | | | (A, 8) (E, 0)
+7 | (bob, E) | UPDATE bob | (E, 3) | | (E, 0 + 3) | (A, 8 - 3) | (A, **5**) (E, **3**)
+
+### Joining
+
+Streams and tables can also be joined. Many stream processing applications in practice are coded as streaming joins. For example, applications backing an online shop might need to access multiple, updating database tables (e.g. sales prices, inventory, customer information) in order to enrich a new data record (e.g. customer transaction) with context information. That is, scenarios where you need to perform table lookups at very large scale and with a low processing latency. Here, a popular pattern is to make the information in the databases available in Kafka through so-called _change data capture_ in combination with [Kafka's Connect API](../../#connect), and then implementing applications that leverage the Streams API to perform very fast and efficient local joins of such tables and streams, rather than requiring the application to make a query to a remote database over the network for each record. In this example, the KTable concept in Kafka Streams would enable you to track the latest state (e.g., snapshot) of each table in a local state store, thus greatly reducing the processing latency as well as reducing the load of the remote databases when doing such streaming joins.
+
+The following join operations are supported, see also the diagram in the overview section of Stateful Transformations. Depending on the operands, joins are either windowed joins or non-windowed joins.
+
+Join operands | Type | (INNER) JOIN | LEFT JOIN | OUTER JOIN
+---|---|---|---|---
+KStream-to-KStream | Windowed | Supported | Supported | Supported
+KTable-to-KTable | Non-windowed | Supported | Supported | Supported
+KTable-to-KTable Foreign-Key Join | Non-windowed | Supported | Supported | Not Supported
+KStream-to-KTable | Non-windowed | Supported | Supported | Not Supported
+KStream-to-GlobalKTable | Non-windowed | Supported | Supported | Not Supported
+KTable-to-GlobalKTable | N/A | Not Supported | Not Supported | Not Supported
+
+Each case is explained in more detail in the subsequent sections.
+
+#### Join co-partitioning requirements
+
+For equi-joins, input data must be co-partitioned when joining. This ensures that input records with the same key from both sides of the join, are delivered to the same stream task during processing. **It is your responsibility to ensure data co-partitioning when joining**. Co-partitioning is not required when performing KTable-KTable Foreign-Key joins and Global KTable joins.
+
+The requirements for data co-partitioning are:
+
+ * The input topics of the join (left side and right side) must have the **same number of partitions**.
+ * All applications that _write_ to the input topics must have the **same partitioning strategy** so that records with the same key are delivered to same partition number. In other words, the keyspace of the input data must be distributed across partitions in the same manner. This means that, for example, applications that use Kafka's [Java Producer API](../../#producerapi) must use the same partitioner (cf. the producer setting `"partitioner.class"` aka `ProducerConfig.PARTITIONER_CLASS_CONFIG`), and applications that use the Kafka's Streams API must use the same `StreamPartitioner` for operations such as `KStream#to()`. The good news is that, if you happen to use the default partitioner-related settings across all applications, you do not need to worry about the partitioning strategy.
+
+
+
+Why is data co-partitioning required? Because KStream-KStream, KTable-KTable, and KStream-KTable joins are performed based on the keys of records (e.g., `leftRecord.key == rightRecord.key`), it is required that the input streams/tables of a join are co-partitioned by key.
+
+There are two exceptions where co-partitioning is not required. For KStream-GlobalKTable joins joins, co-partitioning is not required because _all_ partitions of the `GlobalKTable`'s underlying changelog stream are made available to each `KafkaStreams` instance. That is, each instance has a full copy of the changelog stream. Further, a `KeyValueMapper` allows for non-key based joins from the `KStream` to the `GlobalKTable`. KTable-KTable Foreign-Key joins also do not require co-partitioning. Kafka Streams internally ensures co-partitioning for Foreign-Key joins.
+
+**Note**
+
+**Kafka Streams partly verifies the co-partitioning requirement:** During the partition assignment step, i.e. at runtime, Kafka Streams verifies whether the number of partitions for both sides of a join are the same. If they are not, a `TopologyBuilderException` (runtime exception) is being thrown. Note that Kafka Streams cannot verify whether the partitioning strategy matches between the input streams/tables of a join - it is up to the user to ensure that this is the case.
+
+**Ensuring data co-partitioning:** If the inputs of a join are not co-partitioned yet, you must ensure this manually. You may follow a procedure such as outlined below. It is recommended to repartition the topic with fewer partitions to match the larger partition number of avoid bottlenecks. Technically it would also be possible to repartition the topic with more partitions to the smaller partition number. For stream-table joins, it's recommended to repartition the KStream because repartitioning a KTable might result in a second state store. For table-table joins, you might also consider to size of the KTables and repartition the smaller KTable.
+
+ 1. Identify the input KStream/KTable in the join whose underlying Kafka topic has the smaller number of partitions. Let's call this stream/table "SMALLER", and the other side of the join "LARGER". To learn about the number of partitions of a Kafka topic you can use, for example, the CLI tool `bin/kafka-topics` with the `--describe` option.
+
+ 2. Within your application, re-partition the data of "SMALLER". You must ensure that, when repartitioning the data with `repartition`, the same partitioner is used as for "LARGER".
+
+> * If "SMALLER" is a KStream: `KStream#repartition(Repartitioned.numberOfPartitions(...))`.
+> * If "SMALLER" is a KTable: `KTable#toStream#repartition(Repartitioned.numberOfPartitions(...).toTable())`.
+
+ 3. Within your application, perform the join between "LARGER" and the new stream/table.
+
+
+
+
+#### KStream-KStream Join
+
+KStream-KStream joins are always windowed joins, because otherwise the size of the internal state store used to perform the join - e.g., a sliding window or "buffer" - would grow indefinitely. For stream-stream joins it's important to highlight that a new input record on one side will produce a join output _for each_ matching record on the other side, and there can be _multiple_ such matching records in a given join window (cf. the row with timestamp 15 in the join semantics table below, for example).
+
+Join output records are effectively created as follows, leveraging the user-supplied `ValueJoiner`:
+
+
+ KeyValue leftRecord = ...;
+ KeyValue rightRecord = ...;
+ ValueJoiner joiner = ...;
+
+ KeyValue joinOutputRecord = KeyValue.pair(
+ leftRecord.key, /* by definition, leftRecord.key == rightRecord.key */
+ joiner.apply(leftRecord.value, rightRecord.value)
+ );
+
+Transformation | Description
+---|---
+**Inner Join (windowed)**
+
+ * (KStream, KStream) -> KStream
+
+| Performs an INNER JOIN of this stream with another stream. Even though this operation is windowed, the joined stream will be of type `KStream` rather than `KStream, ...>`. [(details)](/37/javadoc/org/apache/kafka/streams/kstream/KStream.html#join-org.apache.kafka.streams.kstream.KStream-org.apache.kafka.streams.kstream.ValueJoiner-org.apache.kafka.streams.kstream.JoinWindows-) **Data must be co-partitioned** : The input data for both sides must be co-partitioned. **Causes data re-partitioning of a stream if and only if the stream was marked for re-partitioning (if both are marked, both are re-partitioned).** Several variants of `join` exists, see the Javadocs for details.
+
+
+ import java.time.Duration;
+ KStream left = ...;
+ KStream
Kafka protocol guide
- -This document covers the wire protocol implemented in Kafka. It is meant to give a readable guide to the protocol that covers the available requests, their binary format, and the proper way to make use of them to implement a client. This document assumes you understand the basic design and terminology described here
- --
-
- Preliminaries - - -
- The Protocol - - -
- Evolving the Protocol
-
-
-
- The Request Header -
- Versioning -
- Constants
-
-
-
- Error Codes -
- Api Keys -
- - The Messages -
- Some Common Philosophical Questions -
Preliminaries
- -Network
- -Kafka uses a binary protocol over TCP. The protocol defines all APIs as request response message pairs. All messages are size delimited and are made up of the following primitive types.
- -The client initiates a socket connection and then writes a sequence of request messages and reads back the corresponding response message. No handshake is required on connection or disconnection. TCP is happier if you maintain persistent connections used for many requests to amortize the cost of the TCP handshake, but beyond this penalty connecting is pretty cheap.
- -The client will likely need to maintain a connection to multiple brokers, as data is partitioned and the clients will need to talk to the server that has their data. However it should not generally be necessary to maintain multiple connections to a single broker from a single client instance (i.e. connection pooling).
- -The server guarantees that on a single TCP connection, requests will be processed in the order they are sent and responses will return in that order as well. The broker's request processing allows only a single in-flight request per connection in order to guarantee this ordering. Note that clients can (and ideally should) use non-blocking IO to implement request pipelining and achieve higher throughput. i.e., clients can send requests even while awaiting responses for preceding requests since the outstanding requests will be buffered in the underlying OS socket buffer. All requests are initiated by the client, and result in a corresponding response message from the server except where noted.
- -The server has a configurable maximum limit on request size and any request that exceeds this limit will result in the socket being disconnected.
- -Partitioning and bootstrapping
- -Kafka is a partitioned system so not all servers have the complete data set. Instead recall that topics are split into a pre-defined number of partitions, P, and each partition is replicated with some replication factor, N. Topic partitions themselves are just ordered "commit logs" numbered 0, 1, ..., P-1.
- -All systems of this nature have the question of how a particular piece of data is assigned to a particular partition. Kafka clients directly control this assignment, the brokers themselves enforce no particular semantics of which messages should be published to a particular partition. Rather, to publish messages the client directly addresses messages to a particular partition, and when fetching messages, fetches from a particular partition. If two clients want to use the same partitioning scheme they must use the same method to compute the mapping of key to partition.
- -These requests to publish or fetch data must be sent to the broker that is currently acting as the leader for a given partition. This condition is enforced by the broker, so a request for a particular partition to the wrong broker will result in an the NotLeaderForPartition error code (described below).
- -How can the client find out which topics exist, what partitions they have, and which brokers currently host those partitions so that it can direct its requests to the right hosts? This information is dynamic, so you can't just configure each client with some static mapping file. Instead all Kafka brokers can answer a metadata request that describes the current state of the cluster: what topics there are, which partitions those topics have, which broker is the leader for those partitions, and the host and port information for these brokers.
- -In other words, the client needs to somehow find one broker and that broker will tell the client about all the other brokers that exist and what partitions they host. This first broker may itself go down so the best practice for a client implementation is to take a list of two or three URLs to bootstrap from. The user can then choose to use a load balancer or just statically configure two or three of their Kafka hosts in the clients.
- -The client does not need to keep polling to see if the cluster has changed; it can fetch metadata once when it is instantiated cache that metadata until it receives an error indicating that the metadata is out of date. This error can come in two forms: (1) a socket error indicating the client cannot communicate with a particular broker, (2) an error code in the response to a request indicating that this broker no longer hosts the partition for which data was requested.
--
-
- Cycle through a list of "bootstrap" Kafka URLs until we find one we can connect to. Fetch cluster metadata. -
- Process fetch or produce requests, directing them to the appropriate broker based on the topic/partitions they send to or fetch from. -
- If we get an appropriate error, refresh the metadata and try again. -
Partitioning Strategies
- -As mentioned above the assignment of messages to partitions is something the producing client controls. That said, how should this functionality be exposed to the end-user?
- -Partitioning really serves two purposes in Kafka:
--
-
- It balances data and request load over brokers -
- It serves as a way to divvy up processing among consumer processes while allowing local state and preserving order within the partition. We call this semantic partitioning. -
For a given use case you may care about only one of these or both.
- -To accomplish simple load balancing a simple approach would be for the client to just round robin requests over all brokers. Another alternative, in an environment where there are many more producers than brokers, would be to have each client chose a single partition at random and publish to that. This later strategy will result in far fewer TCP connections.
- -Semantic partitioning means using some key in the message to assign messages to partitions. For example if you were processing a click message stream you might want to partition the stream by the user id so that all data for a particular user would go to a single consumer. To accomplish this the client can take a key associated with the message and use some hash of this key to choose the partition to which to deliver the message.
- -Batching
- -Our APIs encourage batching small things together for efficiency. We have found this is a very significant performance win. Both our API to send messages and our API to fetch messages always work with a sequence of messages not a single message to encourage this. A clever client can make use of this and support an "asynchronous" mode in which it batches together messages sent individually and sends them in larger clumps. We go even further with this and allow the batching across multiple topics and partitions, so a produce request may contain data to append to many partitions and a fetch request may pull data from many partitions all at once.
- -The client implementer can choose to ignore this and send everything one at a time if they like.
- -Compatibility
- -Kafka has a "bidirectional" client compatibility policy. In other words, new clients can talk to old servers, and old clients can talk to new servers. This allows users to upgrade either clients or servers without experiencing any downtime. - -
Since the Kafka protocol has changed over time, clients and servers need to agree on the schema of the message that they are sending over the wire. This is done through API versioning. - -
Before each request is sent, the client sends the API key and the API version. These two 16-bit numbers, when taken together, uniquely identify the schema of the message to follow. - -
The intention is that clients will support a range of API versions. When communicating with a particular broker, a given client should use the highest API version supported by both and indicate this version in their requests.
- -The server will reject requests with a version it does not support, and will always respond to the client with exactly the protocol format it expects based on the version it included in its request. The intended upgrade path is that new features would first be rolled out on the server (with the older clients not making use of them) and then as newer clients are deployed these new features would gradually be taken advantage of.
- -Note that KIP-482 tagged fields can be added to a request without incrementing the version number. This offers an additional way of evolving the message schema without breaking compatibility. Tagged fields do not take up any space when the field is not set. Therefore, if a field is rarely used, it is more efficient to make it a tagged field than to put it in the mandatory schema. However, tagged fields are ignored by recipients that don't know about them, which could pose a challenge if this is not the behavior that the sender wants. In such cases, a version bump may be more appropriate. - -
Retrieving Supported API versions
-In order to work against multiple broker versions, clients need to know what versions of various APIs a - broker supports. The broker exposes this information since 0.10.0.0 as described in KIP-35. - Clients should use the supported API versions information to choose the highest API version supported by both client and broker. If no such version - exists, an error should be reported to the user.
-The following sequence may be used by a client to obtain supported API versions from a broker.
--
-
- Client sends
ApiVersionsRequestto a broker after connection has been established with the broker. If SSL is enabled, - this happens after SSL connection has been established.
- - On receiving
ApiVersionsRequest, a broker returns its full list of supported ApiKeys and - versions regardless of current authentication state (e.g., before SASL authentication on an SASL listener, do note that no - Kafka protocol requests may take place on an SSL listener before the SSL handshake is finished). If this is considered to - leak information about the broker version a workaround is to use SSL with client authentication which is performed at an - earlier stage of the connection where theApiVersionRequestis not available. Also, note that broker versions older - than 0.10.0.0 do not support this API and will either ignore the request or close connection in response to the request.
- - If multiple versions of an API are supported by broker and client, clients are recommended to use the latest version supported - by the broker and itself. -
- Deprecation of a protocol version is done by marking an API version as deprecated in the protocol documentation. -
- Supported API versions obtained from a broker are only valid for the connection on which that information is obtained. - In the event of disconnection, the client should obtain the information from the broker again, as the broker might have been - upgraded/downgraded in the mean time. -
SASL Authentication Sequence
-The following sequence is used for SASL authentication: -
-
-
- Kafka
ApiVersionsRequestmay be sent by the client to obtain the version ranges of requests supported by the broker. This is optional.
- - Kafka
SaslHandshakeRequestcontaining the SASL mechanism for authentication is sent by the client. If the requested mechanism is not enabled - in the server, the server responds with the list of supported mechanisms and closes the client connection. If the mechanism is enabled - in the server, the server sends a successful response and continues with SASL authentication.
- - The actual SASL authentication is now performed. If
SaslHandshakeRequestversion is v0, a series of SASL client and server tokens corresponding to the mechanism are sent - as opaque packets without wrapping the messages with Kafka protocol headers. IfSaslHandshakeRequestversion is v1, theSaslAuthenticate- request/response are used, where the actual SASL tokens are wrapped in the Kafka protocol. The error code in the final message from the broker will indicate if authentication succeeded or failed.
- - If authentication succeeds, subsequent packets are handled as Kafka API requests. Otherwise, the client connection is closed. -
For interoperability with 0.9.0.x clients, the first packet received by the server is handled as a SASL/GSSAPI client token if it is not a valid -Kafka request. SASL/GSSAPI authentication is performed starting with this packet, skipping the first two steps above.
- - -The Protocol
- -Protocol Primitive Types
- -The protocol is built out of the following primitive types.
- - -Notes on reading the request format grammars
- -The BNFs below give an exact context free grammar for the request and response binary format. The BNF is intentionally not compact in order to give human-readable name. As always in a BNF a sequence of productions indicates concatenation. When there are multiple possible productions these are separated with '|' and may be enclosed in parenthesis for grouping. The top-level definition is always given first and subsequent sub-parts are indented.
- -Common Request and Response Structure
- -All requests and responses originate from the following grammar which will be incrementally describe through the rest of this document:
- -RequestOrResponse => Size (RequestMessage | ResponseMessage)
- Size => int32| Field | Description |
|---|---|
| message_size | The message_size field gives the size of the subsequent request or response message in bytes. The client can read requests by first reading this 4 byte size as an integer N, and then reading and parsing the subsequent N bytes of the request. |
Record Batch
-A description of the record batch format can be found here.
- -Constants
- -Error Codes
-We use numeric codes to indicate what problem occurred on the server. These can be translated by the client into exceptions or whatever the appropriate error handling mechanism in the client language. Here is a table of the error codes currently in use:
- - -Api Keys
-The following are the numeric codes that the ApiKey in the request can take for each of the below request types.
- - -The Messages
- -This section gives details on each of the individual API Messages, their usage, their binary format, and the meaning of their fields.
- - -Some Common Philosophical Questions
- -Some people have asked why we don't use HTTP. There are a number of reasons, the best is that client implementors can make use of some of the more advanced TCP features--the ability to multiplex requests, the ability to simultaneously poll many connections, etc. We have also found HTTP libraries in many languages to be surprisingly shabby.
- -Others have asked if maybe we shouldn't support many different protocols. Prior experience with this was that it makes it very hard to add and test new features if they have to be ported across many protocol implementations. Our feeling is that most users don't really see multiple protocols as a feature, they just want a good reliable client in the language of their choice.
- -Another question is why we don't adopt XMPP, STOMP, AMQP or an existing protocol. The answer to this varies by protocol, but in general the problem is that the protocol does determine large parts of the implementation and we couldn't do what we are doing if we didn't have control over the protocol. Our belief is that it is possible to do better than existing messaging systems have in providing a truly distributed messaging system, and to do this we need to build something that works differently.
- -A final question is why we don't use a system like Protocol Buffers or Thrift to define our request messages. These packages excel at helping you to managing lots and lots of serialized messages. However we have only a few messages. Support across languages is somewhat spotty (depending on the package). Finally the mapping between binary log format and wire protocol is something we manage somewhat carefully and this would not be possible with these systems. Finally we prefer the style of versioning APIs explicitly and checking this to inferring new values as nulls as it allows more nuanced control of compatibility.
- - - - diff --git a/docs/quickstart.html b/docs/quickstart.html deleted file mode 100644 index e0c842793473b..0000000000000 --- a/docs/quickstart.html +++ /dev/null @@ -1,406 +0,0 @@ - - - - - - - diff --git a/docs/security.html b/docs/security.html deleted file mode 100644 index 895f2b0b0a7a6..0000000000000 --- a/docs/security.html +++ /dev/null @@ -1,2436 +0,0 @@ - - - - - diff --git a/docs/security/_index.md b/docs/security/_index.md new file mode 100644 index 0000000000000..734b80a622f17 --- /dev/null +++ b/docs/security/_index.md @@ -0,0 +1,10 @@ +--- +title: Security +description: +weight: 7 +tags: ['kafka', 'docs', 'security'] +aliases: +keywords: +type: docs +--- + diff --git a/docs/security/authentication-using-sasl.md b/docs/security/authentication-using-sasl.md new file mode 100644 index 0000000000000..faedcc03f3d6a --- /dev/null +++ b/docs/security/authentication-using-sasl.md @@ -0,0 +1,500 @@ +--- +title: Authentication using SASL +description: Authentication using SASL +weight: 4 +tags: ['kafka', 'docs'] +aliases: +keywords: +type: docs +--- + + 1. #### JAAS configuration + +Kafka uses the Java Authentication and Authorization Service ([JAAS](https://docs.oracle.com/javase/8/docs/technotes/guides/security/jaas/JAASRefGuide.html)) for SASL configuration. + + 1. ##### JAAS configuration for Kafka brokers + +`KafkaServer` is the section name in the JAAS file used by each KafkaServer/Broker. This section provides SASL configuration options for the broker including any SASL client connections made by the broker for inter-broker communication. If multiple listeners are configured to use SASL, the section name may be prefixed with the listener name in lower-case followed by a period, e.g. `sasl_ssl.KafkaServer`. + +`Client` section is used to authenticate a SASL connection with zookeeper. It also allows the brokers to set SASL ACL on zookeeper nodes which locks these nodes down so that only the brokers can modify it. It is necessary to have the same principal name across all brokers. If you want to use a section name other than Client, set the system property `zookeeper.sasl.clientconfig` to the appropriate name (_e.g._ , `-Dzookeeper.sasl.clientconfig=ZkClient`). + +ZooKeeper uses "zookeeper" as the service name by default. If you want to change this, set the system property `zookeeper.sasl.client.username` to the appropriate name (_e.g._ , `-Dzookeeper.sasl.client.username=zk`). + +Brokers may also configure JAAS using the broker configuration property `sasl.jaas.config`. The property name must be prefixed with the listener prefix including the SASL mechanism, i.e. `listener.name.{listenerName}.{saslMechanism}.sasl.jaas.config`. Only one login module may be specified in the config value. If multiple mechanisms are configured on a listener, configs must be provided for each mechanism using the listener and mechanism prefix. For example, + + listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \ + username="admin" \ + password="admin-secret"; + listener.name.sasl_ssl.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ + username="admin" \ + password="admin-secret" \ + user_admin="admin-secret" \ + user_alice="alice-secret"; + +If JAAS configuration is defined at different levels, the order of precedence used is: + * Broker configuration property `listener.name.{listenerName}.{saslMechanism}.sasl.jaas.config` + * `{listenerName}.KafkaServer` section of static JAAS configuration + * `KafkaServer` section of static JAAS configuration +Note that ZooKeeper JAAS config may only be configured using static JAAS configuration. + +See GSSAPI (Kerberos), PLAIN, SCRAM or OAUTHBEARER for example broker configurations. + + 2. ##### JAAS configuration for Kafka clients + +Clients may configure JAAS using the client configuration property sasl.jaas.config or using the static JAAS config file similar to brokers. + + 1. ###### JAAS configuration using client configuration property + +Clients may specify JAAS configuration as a producer or consumer property without creating a physical configuration file. This mode also enables different producers and consumers within the same JVM to use different credentials by specifying different properties for each client. If both static JAAS configuration system property `java.security.auth.login.config` and client property `sasl.jaas.config` are specified, the client property will be used. + +See GSSAPI (Kerberos), PLAIN, SCRAM or OAUTHBEARER for example configurations. + + 2. ###### JAAS configuration using static config file + +To configure SASL authentication on the clients using static JAAS config file: + 1. Add a JAAS config file with a client login section named `KafkaClient`. Configure a login module in `KafkaClient` for the selected mechanism as described in the examples for setting up GSSAPI (Kerberos), PLAIN, SCRAM or OAUTHBEARER. For example, GSSAPI credentials may be configured as: + + KafkaClient { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=true + storeKey=true + keyTab="/etc/security/keytabs/kafka_client.keytab" + principal="kafka-client-1@EXAMPLE.COM"; + }; + + 2. Pass the JAAS config file location as JVM parameter to each client JVM. For example: + + -Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf + + 2. #### SASL configuration + +SASL may be used with PLAINTEXT or SSL as the transport layer using the security protocol SASL_PLAINTEXT or SASL_SSL respectively. If SASL_SSL is used, then SSL must also be configured. + + 1. ##### SASL mechanisms + +Kafka supports the following SASL mechanisms: + * GSSAPI (Kerberos) + * PLAIN + * SCRAM-SHA-256 + * SCRAM-SHA-512 + * OAUTHBEARER + 2. ##### SASL configuration for Kafka brokers + + 1. Configure a SASL port in server.properties, by adding at least one of SASL_PLAINTEXT or SASL_SSL to the _listeners_ parameter, which contains one or more comma-separated values: + + listeners=SASL_PLAINTEXT://host.name:port + +If you are only configuring a SASL port (or if you want the Kafka brokers to authenticate each other using SASL) then make sure you set the same SASL protocol for inter-broker communication: + + security.inter.broker.protocol=SASL_PLAINTEXT (or SASL_SSL) + + 2. Select one or more supported mechanisms to enable in the broker and follow the steps to configure SASL for the mechanism. To enable multiple mechanisms in the broker, follow the steps here. + 3. ##### SASL configuration for Kafka clients + +SASL authentication is only supported for the new Java Kafka producer and consumer, the older API is not supported. + +To configure SASL authentication on the clients, select a SASL mechanism that is enabled in the broker for client authentication and follow the steps to configure SASL for the selected mechanism. + +Note: When establishing connections to brokers via SASL, clients may perform a reverse DNS lookup of the broker address. Due to how the JRE implements reverse DNS lookups, clients may observe slow SASL handshakes if fully qualified domain names are not used, for both the client's `bootstrap.servers` and a broker's `advertised.listeners`. + + 3. #### Authentication using SASL/Kerberos + + 1. ##### Prerequisites + + 1. **Kerberos** +If your organization is already using a Kerberos server (for example, by using Active Directory), there is no need to install a new server just for Kafka. Otherwise you will need to install one, your Linux vendor likely has packages for Kerberos and a short guide on how to install and configure it ([Ubuntu](https://help.ubuntu.com/community/Kerberos), [Redhat](https://access.redhat.com/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/installing-kerberos.html)). Note that if you are using Oracle Java, you will need to download JCE policy files for your Java version and copy them to $JAVA_HOME/jre/lib/security. + 2. **Create Kerberos Principals** +If you are using the organization's Kerberos or Active Directory server, ask your Kerberos administrator for a principal for each Kafka broker in your cluster and for every operating system user that will access Kafka with Kerberos authentication (via clients and tools). +If you have installed your own Kerberos, you will need to create these principals yourself using the following commands: + + > sudo /usr/sbin/kadmin.local -q 'addprinc -randkey kafka/{hostname}@{REALM}' + > sudo /usr/sbin/kadmin.local -q "ktadd -k /etc/security/keytabs/{keytabname}.keytab kafka/{hostname}@{REALM}" + + 3. **Make sure all hosts can be reachable using hostnames** \- it is a Kerberos requirement that all your hosts can be resolved with their FQDNs. + 2. ##### Configuring Kafka Brokers + + 1. Add a suitably modified JAAS file similar to the one below to each Kafka broker's config directory, let's call it kafka_server_jaas.conf for this example (note that each broker should have its own keytab): + + KafkaServer { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=true + storeKey=true + keyTab="/etc/security/keytabs/kafka_server.keytab" + principal="kafka/kafka1.hostname.com@EXAMPLE.COM"; + }; + + // Zookeeper client authentication + Client { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=true + storeKey=true + keyTab="/etc/security/keytabs/kafka_server.keytab" + principal="kafka/kafka1.hostname.com@EXAMPLE.COM"; + }; + +`KafkaServer` section in the JAAS file tells the broker which principal to use and the location of the keytab where this principal is stored. It allows the broker to login using the keytab specified in this section. See notes for more details on Zookeeper SASL configuration. + 2. Pass the JAAS and optionally the krb5 file locations as JVM parameters to each Kafka broker (see [here](https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/KerberosReq.html) for more details): + + -Djava.security.krb5.conf=/etc/kafka/krb5.conf + -Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf + + 3. Make sure the keytabs configured in the JAAS file are readable by the operating system user who is starting kafka broker. + 4. Configure SASL port and SASL mechanisms in server.properties as described here. For example: + + listeners=SASL_PLAINTEXT://host.name:port + security.inter.broker.protocol=SASL_PLAINTEXT + sasl.mechanism.inter.broker.protocol=GSSAPI + sasl.enabled.mechanisms=GSSAPI + +We must also configure the service name in server.properties, which should match the principal name of the kafka brokers. In the above example, principal is "kafka/kafka1.hostname.com@EXAMPLE.com", so: + + sasl.kerberos.service.name=kafka + + 3. ##### Configuring Kafka Clients + +To configure SASL authentication on the clients: + 1. Clients (producers, consumers, connect workers, etc) will authenticate to the cluster with their own principal (usually with the same name as the user running the client), so obtain or create these principals as needed. Then configure the JAAS configuration property for each client. Different clients within a JVM may run as different users by specifying different principals. The property `sasl.jaas.config` in producer.properties or consumer.properties describes how clients like producer and consumer can connect to the Kafka Broker. The following is an example configuration for a client using a keytab (recommended for long-running processes): + + sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \ + useKeyTab=true \ + storeKey=true \ + keyTab="/etc/security/keytabs/kafka_client.keytab" \ + principal="kafka-client-1@EXAMPLE.COM"; + +For command-line utilities like kafka-console-consumer or kafka-console-producer, kinit can be used along with "useTicketCache=true" as in: + + sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \ + useTicketCache=true; + +JAAS configuration for clients may alternatively be specified as a JVM parameter similar to brokers as described here. Clients use the login section named `KafkaClient`. This option allows only one user for all client connections from a JVM. + 2. Make sure the keytabs configured in the JAAS configuration are readable by the operating system user who is starting kafka client. + 3. Optionally pass the krb5 file locations as JVM parameters to each client JVM (see [here](https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/KerberosReq.html) for more details): + + -Djava.security.krb5.conf=/etc/kafka/krb5.conf + + 4. Configure the following properties in producer.properties or consumer.properties: + + security.protocol=SASL_PLAINTEXT (or SASL_SSL) + sasl.mechanism=GSSAPI + sasl.kerberos.service.name=kafka + + 4. #### Authentication using SASL/PLAIN + +SASL/PLAIN is a simple username/password authentication mechanism that is typically used with TLS for encryption to implement secure authentication. Kafka supports a default implementation for SASL/PLAIN which can be extended for production use as described here. + +Under the default implementation of `principal.builder.class`, the username is used as the authenticated `Principal` for configuration of ACLs etc. + 1. ##### Configuring Kafka Brokers + + 1. Add a suitably modified JAAS file similar to the one below to each Kafka broker's config directory, let's call it kafka_server_jaas.conf for this example: + + KafkaServer { + org.apache.kafka.common.security.plain.PlainLoginModule required + username="admin" + password="admin-secret" + user_admin="admin-secret" + user_alice="alice-secret"; + }; + +This configuration defines two users (_admin_ and _alice_). The properties `username` and `password` in the `KafkaServer` section are used by the broker to initiate connections to other brokers. In this example, _admin_ is the user for inter-broker communication. The set of properties `user__userName_` defines the passwords for all users that connect to the broker and the broker validates all client connections including those from other brokers using these properties. + 2. Pass the JAAS config file location as JVM parameter to each Kafka broker: + + -Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf + + 3. Configure SASL port and SASL mechanisms in server.properties as described here. For example: + + listeners=SASL_SSL://host.name:port + security.inter.broker.protocol=SASL_SSL + sasl.mechanism.inter.broker.protocol=PLAIN + sasl.enabled.mechanisms=PLAIN + + 2. ##### Configuring Kafka Clients + +To configure SASL authentication on the clients: + 1. Configure the JAAS configuration property for each client in producer.properties or consumer.properties. The login module describes how the clients like producer and consumer can connect to the Kafka Broker. The following is an example configuration for a client for the PLAIN mechanism: + + sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ + username="alice" \ + password="alice-secret"; + +The options `username` and `password` are used by clients to configure the user for client connections. In this example, clients connect to the broker as user _alice_. Different clients within a JVM may connect as different users by specifying different user names and passwords in `sasl.jaas.config`. + +JAAS configuration for clients may alternatively be specified as a JVM parameter similar to brokers as described here. Clients use the login section named `KafkaClient`. This option allows only one user for all client connections from a JVM. + + 2. Configure the following properties in producer.properties or consumer.properties: + + security.protocol=SASL_SSL + sasl.mechanism=PLAIN + + 3. ##### Use of SASL/PLAIN in production + + * SASL/PLAIN should be used only with SSL as transport layer to ensure that clear passwords are not transmitted on the wire without encryption. + * The default implementation of SASL/PLAIN in Kafka specifies usernames and passwords in the JAAS configuration file as shown here. From Kafka version 2.0 onwards, you can avoid storing clear passwords on disk by configuring your own callback handlers that obtain username and password from an external source using the configuration options `sasl.server.callback.handler.class` and `sasl.client.callback.handler.class`. + * In production systems, external authentication servers may implement password authentication. From Kafka version 2.0 onwards, you can plug in your own callback handlers that use external authentication servers for password verification by configuring `sasl.server.callback.handler.class`. + 5. #### Authentication using SASL/SCRAM + +Salted Challenge Response Authentication Mechanism (SCRAM) is a family of SASL mechanisms that addresses the security concerns with traditional mechanisms that perform username/password authentication like PLAIN and DIGEST-MD5. The mechanism is defined in [RFC 5802](https://tools.ietf.org/html/rfc5802). Kafka supports [SCRAM-SHA-256](https://tools.ietf.org/html/rfc7677) and SCRAM-SHA-512 which can be used with TLS to perform secure authentication. Under the default implementation of `principal.builder.class`, the username is used as the authenticated `Principal` for configuration of ACLs etc. The default SCRAM implementation in Kafka stores SCRAM credentials in Zookeeper and is suitable for use in Kafka installations where Zookeeper is on a private network. Refer to Security Considerations for more details. + + 1. ##### Creating SCRAM Credentials + +The SCRAM implementation in Kafka uses Zookeeper as credential store. Credentials can be created in Zookeeper using `kafka-configs.sh`. For each SCRAM mechanism enabled, credentials must be created by adding a config with the mechanism name. Credentials for inter-broker communication must be created before Kafka brokers are started. Client credentials may be created and updated dynamically and updated credentials will be used to authenticate new connections. + +Create SCRAM credentials for user _alice_ with password _alice-secret_ : + + > bin/kafka-configs.sh --zookeeper localhost:2182 --zk-tls-config-file zk_tls_config.properties --alter --add-config 'SCRAM-SHA-256=[iterations=8192,password=alice-secret],SCRAM-SHA-512=[password=alice-secret]' --entity-type users --entity-name alice + +The default iteration count of 4096 is used if iterations are not specified. A random salt is created and the SCRAM identity consisting of salt, iterations, StoredKey and ServerKey are stored in Zookeeper. See [RFC 5802](https://tools.ietf.org/html/rfc5802) for details on SCRAM identity and the individual fields. + +The following examples also require a user _admin_ for inter-broker communication which can be created using: + + > bin/kafka-configs.sh --zookeeper localhost:2182 --zk-tls-config-file zk_tls_config.properties --alter --add-config 'SCRAM-SHA-256=[password=admin-secret],SCRAM-SHA-512=[password=admin-secret]' --entity-type users --entity-name admin + +Existing credentials may be listed using the _\--describe_ option: + + > bin/kafka-configs.sh --zookeeper localhost:2182 --zk-tls-config-file zk_tls_config.properties --describe --entity-type users --entity-name alice + +Credentials may be deleted for one or more SCRAM mechanisms using the _\--alter --delete-config_ option: + + > bin/kafka-configs.sh --zookeeper localhost:2182 --zk-tls-config-file zk_tls_config.properties --alter --delete-config 'SCRAM-SHA-512' --entity-type users --entity-name alice + + 2. ##### Configuring Kafka Brokers + + 1. Add a suitably modified JAAS file similar to the one below to each Kafka broker's config directory, let's call it kafka_server_jaas.conf for this example: + + KafkaServer { + org.apache.kafka.common.security.scram.ScramLoginModule required + username="admin" + password="admin-secret"; + }; + +The properties `username` and `password` in the `KafkaServer` section are used by the broker to initiate connections to other brokers. In this example, _admin_ is the user for inter-broker communication. + 2. Pass the JAAS config file location as JVM parameter to each Kafka broker: + + -Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf + + 3. Configure SASL port and SASL mechanisms in server.properties as described here. For example: + + listeners=SASL_SSL://host.name:port + security.inter.broker.protocol=SASL_SSL + sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256 (or SCRAM-SHA-512) + sasl.enabled.mechanisms=SCRAM-SHA-256 (or SCRAM-SHA-512) + + 3. ##### Configuring Kafka Clients + +To configure SASL authentication on the clients: + 1. Configure the JAAS configuration property for each client in producer.properties or consumer.properties. The login module describes how the clients like producer and consumer can connect to the Kafka Broker. The following is an example configuration for a client for the SCRAM mechanisms: + + sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \ + username="alice" \ + password="alice-secret"; + +The options `username` and `password` are used by clients to configure the user for client connections. In this example, clients connect to the broker as user _alice_. Different clients within a JVM may connect as different users by specifying different user names and passwords in `sasl.jaas.config`. + +JAAS configuration for clients may alternatively be specified as a JVM parameter similar to brokers as described here. Clients use the login section named `KafkaClient`. This option allows only one user for all client connections from a JVM. + + 2. Configure the following properties in producer.properties or consumer.properties: + + security.protocol=SASL_SSL + sasl.mechanism=SCRAM-SHA-256 (or SCRAM-SHA-512) + + 4. ##### Security Considerations for SASL/SCRAM + + * The default implementation of SASL/SCRAM in Kafka stores SCRAM credentials in Zookeeper. This is suitable for production use in installations where Zookeeper is secure and on a private network. + * Kafka supports only the strong hash functions SHA-256 and SHA-512 with a minimum iteration count of 4096. Strong hash functions combined with strong passwords and high iteration counts protect against brute force attacks if Zookeeper security is compromised. + * SCRAM should be used only with TLS-encryption to prevent interception of SCRAM exchanges. This protects against dictionary or brute force attacks and against impersonation if Zookeeper is compromised. + * From Kafka version 2.0 onwards, the default SASL/SCRAM credential store may be overridden using custom callback handlers by configuring `sasl.server.callback.handler.class` in installations where Zookeeper is not secure. + * For more details on security considerations, refer to [RFC 5802](https://tools.ietf.org/html/rfc5802#section-9). + 6. #### Authentication using SASL/OAUTHBEARER + +The [OAuth 2 Authorization Framework](https://tools.ietf.org/html/rfc6749) "enables a third-party application to obtain limited access to an HTTP service, either on behalf of a resource owner by orchestrating an approval interaction between the resource owner and the HTTP service, or by allowing the third-party application to obtain access on its own behalf." The SASL OAUTHBEARER mechanism enables the use of the framework in a SASL (i.e. a non-HTTP) context; it is defined in [RFC 7628](https://tools.ietf.org/html/rfc7628). The default OAUTHBEARER implementation in Kafka creates and validates [Unsecured JSON Web Tokens](https://tools.ietf.org/html/rfc7515#appendix-A.5) and is only suitable for use in non-production Kafka installations. Refer to Security Considerations for more details. + +Under the default implementation of `principal.builder.class`, the principalName of OAuthBearerToken is used as the authenticated `Principal` for configuration of ACLs etc. + 1. ##### Configuring Kafka Brokers + + 1. Add a suitably modified JAAS file similar to the one below to each Kafka broker's config directory, let's call it kafka_server_jaas.conf for this example: + + KafkaServer { + org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required + unsecuredLoginStringClaim_sub="admin"; + }; + +The property `unsecuredLoginStringClaim_sub` in the `KafkaServer` section is used by the broker when it initiates connections to other brokers. In this example, _admin_ will appear in the subject (`sub`) claim and will be the user for inter-broker communication. + 2. Pass the JAAS config file location as JVM parameter to each Kafka broker: + + -Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf + + 3. Configure SASL port and SASL mechanisms in server.properties as described here. For example: + + listeners=SASL_SSL://host.name:port (or SASL_PLAINTEXT if non-production) + security.inter.broker.protocol=SASL_SSL (or SASL_PLAINTEXT if non-production) + sasl.mechanism.inter.broker.protocol=OAUTHBEARER + sasl.enabled.mechanisms=OAUTHBEARER + + 2. ##### Configuring Kafka Clients + +To configure SASL authentication on the clients: + 1. Configure the JAAS configuration property for each client in producer.properties or consumer.properties. The login module describes how the clients like producer and consumer can connect to the Kafka Broker. The following is an example configuration for a client for the OAUTHBEARER mechanisms: + + sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \ + unsecuredLoginStringClaim_sub="alice"; + +The option `unsecuredLoginStringClaim_sub` is used by clients to configure the subject (`sub`) claim, which determines the user for client connections. In this example, clients connect to the broker as user _alice_. Different clients within a JVM may connect as different users by specifying different subject (`sub`) claims in `sasl.jaas.config`. + +JAAS configuration for clients may alternatively be specified as a JVM parameter similar to brokers as described here. Clients use the login section named `KafkaClient`. This option allows only one user for all client connections from a JVM. + + 2. Configure the following properties in producer.properties or consumer.properties: + + security.protocol=SASL_SSL (or SASL_PLAINTEXT if non-production) + sasl.mechanism=OAUTHBEARER + + 3. The default implementation of SASL/OAUTHBEARER depends on the jackson-databind library. Since it's an optional dependency, users have to configure it as a dependency via their build tool. + 3. ##### Unsecured Token Creation Options for SASL/OAUTHBEARER + + * The default implementation of SASL/OAUTHBEARER in Kafka creates and validates [Unsecured JSON Web Tokens](https://tools.ietf.org/html/rfc7515#appendix-A.5). While suitable only for non-production use, it does provide the flexibility to create arbitrary tokens in a DEV or TEST environment. + * Here are the various supported JAAS module options on the client side (and on the broker side if OAUTHBEARER is the inter-broker protocol): JAAS Module Option for Unsecured Token Creation | Documentation +---|--- +`unsecuredLoginStringClaim_
-
-
-
-
diff --git a/docs/streams/architecture.md b/docs/streams/architecture.md
new file mode 100644
index 0000000000000..415eb3c80749e
--- /dev/null
+++ b/docs/streams/architecture.md
@@ -0,0 +1,81 @@
+---
+title: Architecture
+description:
+weight: 5
+tags: ['kafka', 'docs']
+aliases:
+keywords:
+type: docs
+---
+
+Kafka Streams simplifies application development by building on the Kafka producer and consumer libraries and leveraging the native capabilities of Kafka to offer data parallelism, distributed coordination, fault tolerance, and operational simplicity. In this section, we describe how Kafka Streams works underneath the covers.
+
+The picture below shows the anatomy of an application that uses the Kafka Streams library. Let's walk through some details.
+
+
+
+## Stream Partitions and Tasks
+
+The messaging layer of Kafka partitions data for storing and transporting it. Kafka Streams partitions data for processing it. In both cases, this partitioning is what enables data locality, elasticity, scalability, high performance, and fault tolerance. Kafka Streams uses the concepts of **partitions** and **tasks** as logical units of its parallelism model based on Kafka topic partitions. There are close links between Kafka Streams and Kafka in the context of parallelism:
+
+ * Each **stream partition** is a totally ordered sequence of data records and maps to a Kafka **topic partition**.
+ * A **data record** in the stream maps to a Kafka **message** from that topic.
+ * The **keys** of data records determine the partitioning of data in both Kafka and Kafka Streams, i.e., how data is routed to specific partitions within topics.
+
+
+
+An application's processor topology is scaled by breaking it into multiple tasks. More specifically, Kafka Streams creates a fixed number of tasks based on the input stream partitions for the application, with each task assigned a list of partitions from the input streams (i.e., Kafka topics). The assignment of partitions to tasks never changes so that each task is a fixed unit of parallelism of the application. Tasks can then instantiate their own processor topology based on the assigned partitions; they also maintain a buffer for each of its assigned partitions and process messages one-at-a-time from these record buffers. As a result stream tasks can be processed independently and in parallel without manual intervention.
+
+Slightly simplified, the maximum parallelism at which your application may run is bounded by the maximum number of stream tasks, which itself is determined by maximum number of partitions of the input topic(s) the application is reading from. For example, if your input topic has 5 partitions, then you can run up to 5 applications instances. These instances will collaboratively process the topic's data. If you run a larger number of app instances than partitions of the input topic, the "excess" app instances will launch but remain idle; however, if one of the busy instances goes down, one of the idle instances will resume the former's work.
+
+It is important to understand that Kafka Streams is not a resource manager, but a library that "runs" anywhere its stream processing application runs. Multiple instances of the application are executed either on the same machine, or spread across multiple machines and tasks can be distributed automatically by the library to those running application instances. The assignment of partitions to tasks never changes; if an application instance fails, all its assigned tasks will be automatically restarted on other instances and continue to consume from the same stream partitions.
+
+**NOTE:** Topic partitions are assigned to tasks, and tasks are assigned to all threads over all instances, in a best-effort attempt to trade off load-balancing and stickiness of stateful tasks. For this assignment, Kafka Streams uses the [StreamsPartitionAssignor](https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamsPartitionAssignor.java) class and doesn't let you change to a different assignor. If you try to use a different assignor, Kafka Streams ignores it.
+
+The following diagram shows two tasks each assigned with one partition of the input streams.
+
+
+
+
+## Threading Model
+
+Kafka Streams allows the user to configure the number of **threads** that the library can use to parallelize processing within an application instance. Each thread can execute one or more tasks with their processor topologies independently. For example, the following diagram shows one stream thread running two stream tasks.
+
+
+
+Starting more stream threads or more instances of the application merely amounts to replicating the topology and having it process a different subset of Kafka partitions, effectively parallelizing processing. It is worth noting that there is no shared state amongst the threads, so no inter-thread coordination is necessary. This makes it very simple to run topologies in parallel across the application instances and threads. The assignment of Kafka topic partitions amongst the various stream threads is transparently handled by Kafka Streams leveraging [Kafka's coordination](https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal) functionality.
+
+As we described above, scaling your stream processing application with Kafka Streams is easy: you merely need to start additional instances of your application, and Kafka Streams takes care of distributing partitions amongst tasks that run in the application instances. You can start as many threads of the application as there are input Kafka topic partitions so that, across all running instances of an application, every thread (or rather, the tasks it runs) has at least one input partition to process.
+
+As of Kafka 2.8 you can scale stream threads much in the same way you can scale your Kafka Stream clients. Simply add or remove stream threads and Kafka Streams will take care of redistributing the partitions. You may also add threads to replace stream threads that have died removing the need to restart clients to recover the number of thread running.
+
+
+
+
+## Local State Stores
+
+Kafka Streams provides so-called **state stores** , which can be used by stream processing applications to store and query data, which is an important capability when implementing stateful operations. The [Kafka Streams DSL](/37/streams/developer-guide/dsl-api.html), for example, automatically creates and manages such state stores when you are calling stateful operators such as `join()` or `aggregate()`, or when you are windowing a stream.
+
+Every stream task in a Kafka Streams application may embed one or more local state stores that can be accessed via APIs to store and query data required for processing. Kafka Streams offers fault-tolerance and automatic recovery for such local state stores.
+
+The following diagram shows two stream tasks with their dedicated local state stores.
+
+
+
+
+## Fault Tolerance
+
+Kafka Streams builds on fault-tolerance capabilities integrated natively within Kafka. Kafka partitions are highly available and replicated; so when stream data is persisted to Kafka it is available even if the application fails and needs to re-process it. Tasks in Kafka Streams leverage the fault-tolerance capability offered by the Kafka consumer client to handle failures. If a task runs on a machine that fails, Kafka Streams automatically restarts the task in one of the remaining running instances of the application.
+
+In addition, Kafka Streams makes sure that the local state stores are robust to failures, too. For each state store, it maintains a replicated changelog Kafka topic in which it tracks any state updates. These changelog topics are partitioned as well so that each local state store instance, and hence the task accessing the store, has its own dedicated changelog topic partition. [Log compaction](/37/#compaction) is enabled on the changelog topics so that old data can be purged safely to prevent the topics from growing indefinitely. If tasks run on a machine that fails and are restarted on another machine, Kafka Streams guarantees to restore their associated state stores to the content before the failure by replaying the corresponding changelog topics prior to resuming the processing on the newly started tasks. As a result, failure handling is completely transparent to the end user.
+
+Note that the cost of task (re)initialization typically depends primarily on the time for restoring the state by replaying the state stores' associated changelog topics. To minimize this restoration time, users can configure their applications to have **standby replicas** of local states (i.e. fully replicated copies of the state). When a task migration happens, Kafka Streams will assign a task to an application instance where such a standby replica already exists in order to minimize the task (re)initialization cost. See `num.standby.replicas` in the [**Kafka Streams Configs**](/37/#streamsconfigs) section. Starting in 2.6, Kafka Streams will guarantee that a task is only ever assigned to an instance with a fully caught-up local copy of the state, if such an instance exists. Standby tasks will increase the likelihood that a caught-up instance exists in the case of a failure.
+
+You can also configure standby replicas with rack awareness. When configured, Kafka Streams will attempt to distribute a standby task on a different "rack" than the active one, thus having a faster recovery time when the rack of the active tasks fails. See `rack.aware.assignment.tags` in the [**Kafka Streams Developer Guide**](/37/streams/developer-guide/config-streams.html#rack-aware-assignment-tags) section.
+
+There is also a client config `client.rack` which can set the rack for a Kafka consumer. If brokers also have their rack set via `broker.rack`, then rack aware task assignment can be enabled via `rack.aware.assignment.strategy` (cf. [**Kafka Streams Developer Guide**](/37/streams/developer-guide/config-streams.html#rack-aware-assignment-strategy)) to compute a task assignment which can reduce cross rack traffic by trying to assign tasks to clients with the same rack. Note that `client.rack` can also be used to distribute standby tasks to different racks from the active ones, which has a similar functionality as `rack.aware.assignment.tags`. Currently, `rack.aware.assignment.tag` takes precedence in distributing standby tasks which means if both configs present, `rack.aware.assignment.tag` will be used for distributing standby tasks on different racks from the active ones because it can configure more tag keys.
+
+ * [Documentation](/documentation)
+ * [Kafka Streams](/streams)
+
+
diff --git a/docs/streams/core-concepts.html b/docs/streams/core-concepts.html
deleted file mode 100644
index d9a2851e2713f..0000000000000
--- a/docs/streams/core-concepts.html
+++ /dev/null
@@ -1,387 +0,0 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/docs/streams/core-concepts.md b/docs/streams/core-concepts.md
new file mode 100644
index 0000000000000..8cf3bbbc7b09f
--- /dev/null
+++ b/docs/streams/core-concepts.md
@@ -0,0 +1,158 @@
+---
+title: Core Concepts
+description:
+weight: 4
+tags: ['kafka', 'docs']
+aliases:
+keywords:
+type: docs
+---
+
+Kafka Streams is a client library for processing and analyzing data stored in Kafka. It builds upon important stream processing concepts such as properly distinguishing between event time and processing time, windowing support, and simple yet efficient management and real-time querying of application state.
+
+Kafka Streams has a **low barrier to entry** : You can quickly write and run a small-scale proof-of-concept on a single machine; and you only need to run additional instances of your application on multiple machines to scale up to high-volume production workloads. Kafka Streams transparently handles the load balancing of multiple instances of the same application by leveraging Kafka's parallelism model.
+
+Some highlights of Kafka Streams:
+
+ * Designed as a **simple and lightweight client library** , which can be easily embedded in any Java application and integrated with any existing packaging, deployment and operational tools that users have for their streaming applications.
+ * Has **no external dependencies on systems other than Apache Kafka itself** as the internal messaging layer; notably, it uses Kafka's partitioning model to horizontally scale processing while maintaining strong ordering guarantees.
+ * Supports **fault-tolerant local state** , which enables very fast and efficient stateful operations like windowed joins and aggregations.
+ * Supports **exactly-once** processing semantics to guarantee that each record will be processed once and only once even when there is a failure on either Streams clients or Kafka brokers in the middle of processing.
+ * Employs **one-record-at-a-time processing** to achieve millisecond processing latency, and supports **event-time based windowing operations** with out-of-order arrival of records.
+ * Offers necessary stream processing primitives, along with a **high-level Streams DSL** and a **low-level Processor API**.
+
+
+
+We first summarize the key concepts of Kafka Streams.
+
+## Stream Processing Topology
+
+ * A **stream** is the most important abstraction provided by Kafka Streams: it represents an unbounded, continuously updating data set. A stream is an ordered, replayable, and fault-tolerant sequence of immutable data records, where a **data record** is defined as a key-value pair.
+ * A **stream processing application** is any program that makes use of the Kafka Streams library. It defines its computational logic through one or more **processor topologies** , where a processor topology is a graph of stream processors (nodes) that are connected by streams (edges).
+ * A [**stream processor**](/37/streams/developer-guide/processor-api#defining-a-stream-processor) is a node in the processor topology; it represents a processing step to transform data in streams by receiving one input record at a time from its upstream processors in the topology, applying its operation to it, and may subsequently produce one or more output records to its downstream processors.
+
+There are two special processors in the topology:
+ * **Source Processor** : A source processor is a special type of stream processor that does not have any upstream processors. It produces an input stream to its topology from one or multiple Kafka topics by consuming records from these topics and forwarding them to its down-stream processors.
+ * **Sink Processor** : A sink processor is a special type of stream processor that does not have down-stream processors. It sends any received records from its up-stream processors to a specified Kafka topic.
+
+Note that in normal processor nodes other remote systems can also be accessed while processing the current record. Therefore the processed results can either be streamed back into Kafka or written to an external system. 
+
+Kafka Streams offers two ways to define the stream processing topology: the [**Kafka Streams DSL**](/37/streams/developer-guide/dsl-api.html) provides the most common data transformation operations such as `map`, `filter`, `join` and `aggregations` out of the box; the lower-level [**Processor API**](/37/streams/developer-guide/processor-api.html) allows developers define and connect custom processors as well as to interact with state stores.
+
+A processor topology is merely a logical abstraction for your stream processing code. At runtime, the logical topology is instantiated and replicated inside the application for parallel processing (see [**Stream Partitions and Tasks**](/37/streams/architecture#streams_architecture_tasks) for details).
+
+## Time
+
+A critical aspect in stream processing is the notion of **time** , and how it is modeled and integrated. For example, some operations such as **windowing** are defined based on time boundaries.
+
+Common notions of time in streams are:
+
+ * **Event time** \- The point in time when an event or data record occurred, i.e. was originally created "at the source". **Example:** If the event is a geo-location change reported by a GPS sensor in a car, then the associated event-time would be the time when the GPS sensor captured the location change.
+ * **Processing time** \- The point in time when the event or data record happens to be processed by the stream processing application, i.e. when the record is being consumed. The processing time may be milliseconds, hours, or days etc. later than the original event time. **Example:** Imagine an analytics application that reads and processes the geo-location data reported from car sensors to present it to a fleet management dashboard. Here, processing-time in the analytics application might be milliseconds or seconds (e.g. for real-time pipelines based on Apache Kafka and Kafka Streams) or hours (e.g. for batch pipelines based on Apache Hadoop or Apache Spark) after event-time.
+ * **Ingestion time** \- The point in time when an event or data record is stored in a topic partition by a Kafka broker. The difference to event time is that this ingestion timestamp is generated when the record is appended to the target topic by the Kafka broker, not when the record is created "at the source". The difference to processing time is that processing time is when the stream processing application processes the record. **For example,** if a record is never processed, there is no notion of processing time for it, but it still has an ingestion time.
+
+
+
+The choice between event-time and ingestion-time is actually done through the configuration of Kafka (not Kafka Streams): From Kafka 0.10.x onwards, timestamps are automatically embedded into Kafka messages. Depending on Kafka's configuration these timestamps represent event-time or ingestion-time. The respective Kafka configuration setting can be specified on the broker level or per topic. The default timestamp extractor in Kafka Streams will retrieve these embedded timestamps as-is. Hence, the effective time semantics of your application depend on the effective Kafka configuration for these embedded timestamps.
+
+Kafka Streams assigns a **timestamp** to every data record via the `TimestampExtractor` interface. These per-record timestamps describe the progress of a stream with regards to time and are leveraged by time-dependent operations such as window operations. As a result, this time will only advance when a new record arrives at the processor. We call this data-driven time the **stream time** of the application to differentiate with the **wall-clock time** when this application is actually executing. Concrete implementations of the `TimestampExtractor` interface will then provide different semantics to the stream time definition. For example retrieving or computing timestamps based on the actual contents of data records such as an embedded timestamp field to provide event time semantics, and returning the current wall-clock time thereby yield processing time semantics to stream time. Developers can thus enforce different notions of time depending on their business needs.
+
+Finally, whenever a Kafka Streams application writes records to Kafka, then it will also assign timestamps to these new records. The way the timestamps are assigned depends on the context:
+
+ * When new output records are generated via processing some input record, for example, `context.forward()` triggered in the `process()` function call, output record timestamps are inherited from input record timestamps directly.
+ * When new output records are generated via periodic functions such as `Punctuator#punctuate()`, the output record timestamp is defined as the current internal time (obtained through `context.timestamp()`) of the stream task.
+ * For aggregations, the timestamp of a result update record will be the maximum timestamp of all input records contributing to the result.
+
+
+
+You can change the default behavior in the Processor API by assigning timestamps to output records explicitly when calling `#forward()`.
+
+For aggregations and joins, timestamps are computed by using the following rules.
+
+ * For joins (stream-stream, table-table) that have left and right input records, the timestamp of the output record is assigned `max(left.ts, right.ts)`.
+ * For stream-table joins, the output record is assigned the timestamp from the stream record.
+ * For aggregations, Kafka Streams also computes the `max` timestamp over all records, per key, either globally (for non-windowed) or per-window.
+ * For stateless operations, the input record timestamp is passed through. For `flatMap` and siblings that emit multiple records, all output records inherit the timestamp from the corresponding input record.
+
+
+
+## Duality of Streams and Tables
+
+When implementing stream processing use cases in practice, you typically need both **streams** and also **databases**. An example use case that is very common in practice is an e-commerce application that enriches an incoming _stream_ of customer transactions with the latest customer information from a _database table_. In other words, streams are everywhere, but databases are everywhere, too.
+
+Any stream processing technology must therefore provide **first-class support for streams and tables**. Kafka's Streams API provides such functionality through its core abstractions for [streams](/37/streams/developer-guide/dsl-api#streams_concepts_kstream) and [tables](/37/streams/developer-guide/dsl-api#streams_concepts_ktable), which we will talk about in a minute. Now, an interesting observation is that there is actually a **close relationship between streams and tables** , the so-called stream-table duality. And Kafka exploits this duality in many ways: for example, to make your applications [elastic](/37/streams/developer-guide/running-app#elastic-scaling-of-your-application), to support [fault-tolerant stateful processing](/37/streams/architecture#streams_architecture_recovery), or to run [interactive queries](/37/streams/developer-guide/interactive-queries#interactive-queries) against your application's latest processing results. And, beyond its internal usage, the Kafka Streams API also allows developers to exploit this duality in their own applications.
+
+Before we discuss concepts such as [aggregations](/37/streams/developer-guide/dsl-api#aggregating) in Kafka Streams, we must first introduce **tables** in more detail, and talk about the aforementioned stream-table duality. Essentially, this duality means that a stream can be viewed as a table, and a table can be viewed as a stream. Kafka's log compaction feature, for example, exploits this duality.
+
+A simple form of a table is a collection of key-value pairs, also called a map or associative array. Such a table may look as follows:
+
+ The **stream-table duality** describes the close relationship between streams and tables.
+
+ * **Stream as Table** : A stream can be considered a changelog of a table, where each data record in the stream captures a state change of the table. A stream is thus a table in disguise, and it can be easily turned into a "real" table by replaying the changelog from beginning to end to reconstruct the table. Similarly, in a more general analogy, aggregating data records in a stream - such as computing the total number of pageviews by user from a stream of pageview events - will return a table (here with the key and the value being the user and its corresponding pageview count, respectively).
+ * **Table as Stream** : A table can be considered a snapshot, at a point in time, of the latest value for each key in a stream (a stream's data records are key-value pairs). A table is thus a stream in disguise, and it can be easily turned into a "real" stream by iterating over each key-value entry in the table.
+
+
+
+Let's illustrate this with an example. Imagine a table that tracks the total number of pageviews by user (first column of diagram below). Over time, whenever a new pageview event is processed, the state of the table is updated accordingly. Here, the state changes between different points in time - and different revisions of the table - can be represented as a changelog stream (second column).
+
+
+
+Interestingly, because of the stream-table duality, the same stream can be used to reconstruct the original table (third column):
+
+
+
+The same mechanism is used, for example, to replicate databases via change data capture (CDC) and, within Kafka Streams, to replicate its so-called state stores across machines for fault-tolerance. The stream-table duality is such an important concept that Kafka Streams models it explicitly via the KStream, KTable, and GlobalKTable interfaces.
+
+## Aggregations
+
+An **aggregation** operation takes one input stream or table, and yields a new table by combining multiple input records into a single output record. Examples of aggregations are computing counts or sum.
+
+In the `Kafka Streams DSL`, an input stream of an `aggregation` can be a KStream or a KTable, but the output stream will always be a KTable. This allows Kafka Streams to update an aggregate value upon the out-of-order arrival of further records after the value was produced and emitted. When such out-of-order arrival happens, the aggregating KStream or KTable emits a new aggregate value. Because the output is a KTable, the new value is considered to overwrite the old value with the same key in subsequent processing steps.
+
+## Windowing
+
+Windowing lets you control how to _group records that have the same key_ for stateful operations such as `aggregations` or `joins` into so-called _windows_. Windows are tracked per record key.
+
+`Windowing operations` are available in the `Kafka Streams DSL`. When working with windows, you can specify a **grace period** for the window. This grace period controls how long Kafka Streams will wait for **out-of-order** data records for a given window. If a record arrives after the grace period of a window has passed, the record is discarded and will not be processed in that window. Specifically, a record is discarded if its timestamp dictates it belongs to a window, but the current stream time is greater than the end of the window plus the grace period.
+
+Out-of-order records are always possible in the real world and should be properly accounted for in your applications. It depends on the effective `time semantics ` how out-of-order records are handled. In the case of processing-time, the semantics are "when the record is being processed", which means that the notion of out-of-order records is not applicable as, by definition, no record can be out-of-order. Hence, out-of-order records can only be considered as such for event-time. In both cases, Kafka Streams is able to properly handle out-of-order records.
+
+## States
+
+Some stream processing applications don't require state, which means the processing of a message is independent from the processing of all other messages. However, being able to maintain state opens up many possibilities for sophisticated stream processing applications: you can join input streams, or group and aggregate data records. Many such stateful operators are provided by the [**Kafka Streams DSL**](/37/streams/developer-guide/dsl-api.html).
+
+Kafka Streams provides so-called **state stores** , which can be used by stream processing applications to store and query data. This is an important capability when implementing stateful operations. Every task in Kafka Streams embeds one or more state stores that can be accessed via APIs to store and query data required for processing. These state stores can either be a persistent key-value store, an in-memory hashmap, or another convenient data structure. Kafka Streams offers fault-tolerance and automatic recovery for local state stores.
+
+Kafka Streams allows direct read-only queries of the state stores by methods, threads, processes or applications external to the stream processing application that created the state stores. This is provided through a feature called **Interactive Queries**. All stores are named and Interactive Queries exposes only the read operations of the underlying implementation.
+
+
+
+
+# Processing Guarantees
+
+In stream processing, one of the most frequently asked question is "does my stream processing system guarantee that each record is processed once and only once, even if some failures are encountered in the middle of processing?" Failing to guarantee exactly-once stream processing is a deal-breaker for many applications that cannot tolerate any data-loss or data duplicates, and in that case a batch-oriented framework is usually used in addition to the stream processing pipeline, known as the [Lambda Architecture](https://en.wikipedia.org/wiki/Lambda_architecture). Prior to 0.11.0.0, Kafka only provides at-least-once delivery guarantees and hence any stream processing systems that leverage it as the backend storage could not guarantee end-to-end exactly-once semantics. In fact, even for those stream processing systems that claim to support exactly-once processing, as long as they are reading from / writing to Kafka as the source / sink, their applications cannot actually guarantee that no duplicates will be generated throughout the pipeline.
+Since the 0.11.0.0 release, Kafka has added support to allow its producers to send messages to different topic partitions in a [transactional and idempotent manner](https://kafka.apache.org/#semantics), and Kafka Streams has hence added the end-to-end exactly-once processing semantics by leveraging these features. More specifically, it guarantees that for any record read from the source Kafka topics, its processing results will be reflected exactly once in the output Kafka topic as well as in the state stores for stateful operations. Note the key difference between Kafka Streams end-to-end exactly-once guarantee with other stream processing frameworks' claimed guarantees is that Kafka Streams tightly integrates with the underlying Kafka storage system and ensure that commits on the input topic offsets, updates on the state stores, and writes to the output topics will be completed atomically instead of treating Kafka as an external system that may have side-effects. For more information on how this is done inside Kafka Streams, see [KIP-129](https://cwiki.apache.org/confluence/display/KAFKA/KIP-129%3A+Streams+Exactly-Once+Semantics).
+As of the 2.6.0 release, Kafka Streams supports an improved implementation of exactly-once processing, named "exactly-once v2", which requires broker version 2.5.0 or newer. This implementation is more efficient, because it reduces client and broker resource utilization, like client threads and used network connections, and it enables higher throughput and improved scalability. As of the 3.0.0 release, the first version of exactly-once has been deprecated. Users are encouraged to use exactly-once v2 for exactly-once processing from now on, and prepare by upgrading their brokers if necessary. For more information on how this is done inside the brokers and Kafka Streams, see [KIP-447](https://cwiki.apache.org/confluence/display/KAFKA/KIP-447%3A+Producer+scalability+for+exactly+once+semantics).
+To enable exactly-once semantics when running Kafka Streams applications, set the `processing.guarantee` config value (default value is **at_least_once**) to **StreamsConfig.EXACTLY_ONCE_V2** (requires brokers version 2.5 or newer). For more information, see the [Kafka Streams Configs](/37/streams/developer-guide/config-streams.html) section.
+
+## Out-of-Order Handling
+
+Besides the guarantee that each record will be processed exactly-once, another issue that many stream processing applications will face is how to handle [out-of-order data](https://dl.acm.org/citation.cfm?id=3242155) that may impact their business logic. In Kafka Streams, there are two causes that could potentially result in out-of-order data arrivals with respect to their timestamps:
+
+ * Within a topic-partition, a record's timestamp may not be monotonically increasing along with their offsets. Since Kafka Streams will always try to process records within a topic-partition to follow the offset order, it can cause records with larger timestamps (but smaller offsets) to be processed earlier than records with smaller timestamps (but larger offsets) in the same topic-partition.
+ * Within a [stream task](/37/streams/architecture#streams_architecture_tasks) that may be processing multiple topic-partitions, if users configure the application to not wait for all partitions to contain some buffered data and pick from the partition with the smallest timestamp to process the next record, then later on when some records are fetched for other topic-partitions, their timestamps may be smaller than those processed records fetched from another topic-partition.
+
+
+
+For stateless operations, out-of-order data will not impact processing logic since only one record is considered at a time, without looking into the history of past processed records; for stateful operations such as aggregations and joins, however, out-of-order data could cause the processing logic to be incorrect. If users want to handle such out-of-order data, generally they need to allow their applications to wait for longer time while bookkeeping their states during the wait time, i.e. making trade-off decisions between latency, cost, and correctness. In Kafka Streams specifically, users can configure their window operators for windowed aggregations to achieve such trade-offs (details can be found in [**Developer Guide**](/37/streams/developer-guide)). As for Joins, users may use [versioned state stores](/37/streams/developer-guide/dsl-api.html#versioned-state-stores) to address concerns with out-of-order data, but out-of-order data will not be handled by default:
+
+ * For Stream-Stream joins, all three types (inner, outer, left) handle out-of-order records correctly.
+ * For Stream-Table joins, if not using versioned stores, then out-of-order records are not handled (i.e., Streams applications don't check for out-of-order records and just process all records in offset order), and hence it may produce unpredictable results. With versioned stores, stream-side out-of-order data will be properly handled by performing a timestamp-based lookup in the table. Table-side out-of-order data is still not handled.
+ * For Table-Table joins, if not using versioned stores, then out-of-order records are not handled (i.e., Streams applications don't check for out-of-order records and just process all records in offset order). However, the join result is a changelog stream and hence will be eventually consistent. With versioned stores, table-table join semantics change from offset-based semantics to [timestamp-based semantics](/37/streams/developer-guide/dsl-api.html#versioned-state-stores) and out-of-order records are handled accordingly.
+
+
+
+ * [Documentation](/documentation)
+ * [Kafka Streams](/streams)
+
+
diff --git a/docs/streams/developer-guide/_index.md b/docs/streams/developer-guide/_index.md
new file mode 100644
index 0000000000000..8879fea791903
--- /dev/null
+++ b/docs/streams/developer-guide/_index.md
@@ -0,0 +1,10 @@
+---
+title: Streams Developer Guide
+description:
+weight: 10
+tags: ['kafka', 'docs', 'streams', 'developer-guide']
+aliases:
+keywords:
+type: docs
+---
+
diff --git a/docs/streams/developer-guide/app-reset-tool.html b/docs/streams/developer-guide/app-reset-tool.html
deleted file mode 100644
index a877b54f36e18..0000000000000
--- a/docs/streams/developer-guide/app-reset-tool.html
+++ /dev/null
@@ -1,203 +0,0 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/docs/streams/developer-guide/app-reset-tool.md b/docs/streams/developer-guide/app-reset-tool.md
new file mode 100644
index 0000000000000..bff72ca9e2207
--- /dev/null
+++ b/docs/streams/developer-guide/app-reset-tool.md
@@ -0,0 +1,135 @@
+---
+title: Application Reset Tool
+description:
+weight: 13
+tags: ['kafka', 'docs']
+aliases:
+keywords:
+type: docs
+---
+
+You can reset an application and force it to reprocess its data from scratch by using the application reset tool. This can be useful for development and testing, or when fixing bugs.
+
+The application reset tool handles the Kafka Streams [user topics](manage-topics.html#streams-developer-guide-topics-user) (input, output, and intermediate topics) and [internal topics](manage-topics.html#streams-developer-guide-topics-internal) differently when resetting the application.
+
+Here's what the application reset tool does for each topic type:
+
+ * Input topics: Reset offsets to specified position (by default to the beginning of the topic).
+ * Intermediate topics: Skip to the end of the topic, i.e., set the application's committed consumer offsets for all partitions to each partition's `logSize` (for consumer group `application.id`).
+ * Internal topics: Delete the internal topic (this automatically deletes any committed offsets).
+
+
+
+The application reset tool does not:
+
+ * Reset output topics of an application. If any output (or intermediate) topics are consumed by downstream applications, it is your responsibility to adjust those downstream applications as appropriate when you reset the upstream application.
+ * Reset the local environment of your application instances. It is your responsibility to delete the local state on any machine on which an application instance was run. See the instructions in section Step 2: Reset the local environments of your application instances on how to do this.
+
+
+
+Prerequisites
+
+
+ * All instances of your application must be stopped. Otherwise, the application may enter an invalid state, crash, or produce incorrect results. You can verify whether the consumer group with ID `application.id` is still active by using `bin/kafka-consumer-groups`. When long session timeout has been configured, active members could take longer to get expired on the broker thus blocking the reset job to complete. Use the `--force` option could remove those left-over members immediately. Make sure to shut down all stream applications when this option is specified to avoid unexpected rebalances.
+
+ * Use this tool with care and double-check its parameters: If you provide wrong parameter values (e.g., typos in `application.id`) or specify parameters inconsistently (e.g., specify the wrong input topics for the application), this tool might invalidate the application's state or even impact other applications, consumer groups, or your Kafka topics.
+
+ * You should manually delete and re-create any intermediate topics before running the application reset tool. This will free up disk space in Kafka brokers.
+
+ * You should delete and recreate intermediate topics before running the application reset tool, unless the following applies:
+
+> * You have external downstream consumers for the application's intermediate topics.
+> * You are in a development environment where manually deleting and re-creating intermediate topics is unnecessary.
+
+
+
+
+# Step 1: Run the application reset tool
+
+Invoke the application reset tool from the command line
+
+Warning! This tool makes irreversible changes to your application. It is strongly recommended that you run this once with `--dry-run` to preview your changes before making them.
+
+
+
-
-
-
-
-
-
-
-
-
diff --git a/docs/streams/developer-guide/config-streams.md b/docs/streams/developer-guide/config-streams.md
new file mode 100644
index 0000000000000..ac6555992b9dc
--- /dev/null
+++ b/docs/streams/developer-guide/config-streams.md
@@ -0,0 +1,627 @@
+---
+title: Configuring a Streams Application
+description:
+weight: 2
+tags: ['kafka', 'docs']
+aliases:
+keywords:
+type: docs
+---
+
+Kafka and Kafka Streams configuration options must be configured before using Streams. You can configure Kafka Streams by specifying parameters in a `java.util.Properties` instance.
+
+ 1. Create a `java.util.Properties` instance.
+
+ 2. Set the parameters. For example:
+
+ import java.util.Properties;
+ import org.apache.kafka.streams.StreamsConfig;
+
+ Properties settings = new Properties();
+ // Set a few key parameters
+ settings.put(StreamsConfig.APPLICATION_ID_CONFIG, "my-first-streams-application");
+ settings.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-broker1:9092");
+ // Any further settings
+ settings.put(... , ...);
+
+
+
+
+# Configuration parameter reference
+
+This section contains the most common Streams configuration parameters. For a full reference, see the [Streams](/37/javadoc/org/apache/kafka/streams/StreamsConfig.html) Javadocs.
+
+ * Required configuration parameters
+ * application.id
+ * bootstrap.servers
+ * Recommended configuration parameters for resiliency
+ * acks
+ * replication.factor
+ * Optional configuration parameters
+ * acceptable.recovery.lag
+ * default.deserialization.exception.handler
+ * default.key.serde
+ * default.production.exception.handler
+ * default.timestamp.extractor
+ * default.value.serde
+ * default.windowed.key.serde.inner
+ * default.windowed.value.serde.inner
+ * max.task.idle.ms
+ * max.warmup.replicas
+ * num.standby.replicas
+ * num.stream.threads
+ * probing.rebalance.interval.ms
+ * processing.guarantee
+ * rack.aware.assignment.non_overlap_cost
+ * rack.aware.assignment.strategy
+ * rack.aware.assignment.tags
+ * rack.aware.assignment.traffic_cost
+ * replication.factor
+ * rocksdb.config.setter
+ * state.dir
+ * topology.optimization
+ * Kafka consumers and producer configuration parameters
+ * Naming
+ * Default Values
+ * enable.auto.commit
+
+
+
+## Required configuration parameters
+
+Here are the required Streams configuration parameters.
+
+Parameter Name | Importance | Description | Default Value
+---|---|---|---
+application.id | Required | An identifier for the stream processing application. Must be unique within the Kafka cluster. | None
+bootstrap.servers | Required | A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. | None
+
+### application.id
+
+> (Required) The application ID. Each stream processing application must have a unique ID. The same ID must be given to all instances of the application. It is recommended to use only alphanumeric characters, `.` (dot), `-` (hyphen), and `_` (underscore). Examples: `"hello_world"`, `"hello_world-v1.0.0"`
+>
+> This ID is used in the following places to isolate resources used by the application from others:
+>
+> * As the default Kafka consumer and producer `client.id` prefix
+> * As the Kafka consumer `group.id` for coordination
+> * As the name of the subdirectory in the state directory (cf. `state.dir`)
+> * As the prefix of internal Kafka topic names
+>
+
+>
+> Tip:
+> When an application is updated, the `application.id` should be changed unless you want to reuse the existing data in internal topics and state stores. For example, you could embed the version information within `application.id`, as `my-app-v1.0.0` and `my-app-v1.0.2`.
+
+### bootstrap.servers
+
+> (Required) The Kafka bootstrap servers. This is the same [setting](http://kafka.apache.org/documentation.html#producerconfigs) that is used by the underlying producer and consumer clients to connect to the Kafka cluster. Example: `"kafka-broker1:9092,kafka-broker2:9092"`.
+
+## Recommended configuration parameters for resiliency
+
+There are several Kafka and Kafka Streams configuration options that need to be configured explicitly for resiliency in face of broker failures:
+
+Parameter Name | Corresponding Client | Default value | Consider setting to
+---|---|---|---
+acks | Producer | `acks=1` | `acks=all`
+replication.factor (for broker version 2.3 or older)/td> | Streams | `-1` | `3`
+min.insync.replicas | Broker | `1` | `2`
+num.standby.replicas | Streams | `0` | `1`
+
+Increasing the replication factor to 3 ensures that the internal Kafka Streams topic can tolerate up to 2 broker failures. Changing the acks setting to "all" guarantees that a record will not be lost as long as one replica is alive. The tradeoff from moving to the default values to the recommended ones is that some performance and more storage space (3x with the replication factor of 3) are sacrificed for more resiliency.
+
+### acks
+
+> The number of acknowledgments that the leader must have received before considering a request complete. This controls the durability of records that are sent. The possible values are:
+>
+> * `acks=0` The producer does not wait for acknowledgment from the server and the record is immediately added to the socket buffer and considered sent. No guarantee can be made that the server has received the record in this case, and the `retries` configuration will not take effect (as the client won't generally know of any failures). The offset returned for each record will always be set to `-1`.
+> * `acks=1` The leader writes the record to its local log and responds without waiting for full acknowledgement from all followers. If the leader immediately fails after acknowledging the record, but before the followers have replicated it, then the record will be lost.
+> * `acks=all` The leader waits for the full set of in-sync replicas to acknowledge the record. This guarantees that the record will not be lost if there is at least one in-sync replica alive. This is the strongest available guarantee.
+>
+
+>
+> For more information, see the [Kafka Producer documentation](https://kafka.apache.org/#producerconfigs).
+
+### replication.factor
+
+> See the description here.
+
+### num.standby.replicas
+
+> See the description here.
+
+
+ Properties streamsSettings = new Properties();
+ // for broker version 2.3 or older
+ //streamsSettings.put(StreamsConfig.REPLICATION_FACTOR_CONFIG, 3);
+ streamsSettings.put(StreamsConfig.topicPrefix(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG), 2);
+ streamsSettings.put(StreamsConfig.producerPrefix(ProducerConfig.ACKS_CONFIG), "all");
+ streamsSettings.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
+
+## Optional configuration parameters
+
+Here are the optional [Streams](/37/javadoc/org/apache/kafka/streams/StreamsConfig.html) javadocs, sorted by level of importance:
+
+> * High: These parameters can have a significant impact on performance. Take care when deciding the values of these parameters.
+> * Medium: These parameters can have some impact on performance. Your specific environment will determine how much tuning effort should be focused on these parameters.
+> * Low: These parameters have a less general or less significant impact on performance.
+>
+
+
+Parameter Name | Importance | Description | Default Value
+---|---|---|---
+acceptable.recovery.lag | Medium | The maximum acceptable lag (number of offsets to catch up) for an instance to be considered caught-up and ready for the active task. | 10000
+application.server | Low | A host:port pair pointing to an embedded user defined endpoint that can be used for discovering the locations of state stores within a single Kafka Streams application. The value of this must be different for each instance of the application. | the empty string
+buffered.records.per.partition | Low | The maximum number of records to buffer per partition. | 1000
+cache.max.bytes.buffering | Medium | Maximum number of memory bytes to be used for record caches across all threads. | 10485760 bytes
+client.id | Medium | An ID string to pass to the server when making requests. (This setting is passed to the consumer/producer clients used internally by Kafka Streams.) | the empty string
+commit.interval.ms | Low | The frequency in milliseconds with which to save the position (offsets in source topics) of tasks. | 30000 milliseconds
+default.deserialization.exception.handler | Medium | Exception handling class that implements the `DeserializationExceptionHandler` interface. | `LogAndContinueExceptionHandler`
+default.key.serde | Medium | Default serializer/deserializer class for record keys, implements the `Serde` interface. Must be set by the user or all serdes must be passed in explicitly (see also default.value.serde). | `null`
+default.production.exception.handler | Medium | Exception handling class that implements the `ProductionExceptionHandler` interface. | `DefaultProductionExceptionHandler`
+default.timestamp.extractor | Medium | Timestamp extractor class that implements the `TimestampExtractor` interface. | See Timestamp Extractor
+default.value.serde | Medium | Default serializer/deserializer class for record values, implements the `Serde` interface. Must be set by the user or all serdes must be passed in explicitly (see also default.key.serde). | `null`
+default.windowed.key.serde.inner | Medium | Default serializer/deserializer for the inner class of windowed keys, implementing the `Serde` interface. | null
+default.windowed.value.serde.inner | Medium | Default serializer/deserializer for the inner class of windowed values, implementing the `Serde` interface. | null
+default.dsl.store | Low | [DEPRECATED] The default state store type used by DSL operators. Deprecated in favor of `dsl.store.suppliers.class` | `ROCKS_DB`
+dsl.store.suppliers.class | Low | Defines a default state store implementation to be used by any stateful DSL operator that has not explicitly configured the store implementation type. Must implement the `org.apache.kafka.streams.state.DslStoreSuppliers` interface. | `BuiltInDslStoreSuppliers.RocksDBDslStoreSuppliers`
+max.task.idle.ms | Medium | This config controls whether joins and merges may produce out-of-order results. The config value is the maximum amount of time in milliseconds a stream task will stay idle when it is fully caught up on some (but not all) input partitions to wait for producers to send additional records and avoid potential out-of-order record processing across multiple input streams. The default (zero) does not wait for producers to send more records, but it does wait to fetch data that is already present on the brokers. This default means that for records that are already present on the brokers, Streams will process them in timestamp order. Set to -1 to disable idling entirely and process any locally available data, even though doing so may produce out-of-order processing. | 0 milliseconds
+max.warmup.replicas | Medium | The maximum number of warmup replicas (extra standbys beyond the configured num.standbys) that can be assigned at once. | 2
+metric.reporters | Low | A list of classes to use as metrics reporters. | the empty list
+metrics.num.samples | Low | The number of samples maintained to compute metrics. | 2
+metrics.recording.level | Low | The highest recording level for metrics. | `INFO`
+metrics.sample.window.ms | Low | The window of time in milliseconds a metrics sample is computed over. | 30000 milliseconds (30 seconds)
+num.standby.replicas | High | The number of standby replicas for each task. | 0
+num.stream.threads | Medium | The number of threads to execute stream processing. | 1
+probing.rebalance.interval.ms | Low | The maximum time in milliseconds to wait before triggering a rebalance to probe for warmup replicas that have sufficiently caught up. | 600000 milliseconds (10 minutes)
+processing.guarantee | Medium | The processing mode. Can be either `"at_least_once"` (default) or `"exactly_once_v2"` (for EOS version 2, requires broker version 2.5+). Deprecated config options are `"exactly_once"` (for EOS version 1) and `"exactly_once_beta"` (for EOS version 2, requires broker version 2.5+). | See Processing Guarantee
+poll.ms | Low | The amount of time in milliseconds to block waiting for input. | 100 milliseconds
+rack.aware.assignment.tags | Medium | List of tag keys used to distribute standby replicas across Kafka Streams clients. When configured, Kafka Streams will make a best-effort to distribute the standby tasks over clients with different tag values. | the empty list
+replication.factor | Medium | The replication factor for changelog topics and repartition topics created by the application. The default of `-1` (meaning: use broker default replication factor) requires broker version 2.4 or newer. | -1
+retry.backoff.ms | Medium | The amount of time in milliseconds, before a request is retried. This applies if the `retries` parameter is configured to be greater than 0. | 100 milliseconds
+rocksdb.config.setter | Medium | The RocksDB configuration. |
+state.cleanup.delay.ms | Low | The amount of time in milliseconds to wait before deleting state when a partition has migrated. | 600000 milliseconds (10 minutes)
+state.dir | High | Directory location for state stores. | `/${java.io.tmpdir}/kafka-streams`
+task.timeout.ms | Medium | The maximum amount of time in milliseconds a task might stall due to internal errors and retries until an error is raised. For a timeout of `0 ms`, a task would raise an error for the first internal error. For any timeout larger than `0 ms`, a task will retry at least once before an error is raised. | 300000 milliseconds (5 minutes)
+topology.optimization | Medium | A configuration telling Kafka Streams if it should optimize the topology and what optimizations to apply. Acceptable values are: `StreamsConfig.NO_OPTIMIZATION` (`none`), `StreamsConfig.OPTIMIZE` (`all`) or a comma separated list of specific optimizations: (`StreamsConfig.REUSE_KTABLE_SOURCE_TOPICS` (`reuse.ktable.source.topics`), `StreamsConfig.MERGE_REPARTITION_TOPICS` (`merge.repartition.topics`)). | ` NO_OPTIMIZATION`
+upgrade.from | Medium | The version you are upgrading from during a rolling upgrade. | See Upgrade From
+windowstore.changelog.additional.retention.ms | Low | Added to a windows maintainMs to ensure data is not deleted from the log prematurely. Allows for clock drift. | 86400000 milliseconds (1 day)
+window.size.ms | Low | Sets window size for the deserializer in order to calculate window end times. | null
+
+### acceptable.recovery.lag
+
+> The maximum acceptable lag (total number of offsets to catch up from the changelog) for an instance to be considered caught-up and able to receive an active task. Streams will only assign stateful active tasks to instances whose state stores are within the acceptable recovery lag, if any exist, and assign warmup replicas to restore state in the background for instances that are not yet caught up. Should correspond to a recovery time of well under a minute for a given workload. Must be at least 0.
+>
+> Note: if you set this to `Long.MAX_VALUE` it effectively disables the warmup replicas and task high availability, allowing Streams to immediately produce a balanced assignment and migrate tasks to a new instance without first warming them up.
+
+### default.deserialization.exception.handler
+
+> The default deserialization exception handler allows you to manage record exceptions that fail to deserialize. This can be caused by corrupt data, incorrect serialization logic, or unhandled record types. The implemented exception handler needs to return a `FAIL` or `CONTINUE` depending on the record and the exception thrown. Returning `FAIL` will signal that Streams should shut down and `CONTINUE` will signal that Streams should ignore the issue and continue processing. The following library built-in exception handlers are available:
+>
+> * [LogAndContinueExceptionHandler](/37/javadoc/org/apache/kafka/streams/errors/LogAndContinueExceptionHandler.html): This handler logs the deserialization exception and then signals the processing pipeline to continue processing more records. This log-and-skip strategy allows Kafka Streams to make progress instead of failing if there are records that fail to deserialize.
+> * [LogAndFailExceptionHandler](/37/javadoc/org/apache/kafka/streams/errors/LogAndFailExceptionHandler.html). This handler logs the deserialization exception and then signals the processing pipeline to stop processing more records.
+>
+
+>
+> You can also provide your own customized exception handler besides the library provided ones to meet your needs. For example, you can choose to forward corrupt records into a quarantine topic (think: a "dead letter queue") for further processing. To do this, use the Producer API to write a corrupted record directly to the quarantine topic. To be more concrete, you can create a separate `KafkaProducer` object outside the Streams client, and pass in this object as well as the dead letter queue topic name into the `Properties` map, which then can be retrieved from the `configure` function call. The drawback of this approach is that "manual" writes are side effects that are invisible to the Kafka Streams runtime library, so they do not benefit from the end-to-end processing guarantees of the Streams API:
+>
+>
+> public class SendToDeadLetterQueueExceptionHandler implements DeserializationExceptionHandler {
+> KafkaProducer
-
-
-
-
-
-
-
-
-
diff --git a/docs/streams/developer-guide/datatypes.md b/docs/streams/developer-guide/datatypes.md
new file mode 100644
index 0000000000000..5c40e4d1d7b11
--- /dev/null
+++ b/docs/streams/developer-guide/datatypes.md
@@ -0,0 +1,127 @@
+---
+title: Data Types and Serialization
+description:
+weight: 6
+tags: ['kafka', 'docs']
+aliases:
+keywords:
+type: docs
+---
+
+Every Kafka Streams application must provide Serdes (Serializer/Deserializer) for the data types of record keys and record values (e.g. `java.lang.String`) to materialize the data when necessary. Operations that require such Serdes information include: `stream()`, `table()`, `to()`, `repartition()`, `groupByKey()`, `groupBy()`.
+
+You can provide Serdes by using either of these methods, but you must use at least one:
+
+ * By setting default Serdes in the `java.util.Properties` config instance.
+ * By specifying explicit Serdes when calling the appropriate API methods, thus overriding the defaults.
+
+
+
+
+
+
+# Configuring Serdes
+
+Serdes specified in the Streams configuration are used as the default in your Kafka Streams application. Because this config's default is null, you must either set a default Serde by using this configuration or pass in Serdes explicitly, as described below.
+
+
+ import org.apache.kafka.common.serialization.Serdes;
+ import org.apache.kafka.streams.StreamsConfig;
+
+ Properties settings = new Properties();
+ // Default serde for keys of data records (here: built-in serde for String type)
+ settings.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
+ // Default serde for values of data records (here: built-in serde for Long type)
+ settings.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass().getName());
+
+# Overriding default Serdes
+
+You can also specify Serdes explicitly by passing them to the appropriate API methods, which overrides the default serde settings:
+
+
+ import org.apache.kafka.common.serialization.Serde;
+ import org.apache.kafka.common.serialization.Serdes;
+
+ final Serde
-
-
-
-
-
-
-
-
-
diff --git a/docs/streams/developer-guide/dsl-api.md b/docs/streams/developer-guide/dsl-api.md
new file mode 100644
index 0000000000000..c24faa4d506de
--- /dev/null
+++ b/docs/streams/developer-guide/dsl-api.md
@@ -0,0 +1,2375 @@
+---
+title: Streams DSL
+description:
+weight: 3
+tags: ['kafka', 'docs']
+aliases:
+keywords:
+type: docs
+---
+
+The Kafka Streams DSL (Domain Specific Language) is built on top of the Streams Processor API. It is the recommended for most users, especially beginners. Most data processing operations can be expressed in just a few lines of DSL code.
+
+ * Aggregating
+ * Joining
+ * Join co-partitioning requirements
+ * KStream-KStream Join
+ * KTable-KTable Equi-Join
+ * KTable-KTable Foreign-Key Join
+ * KStream-KTable Join
+ * KStream-GlobalKTable Join
+ * Windowing
+ * Hopping time windows
+ * Tumbling time windows
+ * Sliding time windows
+ * Session Windows
+ * Window Final Results
+ * Applying processors and transformers (Processor API integration)
+ * Naming Operators in a Streams DSL application
+ * Controlling KTable update rate
+ * Using timestamp-based semantics for table processors
+ * Writing streams back to Kafka
+ * Testing a Streams application
+ * Kafka Streams DSL for Scala
+ * Sample Usage
+ * Implicit Serdes
+ * User-Defined Serdes
+
+
+
+# Overview
+
+In comparison to the [Processor API](processor-api.html#streams-developer-guide-processor-api), only the DSL supports:
+
+ * Built-in abstractions for [streams and tables](../core-concepts.html#streams_concepts_duality) in the form of KStream, KTable, and GlobalKTable. Having first-class support for streams and tables is crucial because, in practice, most use cases require not just either streams or databases/tables, but a combination of both. For example, if your use case is to create a customer 360-degree view that is updated in real-time, what your application will be doing is transforming many input _streams_ of customer-related events into an output _table_ that contains a continuously updated 360-degree view of your customers.
+ * Declarative, functional programming style with stateless transformations (e.g. `map` and `filter`) as well as stateful transformations such as aggregations (e.g. `count` and `reduce`), joins (e.g. `leftJoin`), and windowing (e.g. session windows).
+
+
+
+With the DSL, you can define [processor topologies](../core-concepts.html#streams_topology) (i.e., the logical processing plan) in your application. The steps to accomplish this are:
+
+ 1. Specify one or more input streams that are read from Kafka topics.
+ 2. Compose transformations on these streams.
+ 3. Write the resulting output streams back to Kafka topics, or expose the processing results of your application directly to other applications through [interactive queries](interactive-queries.html#streams-developer-guide-interactive-queries) (e.g., via a REST API).
+
+
+
+After the application is run, the defined processor topologies are continuously executed (i.e., the processing plan is put into action). A step-by-step guide for writing a stream processing application using the DSL is provided below.
+
+For a complete list of available API functionality, see also the [Streams](/37/javadoc/org/apache/kafka/streams/package-summary.html) API docs.
+
+### KStream
+
+Only the **Kafka Streams DSL** has the notion of a `KStream`.
+
+A **KStream** is an abstraction of a **record stream** , where each data record represents a self-contained datum in the unbounded data set. Using the table analogy, data records in a record stream are always interpreted as an "INSERT" \-- think: adding more entries to an append-only ledger -- because no record replaces an existing row with the same key. Examples are a credit card transaction, a page view event, or a server log entry.
+
+To illustrate, let's imagine the following two data records are being sent to the stream:
+
+("alice", 1) --> ("alice", 3)
+
+If your stream processing application were to sum the values per user, it would return `4` for `alice`. Why? Because the second data record would not be considered an update of the previous record. Compare this behavior of KStream to `KTable` below, which would return `3` for `alice`.
+
+### KTable
+
+Only the **Kafka Streams DSL** has the notion of a `KTable`.
+
+A **KTable** is an abstraction of a **changelog stream** , where each data record represents an update. More precisely, the value in a data record is interpreted as an "UPDATE" of the last value for the same record key, if any (if a corresponding key doesn't exist yet, the update will be considered an INSERT). Using the table analogy, a data record in a changelog stream is interpreted as an UPSERT aka INSERT/UPDATE because any existing row with the same key is overwritten. Also, `null` values are interpreted in a special way: a record with a `null` value represents a "DELETE" or tombstone for the record's key.
+
+To illustrate, let's imagine the following two data records are being sent to the stream:
+
+("alice", 1) --> ("alice", 3)
+
+If your stream processing application were to sum the values per user, it would return `3` for `alice`. Why? Because the second data record would be considered an update of the previous record.
+
+**Effects of Kafka's log compaction:** Another way of thinking about KStream and KTable is as follows: If you were to store a KTable into a Kafka topic, you'd probably want to enable Kafka's [log compaction](http://kafka.apache.org/documentation.html#compaction) feature, e.g. to save storage space.
+
+However, it would not be safe to enable log compaction in the case of a KStream because, as soon as log compaction would begin purging older data records of the same key, it would break the semantics of the data. To pick up the illustration example again, you'd suddenly get a `3` for `alice` instead of a `4` because log compaction would have removed the `("alice", 1)` data record. Hence log compaction is perfectly safe for a KTable (changelog stream) but it is a mistake for a KStream (record stream).
+
+We have already seen an example of a changelog stream in the section [streams and tables](../core-concepts.html#streams_concepts_duality). Another example are change data capture (CDC) records in the changelog of a relational database, representing which row in a database table was inserted, updated, or deleted.
+
+KTable also provides an ability to look up _current_ values of data records by keys. This table-lookup functionality is available through **join operations** (see also **Joining** in the Developer Guide) as well as through **Interactive Queries**.
+
+### GlobalKTable
+
+Only the **Kafka Streams DSL** has the notion of a **GlobalKTable**.
+
+Like a **KTable** , a **GlobalKTable** is an abstraction of a **changelog stream** , where each data record represents an update.
+
+A GlobalKTable differs from a KTable in the data that they are being populated with, i.e. which data from the underlying Kafka topic is being read into the respective table. Slightly simplified, imagine you have an input topic with 5 partitions. In your application, you want to read this topic into a table. Also, you want to run your application across 5 application instances for **maximum parallelism**.
+
+ * If you read the input topic into a **KTable** , then the "local" KTable instance of each application instance will be populated with data **from only 1 partition** of the topic's 5 partitions.
+ * If you read the input topic into a **GlobalKTable** , then the local GlobalKTable instance of each application instance will be populated with data **from all partitions of the topic**.
+
+
+
+GlobalKTable provides the ability to look up _current_ values of data records by keys. This table-lookup functionality is available through `join operations`. Note that a GlobalKTable has **no** notion of time in contrast to a KTable.
+
+Benefits of global tables:
+
+ * More convenient and/or efficient **joins** : Notably, global tables allow you to perform star joins, they support "foreign-key" lookups (i.e., you can lookup data in the table not just by record key, but also by data in the record values), and they are more efficient when chaining multiple joins. Also, when joining against a global table, the input data does not need to be **co-partitioned**.
+ * Can be used to "broadcast" information to all the running instances of your application.
+
+
+
+Downsides of global tables:
+
+ * Increased local storage consumption compared to the (partitioned) KTable because the entire topic is tracked.
+ * Increased network and Kafka broker load compared to the (partitioned) KTable because the entire topic is read.
+
+
+
+# Creating source streams from Kafka
+
+You can easily read data from Kafka topics into your application. The following operations are supported.
+
+Reading from Kafka | Description
+---|---
+**Stream**
+
+ * _input topics_ -> KStream
+
+| Creates a KStream from the specified Kafka input topics and interprets the data as a record stream. A `KStream` represents a _partitioned_ record stream. [(details)](/37/javadoc/org/apache/kafka/streams/StreamsBuilder.html#stream\(java.lang.String\)) In the case of a KStream, the local KStream instance of every application instance will be populated with data from only **a subset** of the partitions of the input topic. Collectively, across all application instances, all input topic partitions are read and processed.
+
+
+ import org.apache.kafka.common.serialization.Serdes;
+ import org.apache.kafka.streams.StreamsBuilder;
+ import org.apache.kafka.streams.kstream.KStream;
+
+ StreamsBuilder builder = new StreamsBuilder();
+
+ KStream
-
-
-
-
-