|
| 1 | += spring-batch-s3 |
| 2 | +:toc: |
| 3 | +:icons: font |
| 4 | +:source-highlighter: highlightjs |
| 5 | + |
| 6 | +https://spring.io/projects/spring-batch[Spring Batch] extension for https://aws.amazon.com/it/s3/[Amazon S3] which contains `S3ItemReader` and `S3ItemWriter` implementations for reading from and writing to S3 buckets, including support for multipart uploads. Other S3-compatible technologies (like as Google Cloud Storage, MinIO, etc.) may be supported, but they are not guaranteed to work. |
| 7 | + |
| 8 | +*Note*: these writers are based on the *AWS SDK V2*. |
| 9 | + |
| 10 | +== Installation |
| 11 | + |
| 12 | +To use the `spring-batch-s3` extension, you need to add the following dependency to your Maven or Gradle project: |
| 13 | + |
| 14 | +=== Maven |
| 15 | + |
| 16 | +[source,xml] |
| 17 | +---- |
| 18 | +<dependency> |
| 19 | + <groupId>org.springframework.batch.extensions</groupId> |
| 20 | + <artifactId>spring-batch-s3</artifactId> |
| 21 | + <version>${spring-batch-extensions.version}</version> |
| 22 | +</dependency> |
| 23 | +---- |
| 24 | + |
| 25 | +=== Gradle |
| 26 | + |
| 27 | +[source,groovy] |
| 28 | +---- |
| 29 | +implementation 'org.springframework.batch.extensions:spring-batch-s3:${springBatchExtensionsVersion}' |
| 30 | +---- |
| 31 | + |
| 32 | +== Known limitations |
| 33 | + |
| 34 | +* The `S3ItemReader` and `S3ItemWriter` are designed to work with the synchronous AWS S3 client (`S3Client`). They do not support the asynchronous client (`S3AsyncClient`) at this time. |
| 35 | + |
| 36 | +== Pre-requisites |
| 37 | + |
| 38 | +In order to set up these components you need to provide some additional beans in your Spring Batch configuration: |
| 39 | + |
| 40 | +* An `S3Client` bean to interact with AWS S3. |
| 41 | +* In case you want to use the `S3ItemReader`: an instance of `S3Deserializer` for the data you want to read. |
| 42 | +* In case you want to use the `S3ItemWriter`: an instance of `S3Serializer` for the data you want to write. |
| 43 | + |
| 44 | +There are two examples of implementation for both `S3Serializer` and `S3Deserializer` provided in this project: |
| 45 | + |
| 46 | +* `S3StringSerializer`: take a `String` as input and writes it to S3 as a UTF-8 encoded byte array. The write functions add a line termination character at the end of each string. |
| 47 | +* `S3StringDeserializer`: takes a UTF-8 encoded byte array from S3 and converts it to a `String`. The implementation of this deserializer is *stateful* because lines may arrive in different chunks. |
| 48 | + |
| 49 | +More details in the JavaDocs of the classes. |
| 50 | + |
| 51 | +=== Configuration of the `S3Client` |
| 52 | + |
| 53 | +To use the `S3ItemReader` and `S3ItemWriter`, you need to configure the AWS S3 client by providing a `S3Client`. Checkout the https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/http-configuration-apache.html[AWS SDK for Java] documentation for more details on how to configure this. |
| 54 | + |
| 55 | +=== Configure `S3Serializer` |
| 56 | + |
| 57 | +`S3StringSerializer` is a simple implementation of `S3Serializer` that takes a `String` as input and writes it to S3 as a UTF-8 encoded byte array. You are encouraged to implement your own serializer if you need to handle different data types or formats. |
| 58 | + |
| 59 | +==== Java Config |
| 60 | + |
| 61 | +[source,java] |
| 62 | +---- |
| 63 | +@Bean |
| 64 | +S3Serializer<String> s3Serializer() { |
| 65 | + return new S3StringSerializer(); |
| 66 | +} |
| 67 | +---- |
| 68 | + |
| 69 | +=== Configure `S3Deserializer` |
| 70 | + |
| 71 | +Similarly, `S3StringDeserializer` is a simple implementation of `S3Deserializer` that takes a UTF-8 encoded byte array from S3 and converts it to a `String`. You can implement your own deserializer if you need to handle different data types or formats. |
| 72 | + |
| 73 | +In case you don't want to implement your serializer checkout the "Alternatives readers" section below. |
| 74 | + |
| 75 | +==== Java Config |
| 76 | + |
| 77 | +[source,java] |
| 78 | +---- |
| 79 | +@Bean |
| 80 | +S3Deserializer<String> s3Deserializer() { |
| 81 | + return new S3StringDeserializer(); |
| 82 | +} |
| 83 | +---- |
| 84 | + |
| 85 | +== Configuration of `S3ItemReader` |
| 86 | + |
| 87 | +Given the `S3Client` and `S3Deserializer` beans, you can now configure the `S3ItemReader`. |
| 88 | + |
| 89 | +=== Java Config |
| 90 | + |
| 91 | +To configure the `S3ItemReader`, you need to set up the AWS S3 client and specify the bucket and object key from which you want to read data. |
| 92 | +[source,java] |
| 93 | +---- |
| 94 | +@Bean |
| 95 | +ItemReader<String> downloadItemReader() throws Exception { |
| 96 | + return new S3ItemReader.Builder<String>() |
| 97 | + .s3Client(s3Client()) |
| 98 | + .bucketName("bucket_name") |
| 99 | + .objectKey("object_key") |
| 100 | + .deserializer(s3Deserializer()) |
| 101 | + .bufferSize((int) DataSize.ofMegabytes(1).toBytes()) // Default 128 Bytes |
| 102 | + .build(); |
| 103 | +} |
| 104 | +---- |
| 105 | + |
| 106 | +There is also an additional option to set the `bufferSize` which is the size of the buffer used to read data from S3. The default value is 128 bytes, but you can increase it to improve memory consumption The bast value for this parameter is the average length of the lines in your file. |
| 107 | + |
| 108 | +=== Alternative reader |
| 109 | + |
| 110 | +Instead of `S3ItemReader` you can also use `FlatFileItemReader` with `InputStreamResources` to read files from S3 as well. |
| 111 | +To do so this package exposes a `S3InputStream` that can be used for that purpose. Below an example: |
| 112 | + |
| 113 | +[source,java] |
| 114 | +---- |
| 115 | +@Bean |
| 116 | +ItemReader<String> itemReader() throws Exception { |
| 117 | + final var inputStreamResource = new InputStreamResource( |
| 118 | + new S3InputStream(s3Client(), |
| 119 | + "bucket_name", |
| 120 | + "object_key")); |
| 121 | +
|
| 122 | + return new FlatFileItemReaderBuilder<String>() |
| 123 | + .name("itemReader") |
| 124 | + .resource(inputStreamResource) |
| 125 | + .lineMapper(new PassThroughLineMapper( )) |
| 126 | + .build(); |
| 127 | +} |
| 128 | +---- |
| 129 | + |
| 130 | +== Configuration of `S3ItemWriter` |
| 131 | + |
| 132 | +Given the `S3Client` and `S3Serializer` beans, you can now configure the `S3ItemWriter`. |
| 133 | + |
| 134 | +=== Java Config |
| 135 | + |
| 136 | +To configure the `S3ItemWriter`, you need to set up the AWS S3 client and specify the bucket and object key to which you want to write data. |
| 137 | +[source,java] |
| 138 | +---- |
| 139 | +@Bean |
| 140 | +ItemWriter<String> uploadItemWriter() throws IOException { |
| 141 | + return new S3ItemWriter.Builder<String>() |
| 142 | + .s3Client(s3Client()) |
| 143 | + .bucketName("bucket_name") |
| 144 | + .objectKey("object_key") |
| 145 | + .multipartUpload(true) // Default is false |
| 146 | + .partSize((int) DataSize.ofMegabytes(10).toBytes()) // Default is 5 MB |
| 147 | + .contentType("text/csv") // Default is application/octet-stream |
| 148 | + .serializer(s3Serializer()) |
| 149 | + .build(); |
| 150 | +} |
| 151 | +---- |
| 152 | + |
| 153 | +There are several additional options you can set for the `S3ItemWriter`: |
| 154 | + |
| 155 | +* `multipartUpload`: If set to `true`, the writer will use multipart upload for large files. The default is `false`. |
| 156 | +* `partSize`: The size of each part in a multipart upload. The default is 5 MB. |
| 157 | +* `contentType`: The content type of the uploaded file. The default is `application/octet-stream`. |
0 commit comments