generated from amazon-archives/__template_Custom
-
Notifications
You must be signed in to change notification settings - Fork 310
Open
Description
Describe the bug
AvroInputCodec throws an exception when fields of type "fixed" appear in an Avro file. These contain a fixed length byte array, which I would expect to be ingested by Data Prepper like the variable length "bytes" fields are now.
I believe "bytes" fields are only implicitly handled now because the output is parsed by Jackson which can read either byte[] or ByteBuffer.
To Reproduce
Steps to reproduce the behavior:
- Create an Avro file with the a "fixed" field type in the schema.
e.g. Avro schema
{
"type": "record",
"name": "TestRecord",
"fields": [
{
"name": "fixedField",
"type": {
"type": "fixed",
"name": "FixedType",
"size": 16
}
}
]
}e.g. Python script to create a test Avro file with one record
import fastavro
import io
schema = {
"type": "record",
"name": "TestRecord",
"fields": [
{
"name": "fixedField",
"type": {
"type": "fixed",
"name": "FixedType",
"size": 16
}
}
]
}
records = [
{"fixedField": b"\x00" * 16}
]
parsed_schema = fastavro.parse_schema(schema)
with open("test_fixed_field.avro", "wb") as f:
fastavro.writer(f, parsed_schema, records)
print("Wrote test_fixed_field.avro")- Attempt to ingest this file with Data Prepper
test-avro-fixed-field-pipeline:
source:
file:
path: "test_fixed_field.avro"
codec:
avro: {}
sink:
- stdout:
codec:
json:
pretty_print: true
Expected behavior
I would expect Data Prepper to read this valid Avro file and convert the "fixed" length field to a byte array, like it does for a variable length "bytes" field.
Screenshots
data-prepper-ingest | 2026-03-03T20:55:35,494 [scanning-fingerprint-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.file.FileSource - Starting file source with /usr/share/data-prepper/data/test_fixed.avro path.
data-prepper-ingest | 2026-03-03T20:55:35,858 [file-source] ERROR org.opensearch.dataprepper.plugins.codec.avro.AvroInputCodec - An exception has occurred while parsing avro InputStream
data-prepper-ingest | java.lang.IllegalArgumentException: Not an enum: {"type":"fixed","name":"FixedType","size":16} (through reference chain: java.util.HashMap["fixedField"]->org.apache.avro.generic.GenericData$Fixed["schema"]->org.apache.avro.Schema$FixedSchema["enumSymbols"])
data-prepper-ingest | at com.fasterxml.jackson.databind.ObjectMapper.valueToTree(ObjectMapper.java:3633)
data-prepper-ingest | at org.opensearch.dataprepper.model.event.JacksonEvent.getInitialJsonNode(JacksonEvent.java:149)
data-prepper-ingest | at org.opensearch.dataprepper.model.event.JacksonEvent.<init>(JacksonEvent.java:114)
data-prepper-ingest | at org.opensearch.dataprepper.model.log.JacksonLog.<init>(JacksonLog.java:23)
data-prepper-ingest | at org.opensearch.dataprepper.model.log.JacksonLog$Builder.build(JacksonLog.java:55)
data-prepper-ingest | at org.opensearch.dataprepper.model.log.JacksonLog$Builder.build(JacksonLog.java:41)
data-prepper-ingest | at org.opensearch.dataprepper.event.DefaultLogEventBuilderFactory$DefaultLogEventBuilder.build(DefaultLogEventBuilderFactory.java:39)
data-prepper-ingest | at org.opensearch.dataprepper.event.DefaultLogEventBuilderFactory$DefaultLogEventBuilder.build(DefaultLogEventBuilderFactory.java:25)
data-prepper-ingest | at org.opensearch.dataprepper.plugins.codec.avro.AvroInputCodec.parseAvroStream(AvroInputCodec.java:74)
data-prepper-ingest | at org.opensearch.dataprepper.plugins.codec.avro.AvroInputCodec.parse(AvroInputCodec.java:56)
data-prepper-ingest | at org.opensearch.dataprepper.plugins.source.file.FileSource$CodecFileStrategy.start(FileSource.java:182)
data-prepper-ingest | at org.opensearch.dataprepper.plugins.source.file.FileSource.lambda$start$0(FileSource.java:88)
data-prepper-ingest | at java.base/java.lang.Thread.run(Thread.java:840)
data-prepper-ingest | Caused by: com.fasterxml.jackson.databind.JsonMappingException: Not an enum: {"type":"fixed","name":"FixedType","size":16} (through reference chain: java.util.HashMap["fixedField"]->org.apache.avro.generic.GenericData$Fixed["schema"]->org.apache.avro.Schema$FixedSchema["enumSymbols"])
data-prepper-ingest | at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:400)
data-prepper-ingest | at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:359)
data-prepper-ingest | at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:324)
data-prepper-ingest | at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:765)
data-prepper-ingest | at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:183)
data-prepper-ingest | at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:732)
data-prepper-ingest | at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:760)
data-prepper-ingest | at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:183)
data-prepper-ingest | at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:807)
data-prepper-ingest | at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeWithoutTypeInfo(MapSerializer.java:763)
data-prepper-ingest | at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:719)
data-prepper-ingest | at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:34)
data-prepper-ingest | at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:503)
data-prepper-ingest | at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:342)
data-prepper-ingest | at com.fasterxml.jackson.databind.ObjectMapper.valueToTree(ObjectMapper.java:3628)
data-prepper-ingest | ... 12 more
data-prepper-ingest | Caused by: org.apache.avro.AvroRuntimeException: Not an enum: {"type":"fixed","name":"FixedType","size":16}
data-prepper-ingest | at org.apache.avro.Schema.getEnumSymbols(Schema.java:303)
data-prepper-ingest | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
data-prepper-ingest | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
data-prepper-ingest | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
data-prepper-ingest | at java.base/java.lang.reflect.Method.invoke(Method.java:569)
data-prepper-ingest | at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:688)
data-prepper-ingest | at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:760)
data-prepper-ingest | ... 23 more
Environment (please complete the following information):
- OS: Official Data Prepper Docker Hub linux/amd64 image
- Version [2.14.0]
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Type
Projects
Status
In review