Skip to content

[BUG] AvroInputCodec exception on fixed type Avro fields #6602

@p1ck

Description

@p1ck

Describe the bug
AvroInputCodec throws an exception when fields of type "fixed" appear in an Avro file. These contain a fixed length byte array, which I would expect to be ingested by Data Prepper like the variable length "bytes" fields are now.

I believe "bytes" fields are only implicitly handled now because the output is parsed by Jackson which can read either byte[] or ByteBuffer.

To Reproduce
Steps to reproduce the behavior:

  1. Create an Avro file with the a "fixed" field type in the schema.

e.g. Avro schema

{
  "type": "record",
  "name": "TestRecord",
  "fields": [
    {
      "name": "fixedField",
      "type": {
        "type": "fixed",
        "name": "FixedType",
        "size": 16
      }
    }
  ]
}

e.g. Python script to create a test Avro file with one record

import fastavro
import io

schema = {
    "type": "record",
    "name": "TestRecord",
    "fields": [
        {
            "name": "fixedField",
            "type": {
                "type": "fixed",
                "name": "FixedType",
                "size": 16
            }
        }
    ]
}

records = [
    {"fixedField": b"\x00" * 16}
]

parsed_schema = fastavro.parse_schema(schema)

with open("test_fixed_field.avro", "wb") as f:
    fastavro.writer(f, parsed_schema, records)

print("Wrote test_fixed_field.avro")
  1. Attempt to ingest this file with Data Prepper
test-avro-fixed-field-pipeline:
  source:
    file:
      path: "test_fixed_field.avro"
      codec:
        avro: {}
  sink:
    - stdout:
        codec:
          json:
            pretty_print: true

Expected behavior
I would expect Data Prepper to read this valid Avro file and convert the "fixed" length field to a byte array, like it does for a variable length "bytes" field.

Screenshots

data-prepper-ingest  | 2026-03-03T20:55:35,494 [scanning-fingerprint-pipeline-sink-worker-2-thread-1] INFO  org.opensearch.dataprepper.plugins.source.file.FileSource - Starting file source with /usr/share/data-prepper/data/test_fixed.avro path.
data-prepper-ingest  | 2026-03-03T20:55:35,858 [file-source] ERROR org.opensearch.dataprepper.plugins.codec.avro.AvroInputCodec - An exception has occurred while parsing avro InputStream 
data-prepper-ingest  | java.lang.IllegalArgumentException: Not an enum: {"type":"fixed","name":"FixedType","size":16} (through reference chain: java.util.HashMap["fixedField"]->org.apache.avro.generic.GenericData$Fixed["schema"]->org.apache.avro.Schema$FixedSchema["enumSymbols"])
data-prepper-ingest  |  at com.fasterxml.jackson.databind.ObjectMapper.valueToTree(ObjectMapper.java:3633)
data-prepper-ingest  |  at org.opensearch.dataprepper.model.event.JacksonEvent.getInitialJsonNode(JacksonEvent.java:149)
data-prepper-ingest  |  at org.opensearch.dataprepper.model.event.JacksonEvent.<init>(JacksonEvent.java:114)
data-prepper-ingest  |  at org.opensearch.dataprepper.model.log.JacksonLog.<init>(JacksonLog.java:23)
data-prepper-ingest  |  at org.opensearch.dataprepper.model.log.JacksonLog$Builder.build(JacksonLog.java:55)
data-prepper-ingest  |  at org.opensearch.dataprepper.model.log.JacksonLog$Builder.build(JacksonLog.java:41)
data-prepper-ingest  |  at org.opensearch.dataprepper.event.DefaultLogEventBuilderFactory$DefaultLogEventBuilder.build(DefaultLogEventBuilderFactory.java:39)
data-prepper-ingest  |  at org.opensearch.dataprepper.event.DefaultLogEventBuilderFactory$DefaultLogEventBuilder.build(DefaultLogEventBuilderFactory.java:25)
data-prepper-ingest  |  at org.opensearch.dataprepper.plugins.codec.avro.AvroInputCodec.parseAvroStream(AvroInputCodec.java:74)
data-prepper-ingest  |  at org.opensearch.dataprepper.plugins.codec.avro.AvroInputCodec.parse(AvroInputCodec.java:56)
data-prepper-ingest  |  at org.opensearch.dataprepper.plugins.source.file.FileSource$CodecFileStrategy.start(FileSource.java:182)
data-prepper-ingest  |  at org.opensearch.dataprepper.plugins.source.file.FileSource.lambda$start$0(FileSource.java:88)
data-prepper-ingest  |  at java.base/java.lang.Thread.run(Thread.java:840)
data-prepper-ingest  | Caused by: com.fasterxml.jackson.databind.JsonMappingException: Not an enum: {"type":"fixed","name":"FixedType","size":16} (through reference chain: java.util.HashMap["fixedField"]->org.apache.avro.generic.GenericData$Fixed["schema"]->org.apache.avro.Schema$FixedSchema["enumSymbols"])
data-prepper-ingest  |  at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:400)
data-prepper-ingest  |  at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:359)
data-prepper-ingest  |  at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:324)
data-prepper-ingest  |  at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:765)
data-prepper-ingest  |  at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:183)
data-prepper-ingest  |  at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:732)
data-prepper-ingest  |  at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:760)
data-prepper-ingest  |  at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:183)
data-prepper-ingest  |  at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:807)
data-prepper-ingest  |  at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeWithoutTypeInfo(MapSerializer.java:763)
data-prepper-ingest  |  at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:719)
data-prepper-ingest  |  at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:34)
data-prepper-ingest  |  at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:503)
data-prepper-ingest  |  at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:342)
data-prepper-ingest  |  at com.fasterxml.jackson.databind.ObjectMapper.valueToTree(ObjectMapper.java:3628)
data-prepper-ingest  |  ... 12 more
data-prepper-ingest  | Caused by: org.apache.avro.AvroRuntimeException: Not an enum: {"type":"fixed","name":"FixedType","size":16}
data-prepper-ingest  |  at org.apache.avro.Schema.getEnumSymbols(Schema.java:303)
data-prepper-ingest  |  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
data-prepper-ingest  |  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
data-prepper-ingest  |  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
data-prepper-ingest  |  at java.base/java.lang.reflect.Method.invoke(Method.java:569)
data-prepper-ingest  |  at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:688)
data-prepper-ingest  |  at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:760)
data-prepper-ingest  |  ... 23 more

Environment (please complete the following information):

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status

In review

Relationships

None yet

Development

No branches or pull requests

Issue actions