Skip to content

How to stop RowFileInputReader after matching a specific line pattern (Text Header + Large Binary Tail)#746

Description

@I-am-a-monster

Hi everyone,

I am using kafka-connect-file-pulse to process files that have a specific hybrid structure:

  1. A multi-line text header
  2. An empty line or custom characters line as a line separator
  3. A large amount of binary data.

My Goal: I want to ingest the entire header as a single Kafka record and then immediately stop reading the file to avoid the overhead of processing the large binary tail.

Possible Configuration:
I am thinking of using a MultiRowFilter to group the header and a DropFilter to discard the binary records,
While the DropFilter prevents the binary data from being sent to Kafka, the RowFileInputReader still continues to read the entire file line-by-line which isn't convenient in case of large data.

So I would like to know :

  • Is there a way to signal the reader to stop/close the file stream as soon as a MultiRowFilter finishes or a specific condition is met?
  • Is there a built-in OffsetPolicy or a specific Reader configuration that can limit the read-range based on a pattern rather than a fixed byte-count?
  • If this isn't supported out-of-the-box, would a custom Filter be able to trigger a file-close, or is that handled strictly at the Reader level?

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions