[BUG] Invalid document version from events is still included in opensearch bulk requests

[graytaylor0]

Describe the bug
Using external versioning with document_version and document_version_type in the opensearch sink with invalid version fields sends the event to DLQ, but then also includes it in the bulk request.

Expected behavior
Send the event to DLQ, then do not add it to the bulk request:

data-prepper/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/OpenSearchSink.java

Line 484 in 5fec579

logFailureForDlqObjects(List.of(createDlqObjectFromEvent(event, indexName, errorMessage)), e);

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: [e.g. Ubuntu 20.04 LTS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[dlvenable]

It looks like we do a pre-validation before the bulk request. Then it still goes into the bulk request.

I'm going to guess it ends up in the DLQ twice. @graytaylor0 , Is that the case?

[graytaylor0]

@dlvenable Yes it goes into the DLQ twice, but the bigger issue is that opensearch had a bug that caused one invalid version document to fail the entire bulk request so the entire bulk request gets sent to DLQ