Skip to content

fix(nlp): collapse runaway phrase repetition in automatic transcripts DEV-2733 - #7500

Draft
platreth wants to merge 1 commit into
mainfrom
hugo/dev-2733-asr-repetition-collapse
Draft

fix(nlp): collapse runaway phrase repetition in automatic transcripts DEV-2733#7500
platreth wants to merge 1 commit into
mainfrom
hugo/dev-2733-asr-repetition-collapse

Conversation

@platreth

Copy link
Copy Markdown
Contributor

📣 Summary

Automatic transcripts no longer fill up with the same short phrase repeated hundreds of times.

📖 Description

When automatic transcription got stuck on a short phrase (reported with Swahili audio mixed with French), the transcript came back as pages of the same two words repeated until the end of the recording. Those runaway repeats are now trimmed and the rest of the transcript is kept as is.

💭 Notes

  • Google STT v2 (chirp_3) can fall into a decoder loop on low-resource languages: the ticket's audio yields na niko, about 300 times, even with the sw:auto override from DEV-2403 active, so remapping the language code doesn't prevent it.
  • The fix clamps consecutive repeats of any phrase of 1 to 5 words in _read_batch_result(), the single choke point for both completion paths. Comparison ignores case and punctuation drift; a transcript with nothing to trim comes back byte-identical.
  • The threshold is a new Constance key, ASR_MAX_CONSECUTIVE_REPEATS (default 5, 0 disables), so it can be tuned without a deploy, same as ASR_LANGUAGE_CODE_OVERRIDES.
  • When trimming happens we log a warning with the before/after character counts, never the transcript text.

👀 Preview steps

Backend only, reproducible with the bundled tests (in the kpi container):

  1. pytest kobo/apps/subsequences/tests/test_repetition.py kobo/apps/subsequences/integrations/tests/test_google_transcribe.py -q

🔴 On main, a batch result whose transcript is <valid text> na niko, na niko, … (×300) is stored verbatim: 2,739 characters of loop.
🟢 On this PR, the same payload is stored as the valid text plus 5 repeats (84 characters), and a warning with the sizes is logged.

@platreth platreth self-assigned this Aug 28, 2026
@platreth
platreth requested a review from noliveleger August 28, 2026 10:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant