Skip to content

fix(kana): normalize full-width and invisible chars in romaji answer … - #26301

Open
Aadityaotaku wants to merge 3 commits into
lingdojo:mainfrom
Aadityaotaku:fix/kana-hitoro-input-24308
Open

fix(kana): normalize full-width and invisible chars in romaji answer …#26301
Aadityaotaku wants to merge 3 commits into
lingdojo:mainfrom
Aadityaotaku:fix/kana-hitoro-input-24308

Conversation

@Aadityaotaku

Copy link
Copy Markdown
Contributor

📝 Description

Fixes a bug where typing correct romaji (e.g. hitoro for ひとろ) was rejected as wrong on mobile devices.

Root cause: Android and other mobile IMEs/keyboards sometimes produce full-width romaji characters (e.g. hitoro instead of hitoro) or inject invisible Unicode characters like zero-width space (U+200B) and BOM (U+FEFF) into the input. The previous NFC normalization does not convert full-width latin letters to their half-width ASCII equivalents, so the comparison always failed.

Changes in isKanaInputAnswerCorrect.ts:

  • Upgraded Unicode normalization from NFCNFKC so full-width characters (e.g. h) are mapped to ASCII equivalents before comparison
  • Added explicit stripping of invisible characters: zero-width space (U+200BU+200D) and BOM (U+FEFF) that some IMEs silently inject

Both the normal (kana → romaji) and reverse (romaji → kana) modes are fixed, as well as single-char and multi-char prompt paths.

🔗 Related Issue

Closes #24308

✅ Pre-Submission Checklist

  • I have starred the repo ⭐
  • My code follows the project's code style and uses cn() utility where needed
  • My commit messages follow the Conventional Commits format
  • This PR is against the main branch

🎯 Type of Change

  • fix: Bug fix (non-breaking change which fixes an issue)

🧪 How Has This Been Tested?

Test Steps:

  1. Open the Kana dojo in Input mode
  2. Encounter a multi-character prompt (e.g. ひとろ)
  3. Using a mobile keyboard or browser console, paste full-width romaji: hitoro
  4. Submit — should now be accepted as correct
  5. Also verified that standard half-width hitoro still works correctly

Manual Test Checklist:

  • Tested in Kana dojo (Input mode)
  • Normal mode (kana → type romaji)
  • Reverse mode (romaji → type kana)
  • Single-character prompts
  • Multi-character prompts

📸 Screenshots/Videos (if applicable)

Before: Typing hitoro (or full-width equivalent from mobile) for ひとろ was rejected with "Wrong! Correct answer: hitoro"

After: Both hitoro and hitoro are accepted as correct

📦 Additional Context

The fix is a one-line change to the normalization call — switching .normalize('NFC') to .replace(/[\u200B-\u200D\uFEFF]/g, '').normalize('NFKC') — applied consistently across all four comparison paths in the function.

…validation

Mobile keyboards (especially on Android) sometimes produce full-width
romaji characters (e.g. hitoro instead of hitoro) or inject
invisible Unicode characters (zero-width space U+200B, BOM U+FEFF).

The previous NFC normalization did not convert full-width latin letters
to their standard half-width equivalents, causing correct answers to be
rejected as wrong.

Fixes:
- Replace NFC with NFKC normalization so full-width characters are
  mapped to their ASCII equivalents before comparison
- Strip zero-width space (U+200B-U+200D) and BOM (U+FEFF) characters
  that some IMEs inject silently

Fixes lingdojo#24308
Copilot AI review requested due to automatic review settings July 28, 2026 05:52
@tentoumushii

Copy link
Copy Markdown
Collaborator

🎉 Thanks for your Pull Request, @Aadityaotaku!

We appreciate your contribution to KanaDojo!

Pre-merge checklist:

  • You starred our repo ⭐
  • Code follows project style guidelines
  • Changes have been tested locally
  • PR title is descriptive
  • If this closes an issue, it's linked with Closes #<number>

A maintainer will review your PR shortly. In the meantime, make sure all CI checks pass. You can run npm run check locally to match CI.

ありがとうございます! 🙏

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes incorrect answer rejection in Kana “Input” mode caused by mobile IMEs producing full-width romaji characters and/or injecting invisible Unicode characters into the user’s input. It updates the input/answer normalization logic in features/Kana/lib/ so comparisons behave consistently across devices.

Changes:

  • Switches Unicode normalization from NFC to NFKC for romaji/kana comparisons (enabling full-width → ASCII compatibility folding).
  • Strips invisible characters (U+200B–U+200D and U+FEFF) from the user input before comparison.
  • Applies the updated normalization consistently across reverse-mode, direct-match, and alternative-answer matching paths.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread features/Kana/lib/isKanaInputAnswerCorrect.ts Outdated
Comment thread features/Kana/lib/isKanaInputAnswerCorrect.ts Outdated
Copilot AI review requested due to automatic review settings July 30, 2026 13:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

features/Kana/lib/isKanaInputAnswerCorrect.ts:52

  • The comment above the normalization logic still describes NFC/NFD normalization, but the implementation now removes specific invisible Unicode characters and applies NFKC (compatibility) normalization. Updating the comment will prevent future readers from assuming NFC behavior (e.g., no full-width → ASCII mapping).
  // Normalize Unicode form and strip surrounding whitespace so that input
  // from an IME or copy-paste (which may arrive in a different NFC/NFD form)
  // is compared on equal footing with the stored answer.
  const normalizedInput = inputValue.trim().replace(/[\u200B-\u200D\uFEFF]/g, '').normalize('NFKC');

features/Kana/tests/isKanaInputAnswerCorrect.test.ts:262

  • This test’s comment says it is using full-width uppercase Latin letters (U+FF2B/U+FF21), but the actual input uses the lowercase code points (U+FF4B/U+FF41). Either update the comment or (preferably) switch the input to the mentioned code points to also cover the toLowerCase path.
    // K (U+FF2B) A (U+FF21) are full-width Latin letters an IME may emit
    expect(
      isKanaInputAnswerCorrect({
        inputValue: '\uFF4B\uFF41', // ka in full-width
        correctChar: 'か',

features/Kana/tests/isKanaInputAnswerCorrect.test.ts:336

  • The comment says the input includes a trailing U+FEFF, but the string literal currently does not include it. This makes the test less representative of the stated scenario.
    // U+200B before, U+200D in middle, U+FEFF after
    expect(
      isKanaInputAnswerCorrect({
        inputValue: '\u200Bs\u200Di\u200B',
        correctChar: 'し',

features/Kana/tests/isKanaInputAnswerCorrect.test.ts:386

  • This reverse-mode test name/comments mention “full-width kana input”, but the test input is regular kana with invisible characters around it. Renaming it to match what’s actually being validated will make failures easier to interpret.
  it('normalises full-width kana input in reverse mode', () => {
    // Full-width a (U+FF41) would not appear in kana reverse mode in practice,
    // but NFKC is applied uniformly; verify it does not break a correct kana answer.
    expect(

Signed-off-by: Aaditya parab <parabaaditya785@student.sfit.ac.in>
Copilot AI review requested due to automatic review settings July 31, 2026 23:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (2)

features/Kana/tests/isKanaInputAnswerCorrect.test.ts:412

  • This test name/comment refers to “full-width kana input” and shows an example full-width Latin 'a', but the actual input under test is hiragana with invisible characters (U+200B/U+200D). Renaming/rewording this keeps the test intent clear and avoids confusion for future debugging.
  it('normalises full-width kana input in reverse mode', () => {
    // Full-width a (U+FF41) would not appear in kana reverse mode in practice,
    // but NFKC is applied uniformly; verify it does not break a correct kana answer.

features/Kana/lib/isKanaInputAnswerCorrect.ts:63

  • lowerTarget is normalized with NFKC, but normalizedInput (and reverse-mode comparison) still uses NFC and does not strip IME-injected invisible characters. This means full-width romaji (e.g. hitoro) and U+200B/U+FEFF inputs will still fail to match, and the new normalization tests will fail. Normalize/strip the input (and reverse-mode target) consistently using the same pipeline (strip invisibles + strip whitespace + NFKC), then lowercase for romaji comparisons.
  // Normal mode: user types romaji. Compare case- and Unicode-insensitively.
  const lowerInput = normalizedInput.toLowerCase();
  const lowerTarget = targetChar.toLowerCase().normalize('NFKC');

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Kana lesson accepts incorrect reading for ひとろ

3 participants