fix: lone quote in skipped line no longer consumes rest of file - #634
Closed
LeonidasZhak wants to merge 2 commits into
Closed
LeonidasZhak wants to merge 2 commits into
LeonidasZhak wants to merge 2 commits into
Conversation
Add a 'Multi-byte characters' section to vroom_fwf() documentation explaining that field positions are byte offsets, not character positions. For multi-byte encodings like UTF-8, these differ. Add a test documenting the current byte-based behavior with the degree symbol (°), which is 1 character but 2 bytes in UTF-8. Closes #622.
When is used and a skipped line contains a lone (unclosed) quote, would set and never reset it, causing all subsequent newlines to be treated as embedded within a quoted field. This made the skip consume the entire file. The fix detects when EOF is reached while is still true (i.e., the quote was never closed) and falls back to finding the first newline from the original start position. Regression test for tidyverse/readr#1577.
Author
|
I am withdrawing this PR as part of a broader cleanup of an oversized automated contribution batch. I am sorry for the review noise and the pressure this may have put on maintainers. Going forward I will keep contributions to this project much more limited and higher-signal, ideally only one or two focused PRs at a time. Thank you for maintaining the project. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes tidyverse/readr#1577
Problem
When
skipis used and a skipped line contains a lone (unclosed) quote character,find_next_non_quoted_newlinesetsin_quote = trueand never resets it. This causes all subsequent newlines to be treated as embedded within a quoted field, making the skip consume the entire file.Fix
In
find_next_non_quoted_newline(src/utils.h), when EOF is reached whilein_quoteis still true (indicating an unclosed quote), fall back to finding the first newline from the original start position. This ensures a lone quote in a skipped line doesn't consume the rest of the file.Changes
src/utils.h: Add fallback logic infind_next_non_quoted_newlinefor unclosed quotestests/testthat/test-vroom.R: Add regression testTesting