Skip to content

Commit f662a72

Browse files
committed
replace the SPACE_PLACEHOLDER by a more exotic character
1 parent 89b577d commit f662a72

1 file changed

Lines changed: 2 additions & 1 deletion

File tree

src/tokenize.cxx

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -162,7 +162,8 @@ namespace Tokenizer {
162162
const UnicodeString type_unknown = "UNKNOWN";
163163
const UnicodeString type_unanalyzed = "UNANALYZED";
164164

165-
UChar32 SPACE_PLACEHOLDER = U'Ž';
165+
UChar32 SPACE_PLACEHOLDER = U'ǯ'; // LATIN SMALL LETTER EZH WITH CARON
166+
// found in languages: sami4 [Skolt Sámi]; (sorry Finland)
166167

167168
UnicodeString filter_placeholder( const UnicodeString& in ){
168169
/// filter out the SPACE_PACEHOLDER characters

0 commit comments

Comments
 (0)