Data Cleaning

Issue: Non-dialog introductory text

Several dialog sets begin with a description of the setting.

Sometimes, that description includes non-attributed dialog.

Approach 1 Attribute the dialog to a narrator.

Regex to capture unattributed text at the beginning of a conversation.

(?:<START>\s*)(?<!{{(?:user|char)}})([\s\S]*?)(?=\s*{{(?:char|user)}}|<START>|\n\n<START>) vs-code escaped version: (?:<START>\s*)(?<!\{\{(?:user|char)\}\})([\s\S]*?)(?=\s*\{\{(?:char|user)\}\}|<START>|\n\n<START>)

• Start matching upon encountering <START> (?:<START>\s*) details - And match any trailing whitespace between and the first line of text.

• Stop matching if... (?<!{{(?:user|char)}})

  details - Negative lookbehind

• Capture this... ([\s\S]*?)

  details - Ungreedy match - Capture any character or whitespace - Removing ? makes it match past the following stop sequence.

• Stop match at... (?=\s*{{(?:char|user)}}|<START>|\n\n<START>)

  details - Positive lookahead. - \s* might not be needed.

(old): (?:<START>\s*)([\s\S]*)(?=\n\{\{char\}\})

- Match beginning at <START>
    - Followed by any amount of whitespace
- Capture character and whitespace that does not contain `{{char}}` or `{{user}}`. 
- End capture at `\n{{char|user}}`

Issue: Non-dialog sets

Some sets have no dialog at all.

Possible solution

Regex to identify blocks of dialog with no user or char.

Regex: (<START>\s*(?:(?!\{\{char\}\}|\{\{user\}\}).|\n)*)(?=<START>)

- Match starting at <START>
- Require a white-space element
- Non-matching container group
    - (Negative lookahead). Stop matching if `{{char}}` or `{{user}}` is found.
    - Match any character or newline
        - Match zero or more times
- End match at <START>

Issue: Incorrect whitespace formatting for

Some dialog items have only one linebreak between the text and the next . Example of incorrect spacing:

{{char}}: The enquirer is the answer and no other answer can come. What comes afresh cannot be true. What always is, is true. 

<START>

To correct this spacing, these regex expressions are proposed:

Regex search pattern: (?<!\n)\n\n<START> Regex replace pattern: \n\n\n<START>