Current inline directive syntax should not be enabled by default

[DavidMulder0]

Initial checklist

• I read the support docs
• I read the contributing guide
• I agree to follow the code of conduct
• I searched issues and discussions and couldn’t find anything (or linked relevant results below)

Affected package

latest

Steps to reproduce

Although there are MANY people reporting how completely broken this extension is, somehow all relevant issues link to other closed issues and closed pull requests just saying that it should be discussed elsewhere.

By far the best work on this is done by @GauBen in remarkjs/remark-directive#19 (comment)

Hi there! Sorry for unearthing this issue but it seems to be the most appropriate place to post my two cents. Please let me cook.

When this many people have been bit by a feature, we might start considering it's a bug.

The most interesting thing to note is that everyone complains about text directives (:directive), and for one very legitimate reason: :\w+ may appear naturally in markdown prose (10:30 AM, localhost:8080, Mitglieder:innen, John 15:13...). ::leaf directives and :::container directives have no such issue because of intentionality: starting a new line with at least two colons is never accidental.

I see two different approaches to address the way this feature behaves.

Syntactical Solutions

We may first approach this problem with a syntactical solution: since the underlying discussion is not settled we are free to implement what we want until a standard is created.

1. Don't allow "bare" text directives. We'll call a bare directive a directive without label and attributes. As currently implemented, :directive, :directive[], :directive{} and :directive[]{} behave the same. This proposal suggests that :directive is no longer parsed as a directive.
   This solves the intentionality principle: you don't see :\w+[] or :\w+{} naturally appear in prose.
   Also, as a side-note, I don't see the use for text directives without parameters. It probably means you want to insert something in your text, and for that, you can use the :emoji: syntax, which removes all ambiguity. All examples in the original post have parameters. Maybe some people abuse the syntax by using non-static directive names (:123 instead of :pr[123], :boolean instead of :badge[boolean]), but that goes against the concept of markdown directives.
2. Use another starting character. The proposal mentions !, @ or ::, but many keyboard-accessible characters could do the trick (maybe %?). The constraint being it should not appear in the middle of words.
   I find this solution less convincing than the first one, mostly because it does not solve the intentionality principle as well, and many platforms already use @ or ! (mention a user, link to GitLab issue...). Coupled with 1 is perfectly fine though.
3. Ask users to escape :. Just kidding, this is a terrible idea. The point of markdown extensions is to add features to markdown, and that might happen after you have written a significant amount of markdown. If it breaks existing markdown, it's a design flaw: it means that the syntax is not specific enough, and we're back to the intentionality principle.
   \: being the normal way to insert colons is user-hostile.

Technical Solutions

If we want to keep the syntax untouched, the other leverage we have is the implementation one: the way micromark-extension-directive behaves when parsing markdown.

Since directives are extension points for markdown (one syntax for arbitrary extensions in markdown), users would always read the documentation (or other markdown files) to know what's available for them to use. You cannot expect all platforms to implement the same directives.

What should a user expect when using an unknown directive?

• An error: this goes against the philosophy of markdown. A markdown parser first tries to fix the markdown (e.g. If the end of the containing block (or document) is reached and no closing code fence has been found, the code block contains all of the lines after the opening code fence until the end of the containing block (or document).), and if it fails, it leaves the input tokens as-is.

• Random <div></div> inserted: nope, nobody wants that, ever. It does not make sense, and might even not be noticed in long documents. (happened to me, the issue stayed under the radar for two weeks)

  It may even produce invalid HTML
  

• Nothing: pretty much what you expect when you make a typo in markdown. Forgetting the space after # for your title? You get <p>#title</p>.

Typing ::youtub[dQw4w9WgXcQ] instead ::youtube[dQw4w9WgXcQ] should result in <p>::youtub[dQw4w9WgXcQ]</p>, I'd argue that the outcome that makes the most sense.

To implement that, we could add some kind of filter to micromark-extension-directive so that it's not :\w+ that gets parsed into a textDirective node, but only the specific directives supported by the platform in question. Other directive-like symbols remain as-is in the output document.

directive({
  /**
   * @type {(type: 'text' | 'leaf' | 'container', name: string) => boolean}
   * @default () => true
   */
  filter: (type, name) => {
    if (type == 'text' && name == 'year') return true;
    if (type == 'leaf' && name == 'youtube') return true;
    if (type == 'container') return true;
    return false; // All other directives are not parsed
  }
})

Closing Words

Thanks for taking the time to read my proposal. I'm okay with any solution that would not require updating my markdown files while still being able to add features to them over time.

My current workaround is patching micromark-extension-directive to only enable container directives, which is as bad and unsafe as it looks:

import { directiveFromMarkdown } from 'mdast-util-directive';
import { directive } from 'micromark-extension-directive';

unified().use(function remarkDirective() {
  const data = this.data();
  const { flow } = directive();
  // Remove `directiveText` and `directiveLeaf` manually
  (data.micromarkExtensions ??= []).push({ flow: { [58]: flow[58][0] } });
  (data.fromMarkdownExtensions ??= []).push(directiveFromMarkdown());
});

It works ok for now, but an official (and safe) solution would be more than welcome.

And then there is @wooorm who somehow believes this is only about numbers (common counter examples: spreadsheet formulas (A1:A10), company names and other names (re:zero, NYSE:SHEL)).

Actual behavior

Depending on the exact setup the result might differ, but fundamentally the golden rule of markdown is broken: The source syntax reads VERY differently from the rendered output including completely changing the meaning (e.g. "common counter examples: spreadsheet formulas (A1), company names and other names (re, NYSE)").

Expected behavior

Normal human text should not be affected by a generic extension.

 - To sum all values use sum(A1:A10)
 - The stock of NYSE:SHEL went up despite the news
 - I watched re:zero all night
 - At 3:15PM we left to the hotel
 - See DOI:10.1000/xyz123 for more information
 - The 3:4 TV looked good on their wall

Runtime

No response

Package manager

No response

Operating system

No response

Build and bundle tools

No response

[wooorm]

Inline directives should be enabled by default. It sounds like you are not understanding the purpose of this syntax extension, which attempts to bring about one syntax for everyone.

You point to that you do not like the design goals of markdown. That is okay. You can use other things. The goal here is not to destroy markdown for your preferences.

Some of your examples are code, for which you can use `code`.
Other examples are indeed numbers.
As mentioned in other PRs, you can make an issue or a PR for those.
You can also always escape things if you don’t want directives.

[angrybacon]

(disclaimer: I don't use micromark-extension-directive directly but remark-directive)

the golden rule of markdown is broken

I also come from the same premise. As in, my Markdown content containing directive syntax should yield valid output whether I do add support for directives or not.

you are not understanding the purpose of this syntax extension

Is my above premise false? If it is, then I don't consider it an extension. Since it doesn't extend behavior but replaces it, if that makes sense. If that's still considered the route this package wants to go with, I would highlight it in more places because issues raised on the subject are proof that this isn't intuitive for everyone.

[wooorm]

Valid input is still valid input, there are no syntax errors.
Every character in markdown (or HTML) does something, which implies that changing any character to do something else, changes things.
I repeatedly mention that it is fine to discuss the semantics of characters: something loose enough to support non-english languages (which is a good cause that warranted this) while strict enough to not run into your case (which is what?). Please focus on improving things for everyone thank you.

[DavidMulder0]

Inline directives should be enabled by default. It sounds like you are not understanding the purpose of this syntax extension, which attempts to bring about one syntax for everyone.

They should indeed be enabled by default if they didn't actively break existing markdown documents. I wholeheartedly agree that it would be great if they are enabled, but as-is they make this extension "dangerous" and "useless" (unless you are literally authoring new markdown documents from scratch). If you have existing content there is a serious risk that normal content is broken by the inclusion of this extension. (Broken is putting it generously, this extension might be silently changing the meaning of the content, which is a terrifying prospect for any serious document).

You point to that you do not like the design goals of markdown.

I make the literal opposite point: I love the design goal of markdown, and this extension does not follow it. It renders source and output VERY differently for normal prose.

---

Personally I would be in favour of either changing the syntax, or Technical Solution #1 by @GauBen .

[github-actions]

Hi! This was closed. Team: If this was fixed, please add phase/solved. Otherwise, please add one of the no/* labels.

[DavidMulder0]

(First of all, not sure why Github autoclosed this, but the important part is:)

I lost hope that this bug/issue will get solved. I was using @GauBen 's solution so far, but I really did want Text Directives, so I forked this project just to make this one change to remove bare Text Directives.

The updated code is available on

https://github.com/DavidMulder0/micromark-extension-directive

This can now be installed using

npm install github:DavidMulder0/remark-directive#3fed4d584b97d16c8ed00cb5899fb14c54d197c0

(the commit id is probably not needed, but I somehow hit an issue with either npm/github caching)

I might or might not publish to npm as a fork at some point. I am very much not vouching for the quality for the changes I made, as I was unfamiliar with the codebase, but 🤷.

[ChristianMurphy]

not sure why Github autoclosed this

You put the word "fixes" then this github id.

The updated code is available on
https://github.com/DavidMulder0/micromark-extension-directive

Thanks for sharing!
And an extension is the right approach.
This project can't be everything to everyone, customizations can live in other repos.