skip yaml (front matter)?

[ec1oud]

Markdown can have yaml headers (yeah it's not in commonmark, maybe not a good idea either... but it happened). I think maybe md4c should skip parsing of "front matter" but also make it available for further parsing with a yaml parser. I.e. if the first line of the file is just three hyphens, that's a special case, not a thematic break, and everything from there until the next line with three hyphens can be assumed to be front matter. (I guess someone was thinking that nobody starts a document with a thematic break, so it can be treated as a special case.) But I'm not sure where's the most-formal specification (to verify that it must begin on the first line, must be three hyphens, and so on).

https://docs.github.com/en/contributing/writing-for-github-docs/using-yaml-frontmatter
https://jekyllrb.com/docs/front-matter/
readthedocs/commonmark.py#208
https://assemble.io/docs/YAML-front-matter.html

ATM I'm trying to figure out what Qt should do with this, since it's currently a mess if you give such a document to https://doc-snapshots.qt.io/qt6-6.7/qml-qtquick-textdocument.html#source-prop I figured if md4c would give me that chunk, I can stash it away somehow so that QTextMarkdownWriter can re-write it verbatim when the file is saved; but Qt doesn't have a yaml parser, so an application that cares about that might need to subclass QTextDocument, retrieve the saved front-matter string, and parse it with https://github.com/jbeder/yaml-cpp or something like that. But QTD won't even skip it yet, so it's going to be a bit less convenient for now.

[mity]

I'm not familiar with Jekyll or its front matters or whatever other tools support it.

But if I understand it correctly, it's some sort of templating system and the document can contain references to variables defined in the front-matter. Also note the templating system itself seems to be more or less independent of the document format after the front-matter: E.g. you might have a static HTML page below the front-matter instead of Markdown.

Consequently it seems to me that such raw documents shouldn't really be passed to Markdown or HTML or whatever parser before the templating system does its job, i.e. replaces the variable references with their respective value, removes the front-matter and then passes the result to the actual document parser, here MD4C.

[ec1oud]

I think yaml front matter is turning into the standard way to add metadata, not only for templating systems. https://obsidian.md/ does that, and I'm trying to prototype a similar app with Qt.

But ok I will see if I can get Qt to deal with it ahead of time, for now. (just the front matter, not template replacement)

Turning a line of yaml into an H2 is just wrong.

---
birth: !timestamp '2024-01-09 19:58:17.324717822 -0700'
---
blah blah

becomes (with md2html)

<hr>
<h2>birth: !timestamp '2024-01-09 19:58:17.324717822 -0700'</h2>
<p>blah blah</p>

[mity]

I think yaml front matter is turning into the standard way to add metadata

I see. Then it might make sense. With the objection that no validation of the stuff between the two --- markers would be performed. Incorporating YAML parser into MD4C because of this seems as a heavy overkill to me.

What should MD4C with it?

Skipping it silently, propagating it as a code block (possibly with MD_BLOCK_CODE_DETAIL::info and/or MD_BLOCK_CODE_DETAIL::lang set to something sensible), or do we need some special MD_BLOCK_TYPE for it?

Also as it's not in vanilla CommonMark spec, it would likely be recognized only with some new parser option.

[ec1oud]

I'm not sure yet what the md4c block type should be. --- is not a normal code block fence, so probably we shouldn't confuse it with a code block. There could be yaml in a code block somewhere else in the document, which presumably would not have the same meaning.

Maybe other people will have comments on this in the meantime.

I'm playing with https://codereview.qt-project.org/c/qt/qtbase/+/529543 for now.

[Moonbase59]

Just to add to this, because I’ve display problems in KOReader (koreader/koreader#14023, koreader/koreader#14012):

YAML headers in Markdown start with a --- on the first line, but can be ended by both a line with --- or a line with ...!

These are typically used to store metadata about the Markdown file, like author, title, date, etc. Well-known document conversion tools like Pandoc also use this information, for instance to produce HTML or EPUB metadata.

To be honest, even when just jotting down some notes in Markdown, I use this header, and of course without handling it, the output just looks bad.

Example (just took any random of my Markdown files):

---
title: A Short History of Drabble
author: Moonbase59
date: 2021-01-19
lang: en-us
...

Another example is in the Pandoc documentation.

I attach a sample HTML output, so you can see even HTML metadata can be generated from above header. Output was generated using the command:

$ pandoc -f commonmark_x -t html --standalone --toc=false -o 'A Short History of Drabble.html' 'A Short History of Drabble - YAML.md' 

A Short History of Drabble.html.zip

Hope this helps!

[ec1oud]

Qt is still skipping front matter, to avoid sending it to md4c. https://code.qt.io/cgit/qt/qtbase.git/tree/src/gui/text/qtextmarkdownimporter.cpp#n189

But this is the first I've heard that it's legal to end with ... But it's in the spec https://yaml.org/spec/1.2.2/#22-structures :

YAML uses three dashes (“---”) to separate directives from document content. This also serves to signal the start of a document if no directives are present. Three dots ( “...”) indicate the end of a document without starting a new one, for use in communication channels.

OK, that doesn't seem appropriate in markdown, as I read it. When we slap yaml onto the front of the document, we want to "separate directives from document content". We are not ending the yaml without starting a new document: we've got markdown coming after that. And what is meant by "communcation channels"? A stream (open socket) as opposed to a file?

I don't think it's wise to interpret the spec in such a way that you can't find the end of the YAML without having a YAML parser. We are not planning to ship one as part of Qt, so far.

You think it's also permissible to just end with a blank line? Does it mean I need to change Qt to look for --- or ... or just a blank line? Something is likely to go wrong, if I try to do that.

[Moonbase59]

You think it's also permissible to just end with a blank line?

The header? Nope. Must be either --- or .... They have a slightly different semantic meaning (because you could have several blocks of data in a YAML file), but all that isn’t of interest for a Markdown metadata YAML header block (which is always only ONE block).

The idea with the "communication channel" is that --- signifies the end of a data block, and ... the end of a (YAML) document. Thus, if you used a pipe instead of file access, you wouldn’t have to close & reopen the pipe to start the next YAML document. Multiple could be sent, one after the other, and a YAML parser would know where the separate documents ended.

In theory, the YAML data could even be amidst the Markdown somewhere, but fortunately all parsers I know of have decided on a "de-facto standard" that the Markdown YAML header MUST start with --- in the FIRST LINE of the document. (And can end with either of above.)

Especially with Markdown, which can use --- (besides others) to indicate a thematic break (horizonal ruler), this makes parsing much easier. Only the ONE "edge case" is left over, that someone wants to start a document with a thematic break. A simple "don’t do that" or "put it on the SECOND LINE" should suffice for these rare cases.

This should make it pretty easy just ignoring the header, as a first step. Or maybe have a switch and a "simple" rendering option (table?) like Obsidian does.

An ignore the header switch (as a first move) would be great, since unfortunately a YAML comment looks like

# This is a YAML coment

and would render a lot of H1 headings and other stuff…

Here is a real Markdown header I use to produce either HTML or an EPUB3 using Pandoc, as a real-world example:

---
# For HTML:
# title: "A Short History of Drabble"
# author: "Matthias C. Hormann"
# For EPUB:
title:
- type: main
  text: A Short History of Drabble
  file-as: Short History of Drabble, A
- type: subtitle
  text: All in a Hundred Words
- type: edition
  text: '2\. Edition'
creator:
- role: author
  text: Matthias C. Hormann
  file-as: Hormann, Matthias C.
publisher: Eigenverlag
rights: Copyright © 2021 Matthias C. Hormann, CC-BY-NC
date: 2021-01-19
# TBD: The real ISBN (this is just a fake for testing)
identifier:
- scheme: ISBN-13
  text: 9781234567897
lang: en-US
description: |
  What is a Drabble, and its history.
subject:
- Test
- EPUB
- E-Book
- Drabble
belongs-to-collection: Tests
group-position: 3.0
# cover-image: "none.jpg"
# css: "none.css"
#
# pandoc -f commonmark_x -t epub --standalone --toc=true -o 'A Short History of Drabble.epub' 'A Short History of Drabble - YAML.md' 
...

Guess you can imagine how horrible this would look if rendered…

The above was actually done as a test to see if Pandoc would produce an ebook with all the correct metadata, and Calibre, Foliate, Thorium, KOReader as well as real e-readers would pick up the metadata correctly. They do.

I don’t really know how KOReader is handling all that. In theory, there are two possibilitis:

• expect md4c to handle all, or
• pre-parse the YAML header (if existent) to get at the metadata for their own library, then pass you "the rest" (without header) for doing a partial rendering. This is probably how I would do it, but can’t speak for others here. ;-)

[ec1oud]

Well Qt has already shipped the front matter removal feature for a while; so it sounds like I have to make it deal with the ... end-of-document marker for the sake of complete compatibility.

Could a markdown document start with several YAML blocks though?

---
title:
  text: A dark and stormy night
---
# metadata for some other context / purpose
foo: blah
...
# A dark and stormy night
Some text here...

Then we'd have a problem in Qt, because the YAML would be considered to end on line 4 and the rest would get sent to md4c. Detecting two blocks would require read-ahead. And that's a fundamental ambiguity if you're never quite sure whether --- ends the YAML completely or you should keep looking for something that still looks like YAML and then a ....

But if you think md4c should also have its own solution, then we get back to mity's question earlier: what exactly should it do? Treat the YAML as a code block or do something else?

[Moonbase59]

Although your example was syntactically okay for a YAML file, a Markdown parser would end the metadata block at the first --- and use only the title object as metadata.

Let’s take Pandoc as a reference, generating partial HTML5 here:

$ pandoc -f commonmark_x -t html5 test.md 
<h1 id="metadata-for-some-other-context--purpose">metadata for some
other context / purpose</h1>
<p>foo: blah …</p>
<h1 id="a-dark-and-stormy-night">A dark and stormy night</h1>
<p>Some text here…</p>
$

Without a YAML parser, md4c could not do too much with the metadata, right?

• So the first step should be a switchable ignore, so just don’t display anything from the header.
• The second step could actually be displaying the header as a code block, as you suggest. It wouldn’t hurt anything, and the user could at least see the header info. A novel approach that appeals to me. Much better than having the header interpreted as Markdown, surely!
• For actually displaying something semantically meaningful, a YAML parser would be needed—probably too much? It would also have to be decided what to display, and how. That’s pretty easy for strings and numbers, but gets complicated when we talk objects (like the title in your example). Obsidian does something like that, for key/value pairs they find relevant.
• Does md4c create full documents or just partial HTML renders? If it was full documents, a subset of the metadata could be used to set HTML metadata, like the page’s title, for example.