Emacs 31: An unofficial guide to Markdown-ts-mode

Intro

So, Emacs 31 has been released, and a lot of shiny new stuff is there, ready for us to play with.

You probably heard of this new markdown-ts-mode and decided to check it out. And guess what? On Emacs version 31, this is marked as an experimental mode. What does this mean? Should you use it or not? Is this ready? Is this just a sketch of a mode?

Treat this post as a quick guide to getting this mode up and running and helping yourself find answers to these questions.

Where is it in terms of features?

This is an experimental mode, right? You need to opt in, so probably not everything will work flawlessly yet, and it needs more testing and feedback.

That said, don't let this title mislead you. This does not mean the mode is premature in terms of features. As you will see, this is a very feature-rich mode. This mode already covers all of the https://commonmark.org/ spec, as well as most of https://github.github.com/gfm/, with some extras like code blocks even for non-ts-modes, like elisp, table of contents utilities, and interfaces with external converters, such as pandoc and gfm.

Before deep diving into it yourself, you may need some help simply turning this mode on. Tree-sitter is tricky. It might even be your first time with tree-sitter, so a quick "install guide" is on our agenda.

Where is it? Do I need to install the mode?

Experimental means Emacs does not enable the mode by default so it is not there waiting for you to simply open a .md file or call it with M-x markdown-ts-mode RET. You need to load this library.

As always on Emacs, there's more than one way of doing everything, I am a big fan of use-package so I tend to use it to organize my init file. Here is my suggested initial setup:

Or if you keep use-package out of your tool belt:

Now both the mode and the x (nice extra goodies) libraries are loaded, and you can simply visit your Markdown files using it.

If you want to experiment with it without touching your own configuration, do the following:

1. Save the above content in a file like testing.el.
2. Call emacs with emacs -Q --load 'testing.el'.

And there you have it, a bare Emacs session with your testing ground set up. This is what I will use for the rest of this guide.

IMPORTANT: there's NO NEED to download or add this package to your package manager. The (now very old and archived) MELPA Repository will refuse to install on Emacs version 31 onward and is very, very poor in terms of features. If you are using this, you're not using the new built-in markdown-ts-mode. Right? Let's continue.

Opening our first markdown file

In order for you to "see what I see", we need some pictures. If it is the first time you're using a tree-sitter-based mode, let me warn you: although tree-sitter is wonderful, fast, and feature-rich, it comes with its own set of tasks to complete and perhaps debugging skills if it needs help. I will try to cover some here; I will forget others for sure.

For this guide, I will be using this test file.

The repository where it is hosted is our laboratory. No code lives there, remember, all code is in Emacs itself.

Now go ahead and open the test.md file.

IMPORTANT: At this point, many things can happen. If you have the grammar for markdown installed in your system, the file is already opened. You could, though, be prompted, as I am here, with this:

It means Emacs hasn't found a grammar for markdown in my system, in this case in ~/.emacs.d/tree-sitter/ (which is the default when I start Emacs with emacs -Q ...). Emacs will offer to install it, which means downloading and compiling it from a repository already defined in markdown-ts-mode's source code. Let's install it with y. Emacs will clone the grammar repository, compile it, and continue to the second grammar. Yes, markdown uses two grammars: the main one and one for inline parsing. I will allow Emacs to install the second one with y.

Success!

What you should be seeing:

If not, here is what you should check if something went wrong:

1. Is Emacs compiled with the tree-sitter flag? Use M-: (featurep 'treesit) RET and check if it returns t.

2. Do you have the tooling used for "compiling" grammars, like make, gcc, and others?

3. Tree-sitter needs a package in your distro, usually named tree-sitter-cli which provides a tree-sitter binary, you can check you have it with tree-sitter --version.

This is a common headache for all tree-sitter modes. Many people like NOT to compile their own grammars, but instead use some compiled file from a place they trust, like their own distro repository, or packages with hundreds of pre-compiled grammars. I will not dive into it; there are many ways of acquiring grammars, and I will stick with "build it yourself" for this guide.

See, I kind of tricked you there. I told you that you should be seeing that, but actually, the "do you see what I see" should look like this:

We provide the full file in here, with several default themes so you can compare whether your setup is complete.

So, what happened?

This is part of the reason markdown-ts-mode is very special.

This mode can work not only with markdown, but with all other -ts-modes available! Keep this in mind; we will talk about code blocks in a while. For now, we need to understand a few things.

In your test.md file, we have a special header. It is very common to have toml or yaml as headers of markdown files.

This little guy here:

Needs something else to fontify (aka be painted with colors by Emacs). Can you figure out what is missing? If your answer is "we need a grammar for YAML!", kudos!

Whenever something does not fontify correctly in -ts-modes, you're probably missing a grammar. And as markdown-ts-mode is made to work with all available ts-modes, this is no exception.

Let's install our yaml grammar with our trusty M-x treesit-install-language-grammar RET yaml.

You might see now what I am seeing:

Let's agree to it with y. Hmm, it looks like this time, something went wrong with yaml-ts-mode trying to register its preferred grammar with treesit-install, as there are no suggestions. We could provide it manually. But let's check something first. Taking a look at yaml-ts-mode.el, we can check which grammar it expects in its source code:

Awesome! Let's simply evaluate that block and try to install the grammar again. Or manually provide the source https://github.com/tree-sitter-grammars/tree-sitter-yaml to our already-started interactive session, as I did this time:

We then keep going with the defaults with RET RET RET... until the library is installed.

After that, reload markdown-ts-mode, or use C-x x g, or re-open the file you're visiting.

What we did here by visiting the source code is pretty rare, and most -ts-modes will automatically suggest the repository from which they are going to compile. It was nice that this happened, so I can show you what to do.

Now what? We need to do the same M-x treesit-install-language-grammar for every block without fontification that we encounter. If you'd like, for our test file we could use C-x x f to force fontification and be prompted for every missing grammar used by this file.

By now, you should see the entire document fontified as in here. Same as previous image:

A note on grammars

A -ts-mode is only as good as the tree-sitter grammar behind it. This means every -ts-mode needs to constantly keep up with improvements to the grammar, which is shared by any editor or program wanting to use tree-sitter to parse the language.

This also means we are, at some point, dependent on the grammar for certain constraints and features. Almost all -ts-mode code in Emacs is filled with notes on limitations and the reasoning behind why and how something obscure is treated the way it is.

Emacs mode authors and maintainers always try to suggest the grammar and the SHA commit the ts-mode is prepared to use, either in comments or in the code inside the mode, which is the same as you saw for the yaml suggestion. Part of maintaining ts-modes is keeping up with newer grammar version changes. We try our best to keep it updated with the latest versions, but the one we tested against and that should work as expected is the one in the source file of the mode.

This is why I think compiling it yourself interactively with Emacs is the best possible way to guarantee a nice experience.

Specifically for markdown-ts-mode, we're using the grammars provided by https://github.com/tree-sitter-grammars/tree-sitter-markdown, as this is the most complete, maintained, and broadly adopted one, both by code editors and programs in general. This doesn't mean it is free of bugs or limitations. Again, we do our best to work around these limitations and even contribute issues to the grammar and to the core tree-sitter library.

I can finally open a markdown file!

Congrats! Now what? How often do I need to do all of this? Only once, the first time you use a -ts-mode, or never if you already have grammars installed by some other method.

Now let's see what markdown-ts-mode already provides.

A quick look at markdown-ts-mode features

We (BTW, this mode is authored by me and Stéphane Marks) provided an easy-menu feature for quick discoverability of functionalities.

You can access it by clicking on Markdown in the mode-line, or, if you have menu-bar-mode enabled, on the menu bar, or even Ctrl + Right click (whatever Emacs maps your OS input to) on a buffer using markdown-ts-mode.

This is actually this guide's TL;DR, if you want to stop now and explore it yourself (spoilers ahead).

Editing

The fastest way to learn the mode is to type a little of everything. Below is a speed run: what you write, what key does it for you.

Marks (emphasis)

Markdown is plain text, so you can always type the markers yourself:

When you want	You write
bold	**bold**
bold, alt	__bold__
italic	*italic*
italic, alt	_italic_
bold + italic	***both***
strikethrough	~~gone~~
inline code	`code`

Or let the mode do it: C-c C-x C-f (markdown-ts-emphasize) then a single key:

• b bold, B bold with underscores
• i italic, I italic with underscores
• a bold + italic
• s strikethrough
• c inline code
• SPC remove emphasis at point

If a region is active, the formatting wraps the region. With no region, it wraps the word at point, or inserts the pair and drops point in the middle.

Tip: C-c C-x RET (markdown-ts-toggle-hide-markup) hides the markers themselves, so **bold** shows as bold. Very nice for reading while editing, like default org-mode.

Another tip: M-q fills correctly even inside lists and quotes.

Headings

Type them: #, ##, ... up to ######. Setext headings (=== and --- underlines) are recognized, too.

Promote and demote without retyping the hashes:

• M-<left> promote (markdown-ts-promote)
• M-<right> demote (markdown-ts-demote)

And move a whole section, body and children included:

• M-<up> (markdown-ts-move-subtree-up)
• M-<down> (markdown-ts-move-subtree-down)

TAB on a heading cycles its visibility (outline folding). The mode is an outline-minor-mode citizen, so folding just works. S-TAB on a heading will cycle the visibility of all headings.

IMPORTANT: By now, you can see this mode tries, when possible, to draw parallels with org-mode, so Emacs users used to it can have fewer problems adapting to markdown. If these bindings don't suit you, everything can be customized.

Listings (lists and checkboxes)

Type - item, + item, * item or 1. item.

• M-RET new list item (markdown-ts-insert-list-item)
• RET is smart: markdown-ts-newline continues the list for you
• M-<left> / M-<right> promote/demote the item
• C-c C-r renumber an ordered list (markdown-ts-renumber-list)
• C-c C-c toggle a task checkbox (markdown-ts-toggle-checkbox)
• M-q fills correctly inside an item

Task lists are the GFM ones:

Raw mode:

With markup hidden:

Note the bullets and boxes you see if you toggled C-c C-x RET are display only. The buffer still holds - and [x]. See markdown-ts-unordered-list-marker, markdown-ts-checked-checkbox and markdown-ts-unchecked-checkbox.

Blocks

C-c C-, (markdown-ts-insert-structure) then one key:

• ` fenced code block, prompts for the language
• ~ tilde fenced code block
• q block quote
• d divider (thematic break)
• t table

If a region is active, it wraps the region instead of inserting an empty block.

With markup hidden:

Code blocks

This is the party trick. A fenced block tagged with a language is fontified by that language's own mode:

Missing colors typically means a missing grammar, same story as the yaml header earlier.

Better than colors: put point inside the block and you are in markdown-ts-code-block-in-context-mode (lighter [code] in the mode-line). Inside it:

• TAB indents like the language does
• RET newline and indent like the language does
• M-q fills like the language does
• M-. jumps to definition via xref

Move to the next/previous blocks with C-c C-v n and C-c C-v p.

Non tree-sitter modes work too, elisp included. Knobs: markdown-ts-code-block-modes, markdown-ts-default-code-block-mode, markdown-ts-fontify-code-blocks-natively.

An example raw:

With markup hidden:

Tables

Insert one with C-c C-, t or M-x markdown-ts-table-insert-table, which asks you to specify the number of rows and columns to insert.

Inside a table you are in markdown-ts-in-table-mode (lighter [table]) and the keys change:

• TAB / S-TAB next / previous cell (also formats your table)
• RET / S-RET next / previous row
• M-RET insert row below
• M-<up> / M-<down> move row
• M-<left> / M-<right> move column
• M-S-<up> insert row above, M-S-<down> delete row
• M-S-<right> insert column left, M-S-<left> delete column
• C-c C-c align the whole table
• C-c C-t a set column alignment (left, center, right)
• C-c C-t t transpose the table

Plus, from the menu: clone rows and columns, CSV/TSV import of a region and CSV/TSV export of the table.

NOTE: There are some limitations when working with tables at the moment, mostly due to how the grammar parses them, so you may bump into unfontified stuff while typing. All valid tables according to the GFM spec should be good to use, though.

Links and images

Links are the usual [text](url) and [text][ref]. Fragment links like [intro](#intro) are clickable and jump to the heading in the buffer, using GitHub style slugs by default.

Images render inline. C-c C-x C-v toggles them (markdown-ts-toggle-inline-images). See markdown-ts-image-max-width and markdown-ts-display-remote-inline-images for how big and whether remote URLs are fetched.

Markdown:

After C-c C-x C-v:

After C-c C-x RET:

Moving around

• TAB cycle folding at point
• C-c C-n / C-c C-p next / previous heading
• C-c C-u up to parent heading
• C-c C-f / C-c C-b next / previous heading, same level
• M-x imenu jump to any heading or named code block by completion
• C-c C-v n / C-c C-v p next / previous code block

markdown-ts-default-folding decides how a file opens: everything shown, or folded.

markdown-ts-view-mode

M-x markdown-ts-view-mode read-only mode with a single key navigation: n, p, u, f, b, TAB. Good for reading a README without fear of typing into it.

Everything below lives in markdown-ts-mode-x.el, which is why we loaded it back in the setup.

TOC

A table of contents is delimited by HTML comments, so it survives rendering anywhere:

• M-x markdown-ts-toc-insert-template inserts those markers, basic or complete (the complete one lists every parameter with its default)
• M-x markdown-ts-toc-generate fills them in, and refills on every call
• M-x markdown-ts-toc-clear empties, markdown-ts-toc-clear-and-remove also removes the markers
• M-x markdown-ts-toc-update-before-save-mode regenerates on save

Parameters go inline in the opening comment: min-depth, max-depth, candidates, from, style, indent, no-link, relative-depth, ignore. A buffer can hold more than one table with different parameters. Candidates are not only headings, list items, setext headers and named code blocks can feed a table too.

Raw:

With markup hidden:

Exporting

M-x markdown-ts-convert converts the buffer, markdown-ts-convert-file a file. You get asked for the format and the converter, unless you set markdown-ts-default-converter. Supported out of the box:

• PDF via pandoc
• HTML via pandoc, cmark, cmark-gfm, markdown, markdown.pl

With a prefix argument the result is displayed, by default with eww. See markdown-ts-convert-display-function to open in a browser instead. That is your somewhat 'live' preview. Converting is not (yet) automatically when you make changes, maybe in the future.

Example using eww, split manually made for this demo:

Spec at hand

M-x markdown-ts-browse-commonmark-spec and M-x markdown-ts-browse-gfm-spec open the specs, for when you need to settle an argument.

Experiment with eglot and eldoc

This is still experimental within the experimental, so don't blame eglot's author if something goes wrong. Send a bug report to markdown-ts-mode instead.

If you set this:

Eglot will try to render documentation (usually Markdown provided by the LSP server) using markdown-ts-mode.

Again, we are still shaving off some rough edges here, and results may vary. Please do help us test this, though.

Play with options

M-x customize-group RET markdown-ts RET and go through it. Some of the customs worth a look at first:

• markdown-ts for display: markup hiding, ellipsis, bullets, checkboxes, thematic break and hard line break characters, inline images, folding on open
• code blocks: markdown-ts-code-block-modes, markdown-ts-default-code-block-mode, markdown-ts-enable-code-block-context-mode
• tables: markdown-ts-enable-table-mode, markdown-ts-table-auto-align, markdown-ts-table-default-column-width
• markdown-ts-convert for exporting
• markdown-ts-toc for tables of contents

Faces are customizable too, one per Markdown element.

How you can help

The best way you can help is simply by using it. Try it with your Markdown files, play with the different features, and see what needs improvement or what breaks.

If you find something that doesn't work as expected, please report it as a bug from Emacs itself with M-x report-emacs-bug RET. Include a small example that reproduces the problem whenever possible. This is especially useful for issues involving fontification, tree-sitter grammars, tables, code blocks, or interactions with other modes.

We're still polishing the rough edges, so bug reports, feedback, and real-world testing are very welcome.

I found a bug, is it because markdown-ts-mode is buggy?

Some of the surprises you may hit while using markdown-ts-mode might be the mode, some might be the grammar, some might come from how tree-sitter is integrated into Emacs, or from the tree-sitter ecosystem as a whole. Knowing about this upfront helps understanding that debugging is challenging.

Grammars are a shared, external asset

A grammar is not written for Emacs. The very same tree-sitter-markdown is consumed by other editors and tools, so any change to it is negotiated among all of its users. That is great for the ecosystem, and it also means a fix we would like to see may take a while to land, or may never land in the shape we would prefer. When that happens, we work around it inside the mode as best we can, and report the issue upstream.

So, if you find something that looks like a mode bug and the answer turns out to be "the grammar parses it this way", now you know where that answer comes from. Please do report it anyway, we would rather hear about it twice than not at all.

Building grammars has its own quirks too. Not every grammar builds with make and a C compiler alone: several are generated from a JavaScript definition, so their build path expects the tree-sitter CLI, and sometimes a Node.js installation, to be available. This is a good part of why pre-compiled grammar bundles and distro packages are so popular. As said before, I still prefer compiling them interactively from Emacs, but now you know why your distro may be pulling in more than you expected.

Indirect buffers

This one deserves an explicit warning, because it surprises people: tree-sitter and indirect buffers do not get along.

1. Parsers are not shared with indirect buffers. They belong to the base buffer, and an indirect buffer starts with none. You either copy them over manually, or re-instantiate them by enabling a major mode in the indirect buffer.

2. Font-lock in indirect buffers is not supported at all. This is a limitation in Emacs itself.

The practical consequence is that (at least at the moment of this writing) if you use a package that clones a region into an indirect buffer, expect no fontification there. This is not specific to markdown-ts-mode, it applies to every -ts-mode, and it is not something we can fix from the mode's side.

Further reading

If this guide got you interested, there is a lot of good material out there about writing and using tree-sitter modes. Stéphane Marks, my partner in crime on this mode, put together the list below, and it is too good to keep to ourselves. Some of it may be a little stale by now, tree-sitter moves fast, but the reasoning in these articles holds up:

• Pulsar Edit's tree-sitter series, another editor going through the same journey
• Building Emacs major modes with tree-sitter: lessons learned
• Tree-sitter modes still need a syntax table
• Let's write a tree-sitter major mode
• Making an Emacs major mode for Cabal using tree-sitter

And, of course, the notes from the people who built all of this into Emacs, Yuan Fu and Juri Linkov, which are the closest thing we have to a canonical reference:

• Emacs 30 tree-sitter notes
• admin/notes/tree-sitter in the Emacs repository

Is this going to be out of the experimental tag on next Emacs release?

In this post beginning I wrote:

What this means? Should you use it or not? Is this ready? Is this just a sketch of a mode?

Now you probably have a better answer.

experimental does not mean markdown-ts-mode is just a sketch or that it is missing the basic features you would expect from a Markdown mode. It means the mode is still evolving, and we are not yet ready to promise that its API, behavior, or some of its features won't change.

So, should you use it? Yes! If you are comfortable with the experimental label, please give it a try. The more people using it with different Markdown files, configurations, and workflows, the easier it is for us to find issues and fix it.

Will it be out of experimental in the next Emacs release? Maybe, we sure are working towards it! We will see. There are still things to polish, limitations to work around, and feedback to process before we can make that call.

For now, consider this your invitation to play with it. And if you find something weird, don't just work around it, let us know. That's how we get it ready.