Clojure and Markdown (and Javascript and Java and...)

Writing up a blog replacement for Wordpress (in Clojure) is coming along nicely. Clojure + Compojure are awesome. Most fun I've had making a website in a long while.

One problem I've run across is that I want to use Markdown for both post content and visitor commenting. I like Stack Overflow's live Javascript previews, so you can type text in Markdown and see what it'll look like as HTML, as you type it.

As I mentioned before, Markdown is very nice because it (partly) solves one longstanding issue I've had with programming blogs (including my own), namely the proper escaping of HTML and the proper formatting of source code. In Markdown you just put code in backquotes or indent it four spaces and there you go, properly escaped. Markdown is also easy and to type and read, which is a plus. I hate hate hate writing HTML by hand.

Anyways, there is no Markdown parser for Clojure, so I was going to write one. (There is MarkdownJ but it has unresolved issues.) The problem with writing my own Markdown parser in Clojure is that Markdown is not a well-specified language. There is no "official" grammar, just an informal "Here's how it works" description and a really ugly reference implementation in Perl. Most implementations (with the exception of peg-markdown and pandoc and friends) are implemented as a bunch of global regex-replacements passed repeatedly over some text.

The result is that there are a lot of Markdown parsers in a lot of languages, and they all give slightly different results in a lot of corner cases (and a lot of not-so-corner cases). The best I could do in Clojure is pick one implementation and try my best to match it.

Now, for each blog post, the server needs to store both the Markdown text and the HTML text. It needs the Markdown because if someone wants to edit content later, they need to edit the raw Markdown. It needs the HTML so that it can be cached and served to people viewing the website, obviously.

But a consequence of the above mess is that if you use a Javascript Markdown library (i.e. Showdown) to show a live preview, and then use a different Markdown library (my own or any other) to do server-side parsing of the text after it's POSTed, there's a good chance that the preview isn't going to match the real output.

One non-solution to this is to do all the parsing client-side, and POST both the Markdown and the post-Markdown HTML to the server so both can be stored, so no server-side parsing is necessary. Aside from being a horrid idea, it's a huge security risk because it leaves open the possibility of someone POSTing some clean Markdown along with some evil, un-matching HTML.

Another non-solution is to do all the parsing server-side and use AJAX to send the preview back to the client. That wouldn't be nearly as smooth or responsive as I want; on Stack Overflow for example the preview updates instantly after every keyup event in the textarea.

The ideal solution is use the same Javascript library client-side for previews, and server-side for parsing the text. Then the preview and content have a very high chance of matching. This requires some way to run Javascript on the server. Thanks to Clojure and Java and Rhino, this turns out to be trivial.

(ns bcc.markdown
  (:import (org.mozilla.javascript Context ScriptableObject)))

(defn markdown-to-html [txt]
  (let [cx (Context/enter)
        scope (.initStandardObjects cx)
        input (Context/javaToJS txt scope)
        script (str (slurp "showdown.js")
                    "new Showdown.converter().makeHtml(input);")]
    (try
     (ScriptableObject/putProperty scope "input" input)
     (let [result (.evaluateString cx scope script "<cmd>" 1 nil)]
       (Context/toString result))
     (finally (Context/exit)))))

This also saves me from having to write a Markdown parser in Clojure, for which I am thankful.

Once again I'm also thankful we live in times when CPU cycles are cheap and abundant. I'm running a Markdown parser, in Javascript, in Java, via Clojure, and this still runs essentially instantly even for very large input strings. If I had any chance of my blog becoming famous and getting a million hits a day, it might matter, but in real life I'm set.