The Old

My site is looking a little different now. I was previously using the Blades static site generator, which is a nice piece of kit, and comes with a nice looking blue retro theme out of the box. But I was running into some behavior I didn't like and I don't know rust, so I couldn't make the changes I wanted easily. Some were pretty easy to work around, the feed generators were including 'garbage' pages, and I toggled that off and wrote my own RSS/ATOM python script using FeedGenerator and BeautifulSoup . But then I wanted to alter the way the image gallery was being created, and that was more complicated, I managed to make some small tweaks that were painful enough to convince me that I didn't actually want to change anything else. Then I was hitting some of the limits of in-line html embedded in TOML, which required me to move the html into a template instead. There was also the CSS, which does look good, and works well on a variety of devices, but as someone who does approximately 0 website frontend development I was struggling with finding where to place any changes I wanted to make in the 6000 line stylesheet (also I was obviously destroying the CSS when I did make those changes because I know nothing about frontend development).

What I Wanted

Templates that can be inserted into other templates
Headers/Footers/Sidebars inserted into all generated pages
Write HTML for the post body
No pagination of the index page
Keep the same general look
Replace python feed generation
0 javascript
Tags and Categories
Auto-updating lists of tags and Categories
ATOM/RSS feeds with full body content
Debug site build that works on localhost/
Debug ATOM/RSS feed that lets me spot problems locally
Target firefox browser, no chrome-only features
Look okay-ish on mobile
Look okay-ish on terminal browsers (lynx, w3m, etc)
Keep dependencies minimal
Keep build time short

What I Wrote/What I Used

Website generator:


makefile -> m4 invocation -> includes (sub)templates -> expands macros -> *.html

Feed generator:


libxml + C code -> loop thru *.html -> loop thru <html> -> atom.xml rss.xml

This is the rough gist, the website generator more or less takes a mypost.m4 (contains date, title, author, template selection, etc), and a mypost.html (contains the literal html I want in the body), and then keep expanding macros until it outputs the full page you see here. List generation (front page, tag list, category list) is messy and forces us to do two passes, first the input mypost.m4+mypost.html are built. Then any new content labels or tag labels are are built into the header/sidebar sub(-sub-sub) templates, and their respective pages are generated (such as the page that lists all posts under the category computer). Finally another pass is done to rebuild the mypost.m4+mypost.html to include the newly updated header/sidebar. This is done via makefile, which mainly calls m4, but also relies on very light bash for looping, and a fair bit of standard unix tools, a quick glance shows cat, sed, cut, tr, sed, awk, printf, mv, cp, find, as well as pipes, and redirection operators.

The feed generator grabs only posts that I want to add to the feed, filenames passed via pipe/stdin or launch arguments. It takes the already built .html of the website generator and will grab info from either the filename (url), the contents of the files (find <title>, <article>, etc), or from a structure of hard-coded strings for boilerplate that goes into the <channel>/<feed> tags.

Lessons (Bad):

Do not write a static site generator in m4. Do not do it. Not even as a joke. I originally wanted a fully m4 implementation, and I got stuck trying to reimplement sort -u -r and gave up, ripped it out, ripped out some more stuff and replaced it with calls to sane unix utilities in the makefile. Pick any language you want as long as it has good debug utilities, trying to make sense of the output from m4 --debug=V will make you go nuts. In the end I managed to get the right level of quoting necessary for everything to work properly with my inputs except for article subtitles. If you look closely at the previous posts what looks like a ' (ascii single quote) is really a ’ (unicode RIGHT SINGLE QUOTATION MARK). Embarrassing.

m4 also has no good way to match (or match+1tab) the indentation level of the line preceding it, so you end up with some incredibly wonky html, you could pass the output through some sort of html linter in the final pass, but come on, if you're writing websites with a 50 year old macro language, you're not going to do that. If you're a real dummy you're also going to screw up your whitespace removing dnl statements, and end up with giant sections of blank newline in your output as well.

This leads me to my second warning, sure, on some level html is just text and you can use text processing tools to generate html, but go pick whatever library your language has for parsing html and just use that instead.

Writing html by hand means keeping a tab open with html character entities, so when writing a > or < you'll need to replace it with > or <. This stinks a little bit, but I write everything in a regular text file using vim first, and then use :%s to swap them all in one go, and then I'll use one or two macros to wrap everything in the tags I want. For someone with a different editing system this could be far more annoying to deal with.

Lessons (Good):

You do get 'free' sub-templating to infinite depth by using m4, and I really like how that works while I'm building up the website and experimenting, the trade-offs aren't worth it, but I would say it's worthwhile to add that feature to an site generator in a way that doesn't rely on macros.

I thought I would get sick of writing HTML by hand in the post body, but I really prefer it to whatever TOML/Markdown/generic markup auto-generation I've used in the past. If you want a heading it's not that hard to type <h1> </h1>, or <p> </p> for paragraph. The inline html that these markup converters allow seem to break if you try to really throw anything more fiddly at them, at which point you're dealing with writing the html you want and fighting the weird rules of your converter on top of it.

libxml is pretty sparse on documentation but I got to behave the way I wanted. I think the next re-write of the site will be in C with only libxml, it can read snippets of xml/html syntax and insert them into other snippets of xml/html, which is all I want. The code I wrote for the feed generation isn't the best but is miles ahead of the site generator code. I may go back and clean it a bit and .tar it up in case someone else has been looking for a way to keep their feed up to date that doesn't require remembering how python's venvs work. The libxml parser also threw a bunch of warnings that I was feeding it invalid html, which made me realize that I was writing invalid in-line html that the browser would render correct because the browser will always want to fail as gracefully as possible.

Through this process I also gained a good-enough understanding of html and css that I could doodle on a piece of paper the layout that I wanted and then get most of the way there to making that happen. I even got pretty okay page layouts for various mobile sizes.

I also learned that the right way to use multi-line <code> blocks is to wrap them in a <pre> tag (or vice-versa) and then feed readers will handle them the way you imagine they should.

Lessons (Neutral):

Middle of the road dependency-wise I think, I can safely assume m4, make, bash, and the standard unix utilities will be around working the same for a good long while, libxml is getting dynamically linked but I could link it statically if I felt like that was something to worry about.

Build times are fine, nothing to write home about, time(1) reports:

real    0m0.635s
user    0m0.343s
sys     0m0.444s

Lessons (Moronic):

I spent a fair amount of time making sure that all references to pages/images could swap between http://localhost/ and the real https://roadsidepicnic.net in the site generator. I thought this was important for debugging my atom/rss feeds. A wiser man than I would have just run sed on all the files in the output folder and accomplished the same thing.

Serving This Website:

Originally I was using cloudflare pages for hosting, it's easy, it's free, it does what it says on the tin. After this ordeal I was more interested in how my actual webpages were being served. I used nginx locally for testing and figured why not just run nginx on my VPS? So for now that's how my page is reaching you, no CDN caching, no nothing. To be seen if it survives the bot-filled internet of [CURRENT YEAR].