Implementing the Indieweb on a static website

Jump to the main things

• Basic theory
• Micropub
• Backfeed
• Receiving Webmentions
• Formatting received Webmentions
• Sending Webmentions

IndieWeb

The IndieWeb is a collection of web technologies actively being worked on to share content between websites and break out from the silo’ed social design you may be familiar with.

All you need to join the IndieWeb is a Website and a URL then you can get involved. In fact, If you run your blog on a common platform such as Wordpress, you might even find that it is already supported.

My site is currently built on Jekyll, but in the near future I would like to migrate to Eleventy. Code is stored in Github and deployed/hosted by Netlify.The techniques I am about to describe should work for other static blogging platforms as well such as Hugo, Grav etc.

This post focuses on how I implemented the following collection of IndieWeb technologies on my static site.

• POSSE^[1]
• PESOS^[2]
• Webmention^[3]

Basic theory

I like to keep things simple. I am a big proponent of POSIWID, (Stafford Beer was a smart man).

It’s no secret that IndieWeb technologies can get complicated if you don’t use plug-ins as is.

I decided that all approaches I used must respect my current blog pattern:

1. Post a file in to Github.
2. Posting a file will cause the Netlify Webhook^[4] to activate and pull the code on to their servers which causes a rebuild and deploy.
3. Avoid client-side JavaScript. Everything is supposed to be static, lets keep it that way. It’s less complicated to cache for Service Workers and reduces potential errors.
4. Platform agnostic. I don’t want any logic responsible for IndieWeb technologies tied together with my blogging platform. Blogs are in markdown to keep them portable to any platform, so let’s keep it that way.

With this format decided it becomes clear that another actor is required.

My blog has no real dynamic aspect other than templates. Thus we need something that captures the data, transforms it in to what we need and pushes it in to the Github API (a translator and formatting engine).

Upon pushing the content in to the Github API, this will trigger a rebuild and deploy the site cleanly with no intermediate action needed. Nice!

I am most comfortable with Node.JS (but the techniques I am going to describe can work in most languages) so I spun up an IndieWeb server on Heroku.

The very knowledgable Phil Hawksworth pointed out recently I could possibly port this code to Lambda functions and remove the Heroku dependency entirely. It’s on the list for a future update and I would love to hear what other people think of this approach.

Microformats

Before we delve deep in to the code its worth noting a few more important things. The IndieWeb is heavily dependant upon Microformats and properly formed HTML. It is these tags it uses to work out what code is related to what. So spend time marking up your code correctly and read the Indieweb Getting started guide. In particular focus on h-card and h-entry.

Security

One last note on security. If you are opening up your website to code from an external source make sure you can only let in the code you want. If a bad actor discovers your endpoint and can submit content easily they might spam your website and no one wants that.

Luckily the smart IndieWeb folk have thought of the first point, so make sure in all your dealings with external sources you use IndieAuth. You will see me mention IndieAuth a few times below. Get familiar with it and how it handles the token request. It’s important you get this right.

It’s also good practice to rate limit your endpoint. Since our IndieWeb server is a separate Microservice^[5] to our website we don’t have to worry about a bad actor bringing our blog down, however we still need to protect our IndieWeb server from attacks such as denial of service.

Finally. Don’t store any of your keys in anything that can be traced such as Github. Learn about environmental variables and use something such as dotenv to manage them.

Micropub

Before you start, don’t forget to add the correct meta tags to your blog home page. You need one for your authorisation endpoint, one for your token endpoint and one to point to your Micropub endpoint.

You can see I am using IndieAuth as my token and authorisation provider. It’s the quickest way to get up and running but there are other options if you prefer them.

Here are my tags below as an example. Every time a service hits my website looking for how to send me Micropub content these tags tell it where to go and what to do. You can view source on my page and look in the header for a detailed example.

<link rel="authorization_endpoint" href="https://indieauth.com/auth" />
<link rel="token_endpoint" href="https://tokens.indieauth.com/token" />
<link rel="micropub" href="https://mastr-cntrl.herokuapp.com/micropub" />

Micropub is an open API standard (W3C Recommendation) that is used to create, update, and delete posts on one's own domain using third-party clients, and supersedes both MetaWeblog and AtomPub.

Web apps and native apps (e.g. iPhone, Android) can use Micropub to post and edit articles, short notes, comments, likes, photos, events, or other kinds of posts to your own site.

Put simply you can write a blog post on another authorised platform and submit it in to your website. This includes attached media, such as images. Finally the content injected in to your website can be optionally syndicated to other platforms with your blog post being the originator.

You can read the W3C spec if you want to dig deeper.

That’s quite a lot of concepts. Lets break it down.

1. I can log in to an IndieWeb authorised (and trusted) platform such as Quill.
2. Once logged in using IndieAuth I can write a blog post.
3. At the point of logging in, the platform will also look for a media endpoint and syndication targets.
4. This content is POST’ed to your micropub endpoint usually in JSON.
5. Once your micropub endpoint verifies the Token and accepts the content it parses the JSON body and formats accordingly.
6. If the content sent specifies a media endpoint your server will need to handle the request and upload the images sequentially “somewhere”.
7. If the content sent specifies syndication your server will need to handle POST’ing that content to the respective source or API.
8. In my case the content is then Base64 encoded and POST’ed in to the Github API, causing a rebuild and deploy. Although you may handle this last step differently depending upon your implementation.

Here is the basic flow my Micropub endpoint takes. In terms of where the “smart stuff” happens. This is all happening in Mastr-Cntrl^[6]. It captures the data, formats it then POST’s it in to the Github API. What keeps this “clean” is following the process as if we had coded the page by hand and committed it using Git^[7].

Media endpoint

Not essential, but useful if you want to own your imagery.

If this isn’t included a link may be included in the JSON to the original image by the provider, or you may have to add images manually to your post later.

Syndication endpoint

If this isn’t included then you will not be able to syndicate to other targets at the same time on publish. You could do this manually later. While I am finishing my own endpoint, I am cheating by using a Zapier Zap that watches a custom RSS^[8] (ATOM^[9]) feed. When a new item appears it syndicates the content to Twitter or Medium.

According to the W3C specification you need to declare these in the following way:

GET /micropub?q=config
Authorization: Bearer xxxxxxxxx
Accept: application/json

HTTP/1.1 200 OK
Content-type: application/json

{
  "media-endpoint": "https://media.example.com/micropub",
  "syndicate-to": [
    {
      "uid": "https://myfavoritesocialnetwork.example/aaronpk",
      "name": "aaronpk on myfavoritesocialnetwork",
      "service": {
        "name": "My Favorite Social Network",
        "url": "https://myfavoritesocialnetwork.example/",
        "photo": "https://myfavoritesocialnetwork.example/img/icon.png"
      },
      "user": {
        "name": "aaronpk",
        "url": "https://myfavoritesocialnetwork.example/aaronpk",
        "photo": "https://myfavoritesocialnetwork.example/aaronpk/photo.jpg"
      }
    }
  ]
}

In essence, your server needs to respond to a GET request on a \micropub endpoint when the parameter q=config is passed with a JSON file that specifies your media-endpoint destination and your social network profiles.

Note that in order for the endpoint to return anything, you must have first authorised the request using IndieAuth.

Backfeed

Backfeed is the process of syndicating interactions on your POSSE copies back (AKA reverse syndicating) to your original posts.

I currently use Backfeed via ownyourswarm and ownyourgram to syndicate content from those service back on to my website.

Backfeeding via these services works exactly the same as using Micropub and uses the same endpoint. The content is automatically published to your site a few moments after posting. If you are using Swarm badges and comments are sent as Webmentions. Neat!

The great thing about controlling your own content is that you can display it how you wish. For example, in this checkin I took a photo, so I can make it the focus of the post. However in this checkin I didn’t take a photo so instead I generate a static map image from Mapbox using the Longitude and Latitude data.

A quick note if you live in Europe around Backfeeding Webmentions via Brid.gy^[10]. There may be GDPR implications. I am avoiding this practice in the short term, and only publishing Webmentions sent to me directly to ensure intent that the content should be displayed on my website.

Formatting a Micropub post

Once we have got our content we need to map it in to a file, assign it the correct template, and save it as a markdown file.

In Mastr-Cntrl^[6:1] I use a formatter for each type. Here is an example of how I format a Note.

Currently I have commented out photos. This is because my media endpoint code is still not finished, once it’s done I’ll update the post. In the meantime I can comment it out and side-step using photos.

It’s fun to note that you can slowly build up your confidence with your Micropub endpoint. I’ve barely touched on a tiny amount of functionality it can perform. As I get more confident I expand what my formatter excepts and how I will capture and transform that data. Likes, Bookmarks, RSVP’s and more can all be managed this way.

The formatter maps the JSON received to a variable. It then performs a try catch to see if it exists. If it exists it is added in to the front matter^[11] for the markdown file. Otherwise substitute date is generated or the field is left blank.

This process captures all the blog post data in one place while keeping things agnostic in the standard Markdown and front matter combination all major static blogging platforms support.

Other things to note. Because my method involves POST’ing a markdown file in to the Github API, I need to Base64 encode the file. This is a requirement of working with the Github API. It’s worth familiarising yourself with the API if you want to use this method, otherwise it is not necessary for your implementation.

For day to day usage posting notes I have been using Indigenous (iOS/Android) by Eddie Hinkle. Which also has an IndieWeb Microsub^[12] reader built in. It’s a great workflow that I really like.

Receiving Webmentions

First we need to tell Webmention services where our Webmention endpoint is.

Here is mine as an example.

<link rel="webmention" href="https://webmention.io/vincentp.me/webmention" />

Notice I am not pointing this at Mastr-Cntrl^[6:2]. Instead I am pointing this at the free Webmention service Webmention.io.

The reason I am doing this is because capturing Webmentions, filtering out Spam and parsing web pages is tricky business. Webmention.io handles all the hard stuff then captures my Webmentions, holding on to them temporarily. There is an API you can manually fetch from, or you can add a secret token and your Webmention endpoint in the dashboard (once you have an account) causing the service to automatically POST you the data once it receives it. Which is what I do.

This is my basic flow for receiving Webmentions using a static site and Github with Netlify set-up.

In this flow the “magic” happens in the same fashion as my Micropub process. The IndieWeb server captures the data and formats it. It is then POST’ed in to the Github API. This triggers a rebuild and deploy of the site containing the new data.

The difference here is that I am storing the data in a JSON file in the data folder for Jekyll and my Webmentions are managed by Webmention.io a free service to handle receiving Webmentions.

I’ve opted in the short term to store Webmentions in JSON because the whole point of using a static site is to avoid a database driven website. Keeping the data in an open format permits me to port this to anything else with minimum effort should I opt to at a later date.

This approach does have a downside, over time the file may become quite large. Especially if your site gets a lot of traffic. There are other options I am exploring here such as saving the Webmentions in a file alongside each blog post, then making the template call the file specifically on build. I’m not sure yet if this will work out quicker when building the site and I’ll post an update once I’ve experimented further.

Formatting and storing Webmentions for a static site

Now we have the Webmentions captured how do we format them and output them on our site?

I include my Webmentions logic on each template I wish to use. Using liquid templates can be a pain for more complex logic. This is how I solved it, there may be a better solution!

{% assign comments = site.data.webmentions.links %}
{% assign likesAvailable = false %}
{% assign mentionsAvailable = false %}
{% assign repostsAvailable = false %}
{% assign repliesAvailable = false %}
{% assign bookmarkAvailable = false %}
{% assign rsvpAvailable = false %}

{% for comment in comments %}
    {% case comment.wm-property %}
        {% when 'in-reply-to' %}
            {% if thisUrl == comment.in-reply-to %}
                {% assign repliesAvailable = true %}
            {% endif %}
        {% when 'like-of' %}
            {% if thisUrl == comment.like-of %}
                {% assign likesAvailable = true %}
            {% endif %}
        {% when 'mention-of' %}
            {% if thisUrl == comment.mention-of %}
                {% assign mentionsAvailable = true %}
            {% endif %}
         {% when 'repost-of' %}
            {% if thisUrl == comment.repost-of %}
                {% assign repostsAvailable = true %}
            {% endif %}
        {% when 'bookmark-of' %}
            {% if thisUrl == comment.bookmark-of %}
                {% assign bookmarkAvailable = true %}
            {% endif %}
        {% when 'rsvp' %}
            {% if thisUrl == comment.rsvp %}
                {% assign rsvpAvailable = true %}
            {% endif %}
        {% else %}
    {% endcase %}
{% endfor%}

First I assign the Webmentions to a variable, in this case comments. Next I assign a variable for each Webmention type and declare it to be false. Then I loop over the Webmentions and look for an occurrence of each type that matches the URL of the page being rendered. If it exists I output true.

Next I include blocks of code where I wish to output the results. For example, this is how I am outputting replies. All my code is open source, and you can review my templates on my Github repo).

{% if repliesAvailable == true %}
    <h2 class="lh-one f5 red mb7 pa0">Conversation</h2>
    {% for comment in comments %}
        {% if thisUrl == comment.in-reply-to %}
                <article class="h-cite p-comment reply-relation mb5">
                    <h3 class="f6 u-author h-card ma0 pa0">
                        <img src="{{comment.author.photo }}" alt="{{comment.author.url  | replace: 'http://', 'https://'}}" class="u-photo photo br-circle fl mr7" width="30" height="30">
                        <a href="{{comment.author.url  | replace: 'http://', 'https://' }}" class="u-in-reply-to p-name u-url fl mt--fixed-4" rel="external">{{comment.author.name }}</a>
                    </h3>
                    <div class="e-content p-summary p-name reply-content cf mv5">
                        <p class="f6">{{comment.content.text}}</p>
                        <p class="f7">{% if comment.swarm-coins %}<img src="{{site.url}}/images/swarm/coin-icon.png" width="16" height="16" alt="coin icon" class="fl mr8">{{comment.swarm-coins}} coins<br>{% endif %}<a href="{{comment.url}}" rel="external"><time class="published-datetime f7" datetime="{{ comment.published | date_to_xmlschema }}">posted on {{comment.published | date_to_long_string: "ordinal" }} at {{comment.published | date: "%H" }}:{{ comment.published | date: "%M%P" }}</time></a></p>
                    </div>
                </article>
            {% endif %}
     {% endfor%}
{% endif %}

Finally, one last Jekyll specific little trick. It’s quite hard to test Webmentions locally. You need the code to match your live URLs but locally this is now localhost. This will cause no Webmentions to show.

We can remedy this by doing the following before the Webmentions code:

{% if site.webmentionTestUrl == false %}
    {% assign thisUrl =  page.url | absolute_url %}
{% else %}
    {% assign thisUrl =  "https://vincentp.me" | append: page.url %}
{% endif %}

This looks for a mysterious variable called webmentionTestUrl and checks to see if it is set to false. If it is, we are running localhost so it will append the live URL and fake it.

We assign the secret variable by passing multiple config files when running the site locally. You can view how I call this in my package.json file and here is the dev config.

The dev only config contains the variable set to true. On Netlify I only pass in the live config and set the variable to false.

Sending Webmentions

To send a Webmention you require a Source and a Target. Sending these parameters to a server that supports Webmentions will cause the service to visit the Source URL and look for a link to the target. If it finds it, it will look for Microformats to denote what to publish on it’s own site. I let Telegraph handle this for me.

Telegraph is a service that handles sending Webmentions. It also handles parsing Microformats and so forth. All I need to do is pass it my Source and Target URLs (plus my secret Token defined in the API dashboard).

Sending Webmentions is the trickiest problem to solve with a static site. Netlify allows you to use a Webhook on publish, which is great. But this is a dumb ping and not intelligent. How can we tell which post published is trying to send a Webmention?

It took a while for me to create a solution that respected the regeneration pattern, kept things simple and allowed me to craft a blog post manually or via Micropub and have a Webmention sent on publish.

The solution I created relies on creating a secret (or not so secret now) JSON feed on build. When this feed is created it compares two dates. A current “Published” date. Which is the last time Mastr-Cntrl^[6:3] sent a Webmention and the dates of the blog posts published since that date.

If a post created is after this date and contains front matter signalling a Webmention, the intended recipient URL (obtained from the frontmatter post) is assigned to the “Target”, and the post URL is assigned the “Source” and added to this feed.

When the site is published the feed is updated if the requirements are met and after it all compiles Netlify fires the Webhook POST to Mastr-Cntrl^[6:4]. Mastr-Cntrl then checks to see if this feed is empty or has content waiting. If the feed is empty it does nothing. If the feed contains content it grabs the source and target URLS and POST’s them to Telegraph.

After it receives the correct response from the API, it then GET’s my data file that contains the “last published date”. This file is overwritten with the current date-time, and the file is re-submitted back in to Github API.

By resubmitting the data file, this causes the site to regenerate. Now the date-time has been updated when the site re-builds the Webmentions to send feed will be empty and the site will re-publish. Mast-Cntrl will once again receive a Webhook POST from Netlify but this time the feed is empty and the process ends.

This does mean when you build the site to send a Webmention it builds twice, but this is OK and pretty quick. It seems to work well so far!

Here is the basic flow for what I described.

In terms of code. Here is how I build the custom feed.

---
layout: null
---
{% assign publishTime = site.data.published.time %}{
    "published":"{{publishTime}}",
    "webmentions": {% for post in site.posts %}{% assign postTime = post.date | date_to_xmlschema %}{% if postTime >= publishTime %}{% if post.replyUrl or post.like or post.repost or post.rsvp or post.bookmark -%}{% if forloop.first != true %},{% endif %}{
            "source": "{{site.url}}{{ post.url }}",
            "target": "{{post.replyUrl}}{{post.like}}{{post.repost}}{{post.rsvp}}{{post.bookmark}}"
        }{% endif %}{% endif %}{% endfor %}
}

I separate out the date in to a data file. I do this because it makes updating the file easier. I can overwrite it with the current date-time and be done. I don’t have to worry about parsing the file or accidentally breaking the feed logic.

"time: 2018-11-12T22:51:04"

I’ve left this open to adding multiple feeds to send. In the future I hope to iterate over this feed to send multiples at once. Currently I am just sending one at a time while I test this approach works and I am happy with it.

For sending Webmentions Mastr-Cntrl^[6:5] is only responsible for POSTing the JSON and updating the date-time file in Github API. Telegraph handles sending the Webmention and all the complicated stuff.

When I create a post or format it via Micropub and Mastr-Cntrl^[6:6]. I add some front matter to signify this is a Webmention. This is the code that causes it to appear in the “To send feed”. In this case below, the replyUrl is the target for a reply.

---
layout: "notes"
title: "Note for 2018-11-12T22:17:45"
date: "2018-11-12T22:17:45"
replyUrl: "https://aaronparecki.com/2018/06/30/11/your-first-webmention"
replyName: "aaronparecki.com"
meta: "testing sending a webmention for the first time"
category: "Notes"
tags:
- indieweb
syndication: ""
location: ""
---
testing sending a webmention for the first time

I’ve only just got started stringing these technologies together and I am enjoying the challenge. There is still much to do and much of my implementation is early in its inception, but I hope it provides some inspiration to others.

All my code is open source.

• Website
• Mastr-Cntrl

Please dig through them for examples of how things are done, or view source on a blog page if you want to view my Microformat implementation to help you further.

…And of course, if you have a question feel free to send me a Webmention!

Always more to do

• Storing Webmentions in one JSON file over time will cause build times to grow. It might be better to store Webmentions in a flat file alongside the post. I am still working on this solution, feel free to send your replies!
• Implement syndication the same as sending Webmentions.
• I would like to port the code over to Lambda functions in the future and remove the Heroku dependency.
• Moving to Eleventy next, this may change my implementation in some key ways. I’ll update this post if that is the case.
• IndieWeb Reader. This is my ultimate aim. I never stopped using feeds to consume content and have stuck with them. The IndieWeb Reader is bringing feeds back! I need to re-design my homepage to enrich my content through an indieweb reader.
• Content Sharding to improve performance is something that I am exploring.