From Indieweb Monolith to Microservices

This post is part of a series where you can follow along as I redesign the blog and my IndieWeb Server.

Quick recap on my old architecture

A year ago I began a journey to really rethink how Indieweb functionality could work with a static website.

The hardest part to this process was the architecture, technically I need to be able to create an easy to support system, that is simple to deploy but flexible to allow me to do the interesting things I want to do with data.

I think (finally!) the major technical work is coming to an end so I wanted to share my thought process and learnings from what I have done.

Originally I had my static site deploying to Netlify. This was supported by my IndieWeb server running on NodeJs on Heroku which:

• Listened for webmentions, POSTing them in to the Github API (triggering a redeploy with the content added).
• Sent webmentions by recieving a GET request on deploy from Netlify, then polling an RSS feed for content before forwarding content to Telegraph for sending.
• Recieved Micropub content, formatting in to Markdown and POSTing in to the Github API (triggering a redeploy with the content added).
• Recieved Media (it only supports jpg images at present) and POSTs in to the Github API (triggering a redeploy with the content added).
• Recieved checkins via ownyourswarm and converting to markdown posts and POSTs in to the Github API (triggering a redeploy with the content added).

I would syndicate content by using an XML feed that was periodically polled by Zapier and that would send content to various places.

It all worked but it was incredibly brittle. Problems ranged from:

• If I broke webmentions in any way, or the server went down, the trigger from Netlify would fire 10 times then silently fail (cmon Netlify at least tell me something went wrong!). Causing a backlog.
• Something broken in one area of the NodeJs server would mean other functionality could stop working as well. (I may have been able to isolate my code and handle it better, but there is only so much free time I have)
• Hard to extend the code. Partly I had a messy codebase, but also when your service “does all the things” it leads to complexity in handling those things and accounting for all the consequences. Less is better and encapsualted code is even better.
• Things doing multiple things, leading to collisions or race conditions.

Complexity was also a big driver, I don’t tinker in the code constantly, so even with good naming conventions and logic it can lead to accidentally missing unintended consequences around changing behaviour.

Learning

I have written in the past around the possibility of combining Lambda functions (or Netlify functions) in to my process. My theory was that on deploying the website, the functions would fire and do things like send a webmention or syndicate content. This was to reduce complexity by moving code from Mastr Cntrl and in to simple “single purpose” functions.

I’ll cut to the chase here. I managed to write a working Netlify function that would send webmentions and another that would syndicate content similar to Max Böck’s example here. It wasn’t a fun process at all however. The Netlify Dev CLI was incredibly buggy. I had to re-install Node multiple times, reauthentice and numerous other issues. I burned through hours and hours of free time I couldn’t really spare.

The buggy CLI aside (which I assume will be fixed at some point) I didn’t proceed with this direction for bigger reasons. Once I had the code working I took a step back and realised I had begun to build a dependancy in my build process. When I set out on the path to include Indieweb functionality I always took the approach, that it was the content I wanted to own, not the process.

1. Indieweb approaches may fall out of favour, the web could change radically, the future is unknown. I want to keep the separation of my static site and not intertwine the logic of these processes. It is one of the reasons I offload the creation of Micropub posts to my server and deliver the static file as if I had created it. At a later date I may want to abandon Indieweb and if I ever chose to do so it should be simple and easy to walk away from. Otherwise I am no better than the platforms I am “liberating” my content from.

2. A personal coding thing, perhaps its my experience level, I really didn’t like the way Lambda functions are formed and how they work. It all just felt a bit wrong and this needs to be at least enjoyable to do or I will get bored and go do something else.

3. You can only run the code for certain conditions. I wanted an “always on” system, which would mean further splitting up my code but not in a logical way. Sending would work, recieving would need to be handled a different way, probably by a separate server or process.

I would encourage you to explore Netlify functions for yourself and decide if they (can or would) work for you. They didn’t meet my needs, but they might be exactly what you are looking for. Plenty of people are happy using them!

Moving to Microservices

Once I decided that lambda functions were not right for me, I began to reassess the best way to reduce complexity, eventually deciding upon a Microservices approach.

I decided upon separating my concerns in to the following:

• A service to handle Micropub (recieving content and formatting it)
• A service to handle Webmentions (sending and recieving)
• A service to syndicate content

I could go further and have 2 separate services to send and recieve webmentions etc. My experience with Microservice arhcitecture has been that it pays to break them down and keep them simple within reason. You don’t want to break down services so small that you end up with a distributed system. The aim over a Microservice approach is to try to make it so that each service is not dependant on the other service working and does not possess knowledge of it’s internal working.

By cutting up the responsibilities this way I can have each service work independantly, and where sending content is concerned (syndication and webmentions) I can provide a feed on my static site, that is simply checked periodically triggered by a webhook for content to send.

A further benefit to this approach is that in the future I intend to try and bring in content from my other services such as Scrobbles from LastFm and more. I may add another service to manage this content and insert it in to my website. When I get to adding this the impact against the other services should be zero in both performance and complexity. Life is alot simpler.

My New Microservices Architecture

This is my new architecture (you can click on the image for a bigger version).

At first glace it could seem a little overwhelming, but it is simpler than it first appears. Content outlined in dotted lines are the microservices. It is easiest to explain by stepping through a few basic examples.

How I Micropub a note and syndicate to Twitter

1. Browse to a Micropub service
2. Sign in using Indie Auth
3. Type a note
4. Select the syndication target (in this example it would be Twitter, but I can select multiple if I wish).
5. Hit the post button
6. The content is sent to my Micropub service (Mastr-Cntrl)
7. Mastr-Cntrl receives the code, determines the Micropub type and passes it to the correct formatter.
8. The content is POST’ed in to my vincentp.me Github repo as a Markdown file
9. Netlify detects a change and rebuilds the website and deploys it
10. When the website is regenerated there is a JSON feed that is updated with any new entries for syndication. Syndication items are detected by using frontmatter in the markdown post.
11. My MC-Syndication service periodically checks the JSON feed for new items is triggered by a webhook on publish (If it finds an item with a datetime more recent than it’s last-sent datetime this is a new item).
12. MC-Syndication detects the new item(s), and POST’s each to their target. Targets are specified inside MC-syndication, so I only need to use a short-hand (This also means if details change at a later date I only need to update one location).
13. Once it has finished POSTing items it gets the most recent date and updates it’s last-sent markdown file to the same date.
14. MC-Syndication POSTs the update to the last-sent markdown file in to the MC-syndication repo.

As you can see in this example Mastr-Cntrl handles the authentication, presents the syndication targets and recieves the content. It formats the content and POST’s it in to my repo. It has no knowledge over syndication whatsoever.

MC-Syndication doesn’t have any knowledge of the Micropub part of the process, the whole time it just waits for the webhook to be triggered periodically pings and then checks my JSON feed on vincentp.me, if it “sees” something to do, it takes the JSON feed content and POST’s it to the target and updates it’s last-sent datetime in Github.

If Mastr-Cntrl stopped working, then my syndication service is unaffected. Similiary if there was some issue syndicating (perhaps Twitter denied access). Then my website is unaffected. Additionally, if I decided to stop syndicating in the future, I could just switch the Mc-Syndication service off and nothing else if affected.

Sending a Webmention

1. Create a post with a frontmatter property “webmentionTarget” the value is the URL I am targeting
2. Publish the post
3. When the website is regenerated there is a JSON feed that is updated with any new entries for sending webmentions
4. My MC-Webmention service periodically checks is triggered by a webhook to check the JSON feed for new items (If it finds an item with a datetime more recent than it’s last-sent datetime this is a new item).
5. MC-Webmention detects the new item(s), and POST’s each to Telegraph for sending.
6. Once it has finished POSTing items to Telegraph (avoiding the rate limit) It gets the most recent date and updates it’s last-sent markdown file to the same date.
7. MC-Webmention POSTs the update to the last-sent markdown file in to the MC-Webmention repo.

As you can probably tell this pattern is very similar to the syndication example above. The MC-Webmention service has no understanding other than to check for a new item in the feed. When it finds one, take the content and POST it. For the webmention service this is much simpler since I use Telegraph to do the sending there is only 1 target.

Recieving a Webmention

1. When someone sends me a webmention, it is redirected to https://webmention.io/.
2. The Webmention is then sent to MC-Webmention from https://webmention.io/.
3. MC-Webmention recieves the content, turns it in to a JSON file and POSTs it in to the correct location based on its Webmention Type.
4. My vincentp.me website then regenerates and outputs the webmention on the correct page.

The process can take a a few minutes to filter through, it works seamlessly and is relatively unchanged from my first implementation in the Monolith. The only difference now is that the Webmention Microservice is responsible instead of a non-related Service.

New Architecture Summary

Mastr Cnrtl is now a single-use microservice recieving Micropub content, formatting and POSTing to my Github.

I have built the Webmention service this is recieving webmentions from Webmention.io. It also sends webmentions from my feed successfully.

The Syndication service is completed with one integration (Twitter). Other servces are in the works such as syndicating bookmarks to Pinboard and it will be easy to extend to other services moving forward. The code is already in place on my website for the feed to except multiple syndication targets.

I feel really confident for the first time that I have a strong base to build upon for the redesign work, which should happen quicker and more incrementally than the wholesale architecture work that has taken much longer to get right.

The hardest part is done and I can’t wait to get started on that redesign. I have a wealth of ideas that I am excited to dig in to and share soon.