Embedded Microservices & Fireside

[this is a technical article]

Suppose you’re a software development agency working with several clients, a serial entrepreneur, or an indie hacker. You find yourself building the same thing multiple times over, each time slightly differently. Fixing an issue in one place then requires fixing it everywhere. If the fix is non-trivial, you end up forgetting a small change somewhere and move on.

Over time, you wonder if it makes sense to abstract it into a library, but realize that some things just don’t make sense as one—for example, anything that involves a database schema (like, a blog or some authentication mechanism). So you wonder if you should turn it into a microservice, but then you will need to maintain multiple microservices for each app unless you want to introduce multitenancy, which will require new authorization mechanisms, communication channels, added latency, etc. In either case, the additional complexity is not worth the trouble. Finally, you settle on making a code template, either as a gist or maybe even a separate repo. Next time you have to use it, you do quite a bit of copy-pasting and you’re good to go.

But there might be a better way.

Introducing Fireside

I was in a similar situation—I had to maintain two separate codebases that both implemented a similar e-commerce core. The functionality was deeply integrated and there was no point in making it a library or in splitting it out into a separate app. I wanted it to remain an explicit part of each codebase, but have the ability to modify both at once. Around that time, Zach Daniel started working on Igniter—an AST-aware code generation and project patching library for Elixir. I thought it would be great if I could use it to solve my problem and build a tool that would let me develop the e-commerce core in one place and magically teleport it back into my codebases. I called it Fireside (see the fire theme?).

The way it works is as follows:

1. First, develop some functionality that you would like to be available in multiple Elixir apps. In my case, it was the e-commerce core, which I called Shopifex. (I open-sourced it, so you can follow along.) Then, turn it into a Fireside component with a simple config file. For example,

defmodule MyComponent.FiresideConfig do
  def config do
    [
      name: :my_component,
      version: 1,
      files: [
        required: [
          "lib/my_component/context.ex",
          "lib/my_component/context/**/*.{ex,exs}",
          "test/my_component/**/*_test.{ex,exs}"
        ],
        optional: [
          "lib/shopifex_web/endpoint.ex"
        ]
      ]
    ]
  end
end

Refer to Creating components for more info.

2. Then, (assuming you already added Fireside as a dependency), run

mix fireside.install shopifex@github:ibarakaiev/shopifex

What this will do is:

• Bring over the dependencies listed in the mix.exs of shopifex.
• In case the dependencies themselves use Igniter, they will install whatever code is necessary to function properly (i.e. set up the Repo module). Don’t worry if you didn’t get this part.
• Bring over all the files that the component exports.
• Setup any additional configuration with Igniter, such as changing config/config.exs. (This is also optional and will probably not be necessary in a lot of potential Fireside use cases.)

     ...|
23 23   |  defp deps do
24 24   |    [

   25 + |      {:ash_state_machine, "~> 0.2.5"},
   26 + |      {:ash_archival, "~> 1.0"},
   27 + |      {:ash_admin, "~> 0.11"},
   28 + |      {:ash_money, "~> 0.1"},
   29 + |      {:ash_postgres, "~> 2.0"},
   30 + |      {:ash, "~> 3.0"},

25 31   |      {:fireside, "~> 0.1"}
26 32   |    ]
     ...|

These dependencies should be installed before continuing. 
Modify mix.exs and install? [Yn] y

...

254 |     @tag resource: resource
255 |     test "implements title!/1", %{resource: resource} do
256 |       assert function_exported?(resource, :title, 1)
257 |     end
258 |   end
259 | end
260 |

The following tasks will be run after the above changes:

* ash.codegen setup_shopifex

Proceed with changes? [Yn] y

...
Generating Migrations:
* creating priv/repo/migrations/20240816193041_setup_shopifex.exs

"shopifex" (version: 1) has been successfully installed.

Make sure to run `mix ash.migrate`.

And voilà: you have the full code within your codebase. At every step, Fireside will confirm the changes with you before applying, and it will even create the migrations for you (at least in the case of Shopifex). It will also make sure your git changes are committed or stashed before starting in case you later need to revert the changes.

The imported files will also have their prefixes replaced to match your app. For example, if your Elixir project is MyStore, Shopifex.Carts from Shopifex will become:

#! fireside: DO NOT EDIT this file. Run `mix fireside.unlock shopifex` if you want to stop syncing.
defmodule MyStore.Carts do
  @moduledoc false
  use Ash.Domain, extensions: [AshAdmin.Domain]

  admin do
    show?(true)
  end

  resources do
    resource MyStore.Carts.Cart
    resource MyStore.Carts.CartItem
  end
end

This is possible because Fireside knows your AST and is not just simple code insertion and replacement.

Updating the component

Now, for the best part. Suppose Shopifex has a new version that fixes some bugs, adds some new configuration, and requires a database migration. For complex things, it defines an upgrade/3 function as follows:

def upgrade(igniter, 1, 2) do
  igniter
  |> Ash.Igniter.codegen("upgrade_shopifex_to_v2")
  |> Igniter.add_notice("Make sure to run `mix ash.migrate`.")
end

This is an app migration (like a database migration, but for your app) that targets an isolated part of your codebase. Getting the changes is as simple as running:

mix fireside.update shopifex

Fireside will look up the original source (in this case, Github), fetch all changes, substitute files, and run the app migration, if provided. Furthermore, Fireside will check if any of the files were changed in the meantime by comparing the AST with their original AST. If they diverged, Fireside will abort the update in order to not overwrite any intentional local changes.

What’s possible?

To monolith or not to monolith?

Why settle? The Monolith vs Microservices debate is still alive precisely because both sides have a point. Larger teams benefit from the separation of concerns, while smaller teams benefit from less friction. Still, everyone benefits from less complexity. With Fireside, if you have a monolith, you can split it up into microservices while having all of them inside your monolith.

Installable tutorials

Next time, when writing a tutorial, you can make the code easily installable as a Fireside component. I personally would have really benefitted from having easily-available code for SDKs with Req: Stripe and S3 with Tigris. I might actually create Fireside components for those if I end up having to implement either of them again.

Sell your by-products

You’ve read REWORK and now want to sell your code by-products. Make them Fireside components in a private Github repo! This is apparently a profitable business. Fireside supports an --unlocked mode, where it will install the code once and not track it. This means that the users won’t get the remote updates, but often they don’t need to anyway. In such cases, Fireside will simply be a smart template installation tool.

Help the world

Help others not reinvent the wheel and open source parts of your codebase. This doesn’t need to be charitable, either: if others end up using and improving your components, you’ll get their changes!

Afterword

As a biased creator of Fireside, I am optimistic that I’m not the only one for whom Fireside solves a problem. If you believe the same, feel free to star it on Github and give it a try. My hope is that it improves developer experience and productivity in the Elixir ecosystem.

Fireside is in early stages and might not work as expected. But it is not a runtime dependency so it will surely not break anything for you.