If you've ever tried using Go plugins raise your hand.
If you've ever tried doing plugins in Go, raise your hand.
If you think that the following rant will be interesting, raise your hand.
If you raised your hand, press [Read More]:

This is a tale of pain and sorrow, the sorrow of discovering that what could be a wonderful feature is woefully incomplete, and won't be for a very long time...

Go plugins are a cool feature: dynamically load pre-compiled code, and interact with it in a useful and relatively performant way (e.g. for dynamically extending the capabilities of your program). So far it sounds great, I know right?

Now let me list off some issues (in order of me remembering them):
1. You can't unload them (due to some bs about dlopen), so you need to restart the application...
2. They bundle the stdlib like a regular Go binary, despite the fact that they're meant to be dynamic!
3. #2 wouldn't be so bad if they didn't also require identical versions of all dependencies in both binaries (meaning you'd need to vendor the dependencies, and also hope you are using the right Go version).
4. You need to use -trimpath or everything dies...

All in all, they are broken and no one is rushing to fix it (literally, the Go team said they aren't really supporting it currently...).

So what other options are there for making plugins in Go?

There's the Hashicorp method of using RPC, where you have two separate applications one the plugin, one the plugin server, and they communicate over RPC. I don't like it. Why? Because it feels like a hack, it's not really efficient and it carries a fear of a limitation that I don't like...

Then we come to a somewhat more clever approach: using Lua (or any other scripting language), it's well known, it's what everyone uses (at least in games...). But, it simply is too hard to use, all the Go Lua VMs I could find were simply too hard to set up...

Now we come to the most creative option I've seen yet: WASM. Now you ask "WASM!? But that's a web thing, how are you gonna make that work?" Indeed, my son, it is a web thing, but that doesn't mean I can't use it! Someone made a WASM VM for Go, and the pros are that you can use any WASM supporting language (i.e. any/all of them). Problem inefficient, PITA to use, and also suffers from the same issues that were preventing me from using Lua.

Enter Yaegi, a Go interpreter created by the same guys who made (and named) Traefik. Yes, you heard me right, an INTERPRETER (i.e. like python) so while it's not super performant (and possibly suffering from large inefficiency issues), it's very easy to set up, and it means that my plugins can still be written in Go (yay)! However, don't think this method doesn't have its own issues, there's still the problem of effectively abstracting different types of plugins without requiring too much boilerplate (a hard problem that I'm actively working on, commits coming soon). However, this still feels to be the best option.

As you can see, doing plugins in Go is a very hard problem. In the coming weeks (hopefully), I'm going to (attempt to at least) benchmark all the different options, as well as publish a library that should help make using Yaegi based plugins easier. All of this stuff will go (see what I did there 😉) in a nice blog post that better explains the issues and solutions. But until then I have some coding to do...

Have a good night(/day)!

I've ranted about this before, but here we go again:

Go Plugins.

I was racking my brains trying to figure out how one could possibly implement plugins easily in Go.

I had a look at using RPC, which requires far to much boilerplate to be realistic. I looked at using Lua, but there doesn't seem to be a straight forward way of using it. I was even about to go with using WASM (yes, WASM). But then I came across Yaegi ("Yet another elegant Go interpreter", you heard right: "interpreter"), Yaegi is also very easy to use.

There are a few issues (including some I haven't solved yet), including flexibility (multiple types of plugins), module support, etc. Fortunately, Traefik just released their plugin system which is based on Yaegi (same company), and I got to learn a few tricks from them.

Here's how module loading works: The developer vendors their dependencies and pushes them to a repo. The user downloads the repo as a zip and saves it to the plugins folder. I hash the zip, unzip it to a cache, and set the the GOPATH for the interpreter to be that extracted folder. I then load the module (which is defined by a config file in the folder), and save it for later. This is the relatively easy part.

The hard part is allowing for different types of plugins. It looks easy, but Go has a strict typing system, makes things complicated. I'm in the process of solving this problem, and so far it should go like this: Check that the plugin fits an arbitrary interface, and if it does, we're good the go. I will just have to apply the returned plugin to that interface. I don't like this method for a few reasons, but hopefully with generics it will become a bit more clean.

So remember rant from yesterday about Go plugins? Well turns out that a new version of Traefik was just released with plugins based on Yaegi!

"It can't all be good news" you say, suddenly paying attention. You're right, their implementation is a bit weird and I don't fully understand it... So I've decided to try to ask for help from the guy who wrote the feature, let's see if he responds...