Learn More
Resend Email
8
pseudonim
4y

If you've ever tried using Go plugins raise your hand.
If you've ever tried doing plugins in Go, raise your hand.
If you think that the following rant will be interesting, raise your hand.
If you raised your hand, press [Read More]:

This is a tale of pain and sorrow, the sorrow of discovering that what could be a wonderful feature is woefully incomplete, and won't be for a very long time...

Go plugins are a cool feature: dynamically load pre-compiled code, and interact with it in a useful and relatively performant way (e.g. for dynamically extending the capabilities of your program). So far it sounds great, I know right?

Now let me list off some issues (in order of me remembering them):
1. You can't unload them (due to some bs about dlopen), so you need to restart the application...
2. They bundle the stdlib like a regular Go binary, despite the fact that they're meant to be dynamic!
3. #2 wouldn't be so bad if they didn't also require identical versions of all dependencies in both binaries (meaning you'd need to vendor the dependencies, and also hope you are using the right Go version).
4. You need to use -trimpath or everything dies...

All in all, they are broken and no one is rushing to fix it (literally, the Go team said they aren't really supporting it currently...).

So what other options are there for making plugins in Go?

There's the Hashicorp method of using RPC, where you have two separate applications one the plugin, one the plugin server, and they communicate over RPC. I don't like it. Why? Because it feels like a hack, it's not really efficient and it carries a fear of a limitation that I don't like...

Then we come to a somewhat more clever approach: using Lua (or any other scripting language), it's well known, it's what everyone uses (at least in games...). But, it simply is too hard to use, all the Go Lua VMs I could find were simply too hard to set up...

Now we come to the most creative option I've seen yet: WASM. Now you ask "WASM!? But that's a web thing, how are you gonna make that work?" Indeed, my son, it is a web thing, but that doesn't mean I can't use it! Someone made a WASM VM for Go, and the pros are that you can use any WASM supporting language (i.e. any/all of them). Problem inefficient, PITA to use, and also suffers from the same issues that were preventing me from using Lua.

Enter Yaegi, a Go interpreter created by the same guys who made (and named) Traefik. Yes, you heard me right, an INTERPRETER (i.e. like python) so while it's not super performant (and possibly suffering from large inefficiency issues), it's very easy to set up, and it means that my plugins can still be written in Go (yay)! However, don't think this method doesn't have its own issues, there's still the problem of effectively abstracting different types of plugins without requiring too much boilerplate (a hard problem that I'm actively working on, commits coming soon). However, this still feels to be the best option.

As you can see, doing plugins in Go is a very hard problem. In the coming weeks (hopefully), I'm going to (attempt to at least) benchmark all the different options, as well as publish a library that should help make using Yaegi based plugins easier. All of this stuff will go (see what I did there 😉) in a nice blog post that better explains the issues and solutions. But until then I have some coding to do...

Have a good night(/day)!

rant

go plugins

go

golang

why

plugins

yaegi

long rant

etc.
Favorite
Ranter
Comments
  • 4
    4y
    The thing with Unloading is actualy very justified, as this makes it easier to make hard guarantees on how program and Plugins behave at runtime.
    It also prevents errors, caused by Plugins that were developed without unloading in mind.
    That's literally the reason, why dlclose is a No-Op in musl-libc.
  • 0
    4y
    @metamourge but isn't that just a race condition (which happens to have rather catastrophic consequences)?
  • 0
    4y
    What about writing your extension with a C interface? Annoying for sure, as each extend than has its own GC and cannot easily pass objects, but that should be your only problem than.
  • 1
    4y
    Didn't know there where go plugins yet. They where always adamant about using RPC instead of binary lib hell. Although they also voice that there is a valid case for dynamic loading, just not a solution yet. Same stance on generics. I would use.

    I would go rpc unless that is unviable than I would go the C route as @sbiewald suggested.
  • 0
    4y
    @pseudonim why would not doing anything be a race condition?

    Also points 2 and 3 are pretty much needed to prevent runtime failures from different dependency versions.
  • 0
    4y
    @sbiewald well, to put it mildly, cgo had it's own issues (I haven't really dabbled with it, so I can't tell you what they are), and I don't know C (gasp!) and I'm not willing to learn it for this.
  • 0
    4y
    @korrat I meant doing something.

    Also, regarding 2 and 3, they were "officially" known issues. The Go plugins implementation was never great.
  • 0
    4y
    @pseudonim I'm confused. Where do you see a race condition happening with dlclose?
  • 1
    4y
    @korrat the race condition would come about if the plugin is still being used is some fashion (e.g. in a goroutine) when it gets unloaded.
  • 0
    4y
    @pseudonim yes it would. The same way freeing memory when it is still in use somewhere else causes a race condition. Therefore, not being able to unload plugins eliminates bugs.
  • 1
    4y
    @korrat unless of course you allow unloading to halt until it can be determined to not be used. But that's its own discussion...
  • 0
    4y
    @pseudonim that's kind of what glibc does. There each library loaded using dlopen has a reference count. dlopen increments that count, dlclose decrements it. When the count reaches 0, the library is unloaded. But of course, with it being C that won't protect you from use-after-free type of situation or a race condition like the one you described.
  • -1
    2y
    Window feeders are a great option if you want to get up close and personal with your hummingbirds. They consist of a small plastic or glass reservoir that attaches to your window with suction cups. The birds feed by inserting their beaks into the feeding port. Window feeders are easy to use and maintain, but they can be prone to leaking and can be more difficult to clean. https://hummingbirdfeedercenter.com/...
Related Rants
devRant © 2021 Hexical Labs LLC
Privacy Policy  |  Terms of Service
Add Comment