Where should refactors go? feature/, bugfix/ or hotfix/ branches?

refactor/

Definitely not hotfix since it's for fixing bugs on master.

Then it depends on what your refactor purpose is: is it to fix a bug? Then bugfix. Or to improve something? Then feature.

Of course you can always choose to diverge from gitflow and use refractor like @MrCSharp suggested.

@Commodore knows the stuff.