He @ArcSlash,
what I would suggest would be a dropdown/select box, where the user can select either a color from the assets panel or “custom”, in which case below, an input where the user can enter a custom hex code, appears. This gives the most flexibility with a reasonable amount of work for how it is right now, in my opinion.
Hopefully, a color input will get implemented for XD relatively soon (cf. Color picker in plugin UI), in which case we’ll just be able to use that…
I hope this helps,
Happy coding,
Pablo