Flyouts can only have a single separator and no nesting?

The flyout menu is, in fact, a regular InDesign menu, and if you manage its contents (menu items, menu separators, submenus etc.) as you would, for instance, a custom menu in InDesign’s main menu, then you can have all the things you want – there are no problems. In the uxp code for your panel, you just need minimal code to trigger the creation of the menu. I explained how to do it, with an attached sample plugin so anyone can see/hack the code, here:

https://forums.creativeclouddeveloper.com/t/dynamic-flyout-menus-in-uxp-plugins-for-indesign/8596/1

I came to this approach after trying but failing to control/manage the flyout menu from the uxp side. You have discovered some of the “weird issues”. There are others (all specific to InDesign plugins BTW, not Photoshop plugins). I came to the conclusion that there were too many. Thankfully they can be sidestepped.

Philip