I’m pretty sure I saw this mentioned somewhere, but can’t find.
The problem is, that there’s no way to disable <sp-action-button/>. I mean it kind of disables the button, but it’s pretty much useless, because it still interacts with the label. E.g. if you click directly on the label, button behaves as if it wasn’t disabled.