Cannot delete Field Groups

It looks like once a field group is created, it cannot be removed. I have deleted all the field in a field group to see if that was the reason, but it still didn’t work. It gives me the green success bar and says “The following field groups were removed” - but none are listed or removed.