Hey there.
Are you also delegating authentication to an external identity provider like LDAP? As noted in the documentation on Delegated Authentication:
When group mapping is configured, the delegated authentication source becomes the only place to manage group membership, and the user’s groups are re-fetched with each log in.
Meaning that if you are manually managing group membership and have ldap.group.* parameters configured, group membership will be revoked for users upon login if that group doesn’t also exist in LDAP (the same applies for other delegated authentication mechanism as well: SAML, GitHub, etc.)