zaratri I doubt that would be an issue, particularly with so few numbers.
The admin panel UI is probably not optimized for hundreds or thousands of groups but it should still work.
On the user side I don't think the total number of groups will have any impact. The only problem I could think of is if you start assigning hundreds of groups to a single user, as that would make the API database queries and responses larger and larger.
There could also be performance issues if you start assigning a permission to hundreds of groups at once. But if you only use the groups cosmetically this is irrelevant.
The third party extensions you use are also important here, some might retrieve the groups for their own features and cause performance issues. But I doubt there would be any issues with less than 100 groups, and even a few thousands should not be an issue unless a feature was very badly implemented.