Currently, when you have a test class with a class level public modifier, it gets marked as bad smell. This is the same with a public modifier on test methods. (java:S5786)
On test methods, this is absolutly desired and as you do not want to have a public modifier on them.
Forcing a package private modifier on class level thought, does not improve anything and produces unwanted side effecs such as the loss of automatically file renaming on class renaming, as a single non public class is not associated to the file name anymore.
Should this rule be adjusted?