Custom java rule not persisting issue (getting InMemorySensorStorage instead of DefaultSensorStorage

juanpablo-santos · July 24, 2023, 9:28am

Hi all, I’ve been trying to deploy a java custom check, and for that I’ve began with the source code of SuppressWarningsCheck, modifying it slightly so instead a comma separated list it could also read a regular expression. The code is the same sans the aforementioned change.

I’ve got my rule registrar, plugin defintiion, etc. The unit tests are also doing fine.

However, when launching a mvn sonar:sonar, I’ve stumbled upon a weird issue, the reportIssue method isn’t reporting anything. Following the chain call, it all goes ok until storage.store(this) on line 151 of DefaultIssue.java: storage class corresponds to InMemorySensorStorage, instead of DefaultSensorStorage, which is the class that the original check gets injected, which also registers the issue so that SQ is able to show it.

So, clearly there’s something missing on my plugin, but I fail to see what and/or where. The issue happens both on a local / corporate SQ 10.1 CE server.

FTR I’ve been following https://github.com/SonarSource/sonar-java/blob/master/docs/CUSTOM_RULES_101.md#rule-activation in order to get the plugin running.

angelo.buono · July 25, 2023, 12:03pm

Hello @juanpablo-santos,

Thanks for reaching out; I’m going to help you by describing quickly the steps I do to create and add a custom plugin to a local SonarQube instance:

enter the sonar-java/docs/java-custom-rules-example/ directory and work from there
implement YourRule.java in src/main/java/org/sonar/samples/java/checks
define the YourRule.java with test samples in src/test/files
define the YourRuleTest.java, for unit tests, in src/test/java/org/sonar/samples/java/checks`
define the Rule Metadata in YourRule.html and YourRule.json in src/main/resources/org/sonar/l10n/java/rules/java
add YourRule.java to the getJavaChecks() method in src/main/java/org/sonar/samples/java/RulesList.java

Now, you are ready to “deploy” your plugin! All you need to do is the following:

generate the jar for java-custom-rules-example for your version (i.e. mvn clean install)
copy the generated jar (from the target dir ) to YOUR_LOCAL_SONAR_QUBE/extensions/plugin
restart SonarQube

Note: the java-custom-rules-example plugin may have a different version from the sonar-java-plugin in your SonarQube instance. In such a case, you will find a warning similar to the following in the SonarQube logs:

WARN  ce[][o.s.s.p.PluginRequirementsValidator] Plugin Java Custom Rules [javacustom] is ignored because the version 7.22.0 of the required plugin [java] is not installed

To fix this issue, you will need to update the sonar-java-plugin:

enter the project root directory, i.e. sonar-java/, and work from there
generate the jar for sonar-java-plugin for your version (i.e. mvn clean install)
remove the existing sonar-java-plugin from YOUR_LOCAL_SONAR_QUBE/lib/extensions
copy the generated jar (from the target dir in sonar-java) in YOUR_LOCAL_SONAR_QUBE/lib/extensions
restart SonarQube
optionally take a look at Install a plugin

I feel you might have missed steps 7-9, or you stump into the issue with the incompatible versions of plugins. You should be able to overcome them by following the above instructions.

Cheers,

juanpablo-santos · July 25, 2023, 12:24pm

Hi Angelo,

yeah I’ve been following that steps. In fact the plugin gets registereed, our properties appear, the rules appear, everything is there. Even the analysis executes the check, so everything is there. The problem is the issue is not persisted. Enabling debug and going through the stack call, the only difference with the original rule, is that the original rule gets a DefaultStorageSensor, whereas the new rule gets an InMemoryStorageSensor.

This (corporate) plugin also contains other custom Sensors, to dos some checks on specific files, and they are able to register issues (although in these cases we register issues througn issue.at(…)…save()). So the plugin seems to be ok, except for the java rule.

Some more checks display additonal weirdness, the rule is basically like this:

public void visitNode(Tree tree) {
        AnnotationTree annotationTree = (AnnotationTree) tree;
        initAllowedWarnings();

        if (isJavaLangSuppressWarnings(annotationTree)) {
            if ((allowedWarnings == null || allowedWarnings.isEmpty()) && allowedWarningsPattern == null) {
                reportIssue(annotationTree.annotationType(), "Suppressing warnings is not allowed"); //////////////////////////////// doesn't register issue InMemoryStorageSensor injected
            } else {
                List<String> suppressedWarnings = getSuppressedWarnings(annotationTree.arguments().get(0));
                LOG.warn( "{} -> {}", suppressedWarnings, warningsCommaSeparated );
                List<String> issues = suppressedWarnings.stream()
                                                        .filter(currentWarning -> filterWarnings(allowedWarningsPattern, allowedWarnings, currentWarning))
                                                        .collect(Collectors.toList());
                if (!issues.isEmpty()) {
                    LOG.warn( "{} -> {}", suppressedWarnings, issues );
                    StringBuilder sb = new StringBuilder("Suppressing the '").append(String.join(", ", issues))
                                                                             .append("' warning")
                                                                             .append(issues.size() > 1 ? "s" : "")
                                                                             .append(" is not allowed");
                    reportIssue(annotationTree.annotationType(), sb.toString()); //////////////////////////////// doesn't register issue InMemoryStorageSensor injected
                    addIssue( 1, sb.toString() ); // does register the issue at that line, wat?
                }
            }
        }
    }

we can use addIssue to successfully register the issue, but we can use only from line 1 to the line previous to where the annotation is. I.e., the forbidden annotation is on line 15, we can use addIssue up to line 14. Given that the annotation might be in it’s own line, preceded or not by other annotations, we’d like to register the issue on the precise line if possible.

angelo.buono · July 25, 2023, 2:08pm

Hi @juanpablo-santos,

Thank you for clarifying the problem; I have a few questions:

Do the unit tests correctly report the expected issues at the expected location?
What did you debut exactly? The unit tests of your custom plugin?
Could you provide the log, the stack call, and any other information that can help with the investigation?

It is preferred to use reportIssue since it will get the exact position from the Tree element. I can’t tell you why this is not happening in your case; more investigation is required once you provide more details.

juanpablo-santos · July 25, 2023, 2:55pm

Hi @angelo.buono !

yes, the unit tests run fine, in fact they’re the same as in SuppressWarningsCheckTests, plus a few ones covering the new modifications.

As for the debugging, I launched an mvnDebug sonar:sonar, and attached a remote debug session inside the IDE, placed a few breakpoints and went down the route to see what reportIssue was doing internally.

Regarding the log, it doesn’t show anything unusual as the analysis is done correctly, whatever happens or why that sensor is injected, happens and is swallowed silently. I suppose that, as far as the sonar:sonar analysis is concerned, the rule is reported. Is just that, the implementation where it is reported, doesn’t persist the issue.

Comparing the java rule part of our plugin with the 101 one, the only difference I can see is the rules registration, whereas the 101 version uses some json files to define the rule, our RuleDefinition ends up doing something like this:

void registerJavaRepositoryRules( final Context context ) {
        final NewRepository newJavaRepository = context.createRepository( JAVA_REPO_KEY, "java" )
                                                       .setName( "Corporate Java Rules" );
        newJavaRepository.createRule( RULE_JAVA_SUPPRESS_WARNINGS.rule() )
                         .setActivatedByDefault( true )
                         .setName( "Detects the @SuppressWarnings annotation with any of the following parameters all, java:* or squid:*" )
                         .setSeverity( Severity.BLOCKER )
                         .setTags( "corporate", "java" )
                         .setType( RuleType.BUG )
                         .setHtmlDescription( html.extractHtmlDescriptionFor( RULE_JAVA_SUPPRESS_WARNINGS.rule() ) )
                         .createParam( "listOfWarnings" ).setType( RuleParamType.STRING );

        newJavaRepository.done();
    }

(we register other non java rules the same way) given that the rule is correctly registered and used, I suppose that this is ok. Everything else follows the instructions for the 101 rule. One could thing I could try is to inject the SensorContext in our check, and see what comes back. Also if I get enough time perhaps I could try to upload here a minimal plugin sample that showcases this behaviour, but I’m not sure I’ll be able to do it before september (a lot of things to finish before holidays in august)

HTH pinpointing where the issue might be

angelo.buono · July 27, 2023, 3:11pm

Hello @juanpablo-santos, can you tell me which version of sonar-java-plugin and sonar-qube are you using in the pom.xml?

There was a recent update to fix a compatibility issue, your problem may be related to it.

juanpablo-santos · July 27, 2023, 5:48pm

Hi @angelo.buono! thanks for the feedback

we’re using SQ 10.1; as for the plugin, it uses the following dependencies:

org.sonarsource.api.plugin:sonar-plugin-api:9.17.0.587:provided
org.sonarsource.sonarqube:sonar-plugin-api-impl:10.1.0.73491:provided
org.sonarsource.java:sonar-java-plugin:7.20.0.31692:provided
org.sonarsource.java:java-checks-testkit:7.20.0.31692:test

IIRC, those were the versions used inside that version of SQ

Last, the version used for the sonar-packaging-maven-plugin is 1.21.0.505

cheers

juanpablo-santos · July 31, 2023, 10:45am

Hi @angelo.buono

I’m attaching a minimal sample that showcases the issue:
storage-sensor-issue.zip (23.5 KB)

Note that the rule in there is basically java:s1309 sans a few modifications

please do let me know if that helps to pinpoint the issue (after today, most probably I won’t be able to retake this until september).

HTH,
juan pablo

angelo.buono · August 29, 2023, 1:54pm

Hello @juanpablo-santos, thanks for the sample code. It is very useful to understand and reproduce the problem you are experiencing.

I will need more time to investigate the issue that is related to the logic in org.sonar.java.filters.SuppressWarningFilter that is suppressing the issues detected by your rule. In order to confirm it I tested by disabling the filter and I can see the expected issues reported correctly.

I will investigate more the filtering and I will come back to you as soon as possible.

juanpablo-santos · September 15, 2023, 10:44am

Hi @angelo.buono!

is there anything that we can do to workaround the issue from the plugin side?

Out of curiosity, exactly what is happening with the SuppressWarningsFilter, how does it affect the plugin execution? IIUC, the issue was that the rule was getting a SensorStorage mock, instead of the real one, before it’s execution, so on execution issues weren’t persisted; but, I’ve understood correctly, SuppressWarningsFilter suppresses the issue after rule execution (?).

thx in advance!

angelo.buono · October 12, 2023, 2:24pm

Hello @juanpablo-santos, I recently tried to reproduce the issue using the SonarQube Community Edition 10.1 with the latest version of sonar-java-plugin, and I was able to analyze correctly the project you shared:
storage-sensor-issue.zip (10.3 MB).

I did slightly modify it to be able to see the expected issues when analyzing it.

Could you please try to build sonar-java-plugin and use that version of the sonar-java-plugin.jar? It will be available in the target folder within the sonar-java-plugin module.
Alternatively, you can try using the latest release sonar-java-plugin-7.25.0.32245.jar, but I haven’t tried it.

In order to upload the sonar-java-plugin you simply need to:

Remove the existing sonar-java-plugin.jar from <your_sonarqube_folder>/lib/extensions
Copy the sonar-java-plugin.jar to <your_sonarqube_folder>/lib/extensions
Restart SonarQube

juanpablo-santos · October 28, 2023, 12:30pm

Hi @angelo.buono

apologies for the delay answering back. Glad this is fixed on the sonar-java-plugin side As the current sonar-java-plugin release is 7.27 now, I imagine that the fixed code should be bundled on a release right now (we can’t use non-release plugins on our instance, due to some requirements, we expect them to be “released”, that is f.ex., available from maven central).

Seeing that latest SQ (10.2.1 as of now) contains the 7.24 version of the java plugin, perhaps we’ll wait until the SQ server is bundled with the latest java version to remove the workaround in our plugin.

Out of curiosity, does this issue has an associated jira ticket or something? so we can track what was happening and in what version was the fix pushed, so we can use that on our SQ instance.

thanks for the continued support!

angelo.buono · October 30, 2023, 7:55am

Hello @juanpablo-santos, there is no Jira because no explicit change was implemented for this issue. I was resuming the investigation and I noticed that the issue was not reproducing anymore.

Note that you can install any released version of sonar-java-plugin in your sonar qube, there is no need to wait for the next release of sonar qube.

I wish you good luck then.

Cheers

system · November 6, 2023, 7:55am

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.