Mutation analysis problem


(Teswar) #1

I’m reusing the PITest reports. In sonarqube developer 7.5 is working. But recently I upgraded to 7.6. Still mutation giving same error


Sonar-pitest-plugin-0.9.1 is not working with SonarQube 7.6
SonarQube 7.6 released
(Gerald Mücke) #2

Hi Venkat,

which version of the mutation-analysis-plugin are you using?
Version <= 1.3 is not compatible with Sonarqube 7.6.

But there has been a new release 1.4 that fixed compatibility issues with SonarQube 7.6. Please upgrade if you haven’t done so, yet. And let me know, if the problem still exists.

Regard,
Gerald


(Teswar) #3

Hi Gerald,

I upgraded to mutation-analysis-plugin to 1.4 in SonarQube 7.6. The above error with these plugins after upgrade.

Thanks,
Venkat


(Gerald Mücke) #4

Could you please re-run the analysis with maven debug flags on (-X -e) and attach the output log?


(Teswar) #5

I’m using Gradle. Sure Gerald. I’ll attach


(Teswar) #6

It is giving hard time. We upgraded to Sonarqube 7.6 but with PItmutation plugin sonarqube not even doing analysis. Can we degrade back to 7.5.
Having build issues.
I’m using PIT plugin ```
id “info.solidsoft.pitest” version “1.4.0”


(Gerald Mücke) #7

Hm, I don’t know Gradle very well, or have even heard of the info.solidsoft.pitest Gradle plugin before. I use Maven and the official maven-pitest plugin. The default configuration of the mutation-testing-plugin for SonarQube uses the a lot of the conventions of maven/maven-pitest.
Anyway, as the info.solidsoft.pitest seems to produce reports using pitest itself - at least that’s my impression from a quick glance at the docs - the mutation-analysis-plugin should be able to process the reports.
Nevertheless, I neither have a Gradle setup to reproduce anything, nor does your problem description contain much details for any further analysis. So unless you provide a proper problem description including any kind of log files, or even better, a simple example project (i.e. on GitHub) that I could use to verify your problem, I won’t be able to provide any kind of help.


(Teswar) #8

SonarQube Developer 7.6 is working without Mutation plugin. When we add these plugins it is not working (whether we are doing pit analysis or not).
If we add these plugins mutation 1.4 then it is not analysing anything. Task initially itself is failing. Not loading any properties.


(Gerald Mücke) #9

How do you add the plugin? Did you upgrade via marketplace or copied it into the plugins folder? Is there any old version (i.e. 1.3) still there?
The reason I ask: the version 1.4 doesn’t use the org.sonar.api.profiles.RulesProfile anywhere in the code. So the error you’re showing can not be caused by the 1.4 version of the plugin.
Could you provide

  • the list of plugins installed from the plugins folder
  • describe the way, how you installed the plugin
  • the screenshot you’re showing must be part of a log,
    • which log is it?
    • could you attach this log?

(Teswar) #10

Okay, This log is Jenkins console output.
After downloading sonarqube 7.6, the mutation plugin 1.4 added in plugins folder.
There is no older versions. All Plugins with default sonarqube 7.6 and the latest mutation plugins.


(Gerald Mücke) #11

It seems you’re using the wrong plugin.

The output shows “org.sonar.plugins.pitest.scanner.PitestSensor”, but this class exists is only the old pitest-plugin (latest version is 0.9.1). But this plugin is discontinued in favor of the mutation-analysis-plugin. In the latter, the name of the class is “ch.devcon5.sonar.plugins.mutationanalysis.sensors.PitestSensor”. You can install it via the SonarQube Marketplace or install it manually from https://github.com/devcon5io/mutation-analysis-plugin/releases/tag/v1.4.

May I ask you again which plugin version are you using - and I am talking of the mutation-analysis/pitest-plugin in SonarQube, not in Gradle, maven or whatever. You find the plugin version in the sonarqube marketplace or in the plugins folder your sonarqube installation.
Please, provide a list of the plugins in the plugins folder.


(Teswar) #12

These are the plugins I copied to SonarQube 7.6


(Gerald Mücke) #13

alright, there you are, you can safely remove the sonar-pitest-plugin-0.9.1 and configure the mutation-analysis-plugin, see here https://github.com/devcon5io/mutation-analysis-plugin/wiki/Installation


(Teswar) #14

Sure Sir,
One more thing, we are doing pit analysis on Jenkins master slave and then reusing the PIT reports to SonarQube.
So for this purpose only mutation-analysis plugin is enough?


(Gerald Mücke) #15

It’s basically the same as the old plugin, works the same way, but provides a bit more information/metrics/rules. It also processes the reports produced by pit, nothing else.


(Teswar) #16

Sure, Thank you Gerald. It is still building. I’ll give a update on this. Looks like sonar-PITEST-0.9.1 is not compatible with SonarQube 7.6


(Teswar) #17

Hi Gerald,
With mutation-analysis-plugin SonarQube is taking long time. I activated the rule

Is there any rule will take less time.
image


(Teswar) #18

Hi Gerald,

I can see discussions. Sonarsource is removed pitest plugin. The only concern with mutation-analysis-plugin, it is taking so long when compared to pitest plugin.
In pitest plugin I used the rule as follows “survived mutant”

Is it possible can we add this rule to mutation-analysis-plugin so that no dependent on pitest plugin. Please.
With mutation-analysis-plugin we have rule named “survived mutants”. this rule has aggregate, mutation, pitest and test quality where as pitest plugin has test-quality.
Which makes us saving 14 mins of time.


(Gerald Mücke) #19

The plugin does - same as the old sonar-pitest-plugin - “just” parse the reports produced by the pitest-plugin, which is reasonably fast. A single screenshot of a jenkins step that takes 24min is no proof of “slow”, you could as well have a Thread.sleep(1440000) in there (just kidding).

Mutation testing - done by Pitest (not the mutation-analysis-plugin) - takes a long time to compute, the larger your codebase, the longer it’ll take. If you disable the Pitest analysis (not the plugin), you won’t have a report, which the mutation-analysis-plugin (for sonarqube) should parse - this is faster, but you won’t get the metrics.

There are other ways of speeding up the mutation testing, like incremental analysis, limiting the scope of mutated classes, limiting the applied mutation operators, but it all depends on your project’s requirements or what kind of information you want to draw from the analysis. There is no single golden rule.

To make things a bit clearer for your, the process is as follows:

  1. you run the mutation analysis using PITest (either in Maven or Gradle)
    1. produces a mutations.xml
  2. you run the sonarqube analysis (in your Build Environment)
    1. The mutation-analysis-plugin will be downloaded from Sonarqube to your build environment (~1MB)
    2. The mutation-analysis-plugin searches for the mutations.xml
    3. The mutation-analysis-plugin parses mutations.xml
    4. Depending on your ruleset, the mutation-analysis-plugin will create issues for each mutation in the xml
  3. Sonarqube Compute Engine computes the metrics for your analysis-run (Server-Side)

The first of these three steps is by far the longest but is required in order to do the second and third.