Mutation-Analysis-Plugin

SonarQube Server / Community Build Plugin Development
1

Hi all,

I have released version 1.2 of the mutation-analysis-plugin this morning under LGPLv3, download+source available on github: mutation-analysis-plugin

The previous releases have been developed under a commercial license/closed source of my company. But since the business outcome was below expectation, I decided to open source it, so that more people can benefit from it’s features and hopefully I’ll get feedback (bugs, feature requests…) from a broader project usage.

Now to the more complicated aspects.
The plugin originated in an early fork (+3yrs) of the sonar-pitest plugin but ended up in a complete rewrite of the plugin. Version 1.2 is the 3rd release.

Now, both plugins have kind of overlap, with sonar-pitest offering “just” a subset of the mutation-analysis-plugin. Additional features are individual rules per mutant (including documentation), a set of additional metrics and (experimental) maven-multi-module support, and a built-in quality profile, but it offers no Kotlin support.

For me the question is now, how to proceed?

For one, I would like to see the plugin being listed in the market place, which makes it more visible and installation more convenient to a wider audience. As far as I’ve understood, the plugin must meet the requirements and has to undergo a check by SonarSource staff. So what would be the next step in that direction?

But I would also like to team up with the developer(s?) of the sonar-pitest plugin efforts as it doesn’t make much sense to me to have two separate plugins with overlapping functionality.
I’ve seen the ownership of the plugin went to Vinod Anandan, who is not yet in this forum, and @Brad Flood is occasionally pushing minor updates to the plugin.
My suggestion would be, to end the sonar-pitest plugin lifecycle with version 0.9 or have a final release of 1.0 or and then switch over to 1.2 of the mutation-analysis-plugin.

Any thoughts, questions or recommendations on this are much appreciated?

Regards,
Gerald

1 Like
2

Hi Gerald,

To start the process on getting your plugin added, you just have to ask for it.

I might have taken this message as an implicit request, but you seem to be choosing from 2 diverging paths, so I’ll wait for an explicit request.

Ann

3

Hi Gerald,

We have no real objection to ceasing work on sonar-pitest-plugin and putting our effort into the mutation-analysis-plugin. The code looks good and I would welcome the improvements you’ve made since forking the project a few years back.

My only concern is related to transfer of the project if at some point your company stopped supporting it. We do need an actively maintained pitest plugin for use on client engagements.

It appears your intent is to bring this under the SonarQube Community guidelines. If I understand the community process, SonarSource would ultimately have the ability to transfer ownership in the event your company stopped being responsive.

Ann, can you confirm that there is (or can be) some control in place to guard against this project being abandoned?

If there is such a control in place, and if you intend to continue licensing under LGPL v3, we have no objection to the mutation-analysis-plugin replacing the sonar-pitest-plugin as the preferred (and only) pitest plugin in the marketplace.

And if this is the case, I look forward to working with you if you are ever in need of help!

Brad

4

Hi Gerald,

I have the same view as Brad. We just want to make sure that the plugin will be maintained and the community can depend on it.

Thanks,

Vinod

5

Hi all,

I searched the plugin guidelines and found nothing regarding ownership transfer to/by SonarSource, apart from the deprecation notification.

I see two worst case scenarios:

  • Me or my company (it’s a one-man-show) is suddenly getting out of business for any unfortunate & unforeseen event and the repository is becoming effectively abandoned. In that case it might be hard to actively transfer the ownership without a deputy.

  • Me or my company decide to abandon the plugin and close source future versions of it. In this rather unfriendly case it’s up to the community to keep it running & maintained on a fork (same as above) with respect to the license.

  • A third scenario, not really worst-case, is that me/my company looses interest in the plugin and forget to deprecated it or to transfer ownership. But the end result is the same as above.

For all cases, a pragmatic approach would be, to fork the project (at that point or in advance) and update the market place pointer to the new repository location. Any owner of a forked repository has full rights on the fork, with (legal) limits being the LGPLv3 + the copyrights. I assume the biggest limitation is to not being able to dual-license / commercialize it due to the copyright. So no ownership transfer of the Github repository is needed.

So in all cases, for the end-users/maintainers/committers it’s not important who is the owner of the original repository, but what the community & SonarSource agreed upon, which of the forked repositories is the master to get the releases from.

Btw, the above mentioned risks also apply to SonarSource/SonarQube itself, with limitation, that it’s not only depending on a single person.

I think the LGPLv3 is the key to business continuity and community engagement for all usages or client engagements (and it’s basically one of the reasons, I open sourced it, because customers asked for how to ensure business continuity)

The alternatives would be

  • contribute it to a foundation (Eclipse, Apache), which I think is a bit of overhead for such a small project + I doubt we meet the minimum requirements (Eclipse basic fees are simply to high for a small company like me and Apache projects kind of need a critical mass of contributors)
  • own it by SonarSource. If I remember correctly that was the case before and now has become the nursing home of deprecated/abandoned projects before ownership of any project is transferred to new owners.
  • have a newly and dedicated GitHub organization, which is the team of committers. Downside is, it’s no legal entity that can hold a copyright + I (as a company) have less visibility (which is kind of crucial if you run a one-man consulting company - sorry, but we gotta pay the bills somehow :wink: )
  • me/my company grants admin rights to any non-company member, which allows a transfer of the ownership, but exposing my company to the risk of unwanted and irrevocable loss of intellectual property (loss of copyright) as there are no legal ties to any non-company members.

I wonder, how is it done with the current sonar-pitest plugin, it’s owned by you, Vinod, right? How would you handle transfer of ownership in case you, Vinod, abandon the plugin for any reasons?

But I’m also curious what’s SonarSource take on this, because this question is not special to this plugin, but to all individually held plugins.

Regards,
Gerald

6

Hi all,

Gerald has given a pretty thorough answer, but I’ll weigh in on a couple points (mostly because I have the feeling some of you are waiting for “official” word).

First, getting a plugin into the Marketplace in no way cedes repository ownership or management to SonarSource. It’s true that the SonarQubeCommunity organization in GitHub holds some abandoned plugins. Some of those are there as the remnant of old processes, and others because that’s where we move our plugin repos when we decide to stop maintaining them. Active maintainers of any of those repos can ask for a transfer of ownership, and we’ll happily comply. (Ditto anyone who would like to take up maintenance of an abandoned repo!)

Second, we have nothing in place to prevent plugin abandonment. I think the list of deprecated plugins bears witness to that. That said, I think what Gerald outlined for the “what if you disappear” scenario is pretty fair and accurate. If the plugin where to stop being maintained and another community member were to say “I’ve tried to contact Gerald and he’s not responding. I’d like to take the plugin over, could you switch to my fork” … I would probably wait a little while to give Gerald a chance to object. I might walk through the requirements again just to make sure no one’s trying to pull a fast one. And then if it seemed reasonable, I’d probably make the switch. Of course, it’s all speculative at this point so this is not a promise of future behavior. :slight_smile:

Finally, since we are talking about pointing an existing Marketplace plugin to a new repo, I probably will walk through the requirements again just to make sure, but since the existing maintainers seem to agree with this, I don’t see any other obstacles on my side if such a request is made.

Ann

7

Hi Gerlad,

I am an OWASP volunteer, I maintain the “OWASP SonarQube” project.

I found the importance of mutation testing as well as the value that this will bring to security. That’s the reason I’ve decided to include the pitest plugin in the OWASP SonarQube project and contacted the maintainers to accept the pull request to keep up the compatibility. Unfortunately, no one replied or accepted the pull request and eventually I had to take ownership of the project with the help of Brad.

We were planning to move this project to OWASP. OWASP has it’s own GitHub Organization and a proper project management structure. We don’t necessarily need to move this project under OWASP. As you may know many people are depend on this plugin.

If you can collaboratively work and consider the pull requests and roadmaps inputs for the benefits of the community, it will be useful. If you want we can also set up a Special Interest Group (SIG) for this purpose.

Please let me know your thoughts,

Vinod

8

Hi Vinod,

I’m coming more from tester perspective, where mutation analysis has a lot to offer, too, not just in a security context. So I’d rather keep it separated from OWASP.

But in order to keep the plugin development running and trying to avoid being a bottleneck for PRs or single point of failure, and given we all find kind of an agreement on and how to coordinate the work ahead, I’d suggest I’ll grant you and Brad write permission on the repository, so you should be able to push, merge PRs and create releases / tags, too, not just me. The admin permissions, however, I would rather prefer to keep in the company.

Would that be acceptable for you and Brad?

@Ann, as the mutation-analysis-plugin does not yet cover all functionality of the sonar-pitest plugin (the Kotlin support in particular), I’d suggest to keep the sonar-pitest plugin until we added that missing functionality and Vinod & Brad agree to deprecate and remove it, and have the mutation-analysis-plugin separately, so users have some time to phase over.
So if there are no further objections, I’d like to explicitly request to add the plugin to the marketplace :slight_smile:

Gerald

9

Hi Gerald,

Request received and queued. At this point it looks like I’ll get to this sometime next week.

Ann

10

Hi Gerald,

Your suggestion is fine with us.

Thanks,

Vinod

11

Hi Gerald,

The SonarCloud badge in your readme links to … the project you forked from? (https://sonarcloud.io/dashboard?id=org.sonarsource.pitest%3Asonar-pitest-plugin) At any rate it looks like it belongs to Vinod, not you. Can you point me to your SonarCloud project?

In testing your plugin, I see that you set your profile as the default, and it ends up that way in what I guess is a last-saved-wins manner. You’re going to need to remove that default-ness.

After doing the quickstart, I got this in my analysis log:

[WARNING] No XML PIT report found in directory /home/ganncamp/workspace/sonar-rule-api/target/pit-reports !
[WARNING] Checkout plugin documentation for more detailed explanations: http://docs.codehaus.org/display/SONAR/Pitest

You probably want to update the URL in the warning (Codehaus has been dead for several years).

But also, how do I actually make this work? I’ve got a target/pit-reports directory, and the dated subdir has an index.html, but there’s nary an xml file in sight, and an analysis with only your rules yields 0 issues.

Ann

12

Hi Ann,

  • Badge: fixed - the image source was correct, but not the link itself - stupid copy&paste mistake :slight_smile:
  • Log: link fixed, pointing now to the github repo
  • Quality Profile: I removed the “setAsDefault”

I released a fix version 1.2.1 here https://github.com/devcon5io/mutation-analysis-plugin/releases/tag/v1.2.1

Regarding the reports: you need to enable XML report generation, apparently that is not generated by default from pitest. That’s the same for the sonar-pitest plugin. The actual mutation analysis is done by pitest. The sonar plugin has a sensor for the (xml) reports generated by pitest.
XML report generation is enabled by adjusting the pitest plugin configuration section in your pom. I updated the documentation accordingly to be more precise.

Enabling XML report generation during pitest analysis:

<plugin>
	<groupId>org.pitest</groupId>
	<artifactId>pitest-maven</artifactId>
	<version>1.4.0</version>
	<configuration>
		<targetClasses>
			<param>ch.devcon5.sonar.plugins.mutationanalysis.*</param>
		</targetClasses>
		<targetTests>
			<param>ch.devcon5.sonar.plugins.mutationanalysis.*</param>
		</targetTests>
		<timestampedReports>
			false
		</timestampedReports>
		<outputFormats>
			<param>XML</param> <!-- <<< this is important for the plugin -->
			<param>HTML</param>
		</outputFormats>
	</configuration>
	<executions>
		<execution>
			<goals>
				<goal>mutationCoverage</goal>
			</goals>
			<phase>verify</phase>
		</execution>
	</executions>
</plugin>

The mutation-analysis-plugin itself is a fully working example, run the mutation analysis using the pit maven profile via

mvn clean install sonar:sonar -Ppit

One word of warning: all surviving mutations are counted as “bugs” and therefore may deteriorate an otherwise A-grade reliability rating :slight_smile:

Vinod and Brad should have write permissions on the repository by now.
I won’t do any code related stuff in the next two weeks because of vacation time, but will check this forum from time to time.

13

Hi Gerald,

Thanks for the update to the docs; I got it to work. \o/

This new release looks good, and I’ve added the plugin to the Marketplace.

I do have some advice tho. Up to you whether to take it or leave it. :slight_smile:

For testing, I crafted a profile that contains all of and only your rules. Yes, I know that includes some experimental rules you left off by default on purpose. I wanted to see how bad “experimental” really was. :slight_smile:

What I notice is that at many places in the code I have seemingly duplicate issues: same file, line, and message. In reality, they come from different rules (and now that I look closely I see that one of each pair is a bug and the other a code smell), but that was hard to discern at first. If possible, I recommend you vary the messages so that other users don’t have the same impression.

Regarding your rule remediation functions, I’m looking at mutation.analysis:mutant.survived and seeing a Linear with Offset function with a base of 15 min + 15 min to kill the mutant(s). Since this rule raises an issue on a single call whose removal didn’t seem to affect the outcome, then this will always come to a total of 30 min. Simpler then to make it a Constant remediation of 30 min.

On the other hand mutation.analysis:mutant.coverage is a great use of Linear + Offset. However, you probably want to update the description of the linear value slightly to something like “per mutant to kill” so that in the rule it reads “Linear with offset: 15min + 15min Effort per mutant to kill”

Ann

14

Hi Ann,

That is true, there are redundant rules, one is a non-specific rule “Surviving Mutant” and the other is a specific rule “Surviving Mutant XYZ”, if BOTH are enabled, they appear on the very same position. What’s not correct is that one is a Code Smell (the non-specific version) and the other a Bug (the specific version). I think that is incorrect and should be consistent. However, it’s not recommended to enable both at the same time. I think, I have to improve the documentation a bit in that regard.
Further, I think it makes some sense to deprecate the non-specific mutation rules, too, as the specific rules are a better replacement.

Regarding the debt remediation:
As far as I have understood - and please correct me if I’m wrong - the linearWithOffset function is something like n*x+m
n,m are time values (like 15min), m is fixed (15min), n is configurable (default value is 15min)
x is the gap specified when the issue is created. The base value is configurable (default 1.0), for the coverage threshold it’s x*M, whith M being the number of missing killed mutations.

Does this make sense to you?

So there should be a difference between what is currently implemented and a constant remediation of 30min, depending on what the user configured in the plugin settings. If nothing is configured, the defaults equal a constant remediation of 30min.

15

Hi Ann,

the plugin seems not to be installable on SonarQube 6.7.x version using the market place. Versions 7.x are fine.

The plugin is built against the 6.7 API and is compatible with the 6.7.x LTS and the 7.x versions.

It seems the update-center.properties only lists 7.x version
mutationanalysis.1.2.1.sqVersions=7.1,7.2,7.2.1

Would you be so kind as to look into the marketplace configuration on your side?

There is also an open issue regarding this problem

16

Hi Gerald,

Looking back, it seems that you didn’t specify compatibility so I guess I went with what I tested on. I’ve updated the Marketplace files.

Regarding remediation, your analysis is correct.

Ann

1 Like
17

Any new version of this plug-in is planned? The latest version 1.8 is not compatible with SonarQube 2025.1 as it was built on Java 11 whereas it is required to have at least Java 17 now.