Hi Ann!
thanks a lot for taking time to write a detailed response!
I’ll go and request changes to the docs as per points that you raised, but please let me know if anything else is required besides that.
It appears that the same issues are raised in both the json & yaml files…?
Indeed the content of the files is the same, although one is in JSON format and the other in YAML. My intention for providing samples in two different formats is to demonstrate plugin handling both, while keeping the test repo small. I’ve added a new OpenAPI example “pixi.json” to the repo.
You’ve added metrics but I suspect not aggregated them correctly. For the test project you provided, I see an Audit Score of 13 for petstore.json and the same for pestore.yaml. Aaaand a total of 13 for the project. That’s not how it’s supposed to work.
What’s the difference supposed to be between Audit Score and Audit Score (data)?
And for that matter, what is Audit Score (security) supposed to be? I get a 0 on that.
So, our Audit Score is a metric which can reach maximum 100 points, and is reduced for every issue encountered in the OpenAPI file. It is split into two sections “Security analysis” and “Data validation” which are independent and are summed to calculate total Audit Score More details
This are the scores that our platform provides, and when implementing the SonarQube plugin, we decided to report each as it’s own Metric for every scanned file, and then choose a lowest one for the project.
The reason for doing that is to allow user to configure a Quality Gate to fail if any OpenAPI with the score below of a certain threshold have been encountered.
The “pixi.json” sample I’ve added should get 26 points in security score, 69 points in data and a total score of 95.
I’m seeing Lines of Code reported for the json and yaml files, but no rules for them. This makes me wonder several things.
It seems that you’ve declared an “OpenAPI” language…?
Without giving the user the ability to manage its file extensions…? …Okay, I finally read the rest of the docs and found this under OpenAPI → OpenAPI file suffixes. As you may have guessed by now this is not where I expected to find them. I’m assuming it will be the same for other users
How do users control what rules are applied?
So, there are few points to address: OpenAPI files typically come in files with .json .yaml or .yml files. However, there are many other file types which share these extensions, therefore in order to find OpenAPI files in the project we have to examine every file with one of these extensions.
Each file which have found to have OpenAPI content is uploaded to the 42Crunch platform for analysis, and results are reported to SonarQube.
The extensions and the exclusions for the search are managed under “OpenAPI” tab as you’ve found. You mentioned that it’s not where you’ve expected to find them. Could you be more specific? Originally I wanted to place it under Languages → OpenAPI, but it seemed that the languages list was hardcoded in the UI (please correct me if I’m wrong), so it went into it’s own tab.
In fact, on closer examination, I see that step 3 in your docs under “Fine-tune the plugin” is to exclude json & yaml files you don’t want uploaded. Aaand it’s not super clear from the docs that this is a specific configuration under OpenAPI rather than a garden-variety exclusion.
Yeah, I agree that the docs focus mainly on OpenAPI tab controls. I’ll update these to talk more specifically about OpenAPI → Excluded filepaths.
How will your plugin interact if the YAML analyzer already in the Marketplace is loaded?
It should not interact with it, YAML analyzer provides generic YAML checks (for syntax errors or stying issue) and our plugin focuses specifically on OpenAPI, so there is hardly any overlap.
I guess 42crunch is a commercial enterprise. Cool. So is SonarSource. But your plugin readme doesn’t indicate that. It only talks about creating a free account.
IMO your README should also be upfront about the fact that project files are uploaded to, processed on, and subsequently stored on an external server
Is there anything in particular you would like me to add there? Prerequisite to the plugin use is to signup for the account, which shows TOC, data privacy poilcy, etc.
The documentation linked to from the readme lists as step 5: “Run SonarQube”. Since SonarQube has to be running to do steps 2-4, I guess you mean “Run analysis”?
Yes, thanks for spotting this! We’ll update the docs.
What happens to analysis if the network is slow or the server unavailable?
If the network is slow, the analysis will take longer to complete. In case of network failure, analysis will fail as well.
The documentation says: By default, the plugin is automatically applied to each SonarQube project on the server.
there’s a way to make it not apply…? (Or is this the exclusions thing?)
this will happen immediately/automatically (the way it’s currently written) when what I think you mean is that each project’s files will be automatically uploaded and processed during analysis
The intention for this part of the documentation is to communicate that there are no specific steps needed to enable the plugin for any SonarQube project (besides of copying the plugin jar to lib/extensions).
I guess this is a normal behaviour for any SonarQube plugin, and there is no real need to stress that.
Same as for the analysis, I’ll correct the documentation to reflect this.
TBH, I’m a bit confounded by your approach. You’re picking up one or two specific files for parsing in order to raise issues based on their contents. This is the general description of a report-parsing plugin, yet you have structured this like a language analyzer. IMO, you should reconsider your approach: instead of grabbing everything that’s not excluded, set a default “report path” and allow the user to override that with configs, similar to how the JaCoCo plugin works. You can still report issues on files you don’t “own” by extension. Java analysis does that for pom.xml files.
I’m not sure I agree with that. My understanding is that report parsing plugins consume a file produced by some external tool (code coverage, linter what not) typically saved in a well known location and communicate it’s contents to SonarQube.
In our case the location or the number of OpenAPI files in the project are not known, and there are in some cases projects with tents or hundreds of OpenAPI files in them.
Regards,
Anton
I’m not a SonarQube expert though, and I’m very willing to tweak the plugin to get it approved. However there are few things which are not under my control - the need to upload files for analysis and the scoring mechanism.