[First release] cnesreport 3.0.1

Hello every one, we would like to submit you a new plugin we have developed for SonarQube. Here come required information:

It is the first release of the plugin, so the plugin should be added to the Plugin Library page.

Thanks a lot for taking a look on this plugin.

Louis

Hi Louis,

Welcome to the community and congrats on getting to this milestone with your first plugin. Before I can add it to the Marketplace, I have to test it first. I wanted to let you know that I have other priorities right now, but I will come back to this “soon”.

 
Ann

Hi Louis,

Sorry it has taken me a while to get to this.

You will find below the narrative of quirks I found and comments I had as I tested your jar/plugin. I won’t make you read to the end though to get the upshot.

There are currently 3 barriers to adding this plugin to the Marketplace.

  1. You’re failing the Quality Gate on SonarCloud. When I look at your QG status graph, what I see reflects active development. So okay, maybe you failed since the release but you were fine when you released? Well, no. The timestamp of the GH tag (normalized for my time zone, I think) is 23 Aug. 8:26 a.m. Latest analysis, where your QG went red, was 23 Aug. 8:24 a.m. So 2 min before. TBH, I was looking for a way to accept that you were Passing when you released, but the timestamps won’t allow me to do that. You’ll need to release 3.0.1 anyway (more below) so you’ve got another chance at this.

  2. I’ve requested some changes in your PR. This part is easily handled, though. (You’ll obviously also need to make version number changes)

  3. It appears that when you craft a report bundle you first write the reports to SonarQube’s temp directory before streaming the zip out to the user. To my knowledge, this directory is not cleaned up, so with your plugin the user runs the risk of running out of disk space. There is nothing in the SQ docs about performing maintenance in this directory (because there’s no need) and nothing in your README that I noticed. For me accumulating these files in the file system is a blocker, and even adding a notice in your README wouldn’t be sufficient.

So… those three things must be done to enter the Marketplace. Below I’ve described some additional things you might want to handle eventually.


I find the dual nature of your jar/plugin interesting, so I tried it first standalone. A default command, java -jar sonar-cnes-report.jar yielded this:

SonarQube URL: http://localhost:9000
SonarQube online: true
Detected SonarQube version: 7.9.0
[ERROR] The 'component' parameter is missing

Fair enough. Time to ask for help to see how to pass component…
java -jar sonar-cnes-report.jar -h

usage: java -jar cnesreport.jar [-a <arg>] [-c] [-d <arg>] [-e] [-f <arg>] [-h] [-l <arg>] [-m <arg>] [-n <arg>] [-o <arg>] [-p
       <arg>] [-r <arg>] [-s <arg>] [-t <arg>] [-v] [-w] [-x <arg>]
Generate editable reports for SonarQube projects.

 -a,--author <arg>                 Name of the report writer.
 -c,--disable-conf                 Disable export of quality configuration used during analysis.
 -d,--date <arg>                   Date for the report. Default: current date.
 -e,--disable-spreadsheet          Disable spreadsheet generation.
 -f,--disable-csv <arg>            Disable CSV generation
 -h,--help                         Display this message.
 -l,--language <arg>               Language of the report. Values: en_US, fr_FR. Default: en_US.
 -m,--disable-markdown <arg>       Disable Markdown generation
 -n,--template-markdown <arg>      Path to the report template in markdown. Default: usage of internal template.
 -o,--output <arg>                 Output path for exported resources.
 -p,--project <arg>                SonarQube key of the targeted project.
 -r,--template-report <arg>        Path to the report template. Default: usage of internal template.
 -s,--server <arg>                 Complete URL of the targeted SonarQube server.
 -t,--token <arg>                  SonarQube token of the SonarQube user who has permissions on the project.
 -v,--version                      Display current version.
 -w,--disable-report               Disable report generation.
 -x,--template-spreadsheet <arg>   Path to the spreadsheet template. Default: usage of internal template.


Please report issues at https://github.com/lequal/sonar-cnes-report/issues

No where in there do I see a flag for passing “The ‘component’ parameter”. :joy:

A further browse of your docs yields the fact that you’re looking for a project id (-p/--project). It might be helpful to synch up the error message with the flag names. :slight_smile:

Once I passed in a project id I got my 4 reports.

LibreOffice 6.0 gave me an error on opening the .docx:

An error occurred during opening the file. This may be caused by incorrect file contents.
The error details are:
SAXException: [word/document.xml line 271]: unknown error
Proceeding with import may cause data loss or corruption, and application may become unstable or crash.

I okay-ed past that and it seemed to open fine, though.

Since this is my localhost test server, I don’t generally bother to generate and include test coverage reports. I think that threw the report generator:

The metric section doesn’t make a lot of sense to me:

And the Issues part of the report makes me wish for a “The Scream” emoji. :scream: doesn’t quite do my feelings justice.

But whatever. It seems that the report generator doesn’t know how to handle hotspots (or maybe that’s what my initial error was about):
Selection_660

Now, on to the SQ plugin!

Once installed, I find… nothing in the project homepage. Okay let’s log in as an admin. Nope, nothing in the admin menu. Nothing in the admin configs. Hmm… maybe the web api? Yep!

It would be helpful if this were explicit in the project README.

So I try the web service and find that token is required even for public projects. That’s particularly odd when it’s not required on the standalone side. :frowning: However, the service docs do say that it’s required, so okay.

Now I’m providing all 3 params: http://localhost:9000/api/cnesreport/report?key=bsl&author=ann&token=8065ed6dfdf4a721b90262f8ea93107fc7ae3646 (token not redacted because it’s already deleted :stuck_out_tongue_winking_eye:) and I get

{
  "errors": [
    {
      "msg": "An error has occurred. Please contact your administrator"
    }
  ]
}

sonar.log:

Aug 28, 2019 3:05:17 PM fr.cnes.sonar.report.ReportCommandLine execute
INFO: Detected SonarQube version: 7.9.0

web.log:

Fail to process request http://localhost:9000/api/cnesreport/report?key=bsl&author=ann&token=8065ed6dfdf4a721b90262f8ea93107fc7ae3646
java.lang.NullPointerException: null
	at fr.cnes.sonar.report.providers.LanguageProvider.getLanguage(LanguageProvider.java:67)
	at fr.cnes.sonar.report.providers.ProjectProvider.getProject(ProjectProvider.java:80)
	at fr.cnes.sonar.report.factory.ReportModelFactory.create(ReportModelFactory.java:103)
	at fr.cnes.sonar.report.ReportCommandLine.execute(ReportCommandLine.java:131)
	at fr.cnes.sonar.plugin.ws.ExportTask.handle(ExportTask.java:78)
	at org.sonar.server.ws.WebServiceEngine.execute(WebServiceEngine.java:110)
...

When I pick a different project, I do get my report bundle. I think the NPE was caused by my (lazily) picking a project with a super-short key that uses a language no longer present in my instance. Still, that shouldn’t be a barrier to normal operation as long as I don’t want to analyze the project again. So this should be cleaned up but I guess it’s not a deal breaker since it’s a corner case.

Also, I find the logs to be rather chatty. All this for a single web service call:

Aug 28, 2019 3:13:18 PM fr.cnes.sonar.report.ReportCommandLine execute
INFO: SonarQube URL: http://localhost:9000
Aug 28, 2019 3:13:18 PM fr.cnes.sonar.report.ReportCommandLine execute
INFO: SonarQube online: true
Aug 28, 2019 3:13:18 PM fr.cnes.sonar.report.ReportCommandLine execute
INFO: Detected SonarQube version: 7.9.0
Aug 28, 2019 3:13:25 PM fr.cnes.sonar.report.ReportCommandLine execute
INFO: Report generation: SUCCESS
Aug 28, 2019 3:13:25 PM fr.cnes.sonar.plugin.tools.FileTools deleteFolder
WARNING: 2019-08-28-SonarSource :: Rule API-analysis-report.docx
Aug 28, 2019 3:13:25 PM fr.cnes.sonar.plugin.tools.FileTools deleteFolder
WARNING: sonar.properties
Aug 28, 2019 3:13:25 PM fr.cnes.sonar.plugin.tools.FileTools deleteFolder
WARNING: wrapper.conf
Aug 28, 2019 3:13:25 PM fr.cnes.sonar.plugin.tools.FileTools deleteFolder
WARNING: conf
Aug 28, 2019 3:13:25 PM fr.cnes.sonar.plugin.tools.FileTools deleteFolder
WARNING: 2019-08-28-SonarSource :: Rule API-issues-report.csv
Aug 28, 2019 3:13:25 PM fr.cnes.sonar.plugin.tools.FileTools deleteFolder
WARNING: 2019-08-28-SonarSource :: Rule API-issues-report.xlsx
Aug 28, 2019 3:13:25 PM fr.cnes.sonar.plugin.tools.FileTools deleteFolder
WARNING: 2019-08-28-SonarSource :: Rule API-analysis-report.md
Aug 28, 2019 3:13:25 PM fr.cnes.sonar.plugin.tools.FileTools deleteFolder
WARNING: /path/to/sonarqube-7.9/temp/cnesreport532079384087107079186198403164170

And then when I check the path in that last, WARNING log line I discover that you’re writing the reports to disk in SonarQube’s temp directory before streaming them out to the requester. And that is not cool because to my knowledge, nothing cleans this up. The user runs the risk of running out of disk space without ever realizing why.


Let me know if you have questions or want the full stacktrace from the NPE.

 
Ann

Hi Ann,

Thanks for your answer.

I’m going to do the best to release a new version that can be published in the marketplace!

  • For the QG status, I’m going to fix that in next release.
  • I’ve seen the requested changes for the PR, I will change it when publishing next release.
  • For the temp directory, we need it to generate files, but is it possible to publish the plugin if it cleans the generated files after sending them to the client ?
  • I will update the logs to replace “component” by “projects”
  • I will investigate for the LibreOffice issue, if it possible I’m going to fix it on the next release, if not, I will open an issue on GitHub to fix it later.

Finally, for the NPE issue I’m going to investigate, you should receive a message later if a need more information !

Louis

Hi Louis,

I don’t (much) care about what happens in the middle as long as SQ itself still runs properly and I don’t find my file system cluttered at the end of testing. :smile:

Since you’re still going to be writing to the file system, I think you should add a notice to your docs that files are temporarily written to $SONARQUBE-HOME/temp so that people have a clue where to look just in case something goes sideways.

Also, would you make the default level logs a little less chatty? Some people are going to be buffaloed by all those WARNING messages.

 
Ann

Hi Ann,

Sorry for the delay but the plugin in updated to 3.1.0!

  • Now, temp directory is cleaned and a notice is added in the README file.
  • NPE is fixed, tell us if you find another issue :wink:
  • Logs are not less chatty but warnings are fixed.
  • Some displays are changed, for example, if you run java -jar sonar-cnes-report the error message is now: Please provide a project with the -p argument, you can also use -h argument to display help.
  • Test coverage is improved, so Quality Gate is now green!
  • PR is updated

For the docx error, I don’t know why there is this issue, but it still work when we click on “yes” (LibreOffice) and with Microsoft Word. So I continue to investigate and I will open an issue.

I hope that this new version can be added to marketplace :slight_smile:
Louis

Hi Louis,

FYI I’ll need to find another chunk of time for retesting, so it could be a while before I come back to you on this.

 
Ann

Hi Louis,

I’ve found my chunk of time. So you know, I didn’t re-test the jar in stand-alone mode because my curiosity was already satisfied & it’s really out of my scope. I did test the plugin & on that I’ve got some more feedback for you:

  1. I’m still seeing artifacts left in in $SONARQUBE-HOME/temp after report generation. It looks like you’ve cleaned up the reports themselves but forgotten the contents of the conf subdirectory: an xml file and the Quality Gate JSON file. The file space hit is much smaller, but this will still accumulate over time.

  2. It didn’t hit me until I downloaded the jar for this second version that you don’t have the version number in the jar name. That’s only an issue because of the way the Marketplace metadata generator works. If all the jars for all your versions are named the same, then you end up with jumbled data in the Marketplace because the metadata generator reads some of its data directly from the jar (which jar? good question). So… you’ll want to add some unique-ifier to your jar name and version number is as good a pattern as any. For first release this can pass, but it must be changed by your second release or we’ll have a real mess.

  3. I saw that you added a UI page for this. And it filters the project list based on your Browse permissions! Very nice!

  4. Via the UI page, I selected the same missing-language project to run reports for. I got back a JSON error, and in web.log this:

2019.09.09 10:19:05 ERROR web[AW0WY+CPRH/OZscpAAA8][o.s.s.w.WebServiceEngine] Fail to process request http://localhost:9000/api/cnesreport/report?key=bsl&author=Administrator&token=f2296be3347c3eb49b9fb1e3149b9039b3b69b17&generation=Generate
java.lang.NullPointerException: null
	at fr.cnes.sonar.report.model.Report.getQualityProfilesName(Report.java:290)
	at fr.cnes.sonar.report.exporters.docx.DataAdapter.loadPlaceholdersMap(DataAdapter.java:443)
	at fr.cnes.sonar.report.exporters.docx.DocXExporter.export(DocXExporter.java:140)
...

with no ‘Caused by’. And BTW it worked fine when I picked a different project.

  1. I noticed that the colors in the issue severity graph seem off. I would expect Blocker to be red, not Info. No idea how much control over this you have, but FWIW:
    Selection_698

Of these points, only #1 must be addressed before Marketplace entry. #2 must be addressed before your 2nd marketplace release, but would be better addressed with the first one.

 
Ann

Hi Ann

Thanks for your time.

  • For #1 it’s strange, all is cleaned on my sonarqube instance, i’m going to investigate about that. What is your sonarqube version ?
  • For #2 we will fix that
  • For #4 & #5 i’m going to open an issue for now but I will take a look if I have some time.

Louis

Hi Louis,

I tested with 7.9(.0) CE. I’m on Linux, BTW.

 
Ann

Hi Ann,

I can’t reproduce the bug, I also use SonarQube 7.9CE (with Docker)
If it’s not to late is it possible to have some info about files ? Something like ls -latr on the folders that are not deleted to see if there is something strange about filenames or permission that prevent deleting.

If there is no sensitive informations, you can also send us your files that are not deleted.

Meanwhile, I’ll continue to investigate about this issue.

Thanks

Louis

Hi Louis,

I had already cleaned up the files. So I tried to reproduce and I couldn’t. So I guess we’re good on that point. Do you want to proceed with release with no version number in your jar name, or issue a new jar?

 
Ann

Hi ann

The PR is updated, we renamed the jar to include version number.

Tell me if there is anything else you want to change before publishing. We also keep all your suggestions for future releases.

Louis

You’re in!

 
:slightly_smiling_face:
Ann

Nice! :smiley:
Thanks for releasing plugin in the marketplace and thanks for your help!

Louis

1 Like

Hi Louis,

your plugin might be useful for many users, as it is often asked for creating reports, export issues to Excel, csv … etc.
Tried with Sonarqube 7.9.1 Enterprise test instance on Windows server 2012, but unfortunately it doesn’t work from Sonarqube web ui, whereas the CLI Version works with the same settings.

browser shows
{"errors":[{"msg":"An error has occurred. Please contact your administrator"}]}

sonar.log has

Sep. 18, 2019 1:41:07 NACHM. fr.cnes.sonar.report.ReportCommandLine execute
INFO: SonarQube online: false
Sep. 18, 2019 1:45:12 NACHM. fr.cnes.sonar.report.ReportCommandLine execute
INFO: SonarQube URL: https://somesonarhost.foobar.de
Sep. 18, 2019 1:45:12 NACHM. fr.cnes.sonar.report.providers.SonarQubeInfoProvider getSonarQubeStatus
SEVERE: Impossible to reach SonarQube instance.
fr.cnes.sonar.report.exceptions.SonarQubeException: Impossible to reach SonarQube instance.
at fr.cnes.sonar.report.providers.RequestManager.get(RequestManager.java:127)
at fr.cnes.sonar.report.providers.AbstractDataProvider.stringRequest(AbstractDataProvider.java:321)
at fr.cnes.sonar.report.providers.AbstractDataProvider.request(AbstractDataProvider.java:275)
at fr.cnes.sonar.report.providers.SonarQubeInfoProvider.getSonarQubeStatus(SonarQubeInfoProvider.java:67)
at fr.cnes.sonar.report.factory.ServerFactory.create(ServerFactory.java:70)
at fr.cnes.sonar.report.ReportCommandLine.execute(ReportCommandLine.java:117)
at fr.cnes.sonar.plugin.ws.ExportTask.handle(ExportTask.java:78)
at org.sonar.server.ws.WebServiceEngine.execute(WebServiceEngine.java:110)
at org.sonar.server.ws.WebServiceFilter.doFilter(WebServiceFilter.java:88)
at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:126)
at org.sonar.server.platform.web.MasterServletFilter.doFilter(MasterServletFilter.java:95)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.user.UserSessionFilter.doFilter(UserSessionFilter.java:87)
at org.sonar.server.user.UserSessionFilter.doFilter(UserSessionFilter.java:71)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.CacheControlFilter.doFilter(CacheControlFilter.java:76)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.SecurityServletFilter.doHttpFilter(SecurityServletFilter.java:76)
at org.sonar.server.platform.web.SecurityServletFilter.doFilter(SecurityServletFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.RedirectFilter.doFilter(RedirectFilter.java:58)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.requestid.RequestIdFilter.doFilter(RequestIdFilter.java:63)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.sonar.server.platform.web.RootFilter.doFilter(RootFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at ch.qos.logback.access.tomcat.LogbackValve.invoke(LogbackValve.java:256)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Fail to request https://somesonarhost.foobar.de/api/system/status
at org.sonarqube.ws.client.HttpConnector.doCall(HttpConnector.java:176)
at org.sonarqube.ws.client.HttpConnector.get(HttpConnector.java:112)
at org.sonarqube.ws.client.HttpConnector.call(HttpConnector.java:99)
at fr.cnes.sonar.report.providers.RequestManager.get(RequestManager.java:125)
… 52 more
Caused by: java.io.IOException: Unexpected response code for CONNECT: 403
at okhttp3.internal.connection.RealConnection.createTunnel(RealConnection.java:358)
at okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.java:197)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:145)
at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:192)
at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121)
at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:100)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
at okhttp3.RealCall.execute(RealCall.java:69)
at org.sonarqube.ws.client.HttpConnector.doCall(HttpConnector.java:173)
… 55 more

Our instance uses Apache as reverse proxy, it’s access.log has

xx.xx.xxx.xxx - - [18/Sep/2019:14:21:55 +0200] “GET /api/cnesreport/report?key=de.foo.bar%3AFoobar&author=Rebhan%2C+Gilbert±+xx123&token=xxxxxxxxxxxxxxxxx&generation=Generate HTTP/1.1” 500 79 “https://somesonarhost.foobar.de/extension/cnesreport/report” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/18.17763”

Another problem, missed a possibillity to set sonarhost and token for the plugin in web ui, had to set it directly in the %SONARQUBE_HOME%\extensions\plugins\sonar-cnes-report.jar
two times = plugin.properties, report.properties ?! Also setting the sonarhost one time should be sufficient.

Regarding the ouput, it has to be polished, might be done by users itself, didn’t try it yet.

Gilbert