Jenkins integration works but SonarQube server shows analysis error?

jblaine · July 25, 2019, 7:00pm

SonarQube Community 7.9.1
Sonar Scanner 4.0.0.1744
SonarQube Plugin for Jenkins 2.9
Jenkins 2.176.2 LTS

Hi everyone, I’m stumped. If anyone has any ideas I would love to hear them.

Jenkins build job output looks great. Scan completes and all HTTP requests to our SonarQube server result in 200 status codes:

    ...
    [Pipeline] withSonarQubeEnv
    Injecting SonarQube environment variables using the configuration: sonarqube-dev
    [Pipeline] {
    [Pipeline] sh
    + /var/lib/jenkins/tools/hudson.plugins.sonar.SonarRunnerInstallation/SonarQube-Scanner-4/bin/sonar-scanner -X -Dsonar.projectKey=R701-Tools:ssl-certificates
    14:29:38.339 INFO: Scanner configuration file: /var/lib/jenkins/tools/hudson.plugins.sonar.SonarRunnerInstallation/SonarQube-Scanner-4/conf/sonar-scanner.properties
    ...
    14:29:40.253 INFO: Load project settings for component key: 'R701-Tools:ssl-certificates'
    14:29:40.269 DEBUG: GET 200 https://sonarqube-dev.our.org/api/settings/values.protobuf?component=R701-Tools%3Assl-certificates | time=15ms
    14:29:40.270 INFO: Load project settings for component key: 'R701-Tools:ssl-certificates' (done) | time=17ms
    14:29:40.322 INFO: Load quality profiles
    14:29:40.350 DEBUG: GET 200 https://sonarqube-dev.our.org/api/qualityprofiles/search.protobuf?projectKey=R701-Tools%3Assl-certificates | time=27ms
    14:29:40.372 INFO: Load quality profiles (done) | time=49ms
    ...
    14:29:41.591 INFO: Quality profile for shell: ShellCheck
    14:29:41.591 INFO: ------------- Run sensors on module R701-Tools:ssl-certificates
    14:29:41.768 INFO: Load metrics repository
    14:29:41.780 DEBUG: GET 200 https://sonarqube-dev.our.org/api/metrics/search?f=name,description,direction,qualitative,custom&ps=500&p=1 | time=9ms
    14:29:41.799 INFO: Load metrics repository (done) | time=31ms
    ...
    14:29:44.671 DEBUG: Issue SC2006 saved for update-vault-cert.sh
    14:29:44.671 DEBUG: Issue SC2086 saved for update-vault-cert.sh
    14:29:44.672 DEBUG: Issue SC2086 saved for update-vault-cert.sh
    14:29:44.672 DEBUG: Issue SC2086 saved for update-vault-cert.sh
    14:29:44.672 DEBUG: Issue SC2086 saved for update-vault-cert.sh
    14:29:44.672 DEBUG: Count lines in update-vault-cert.sh
    14:29:44.682 INFO: Sensor ShellCheck Sensor [shellcheck] (done) | time=1193ms
    ...
    14:29:44.938 INFO: Analysis report generated in /var/lib/jenkins/workspace/jblaine-Data/r701-tools-ssl-certificates-pipeline/.scannerwork/scanner-report
    14:29:44.938 DEBUG: Upload report
    14:29:44.969 DEBUG: POST 200 https://sonarqube-dev.our.org/api/ce/submit?projectKey=R701-Tools%3Assl-certificates | time=30ms
    14:29:44.971 INFO: Analysis report uploaded in 33ms

But results in “The last analysis has failed. More details available on the [Background Tasks] page.” and here is the error report once I drill down to the task in the SonarQube web UI:

Error Details
java.lang.IllegalStateException: Unable to load component class org.sonar.server.webhook.WebHooksImpl
	at org.sonar.core.platform.ComponentContainer$ExtendedDefaultPicoContainer.getComponent(ComponentContainer.java:65)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:621)
	at org.picocontainer.containers.ImmutablePicoContainer.getComponent(ImmutablePicoContainer.java:40)
	at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:718)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:647)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:632)
	at org.picocontainer.parameters.BasicComponentParameter$1.resolveInstance(BasicComponentParameter.java:118)
	at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:136)
	at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78)
	at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309)
	at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335)
	at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
	at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364)
	at org.picocontainer.injectors.AbstractInjectionFactory$LifecycleAdapter.getComponentInstance(AbstractInjectionFactory.java:56)
	at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
	at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91)
	at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:699)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:647)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:621)
	at org.picocontainer.parameters.CollectionComponentParameter.getArrayInstance(CollectionComponentParameter.java:334)
	at org.picocontainer.parameters.CollectionComponentParameter.access$100(CollectionComponentParameter.java:49)
	at org.picocontainer.parameters.CollectionComponentParameter$1.resolveInstance(CollectionComponentParameter.java:139)
	at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:141)
	at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78)
	at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309)
	at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335)
	at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
	at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364)
	at org.picocontainer.injectors.AbstractInjectionFactory$LifecycleAdapter.getComponentInstance(AbstractInjectionFactory.java:56)
	at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
	at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91)
	at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:699)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:647)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:632)
	at org.picocontainer.parameters.BasicComponentParameter$1.resolveInstance(BasicComponentParameter.java:118)
	at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:136)
	at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78)
	at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309)
	at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335)
	at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
	at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364)
	at org.picocontainer.injectors.AbstractInjectionFactory$LifecycleAdapter.getComponentInstance(AbstractInjectionFactory.java:56)
	at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
	at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91)
	at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:699)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:647)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:678)
	at org.sonar.core.platform.ComponentContainer.getComponentByType(ComponentContainer.java:281)
	at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:209)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:191)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:158)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:133)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:85)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Unable to load component class org.sonar.server.webhook.WebhookCallerImpl
	at org.sonar.core.platform.ComponentContainer$ExtendedDefaultPicoContainer.getComponent(ComponentContainer.java:65)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:632)
	at org.picocontainer.parameters.BasicComponentParameter$1.resolveInstance(BasicComponentParameter.java:118)
	at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:136)
	at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78)
	at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309)
	at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335)
	at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
	at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364)
	at org.picocontainer.injectors.AbstractInjectionFactory$LifecycleAdapter.getComponentInstance(AbstractInjectionFactory.java:56)
	at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
	at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91)
	at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:699)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:647)
	at org.sonar.core.platform.ComponentContainer$ExtendedDefaultPicoContainer.getComponent(ComponentContainer.java:63)
	... 61 more
Caused by: java.lang.IllegalStateException: Unable to load component class okhttp3.OkHttpClient
	at org.sonar.core.platform.ComponentContainer$ExtendedDefaultPicoContainer.getComponent(ComponentContainer.java:65)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:632)
	at org.picocontainer.parameters.BasicComponentParameter$1.resolveInstance(BasicComponentParameter.java:118)
	at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:136)
	at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78)
	at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309)
	at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335)
	at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
	at org.picocontainer.injectors.ConstructorInjector.getComponentInstance(ConstructorInjector.java:364)
	at org.picocontainer.injectors.AbstractInjectionFactory$LifecycleAdapter.getComponentInstance(AbstractInjectionFactory.java:56)
	at org.picocontainer.behaviors.AbstractBehavior.getComponentInstance(AbstractBehavior.java:64)
	at org.picocontainer.behaviors.Stored.getComponentInstance(Stored.java:91)
	at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:699)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:647)
	at org.sonar.core.platform.ComponentContainer$ExtendedDefaultPicoContainer.getComponent(ComponentContainer.java:63)
	... 75 more
Caused by: java.lang.IllegalStateException: Unable to load component class okhttp3.OkHttpClient
	at org.sonar.core.platform.ComponentContainer$ExtendedDefaultPicoContainer.getComponent(ComponentContainer.java:65)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:621)
	at org.picocontainer.containers.ImmutablePicoContainer.getComponent(ImmutablePicoContainer.java:40)
	at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:718)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:647)
	at org.sonar.core.platform.ComponentContainer$ExtendedDefaultPicoContainer.getComponent(ComponentContainer.java:63)
	... 89 more
Caused by: java.lang.IllegalStateException: Unable to load component class okhttp3.OkHttpClient
	at org.sonar.core.platform.ComponentContainer$ExtendedDefaultPicoContainer.getComponent(ComponentContainer.java:65)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:621)
	at org.picocontainer.containers.ImmutablePicoContainer.getComponent(ImmutablePicoContainer.java:40)
	at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:718)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:647)
	at org.sonar.core.platform.ComponentContainer$ExtendedDefaultPicoContainer.getComponent(ComponentContainer.java:63)
	... 94 more
Caused by: java.lang.IllegalStateException: Unable to load component class okhttp3.OkHttpClient
	at org.sonar.core.platform.ComponentContainer$ExtendedDefaultPicoContainer.getComponent(ComponentContainer.java:65)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:621)
	at org.picocontainer.containers.ImmutablePicoContainer.getComponent(ImmutablePicoContainer.java:40)
	at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:718)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:647)
	at org.sonar.core.platform.ComponentContainer$ExtendedDefaultPicoContainer.getComponent(ComponentContainer.java:63)
	... 99 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class sun.security.ssl.SSLContextImpl$TLSContext
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:315)
	at java.base/java.security.Provider$Service.getImplClass(Provider.java:1848)
	at java.base/java.security.Provider$Service.newInstance(Provider.java:1824)
	at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
	at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
	at java.base/javax.net.ssl.SSLContext.getInstance(SSLContext.java:168)
	at org.sonarqube.ws.client.OkHttpClientBuilder.systemDefaultSslSocketFactory(OkHttpClientBuilder.java:234)
	at org.sonarqube.ws.client.OkHttpClientBuilder.build(OkHttpClientBuilder.java:194)
	at org.sonar.server.util.OkHttpClientProvider.provide(OkHttpClientProvider.java:66)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.picocontainer.injectors.MethodInjector.invokeMethod(MethodInjector.java:129)
	at org.picocontainer.injectors.MethodInjector.access$000(MethodInjector.java:39)
	at org.picocontainer.injectors.MethodInjector$2.run(MethodInjector.java:113)
	at org.picocontainer.injectors.AbstractInjector$ThreadLocalCyclicDependencyGuard.observe(AbstractInjector.java:270)
	at org.picocontainer.injectors.MethodInjector.decorateComponentInstance(MethodInjector.java:120)
	at org.picocontainer.injectors.CompositeInjector.decorateComponentInstance(CompositeInjector.java:58)
	at org.picocontainer.injectors.Reinjector.reinject(Reinjector.java:142)
	at org.picocontainer.injectors.ProviderAdapter.getComponentInstance(ProviderAdapter.java:96)
	at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:699)
	at org.picocontainer.DefaultPicoContainer.getComponent(DefaultPicoContainer.java:647)
	at org.sonar.core.platform.ComponentContainer$ExtendedDefaultPicoContainer.getComponent(ComponentContainer.java:63)
	... 104 more

ganncamp · July 26, 2019, 3:04pm

Hi,

So first, analysis happens in two steps. The first is that your scanner gathers issues and file-level metrics, balls those up into a report and submits that report to the server. That’s what’s succeeding for you on Jenkins. The second part is that the server processes the report, saving the issues and file-level metrics, and calculating and saving higher-level metrics. That’s the part that’s failing.

I haven’t seen the error you’re reporting before. Could you check your server logs to see if there’s anything interesting there?

Thx,
Ann

jblaine · July 26, 2019, 4:03pm

Hi Ann,

The only thing added to the SonarQube server logs during this Jenkins build is the following in logs/access.log.

1xx.83.10.243 - - [26/Jul/2019:11:33:11 -0400] "GET /api/server/version HTTP/1.1" 200 - "-" "Scanner for Jenkins" "AWwlvFkCryErk0j7AANu"
1xx.83.10.243 - - [26/Jul/2019:11:33:11 -0400] "GET /api/ce/task?id=AWwqFejoR-T6EkKLWTZc HTTP/1.1" 403 46 "-" "Scanner for Jenkins" "AWwlvFkCryErk0j7AANv"
1xx.83.10.243 - - [26/Jul/2019:11:33:11 -0400] "GET /api/server/version HTTP/1.1" 200 - "-" "Scanner for Jenkins" "AWwlvFkCryErk0j7AANw"
1xx.83.10.243 - - [26/Jul/2019:11:33:11 -0400] "GET /api/ce/task?id=AWwqY_YQR-T6EkKLWTZi HTTP/1.1" 403 46 "-" "Scanner for Jenkins" "AWwlvFkCryErk0j7AANx"
1xx.83.10.243 - - [26/Jul/2019:11:33:11 -0400] "GET /api/server/version HTTP/1.1" 200 - "-" "Scanner for Jenkins" "AWwlvFkCryErk0j7AANy"
1xx.83.10.243 - - [26/Jul/2019:11:33:11 -0400] "GET /api/ce/task?id=AWwu6aG-R-T6EkKLWTZl HTTP/1.1" 403 46 "-" "Scanner for Jenkins" "AWwlvFkCryErk0j7AANz"

So the 403 codes are telling, but… I have confirmed several times that the token being used in Jenkins (credential name “jblaine-SonarQube-token”) is correctly configured as a “Secret Text” type and that it’s value matches SonarQube’s token for user jblaine. The “jblaine-SonarQube-token” credential is chosen in the “Prepare SonarQube Scanner environment” section of the Jenkins job.

Yes, we do have “anonymous” access turned off for our SonarQube instance.

jblaine · July 26, 2019, 4:07pm

FWIW, the pipeline script is:

node {
  stage('SCM') {
    git 'https://gitlab.our.org/R701-Tools/ssl-certificates.git'
  }
  stage('SonarQube analysis') {
    def scannerHome = tool 'SonarQube-Scanner-4';
    withSonarQubeEnv(credentialsId: 'jblaine-SonarQube-token') {
      sh "${scannerHome}/bin/sonar-scanner -Dsonar.projectKey=R701-Tools:ssl-certificates"
    }
  }
}

ganncamp · July 26, 2019, 4:10pm

Hi,

It looks like you’ve checked your access log. Could you check your other server logs, please?

Thx,
Ann

jblaine · July 26, 2019, 4:14pm

I watched all of the logs. Those are the only lines logged in any of the logs during the Jenkins build timeframe.

jblaine · July 26, 2019, 4:16pm

Well, yes, ce.log does have the same Java traceback (java.lang.IllegalStateException: Unable to load component class org.sonar.server.webhook.WebHooksImpl) I pasted in the original post. I didn’t see a need to reproduce that.

jblaine · July 26, 2019, 4:20pm

I have also confirmed that SONAR_AUTH_TOKEN is in fact being set to the proper token in the Jenkins build job. I did that by changing my “sh” command temporarily to:

sh "echo $SONAR_AUTH_TOKEN | od -c"

Which bypasses the Jenkins-builtin credential masking and showed (altered for privacy):

[Pipeline] sh
+ echo ******
+ od -c
0000000   6   3   c   8   b   6   7   2   b   a   5   c   f   b   6   c
0000020   2   2   8   a   1   f   0   5   9   2   e   3   1   e   1   2
0000040   6   1   1   f   9   c   7   f  \n
0000051

jblaine · July 29, 2019, 5:29pm

With DEBUG level logging, I see the following auth success using my token from our Jenkins server (1xx.83.11.135):

2019.07.26 16:53:27 DEBUG web[AWwlvFkCryErk0j7AAhe][auth.event] login success [method|BASIC_TOKEN][provider|LOCAL|local][IP|1xx.83.10.
243|1xx.83.11.135][login|jblaine]

ganncamp · July 29, 2019, 5:34pm

Hi,

I’ve been hoping for some internal help on this (it may still come). In the meantime, I’m wondering if your instance isn’t just borked up. I’m wondering how must effort it would be for you to re-explode the jar & re-configure just to see…

Ann

jblaine · July 29, 2019, 9:26pm

Strange. I stopped, re-unzipped, restored our sonar.properties and wrapper.conf, started, installed the plugins we use, and all is fine now. I wonder if a mere restart would have solved it.

Thanks for the assistance, Ann. This gets us over another evaluation hurdle for SonarQube.

ganncamp · July 30, 2019, 11:42am

Hi,

I’m glad you got it working! Let us know (in new threads please ) if you have any other problems.

Ann