Jenkins OkHttp thread remain active when branch analysis is no more present

  • versions
    SonarQube Enterprise Edition Version 7.9.6 (build 41879)

  • error observed
    Jenkins is searching all branches analysis using OkHttp thread and in case the analysis was canceled on Sonarqube, the thread remain into the pool for 5 minutes. Repeating this we reach the 500 limit threads of Jenkins after the Jenkins cannot initialize new thread

  • steps to reproduce
    Jenkins process a feature branch and do an analysis with sonarqube.
    Analysis is OK and published in sonarqube
    After a while (Key: sonar.dbcleaner.daysBeforeDeletingInactiveShortLivingBranches) the analysis is canceled.
    The web page of the organization and project in Jenkins is searching all information’s about each branches, including the old branches where the analysis has been deleted.
    The OkHttp thread used remain active for 5 minutes.
    Repeating the reload of the web page increase the number of Okhttp thread and reaches 500 threads
    At that point Jenkins is not able to initialize new thread and is stuck

We found this problem when many people was loading the web page of the projects during a consolidation release. People was checking the progress of the pipeline in jenkins and suddenly the Jenkins was crashing.
As remediation in short term, we have reduced the number of old project by filtering it from Jenkins, but we cannot guarantee a stable situation for ever.

Hi Stefano.

I’m having trouble understanding the issue.

  • Is the issue related to the Jenkins plugin ‘SonarScanner for Jenkins’? If so, what’s the version of Jenkins and the SonarScanner for Jenkins?
  • Do you have any error stack trace that you could show us? Also perhaps your pipeline configuration?
  • What’s the role of the key sonar.dbcleaner.daysBeforeDeletingInactiveShortLivingBranches in this problem? Is it that when a branch gets deleted, there’s a connection from Jenkins that hangs? However at the beginning you mention that it’s due to a canceled analysis?

Hi Duarte,
I understand, we are trying several month to solve the issue…

  • I don’t think the problem is related to the plugin “SonarScanner for Jenkins”, because the problem happen without a running analysis. What I don’t know, if Jenkins is using the plugin to get information from Sonarqube.
  • It’s a GitHub Organization pipeline, it scans a GitHub organization for all repositories matching some defined markers. As soon you enters on the Organization (https://jenkins-server/jenkins/job/OrganizationName/) you have the list of organization’s project. When you select a project (https://jenkins-server/jenkins/job/OrganizationName/job/ProjectName/) you have the branch list. At that point, in case some branches exist in Jenkins and the analysis in Sonarqube have been cancelled, we found additional OkHttp thread on the Jenkins server. The thread remain for 5 minutes and after disappears. If different people enter on that web page on the same time (<5min), the thread count increase and if reaches 500 the Jenkins server is stuck.
    Unfortunately I didn’t found any error stack trace
  • The role of the key is that we have the problem when the branch in Jenkins is present and the analysis in sonarqube has been cancelled. The key define the time before delete the analysis.

I think it’s a communication problem between Jenkins and Sonarqube, I suppose Jenkins ask some details about the analysis, Sonarqube answer with an error or an exception, but Jenkins is not handling correctly the error or the exception, and didn’t release the thread.