SonarQube DB Upgrade Issue

chinnu_yallapalli · April 8, 2021, 8:37am

Hello Team,

We have hosted SonarQube on AWS environment. We are using RDS PostgreSQL DB. We are trying to upgrade RDS version, after that we are not able to scan the code. It was redirecting to different URL, then the Bamboo build plan getting failed. From supported platform for SQ we find that PostgreSQL will support & it is in green, but still it is failing.

Doc : Prerequisites and Overview | SonarQube Docs
SonarQube Current Version - 8.4.2
Scanner Version - 4.3.0.2102
Sonar for Bamboo Plugin version - 2.9.0
PostgreSQL current version - 9.6.20
PostgreSQL latest Version - 12.5

We went through all the logs (sonar.log, ce.log, es.log, access.log, web.log), we didn’t find anything. We tested it manually from CLI, through Bamboo plans both are failing. But post upgrade of RDS CPU went to 100 %. So we restarted RDS then it came decrease to 4 %.

We have taken sample code from sonar-scanner-cli/sonarscanner.md at master · SonarSource/sonar-scanner-cli · GitHub website. It is working for current version of RDS on Production but not on Testing env which is 12.5 RDS. Expected URL after scanning process is : https://sonarqube.com/dashboard?id=org.sonarqube%3Asonarqube-scanner

We are sharing error logs as mentioned below.

build	06-Apr-2021 12:57:42	12:57:42.410 INFO: EXECUTION FAILURE
build	06-Apr-2021 12:57:42	12:57:42.410 INFO: ------------------------------------------------------------------------
build	06-Apr-2021 12:57:42	12:57:42.410 INFO: Total time: 1:09.526s
build	06-Apr-2021 12:57:42	12:57:42.578 INFO: Final Memory: 28M/207M
error	06-Apr-2021 12:57:42	12:57:42.584 ERROR: Error during SonarScanner execution
error	06-Apr-2021 12:57:42	java.lang.IllegalStateException: java.lang.IllegalStateException: Fail to request https://sonarqube.com/batch/project.protobuf?key=org.sonarqube%3Asonarqube-scanner
error	06-Apr-2021 12:57:42		at org.sonar.scanner.scan.filesystem.MetadataGenerator.setMetadata(MetadataGenerator.java:69)
error	06-Apr-2021 12:57:42		at org.sonar.scanner.scan.filesystem.FileIndexer.lambda$indexFile$0(FileIndexer.java:136)
error	06-Apr-2021 12:57:42		at org.sonar.api.batch.fs.internal.DefaultInputFile.checkMetadata(DefaultInputFile.java:91)
error	06-Apr-2021 12:57:42		at org.sonar.api.batch.fs.internal.DefaultInputFile.charset(DefaultInputFile.java:230)
error	06-Apr-2021 12:57:42		at org.sonar.plugins.php.PhpExclusionsFileFilter$AverageLineLengthCalculator.fileLines(PhpExclusionsFileFilter.java:110)
error	06-Apr-2021 12:57:42		at org.sonar.plugins.php.PhpExclusionsFileFilter$AverageLineLengthCalculator.getAverageLineLength(PhpExclusionsFileFilter.java:95)
error	06-Apr-2021 12:57:42		at org.sonar.plugins.php.PhpExclusionsFileFilter.accept(PhpExclusionsFileFilter.java:59)
error	06-Apr-2021 12:57:42		at org.sonar.scanner.scan.filesystem.FileIndexer.accept(FileIndexer.java:261)
error	06-Apr-2021 12:57:42		at org.sonar.scanner.scan.filesystem.FileIndexer.indexFile(FileIndexer.java:140)
error	06-Apr-2021 12:57:42		at org.sonar.scanner.scan.filesystem.ProjectFileIndexer$IndexFileVisitor.visitFile(ProjectFileIndexer.java:242)
error	06-Apr-2021 12:57:42		at org.sonar.scanner.scan.filesystem.ProjectFileIndexer$IndexFileVisitor.visitFile(ProjectFileIndexer.java:214)
error	06-Apr-2021 12:57:42		at java.nio.file.Files.walkFileTree(Files.java:2670)
error	06-Apr-2021 12:57:42		at org.sonar.scanner.scan.filesystem.ProjectFileIndexer.indexDirectory(ProjectFileIndexer.java:210)
error	06-Apr-2021 12:57:42		at org.sonar.scanner.scan.filesystem.ProjectFileIndexer.indexFiles(ProjectFileIndexer.java:197)
error	06-Apr-2021 12:57:42		at org.sonar.scanner.scan.filesystem.ProjectFileIndexer.index(ProjectFileIndexer.java:159)
error	06-Apr-2021 12:57:42		at org.sonar.scanner.scan.filesystem.ProjectFileIndexer.indexModulesRecursively(ProjectFileIndexer.java:138)
error	06-Apr-2021 12:57:42		at org.sonar.scanner.scan.filesystem.ProjectFileIndexer.index(ProjectFileIndexer.java:105)
error	06-Apr-2021 12:57:42		at org.sonar.scanner.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:348)
error	06-Apr-2021 12:57:42		at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:137)
error	06-Apr-2021 12:57:42		at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:123)
error	06-Apr-2021 12:57:42		at org.sonar.scanner.bootstrap.GlobalContainer.doAfterStart(GlobalContainer.java:144)
error	06-Apr-2021 12:57:42		at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:137)
error	06-Apr-2021 12:57:42		at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:123)
error	06-Apr-2021 12:57:42		at org.sonar.batch.bootstrapper.Batch.doExecute(Batch.java:72)
error	06-Apr-2021 12:57:42		at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:66)
error	06-Apr-2021 12:57:42		at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:46)
build	06-Apr-2021 12:57:42	12:57:42.584 INFO: ------------------------------------------------------------------------
error	06-Apr-2021 12:57:42		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
error	06-Apr-2021 12:57:42		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
error	06-Apr-2021 12:57:42		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
error	06-Apr-2021 12:57:42		at java.lang.reflect.Method.invoke(Method.java:498)
error	06-Apr-2021 12:57:42		at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
error	06-Apr-2021 12:57:42		at com.sun.proxy.$Proxy0.execute(Unknown Source)
error	06-Apr-2021 12:57:42		at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:189)
error	06-Apr-2021 12:57:42		at org.sonarsource.scanner.api.EmbeddedScanner.execute(EmbeddedScanner.java:138)
error	06-Apr-2021 12:57:42		at org.sonarsource.scanner.cli.Main.execute(Main.java:112)
error	06-Apr-2021 12:57:42		at org.sonarsource.scanner.cli.Main.execute(Main.java:75)
error	06-Apr-2021 12:57:42		at org.sonarsource.scanner.cli.Main.main(Main.java:61)
error	06-Apr-2021 12:57:42	Caused by: java.lang.IllegalStateException: Fail to request https://sonarqube.com/batch/project.protobuf?key=org.sonarqube%3Asonarqube-scanner
error	06-Apr-2021 12:57:42		at org.sonarqube.ws.client.HttpConnector.doCall(HttpConnector.java:200)
error	06-Apr-2021 12:57:42		at org.sonarqube.ws.client.HttpConnector.get(HttpConnector.java:124)
error	06-Apr-2021 12:57:42		at org.sonarqube.ws.client.HttpConnector.call(HttpConnector.java:111)
error	06-Apr-2021 12:57:42		at org.sonar.scanner.bootstrap.DefaultScannerWsClient.call(DefaultScannerWsClient.java:73)
error	06-Apr-2021 12:57:42		at org.sonar.scanner.repository.DefaultProjectRepositoriesLoader.load(DefaultProjectRepositoriesLoader.java:49)
error	06-Apr-2021 12:57:42		at org.sonar.scanner.repository.ProjectRepositoriesSupplier.get(ProjectRepositoriesSupplier.java:47)
error	06-Apr-2021 12:57:42		at org.sonar.scanner.scan.filesystem.StatusDetection.checkChangedWithProjectRepositories(StatusDetection.java:52)
error	06-Apr-2021 12:57:42		at org.sonar.scanner.scan.filesystem.StatusDetection.status(StatusDetection.java:48)
error	06-Apr-2021 12:57:42		at org.sonar.scanner.scan.filesystem.MetadataGenerator.setMetadata(MetadataGenerator.java:66)
error	06-Apr-2021 12:57:42		... 36 more
error	06-Apr-2021 12:57:42	Caused by: java.net.SocketTimeoutException: timeout
error	06-Apr-2021 12:57:42		at okio.Okio$4.newTimeoutException(Okio.java:232)
error	06-Apr-2021 12:57:42		at okio.AsyncTimeout.exit(AsyncTimeout.java:286)
error	06-Apr-2021 12:57:42		at okio.AsyncTimeout$2.read(AsyncTimeout.java:241)
error	06-Apr-2021 12:57:42		at okio.RealBufferedSource.indexOf(RealBufferedSource.java:358)
error	06-Apr-2021 12:57:42		at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:230)
error	06-Apr-2021 12:57:42		at okhttp3.internal.http1.Http1ExchangeCodec.readHeaderLine(Http1ExchangeCodec.java:242)
error	06-Apr-2021 12:57:42		at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.java:213)
error	06-Apr-2021 12:57:42		at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.java:115)
error	06-Apr-2021 12:57:42		at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:94)
error	06-Apr-2021 12:57:42		at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
error	06-Apr-2021 12:57:42		at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
error	06-Apr-2021 12:57:42		at org.sonarqube.ws.client.OkHttpClientBuilder.addHeaders(OkHttpClientBuilder.java:208)
error	06-Apr-2021 12:57:42		at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
error	06-Apr-2021 12:57:42		at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:43)
error	06-Apr-2021 12:57:42		at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
error	06-Apr-2021 12:57:42		at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
error	06-Apr-2021 12:57:42		at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
error	06-Apr-2021 12:57:42		at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
error	06-Apr-2021 12:57:42		at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
error	06-Apr-2021 12:57:42		at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
error	06-Apr-2021 12:57:42		at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
error	06-Apr-2021 12:57:42		at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
error	06-Apr-2021 12:57:42		at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
error	06-Apr-2021 12:57:42		at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
error	06-Apr-2021 12:57:42		at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
error	06-Apr-2021 12:57:42		at okhttp3.RealCall.execute(RealCall.java:81)
error	06-Apr-2021 12:57:42		at org.sonarqube.ws.client.HttpConnector.doCall(HttpConnector.java:198)
error	06-Apr-2021 12:57:42		... 44 more
error	06-Apr-2021 12:57:42	Caused by: java.net.SocketException: Socket closed
error	06-Apr-2021 12:57:42		at java.net.SocketInputStream.read(SocketInputStream.java:204)
error	06-Apr-2021 12:57:42		at java.net.SocketInputStream.read(SocketInputStream.java:141)
error	06-Apr-2021 12:57:42		at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
error	06-Apr-2021 12:57:42		at sun.security.ssl.InputRecord.read(InputRecord.java:503)
error	06-Apr-2021 12:57:42		at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
error	06-Apr-2021 12:57:42		at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933)
error	06-Apr-2021 12:57:42		at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
error	06-Apr-2021 12:57:42		at okio.Okio$2.read(Okio.java:140)
error	06-Apr-2021 12:57:42		at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
error	06-Apr-2021 12:57:42		... 68 more
simple	06-Apr-2021 12:57:42	Failing task since return code of [/app/sonarqube/sonar-scanner-4.3.0.2102/bin/sonar-scanner -X -D sonar.login=****** -Dsonar.host.url=<sonar URL> -Dsonar.login=****** was 1 while expected 0
simple	06-Apr-2021 12:57:42	SONAR4BAMBOO: was not able to find a SonarQube result URL
simple	06-Apr-2021 12:57:42	Finished task 'Scanning' with result: Failed
simple	06-Apr-2021 12:57:42	Running post build plugin 'NCover Results Collector'
simple	06-Apr-2021 12:57:42	Running post build plugin 'Artifactory Release Management Staging'
simple	06-Apr-2021 12:57:42	Running post build plugin 'Artifactory Build Info Copier'
simple	06-Apr-2021 12:57:42	Running post build plugin 'Artifactory Gradle Properties Info Copier'
simple	06-Apr-2021 12:57:42	Running post build plugin 'Clover Results Collector'
simple	06-Apr-2021 12:57:42	Running post build plugin 'Sauce OnDemand Post Runner'
simple	06-Apr-2021 12:57:42	Running post build plugin 'Docker Container Cleanup'
simple	06-Apr-2021 12:57:42	Running post build plugin 'npm Cache Cleanup'
simple	06-Apr-2021 12:57:42	Running post build plugin 'Artifact Copier'
simple	06-Apr-2021 12:57:42	Finalising the build...
simple	06-Apr-2021 12:57:42	Stopping timer.
simple	06-Apr-2021 12:57:42	Build BSHAAS-TES-JOB1-6 completed.
simple	06-Apr-2021 12:57:42	Running on server: post build plugin 'NCover Results Collector'
simple	06-Apr-2021 12:57:42	Running on server: post build plugin 'Build Hanging Detection Configuration'
simple	06-Apr-2021 12:57:42	Running on server: post build plugin 'Bamboo project link creator'
simple	06-Apr-2021 12:57:42	Running on server: post build plugin 'Clover Delta Calculator'
simple	06-Apr-2021 12:57:42	Running on server: post build plugin 'Maven Dependencies Postprocessor'
simple	06-Apr-2021 12:57:42	All post build plugins have finished
simple	06-Apr-2021 12:57:42	Generating build results summary...
simple	06-Apr-2021 12:57:42	Saving build results to disk...
simple	06-Apr-2021 12:57:42	Store variable context...
simple	06-Apr-2021 12:57:42	Indexing build results...
simple	06-Apr-2021 12:57:42	Finished building BSHAAS-TES-JOB1-6.

Kindly help us on this.

Thanks & Regards
Revathi

Tobias_Trabelsi · April 9, 2021, 7:42am

Hi @chinnu_yallapalli

i think during the RDS upgrade some indexes are delete. we document an additional database maintenance here, so please try to run vacuum(full, analyze); on your sonarqube database and on some occasions a reindex database was also needed to increase the performance of the database.

chinnu_yallapalli · April 7, 2021, 10:55am

Hello @Tobias_Trabelsi
Thanks for the quick help, We are trying to execute above command but it’s not allowing with sonar user, but we are not sure which is super user. We are seeing these warnings
WARNING: skipping “pg_toast_xxxx” — only superuser can vacuum it
WARNING: skipping “pg_toast_xxxx” — only superuser can vacuum it
Is it possible to find super user from DB, is there any command to find ? or else is it possible to provide super user previledges to sonar user ?
Please let us know if there is any way.

Tobias_Trabelsi · April 9, 2021, 7:42am

Hi @chinnu_yallapalli ,

uhm yes and no… this is up to your RDS configuration. in the RDS configuration overview there is a field for the admin user, but credentials are not exposed anywhere on aws as far as i know and you might need to reset the password as per aws docs .

chinnu_yallapalli · April 7, 2021, 1:28pm

Hello @Tobias_Trabelsi
Thanks for the help, is working fine for us. It got fixed. But one question as you mention in the last line ( on your sonarqube database and on some occasions a reindex database was also needed to increase the performance of the database), if we want to do reindex manually how we can start it, why because during application upgrade we went to /setup but now it is showing like database is upto date. Please let us know how to initiate manually or through CLI. So it would be help full while doing it on Production.

Tobias_Trabelsi · April 7, 2021, 1:38pm

did it also solve your problem that the server timed out during a scan as i initially assumed?

same as the vacuum. you connect to the RDS database with the tool of your choice and issue the SQL command reindex database $DATABASE_NAME; where $DATABASE_NAME needs to be replaced with the name of your database

psql -U $USERNAME -h $RDS_URL -p 5432 -W
psql (12.5 (Ubuntu 12.5-1.pgdg20.04+1), server 12.5)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

database=> reindex database $DATABASE_NAME;

there is more information in the postgresql documentation

chinnu_yallapalli · April 8, 2021, 8:36am

Hello @Tobias_Trabelsi
Thanks for the command, it is working.

We really satisfied with your support.

system · April 15, 2021, 8:36am

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.