SQ 6.7.7: Increasing pending tasks - compute engine stopped processing tasks

(Mirko Friedenhagen) #1

Hello,

looking at the “Compute Engine” section of System Info I see:

Compute Engine Tasks

Pending
40
In Progress
1
Processed With Error
2
Processed With Success
891
Processing Time (ms)
4347759
Worker Count
1
Max Worker Count
10

New tasks are not processed anymore and some projects started to fail as they use the build-breaker plugin to wait for background task completion.

I now restarted SQ and the queue size decreases again, however the worker-count was always 1 and the CE seems not to spawn more workers. Where may I look for problems?

These are the settings diverging from the default ones, the host has 8GB RAM and 4 vCPUs, the database is PostgreSQL 10 running on another host.

# Security Settings
sonar.security.realm=${env:SECURITY_REALM}
sonar.security.localUsers=admin,sonar
sonar.security.savePassword=true
ldap.url=ldaps://ldap.example.org:636
ldap.bindDn=XXXXX
ldap.bindPassword=${env:LDAP_PASSWORD}
ldap.user.baseDn=XXXX
ldap.user.realNameAttribute=cn
ldap.user.emailAttribute=mail
ldap.group.baseDn=YYYYY
ldap.group.request: (&(objectClass=organizationalUnit)(member={dn}))
ldap.group.idAttribute=cn

# Logging
sonar.log.console=true
sonar.path.logs=/opt/sonarqube/log/

# Delay (in seconds) between processing of notification queue
sonar.notifications.delay=60

# UPDATE CENTER
sonar.updatecenter.activate=true
http.proxyHost=${env:UI_HTTP_PROXY_HOST}
http.proxyPort=3128
https.proxyHost=${env:UI_HTTP_PROXY_HOST}
https.proxyPort=3128
http.nonProxyHosts=127.0.0.1|localhost|*.lan

# Compute-Engine
sonar.ce.javaOpts=-Xmx1g -Xms256m -XX:+HeapDumpOnOutOfMemoryError

# JVM options of Elasticsearch process
# https://www.elastic.co/guide/en/elasticsearch/reference/5.5/vm-max-map-count.html
sonar.search.javaOpts=-Xms1g -Xmx1g -XX:+HeapDumpOnOutOfMemoryError

Thanks
Mirko

(Julien Lancelot) #3

Hi @mfriedenhagen,

Could you please have a look in logs/ce.log for lines containing ERROR ?

Regards,
Julien Lancelot

(Mirko Friedenhagen) #4

Hello @julienlancelot,

I took a look at the CE.log and found a lot of error messages indicating problems when sending Email:

2019.05.06 12:27:14 ERROR ce[AWqNFC18pN5NaeWrS4L2][o.s.s.n.e.EmailNotificationChannel] Unable to send email
org.apache.commons.mail.EmailException: Sending the email to the following server failed : mri.EXAMPLE.COM:25
	at org.apache.commons.mail.Email.sendMimeMessage(Email.java:1469)
	at org.apache.commons.mail.Email.send(Email.java:1496)
	at org.sonar.server.notification.email.EmailNotificationChannel.send(EmailNotificationChannel.java:184)
	at org.sonar.server.notification.email.EmailNotificationChannel.deliver(EmailNotificationChannel.java:129)
	at org.sonar.server.notification.email.EmailNotificationChannel.deliver(EmailNotificationChannel.java:104)
	at org.sonar.server.notification.NotificationService.dispatch(NotificationService.java:84)
	at org.sonar.server.notification.NotificationService.deliver(NotificationService.java:73)
	at org.sonar.server.computation.task.projectanalysis.step.SendIssueNotificationsStep.sendIssueChangeNotification(SendIssueNotificationsStep.java:135)
	at org.sonar.server.computation.task.projectanalysis.step.SendIssueNotificationsStep.processIssues(SendIssueNotificationsStep.java:124)
	at org.sonar.server.computation.task.projectanalysis.step.SendIssueNotificationsStep.doExecute(SendIssueNotificationsStep.java:100)
	at org.sonar.server.computation.task.projectanalysis.step.SendIssueNotificationsStep.execute(SendIssueNotificationsStep.java:90)
	at org.sonar.server.computation.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:64)
	at org.sonar.server.computation.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:52)
	at org.sonar.server.computation.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:73)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.executeTask(CeWorkerImpl.java:134)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:97)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.withCustomizedThreadName(CeWorkerImpl.java:81)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:73)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:43)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.mail.SendFailedException: Invalid Addresses
	at com.sun.mail.smtp.SMTPTransport.rcptTo(SMTPTransport.java:2014)
	at com.sun.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:1258)
	at javax.mail.Transport.send0(Transport.java:255)
	at javax.mail.Transport.send(Transport.java:124)
	at org.apache.commons.mail.Email.sendMimeMessage(Email.java:1459)
	... 26 common frames omitted
Caused by: com.sun.mail.smtp.SMTPAddressFailedException: 550 unrouteable address

	at com.sun.mail.smtp.SMTPTransport.rcptTo(SMTPTransport.java:1867)
	... 30 common frames omitted

I checked the configuration and was able to send a test email from https://sonarqube.EXAMPLE.COM/admin/settings, so maybe the mail was sent to an inactive former employee. Unfortunately neither the project which triggered the email nor the recipient are logged. I am not in charge of mri.EXAMPLE.COM, so would have to configure a MTA-proxy for more details.

Another error I saw:

2019.05.06 12:00:31 ERROR ce[AWqNBE7MpN5NaeWrS4Lp][o.s.c.t.CeWorkerImpl] Failed to execute task AWqNBE7MpN5NaeWrS4Lp
org.sonar.server.computation.task.projectanalysis.component.VisitException: Visit failed for Component {key=COM.EXAMPLE:MYAPP:src/main/java/COM/EXAMPLE/sse/crmcase/adapter/CrmCaseAdapter.java,type=FILE}  located COM.EXAMPLE:MYAPP:src/main/java/COM/EXAMPLE/sse/crmcase/adapter(type=DIRECTORY)->COM.EXAMPLE:MYAPP(type=PROJECT)
    at org.sonar.server.computation.task.projectanalysis.component.VisitException.rethrowOrWrap(VisitException.java:44)
    at org.sonar.server.computation.task.projectanalysis.component.PathAwareCrawler.visit(PathAwareCrawler.java:52)
    at org.sonar.server.computation.task.projectanalysis.component.PathAwareCrawler.visitChildren(PathAwareCrawler.java:87)
    at org.sonar.server.computation.task.projectanalysis.component.PathAwareCrawler.visitImpl(PathAwareCrawler.java:70)
    at org.sonar.server.computation.task.projectanalysis.component.PathAwareCrawler.visit(PathAwareCrawler.java:50)
    at org.sonar.server.computation.task.projectanalysis.component.PathAwareCrawler.visitChildren(PathAwareCrawler.java:87)
    at org.sonar.server.computation.task.projectanalysis.component.PathAwareCrawler.visitImpl(PathAwareCrawler.java:70)
    at org.sonar.server.computation.task.projectanalysis.component.PathAwareCrawler.visit(PathAwareCrawler.java:50)
    at org.sonar.server.computation.task.projectanalysis.step.NewCoverageMeasuresStep.execute(NewCoverageMeasuresStep.java:109)
    at org.sonar.server.computation.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:64)
    at org.sonar.server.computation.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:52)
    at org.sonar.server.computation.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:73)
    at org.sonar.ce.taskprocessor.CeWorkerImpl.executeTask(CeWorkerImpl.java:134)
    at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:97)
    at org.sonar.ce.taskprocessor.CeWorkerImpl.withCustomizedThreadName(CeWorkerImpl.java:81)
    at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:73)
    at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:43)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: There's no changeset on line 42
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
    at org.sonar.server.computation.task.projectanalysis.scm.ScmInfoImpl.getChangesetForLine(ScmInfoImpl.java:66)
    at org.sonar.server.computation.task.projectanalysis.scm.DbScmInfo.getChangesetForLine(DbScmInfo.java:71)
    at org.sonar.server.computation.task.projectanalysis.step.NewCoverageMeasuresStep$NewCoverageCounter.initialize(NewCoverageMeasuresStep.java:274)
    at org.sonar.server.computation.task.projectanalysis.formula.FormulaExecutorComponentVisitor.processLeaf(FormulaExecutorComponentVisitor.java:165)
    at org.sonar.server.computation.task.projectanalysis.formula.FormulaExecutorComponentVisitor.process(FormulaExecutorComponentVisitor.java:142)
    at org.sonar.server.computation.task.projectanalysis.formula.FormulaExecutorComponentVisitor.visitFile(FormulaExecutorComponentVisitor.java:122)
    at org.sonar.server.computation.task.projectanalysis.component.PathAwareCrawler.visitNode(PathAwareCrawler.java:105)
    at org.sonar.server.computation.task.projectanalysis.component.PathAwareCrawler.visitImpl(PathAwareCrawler.java:73)
    at org.sonar.server.computation.task.projectanalysis.component.PathAwareCrawler.visit(PathAwareCrawler.java:50)
    ... 23 common frames omitted
2019.05.06 12:00:31 ERROR ce[AWqNBE7MpN5NaeWrS4Lp][o.s.c.t.CeWorkerImpl] Executed task | project=COM.EXAMPLE:MYAPP | type=REPORT | id=AWqNBE7MpN5NaeWrS4Lp | submitter=XXX | time=429ms
(Julien Lancelot) #6

I’ve had a better look at your issue, and I now understand that your real issues is really about the 40 pending tasks and the fact that you are not able to activate more workers.

Such issues can be very tricky to investigate, and as you are using at least an enterprise edition, you may have access to our support team which is very good to deal with such issue.

So I highly recommend you to get in touch with them.