New ce thread dump: ce-aug9.txt (40.9 KB)
Let me know if you want me to try something else.
Thanks. Once again, the logging appender is blocked trying to write logs. It’s not a lock problem, since it’s actually holding the lock.
There are some reports that the receiving end of pipes can stop consuming data causing the writer to block: http://www.jppf.org/tracker/tbg/jppf/issues/JPPF-343 https://issues.apache.org/jira/browse/LOG4J2-880
It’s unclear in what conditions that would happen but I’d try to make sure nothing is logged to the console (stdout/stderr). Did you set any property related to logging in sonar.properties? I think by default nothing is logged to stdout/stderr by default, only to files.
I did not change any of the logging configurations.
I will try to find some time to play with the log settings.
Looking at ConsoleTarget it uses System.out for its output.
So if something in the CE “hijacks” System.out, the problem may be similar to https://jira.qos.ch/browse/LOGBACK-504
Not sure if related. But that bug came up when searching.
Anyway, the curious thing is that the logback hang is reported to trigger when there is large amounts of output.
And I am looking at log-files of a couple of MB. Hardly a lot of data these days.
I will try to increase the log level to see if it triggers the bug in day-to-day operations (instead of just weekends when we build all projects).
Oops! I did change the logging.
But not via the log file, which is where I looked.
I start the java process with -Dsonar.log.console=true - I will try to remove that for a while.
Ok, that could be it.
What I suspect is that the reader of the pipe is not picking up the data, the pipe gets full and the logger hangs trying to write to it. The logger could probably do a better job handling that without blocking while holding a lock, but ultimately the problem is caused by a mismanagement of the pipes attached to the process.
Too early to tell if it makes a difference.
But FYI you have the same option in your official docker image. That is where I copied it from.
I will return with status next week.
SonarQube kept running over the weekend.
So the container option for console log may be the culprit.
I will trigger a full build (like weekends) tonight to see if it keeps running.
@dmeneses SonarQube survived another build storm without hanging.
So for my money, that console output option was the problem.
You should probably change your official docker image.
Cheers!
1 Like
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.