Bitbucket memory limit exceeded

SonarQube Cloud
,
1
  • Using Bitbucket and bitbucket pipelines
  • Python project
  • Error:

Container ‘docker’ exceeded memory limit.

Sonar command:

      - pipe: sonarsource/sonarcloud-scan:1.0.1
        variables:
          DEBUG: "true"
          SONAR_TOKEN: ${SONARCLOUD_TOKEN}
          EXTRA_ARGS: -Dsonar.host.url=https://sonarcloud.io
            -Dsonar.organization=<redacted>
            -Dsonar.projectKey=<redacted>
            -Dsonar.projectVersion=${BITBUCKET_BRANCH}-${BITBUCKET_BUILD_NUMBER}
            -Dsonar.pullrequest.bitbucketcloud.repository=${BITBUCKET_REPO_UUID}
            -Dsonar.pullrequest.bitbucketcloud.owner=${BITBUCKET_REPO_OWNER_UUID}
            -Dsonar.python.xunit.reportPath=test-results/results.xml
            -Dsonar.python.coverage.reportPaths=test-results/coverage.xml
            -Dsonar.sources=src
            -Dsonar.exclusions=src/tests/**
            -Dsonar.python.bandit.reportPaths=test-results/bandit-report.json
            -Dsonar.python.pylint.reportPath=test-results/pylint.txt

Last successful run: 2020-06-16 00:03 UTC
First failed run: 2020-06-16 14:35 UTC

Everything was working fine yesterday. Today every pipeline run across several branches is failing with the docker memory error - including rerunning builds that worked yesterday. It always fails in the midst of the rules execution.

What I’ve tried: I’m already using the size: 2x option. I’ve tried setting the memory up to 10G instead of the 8 that is the default. That seems to have little to no effect. Also, normally we don’t have the DEBUG option enabled. I’ve confirmed that we didn’t make any configuration changes recently.

Last few logs from the debug for one of the failed runs:

20:05:43.206 INFO: 117/117 source files have been analyzed
20:05:43.206 INFO: Starting rules execution
20:05:43.207 INFO: 117 source files to be analyzed
20:05:44.220 DEBUG: Not enough content in 'src/extraction/__init__.py' to have CPD blocks, it will not be part of the duplication detection
20:05:55.006 INFO: 7/117 files analyzed, current file: src/assignment/condition_value_preprocess_functions.py
20:06:05.409 INFO: 8/117 files analyzed, current file: src/models/main_model.py
20:06:15.781 INFO: 8/117 files analyzed, current file: src/models/main_model.py
20:06:26.069 INFO: 8/117 files analyzed, current file: src/models/main_model.py
20:06:37.022 INFO: 8/117 files analyzed, current file: src/models/main_model.py
20:06:49.771 INFO: 8/117 files analyzed, current file: src/models/main_model.py
20:07:01.451 INFO: 8/117 files analyzed, current file: src/models/main_model.py
20:07:11.938 INFO: 8/117 files analyzed, current file: src/models/main_model.py
20:07:22.109 INFO: 8/117 files analyzed, current file: src/models/main_model.py
20:07:32.790 INFO: 8/117 files analyzed, current file: src/models/main_model.py
20:07:43.428 INFO: 8/117 files analyzed, current file: src/models/main_model.py
20:07:54.300 INFO: 8/117 files analyzed, current file: src/models/main_model.py
20:08:05.310 INFO: 8/117 files analyzed, current file: src/models/main_model.py
20:08:16.227 INFO: 8/117 files analyzed, current file: src/models/main_model.py
20:08:26.912 INFO: 8/117 files analyzed, current file: src/models/main_model.py
20:08:37.566 INFO: 8/117 files analyzed, current file: src/models/main_model.py
20:08:48.587 INFO: 8/117 files analyzed, current file: src/models/main_model.py
time="2020-06-16T20:08:51Z" level=error msg="error waiting for container: unexpected EOF"
2

Hi @David_Price,
thank you for reporting.
Is it possible for you to share (even privetely in direct message) what’s inside “src/models/main_model.py” file?

As a workaround, you may try increasing the maximum memory allocation for the JVM (flag -Xmx)
by setting environment variable SONAR_SCANNER_OPTS

For example:

export SONAR_SCANNER_OPTS="-Xmx2048m"
3

I could share it privately, but I’m not sure it is relevant. every time it runs it seems to be a different file - this was just one example.

4

I’d like to explore what possible changes may have occurred that could possibly lead to this problem.

  • We know our code didn’t cause the issue since it affects all branches and running any pipeline that was previously successful now errors
  • We know the version of sonarcloud didn’t change because we are specifying the version number in the command (namely: 1.0.1)
  • I assume that configuration of rules might be a possible cause? Is there anything I can look for or try changing to correct things there?
  • Is it possible that bitbucket might be doing something differently now that could have caused this?
  • Are there other things that could have triggered the issue?

I just ran it with the suggested JVM memory settings, and it did run longer than normal but still crashed with the same error. This time it spent 7 minutes on the first file, 1 minute on the second and crashed 20 seconds into the 3rd file (none of which is the src/models/main_model.py that we saw on the earlier example.

Is there any extra debugging we could turn on to figure out more information about why it seems to be taking so long applying rules?

5

Hey @David_Price,

We recently deployed a new version of the python analyzer (2.12) on SonarCloud and we’ve just discovered a memory problem in one of the new rule (S5806).

The fix will be available at the end of next week with the release of new version of Python analyzer.

In the meantime could you try to deactivate rule S5806 and see if that helps?

6

Since that was a new rule, we hadn’t added it to our profile yet, so I think it is already disabled?

7

It depends how you defined your quality profile: if it inherits from “Sonar Way” then you’ll have the new rule activated by default.

Can you please double check your quality profile?

8

We don’t inherit. However, we did get the JVM settings to work to get through the issue - we just had to set it as a variable rather than a environment variable (and 512 seems to be enough for us). i.e.:

      - pipe: sonarsource/sonarcloud-scan:1.0.1
        variables:
          SONAR_SCANNER_OPTS: -Xmx512m
          SONAR_TOKEN: ${SONARCLOUD_TOKEN}
          EXTRA_ARGS: -Dsonar.host.url=https://sonarcloud.io
9

I see, thanks for sharing!

10

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.