Background tasks error due to lines limit on pipeline re-run

Gaston_Gamietea · November 25, 2022, 2:17am

Hello,
We are using sonarcloud and we have 79k LOC divided in 6 projects where each project has 4 long lived branches (master, staging, qa, develop) and is currently supporting PR analysis.
Last week we sent a PR on a brand new project that has 2.1k LOC (included in the 79k) and we started getting this error…

This analysis will make your organization 'XYZ' to reach the maximum 
allowed lines limit (having 161226 lines).
Please contact the administrator of the organization to resolve this issue.

and when we re-run the same pipeline we are getting…

This analysis will make your organization 'XYZ' to reach the maximum 
allowed lines limit (having 182398 lines). 
Please contact the administrator of the organization to resolve this issue.

there is clearly something wrong here, by just re running the pipeline I see +21k LOCs in a project that only has 2.1k LOCs? and is worth to mention that there is no new code, we are just adding a long lived branch.

Something is not right or I’m not understanding this… can anyone provide any help or guidance? this is the first time I see this problem, it’s also my first time using the cloud version, not sure if it was a wise choice

Pipeline YAML

definitions:
  caches:
    sonar: ~/.sonar/cache # Caching SonarCloud artifacts will speed up your build
  steps:
    - step: &build-test-custom-sonar
        name: Build, test and analyze on SonarCloud (Custom run)
        caches:
          - node
          - sonar
        script:
          - npm install --quiet --legacy-peer-deps
          - npm run coverage -- --no-watch --no-progress --browsers=ChromeHeadlessNoSandbox
          - npm run sonar

pipelines:
  default:
    - step: *build-test-custom-sonar

ALM used (Bitbucket Cloud)
CI system used (Bitbucket Cloud)
Monorepo Project - Angular (JS/TS)

ganncamp · November 28, 2022, 6:55pm

Hi,

Can you provide the two analysis logs, redacted as necessary? I think they’ll help diagnose the source of the extra ~21k LOC.

Ann

Gaston_Gamietea · November 29, 2022, 1:23pm

first-scan.txt (4.0 KB)
second-scan.txt (4.0 KB)
Thanks for your reply, here you have both SonarScanner Context data…

ganncamp · November 29, 2022, 2:45pm

Hi,

I was looking for the analysis logs rather than the context files.

Tangentially, I see some interesting things in your context:

- sonar.language=js - this parameter hasn’t worked for years. Analysis is always* multi-lingual
- sonar.sources.inclusions=apps/**/src/app I see this, but not a definition of sonar.sources, which means you’re allowing it to default to .. In a monorepo, I suspect that’s not actually what you want
- sonar.test.inclusions=**/*.spec.ts
- sonar.tests.inclusions=apps/**/src/app
Only one of these keys is correct (the first one). But I find it odd that the patterns don’t match.
Additionally, I’m not seeing a definition of the basic sonar.tests.

Ann

^{* There are ways}

Gaston_Gamietea · November 30, 2022, 12:31am

Thank for your answer…

Here is the sonar-project.properties where we add the sources and set inclusions and exclusions
props.txt (865 Bytes)

Here is the first run log (sorry that I misunderstood)
pipelineLog-run-1.txt (24.9 KB)

Here is the second run log
pipelineLog-run-2.txt (25.5 KB)

I get that perhaps the configuration might need to be adjusted but is that a reason to have +21k new LOC? I don’t see how…

ganncamp · November 30, 2022, 3:17pm

Hi,

Thanks for the logs.

Unfortunately, I’m not finding them as helpful as I had hoped.

I’m looking into what may have changed on the SonarCloud side. Could you give me dates for these two analyses?

Ann

ganncamp · November 30, 2022, 4:22pm

As a followup, to be clear, the first analysis log is from one that did not go over the license LOC and the second is one that did?

Ann

Gaston_Gamietea · November 30, 2022, 4:48pm

Hello,
Here are the details…

Gaston_Gamietea · November 30, 2022, 4:52pm

the first execution throw the error but when I tried to re-run the pipeline I see the same error but this time had +21k new LOC…

ganncamp · November 30, 2022, 5:23pm

Hi,

Let’s back up a little.

You have two failing analyses of a 2.1k project, both reporting 100k+ LOC. I agree that the 21k difference between the two LOC counts is strange. At the same time, I think there’s obviously a bigger problem. If we take the smaller of the two analyses as a baseline: where did the extra 159.1k LOC come from?

So now I’d like to come back to this:

You asserted that sonar.sources is set in your sonar-project.properties file, and thoughtfully provided it. In that file I see this:

sonar.sources=apps/ui-1/src/app,apps/ui-2/src/app,apps/ui-3/src/app,apps/apis/CXXX/src/app,apps/apis/FXXX/src/app,apps/apis/IXXX/src/app,apps/apis/RXXX/src/app,apps/apis/SXXX/src/app,apps/apis/TAXXX/src/app,apps/apis/TXXX/src/app
sonar.sources.inclusions=**/*.ts
...
sonar.exclusions=**/*.html,**/*.scss

Which doesn’t match at all the final analysis properties seen by the server:

Project scanner properties:
  - sonar.host.url=https://sonarcloud.io
  - sonar.javascript.lcov.reportPaths=coverage/apps
  - sonar.language=js
  - sonar.organization=XYZ-ORGANIZATION
  - sonar.projectBaseDir=/opt/atlassian/pipelines/agent/build
  - sonar.projectKey=XYZ-ORGANIZATION_XYZ-ORGANIZATION-monorepo
  - sonar.projectName=Monorepo
  - sonar.projectVersion=5.0.0
  - sonar.scanner.app=ScannerNpm
  - sonar.scanner.appVersion=2.8.2
  - sonar.scm.provider=git
  - sonar.sourceEncoding=UTF-8
  - sonar.sources.inclusions=apps/**/src/app
  - sonar.test.inclusions=**/*.spec.ts
  - sonar.tests.inclusions=apps/**/src/app
  - sonar.typescript.lcov.reportPaths=coverage/apps
  - sonar.working.directory=/opt/atlassian/pipelines/agent/build/.scannerwork

Note that nowhere in there is sonar.exclusions, and sonar.inclusions has an entirely different value.

From this, I conclude

your sonar-project.properties file is not being picked up
sonar.sources is defaulting to ., rather than the carfully curated list you provided
- Is it possible there are extra random files in .? The two analysis logs you provided don’t indicate that they find different numbers of files, but at this point I feel like everything’s up for grabs.

I think we need to step back to making sure analysis is running with the properties you intended, and if there are still LOC anomalies, we can look at them then.

Ann

Gaston_Gamietea · December 1, 2022, 7:20pm

Thank you so much Ann…

Indeed, I’ve just run a new clean execution of the same branch and now the sonar context is picking up the correct properties file , not sure what happened with those 2 executions, but you were right seeing it from the +150k

I’ve notice that in the FAILED execution the context had this value… notice that server settings has one prop…

Project server settings:
  - sonar.branch.longLivedBranches.regex=(master|staging|qa|develop)
Project scanner properties:
  - sonar.host.url=https://sonarcloud.io
  - sonar.javascript.lcov.reportPaths=coverage/apps
  - sonar.language=js
  - sonar.organization=xyz-organization
  - sonar.projectBaseDir=/opt/atlassian/pipelines/agent/build
  - sonar.projectKey=xyz-organization_xyz-organization-monorepo
  - sonar.projectName=Monorepo
  - sonar.projectVersion=5.0.0
  - sonar.scanner.app=ScannerNpm
  - sonar.scanner.appVersion=2.8.2
  - sonar.scm.provider=git
  - sonar.sourceEncoding=UTF-8
  - sonar.sources.inclusions=apps/**/src/app
  - sonar.test.inclusions=**/*.spec.ts
  - sonar.tests.inclusions=apps/**/src/app
  - sonar.typescript.lcov.reportPaths=coverage/apps
  - sonar.working.directory=/opt/atlassian/pipelines/agent/build/.scannerwork

here is the success execution, notice that server settings has two… not sure if this could be something? maybe?

Project server settings:
  - sonar.branch.longLivedBranches.regex=(master|staging|qa|develop)
  - sonar.pullrequest.github.summary_comment=false
  - sonar.pullrequest.provider=BitbucketCloud
Project scanner properties:
  - sonar.exclusions=**/*.html,**/*.scss
  - sonar.host.url=https://sonarcloud.io
  - sonar.javascript.lcov.reportPaths=./coverage/lcov.info
  - sonar.language=js
  - sonar.organization=xyz-organization
  - sonar.projectBaseDir=/opt/atlassian/pipelines/agent/build
  - sonar.projectKey=xyz-organization_xyz-organization-monorepo
  - sonar.projectName=Monorepo
  - sonar.projectVersion=5.0.0
  - sonar.scanner.app=ScannerNpm
  - sonar.scanner.appVersion=2.8.2
  - sonar.scm.provider=git
  - sonar.sourceEncoding=UTF-8
  - sonar.sources=apps/ui-1/src/app,apps/ui-2/src/app,apps/ui-2/src/app,apps/apis/cxxx/src/app,apps/apis/fxxx/src/app,apps/apis/ixxx/src/app,apps/apis/rxxx/src/app,apps/apis/sxxx/src/app,apps/apis/txxx/src/app,apps//apis/trxxx/src/app
  - sonar.sources.inclusions=**/*.ts
  - sonar.test.inclusions=**/*.spec.ts
  - sonar.tests=apps/ui-1/src/app,apps/ui-2/src/app,apps/ui-2/src/app,apps/apis/cxxx/src/app,apps/apis/fxxx/src/app,apps/apis/ixxx/src/app,apps/apis/rxxx/src/app,apps/apis/sxxx/src/app,apps/apis/txxx/src/app,apps//apis/trxxx/src/app
  - sonar.working.directory=/opt/atlassian/pipelines/agent/build/.scannerwork

Anyway, pipeline is working as expected now, the issue is no longer there…
Thank you… happy to provide more details if needed

ganncamp · December 2, 2022, 1:06pm

Hi,

I’m glad this is working for you.

Just to tidy up, the difference in the server settings in interesting, but since that’s just a report - from the server - of the values set on the server, I’m going to guess that maybe you added settings before the latest run…?

Anyway, that’ll be something to look at if there are future problems, but for now I don’t think it’s something we need to worry about.

Ann

Gaston_Gamietea · December 2, 2022, 2:24pm

Thanks again Ann, without your help I’d still be thinking what’s happening…

Indeed, I’ve changed the server props after the error, it was one of those things that you do during the troubleshooting process.

I’ll mark your previous comment as the solution because that one took me to the right place to have a closer look at the config…

Once again, thank you for your time and response…
Happy holidays

system · December 9, 2022, 2:25pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.