SonarQube New Code analysis is off by 5 hours

Hi,

Did you get a change to look at my last message. Do you need any other clarification on this issue?

Wouter,
Is there a chance to take this as an issue or a bug instead of ‘Help’ and verify/ get help at the earliest

@Wouter_Admiraal @ganncamp Can we check this issue as we are having lot of impact on the analysis and see if this has to be considered as a bug? Your help and Effort is appreciated…

@Wouter_Admiraal @sai

Was this issue ever resolved ? we just upgraded to 8.9 with postgres 12.4, and are having the same issue, sonarqube 8.9 is virtually unusable.

Thanks

@kdivecha No, Issue still persist. I’m waiting for SonarQube Team to verify and provide a fix.

@sai @Alexander_Dragun

We followed Alexander’s suggestion below, it seemed to help with background tasks, try increasing postgres work_mem to 64mb (default is 4mb) assuming you are using postgres, we use aws rds for postgres so we did it via rds param group, reboot sonarqube after updating work_mem just in case.

Thanks

Hello, also I would recommend considering re-analyzing the statistics for the postgres query engine, since in my case, the main problem was that the query planner was building incorrect query plans due to incorrect statistics.

1 Like

We are not using Postgres and as explained above, its nothing related to memory size… I don’t have any issues related to background task because i can see the analysis happening but the analysis on new code wasn’t showing for the 1st 5 hr period (compare from base code) and unable to check with quality Gate.

@sai

  • What is the current new code definition of your project? You can find it under Project Settings > New code.
  • Could you reproduce the issue with all 3 analyses you mentioned (1PM/1:30:PM/6:30PM)? Are those analyses sharing the same version? Please provide activity page screenshots once it is reproduced.

++

@Philippe_Perrin ,
Thanks for the reply.
Please find the details below.

  1. Analysis one:

As you can see total lines of code for the 1st Analysis is 1,343

  1. Analysis two: Added few more lines of code and ran the analysis in less than 30 Min from the initial analysis, but the below screenshot shows 0 new lines

As you can see, the total lines of code is now 1,359. And Lines to cover is still 0 On New Code

  1. Analysis Three: We ran this analysis after few hours which is more that 6 hrs. Added few more lines and now it started showing the Analysis on new code.

In the below screenshot, total lines are now 1,367 and you can see total new lines are 8 and they are not covered

  • Sonar Activity Screenshot: As you can see there is increase in the total lines between 1st and 2nd analysis which didn’t showup. But in the 3rd analysis which we ran today morning (time gap is more than 6 hrs), shows the correct analysis on the new code.

Let me know if you need any other explanation.

Hi sai,

Terribly sorry for the late reply, I’m just back from paternity leave, and did not think of passing this to a colleague before I left.

I’m re-reading all this, and I’m still very, very confused about this “5 hours” thing. Analyses are a sequential thing. SonarQube doesn’t care about the time between different analyses. You could send several of them only seconds apart, it will work. In fact, that’s exactly what we do with our automated QA’s, which are run dozens, if not hundreds of times every day. The fact you have to wait 5 hours before sending a 3rd analysis, which would then only detect new code, sounds like a red herring. I doubt the timing between analyses has much to do with it, unless your Background Tasks queue is completely saturated.

Now, I’m not saying there’s not a bug here somewhere. But I have the feeling concentrating on this “5h” is leading us astray, at least as far as New Code detection is concerned.

I have some extra questions:

  • You said you were using TFS. Are you also using TFS as your SCM (meaning, you’re not using Git or SVN)? If yes, what plugin are you using (as SonarQube doesn’t provide TFS SCM support out of the box)?
  • Is this happening on several projects? Or only a single one? Or perhaps a specific category of projects (language, framework, etc) ?
  • If you submit those 3 analyses in rapid succession, do you see all 3 of them in the Background Tasks? How long before they’re all treated by SonarQube?
  • Are you using DataCenter Edition? If so, do you have several nodes?

@Wouter_Admiraal

I worked on Different Sonarqube Versions before and never had this issue. All i want from you is how can i find whether its Sonarqube Bug or do i need to make any settings inorder to work. I keep saying 5 hours because any new code which is analyzed after it is treated as new code, and anything before that is never considered as new code (It doesn’t matter how many times you run like 100 or 200). I have sent many screenshots with examples for the reference.
All i need is Why this is happening and how can i check and fix it.

For other questions

  • Yes, we use TFVC as our SCM. That being said, we are able to get the analysis on new code after 5 hr period. So i think there is no issue here.
  • It is happening on all the Newly added projects to Sonarqube. We are using C# with .net framework 4.7 or above, .net core 2.2 & 3.1
  • All background tasks are getting completed in regular manner (running about 1 min or so depending on the size of the project). As i said before, its not just 3 but as many as we run for the 1st 5 hr period.
  • Yes, we are using Datacenter Edition. We only have 1 node Instance

Let me know if you need any other information.

OK, and have you tried adjusting your new code period setting? Right now, I see it’s set to 400 days, which might be part of the problem (would be a bug, in that case). I’ve tried reproducing with such a high value, but no luck. Is there a reason for using 400 days as a new code period?

We tried setting it to use Previous Version but it didn’t work. (Worked after 5 hr period again). So everything is revolving around this time period. We also tried setting the Java Time Zone to use CST, New York and UTC and none of them worked. Also tried commenting this setting in sonar.properties file

Hi,

The “new code” in a branch is always the result of comparing the last modification date of each line (usually collected by the scanner from the SCM) with a threshold which was calculated based on the “New Code Period” setting. One of the dates may be wrong due to the timezone.

Could you please check the date when the New Code starts by checking the field leakPeriodDate in the response of
https://my.sonarqube.com/api/components/show?component=[projectKey]&branch=[branch]

Then, for the source code file where you expected to find lines classified as ‘new’, please request:
https://my.sonarqube.com/api/sources/lines?key=[fileKey]&from=[lineNumber]&to=[lineNumber]&branch=[branch]

And check the field scmDate. The file key will be shown in the URL bar when you navigate to the file in SonarQube web interface.

Above is the total analysis details. Let me know your findings

So it looks like the lines were classified as “new” when you get those responses.

Were you expecting the change date of those lines to be 2021-07-13T08:22:05-0500?

My Bad… The one’s above which I mentioned and classified as “New” is the correct response. I was able to see the analysis on new code in sonarqube.

I also updated few lines of code in the project (Between 5 Hr time range) but they were not showing as new code as you can see below.

{
			"line": 47,
			"code": "        <span class=\"k\">public</span> <span class=\"h\">IEnumerable</span>&lt;<span class=\"h\">RefAddressType</span>&gt; <span class=\"sym-10 sym\">GetRefAddressType</span>()",
			"scmRevision": "302208",
			"scmAuthor": "xxxxxx",
			"scmDate": "2021-07-12T08:37:53-0500",
			"duplicated": false,
			"isNew": false
		},
		{
			"line": 48,
			"code": "        {",
			"scmRevision": "302208",
			"scmAuthor": "xxxxxxx",
			"scmDate": "2021-07-12T08:37:53-0500",
			"duplicated": false,
			"isNew": false
		},
		{
			"line": 49,
			"code": "            <span class=\"sym-3 sym\">_logger</span>.LogInfo(<span class=\"s\">\"GetRefAddressType\"</span>, <span class=\"s\">\"Get Ref Address Type started!\"</span>);",
			"scmRevision": "302208",
			"scmAuthor": "xxxxxxx",
			"scmDate": "2021-07-12T08:37:53-0500",
			"utLineHits": 0,
			"lineHits": 0,
			"duplicated": false,
			"isNew": false
		},

Hope this gives you the correct and detailed report.

Thanks.
So the New Code Period started at 2021-07-12T13:28:05-0500 and those lines have a changed date, as reported by the SCM, set to 2021-07-12T08:37:53-0500.
Based on that data it makes sense that they are not considered as “new”.

What one of the dates did you expect to be different?

Thanks