SonarCloud Code Analysis Expected- onoging issues

SonarQube Cloud
,
1

Hi

We are using Github Action for our CI workflows and one of the steps includes SonarCloud scanner using sonarqube.
we have a wierd situation while once the status check of SonarCloud just suck on waiting to be reported although everything seems to be analyzed properly.

basically, when triggering a CI i see the following:

  • sonarqube step ended successfully (pipeline-sonar status check)
  • on SonarCloud UI i see the pr was analyzed successfully
  • i also view the ‘Background Tasks’ on this repository and saw the Successful operation for this specific PR.
  • i do see link for “view the PR” on the SonarCloud UI for this PR which redirect me to the PR on github
  • we do have sonarcloud bot comment regarding the succesfull analysis (“Kudos, SonarCloud Quality Gate passed!”)
  • but still the Status check for the Analysis stuck on waiting for status to be reported.

image
image1716×474 66.4 KB

the main problem , is that it sometimes helps to retrigger the workflow, and sometimes its just works :
so i cant explain why its happening and not sure where to dig more.

i’ve reviewed few other tickets regarding this but none of them included solution for us.
when searching on the web for related issues i’ve noticed a comment about the fact that SonarCloud annotate PR based on the Sha1 in the working tree.

so i’ve reviewed our CI process and saw that when we use the default action of checking out the PR, it ends up with merge commit. so started to think that maybe this cause the PR not to be annotated cause this merge commit and the branch specific commit are not the same.

so i’ve updated our CI to checkout pull request HEAD commit instead of merge commit.
like the following:

    - if: github.event_name == 'pull_request'
            name: Checkout PR
            uses: actions/checkout@v2
            with:
              ref: ${{ github.event.pull_request.head.sha }}

this looks like solved the problem week ago and all the PR got annotated as we expected them to be.
ugly but…what can you do. this last for a week.

problem with SonarCloud started again yesterday. suddenly all sonarCloud steps includes the following message:
“Could not find ref: master in refs/heads, refs/remotes/upstream or refs/remotes/origin”
that’s make sense as we only fetch the exact commit of the PR, but now it only started to yell about it now?

anyway i’ve updated our CI to includes all history so it will have this ref of master also.

        - if: github.event_name == 'pull_request'
                name: Checkout PR
                uses: actions/checkout@v2
                with:
                  ref: ${{ github.event.pull_request.head.sha }}
                  fetch-depth: 0

it solved the issue for one PR but again on other PRs i still see this instability again, where PR got analyzed on the Sonar but the status on the PR stuck on “waiting for status to be reported”.

what’s going on?
what am i missing here?

thanks!

1 Like
2

Hi @clevi

Is this possible for you to give us some background tasks IDs where the status check was not updated correctly ? We’re gonna have a look at our internal logs to try to troubleshoot.

Thanks in advance !
Mickaël

3

Hi Mickael,

thanks alot for your response!
i can share the background tasks yes, the thing its a private project.
is that doable from your side?

anyway Background Task ID=AXNTDQPDg7PgyfVVmUmY

let me know if you want me to share details in personal thread.
thanks!
Chen

4

Thanks !

Do you have the date/time related to this background task as well ?

5

hi

sure,
July 15, 2020 Submitted: 6:17:08 PM, Started: 6:17:08 PM Finished: 6:17:16 PM

10x

6

Perfect.

Well i didn’t see anything particular, except that this specific background task has been executed while GitHub was having hard time (it was the case multiple times last week), so we might have failed calling the webhook to notify the check to be completed.

Is the problem still occurring since yesterday for example ?

7

Hi

yes , its just happened again.

taskID: AXNnCDFqFpdrw9Wpeljb
TimeFrame: July 19, 2020 subbmitted 3:24:17 PM , started: 3:24:17 PM, Finished: 3:24:24 PM

no warning on the analysis nor on the github action itself, but still analysis passed but status check on github is waiting for response.

its becomes pretty much magic.
what can we do?

thanks
Chen

9

If I understood your messages above, this happens on some PRs, not always.

When this happens to a PR, if you trigger further analyses on the PR (by pushing new commits, for example, or close+reopen), does it ever recover? (get out of “Analysis Expected” state) In other words, is the behavior intermittent, or consistent?

11

yes it does.
that’s exactly the main problem, its intermittent and makes our CI really unstable.
i assume you imagine that re-triggering the CI few times before we get proper notification, is not ideal.
do you have any idea why its started happening the last few weeks/months?

is there anything we can do?
thanks
Chen

13

Thanks for confirming!

No, re-triggers are certainly not an option, I just asked to clarify the intermittent nature of the problem.

In our logs we see that the analysis of task AXNnCDFqFpdrw9Wpeljb has correctly triggered the expected GitHub API calls to create a Check Run for commit 90b43147ff996b4d5df512db85816796800bea2f, and the call to GitHub was successful (status 201). Now, is this in fact the HEAD of the PR where you are expecting the check to arrive?

14

yes it looks like it, although this PR already included additional commits and merged.

its happening on other repos currently, so maybe we can check something more alive now?
thanks

15

current issue.

taskID: AXNr2PoPqjKt_9csjCrF
TimeFrame: July 20, 2020 , 1:50:48 PM (finished after4s)

i’ve checked and PR head commit is 51e98e22aa76fc03eefcabec77addef774f57ea2

also the CI is checking out this exact ref, also seen on the log:
checking out ref: commit 51e98e22aa76fc03eefcabec77addef774f57ea2

i get the same issue, i see this PR successfully decorated 17 minutes ago
Kudos, SonarCloud Quality Gate passed!

but status check is still waiting.

thanks
Chen

16

Hi @clevi and thanks,

The analysis you gave was successful on our side, GitHub check was posted well too.
The only thing is that we processed revision 1fcb73787efa6ac28e36a93daf0daae8ad414ba6 with that analysis ID.

Can you figure out where does it come from ?

1 Like
17

Hi @mickaelcaro,
sorry for the late response.

I Could not find this revision on this PR or the mentioned branch.

but i’ve used the Github API to search it and found that its a merge commit which has pretty much the same timestamp of the 51e9… sha1 and includes the following message:
"message": "Merge 51e98e22aa76fc03eefcabec77addef774f57ea2 into 04641147164b97073172648b10ac7aad93adb0c9"

the 51e98e22aa76fc03eefcabec77addef774f57ea2 is the sha1 of this PR.
the 04641147164b97073172648b10ac7aad93adb0c9 is the SHA1 of master.
so it looks to me that it will be the SHA1 of the commit once i’ll merge to master this PR.
(did not know git creates this before actually merging…or i’m missing something).

anyway why is Sonar doesnt analyzed the PR sha1?
not sure what else should i change on the flow so it will analyze the right commit.

is any of these make sense to you?

thanks

18

Hi @clevi

No at least we don’t create any commit beforehand, we are just analyzing “normally” the proper one that has been triggered for a build.

I did some researches and stumbled accross this issue opened on the github/checkout GH action. I think it will make sense for you : https://github.com/actions/checkout/issues/27

TLDR the GitHub/Checkout action will create a phantom merge commit to be sure that your code will build correctly once merge into the target branch.

There’s maybe something to dig into in that matter.

HTH,
Mickaël

19

will check that. (although its relevant to v1 and we already are using v2).

but , in order for us to be able to check the flow properly , is there any way we can get the sha1 the sonar is actually analyzed? like you did in the logs? some verbose flag that will get us this information so i’ll be able to compare that?

thanks

20

No unfortunately that’s still part of our internal logs, but we’re working on making it available in the UI.

21

The scanner analyzes whatever it finds in the working tree, and creates / updates a Check Run for the corresponding SHA1. And I think it makes sense this way, otherwise some line number references in the scanner report may not be consistent with the actual code, which would be confusing. Using the SHA1 of the working tree seems the sane option.

That said, I do recall making a similar exception with Travis CI (maybe), where PR builds use a special ref generated on GitHub called MERGE_HEAD, which is commit generated from the merge (maybe the same thing @mickaelcaro referred to as “phantom commit”). We (used to?) have a check if the analyzed commit matches MERGE_HEAD, then we decorate its parent instead, which should be the HEAD of the PR. Although the scanner report may become inaccurate as explained earlier, in practice it’s unlikely and nobody complained about it. So maybe we can find a similar improvement. (I’m not familiar with GitHub Actions, I’m wondering if they’ve changed something recently, because we haven’t seen such issues before, and recently we start seeing more and more.)

For the time being, your safest option is to make sure the scanner gets executed in a working tree of the SHA1 that corresponds to the HEAD of the PR.

22

thanks Janos,

that’s exactly the change i’ve made on the CI recently but this fixed doesnt seems to be the final solution as we still see this issue.

we used to use the default v1 version of the checkout github aciton, which does the same thing you’ve mentioned , checkout the PR and does a merge commit with master, and runs the CI on this commit.
i have no idea (if nothing changes on sonar side) how it worked in the past without us touching anything.
cause based on what you guys said, it means it should have never worked as the scanner would run on the merged commit which is what we have on the working tree, but when compares to the SHA1 of the head of the branch it would see its not the same and will fail to notify the status check.

i still fail to understand why this ever worked, and also why its still intermittent issue and not constant.

but anyway, few weeks ago i’ve started to see the wierd ‘waiting issue’ again, and updated the CI with v2 of checkout action, while making sure we checkout specfiically the head of the branch, so sonar will analyze this exact commit and will also be able to notify about it (working tree commit will match the head of the branch).

the thing is, this solution also unstable, and although that’s exactly what i did, you saw that the last time this issue happeend, mickael mentioned sonar is analyzing a merge commit.

so to be honest, i lost track here,
and i also unable to identify which commit exactly sonar is analyzing as this information not available.

i also added a command right before running sonarScanner which will print current commit (git --no-pager show --oneline ) and i got the same commit as i expected to see, the sha1 of the PR, so i still dont understand why the scannar analyzes the merged commit. (which doesnt exist anywhere on the working tree).

can we discuss offline to dig into this a bit more, while i’ll provide 1-2 more examples?
i can make any changes needed, its just hard to see the outcome when i cannot see what the scanner is analyzed.

thanks

23

I think, when the merge would be a fast forward, then the MERGE_HEAD would be the same SHA1 as the HEAD of the PR (no merge commit). By this logic, it would always work in a PR that is freshly rebased on its target, and never work when the target has any newer commits since the PR branch started.

Can you please share the relevant CI configuration how you did this?