We have a C++ project we run CI against using Azure DevOps and SQ server. In a feature branch we upgraded to C++ 20 from 14. The SQ scan passed on the branch. The scan also passed on the PR (branch ‘merge’). But after merging the PR into master the Quality Gate failed on the master branch!
Sample issues that have appeared are cpp:S6186 and cpp:S6005. Note these are both issues that would appear after upgrading to C++ 20 or 17 respectively.
“New code” for this project is defined as against the previous version (git release tag) for the master branch, and for all other branches ‘new code’ is relative to master as the reference branch.
Please can someone explain why SQ passed the feature and PR branches but not the master branch? This kind of defeats the purpose of using SQ to check for issues if they only pop up after they’ve been merged into master!
I won’t say this is a “classic” example of the genre, but it’s a pretty good one. PR analysis raises issues on the code that was changed in the PR. Your PR to upgrade to C++ 20 changed very little - if any - code if I’m not mistaken. Probably it just a few configurations, right?
Unfortunately - as you’ve discovered - PR analysis really isn’t meant for this sort of use case. I know the horse has already left the barn, but for next time, I would advise doing a branch analysis, rather than just a PR analysis for this sort of “change the underlying rules” scenario.
Thanks Ann, but I did do a branch analysis - that’s what I meant by feature branch above - and SQ passed it. And SQ passed the PR too. It’s only once the changes were merged into master that SQ failed it. So SQ was perfectly capable of detecting the changes.
It was the g++ compiler flag option in the Makefile that was changed (from -std=c++14 to -std=c++20). So it wasn’t C++ code as such that was changed, but the Makefile is under source control and was listed among the changed files for the feature branch and the PR (some C++ code was changed as well).
So why did SQ only fail the Quality Gate after the merge into master?
The feature branch no longer exists, but the analysis results for it are still in SQ at the moment. Of the few issues I’ve sampled (> 750 were reported!!) they do seem to be listed by SQ in the feature branch. They just didn’t cause the Quality Gate to fail even though they were effectively new issues because the since-c++17 and since-c++20 rules were now being triggered.
Correct. Only in the master branch after the PR was merged in.
I already mentioned this in the OP. All branches except master use master as the reference branch. Master itself uses ‘previous version’ (based on the value of sonar-scanner -Dsonar.projectVersion=<git_tag_of_latest_release>). But many of the files which SQ is now complaining about have not changed since the last release - i.e., they’re not new code.
It’s like that portal card you pointed me at (and which I upvoted) - they’re new issues in old code.
Well… digging deeper into the docs… The reference branch discussion talks about using SCM data to detect new code. So I suppose it’s the same problem as with PR analysis and that portal card is really the best path forward if we can ever get it acted on.
As a side note, I pinged the C-Family Squad about your situation and was told they faced exactly the same problem with their upgrade to C++20. In case it helps, their “solution” was to
bulk add a tag to all the new issues and accept them temporarily. We then created a JIRA ticket to handle them when we can allocate the necessary time. See CPP-5557
I’m sorry that, even retroactively, I don’t have a good answer for you. And I’m going to flag this for the PMs so they’re aware of the frustration.
Thanks Ann - I really appreciate you flagging this issue, as it really caught our dev team out when we saw SQ failing the master branch with a whole bunch of new issues!
Yes, that’s what I did too to get the builds on master passing for now - bulk accept and tag them and link them to a separate DevOps work item for addressing the issues some time in the future.