First of all I’m a big fan of SonarQube and have been using it successfully since a couple of years in different companies/projects as well private as corporate ones. And please excuse this wall-of-text
At my current organization as well as in some projects before, I hear people start to get annoyed of the system, which brings me here to describe the situation and suggest an improvement regarding the analysis results or the API responses respectively.
As many others have reported, we have a problem with the hard-coded 20 line limit which results in a “Quality Gate OK” if the changes are free of bugs and vulnerabilities.
Of course this depends on the “New Code Period” configuration which we want to be set on “Previous Version”.
Short Set-Up Description:
- SonarQube Server 8.5.1
- We are protecting our master branch by Build Policies which includes a successfull QG -> Only PRs are breaking builds (previously the main branch build got broken too if QG criterie were not met)
- We are one of the key players in the public health system in Germany and luckily data protection is still a valuable asset and it is enforced by law (don’t get me started on politics). Therefore minor changes of code might have a big impact on data security.
The last couple of months one of our Dev teams had to do quite a lot of minor changes. Unfortunately not all of them got covered by sufficient tests as there are some developers with less experience in writing good and meaningful tests.
As a consequence of the 20 line limit, the “bug and vulnerability”-free changes found their way into the main branch but of course the overall coverage dropped a little, someday even below the threshold configured in the quality gate.
Fast-forward a couple weeks, another developer opens a new feature branch which changes more than 20 lines and he/she gets forced to cover also the code of some earlier smaller feature where tests are missing or being incomplete. Only to raise their coverage (depending on the code changes) to be able to have a successfull PR build.
In my opinion if the health and the communication of the team is good enough, this is not a problem as everyone works towards the same goal. Luckily we are in this situation where people are willing to fix the carelessness of other team members. But still everyone is pretty annoyed if something like this happens as it costs extra time to cover someone else’ code changes. Some are even now requesting to completly disable the breaking of builds in Pull Requests. We all know the likely consequences in a couple of months… And degregading SonarQube to be an optional tool where you could check your code if you like leads to the same problems we are facing with a lot of legacy code regarding maintenance (over the years ~20 different devs touched it with completly different style)
In case a feature branch has less than 20 lines of changes, SonarQube does show an appropriate message inside its UI. But I guess, as it is in my company, most developers are kinda lazy and don’t explicitly go over to SonarQube to look for their analysis results. Especially if you have a CI/CD system in place which shows you “green thumbs up” almost nobody looks at ther results. In case of hitting the 20 line limit the CI system gets a QG “OK” message from the SonarQube server. A couple versions earlier there have been more quality gate statuses returned from the server (OK, WARN, ERROR) but the WARN state unfortunately has been removed.
On top of it and to my regret, we moved from Jenkins to Azure DevOps (On-Premise). In Jenkins developers have an easy quick access to SonarQube which had been used by quite a lot developers whereas in Azure DevOps the links to the SonarQube project or the analysis result of a build are shown solely in the Build result view (guess we will have to place some links somewhere else as well). If people are using the (broken) german language pack they even have to scroll down to see the results the SonarQube plugin presents.
So my proposal is the following (either or both would solve the problem):
- make the 20 line limit configurable or
- re-introduce the WARN state which previously got returned if there is something fishy (not broken) about the code or the analysis result, e.g. if you hit the 20 line limit and the expected coverage might be less than the one configured in the QG.
With the WARN state returned there is again the possibility to show corresponding warnings inside the CI systems so that the developers and the PR reviewers are made aware of the possible upcoming situation in future feature branches/PRs
Let me say, that I can understand the introduction of this line limit as I had to hotfix quite a couple production critical bugs in different systems when I was a freelancing developer.
And of course this situation can be eased to some degree if everyone would simply look into the SonarQube analysis results especially if they know that they have changed less than 20 lines. But as you probably know as well, people tend to be lazy and like the comfortable way (nothing wrong with that ) and software can often be adapted more easily than people’s minds.
And I am not seeing me coding workarounds (custom api calls and logic) as this has to be maintained as well.
Do you have any plans on integrating the configuration mentiones above?
I’m looking forward to your reply.
Edit: Might the sonar-build-breaker be at help with the WARN state?