Bug report: Web API endpoint search_gitlab_repos lists old component data instead of new

lfvjimisola · May 20, 2024, 8:25am

I found what I believe is a bug in one of the endpoint api/alm_integrations/search_gitlab_repos

It will return the GitLab project’s original values for

“sqProjectKey”: “x_y_z_int001_code_AXwmuUqhccjl3LgiYrr0”,
“sqProjectName”: “x.lips3.test.integrations.int001”

instead of the current ones (see e.g. components/show):

“component”: {
“key”: “x_y_z_ms001_code_AXwmuUqhccjl3LgiYrr0”

The two keys are two different components, but the api/alm_integrations/search_gitlab_repos is listing the old (incorrect one).

After some more resource, I found out that there were in fact two components that had the same GitLab project id and the older one was the one showing up in search_gitlab_repos instead of the current one.

I consider this a bug. The API should show the information for the current GitLab projects.

lfvjimisola · May 23, 2024, 4:39pm

@ganncamp Just ping so that the thread does not close due to 7 days.

bogdana · May 28, 2024, 11:16am

Hi @lfvjimisola ,

Thank you for reporting it. I’ll try to look closer at it sometime this week and will get back to you.

Best regards,
Bogdana

lfvjimisola · June 11, 2024, 9:50am

Any updates? I had a meeting with PO Python ecosystem who said that there was something going on behind the scenes (from what she could see), but nothing has been communicated here.

It’s a blocker for us since you, as expected, don’t get the accurate (i.e. up-to-date) information.

bogdana · June 18, 2024, 7:43am

Hi @lfvjimisola,

When you say that:

Does this mean that the old one no longer exists? In the result, do you see just one component? Also, can you maybe share the full request/response? I can open personal messages for that.

Best regards,
Bogdana

lfvjimisola · June 18, 2024, 9:30am

Hi,

But, components exists in JFrog for the particular GitLab project id. But, in GitLab one no longer exists as the group id and artifact id in pom.xml has changed.

I’m not sure if I can reproduce since I had to remove the old component in order to be able to continue.

To reproduce this should work,

create a minimal maven project, artifact id = artifact_org
Create the project in SonarQube (GitLab linked)
Run sonarscanner, confirm that there is data for the project in SonarQube
Change the artifact id (artifact_new) in pom.xml
Run sonarscanner

Now use the API (api/alm_integrations/search_gitlab_repos) and check which component the API returns. In our case it was artifact_org and not artifact_new.

BR,
Jimisola

bogdana · June 18, 2024, 1:15pm

Could you please also clarify the following:

What do you mean by artifact_org?
Are you on Community Edition?
How do you run sonarscanner?
How do you use api/alm_integrations/search_gitlab_repos?

Best regards,
Bogdana

lfvjimisola · June 18, 2024, 1:59pm

<groupId>com.mycompany.app</groupId>
<artifactId>artifact_org</artifactId>

is changed to e.g.

<groupId>com.mycompany.app</groupId>
<artifactId>artifact_new</artifactId>

No, Developer Edition.

maven projects: maven plugin
python projects: pysonar-scanner (ping @jean.jimbo ;))
others: sonarscanner cli

In Java, like this but you can just use the endpoints directly to confirm.

private Mono<String> fetchProjectKey(int projectId, String projectPath, String projectName) {
		return webClient.get()
			.uri(uriBuilder -> uriBuilder.path("api/alm_integrations/search_gitlab_repos")
				.queryParam("almSetting", "LFV GitLab")
				.queryParam("p", "1")
				.queryParam("ps", 100)
				.queryParam("projectName", projectName)
				.build())
			.retrieve()
			.bodyToMono(Map.class)
			.doOnNext(response -> LOG.info("Response for search_gitlab_repos: {}", response))
			.flatMap(response ->

			{
				List<Map<String, Object>> repositories = (List<Map<String, Object>>) response.get("repositories");
				return repositories.stream().filter(repo -> repo.get("id").equals(projectId)).findFirst().map(repo -> {
					String fullSlug = repo.get("pathSlug") + "/" + repo.get("slug");
					LOG.warn("fullSlug: {}", fullSlug);
					if (!fullSlug.equals(projectPath)) {
						LOG.warn(
								"Project with ID {} has most likely changed name, token, key and/or URL as there is a difference in projectPath '{}' vs full slug '{}'",
								projectId, projectPath, fullSlug);
						return Mono.<String>empty();
					}
					return Mono.justOrEmpty((String) repo.get("sqProjectKey"));
				}).orElse(Mono.empty());
			});
	}

bogdana · June 19, 2024, 2:27pm

Hi @lfvjimisola,

I think I finally understand what might be the problem here, it’s sonar.projectKey, which isn’t explicitly specified. Your project key is a unique identifier for your project. If you are using Maven, ensure the key matches the “groupId:artifactId” format as it’s a default value for projectKey. When artifactId was changed, it created new SQ project, which isn’t bound to gitlab and won’t appear in api/alm_integrations/search_gitlab_repos results.
Please, let me know whether it makes sense.

Best regards,
Bogdana

lfvjimisola · June 20, 2024, 7:07pm

As I explained to your colleague, we have created a system to auto-detect sonar key and token without the use of sonar-project.properties.

I could be wrong here, but I don’t think that you are right

The new project is bound to GitLab because it shows if I delete the original component.
The bug here is that there are two components that have a link to the same GitLab project and, for the given endpoint, SonarQube returns the original (non-current) one.

bogdana · June 26, 2024, 10:19am

Hi @lfvjimisola,

From the SQ side, we can’t be sure what the current project is for you. I’d suggest having one project under one Gitlab id and you can achieve it either by deleting the incorrect one or by unbinding it (Project Settings → DevOps Platform Integration).

Best regards,
Bogdana

lfvjimisola · July 16, 2024, 1:12pm

I understand that you can’t be sure which is the current/correct project for the user, but your current “solution” is quite bad. Options are:

as a user it is not obvious that GitLab ID are reused by SonarQube
you are currently returning something whereas there are other BETTER options such as:
a. return the id for the last seen, seen as in last one that you received data for (makes a lot more sense than for the oldest one)
b. return all SonarQube projects linked to a GitLab project id (leave it to the user to decide which one is correct)
c. raise an error message

BR,
Jimisola

bogdana · September 13, 2024, 7:50am

Hi @lfvjimisola

Thank you for your ideas, we’ve created a ticket for this.

Best regards,
Bogdana

lfvjimisola · October 4, 2024, 7:51am

I don’t have access to that Jira althought I have an account.

" xxx@lfv.se doesn’t have access to Jira on sonarsource.atlassian.net .

Can I get that somehow so that I can keep track of this ticket?

Colin · October 4, 2024, 7:58am

I think you’ll need to be logged out of the Atlassian cloud in order to view the ticket since you don’t belong to our Jira. I just opened the ticket in incognito mode so it’s definitely accessible to the public.

It’s not possible to track/watch the ticket without being a member of our Jira… and only we can be members of our Jira (licenses )

lfvjimisola · October 10, 2024, 4:40pm

I can see it but I can’t track/watch which is what I wanted to do.

That said, this issue is really a blocker for us and has been for 6 months or so now.
To my knowledge there is no workaround either.

If you are able then it would be very useful to be able to information such as

created
last update

In order to determine which one that is the current one.

What is the likeliness of anything happening to this bug any time soon?

Colin · October 11, 2024, 6:17am

Hey there.

It might be an issue we can tackle before the upcoming LTA, but I can’t share any guarantee.

I understand the limitations on this one endpoint are frustrating for your automation, but the core functionality of SonarQube is still intact. This is something you’ll have to work around for now.

lfvjimisola · October 11, 2024, 8:32pm

Ok. Fingers crossed it gets in before the upcoming LTA.

Well, define core and define intact…

And just to be clear, there is no workaround for this to my knowledge - I’ve spent hours trying out your APIs without any luck. If there is a workaround (e.g. internal API) then please advise.
But, I’ll go through the web interface again and see what API calls there is to sniff.