Not sure where this goes but could someone make it easier to get data out of Sonarqube? The API to Sonarqube is not designed to pull large amounts of data out without DOSing the server. IaaC waf blocks the transactions. Pulling data to a SIEM ether DOS or over 4 hours of calls as you have to page through everything and some APIs build on others so double the traffic.
The only solutions I have is:
give the option not to use paging or unlimited entries per call.
Changelog should be able to use a wild card to pull everything.
A way to flag items that have been pulled.
Idealy a way to stream in to a SIEM of your choice would be best but whatever comes to make the data correlation of your choice.
Sure,
To set the stage I have a lot of apps (monolithics and microservices) and even more bugs and code smells. For this example, the SIEM does not matter because it should work for most. I will use terraform for IaaC as it has a plugin for this task and Sonarqube scans it.
API issue 1 IaaC
I just moved my rule maintenance to IaaC so I could have easy change tracking in the GIT (Lab or hub it doesn’t matter) format. This allows the devs to make changes to fit their needs and gives security an audit trail and alerting based on the criteria we need. Yes, you can do some of this VIA the interface BUT we are pushing IaaC for everything we do.
The initial setup of this was over 4500 API calls. Going forward it’s only 2500. Short term I dont think there is any solution for this piece. Also Sonar does not support the TF modual. But should. Terraform Registry
API issue 2 SIEM
I am wanting to aggregate my large amount of data to paint a wholistic picture of how the Org is performing and automate action their in with the logs from the system. So we want and are to pulling the following:
These alert to if there is a system issue: If something goes sideways\down it can be caught with the polling here or lack thereof.
system/health
system/status
system/db_migration_status
ce/activity <-Paging 30ish pulls a time.
Reporting:
measures/component <-good for reports going to excetives but that’s about it.
issues/search <-Good for Pulling Vulns, bugs and codesmells but its a DOS on the API
hotspots/search <-Good for Pulling hotspots but its a DOS on the Project and this API
issues/changelog <-Great API for info but it is the Mother of all DOS (Issues+(Projects+Hotspots)=API Calls
We were able to do this VIA Python and a beast (16 core 32gb ram) of a server and took 4 hours and were not able to pull everything we were looking for consistently. Even pulling just master did fail at times. The interface is lovely to Sonarqube but it sucks to build reports from. There has to be a better way. Issues and hotspots I would just have an option to dump everything (like ps=0 or *) instead of the 500 max and changelog allow for a * in project with the same in the page size.
If i could pull these in to the SIEM, I could coralate between may DAST, SCA, and crowd source pentester to both give leadership a wholistic picture and make some things actionable VIA Micros or AI decisions to block at the WAF or firewall VIA their respective API calls.
Also if you would like more detailed information just DM me.