Hi all,
Join us for our October AI-focused Webinar!
Join @manish.k & @alexander.rage to gain insights on harnessing the full potential of GenAI while ensuring a secure and efficient development process.
Title: > Introducing Sonar AI Code Assurance and AI CodeFix
Date & Time: October 16th, 5pm CEST / 10am CDT.
Speakers: Manish Kapur, Sr Director, Product and Solutions & Alexander Rage, Product Manager.
Hi all,
Thanks to all who attended our session! Find below all the questions that were asked during our webinar:
Q: How can you ensure security when using AI CodeFix? Will my code be shared with external parties?
A: We are sending a code snippet to the LLM and receiving the code fix suggested response through a secure channel. Additionally, Sonar does not use or permit the use of your private code to train any LLM. With our secure practices to handle your data, we’re confident none of your code will be leaked if you wish for it to remain private.
Q: What specific LLM does SonarQube use for training? How do you ensure it uses the most updated components/bug fixes in supported programming languages?
A: We are currently using an OpenAI LLM for the suggested coding fixes. We have tested several LLMs, and this one is giving us the best results based on our stringent criteria. If we find an LLM that we discover is better, we will switch to the best-performing LLM.
Q: What are the privacy considerations to be aware of when using Sonar AI features on private code repositories?
A: AI CodeFix submits a small code snippet (not all your code) to the LLM to generate fix suggestions. Sonar does not use or permit the use of your private code to train any LLM.
Q: How do you rate your security focus compared to Veracode or Checkmarx? How about security-related metrics?
A: We believe that our solutions have strong security coverage and we are continuously adding more security rules and enhancing our secrets detection capabilities. Sonar’s deeper SAST is a powerful feature that allows you to detect unknown and deeply hidden vulnerabilities that can arise out of user code interaction with open-source libraries. This feature is enabled by default and is available in all commercial editions of our products.
Q: Our lawyers are very concerned about the license risk of third-party components. So, we’re not allowed to use AI-generated code unless we can assure them the AI didn’t accidentally use a snippet of open-source code from GitHub, etc., that falls under a copyleft license, such as AGPL. Does this AI Assurance product also look at license info?
A: AI Code Assurance currently does not identify license or copywrite violations. It is an extremely hard issue to find this reliably. However, we know this is an important problem and we are actively investigating a solution.
Q: Will it scan my security settings in the development environment and use my code to train your model?
A: No, your code will not be used for training purposes. Sonar does not use or permit the use of your private code to train any LLM.
Q: How do you deal with data security and privacy?
A: We are testing and evaluating the quality of LLM to provide secure and reliable results for the code fix suggestions. In terms of privacy, the LLM we are using is trained on data in the public domain. Since we rely on an OpenAI model for the code fix suggestions, we are thus relying on the model to determine what’s allowed from the public domain.
Q: How do you ensure that your fix code does not affect the rights/licenses of third parties?
A: Currently, we have no way to know the origin of the suggested fix results from AI CodeFix. We’re investigating this because it’s an important problem to solve.
Q: Where is my code analyzed? it this a 100% in-house solution? Do we need to connect to external servers for this? we use SonarQube.
A: Code analysis in SonarQube is done using analyzers included in the product. You can run local analyzers or integrate with common DevOps platforms such as GitHub, Azure DevOps, etc.
Q: So, AI (OpenAPI) is only used for AI CodeFix as far as I can understand?
Before enabling this feature, can it be made explicit that data is, in fact, shared with company X? Additionally, can you ensure that you do not introduce AI from third parties without explicitly confirming by the customer outside of “AI CodeFix”?
A: To enable AI CodeFix in SonarQube and SonarCloud, you must accept the terms of service, which explain that we’re using a third-party LLM. Sonar does not send any customer data to other third-party LLMs outside of the AI CodeFix feature.
Q: If we are using SonarQube Server, does anything get sent to the cloud for processing?
A: Yes, we are sending small code snippets to an OpenAI model that generates code fix suggestions.
Q: Since you are using OpenAI - how do you protect our code from being exploited against us?
A: Information including a snippet of your code is sent to the LLM to generate a code fix suggestion in a secure way. That information is sent over a secure connection. The response is sent back over a secure connection. The information we send is not persisted in the LLM or used for training purposes.
Q: Without sending the context of the whole project, it seems that suggested solutions will not be as congruent with existing code in the project as they could be. I understand not sharing widely for security reasons, but it would be helpful to share more context for accuracy and usefulness.
A: You are right. We are committed to not sharing your complete code. Even if the complete code was sent, LLMs can still hallucinate. This is an area where we constantly researching to improve the accuracy of code fix suggestions without compromising your data privacy. One of the ways you can help us is by taking the in-app survey and providing us feedback on the code fix suggestions.
Q: If you are using an OpenAI LLM that would give problems because their term of service means that all code is no longer private, how are you securing customer code?
A: According to the OpenAI Terms of Service, you retain ownership of all Input (including your code). https://openai.com/policies/row-terms-of-use/
Q: Is SonarSource collecting Community data? How are prompts managed? Stored? Evaluated?
A: We handle the information we send to the LLM in a proprietary way to get the best response from the LLM.
Q: Will AI Code assurance also check if the code has any license/proprietary issues?
A: Not at this time. We are investigating this capability as it is an important problem to solve.
Q: Are these AI features available in SonarQube only or also in SonarCloud and SonarLint?
A: Sonar AI CodeFix is available in both SonarQube (Enterprise Edition and higher) and SonarCloud (Team and Enterprise plans). Sonar AI Code Assurance is available in SonarQube commercial editions and is planned to be released in SonarCloud within a month or so.
AI CodeFix is not available in SonarLint as of now. We are looking into this. However, as of today, you can open code fix suggestions generated in SonarQube and SonarCloud directly in the IDE if SonarLint is in connected mode.
Q: Which is the latest version of SonarLint that does not contain AI Coding Assistant?"
A: SonarLint does not contain an AI coding assistant. However, you can use one in parallel with SonarLint.
Q: Will CodeFix be included within the SonarQube Edition after the Early Access too? Or will this become a separate product
A: AI CodeFix will continue to be available to users with SonarQube Enterprise Edition and higher after the Early Access ends. Stay tuned for more details about how we will make it available.
Q: Is this feature available in an open-source version?
A: No, AI features are only available in our commercial editions.
Q: is SonarQube the same product as SonarCloud?
A: SonarCloud has most of the core functionality of SonarQube, and we are continuously adding more to achieve as much feature parity as possible between the two.
Q: Are the additional costs planned once it is no longer early access?
A: Yes, it is free in early access. We don’t yet have any details to share about AI CodeFix when it becomes a GA feature.
Q: Can SonarLint generate a fix for an issue detected in real-time by SonarLint?
A: AI CodeFix is a feature in SonarQube or SonarCloud. If you have set up SonarQube or SonarCloud in connected mode with SonarLint, you can click on an issue in the SonarQube or SonarCloud UI. This will open your IDE to the line of code where the issue is located and apply the fix.
Q: How is SonarLint different from the SonarQube AI CodeFix? As I understand, CodeFix suggests the fix, and SonarLint does the same thing.
A: SonarLint does not have the ability to suggest an AI-generated code fix. SonarLint can detect an issue and tell you why it’s an issue, but not suggest a fix for you.
Q: To follow up on your answer above, does that mean the AI Code Assurance and AI CodeFix are not included in SonarQube 10.7? Thank you.
A: AI CodeFix is available in SonarQube 10.7 Enterprise Edition and higher in Early Access. AI Code Assurance is available in all commercial editions of SonarQube 10.7.
Q: We have SonarQube Enterprise 9.9.7. Do we have to enable this feature somewhere, or is it automatically running by default?
A: AI CodeFix and AI Code Assurance are not available in SonarQube 9.9.x. They are available starting from version 10.7.
Q: It is mentioned that SQ 10.8 would be required for these new capabilities. Is 10.8 available already or is 10.7 meant with this?
A: AI CodeFix and AI Code Assurance are both available from the 10.7 release, which came out at the beginning of October.
Q: I am an existing SonarCloud user. I have not upgraded to Teams or Enterprise, as I am licensed for some years to come. Can I use CodeFix now?
A: Yes, AI CodeFix is available for free in all SonarCloud paid plans during the early access period.
Q: I also hear the cloud solution is not fully formed with all the features?
A: Our goal is to have feature parity between SonarQube and SonarCloud as much as feasible. We have already added quite a few features in SonarCloud such as reporting, SSO, portfolios, etc. Take a look at the SonarCloud Enterprise Plan features to learn more.
Q: What programming languages are supported?
A: In this early access, AI CodeFix supports Java, JavaScript, TypeScript, C#, Python, C, and C++. We will be adding more languages in future releases.
Q: Will there be an integration with Azure DevOps so that AI CodeFix can be triggered within PRs instead of SonarQube?
A: Yes, this is planned for a future release.
Q: Noticed this feature came to SonarQube before SonarCloud. Is that the expected release approach in the future for new features?
A: No. In general, we try to time the release of features at about the same time for SonarCloud and SonarQube. However, their release cadence is different, so is unlikely that features will be released on the same day for the two products. In the case of AI Code Assurance, the timing of availability is a few weeks apart.
Q: What is the license model (seats or consumption)?
A: These features do not introduce any changes to the license model. SonarCloud and SonarQube are based on Lines of Code as before.
Q: Can I use my custom models (e.g. trained for my own coding standards)?
A: Not as of now but we are looking into providing a “bring your own LLM” option in a future release.
Q: will this be a chargeable SKU in the longer term on SonarCloud? (post early access)
A: We don’t yet have any details to share about AI CodeFix when it becomes a GA feature.
Q: Will AI Code Assurance become available in the Enterprise Edition? Any rough timeframes?
A: AI Code Assurance is already available in SonarQube Enterprise (10.7) and will soon be available in SonarCloud Enterprise.
Q: Will using AI CodeFix in SonarCloud be included in the subscription or billed separately?
A: Currently AI CodeFix is available for free during early access in SonarCloud paid plans.
Q: For SonarQube, will these features be available in an air-gapped environment?
A: AI CodeFix requires internet access to the external LLM, so fix suggestions will not be available in an air-gapped environment. AI Code Assurance works in an air-gapped environment.
Q: We are on v9.9.2, should I be upgrading to the new LTS?
A: Yes, we strongly recommend upgrading to the latest LTA (long term active) release to take advantage of all the new features. The next LTA is planned for early 2025.
Q: From which LTS version onwards can this AI code assistant provide fixes?
A: AI CodeFix is available starting SonarQube 10.7 and will be included in the next LTA (long term active) release which is planned for early 2025.
Q: We understood that 10.7 is not an LTS version. How do you expect large customers like ours to use this?
A: Correct. The SonarQube 10.7 release is not an LTA (long term active) release. These features will be included in the next SonarQube LTA release planned for early 2025. Making them available in the pre-LTA release gives you the opportunity to use them before you move to the LTA release.
Q: We’re currently on 9.9 LTS. When can we expect 10.X LTS?
A: The next SonarQube LTA (long term active) release is planned for early 2025.
Q: How much of my project’s context is taken into account when making suggestions—the class file or the entire project?
A: Only a small snippet of your code is sent to the LLM. We do not send the entire project code.
Q: Can SonarQube detect which part of the code is generated by a tool such as ChatGPT within a project?
A: In this current release, a SonarQube user must manually identify which projects contain AI-generated code. It is extremely difficult to reliably and consistently identify AI-generated code within a project, but we are actively trying to solve this important problem.
Q: Is the evaluation of AI-generated code similar to the non-AI version, or we are talking about more software quality metrics covered in AI-generated codes?
A: Code is code, whether it is AI-generated or developer-written. Because AI-generated code is trained on human-written code, the same issues are found in both. Our rules cover issues found in AI-generated code. SonarQube performs the same rigorous analysis of code regardless of its source.
Q: How does SQ know the code is AI-generated, and how does the verification differ from non-AI code?
A: Code is code, whether it is AI-generated or not. With AI Code Assurance, you are able to tag projects containing AI-generated code and once this is done, it has to pass a strict Quality Gate. To enable AI Code Assurance, you need to manually tag projects that include AI-generated code. Today we don’t automatically detect AI-generated code.
Q: Can SonarCloud’s AI suggest code optimizations based on past scan results?
A: No, only a small code snippet is sent to the LLM. No customer code is used to train the LLM
Q: What’s the difference between scanning all codes in my project and using the AI Code Assurance feature? Will I be able to define the quality gate differently?
A: With AI Code Assurance, as of today, it enforces the use of Sonar Way Quality Gate. We are looking into more customization options in a future iteration of this feature.
Q: Will SonarQube be able to track/detect AI-generated code?
A: Detecting and tracking specific AI-generated lines of code is not yet available in SonarQube. We are evaluating this as a future improvement to AI Code Assurance.
Q: Your customer may not be able to leverage this feature, so how soon do you expect you to have a more stable version available?
A: We are confident about the stability and quality of AI CodeFix’s results during early access. For general availability, we are planning further improvements and will expand it to cover more rules.
Q: How are the rules for Code Assurance different than the “clean as you code” rules?
A: The rules used by the analyzers are the same whether the code is human-written or AI-generated because the types of issues found in the code are the same. AI Code Assurance requires the use of the Sonar way quality gate conditions for Sonar to ensure the code is free of issues. We are looking into offering more customization options. Do share your feedback if you have any specific needs.
Q: Should any code, indifferent to the source, not be treated the same?
A: Yes, we agree. The actual code analysis is the same for both human-written and AI-generated code because code is code. However, with AI Code Assurance, we are enforcing our strict Sonar way quality gate so that Sonar can ensure the quality of the code meets our strict standards when the project includes AI-generated code.
Q: Enabling these features can be done project-specific or all projects as a whole?
A: AI Code Assurance is enabled on a per-project basis, and AI CodeFix is enabled at the organization level.
Q: Will this only work with the “Sonar way” profile or why is that being enforced?
A: Today, AI Code Assurance enforces the Sonar way quality gate so that Sonar can ensure the code in projects with AI-generated code meets our strict standards, and we can apply the AI Code Assured badge to the project. We are looking into adding customization options. Do let us know if you have any inputs or feedback here and we will be happy to evaluate your requirements,
Q: What does the new gate do differently than a normal one? Is it analyzing against the same rulesets already available, but just a stricter gate? Or is it using additional rules that are special to AI code?
A: We currently use the Sonar Way Quality Gate but are looking into more customization options. It is a strict gate, as it will pass only if your code has zero issues.
Q: Using AI Code Assurance, can you no longer use your own custom quality gates? (The demo seemed to imply that you have to use the “Sonar Way” quality gate.)
A: You can always use your own custom quality gate. However, AI Code Assurance enforces the Sonar way quality gate so we can be sure the code passes our quality standard in order to apply the AI Code Assured badge to a project with AI-generated code. We are looking into offering some customization options. Do let us know if you have any specific requirements,
Q: When analyzing code, like it does now, how does it even matter if it is AI-generated or not? Shouldn’t the code be analyzed the same way regardless of whether it is generated by AI?
A: Correct, there is no difference between issues in AI-generated code and developer-written code, so code analysis is the same. For Sonar to mark a project as AI Code Assured, we enforce our strict Sonar way quality gate to ensure the code passes our high standard for quality.
Q: How does SonarQube use AI to improve false positive detection in code scanning?
A: We are investing in research and development to improve detection accuracy by leveraging ML. Stay tuned to hear more on this in the future, but it’s an excellent question. Thanks for asking.
Q: How do you compare Autofix with the recent features of GitHub CoPilot Autofix (security now, I assume more general code fixes in the future)? Can Sonar AI Assistant and GH Autofix work together and be complementary?
A: GitHub’s CoPilot Autofix is for security vulnerability fixes only. Sonar AI CodeFix generates fixes for not only security issues but also bugs and code quality issues. I don’t see a reason why the two cannot play together but we have not tested it.
Q: Can codefix be forced to use a specific LLM / enterprise-vetted AI?
A: Not as of now but we are looking into providing a “bring your own LLM” option in a future release.
Q: Pressing generate AI fix is simply a prompt to ChatGPT or does SonarQube handle it differently?
A: We handle the data in a proprietary way that is more than just a prompt. We do not use ChatGPT directly.
Q: How to enable AI Code Fix?
A: Go to the organization settings, review the early access terms and conditions, and enable it. Docs - https://docs.sonarsource.com/sonarqube/latest/instance-administration/system-functions/enabling-ai-generated-fix-suggestions/
Q: Is this LLM going to be part of the Data Center product or require an internet connection?
A: Yes, AI CodeFix is available in SonarQube Data Center Edition as an early access feature. As of today, it does require an internet connection to communicate with the LLM. Docs - https://docs.sonarsource.com/sonarqube/latest/instance-administration/system-functions/enabling-ai-generated-fix-suggestions/
Q: If the fix requires changes in another file, is SonarCloud AI CodeFix able to suggest that?
A: The suggested code fix is only intended for the location where the analyzer found that specific issue. The context of the fix suggestion is only for that segment of code.
Q: I know CodeFix is in a very early stage. But are there any plans to provide a backend endpoint for on-premise LLMs for CodeFix?
A: Yes, we are looking into an option to “bring your own LLM” in a future release. Please don’t hesitate to tell our support team or your sales rep your requirements.
Q: Does SonarQube do a different type of scan against the codebase if “AI Code Assurance” is turned on? If so, what the differences between that scan compared to a regular scan?
A: No, the scan stays the same. We enforce the Sonar way quality gate to ensure that the AI-generated code meets the Sonar’s high-quality standard.
Q: What are the differences between AI Projects and non-AI-Projects in regards to finding issues?
A: Code analysis remains the same. There is no difference between issues generated by AI and those introduced by a developer.
Q: Will SQ use AI itself to find issues if my project is an AI project? Will SQ also use AI on projects that are not AI-tagged? What difference Assurance vs. CodeFixes for the AI in the background?
A: Today, our analyzers do not use AI to detect issues. We have been and will continue to research the use of LLMs for issue detection. We’ve found that because of LLM hallucinations, the results from current LLMs introduce a high number of false positives, which is counterproductive. LLMs are changing constantly, so this may quickly change.
Q: Can AI Codefix be used for multiple similar issues at the same time? Working with legacy code, there may be hundreds of similar issues in the codebase, so clicking them through one by one is still time-consuming.
A: No AI CodeFix can only be used for a single issue at a time.
Q: Does AI CodeFix support GitLab?
A: Yes, you can use AI CodeFix in SonarQube for projects imported from GitLab.
Q: What is the purpose of Sonar Assurance?
A: The purpose of AI Code Assurance is for teams to be aware when a project contains AI-generated code and to be sure that the AI-generated code has passed Sonar’s high-quality standards. This way your organization can be sure that the AI-generated code is free of issues.
Q: AI Code Assurance is just a tag for AI projects?
A: Tagging a project helps everyone on the team identify which projects contain AI-generated code, but that’s just the beginning. Once tagged, AI Code Assurance enforces a strict quality gate on the project. When the quality gate is successfully passed, AI Code Assurance applies an AI Code Assured badge so the team knows the project has been assured by Sonar. The badge can also be used externally to identify which projects have been AI Code Assured by Sonar.
Q: How customizable is Code Assurance? Can be it seen as a logical continuation of the static code analysis toolchain? What are the main differences between traditional static code analysis and AI Code Assurance?
A: With AI Code Assurance, SonarQube enforces the use of Sonar Way Quality Gate. This feature can be turned on at the project level. We are looking into more customization options in future iterations of this feature. This feature is still taking advantage of traditional static analysis.
Q: Does the AI CodeFix change the SonarQube server hardware requirements or is it calling out to an external service?
A: No additional hardware is required. SonarQube calls the OpenAI model for the fix suggestion.
Q: Can you provide some self-enablement documentation that we can share with developers to use Code Fix?
A: Here you can read about using AI CodeFix.
Q: Is there any difference between enabling the checkbox “Contains AI-Generated Code” and just manually setting the built-in quality gate to the projects generated by AI? Does it add any value on top of this?
A: A special AI Assurance-related badge is displayed on applicable projects in the projects list when the project passes our strict quality gate. Users can also take advantage of AI Code Assurance status badge that can be appended to repositories. We are evaluating further improvements.
Q: If Sonar gives a false positive, does the AI code fix say that it is not a bug?
A: No. The analysis of issues and the fix suggestions are separate from each other.
Q: So the reason to distinct between developer code and AI code is the quality gate applied?
A: The same quality gate (Sonar way) is applied to all code in a project where this feature is enabled. AI code calls for a high-quality standard to avoid risks. In order for Sonar to be confident that the project is AI Code Assured, the project has to pass our strict quality gate.
Q: As far as I know the current quality gate proposed by Sonar is “0 new issues”. How can a quality gate be more strict than “0 new issues”?
A: A quality gate can be more strict if you also have conditions on overall code, coverage limit, and many other conditions. Check our documentation to learn more.
Q: Will there be a rule category only for AI-generated code when creating quality profiles?
A: No there will not be a rule category only AI-generated code.
Q: Do we need to analyze the code again using SonarQube as suggested by the AI CodeFix?
A: If you have set up automated analysis with every code change, once you apply a change from a suggested code fix from AI CodeFix, an analysis will be retriggered.
Q: Is code fix available for non-AI code?
A: AI CodeFix is available for issues detected in any code.
Q: Is marking a project as AI code any different than setting it to use the Sonarway gate?
A: The difference is that when you mark a project as containing AI-generated code and the project passes the Sonar way quality gate, Sonar gives the project an AI Code Assured badge. This badge shows the team and your organization that the project has passed Sonar’s high-quality standard when it contains AI-generated code.
Q: Can we use Sonar AI CodeFix without enabling AI Code Assurance?
A: Yes, you can use only one or both. It is up to you.
Q: When you generate a fix for an issue using the LLM, is the fix itself run through SonarCloud/Qube’s quality gates to ensure that it is free of issues?
If not, why not? wouldn’t it be the obvious choice?
A: If you have set up automated analysis with every code change, once you apply a change from a suggested code fix from AI CodeFix, an analysis will be retriggered.
Q: How do you think IDEs will be able to scale up with extended plugin capability?
A: We are continually testing the performance of each IDE with our SonarLint plugin to ensure no significant performance degradation.
Q: Can we point AI CodeFix to our own internal secured LLM proxy rather than to OpenAI directly?
A: Not currently. We are looking into providing a “bring your own LLM” option in a future release.
Q: How is AI code assurance different from quality Gate?
A: The difference is that when you mark a project as containing AI-generated code and the project passes the Sonar way quality gate, Sonar gives the project an AI Code Assured badge. This badge shows the team and your organization that the project has passed Sonar’s high-quality standard when it contains AI-generated code.
Q: In a self-hosted deployment context, can we specify our own OpenAI inference endpoint and token? Or is it fixed or going through a sonar source?
A: This is not possible as of now. Thanks for your question, we will look into it.
Q: How does this work for SonarQube users? Do we need to open our platform to the internet?
A: Yes, you need to open an internet connection so that SonarQube can send a query to the LLM. https://docs.sonarsource.com/sonarqube/latest/instance-administration/system-functions/enabling-ai-generated-fix-suggestions/
Q: How is an AI-generated code bug different from a regular code bug?
A: AI-generated issues and human-written issues are no different because the LLM is trained on human-written code.
Q: How does AI enhance code quality analysis in SonarQube?
A: Today we do not use AI in our analyzers. It’s something we’re investigating ongoing, but based on our findings the results create more false positives which is counterproductive.
Q: Will it also provide the AI Code fix for the general issues(line BufferOverFlow/NullPointerr…) or just the security issues?
A: AI CodeFix early access suggests code fixes for a select number of issues, such as bugs, vulnerabilities, and code quality issues, as part of the feature’s free trial. When we make AI CodeFix generally available, it will cover more issues. Keep a look out for the details when we announce the general availability of AI CodeFix.
Q: Is there any plan to enable AI Code Assurance at the development level (maybe with SonarLint) so that it can be verified before the developer creates a PR?
A: We will continually enhance AI Code Assurance. This is a great suggestion, and we’ll add it to the backlog for future consideration.
Q: AI code fix seems to be an issue-per-issue in code. Are you looking at a bulk fix (all issues) feature? Mostly thinking of fixing issues in legacy code.
A: Bulk fixes are much more complicated than fixing a single issue at a time. However, we are researching and evaluating this area to add this capability.