And a thorough description of the problem / question:
I am scanning a COBOL source program and have indicated the absolute path to the downloaded copybooks via the property sonar.cobol.db2include.directories.
Getting this error:
Error during execution of SonarLint analysis
java.lang.IllegalStateException: Cannot save issue on C:\Users\jtfernan\Compuware\Workbench\workspace\.com.compuware.hostexplorer\dipd.ipc.us.aexp.com_1224\ISPW.ISPP.TMPH.UT11.SQC\TESTSQC1.cpy: cannot find a matching InputFile.
at com.sonarsource.cobol.plugin.squid.I.A(Unknown Source)
at com.sonarsource.cobol.plugin.squid.I.A(Unknown Source)
at com.sonarsource.cobol.plugin.squid.I.A(Unknown Source)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at com.sonarsource.cobol.E.I.B(Unknown Source)
at com.sonarsource.cobol.squid.CobolAstScanner.A(Unknown Source)
at com.sonarsource.cobol.squid.CobolAstScanner.scanFiles(Unknown Source)
at com.sonarsource.cobol.plugin.squid.H.execute(Unknown Source)
at org.sonarsource.sonarlint.core.analyzer.sensor.SensorsExecutor.executeSensor(SensorsExecutor.java:80)
at org.sonarsource.sonarlint.core.analyzer.sensor.SensorsExecutor.execute(SensorsExecutor.java:71)
at org.sonarsource.sonarlint.core.container.analysis.AnalysisContainer.doAfterStart(AnalysisContainer.java:133)
at org.sonarsource.sonarlint.core.container.ComponentContainer.startComponents(ComponentContainer.java:126)
at org.sonarsource.sonarlint.core.container.ComponentContainer.execute(ComponentContainer.java:111)
at org.sonarsource.sonarlint.core.container.storage.StorageAnalyzer.analyze(StorageAnalyzer.java:77)
at org.sonarsource.sonarlint.core.container.storage.StorageContainerHandler.analyze(StorageContainerHandler.java:80)
at org.sonarsource.sonarlint.core.ConnectedSonarLintEngineImpl.lambda$analyze$0(ConnectedSonarLintEngineImpl.java:159)
at org.sonarsource.sonarlint.core.AbstractSonarLintEngine.withModule(AbstractSonarLintEngine.java:87)
at org.sonarsource.sonarlint.core.ConnectedSonarLintEngineImpl.lambda$analyze$1(ConnectedSonarLintEngineImpl.java:157)
at org.sonarsource.sonarlint.core.ConnectedSonarLintEngineImpl.withReadLock(ConnectedSonarLintEngineImpl.java:371)
at org.sonarsource.sonarlint.core.ConnectedSonarLintEngineImpl.withReadLock(ConnectedSonarLintEngineImpl.java:361)
at org.sonarsource.sonarlint.core.ConnectedSonarLintEngineImpl.analyze(ConnectedSonarLintEngineImpl.java:155)
at org.sonarlint.eclipse.core.internal.engine.connected.ConnectedEngineFacade.lambda$15(ConnectedEngineFacade.java:392)
at org.sonarlint.eclipse.core.internal.engine.connected.ConnectedEngineFacade.withEngine(ConnectedEngineFacade.java:185)
at org.sonarlint.eclipse.core.internal.engine.connected.ConnectedEngineFacade.runAnalysis(ConnectedEngineFacade.java:391)
at org.sonarlint.eclipse.core.internal.jobs.AnalyzeConnectedProjectJob.runAnalysis(AnalyzeConnectedProjectJob.java:70)
at org.sonarlint.eclipse.core.internal.jobs.AnalyzeConnectedProjectJob.runAnalysis(AnalyzeConnectedProjectJob.java:1)
at org.sonarlint.eclipse.core.internal.jobs.AbstractAnalyzeProjectJob.run(AbstractAnalyzeProjectJob.java:400)
at org.sonarlint.eclipse.core.internal.jobs.AbstractAnalyzeProjectJob.runAnalysisAndUpdateMarkers(AbstractAnalyzeProjectJob.java:208)
at org.sonarlint.eclipse.core.internal.jobs.AbstractAnalyzeProjectJob.doRun(AbstractAnalyzeProjectJob.java:162)
at org.sonarlint.eclipse.core.internal.jobs.AbstractSonarProjectJob.run(AbstractSonarProjectJob.java:45)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:60)
Hi Colin,
Thanks for the feedback. I did the upgrades and I am now in SonarLint for Eclipse v8.1.0 and SonarQube v9.9.1.
I still get the same error message:
Error executing sensor: 'CobolSquidSensor'
java.lang.IllegalStateException: Cannot save issue on C:\Users\jtfernan\Compuware\Workbench\workspace\.com.compuware.hostexplorer\dipd.ipc.us.aexp.com_1224\ISPW.ISPP.TMPH.UT11.SQC\TESTSQC1.cpy: cannot find a matching InputFile.
at com.sonarsource.cobol.plugin.squid.H.A(Unknown Source)
at com.sonarsource.cobol.plugin.squid.H.A(Unknown Source)
at com.sonarsource.cobol.plugin.squid.H.A(Unknown Source)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at com.sonarsource.cobol.J.I.B(Unknown Source)
at com.sonarsource.cobol.squid.CobolAstScanner.A(Unknown Source)
at com.sonarsource.cobol.squid.CobolAstScanner.A(Unknown Source)
at com.sonarsource.cobol.squid.CobolAstScanner.scanFiles(Unknown Source)
at com.sonarsource.cobol.plugin.squid.G.execute(Unknown Source)
at org.sonarsource.sonarlint.core.analysis.container.analysis.sensor.SensorsExecutor.executeSensor(SensorsExecutor.java:75)
at org.sonarsource.sonarlint.core.analysis.container.analysis.sensor.SensorsExecutor.execute(SensorsExecutor.java:66)
at org.sonarsource.sonarlint.core.analysis.container.analysis.AnalysisContainer.doAfterStart(AnalysisContainer.java:122)
at org.sonarsource.sonarlint.core.plugin.commons.container.SpringComponentContainer.startComponents(SpringComponentContainer.java:182)
at org.sonarsource.sonarlint.core.plugin.commons.container.SpringComponentContainer.execute(SpringComponentContainer.java:161)
at org.sonarsource.sonarlint.core.analysis.container.module.ModuleContainer.analyze(ModuleContainer.java:71)
at org.sonarsource.sonarlint.core.analysis.command.AnalyzeCommand.execute(AnalyzeCommand.java:58)
at org.sonarsource.sonarlint.core.analysis.command.AnalyzeCommand.execute(AnalyzeCommand.java:32)
at org.sonarsource.sonarlint.core.analysis.AnalysisEngine$AsyncCommand.execute(AnalysisEngine.java:153)
at org.sonarsource.sonarlint.core.analysis.AnalysisEngine.executeQueuedCommands(AnalysisEngine.java:71)
at java.base/java.lang.Thread.run(Thread.java:829)
There is another one that pops up this time:
Error during execution of SonarLint analysis
java.lang.NullPointerException
at java.base/java.util.Objects.requireNonNull(Objects.java:221)
at java.base/java.util.Optional.flatMap(Optional.java:295)
at org.sonarlint.eclipse.core.internal.tracking.ProjectIssueTracker.lambda$3(ProjectIssueTracker.java:142)
at java.base/java.util.HashMap.forEach(HashMap.java:1337)
at org.sonarlint.eclipse.core.internal.tracking.ProjectIssueTracker.trackWithServerIssues(ProjectIssueTracker.java:141)
at org.sonarlint.eclipse.core.internal.jobs.AnalyzeConnectedProjectJob.trackFileIssues(AnalyzeConnectedProjectJob.java:90)
at org.sonarlint.eclipse.core.internal.jobs.AbstractAnalyzeProjectJob.trackIssues(AbstractAnalyzeProjectJob.java:324)
at org.sonarlint.eclipse.core.internal.jobs.AnalyzeConnectedProjectJob.trackIssues(AnalyzeConnectedProjectJob.java:78)
at org.sonarlint.eclipse.core.internal.jobs.AbstractAnalyzeProjectJob.lambda$19(AbstractAnalyzeProjectJob.java:295)
at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2289)
at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2311)
at org.sonarlint.eclipse.core.internal.jobs.AbstractAnalyzeProjectJob.updateMarkers(AbstractAnalyzeProjectJob.java:295)
at org.sonarlint.eclipse.core.internal.jobs.AbstractAnalyzeProjectJob.runAnalysisAndUpdateMarkers(AbstractAnalyzeProjectJob.java:202)
at org.sonarlint.eclipse.core.internal.jobs.AbstractAnalyzeProjectJob.doRun(AbstractAnalyzeProjectJob.java:156)
at org.sonarlint.eclipse.core.internal.jobs.AbstractSonarJob.run(AbstractSonarJob.java:37)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:60)
Wow, rarely do I have that easy of a time getting people to upgrade
COBOL analysis in the IDE is a complicated subject and likely I will need to ping the relevant team.
But first, I wanted to know what made you choose to use sonar.cobol.db2include.directories (I wasn’t aware this existed) as opposed to sonar.cobol.copy.directories?
Hi Colin,
Our live server is on SonarQube Version 8.9.6 but the test/QA server is actually on SonarQube v9.9.1 already so all I did was to upgrade SonarLint on the client side (my machine) and bind my project to the test/QA server.
I use sonar.cobol.db2include.directories because we use the DB2 pre-compile directive EXEC SQL INCLUDE ... END-EXEC in our programs especially for SQL copybooks/statements in the procedure division. We may actually need to use both (sonar.cobol.copy.directories and sonar.cobol.db2include.directories) as we also have layout copybooks declared with the COBOL compile directive COPY in the WS/linkage sections.
Hi Colin,
I’ve set both properties (sonar.cobol.copy.directories and sonar.cobol.db2include.directories ) but still getting the 2 errors I’ve indicated in one of the previous replies.
We had some collaboration with Topaz developers few years ago, and it was supposed to make this process easier (automatically providing copybooks to the analysis).
I am not sure how it works today, as we don’t have access to Topaz IDE, not to the Topaz/SonarLint integration.
Would you mind to enable verbose log output in the SonarLint console, and share the beginning when triggering an analysis. Should be something like:
Here I have the impression you are trying to configure manually the path to the copybooks, and I am wondering if it is not colliding with Topaz built-in integration (but again, difficult for me to reproduce).
Can you try to remove any property you may have defined in SonarLint (sonar.cobol.db2include.directories and sonar.cobol.copy.directories) and run a new analysis of a single Cobol file (just open the file in an editor)? Then share the logs like earlier. I would like to see if Topaz is somehow automatically passing the property sonar.cobol.copy.directories.
Hi Julien,
Seems you are right. Topaz is automatically passing the property sonar.cobol.copy.directories . This property caters to copybooks declared using the COPY directive. I also need the sonar.cobol.db2include.directories to work so copybooks declared via the DB2 pre-compile directive EXEC SQL INCLUDE ... END-EXEC get included in the scan. Is that possible?
Hi,
I have tried going back to SonarLint 6.2 and it looks like it uses the same analyzer property (sonar.cobol.copy.directories) for both copybooks declared using the COPY directive and those declared via the EXEC SQL INCLUDE ... END-EXEC directive. It also seems that it is able to include copybooks in the scan/analysis and detect issues in them. However, these issues are somehow skipped.
Initializing metadata of file zosfs://dipd.ipc.us.aexp.com:1224/ISPW.ISPP.TMPH.DV11.SRC/JJSS09B?PDSMEMBER
Skipping the issue "SQLWhereInSelect" at copybook location TESTSQC1.cpy:6 (6:15,8:29) which is not imported in SonarQube.
Skipping the issue "BinaryDependingOn" at copybook location TESTCPY1.cpy:4 (4:38,4:52) which is not imported in SonarQube.
2 issues where skipped on non-imported copybooks. Rerun with DEBUG log levels for details.
Would you know what is the missing link here? What does it mean by ‘not imported in SonarQube’ or ‘non-imported copybooks’?
I don’t think this property existed when we did our collaboration with Compuware to have Topaz/SonarLint integration. So this might be an new feature request to fill on their side. On the SonarLint side, the API we developed for third-party vendor like Topaz is permissive enough to pass any analysis property.
It means that the copybooks are not part of the analysis scope, because we run on-the-fly analysis on the file currently opened in the editor. Copybooks are in this case considered as “external” code (like a dependency).
What might work (depending on how it is implemented on the Topaz side) is to select both the program and copybook files in the IDE resource explorer, and manually trigger a SonarLint analysis on the 2 files:
In general, SonarLint is not currently able to do on-the-fly + cross-file analysis (= editing code in one file leading to the report of an issue in another file).
There’s this sonar.sources property which in combination with sonar.cobol.file.suffixes is supposed to facilitate raising issues against copybooks. Would you be able to confirm this works only with SonarQube and SonarCloud but not with SonarLint?
Would you know of any future plans for this? Thanks!
If I do this, I can see in the logs the it initiates separate analysis/scans for the program and the copybook. On the copybooks analysis, I get a parsing error right at the very first active line of the copybook.
1 source files to be analyzed
Unable to parse COBOL source file : C:\Users\jtfernan\Compuware\Workbench\workspace\.com.compuware.hostexplorer\dipd.ipc.us.aexp.com_1224\ISPW.ISPP.TMPH.UT11.CPY\TESTCPY1.cpy at line 1
Probably because the copybook is not syntactically valid by itself. But I wonder how does it work on your IDE and example (P123.cbl and CP456.cpy) shown.
The analysis logs on the program still indicates issues skipped non-imported copybooks.
Normally, even if both program and copybook are part of the input files, our Cobol analyzer should not attempt to analyze copybook alone.
This should be controlled by the sonar.cobol.copy.suffixes property.
In one of your previous message, your log was containing:
If you select both the program and the copybook, it is expected that SonarLint processes the two. What is confusing in your last log is that it seems that only the copybook is processed here.
What should normally happen is that SonarLint will process the two files, but thanks to the configured file extensions, the SonarCobol Sensor will be able to differentiate programs from copybooks.
But now I realize that my reproducer is likely to not match your case, because I don’t have Topaz nor any real mainframe, so both my program and copybook are local files. Looking at the URI of your copybook: zosfs://dipd.ipc.us.aexp.com:1224/ISPW.ISPP.TMPH.UT11.CPY/TESTCPY1?PDSMEMBER#;Copybook it is likely that our Cobol analyzer might get confused and not properly able to determine what is copybook and what is not.
May I ask you to clear all the analysis properties you have manually defined in SonarLint, enable both verbose and analysis logs, and trigger the following analyses:
copybook alone
program alone
copybook + program
then share the logs with me (you can send a private message if you are concerned about privacy). That will be really helpful to find a solution.
Apologies if my statement was confusing and I’ve actually shared partial logs. When I selected both the program and the copybook, I can see from the logs that it started analysis for both the program and the copybook separately.
Hi Julien,
Without the sonar.cobol.copy.directories property and analysing the copybook alone results to parsing error:
Trigger: MANUAL
Server excluded sources:
**/target/**
**/node_modules/**
Clear markers on 0 excluded files
SonarLint processing file /TMPH AXD0/CPY/TESTCPY1.cpy...
Connected mode (using configuration of 'ispw_137959725' in connection 'sonar.aexp.com')
Starting analysis with configuration:
[
projectKey: ispw_137959725
baseDir: C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0
extraProperties: {sonar.cobol.copy.suffixes=cpy,cbl,cob,cobol,copy, sonar.cobol.file.suffixes=cbl,cob,cobol,copy,cpy, sonar.pli.file.suffixes=ibmpli,pl1,pli,plm,inc}
moduleKey: null
inputFiles: [
ispw://dipd.ipc.us.aexp.com:1224/ISPP/AXD0/TMPH/UT11/CPY/TESTCPY1?taskid=7E767A91665F&user=RE9838A (UTF-8)
]
]
Creating container for module with key=null
TypeScript sensor excluded
Start analysis
Index files
Language of file 'ispw://dipd.ipc.us.aexp.com:1224/ISPP/AXD0/TMPH/UT11/CPY/TESTCPY1?taskid=7E767A91665F&user=RE9838A' is detected to be 'COBOL'
1 file indexed
Available languages:
* Python => "py"
* Kotlin => "kotlin"
* RPG => "rpg"
* PL/I => "pli"
* T-SQL => "tsql"
* Apex => "apex"
* Secrets => "secrets"
* JavaScript => "js"
* TypeScript => "ts"
* PL/SQL => "plsql"
* Ruby => "ruby"
* Scala => "scala"
* Java => "java"
* COBOL => "cobol"
* HTML => "web"
* JSP => "jsp"
* XML => "xml"
* PHP => "php"
* ABAP => "abap"
Quality profiles:
* abap: 'Sonar way' (62 rules)
* apex: 'Sonar way' (38 rules)
* cobol: 'Triumph Custom Profile' (75 rules)
* java: 'AEXP-SECURITY V2' (440 rules)
* js: 'Advance JavaScript V2' (137 rules)
* jsp: 'FindBugs Security JSP' (0 rules)
* kotlin: 'Sonar way' (29 rules)
* php: 'Sonar way' (126 rules)
* pli: 'Sonar way' (16 rules)
* plsql: 'Sonar way' (130 rules)
* py: 'Sonar way' (128 rules)
* rpg: 'Sonar way' (29 rules)
* ruby: 'Sonar way' (28 rules)
* scala: 'Sonar way' (28 rules)
Unable to find the quality profile AWR18vZEnG3ibnjvVBw- in the SonarLint storage. You should update the storage, or ignore this message if the profile is empty.
* ts: 'Sonar way' (0 rules)
* tsql: 'Sonar way' (53 rules)
* web: 'Sonar way' (27 rules)
* xml: 'Sonar way' (6 rules)
Setting filesystem encoding: UTF-8
'JavaSquidSensor' skipped because there is no related file in current project
Execute Sensor: CobolSquidSensor
About to analyse COBOL Programs using 'ibm-enterprise-cobol' dialect and 'fixed' source code format.
1 source files to be analyzed
Unable to parse COBOL source file : C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\CPY\TESTCPY1.cpy at line 1
Original contents starting from line 1 till line 7:
01 LINE-ITEM-TABLE.
05 LINE-ITEM-COUNT PIC 99.
05 LINE-ITEMS OCCURS 0 TO 25 TIMES
DEPENDING ON LINE-ITEM-COUNT.
10 QUANTITY PIC 9999.
10 DESCRIPTION PIC X(30).
10 UNIT-PRICE PIC S9(5)V99.
Preprocessed contents:
Parse error at line 1:
1: 01 LINE-ITEM-TABLE .
^
2: 05 LINE-ITEM-COUNT PIC 99 .
3: 05 LINE-ITEMS OCCURS 0 TO 25 TIMES
4: DEPENDING ON LINE-ITEM-COUNT .
5: 10 QUANTITY PIC 9999 .
6: 10 DESCRIPTION PIC X ( 30
1/1 source files have been analyzed
Initializing metadata of file ispw://dipd.ipc.us.aexp.com:1224/ISPP/AXD0/TMPH/UT11/CPY/TESTCPY1?taskid=7E767A91665F&user=RE9838A
'Python Sensor' skipped because there is no related file in current project
'Kotlin Sensor' skipped because there is no related file in current project
'RPG sensor' skipped because there is no related file in current project
'PliSquidSensor' skipped because there is no related file in current project
'T-SQL Sensor' skipped because there is no related file in current project
'Apex Sensor' skipped because there is no related file in current project
Execute Sensor: Sonar Secrets Detection Sensor
'JavaScript analysis' skipped because there is no related file in current project
'PL/SQL Sensor' skipped because there is no related file in current project
'Ruby Sensor' skipped because there is no related file in current project
'Scala Sensor' skipped because there is no related file in current project
Execute Sensor: JavaXmlSensor
Execute Sensor: HTML
'XML Sensor' skipped because there is no related file in current project
'PHP sensor' skipped because there is no related file in current project
'Analyzer for "php.ini" files' skipped because there is no related file in current project
'AbapSquidSensor' skipped because there is no related file in current project
Found 0 issue(s)
Done in 4693 ms
GET 200 https://sonar.aexp.com/api/developers/search_events?projects=ispw_137959725&from=2023-11-17T00%3A05%3A26-0700 | response time=1460ms
GET 200 https://sonar.aexp.com/api/developers/search_events?projects=ispw_137959725&from=2023-11-17T00%3A05%3A26-0700 | response time=1380ms
