With the sonar.cobol.copy.directories property and analysing the copybook alone, here are the logs:
GET 200 https://sonar.aexp.com/api/developers/search_events?projects=ispw_137959725&from=2023-11-17T00%3A05%3A26-0700 | response time=1339ms
Trigger: MANUAL
Server excluded sources:
**/target/**
**/node_modules/**
Clear markers on 0 excluded files
SonarLint processing file /TMPH AXD0/CPY/TESTCPY1.cpy...
Connected mode (using configuration of 'ispw_137959725' in connection 'sonar.aexp.com')
Starting analysis with configuration:
[
projectKey: ispw_137959725
baseDir: C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0
extraProperties: {sonar.cobol.copy.suffixes=cpy,cbl,cob,cobol,copy, sonar.cobol.file.suffixes=cbl,cob,cobol,copy,cpy, sonar.pli.file.suffixes=ibmpli,pl1,pli,plm,inc, sonar.cobol.copy.directories=CPY,SQC}
moduleKey: null
inputFiles: [
ispw://dipd.ipc.us.aexp.com:1224/ISPP/AXD0/TMPH/UT11/CPY/TESTCPY1?taskid=7E767A91665F&user=RE9838A (UTF-8)
]
]
Creating container for module with key=null
TypeScript sensor excluded
Start analysis
Index files
Language of file 'ispw://dipd.ipc.us.aexp.com:1224/ISPP/AXD0/TMPH/UT11/CPY/TESTCPY1?taskid=7E767A91665F&user=RE9838A' is detected to be 'COBOL'
1 file indexed
Available languages:
* Python => "py"
* Kotlin => "kotlin"
* RPG => "rpg"
* PL/I => "pli"
* T-SQL => "tsql"
* Apex => "apex"
* Secrets => "secrets"
* JavaScript => "js"
* TypeScript => "ts"
* PL/SQL => "plsql"
* Ruby => "ruby"
* Scala => "scala"
* Java => "java"
* COBOL => "cobol"
* HTML => "web"
* JSP => "jsp"
* XML => "xml"
* PHP => "php"
* ABAP => "abap"
Quality profiles:
* abap: 'Sonar way' (62 rules)
* apex: 'Sonar way' (38 rules)
* cobol: 'Triumph Custom Profile' (75 rules)
* java: 'AEXP-SECURITY V2' (440 rules)
* js: 'Advance JavaScript V2' (137 rules)
* jsp: 'FindBugs Security JSP' (0 rules)
* kotlin: 'Sonar way' (29 rules)
* php: 'Sonar way' (126 rules)
* pli: 'Sonar way' (16 rules)
* plsql: 'Sonar way' (130 rules)
* py: 'Sonar way' (128 rules)
* rpg: 'Sonar way' (29 rules)
* ruby: 'Sonar way' (28 rules)
* scala: 'Sonar way' (28 rules)
Unable to find the quality profile AWR18vZEnG3ibnjvVBw- in the SonarLint storage. You should update the storage, or ignore this message if the profile is empty.
* ts: 'Sonar way' (0 rules)
* tsql: 'Sonar way' (53 rules)
* web: 'Sonar way' (27 rules)
* xml: 'Sonar way' (6 rules)
Setting filesystem encoding: UTF-8
'JavaSquidSensor' skipped because there is no related file in current project
Execute Sensor: CobolSquidSensor
About to analyse COBOL Programs using 'ibm-enterprise-cobol' dialect and 'fixed' source code format.
0 source files to be analyzed
0/0 source files have been analyzed
Initializing metadata of file ispw://dipd.ipc.us.aexp.com:1224/ISPP/AXD0/TMPH/UT11/CPY/TESTCPY1?taskid=7E767A91665F&user=RE9838A
'Python Sensor' skipped because there is no related file in current project
'Kotlin Sensor' skipped because there is no related file in current project
'RPG sensor' skipped because there is no related file in current project
'PliSquidSensor' skipped because there is no related file in current project
'T-SQL Sensor' skipped because there is no related file in current project
'Apex Sensor' skipped because there is no related file in current project
Execute Sensor: Sonar Secrets Detection Sensor
'JavaScript analysis' skipped because there is no related file in current project
'PL/SQL Sensor' skipped because there is no related file in current project
'Ruby Sensor' skipped because there is no related file in current project
'Scala Sensor' skipped because there is no related file in current project
Execute Sensor: JavaXmlSensor
Execute Sensor: HTML
'XML Sensor' skipped because there is no related file in current project
'PHP sensor' skipped because there is no related file in current project
'Analyzer for "php.ini" files' skipped because there is no related file in current project
'AbapSquidSensor' skipped because there is no related file in current project
Found 0 issue(s)
0 entries removed from the store
Done in 2531 ms
It seems to identify the component as COBOL and executes the CobolSquidSensor but ends up with “0 source files to be analyzed”.
This is the expected behavior, since copybooks alone are not necessarily complete Cobol programs, our parser can’t analyze them outside of the context they are “imported” into.
Hi Julien,
Analyzing the program alone and without the sonar.cobol.copy.directories results in all copybooks not being found:
Trigger: MANUAL
Clear markers on 0 excluded files
SonarLint processing file /TMPH AXD0/SRC/JJSS09B.cbl...
Connected mode (using configuration of 'ispw_137959725' in connection 'sonarstage.aexp.com')
Starting analysis with configuration:
[
projectKey: ispw_137959725
baseDir: C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0
extraProperties: {sonar.cobol.copy.suffixes=cpy,cbl,cob,cobol,copy, sonar.cobol.file.suffixes=cob,cbl,cpy,cobol,copy, sonar.pli.file.suffixes=ibmpli,pl1,pli,plm,inc}
moduleKey: null
inputFiles: [
ispw://dipd.ipc.us.aexp.com:1224/ISPP/AXD0/TMPH/DV11/SRC/JJSS09B?taskid=7E7AFB22B1D1&user=RE9838A (UTF-8)
]
]
Creating container for module with key=null
Start analysis
Index files
Language of file 'ispw://dipd.ipc.us.aexp.com:1224/ISPP/AXD0/TMPH/DV11/SRC/JJSS09B?taskid=7E7AFB22B1D1&user=RE9838A' is detected to be 'COBOL'
1 file indexed
Available languages:
* Python => "py"
* Kotlin => "kotlin"
* RPG => "rpg"
* PL/I => "pli"
* T-SQL => "tsql"
* Apex => "apex"
* Secrets => "secrets"
* PL/SQL => "plsql"
* Ruby => "ruby"
* Scala => "scala"
* Java => "java"
* COBOL => "cobol"
* HTML => "web"
* JSP => "jsp"
* XML => "xml"
* PHP => "php"
* ABAP => "abap"
Quality profiles:
* abap: 'Sonar way' (61 rules)
* apex: 'EPSAM Standard v2' (42 rules)
* cobol: 'TIRUMPH_CUS' (75 rules)
* java: 'AEXP-SECURITY V2' (471 rules)
* jsp: 'FindBugs Security JSP (outdated copy since November 10 2023 at 01:21 PM)' (0 rules)
* kotlin: 'Sonar way' (79 rules)
* php: 'Sonar way' (173 rules)
* pli: 'Sonar way' (17 rules)
* plsql: 'Sonar way' (130 rules)
* py: 'Sonar way' (169 rules)
* rpg: 'Sonar way' (29 rules)
* ruby: 'Sonar way' (28 rules)
* scala: 'Sonar way' (28 rules)
* secrets: 'Sonar way' (7 rules)
* tsql: 'Sonar way' (53 rules)
* web: 'Sonar way' (26 rules)
* xml: 'Sonar way' (15 rules)
Setting filesystem encoding: UTF-8
'JavaSensor' skipped because there is no related file in current project
Execute Sensor: CobolSquidSensor
About to analyse COBOL Programs using 'ibm-enterprise-cobol' dialect and 'fixed' source code format.
Analysis cache is disabled, we are in SonarLint context
1 source file to be analyzed
Unable to find copybook 'CPOOWSL3' (at line=34 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Unable to find copybook 'WERRL003' (at line=36 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Unable to find copybook 'WFNDERR' (at line=38 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Unable to find copybook 'CPBU07L' (at line=40 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Unable to find copybook 'NTCPLA' (at line=43 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Unable to find copybook 'NTPPMX' (at line=45 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Unable to find copybook 'NTCOBD' (at line=47 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Unable to find copybook 'TESTCPY1' (at line=49 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Unable to find copybook 'TESTCPY2' (at line=54 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Unable to find copybook 'TESTSQC1' (at line=62 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Unable to find copybook 'NPTT32U' (at line=153 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Initializing metadata of file ispw://dipd.ipc.us.aexp.com:1224/ISPP/AXD0/TMPH/DV11/SRC/JJSS09B?taskid=7E7AFB22B1D1&user=RE9838A
1/1 source file has been analyzed
Analysis time: 2407ms
Decoration time: 0ms
'Python Sensor' skipped because there is no related file in current project
'Kotlin Sensor' skipped because there is no related file in current project
'RPG sensor' skipped because there is no related file in current project
'PliSquidSensor' skipped because there is no related file in current project
'T-SQL Sensor' skipped because there is no related file in current project
'Apex Sensor' skipped because there is no related file in current project
Execute Sensor: Sonar Secrets Detection Sensor
'PL/SQL Sensor' skipped because there is no related file in current project
'Ruby Sensor' skipped because there is no related file in current project
'Scala Sensor' skipped because there is no related file in current project
Execute Sensor: HTML
'XML Sensor' skipped because there is no related file in current project
'PHP sensor' skipped because there is no related file in current project
'Analyzer for "php.ini" files' skipped because there is no related file in current project
'AbapSquidSensor' skipped because there is no related file in current project
Found 1 issue(s)
Download server issues for /TMPH AXD0/SRC/JJSS09B.cbl
GET 404 https://sonarstage.aexp.com/batch/issues?key=ispw_137959725%3ASRC%2FJJSS09B.cbl | response time=1074ms
Downloaded issues in 1075ms
Done in 6715 ms
GET 200 https://sonar.aexp.com/api/developers/search_events?projects=ispw_137959725&from=2023-11-30T18%3A01%3A27-0700 | response time=1362ms
GET 200 https://sonar.aexp.com/api/developers/search_events?projects=ispw_137959725&from=2023-11-30T18%3A01%3A27-0700 | response time=1367ms
Analyzing the program alone with a manually defined sonar.cobol.copy.directories property results in some copybooks being found, issues detected but skipped:
Trigger: MANUAL
Clear markers on 0 excluded files
SonarLint processing file /TMPH AXD0/SRC/JJSS09B.cbl...
Connected mode (using configuration of 'ispw_137959725' in connection 'sonarstage.aexp.com')
Starting analysis with configuration:
[
projectKey: ispw_137959725
baseDir: C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0
extraProperties: {sonar.cobol.copy.suffixes=cpy,cbl,cob,cobol,copy, sonar.cobol.file.suffixes=cob,cbl,cpy,cobol,copy, sonar.pli.file.suffixes=ibmpli,pl1,pli,plm,inc, sonar.cobol.copy.directories=CPY,SQC}
moduleKey: null
inputFiles: [
ispw://dipd.ipc.us.aexp.com:1224/ISPP/AXD0/TMPH/DV11/SRC/JJSS09B?taskid=7E7AFB22B1D1&user=RE9838A (UTF-8)
]
]
Creating container for module with key=null
Start analysis
Index files
Language of file 'ispw://dipd.ipc.us.aexp.com:1224/ISPP/AXD0/TMPH/DV11/SRC/JJSS09B?taskid=7E7AFB22B1D1&user=RE9838A' is detected to be 'COBOL'
1 file indexed
Available languages:
* Python => "py"
* Kotlin => "kotlin"
* RPG => "rpg"
* PL/I => "pli"
* T-SQL => "tsql"
* Apex => "apex"
* Secrets => "secrets"
* PL/SQL => "plsql"
* Ruby => "ruby"
* Scala => "scala"
* Java => "java"
* COBOL => "cobol"
* HTML => "web"
* JSP => "jsp"
* XML => "xml"
* PHP => "php"
* ABAP => "abap"
Quality profiles:
* abap: 'Sonar way' (61 rules)
* apex: 'EPSAM Standard v2' (42 rules)
* cobol: 'TIRUMPH_CUS' (75 rules)
* java: 'AEXP-SECURITY V2' (471 rules)
* jsp: 'FindBugs Security JSP (outdated copy since November 10 2023 at 01:21 PM)' (0 rules)
* kotlin: 'Sonar way' (79 rules)
* php: 'Sonar way' (173 rules)
* pli: 'Sonar way' (17 rules)
* plsql: 'Sonar way' (130 rules)
* py: 'Sonar way' (169 rules)
* rpg: 'Sonar way' (29 rules)
* ruby: 'Sonar way' (28 rules)
* scala: 'Sonar way' (28 rules)
* secrets: 'Sonar way' (7 rules)
* tsql: 'Sonar way' (53 rules)
* web: 'Sonar way' (26 rules)
* xml: 'Sonar way' (15 rules)
Setting filesystem encoding: UTF-8
'JavaSensor' skipped because there is no related file in current project
Execute Sensor: CobolSquidSensor
About to analyse COBOL Programs using 'ibm-enterprise-cobol' dialect and 'fixed' source code format.
Analysis cache is disabled, we are in SonarLint context
1 source file to be analyzed
Unable to find copybook 'CPOOWSL3' (at line=34 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Unable to find copybook 'WERRL003' (at line=36 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Unable to find copybook 'WFNDERR' (at line=38 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Unable to find copybook 'CPBU07L' (at line=40 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Unable to find copybook 'NTCPLA' (at line=43 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Unable to find copybook 'NTPPMX' (at line=45 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Unable to find copybook 'NTCOBD' (at line=47 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Unable to find copybook 'NPTT32U' (at line=153 in file='C:\Users\jtfernan\Compuware\Workbench\workspace\TMPH AXD0\SRC\JJSS09B.cbl'.)
Initializing metadata of file ispw://dipd.ipc.us.aexp.com:1224/ISPP/AXD0/TMPH/DV11/SRC/JJSS09B?taskid=7E7AFB22B1D1&user=RE9838A
1/1 source file has been analyzed
Skipping the issue "UseCompute" at copybook location TESTCPY2.cpy:1 (1:8,1:42) which is not imported in SonarQube.
Skipping the issue "SQLWhereInSelect" at copybook location TESTSQC1.cpy:6 (6:15,8:29) which is not imported in SonarQube.
Skipping the issue "BinaryDependingOn" at copybook location TESTCPY1.cpy:4 (4:38,4:52) which is not imported in SonarQube.
3 issues where skipped on non-imported copybooks. Rerun with DEBUG log levels for details.
Analysis time: 2398ms
Decoration time: 0ms
'Python Sensor' skipped because there is no related file in current project
'Kotlin Sensor' skipped because there is no related file in current project
'RPG sensor' skipped because there is no related file in current project
'PliSquidSensor' skipped because there is no related file in current project
'T-SQL Sensor' skipped because there is no related file in current project
'Apex Sensor' skipped because there is no related file in current project
Execute Sensor: Sonar Secrets Detection Sensor
'PL/SQL Sensor' skipped because there is no related file in current project
'Ruby Sensor' skipped because there is no related file in current project
'Scala Sensor' skipped because there is no related file in current project
Execute Sensor: HTML
'XML Sensor' skipped because there is no related file in current project
'PHP sensor' skipped because there is no related file in current project
'Analyzer for "php.ini" files' skipped because there is no related file in current project
'AbapSquidSensor' skipped because there is no related file in current project
Found 1 issue(s)
Download server issues for /TMPH AXD0/SRC/JJSS09B.cbl
GET 404 https://sonarstage.aexp.com/batch/issues?key=ispw_137959725%3ASRC%2FJJSS09B.cbl | response time=1492ms
Downloaded issues in 1493ms
Done in 7192 ms
GET 200 https://sonar.aexp.com/api/developers/search_events?projects=ispw_137959725&from=2023-11-30T18%3A01%3A27-0700 | response time=1396ms
GET 200 https://sonar.aexp.com/api/developers/search_events?projects=ispw_137959725&from=2023-11-30T18%3A01%3A27-0700 | response time=1344ms
Sorry for the slow response rate, I am quite busy with the end of year approaching.
To sum-up, we are aware of the limitation that a copybook can’t be analyzed alone, out of any program context. That’s why SonarLint won’t report any issue if you open a copybook alone.
Regarding your last test (Analyzing the program alone with a manually defined sonar.cobol.copy.directories property results in some copybooks being found)
is there any difference between the copybooks that are found and the ones that are not found?
Based on the logs, the basedir of the IDE project seems to be:
But looking at the URI of the program, I think this is a remote project on mainframe (ispw://dipd.ipc.us.aexp.com:1224/ISPP/AXD0/TMPH/DV11/SRC/JJSS09B?taskid=7E7AFB22B1D1&user=RE9838A)
So do you know if the copybooks are physically present in the local filesystem in folders:
Hi Julien,
Those copybooks that are physically present (3 of them) in the local filesystem are found during the scan.
Skipping the issue "UseCompute" at copybook location TESTCPY2.cpy:1 (1:8,1:42) which is not imported in SonarQube.
Skipping the issue "SQLWhereInSelect" at copybook location TESTSQC1.cpy:6 (6:15,8:29) which is not imported in SonarQube.
Skipping the issue "BinaryDependingOn" at copybook location TESTCPY1.cpy:4 (4:38,4:52) which is not imported in SonarQube.
thank you for your response. Please be a bit patient with us answering as it is still the Holiday season and we are out of the office for the next week and will be only back (including Julien) on January 15th.
Thank you very much for your patience and sorry for it taking so very long on our side.
In order to finally help you with this issue you provided in the first place (and the ones coming along the way) I want to provide you with the following information:
The sonar.cobol.db2include.directories property seems to not be set automatically by Topaz Workbench, you have to configure it manually. I suggest that you reach out to BMC for them to implement, as @Julien_HENRY said it wasn’t around back when they had the collaboration, that it is set automatically for the project.
As the sonar.cobol.copy.directories seem to not pick up all the copybook directories automatically, you have to configure it manually. Maybe reach out to BMC on this as well for them to pick up more directories automatically without the need to configure them on your side (if possible).
For the copybooks that are skipped / not found, please make sure that the paths are correct and that they are present locally (in the file system, not only displayed in the IDE). If they are not present locally, either download them (if that is possible for you / through the IDE) or reach out to BMC for them to implement them being downloaded automatically whenever you trigger an analysis so it is made sure that everything is local once the actual analysis in SonarLint starts (e.g. they can cache it an add that cache folder to sonar.cobol.copy.directories).
When opening a COBOL program (not a copybook) on its own (e.g. opening the editor) it won’t report issues on the copybooks as the copybooks are not considered as input files (=source) but more like dependencies for the current analysis scope.
When you want to analyze and have issues reported on a COBOL program including all the copybooks it references (like SonarQube does), select them all (if possible) in the IDE, and from the context menu trigger the SonarLint → Analyze action. For the ones you cannot select we’re sorry, but you cannot take them into account for the analysis.
With this, there shouldn’t be any more errors shown.
As this thread got quite long I might not covered every case you had but I hope I covered the initial issues and the ones that arose along the way.
If you find any other issues or my suggestions are not working as expected, please feel free to get back to us.