Intermittent exception "unexpanded response file" with IAR on Windows (8.9 LTS)

SonarQube Server / Community Build
, , ,
1

Intermittent exception “unexpanded response file” with IAR on Windows (SonarQube 8.9)

I took the recommended course of action and spun up a SonarQube 8.9 instance with the bundled
CFamily plugin, but still got the same exception.

Previous Thread:

When running sonar-scanner over an IAR build output on Windows, an “unexpanded response file”
exception is sometimes raised.

java.lang.IllegalStateException: unexpanded response file
	at com.sonar.cpp.analyzer.IarDriver.onCapture(IarDriver.java:95)
	at com.sonar.cpp.plugin.CFamilySensor.process(CFamilySensor.java:586)
	at com.sonar.cpp.plugin.CFamilySensor.process(CFamilySensor.java:366)
	at com.sonar.cpp.plugin.CFamilySensor.execute(CFamilySensor.java:175)
... (full stacktrace included below)

I added a retry to the sonar-scanner step, but the same exception would always happen in
that case. Maybe this suggests the build wrapper output goes bad? I checked the
build-wrapper.log and found the following error:

ERROR: error opening file for UTF-16 checking: C:\Users\bamboo\AppData\Local\Temp\EW8A2F.tmp
error reading file: C:\Users\bamboo\AppData\Local\Temp\EW8A2F.tmp

As you would expect, this is not present when the build succeeds. The particular file that
causes the exception is inconsistent, and given the same commit, sometimes the exception will
not occur at all. The file in question exists on the filesystem, is accessible, and according
to notepad++ is UTF-8 with BOM, if that matters.

I attempted to exclude the executables and paths involved from Microsoft Defender, thinking
that maybe it was holding a read lock when scanning files that sonar-scanner attempted to use,
but this did not seem to solve the issue either.

Are there any ideas as to what may be going wrong here? Is this addressed in the next LTS
version? I can send the build-wrapper.log and build-wrapper-dump.json in a private chat if
it helps.

Versions

Sonar

  • CSS Code Quality and Security 1.4.2.2002 (cssfamily)
  • PL/SQL Code Quality and Security 3.6.1.3873 (plsql)
  • Scala Code Quality and Security 1.8.3.2219 (sonarscala)
  • C# Code Quality and Security 8.22.0.31243 (csharp)
  • Vulnerability Analysis 8.9.0.11439 (security)
  • Java Code Quality and Security 6.15.1.26025 (java)
  • HTML Code Quality and Security 3.4.0.2754 (web)
  • Flex Code Quality and Security 2.6.1.2564 (flex)
  • XML Code Quality and Security 2.2.0.2973 (xml)
  • VB.NET Code Quality and Security 8.22.0.31243 (vbnet)
  • Swift Code Quality and Security 4.3.1.4892 (swift)
  • CFamily Code Quality and Security 6.20.0.31240 (cpp)
  • Python Code Quality and Security 3.4.1.8066 (python)
  • Go Code Quality and Security 1.8.3.2219 (go)
  • JaCoCo 1.1.1.1157 (jacoco)
  • Kotlin Code Quality and Security 1.8.3.2219 (kotlin)
  • T-SQL Code Quality and Security 1.5.1.4340 (tsql)
  • JavaScript/TypeScript Code Quality and Security 7.4.2.15501 (javascript)
  • Ruby Code Quality and Security 1.8.3.2219 (ruby)
  • Vulnerability Rules for C# 8.9.0.11439 (securitycsharpfrontend)
  • Vulnerability Rules for Java 8.9.0.11439 (securityjavafrontend)
  • License for SonarLint 8.9.0.43852 (license)
  • Vulnerability Rules for JS 8.9.0.11439 (securityjsfrontend)
  • Vulnerability Rules for Python 8.9.0.11439 (securitypythonfrontend)
  • PHP Code Quality and Security 3.17.0.7439 (php)
  • ABAP Code Quality and Security 3.9.1.3127 (abap)
  • Vulnerability Rules for PHP 8.9.0.11439 (securityphpfrontend)

Other

Windows 10 10.0 amd64
iccarm.exe ANSI C/C++ Compiler V8.11.3.13950/W32 for ARM
IarBuild.exe IAR Command Line Build Utility V8.0.10.5001
Java 1.8.0_201 Oracle Corporation (64-bit)
Atlassian Bamboo version 7.1

Backtrace

error	14-Jul-2021 12:50:57	java.lang.IllegalStateException: unexpanded response file
error	14-Jul-2021 12:50:57		at com.sonar.cpp.analyzer.IarDriver.onCapture(IarDriver.java:95)
error	14-Jul-2021 12:50:57		at com.sonar.cpp.plugin.CFamilySensor.process(CFamilySensor.java:586)
error	14-Jul-2021 12:50:57		at com.sonar.cpp.plugin.CFamilySensor.process(CFamilySensor.java:366)
error	14-Jul-2021 12:50:57		at com.sonar.cpp.plugin.CFamilySensor.execute(CFamilySensor.java:175)
error	14-Jul-2021 12:50:57		at org.sonar.scanner.sensor.AbstractSensorWrapper.analyse(AbstractSensorWrapper.java:48)
error	14-Jul-2021 12:50:57		at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:85)
error	14-Jul-2021 12:50:57		at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:62)
error	14-Jul-2021 12:50:57		at org.sonar.scanner.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:82)
error	14-Jul-2021 12:50:57		at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:137)
error	14-Jul-2021 12:50:57		at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:123)
error	14-Jul-2021 12:50:57		at org.sonar.scanner.scan.ProjectScanContainer.scan(ProjectScanContainer.java:392)
error	14-Jul-2021 12:50:57		at org.sonar.scanner.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:388)
error	14-Jul-2021 12:50:57		at org.sonar.scanner.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:357)
error	14-Jul-2021 12:50:57		at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:137)
error	14-Jul-2021 12:50:57		at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:123)
error	14-Jul-2021 12:50:57		at org.sonar.scanner.bootstrap.GlobalContainer.doAfterStart(GlobalContainer.java:150)
error	14-Jul-2021 12:50:57		at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:137)
error	14-Jul-2021 12:50:57		at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:123)
error	14-Jul-2021 12:50:57		at org.sonar.batch.bootstrapper.Batch.doExecute(Batch.java:72)
error	14-Jul-2021 12:50:57		at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:66)
error	14-Jul-2021 12:50:57		at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:46)
error	14-Jul-2021 12:50:57		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
error	14-Jul-2021 12:50:57		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
error	14-Jul-2021 12:50:57		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
error	14-Jul-2021 12:50:57		at java.lang.reflect.Method.invoke(Method.java:498)
error	14-Jul-2021 12:50:57		at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
error	14-Jul-2021 12:50:57		at com.sun.proxy.$Proxy0.execute(Unknown Source)
error	14-Jul-2021 12:50:57		at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:171)
error	14-Jul-2021 12:50:57		at org.sonarsource.scanner.api.EmbeddedScanner.execute(EmbeddedScanner.java:128)
error	14-Jul-2021 12:50:57		at org.sonarsource.scanner.cli.Main.execute(Main.java:111)
error	14-Jul-2021 12:50:57		at org.sonarsource.scanner.cli.Main.execute(Main.java:75)
error	14-Jul-2021 12:50:57		at org.sonarsource.scanner.cli.Main.main(Main.java:61)

build-wrapper.log

...
process created with pid: 8148
image path name: <C:\Program Files (x86)\IAR Systems\ARM\8113\arm\bin\iccarm.exe>
command line: <"C:/Program Files (x86)\IAR Systems\ARM\8113\arm\bin\iccarm.exe" "--IDE2" "-f" "C:\Users\bamboo\AppData\Local\Temp\EWB6CA.tmp">
working directory: <C:\MSA\build-dir\HOP-HFST0-JOB1\Source\Rx\HardwareLayer\PE\Project\Generated_Code\>
isWow64: 1
{{ ERROR: error opening file for UTF-16 checking: C:\Users\bamboo\AppData\Local\Temp\EWB6CA.tmp}}
error reading file: C:\Users\bamboo\AppData\Local\Temp\EWB6CA.tmp
...

sonar-project.properties

sonar.projectKey=<...redacted...>
sonar.projectName=<...redacted...>
sonar.sources=Source
sonar.exclusions=<...redacted...>
sonar.language=c
sonar.c.std=c99
sonar.sourceEncoding=windows-1252
sonar.links.homepage=<...redacted...>
sonar.links.ci=<...redacted...>
sonar.links.issue=<...redacted...>
sonar.links.scm=<...redacted...>
sonar.cfamily.build-wrapper-output=Source/BuildOutput/sonar
sonar.scm.exclusions.disabled = True

Sonar launch configuration

git bash for windows:

"C:\BuildTools\sonar-scanner\bin\sonar-scanner.bat" \
	-Dsonar.branch.name=develop \
	-Dsonar.projectVersion=0.0.15 \
	-Dsonar.verbose=true \
	-Dsonar.cfamily.build-wrapper-output=BuildOutput/sonar-build-wrapper \
	-Dsonar.host.url=http://10.16.5.212:9000 \
	-X -e

Windows Defender Exclusions

Folder: C:\MSA\build-dir
File type: C:\Users\bamboo\AppData\Local\Temp\EW*.tmp
Process: C:\BuildTools\sonar-build-wrapper-win-x86\build-wrapper-win-x86-64.exe
Process: C:\BuildTools\sonar-scanner\bin\sonar-scanner.bat
Process: C:\Program Files (x86)\IAR Systems*
Process: C:\Program Files (x86)\IAR Systems\ARM\8113\arm\bin\iccarm.exe
Process: C:\Program Files (x86)\IAR Systems\ARM\8113\common\bin\IarBuild.exe
Process: C:\Program Files\Git\bin\git.exe
Process: C:\VCAST2020*

2

Hello @JonPovirk,

I sent you a PM for you to share the files.

4

Hello @JonPovirk ,

I can see in the logs that the build-wrapper version you are using is not consistent with the CFamily plugin:

build-wrapper-dump.json was generated using old build-wrapper version,
which does not match analyzer 6.20.0.31240 version.
Please download matching version from the server
*YourSonarQubeURL*/static/cpp/build-wrapper-win-x86.zip

Could you use the build-wrapper from the link please?

5

Whoops, my mistake – my apologies! Unfortunately that didn’t quite do the trick. I sent the failure I encountered under the latest build-wrapper in the PM. Thanks for your prompt responses!

7

Hello @JonPovirk ,

For some reason, it seems that sometimes we do not succeed in expanding one of the temporary file in C:\Users\bamboo\AppData\Local\Temp.
Could you try the following please:

  • Find in the build-wrapper.log the name of the file that causes the error (as you did before, e.g. EW8A2F.tmp)
  • Find this file in the build-wrapper.json (it should appear at one place) and remove the whole paragraph that corresponds to this file, i.e.
{
"compiler":"iar",
"executable":"XXX",
"cmd": [
"XXX",
"XXX",
"XXX",
"C:\\Users\\bamboo\\AppData\\Local\\Temp\\EW51D3.tmp"],
"cwd":"XXX",
"env": [
"XXX",
"big-list-of-items",
"XXX"]},

And then run sonar-scanner command.
Can you tell me what happens in this case?

In addition to this, do you see any encoding difference between the file that causes the error and the other bamboo temporary files?

Thank you

8

In fact, would you be able to entirely disable Windows Defender?
It seems that Windows Defender often blocks us from accessing some files, even when exclusions are set.

Thank you

9

I am having pretty much identical issue, with the same symptoms

  • failure is not 100% on the same code base
  • project is IAR based (8.32 in our case)
  • java.lang.IllegalStateException: unexpanded response file

I have tried spinning up a new VM for this, and installing IAR on it again = same results
I have completely (as far as I was able to find settings for that on the internet) disabled WIndows Defender = issue is not fixed
I have added ‘exclusions’ to Windows Defender, even though it is already disabled, and as expected that did not solve the issue as well.

Wed Jul 28 19:25:35 2021 - ERROR: error opening file for UTF-16 checking: C:\Users\builder\AppData\Local\Temp\EW6ECD.tmp
Wed Jul 28 19:25:35 2021: error reading file: C:\Users\builder\AppData\Local\Temp\EW6ECD.tmp
12

Apologies for the delay, I was on vacation last week.

From the log:

Tue Aug 03 10:52:45 2021 - ERROR: error opening file for UTF-16 checking: C:\Users\bamboo\AppData\Local\Temp\EW587C.tmp
Tue Aug 03 10:52:45 2021: error reading file: C:\Users\bamboo\AppData\Local\Temp\EW587C.tmp

EW587C.tmp was not present when I inspected the TMP directory upon failure, but some EW*.tmp files were. According to
Notepad++, these were in UTF-8 BOM encoding (CRLF line endings).

I removed the EW587C.tmp JSON object, disabled Windows Defender,
image
and manually ran the sonar-scanner command:

C:\BuildTools\sonar-scanner\bin\sonar-scanner.bat -Dsonar.branch.name=bugfix/1598-diagnose-sonar-unexpanded-response-file -Dsonar.projectVersion=0.0.0 -Dsonar.verbose=true -Dsonar.cfamily.build-wrapper-output=BuildOutput/sonar-build-wrapper -Dsonar.host.url=http://10.16.5.212:9000 -X -e

It was at this point where I noticed there was another instance of error reading file: in the log:

Tue Aug 03 10:56:14 2021 - ERROR: error opening file for UTF-16 checking: C:\Users\bamboo\AppData\Local\Temp\EW854A.tmp
Tue Aug 03 10:56:14 2021: error reading file: C:\Users\bamboo\AppData\Local\Temp\EW854A.tmp

This file was also not present in TMP. I deleted its entry in the JSON file as well and re-ran.

At this point, sonar-scanner was able to execute successfully.

13

Hello @JonPovirk,

Thanks for your detailed reply!

The error you are facing simply means that the build-wrapper wasn’t able to access the file. So the question we are trying to answer is why we weren’t able to access this file on your machine. The most common reason for such error is software(usually an antivirus) not allowing the build-wrapper to access the required file.

Now after you disabled Windows defender completely. Can you try again to run build-wrapper then the scanner(Without editing the JSON file in between)?

If it succeeds it means Windows defender was the problem.

If it fails, we will have to investigate further. Here are some suggestions:

  • Do you have antivirus other than windows defender installed? if yes disable it.

  • Try to monitor the state of the inaccessible files and the process that are trying to access them using software of your choice during build-wrapper execution. This way you can find out the possible reasons why the file couldn’t be accessed.

  • If all previous approaches fail, the last resort is to try to build a small example that can reproduce the issue on different machines. This way we can easily reproduce it on our side and investigate.

Thanks,

1 Like
14

Hi Abbas,

These were good suggestions! It took some time to get a proper run completed, (partly because windows defender will helpfully turn itself back on after a little while, so I needed to check it didn’t happen mid-build) but I think we now might just have what we need.

Before the build wrapper ran, I disabled anti-virus and “Automatic sample submission” for good measure.

windefend
windefend594×688 59.6 KB

I didn’t see any other anti-virus installed on the system. This machine is a little older, and didn’t have the additional AV installs that our modern workstations do.

Next, I setup a SysInternals Procmon filter to watch all file events in %TMP%

settings
settings1439×745 97.3 KB

Finally, rinse and repeat until a failure occurred while AV is disabled and the monitor is running:

Thu Aug 05 16:07:18 2021: process created with pid: 7536
Thu Aug 05 16:07:18 2021: image path name: <C:\Program Files (x86)\IAR Systems\ARM\8113\arm\bin\iccarm.exe>
Thu Aug 05 16:07:18 2021: command line: <"C:/Program Files (x86)\IAR Systems\ARM\8113\arm\bin\iccarm.exe" "--IDE2" "-f" "C:\Users\bamboo\AppData\Local\Temp\EW44D.tmp">
Thu Aug 05 16:07:18 2021: working directory: <C:\MSA\build-dir\HOP-TST0-JOB1\Source\Rx\Common\CM_HardwareTest\libhardwaretest\>
Thu Aug 05 16:07:18 2021: isWow64: 1
Thu Aug 05 16:07:18 2021 - ERROR: error opening file for UTF-16 checking: C:\Users\bamboo\AppData\Local\Temp\EW44D.tmp
Thu Aug 05 16:07:18 2021: error reading file: C:\Users\bamboo\AppData\Local\Temp\EW44D.tmp

After a short intermission to enable dark theme, I finally found some useful information:

SharingViolation
SharingViolation1917×1045 403 KB

A SHARING VIOLATION happens at 4:07:18.4756030 PM, matching the error timestamp in the log.

image
image1074×587 73.1 KB

We can also see iccarm.exe (PID 7536) that the build-wrapper mentioned in the process tree.

I have the monitor file saved to disk, so I can further scrutinize it if it’s helpful, or even send you the whole thing (though it’s huge so we may need to find a different means of transmission).

Hopefully this sheds some light on what’s going on. Thanks for your time!

16

Hi @JonPovirk ,

thank you for your this detailed report.

In the log you provided the SHARING_VIOLATION happens for file EW44D which has not been closed by IarBuild.exe process (i.e. EW45E gets closed and hence no sharing violation).

In order to understand if it is a bug on the IarBuild process which is not closing the file, could you provide the following part of the file access log to see if iccarm.exe successfully can access an unclosed file?

17

Hi Massimo,

Apologies again for the delay, I was off at the end of last week. That should be my last vacation for a while though, so fewer delays going forward.

Filtering by EW44D, iccarm and IarBuild appear to do quite a bit of work on the file before they’re finally through with it.

image
image1920×1066 388 KB

8/5/2021 4:07:21.4375254 PM is the last time IarBuild.exe touches the file:

image
image2266×265 61.2 KB

Let me know if there’s anything else I can do to help.

18

Hi @JonPovirk,

thank you for your detailed report. Few questions:

  • have you always faced this sharing violation using the build-wrapper?
  • or did it start after upgrading to a newer version of IarBuild?
  • would it be possible to check if a previous/newer version of IarBuild has the same symptoms?
19

Hi Massimo,

This is our first IAR project we updated to use the build wrapper. To my knowledge, we have not upgraded compiler versions since the project began.

I attempted to build the project with the next highest version of IAR we had installed (8.10.1), but unfortunately the project fails to build due to a number of vendor-provided files depending on features in the newer version. Unfortunately, backporting the project to older versions of IAR probably isn’t on the table. From myself and Konstantin, we know this issue effects at least versions 8.11.3 and 8.32 respectively.

For convenience, I’ve gathered the version numbers from some previous threads:

Unfortunately I couldn’t find any info on IAR 7.X, but others are at least seeing the same issue on 8.X and 9.X.

21

Hello @JonPovirk,

Apologies for the late reply.

I wouldn’t say they are the same issue. one of them is a totally different issue and the others has similar error messages but seem to be solved.

What I can notice from the shared monitoring log is the error happens when the file is not closed by the IAR process. This error is due to the fact that build-wrapper requires exclusive ownership to the files it access and when the file is not closed build-wrapper doesn’t get that.

To proceed we can do two things:

  • On the SonarSource side, investigate the possibility of not requiring exclusive access by build-wrapper. The ticket to follow.
  • On your side, It might be worth it to reach out to the compiler support asking them if this is expected behavior on their side. Maybe it is unintentional and fixing it would make the compiler compatible with more developer tools that require exclusive access.

Thanks,

2 Likes
22

We came across this discussion while investigating, why the analysis of a parallelized IAR build fails with exactly the error reported on this discussion, while the analysis of a non-parallelized IAR build works without issues.

So maybe non-parallelized IAR builds can be a workaround until the issue gets fixed either from SonarSource or from IAR side. FYI @JonPovirk.

3 Likes
23

Thanks a lot for looking into this everyone! We are indeed running a parallel build. I’ll try the single threaded build soon!

1 Like
24

I managed to run the build in single-threaded mode 20 times this weekend without encountering any failures. This project is small enough where lack of parallel builds isn’t a big deal (and indeed I would expect most embedded projects compiled under IAR are probably in the same boat).

2 Likes
25

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.