C** build wrapper - libinterceptor-${PLATFORM}.so from LD_PRELOAD cannot be preloaded

vogtlaender · February 3, 2021, 5:03pm

SonarQube, version 7.9.5 (build 38598)
build-wrapper, version 6.3 (linux-x86)
Tested with Ubuntu 18.04, 20.10 and Debian 10.7 in VirtualBox VM (Ubuntu 20.10 host)

When running gcc cross-compiler from a custom arm-toolchain (created with Yocto) inside a virtual machine I get the following error

./build-wrapper-linux-x86/build-wrapper-linux-x86-64 --out-dir sq ${CC} --version
ERROR: ld.so: object '/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-${PLATFORM}.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
...

Please note that this is not CPP-2027

Running with strace shows that the build-wrapper tries to load the file “” which of course fails.:

openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-i686.so", O_RDONLY) = 3
openat(AT_FDCWD, "/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-x86_64.so", O_RDONLY) = 3
[pid 155223] openat(AT_FDCWD, "/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-x86_64.so", O_RDONLY|O_CLOEXEC) = 3
[pid 155223] openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
[pid 155223] openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
[pid 155223] openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
[pid 155224] openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
...

Calling other executables (e.g. “ls”) works without issues:

openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-i686.so", O_RDONLY) = 3
openat(AT_FDCWD, "/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-x86_64.so", O_RDONLY) = 3
[pid 36179] openat(AT_FDCWD, "/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-x86_64.so", O_RDONLY|O_CLOEXEC) = 3
[pid 36179] openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
[pid 36179] openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
[pid 36179] openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
[pid 36179] write(3, "LD_PRELOAD=", 11 <unfinished ...>
[pid 36178] <... read resumed>"LD_PRELOAD=", 11) = 11
[pid 36178] openat(AT_FDCWD, "sq/build-wrapper.log", O_WRONLY|O_CREAT|O_APPEND, 0666     36179: binding file ./build-wrapper-linux-x86/build-wrapper-linux-x86-64 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `malloc') = 4
[pid 36181] openat(AT_FDCWD, "/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-x86_64.so", O_RDONLY|O_CLOEXEC) = 3
....

Strangely, the LD_PRELOAD error does not occur on the host system itself. Only on the virtual machine s(and also on the virtual build server) are affected. I’m using the same tool-chain in all systems so there should not be any differences regarding the compiler executable.
On my local system I had to create libinterceptor-haswell.so in order to make it work, though (which is also weird because “uname -i” as well as “LD_SHOW_AUXV=1 sleep 1” returns AT_PLATFORM = “x86_64”, I don’t understand why $PLATFORM is “haswell” in this case).

mpaladin · February 4, 2021, 9:51am

Hi @vogtlaender ,

Note that you can update your cfamily analyzer and build-wrapper and have the haswell issue fixed.

The issue you are facing might be related with the vm created by yocto. It is quite strange that you are getting an issue invoking the compiler and not ls. Would it be easy to reproduce it on my side by creating a vm the same way you are doing?

vogtlaender · February 5, 2021, 9:33am

I prepared a minimal example:
build-wrapper.zip (645.2 KB)

In order to trigger the issue install a Ubuntu (e.g. 20.10) inside a VirtualBox VM (leaving options at default, I only increased RAM to 8GB and number of processors to 4)

Then extract build-wrapper.zip in /tmp/sonarqube and run:

$> ./build-wrapper-linux-x86/build-wrapper-linux-x86-64 --out-dir sq ./ld-linux-x86-64.so.2 ./col
ERROR: ld.so: object '/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-${PLATFORM}.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
./col: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory

I hope the missing shared objects from the tool-chain (“error while loading shared libraries: libc.so.6”) are not a problem. If you need the whole tool-chain (which is ~1GB) please let me know.

Running with strace produces:

$> strace -f -E LD_DEBUG=all ./build-wrapper-linux-x86/build-wrapper-linux-x86-64 --out-dir sq ./ld-linux-x86-64.so.2 ./col 2>&1 | grep -e "openat(" -e LD_PRELOAD
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "sq", O_RDONLY)        = 3
openat(AT_FDCWD, "sq", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
openat(AT_FDCWD, "/tmp/build-wrapper-socket.AwE0PB", O_RDWR|O_CREAT|O_EXCL, 0600) = 3
openat(AT_FDCWD, "sq/build-wrapper.log", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-i686.so", O_RDONLY) = 4
openat(AT_FDCWD, "/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-x86_64.so", O_RDONLY) = 4
openat(AT_FDCWD, "/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-haswell.so", O_RDONLY) = 4
[pid  2767] openat(AT_FDCWD, "/tmp/build-wrapper-socket.AwE0PB", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid  2767] openat(AT_FDCWD, "sq/build-wrapper-dump.json", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4
[pid  2769] openat(AT_FDCWD, "/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-x86_64.so", O_RDONLY|O_CLOEXEC) = 4
[pid  2769] openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
[pid  2769] openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 4
[pid  2769] openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 4
[pid  2769] write(4, "LD_PRELOAD=", 11 <unfinished ...>
[pid  2767] <... read resumed> "LD_PRELOAD=", 11) = 11
[pid  2770] openat(AT_FDCWD, "./col", O_RDONLY|O_CLOEXEC) = 4
[pid  2770] openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  2770] writev(2, [{iov_base="ERROR: ld.so: object '", iov_len=22}, {iov_base="/tmp/sonarqube/build-wrapper-lin"..., iov_len=68}, {iov_base="' from ", iov_len=7}, {iov_base="LD_PRELOAD", iov_len=10}, {iov_base=" cannot be preloaded (", iov_len=22}, {iov_base="cannot open shared object file", iov_len=30}, {iov_base="): ignored.\n", iov_len=12}], 7ERROR: ld.so: object '/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-${PLATFORM}.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
[pid  2770] openat(AT_FDCWD, "/opt/angstrom/2017.01/sysroots/x86_64-slvsdk-linux/lib/tls/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  2770] openat(AT_FDCWD, "/opt/angstrom/2017.01/sysroots/x86_64-slvsdk-linux/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  2770] openat(AT_FDCWD, "/opt/angstrom/2017.01/sysroots/x86_64-slvsdk-linux/usr/lib/tls/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  2770] openat(AT_FDCWD, "/opt/angstrom/2017.01/sysroots/x86_64-slvsdk-linux/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  2770] openat(AT_FDCWD, "/opt/angstrom/2017.01/sysroots/x86_64-slvsdk-linux/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

When not running inside the VM (but also on a Ubuntu 20.10 system) libinterceptor-haswell.so gets loaded and there is no LD_PRELOAD error:

$> strace -f -E LD_DEBUG=all ./build-wrapper-linux-x86/build-wrapper-linux-x86-64 --out-dir sq ./ld-linux-x86-64.so.2 ./col 2>&1 | grep -e "openat(" -e LD_PRELOAD
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "sq", O_RDONLY)        = 3
openat(AT_FDCWD, "sq", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
openat(AT_FDCWD, "/tmp/build-wrapper-socket.KBmHYg", O_RDWR|O_CREAT|O_EXCL, 0600) = 3
openat(AT_FDCWD, "sq/build-wrapper.log", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/home/vogtlaender/dev/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-i686.so", O_RDONLY) = 4
openat(AT_FDCWD, "/home/vogtlaender/dev/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-x86_64.so", O_RDONLY) = 4
openat(AT_FDCWD, "/home/vogtlaender/dev/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-haswell.so", O_RDONLY) = 4
[pid 1949249] openat(AT_FDCWD, "/tmp/build-wrapper-socket.KBmHYg", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 1949249] openat(AT_FDCWD, "sq/build-wrapper-dump.json", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4
[pid 1949250] openat(AT_FDCWD, "/home/vogtlaender/dev/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-haswell.so", O_RDONLY|O_CLOEXEC) = 4
[pid 1949250] openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
[pid 1949250] openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 4
[pid 1949250] openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 4
[pid 1949250] write(4, "LD_PRELOAD=", 11 <unfinished ...>
[pid 1949249] <... read resumed>"LD_PRELOAD=", 11) = 11
[pid 1949251] openat(AT_FDCWD, "./col", O_RDONLY|O_CLOEXEC) = 4
[pid 1949251] openat(AT_FDCWD, "/home/vogtlaender/dev/tmp/sonarqube/build-wrapper-linux-x86/libinterceptor-haswell.so", O_RDONLY|O_CLOEXEC) = 4
[pid 1949251] openat(AT_FDCWD, "/opt/angstrom/2017.01/sysroots/x86_64-slvsdk-linux/lib/tls/haswell/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 1949251] openat(AT_FDCWD, "/opt/angstrom/2017.01/sysroots/x86_64-slvsdk-linux/lib/tls/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 1949251] openat(AT_FDCWD, "/opt/angstrom/2017.01/sysroots/x86_64-slvsdk-linux/lib/haswell/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 1949251] openat(AT_FDCWD, "/opt/angstrom/2017.01/sysroots/x86_64-slvsdk-linux/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 1949251] openat(AT_FDCWD, "/opt/angstrom/2017.01/sysroots/x86_64-slvsdk-linux/usr/lib/tls/haswell/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 1949251] openat(AT_FDCWD, "/opt/angstrom/2017.01/sysroots/x86_64-slvsdk-linux/usr/lib/tls/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 1949251] openat(AT_FDCWD, "/opt/angstrom/2017.01/sysroots/x86_64-slvsdk-linux/usr/lib/haswell/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 1949251] openat(AT_FDCWD, "/opt/angstrom/2017.01/sysroots/x86_64-slvsdk-linux/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 1949251] openat(AT_FDCWD, "/opt/angstrom/2017.01/sysroots/x86_64-slvsdk-linux/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

mpaladin · February 11, 2021, 3:29pm

Hi @vogtlaender,

so far I didn’t have time to try to investigate. The guess on my side is Ubuntu 20.10 and its glibc version which misbehave when running within virtualbox. Would it be possible for you to give it a try on LTS version of Ubuntu like 18.04 or 20.04?

vogtlaender · February 11, 2021, 4:26pm

Hi @mpaladin ,

I already tried with Ubuntu 18.04, Ubuntu 20.04, Arch Linux and also Debian 10.7 (yes, I was desperate ). Same error on all Systems. The Debian System is running on a Linux-VServer, all other Systems on VirtualBox.

mpaladin · February 15, 2021, 4:05pm

Hi @vogtlaender ,

does it mean that you tried the following (at the same time)?
Host: Ubuntu 18.04
Virtualbox VM: Ubuntu 18.04

vogtlaender · February 15, 2021, 4:52pm

Hi @mpaladin

no, my setup ist currently:
Host: Ubuntu 20.10
VirtualBox VM: Ubuntu 18.04

mpaladin · February 16, 2021, 8:20am

Hi @vogtlaender ,

would you be able to try with Ubuntu 18.04 both as Host and VM?