Scanner skips child processes of newt project

Brief: I’m trying to run the sonarcscanner on a mynewt project. Mynewt is an embedded RTOS that’s written in C and uses GCC to build. The scanner is unable to detect the child processes of the ‘newt’ build automation tool. I suspect it may be related to mynewt’s build automation tool being built in golang.

Environment:
OS: Ubuntu 19.04
Wrapper: build-wrapper, version 6.5 (linux-x86)
SonarCFamily plugin: 6.5.0 (build 12506)
Scanner: SonarQube Scanner 4.2.0.1873 6.5.0 (build 12506)
SonarQube: Developer Edition Version 8.1 (build 31237)

Note: I’m using the workaround for the libinterceptor-haswell.so issue (https://jira.sonarsource.com/browse/CPP-2027)

Discussion: If I understand how SonarScanner works, it just intercepts the glibc syscalls, right? I’m wondering if the problem is related to newt being a golang program that sends most of its syscalls through libpthread instead of glibc. Is this a problem that’s perhaps been solved on the golang scanner already?

strace:

strace -f -o trace.log ./build-wrapper-linux-x86-64 --out-dir sonar-out newt build my_blinky_sim

(... many lines skipped)
Thu Jan 02 17:19:02 2020: executing: <newt build my_blinky_sim>
Thu Jan 02 17:19:02 2020: initializing json file
Thu Jan 02 17:19:02 2020: process created with pid: 4388
Thu Jan 02 17:19:02 2020: parent pid: 4385
Thu Jan 02 17:19:02 2020: working directory: </home/gfreese/ws/mynewt-sonar-test>
Thu Jan 02 17:19:02 2020: executable: </home/gfreese/ws/mynewt-sonar-test/build-wrapper-linux-x86-64>
Thu Jan 02 17:19:02 2020: argv[0]: <./build-wrapper-linux-x86-64>
Thu Jan 02 17:19:02 2020: argv[1]: <-c>
Thu Jan 02 17:19:02 2020: argv[2]: <>
Thu Jan 02 17:19:02 2020: argv[3]: <newt>
Thu Jan 02 17:19:02 2020: argv[4]: <build>
Thu Jan 02 17:19:02 2020: argv[5]: <my_blinky_sim>
Thu Jan 02 17:19:02 2020: skipping process with pid: 4388
Thu Jan 02 17:19:02 2020: process created with pid: 4389
Thu Jan 02 17:19:02 2020: parent pid: 4388
Thu Jan 02 17:19:02 2020: working directory: </home/gfreese/ws/mynewt-sonar-test>
Thu Jan 02 17:19:02 2020: executable: </home/gfreese/opt/newt/newt>
Thu Jan 02 17:19:02 2020: argv[0]: <newt>
Thu Jan 02 17:19:02 2020: argv[1]: <build>
Thu Jan 02 17:19:02 2020: argv[2]: <my_blinky_sim>
Thu Jan 02 17:19:02 2020: skipping process with pid: 4389
Thu Jan 02 17:19:15 2020: finalizing json file
Thu Jan 02 17:19:15 2020: returned with code: 0

Hi @grantfreese,

is this the log with the {{haswell}} workaround in place?

Is /home/gfreese/opt/newt/newt statically linked? We do intercept glibc syscalls using dynamic libraries which means that we cannot follow statically linked binaries.

Hi Massimo,

Yes- this is with the Haswell workaround in place. It fails much earlier on without the workaround.

Newt is dynamically linked.
gfreese@pop-os:~$ ldd ~/opt/newt/newt
linux-vdso.so.1 (0x00007fff7afe3000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f20f53cf000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f20f51e4000)
/lib64/ld-linux-x86-64.so.2 (0x00007f20f5416000)

Hi @grantfreese,

?

Newt is not statically linked. It’s dynamically linked. The ldd output in my prior reply shows the libraries it dynamically links with.

Hi @grantfreese,

sorry, I missed the ldd output, my bad.

Are you using newt docker container as described in Everything You Need in a Docker Container — Apache Mynewt latest documentation?

No worries =). No, I’m not running the docker container. It’s just running straight on my Ubuntu 19.04 system (sidenote- the ldd output was from a different machine).

So… does your scanner for golang operate any differently? I’m assuming it would be hooking libpthread, right? Is it possible to get this scanner to do that?

Hi @grantfreese,

another stupid question: are you running newt clean my_blinky_sim before?

I quickly tried on MacOS and everything works as expected, newt is forking processes to invoke the compiler.

Yeah- I’m actually running ‘newt clean all’.

Hmmm… I’ll give it try on an ubuntu 18.04 VM tomorrow and see if I get different results.

Hi @grantfreese,

I confirm the issue on Linux, I tried the following simple go program:

package main
import (
  "fmt"
  "log"
  "os/exec"
)
func main() {
  fmt.Printf("hello\n")
  cmd := exec.Command("gcc", "-c", "main.c")
  stdoutStderr, err := cmd.CombinedOutput()
  if err != nil {
    log.Fatal(err)
  }
  fmt.Printf("stdoutStderr: %s\n", stdoutStderr)
}

running ./build-wrapper-linux-x86/build-wrapper-linux-x86-64 --out-dir cfam-out go run main.go doesn’t see the gcc call. I guess it is the way that go fork and exec programs on linux, I need to investigate.

Hi @grantfreese,

I created a ticket to track this bug: CPP-2372

Hey Massimo,

Would you happen to have an ETA available for this bug?

Also, is build-wrapper-linux open source? I didn’t see a link from its page to the source.

Hi @grantfreese,

No, and from a brief look it doesn’t look that obvious and may be related to linking of executables.

build-wrapper is not open-source.