Sonar Wrapper taking agent down #4435

1 #1

MFalkner · 3 years ago

I've got cmake build that works just fine.
However, if I add the SonarCube wrapper the build takes down the QuickBuild agent.
The agent is running on Ubuntu, Quickbuild version 12.

So this works just fine:

$cmake_bin --build . -- -j $EXTRA_COMPILE

But this takes doen the agent:

$cmake_wrapper $cmake_bin --build . -- -j $EXTRA_COMPILE

whereas sonar_build_wrapper="$HOME/dev/build-wrapper-linux-x86/build-wrapper-linux-x86-64".

The same sonar build executed from a bash shell works just fine and does not show any error.

It seems like the agent receives a SIGKILL:

jvm 1 | 2022-08-26 17:41:01,380 INFO Workspace: /home/fusionuser/dev/buildagent/workspace/Fusion/GPR/GPR3/APP/GPR3_APP
wrapper | JVM received a signal SIGKILL (9).
wrapper | JVM process is gone.
wrapper | JVM exited unexpectedly.
wrapper | Reloading Wrapper configuration...
wrapper | JVM process is gone.
wrapper | Launching a JVM...
jvm 2 | WrapperManager: Initializing...

Any idea what could be the cause?
How could I debug this?
How could I avoid this?

solved #6
replies 5
views 1019
stars 0

robinshen ADMIN · 3 years ago

Both QB agent and Solar wrapper uses JSW (a third party service wrapper) to do its work. Looks like that JSW used by QB agent is being interfered by that of Solar wrapper. Is this issue happening even run Solar wrapper outside of QB? If so, can you give me detail steps to reproduce so that I can report this to JSW support?

MFalkner · 3 years ago

It happens only when built by the QB agent.
I tried to create a small example to reproduce the problem, but it always worked.
In the 'real' case, I had some parallel builds running. Removing those, the Sonar build started to work as well.
I assume, there is some conflicting access with the parallel build happening, that causes the agent to stop.
Build 1 builds in build.xxx.target
Build 2 builds in build.zzz.target
Build 3 builds in build.test
Build 4 builds in build.sonar
Usually, this is no problem when using cmake, as all the files for the build are created it the build directory.
Only after adding the Sonar build, the problem appeared. Running Builds 1-3 in parallel is no problem.
So far I was not able to find the conflict, as all builds use their own build directory, on the same source though.

Anyway, I can live with that for now, building the sonar report in a separate configuration.

robinshen ADMIN · 3 years ago

Thanks for the investigation. Let me know if there is more clue.

MFalkner · 2 years ago

Finally, I got to the bottom of this issue:
The cause is this great Ubuntu feature called systemd-oomd. It tries to manage out-of-memory issues by killing jobs randomly.
After disabling systemd-oomd, adding some physical memory and increasing the swap file size, the problem is gone.

robinshen ADMIN · 2 years ago

Thanks for letting me know.