Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

Unusual Performance Issue Starting Steps, QB9.23 #4322

tomz ·

We have recently noticed a very sizable delay between a build step's execution condition being successful, and the step running. This the time delay is usually between 4 and 10 seconds, and this can result in this delay being a sizable portion of the total build time. Composition steps tend to be more problematic than individual ones.
We have tested the issue using a build configuration consisting of a composite step containing a single dummy step, and the composite step uses a repeat parameter to run numerous times. The dummy steps happen near instantaneously, but the composition steps take about 4 seconds to start. The logs showing the delay look like this:

09:31:23,021 INFO - Step execute condition satisfied, executing...
09:31:26,832 INFO - Executing pre-execute action...

Another mystery about this problem is that we have several Quickbuild servers running on fairly similar hardware and this only seems to be occurring on one of our production servers (though it is also the most active). There may be some minor Quickbuild configuration differences between these servers, as they are not clones of one another.
It isn’t clear what the build system is doing during this delay, and it occurs even if the composite step and child step are on the same agent. Can you provide suggestions to shorten or eliminate this delay, or help us profile what is going on here?
Some specs of what we are using:

  • Quickbuild 9.0.23 over SSL
  • Linux x64 OS
  • 95GB RAM, 2.7GHzx12 CPU VM
  • JVM max heap 28G, JVM max mem 28G (we're literally passing in -Xmx32768m via wrapper.conf so this is odd)
  • OpenJDK 1.8.0.131
  • replies 7
  • views 1642
  • stars 0
robinshen ADMIN ·

During this time QB has to determine which node to run the step. Does the node selection setting and node preference setting in child step executes custom script?

tomz ·

In the test, the entire thing runs on the server with no node-selection resources nor scripting.

robinshen ADMIN ·

Is this happening even if system is not busy (no many jobs in queue)? Also are you able to reproduce with a sample configuration?

tomz ·

It only reproduces on one specific server. It is a production server that is generally busy with at least 20 jobs running or in queue or more, except in the middle of the night. I will schedule the test to run in the overnight tonight to see if it has any better performance. Even if it did, it still seems odd that server queues would impact performance so much.

I am able to reproduce the problem using the sample configuration described in the initial question.

After running overnight, each composite step had a 5 second wait. Running it this morning during a busy queue (roughly 60 items) each step was taking up to 15 seconds! It appears to be getting worse.

robinshen ADMIN ·

Please export the sample configuration and send to me for diagnostics [robin AT pmease DOT com].

robinshen ADMIN ·

Also are you using cloud profile?

robinshen ADMIN ·

Please upgrade to latest QB10 version. It optimizes the logic of assigning nodes for node selection rule "On the same node running parent step", and should be much faster.