We have recently noticed a very sizable delay between a build step's execution condition being successful, and the step running. This the time delay is usually between 4 and 10 seconds, and this can result in this delay being a sizable portion of the total build time. Composition steps tend to be more problematic than individual ones.
We have tested the issue using a build configuration consisting of a composite step containing a single dummy step, and the composite step uses a repeat parameter to run numerous times. The dummy steps happen near instantaneously, but the composition steps take about 4 seconds to start. The logs showing the delay look like this:
09:31:23,021 INFO - Step execute condition satisfied, executing...
09:31:26,832 INFO - Executing pre-execute action...
Another mystery about this problem is that we have several Quickbuild servers running on fairly similar hardware and this only seems to be occurring on one of our production servers (though it is also the most active). There may be some minor Quickbuild configuration differences between these servers, as they are not clones of one another.
It isn’t clear what the build system is doing during this delay, and it occurs even if the composite step and child step are on the same agent. Can you provide suggestions to shorten or eliminate this delay, or help us profile what is going on here?
Some specs of what we are using:
- Quickbuild 9.0.23 over SSL
- Linux x64 OS
- 95GB RAM, 2.7GHzx12 CPU VM
- JVM max heap 28G, JVM max mem 28G (we're literally passing in -Xmx32768m via wrapper.conf so this is odd)
- OpenJDK 1.8.0.131