Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

QB Steps Execute Out of Order #4162

jmuirhead ·

Hi Robin:

Quickbuild Version: 8.0.5

We at Intel are seeing a bit of a disturbing problem; occasionally our configuration steps run out of order. We've seen this only on Windows agents -- virtual machines on a Hyper-V host. At least 95% of the times the build steps execute correctly. But from time-to-time, for whatever reason, they execute out of order. We have >350 Windows Server 2016 virtual machine agents on Hyper-V hosts attached to our QB Server. Our Windows Administrators are investigating the possibility the issue is associated with our Hyper-V setup. Before diving into an upgrade to version 8 latest or version 9, would it be possible to have a quick conference call to filter out any possibilities this is not a Quickbuild issue? This issue is rapidly gaining visibility with our firmware development management team, so we'd like to get it addressed as soon a possible.

Attached is a screenshot of the out-of-order step execution (out_of_order_execution.png), and a screenshot of how the steps are actually defined in the QB configuration (defined_step_order.png).

Thanks Robin.

James Muirhead
jamesx.muirhead@intel.com
out_of_order_execution.pngdefined_step_order.png

  • replies 4
  • views 62
  • stars 0
robinshen ADMIN ·

Seems like a QB side issue. But never encountered this type of issue in 13 years of QB. Does the build log exhibit the same out-of-order issue? To further investigate the issue, you might need to upgrade to the version with extra debug logging.

We can arrange a online meeting, but may not be very helpful at this point.

robinshen ADMIN ·

Also it might be of great help if you can send me backup of your database (via QB administration / database backup) mentioning normal and out-of-order build id

jmuirhead ·

Thanks Robin. A little more research has turned up some new information. First, I forgot to mention that no agents have a static IP address. All use DHCP. Also, this problem appears to only have occurred during our busiest time of day. During those time frames all 350+ Windows build agents are basically continuously running builds. The 'Resource Heat Map' is black.

Digging into the log of the failed build shown above indicates that the steps did execute in the proper order. What we're currently looking into is whether or not this could be related to DHCP lease expiration/renewal. IOW, from QB's perspective, what happens if a build agent, or agents, IP address changes in the middle of a build?

robinshen ADMIN ·

Changing the ip in the middle of the build will confuse the server as the server recognizes the agent via ip address and port number. It will certainly fail the build. Subsequent builds should be successful though as long as ip address remains unchanged.