Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

Builds timeout when waiting for other builds #3826

tardis4500 ·

I notice that if a build reaches a step where it is waiting on another build, it might timeout even though the build itself didn't. Is there a way to have the build timeout only include the actual time to run the steps?

  • replies 12
  • views 1560
  • stars 0
robinshen ADMIN ·

It is difficult to count actual timed used by a step, for instance the step itself might be a script containing logic to wait for other builds. Also it is meaningful to include time used by other builds as it can serve to monitor spawned builds.

tardis4500 ·

Unfortunately, this makes using the build timeouts worthless for our situation. Most of the builds take less than thirty minutes but I have one that takes over an hour. But since they all share the same build servers, I have to set the build timeout to be the sum of all builds (~4hours). The only other solution is to add more build servers but that's not an option for cost reasons.

robinshen ADMIN ·

If it is a problem a limited resource, you do not need to have one build waiting for another via "trigger build" step. Instead you may configure resources and configure these builds to require the resources. This way these builds are independent and build execution time will be calculated independently.

tardis4500 ·

They are independent builds but they have steps which need on a common resource. When a build gets to a step that needs a resource that is unavailable because another build is using it, the time it waits for that resource is counted against the build time. Would it be possible to add an option to a step like, "Don't count time waiting for a resource again the build time"?

robinshen ADMIN ·

You may specify step timeout separately in this case while leave build timeout to be a large value.

tardis4500 ·

That the reverse of what I do. Let me try to be clearer. I have two independent builds, one which I know should only last 15 minutes, the other which lasts an hour. The builds use different masters but both have a build step inside the master which runs on a common resource. If the longer build is running its step which needs that resource when the shorter build starts, the shorter build will wait until the resource is available which will cause the shorter build to timeout.

robinshen ADMIN ·

I think it is difficult to count independent step execution time for build timeout, as:

  1. QB may have multiple steps running concurrently: some might be running, and some might be waiting
  2. What is worse is that QB often can not know details of step execution: for instance it might be executing a custom groovy script waiting for some resource after doing some tasks and before continue with next task

If you have common tasks executing on master and you do not want to count for its waiting time, you may create a separate configuration doing this common task and have your current independent builds dependent on this common configuration. Then time spent on common configuration will no longer be counted into downstream configurations.

tardis4500 ·

I don't think that would work in my case since the steps don't really do the same thing, they just need a common resource. In my case I have builds which get built across multiple platforms. I have a parallel loop which uses a variable to determine which resources the build gets built on. So I might have two builds which need the "Linux26x86_64 Build Engine" resource in one of their build steps but I don't want to have 10 of them (one for every build). So I keep around 2 or 3 of them but if I get 5 builds running at the same time which get built on that platform, I may get one build timeout because it had to wait to long. I know it might not be easy but it would be helpful to have a setting which told QuickBuild not to count time waiting for resources. As it is now I have to determine the worst case scenario and set a build timeout of 4-5 hours even though a build should only take 30 minutes. So if I get a build which is really stuck I don't find out until far too late.

robinshen ADMIN ·

Sorry I'd rather not to implement this feature as it will be logically flawed and can cause confusions in concurrent steps. Having a dependency build waiting for the common resource is the way to go, and you can control the dependency build via variables so that it behaves differently when used by different configurations.

tardis4500 ·

I understand your concerns about implementing the feature but I really don't think a dependency build would work well. I want to keep the build logs with the builds that run them. Having the logs for all builds interspersed in the dependency build would be a but of a tracking nightmare and lead to potentially deleting logs that are needed when the dependency build is cleaned up. And I'm not even sure that would help at all unless I wrote a script to change the timeout on the dependency build itself based on the build that was calling it. For some builds I would want the timeout on the dependency build to be 10 minutes for other 2 hours.

drdt ·

Pass the timeout as an option when triggering the dependency build?

I didn't like it either; my solution was to have a step within a step; the parent step grabs the resource with a very long timeout, and then the child step is triggered with its own timeout once we know it isn't waiting on anything.

tardis4500 ·

I have a variation on that that might work. The actual build step is inside a container step that gets the resource. This means I should be able to set the build to be long and use variables to set the build step timeout on a project-by-project basis.