Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

QuickBuild

how to handle jobs that use more than one core #4510

1 #1
drdt ·

I have a situation which might be beyond the design capabilites of QB.

All of our docker build agents are set up with a 1-1 mapping between resource count and CPU cores (minus one for over head). So an 8-core machine has a resource count of 7, a 24-core machine supports 23, etc. This usually works well fbecause most of our builds are single-threaded; using the resource counts, we guarantee that each job gets one CPU all to its own.

However, we have a new type of job which takes advantage of parallelism in 'gmake' to complete builds faster. These jobs get the best performance when they have access to 3-5 cores (more than that gives diminishing returns).

We tried changing the mapping of resources to cores to 4-1, but then my jobs that run single-threaded were also allocated four cores, wasting precious time. I could have different machines for each type of job, but then on days when there are no big jobs running, agents are sitting idle. In theory any machine should be able to run a combination of both types of jobs, making optimal use of the CPU.

Do you have any suggestions how I could accomplish my goals?

  • replies 7
  • views 492
  • stars 0
#2
robinshen ADMIN ·

You may still define resource as before, but specify node selection setting to use an AND operation to add number of criterias to require same resource multiple times. I also filed an improvement request to be able to require more than one resource in a single criteria to make this task easier.

#3
robinshen ADMIN ·
#4
drdt ·

Thanks, I am very looking forward to that improvement. Do we have a time frame? Is it likely to be a patch to 13.0 or a future version?

#5
drdt ·

Meanwhile, I tried to be clever and define a new resource that chains off the other resource:

bigDockerHost =
-- Nodes matching all specified criterias
---- Node matches another resource: anyDockerHost
---- Node matches another resource: anyDockerHost
---- Node matches another resource: anyDockerHost
---- Node matches another resource: anyDockerHost

I thought this would reserve four instances of the anyDockerHost resource, but it doesn't reserve any.
Is this behavior that has changed, or did it always work this way and I just misunderstood it?

#6
robinshen ADMIN ·

Thanks, I am very looking forward to that improvement. Do we have a time frame? Is it likely to be a patch to 13.0 or a future version?

I plan to get it into a patch release of QB 13, probably in the end of this month.

I thought this would reserve four instances of the anyDockerHost resource, but it doesn't reserve any.
Is this behavior that has changed, or did it always work this way and I just misunderstood it?

This is expected behavior, in this case, only node selection rule of other resources are checked. The resource itself will not be reserved.

1 Reply
#7
drdt ·

Thanks, we will adjust our expectations accordingly.

#8
robinshen ADMIN ·

The ability to reserve more than one instances of resource in a single resource requirement has been added in latest QB version.

Log in to reply