Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

distribute builds/test to different agents when using repeat params #3843

HealingQuickly ·

Hello Robin

I have a question about the distributed builds (using parallel step).
I understand that if you have (let's say) 2 kinds of builds, one for linux and one for windows, we can use parallel step to wrap two child steps, with different node selection schemes.


My situation is, I have just one kind of builds, it can be run on any of my agents. And now I'm trying to run the same build multiple times with different arguments, and I'm using repeat parameter to iterate through all the different arguments. And I found out that when I put this build step inside a parallel step, when it repeats (let's say 5 times), they will all be run in parallel.


My question is, how can I distribute the repeating builds to different agents? If I specify the node selection scheme for the repeating step (the one inside the parallel wrapper), will they be distributed to multiple agents as long as I have more than one?
If I only had one agent for the build, it will still be run on parallel, I guess it's not good if the machine cannot handle too many arguments (what if I had 100 arguments to loop from)? In this case, how can I limit the amount of builds on the only build agent?


In addition, which one is faster, checkout every time on an agent, or use Input/Output file transfer?
Thank you very much!

  • solved #2
  • replies 7
  • views 1231
  • stars 0
robinshen ADMIN ·

If multiple agents match the node selection rule, steps will be distributed to them based on machine load. You may also limit the concurrency level via the number of workers in parallel step. Checkout on agent should be faster as SCM tool can checkout incrementally, while input/ouput needs to pull all files every time.

HealingQuickly ·

Hi Robin

Would you please take a look at the screen shot of mine. I tried to do some tests on the worker number settings of parallel steps. I have only one step inside, which repeats by parameters.
Image.jpg

I set the worker number to 2, and the groovy step inside indeed only run 2 at most at a time. But my question here is, how come they are not distributed to different machines? As you can see, I set the node selection scheme to "any node", the parameter has 4 values, and I expected to have two machines that run 2 steps in parallel, so that all 4 steps get run in parallel on 2 machines.
Instead, I see the step gets executed on the same machine, and the rest 2 param values waited for one of the first two to complete, then execute.

Could you tell me why this is, and how I should set the steps so that I can have 2 machines run 2 steps in parallel at the same time?

Thank you very much!


Updates
Now I set the worker number to 4, and it runs 4 steps in parallel on different machines, so the worker number is for the total parallel steps that are able to run. But sometimes one machine gets to run two steps at the same time. This seems to be distributed based on which one is the least loaded machine? Here is a strange question, if certain machine has not much memories left, and it's the only machine available, and the step needs a very heavy load of work to be done. Will QB prevent the machine from running the step and potentially being frozen, and wait for other machines that deallocated more memories?

Thank you!

robinshen ADMIN ·

QB tries to use least loaded node in this case. However the load calculation might not be very accurate: it first benchmarks the agent when QB starts, and then divides by number of steps (also counting steps from other builds) currently running on the machine.

HealingQuickly ·

Hi Robin

I'd appreciate it if you could answer me this old question.

How exactly does distributed build work? First of all, where do I do checkout?
If I specify the node selection for checkout step, how do the other agents find the checkout files?

Thank you!

robinshen ADMIN ·

It is all up to you. For instance, you may checkout on agentA, and arrange subsequent steps to run on agentA. You may also run subsequent steps on agentB, and transfer checkout files (or part of them) from agentA to agentB via step input/output files. Here is some tutorials:
http://wiki.pmease.com/display/QB70/Distributed+Builds

HealingQuickly ·

Thank you

I've checked these examples. You previously said it's faster to do checkout on agentB (I do have a lot of files that running a build needs). That's what I'll do. But it seems there will be multiple copies of the checkout files on different agents now.

robinshen ADMIN ·

Unless absolutely necessary, I'd suggest to put checkout step and build step on same agent for performance reason. But QB is flexible if you really want to put different steps on different nodes.