Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

Running several builds in parallel on the same agent #4029

nini10 ·

Hi,

In short:
Is it possible to run several builds (in same configuration) in parallel on same agent?

More details:
Master step of the configuration is fixed and running all builds on the same node. I want to execute several builds which will running in parallel on the same agent.
Currently, after first build start running, next ones waiting until it done (configuration configure as concurrency), as I understand the next build wait because can't run several builds on the same host workspace.

How can it be solve?

Thanks!

  • replies 11
  • views 946
  • stars 0
robinshen ADMIN ·

Unfortunately this is mandatory in QB to prevent possible race conditions on using same workspace by multiple concurrent builds.

nini10 ·

Well, I change the workspace folder to be unique (using variable), but it still doesn't run 2 or more builds in parallel.
What could be the problem?

robinshen ADMIN ·

That will not work either as QB internally locks against the configuration id. Also please do not use different workspace for different builds of same configuration. That will confuse QB and can result in unexpected results.

nini10 ·

Ok, using different workspace is not an option.
Can you please recommend what is the right way to achieve this functionality - run several builds (in same configuration) in parallel on same agent?

robinshen ADMIN ·

Running several builds of same configuration concurrently on same agent is not supported. What is your usage scenario?

nini10 ·

I have one host that serve as tests execution manager and it runs several tests, in parallel on many EC2 instances. Each build could be run several hours.
The tests execution manager using "pytest" and could run many sessions independent.
So, what I want is to allow several builds to run concurrency on same QB configuration - while each build run on different EC2 instances.

Can you please help me to find solution for this?

robinshen ADMIN ·

You may consider installing multiple build agents on that host, and each build agent executes its "pytest" to do the job.

nini10 ·

This solution could works, but it doesn't scale while it depend on the number of build agents.
Can you please help me to think on more scale solution?

Thanks for your help

robinshen ADMIN ·

No other approaches I can think of if the host is a the only host to orchestrate the test execution.

nini10 ·

How about create main configuration, lets call it "RunnersManager" and "N" runner configurations, lets call each of them "Runner".
The "RunnerManager" will check for "idle" runner and trigger the "pytest" of it. If all are busy it waits for the first one that done.
By doing this, for any runner will be different workspace and they could run in parallel.

What do you think?

If you think it could work, can you please provide example code for how the "RunnerManager" could iterate all the "Runner" and check it busy or idle?

Thanks you for great support

robinshen ADMIN ·

If using multiple runner configurations is an option, you may create a grid resource say "runners" with for instance "100" on the test manager host, and then edit "node selection setting" of master step of these runners (can inherit from parent conf for simplicity) to require the resource. This way, only 100 runners are allowed to run concurrently on the test manager