Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

running concurrent builds #4478

drdt ·

I find a couple places where QB is not behaving as I expected. Can you help me understand?

We want to run concurrent jobs of the same configuration (on-demand builds requested by users with shelved changes). I know that QB locks the workspace so this cannot happen, so we have modified our workspace definition to include a unique subdirectory. (Our workspaces are on a network drive, so all agents share the same workspace within a job).

So, for our configuration root/test/with-shelf, the build runs in folder /nfs-drive/root/test/with-shelf/shelf-number. I expect this to mean that no matter how many builds we trigger, each will get its own workspace and can therefore run independently. However, in spite of this, each job waits for the next to finish before proceeding.

We discovered quite accidentally that, if we run the master step on "any linux node" instead of "on qb server", the problem is resolved. (However, this introduces a new problem, which I will raise in another thread). In fact, it seems that concurrent jobs can share the same workspace as long as the master steps run on different agents, potentially causing collision. This is not the behavior that I was expecting.

What is going on here?

  • replies 7
  • views 233
  • stars 0
robinshen ADMIN ·

QB expects workspace of a configuration remains unchanged on a particular agent or server. Parameterize workspace for different run of the job may cause unexpected results. To run concurrent builds of same configuration, make sure you have multiple agents and these agents are not sharing workspace. To pass files between agents, use the input/output files facility of step.

drdt ·

Unfortunately, that is going to be a problem for us. We are right now dealing with a workspace of 23gb, which is built on multiple platforms. Our solution (even before QuickBuild) to check it out onto a network drive and then run the build on each platform via NFS.

Copying it to each agent will require major increases to our disk space requirement as well as slow the build down. I actually tried this, it takes about a half hour to copy the source files to the agent, and another 45m to copy the binaries back.

Another option might be to have the network drive be different from the agent workspace, so we basically do the build outside of QuickBuild's control. Then we can have a final step on the server which copies the output files from the network to the workspace for publication. Do you see a risk in trying it this way?

robinshen ADMIN ·

That might work. But you will need to control concurrent access to that shared folder carefully, so that no two steps are reading/writing to this shared folder in the same time.

drdt ·

Alas, that will not work either. It turns out that, no matter what I do, no two jobs of the same configuration can use the same machine, even for a child step. So, when I kick off my five parallel steps that each run on a different platform, they conflict with the other simultaneous jobs and have to wait for each other, even though the machines can each handle up to 12 simultaneous.

I would end up needing one box of each platform for every build I want to run:

master on server-1 --> checkout --> run builds on (macosx-1, rhel7-1, rhel8 on docker-1, rhel9 on docker-1, aix-1)
master on server-2 --> checkout --> run builds on (macosx-2, rhel7-2, rhel8 on docker-2, rhel9 on docker-2, aix-2)

This is really a hard restriction to deal with. Is there any way to override this restriction, or do you have another suggestion? I think I could get around it by installing 10+ agents on each box, but that would be an administrative nightmare.

robinshen ADMIN ·

This is a basic design consideration that builds of same configuration can not run concurrently on same machine. Changing this may break many things. Installing multiple build agents on same machine is a good idea. Since agents can upgrade themselves, you can forget it once installed. We even have customers running 1k+ agents.

drdt ·

The approach we are trying now is to create a new child configuration for each such job. The configurations are per-user, so each user can only request one build of this type at a time (don't want anyone to go crazy on us).

However, this requires a prompted variable to select the target config, and then populate other prompted variable defaults based on this. To this end I am now researching beans. Can you point me to some good documentation for this? I looked in the v13 documentation, and found the "Bean Editor" instructions, but it seems incomplete (and there is a dead link "QBTEAM:Write a Simple Plugin").

Is there better information anywhere?

robinshen ADMIN ·