running concurrent builds #4478

1 #1

drdt · 2 weeks ago

I find a couple places where QB is not behaving as I expected. Can you help me understand?

We want to run concurrent jobs of the same configuration (on-demand builds requested by users with shelved changes). I know that QB locks the workspace so this cannot happen, so we have modified our workspace definition to include a unique subdirectory. (Our workspaces are on a network drive, so all agents share the same workspace within a job).

So, for our configuration root/test/with-shelf, the build runs in folder /nfs-drive/root/test/with-shelf/shelf-number. I expect this to mean that no matter how many builds we trigger, each will get its own workspace and can therefore run independently. However, in spite of this, each job waits for the next to finish before proceeding.

We discovered quite accidentally that, if we run the master step on "any linux node" instead of "on qb server", the problem is resolved. (However, this introduces a new problem, which I will raise in another thread). In fact, it seems that concurrent jobs can share the same workspace as long as the master steps run on different agents, potentially causing collision. This is not the behavior that I was expecting.

What is going on here?

replies 3
views 30
stars 0

robinshen ADMIN · 2 weeks ago

QB expects workspace of a configuration remains unchanged on a particular agent or server. Parameterize workspace for different run of the job may cause unexpected results. To run concurrent builds of same configuration, make sure you have multiple agents and these agents are not sharing workspace. To pass files between agents, use the input/output files facility of step.

1 #3

drdt · 4 days ago

Unfortunately, that is going to be a problem for us. We are right now dealing with a workspace of 23gb, which is built on multiple platforms. Our solution (even before QuickBuild) to check it out onto a network drive and then run the build on each platform via NFS.

Copying it to each agent will require major increases to our disk space requirement as well as slow the build down. I actually tried this, it takes about a half hour to copy the source files to the agent, and another 45m to copy the binaries back.

Another option might be to have the network drive be different from the agent workspace, so we basically do the build outside of QuickBuild's control. Then we can have a final step on the server which copies the output files from the network to the workspace for publication. Do you see a risk in trying it this way?

robinshen ADMIN · 3 days ago

That might work. But you will need to control concurrent access to that shared folder carefully, so that no two steps are reading/writing to this shared folder in the same time.