Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

need to decide at run-time whether or not to copy files to local #4449

drdt ·

I have a puzzle. For a majority of our builds, we use network storage for our build workspaces, so all of our agents look in a network location: \nas-storage\qbdir\workspaces\servername\root\config\path. This way, multiple agents can share files via the workspace.

However, some operations are so big, it is faster to copy the source files to a local disk, do the build, and copy the results back. We accomplish this via the 'Fetch Input Files' and 'Send Output Files' options under Advanced settings. Our build scripts are clever enough to do the right thing once the files are there (using the local directory instead of the workspace) depending on a variable "flagUseLocalWinWorkspace".

My problem is that, I cannot programmatically choose whether or not to fetch/send the files. And, if I use fetch/send on a node that is using network storage, it will overwrite the files it is trying to copy, resulting in chaos. I would like to be able to have "Fetch if specified script evaluates to true" but that is not an option, it is just a check-box.

I get around this by having two identical steps, one which transfers files and one which does not, and run one or the other based on the "flagUseLocalWinWorkspace" variable. However, that means any time I change those steps I have to copy changes very carefully by hand.

Is this something which can be made better? Does anyone have a more clever way to work around it?

  • replies 3
  • views 35
  • stars 0
robinshen ADMIN ·

You may call grid.transferFiles programmatically. Its signature:

void transferFiles(GridNode fromNode, String srcDir, 
    		String filePatterns, GridNode toNode, String destDir, 
    		boolean compress, boolean failUponDuplication, 
    		String lockForReadPath, String lockForWritePath);

To mimic fetch input files in child step, fromNode should be specified as step.parent.node, and toNode should be specified as grid.localNode, and vise versa for send output files.

drdt ·

Oh, this would be fantastic. I should implement this in the Pre-Execute script, right? I am currently using Pre-Execute script to 'Clean workspace on current node' before the copy. What would be the call to do that programmatically? So I can do:

if (flagUseLocalWinWorkspace) {
clean-workspace-on-current-node
copy-files-from-parent-node-to-current-node
}

robinshen ADMIN ·

Yes that is the place to call it. To clean workspace programmatically:

import com.pmease.quickbuild.util.FileUtils;

FileUtils.cleanDir(configuration.workspaceDir);