Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

need to decide at run-time whether or not to copy files to local #4449

drdt ·

I have a puzzle. For a majority of our builds, we use network storage for our build workspaces, so all of our agents look in a network location: \nas-storage\qbdir\workspaces\servername\root\config\path. This way, multiple agents can share files via the workspace.

However, some operations are so big, it is faster to copy the source files to a local disk, do the build, and copy the results back. We accomplish this via the 'Fetch Input Files' and 'Send Output Files' options under Advanced settings. Our build scripts are clever enough to do the right thing once the files are there (using the local directory instead of the workspace) depending on a variable "flagUseLocalWinWorkspace".

My problem is that, I cannot programmatically choose whether or not to fetch/send the files. And, if I use fetch/send on a node that is using network storage, it will overwrite the files it is trying to copy, resulting in chaos. I would like to be able to have "Fetch if specified script evaluates to true" but that is not an option, it is just a check-box.

I get around this by having two identical steps, one which transfers files and one which does not, and run one or the other based on the "flagUseLocalWinWorkspace" variable. However, that means any time I change those steps I have to copy changes very carefully by hand.

Is this something which can be made better? Does anyone have a more clever way to work around it?

  • replies 10
  • views 516
  • stars 0
robinshen ADMIN ·

You may call grid.transferFiles programmatically. Its signature:

void transferFiles(GridNode fromNode, String srcDir, 
    		String filePatterns, GridNode toNode, String destDir, 
    		boolean compress, boolean failUponDuplication, 
    		String lockForReadPath, String lockForWritePath);

To mimic fetch input files in child step, fromNode should be specified as step.parent.node, and toNode should be specified as grid.localNode, and vise versa for send output files.

drdt ·

Oh, this would be fantastic. I should implement this in the Pre-Execute script, right? I am currently using Pre-Execute script to 'Clean workspace on current node' before the copy. What would be the call to do that programmatically? So I can do:

if (flagUseLocalWinWorkspace) {
clean-workspace-on-current-node
copy-files-from-parent-node-to-current-node
}

robinshen ADMIN ·

Yes that is the place to call it. To clean workspace programmatically:

import com.pmease.quickbuild.util.FileUtils;

FileUtils.cleanDir(configuration.workspaceDir);
drdt ·

I am trying to do this, but I think there is something I don't know about this function:

// fetch input files
grid.transferFiles( 
  step.parent.node, "dev", "**,-built/**,-kits/**", // remote files to copy in
  grid.localNode, "dev",                            // local folder
  false, false, "", ""                              // additional flags
);

I am getting this error. Do I need to use absolute path?

HTTP ERROR 500 com.pmease.quickbuild.QuickbuildException: Can not find source directory for file transferring: dev
URI:	/file_transfer
STATUS:	500
MESSAGE:	com.pmease.quickbuild.QuickbuildException: Can not find source directory for file transferring: dev
SERVLET:	com.pmease.quickbuild.grid.FileTransferServlet-2cac4107
CAUSED BY:	com.pmease.quickbuild.QuickbuildException: Can not find source directory for file transferring: dev
drdt ·

Actually I got past that; I was right, I needed the absolute path. Now I am getting this error:

13:25:05,079 ERROR - Step 'master>2...' is failed: id to load is required for loading

I don't see a place to enter any kind of ID. What am I missing?

parentDir = sprintf( "%s/dev", step.parent.node.getWorkspaceDir().getPath() );
localDir = sprintf( "%s/dev", grid.localNode.getWorkspaceDir().getPath() );
filesToCopy = "**,-built/**,-kits/**";
grid.transferFiles( 
  step.parent.node, parentDir,              // remote folder
  filesToCopy,                              // copy wildcards
  grid.localNode, localDir,                 // local folder
  false, false, "", ""                      // additional flags
);
robinshen ADMIN ·

Please try below script:

groovy:

def parentDir = sprintf( "%s/dev", step.parent.node.getWorkspaceDir(configuration.id));
def localDir = sprintf( "%s/dev", grid.localNode.getWorkspaceDir.absolutePath);
filesToCopy = "**,-built/**,-kits/**";
grid.transferFiles( 
  step.parent.node, parentDir,
  filesToCopy, 
  grid.localNode, localDir,                            
  false, false, null, null                              
);
drdt ·

Thanks, this worked:

parentDir = sprintf( "%s/dev", step.parent.node.getWorkspaceDir( configuration.id) );
localDir = sprintf( "%s/dev", grid.localNode.getWorkspaceDir( configuration.id ) );

I guess I could have had localDir be just configuation.getWorkspace()?

robinshen ADMIN ·

It expects a string, so use configuration.getWorkspace().getAbsolutePath() instead

drdt ·

Strangely, it works fine without the absolute path. This is my final script:

groovy:
  import java.io.File;
  import org.apache.commons.io.FileUtils;

  // if building on a windows machine, using a local workspace, 
  //   copy source files from the network drive
  if (vars.get( "sysIsWindowsNode" ).asBoolean() && 
      vars.get( "flagUseLocalWinWorkspace" ).asBoolean()) {

    // path to working directory is different on local node and on server
	parentDir = sprintf( "%s/dev", step.parent.node.getWorkspaceDir( configuration.id ) );
	localDir = sprintf( "%s/dev", configuration.workspaceDir );

    // make sure we have a clean working area
	oLocalDir = new File( localDir );
	if (oLocalDir.exists()) FileUtils.cleanDirectory( oLocalDir );
	
	// fetch input files
	filesToCopy = "**,-built/**,-kits/**";
	grid.transferFiles( 
	  step.parent.node, parentDir,              // remote folder
	  filesToCopy,                              // copy wildcards
	  grid.localNode, localDir,                 // local folder
	  false, false, "", ""                      // additional flags
	);	
  }
robinshen ADMIN ·

Groovy calls toString when calling springf which essentially calls getAbsolutePath()