Your browser was unable to load all of the resources. They may
have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.
We have a configuration that has twice this week failed the Fetch Input Files sending the workspace from a linux agent to a Windows agent. The QB step error is
java.io.EOFException: Unexpected end of ZLIB input stream
caused by: Unexpected end of ZLIB input stream
The step doing the fetch (windows) has an empty step log. The parent step (linux) just has a generic "unsatisfaction of conditions" exception. The quickbuild.log on the linux agent has this
2011-12-02 04:30:32,266 [1513218974@qtp-2003560079-1064] INFO com.pmease.quickbuild.grid.FileTransferServl
et - Serving files (base dir: /d01/QuickBuild/workspace3/root/lgcbuilds/Installers/dssdksetup/DS_5000_8_1_
0, file patterns: dssetup/DSSDK_5000_8_1_0/build/**,-dssetup/DSSDK_5000_8_1_0/build/module/bin/win32/**,-d
ssetup/DSSDK_5000_8_1_0/build/module/bin/win32vc6/**,-dssetup/DSSDK_5000_8_1_0/build/module/bin/win32vc9/*
*,-dssetup/DSSDK_5000_8_1_0/build/module/bin/win64vc9/**,-dssetup/DSSDK_5000_8_1_0/build/module/bin/linux/
**,-dssetup/DSSDK_5000_8_1_0/build/module/bin/linux64/**,-dssetup/DSSDK_5000_8_1_0/build/module/bin/linux6
4rh5/**,-dssetup/DSSDK_5000_8_1_0/build/module/bin/solaris/**,dssetup/DSSDK_5000_8_1_0/dependencies/**,-ds
setup/DSSDK_5000_8_1_0/dependencies/**/bin/win32vc6/**,-dssetup/DSSDK_5000_8_1_0/dependencies/**/bin/win32
vc9/**,-dssetup/DSSDK_5000_8_1_0/dependencies/**/bin/win64vc9/**,-dssetup/DSSDK_5000_8_1_0/dependencies/**
/bin/linux/**,-dssetup/DSSDK_5000_8_1_0/dependencies/**/bin/linux64/**,-dssetup/DSSDK_5000_8_1_0/dependenc
ies/**/bin/linux64rh5/**,-dssetup/DSSDK_5000_8_1_0/dependencies/**/bin/solaris/**)
2011-12-02 04:30:51,215 [1513218974@qtp-2003560079-1064] ERROR org.mortbay.log - /file_transfer
Problem creating TAR: request to write '65536' bytes exceeds size in header of '114913280' bytes for entr
y 'dssetup/DSSDK_5000_8_1_0/build/stage/linux64/DS-SDK_5000.8.1.0_linux64.tar'
We aren't seeing the error consistently (2 out of 15 builds since Monday). Could probably modify the pattern for the input files to exclude build/stage/linux64 since I'm sending to a Windows agent, but I'd like to understand the actual root cause of the error.
-
replies
5
-
views
2751
-
stars
0
-
The most probable reason is that while QB is packaging the files for transferring, another concurrent running is modifying it. For instance, if you set pattern as "**", and another step is modifying files in the same workspace, such error may occur.
The file in question is created in the build's workspace 2 sibling steps prior to the step that is trying to read the file to send to the agent. And that prior sibling step is simply copying the file from (it's an artifact of an Ivy module dependency, put into this build's workspace with the ivy:retrieve Ant task). There was a file handle threading issue we had seen in another place that was fixed in a later release so I was hoping our upgrade to 3.1.67 this past weekend would fix this issue too, but I am still seeing it. I've added some more '-' expressions to the input files to exclude dependencies/**/stage/platform so taht all platforms other than the platform of the agent being sent to are excluded, so that should protect us. But I'm still curious as to what is causing the problem. The step that is sending the files is a parallel repeating step, so it's sending the platform specific files to agents of each required platform in parallel. Could the threads from other platforms having an open file handle for reading the file cause this issue?
Generally you should make sure that while a step is traversing workspace directories or reading workspace files (input files option will use both), no other steps should be writing to the workspace. If you do have the requirement, please separate them into different folder explicitly to avoid directory/file concurrent modification issue, for instance, if you have below step structure:
composite step for windows:
step 1: reads file into workspace using ivy
step 2: transfers read files into another agent
composite step for linux:
step 1: reads file into workspace using ivy
step 2: transfers read files into another agent
If the composite steps are running concurrently, please make sure that you run a step prior to these composite steps to create two separate sub folder, for instance, "windows" and "linux" in the workspace, and then make sure that all windows steps are operating directories under "windows", and linux steps are operating against "linux" sub directory. Simply using file pattern to adhere to specific directory is not sufficient, so for windows step 2, please specify input setting like below:
source directory: windows
file pattern: **
And avoid of specifying file pattern as "windows/**"
We have a parent step called platforms, parallel composition, that runs on a node matching two attributes to variables (os.platform to masterStepPlatform and type to nodeType), within that composite step called platform, sequential composition, with a repeat parameter repeat.platform using the variable supportedPlatforms, which is a list of the same values out nodes would have in their os.platform attribute. This step matches up the os.platform attribute to the repeat.platform parameter, and the nodeType variable to the type attribute. The masterStepPlatforms will always be a subset of the supportedPlatforms, so one of the nodes running the platform step will be the same platform as the one running the parent platforms step. I believe it always uses the same node for those two steps. Will QuickBuild actually send the workspace back to itself in this scenario? That is the only way anything could be writing to the workspace that is being read to send files to all the other platforms. And it's only a problem for sending the dependencies/stage/linux64/ files to the win64 node, even though I have now excluded that folder from being sent to that node.
For the case of platform node being the same as platforms node, file copying will only happen if source directory and destination directory are specified differently in input file option. If files under "dependencies/stage/linux64" are touched as result of this, and if another node is requesting input files with empty source directory (means workspace itself) with pattern "dependencies/stage/win64/**", this EOF error will likely to happen as touched files will interfere with pattern match. Specifying source directory as "dependencies/stage/win64", with pattern "**" should be safe in this case.
