This has now been fixed in 7.0.19:
https://build.pmease.com/build/3860

Thank you so much. QB customer support is the best!!!

Hmmm, looks like we may have a regression.

We are now seeing errors when providing a commit hash. This used to work fine before upgrade to 7.0.19. We are also seeing the same error when providing a branch name, e.g. origin/feature-xyz. Here's an exception when we provide a commit hash (on a branch that's not master):

2017-07-28 11:36:23,354 [pool-1-thread-1401] INFO  com.pmease.quickbuild.DefaultBuildEngine - Processing build request (configuration:gaikai/ebuilder/core/core taikai, request id:39cc5610-643b-4894-afc0-d3bea1e06763)
2017-07-28 11:36:23,363 [pool-1-thread-1401] INFO  com.pmease.quickbuild.DefaultBuildEngine - Checking build condition on node (address: qba23:8811, ip: 10.0.31.164)...
2017-07-28 11:36:23,963 [pool-1-thread-1401] ERROR com.pmease.quickbuild.DefaultBuildEngine - Error processing build request.
    java.lang.RuntimeException: Error executing check condition job.
        at com.pmease.quickbuild.CheckConditionTask.reduce(CheckConditionTask.java:39)
        at com.pmease.quickbuild.CheckConditionTask.reduce(CheckConditionTask.java:16)
        at com.pmease.quickbuild.grid.GridTaskFuture.get(GridTaskFuture.java:116)
        at com.pmease.quickbuild.DefaultBuildEngine.process(DefaultBuildEngine.java:393)
        at com.pmease.quickbuild.DefaultBuildEngine.access$000(DefaultBuildEngine.java:141)
        at com.pmease.quickbuild.DefaultBuildEngine$2.run(DefaultBuildEngine.java:1220)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
    Caused by: java.lang.RuntimeException: Failed to evaluate below expression in configuration 'gaikai/ebuilder/core/core taikai':
    groovy:
    cmd='install_github_ssh_keys'
    logger.info('snapshot taking on: ' + node.getHostName() + ' exec: ' + cmd)
    assert util.execute(cmd) == 0, "Could not install keys"
    for (repo in configuration.getReferencedRepositories()) {
     repo.takeSnapshot();
    }
        at com.pmease.quickbuild.util.ExceptionUtils.wrapException(ExceptionUtils.java:87)
        at com.pmease.quickbuild.DefaultScriptEngine.evaluate(DefaultScriptEngine.java:321)
        at com.pmease.quickbuild.DefaultScriptEngine.evaluate(DefaultScriptEngine.java:74)
        at com.pmease.quickbuild.setting.configuration.snapshot.ScriptSnapshotTaking.takeSnapshot(ScriptSnapshotTaking.java:39)
        at com.pmease.quickbuild.model.Configuration.takeSnapshot(Configuration.java:1870)
        at com.pmease.quickbuild.CheckConditionJob.execute(CheckConditionJob.java:35)
        at com.pmease.quickbuild.grid.GridJob.run(GridJob.java:106)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        ... 3 more
    Caused by: com.pmease.quickbuild.QuickbuildException: Failed to run command: git log -1 --pretty=format:%H 41c5c510f92381c3589b5a1179f7f18bf47baab1 
    Command return code: 128
    Command error output: fatal: bad object 41c5c510f92381c3589b5a1179f7f18bf47baab1
        at com.pmease.quickbuild.plugin.basis.BasisPlugin$30.evaluate(BasisPlugin.java:380)
        at com.pmease.quickbuild.DefaultScriptEngine.evaluate(DefaultScriptEngine.java:305)
        ... 10 more

More information:

The problem seems to be only with branches or commits on branches that are NEWER than the current master branch. The same experiment with commits and branches that are OLDER than the current master branch succeeds. This looks like QB may be discarding commits made on other branches which took place after the current master branch was pushed.

I tried but it works at my side. What I am doing:

1. create a test configuration and define a git repository, with branch left empty (defaults to master). Commit/tag field is also left empty.
2. add a checkout step against the git repository and run a build. The build succeeds.
3. now create a new branch at git side and make a new commit.
4. edit the git repository setting at QB side, and specify commit/tag as hash of the new commit. Run the build again and it also succeeds.

Is there any difference in your configuration setting?

Yes. I am doing this:

1. I'm on a master branch. Start a build. It succeeds.
2. Create a bogus file. git add; git commit; git push;
3. Now go back by a commit. git checkout <hash of the previous commit>
4. Create some other file. git add; git commit; (it will create an orphaned commit)
5. Create a branch for this orphaned commit and push. git branch bogus-branch; git push -u origin bogus-branch
6. git checkout bogus-branch (make sure the commit is on the bogus branch)
7. git push (should say it's up-to-date, I am just making sure it got pushed)
8. Now try a build with the commit hash equal to the HEAD of the bogus branch. Build fails:

2017-07-28 18:43:54 - Failed to run command: git log -1 --pretty=format:%H bddd93011bc00bfc0a58898a522d202e0b661782 
Command return code: 128
Command error output: fatal: bad object bddd93011bc00bfc0a58898a522d202e0b661782

I followed your procedure exactly and it still works at my side. Can you please test with a new installation in case some other setting affected the result? Also make sure you are working with QB 7.0.20

Interesting. I've upgraded to 7.0.21 with the same result, failure. I did not try fresh installation. I trust your results. We have a large system which I cannot rebuild from scratch. We have to fix it in this system. So this is turning into a question of how to debug this.

I temporarily removed the QB repository check out step with a bash script like this (url and repo name changed):

git clone git@local_github.com:org/some_repo
cd some_repo
ver=${vars.getValue('some_repo_version')}
git log -1 --pretty=format:%H $ver
git checkout $ver

This works. In particular, 'git log -1 --pretty=format:%H' returns correct commit hash. After that, the checkout works, and the project builds fine.

I was suspecting our script in 'Snapshot Taking' in Advanced Settings. We use it do distribute github keys:

groovy:
cmd='install_github_ssh_keys'
logger.info('snapshot taking on: ' + node.getHostName() + '  exec: ' + cmd)
assert util.execute(cmd) == 0, "Could not install keys"
for (repo in configuration.getReferencedRepositories()) {
    repo.takeSnapshot();
}

You can see this script in the log of my first post. I temporarily removed it, but it made no difference.

So, it looks like the problem is somewhere between QB taking repository snapshots and checking out the desired version. I am not sure how to debug this. Would you have any suggestions?

At snapshot taking stage, QB will clone the repository first (if it does not exist yet, or has been cleaned), and then run below command to make sure attached commits of all branches and tags are retrieved:
git fetch --all --tags

Can you please turn on debug logging via "conf/log4j.properties" on the machine running the master step of this failing build (snapshot taking stage will happen on the machine running master step) by adding below line:
log4j.logger.com.pmease.quickbuild.execution = DEBUG

Then try to reproduce the issue and check server log or agent log (if master step runs on agent) to see if it contains the expected "git fetch" command.

I don't see the 'git fetch' command but I also don't see any DEBUG lines despite settings in log4j.properties. I pegged the build to agent qba29. Here are top lines in the log4j.properties on that agent:

# accept log levels: TRACE, DEBUG, INFO, WARN, ERROR
log4j.logger.com.pmease.quickbuild=DEBUG
log4j.logger.com.pmease.quickbuild.execution=DEBUG

log4j.rootLogger=WARN,file,console

[...]

I restarted the agent after making the changes. Not sure if it's necessary.

The build fails every time, so it was easy to reproduce. Here's the log:

2017-08-01 11:54:58,114 [pool-1-thread-151] INFO  com.pmease.quickbuild.DefaultBuildEngine - Processing build request (configuration:xxxxxx/xxxxxx/xxxxxx/xxxxxx one-step-build, request id:b5f42fa5-540d-4bfd-86a0-463a50b656f5)
2017-08-01 11:54:58,137 [pool-1-thread-151] INFO  com.pmease.quickbuild.DefaultBuildEngine - Checking build condition on node (address: qba29:8811, ip: 10.0.31.253)...
2017-08-01 11:55:00,681 [pool-1-thread-151] ERROR com.pmease.quickbuild.DefaultBuildEngine - Error processing build request.
    java.lang.RuntimeException: Error executing check condition job.
        at com.pmease.quickbuild.CheckConditionTask.reduce(CheckConditionTask.java:39)
        at com.pmease.quickbuild.CheckConditionTask.reduce(CheckConditionTask.java:16)
        at com.pmease.quickbuild.grid.GridTaskFuture.get(GridTaskFuture.java:116)
        at com.pmease.quickbuild.DefaultBuildEngine.process(DefaultBuildEngine.java:393)
        at com.pmease.quickbuild.DefaultBuildEngine.access$000(DefaultBuildEngine.java:141)
        at com.pmease.quickbuild.DefaultBuildEngine$2.run(DefaultBuildEngine.java:1220)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
    Caused by: java.lang.RuntimeException: Failed to evaluate below expression in configuration 'xxxxxx/xxxxxx/xxxxxx/xxxxxx one-step-build':
    groovy:
    cmd='install_github_ssh_keys'
    logger.info('snapshot taking on: ' + node.getHostName() + ' exec: ' + cmd)
    assert util.execute(cmd) == 0, "Could not install keys"
    for (repo in configuration.getReferencedRepositories()) {
     repo.takeSnapshot();
    }
        at com.pmease.quickbuild.util.ExceptionUtils.wrapException(ExceptionUtils.java:87)
        at com.pmease.quickbuild.DefaultScriptEngine.evaluate(DefaultScriptEngine.java:321)
        at com.pmease.quickbuild.DefaultScriptEngine.evaluate(DefaultScriptEngine.java:74)
        at com.pmease.quickbuild.setting.configuration.snapshot.ScriptSnapshotTaking.takeSnapshot(ScriptSnapshotTaking.java:39)
        at com.pmease.quickbuild.model.Configuration.takeSnapshot(Configuration.java:1870)
        at com.pmease.quickbuild.CheckConditionJob.execute(CheckConditionJob.java:35)
        at com.pmease.quickbuild.grid.GridJob.run(GridJob.java:106)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more
    Caused by: com.pmease.quickbuild.QuickbuildException: Failed to run command: git log -1 --pretty=format:%H bddd93011bc00bfc0a58898a522d202e0b661782 
    Command return code: 128
    Command error output: fatal: bad object bddd93011bc00bfc0a58898a522d202e0b661782
        at com.pmease.quickbuild.plugin.basis.BasisPlugin$30.evaluate(BasisPlugin.java:380)
        at com.pmease.quickbuild.DefaultScriptEngine.evaluate(DefaultScriptEngine.java:305)
        ... 10 more

Sorry, I was looking in the wrong place. Yes, there is DEBUG trace:

2017-08-01 18:54:59,935 [pool-1-thread-1] INFO com.pmease.quickbuild.repositorysupport.Repository - Taking snapshot of repository 'releng one-step-build'...
2017-08-01 18:54:59,935 [pool-1-thread-1] INFO com.pmease.quickbuild.repositorysupport.Repository - Determining head revision for repository: releng one-step-build
2017-08-01 18:54:59,950 [pool-1-thread-1] DEBUG com.pmease.quickbuild.plugin.scm.helper.ScmCli - sync working directory [/opt/qb/buildagent/workspace/1541/one-step-build]...
2017-08-01 18:54:59,951 [pool-1-thread-1] DEBUG com.pmease.quickbuild.plugin.scm.helper.AbstractRepository - Detect whether repository settings have been changed ...
2017-08-01 18:54:59,952 [pool-1-thread-1] DEBUG com.pmease.quickbuild.plugin.scm.helper.AbstractRepository - Loading scm rc file ...
2017-08-01 18:55:00,049 [pool-1-thread-1] DEBUG com.pmease.quickbuild.execution.Commandline - Executing command: git remote prune origin
2017-08-01 18:55:00,049 [pool-1-thread-1] DEBUG com.pmease.quickbuild.execution.Commandline - Command working directory: /opt/qb/buildagent/workspace/1541/one-step-build
2017-08-01 18:55:00,297 [pool-1-thread-1] DEBUG com.pmease.quickbuild.execution.Commandline - Executing command: git fetch --all --tags
2017-08-01 18:55:00,297 [pool-1-thread-1] DEBUG com.pmease.quickbuild.execution.Commandline - Command working directory: /opt/qb/buildagent/workspace/1541/one-step-build
2017-08-01 18:55:00,299 [pool-1-thread-4] DEBUG com.pmease.quickbuild.execution.LineConsumer - Fetching origin
2017-08-01 18:55:00,447 [pool-1-thread-1] DEBUG com.pmease.quickbuild.execution.Commandline - Executing command: git log -1 --pretty=format:%H bddd93011bc00bfc0a58898a522d202e0b661782
2017-08-01 18:55:00,447 [pool-1-thread-1] DEBUG com.pmease.quickbuild.execution.Commandline - Command working directory: /opt/qb/buildagent/workspace/1541/one-step-build
2017-08-01 18:55:00,451 [pool-1-thread-4] ERROR com.pmease.quickbuild.execution.LineConsumer - fatal: bad object bddd93011bc00bfc0a58898a522d202e0b661782

Hmm, looking at the DEBUG log, it seems like QB tries 'git log' in the context of working directory 'workspace/1541'. Looking at the agent, that directory exists and the one-step-build repository exists in that directory,too. BUT... this is the OLD repository from a build some four weeks ago. Running 'git log' in that repo fails because that repository was not refreshed. I don't know how this is supposed to work in QB but if I remove this directory, then the subsequent build succeeds!

It looks like QB, for some reason, does not refresh old repositories.

Googling a little, I found out that git fetch --tags does not fetch the objects for git versions prior to 1.9. We are running git 1.8.3.1.

Indeed, running git fetch --all --tags in that old repository does not retrieve the objects, so 'git log' keeps failing. If I run git fetch (without --all --tags) then it fetches the objects and 'git log' succeeds!

I believe this is the problem.

The command being executed exists in agent log. What you are posting is server log. Agent log can be accessed from grid node details in grid nodes page (or can be opened from terminal in <build agent>/logs/quickbuild.log)

Yes, I found it eventually, see my subsequent posts. I also think I found the problem. QB runs 'git fetch --all --tags' but this command will not retrieve objects for git versions less than 1.9. We are running 1.8.3.1 (centos 7) which explains why the builds fail.

Thanks for the finding. I'd suggest to upgrade to git 1.9, as merely running "git fetch" will not retrieve commits attaching to tag or non-current branch. So we added "-all -tags" to make sure all possible commits are retrieved to avoid the "bad object" issue.