invalid XML character from REST response #1591

jgeorgeson · 1 decade ago

We're using v3.0.14 running on 64 bit RH4. We have an ant task to loop recursively thru the configs and search for a variable with a given name. This task has been working well for about a week. We just started seeing a bunch of configurations failing with socket exceptions (500, unexpected end of file) and noticed the variable searching task is now giving exceptions for invalid xml. I'm not asking for help debugging our Ant task, because that's been working fine. I'm asking for help debugging any potential issues with our server and/or it's database because this invalid response error seems extremely co-incidentally timed with the SocketException errors.

Action URL in getURLElement:/configurations?parent_id=14250
Sep 27, 2011 11:32:07 AM com.lgc.btlite.Qb3FindVariableTask getURLElement
SEVERE: Cannot read from QB server: http://buildmaster3.lgc.com:8080...
org.dom4j.DocumentException: Error on line 42 of document  : Character reference "&#0" is an invalid XML character. Nested exception: Character reference "&#0" is an invalid XML character.
        at org.dom4j.io.SAXReader.read(SAXReader.java:482)
        at org.dom4j.io.SAXReader.read(SAXReader.java:343)
        at com.lgc.btlite.Qb3FindVariableTask.getURLElement(Qb3FindVariableTask.java:122)

The call in our code is

	  protected static Element getURLElement(String actionURL)
	  {
	    Element retVal = null;
	    try {
	      System.out.print("\nAction URL in getURLElement:" + actionURL +"\n");
	      HttpURLConnection conn = getURLConnection(actionURL);
	      conn.setReadTimeout(CONNECTION_TIMEOUT);
	      conn.setDoInput(true);
	      conn.setDoOutput(false);
	      conn.setRequestMethod("GET");
	      InputStream is = conn.getInputStream();
	      retVal = _sax.read(is).getRootElement();                           // line 122
	      is.close();
	    } catch(Throwable t) {
	      _logger.log(Level.SEVERE,"Cannot read from QB server: " + QB_SERVER_URL + "...",t);
	    }
	    return(retVal);
	  }

The getURLConnection method appends the actionURL to the base URL for the rest API and establishes the HTTP connection. I put the URL into a broswer and it renders this

replies 5
views 3522
stars 0

jgeorgeson · 1 decade ago

Here is the full XML of the response, the offending character is in an <errorMessage> entity for an LDAP error. Can you elaborate on this error in this context? That may help us identify the SocketExceptions.

<?xml version="1.0" encoding="UTF-8"?>

<list>
  <com.pmease.quickbuild.model.Configuration>
    <id>14252</id>
    <parent>14250</parent>
    <queue>7</queue>
    <name>docs</name>
    <versionManagerDOM>
      <com.pmease.quickbuild.setting.configuration.version.UseSpecifiedVersion revision="0.0.0">
        <version>1.0.0</version>
      </com.pmease.quickbuild.setting.configuration.version.UseSpecifiedVersion>
    </versionManagerDOM>
    <runMode>SEQUENTIAL</runMode>
    <schedule class="com.pmease.quickbuild.taskschedule.schedule.CronSchedule">
      <paused>true</paused>
      <expression>0 0 4 * * ?</expression>
    </schedule>
    <statusDate>2011-08-12T07:16:37.000-05:00</statusDate>
    <pluginSettingDOMs/>
    <stepDOMs/>
    <repositoryDOMs/>
    <variables/>
    <notifications/>
    <promotions/>
  </com.pmease.quickbuild.model.Configuration>
  <com.pmease.quickbuild.model.Configuration>
    <id>14253</id>
    <parent>14250</parent>
    <queue>18</queue>
    <name>sandbox</name>
    <versionManagerDOM>
      <com.pmease.quickbuild.setting.configuration.version.UseSpecifiedVersion revision="0.0.0">
        <version>1.0.0</version>
      </com.pmease.quickbuild.setting.configuration.version.UseSpecifiedVersion>
    </versionManagerDOM>
    <runMode>SEQUENTIAL</runMode>
    <schedule class="com.pmease.quickbuild.taskschedule.schedule.NoSchedule">
      <paused>true</paused>
    </schedule>
    <statusDate>2011-09-27T04:02:51.415-05:00</statusDate>
    <errorMessage>javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C090334, comment: AcceptSecurityContext error, data 775, vece]
caused by: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C090334, comment: AcceptSecurityContext error, data 775, vece]</errorMessage>
    <pluginSettingDOMs/>
    <stepDOMs/>
    <repositoryDOMs/>
    <variables/>
    <notifications/>
    <promotions/>
  </com.pmease.quickbuild.model.Configuration>
  <com.pmease.quickbuild.model.Configuration>
    <id>14251</id>
    <parent>14250</parent>
    <queue>6</queue>
    <name>rsync</name>
    <versionManagerDOM>
      <com.pmease.quickbuild.setting.configuration.version.UseSpecifiedVersion revision="0.0.0">
        <version>1.0.0</version>
      </com.pmease.quickbuild.setting.configuration.version.UseSpecifiedVersion>
    </versionManagerDOM>
    <runMode>SEQUENTIAL</runMode>
    <schedule class="com.pmease.quickbuild.taskschedule.schedule.NoSchedule">
      <paused>true</paused>
    </schedule>
    <statusDate>2011-09-27T05:21:01.245-05:00</statusDate>
    <pluginSettingDOMs/>
    <stepDOMs/>
    <repositoryDOMs/>
    <variables/>
    <notifications/>
    <promotions/>
  </com.pmease.quickbuild.model.Configuration>
</list>

jgeorgeson · 1 decade ago

So looking in the dashboard, the errorMessage body is displayed as a clearable configuration error. It is present for 4 of our configurations. We have no way of telling (other than by the luck of having run the ant task yesterday and not seeing this issue then) when these occurred, or any context for the error. Any insight on what happens between the scheduler (or a user) triggering a build of a config and the master step starting that would cause this error would be appreciated.

robinshen ADMIN · 1 decade ago

During sending notification stage of a build, QB may query LDAP server to get user email when it can not find some committers in QB user database. If configured LDAP binding user can not authenticate to LDAP server, this error will occur. You may check details of the error by clicking the log icon besides the error message title.

I am curious though: it seems that the character sequence "&#0" prevents the XML response to be returned correctly via RESTful service, but I paste the same LDAP error message into my configuration, and RESTful service can return the XML correctly. Also you paste a XML snippet which also seems fine with that character sequence.

jgeorgeson · 1 decade ago

Ok. It must either be something funky configured on these specific configurations then as we would be having some serious widespread issues if that bind account couldn't authenticate. I obtained the XML snippet by loading the non RESTful URL in the browser and then hitting ctrl-U to see the page source. So this doesn't appear to be related to our SocketExceptions after all. We found that our VM infrastructure had put the majority of our VMs on one server and overloading the resources. We've rebalanced the load so our builds tonight should hopefully finish clean. I'll poke around for any odd settings on those configs with the LDAP config error. Regarding the log icon, we pretty much have to catch the config errors as they happen to see them in the system log. These config errors were no more than 30 hours old and already pushed out of the system log. I even logged into the server machine and checked the backup console.log and quickbuild.log files.

robinshen ADMIN · 1 decade ago

You may edit "conf/log4j.properties" to preserve more entries in system log, for instance:

log4j.appender.file.MaxFileSize=2500KB
log4j.appender.file.MaxBackupIndex=10

Configuration XML output is only available via RESTful API. So I am wondering which non-RESTful url you are using so that you can use Ctrl-U to get the XML.