Testing a RELAX NG Schema Using XSpec

Also, a Possibly Unexpected Use of x:call

You might know that XSpec can test a Schematron schema using dedicated elements like <x:expect-assert> and <x:expect-valid>. Schematron is only one of several schema languages in the XML area. Another schema language is RELAX NG, sometimes written as RelaxNG or RNG. Schematron often complements other schema languages, so you might have both a RELAX NG schema and a Schematron schema for the same set of XML documents.

XSpec does not have dedicated features for testing a RELAX NG schema. Even so, you can test a RELAX NG schema with XSpec, if you get help from the validation capabilities in BaseX and Jing software. This topic illustrates how. Along the way, we will also see a possibly unexpected use of the XSpec <x:call> element, to call a function you aren’t interested in testing.

A Sample RELAX NG Schema

Before we look at the sample RELAX NG schema for this topic, let’s look at a sample XML document that conforms to the schema. The XML document describes a palette of colors, where each color has a name and an optional triplet of RGB (red, green, blue) values as decimal numbers.

Example 1. Valid XML Document, document-valid.xml

<palette>
  <color red="0" green="138" blue="194">
    <name>Medium blue</name>
  </color>
  <color red="255" green="109" blue="109">
    <name>Coral</name>
  </color>
  <color>
    <name>Pale yellow</name>
  </color>
</palette>

RELAX NG has two syntaxes, an XML syntax and a more compact, non-XML syntax. Here is a schema that uses the compact syntax.

Example 2. RELAX NG Schema, palette-schema.rnc

element palette {
  element color {
    (attribute red { xsd:nonNegativeInteger },
     attribute green { xsd:nonNegativeInteger },
     attribute blue { xsd:nonNegativeInteger })?,
    element name { text }
  }+
}

The schema defines an element named <palette> that contains one or more child elements named <color>. Each <color> element stores an optional RGB triplet for the color as individual attributes and stores the color name in a child element called <name>.

We can validate the sample XML document against this schema using a RELAX NG processor like Jing or an IDE like Oxygen XML Editor (which includes Jing as part of the product). Either way, the validator says the document is valid. If we insert an extra <name> element, or if we remove one of the three attributes of <color> while leaving the other two in place, then the validator produces an error message that tells us the document is invalid.

What Does Testing a Schema Mean?

To test a schema, we will validate documents against the schema and check that the validation results are what we expect. If we expect a document to be valid, the validation results should say so. If we expect a document to be invalid, the validation results should list at least one validation error. We can optionally check for specific validation errors.

How to Validate the Document in XSpec

BaseX is an XQuery processor that includes its own functions for validating documents against a RELAX NG schema. In particular, the validate:rng-report function validates a document against a schema and returns validation results in XML format. Validation errors do not halt processing. Within an XSpec scenario, we can call the validate:rng-report function like this:

Example 3. Calling validate:rng-report in XSpec

<x:call function="validate:rng-report">
  <x:param name="input" href="document-valid.xml"/>
  <x:param name="schema"
    select="'palette-schema.rnc' => resolve-uri($x:xspec-uri)"/>
  <x:param name="compact" select="true()"/>
</x:call>

The part that says 'palette-schema.rnc' => resolve-uri($x:xspec-uri) assumes the schema is in the same directory as the XSpec file. If the schema were in a sibling directory named schemas, for instance, the expression would say '../schemas/palette-schema.rnc' => resolve-uri($x:xspec-uri) instead.

The compact parameter is true because our schema uses the compact syntax for RELAX NG. If the schema used the XML syntax, we would set this parameter value to false() or omit the third <x:param> element.

The Test Target

The validate:rng-report function is built into BaseX. To access this function, we need to run the test with BaseX, but we don’t need to write any XQuery code of our own. The XSpec test file needs to point to some XQuery module as a test target, and it can be a nearly empty XQuery module like this:

Example 4. Trivial XQuery Module

xquery version "3.1";
module namespace noop = "urn:x-xspectacles:functions:noop";

In XSpec, the <description> element’s start tag looks like the following, assuming the XQuery module file is named no-op.xqm:

Example 5. XSpec Top Element Tag

<x:description
  xmlns:validate="http://basex.org/modules/validate"
  xmlns:x="http://www.jenitennison.com/xslt/xspec"
  xmlns:xv="urn:x-xspectacles:xspec:variables"
  query="urn:x-xspectacles:functions:noop"
  query-at="no-op.xqm">

The xmlns:validate="http://basex.org/modules/validate" namespace declaration avoids errors in XSpec, although BaseX itself already knows that the prefix validate is bound to that particular namespace URI. The way we knew what URI to use in our namespace declaration is by reading the BaseX documentation for validation functions.

RELAX NG Validation Results from BaseX

For a schema-valid document, the result of calling validate:rng-report is simple:

Example 6. Result for Valid Document

<report>
  <status>valid</status>
</report>

For a schema-invalid document, the result of calling validate:rng-report includes a status and messages. Here is a sample document that does not conform to our RELAX NG schema because the first color has an incomplete set of RGB values and the last color has two names:

Example 7. Invalid Document, document-invalid.xml

<palette>
  <color green="138" blue="194">
    <name>Medium blue</name>
  </color>
  <color red="255" green="109" blue="109">
    <name>Coral</name>
  </color>
  <color>
    <name>Pale yellow</name>
    <name>Light yellow</name>
  </color>
</palette>

The corresponding result from validate:rng-report looks like this:

Example 8. Validation Result for document-invalid.xml

<report>
  <status>invalid</status>
  <message level="Error"
    line="2"
    column="33"
    url="file:///C:/.../document-invalid.xml"
    >element "color" missing required attribute "red"</message>
  <message level="Error"
    line="10"
    column="11"
    url="file:///C:/.../document-invalid.xml"
    >element "name" not allowed here; expected the element end-tag</message>
</report>

How to Verify RELAX NG Validation Results

The simplest verification approach is to check the valid vs. invalid status. Checking for specific validation errors requires a little more work. We’ll look at both kinds of verifications.

Valid vs. Invalid Status

For a document we expect to be valid, we can verify its validity in XSpec using the following <x:expect> element.

<x:expect label="Valid"
  test="$x:result/status/text()">valid</x:expect>

For a document we expect to be invalid, we can verify its invalidity in an analogous way.

<x:expect label="Not valid"
  test="$x:result/status/text()">invalid</x:expect>

Tip: If an XSpec test suite validates many documents, each of these <x:expect> elements is a good candidate for reuse; see Code Reuse in XSpec.

If verification fails and we want to see the full validation results in the XSpec test report to aid failure investigation, we can temporarily add the following <x:expect> element and rerun the test.

<x:expect label="Show full validation results"/>

Verifying Validation Errors

For a document we expect to be invalid, we can go further than merely checking that the status is invalid. We can check the full or partial text of error messages that we expect.

First, it is convenient to create some XSpec variables that store the following:

• A sequence of strings representing the text of all the validation errors
• A string with the full or partial text of each expected error message

Example 9. XSpec Variables

<x:variable name="xv:actual-error-strings" as="xs:string+"
  select="$x:result/message/string()"/>
<x:variable name="xv:expected-string1" as="xs:string"
  >element "color" missing required attribute "red"</x:variable>
<x:variable name="xv:expected-string2" as="xs:string"
  >element "name" not allowed here; expected the element end-tag</x:variable>
<x:variable name="xv:expected-string2-partial" as="xs:string"
  >element "name" not allowed here;</x:variable>

Now, we can check that $xv:expected-string1 is a member of the sequence $xv:actual-error-strings by using the = operator. (When one operand is a single string and the other is a sequence of multiple strings, the = operator returns true if the single string is a member of the sequence.) While we could instead verify that the first message has certain text and the second message has certain other text, we prefer not to depend on Jing and BaseX returning messages in a particular order.

Example 10. Verify that Result Includes a Particular Validation Error

<x:expect label="Check for validation error 1"
  test="$xv:expected-string1 = $xv:actual-error-strings"/>

We can do the same for $xv:expected-string2.

Alternatively, we can check that $xv:actual-error-strings has a member string that contains $xv:expected-string2-partial as a substring.

Example 11. Verify a Substring Within Validation Results

<x:expect label="Check for partial validation error 2"
  test="some $s in $xv:actual-error-strings satisfies
  contains($s, $xv:expected-string2-partial)"/>

Running the XSpec Tests with BaseX

The validate:rng-report function is specific to BaseX, so we use BaseX to run our XSpec tests. In the XSpec wiki, Run an XSpec test for XQuery with BaseX standalone provides general instructions. In addition, a test that uses validate:rng-report requires the classpath to include an additional library named Jing, when BaseX is running the test. One way to get Jing into the classpath when BaseX runs the test is to include the Jing location in the same argument that provides the BaseX location:

-p basex-jar="%BASEX_HOME%\lib\jing-20220510.jar;%BASEX_HOME%\BaseX.jar"

Without Jing, BaseX returns this error:

[validate:not-found] RelaxNG validation is not available.

My commands on Windows, after I navigate to the directory containing my RELAX NG schema, XQuery, and XSpec files (not the directory containing the XSpec implementation), look like the following.

set XSPEC_HOME_URI=file:///C:/.../xspec/
set BASEX_HOME=C:\...\BaseX111
set XMLCALABASH_JAR=C:\...\xmlcalabash-1.5.7-120\xmlcalabash-1.5.7-120.jar

java -jar "%XMLCALABASH_JAR%" -i source=./relaxng-test.xspec -p xspec-home=%XSPEC_HOME_URI% -p basex-jar="%BASEX_HOME%\lib\jing-20220510.jar;%BASEX_HOME%\BaseX.jar"  -o result=./relaxng-test-result.html %XSPEC_HOME_URI%src\harnesses\basex\basex-standalone-xquery-harness.xproc

I abbreviated dependency locations using ..., but you can see the data format. Fill in the paths that point to the dependency locations on your own system.

You might see the following output:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

If the HTML report was generated, then the process worked and you can ignore that output.

FYI: BaseX Has Its Own Testing Functions

BaseX has its own testing capabilities, making it possible for you to test a RELAX NG schema using BaseX without XSpec. If you already use BaseX XQuery-based tests for XQuery code, you’ll probably want to continue in that vein for testing your RELAX NG schema. You might write an XQuery test module that looks like this:

Example 12. RELAX NG Test Using BaseX Testing Capabilities

xquery version "3.1";
module namespace mytest = "urn:x-xspectacles:basex:relaxng";

declare variable $mytest:rng-schema as xs:string :=
  'palette-schema.rnc';

declare %unit:test function mytest:valid-doc() {
  let $s as document-node() := doc('document-valid.xml')
  let $val as element(report) :=
    validate:rng-report($s, $mytest:rng-schema, true())
  return unit:assert-equals($val/status/string(), 'valid', $val)
};

declare %unit:test function mytest:invalid-doc() {
  let $s as document-node() := doc('document-invalid.xml')
  let $val as element(report) :=
    validate:rng-report($s, $mytest:rng-schema, true())
  let $actual-error-strings as xs:string+ :=
    $val/message/string()
  let $expected-string1 as xs:string :=
    'element "color" missing required attribute "red"'
  let $expected-string2 as xs:string :=
    'element "name" not allowed here; expected the element end-tag'
  let $expected-string2-partial as xs:string :=
    'element "name" not allowed here'
  return (
    unit:assert-equals($val/status/string(), 'invalid', $val),
    unit:assert($actual-error-strings = $expected-string1, $val),
    unit:assert(some $s in $actual-error-strings satisfies
      contains($s, $expected-string2-partial), $val)
  )
};

On the other hand, if you already use XSpec and aren’t interested in switching to a different kind of test code just for RELAX NG testing, I hope you found this topic useful.

Key Takeaways

• The BaseX XQuery processor has a function that validates a document against a RELAX NG schema and returns validation results. You can use this function in XSpec to test your schema.
• The XSpec code for testing a RELAX NG schema in this way differs from code in an XSpec test for a Schematron schema, but the underlying idea is the same: validate documents and verify that the validation results are what you expect.
• The technique in this topic (and also in Testing Invisible XML Using XSpec, by the way) uses a function to unlock testing capabilities. The function isn’t really the thing you’re testing; that is, testing the behavior of a BaseX function or a standard XPath function is not your goal. Instead, the point is that calling the function with certain parameters produces a result that sheds light on what you’re really trying to test. In this case, calling validate:rng-report with certain parameters sheds light on your schema. Because XSpec has syntax for calling the function and verifying characteristics of the function’s result, XSpec can stretch beyond its XSLT/XQuery/Schematron zone and become a viable vocabulary for testing RELAX NG.

Code is downloadable from https://github.com/galtm/xspectacles/ on GitHub, in the src/relaxng folder.