Lexical Preservation Format Reference

Lexical Preservation Format Reference

The underpinning XML Compare comparator only understands three types of XML node: element, attribute, and text nodes. Therefore, all other types of node need to be converted into these types of node if they are to be retained. In particular, processing instruction and comment nodes need to be converted if they are to be retained.

In addition to the XML nodes within a document, there are other types of information that you might need to retain, such as the document's DOCTYPE, CDATA Sections, and entity references. This information is frequently available from an XML parser via its SAX parsing event interface, and thus can be converted into XML nodes that the underpinning comparator uses.

In general, the retained information is stored in element nodes that are added to one of these namespaces.

• xmlns:pi='http://www.deltaxml.com/ns/processing-instructions' — for retaining processing instructions (PIs), where the element's name and content mirror the PI's tag and content respectively.

• xmlns:er='http://www.deltaxml.com/ns/entity-references' — for retaining entity references, where the element's name and content mirror the entity reference's name and content respectively.

• xmlns:preserve='http://www.deltaxml.com/ns/preserve' — for retaining all other sorts of information, where the element's name represents the sort of information being retained (e.g. doctype, cdata, or comment).

The Lexical Preservation Concepts guide and the lexical-preservation.rng RELAX NG schema provide a top-level overview of what can be preserved and a semi-formal definition of the format respectively. This is supplemented by the remainder of this document which informally defines the format by example.

Example input document:

(01)
(02)  <!-- A pre DOCTYPE comment -->
(03)  <!DOCTYPE article SYSTEM "http://www.docbook.org/xml/4.5/docbookx.dtd"
(04)  [ <!ENTITY % paramEnt "
(05)      <!ATTLIST simpara level (unknown|novice|trainee|practitioner|expert) 'unknown'>
(06)    ">
(07)    <!ELEMENT exampleElement (#PCDATA)>
(08)    <!ATTLIST exampleElement yesNo (yes|no) 'no'>
(09)    %paramEnt;
(10)    <!ENTITY genEnt "an <emphasis role='bold'>internal (parsed) general</emphasis> entity.">
(11)  ]>
(12)   <?myPI Content of the processing instruction.?>
(13)   <article>
(14)     <title>Lexical Preservation Filter Demo</title>
(15)     <!-- In the following paragraph we reference the entity &genEnt; -->
(16)     <para>This paragraph references &genEnt;</para>
(17)     <para><![CDATA[Content of the CDATA Section text]]></para>
(18)     <simpara>An overridden simpara with a defaulted level attribute.</simpara>
(19)  </article>
(20)  <!-- A post XML body comment -->

The lexical preservation conversion of the example file is discussed in stages where the lines under consideration are reproduced along with the output where everything is being preserved. For clarity, the whitespace aspect of the preservation is not maintained.

XML Declaration

The XML declaration data is stored in a preserve:xmldecl element, which is a child of the root element. Here the XML declaration attributes are converted into element attributes with the same name.

(01)
(13)  <article>

To compare and preserve the XML declaration, it is added to the body of the document, as illustrated by line labelled (01b) below. Further, the namespaces that are used by this lexical preservation filter (13b to 13e) are attached to the root element (13a).

(01a)
(13a)  <article
(13b)    xmlns:deltaxml="http://www.deltaxml.com/ns/well-formed-delta-v1"
(13c)    xmlns:preserve="http://www.deltaxml.com/ns/preserve"
(13d)    xmlns:er="http://www.deltaxml.com/ns/entity-references"
(13e)    xmlns:pi="http://www.deltaxml.com/ns/processing-instructions">
(01b)    <preserve:xmldecl xml-version="1.0" encoding="UTF-8"/>

For convenience, all three namespaces for the retained information are declared, along with the main DeltaNova XML versioning namespace.

Comments and Processing Instructions

XML comments and processing instructions (PIs) are encoded into preserve:comment and pi:tag elements, where tag is replaced by the name of the PI. Encoded XML comments and PIs that appear outside the root element are contained in a preserve:pi-and-comment element, which is a child of the root element. There can be up to three preserve:pi-and-comment elements, distinguished by their region attribute value:

• BEFORE_DTD — PIs and comments before the DOCTYPE/Internal-Subset declaration.

• AFTER_DTD — PIs and comments after the DOCTYPE/Internal-Subset declaration, but before the root element.

• AFTER_BODY — PIs and comments after the root element (XML body has been completed).

Processing Instructions and Comments before the DOCTYPE declaration

(02)  <!-- A pre DOCTYPE comment -->

To compare and preserve comments and processing instructions that occur before the DOCTYPE declaration, a preserve:pi-and-comment block is introduced, with mode BEFORE_DTD.

(02a)    <preserve:pi-and-comment region="BEFORE_DTD">
(02b)      <preserve:comment> A pre DOCTYPE comment </preserve:comment>
(02c)    </preserve:pi-and-comment>

DOCTYPE and the Internal Subset Including Entity Refs

DOCTYPE information is converted into a preserve:doctype element that is a child of the root element, where the root element name, public identifier and system identifier are stored as attributes (if they exist). Furthermore, the content of the element contains the retained internal subset information.

The internal subset can contain DTD declarations and entity references. Each DTD declaration type is given its own representation element:

• Element Decl — converted into a preserve:elementDecl element, where the element's attributes are used to store its name and model data.

• Attribute Decl — converted into a preserve:attributeDecl element, where the element's attributes are used to store its name, associated element name, type, and default value.

• Internal Parsed Parameter Entity Declaration — converted into a preserve:internalParsedParameterEntityDecl element, where the element's attributes are used to store the entity declaration's name and value.

• Internal Parsed General Entity Declaration — converted into a preserve:internalParsedGeneralEntityDecl element, in a similar manner to the other entity declarations.

• Entity Reference (usage) — converted into an er:n element, where n is the name of the entity reference and the parameter attribute records whether this entity reference is referring to a parameter entity declaration. The content of these elements may contain the entity replacement text, if this has been asked for.

All the DTD declaration conversion elements also contain a deltaxml:key that uniquely identifies that declaration by its name.

The content of the entity declarations is escaped using the ASCII exclamation mark ! characters, where:

This special form of escaping ensures that it does not interfere with standard XML entity encoding mechanisms.

The doctype and internal subset conversions are illustrated in lines (3) to (11) of the example:

(03)  <!DOCTYPE article SYSTEM "http://www.docbook.org/xml/4.5/docbookx.dtd"
(04)  [ <!ENTITY % paramEnt "
(05)      <!ATTLIST q level (unknown|novice|trainee|practitioner|expert) 'unknown'>
(06)    ">
(07)    <!ELEMENT exampleElement (#PCDATA)>
(08)    <!ATTLIST exampleElement yesNo (yes|no) 'no'>
(09)    %paramEnt;
(10)    <!ENTITY genEnt "an <emphasis role='bold'>internal (parsed) general</emphasis> entity.">
(11)  ]>

These lines are converted to the following XML:

(03a)    <preserve:doctype name="article" systemId="http://www.docbook.org/xml/4.5/docbookx.dtd">
(04a)      <preserve:internalParsedParameterEntityDecl name="paramEnt"
               deltaxml:key="entity_par_paramEnt"
(05a)          value="!(*lt!)!!ATTLIST simpara level (unknown|...) !(*apos!)unknown!(*apos!)!(*gt!)"
(06a)      />
(07a)      <preserve:elementDecl name="exampleElement"
               deltaxml:key="element_exampleElement" model="(#PCDATA)"/>
(08a)      <preserve:attributeDecl name="yesNo"
               deltaxml:key="attribute(exampleElement,yesNo)"
(08b)          eName="exampleElement" type="(yes|no)" value="no"/>
(09a)      <er:paramEnt parameter="yes">
(05b)        <preserve:attributeDecl name="level"
                 deltaxml:key="attribute(simpara,level)"
(05c)            eName="simpara" type="(unknown|...)" value="unknown"/>
(09b)      </er:paramEnt>
(10a)      <preserve:internalParsedGeneralEntityDecl name="genEnt"
               deltaxml:key="entity_gen_genEnt"
(10b)          value="an !(*lt!)emphasis role=!(*apos!)bold!(*apos!)!(*gt!)internal (parsed) general!(*lt!)/emphasis!(*gt!) entity."/>
(03b)    </preserve:doctype>

Comments, Processing Instructions, and Entity References Revisited

In this section we look at some more examples of comments, processing instructions, and entity references. First we examine a processing instruction that is declared between the DOCTYPE and the XML body.

(12)  <?myPI Content of the processing instruction.?>

This is encoded as follows:

(12a)    <preserve:pi-and-comment region="AFTER_DTD">
(12b)      <pi:myPI>Content of the processing instruction.</pi:myPI>
(12c)    </preserve:pi-and-comment>

The main content of the document can now be processed. The following three lines contain some normal content, a comment, and a line that contains an entity reference.

(14)    <title>Lexical Preservation Filter Demo</title>
(15)    <!-- In the following paragraph we reference the entity &genEnt; -->
(16)    <para>This paragraph references &genEnt;</para>

The normal content is left unchanged, the comment is encoded into a preserve comment block, and the entity reference is encoded into an entity reference block that contains its replacement text.

(14a)    <title>Lexical Preservation Filter Demo</title>
(15a)    <preserve:comment> In the following paragraph we reference the entity &genEnt; </preserve:comment>
(16a)    <para>This paragraph references <er:genEnt>an <emphasis role="bold">internal (parsed) general</emphasis> entity.</er:genEnt></para>

CDATA section and Defaulted attributes

The CDATA section is converted into a preserve:cdata element whose content contains the CDATA characters.

The preserve:defaultAttributes attribute is added to any element that contains defaulted attributes. Its job is to record the attribute names, so that they can be stripped out later.

(17)    <para><![CDATA[Content of the CDATA Section text]]></para>
(18)    <simpara>An overridden simpara with a defaulted level attribute.</simpara>

These lines are converted to:

(17a)    <para><preserve:cdata>Content of the CDATA Section text</preserve:cdata></para>
(18a)    <simpara level="unknown" preserve:defaultAttributes="{}level">
            An overridden simpara with a defaulted level attribute.
         </simpara>

Completing the Example File

The last few lines of the example input complete the body of the document and add a final post comment.

(19)  </article>
(20)  <!-- A post XML body comment -->

These lines are converted as follows:

(20a)    <preserve:pi-and-comment region="AFTER_BODY">
(20b)      <preserve:comment> A post XML body comment </preserve:comment>
(20c)    </preserve:pi-and-comment>
(19a)  </article>

Implementation Notes

The LexicalPreservationBase class provides the underpinning implementation of the input filtering. The LexicalPreservationBase base class is specialised for use in the DocumentComparator, DataComparator and PipelinedComparator. These specialisations enable the lexical preservation to be configured by the LexicalPreservationConfig class.

When using the pipeline configuration formats DXP or DCP, the lexicalPreservation element is used to set the properties of the LexicalPreservationConfig class declaratively.