Running a Word-by-word Text Comparison
This section explains the concepts behind word-by-word text comparison using a sample. See sample resources and a description on how to run it here.
When comparing documents containing text, DeltaNova Compare treats each block of text as a single node. This can lead to large amounts of change when in fact only certain words within the text have been changed. Consider the following document and the changes made to it:
Example 1: an XML document containing text (iiinput1.xml`` in Bitbucket, https://bitbucket.org/deltaxml/word-by-word)
<document>
<para>A pangram uses all the letters of the alphabet. It is often used to test typewriters
or computer keyboards, for example: The quick brown fox jumps over the lazy dog.</para>
</document>
Example 2: a modified version of the document (iiinput2.xml`` in Bitbucket, https://bitbucket.org/deltaxml/word-by-word)
<document>
<para>A pangram uses all the letters of the alphabet and is often used to test typewriters
or computer keyboards, for example: A quick movement of the enemy will jeopardize six gunboats.</para>
</document>
The sample code shows how the same comparison can be run using either the Pipelined Comparator or the Document Comparator. The description of specific filters here applies only to the Pipelined Comparator; the equivalent 'word-by-word' and 'orphaned word' features are built-in to the Document Comparator and are controlled via its API or a DCP XML configuration file.
Comparing Text Word-by-word
While the result of a plain comparison is technically correct, it is not particularly useful for displaying what has actually changed. A much better approach would be to compare the text on a word by word basis.
Word-by-word Filters
DeltaNova Compare includes Java filters to split text into individual words before comparing them and also to convert the split words back into larger chunks of text. This allows the comparison to show only those words that have changed. These filters are com.deltaxml.pipe.filters.dx2.wbw.WordInfilter and com.deltaxml.pipe.filters.dx2.wbw.WordOutfilter.
Word-by-word Attributes
When word-by-word filters are in place, the default is for the word by word feature to affect all parts of the document. However, it is possible to specify which parts of a document are affected or unaffected through the use of a deltaxml:word-by-word attribute. This attribute has permitted values of true or false and may be attached to any element in an input document.
Orphaned Words
The OrphanedWordOutfilter is a post-processing filter that will detect unchanged words in the middle of changes and duplicate them as an added and deleted copy of the same word.
This makes the result easier to read by grouping fragmented changes together.
Filter Ordering
The order of the word by word filters is important. Because they convert each word, space and potentially punctuation character into an element, they can dramatically increase the size of the XML document tree. The word filters are written as Java streaming filters that do not load the tree into memory and are placed as close as possible on either side of the Java-implemented comparison stage.
Input Filters
The WordInfilter is typically used as the last input filter before the comparison on either input filter chain.
Output Filters
The minimal requirement for output filters is to use the WordOutfilter as the first output filter. If the orphaned word processing is required, OrphanedWordOutfilter should be used as the first output filter, followed by WordOutfilter. All other output filters (particularly XSLT filters) should go after the word filters.
Configuring the Orphaned Words Filter
The OrphanedWordOutfilter has two configuration parameters that change its behaviour: orphanedLengthLimit and orphanedThresholdPercentage.
The orphanedLengthLimit Parameter
This parameter specifies the maximum number of consecutive unchanged words that could be treated as orphaned words. Its default value is 2 and it should be kept as a fairly small number to avoid all words being treated as orphans and the advantage of word by word comparison being lost.
The orphanedThresholdPercentage Parameter
This specifies a threshold that is used in a calculation that assesses the size of the unchanged word section in relation to the changed words either side of it. The default value is 20 — i.e. the unchanged words can count for no more than 20 percent of the total count of changed and unchanged words. The calculated value that is compared against this percentage is:
unchanged words / (changed words before + unchanged words + changed words after) * 100
This calculated value must be less than the value of orphanedThresholdPercentage for the unchanged words to be treated as orphans.
Running the Sample
See the sample resources and a description on how to run it for Java here.
The resources must be checked out, cloned or downloaded and unzipped into the samples directory of the DeltaNova Compare release. They should be located such that they are two levels below the top level release directory, for example DeltaXML-XML-Compare-10_0_0_j/samples/word-by-word.