<chapter id="record-model-alvisxslt">
- <!-- $Id: recordmodel-alvisxslt.xml,v 1.7 2006-04-25 12:26:26 marc Exp $ -->
- <title>ALVIS XML Record Model and Filter Module</title>
+ <!-- $Id: recordmodel-alvisxslt.xml,v 1.15 2007-02-02 11:10:08 marc Exp $ -->
+ <title>ALVIS &xml; Record Model and Filter Module</title>
<para>
The record model described in this chapter applies to the fundamental,
- structured XML
+ structured &xml;
record type <literal>alvis</literal>, introduced in
- <xref linkend="componentmodulesalvis"/>. The ALVIS XML record model
+ <xref linkend="componentmodulesalvis"/>. The ALVIS &xml; record model
is experimental, and it's inner workings might change in future
- releases of the Zebra Information Server.
+ releases of the &zebra; Information Server.
</para>
<para> This filter has been developed under the
</para>
- <sect1 id="record-model-alvisxslt-filter">
+ <section id="record-model-alvisxslt-filter">
<title>ALVIS Record Filter</title>
<para>
- The experimental, loadable Alvis XML/XSLT filter module
+ The experimental, loadable Alvis &xml;/&xslt; filter module
<literal>mod-alvis.so</literal> is packaged in the GNU/Debian package
<literal>libidzebra1.4-mod-alvis</literal>.
It is invoked by the <filename>zebra.cfg</filename> configuration statement
</screen>
In this example on all data files with suffix
<filename>*.xml</filename>, where the
- Alvis XSLT filter configuration file is found in the
+ Alvis &xslt; filter configuration file is found in the
path <filename>db/filter_alvis_conf.xml</filename>.
</para>
- <para>The Alvis XSLT filter configuration file must be
- valid XML. It might look like this (This example is
- used for indexing and display of OAI harvested records):
+ <para>The Alvis &xslt; filter configuration file must be
+ valid &xml;. It might look like this (This example is
+ used for indexing and display of &oai; harvested records):
<screen>
<?xml version="1.0" encoding="UTF-8"?>
<schemaInfo>
<schema name="index" identifier="http://indexdata.dk/zebra/xslt/1"
stylesheet="xsl/oai2index.xsl" />
<schema name="dc" stylesheet="xsl/oai2dc.xsl" />
- <!-- use split level 2 when indexing whole OAI Record lists -->
+ <!-- use split level 2 when indexing whole &oai; Record lists -->
<split level="2"/>
</schemaInfo>
</screen>
names defined in the <literal>name</literal> attributes must be
unique, these are the literal <literal>schema</literal> or
<literal>element set</literal> names used in
- <ulink url="http://www.loc.gov/standards/sru/srw/">SRW</ulink>,
- <ulink url="http://www.loc.gov/standards/sru/">SRU</ulink> and
- Z39.50 protocol queries.
+ <ulink url="http://www.loc.gov/standards/sru/srw/">&srw;</ulink>,
+ <ulink url="&url.sru;">&sru;</ulink> and
+ &z3950; protocol queries.
The paths in the <literal>stylesheet</literal> attributes
are relative to zebras working directory, or absolute to file
system root.
</para>
<para>
The <literal><split level="2"/></literal> decides where the
- XML Reader shall split the
+ &xml; Reader shall split the
collections of records into individual records, which then are
- loaded into DOM, and have the indexing XSLT stylesheet applied.
+ loaded into &dom;, and have the indexing &xslt; stylesheet applied.
</para>
<para>
- There must be exactly one indexing XSLT stylesheet, which is
+ There must be exactly one indexing &xslt; stylesheet, which is
defined by the magic attribute
<literal>identifier="http://indexdata.dk/zebra/xslt/1"</literal>.
</para>
- <sect2 id="record-model-alvisxslt-internal">
+ <section id="record-model-alvisxslt-internal">
<title>ALVIS Internal Record Representation</title>
- <para>When indexing, an XML Reader is invoked to split the input
- files into suitable record XML pieces. Each record piece is then
- transformed to an XML DOM structure, which is essentially the
- record model. Only XSLT transformations can be applied during
+ <para>When indexing, an &xml; Reader is invoked to split the input
+ files into suitable record &xml; pieces. Each record piece is then
+ transformed to an &xml; &dom; structure, which is essentially the
+ record model. Only &xslt; transformations can be applied during
index, search and retrieval. Consequently, output formats are
- restricted to whatever XSLT can deliver from the record XML
- structure, be it other XML formats, HTML, or plain text. In case
- you have <literal>libxslt1</literal> running with EXSLT support,
+ restricted to whatever &xslt; can deliver from the record &xml;
+ structure, be it other &xml; formats, HTML, or plain text. In case
+ you have <literal>libxslt1</literal> running with E&xslt; support,
you can use this functionality inside the Alvis
- filter configuration XSLT stylesheets.
+ filter configuration &xslt; stylesheets.
</para>
- </sect2>
+ </section>
- <sect2 id="record-model-alvisxslt-canonical">
+ <section id="record-model-alvisxslt-canonical">
<title>ALVIS Canonical Indexing Format</title>
- <para>The output of the indexing XSLT stylesheets must contain
+ <para>The output of the indexing &xslt; stylesheets must contain
certain elements in the magic
<literal>xmlns:z="http://indexdata.dk/zebra/xslt/1"</literal>
- namespace. The output of the XSLT indexing transformation is then
- parsed using DOM methods, and the contained instructions are
+ namespace. The output of the &xslt; indexing transformation is then
+ parsed using &dom; methods, and the contained instructions are
performed on the <emphasis>magic elements and their
subtrees</emphasis>.
</para>
z:id="oai:JTRS:CP-3290---Volume-I"
z:rank="47896"
z:type="update">
- <z:index name="oai:identifier" type="0">
+ <z:index name="oai_identifier" type="0">
oai:JTRS:CP-3290---Volume-I</z:index>
- <z:index name="oai:datestamp" type="0">2004-07-09</z:index>
- <z:index name="oai:setspec" type="0">jtrs</z:index>
- <z:index name="dc:all" type="w">
- <z:index name="dc:title" type="w">Proceedings of the 4th
+ <z:index name="oai_datestamp" type="0">2004-07-09</z:index>
+ <z:index name="oai_setspec" type="0">jtrs</z:index>
+ <z:index name="dc_all" type="w">
+ <z:index name="dc_title" type="w">Proceedings of the 4th
International Conference and Exhibition:
World Congress on Superconductivity - Volume I</z:index>
- <z:index name="dc:creator" type="w">Kumar Krishen and *Calvin
+ <z:index name="dc_creator" type="w">Kumar Krishen and *Calvin
Burnham, Editors</z:index>
</z:index>
</z:record>
</screen>
</para>
- <para>This means the following: From the original XML file
- <literal>one-record.xml</literal> (or from the XML record DOM of the
+ <para>This means the following: From the original &xml; file
+ <literal>one-record.xml</literal> (or from the &xml; record &dom; of the
same form coming from a splitted input file), the indexing
- stylesheet produces an indexing XML record, which is defined by
+ stylesheet produces an indexing &xml; record, which is defined by
the <literal>record</literal> element in the magic namespace
<literal>xmlns:z="http://indexdata.dk/zebra/xslt/1"</literal>.
- Zebra uses the content of
+ &zebra; uses the content of
<literal>z:id="oai:JTRS:CP-3290---Volume-I"</literal> as internal
record ID, and - in case static ranking is set - the content of
<literal>z:rank="47896"</literal> as static rank. Following the
</para>
<para>In this example, the following literal indexes are constructed:
<screen>
- oai:identifier
- oai:datestamp
- oai:setspec
- dc:all
- dc:title
- dc:creator
+ oai_identifier
+ oai_datestamp
+ oai_setspec
+ dc_all
+ dc_title
+ dc_creator
</screen>
where the indexing type is defined in the
<literal>type</literal> attribute
the same character normalization map <literal>w</literal>.
</para>
<para>
- Finally, this example configuration can be queried using PQF
- queries, either transported by Z39.50, (here using a yaz-client)
+ Finally, this example configuration can be queried using &pqf;
+ queries, either transported by &z3950;, (here using a yaz-client)
<screen>
<![CDATA[
Z> open localhost:9999
Z> elem dc
Z> form xml
Z>
- Z> f @attr 1=dc:creator Kumar
- Z> scan @attr 1=dc:creator adam
+ Z> f @attr 1=dc_creator Kumar
+ Z> scan @attr 1=dc_creator adam
Z>
- Z> f @attr 1=dc:title @attr 4=2 "proceeding congress superconductivity"
- Z> scan @attr 1=dc:title abc
+ Z> f @attr 1=dc_title @attr 4=2 "proceeding congress superconductivity"
+ Z> scan @attr 1=dc_title abc
]]>
</screen>
or the proprietary
extentions <literal>x-pquery</literal> and
<literal>x-pScanClause</literal> to
- SRU, and SRW
+ &sru;, and &srw;
<screen>
<![CDATA[
- http://localhost:9999/?version=1.1&operation=searchRetrieve&x-pquery=%40attr+1%3Ddc%3Acreator+%40attr+4%3D6+%22the
- http://localhost:9999/?version=1.1&operation=scan&x-pScanClause=@attr+1=dc:date+@attr+4=2+a
+ http://localhost:9999/?version=1.1&operation=searchRetrieve&x-pquery=%40attr+1%3Ddc_creator+%40attr+4%3D6+%22the
+ http://localhost:9999/?version=1.1&operation=scan&x-pScanClause=@attr+1=dc_date+@attr+4=2+a
]]>
</screen>
- See <xref linkend="server-sru"/> for more information on SRU/SRW
- configuration, and <xref linkend="gfs-config"/> or
- <ulink url="http://www.indexdata.dk/yaz/doc/tools.tkl#tools.cql">
- the YAZ manual CQL section</ulink>
- for the details
- of the YAZ frontend server
- <ulink url="http://www.loc.gov/standards/sru/cql/">CQL</ulink>
- configuration.
+ See <xref linkend="zebrasrv-sru"/> for more information on &sru;/&srw;
+ configuration, and <xref linkend="gfs-config"/> or the &yaz;
+ <ulink url="&url.yaz.cql;">&cql; section</ulink>
+ for the details or the &yaz; frontend server.
</para>
<para>
Notice that there are no <filename>*.abs</filename>,
- <filename>*.est</filename>, <filename>*.map</filename>, or other GRS-1
+ <filename>*.est</filename>, <filename>*.map</filename>, or other &grs1;
filter configuration files involves in this process, and that the
literal index names are used during search and retrieval.
</para>
- </sect2>
- </sect1>
+ </section>
+ </section>
- <sect1 id="record-model-alvisxslt-conf">
+ <section id="record-model-alvisxslt-conf">
<title>ALVIS Record Model Configuration</title>
- <sect2 id="record-model-alvisxslt-index">
+ <section id="record-model-alvisxslt-index">
<title>ALVIS Indexing Configuration</title>
<para>
As mentioned above, there can be only one indexing
stylesheet, and configuration of the indexing process is a synonym
- of writing an XSLT stylesheet which produces XML output containing the
+ of writing an &xslt; stylesheet which produces &xml; output containing the
magic elements discussed in
<xref linkend="record-model-alvisxslt-internal"/>.
Obviously, there are million of different ways to accomplish this
<para>
Stylesheets can be written in the <emphasis>pull</emphasis> or
the <emphasis>push</emphasis> style: <emphasis>pull</emphasis>
- means that the output XML structure is taken as starting point of
- the internal structure of the XSLT stylesheet, and portions of
- the input XML are <emphasis>pulled</emphasis> out and inserted
- into the right spots of the output XML structure. On the other
- side, <emphasis>push</emphasis> XSLT stylesheets are recursavly
+ means that the output &xml; structure is taken as starting point of
+ the internal structure of the &xslt; stylesheet, and portions of
+ the input &xml; are <emphasis>pulled</emphasis> out and inserted
+ into the right spots of the output &xml; structure. On the other
+ side, <emphasis>push</emphasis> &xslt; stylesheets are recursavly
calling their template definitions, a process which is commanded
- by the input XML structure, and avake to produce some output XML
+ by the input &xml; structure, and avake to produce some output &xml;
whenever some special conditions in the input styelsheets are
met. The <emphasis>pull</emphasis> type is well-suited for input
- XML with strong and well-defined structure and semantcs, like the
- following OAI indexing example, whereas the
+ &xml; with strong and well-defined structure and semantcs, like the
+ following &oai; indexing example, whereas the
<emphasis>push</emphasis> type might be the only possible way to
- sort out deeply recursive input XML formats.
+ sort out deeply recursive input &xml; formats.
</para>
<para>
A <emphasis>pull</emphasis> stylesheet example used to index
- OAI harvested records could use some of the following template
+ &oai; harvested records could use some of the following template
definitions:
<screen>
<![CDATA[
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:z="http://indexdata.dk/zebra/xslt/1"
- xmlns:oai="http://www.openarchives.org/OAI/2.0/"
- xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/"
+ xmlns:oai="http://www.openarchives.org/&oai;/2.0/"
+ xmlns:oai_dc="http://www.openarchives.org/&oai;/2.0/oai_dc/"
xmlns:dc="http://purl.org/dc/elements/1.1/"
version="1.0">
<xsl:template match="/">
<z:record z:id="{normalize-space(oai:record/oai:header/oai:identifier)}"
z:type="update">
- <!-- you might want to use z:rank="{some XSLT function here}" -->
+ <!-- you might want to use z:rank="{some &xslt; function here}" -->
<xsl:apply-templates/>
</z:record>
</xsl:template>
- <!-- OAI indexing templates -->
+ <!-- &oai; indexing templates -->
<xsl:template match="oai:record/oai:header/oai:identifier">
- <z:index name="oai:identifier" type="0">
+ <z:index name="oai_identifier" type="0">
<xsl:value-of select="."/>
</z:index>
</xsl:template>
<!-- DC specific indexing templates -->
<xsl:template match="oai:record/oai:metadata/oai_dc:dc/dc:title">
- <z:index name="dc:title" type="w">
+ <z:index name="dc_title" type="w">
<xsl:value-of select="."/>
</z:index>
</xsl:template>
<para>
Notice also,
that the names and types of the indexes can be defined in the
- indexing XSLT stylesheet <emphasis>dynamically according to
- content in the original XML records</emphasis>, which has
+ indexing &xslt; stylesheet <emphasis>dynamically according to
+ content in the original &xml; records</emphasis>, which has
opportunities for great power and wizardery as well as grande
disaster.
</para>
<para>
The following excerpt of a <emphasis>push</emphasis> stylesheet
<emphasis>might</emphasis>
- be a good idea according to your strict control of the XML
+ be a good idea according to your strict control of the &xml;
input format (due to rigerours checking against well-defined and
- tight RelaxNG or XML Schema's, for example):
+ tight RelaxNG or &xml; Schema's, for example):
<screen>
<![CDATA[
<xsl:template name="element-name-indexes">
]]>
</screen>
This template creates indexes which have the name of the working
- node of any input XML file, and assigns a '1' to the index.
+ node of any input &xml; file, and assigns a '1' to the index.
The example query
<literal>find @attr 1=xyz 1</literal>
finds all files which contain at least one
- <literal>xyz</literal> XML element. In case you can not control
+ <literal>xyz</literal> &xml; element. In case you can not control
which element names the input files contain, you might ask for
disaster and bad karma using this technique.
</para>
to suffering and pain, and universal
disentigration of your project schedule.
</para>
- </sect2>
+ </section>
- <sect2 id="record-model-alvisxslt-elementset">
+ <section id="record-model-alvisxslt-elementset">
<title>ALVIS Exchange Formats</title>
<para>
An exchange format can be anything which can be the outcome of an
- XSLT transformation, as far as the stylesheet is registered in
- the main Alvis XSLT filter configuration file, see
+ &xslt; transformation, as far as the stylesheet is registered in
+ the main Alvis &xslt; filter configuration file, see
<xref linkend="record-model-alvisxslt-filter"/>.
- In principle anything that can be expressed in XML, HTML, and
+ In principle anything that can be expressed in &xml;, HTML, and
TEXT can be the output of a <literal>schema</literal> or
<literal>element set</literal> directive during search, as long as
the information comes from the
- <emphasis>original input record XML DOM tree</emphasis>
- (and not the transformed and <emphasis>indexed</emphasis> XML!!).
+ <emphasis>original input record &xml; &dom; tree</emphasis>
+ (and not the transformed and <emphasis>indexed</emphasis> &xml;!!).
</para>
<para>
- In addition, internal administrative information from the Zebra
+ In addition, internal administrative information from the &zebra;
indexer can be accessed during record retrieval. The following
example is a summary of the possibilities:
<screen>
</screen>
</para>
- </sect2>
+ </section>
- <sect2 id="record-model-alvisxslt-example">
- <title>ALVIS Filter OAI Indexing Example</title>
+ <section id="record-model-alvisxslt-example">
+ <title>ALVIS Filter &oai; Indexing Example</title>
<para>
The sourcecode tarball contains a working Alvis filter example in
the directory <filename>examples/alvis-oai/</filename>, which
should get you started.
</para>
<para>
- More example data can be harvested from any OAI complient server,
- see details at the OAI
+ More example data can be harvested from any &oai; complient server,
+ see details at the &oai;
<ulink url="http://www.openarchives.org/">
http://www.openarchives.org/</ulink> web site, and the community
links at
<ulink url="http://www.oaforum.org/tutorial/">
http://www.oaforum.org/tutorial/</ulink>.
</para>
- </sect2>
+ </section>
- </sect1>
+ </section>
</chapter>
<!--
-c) Main "alvis" XSLT filter config file:
+c) Main "alvis" &xslt; filter config file:
cat db/filter_alvis_conf.xml
<?xml version="1.0" encoding="UTF8"?>
The split level decides where the SAX parser shall split the
collections of records into individual records, which then are
- loaded into DOM, and have the indexing XSLT stylesheet applied.
+ loaded into &dom;, and have the indexing &xslt; stylesheet applied.
The indexing stylesheet is found by it's identifier.
and so on.
- in db/ a cql2pqf.txt yaz-client config file
- which is also used in the yaz-server <ulink url="http://www.loc.gov/standards/sru/cql/">CQL</ulink>-to-PQF process
+ which is also used in the yaz-server <ulink url="&url.cql;">&cql;</ulink>-to-&pqf; process
see: http://www.indexdata.com/yaz/doc/tools.tkl#tools.cql.map
-- in db/ an indexing XSLT stylesheet. This is a PULL-type XSLT thing,
- as it constructs the new XML structure by pulling data out of the
+- in db/ an indexing &xslt; stylesheet. This is a PULL-type XSLT thing,
+ as it constructs the new &xml; structure by pulling data out of the
respective elements/attributes of the old structure.
Notice the special zebra namespace, and the special elements in this
indicates that a new record with given id and static rank has to be updated.
<z:index name="title" type="w">
- encloses all the text/XML which shall be indexed in the index named
+ encloses all the text/&xml; which shall be indexed in the index named
"title" and of index type "w" (see file default.idx in your zebra
installation)