1 <chapter id="record-model-alvisxslt">
2 <title>ALVIS &acro.xml; Record Model and Filter Module</title>
6 The functionality of this record model has been improved and
7 replaced by the DOM &acro.xml; record model, see
8 <xref linkend="record-model-domxml"/>. The Alvis &acro.xml; record
9 model is considered obsolete, and will eventually be removed
10 from future releases of the &zebra; software.
15 The record model described in this chapter applies to the fundamental,
17 record type <literal>alvis</literal>, introduced in
18 <xref linkend="componentmodulesalvis"/>.
21 <para> This filter has been developed under the
22 <ulink url="http://www.alvis.info/">ALVIS</ulink> project funded by
23 the European Community under the "Information Society Technologies"
28 <section id="record-model-alvisxslt-filter">
29 <title>ALVIS Record Filter</title>
31 The experimental, loadable Alvis &acro.xml;/&acro.xslt; filter module
32 <literal>mod-alvis.so</literal> is packaged in the GNU/Debian package
33 <literal>libidzebra1.4-mod-alvis</literal>.
34 It is invoked by the <filename>zebra.cfg</filename> configuration statement
36 recordtype.xml: alvis.db/filter_alvis_conf.xml
38 In this example on all data files with suffix
39 <filename>*.xml</filename>, where the
40 Alvis &acro.xslt; filter configuration file is found in the
41 path <filename>db/filter_alvis_conf.xml</filename>.
43 <para>The Alvis &acro.xslt; filter configuration file must be
44 valid &acro.xml;. It might look like this (This example is
45 used for indexing and display of &acro.oai; harvested records):
47 <?xml version="1.0" encoding="UTF-8"?>
49 <schema name="identity" stylesheet="xsl/identity.xsl" />
50 <schema name="index" identifier="http://indexdata.dk/zebra/xslt/1"
51 stylesheet="xsl/oai2index.xsl" />
52 <schema name="dc" stylesheet="xsl/oai2dc.xsl" />
53 <!-- use split level 2 when indexing whole OAI Record lists -->
54 <split level="2"/>
59 All named stylesheets defined inside
60 <literal>schema</literal> element tags
61 are for presentation after search, including
62 the indexing stylesheet (which is a great debugging help). The
63 names defined in the <literal>name</literal> attributes must be
64 unique, these are the literal <literal>schema</literal> or
65 <literal>element set</literal> names used in
66 <ulink url="http://www.loc.gov/standards/sru/srw/">&acro.srw;</ulink>,
67 <ulink url="&url.sru;">&acro.sru;</ulink> and
68 &acro.z3950; protocol queries.
69 The paths in the <literal>stylesheet</literal> attributes
70 are relative to zebras working directory, or absolute to file
74 The <literal><split level="2"/></literal> decides where the
75 &acro.xml; Reader shall split the
76 collections of records into individual records, which then are
77 loaded into &acro.dom;, and have the indexing &acro.xslt; stylesheet applied.
80 There must be exactly one indexing &acro.xslt; stylesheet, which is
81 defined by the magic attribute
82 <literal>identifier="http://indexdata.dk/zebra/xslt/1"</literal>.
85 <section id="record-model-alvisxslt-internal">
86 <title>ALVIS Internal Record Representation</title>
87 <para>When indexing, an &acro.xml; Reader is invoked to split the input
88 files into suitable record &acro.xml; pieces. Each record piece is then
89 transformed to an &acro.xml; &acro.dom; structure, which is essentially the
90 record model. Only &acro.xslt; transformations can be applied during
91 index, search and retrieval. Consequently, output formats are
92 restricted to whatever &acro.xslt; can deliver from the record &acro.xml;
93 structure, be it other &acro.xml; formats, HTML, or plain text. In case
94 you have <literal>libxslt1</literal> running with E&acro.xslt; support,
95 you can use this functionality inside the Alvis
96 filter configuration &acro.xslt; stylesheets.
100 <section id="record-model-alvisxslt-canonical">
101 <title>ALVIS Canonical Indexing Format</title>
102 <para>The output of the indexing &acro.xslt; stylesheets must contain
103 certain elements in the magic
104 <literal>xmlns:z="http://indexdata.dk/zebra/xslt/1"</literal>
105 namespace. The output of the &acro.xslt; indexing transformation is then
106 parsed using &acro.dom; methods, and the contained instructions are
107 performed on the <emphasis>magic elements and their
111 For example, the output of the command
113 xsltproc xsl/oai2index.xsl one-record.xml
115 might look like this:
117 <?xml version="1.0" encoding="UTF-8"?>
118 <z:record xmlns:z="http://indexdata.dk/zebra/xslt/1"
119 z:id="oai:JTRS:CP-3290---Volume-I"
121 <z:index name="oai_identifier" type="0">
122 oai:JTRS:CP-3290---Volume-I</z:index>
123 <z:index name="oai_datestamp" type="0">2004-07-09</z:index>
124 <z:index name="oai_setspec" type="0">jtrs</z:index>
125 <z:index name="dc_all" type="w">
126 <z:index name="dc_title" type="w">Proceedings of the 4th
127 International Conference and Exhibition:
128 World Congress on Superconductivity - Volume I</z:index>
129 <z:index name="dc_creator" type="w">Kumar Krishen and *Calvin
130 Burnham, Editors</z:index>
135 <para>This means the following: From the original &acro.xml; file
136 <literal>one-record.xml</literal> (or from the &acro.xml; record &acro.dom; of the
137 same form coming from a split input file), the indexing
138 stylesheet produces an indexing &acro.xml; record, which is defined by
139 the <literal>record</literal> element in the magic namespace
140 <literal>xmlns:z="http://indexdata.dk/zebra/xslt/1"</literal>.
141 &zebra; uses the content of
142 <literal>z:id="oai:JTRS:CP-3290---Volume-I"</literal> as internal
143 record ID, and - in case static ranking is set - the content of
144 <literal>z:rank="47896"</literal> as static rank. Following the
145 discussion in <xref linkend="administration-ranking"/>
146 we see that this records is internally ordered
147 lexicographically according to the value of the string
148 <literal>oai:JTRS:CP-3290---Volume-I47896</literal>.
149 <!-- The type of action performed during indexing is defined by
150 <literal>z:type="update"></literal>, with recognized values
151 <literal>insert</literal>, <literal>update</literal>, and
152 <literal>delete</literal>. -->
154 <para>In this example, the following literal indexes are constructed:
163 where the indexing type is defined in the
164 <literal>type</literal> attribute
165 (any value from the standard configuration
166 file <filename>default.idx</filename> will do). Finally, any
167 <literal>text()</literal> node content recursively contained
168 inside the <literal>index</literal> will be filtered through the
169 appropriate char map for character normalization, and will be
170 inserted in the index.
173 Specific to this example, we see that the single word
174 <literal>oai:JTRS:CP-3290---Volume-I</literal> will be literal,
175 byte for byte without any form of character normalization,
176 inserted into the index named <literal>oai:identifier</literal>,
178 <literal>Kumar Krishen and *Calvin Burnham, Editors</literal>
179 will be inserted using the <literal>w</literal> character
180 normalization defined in <filename>default.idx</filename> into
181 the index <literal>dc:creator</literal> (that is, after character
182 normalization the index will keep the individual words
183 <literal>kumar</literal>, <literal>krishen</literal>,
184 <literal>and</literal>, <literal>calvin</literal>,
185 <literal>burnham</literal>, and <literal>editors</literal>), and
186 finally both the texts
187 <literal>Proceedings of the 4th International Conference and Exhibition:
188 World Congress on Superconductivity - Volume I</literal>
190 <literal>Kumar Krishen and *Calvin Burnham, Editors</literal>
191 will be inserted into the index <literal>dc:all</literal> using
192 the same character normalization map <literal>w</literal>.
195 Finally, this example configuration can be queried using &acro.pqf;
196 queries, either transported by &acro.z3950;, (here using a yaz-client)
199 Z> open localhost:9999
203 Z> f @attr 1=dc_creator Kumar
204 Z> scan @attr 1=dc_creator adam
206 Z> f @attr 1=dc_title @attr 4=2 "proceeding congress superconductivity"
207 Z> scan @attr 1=dc_title abc
211 extensions <literal>x-pquery</literal> and
212 <literal>x-pScanClause</literal> to
213 &acro.sru;, and &acro.srw;
216 http://localhost:9999/?version=1.1&operation=searchRetrieve&x-pquery=%40attr+1%3Ddc_creator+%40attr+4%3D6+%22the
217 http://localhost:9999/?version=1.1&operation=scan&x-pScanClause=@attr+1=dc_date+@attr+4=2+a
220 See <xref linkend="zebrasrv-sru"/> for more information on &acro.sru;/&acro.srw;
221 configuration, and <xref linkend="gfs-config"/> or the &yaz;
222 <ulink url="&url.yaz.cql;">&acro.cql; section</ulink>
223 for the details or the &yaz; frontend server.
226 Notice that there are no <filename>*.abs</filename>,
227 <filename>*.est</filename>, <filename>*.map</filename>, or other &acro.grs1;
228 filter configuration files involves in this process, and that the
229 literal index names are used during search and retrieval.
235 <section id="record-model-alvisxslt-conf">
236 <title>ALVIS Record Model Configuration</title>
239 <section id="record-model-alvisxslt-index">
240 <title>ALVIS Indexing Configuration</title>
242 As mentioned above, there can be only one indexing
243 stylesheet, and configuration of the indexing process is a synonym
244 of writing an &acro.xslt; stylesheet which produces &acro.xml; output containing the
245 magic elements discussed in
246 <xref linkend="record-model-alvisxslt-internal"/>.
247 Obviously, there are million of different ways to accomplish this
248 task, and some comments and code snippets are in order to lead
249 our Padawan's on the right track to the good side of the force.
252 Stylesheets can be written in the <emphasis>pull</emphasis> or
253 the <emphasis>push</emphasis> style: <emphasis>pull</emphasis>
254 means that the output &acro.xml; structure is taken as starting point of
255 the internal structure of the &acro.xslt; stylesheet, and portions of
256 the input &acro.xml; are <emphasis>pulled</emphasis> out and inserted
257 into the right spots of the output &acro.xml; structure. On the other
258 side, <emphasis>push</emphasis> &acro.xslt; stylesheets are recursively
259 calling their template definitions, a process which is commanded
260 by the input &acro.xml; structure, and are triggered to produce some output &acro.xml;
261 whenever some special conditions in the input stylesheets are
262 met. The <emphasis>pull</emphasis> type is well-suited for input
263 &acro.xml; with strong and well-defined structure and semantics, like the
264 following &acro.oai; indexing example, whereas the
265 <emphasis>push</emphasis> type might be the only possible way to
266 sort out deeply recursive input &acro.xml; formats.
269 A <emphasis>pull</emphasis> stylesheet example used to index
270 &acro.oai; harvested records could use some of the following template
274 <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
275 xmlns:z="http://indexdata.dk/zebra/xslt/1"
276 xmlns:oai="http://www.openarchives.org/&acro.oai;/2.0/"
277 xmlns:oai_dc="http://www.openarchives.org/&acro.oai;/2.0/oai_dc/"
278 xmlns:dc="http://purl.org/dc/elements/1.1/"
281 <xsl:output indent="yes" method="xml" version="1.0" encoding="UTF-8"/>
283 <!-- disable all default text node output -->
284 <xsl:template match="text()"/>
286 <!-- match on oai xml record root -->
287 <xsl:template match="/">
288 <z:record z:id="{normalize-space(oai:record/oai:header/oai:identifier)}">
289 <!-- you might want to use z:rank="{some &acro.xslt; function here}" -->
290 <xsl:apply-templates/>
294 <!-- &acro.oai; indexing templates -->
295 <xsl:template match="oai:record/oai:header/oai:identifier">
296 <z:index name="oai_identifier" type="0">
297 <xsl:value-of select="."/>
303 <!-- DC specific indexing templates -->
304 <xsl:template match="oai:record/oai:metadata/oai_dc:dc/dc:title">
305 <z:index name="dc_title" type="w">
306 <xsl:value-of select="."/>
318 that the names and types of the indexes can be defined in the
319 indexing &acro.xslt; stylesheet <emphasis>dynamically according to
320 content in the original &acro.xml; records</emphasis>, which has
321 opportunities for great power and wizardry as well as grande
325 The following excerpt of a <emphasis>push</emphasis> stylesheet
326 <emphasis>might</emphasis>
327 be a good idea according to your strict control of the &acro.xml;
328 input format (due to rigorous checking against well-defined and
329 tight RelaxNG or &acro.xml; Schema's, for example):
332 <xsl:template name="element-name-indexes">
333 <z:index name="{name()}" type="w">
334 <xsl:value-of select="'1'"/>
339 This template creates indexes which have the name of the working
340 node of any input &acro.xml; file, and assigns a '1' to the index.
342 <literal>find @attr 1=xyz 1</literal>
343 finds all files which contain at least one
344 <literal>xyz</literal> &acro.xml; element. In case you can not control
345 which element names the input files contain, you might ask for
346 disaster and bad karma using this technique.
349 One variation over the theme <emphasis>dynamically created
350 indexes</emphasis> will definitely be unwise:
353 <!-- match on oai xml record root -->
354 <xsl:template match="/">
357 <!-- create dynamic index name from input content -->
358 <xsl:variable name="dynamic_content">
359 <xsl:value-of select="oai:record/oai:header/oai:identifier"/>
362 <!-- create zillions of indexes with unknown names -->
363 <z:index name="{$dynamic_content}" type="w">
364 <xsl:value-of select="oai:record/oai:metadata/oai_dc:dc"/>
371 Don't be tempted to cross
372 the line to the dark side of the force, Padawan; this leads
373 to suffering and pain, and universal
374 disintegration of your project schedule.
378 <section id="record-model-alvisxslt-elementset">
379 <title>ALVIS Exchange Formats</title>
381 An exchange format can be anything which can be the outcome of an
382 &acro.xslt; transformation, as far as the stylesheet is registered in
383 the main Alvis &acro.xslt; filter configuration file, see
384 <xref linkend="record-model-alvisxslt-filter"/>.
385 In principle anything that can be expressed in &acro.xml;, HTML, and
386 TEXT can be the output of a <literal>schema</literal> or
387 <literal>element set</literal> directive during search, as long as
388 the information comes from the
389 <emphasis>original input record &acro.xml; &acro.dom; tree</emphasis>
390 (and not the transformed and <emphasis>indexed</emphasis> &acro.xml;!!).
393 In addition, internal administrative information from the &zebra;
394 indexer can be accessed during record retrieval. The following
395 example is a summary of the possibilities:
398 <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
399 xmlns:z="http://indexdata.dk/zebra/xslt/1"
402 <!-- register internal zebra parameters -->
403 <xsl:param name="id" select="''"/>
404 <xsl:param name="filename" select="''"/>
405 <xsl:param name="score" select="''"/>
406 <xsl:param name="schema" select="''"/>
408 <xsl:output indent="yes" method="xml" version="1.0" encoding="UTF-8"/>
410 <!-- use then for display of internal information -->
411 <xsl:template match="/">
413 <id><xsl:value-of select="$id"/></id>
414 <filename><xsl:value-of select="$filename"/></filename>
415 <score><xsl:value-of select="$score"/></score>
416 <schema><xsl:value-of select="$schema"/></schema>
427 <section id="record-model-alvisxslt-example">
428 <title>ALVIS Filter &acro.oai; Indexing Example</title>
430 The source code tarball contains a working Alvis filter example in
431 the directory <filename>examples/alvis-oai/</filename>, which
432 should get you started.
435 More example data can be harvested from any &acro.oai; compliant server,
436 see details at the &acro.oai;
437 <ulink url="http://www.openarchives.org/">
438 http://www.openarchives.org/</ulink> web site, and the community
440 <ulink url="http://www.openarchives.org/community/index.html">
441 http://www.openarchives.org/community/index.html</ulink>.
444 <ulink url="http://www.oaforum.org/tutorial/">
445 http://www.oaforum.org/tutorial/</ulink>.
456 <!-- Keep this comment at the end of the file
461 sgml-minimize-attributes:nil
462 sgml-always-quote-attributes:t
465 sgml-parent-document: "idzebra.xml"
466 sgml-local-catalogs: nil
467 sgml-namecase-general:t