<chapter id="architecture">
- <!-- $Id: architecture.xml,v 1.13 2006-09-22 12:34:45 adam Exp $ -->
- <title>Overview of Zebra Architecture</title>
-
+ <!-- $Id: architecture.xml,v 1.19 2007-02-02 09:58:39 marc Exp $ -->
+ <title>Overview of &zebra; Architecture</title>
<section id="architecture-representation">
<title>Local Representation</title>
<para>
- As mentioned earlier, Zebra places few restrictions on the type of
+ As mentioned earlier, &zebra; places few restrictions on the type of
data that you can index and manage. Generally, whatever the form of
the data, it is parsed by an input filter specific to that format, and
- turned into an internal structure that Zebra knows how to handle. This
+ turned into an internal structure that &zebra; knows how to handle. This
process takes place whenever the record is accessed - for indexing and
retrieval.
</para>
<para>
The RecordType parameter in the <literal>zebra.cfg</literal> file, or
- the <literal>-t</literal> option to the indexer tells Zebra how to
+ the <literal>-t</literal> option to the indexer tells &zebra; how to
process input records.
Two basic types of processing are available - raw text and structured
data. Raw text is just that, and it is selected by providing the
- argument <emphasis>text</emphasis> to Zebra. Structured records are
+ argument <emphasis>text</emphasis> to &zebra;. Structured records are
all handled internally using the basic mechanisms described in the
subsequent sections.
- Zebra can read structured records in many different formats.
+ &zebra; can read structured records in many different formats.
<!--
How this is done is governed by additional parameters after the
"grs" keyword, separated by "." characters.
<section id="architecture-maincomponents">
<title>Main Components</title>
<para>
- The Zebra system is designed to support a wide range of data management
+ The &zebra; system is designed to support a wide range of data management
applications. The system can be configured to handle virtually any
kind of structured data. Each record in the system is associated with
a <emphasis>record schema</emphasis> which lends context to the data
one database, the system poses no such restrictions.
</para>
<para>
- The Zebra indexer and information retrieval server consists of the
+ The &zebra; indexer and information retrieval server consists of the
following main applications: the <command>zebraidx</command>
indexing maintenance utility, and the <command>zebrasrv</command>
information query and retrieval server. Both are using some of the
<para>
The virtual Debian package <literal>idzebra-2.0</literal>
installs all the necessary packages to start
- working with Zebra - including utility programs, development libraries,
+ working with &zebra; - including utility programs, development libraries,
documentation and modules.
</para>
<section id="componentcore">
- <title>Core Zebra Libraries Containing Common Functionality</title>
+ <title>Core &zebra; Libraries Containing Common Functionality</title>
<para>
- The core Zebra module is the meat of the <command>zebraidx</command>
+ The core &zebra; module is the meat of the <command>zebraidx</command>
indexing maintenance utility, and the <command>zebrasrv</command>
information query and retrieval server binaries. Shortly, the core
libraries are responsible for
<varlistentry>
<term>Index Maintenance</term>
<listitem>
- <para> Zebra maintains Term Dictionaries and ISAM index
+ <para> &zebra; maintains Term Dictionaries and ISAM index
entries in inverted index structures kept on disk. These are
optimized for fast inset, update and delete, as well as good
search performance.
</para>
<para>
The Debian package <literal>libidzebra-2.0</literal>
- contains all run-time libraries for Zebra, the
+ contains all run-time libraries for &zebra;, the
documentation in PDF and HTML is found in
<literal>idzebra-2.0-doc</literal>, and
<literal>idzebra-2.0-common</literal>
- includes common essential Zebra configuration files.
+ includes common essential &zebra; configuration files.
</para>
</section>
<section id="componentindexer">
- <title>Zebra Indexer</title>
+ <title>&zebra; Indexer</title>
<para>
The <command>zebraidx</command>
indexing maintenance utility
</section>
<section id="componentsearcher">
- <title>Zebra Searcher/Retriever</title>
+ <title>&zebra; Searcher/Retriever</title>
<para>
This is the executable which runs the Z39.50/SRU/SRW server and
glues together the core libraries and the filter modules to one
The YAZ server frontend is
a full fledged stateful Z39.50 server taking client
connections, and forwarding search and scan requests to the
- Zebra core indexer.
+ &zebra; core indexer.
</para>
<para>
In addition to Z39.50 requests, the YAZ server frontend acts
modules.
</para>
-
- <section id="componentmodulestext">
- <title>TEXT Record Model and Filter Module</title>
+
+ <section id="componentmodulesalvis">
+ <title>ALVIS XML Record Model and Filter Module</title>
+ <para>
+ The Alvis filter for XML files is an XSLT based input
+ filter.
+ It indexes element and attribute content of any thinkable XML format
+ using full XPATH support, a feature which the standard &zebra;
+ GRS SGML and XML filters lacked. The indexed documents are
+ parsed into a standard XML DOM tree, which restricts record size
+ according to availability of memory.
+ </para>
<para>
- Plain ASCII text filter. TODO: add information here.
+ The Alvis filter
+ uses XSLT display stylesheets, which let
+ the &zebra; DB administrator associate multiple, different views on
+ the same XML document type. These views are chosen on-the-fly in
+ search time.
+ </para>
+ <para>
+ In addition, the Alvis filter configuration is not bound to the
+ arcane BIB-1 Z39.50 library catalogue indexing traditions and
+ folklore, and is therefore easier to understand.
</para>
- </section>
+ <para>
+ Finally, the Alvis filter allows for static ranking at index
+ time, and to to sort hit lists according to predefined
+ static ranks. This imposes no overhead at all, both
+ search and indexing perform still
+ <emphasis>O(1)</emphasis> irrespectively of document
+ collection size. This feature resembles Googles pre-ranking using
+ their Pagerank algorithm.
+ </para>
+ <para>
+ Details on the experimental Alvis XSLT filter are found in
+ <xref linkend="record-model-alvisxslt"/>.
+ </para>
+ <para>
+ The Debian package <literal>libidzebra-2.0-mod-alvis</literal>
+ contains the Alvis filter module.
+ </para>
+ </section>
<section id="componentmodulesgrs">
<title>GRS Record Model and Filter Modules</title>
<literal>libidzebra-2.0-mod-grs-xml</literal> includes the
<emphasis>grs.xml</emphasis> filter which uses <ulink
url="&url.expat;">Expat</ulink> to
- parse records in XML and turn them into IDZebra's internal GRS node
+ parse records in XML and turn them into ID&zebra;'s internal GRS node
trees. Have also a look at the Alvis XML/XSLT filter described in
the next session.
</para>
</section>
-
- <section id="componentmodulesalvis">
- <title>ALVIS Record Model and Filter Module</title>
- <para>
- The Alvis filter for XML files is an XSLT based input
- filter.
- It indexes element and attribute content of any thinkable XML format
- using full XPATH support, a feature which the standard Zebra
- GRS SGML and XML filters lacked. The indexed documents are
- parsed into a standard XML DOM tree, which restricts record size
- according to availability of memory.
- </para>
- <para>
- The Alvis filter
- uses XSLT display stylesheets, which let
- the Zebra DB administrator associate multiple, different views on
- the same XML document type. These views are chosen on-the-fly in
- search time.
- </para>
- <para>
- In addition, the Alvis filter configuration is not bound to the
- arcane BIB-1 Z39.50 library catalogue indexing traditions and
- folklore, and is therefore easier to understand.
- </para>
+
+ <section id="componentmodulestext">
+ <title>TEXT Record Model and Filter Module</title>
<para>
- Finally, the Alvis filter allows for static ranking at index
- time, and to to sort hit lists according to predefined
- static ranks. This imposes no overhead at all, both
- search and indexing perform still
- <emphasis>O(1)</emphasis> irrespectively of document
- collection size. This feature resembles Googles pre-ranking using
- their Pagerank algorithm.
+ Plain ASCII text filter. TODO: add information here.
</para>
- <para>
- Details on the experimental Alvis XSLT filter are found in
- <xref linkend="record-model-alvisxslt"/>.
- </para>
- <para>
- The Debian package <literal>libidzebra-2.0-mod-alvis</literal>
- contains the Alvis filter module.
- </para>
- </section>
+ </section>
<!--
<section id="componentmodulessafari">
</para>
</section>
+ <section id="special-retrieval">
+ <title>Retrieval of &zebra; internal record data</title>
+ <para>
+ Starting with <literal>&zebra;</literal> version 2.0.5 or newer, it is
+ possible to use a special element set which has the prefix
+ <literal>zebra::</literal>.
+ </para>
+ <para>
+ Using this element will, regardless of record type, return
+ &zebra;'s internal index structure/data for a record.
+ In particular, the regular record filters are not invoked when
+ these are in use.
+ This can in some cases make the retrival faster than regular
+ retrieval operations (for MARC, XML etc).
+ </para>
+ <table id="special-retrieval-types">
+ <title>Special Retrieval Elements</title>
+ <tgroup cols="2">
+ <thead>
+ <row>
+ <entry>Element Set</entry>
+ <entry>Description</entry>
+ <entry>Syntax</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry><literal>zebra::meta::sysno</literal></entry>
+ <entry>Get &zebra; record system ID</entry>
+ <entry>XML and SUTRS</entry>
+ </row>
+ <row>
+ <entry><literal>zebra::data</literal></entry>
+ <entry>Get raw record</entry>
+ <entry>all</entry>
+ </row>
+ <row>
+ <entry><literal>zebra::meta</literal></entry>
+ <entry>Get &zebra; record internal metadata</entry>
+ <entry>XML and SUTRS</entry>
+ </row>
+ <row>
+ <entry><literal>zebra::index</literal></entry>
+ <entry>Get all indexed keys for record</entry>
+ <entry>XML and SUTRS</entry>
+ </row>
+ <row>
+ <entry>
+ <literal>zebra::index::</literal><replaceable>f</replaceable>
+ </entry>
+ <entry>
+ Get indexed keys for field <replaceable>f</replaceable> for record
+ </entry>
+ <entry>XML and SUTRS</entry>
+ </row>
+ <row>
+ <entry>
+ <literal>zebra::index::</literal><replaceable>f</replaceable>:<replaceable>t</replaceable>
+ </entry>
+ <entry>
+ Get indexed keys for field <replaceable>f</replaceable>
+ and type <replaceable>t</replaceable> for record
+ </entry>
+ <entry>XML and SUTRS</entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+ <para>
+ For example, to fetch the raw binary record data stored in the
+ zebra internal storage, or on the filesystem, the following
+ commands can be issued:
+ <screen>
+ Z> f @attr 1=title my
+ Z> format xml
+ Z> elements zebra::data
+ Z> s 1+1
+ Z> format sutrs
+ Z> s 1+1
+ Z> format usmarc
+ Z> s 1+1
+ </screen>
+ </para>
+ <para>
+ The special
+ <literal>zebra::data</literal> element set name is
+ defined for any record syntax, but will always fetch
+ the raw record data in exactly the original form. No record syntax
+ specific transformations will be applied to the raw record data.
+ </para>
+ <para>
+ Also, &zebra; internal metadata about the record can be accessed:
+ <screen>
+ Z> f @attr 1=title my
+ Z> format xml
+ Z> elements zebra::meta::sysno
+ Z> s 1+1
+ </screen>
+ displays in <literal>XML</literal> record syntax only internal
+ record system number, whereas
+ <screen>
+ Z> f @attr 1=title my
+ Z> format xml
+ Z> elements zebra::meta
+ Z> s 1+1
+ </screen>
+ displays all available metadata on the record. These include sytem
+ number, database name, indexed filename, filter used for indexing,
+ score and static ranking information and finally bytesize of record.
+ </para>
+ <para>
+ Sometimes, it is very hard to figure out what exactly has been
+ indexed how and in which indexes. Using the indexing stylesheet of
+ the Alvis filter, one can at least see which portion of the record
+ went into which index, but a similar aid does not exist for all
+ other indexing filters.
+ </para>
+ <para>
+ The special
+ <literal>zebra::index</literal> element set names are provided to
+ access information on per record indexed fields. For example, the
+ queries
+ <screen>
+ Z> f @attr 1=title my
+ Z> format sutrs
+ Z> elements zebra::index
+ Z> s 1+1
+ </screen>
+ will display all indexed tokens from all indexed fields of the
+ first record, and it will display in <literal>SUTRS</literal>
+ record syntax, whereas
+ <screen>
+ Z> f @attr 1=title my
+ Z> format xml
+ Z> elements zebra::index::title
+ Z> s 1+1
+ Z> elements zebra::index::title:p
+ Z> s 1+1
+ </screen>
+ displays in <literal>XML</literal> record syntax only the content
+ of the zebra string index <literal>title</literal>, or
+ even only the type <literal>p</literal> phrase indexed part of it.
+ </para>
+ <note>
+ <para>
+ Trying to access numeric <literal>Bib-1</literal> use
+ attributes or trying to access non-existent zebra intern string
+ access points will result in a Diagnostic 25: Specified element set
+ 'name not valid for specified database.
+ </para>
+ </note>
+ </section>
+
</chapter>
<!-- Keep this comment at the end of the file
projects / idzebra-moved-to-github.git / blobdiff