<chapter id="introduction">
- <!-- $Id: introduction.xml,v 1.21 2002-11-08 17:00:57 mike Exp $ -->
+ <!-- $Id: introduction.xml,v 1.45 2007-02-02 14:42:44 marc Exp $ -->
<title>Introduction</title>
- <sect1>
+ <section id="overview">
<title>Overview</title>
+ <para>
+ &zebra; is a free, fast, friendly information management system. It can
+ index records in &xml;/&sgml;, &marc;, e-mail archives and many other
+ formats, and quickly find them using a combination of boolean
+ searching and relevance ranking. Search-and-retrieve applications can
+ be written using &api;s in a wide variety of languages, communicating
+ with the &zebra; server using industry-standard information-retrieval
+ protocols or web services.
+ </para>
+ <para>
+ &zebra; is licensed Open Source, and can be
+ deployed by anyone for any purpose without license fees. The C source
+ code is open to anybody to read and change under the GPL license.
+ </para>
+ <para>
+ &zebra; is a networked component which acts as a reliable &z3950; server
+ for both record/document search, presentation, insert, update and
+ delete operations. In addition, it understands the &sru; family of
+ webservices, which exist in &rest; &get;/&post; and truly &soap; flavors.
+ </para>
+ <para>
+ &zebra; is available as MS Windows 2003 Server (32 bit) self-extracting
+ package as well as GNU/Debian Linux (32 bit and 64 bit) precompiled
+ packages. It has been deployed successfully on other Unix systems,
+ including Sun Sparc, HP Unix, and many variants of Linux and BSD
+ based systems.
+ </para>
+ <para>
+ <ulink url="http://www.indexdata.com/zebra/">http://www.indexdata.com/zebra/</ulink>
+ <ulink url="http://ftp.indexdata.dk/pub/zebra/win32/">http://ftp.indexdata.dk/pub/zebra/win32/</ulink>
+ <ulink url="http://ftp.indexdata.dk/pub/zebra/debian/">http://ftp.indexdata.dk/pub/zebra/debian/</ulink>
+ </para>
+
<para>
- <ulink url="http://indexdata.dk/zebra/">
- Zebra</ulink>
+ <ulink url="http://indexdata.dk/zebra/">&zebra;</ulink>
is a high-performance, general-purpose structured text
- indexing and retrieval engine. It reads structured records in a
- variety of input formats (eg. email, XML, MARC) and provides access
+ indexing and retrieval engine. It reads records in a
+ variety of input formats (eg. email, &xml;, &marc;) and provides access
to them through a powerful combination of boolean search
expressions and relevance-ranked free-text queries.
</para>
<para>
- Zebra supports large databases (tens of millions of records,
+ &zebra; supports large databases (tens of millions of records,
tens of gigabytes of data). It allows safe, incremental
- database updates on live systems. Because Zebra supports
- the industry-standard information retrieval protocol, Z39.50,
- you can search Zebra databases using an enormous variety of
+ database updates on live systems. Because &zebra; supports
+ the industry-standard information retrieval protocol, &z3950;,
+ you can search &zebra; databases using an enormous variety of
programs and toolkits, both commercial and free, which understand
this protocol. Application libraries are available to allow
bespoke clients to be written in Perl, C, C++, Java, Tcl, Visual
- Basic, Python, PHP and more - see
- <ulink url="http://zoom.z3950.org/">the ZOOM web site</ulink>
+ Basic, Python, &php; and more - see the
+ <ulink url="&url.zoom;">&zoom; web site</ulink>
for more information on some of these client toolkits.
</para>
<para>
- This document is an introduction to the Zebra system. It explains
+ This document is an introduction to the &zebra; system. It explains
how to compile the software, how to prepare your first database,
and how to configure the server to give you the
functionality that you need.
</para>
- </sect1>
+ </section>
- <sect1 id="features">
- <title>Features</title>
+ <section id="features">
+ <title>&zebra; Features Overview</title>
- <para>
- This is an overview of some of Zebra's most important features:
- </para>
-
- <para>
- <itemizedlist>
- <listitem>
- <para>
- Very large databases: files for indexes, etc. can be
- automatically partitioned over multiple disks.
- </para>
- </listitem>
+ <!--
+ <row>
+ <entry></entry>
+ <entry></entry>
+ <entry></entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry></entry>
+ <entry></entry>
+ <entry></entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry></entry>
+ <entry></entry>
+ <entry></entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ -->
- <listitem>
- <para>
- Arbitrarily complex records. The internal data format
- is an structured format conceptually similar to XML or GRS-1,
- which allows lists, nested structured data elements and
- variant forms of data.
- </para>
- </listitem>
- <listitem>
- <para>
- Robust updating - records can be added and deleted ``on the fly''
- without rebuilding the index from scratch.
- Records can be safely updated even while users are accessing
- the server.
- The update procedure is tolerant to crashes or hard interrupts
- during database updating - data can be reconstructed following
- a crash.
- </para>
- </listitem>
+ <table id="table-features-protocol" frame="top">
+ <title>&zebra; networked protocols</title>
+ <tgroup cols="4">
+ <thead>
+ <row>
+ <entry>Feature</entry>
+ <entry>Availability</entry>
+ <entry>Notes</entry>
+ <entry>Reference</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>Fundamental operation types</entry>
+ <entry>&z3950;/&sru; explain, search, and scan</entry>
+ <entry></entry>
+ <entry><xref linkend="querymodel-operation-types"/></entry>
+ </row>
+ <row>
+ <entry>&z3950; protocol support</entry>
+ <entry>yes</entry>
+ <entry> Protocol facilities supported are:
+ Init, Search, Present (retrieval),
+ Segmentation (support for very large records), Delete, Scan
+ (index browsing), Sort, Close and support for the ``update''
+ Extended Service to add or replace an existing &xml;
+ record. Piggy-backed presents are honored in the search
+ request. Named result sets are supported.</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Web Service support</entry>
+ <entry>&sru_gps;</entry>
+ <entry> The protocol operations <literal>explain</literal>,
+ <literal>searchRetrieve</literal> and <literal>scan</literal>
+ are supported. <ulink url="&url.cql;">&cql;</ulink> to internal
+ query model &rpn;
+ conversion is supported. Extended RPN queries
+ for search/retrieve and scan are supported.</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
- <listitem>
- <para>
- Configurable to understand many input formats.
- A system of input filters driven by
- regular expressions allows most ASCII-based
- data formats to be easily processed.
- SGML, XML, ISO2709 (MARC), and raw text are also
- supported.
- </para>
- </listitem>
- <listitem>
- <para>
- Searching supports a powerful combination of boolean queries as
- well as relevance-ranking (free-text) queries. Truncation,
- masking, full regular expression matching and "approximate
- matching" (eg. spelling mistakes) are all handled.
- </para>
- </listitem>
+ <table id="table-features-search" frame="top">
+ <title>&zebra; search functionality</title>
+ <tgroup cols="4">
+ <thead>
+ <row>
+ <entry>Feature</entry>
+ <entry>Availability</entry>
+ <entry>Notes</entry>
+ <entry>Reference</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>Query languages</entry>
+ <entry>&cql; and &rpn;/&pqf;</entry>
+ <entry>The type-1 Reverse Polish Notation (&rpn;)
+ and it's textual representation Prefix Query Format (&pqf;) are
+ supported. The Common Query Language (&cql;) can be configured as
+ a mapping from &cql; to &rpn;/&pqf;</entry>
+ <entry><xref linkend="querymodel-query-languages-pqf"/>
+ <xref linkend="querymodel-cql-to-pqf"/></entry>
+ </row>
+ <row>
+ <entry>Complex boolean query tree</entry>
+ <entry>&cql; and &rpn;/&pqf;</entry>
+ <entry>Both &cql; and &rpn;/&pqf; allow atomic query parts (&apt;) to
+ be combined into complex boolean query trees</entry>
+ <entry><xref linkend="querymodel-rpn-tree"/></entry>
+ </row>
+ <row>
+ <entry>Field search</entry>
+ <entry>user defined</entry>
+ <entry>Atomic query parts (&apt;) are either general, or
+ directed at user-specified document fields
+ </entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Data normalization</entry>
+ <entry></entry>
+ <entry>Data normalization, text tokenization and character mappings can be
+ applied during indexing and searching</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Predefined field types</entry>
+ <entry></entry>
+ <entry>Data fields can be indexed as phrase, as into word tokenized text,
+ as numeric values, url's, dates, and raw binary data.</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Regular expression matching</entry>
+ <entry>Regexp </entry>
+ <entry>Full regular expression matching and "approximate
+ matching" (eg. spelling mistake corrections) are handled.</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Search truncation</entry>
+ <entry></entry>
+ <entry></entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Fuzzy searches</entry>
+ <entry></entry>
+ <entry>In addition, fuzzy searches are implemented, where one
+ spelling mistake in search terms is matched</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
- <listitem>
- <para>
- Index-only databases: data can be, and usually is, imported
- into Zebra's own storage, but Zebra can also refer to
+
+ <table id="table-features-scan" frame="top">
+ <title>&zebra; index scanning</title>
+ <tgroup cols="4">
+ <thead>
+ <row>
+ <entry>Feature</entry>
+ <entry>Availability</entry>
+ <entry>Notes</entry>
+ <entry>Reference</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>Scan</entry>
+ <entry>yes</entry>
+ <entry><literal>Scan</literal> on a given named index returns all the
+ indexed terms in lexicographical order near the given start term.</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Facetted browsing</entry>
+ <entry>partial</entry>
+ <entry>&zebra; supports <literal>scan inside a hit
+ set</literal> from a previous search, thus reducing the listed
+ terms to the
+ subset of terms found in the documents/records of the hit set.</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Drill-down or refine-search</entry>
+ <entry>partially</entry>
+ <entry>scanning in result sets can be used to implement
+ drill-down in search clients</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+
+ <table id="table-features-presentation" frame="top">
+ <title>&zebra; document presentation</title>
+ <tgroup cols="4">
+ <thead>
+ <row>
+ <entry>Feature</entry>
+ <entry>Availability</entry>
+ <entry>Notes</entry>
+ <entry>Reference</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>Hit count</entry>
+ <entry>yes</entry>
+ <entry>Search results include at any time the total hit count of a given
+ query, either exact computed, or approximative, in case that the
+ hit count exceeds a possible pre-defined hit set truncation
+ level.
+</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Paged result sets</entry>
+ <entry>yes</entry>
+ <entry>Paging of search requests and present/display request can return any
+ successive number of records from any start position in the hit set,
+ i.e. it is trivial to provide search results in successive pages of
+ any size.</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>&xml;ocument transformations</entry>
+ <entry>&xslt; based</entry>
+ <entry> Record presentation can be performed in many pre-defined &xml; data
+ formats, where the original &xml; records are on-the-fly transformed
+ through any preconfigured &xslt; transformation. It is therefore
+ trivial to present records in short/full &xml; views, transforming to
+ RSS, Dublin Core, or other &xml; based data formats, or transform
+ records to XHTML snippets ready for inserting in XHTML pages.</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Binary record transformations</entry>
+ <entry>&marc;, &usmarc;, &marc21; and &marcxml;</entry>
+ <entry></entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Record Syntaxes</entry>
+ <entry></entry>
+ <entry> Multiple record syntaxes
+ for data retrieval: &grs1;, &sutrs;,
+ &xml;, ISO2709 (&marc;), etc. Records can be mapped between record syntaxes
+ and schemas on the fly.</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+
+ <table id="table-features-sort-rank" frame="top">
+ <title>&zebra; sorting and ranking</title>
+ <tgroup cols="4">
+ <thead>
+ <row>
+ <entry>Feature</entry>
+ <entry>Availability</entry>
+ <entry>Notes</entry>
+ <entry>Reference</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>Sort</entry>
+ <entry>numeric, lexicographic</entry>
+ <entry>Sorting on the basis of alpha-numeric and numeric data
+ is supported. Alphanumeric sorts can be configured for different data encodings
+ and locales for European languages. </entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Combined sorting</entry>
+ <entry>yes</entry>
+ <entry>Sorting on the basis of combined sorts e.g. combinations of
+ ascending/descending sorts of lexicographical/numeric/date field data
+ is supported</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Relevance ranking</entry>
+ <entry>TF-IDF like</entry>
+ <entry>Relevance-ranking of free-text queries is supported
+ using a TF-IDF like algorithm.</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Relevence ranking</entry>
+ <entry>TDF-IDF like</entry>
+ <entry></entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+
+
+ <table id="table-features-document" frame="top">
+ <title>&zebra; document model</title>
+ <tgroup cols="4">
+ <thead>
+ <row>
+ <entry>Feature</entry>
+ <entry>Availability</entry>
+ <entry>Notes</entry>
+ <entry>Reference</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>Complex semi-structured Documents</entry>
+ <entry>&xml; and &grs1; Documents</entry>
+ <entry>Both &xml; and &grs1; documents exhibit a &dom; like internal
+ representation allowing for complex indexing and display rules</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Input document formats</entry>
+ <entry>&xml;, &sgml;, Text, ISO2709 (&marc;)</entry>
+ <entry>
+ A system of input filters driven by
+ regular expressions allows most ASCII-based
+ data formats to be easily processed.
+ &sgml;, &xml;, ISO2709 (&marc;), and raw text are also
+ supported.</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Document storage</entry>
+ <entry>Index-only, Key storage, Document storage</entry>
+ <entry>Data can be, and usually is, imported
+ into &zebra;'s own storage, but &zebra; can also refer to
external files, building and maintaining indexes of "live"
- collections.
- </para>
- </listitem>
+ collections.</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
- <listitem>
- <para>
- Zebra is written in portable C, so it runs on most Unix-like systems
- as well as Windows NT. A binary distribution for Windows NT is
- available at
- <ulink url="http://ftp.indexdata.dk/pub/zebra/win32/"/>,
- and pre-built packages are available for some Linux
- distributions:
- Red Hat 7.x RPMs at
- <ulink url="http://ftp.indexdata.dk/pub/zebra/RedHat7.X/"/>
- and Debian packages at
- <ulink url="http://ftp.indexdata.dk/pub/zebra/debian/"/>
- </para>
- </listitem>
-
- </itemizedlist>
-
- </para>
-
- <para>
- Z39.50 protocol support:
- </para>
-
- <para>
- <itemizedlist>
- <listitem>
- <para>
- Protocol facilities: Init, Search, Present (retrieval),
- Segmentation (support for very large records), Delete, Scan
- (index browsing), Sort, Close and support for the ``update''
- Extended Service to add or replace an existing XML record.
- <!-- Adam says:
- * Supported
- You can insert/delete/replace an XML record given an
- "external" ID. Actually this way of doing ES Update was
- meant for an OAI application that Ian Ibbotson had in
- mind to implement. The "update" command in YAZ client
- implements this on the client side. My plan is to make
- this available in ZOOM "extended" soon..
- -->
- </para>
- </listitem>
+ </tbody>
+ </tgroup>
+ </table>
- <listitem>
- <para>
- Piggy-backed presents are honored in the search request - that
- is, a subset of the found records can be returned directly with
- a search response, enabling search and retrieval to happen in a
- single round-trip.
- </para>
- </listitem>
- <listitem>
- <para>
- Named result sets are supported.
- </para>
- </listitem>
- <listitem>
- <para>
- Easily configured to support different application profiles, with
- tables for attribute sets, tag sets, and abstract syntaxes.
- Additional tables control facilities such as element mappings to
- different schema (eg., GILS-to-USMARC).
- </para>
- </listitem>
+ <table id="table-features-scalability" frame="top">
+ <title>&zebra; data size and scalability</title>
+ <tgroup cols="4">
+ <thead>
+ <row>
+ <entry>Feature</entry>
+ <entry>Availability</entry>
+ <entry>Notes</entry>
+ <entry>Reference</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>No of records</entry>
+ <entry>40-60 million</entry>
+ <entry></entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Data size</entry>
+ <entry>100 GB of record data</entry>
+ <entry></entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>File pointers</entry>
+ <entry>64 bit</entry>
+ <entry></entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Scale out</entry>
+ <entry>multiple discs</entry>
+ <entry></entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Performance</entry>
+ <entry><literal>O(n * log N)</literal></entry>
+ <entry> &zebra; query speed and performance is affected roughly by
+ <literal>O(log N)</literal>,
+ where <literal>N</literal> is the total database size, and by
+ <literal>O(n)</literal>, where <literal>n</literal> is the
+ specific query hit set size.</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Average search times</entry>
+ <entry></entry>
+ <entry> Even on very large size databases hit rates of 20 queries per
+ seconds with average query answering time of 1 second are possible,
+ provided that the boolean queries are constructed sufficiently
+ precise to result in hit sets of the order of 1000 to 5.000
+ documents.</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Large databases</entry>
+ <entry>64 file pointers assure that register files can extend
+ the 2 GB limit. Logical files can be
+ automatically partitioned over multiple disks, thus allowing for
+ large databases.</entry>
+ <entry></entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
- <listitem>
- <para>
- Complex composition specifications using Espec-1 (partial support).
- Element sets are defined using the Espec-1 capability,
- and are specified in configuration files as simple element
- requests (and, optionally, variant requests).
- </para>
- </listitem>
- <listitem>
- <para>
- Multiple record syntaxes
- for data retrieval: GRS-1, SUTRS,
- XML, ISO2709 (MARC), etc. Records can be mapped between record syntaxes
- and schemas on the fly.
- </para>
- </listitem>
+ <table id="table-features-updates" frame="top">
+ <title>&zebra; live updates</title>
+ <tgroup cols="4">
+ <thead>
+ <row>
+ <entry>Feature</entry>
+ <entry>Availability</entry>
+ <entry>Notes</entry>
+ <entry>Reference</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>Batch updates</entry>
+ <entry></entry>
+ <entry>It is possible to schedule record inserts/updates/deletes in any
+ quantity, from single individual handled records to batch updates
+ in strikes of any size, as well as total re-indexing of all records
+ from file system. </entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Incremental updates</entry>
+ <entry></entry>
+ <entry></entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Remote updates</entry>
+ <entry>&z3950; extended services</entry>
+ <entry></entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Live updates</entry>
+ <entry></entry>
+ <entry> Data updates are transaction based and can be performed on running
+ &zebra; systems. Full searchability is preserved during life data update due to use
+ of shadow disk areas for update operations. Multiple update transactions at the same time are lined up, to be
+ performed one after each other. Data integrity is preserved.</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Database updates</entry>
+ <entry>live, incremental updates</entry>
+ <entry>Robust updating - records can be added and deleted ``on the fly''
+ without rebuilding the index from scratch.
+ Records can be safely updated even while users are accessing
+ the server.
+ The update procedure is tolerant to crashes or hard interrupts
+ during database updating - data can be reconstructed following
+ a crash.</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <table id="table-features-platforms" frame="top">
+ <title>&zebra; supported platforms</title>
+ <tgroup cols="4">
+ <thead>
+ <row>
+ <entry>Feature</entry>
+ <entry>Availability</entry>
+ <entry>Notes</entry>
+ <entry>Reference</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>Linux</entry>
+ <entry></entry>
+ <entry>GNU Linux (32 and 64bit), journaling Reiser or (better) JFS filesystem
+ on disks. GNU/Debian Linux packages are available</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Unix</entry>
+ <entry>tarball</entry>
+ <entry>Usual tarball install possible on many major Unix systems</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Windows</entry>
+ <entry></entry>
+ <entry>Windows installer packages available</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ <row>
+ <entry>Supported Platforms</entry>
+ <entry>UNIX, Linux, Windows (NT/2000/2003/XP)</entry>
+ <entry>&zebra; is written in portable C, so it runs on most
+ Unix-like systems as well as Windows (NT/2000/2003/XP). Binary
+ distributions are
+ available for GNU/Debian Linux and Windows</entry>
+ <entry><xref linkend=""/></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
- </itemizedlist>
-
- </para>
- </sect1>
+ </section>
- <sect1 id="apps">
- <title>Applications</title>
+ <section id="introduction-apps">
+ <title>References and &zebra; based Applications</title>
<para>
- Zebra has been deployed in numerous applications, in both the
+ &zebra; has been deployed in numerous applications, in both the
academic and commercial worlds, in application domains as diverse
as bibliographic catalogues, geospatial information, structured
vocabulary browsing, government information locators, civic
Notable applications include the following:
</para>
- <sect2>
- <title>DADS - the DTV Article Database Service</title>
+
+ <section id="koha-ils">
+ <title>Koha free open-source ILS</title>
+ <para>
+ <ulink url="http://www.koha.org/">Koha</ulink> is a full-featured
+ open-source ILS, initially developed in
+ New Zealand by Katipo Communications Ltd, and first deployed in
+ January of 2000 for Horowhenua Library Trust. It is currently
+ maintained by a team of software providers and library technology
+ staff from around the globe.
+ </para>
+ <para>
+ <ulink url="http://liblime.com/">LibLime</ulink>,
+ a company that is marketing and supporting Koha, adds in
+ the new release of Koha 3.0 the &zebra;
+ database server to drive its bibliographic database.
+ </para>
+ <para>
+ In early 2005, the Koha project development team began looking at
+ ways to improve &marc; support and overcome scalability limitations
+ in the Koha 2.x series. After extensive evaluations of the best
+ of the Open Source textual database engines - including MySQL
+ full-text searching, PostgreSQL, Lucene and Plucene - the team
+ selected &zebra;.
+ </para>
+ <para>
+ "&zebra; completely eliminates scalability limitations, because it
+ can support tens of millions of records." explained Joshua
+ Ferraro, LibLime's Technology President and Koha's Project
+ Release Manager. "Our performance tests showed search results in
+ under a second for databases with over 5 million records on a
+ modest i386 900Mhz test server."
+ </para>
+ <para>
+ "&zebra; also includes support for true boolean search expressions
+ and relevance-ranked free-text queries, both of which the Koha
+ 2.x series lack. &zebra; also supports incremental and safe
+ database updates, which allow on-the-fly record
+ management. Finally, since &zebra; has at its heart the &z3950;
+ protocol, it greatly improves Koha's support for that critical
+ library standard."
+ </para>
+ <para>
+ Although the bibliographic database will be moved to &zebra;, Koha
+ 3.0 will continue to use a relational SQL-based database design
+ for the 'factual' database. "Relational database managers have
+ their strengths, in spite of their inability to handle large
+ numbers of bibliographic records efficiently," summed up Ferraro,
+ "We're taking the best from both worlds in our redesigned Koha
+ 3.0.
+ </para>
+ <para>
+ See also LibLime's newsletter article
+ <ulink url="http://www.liblime.com/newsletter/2006/01/features/koha-earns-its-stripes/">
+ Koha Earns its Stripes</ulink>.
+ </para>
+ </section>
+
+ <section id="emilda-ils">
+ <title>Emilda open source ILS</title>
<para>
+ <ulink url="http://www.emilda.org/">Emilda</ulink>
+ is a complete Integrated Library System, released under the
+ GNU General Public License. It has a
+ full featured Web-OPAC, allowing comprehensive system management
+ from virtually any computer with an Internet connection, has
+ template based layout allowing anyone to alter the visual
+ appearance of Emilda, and is
+ &xml; based language for fast and easy portability to virtually any
+ language.
+ Currently, Emilda is used at three schools in Espoo, Finland.
+ </para>
+ <para>
+ As a surplus, 100% &marc; compatibility has been achieved using the
+ &zebra; Server from Index Data as backend server.
+ </para>
+ </section>
+
+ <section id="reindex-ils">
+ <title>ReIndex.Net web based ILS</title>
+ <para>
+ <ulink url="http://www.reindex.net/index.php?lang=en">Reindex.net</ulink>
+ is a netbased library service offering all
+ traditional functions on a very high level plus many new
+ services. Reindex.net is a comprehensive and powerful WEB system
+ based on standards such as &xml; and &z3950;.
+ updates. Reindex supports &marc21;, dan&marc; eller Dublin Core with
+ UTF8-encoding.
+ </para>
+ <para>
+ Reindex.net runs on GNU/Debian Linux with &zebra; and Simpleserver
+ from Index
+ Data for bibliographic data. The relational database system
+ Sybase 9 &xml; is used for
+ administrative data.
+ Internally &marcxml; is used for bibliographical records. Update
+ utilizes &z3950; extended services.
+ </para>
+ </section>
+
+ <section id="dads-article-database">
+ <title>DADS - the DTV Article Database
+ Service</title>
+ <para>
DADS is a huge database of more than ten million records, totalling
over ten gigabytes of data. The records are metadata about academic
journal articles, primarily scientific; about 10% of these
</para>
<para>
More information can be found at
- <ulink url="http://www.dtv.dk/help/dads/index_e.htm"/>
+ <ulink url="http://www.dtv.dk/"/> and
+ <ulink url="http://dads.dtv.dk"/>
</para>
- </sect2>
+ </section>
- <sect2>
- <title>NLI-Z39.50 - a Natural Language Interface for Libraries</title>
+ <section id="infonet-eprints">
+ <title>Infonet Eprints</title>
<para>
- Fernuniversität Hagen in Germany have developed a natural
- language interface for access to library databases.
- <ulink url="http://ki212.fernuni-hagen.de/nli/NLIintro.html"/>
- In order to evaluate this interface for recall and precision, they
- chose Zebra as the basis for retrieval effectiveness. The Zebra
- server contains a copy of the GIRT database, consisting of more
- than 76000 records in SGML format (bibliographic records from
- social science), which are mapped to MARC for presentation.
- </para>
+ The InfoNet Eprints service from the
+ <ulink url="http://www.dtv.dk/">
+ Technical Knowledge Center of Denmark</ulink>
+ provides access to documents stored in
+ eprint/preprint servers and institutional research archives around
+ the world. The service is based on Open Archives Initiative metadata
+ harvesting of selected scientific archives around the world. These
+ open archives offer free and unrestricted access to their contents.
+ </para>
<para>
- (GIRT is the German Indexing and Retrieval Testdatabase. It is a
- standard German-language test database for intelligent indexing
- and retrieval systems. See
- <ulink url="http://www.gesis.org/forschung/informationstechnologie/clef-delos.htm"/>)
- </para>
- <para>
- Evaluation will take place as part of the TREC/CLEF campaign 2003
- <ulink url="http://clef.iei.pi.cnr.it or http://www4.eurospider.ch/CLEF/"/>
+ Infonet Eprints currently holds 1.4 million records from 16 archives.
+ The online search facility is found at
+ <ulink url="http://preprints.cvt.dk"/>.
</para>
+ </section>
+
+ <section id="alvis-project">
+ <title>Alvis</title>
<para>
- For more information, contact Johannes Leveling
- <email>Johannes.Leveling@FernUni-Hagen.De</email>
- </para>
- </sect2>
+ The <ulink url="http://www.alvis.info/alvis/">Alvis</ulink> EU
+ project run under the 6th Framework (IST-1-002068-STP)
+ is building a semantic-based peer-to-peer search engine. A
+ consortium of eleven partners from six different European
+ Community countries plus Switzerland and China contribute
+ with expertise in a broad range of specialties including network
+ topologies, routing algorithms, linguistic analysis and
+ bioinformatics.
+ </para>
+ <para>
+ The &zebra; information retrieval indexing machine is used inside
+ the Alvis framework to
+ manage huge collections of natural language processed and
+ enhanced &xml; data, coming from a topic relevant web crawl.
+ In this application, &zebra; swallows and manages 37GB of &xml; data
+ in about 4 hours, resulting in search times of fractions of
+ seconds.
+ </para>
+ </section>
+
- <sect2>
+ <section id="uls">
<title>ULS (Union List of Serials)</title>
<para>
- The M25-Link systems team
- (<ulink url="http://www.m25lib.ac.uk/M25link/"/>)
- are involved in a project called ULS to provide a union catalogue
- for periodicals in 21 member libraries. They do this with an
- unusual architecture which they call a
+ The M25 Systems Team
+ has created a union catalogue for the periodicals of the
+ twenty-one constituent libraries of the University of London and
+ the University of Westminster
+ (<ulink url="http://www.m25lib.ac.uk/ULS/"/>).
+ They have achieved this using an
+ unusual architecture, which they describe as a
``non-distributed virtual union catalogue''.
</para>
<para>
The member libraries send in data files representing their
periodicals, including both brief bibliographic data and summary
- holdings. Then 21 individual Z39.50 targets are created, each
- using Zebra, and all mounted on the single hardware server.
- The live service provides a web gateway allowing Z39.50 searching
- of all of the targets or a selection of them. Zebra's small
+ holdings. Then 21 individual &z3950; targets are created, each
+ using &zebra;, and all mounted on the single hardware server.
+ The live service provides a web gateway allowing &z3950; searching
+ of all of the targets or a selection of them. &zebra;'s small
footprint allows a relatively modest system to comfortably host
the 21 servers.
</para>
More information can be found at
<ulink url="http://www.m25lib.ac.uk/ULS/"/>
</para>
- </sect2>
+ </section>
- <sect2>
+ <section id="nli">
+ <title>NLI-&z3950; - a Natural Language Interface for Libraries</title>
+ <para>
+ Fernuniversität Hagen in Germany have developed a natural
+ language interface for access to library databases.
+ <!-- <ulink
+ url="http://ki212.fernuni-hagen.de/nli/NLIintro.html"/> -->
+ In order to evaluate this interface for recall and precision, they
+ chose &zebra; as the basis for retrieval effectiveness. The &zebra;
+ server contains a copy of the GIRT database, consisting of more
+ than 76000 records in &sgml; format (bibliographic records from
+ social science), which are mapped to &marc; for presentation.
+ </para>
+ <para>
+ (GIRT is the German Indexing and Retrieval Testdatabase. It is a
+ standard German-language test database for intelligent indexing
+ and retrieval systems. See
+ <ulink url="http://www.gesis.org/forschung/informationstechnologie/clef-delos.htm"/>)
+ </para>
+ <para>
+ Evaluation will take place as part of the TREC/CLEF campaign 2003
+ <ulink url="http://clef.iei.pi.cnr.it"/>.
+ <!-- or <ulink url="http://www4.eurospider.ch/CLEF/"/> -->
+ </para>
+ <para>
+ For more information, contact Johannes Leveling
+ <email>Johannes.Leveling@FernUni-Hagen.De</email>
+ </para>
+ </section>
+
+ <section id="various-web-indexes">
<title>Various web indexes</title>
<para>
- Zebra has been used by a variety of institutions to construct
+ &zebra; has been used by a variety of institutions to construct
indexes of large web sites, typically in the region of tens of
millions of pages. In this role, it functions somewhat similarly
to the engine of google or altavista, but for a selected intranet
For example, Liverpool University's web-search facility (see on
the home page at
<ulink url="http://www.liv.ac.uk/"/>
- and many sub-pages) works by relevance-searching a Zebra database
+ and many sub-pages) works by relevance-searching a &zebra; database
which is populated by the Harvest-NG web-crawling software.
</para>
<para>
- For more information, contact John Gilbertson
+ For more information on Liverpool university's intranet search
+ architecture, contact John Gilbertson
<email>jgilbert@liverpool.ac.uk</email>
</para>
- </sect2>
- </sect1>
-
-
- <sect1 id="support">
- <title>Support</title>
- <para>
- You can get support for Zebra from at least three sources.
- </para>
- <para>
- First, there's the Zebra web site at
- <ulink url="http://indexdata.dk/zebra/"/>,
- which always has the most recent version available for download.
- If you have a problem with Zebra, the first thing to do is see
- whether it's fixed in the current release.
- </para>
- <para>
- Second, there's the Zebra mailing list. Its home page at
- <ulink url="http://indexdata.dk/mailman/listinfo/zebralist"/>
- includes a complete archive of all messages that have ever been
- posted on the list. The Zebra mailing list is used both for
- announcements from the authors (new
- releases, bug fixes, etc.) and general discussion. You are welcome
- to seek support there. Join by sending email to
- <email>zebra-request@indexdata.dk</email>. Put the word
- <literal>subscribe</literal> in the body of the message.
- </para>
- <para>
- Third, it's possible to buy a commercial support contract, with
- well defined service levels and response times, from Index Data.
- See
- <ulink url="http://indexdata.dk/support2/"/>
- for details.
- </para>
- </sect1>
+ <para>
+ Kang-Jin Lee
+ has recently modified the Harvest web indexer to use &zebra; as
+ its native repository engine. His comments on the switch over
+ from the old engine are revealing:
+ <blockquote>
+ <para>
+ The first results after some testing with &zebra; are very
+ promising. The tests were done with around 220,000 SOIF files,
+ which occupies 1.6GB of disk space.
+ </para>
+ <para>
+ Building the index from scratch takes around one hour with &zebra;
+ where [old-engine] needs around five hours. While [old-engine]
+ blocks search requests when updating its index, &zebra; can still
+ answer search requests.
+ [...]
+ &zebra; supports incremental indexing which will speed up indexing
+ even further.
+ </para>
+ <para>
+ While the search time of [old-engine] varies from some seconds
+ to some minutes depending how expensive the query is, &zebra;
+ usually takes around one to three seconds, even for expensive
+ queries.
+ [...]
+ &zebra; can search more than 100 times faster than [old-engine]
+ and can process multiple search requests simultaneously
+ </para>
+ <para>
+ I am very happy to see such nice software available under GPL.
+ </para>
+ </blockquote>
+ </para>
+ </section>
+ </section>
+
+
+ <section id="introduction-support">
+ <title>Support</title>
+ <para>
+ You can get support for &zebra; from at least three sources.
+ </para>
+ <para>
+ First, there's the &zebra; web site at
+ <ulink url="&url.idzebra;"/>,
+ which always has the most recent version available for download.
+ If you have a problem with &zebra;, the first thing to do is see
+ whether it's fixed in the current release.
+ </para>
+ <para>
+ Second, there's the &zebra; mailing list. Its home page at
+ <ulink url="&url.idzebra.mailinglist;"/>
+ includes a complete archive of all messages that have ever been
+ posted on the list. The &zebra; mailing list is used both for
+ announcements from the authors (new
+ releases, bug fixes, etc.) and general discussion. You are welcome
+ to seek support there. Join by filling the form on the list home page.
+ </para>
+ <para>
+ Third, it's possible to buy a commercial support contract, with
+ well defined service levels and response times, from Index Data.
+ See
+ <ulink url="&url.indexdata.support;"/>
+ for details.
+ </para>
+ </section>
- <sect1 id="future">
+ <section id="future">
<title>Future Directions</title>
<para>
<listitem>
<para>
- Improved support for XML in search and retrieval. Eventually,
- the goal is for Zebra to pull double duty as a flexible
- information retrieval engine and high-performance XML
- repository.
- </para>
- <para>
- ### Partially done.
- </para>
- </listitem>
-
- <listitem>
- <para>
- Access to search engine through SOAP/RPC API to allow the
- construction of applications without requiring Z39.50 tools.
+ Improved support for &xml; in search and retrieval. Eventually,
+ the goal is for &zebra; to pull double duty as a flexible
+ information retrieval engine and high-performance &xml;
+ repository. The recent addition of XPath searching is one
+ example of the kind of enhancement we're working on.
</para>
<para>
- ### Partially done, thanks to the new SRW/Z39.50 gateway.
+ There is also the experimental <literal>ALVIS &xslt;</literal>
+ &xml; input filter, which unleashes the full power of &dom; based
+ &xslt; transformations during indexing and record retrieval. Work
+ on this filter has been sponsored by the ALVIS EU project
+ <ulink url="http://www.alvis.info/alvis/"/>. We expect this filter to
+ mature soon, as it is planned to be included in the version 2.0
+ release of &zebra;.
</para>
</listitem>
<listitem>
<para>
- Finalisation and documentation of Zebra's C programming
- API, allowing updates, database management and other functions
- not readily expressed in Z39.50. We will also consider
- exposing the API through SOAP.
+ Finalisation and documentation of &zebra;'s C programming
+ &api;, allowing updates, database management and other functions
+ not readily expressed in &z3950;. We will also consider
+ exposing the &api; through &soap;.
</para>
</listitem>
or check the contact info at the end of this manual.
</para>
- </sect1>
+ </section>
</chapter>
<!-- Keep this comment at the end of the file
Local variables:
projects / idzebra-moved-to-github.git / blobdiff