X-Git-Url: http://git.indexdata.com/?a=blobdiff_plain;f=doc%2Fintroduction.xml;h=29d759dc84706a10dfaa74e2ee2e8fbbb2e909ae;hb=dba1046a7e353d3ed66936aa90d8a5fb54a68a9c;hp=6553bc6c43e5c65bc6e44dbe3f903dea221cf006;hpb=5b903e946423cc2c91665853f8478b9d44d7088d;p=idzebra-moved-to-github.git diff --git a/doc/introduction.xml b/doc/introduction.xml index 6553bc6..29d759d 100644 --- a/doc/introduction.xml +++ b/doc/introduction.xml @@ -1,224 +1,592 @@ - + Introduction - +
Overview - - Zebra + Zebra is a high-performance, general-purpose structured text - indexing and retrieval engine. It reads structured records in a - variety of input formats (eg. email, XML, MARC) and allows access - to them through exact boolean search expressions and - relevance-ranked free-text queries. - + indexing and retrieval engine. It reads records in a + variety of input formats (eg. email, XML, MARC) and provides access + to them through a powerful combination of boolean search + expressions and relevance-ranked free-text queries. + - - Zebra supports large databases (more than ten gigabytes of data, - tens of millions of records). It supports safe, incremental - database updates on live systems. You can access data stored in - Zebra using a variety of Index Data tools (eg. YAZ and PHP/YAZ) as - well as commercial and freeware Z39.50 clients and toolkits. - + + Zebra supports large databases (tens of millions of records, + tens of gigabytes of data). It allows safe, incremental + database updates on live systems. Because Zebra supports + the industry-standard information retrieval protocol, Z39.50, + you can search Zebra databases using an enormous variety of + programs and toolkits, both commercial and free, which understand + this protocol. Application libraries are available to allow + bespoke clients to be written in Perl, C, C++, Java, Tcl, Visual + Basic, Python, PHP and more - see the + ZOOM web site + for more information on some of these client toolkits. + - This document is an introduction to the Zebra system. It will tell you - how to compile the software, and how to prepare your first database. - It also explains how the server can be configured to give you the + This document is an introduction to the Zebra system. It explains + how to compile the software, how to prepare your first database, + and how to configure the server to give you the functionality that you need. +
+ +
+ Zebra Features Overview - - - If you find the software interesting, you should visit the - - Zebra web site, where you can join the - - mailing-list - by sending email to - + + + Zebra Features Overview + + + + Feature + Availability + Notes + Reference + + + + + Boolean query language + CQL and RPN/PQF + The type-1 Reverse Polish Notation (RPN) + and it's textual representation Prefix Query Format (PQF) are + supported. The Common Query Language (CQL) can be configured as + a mapping from CQL to RPN/PQF + + + + + Operation types + Z39.50/SRU explain, search, and scan + + + + + Recursive boolean query tree + CQL and RPN/PQF + Both CQL and RPN/PQF allow atomic query parts (APT) to + be combined into complex boolean query trees + + + + Large databases + 64 file pointers assure that register files can extend + the 2 GB limit. Logical files can be + automatically partitioned over multiple disks, thus allowing for + large databases. + + + + + Complex semi-structured Documents + XML and GRS-1 Documents + Both XML and GRS-1 documents exhibit a DOM like internal + representation allowing for complex indexing and display rules + + + + Database updates + live, incremental updates + Robust updating - records can be added and deleted ``on the fly'' + without rebuilding the index from scratch. + Records can be safely updated even while users are accessing + the server. + The update procedure is tolerant to crashes or hard interrupts + during database updating - data can be reconstructed following + a crash. + + + + Input document formats + XML, SGML, Text, ISO2709 (MARC) + + A system of input filters driven by + regular expressions allows most ASCII-based + data formats to be easily processed. + SGML, XML, ISO2709 (MARC), and raw text are also + supported. + + + + Relevance ranking + TF-IDF like + Relevance-ranking of free-text queries is supported + using a TF-IDF like algorithm. + + + + Document storage + Index-only, Key storage, Document storage + Data can be, and usually is, imported + into Zebra's own storage, but Zebra can also refer to + external files, building and maintaining indexes of "live" + collections. + + + + Regular expression matching + Regexp + Full regular expression matching and "approximate + matching" (eg. spelling mistake corrections) are handled. + + + + Search truncation + + + + + + Remote update + Z39.50 extended services + + + + + Supported Platforms + UNIX, Linux, Windows (NT/2000/2003/XP) + Zebra is written in portable C, so it runs on most + Unix-like systems as well as Windows (NT/2000/2003/XP). Binary + distributions are + available for GNU/Debian Linux and Windows + + + + Z39.50 + Z39.50 protocol support + Protocol facilities: Init, Search, Present (retrieval), + Segmentation (support for very large records), Delete, Scan + (index browsing), Sort, Close and support for the ``update'' + Extended Service to add or replace an existing XML + record. Piggy-backed presents are honored in the search + request. Named result sets are supported. + + + + Record Syntaxes + + Multiple record syntaxes + for data retrieval: GRS-1, SUTRS, + XML, ISO2709 (MARC), etc. Records can be mapped between record syntaxes + and schemas on the fly. + + + + Web Service support + SRU GET/POST/SOAP + The protocol operations explain, + searchRetrieve and scan + are supported. CQL to internal + query model RPN conversion is supported. Extended RPN queries + for search/retrieve and scan are supported. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + - +
- - Features - +
+ References and Zebra based Applications - This is an overview of some of the most important features of the - system. + Zebra has been deployed in numerous applications, in both the + academic and commercial worlds, in application domains as diverse + as bibliographic catalogues, geospatial information, structured + vocabulary browsing, government information locators, civic + information systems, environmental observations, museum information + and web indexes. - - + Notable applications include the following: + - - - Supports large databases - files for indices, etc. can be - automatically partitioned over multiple disks. - - - - - Supports arbitrarily complex records - base input format is an - SGML-like syntax which allows nested (structured) data elements, as - well as variant forms of data. +
+ Koha free open-source ILS + + Koha is a full-featured + open-source ILS, initially developed in + New Zealand by Katipo Communications Ltd, and first deployed in + January of 2000 for Horowhenua Library Trust. It is currently + maintained by a team of software providers and library technology + staff from around the globe. + + + LibLime, + a company that is marketing and supporting Koha, adds in + the new release of Koha 3.0 the Zebra + database server to drive its bibliographic database. + + + In early 2005, the Koha project development team began looking at + ways to improve MARC support and overcome scalability limitations + in the Koha 2.x series. After extensive evaluations of the best + of the Open Source textual database engines - including MySQL + full-text searching, PostgreSQL, Lucene and Plucene - the team + selected Zebra. + + + "Zebra completely eliminates scalability limitations, because it + can support tens of millions of records." explained Joshua + Ferraro, LibLime's Technology President and Koha's Project + Release Manager. "Our performance tests showed search results in + under a second for databases with over 5 million records on a + modest i386 900Mhz test server." + + + "Zebra also includes support for true boolean search expressions + and relevance-ranked free-text queries, both of which the Koha + 2.x series lack. Zebra also supports incremental and safe + database updates, which allow on-the-fly record + management. Finally, since Zebra has at its heart the Z39.50 + protocol, it greatly improves Koha's support for that critical + library standard." + + + Although the bibliographic database will be moved to Zebra, Koha + 3.0 will continue to use a relational SQL-based database design + for the 'factual' database. "Relational database managers have + their strengths, in spite of their inability to handle large + numbers of bibliographic records efficiently," summed up Ferraro, + "We're taking the best from both worlds in our redesigned Koha + 3.0. - - - - Robust updating - records can be added and deleted without - rebuilding the index from scratch. - The update procedure is tolerant to crashes or hard interrupts - during register updating - registers can be reconstructed following - a crash. - Registers can be safely updated even while users are accessing - the server. + See also LibLime's newsletter article + + Koha Earns its Stripes. - +
- - - Supports random storage formats. A system of input filters driven by - regular expressions allows you to easily process most ASCII-based - data formats. SGML, XML, ISO2709 (MARC), and raw text are also - supported. - - +
+ Emilda open source ILS + + Emilda + is a complete Integrated Library System, released under the + GNU General Public License. It has a + full featured Web-OPAC, allowing comprehensive system management + from virtually any computer with an Internet connection, has + template based layout allowing anyone to alter the visual + appearance of Emilda, and is + XML based language for fast and easy portability to virtually any + language. + Currently, Emilda is used at three schools in Espoo, Finland. + + + As a surplus, 100% MARC compatibility has been achieved using the + Zebra Server from Index Data as backend server. + +
- - - Supports boolean queries as well as relevance-ranking (free-text) - searching. Right truncation and masking in terms are supported, as - well as full regular expressions. - - +
+ ReIndex.Net web based ILS + + Reindex.net + is a netbased library service offering all + traditional functions on a very high level plus many new + services. Reindex.net is a comprehensive and powerful WEB system + based on standards such as XML and Z39.50. + updates. Reindex supports MARC21, danMARC eller Dublin Core with + UTF8-encoding. + + + Reindex.net runs on GNU/Debian Linux with Zebra and Simpleserver + from Index + Data for bibliographic data. The relational database system + Sybase 9 XML is used for + administrative data. + Internally MARCXML is used for bibliographical records. Update + utilizes Z39.50 extended services. + +
- - - Can import the data into Zebras own storage, or just refer to - external files (good for building indexes of "live" - collections). - - +
+ DADS - the DTV Article Database + Service + + DADS is a huge database of more than ten million records, totalling + over ten gigabytes of data. The records are metadata about academic + journal articles, primarily scientific; about 10% of these + metadata records link to the full text of the articles they + describe, a body of about a terabyte of information (although the + full text is not indexed.) + + + It allows students and researchers at DTU (Danmarks Tekniske + Universitet, the Technical College of Denmark) to find and order + articles from multiple databases in a single query. The database + contains literature on all engineering subjects. It's available + on-line through a web gateway, though currently only to registered + users. + + + More information can be found at + and + + +
- - - Supports multiple concrete syntaxes - for record exchange (depending on the configuration): GRS-1, SUTRS, - XML, ISO2709 (*MARC). Records can be mapped between record syntaxes - and schema on the fly. - - +
+ Infonet Eprints + + The InfoNet Eprints service from the + + Technical Knowledge Center of Denmark + provides access to documents stored in + eprint/preprint servers and institutional research archives around + the world. The service is based on Open Archives Initiative metadata + harvesting of selected scientific archives around the world. These + open archives offer free and unrestricted access to their contents. + + + Infonet Eprints currently holds 1.4 million records from 16 archives. + The online search facility is found at + . + +
- - - Supports approximate matching in registers (ie. spelling mistakes, - etc). - - - - - - Zebra is written in portable C, so it runs on most Unix-like systems - as well as Windows NT - a binary distribution for Windows NT is available. - - - - - - - - - Z39.50 protocol support: - - - - - - - Protocol facilities: Init, Search, Retrieve, Delete, Browse and Sort. +
+ Alvis + + The Alvis EU + project run under the 6th Framework (IST-1-002068-STP) + is building a semantic-based peer-to-peer search engine. A + consortium of eleven partners from six different European + Community countries plus Switzerland and China contribute + with expertise in a broad range of specialties including network + topologies, routing algorithms, linguistic analysis and + bioinformatics. + + + The Zebra information retrieval indexing machine is used inside + the Alvis framework to + manage huge collections of natural language processed and + enhanced XML data, coming from a topic relevant web crawl. + In this application, Zebra swallows and manages 37GB of XML data + in about 4 hours, resulting in search times of fractions of + seconds. - +
- - - Piggy-backed presents are honored in the search-request. - - - +
+ ULS (Union List of Serials) + + The M25 Systems Team + has created a union catalogue for the periodicals of the + twenty-one constituent libraries of the University of London and + the University of Westminster + (). + They have achieved this using an + unusual architecture, which they describe as a + ``non-distributed virtual union catalogue''. + + + The member libraries send in data files representing their + periodicals, including both brief bibliographic data and summary + holdings. Then 21 individual Z39.50 targets are created, each + using Zebra, and all mounted on the single hardware server. + The live service provides a web gateway allowing Z39.50 searching + of all of the targets or a selection of them. Zebra's small + footprint allows a relatively modest system to comfortably host + the 21 servers. + + + More information can be found at + + +
+ +
+ NLI-Z39.50 - a Natural Language Interface for Libraries + + Fernuniversität Hagen in Germany have developed a natural + language interface for access to library databases. + + In order to evaluate this interface for recall and precision, they + chose Zebra as the basis for retrieval effectiveness. The Zebra + server contains a copy of the GIRT database, consisting of more + than 76000 records in SGML format (bibliographic records from + social science), which are mapped to MARC for presentation. + + + (GIRT is the German Indexing and Retrieval Testdatabase. It is a + standard German-language test database for intelligent indexing + and retrieval systems. See + ) + + + Evaluation will take place as part of the TREC/CLEF campaign 2003 + . + + + + For more information, contact Johannes Leveling + Johannes.Leveling@FernUni-Hagen.De + +
+ +
+ Various web indexes + + Zebra has been used by a variety of institutions to construct + indexes of large web sites, typically in the region of tens of + millions of pages. In this role, it functions somewhat similarly + to the engine of google or altavista, but for a selected intranet + or a subset of the whole Web. + + + For example, Liverpool University's web-search facility (see on + the home page at + + and many sub-pages) works by relevance-searching a Zebra database + which is populated by the Harvest-NG web-crawling software. + + + For more information on Liverpool university's intranet search + architecture, contact John Gilbertson + jgilbert@liverpool.ac.uk + + + Kang-Jin Lee + has recently modified the Harvest web indexer to use Zebra as + its native repository engine. His comments on the switch over + from the old engine are revealing: +
- Named result sets are supported. + The first results after some testing with Zebra are very + promising. The tests were done with around 220,000 SOIF files, + which occupies 1.6GB of disk space. - - - Easily configured to support different application profiles, with - tables for attribute sets, tag sets, and abstract syntaxes. - Additional tables control facilities such as element mappings to - different schema (eg., GILS-to-USMARC). + Building the index from scratch takes around one hour with Zebra + where [old-engine] needs around five hours. While [old-engine] + blocks search requests when updating its index, Zebra can still + answer search requests. + [...] + Zebra supports incremental indexing which will speed up indexing + even further. - - - - Complex composition specifications using Espec-1 are partially - supported (simple element requests only). + While the search time of [old-engine] varies from some seconds + to some minutes depending how expensive the query is, Zebra + usually takes around one to three seconds, even for expensive + queries. + [...] + Zebra can search more than 100 times faster than [old-engine] + and can process multiple search requests simultaneously - - - - Element Set Names are defined using the Espec-1 capability of the - system, and are given in configuration files as simple element - requests (and possibly variant requests). + I am very happy to see such nice software available under GPL. - - - - - +
+ +
+
- - - - Applications - - Zebra has been deployed in numerous applications, in both the - academic and commercial worlds, in application domains as diverse - as bibliographic information, geospatial, ### (Help, guys!) - - - Notable applications include the following: - - - - DADS - the DTV Article Database Service + +
+ Support + + You can get support for Zebra from at least three sources. + - DADS is a huge database of ### records, allowing students and - researchers at DTU (###) to search and order articles from several - different databases at once. The database contains - literature on all engineering subjects. It's available on-line - through a web gateway at - http://www.dtv.dk/search/index_e.htm - though only to members of the university. + First, there's the Zebra web site at + , + which always has the most recent version available for download. + If you have a problem with Zebra, the first thing to do is see + whether it's fixed in the current release. - ### Much more information needed. + Second, there's the Zebra mailing list. Its home page at + + includes a complete archive of all messages that have ever been + posted on the list. The Zebra mailing list is used both for + announcements from the authors (new + releases, bug fixes, etc.) and general discussion. You are welcome + to seek support there. Join by filling the form on the list home page. - - + + Third, it's possible to buy a commercial support contract, with + well defined service levels and response times, from Index Data. + See + + for details. + +
- - Future Work + +
+ Future Directions These are some of the plans that we have for the software in the near - and far future, approximately ordered after their relative importance. + and far future, ordered approximately as we expect to work on them. @@ -229,22 +597,26 @@ Improved support for XML in search and retrieval. Eventually, the goal is for Zebra to pull double duty as a flexible information retrieval engine and high-performance XML - repository. + repository. The recent addition of XPath searching is one + example of the kind of enhancement we're working on. - - - - Access to search engine through SOAP/RPC API to allow the - construction of applications without requiring Z39.50 tools. + There is also the experimental ALVIS XSLT + XML input filter, which unleashes the full power of DOM based + XSLT transformations during indexing and record retrieval. Work + on this filter has been sponsored by the ALVIS EU project + . We expect this filter to + mature soon, as it is planned to be included in the version 2.0 + release of Zebra. - Finalisation, documentation of the Zebra API. Consider - exposing the API through SOAP as well (allowing updates, - database management). + Finalisation and documentation of Zebra's C programming + API, allowing updates, database management and other functions + not readily expressed in Z39.50. We will also consider + exposing the API through SOAP. @@ -263,12 +635,17 @@ Programmers thrive on user feedback. If you are interested in a facility that you don't see mentioned here, or if there's something - you think we could do better, please drop us a mail. + you think we could do better, please drop us a mail. Better still, + implement it and send us the patches. + + If you think it's all really neat, you're welcome to drop us a line - saying that, too. You'll find contact info at the end of this file. + saying that, too. You can email us on + info@indexdata.dk + or check the contact info at the end of this manual. - +