X-Git-Url: http://git.indexdata.com/?a=blobdiff_plain;f=doc%2Fintroduction.xml;h=d93603c58421e35fc5f58f94b921c143d807592b;hb=518c06f68ffac6658aa792da45282a165b32ca95;hp=06f69600873187a848295c589291effb2483a347;hpb=9cbe7b17c2f2e9d664111e83edf2a8a9c09b4b38;p=idzebra-moved-to-github.git diff --git a/doc/introduction.xml b/doc/introduction.xml index 06f6960..d93603c 100644 --- a/doc/introduction.xml +++ b/doc/introduction.xml @@ -1,54 +1,47 @@ - + Introduction Overview - The Zebra - server is a high-performance, general-purpose structured text + is a high-performance, general-purpose structured text indexing and retrieval engine. It reads structured records in a - variety of input formats (eg. email, XML, MARC) and allows access - to them through exact boolean search expressions and - relevance-ranked free-text queries. - - - - Zebra supports large databases (more than ten gigabytes of data, - tens of millions of records). It supports incremental, safe - database updates on live systems. You can access data stored in - Zebra using a variety of Index Data tools (eg. YAZ and PHP/YAZ) as - well as commercial and freeware Z39.50 clients and toolkits. - + variety of input formats (eg. email, XML, MARC) and provides access + to them through a powerful combination of boolean search + expressions and relevance-ranked free-text queries. + - This document is an introduction to the Zebra system. It will tell you - how to compile the software, and how to prepare your first database. - It also explains how the server can be configured to give you the - functionality that you need. + Zebra supports large databases (tens of millions of records, + tens of gigabytes of data). It allows safe, incremental + database updates on live systems. Because Zebra supports + the industry-standard information retrieval protocol, Z39.50, + you can search Zebra databases using an enormous variety of + programs and toolkits, both commercial and free, which understand + this protocol. Application libraries are available to allow + bespoke clients to be written in Perl, C, C++, Java, Tcl, Visual + Basic, Python, PHP and more - see + the ZOOM web site + for more information on some of these client toolkits. - + - - If you find the software interesting, you should visit the - - Zebra web site, where you can join the - - mailing-list - by sending email to + This document is an introduction to the Zebra system. It explains + how to compile the software, how to prepare your first database, + and how to configure the server to give you the + functionality that you need. - Features - This is an overview of some of the most important features of the - system. + This is an overview of some of Zebra's most important features: @@ -56,76 +49,67 @@ - Supports large databases - files for indices, etc. can be + Very large databases: files for indexes, etc. can be automatically partitioned over multiple disks. - Supports arbitrarily complex records - base input format is an - SGML-like syntax which allows nested (structured) data elements, as - well as variant forms of data. + Arbitrarily complex records. The internal data format + is an structured format conceptually similar to XML or GRS-1, + which allows lists, nested structured data elements and + variant forms of data. - Robust updating - records can be added and deleted without - rebuilding the index from scratch. + Robust updating - records can be added and deleted ``on the fly'' + without rebuilding the index from scratch. + Records can be safely updated even while users are accessing + the server. The update procedure is tolerant to crashes or hard interrupts - during register updating - registers can be reconstructed following + during database updating - data can be reconstructed following a crash. - Registers can be safely updated even while users are accessing - the server. - Supports random storage formats. A system of input filters driven by - regular expressions allows you to easily process most ASCII-based - data formats. SGML, XML, ISO2709 (MARC), and raw text are also + Configurable to understand many input formats. + A system of input filters driven by + regular expressions allows most ASCII-based + data formats to be easily processed. + SGML, XML, ISO2709 (MARC), and raw text are also supported. - Supports boolean queries as well as relevance-ranking (free-text) - searching. Right truncation and masking in terms are supported, as - well as full regular expressions. + Searching supports a powerful combination of boolean queries as + well as relevance-ranking (free-text) queries. Truncation, + masking, full regular expression matching and "approximate + matching" (eg. spelling mistakes) are all handled. - Can import the data into Zebras own storage, or just refer to - external files (good for building indexes of "live" - collections). + Index-only databases: data can be, and usually is, imported + into Zebra's own storage, but Zebra can also refer to + external files, building and maintaining indexes of "live" + collections. - Supports multiple concrete syntaxes - for record exchange (depending on the configuration): GRS-1, SUTRS, - XML, ISO2709 (*MARC). Records can be mapped between record syntaxes - and schema on the fly. - - - - - - Supports approximate matching in registers (ie. spelling mistakes, - etc). - - - - - Zebra is written in portable C, so it runs on most Unix-like systems - as well as Windows NT - a binary distribution for Windows NT is available. + as well as Windows NT. A binary distribution for Windows NT is + available at + @@ -141,13 +125,18 @@ - Protocol facilities: Init, Search, Retrieve, Delete, Browse and Sort. + Protocol facilities: Init, Search, Present (retrieval), + Segmentation (support for very large records), Delete, Scan + (index browsing), Sort, Close and some Extended Services. - Piggy-backed presents are honored in the search-request. + Piggy-backed presents are honored in the search request - that + is, a subset of the found records can be returned directly with + a search response, enabling search and retrieval to happen in a + single round-trip. @@ -156,6 +145,7 @@ Named result sets are supported. + Easily configured to support different application profiles, with @@ -167,16 +157,19 @@ - Complex composition specifications using Espec-1 are partially - supported (simple element requests only). + Complex composition specifications using Espec-1 (partial support). + Element sets are defined using the Espec-1 capability, + and are specified in configuration files as simple element + requests (and, optionally, variant requests). - Element Set Names are defined using the Espec-1 capability of the - system, and are given in configuration files as simple element - requests (and possibly variant requests). + Multiple record syntaxes + for data retrieval: GRS-1, SUTRS, + XML, ISO2709 (MARC), etc. Records can be mapped between record syntaxes + and schemas on the fly. @@ -186,12 +179,163 @@ + + Applications + + Zebra has been deployed in numerous applications, in both the + academic and commercial worlds, in application domains as diverse + as bibliographic catalogues, geospatial information, structured + vocabulary browsing, government information locators, civic + information systems, environmental observations, museum information + and web indexes. + + + Notable applications include the following: + + + + DADS - the DTV Article Database Service + + DADS is a huge database of more than ten million records, totalling + over ten gigabytes of data. The records are metadata about academic + journal articles, primarily scientific; about 10% of these + metadata records link to the full text of the articles they + describe, a body of about a terabyte of information (although the + full text is not indexed.) + + + It allows students and researchers at DTU (Danmarks Tekniske + Universitet, the Technical College of Denmark) to find and order + articles from multiple databases in a single query. The database + contains literature on all engineering subjects. It's available + on-line through a web gateway, though currently only to registered + users. + + + More information can be found at + + + + + + NLI-Z39.50 - a Natural Language Interface for Libraries + + Fernuniversität Hagen in Germany have developed a natural + language interface for access to library databases. + + In order to evaluate this interface for recall and precision, they + chose Zebra as the basis for retrieval effectiveness. The Zebra + server contains a copy of the GIRT database, consisting of more + than 76000 records in SGML format (bibliographic records from + social science), which are mapped to MARC for presentation. + + + (GIRT is the German Indexing and Retrieval Testdatabase. It is a + standard German-language test database for intelligent indexing + and retrieval systems. See + ) + + + Evaluation will take place as part of the TREC/CLEF campaign 2003 + + + + For more information, contact Johannes Leveling + Johannes.Leveling@FernUni-Hagen.De + + + + + ULS (Union List of Serials) + + The M25-Link systems team + () + are involved in a project called ULS to provide a union catalogue + for periodicals in 21 member libraries. They do this with an + unusual architecture which they call a + ``non-distributed virtual union catalogue''. + + + The member libraries send in data files representing their + periodicals, including both brief bibliographic data and summary + holdings. Then 21 individual Z39.50 targets are created, each + using Zebra, and all mounted on the single hardware server. + The live service provides a web gateway allowing Z39.50 searching + of all of the targets or a selection of them. Zebra's small + footprint allows a relatively modest system to comfortably host + the 21 servers. + + + More information can be found at + + + + + + Various web indexes + + Zebra has been used by a variety of institutions to construct + indexes of large web sites, typically in the region of tens of + millions of pages. In this role, it functions somewhat similarly + to the engine of google or altavista, but for a selected intranet + or a subset of the whole Web. + + + For example, Liverpool University's web-search facility (see on + the home page at + + and many sub-pages) works by relevance-searching a Zebra database + which is populated by the Harvest-NG web-crawling software. + + + For more information, contact John Gilbertson + jgilbert@liverpool.ac.uk + + + + + + + Support + + You can get support for Zebra from at least three sources. + + + First, there's the Zebra web site at + , + which always has the most recent version available for download. + If you have a problem with Zebra, the first thing to do is see + whether it's fixed in the current release. + + + Second, there's the Zebra mailing list. Its home page at + + includes a complete archive of all messages that have ever been + posted on the list. The Zebra mailing list is used both for + announcements from the authors (new + releases, bug fixes, etc.) and general discussion. You are welcome + to seek support there. Join by sending email to + zebra-request@indexdata.dk. Put the word 'subscribe' + in the body of the message. + + + + Third, it's possible to buy a commercial support contract, with + well defined service levels and response times, from Index Data. + See + + + for details. + + + + - Future Work + Future Directions These are some of the plans that we have for the software in the near - and far future, approximately ordered after their relative importance. + and far future, ordered approximately as we expect to work on them. @@ -204,6 +348,9 @@ information retrieval engine and high-performance XML repository. + + ### Partially done. + @@ -211,13 +358,17 @@ Access to search engine through SOAP/RPC API to allow the construction of applications without requiring Z39.50 tools. + + ### Partially done, thanks to the new SRW/Z39.50 gateway. + - Finalisation, documentation of the Zebra API. Consider - exposing the API through SOAP as well (allowing updates, - database management). + Finalisation and documentation of Zebra's C programming + API, allowing updates, database management and other functions + not readily expressed in Z39.50. We will also consider + exposing the API through SOAP. @@ -236,9 +387,14 @@ Programmers thrive on user feedback. If you are interested in a facility that you don't see mentioned here, or if there's something - you think we could do better, please drop us a mail. + you think we could do better, please drop us a mail. Better still, + implement it and send us the patches. + + If you think it's all really neat, you're welcome to drop us a line - saying that, too. You'll find contact info at the end of this file. + saying that, too. You can email us on + info@indexdata.dk + or check the contact info at the end of this manual.