X-Git-Url: http://git.indexdata.com/?a=blobdiff_plain;f=doc%2Fintroduction.xml;h=03401df0eeea89962fa1ddb5757eb71e0d911527;hb=2b17f33968c38dfe50b7710da1c9e5004da9cf8b;hp=2175218889eaa6911e295dc2ea81d4d7b7b975aa;hpb=7d77cebae2b7af01eb7211f4ca9860217b3d32cb;p=idzebra-moved-to-github.git diff --git a/doc/introduction.xml b/doc/introduction.xml index 2175218..03401df 100644 --- a/doc/introduction.xml +++ b/doc/introduction.xml @@ -1,39 +1,49 @@ - + Introduction Overview - The Zebra system is a fielded free-text indexing and retrieval engine with a - Z39.50 front-end. You can use any commercial or free-ware Z39.50 client - to access data stored in Zebra. + + Zebra + is a high-performance, general-purpose structured text + indexing and retrieval engine. It reads structured records in a + variety of input formats (eg. email, XML, MARC) and provides access + to them through a powerful combination of boolean search + expressions and relevance-ranked free-text queries. - + - The Zebra server is our first step towards the development of a fully - configurable, open information system. Eventually, it will be paired - off with a powerful Z39.50 client to support complex information - management tasks within almost any application domain. We're making - the server available now because it's no fun to be in the open - information retrieval business all by yourself. We want to allow - people with interesting data to make their things - available in interesting ways, without having to start out - by implementing yet another protocol stack from scratch. + Zebra supports large databases (tens of millions of records, + tens of gigabytes of data). It allows safe, incremental + database updates on live systems. Because Zebra supports + the industry-standard information retrieval protocol, Z39.50, + you can search Zebra databases using an enormous variety of + programs and toolkits, both commercial and free, which understand + this protocol. Application libraries are available to allow + bespoke clients to be written in Perl, C, C++, Java, Tcl, Visual + Basic, Python, PHP and more - see + the ZOOM web site + for more information on some of these client toolkits. - + - This document is an introduction to the Zebra system. It will tell you - how to compile the software, and how to prepare your first database. - It also explains how the server can be configured to give you the + This document is an introduction to the Zebra system. It explains + how to compile the software, how to prepare your first database, + and how to configure the server to give you the functionality that you need. - If you find the software interesting, you should join the support - mailing-list by sending email to - zebra-request@indexdata.dk. + If you use Zebra, you should visit its + web site, + where you can join the + + mailing-list + by sending email to + ### zebra-subscribe@mailman.indexdata.dk @@ -42,42 +52,44 @@ Features - This is a list of some of the most important features of the - system. + This is an overview of some of Zebra's most important features: + - Supports updating - records can be added and deleted without - rebuilding the index from scratch. - The update procedure is tolerant to crashes or hard interrupts - during register updating - registers can be reconstructed following - a crash. - Registers can be safely updated even while users are accessing - the server. + Very large databases: files for indexes, etc. can be + automatically partitioned over multiple disks. - Supports large databases - files for indices, etc. can be - automatically partitioned over multiple disks. + Arbitrarily complex records. The internal data format + is an structured format conceptually similar to XML or GRS-1, + which allows nested structured data elements and + variant forms of data. - Supports arbitrarily complex records - base input format is an - SGML-like syntax which allows nested (structured) data elements, as - well as variant forms of data. + Robust updating - records can be added and deleted ``on the fly'' + without rebuilding the index from scratch. + Records can be safely updated even while users are accessing + the server. + The update procedure is tolerant to crashes or hard interrupts + during database updating - data can be reconstructed following + a crash. - Supports random storage formats. A system of input filters driven by + Configurable to understand many input formats. + A system of input filters driven by regular expressions allows you to easily process most ASCII-based data formats. SGML, XML, ISO2709 (MARC), and raw text are also supported. @@ -86,25 +98,27 @@ - Supports boolean queries as well as relevance-ranking (free-text) - searching. Right truncation and masking in terms are supported, as - well as full regular expressions. + Searching supports a powerful combination of boolean queries as + well as relevance-ranking (free-text) queries. Truncation, + masking, full regular expression matching and "approximate + matching" (eg. spelling mistakes) are all supported. - - Supports multiple concrete syntaxes - for record exchange (depending on the configuration): GRS-1, SUTRS, - XML, ISO2709 (*MARC). Records can be mapped between record syntaxes - and schema on the fly. - + + Index-only databases: data can be, and usually is, imported + into Zebra's own storage, but Zebra can also refer to + external files, building and maintaining indexes of "live" + collections. + - + - Supports approximate matching in registers (ie. spelling mistakes, - etc). + Zebra is written in portable C, so it runs on most Unix-like systems + as well as Windows NT. A binary distribution for Windows NT is + available. @@ -113,14 +127,15 @@ - Protocol support: + Z39.50 protocol support: - Protocol facilities: Init, Search, Retrieve, Delete, Browse and Sort. + Protocol facilities: Init, Search, Present (retrieval), Delete, + Scan (index browsing) and Sort. @@ -135,6 +150,7 @@ Named result sets are supported. + Easily configured to support different application profiles, with @@ -146,105 +162,192 @@ - Complex composition specifications using Espec-1 are partially - supported (simple element requests only). - - - - - - Element Set Names are defined using the Espec-1 capability of the - system, and are given in configuration files as simple element - requests (and possibly variant requests). + Complex composition specifications using Espec-1 (partial support). + Element sets are defined using the Espec-1 capability, + and are specified in configuration files as simple element + requests (and, optionally, variant requests). - Some variant support (not fully implemented yet). + Multiple record syntaxes + for data retrieval: GRS-1, SUTRS, + XML, ISO2709 (MARC), etc. Records can be mapped between record syntaxes + and schemas on the fly. - - - Zebra runs on most Unix-like systems as well as Windows NT - a binary - distribution for Windows NT is available. - - - + + Applications + + Zebra has been deployed in numerous applications, in both the + academic and commercial worlds, in application domains as diverse + as bibliographic catalogues, geospatial information, structured + vocabulary browsing, government information locators, civic + information systems, environmental observations, museum information + and web indexes. + + + Notable applications include the following: + + + + DADS - the DTV Article Database Service + + DADS is a huge database of more than ten million records, totalling + over ten gigabytes of data. The records are metadata about academic + journal articles, primarily scientific; about 10% of these + metadata records link to the full text of the articles they + describe, a body of about a terabyte of information (although the + full text is not indexed.) + + + It allows students and researchers at DTU (Danmarks Tekniske + Universitet, the Technical College of Denmark) to find and order + articles from multiple databases in a single query. The database + contains literature on all engineering subjects. It's available + on-line through a web gateway, though currently only to registered + users. + + + More information can be found at + + + + + + + + Various web indexes + + Zebra has been used by a variety of institutions to construct + indexes of large web sites, typically in the region of tens of + millions of pages. In this role, it functions somewhat similarly + to the engine of google or altavista, but for a selected intranet + or subset of the whole Web. + + + ### examples, details and numbers, please! + + + + - Future Work + Future Directions These are some of the plans that we have for the software in the near - and far future, approximately ordered after their relative importance. - Items marked with an - asterisk will be implemented before the - last beta release. + and far future, ordered approximately as we expect to work on them. - - - *Complete the support for variants. - - - - - - *Finalize the data element include facility - to support multimedia data elements in records. - - - Add more sophisticated relevance ranking mechanisms. - Add support for soundex and stemming. - Add relevance feedback support. + Improved support for XML in search and retrieval. Eventually, + the goal is for Zebra to pull double duty as a flexible + information retrieval engine and high-performance XML + repository. - Complete EXPLAIN support. + Access to search engine through SOAP/RPC API to allow the + construction of applications without requiring Z39.50 tools. - Add support for very large records by implementing segmentation and/or - variant pieces. + Finalisation and documentation of Zebra's C programming + API, allowing updates, database management and other functions + not readily expressed in Z39.50. We will also consider + exposing the API through SOAP. - Support the Item Update extended service of the protocol. + Improved free-text searching. We're first and foremost octet jockeys and + we're actively looking for organisations or people who'd like + to contribute experience in relevance ranking and text + searching. - - - We want to add a management system that allows you to - control your databases and configuration tables from a graphical - interface. - - Programmers thrive on user feedback. If you are interested in a facility that you don't see mentioned here, or if there's something - you think we could do better, please drop us a mail. + you think we could do better, please drop us a mail. Better still, + implement it and send us the patches. + + If you think it's all really neat, you're welcome to drop us a line saying that, too. You'll find contact info at the end of this file.