X-Git-Url: http://git.indexdata.com/?a=blobdiff_plain;f=doc%2Fintroduction.xml;h=ae5df92f0e29d397e2858a09a93dfcc6d778af70;hb=a4d1fff3d8575f7f3db84630cc408fdd8d051f8f;hp=d93603c58421e35fc5f58f94b921c143d807592b;hpb=518c06f68ffac6658aa792da45282a165b32ca95;p=idzebra-moved-to-github.git diff --git a/doc/introduction.xml b/doc/introduction.xml index d93603c..ae5df92 100644 --- a/doc/introduction.xml +++ b/doc/introduction.xml @@ -1,15 +1,14 @@ - + Introduction - +
Overview - - Zebra + Zebra is a high-performance, general-purpose structured text - indexing and retrieval engine. It reads structured records in a + indexing and retrieval engine. It reads records in a variety of input formats (eg. email, XML, MARC) and provides access to them through a powerful combination of boolean search expressions and relevance-ranked free-text queries. @@ -24,8 +23,8 @@ programs and toolkits, both commercial and free, which understand this protocol. Application libraries are available to allow bespoke clients to be written in Perl, C, C++, Java, Tcl, Visual - Basic, Python, PHP and more - see - the ZOOM web site + Basic, Python, PHP and more - see the + ZOOM web site for more information on some of these client toolkits. @@ -35,9 +34,9 @@ and how to configure the server to give you the functionality that you need. - +
- +
Features @@ -49,7 +48,7 @@ - Very large databases: files for indexes, etc. can be + Very large databases: logical files can be automatically partitioned over multiple disks. @@ -57,7 +56,7 @@ Arbitrarily complex records. The internal data format - is an structured format conceptually similar to XML or GRS-1, + is a structured format conceptually similar to XML or GRS-1, which allows lists, nested structured data elements and variant forms of data. @@ -107,9 +106,18 @@ Zebra is written in portable C, so it runs on most Unix-like systems - as well as Windows NT. A binary distribution for Windows NT is - available at - + as well as Windows (NT/2000/2003). A binary distribution for Windows + is available at + , + and pre-built packages are available for + + GNU/Debian Linux at + . @@ -118,7 +126,7 @@ - Z39.50 protocol support: + Z39.50 protocol support: @@ -127,7 +135,8 @@ Protocol facilities: Init, Search, Present (retrieval), Segmentation (support for very large records), Delete, Scan - (index browsing), Sort, Close and some Extended Services. + (index browsing), Sort, Close and support for the ``update'' + Extended Service to add or replace an existing XML record. @@ -176,11 +185,52 @@ + - + + SRU Web Service support: + + + + + + The protocol operations explain, + searchRetrieve and scan + are supported. + + + + + CQL to internal query model RPN + conversion is supported. + + + + + Multiple XML record formats + for data retrieval are supported, modelled over the GRS-1, SUTRS, + MARC record formats. Records can be mapped between record + schemas on the fly. Arbitrarily complex XSLT transformations + can be applied during record retrieval if one uses the + alvis filter module. + + + + + Extended RPN queries for search/retrieve and scan are supported, + for controlling approximate hit counts, etc. + + + + + + + + +
- - Applications +
+ References and Zebra based Applications Zebra has been deployed in numerous applications, in both the academic and commercial worlds, in application domains as diverse @@ -193,9 +243,110 @@ Notable applications include the following: - - DADS - the DTV Article Database Service + +
+ Koha free open-source ILS + Koha is a full-featured + open-source ILS, initially developed in + New Zealand by Katipo Communications Ltd, and first deployed in + January of 2000 for Horowhenua Library Trust. It is currently + maintained by a team of software providers and library technology + staff from around the globe. + + + LibLime, + a company that is marketing and supporting Koha, adds in + the new release of Koha 3.0 the Zebra + database server to drive its bibliographic database. + + + In early 2005, the Koha project development team began looking at + ways to improve MARC support and overcome scalability limitations + in the Koha 2.x series. After extensive evaluations of the best + of the Open Source textual database engines - including MySQL + full-text searching, PostgreSQL, Lucene and Plucene - the team + selected Zebra. + + + "Zebra completely eliminates scalability limitations, because it + can support tens of millions of records." explained Joshua + Ferraro, LibLime's Technology President and Koha's Project + Release Manager. "Our performance tests showed search results in + under a second for databases with over 5 million records on a + modest i386 900Mhz test server." + + + "Zebra also includes support for true boolean search expressions + and relevance-ranked free-text queries, both of which the Koha + 2.x series lack. Zebra also supports incremental and safe + database updates, which allow on-the-fly record + management. Finally, since Zebra has at its heart the Z39.50 + protocol, it greatly improves Koha's support for that critical + library standard." + + + Although the bibliographic database will be moved to Zebra, Koha + 3.0 will continue to use a relational SQL-based database design + for the 'factual' database. "Relational database managers have + their strengths, in spite of their inability to handle large + numbers of bibliographic records efficiently," summed up Ferraro, + "We're taking the best from both worlds in our redesigned Koha + 3.0. + + + See also LibLime's newsletter article + + Koha Earns its Stripes. + +
+ +
+ Emilda open source ILS + + Emilda + is a complete Integrated Library System, released under the + GNU General Public License. It has a + full featured Web-OPAC, allowing comprehensive system management + from virtually any computer with an Internet connection, has + template based layout allowing anyone to alter the visual + appearance of Emilda, and is + XML based language for fast and easy portability to virtually any + language. + Currently, Emilda is used at three schools in Espoo, Finland. + + + As a surplus, 100% MARC compatibility has been achieved using the + Zebra Server from Index Data as backend server. + +
+ +
+ ReIndex.Net web based ILS + + Reindex.net + is a netbased library service offering all + traditional functions on a very high level plus many new + services. Reindex.net is a comprehensive and powerful WEB system + based on standards such as XML and Z39.50. + updates. Reindex supports MARC21, danMARC eller Dublin Core with + UTF8-encoding. + + + Reindex.net runs on GNU/Debian Linux with Zebra and Simpleserver + from Index + Data for bibliographic data. The relational database system + Sybase 9 XML is used for + administrative data. + Internally MARCXML is used for bibliographical records. Update + utilizes Z39.50 extended services. + +
+ +
+ DADS - the DTV Article Database + Service + DADS is a huge database of more than ten million records, totalling over ten gigabytes of data. The records are metadata about academic journal articles, primarily scientific; about 10% of these @@ -213,46 +364,64 @@ More information can be found at - + and + - +
- - NLI-Z39.50 - a Natural Language Interface for Libraries - - Fernuniversität Hagen in Germany have developed a natural - language interface for access to library databases. - - In order to evaluate this interface for recall and precision, they - chose Zebra as the basis for retrieval effectiveness. The Zebra - server contains a copy of the GIRT database, consisting of more - than 76000 records in SGML format (bibliographic records from - social science), which are mapped to MARC for presentation. - +
+ Infonet Eprints - (GIRT is the German Indexing and Retrieval Testdatabase. It is a - standard German-language test database for intelligent indexing - and retrieval systems. See - ) - + The InfoNet Eprints service from the + + Technical Knowledge Center of Denmark + provides access to documents stored in + eprint/preprint servers and institutional research archives around + the world. The service is based on Open Archives Initiative metadata + harvesting of selected scientific archives around the world. These + open archives offer free and unrestricted access to their contents. + - Evaluation will take place as part of the TREC/CLEF campaign 2003 - + Infonet Eprints currently holds 1.4 million records from 16 archives. + The online search facility is found at + . +
+ +
+ Alvis - For more information, contact Johannes Leveling - Johannes.Leveling@FernUni-Hagen.De - - + The Alvis EU + project run under the 6th Framework (IST-1-002068-STP) + is building a semantic-based peer-to-peer search engine. A + consortium of eleven partners from six different European + Community countries plus Switzerland and China contribute + with expertise in a broad range of specialties including network + topologies, routing algorithms, linguistic analysis and + bioinformatics. + + + The Zebra information retrieval indexing machine is used inside + the Alvis framework to + manage huge collections of natural language processed and + enhanced XML data, coming from a topic relevant web crawl. + In this application, Zebra swallows and manages 37GB of XML data + in about 4 hours, resulting in search times of fractions of + seconds. + +
+ - +
ULS (Union List of Serials) - The M25-Link systems team - () - are involved in a project called ULS to provide a union catalogue - for periodicals in 21 member libraries. They do this with an - unusual architecture which they call a + The M25 Systems Team + has created a union catalogue for the periodicals of the + twenty-one constituent libraries of the University of London and + the University of Westminster + (). + They have achieved this using an + unusual architecture, which they describe as a ``non-distributed virtual union catalogue''. @@ -269,9 +438,39 @@ More information can be found at - +
+ +
+ NLI-Z39.50 - a Natural Language Interface for Libraries + + Fernuniversität Hagen in Germany have developed a natural + language interface for access to library databases. + + In order to evaluate this interface for recall and precision, they + chose Zebra as the basis for retrieval effectiveness. The Zebra + server contains a copy of the GIRT database, consisting of more + than 76000 records in SGML format (bibliographic records from + social science), which are mapped to MARC for presentation. + + + (GIRT is the German Indexing and Retrieval Testdatabase. It is a + standard German-language test database for intelligent indexing + and retrieval systems. See + ) + + + Evaluation will take place as part of the TREC/CLEF campaign 2003 + . + + + + For more information, contact Johannes Leveling + Johannes.Leveling@FernUni-Hagen.De + +
- +
Various web indexes Zebra has been used by a variety of institutions to construct @@ -288,49 +487,80 @@ which is populated by the Harvest-NG web-crawling software. - For more information, contact John Gilbertson + For more information on Liverpool university's intranet search + architecture, contact John Gilbertson jgilbert@liverpool.ac.uk - - - - - - Support - - You can get support for Zebra from at least three sources. - - - First, there's the Zebra web site at - , - which always has the most recent version available for download. - If you have a problem with Zebra, the first thing to do is see - whether it's fixed in the current release. - - - Second, there's the Zebra mailing list. Its home page at - - includes a complete archive of all messages that have ever been - posted on the list. The Zebra mailing list is used both for - announcements from the authors (new - releases, bug fixes, etc.) and general discussion. You are welcome - to seek support there. Join by sending email to - zebra-request@indexdata.dk. Put the word 'subscribe' - in the body of the message. - - - - Third, it's possible to buy a commercial support contract, with - well defined service levels and response times, from Index Data. - See - - - for details. - - + + Kang-Jin Lee + has recently modified the Harvest web indexer to use Zebra as + its native repository engine. His comments on the switch over + from the old engine are revealing: +
+ + The first results after some testing with Zebra are very + promising. The tests were done with around 220,000 SOIF files, + which occupies 1.6GB of disk space. + + + Building the index from scratch takes around one hour with Zebra + where [old-engine] needs around five hours. While [old-engine] + blocks search requests when updating its index, Zebra can still + answer search requests. + [...] + Zebra supports incremental indexing which will speed up indexing + even further. + + + While the search time of [old-engine] varies from some seconds + to some minutes depending how expensive the query is, Zebra + usually takes around one to three seconds, even for expensive + queries. + [...] + Zebra can search more than 100 times faster than [old-engine] + and can process multiple search requests simultaneously + + + I am very happy to see such nice software available under GPL. + +
+ +
+
+ + +
+ Support + + You can get support for Zebra from at least three sources. + + + First, there's the Zebra web site at + , + which always has the most recent version available for download. + If you have a problem with Zebra, the first thing to do is see + whether it's fixed in the current release. + + + Second, there's the Zebra mailing list. Its home page at + + includes a complete archive of all messages that have ever been + posted on the list. The Zebra mailing list is used both for + announcements from the authors (new + releases, bug fixes, etc.) and general discussion. You are welcome + to seek support there. Join by filling the form on the list home page. + + + Third, it's possible to buy a commercial support contract, with + well defined service levels and response times, from Index Data. + See + + for details. + +
- +
Future Directions @@ -346,20 +576,17 @@ Improved support for XML in search and retrieval. Eventually, the goal is for Zebra to pull double duty as a flexible information retrieval engine and high-performance XML - repository. - - - ### Partially done. - - - - - - Access to search engine through SOAP/RPC API to allow the - construction of applications without requiring Z39.50 tools. + repository. The recent addition of XPath searching is one + example of the kind of enhancement we're working on. - ### Partially done, thanks to the new SRW/Z39.50 gateway. + There is also the experimental ALVIS XSLT + XML input filter, which unleashes the full power of DOM based + XSLT transformations during indexing and record retrieval. Work + on this filter has been sponsored by the ALVIS EU project + . We expect this filter to + mature soon, as it is planned to be included in the version 2.0 + release of Zebra. @@ -397,7 +624,7 @@ or check the contact info at the end of this manual. - +