X-Git-Url: http://git.indexdata.com/?p=idzebra-moved-to-github.git;a=blobdiff_plain;f=doc%2Fintroduction.xml;h=69938cbaf24603fa17df16f7f3f3db4e5800ad7e;hp=97780083479be61615791f98c8284b4f3c861f7f;hb=99842ec71f065fd6886daa355923b01d9ce71d26;hpb=ca4101c308ad8427fafba129a4f58a8b38ead63e diff --git a/doc/introduction.xml b/doc/introduction.xml index 9778008..69938cb 100644 --- a/doc/introduction.xml +++ b/doc/introduction.xml @@ -1,5 +1,4 @@ - Introduction
@@ -7,10 +6,10 @@ &zebra; is a free, fast, friendly information management system. It can - index records in &xml;/&sgml;, &marc;, e-mail archives and many other + index records in &acro.xml;/&acro.sgml;, &acro.marc;, e-mail archives and many other formats, and quickly find them using a combination of boolean searching and relevance ranking. Search-and-retrieve applications can - be written using &api;s in a wide variety of languages, communicating + be written using &acro.api;s in a wide variety of languages, communicating with the &zebra; server using industry-standard information-retrieval protocols or web services. @@ -20,10 +19,12 @@ code is open to anybody to read and change under the GPL license. - &zebra; is a networked component which acts as a reliable &z3950; server + &zebra; is a networked component which acts as a + reliable &acro.z3950; server for both record/document search, presentation, insert, update and - delete operations. In addition, it understands the &sru; family of - webservices, which exist in &rest; &get;/&post; and truly &soap; flavors. + delete operations. In addition, it understands the &acro.sru; family of + webservices, which exist in &acro.rest; &acro.get;/&acro.post; and truly + &acro.soap; flavors. &zebra; is available as MS Windows 2003 Server (32 bit) self-extracting @@ -42,7 +43,7 @@ &zebra; is a high-performance, general-purpose structured text indexing and retrieval engine. It reads records in a - variety of input formats (eg. email, &xml;, &marc;) and provides access + variety of input formats (e.g. email, &acro.xml;, &acro.marc;) and provides access to them through a powerful combination of boolean search expressions and relevance-ranked free-text queries. @@ -51,13 +52,13 @@ &zebra; supports large databases (tens of millions of records, tens of gigabytes of data). It allows safe, incremental database updates on live systems. Because &zebra; supports - the industry-standard information retrieval protocol, &z3950;, + the industry-standard information retrieval protocol, &acro.z3950;, you can search &zebra; databases using an enormous variety of programs and toolkits, both commercial and free, which understand this protocol. Application libraries are available to allow bespoke clients to be written in Perl, C, C++, Java, Tcl, Visual - Basic, Python, &php; and more - see the - &zoom; web site + Basic, Python, &acro.php; and more - see the + &acro.zoom; web site for more information on some of these client toolkits. @@ -95,9 +96,16 @@ --> - - &zebra; networked protocols +
+ &zebra; Document Model + +
+ &zebra; document model + + + + Feature @@ -108,42 +116,49 @@ - Fundamental operation types - &z3950;/&sru; explain, search, and scan - - + Complex semi-structured Documents + &acro.xml; and &acro.grs1; Documents + Both &acro.xml; and &acro.grs1; documents exhibit a &acro.dom; like internal + representation allowing for complex indexing and display rules + and + - &z3950; protocol support - yes - Protocol facilities supported are: - Init, Search, Present (retrieval), - Segmentation (support for very large records), Delete, Scan - (index browsing), Sort, Close and support for the ``update'' - Extended Service to add or replace an existing &xml; - record. Piggy-backed presents are honored in the search - request. Named result sets are supported. - + Input document formats + &acro.xml;, &acro.sgml;, Text, ISO2709 (&acro.marc;) + + A system of input filters driven by + regular expressions allows most ASCII-based + data formats to be easily processed. + &acro.sgml;, &acro.xml;, ISO2709 (&acro.marc;), and raw text are also + supported. + - Web Service support - &sru_gps; - The protocol operations explain, - searchRetrieve and scan - are supported. &cql; to internal - query model &rpn; - conversion is supported. Extended RPN queries - for search/retrieve and scan are supported. - + Document storage + Index-only, Key storage, Document storage + Data can be, and usually is, imported + into &zebra;'s own storage, but &zebra; can also refer to + external files, building and maintaining indexes of "live" + collections. + +
+
+ +
+ &zebra; Index Scanning &zebra; index scanning + + + + Feature @@ -231,35 +262,45 @@ Scan - yes + term suggestions Scan on a given named index returns all the - indexed terms in lexicographical order near the given start term. - + indexed terms in lexicographical order near the given start + term. This can be used to create drop-down menus and search + suggestions. + and + + Facetted browsing - partial - &zebra; supports scan inside a hit - set from a previous search, thus reducing the listed - terms to the - subset of terms found in the documents/records of the hit set. - + available + Zebra 2.1 and allows retrieval of facets for + a result set. + + Drill-down or refine-search partially scanning in result sets can be used to implement drill-down in search clients - +
+
+
+ &zebra; Document Presentation &zebra; document presentation + + + + Feature @@ -275,53 +316,88 @@ Search results include at any time the total hit count of a given query, either exact computed, or approximative, in case that the hit count exceeds a possible pre-defined hit set truncation - level. - - + level. + + and + + Paged result sets yes - Paging of search requests and present/display request can return any - successive number of records from any start position in the hit set, - i.e. it is trivial to provide search results in successive pages of - any size. - + Paging of search requests and present/display request + can return any successive number of records from any start + position in the hit set, i.e. it is trivial to provide search + results in successive pages of any size. + - &xml;ocument transformations - &xslt; based - Record presentation can be performed in many pre-defined &xml; data - formats, where the original &xml; records are on-the-fly transformed - through any preconfigured &xslt; transformation. It is therefore - trivial to present records in short/full &xml; views, transforming to - RSS, Dublin Core, or other &xml; based data formats, or transform + &acro.xml; document transformations + &acro.xslt; based + Record presentation can be performed in many + pre-defined &acro.xml; data + formats, where the original &acro.xml; records are on-the-fly transformed + through any preconfigured &acro.xslt; transformation. It is therefore + trivial to present records in short/full &acro.xml; views, transforming to + RSS, Dublin Core, or other &acro.xml; based data formats, or transform records to XHTML snippets ready for inserting in XHTML pages. - + + Binary record transformations - &marc;, &usmarc;, &marc21; and &marcxml; + &acro.marc;, &acro.usmarc;, &acro.marc21; and &acro.marcxml; + post-filter record transformations - Record Syntaxes Multiple record syntaxes - for data retrieval: &grs1;, &sutrs;, - &xml;, ISO2709 (&marc;), etc. Records can be mapped between record syntaxes - and schemas on the fly. - + for data retrieval: &acro.grs1;, &acro.sutrs;, + &acro.xml;, ISO2709 (&acro.marc;), etc. Records can be mapped between + record syntaxes and schemas on the fly. + + + + &zebra; internal metadata + yes + &zebra; internal document metadata can be fetched in + &acro.sutrs; and &acro.xml; record syntaxes. Those are useful in client + applications. + + + + &zebra; internal raw record data + yes + &zebra; internal raw, binary record data can be fetched in + &acro.sutrs; and &acro.xml; record syntaxes, leveraging %zebra; to a + binary storage system + + + + &zebra; internal record field data + yes + &zebra; internal record field data can be fetched in + &acro.sutrs; and &acro.xml; record syntaxes. This makes very fast minimal + record data displays possible. +
+
+
+ &zebra; Sorting and Ranking &zebra; sorting and ranking + + + + Feature @@ -335,9 +411,10 @@ Sort numeric, lexicographic Sorting on the basis of alpha-numeric and numeric data - is supported. Alphanumeric sorts can be configured for different data encodings - and locales for European languages. - + is supported. Alphanumeric sorts can be configured for + different data encodings and locales for European languages. + and + Combined sorting @@ -345,30 +422,40 @@ Sorting on the basis of combined sorts ­ e.g. combinations of ascending/descending sorts of lexicographical/numeric/date field data is supported - + Relevance ranking TF-IDF like Relevance-ranking of free-text queries is supported using a TF-IDF like algorithm. - + - Relevence ranking - TDF-IDF like - - + Static pre-ranking + yes + Enables pre-index time ranking of documents where hit + lists are ordered first by ascending static rank, then by + ascending document ID. +
+
+
+ &zebra; Live Updates - - &zebra; document model + +
+ &zebra; live updates + + + + Feature @@ -379,42 +466,107 @@ - Complex semi-structured Documents - &xml; and &grs1; Documents - Both &xml; and &grs1; documents exhibit a &dom; like internal - representation allowing for complex indexing and display rules - + Incremental and batch updates + + It is possible to schedule record inserts/updates/deletes in any + quantity, from single individual handled records to batch updates + in strikes of any size, as well as total re-indexing of all records + from file system. + - Input document formats - &xml;, &sgml;, Text, ISO2709 (&marc;) - - A system of input filters driven by - regular expressions allows most ASCII-based - data formats to be easily processed. - &sgml;, &xml;, ISO2709 (&marc;), and raw text are also - supported. - + Remote updates + &acro.z3950; extended services + Updates can be performed from remote locations using the + &acro.z3950; extended services. Access to extended services can be + login-password protected. + and + - Document storage - Index-only, Key storage, Document storage - Data can be, and usually is, imported - into &zebra;'s own storage, but &zebra; can also refer to - external files, building and maintaining indexes of "live" - collections. - + Live updates + transaction based + Data updates are transaction based and can be performed + on running &zebra; systems. Full searchability is preserved + during life data update due to use of shadow disk areas for + update operations. Multiple update transactions at the same + time are lined up, to be performed one after each other. Data + integrity is preserved. + -
+
+
+ &zebra; Networked Protocols + + + &zebra; networked protocols + + + + + + + + Feature + Availability + Notes + Reference + + + + + Fundamental operations + &acro.z3950;/&acro.sru; explain, + search, scan, and + update + + + + + &acro.z3950; protocol support + yes + Protocol facilities supported are: + init, search, + present (retrieval), + Segmentation (support for very large records), + delete, scan + (index browsing), sort, + close and support for the update + Extended Service to add or replace an existing &acro.xml; + record. Piggy-backed presents are honored in the search + request. Named result sets are supported. + + + + Web Service support + &acro.sru; + The protocol operations explain, + searchRetrieve and scan + are supported. &acro.cql; to internal + query model &acro.rpn; + conversion is supported. Extended RPN queries + for search/retrieve and scan are supported. + + + + +
+
+
+ &zebra; Data Size and Scalability &zebra; data size and scalability + + + + Feature @@ -428,25 +580,20 @@ No of records 40-60 million - + Data size 100 GB of record data + &zebra; based applications have successfully indexed up + to 100 GB of record data - - - - File pointers - 64 bit - - Scale out multiple discs - + Performance @@ -456,7 +603,7 @@ where N is the total database size, and by O(n), where n is the specific query hit set size. - + Average search times @@ -466,83 +613,32 @@ provided that the boolean queries are constructed sufficiently precise to result in hit sets of the order of 1000 to 5.000 documents. - + Large databases + 64 bit file pointers 64 file pointers assure that register files can extend the 2 GB limit. Logical files can be automatically partitioned over multiple disks, thus allowing for large databases. -
+
- - - &zebra; live updates - - - - Feature - Availability - Notes - Reference - - - - - Batch updates - - It is possible to schedule record inserts/updates/deletes in any - quantity, from single individual handled records to batch updates - in strikes of any size, as well as total re-indexing of all records - from file system. - - - - Incremental updates - - - - - - Remote updates - &z3950; extended services - - - - - Live updates - - Data updates are transaction based and can be performed on running - &zebra; systems. Full searchability is preserved during life data update due to use - of shadow disk areas for update operations. Multiple update transactions at the same time are lined up, to be - performed one after each other. Data integrity is preserved. - - - - Database updates - live, incremental updates - Robust updating - records can be added and deleted ``on the fly'' - without rebuilding the index from scratch. - Records can be safely updated even while users are accessing - the server. - The update procedure is tolerant to crashes or hard interrupts - during database updating - data can be reconstructed following - a crash. - - - - -
+
+ &zebra; Supported Platforms &zebra; supported platforms + + + + Feature @@ -555,35 +651,31 @@ Linux - GNU Linux (32 and 64bit), journaling Reiser or (better) JFS filesystem - on disks. GNU/Debian Linux packages are available - + GNU Linux (32 and 64bit), journaling Reiser or (better) + JFS file system + on disks. NFS file systems are not supported. + GNU/Debian Linux packages are available + Unix - tarball - Usual tarball install possible on many major Unix systems - + tar-ball + &zebra; is written in portable C, so it runs on most + Unix-like systems. + Usual tar-ball install possible on many major Unix systems + Windows - - Windows installer packages available - - - - Supported Platforms - UNIX, Linux, Windows (NT/2000/2003/XP) - &zebra; is written in portable C, so it runs on most - Unix-like systems as well as Windows (NT/2000/2003/XP). Binary - distributions are - available for GNU/Debian Linux and Windows - + NT/2000/2003/XP + &zebra; runs as well on Windows (NT/2000/2003/XP). + Windows installer packages available +
- +
@@ -593,7 +685,7 @@ &zebra; has been deployed in numerous applications, in both the academic and commercial worlds, in application domains as diverse - as bibliographic catalogues, geospatial information, structured + as bibliographic catalogues, Geo-spatial information, structured vocabulary browsing, government information locators, civic information systems, environmental observations, museum information and web indexes. @@ -621,7 +713,7 @@ In early 2005, the Koha project development team began looking at - ways to improve &marc; support and overcome scalability limitations + ways to improve &acro.marc; support and overcome scalability limitations in the Koha 2.x series. After extensive evaluations of the best of the Open Source textual database engines - including MySQL full-text searching, PostgreSQL, Lucene and Plucene - the team @@ -640,7 +732,7 @@ and relevance-ranked free-text queries, both of which the Koha 2.x series lack. &zebra; also supports incremental and safe database updates, which allow on-the-fly record - management. Finally, since &zebra; has at its heart the &z3950; + management. Finally, since &zebra; has at its heart the &acro.z3950; protocol, it greatly improves Koha's support for that critical library standard." @@ -660,6 +752,39 @@ + +
+ Kete Open Source Digital Library and Archiving software + + Kete is a digital object + management repository, initially developed in + New Zealand. Initial development has + been a partnership between the Horowhenua Library Trust and + Katipo Communications Ltd. funded as part of the Community + Partnership Fund in 2006. + Kete is purpose built + software to enable communities to build their own digital + libraries, archives and repositories. + + + It is based on Ruby-on-Rails and MySQL, and integrates the &zebra; server + and the &yaz; toolkit for indexing and retrieval of it's content. + Zebra is run as separate computer process from the Kete + application. + See + how Kete manages + Zebra. + + + Why does Kete wants to use Zebra?? Speed, Scalability and easy + integration with Koha. Read their + detailed + reasoning here. + +
+
Emilda open source ILS @@ -670,12 +795,12 @@ from virtually any computer with an Internet connection, has template based layout allowing anyone to alter the visual appearance of Emilda, and is - &xml; based language for fast and easy portability to virtually any + &acro.xml; based language for fast and easy portability to virtually any language. Currently, Emilda is used at three schools in Espoo, Finland. - As a surplus, 100% &marc; compatibility has been achieved using the + As a surplus, 100% &acro.marc; compatibility has been achieved using the &zebra; Server from Index Data as backend server.
@@ -687,18 +812,18 @@ is a netbased library service offering all traditional functions on a very high level plus many new services. Reindex.net is a comprehensive and powerful WEB system - based on standards such as &xml; and &z3950;. - updates. Reindex supports &marc21;, dan&marc; eller Dublin Core with + based on standards such as &acro.xml; and &acro.z3950;. + updates. Reindex supports &acro.marc21;, dan&acro.marc; eller Dublin Core with UTF8-encoding. Reindex.net runs on GNU/Debian Linux with &zebra; and Simpleserver from Index Data for bibliographic data. The relational database system - Sybase 9 &xml; is used for + Sybase 9 &acro.xml; is used for administrative data. - Internally &marcxml; is used for bibliographical records. Update - utilizes &z3950; extended services. + Internally &acro.marcxml; is used for bibliographical records. Update + utilizes &acro.z3950; extended services. @@ -763,8 +888,8 @@ The &zebra; information retrieval indexing machine is used inside the Alvis framework to manage huge collections of natural language processed and - enhanced &xml; data, coming from a topic relevant web crawl. - In this application, &zebra; swallows and manages 37GB of &xml; data + enhanced &acro.xml; data, coming from a topic relevant web crawl. + In this application, &zebra; swallows and manages 37GB of &acro.xml; data in about 4 hours, resulting in search times of fractions of seconds. @@ -786,9 +911,9 @@ The member libraries send in data files representing their periodicals, including both brief bibliographic data and summary - holdings. Then 21 individual &z3950; targets are created, each + holdings. Then 21 individual &acro.z3950; targets are created, each using &zebra;, and all mounted on the single hardware server. - The live service provides a web gateway allowing &z3950; searching + The live service provides a web gateway allowing &acro.z3950; searching of all of the targets or a selection of them. &zebra;'s small footprint allows a relatively modest system to comfortably host the 21 servers. @@ -800,7 +925,7 @@
- NLI-&z3950; - a Natural Language Interface for Libraries + NLI-&acro.z3950; - a Natural Language Interface for Libraries Fernuniversität Hagen in Germany have developed a natural language interface for access to library databases. @@ -809,8 +934,8 @@ In order to evaluate this interface for recall and precision, they chose &zebra; as the basis for retrieval effectiveness. The &zebra; server contains a copy of the GIRT database, consisting of more - than 76000 records in &sgml; format (bibliographic records from - social science), which are mapped to &marc; for presentation. + than 76000 records in &acro.sgml; format (bibliographic records from + social science), which are mapped to &acro.marc; for presentation. (GIRT is the German Indexing and Retrieval Testdatabase. It is a @@ -835,7 +960,7 @@ &zebra; has been used by a variety of institutions to construct indexes of large web sites, typically in the region of tens of millions of pages. In this role, it functions somewhat similarly - to the engine of google or altavista, but for a selected intranet + to the engine of Google or AltaVista, but for a selected intranet or a subset of the whole Web. @@ -909,81 +1034,7 @@ releases, bug fixes, etc.) and general discussion. You are welcome to seek support there. Join by filling the form on the list home page. - - Third, it's possible to buy a commercial support contract, with - well defined service levels and response times, from Index Data. - See - - for details. -
- - -
- Future Directions - - - These are some of the plans that we have for the software in the near - and far future, ordered approximately as we expect to work on them. - - - - - - - - Improved support for &xml; in search and retrieval. Eventually, - the goal is for &zebra; to pull double duty as a flexible - information retrieval engine and high-performance &xml; - repository. The recent addition of XPath searching is one - example of the kind of enhancement we're working on. - - - There is also the experimental ALVIS &xslt; - &xml; input filter, which unleashes the full power of &dom; based - &xslt; transformations during indexing and record retrieval. Work - on this filter has been sponsored by the ALVIS EU project - . We expect this filter to - mature soon, as it is planned to be included in the version 2.0 - release of &zebra;. - - - - - - Finalisation and documentation of &zebra;'s C programming - &api;, allowing updates, database management and other functions - not readily expressed in &z3950;. We will also consider - exposing the &api; through &soap;. - - - - - - Improved free-text searching. We're first and foremost octet jockeys and - we're actively looking for organisations or people who'd like - to contribute experience in relevance ranking and text - searching. - - - - - - - - Programmers thrive on user feedback. If you are interested in a - facility that you don't see mentioned here, or if there's something - you think we could do better, please drop us a mail. Better still, - implement it and send us the patches. - - - If you think it's all really neat, you're welcome to drop us a line - saying that, too. You can email us on - info@indexdata.dk - or check the contact info at the end of this manual. - - -