X-Git-Url: http://git.indexdata.com/?a=blobdiff_plain;f=doc%2Fintroduction.xml;h=37c5fd2294558903d81b550b2d914bfcb3a3b320;hb=271266c6e2dfeef76ba5724848e435c441d671e7;hp=47096a3c826ef4ceb7d693db1d02080bbc628e0f;hpb=b19b79e382ef8196f1625763db1af3a82b1e0c81;p=idzebra-moved-to-github.git diff --git a/doc/introduction.xml b/doc/introduction.xml index 47096a3..37c5fd2 100644 --- a/doc/introduction.xml +++ b/doc/introduction.xml @@ -1,5 +1,5 @@ - + Introduction
@@ -20,10 +20,12 @@ code is open to anybody to read and change under the GPL license. - &zebra; is a networked component which acts as a reliable &z3950; server + &zebra; is a networked component which acts as a + reliable &z3950; server for both record/document search, presentation, insert, update and delete operations. In addition, it understands the &sru; family of - webservices, which exist in &rest; &get;/&post; and truly &soap; flavors. + webservices, which exist in &rest; &get;/&post; and truly + &soap; flavors. &zebra; is available as MS Windows 2003 Server (32 bit) self-extracting @@ -73,9 +75,38 @@ &zebra; Features Overview - - &zebra; Features Overview + + + +
+ &zebra; Document Model + +
+ &zebra; document model + + + + Feature @@ -86,130 +117,432 @@ - Boolean query language + Complex semi-structured Documents + &xml; and &grs1; Documents + Both &xml; and &grs1; documents exhibit a &dom; like internal + representation allowing for complex indexing and display rules + and + + + + Input document formats + &xml;, &sgml;, Text, ISO2709 (&marc;) + + A system of input filters driven by + regular expressions allows most ASCII-based + data formats to be easily processed. + &sgml;, &xml;, ISO2709 (&marc;), and raw text are also + supported. + + + + Document storage + Index-only, Key storage, Document storage + Data can be, and usually is, imported + into &zebra;'s own storage, but &zebra; can also refer to + external files, building and maintaining indexes of "live" + collections. + + + + + +
+
+ + + +
+ &zebra; Index Scanning + + + &zebra; index scanning + + + + + + + + Feature + Availability + Notes + Reference + + + + + Scan + term suggestions + Scan on a given named index returns all the + indexed terms in lexicographical order near the given start + term. This can be used to create drop-down menus and search + suggestions. + and + + + + + Facetted browsing + partial + &zebra; supports scan inside a hit + set from a previous search, thus reducing the listed + terms to the + subset of terms found in the documents/records of the hit + set. + + + + Drill-down or refine-search + partially + scanning in result sets can be used to implement + drill-down in search clients + + + + +
+
+ +
+ &zebra; Document Presentation + + + &zebra; document presentation + + + + + + + + Feature + Availability + Notes + Reference + + + + + Hit count + yes + Search results include at any time the total hit count of a given + query, either exact computed, or approximative, in case that the + hit count exceeds a possible pre-defined hit set truncation + level. - A system of input filters driven by - regular expressions allows most ASCII-based - data formats to be easily processed. - &sgml;, &xml;, ISO2709 (&marc;), and raw text are also - supported. - + and + + + + + Paged result sets + yes + Paging of search requests and present/display request + can return any successive number of records from any start + position in the hit set, i.e. it is trivial to provide search + results in successive pages of any size. + + + + &xml; document transformations + &xslt; based + Record presentation can be performed in many + pre-defined &xml; data + formats, where the original &xml; records are on-the-fly transformed + through any preconfigured &xslt; transformation. It is therefore + trivial to present records in short/full &xml; views, transforming to + RSS, Dublin Core, or other &xml; based data formats, or transform + records to XHTML snippets ready for inserting in XHTML pages. + + + + + Binary record transformations + &marc;, &usmarc;, &marc21; and &marcxml; + post-filter record transformations + + + + Record Syntaxes + + Multiple record syntaxes + for data retrieval: &grs1;, &sutrs;, + &xml;, ISO2709 (&marc;), etc. Records can be mapped between + record syntaxes and schemas on the fly. + + + + &zebra; internal metadata + yes + &zebra; internal document metadata can be fetched in + &sutrs; and &xml; record syntaxes. Those are useful in client + applications. + + + + &zebra; internal raw record data + yes + &zebra; internal raw, binary record data can be fetched in + &sutrs; and &xml; record syntaxes, leveraging %zebra; to a + binary storage system + + + + &zebra; internal record field data + yes + &zebra; internal record field data can be fetched in + &sutrs; and &xml; record syntaxes. This makes very fast minimal + record data displays possible. + + + + +
+
+ +
+ &zebra; Sorting and Ranking + + + &zebra; sorting and ranking + + + + + + + + Feature + Availability + Notes + Reference + + + + + Sort + numeric, lexicographic + Sorting on the basis of alpha-numeric and numeric data + is supported. Alphanumeric sorts can be configured for + different data encodings and locales for European languages. + and + + + + Combined sorting + yes + Sorting on the basis of combined sorts ­ e.g. combinations of + ascending/descending sorts of lexicographical/numeric/date field data + is supported + Relevance ranking TF-IDF like Relevance-ranking of free-text queries is supported using a TF-IDF like algorithm. - + - Document storage - Index-only, Key storage, Document storage - Data can be, and usually is, imported - into &zebra;'s own storage, but &zebra; can also refer to - external files, building and maintaining indexes of "live" - collections. - + Static pre-ranking + yes + Enables pre-index time ranking of documents where hit + lists are ordered first by ascending static rank, then by + ascending document ID. + + + +
+
+ + +
+ &zebra; Live Updates + + + + &zebra; live updates + + + + + + - Regular expression matching - Regexp - Full regular expression matching and "approximate - matching" (eg. spelling mistake corrections) are handled. - + Feature + Availability + Notes + Reference + + - Search truncation - + Incremental and batch updates - + It is possible to schedule record inserts/updates/deletes in any + quantity, from single individual handled records to batch updates + in strikes of any size, as well as total re-indexing of all records + from file system. + - Remote update + Remote updates &z3950; extended services - - + Updates can be performed from remote locations using the + &z3950; extended services. Access to extended services can be + login-password protected. + and + - Supported Platforms - UNIX, Linux, Windows (NT/2000/2003/XP) - &zebra; is written in portable C, so it runs on most - Unix-like systems as well as Windows (NT/2000/2003/XP). Binary - distributions are - available for GNU/Debian Linux and Windows - + Live updates + transaction based + Data updates are transaction based and can be performed + on running &zebra; systems. Full searchability is preserved + during life data update due to use of shadow disk areas for + update operations. Multiple update transactions at the same + time are lined up, to be performed one after each other. Data + integrity is preserved. + + + +
+
+ +
+ &zebra; Networked Protocols + + + &zebra; networked protocols + + + + + + - &z3950; - &z3950; protocol support - Protocol facilities: Init, Search, Present (retrieval), - Segmentation (support for very large records), Delete, Scan - (index browsing), Sort, Close and support for the ``update'' - Extended Service to add or replace an existing &xml; - record. Piggy-backed presents are honored in the search - request. Named result sets are supported. - + Feature + Availability + Notes + Reference + + - Record Syntaxes + Fundamental operations + &z3950;/&sru; explain, + search, scan, and + update - Multiple record syntaxes - for data retrieval: &grs1;, &sutrs;, - &xml;, ISO2709 (&marc;), etc. Records can be mapped between record syntaxes - and schemas on the fly. - + + + + &z3950; protocol support + yes + Protocol facilities supported are: + init, search, + present (retrieval), + Segmentation (support for very large records), + delete, scan + (index browsing), sort, + close and support for the update + Extended Service to add or replace an existing &xml; + record. Piggy-backed presents are honored in the search + request. Named result sets are supported. + Web Service support @@ -217,68 +550,135 @@ The protocol operations explain, searchRetrieve and scan are supported. &cql; to internal - query model &rpn; conversion is supported. Extended RPN queries + query model &rpn; + conversion is supported. Extended RPN queries for search/retrieve and scan are supported. - + + + +
+
+ +
+ &zebra; Data Size and Scalability + + + &zebra; data size and scalability + + + + + + - - - - + Feature + Availability + Notes + Reference + + + No of records + 40-60 million - - + Data size + 100 GB of record data + &zebra; based applications have successfully indexed up + to 100 GB of record data - - - + Scale out + multiple discs - - + Performance + O(n * log N) + &zebra; query speed and performance is affected roughly by + O(log N), + where N is the total database size, and by + O(n), where n is the + specific query hit set size. - - - + Average search times + Even on very large size databases hit rates of 20 queries per + seconds with average query answering time of 1 second are possible, + provided that the boolean queries are constructed sufficiently + precise to result in hit sets of the order of 1000 to 5.000 + documents. - - + Large databases + 64 bit file pointers + 64 file pointers assure that register files can extend + the 2 GB limit. Logical files can be + automatically partitioned over multiple disks, thus allowing for + large databases. - - - + + +
+
+ +
+ &zebra; Supported Platforms + + + &zebra; supported platforms + + + + + + - - - - + Feature + Availability + Notes + Reference + + + Linux - - - + GNU Linux (32 and 64bit), journaling Reiser or (better) + JFS file system + on disks. NFS file systems are not supported. + GNU/Debian Linux packages are available + + + + Unix + tar-ball + &zebra; is written in portable C, so it runs on most + Unix-like systems. + Usual tar-ball install possible on many major Unix systems + + + + Windows + NT/2000/2003/XP + &zebra; runs as well on Windows (NT/2000/2003/XP). + Windows installer packages available +
- +
@@ -288,7 +688,7 @@ &zebra; has been deployed in numerous applications, in both the academic and commercial worlds, in application domains as diverse - as bibliographic catalogues, geospatial information, structured + as bibliographic catalogues, Geo-spatial information, structured vocabulary browsing, government information locators, civic information systems, environmental observations, museum information and web indexes. @@ -612,73 +1012,6 @@ for details. - - -
- Future Directions - - - These are some of the plans that we have for the software in the near - and far future, ordered approximately as we expect to work on them. - - - - - - - - Improved support for &xml; in search and retrieval. Eventually, - the goal is for &zebra; to pull double duty as a flexible - information retrieval engine and high-performance &xml; - repository. The recent addition of XPath searching is one - example of the kind of enhancement we're working on. - - - There is also the experimental ALVIS &xslt; - &xml; input filter, which unleashes the full power of &dom; based - &xslt; transformations during indexing and record retrieval. Work - on this filter has been sponsored by the ALVIS EU project - . We expect this filter to - mature soon, as it is planned to be included in the version 2.0 - release of &zebra;. - - - - - - Finalisation and documentation of &zebra;'s C programming - &api;, allowing updates, database management and other functions - not readily expressed in &z3950;. We will also consider - exposing the &api; through &soap;. - - - - - - Improved free-text searching. We're first and foremost octet jockeys and - we're actively looking for organisations or people who'd like - to contribute experience in relevance ranking and text - searching. - - - - - - - - Programmers thrive on user feedback. If you are interested in a - facility that you don't see mentioned here, or if there's something - you think we could do better, please drop us a mail. Better still, - implement it and send us the patches. - - - If you think it's all really neat, you're welcome to drop us a line - saying that, too. You can email us on - info@indexdata.dk - or check the contact info at the end of this manual. - - -