X-Git-Url: http://git.indexdata.com/?a=blobdiff_plain;f=doc%2Fadministration.xml;h=829ef7591505428863477952c443e459617edb76;hb=656a766f96dd92939c3604a7bf88f2355d040fc8;hp=cd0572e9f661132d6a3ec5b36588f553bf0c89fd;hpb=7b149568c78c6a73915e282ef10edc95171e66fd;p=idzebra-moved-to-github.git diff --git a/doc/administration.xml b/doc/administration.xml index cd0572e..829ef75 100644 --- a/doc/administration.xml +++ b/doc/administration.xml @@ -1,5 +1,5 @@ - + Administrating Zebra + - Those are in the zebra config file enabled by a directive like (use - only one of these a time!): - - rank: rank-1 # default - rank: rank-static # dummy - rank: zvrank # TDF-IDF like + Dynamic ranking is done at query time rather than + indexing time (this is why we + call it ``dynamic ranking'' in the first place ...) + It is invoked by adding + the Bib-1 relation attribute with + value ``relevance'' to the PQF query (that is, + @attr 2=102, see also + + The BIB-1 Attribute Set Semantics, also in + HTML). + To find all articles with the word Eoraptor in + the title, and present them relevance ranked, issue the PQF query: + + @attr 2=102 @attr 1=4 Eoraptor - Notice that the rank-1 and - zvrank do not use the static rank - information in the list keys, and will produce the same ordering - with our without static ranking enabled. + + + Dynamically ranking using PQF queries with the 'rank-1' + algorithm + + The default rank-1 ranking module implements a + TF/IDF (Term Frequecy over Inverse Document Frequency) like + algorithm. In contrast to the usual defintion of TF/IDF + algorithms, which only considers searching in one full-text + index, this one works on multiple indexes at the same time. + More precisely, + Zebra does boolean queries and searches in specific addressed + indexes (there are inverted indexes pointing from terms in the + dictionary to documents and term positions inside documents). + It works like this: + + + Query Components + + + First, the boolean query is dismantled into it's principal components, + i.e. atomic queries where one term is looked up in one index. + For example, the query + + @attr 2=102 @and @attr 1=1010 Utah @attr 1=1018 Springer + + is a boolean AND between the atomic parts + + @attr 2=102 @attr 1=1010 Utah + + and + + @attr 2=102 @attr 1=1018 Springer + + which gets processed each for itself. + + + + + + Atomic hit lists + + + Second, for each atomic query, the hit list of documents is + computed. + + + In this example, two hit lists for each index + @attr 1=1010 and + @attr 1=1018 are computed. + + + + + + Atomic scores + + + Third, each document in the hit list is assigned a score (_if_ ranking + is enabled and requested in the query) using a TF/IDF scheme. + + + In this example, both atomic parts of the query assign the magic + @attr 2=102 relevance attribute, and are + to be used in the relevance ranking functions. + + + It is possible to apply dynamic ranking on only parts of the + PQF query: + + @and @attr 2=102 @attr 1=1010 Utah @attr 1=1018 Springer + + searches for all documents which have the term 'Utah' on the + body of text, and which have the term 'Springer' in the publisher + field, and sort them in the order of the relevance ranking made on + the body-of-text index only. + + + + + + Hit list merging + + + Fourth, the atomic hit lists are merged according to the boolean + conditions to a final hit list of documents to be returned. + + + This step is always performed, independently of the fact that + dynamic ranking is enabled or not. + + + + + + Document score computation + + + Fifth, the total score of a document is computed as a linear + combination of the atomic scores of the atomic hit lists + + + Ranking weights may be used to pass a value to a ranking + algorithm, using the non-standard BIB-1 attribute type 9. + This allows one branch of a query to use one value while + another branch uses a different one. For example, we can search + for utah in the + @attr 1=4 index with weight 30, as + well as in the @attr 1=1010 index with weight 20: + + @attr 2=102 @or @attr 9=30 @attr 1=4 utah @attr 9=20 @attr 1=1010 city + + + + The default weight is + sqrt(1000) ~ 34 , as the Z39.50 standard prescribes that the top score + is 1000 and the bottom score is 0, encoded in integers. + + + + The ranking-weight feature is experimental. It may change in future + releases of zebra. + + + + + + + Re-sorting of hit list + + + Finally, the final hit list is re-ordered according to scores. + + + + + + + + + + + + + + The rank-1 algorithm + does not use the static rank + information in the list keys, and will produce the same ordering + with or without static ranking enabled. + + + + + + + + Dynamic ranking is not compatible + with estimated hit sizes, as all documents in + a hit set must be accessed to compute the correct placing in a + ranking sorted list. Therefore the use attribute setting + @attr 2=102 clashes with + @attr 9=integer. + + + + + + + + + Dynamically ranking CQL queries + + Dynamic ranking can be enabled during sever side CQL + query expansion by adding @attr 2=102 + chunks to the CQL config file. For example + + relationModifier.relevant = 2=102 + + invokes dynamic ranking each time a CQL query of the form + + Z> querytype cql + Z> f alvis.text =/relevant house + + is issued. Dynamic ranking can also be automatically used on + specific CQL indexes by (for example) setting + + index.alvis.text = 1=text 2=102 + + which then invokes dynamic ranking each time a CQL query of the form + + Z> querytype cql + Z> f alvis.text = house + + is issued. + + + + + + + + + Sorting + + Zebra sorts efficiently using special sorting indexes + (type=s; so each sortable index must be known + at indexing time, specified in the configuration of record + indexing. For example, to enable sorting according to the BIB-1 + Date/time-added-to-db field, one could add the line + + xelm /*/@created Date/time-added-to-db:s + + to any .abs record-indexing configuration file. + Similarly, one could add an indexing element of the form + + + + ]]> + to any alvis-filter indexing stylesheet. + + + Indexing can be specified at searching time using a query term + carrying the non-standard + BIB-1 attribute-type 7. This removes the + need to send a Z39.50 Sort Request + separately, and can dramatically improve latency when the client + and server are on separate networks. + The sorting part of the query is separate from the rest of the + query - the actual search specification - and must be combined + with it using OR. + + + A sorting subquery needs two attributes: an index (such as a + BIB-1 type-1 attribute) specifying which index to sort on, and a + type-7 attribute whose value is be 1 for + ascending sorting, or 2 for descending. The + term associated with the sorting attribute is the priority of + the sort key, where 0 specifies the primary + sort key, 1 the secondary sort key, and so + on. + + For example, a search for water, sort by title (ascending), + is expressed by the PQF query + + @or @attr 1=1016 water @attr 7=1 @attr 1=4 0 + + whereas a search for water, sort by title ascending, + then date descending would be + + @or @or @attr 1=1016 water @attr 7=1 @attr 1=4 0 @attr 7=2 @attr 1=30 1 + + + + Notice the fundamental differences between dynamic + ranking and sorting: there can be + only one ranking function defined and configured; but multiple + sorting indexes can be specified dynamically at search + time. Ranking does not need to use specific indexes, so + dynamic ranking can be enabled and disabled without + re-indexing; whereas, sorting indexes need to be + defined before indexing. + + + + + Extended Services: Remote Insert, Update and Delete + + + Extended services are only supported when accessing the Zebra + server using the Z39.50 + protocol. The SRU protocol does + not support extended services. + + + The extended services are not enabled by default in zebra - due to the - fact that they modify the system. - In order to allow anybody to update, use - - perm.anonymous: rw - + fact that they modify the system. Zebra can be configured + to allow anybody to + search, and to allow only updates for a particular admin user in the main zebra configuration file zebra.cfg. - Or, even better, allow only updates for a particular admin user. For - user admin, you could use: + For user admin, you could use: + perm.anonymous: r perm.admin: rw passwd: passwordfile - And in passwordfile, specify users and - passwords as colon seperated strings: + And in the password file + passwordfile, you have to specify users and + encrypted passwords as colon separated strings. + Use a tool like htpasswd + to maintain the encrypted passwords. admin:secret - + + It is essential to configure Zebra to store records internally, + and to support + modifications and deletion of records: + + storeData: 1 + storeKeys: 1 + + The general record type should be set to any record filter which + is able to parse XML records, you may use any of the two + declarations (but not both simultaneously!) + + recordType: grs.xml + # recordType: alvis.filter_alvis_config.xml + + To enable transaction safe shadow indexing, + which is extra important for this kind of operation, set + + shadow: directoryname: size (e.g. 1000M) + + See for additional information on + these configuration options. + + + It is not possible to carry information about record types or + similar to Zebra when using extended services, due to + limitations of the Z39.50 + protocol. Therefore, indexing filters can not be chosen on a + per-record basis. One and only one general XML indexing filter + must be defined. + + + + + + + + Extended services in the Z39.50 protocol + + + The Z39.50 standard allows + servers to accept special binary extended services + protocol packages, which may be used to insert, update and delete + records into servers. These carry control and update + information to the servers, which are encoded in seven package fields: + + + + Extended services Z39.50 Package Fields + + + + Parameter + Value + Notes + + + + + type + 'update' + Must be set to trigger extended services + + + action + string + + Extended service action type with + one of four possible values: recordInsert, + recordReplace, + recordDelete, + and specialUpdate + + + + record + XML string + An XML formatted string containing the record + + + syntax + 'xml' + Only XML record syntax is supported + + + recordIdOpaque + string + + Optional client-supplied, opaque record + identifier used under insert operations. + + + + recordIdNumber + positive number + Zebra's internal system number, + not allowed for recordInsert or + specialUpdate actions which result in fresh + record inserts. + + + + databaseName + database identifier + + The name of the database to which the extended services should be + applied. + + + + +
+ + + + The action parameter can be any of + recordInsert (will fail if the record already exists), + recordReplace (will fail if the record does not exist), + recordDelete (will fail if the record does not + exist), and + specialUpdate (will insert or update the record + as needed, record deletion is not possible). + + + + During all actions, the + usual rules for internal record ID generation apply, unless an + optional recordIdNumber Zebra internal ID or a + recordIdOpaque string identifier is assigned. + The default ID generation is + configured using the recordId: from + zebra.cfg. + See . + + + + Setting of the recordIdNumber parameter, + which must be an existing Zebra internal system ID number, is not + allowed during any recordInsert or + specialUpdate action resulting in fresh record + inserts. + + + + When retrieving existing + records indexed with GRS indexing filters, the Zebra internal + ID number is returned in the field + /*/id:idzebra/localnumber in the namespace + xmlns:id="http://www.indexdata.dk/zebra/", + where it can be picked up for later record updates or deletes. + + + + A new element set for retrieval of internal record + data has been added, which can be used to access minimal records + containing only the recordIdNumber Zebra + internal ID, or the recordIdOpaque string + identifier. This works for any indexing filter used. + See . + + + + The recordIdOpaque string parameter + is an client-supplied, opaque record + identifier, which may be used under + insert, update and delete operations. The + client software is responsible for assigning these to + records. This identifier will + replace zebra's own automagic identifier generation with a unique + mapping from recordIdOpaque to the + Zebra internal recordIdNumber. + The opaque recordIdOpaque string + identifiers + are not visible in retrieval records, nor are + searchable, so the value of this parameter is + questionable. It serves mostly as a convenient mapping from + application domain string identifiers to Zebra internal ID's. + + + + + + + Extended services from yaz-client + We can now start a yaz-client admin session and create a database: @@ -1059,14 +1716,11 @@ from example/gils/records) and index it: update insert 1 esdd0006.grs + Z> update insert id1234 esdd0006.grs ]]> - The 3rd parameter - 1 here - - is the opaque record ID from Ext update. - It a record ID that we assign to the record - in question. If we do not - assign one, the usual rules for match apply (recordId: from zebra.cfg). + The 3rd parameter - id1234 here - + is the recordIdOpaque package field. Actually, we should have a way to specify "no opaque record id" for @@ -1088,10 +1742,11 @@ - Let's delete the beast: + Let's delete the beast, using the same + recordIdOpaque string parameter: update delete 1 + Z> update delete id1234 No last record (update ignored) Z> update delete 1 esdd0006.grs Got extended services response @@ -1120,8 +1775,14 @@ after each update session in order write your changes from the shadow to the life register space. + + + + + Extended services from yaz-php + - Extended services are also available from the YAZ client layer. An + Extended services are also available from the YAZ PHP client layer. An example of an YAZ-PHP extended service transaction is given here: - - - - - - YAZ Frontend Virtual Hosts - - zebrasrv uses the YAZ server frontend and does - support multiple virtual servers behind multiple listening sockets. - &zebrasrv-virtual; - - - Section "Virtual Hosts" in the YAZ manual. - http://www.indexdata.dk/yaz/doc/server.vhosts.tkl - - - - - - Server Side CQL to PQF Query Translation - - Using the - <cql2rpn>l2rpn.txt</cql2rpn> - YAZ Frontend Virtual - Hosts option, one can configure - the YAZ Frontend CQL-to-PQF - converter, specifying the interpretation of various - CQL - indexes, relations, etc. in terms of Type-1 query attributes. - - - - For example, using server-side CQL-to-PQF conversion, one might - query a zebra server like this: - - querytype cql - Z> find text=(plant and soil) - ]]> - - and - if properly configured - even static relevance ranking can - be performed using CQL query syntax: - - find text = /relevant (plant and soil) - ]]> - - - - - By the way, the same configuration can be used to - search using client-side CQL-to-PQF conversion: - (the only difference is querytype cql2rpn - instead of - querytype cql, and the call specifying a local - conversion file) - - querytype cql2rpn - Z> find text=(plant and soil) - ]]> - - - - - Exhaustive information can be found in the - Section "Specification of CQL to RPN mappings" in the YAZ manual. - - http://www.indexdata.dk/yaz/doc/tools.tkl#tools.cql.map, - and shall therefore not be repeated here. - - + - -