The numbering scheme at the core of eXist 1.0 did limit the maximum size of a document to be stored within the database. In eXist releases 1.1 and later this is fixed.

The current organization of collections and resources causes a number of problems with respect to (a) locking, (b) query performance, (c) update performance. Right now, documents are tightly bound to the collection in which they are contained. Any operation on a document has to go through its parent collection. As a result, locking and access control becomes quite complex as we need to take care of the document and the collection.

There's also a direct dependency between the size of a collection (in terms of the number of documents stored in it) and document update speed. If a collection has a large number of documents, removing a single document becomes very slow.

This problem can be solved by physically decoupling documents and collections:

• a collection should be modelled as a logical unit, not a physical one.
• current integer document ids should be replaced by hierarchical ids, which model the entire collection/document path.
• the link between the resource and the collection is only done through the hierarchical document id.
• locking and access control should be delegated to a central lock manager, which just contains a map DocumentProxy -> Lock

eXist currently uses 2 DOM (document object model) implementations: one for nodes stored in the db, and a different one for in-memory nodes constructed during an XQuery. The two models can not be mixed. To process an XQuery expression on an in-memory node, the query engine needs to create a temporary persistent copy. This costs performance and did cause stability issues in the past.

To solve those problems, we are currently redesigning the in-memory DOM to implement the same interfaces as the persistent DOM. The query engine should be able to mix nodes of both models.

Metadata could include system properties like last-modification date or user-defined metadata. Preferably, metadata records should be ordinary XML documents. The format should not be restricted.

Since version 1.2, there are now alternative index configuration methods which support the optimizer in rewriting a query for best performance. A new modularized indexing architecture allows to plug in arbitrary new indexes into the indexing pipeline. A N-gram and a spatial index module were added as prototypes to test the new architecture.

There will be other index types added in the future, for example:

• combined path indexes to speed up frequently-used XPath expressions
• indexes to support operations on atomic-values, e.g. in order-by expressions or the distinct-values function
• integration of specialized, external index types like the spatial index module mentioned above.

(Align with the XQuery Full-text specification)

The interfaces to the indexing system need to be redesigned to support the query engine in index selection. This includes, for example, statistical information about the frequency of index items. The redesign of the indexing system thus presents a necessary foundation for the query optimizer.

The current architecture is also too limited with respect to text analysis. The general-purpose tokenizer is not suitable for language-dependent analysis. Plans are to replace these classes by Lucene's analyzer. Lucene offers a pluggable architecture in which multiple analyzers can be combined.

No remarks available.

Currently stored in the structural index. Should be moved to the range index.

When dealing with texts in many non-European languages, the token-based full-text index produces insufficient results. Tokenization is currently based on Unicode code points. Most chinese characters, for example, are thus stored as single tokens. Users have to abuse the near() or phrase() function to search for character sequences consisting of more than one character, which is quite slow. It also means that real proximity searches are not available.

An N-gram based index would be much more suitable for these languages. It would also allow additional functionality to be implemented, e.g. to deal with varying spellings. The main question is how the N-gram index would integrate conceptually with the existing full-text functions.

A N-gram index based on the new modularized indexing architecture is in SVN trunk as of July 2007 and in eXist 1.2 and later releases.

eXist now offers spatial indexes in SVN trunk as of July 2007 and in version 1.2 and later releases.

Order-by expressions and other functions that need to access atomized nodes are not supported by indexes.

Maybe part of FT index redesign.

The journal log and the recovery manager should be stable and are covered by extensive tests. However, recovery failures can not be excluded entirely. The tests can't reproduce every possible real-world scenario.

However, some steps remain for eXist to become a fully transactional database system. Transaction support is currently limited to the functionality needed for crash recovery. Though we maintain transactions internally, they are currently not exposed to applications. Also, read operations are not transactional right now.

In order to allow user-defined ACID transactions with support for rollback, all index files would need to be protected by the journaling log. The required functionality is basically available, but the feature is currently not regarded as high-priority.

No remarks available.

No remarks available.

Transactions are maintained internally, but they are not exposed to applications. eXist does not yet support full ACID transactions. Read-only operations bypass the transaction system.

Journal logs are limited to critical data required for recovery. No transaction rollbacks.

No remarks available.

Create a DB repair tool which can handle and resolve inconsistencies in the database structure. It should be possible to recreate the db if at least dom.dbx, collections.dbx and symbols.dbx are more or less intact. If a single document is damaged, it could be filtered out.

We should also think about storing redundant copies of some of the vital data blocks, so we could reconstruct the collection hierarchy from dom.dbx alone.

Main problem: access control and security.

No remarks available.

W3C released yet another working draft August 28 2007 which seems to close in on a actual draft. No merge plan yet.

The currently implemented Unix-like access control scheme is sufficient to protect resources and collections in a multi-user environment. However, it might be too coarse-grained for some types of applications. A more dynamic ACL implementation could help here. Right now, security management forms part of the database core. This is unnecessary. A more modular architecture would allow different security managers to be plugged in. It would be the responsibility of the security manager implementation to handle ACL lists.

Since version 1.1, eXist supports the XACML standard for fine-grained access control to stored XQueries, Java classes etc.

No remarks available.

Need more dynamic ACL structures that can adapt to varying requirements.

No remarks available.

No remarks available.

No remarks available.

No remarks available.

No remarks available.

No remarks available.

No remarks available.

No remarks available.

No remarks available.

No remarks available.

No remarks available.

The XQuery engine as well as the standard function libraries should be updated to align with the XQuery 1.0 recommendation. Basically, almost all core language features are implemented, excluding schema related features, which are currently beyond eXist's scope.

XQuery support in eXist is covered by the official W3C XQuery Test Suite (XQTS) 1.02. Implementing the official XQTS XQuery test suite was a top priority in order to guarantee standard conformance and avoid future regressions.

Updated to the XPath 2.0 and XQuery 1.0 recommendations. Stable, excluding schema-related features.

Updated to XPath 2.0 and XQuery 1.0 recommendations. Stable, excluding schema-related features.

Updated to XPath 2.0 and XQuery 1.0 recommendations.

Though we implement most of the serialization options specified in the XQuery and XSLT serialization spec, some options need to be reworked and should be covered by tests.

With the 1.2 release, eXist features a new query-rewriting optimizer. It analyzes the query at compile time and searches for optimizable subexpressions within the query tree. If it finds an optimizable expression, the optimizer will modify the query and wrap some special instructions around the optimizable code block. Together with the new indexing features (see blog article, the optimizer can achieve dramatic improvements.

However, the optimizer is currently limited to predicate expressions. It does not optimize e.g. "where" clauses in a FLWOR statement. The query-rewriting should thus be extended to recognize other types of expressions beyond predicate statements. In short, we need a better static analysis of the query.

Based on eXist's current indexes, it is also often difficult to decide if a certain optimization path leads to performance improvements or not. Better index statistics could help here. Also, there's a wide range of performance optimizations which could be applied if we had appropriate statistics on node distribution and frequency.

Error reports by the XQuery parser and compiler need to be improved.

Recursive functions may trigger a StackOverflowException. We need to handle tail-recursion.

Remote debugging protocol.

Command line debugger similar to jdb as prototype

Decide which functionality to expose.

XInclude expansion happens at serialization time. Queries across the included document fragments are not possible. Stable, but limited.

No remarks available.

Exposes the entire database functionality.

Does not cover administrative functions, e.g. user-management and permissions. Stable, but further functionality could be exposed.

General functionality tests required

Could be a simpler alternative to the now somewhat bloated XML:DB API

No remarks available.

Should explain in more depth how one can write webapps in XQuery, using the XQueryGenerator with Cocoon or stored XQueries.

calling XQuery scripts stored in the DB; import stored modules into a query passed to the DB.

No remarks available.

Integration with a servlet engine, Cocoon, stand-alone server, embedded use.

No remarks available.

No remarks available.

No remarks available.

XQuery functions search was improved. Missing: search in the documentation. All docs are available in XML docbook.

Migrate the function documentation to XQDoc. Use XQDoc to better document all XQuery examples.

Imperative release with new features from 2006–2007 to replace version 1.0-line. The last release based on Java 1.4.

Based on Java 5.

Provide translations for error messages, console outputs etc. At least, resource bundles should be used, so others can translate them if they want.

All libraries included with eXist need to be checked.

Change build taget in switch for development branch 1.3 to Java 5 after eXist version 1.2 is released.

This will make some more stuff easier. Especially since the jmx monitors and junit 4 tests are by requirements targeted for Java 5 already.

Change the parser to ANTLR3 which is better performant, LL* lookahead capable and processes whitespace in a better manner than ANTLR2 which we currently are using.

Maybe use gunit for testing?

Change from the current spam-ridden and unmaintained wiki to our own Atom-based AtomicWiki.