-<xlog server="chat.freenode.net" channel="#existdb" date="2008-04-02">

<message nick="wolf77" time="09:01:48.467+01:00">Morning. Why the heck should it not be allowed to use fn:id on in-memory nodes? I really like to do this.</message>

<message nick="perig" time="09:03:50.292+01:00">Hi. Because, as per the specs, we have no document ?</message>

<message nick="wolf77" time="09:04:54.379+01:00">Just try the following: edit webapp/xqts/config.xml and set <mode>memory</mode></message>

<message nick="wolf77" time="09:05:40.102+01:00">then you'll see a lot of failures concerning missing fn:id support.</message>

<message nick="wolf77" time="09:06:46.271+01:00">please roll back. I'm relying on this.</message>

<message nick="perig" time="09:06:55.053+01:00">Then, it would be nice to have a short test case to demonstrate the issue</message>

<message nick="wolf77" time="09:07:16.492+01:00">Hey, I spent a day to implement fn:id on in-memory nodes.</message>

<message nick="wolf77" time="09:07:24.383+01:00">Disabling it without asking is not PC</message>

<message nick="wolf77" time="09:07:41.737+01:00">The test cases are in the XQTS.</message>

<message nick="perig" time="09:09:13.351+01:00">Well, they are... if the appropriate switch is set</message>

<message nick="perig" time="09:09:32.037+01:00">I'll roll back (without testing)</message>

<message nick="wolf77" time="09:10:02.197+01:00">Just set the XQTS to in-memory processing and you'll see</message>

<message nick="wolf77" time="09:10:52.055+01:00">From now on, the XQTS has to be run against 2 implementations.</message>

<message nick="perig" time="09:13:58.348+01:00">Just wondering how many we are :-)</message>

<message nick="wolf77" time="09:18:37.497+01:00">Most of the XQTS failures can be fixed easier once we have a new parser. So I would just leave the XQTS as it is for now. eXist has serious problems in other areas.</message>

<message nick="perig" time="09:20:08.631+01:00">I'm working on bugs that are not related to the parser.</message>

<message nick="wolf77" time="09:20:53.893+01:00">What would be most useful for me concerning standard compliance: support for the missing module-related features!</message>

<message nick="wolf77" time="09:23:06.643+01:00">Modules split across several files. Support for "external" in function declarations.</message>

<message nick="wolf77" time="09:29:18.868+01:00">"external" would allow us to provide a proper module definition for those modules implemented in Java, thus simplifying documentation.</message>

<message nick="wolf77" time="09:29:40.027+01:00">We could finally switch to xqdoc then.</message>

<message nick="zOOd" time="09:32:54.441+01:00">hello.</message>

-<message nick="zOOd" time="09:34:53.273+01:00">

<a href="http://pastebin.com/m7dbf0923">http://pastebin.com/m7dbf0923</a>

shows a try of mine to access an exist database. but it fails: <mission id= ...> seems wrong, i dont understand why. any idea ?

</message>

<message nick="sergrinc" time="09:35:42.71+01:00">Hi, good news (GSoC related). I have to research about distributed databases ...</message>

<message nick="perig" time="09:35:55.127+01:00">what's the failure</message>

<message nick="sergrinc" time="09:36:24.036+01:00">for my Databases postgrad. course and planned work for distributed search for ...</message>

<message nick="perig" time="09:36:53.075+01:00">ah ! so obvious, missing quotes</message>

<message nick="sergrinc" time="09:37:14.184+01:00">eXist is part of that! So, GSoC yes or not, I hope to help in the future.</message>

<message nick="zOOd" time="09:39:00.659+01:00">perig: pasted failure. Event with quotes same failure (unexpected token)</message>

<message nick="sergrinc" time="09:39:15.379+01:00">While I polish GSoC application (with your help) I have few questions and suggestions.</message>

<message nick="zOOd" time="09:39:22.869+01:00">replaced with <mission> ... same failure.</message>

<message nick="perig" time="09:40:09.752+01:00"><instrument> has the same problem</message>

<message nick="sergrinc" time="09:41:33.917+01:00">First: referencing nodes is described in XQueryP: An XML Application Development Language ...</message>

<message nick="sergrinc" time="09:42:07.242+01:00">which is development work of XQuery W3C group for XQuery Scripting Extensions.</message>

<message nick="zOOd" time="09:43:32.588+01:00">perig: pasted corrected code. same failure. do you want to try on my sandbox ?</message>

<message nick="sergrinc" time="09:43:47.748+01:00">Especially, two functions are mentioned: fn:ref and fn:deref. I think any eXist</message>

<message nick="perig" time="09:44:24.348+01:00">zOOD : pasted... where ?</message>

<message nick="sergrinc" time="09:44:30.697+01:00">module which would reference nodes should use these functions.</message>

-<message nick="zOOd" time="09:44:51.395+01:00">

perig:

<a href="http://pastebin.com/m31e4633f">http://pastebin.com/m31e4633f</a>

</message>

<message nick="sergrinc" time="09:45:05.224+01:00">E.g. it would be just renaming them in the future if standard changes.</message>

<message nick="zOOd" time="09:45:07.637+01:00">oops, thougt paste bin would keep track of number.</message>

-<message nick="wolf77" time="09:45:14.977+01:00">

sergrinc: are you referring to what is now called XQuery Scripting Extension 1.0?

<a href="http://www.w3.org/TR/xquery-sx-10/">http://www.w3.org/TR/xquery-sx-10/</a>

</message>

-<message nick="sergrinc" time="09:46:39.902+01:00">

wolf77: yes and no. On the main XQuery page there's link above this specs e.g.

<a href="http://2006.xmlconference.org/proceedings/38/presentation.pdf">http://2006.xmlconference.org/proceedings/38/presentation.pdf</a>

</message>

<message nick="wolf77" time="09:47:17.123+01:00">yeah, but I think that's what became xquery-sx</message>

<message nick="perig" time="09:47:56.436+01:00">zOOd : indeed you have a problem with the "id" attribute</message>

<message nick="perig" time="09:48:15.042+01:00">please give me a few mins (task to complete)</message>

<message nick="perig" time="09:48:23.283+01:00">will come back on modules after that ;-)</message>

<message nick="sergrinc" time="09:48:25.616+01:00">wolf77: hmm, but where are some functions proposed then?</message>

<message nick="zOOd" time="09:48:55.737+01:00">perig: ok, thx.</message>

<message nick="wolf77" time="09:49:09.435+01:00">Looks like they dropped them? Which is not a big problem since new functions can be defined anytime.</message>

<message nick="sergrinc" time="09:49:51.042+01:00">I see. OK. Second question: is there any documentation about federated database ...</message>

<message nick="sergrinc" time="09:50:06.994+01:00">in eXist. I mean clusters are not the same thing.</message>

<message nick="perig" time="09:50:10.719+01:00">zOOd : how stupid I am...</message>

<message nick="perig" time="09:50:33.299+01:00">no return clause...</message>

<message nick="perig" time="09:51:58.906+01:00">(never do 3 things at the same time)</message>

<message nick="sergrinc" time="09:52:09.34+01:00">Ups, I've mean't distributed database (parts of database residing on different eXist instances/servers).</message>

<message nick="wolf77" time="09:52:48.763+01:00">sergrinc: well, eXist doesn't support distributed databases yet. You can replicate databases, but that's not the same thing.</message>

<message nick="zOOd" time="09:53:27.616+01:00">perig: right. thx.</message>

<message nick="sergrinc" time="09:56:31.376+01:00"> So, what's the plan for configuring distributed database? Include details in ...</message>

<message nick="sergrinc" time="09:56:42.601+01:00">... conf.xml? And then when searching spawning several threads conditionally ...</message>

<message nick="sergrinc" time="09:56:48.729+01:00">based on one server or several servers?</message>

<message nick="wolf77" time="09:57:25.299+01:00">no, it won't be as easy as that. I see several possibilities:</message>

<message nick="wolf77" time="09:57:52.075+01:00">1) distribute a single query across multiple db instances and merge the results of each query into a main result.</message>

<message nick="wolf77" time="09:58:38.829+01:00">2) distribute a certain query block to multiple db instances, merge the results and post-process them in the main query</message>

<message nick="wolf77" time="09:59:50.065+01:00">from a user point of view, the ideal solution would probably be: specify the db instance to be queried in an URI which can be passed to collection()</message>

-<message nick="wolf77" time="10:00:50.317+01:00">

let $a := collection("

<a href="http://instanceA.com">http://instanceA.com</a>

")//test, $b := collection("

<a href="http://instanceB.com">http://instanceB.com</a>

")//test return $a union $b

</message>

<message nick="sergrinc" time="10:01:38.142+01:00">Yes, that's what I wanted to read :-). Hmm, have you read some academic papers ...</message>

<message nick="sergrinc" time="10:02:12.309+01:00">about including e.g. "execute at (server)" part of query? Not standard, just like to know your opinion.</message>

<message nick="wolf77" time="10:02:33.898+01:00">the query engine would then need to detect that instanceA and instanceB are external db instances. And instead of executing the corresponding path expression on the local instance, it forwards them to the external servers.</message>

<message nick="wolf77" time="10:03:25.909+01:00">no, I have not read any papers about this topic. Maybe there are some, but my considerations are purely practical.</message>

<message nick="sergrinc" time="10:04:47.449+01:00">Aha, logical. That's where that "execute" XQuery comes into picture (so user basically commands that this part of query should be executed on that server. Authors probably wanted to simplify things for parsing query.</message>

<message nick="perig" time="10:05:58.386+01:00">from eXist's point of view, it would consist in having solid resolvers (which are able to detect internal resources vs. external ones) and computing dependencies in order to be able to run/block parallel tasks. </message>

<message nick="sergrinc" time="10:06:02.051+01:00">E.g. if I know the database URI then I probably know its server :-).</message>

-<message nick="wolf77" time="10:07:22.079+01:00">

Instead of using collection(), we could also create a pseudo-function execute("

<a href="http://instanceA.com">http://instanceA.com</a>

", //test). However, collection() is already there and part of the standard.

</message>

<message nick="sergrinc" time="10:07:59.43+01:00">perig: yep. I do not fear about technical part (though nonblocking, timeouts etc are yet to come standard in Java). Just first, XQuery form part is important to agree on.</message>

<message nick="perig" time="10:09:13.18+01:00">Personnally, I'd fear about technically considering resource resolving in eXist again :-)</message>

<message nick="sergrinc" time="10:09:18.558+01:00">Yes, I agree that internal resolver(s) should simply compare running instance URI to those entered in XQuery.</message>

<message nick="wolf77" time="10:10:12.861+01:00">Also, the resolving here has to occur at compile time.</message>

<message nick="sergrinc" time="10:10:13.192+01:00">perig: well, if it's in source, I'll go through it.</message>

<message nick="wolf77" time="10:11:26.381+01:00">Or well, not necessarily. I was just thinking about expressions like collection($someExternalVar)</message>

<message nick="sergrinc" time="10:11:53.368+01:00">wolf77: about technical implementation - Adam has written me about SAX-like returning of results e.g. as they come from different URIs.</message>

<message nick="wolf77" time="10:11:58.765+01:00">but we can handle that at runtime.</message>

<message nick="perig" time="10:12:03.692+01:00">reaading the XQueryP paper, interesting and pretty close to what we have recently discussed on the channel :-)</message>

<message nick="wolf77" time="10:12:27.694+01:00">not close enough to be really helpful though</message>

<message nick="wolf77" time="10:13:27.416+01:00">though the notion of a "block" would certainly help as a basis, e.g. for locking</message>

<message nick="perig" time="10:13:30.97+01:00">reference of copy ? that is the question :-)</message>

<message nick="sergrinc" time="10:13:36.137+01:00">Aha. About streaming returned results in distributed search - this obviously stands only for not ordered results, right?</message>

<message nick="wolf77" time="10:14:24.683+01:00">Streaming back and merging the results is pretty simple</message>

<message nick="wolf77" time="10:14:37.16+01:00">the rules for ordering are pretty clear in XQuery</message>

<message nick="wolf77" time="10:15:47.816+01:00">After calling the external instance, you will get a HTTP response which can be parsed into an in-memory DOM. This can then be further processed like any local document.</message>

<message nick="sergrinc" time="10:16:24.206+01:00">Mhm, I think Adam mean't to start returning results to the user as they are parsed incompletely (not all results gathered from all URIs).</message>

<message nick="sergrinc" time="10:16:43.922+01:00">Sorry for a typo.</message>

<message nick="wolf77" time="10:17:29.414+01:00">Mmmh, streaming results would be a completely different topic. Not a good idea.</message>

<message nick="sergrinc" time="10:17:58.384+01:00">OK :-). It sounded little too futuristic.</message>

<message nick="wolf77" time="10:18:26.562+01:00">Too merge results in document order, you will always need to retrieve the entire set first.</message>

<message nick="sergrinc" time="10:20:27.676+01:00">OK, now I have a lot more information than on Monday (got fever, lost six hours but fortunately GSoC extension appeared).</message>

-<message nick="sergrinc" time="10:21:47.515+01:00">

DiZzZz: just like to ask about the book he was referring, doing PSVI. Last year I've read Querying XML (

<a href="http://www.elsevierdirect.com/product.jsp?isbn=9781558607118">http://www.elsevierdirect.com/product.jsp?isbn=9781558607118</a>

) ...

</message>

<message nick="sergrinc" time="10:23:29.613+01:00">which is still the best on the subject and from one of (Oracle) members of W3C SQL and XQuery groups (Jim Melton).</message>

<message nick="sergrinc" time="10:24:57.313+01:00">I remember that all technical details about Infosets, PSVI etc were clearly explained (better than in formal standard) but I just haven't needed that.</message>

<message nick="sergrinc" time="10:27:36.308+01:00">I'm still not well so leaving for now. Thank you, guys!</message>

<message nick="wolf77" time="10:27:51.669+01:00">Ok, I'll forward your recommendation to DiZzZz</message>

<message nick="a" time="11:55:36.79+01:00">i have a realy BOG problem with the eXist db</message>

<message nick="a" time="11:55:53.703+01:00">I means big problem :)</message>

<message nick="a" time="11:56:55.207+01:00">we are storing some date in the DB, and in case of exception, we are storing the whole stacktrace in a xml-file in the exist-db</message>

<message nick="angel" time="11:58:43.223+01:00">I don't know wath exactly the problem is,</message>

<message nick="angel" time="11:59:16.589+01:00">I get an exception</message>

<message nick="angel" time="11:59:57.524+01:00">here is the stacktrace</message>

<message nick="angel" time="11:59:59.016+01:00">org.xmldb.api.base.XMLDBException: java.lang.StringIndexOutOfBoundsException: String index out of range: -3245</message>

<message nick="angel" time="11:59:59.347+01:00"> at org.exist.xmldb.RemoteXMLResource.getContent(RemoteXMLResource.java:146)</message>

<message nick="angel" time="11:59:59.397+01:00"> at org.exist.client.QueryDialog$QueryThread.run(QueryDialog.java:421)</message>

<message nick="angel" time="11:59:59.427+01:00">Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.StringIndexOutOfBoundsException: String index out of range: -3245</message>

<message nick="angel" time="12:00:00.428+01:00"> at org.apache.xmlrpc.XmlRpcClient$Worker.execute(XmlRpcClient.java:457)</message>

<message nick="angel" time="12:00:02.381+01:00"> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:163)</message>

<message nick="angel" time="12:00:04.384+01:00"> at org.exist.xmldb.RemoteXMLResource.getContent(RemoteXMLResource.java:144)</message>

<message nick="angel" time="12:00:08.389+01:00">Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.StringIndexOutOfBoundsException: String index out of range: -3245</message>

<message nick="angel" time="12:00:11.443+01:00"> at org.apache.xmlrpc.XmlRpcClient$Worker.execute(XmlRpcClient.java:457)</message>

<message nick="angel" time="12:00:13.436+01:00"> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:163)</message>

<message nick="angel" time="12:00:15.389+01:00"> at org.exist.xmldb.RemoteXMLResource.getContent(RemoteXMLResource.java:144)</message>

<message nick="angel" time="12:00:17.371+01:00"> at org.exist.client.QueryDialog$QueryThread.run(QueryDialog.java:421)</message>

<message nick="angel" time="12:07:08.119+01:00">the exception is thrown both if I try to read da data over XQuery and if I try to export the whole xml file or over the WebDAV interface</message>

<message nick="angel" time="12:07:56.525+01:00">any idea how we can get the data from the database</message>

<message nick="angel" time="12:08:22.5+01:00">this is a live enviroment</message>

<message nick="angel" time="12:09:08.453+01:00">and we DON'T HAVE up-to-date backup</message>

<message nick="wolf77" time="12:10:01.625+01:00">could you post the server-side exception please?</message>

<message nick="wolf77" time="12:10:12.429+01:00">the client side error doesn't tell anything</message>

<message nick="wolf77" time="12:13:43.857+01:00">just tell me what the first lines of the relevant exception are</message>

<message nick="wolf77" time="12:13:55.012+01:00">I don't want the whole thing</message>

<message nick="angel" time="12:17:52.654+01:00">2008-04-02 13:21:13,873 [P1-9] DEBUG (WebApplicationHandler.java [dispatch]:471) - call servlet org.exist.xmlrpc.RpcServlet[org.exist.xmlrpc.RpcServlet] </message>

<message nick="angel" time="12:17:52.965+01:00">2008-04-02 13:21:13,882 [P1-9] DEBUG (Paged.java [read]:982) - error while reading page: page: 7539513103360; file = dom.dbx; address = 6db6e001401000; page header = 64; data start = 6db6e001401040 </message>

<message nick="angel" time="12:17:52.985+01:00">java.io.IOException: Invalid argument</message>

<message nick="angel" time="12:17:52.995+01:00"> at java.io.RandomAccessFile.seek(Native Method)</message>

<message nick="angel" time="12:17:53.436+01:00"> at org.exist.storage.btree.Paged$Page.read(Paged.java:968)</message>

<message nick="angel" time="12:17:55.468+01:00"> at org.exist.storage.dom.DOMFile$OverflowDOMPage.streamTo(DOMFile.java:3158)</message>

<message nick="wolf77" time="12:18:46.758+01:00">exceptions in DOMFile (dom.dbx) are pretty bad</message>

<message nick="wolf77" time="12:19:39.139+01:00">I would recommend to stop the db. copy all the files from WEB-INF/data into a safe place. remove the .log files in there, restart the db and try to create a backup.</message>

<message nick="angel" time="12:19:42.333+01:00">I thing the same too</message>

<message nick="wolf77" time="12:20:37.398+01:00">no, you need to create a backup with the Java client</message>

<message nick="wolf77" time="12:21:20.787+01:00">i.e. a real backup of the XML in the database.</message>

<message nick="angel" time="12:21:53.562+01:00">how is this possible with the files from WEB-INF/data folder?</message>

<message nick="angel" time="12:22:29.611+01:00">because we don't have an backupt, which is up-to-date</message>

<message nick="wolf77" time="12:23:34.128+01:00">you want to extract the data, don't you?</message>

<message nick="wolf77" time="12:23:57.6+01:00">right</message>

<message nick="angel" time="12:24:14.603+01:00">do you speak german?</message>

<message nick="wolf77" time="12:24:31.496+01:00">sure, but it won't get easier ;-)</message>

<message nick="wolf77" time="12:24:55.999+01:00">1) shut down eXist</message>

<message nick="wolf77" time="12:25:19.281+01:00">2) for safety, copy all the files from WEB-INF/data somewhere else</message>

<message nick="wolf77" time="12:25:30.837+01:00">3) remove the .log files from WEB-INF/data, keep all the rest</message>

<message nick="wolf77" time="12:25:47.499+01:00">4) launch the eXist client in embedded mode: bin/client.sh -l</message>

<message nick="wolf77" time="12:25:57.984+01:00">5) try to create a backup</message>

<message nick="wolf77" time="12:26:11.312+01:00">6) if you are still getting the exception, you will need to hack the code</message>

<message nick="angel" time="12:27:05.495+01:00">ok, we'll try this now and I give you feedback</message>

<message nick="wolf77" time="12:27:26.544+01:00">ok, I need to go back cooking...</message>

<message nick="angel" time="13:03:24.154+01:00">it doesn't work :(</message>

<message nick="angel" time="13:03:43.901+01:00">we get the same exception</message>

<message nick="perig" time="13:03:54.516+01:00">we wont get very far with such a report :-)</message>

<message nick="angel" time="13:07:08.559+01:00">what information can I provide you?</message>

<message nick="perig" time="13:07:45.86+01:00">you had a 6 step process. Where does it fail ?</message>

<message nick="angel" time="13:08:47.684+01:00">when we try to create a backup, we get thesame exception</message>

<message nick="wolf77" time="13:16:45.523+01:00">so nobody helps with step 6?</message>

<message nick="angel" time="13:18:56.835+01:00">do you means the code hacking?</message>

<message nick="wolf77" time="13:20:53.807+01:00">yeah. you can probably catch that error during the backup and continue.</message>

<message nick="wolf77" time="13:21:07.439+01:00">that way you will only miss the damaged document - hopefully</message>

<message nick="angel" time="13:22:05.795+01:00">I'm mentally ready to try this ;), but the only think I've made with the code is to write one simple extension</message>

<message nick="wolf77" time="13:22:45.205+01:00">well, it has to be done on your version</message>

<message nick="angel" time="13:23:08.957+01:00">we have all the other xml files</message>

<message nick="angel" time="13:23:18.821+01:00">the backup exports them</message>

<message nick="wolf77" time="13:23:29.735+01:00">oh, so it's just this single binary you don't get?</message>

<message nick="angel" time="13:24:07.176+01:00">It's not a binary</message>

<message nick="angel" time="13:24:48.673+01:00">and we can get some data from the corrupted file over WebDAV</message>

<message nick="wolf77" time="13:25:13.857+01:00">ok, that's not bad. I fear we won't be able achieve much more.</message>

<message nick="wolf77" time="13:26:09.272+01:00">(i.e. without going down to the level of single bits and bytes)</message>

<message nick="angel" time="13:26:33.976+01:00">I'll be verry happy ;)</message>

<message nick="wolf77" time="13:27:30.893+01:00">since the WebDAV server is streaming the data, it probably retrieves all the pages which can still be read.</message>

<message nick="wolf77" time="13:27:43.74+01:00">and stops at the damaged one.</message>

<message nick="wolf77" time="13:29:28.753+01:00">the link to the next page is certainly lost. so all we could do is search for the correct page. but that would be a pain.</message>

<message nick="angel" time="13:34:48.287+01:00">I think, more painfully can't be</message>

<message nick="angel" time="13:36:00.034+01:00">I'm ready for every suggestion, that can help us to get some more data from this corrupted file</message>

<message nick="wolf77" time="13:36:33.991+01:00">what eXist are you using?</message>

<message nick="angel" time="13:40:20.398+01:00">version ist 1.1.2dev</message>

<message nick="wolf77" time="13:42:10.097+01:00">which is a version built from some SVN revision</message>

<message nick="angel" time="13:43:52.216+01:00">it was downloaded from the eXist web page and installed with the installer</message>

<message nick="wolf77" time="13:44:36.856+01:00">but the line numbers in the exception do not correspond to my sources in any way</message>

<message nick="wolf77" time="13:44:48.022+01:00">so it seems to be an older version</message>

<message nick="wolf77" time="13:45:44.919+01:00">I don't really understand the exception. Could you post the entire thing?</message>

<message nick="wolf77" time="13:47:06.72+01:00">Obviously, the data is treated as a binary stream.</message>

<message nick="angel" time="13:47:08.152+01:00">project.version=1.0rc2</message>

<message nick="angel" time="13:47:08.172+01:00">project.build=20060907</message>

<message nick="angel" time="13:47:08.202+01:00">svn.revision=4311</message>

<message nick="wolf77" time="13:47:20.048+01:00">ooooh, rev 4311</message>

<message nick="wolf77" time="13:48:04.659+01:00">well, if the data is indeed stored as a binary, you should be able to see the data by looking at dom.dbx with a hex editor</message>

<message nick="