-<xlog server="chat.freenode.net" channel="#existdb" date="2008-02-27">

<message nick="Cassandra" time="00:09:00.16Z">What does it take to become an eXist developer?</message>

<message nick="perig" time="00:28:32.402Z">Blood, Sweat & Tears :-)</message>

<message nick="MoBy" time="10:16:06.342Z">morning all</message>

<message nick="MoBy" time="10:16:13.581Z">i have a question</message>

<message nick="MoBy" time="10:22:48.578Z">i have data spreaded over (mostly mysql) databases</message>

<message nick="MoBy" time="10:23:17.387Z">i query them, build xml and insert this xml in exist</message>

<message nick="MoBy" time="10:24:14.254Z">currently i have build a PHP script with the XML:DB solution to insert the data</message>

<message nick="MoBy" time="10:24:22.736Z">however this seems to be very slow</message>

<message nick="MoBy" time="10:25:03.161Z">1000 inserts in around 30 seconds</message>

<message nick="perig" time="10:25:24.139Z">doesn't seem too bad, does it ?</message>

<message nick="MoBy" time="10:25:57.885Z">mhhh when we go into production i will be inserting around 200k documents every night</message>

<message nick="MoBy" time="10:26:08.89Z">so i need the optimal strategy</message>

<message nick="perig" time="10:26:38.35Z">what is the average size of the insert ?</message>

<message nick="MoBy" time="10:26:57.085Z">couple k</message>

<message nick="perig" time="10:27:35.748Z">do ou plan to use fulltext queries ?</message>

<message nick="wolf77" time="10:28:02.044Z">and you want to insert 100k *new* docs every night?</message>

<message nick="MoBy" time="10:28:33.306Z">yeah, well it might grow to that</message>

<message nick="MoBy" time="10:29:08.974Z">it are small docs spreaded over collections per day</message>

<message nick="MoBy" time="10:29:37.303Z">like; /db/project/transactions/sitename/2008/2/25/</message>

<message nick="wolf77" time="10:30:00.484Z">but then your db could easily grow beyond a few GB?</message>

<message nick="perig" time="10:30:07.905Z">that is 200,000 x 2 kb = 400 Mb a night.</message>

<message nick="MoBy" time="10:30:09.847Z">we want to store unaggregated visitor information for multiple sites</message>

<message nick="MoBy" time="10:30:48.99Z">is that a problem?</message>

<message nick="wolf77" time="10:31:30.517Z">yes. eXist is not just a dump data store. it indexes every document.</message>

<message nick="MoBy" time="10:32:29.056Z">so it will become very slow when it holds millions of docs?</message>

<message nick="wolf77" time="10:33:01.36Z">I would not recommend to keep more than 10 to 50 gb in there</message>

<message nick="MoBy" time="10:34:01.562Z">commercial or opens.</message>

<message nick="wolf77" time="10:34:05.587Z">yes, but they are expensive ;-)</message>

<message nick="wolf77" time="10:34:16.793Z">marklogic can handle huge data sets</message>

<message nick="wolf77" time="10:34:27.207Z">but the question is: does it really make sense for your app?</message>

<message nick="MoBy" time="10:35:06.24Z">wolf77: i can explain the idea quite simple, maybe you can point me in the right direction</message>

<message nick="MoBy" time="10:36:08.474Z">i will be accessing multile databases with visitor information (Mysql, Oracle, MSSQL, etc.) </message>

<message nick="MoBy" time="10:36:29.963Z">these are all stored in different manners (different nodenames, different nodes, etc.)</message>

<message nick="MoBy" time="10:36:50.892Z">i need to access them all and store the data in a generic document</message>

<message nick="MoBy" time="10:37:17.588Z">which allows me to query the different data with a single query</message>

<message nick="MoBy" time="10:39:09.3Z">my idea was to convert all the different data sets to an XML doc. which conforms to a XML Schema</message>

<message nick="MoBy" time="10:39:22.868Z">using some middleware solution</message>

<message nick="MoBy" time="10:40:18.073Z">then i will have an olap database with all the data on which i can run scheduled report queries</message>

<message nick="wolf77" time="10:40:37.029Z">ok, I see. the problem is that every xml db needs a lot of optimizations to run </message>

<message nick="wolf77" time="10:40:37.339Z">properly on large data sets.</message>

<message nick="wolf77" time="10:41:00.05Z">it's not as easy as on a relational system</message>

<message nick="wolf77" time="10:41:30.321Z">do you really need to extract *all* the data or can you already summarize it </message>

<message nick="wolf77" time="10:41:30.462Z">when retrieving it from the rdbms?</message>

<message nick="MoBy" time="10:41:58.98Z">i can aggregate it beforehand until a certain point</message>

<message nick="wolf77" time="10:44:13.924Z">if you could reduce it from 400mb a night to 40, things would become easier ;-)</message>

<message nick="MoBy" time="10:45:56.293Z">mhhh, the problem is, when me/the customer decides that a new kind of report is needed, i cannot do so on historical data, which is a must have since he might want to compare the current month with the same month in the last year</message>

<message nick="MoBy" time="10:46:26.654Z">since i will lose date when i aggregate, certainly on a 1/10 scale with the current data</message>

<message nick="ljo" time="10:48:30.081Z">Maybe you can put a gzipped binary of the data in there for reexploitation?</message>

<message nick="wolf77" time="10:48:34.748Z">well, I was just working with a 27 GB database and performance was ok with some </message>

<message nick="wolf77" time="10:48:34.808Z">tweaks. but it certainly costs a lot.</message>

<message nick="wolf77" time="10:48:53.093Z">(costs in terms of memory and cpu)</message>

<message nick="MoBy" time="10:49:26.498Z">and marklogic can handle a DB of a TB?</message>

<message nick="wolf77" time="10:49:57.941Z">I never tried it, but it is said to be able to. however, you will need some </message>

<message nick="wolf77" time="10:49:58.001Z">tweaking and optimization as well.</message>

<message nick="ljo" time="10:50:14.143Z">well you will need a hefty machine in either case.</message>

<message nick="MoBy" time="10:50:28.793Z">ok, hardware aint an issue</message>

<message nick="MoBy" time="11:14:01.321Z">ok, i might aggreagate every 4 weeks and then delete the transactional data, this meens that the DB grows every 4 weeks up to a max of 20GB</message>

<message nick="MoBy" time="11:14:52.951Z">and then shrinks again to like a max of 1GB</message>

<message nick="MoBy" time="11:15:09.243Z">what kind of hardware would i need for this</message>

<message nick="MoBy" time="11:16:06.501Z">when you think of the fact that i have to insert 150k documents every night and then query the entire DB to generate reports</message>

<message nick="wolf77" time="11:32:10.42Z">it depends on your data and queries: I guess the data will be rather flat and </message>

<message nick="wolf77" time="11:32:10.45Z">regularily structured?</message>

<message nick="MoBy" time="11:32:50.174Z">queries are quite intensive thoug</message>

<message nick="wolf77" time="11:33:17.782Z">if yes, you can work a lot with indexes and you an expect eXist to become </message>

<message nick="wolf77" time="11:33:17.872Z">faster (I have some pending optimizations)</message>

<message nick="MoBy" time="11:33:22.538Z">since they involve alot of documents</message>

<message nick="wolf77" time="11:33:36.137Z">the number of documents alone doesn't matter</message>

<message nick="wolf77" time="11:33:45.91Z">will there be a lot of joins between documents?</message>

<message nick="wolf77" time="11:33:49.184Z">those are slow.</message>

<message nick="wolf77" time="11:34:23.04Z">(for eXist it doesn't matter much if it is querying one large doc or many small </message>

<message nick="MoBy" time="11:34:50.307Z">things like this will happen alot</message>

<message nick="MoBy" time="11:34:52.28Z">collection("/inviso/transactions/Brinkman")//statistics/module[@id='2']/detail[@key='contentitem']</message>

<message nick="MoBy" time="11:35:22.331Z">and counts over results of such a query</message>

<message nick="wolf77" time="11:36:52.353Z">this should be manageable</message>

<message nick="wolf77" time="11:38:34.672Z">30, 40, maybe 50</message>

<message nick="wolf77" time="13:21:22.559Z">Hello again</message>

<message nick="DiZzZz" time="13:35:41.295Z">functions.xq : org.apache.cocoon.ProcessingException: XMLDBException occurred: Call to undeclared function: ngram:contains [at line 73, column 38]: org.xmldb.api.base.XMLDBException: Call to undeclared function: ngram:contains [at line 73, column 38]</message>

<message nick="DiZzZz" time="13:35:54.163Z">did we have recent changes? checking....</message>

<message nick="DiZzZz" time="13:40:56.614Z">hi helicopter</message>

<message nick="DiZzZz" time="13:41:13.827Z">please could you send me some details on the xerces dependancy you found?</message>

<message nick="DiZzZz" time="13:41:40.874Z">which parser, stacktrace. how to install parser, configure... that kind of stuff</message>

<message nick="wolf77" time="13:47:04.674Z">DiZzZz: the ngram index module is required for the function search</message>

<message nick="DiZzZz" time="13:47:38.57Z">sure, but why does it claim the contains() is not there....</message>

<message nick="DiZzZz" time="13:48:15.56Z">checking....</message>

<message nick="wolf77" time="13:53:57.314Z">either it wasn't compiled or disabled in conf.xml</message>

<message nick="DiZzZz" time="14:42:52.37Z">after a clean build it was oki indeed</message>

<message nick="wolf77" time="17:41:48.239Z">perig: did you try Piotr's latest changes with the XQTS?</message>

<message nick="perig" time="17:42:22.295Z">no : will try that in a few mins ; I fear some regressions though...</message>

<message nick="perig" time="17:45:47.023Z">my cousin, who works for the french railway company, tells me that is works much better than expected :-)</message>

<message nick="wolf77" time="17:46:39.334Z">yeah, but last time I tried I was cut off when getting too far south ;-)</message>

<message nick="wolf77" time="17:48:49.071Z">west is .... France</message>

<message nick="newbie2" time="18:28:02.237Z">hello</message>

<message nick="newbie2" time="18:28:33.91Z">is the way to store in database char <?</message>

<message nick="newbie2" time="18:29:13.003Z">i can't find possible escape sequence</message>

<message nick="perig" time="18:30:01.228Z">what "escape sequences" did you try ?</message>

<message nick="newbie2" time="18:30:02.46Z">i need to store character 'lower than' in database</message>

<message nick="newbie2" time="18:30:24.029Z">i tryed to construct element with [!CDATA[</message>

<message nick="newbie2" time="18:30:41.002Z">but it is not possible to use amp as i know</message>

<message nick="perig" time="18:31:22.248Z">why not ? this syntax is quite common in the XML world....</message>

<message nick="newbie2" time="18:31:44.989Z">Does eXist support CDATA xml section?</message>

<message nick="newbie2" time="18:32:21.228Z">can you advise me to try something?</message>

<message nick="perig" time="18:32:45.892Z">I'mafraid you don't say enough to get a valuable answer</message>

<message nick="newbie2" time="18:33:42.979Z">how to store ampersand in database via XQuery?</message>

<message nick="newbie2" time="18:33:47.105Z">is that possible?</message>

<message nick="perig" time="18:34:28.191Z">you have to be aware that you dont store characters, you store *nodes*</message>

<message nick="newbie2" time="18:36:15.156Z">well, some text can be assign to some node... can that text contains &?</message>

<message nick="perig" time="18:36:53.198Z">yes test nodes can contain &, <, >, " or '</message>

<message nick="newbie2" time="18:37:03.682Z">java.rmi.RemoteException: query execution failed: XPST0003 : Ampersands (&) must be escaped (following character was not a name start character).</message>

<message nick="newbie2" time="18:37:37.969Z">let $pText := "&"</message>

<message nick="newbie2" time="18:37:38.019Z">return $pText</message>

<message nick="newbie2" time="18:37:46.13Z">what should i do to make it work?</message>

<message nick="perig" time="18:38:33.854Z">escape it, just like the error message says. However, I'm not sure how rev. 4311 behaves with this...</message>

<message nick="newbie2" time="18:39:58.76Z">you mean i need to work in my escape sequence?</message>

<message nick="newbie2" time="18:40:25.726Z">on the get/set methods</message>

<message nick="newbie2" time="18:40:58.2Z">i thought that should be some way to escape it by database</message>

<message nick="newbie2" time="18:44:30.699Z">may be there is some way to store character by its code</message>

<message nick="newbie2" time="18:44:35.936Z">for example</message>

<message nick="newbie2" time="18:44:55.953Z">it would be just great</message>

<message nick="newbie2" time="18:45:33.314Z">as i know in xml specification there is no way to escape something without ampersand</message>

<message nick="newbie2" time="18:46:00.01Z">am i right?</message>

<message nick="perig" time="19:00:15.142Z">for wolf77 : only 1 regression</message>

<message nick="newbie2" time="19:10:27.704Z">perig: do you know escape sequence for < or >, i know that for " is "", ' -> '', { -> {{, } -> }}</message>

<message nick="newbie2" time="19:11:53.801Z">it's something obvious here but i can't find the solution, i still have not decided to store information in base64 format </message>

<message nick="DiZzZz" time="20:55:25.057Z">huhuhu</message>

<message nick="DiZzZz" time="20:59:44.699Z">ok i commited my last changes of the new validation functions. Schematron, RelaxNG and oNVDL are now supported...... </message>

<message nick="DiZzZz" time="21:00:11.185Z">hmmm not new functions, extensions of the extension functions</message>

<message nick="gbarrei" time="22:08:44.261Z">Hello everyone</message>

<message nick="gbarrei" time="22:09:07.573Z">do you guys have time for a question ?</message>

<message nick="gbarrei" time="22:09:14.472Z">other than the one before</message>

<message nick="gbarrei" time="22:09:32.587Z">I am having problems with a recursive function</message>

<message nick="gbarrei" time="22:09:55.909Z">it is not getting to all the nested nodes it needs to</message>

<message nick="gbarrei" time="22:10:03.239Z">and I can't quite understand why</message>

<message nick="gbarrei" time="22:10:10.609Z">example</message>

<message nick="gbarrei" time="22:12:58.587Z">function myFunc($qst) { <QST> { $qst/@*, for $rsp in $qst/Response return <RSP> { $rsp/@*, $rsp/Elem1, for $rspqst in $rsp return myFunc($qst) } <RSP>, $qst/Elem1 } <Qst></message>

<message nick="gbarrei" time="22:13:18.324Z">cannot reach <QST> nodes that are 5-deep nested</message>

<message nick="gbarrei" time="22:13:21.278Z">whilst </message>

<message nick="gbarrei" time="22:14:13.829Z">function myFunc($qst) { <QST> { $qst/@*, for $rsp in $qst/Response return <RSP> { $rsp/@*, for $rspqst in $rsp return myFunc($qst), $rsp/Elem1 } <RSP>, $qst/Elem1 } <Qst> </message>

<message nick="gbarrei" time="22:15:42.469Z">The difference is the order in which I try to retrieve the elements</message>

<message nick="gbarrei" time="22:16:49.03Z">sorry, IRC doesn't like me</message>

<message nick="perig" time="22:17:14.935Z">let me think</message>

<message nick="perig" time="22:19:11.453Z">first what version of eXist ?</message>

<message nick="gbarrei" time="22:19:21.096Z">still here</message>

<message nick="gbarrei" time="22:21:01.973Z">close to the 1.2.0 release</message>

<message nick="gbarrei" time="22:21:14.07Z">let me check</message>

<message nick="gbarrei" time="22:22:05.87Z">trunk of early november 2007</message>

<message nick="gbarrei" time="22:22:20.81Z">eXist is running embeded</message>

<message nick="gbarrei" time="22:22:28.581Z">using Tomcat as server</message>

<message nick="perig" time="22:22:38.494Z">mmmh.... we've had some troubles with the in-memory tree builder in the past. Not sure if they are all fixed, but we've made some progress here</message>

<message nick="perig" time="22:23:12.901Z">Ideally, you should provide a minimal document to reproduce the sissue</message>

<message nick="gbarrei" time="22:23:30.996Z">XML document and xquery ?</message>

<message nick="perig" time="22:23:34.14Z">and more ideally, one that fits into a let statement</message>

<message nick="perig" time="22:23:54.468Z">... so just one XQuery :-)</message>

<message nick="perig" time="22:24:26.701Z">I could quickly confirm/infirm the bug</message>

<message nick="ljo" time="22:43:55.399Z">night</message>

-<message nick="perig" time="22:44:52.847Z">

ljo : I've found a regression in the XQTS. see

<a href="http://demo.exist-db.org/irclog/irclog.xql">http://demo.exist-db.org/irclog/irclog.xql</a>

, 19:00 (GMT)

</message>

<message nick="perig" time="22:45:43.646Z">this is the only side-effect of Piotr's work that I know of</message>

<message nick="ljo" time="22:46:12.985Z">ok, thanks will look at it tomorrow. Got a really bad cold today.</message>

</xlog>