I'm trying to index several tables (20+) with some of them having 500000+ rows.
The problem is that when indexing the heap memory goes up and the GarbageCollector can't free memory. After a few tables an OutOfMemoryException is thrown.
I'm using JBoss 4.2.3 and spring. The transaction processor used is lucene and the snapshot persister is FSSnapshotPersister.
Can you tell what takes most of the memory? One of the problems of the jdbc device is the fact that, event with the snapshot persister, it still stores the all the rows primary keys and versions in memory, which might explain why it takes so much memory.
That cannot be the problem, because the snapshot file saved in the disk is too small compared to the occupied RAM.
For a 20.000 line table, the snapshot file after indexing is about 1.5MB and the RAM size is 100MB.
This size discrepancy shows that there is some real problem in the release of objects going on.
I was wrong. After a close analyses of the Heap with jvisualvm, the RAM increases with the indexing but is reclaimed successfully by the GC after that.
One thing I noticed is that the RAM is always increasing while indexing a alias (i.e. table), and the GC only kicks in after the indexing is over (and the mirroring starts), until then the RAM just keeps growing until the JVM -Xmx setting. I think this is because the transaction is only closed at the end.
Do you think it would make sense to make available a configuration that at the ResultSetSnapshotEventListener.doCreateAndUpdateFor method called the session.flush() after processing X JdbcAliasRowSnapshots, to avoid this unstoppable increase of RAM while indexing a single alias/table?
I am not sure that this is the reason why the memory is not reclaimed. With the lucene transaction isolation, flush actually does not do much. Also, I thought that the problem is with indexing, and ResultSetSnapshotEventListener is used when mirroring.
We are just using mirroring, so the "indexing" is made the first time it runs the version query and compares what is in the DB to what is on the index (nothing at the first), what causes the mirroring to index all the content.
I think I solved the problem, at least that what it seams in the tests.
The problem was because in this first "indexation" the snapshot list is huge, and it was only available to GC at the very end of the method.
We changed the method to that at a defined flushSize the list already processed items where cleared from the list and System.gc() is called.
This way, we don't need to wait for the end of the method for all the JdbcAliasRowSnapshots of the lists be available for GC, because clearing them form the list makes them available for GC. We also called flush before calling System.gc().
Sports brand Nike (Nike) has announced the new 2010 summer footwear, including the nike air max, Tennis Classic, and Air Max 90 Current Torch three groups. nike shox which are yellow, orange, and blue color, with some bright colors there is a refreshing feeling. Tennis Classic with a white and green color, low-cut shoes suitable for summer wear. And with the nike air jordan of the latest technology Torch nike shoes uk Current Torch shoes are hit with the classic black and white color with the toe position with the Torch materials, mining forefoot nike air max 2011 Free technology coupled with the classic appearance , reflecting the retro and technology integration.