SourceForge.net Logo
Main Overview Wiki Issues Forum Build Fisheye

Welcome, Guest
Guest Settings
Help

Compass Support Forums » Compass » Compass Users

Thread: Toplink OutOfMemoryError: Java heap space, index of 500000 records

This question is answered. Helpful answers available: 1. Correct answers available: 1.


Permlink Replies: 13 - Pages: 1 - Last Post: Dec 15, 2011 7:40 PM Last Post By: anitas
Jirka Hradil

Posts: 17
Registered: 10/2/08
Toplink OutOfMemoryError: Java heap space, index of 500000 records
Posted: Oct 3, 2008 4:46 AM
 
  Click to reply to this thread Reply
Hello,

following the document
http://www.compass-project.org/docs/2.1.0M4/reference/html/gps-embeddedtoplink.html,
I reindexed a database of 500 000 record with one alias, Company. Company has only 2 @SearchableProperty and 2 @SearchableReference. The reindexing went OK and it created an index in a filesystem with size of 800 MB.

When I try to find the indexed records I get the following exception:

java.lang.OutOfMemoryError: Java heap space
java.util.Arrays.copyOfRange(Arrays.java:3209)
java.lang.String.<init>(String.java:216)
org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:104)
org.apache.lucene.index.SegmentTermEnum.term(SegmentTermEnum.java:160)
org.apache.lucene.index.SegmentMergeInfo.next(SegmentMergeInfo.java:66)
org.apache.lucene.index.MultiSegmentReader$MultiTermEnum.next(MultiSegmentReader.java:490)
org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:390)
org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:72)
org.apache.lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl.java:352)
org.apache.lucene.search.FieldSortedHitQueue.comparatorString(FieldSortedHitQueue.java:348)
org.apache.lucene.search.FieldSortedHitQueue$1.createValue(FieldSortedHitQueue.java:201)
org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:72)
org.apache.lucene.search.FieldSortedHitQueue.getCachedComparator(FieldSortedHitQueue.java:168)
org.apache.lucene.search.FieldSortedHitQueue.<init>(FieldSortedHitQueue.java:56)
org.apache.lucene.search.TopFieldDocCollector.<init>(TopFieldDocCollector.java:43)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:122)
org.apache.lucene.search.Hits.getMoreDocs(Hits.java:100)
org.apache.lucene.search.Hits.<init>(Hits.java:77)
org.apache.lucene.search.Searcher.search(Searcher.java:55)
org.compass.core.lucene.engine.transaction.AbstractTransaction.findByQuery(AbstractTransaction.java:205)
org.compass.core.lucene.engine.transaction.lucene.LuceneTransaction.doFind(LuceneTransaction.java:149)
org.compass.core.lucene.engine.transaction.AbstractTransaction.find(AbstractTransaction.java:103)
org.compass.core.lucene.engine.LuceneSearchEngine.find(LuceneSearchEngine.java:351)
org.compass.core.lucene.engine.LuceneSearchEngineQuery.hits(LuceneSearchEngineQuery.java:189)
org.compass.core.impl.DefaultCompassQuery.hits(DefaultCompassQuery.java:158)

Java code for the query is following:

Compass compass = TopLinkHelper.getCompass(entityManagerFactory);
CompassSession session = compass.openSession();
CompassTransaction tr = session.beginTransaction();

CompassHits hits = session.find("search something"); //THIS LINE THROWS THE EXCEPTION

tr.commit();
session.close();

I tried Compass versions 2.0.2 and 2.1.0M4, Toplink Essentials version is V2.1_build_b47.

None of the Compass properties in the persistence.xml were set, except these ones:

<property name="toplink.session.customizer" value="org.compass.gps.device.jpa.embedded.toplink.CompassSessionCustomizer" />
<property name="compass.engine.connection" value="file://tmp" />

Do you any idea what I have to set or change?

Thank you for your help.

Sincerely,

Jirka Hradil
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Toplink OutOfMemoryError: Java heap space, index of 500000 records
Posted: Oct 3, 2008 3:45 PM   in response to: Jirka Hradil in response to: Jirka Hradil
 
  Click to reply to this thread Reply
How much memory are you allocating the application? Are you using sorting?
Jirka Hradil

Posts: 17
Registered: 10/2/08
Re: Toplink OutOfMemoryError: Java heap space, index of 500000 records
Posted: Oct 3, 2008 4:16 PM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
Hello,

thank you for your answer. I'm using default heap size 64 MB in Tomcat and yes, I'm using sorting in the query:

query.addSort("un_name", CompassQuery.SortPropertyType.STRING, asc ? CompassQuery.SortDirection.AUTO : CompassQuery.SortDirection.REVERSE);

Jirka Hradil
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Toplink OutOfMemoryError: Java heap space, index of 500000 records
Posted: Oct 3, 2008 4:25 PM   in response to: Jirka Hradil in response to: Jirka Hradil
Helpful
  Click to reply to this thread Reply
Then you better add more memory since sorting, when using Lucene, requires to load more information to the JVM...
Jirka Hradil

Posts: 17
Registered: 10/2/08
Re: Toplink OutOfMemoryError: Java heap space, index of 500000 records
Posted: Oct 3, 2008 4:45 PM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
Thank you Shay, I'll try it and I'll give you a report. One more question - I want to copy Compass hits inside the transaction (see comments in the code):

List<Company> result = new ArrayList<MyObject>();

CompassTransaction tr = session.beginTransaction();
CompassHits hits = query.hits(); //SHOULD I DETACH HITS HERE AND COPY DETACHED HITS TO RESULT LIST OUTSIDE THE TRANSACTION ?

for (CompassHit hit : hits) {
result.add((MyObject) hit.data()); //IS IT OK WHEN I DOING THIS INSIDE THE TRANSACTION?
}

tr.commit();
session.close();

Thank you again,

Jirka Hradil
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Toplink OutOfMemoryError: Java heap space, index of 500000 records
Posted: Oct 3, 2008 4:59 PM   in response to: Jirka Hradil in response to: Jirka Hradil
 
  Click to reply to this thread Reply
Its perfectly fine to do this within a transaction. Note, you might not want to iterate over all the hits, but maybe do some pagination to the UI side.
Jirka Hradil

Posts: 17
Registered: 10/2/08
Re: Toplink OutOfMemoryError: Java heap space, index of 500000 records
Posted: Oct 3, 2008 7:19 PM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
Shay, when I set up memory from 64 MB to 256M (JAVA_OPTS='-Xmx256M') problem with OutOfMemoryError disappeared, so it works. Tomcat manager shows me approx. 130M used memory now. When I try to access the application with 2 users, it took approx. 240 MB.
Now imagine that I access the application with 500 users whose will use the fulltext frequently and at once. Do you think that is better to sort the document outside the Compass (in a database for example) or the Compass is able to do that?

Thank you for your support, I appreciate it!

Jirka Hradil
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Toplink OutOfMemoryError: Java heap space, index of 500000 records
Posted: Oct 3, 2008 7:28 PM   in response to: Jirka Hradil in response to: Jirka Hradil
 
  Click to reply to this thread Reply
This might happen, as Lucene caches might get reloaded if the index changes, and Lucene uses WeakHashMap for this (it will get cleared once GC runs). Can you try and access the system with more users and see that you don't get an OOM?
Jirka Hradil

Posts: 17
Registered: 10/2/08
Re: Toplink OutOfMemoryError: Java heap space, index of 500000 records
Posted: Oct 4, 2008 8:38 AM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
Hello, I made some progress while the load test with 500 000 records. I found an performance issue with the CompassHit.getData(), which is extremely slow, see the example:

CompassTransaction tr = session.beginTransaction();
CompassHits hits = query.hits(); //I removed any sorting from query

for (CompassHit hit : hits) { //paging is not required here, only one result is returned for the test
hit.getResource().getId(); //GOOD, fast, about 20 ms
hit.getData(); //BAD, extremely slow, about 20 000 ms!
}

tr.commit();
session.close();

I removed sorting completely. Only 1 user accessed the fulltext while testing so it is useless to try it with more users. As I mentioned, for the index storage I use normal file system index. I raised memory allocated to Tomcat to 768 MB but without any effect. It looks like that main problem which degrades the performance is unmarshalling the resource to the object.
As a fallback, I could use Compass to return hit.getResource().getId() only and get the object via Toplink instead of hit.getData() but it degrades the beauty of OSEM in Compass :).

Thanks for any idea.
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Toplink OutOfMemoryError: Java heap space, index of 500000 records
Posted: Oct 4, 2008 10:30 AM   in response to: Jirka Hradil in response to: Jirka Hradil
 
  Click to reply to this thread Reply
Can you post your mapping definitions? Note that references mapping are eagerly loaded, and you might end up loading a big object graph in this case.
Jirka Hradil

Posts: 17
Registered: 10/2/08
Re: Toplink OutOfMemoryError: Java heap space, index of 500000 records
Posted: Oct 11, 2008 3:48 PM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
Sorry for my late answer. I made additional tests and I solved my problem indirectly by using Resource object returned from a hit instead of getting data object. Example:

for (CompassHit hit : hits) {
hit.getResource().getId();
String name = hit.getResource("name");
//now I create instance of a target object and I fill attributes by hand, because hit.getData() is slow
}

As you noticed, using Compass sorting get a lot of memory (in my case sorting of 500 000 records needs approx. 130MB heap) so I decided to turn off the internal sorting and I use own comparator for every object I index. Comparators are extremely fast and they don't need so much memory as internal sorting, 64MB heap suffices.

Finally I saved additional documents to the index to the size of 2 milions and finding a document takes 10-20 ms now, so it is perfectly usable for my application. In the documentation I read a paragraph which said that unmarshalling needs more memory and it is slower than using Property object only, so I can confirm it.
louisvuittonbox

Posts: 411
Registered: 8/23/11
Re: Toplink OutOfMemoryError: Java heap space, index of 500000 records
Posted: Aug 23, 2011 10:03 PM   in response to: Jirka Hradil in response to: Jirka Hradil
 
  Click to reply to this thread Reply
Sports brand Nike (Nike) has announced the new 2010 summer footwear, including the nike air max, Tennis Classic, and Air Max 90 Current Torch three groups. nike shox which are yellow, orange, and blue color, with some bright colors there is a refreshing feeling. Tennis Classic with a white and green color, low-cut shoes suitable for summer wear. And with the nike air jordan of the latest technology Torch nike shoes uk Current Torch shoes are hit with the classic black and white color with the toe position with the Torch materials, mining forefoot nike air max 2011 Free technology coupled with the classic appearance , reflecting the retro and technology integration.
miyao lin

Posts: 97
Registered: 11/20/11
Re: Toplink OutOfMemoryError: Java heap space, index of 500000 records
Posted: Dec 15, 2011 6:26 PM   in response to: louisvuittonbox in response to: louisvuittonbox
 
  Click to reply to this thread Reply
It really a good article.cheap oakley sunglassesFake Oakley Flak JacketOakley Half X sunglassesoakley eyewearOakley Antix sunglassesoakley sunglassesFake Oakley Fuel CellOakley Holbrook sunglassesoakley sunglasses discountedOakley Batwolf sunglassesOakley Antix SunglassesFake Oakley Fuel Cell Sunglassesoakley batwolf sunglassesOakley Bottlecap sunglassesOakley Bottlecap SunglassesFake Oakley Gascan SunglassesOakley Commit SQ SunglassesOakley Commit SQ sunglassesoakley five squared sunglassesFake Oakley PolarizedOakley Half X SunglassesOakley Dispatch sunglasses outletOakley Jupiter SunglassesOakley Encounter sunglassescheap oakley minute sunglassesCheap oakleys Bottlecap SunglassesOakley Pit Boss SunglassesCheap oakleys Fives Squared Sunglassesreplica Oakley Pit Boss SunglassesOakley Enduring Pace sunglassesOakley Polarized SunglassesOakley Fives Squared sunglassesoakley crosshair sunglassesCheap oakleys Flak Jacket SunglassesOakley Dangerous SunglassesCheap oakleys Frogskins SunglassesOakley Dart SunglassesFake Oakley Flak Jacket sunglassesoakley dispatch sunglassesOakley Forsake sunglassesOakley Encounter SunglassesCheap oakleys Fuel Cell SunglassesOakley Enduring Pace SunglassesCheap oakleys Gascan SunglassesOakley Fast Jacket SunglassesOakley Frogskins sunglassesOakley Fives SunglassesFake Oakley Gascan sunglassesoakley flak jacket sunglassesFake Oakley DispatchOakley Forsake SunglassesFake Oakley Fast JacketOakley Frogskins SunglassesOakley Hijinx sunglassesoakley fuel cell sunglassesOakley M Frame sunglassesoakley gascan sunglassesFake Oakley Five SquaredOakley Half Wire SunglassesFake Oakley FrogskinsFake Oakley Half XFake Oakley Polarized sunglassesFake Oakley HijinxOakley Polarized sunglassesFake Oakley HolbrookFake Oakley Straight Jacket sunglassesFake Oakley JupiterOakley Fuel Cell sunglassesFake Oakley JuryOakley Half Jacket sunglassesFake Oakley Monster DogOakley Half Wire sunglasses
anitas

Posts: 95
Registered: 11/20/11
Re: Toplink OutOfMemoryError: Java heap space, index of 500000 records
Posted: Dec 15, 2011 7:40 PM   in response to: Jirka Hradil in response to: Jirka Hradil
 
  Click to reply to this thread Reply
I love this article, it's very well.oakley sunglasses saleFake Oakley Flak Jacket Sunglassesreplica Oakley Polarized SunglassesOakley Batwolf sunglasses outletoakley sunglasses wholesaleReplica Oakley Flak Jacket SunglassesOakley Restless SunglassesOakley Dispatch sunglassesreplica oakley sunglassesReplica Oakley Fuel Cell Sunglassesoakley scalpel sunglassesOakley Flak Jacket sunglassesreplica Oakley Commit SQ SunglassesFake Oakley GascanOakley Split Jacket SunglassesOakley Frogskins sunglasses outletoakley fives squared sunglassesFake Oakley Polarized Sunglassesoakley straight jacket sunglassesOakley Gascan sunglassesOakley Half Jacket SunglassesReplica Oakley Polarized SunglassesOakley Ten SunglassesFake Oakley Hijinx sunglassesreplica Oakley Half Wire SunglassesFake Oakley RadarOakley Zero SunglassesOakley Jawbone sunglassesreplica Oakley Half X SunglassesReplica Oakley sunglassesOakley Juliet sunglassesOakley Hijinx SunglassesCheap oakleys Commit SQ SunglassesOakley Jupiter sunglassesOakley Holbrook SunglassesReplica Oakley Fives SquaredOakley M Frame sunglasses outletoakley jawbone sunglassesReplica Oakley Flak JacketOakley Minute sunglassesoakley juliet sunglassesReplica Oakley FrogskinsOakley Monster Dog sunglassesreplica Oakley Jupiter SunglassesReplica Oakley Fuel CellOakley Oil Rig sunglassesOakley Jury SunglassesReplica Oakley GascanOakley Pit Boss sunglassesoakley m frame sunglassesCheap oakleys Half Wire SunglassesOakley Polarized sunglasses outletoakley minute sunglassesCheap oakleys Half X SunglassesOakley Radar sunglassesreplica oakley minute sunglassesCheap oakleys Hijinx SunglassesOakley Restless sunglassesoakley monster dog sunglassesCheap oakleys Jawbone SunglassesOakley Scalpel sunglassesOakley Necessity SunglassesCheap oakleys Oil Rig SunglassesOakley Straight Jacket sunglassesoakley oil rig sunglassesCheap oakleys Pit Boss SunglassesOakley Ten sunglassesOakley Pit Bull SunglassesCheap oakleys Radar SunglassesOakley Zero sunglassesOakley Plaintiff SunglassesCheap oakleys Straight Jacket Sunglasses
Legend
Guru: 2001 + pts
Expert: 501 - 2000 pts
Super-star: 101 - 500 pts
Assistant: 51 - 100 pts
Participant: 0 - 50 pts
Helpful Answer (5 pts)
Correct Answer (10 pts)

Point your RSS reader here for a feed of the latest messages in all forums