SourceForge.net Logo
Main Overview Wiki Issues Forum Build Fisheye

Welcome, Guest
Guest Settings
Help

Compass Support Forums » Compass » Compass Users

Thread: Java out of memory after index more than 1000000 items

This question is not answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 29 - Pages: 2 [ 1 2 | Next ] - Last Post: Nov 11, 2011 10:27 PM Last Post By: shoes01
meir kalter

Posts: 1
Registered: 2/14/08
Java out of memory after index more than 1000000 items
Posted: Feb 14, 2008 3:01 AM
 
  Click to reply to this thread Reply
We try index to lucene using Compass 1.2.
And have problems with the memory.While running java with: -Xms2048m -Xmx2048m.
Are there any limitaitions with the size of the index?
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Java out of memory after index more than 1000000 items
Posted: Feb 14, 2008 5:09 AM   in response to: meir kalter in response to: meir kalter
 
  Click to reply to this thread Reply
I need a bit more information on what exactly you do. Do you start a transaction and try to index that amount of objects within the same transaction? Are you using Compass Gps (for example, the Hibernate integration)?
meir kalter

Posts: 3
Registered: 3/16/06
Re: Java out of memory after index more than 1000000 items
Posted: Feb 14, 2008 10:42 AM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
I create transaction and commit it for every about 4000 rows of data.
But - i did not close the session, just open it for every group of 4000 rows.
I do not use any type of GPS.

I will try close the session.

The close of session every time- did not solve the problem.

Edited by: meir kalter on Feb 14, 2008 10:47 AM

Edited by: meir kalter on Feb 15, 2008 12:32 AM
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Java out of memory after index more than 1000000 items
Posted: Feb 15, 2008 1:37 PM   in response to: meir kalter in response to: meir kalter
 
  Click to reply to this thread Reply
It might be that 4000 rows are too much, since Compass stores the transactional index in memory. Try to lower it.
meir kalter

Posts: 3
Registered: 3/16/06
Re: Java out of memory after index more than 1000000
Posted: Feb 22, 2008 12:48 AM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
It's does not change the big problem - out of memory.

I will give more details about the problem:

We try to do full index for about 2G rows, while every data - every document contains about 300 Bytes.

JBosss show that every starting of transaction get more memory.

The size of the index of about 900000 rows, when it's complete the work - is more than 0.5 Giga of memory.

Therefore, the real question is:
The memory usage gradually increases, and after some hours of working - we run out of memory.

Edited by: meir kalter on Feb 22, 2008 12:48 AM

Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Java out of memory after index more than 1000000
Posted: Feb 22, 2008 5:40 AM   in response to: meir kalter in response to: meir kalter
 
  Click to reply to this thread Reply
Try and close the session each round as well.
Yuval Sapir

Posts: 8
Registered: 2/27/08
Re: Java out of memory after index more than 1000000
Posted: Mar 5, 2008 2:25 AM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
We tried closing the session as well, and it did not affect the memory usage.
We enabled logging to make sure we actually open a new session and a new transaction each time and indeed we do.
Any other idea would be appreciated.
vam

Posts: 259
Registered: 1/2/07
Re: Java out of memory after index more than 1000000
Posted: Mar 5, 2008 8:21 AM   in response to: Yuval Sapir in response to: Yuval Sapir
 
  Click to reply to this thread Reply
Hey, the problem is that you are using the Jboss transaction system. This happens specially if you are using the Transactional Cache. I had the same problem and changing to Hibernate Local Transactions instead of the JTA or XA fixed the issue.
Yuval Sapir

Posts: 8
Registered: 2/27/08
Re: Java out of memory after index more than 1000000
Posted: Mar 5, 2008 8:42 AM   in response to: vam in response to: vam
 
  Click to reply to this thread Reply
Actually we're using local transactions. among other things we don't use jboss transactions because we're running in weblogic :-)
vam

Posts: 259
Registered: 1/2/07
Re: Java out of memory after index more than 1000000
Posted: Mar 5, 2008 11:07 AM   in response to: Yuval Sapir in response to: Yuval Sapir
 
  Click to reply to this thread Reply
I think this is hibernate issue. Hibernate has two types of caches: A factory cache and a session cache. Compass is using a SessionFactory to create the entity manager used by the CompassSession. Maybe it is a good idea to empty the SessionFactory cache.

You can inject the HibernateFactory used by compass
@PersistenceUnit(unitName="index")
private EntityManagerFactory emf;

create an entity manager, cast to a HibernateSession, get the Factory and remove everything from the cache.
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Java out of memory after index more than 1000000
Posted: Mar 5, 2008 3:09 PM   in response to: Yuval Sapir in response to: Yuval Sapir
 
  Click to reply to this thread Reply
Can you verify in Compass logging that it is actually committing the transaction, and not joining one? For example, you should see, if you use Compass local transaction this:

Committing local transaction on thread ...

And not this:

Not committing the transaction since within a local transaction on thread ....
Yuval Sapir

Posts: 8
Registered: 2/27/08
Re: Java out of memory after index more than 1000000
Posted: Mar 6, 2008 1:43 AM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
I did and it is. it's also creating a new transaction and joining an existing one.
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Java out of memory after index more than 1000000
Posted: Mar 9, 2008 10:01 AM   in response to: Yuval Sapir in response to: Yuval Sapir
 
  Click to reply to this thread Reply
The question is if it starts a transaction, and most of the time joins one. Note, that only when a local transaction is committed, then the memory is flushed and discarded.

Can you post a pseudo code of how you do indexing?
Yuval Sapir

Posts: 8
Registered: 2/27/08
Re: Java out of memory after index more than 1000000
Posted: Mar 16, 2008 4:04 AM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
OK.
Turns out the problem was not with the indexing code at all, but with the code that read the data from the database.
Thanks for all the help.

Yuval

Edited by: Yuval Sapir on Mar 16, 2008 4:04 AM
Konstantyn Smir...

Posts: 9
Registered: 3/17/08
Re: Java out of memory after index more than 1000000
Posted: May 8, 2008 7:19 AM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
How to force compass to start a brand new transaction on beginTransaction() rather then join the existing one?

If that happens only with local transactions, can I use BATCH_INSERT isolation with them?

My code with some 'smart' flushing and optimization looks like:

def cTemplate = new CompassTemplate( compass )
def optimizer = compass.searchEngineOptimizer

//iterating through docs
docs.each{ doc ->

cTemplate.execute( CTI.BATCH_INSERT, [ 'doInCompass':{ cs ->
cs.save doc

if( 0 == index % 10000 ){
cs.flush()
}
}
cs.flush()
} ] as CompassCallback )

if( 0 == index % 250000 ) optimizer.optimize( alias )
}

When I run the code, the used memory grows (ignoring the flushes and commits) and after ~300000 documents hits the Xmx, but for some reason the app doesn't crash with OutOfMemory error and survives processing of ~900000 docs

I also tried to do it outside the CompassTemplate's execute() method by managing the session and transaction manually, but the result remains the same.

Edited by: Konstantyn Smirnov on May 8, 2008 2:22 PM

Edited by: Konstantyn Smirnov on May 8, 2008 9:41 PM
Legend
Guru: 2001 + pts
Expert: 501 - 2000 pts
Super-star: 101 - 500 pts
Assistant: 51 - 100 pts
Participant: 0 - 50 pts
Helpful Answer (5 pts)
Correct Answer (10 pts)

Point your RSS reader here for a feed of the latest messages in all forums