SourceForge.net Logo
Main Overview Wiki Issues Forum Build Fisheye

Welcome, Guest
Guest Settings
Help

Compass Support Forums » Compass » Compass Developers

Thread: Memory problems

This question is not answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 27 - Pages: 2 [ 1 2 | Next ] - Last Post: Nov 26, 2011 12:40 AM Last Post By: ying
Rui Pereira

Posts: 1
Registered: 1/30/09
Memory problems
Posted: Jan 30, 2009 10:38 AM
 
  Click to reply to this thread Reply
I'm trying to index several tables (20+) with some of them having 500000+ rows.
The problem is that when indexing the heap memory goes up and the GarbageCollector can't free memory. After a few tables an OutOfMemoryException is thrown.
I'm using JBoss 4.2.3 and spring. The transaction processor used is lucene and the snapshot persister is FSSnapshotPersister.

Here some compass configurations:

<prop key="compass.transaction.factory">org.compass.spring.transaction.SpringSyncTransactionFactory</prop>
<prop key="compass.engine.useCompoundFile">false</prop>
<prop key="compass.engine.store.jdbc.managed">true</prop>
<prop key="compass.transaction.commitBeforeCompletion">true</prop>
<prop key="compass.transaction.processor">lucene</prop>

<bean id="jdbcGpsDevice" class="com.indexer.util.compass.IndexerResultSetJdbcGpsDevice">
<property name="name" value="jdbcGpsDevice"/>
<property name="dataSource" ref="inputDataSource"/>
<property name="fetchSize" value="${performance.compass.fetchSize}"/>
<property name="mirrorDataChanges" value="true"/>
<property name="saveSnapshotAfterMirror" value="true"/>
<property name="snapshotPersister">
<bean class="org.compass.gps.device.jdbc.snapshot.FSJdbcSnapshotPersister">
<property name="path" value="${compass.snapshot.filepath}"/>
</bean>
</property>
</bean>
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Memory problems
Posted: Jan 30, 2009 3:23 PM   in response to: Rui Pereira in response to: Rui Pereira
 
  Click to reply to this thread Reply
Can you tell what takes most of the memory? One of the problems of the jdbc device is the fact that, event with the snapshot persister, it still stores the all the rows primary keys and versions in memory, which might explain why it takes so much memory.
Fábio Matos

Posts: 16
Registered: 11/17/08
Re: Memory problems
Posted: Jan 31, 2009 2:56 PM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
That cannot be the problem, because the snapshot file saved in the disk is too small compared to the occupied RAM.

Example:
For a 20.000 line table, the snapshot file after indexing is about 1.5MB and the RAM size is 100MB.
This size discrepancy shows that there is some real problem in the release of objects going on.
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Memory problems
Posted: Jan 31, 2009 4:02 PM   in response to: Fábio Matos in response to: Fábio Matos
 
  Click to reply to this thread Reply
ok, if that is not the case, then I need more information as to what takes most of the memory so I can help. Can you profile the app and check?
Fábio Matos

Posts: 16
Registered: 11/17/08
Re: Memory problems
Posted: Feb 1, 2009 4:31 AM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
I was wrong. After a close analyses of the Heap with jvisualvm, the RAM increases with the indexing but is reclaimed successfully by the GC after that.

One thing I noticed is that the RAM is always increasing while indexing a alias (i.e. table), and the GC only kicks in after the indexing is over (and the mirroring starts), until then the RAM just keeps growing until the JVM -Xmx setting. I think this is because the transaction is only closed at the end.

Do you think it would make sense to make available a configuration that at the ResultSetSnapshotEventListener.doCreateAndUpdateFor method called the session.flush() after processing X JdbcAliasRowSnapshots, to avoid this unstoppable increase of RAM while indexing a single alias/table?

Thanks.
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Memory problems
Posted: Feb 1, 2009 12:37 PM   in response to: Fábio Matos in response to: Fábio Matos
 
  Click to reply to this thread Reply
I am not sure that this is the reason why the memory is not reclaimed. With the lucene transaction isolation, flush actually does not do much. Also, I thought that the problem is with indexing, and ResultSetSnapshotEventListener is used when mirroring.
Fábio Matos

Posts: 16
Registered: 11/17/08
Re: Memory problems
Posted: Feb 1, 2009 1:23 PM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
We are just using mirroring, so the "indexing" is made the first time it runs the version query and compares what is in the DB to what is on the index (nothing at the first), what causes the mirroring to index all the content.

I think I solved the problem, at least that what it seams in the tests.
The problem was because in this first "indexation" the snapshot list is huge, and it was only available to GC at the very end of the method.

We changed the method to that at a defined flushSize the list already processed items where cleared from the list and System.gc() is called.
This way, we don't need to wait for the end of the method for all the JdbcAliasRowSnapshots of the lists be available for GC, because clearing them form the list makes them available for GC. We also called flush before calling System.gc().
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Memory problems
Posted: Feb 2, 2009 1:07 PM   in response to: Fábio Matos in response to: Fábio Matos
 
  Click to reply to this thread Reply
Can you you post your changes? If they work, I would love to get them into Compass.
Fábio Matos

Posts: 16
Registered: 11/17/08
Re: Memory problems
Posted: Feb 2, 2009 6:15 PM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
See the attached file.
The only change is the adding of the flushSize property and in the doCreateAndUpdateFor method.

Hope it helps.
louisvuittonbox

Posts: 411
Registered: 8/23/11
Re: Memory problems
Posted: Aug 23, 2011 9:59 PM   in response to: Rui Pereira in response to: Rui Pereira
 
  Click to reply to this thread Reply
Sports brand Nike (Nike) has announced the new 2010 summer footwear, including the nike air max, Tennis Classic, and Air Max 90 Current Torch three groups. nike shox which are yellow, orange, and blue color, with some bright colors there is a refreshing feeling. Tennis Classic with a white and green color, low-cut shoes suitable for summer wear. And with the nike air jordan of the latest technology Torch nike shoes uk Current Torch shoes are hit with the classic black and white color with the toe position with the Torch materials, mining forefoot nike air max 2011 Free technology coupled with the classic appearance , reflecting the retro and technology integration.
hmq123

Posts: 449
Registered: 8/22/11
christian louboutin
Posted: Aug 24, 2011 10:38 PM   in response to: Rui Pereira in response to: Rui Pereira
 
  Click to reply to this thread Reply
How christian louboutin should I heard cheap louboutin shoes that, about your right discount christian louboutin shoes and wrong? How Christian Louboutin Pumps should I heard, was able to Christian Louboutin Ankle Boots smile? How should I heard, you good, for her not to let the http://www.louboutinsshoes2u.com/ tears?
hmq123

Posts: 449
Registered: 8/22/11
Louboutins Shoes Sale
Posted: Aug 24, 2011 10:41 PM   in response to: Rui Pereira in response to: Rui Pereira
 
  Click to reply to this thread Reply
Fresh Louboutins Shoes Sale sweet smile like the Christian Louboutin Shoes wine you drink, the more Cheap Christian Louboutin mellow Christian Louboutin Glitter Pumps life, elegant lingering smile Cheap Louboutin Booties like read the poem, the more http://www.louboutinsshoes4sale.com/ romantic life.
hmq123

Posts: 449
Registered: 8/22/11
Discount Christian Louboutin Shoes
Posted: Aug 24, 2011 10:43 PM   in response to: Rui Pereira in response to: Rui Pereira
 
  Click to reply to this thread Reply
A smile Discount Christian Louboutin Shoes is healthy, the smile Christian Louboutin Shoes is success, but only Christian Louboutin sale smile and life, and life Cheap Wedding Shoes road to the colorful, just Christian Louboutin Ankle Boots won't give his life with http://www.cheap-christianlouboutinpumps.com/ regrets.
hmq123

Posts: 449
Registered: 8/22/11
Cheap Louboutins
Posted: Aug 24, 2011 10:45 PM   in response to: Rui Pereira in response to: Rui Pereira
 
  Click to reply to this thread Reply
A smile is the most Cheap Louboutins moving people expression, Christian Louboutin Shoes life is good and silent Christian Louboutin Discount language, only smile Christian Louboutin Wedding life, the Louboutin Open-toe Booties mood will be relaxed and http://www.cheaplouboutins-shop.com/ cheerful.
hmq123

Posts: 449
Registered: 8/22/11
christian louboutin sale
Posted: Aug 24, 2011 10:47 PM   in response to: Rui Pereira in response to: Rui Pereira
 
  Click to reply to this thread Reply
In middle christian louboutin sale age, experience christian louboutin shoes rains and winds, have discount louboutin sale seen the rainbow. Through Christian Louboutin Peep Toe Pumps pain, also get the Christian Louboutin Sandal Booties joy. Experienced setbacks, and bear witness to the http://www.discountlouboutins-sale.com/ success.
Legend
Guru: 2001 + pts
Expert: 501 - 2000 pts
Super-star: 101 - 500 pts
Assistant: 51 - 100 pts
Participant: 0 - 50 pts
Helpful Answer (5 pts)
Correct Answer (10 pts)

Point your RSS reader here for a feed of the latest messages in all forums