SourceForge.net Logo
Main Overview Wiki Issues Forum Build Fisheye

Welcome, Guest
Guest Settings
Help

Compass Support Forums » Compass » Compass Users

Thread: OutOfMemoryError: Java heap space while indexing 50000 records

This question is answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 31 - Pages: 3 [ 1 2 3 | Next ] - Last Post: Dec 3, 2011 4:47 AM Last Post By: longchamp
nch

Posts: 43
Registered: 3/3/08
OutOfMemoryError: Java heap space while indexing 50000 records
Posted: Mar 6, 2008 9:07 AM
 
  Click to reply to this thread Reply
Hi, there.

I am testing Compass ability to index contents (amount, speed, ...). I downloaded a Wikipedia dump and imported it into a MySQL db.

I configured Compass and a GPS device using Spring, iBatis and Apache Commons DBCP (Compass 2.0.0M2, Lucene 2.3.1, iBatis 2.3.1 and DBCP 1.2.2) and using the Compass Spring controller.

First I tried indexing 10000 records and it worked (45 min). Then I tried 50000, but it throw OutOfMemoryError right at start up, right when iBatis executes the select *.

Many thanks

This is how ApplicationContext-service.xml looks like:
<bean id="compass" class="org.compass.spring.LocalCompassBean">
<property name="classMappings">
<list>
<value>org.myorg.TestingPojo</value>
</list>
</property>
<property name="compassSettings">
<props>
<prop key="compass.engine.connection">file://compass</prop>
<prop key="compass.transaction.factory">org.compass.spring.transaction.SpringSyncTransactionFactory</prop>
<prop key="compass.engine.analyzer.default.type">snowball</prop>
<prop key="compass.engine.analyzer.default.name">English</prop>
<!-- For testing why are we getting OOME -->
<prop key="compass.engine.ramBufferSize">16</prop>
<prop key="compass.engine.maxBufferedDocs">10</prop>
<prop key="compass.engine.maxBufferedDeletedTerms">-1</prop>
<prop key="compass.engine.mergeFactor">10</prop>
<prop key="compass.engine.maxFieldLength">10000</prop>
<prop key="compass.engine.cacheIntervalInvalidation">5000</prop>
<prop key="compass.engine.indexManagerScheduleInterval">60</prop>
<!--prop key="compass.transaction.isolation">batch_insert</prop-->
</props>
</property>
<property name="transactionManager" ref="transactionManager" />
</bean>

<bean id="iBatisGpsDevice" class="org.compass.gps.device.ibatis.SqlMapClientGpsDevice">
<property name="name" value="iBatisDevice" />
<property name="sqlMapClient" ref="sqlMapClient" />
<property name="selectStatementsIds">
<list>
<value>getAllTestingPojos</value>
</list>
</property>
<property name="pageSize" value="50" />
</bean>

<bean id="compassGps" class="org.compass.gps.impl.SingleCompassGps" init-method="start" destroy-method="stop">
<property name="compass">
<ref bean="compass" />
</property>
<property name="gpsDevices">
<list>
<bean class="org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper">
<property name="gpsDevice" ref="iBatisGpsDevice" />
</bean>
</list>
</property>
</bean>

The stack trace:
Caused by: java.lang.OutOfMemoryError: Java heap space
at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198)
at com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java:318)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1366)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2333)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:435)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2040)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1443)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1777)
at com.mysql.jdbc.Connection.execSQL(Connection.java:3249)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1268)
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843)
at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:169)
at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:169)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.ibatis.common.jdbc.logging.PreparedStatementLogProxy.invoke(PreparedStatementLogProxy.java:62)
at $Proxy44.execute(Unknown Source)
at com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:186)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery(GeneralStatement.java:205)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback(GeneralStatement.java:173)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryForList(GeneralStatement.java:123)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForList(SqlMapExecutorDelegate.java:615)
at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForList(SqlMapSessionImpl.java:126)
at com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.queryForList(SqlMapClientImpl.java:103)
at com.ibatis.sqlmap.engine.mapping.statement.PaginatedDataList.getList(PaginatedDataList.java:129)
at com.ibatis.sqlmap.engine.mapping.statement.PaginatedDataList.pageTo(PaginatedDataList.java:89)
at com.ibatis.sqlmap.engine.mapping.statement.PaginatedDataList.(PaginatedDataList.java:48)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForPaginatedList(SqlMapExecutorDelegate.java:673)
at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForPaginatedList(SqlMapSessionImpl.java:137)
at org.compass.gps.device.ibatis.SqlMapClientGpsDevice.doIndex(SqlMapClientGpsDevice.java:117)

nch

Posts: 43
Registered: 3/3/08
Re: OutOfMemoryError: Java heap space while indexing 50000 records
Posted: Mar 7, 2008 2:11 AM   in response to: nch in response to: nch
 
  Click to reply to this thread Reply
I increased the amount of available mem for the VM up to -Xmx1024m and now I can index 100000 records but, definitely not all of the 650000 records the of the Wikipedia dump.
Any ideas on how to ovecome this issue?

Thank you.
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: OutOfMemoryError: Java heap space while indexing 50000 records
Posted: Mar 9, 2008 10:14 AM   in response to: nch in response to: nch
 
  Click to reply to this thread Reply
Strange, Compass does should not hold references during indexing, and it clears them. Can you maybe try and find out what is being kept in the VM?
nch

Posts: 43
Registered: 3/3/08
Re: OutOfMemoryError: Java heap space while indexing 50000 records
Posted: Mar 12, 2008 10:37 AM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
I've been testing a few profiling tools. This is what I observed:

  • Heap memory is the only type of memory space having a relevant impact during indexing.
  • The more records to be indexed from the database, the more heap is being used.
  • Having 64MB of heap space, I can index 10000 records.
  • Having 1024MB of heap space, when I try to index 100000 records the usage of heap space is between 230MB and 400MB.
  • Although I select everything from the contents table, it looks like iBatis divides it into many select statements, so when iBatis performs each of these selects is when it reaches the 400MB pick.
  • Right after a select is performed, the heap consumption decreases to 230MB.
  • Selected the profiler option to display the relation method-object-size and got that
com.mysql.jdbc.Buffer.getBytes(int) was the one with a higher rate of owned objects and size
  • Selected the profiler option for CPU usage and got that the following where the ones running for longer (especially the second one):
com.ibatis.sqlmap.engine.mapping.statement.PaginatedDataList.pageForward()
$Proxy44.execute()

Please, see attached AllocationCallTree.html and BackTrace.html.
So far, I could not figure out what is causing this. I thought setting the page size should do. Any ideas?

Regards
nch

Posts: 43
Registered: 3/3/08
Re: OutOfMemoryError: Java heap space while indexing 50000 records
Posted: Mar 13, 2008 8:33 AM   in response to: nch in response to: nch
 
  Click to reply to this thread Reply
I replaced the iBatis GPS device for a Hibernate one.
Now I can index the 650000 records of the Wikipedia dump.
Unfortunately, I have to stick to iBatis. Is there a way to make an iBatis and a Hibernate compass instances coexist?

Edited by: nch on Mar 13, 2008 8:37 AM
vam

Posts: 259
Registered: 1/2/07
Re: OutOfMemoryError: Java heap space while indexing 50000 records
Posted: Mar 13, 2008 9:19 AM   in response to: nch in response to: nch
 
  Click to reply to this thread Reply
Have you tried to implement your own indexer. It is faily easy to do and you can even go with native SQL if you wanted to. I implemented my own because the one compass provides was not meeting my reqyuirements:
 JpaGpsDevice jpaDevice = new JpaGpsDevice("jpa", emf);
        jpaDevice.setFetchCount(5000);
        jpaDevice.setMirrorDataChanges(false);
        jpaDevice.setInjectEntityLifecycleListener(false);
        gps.addGpsDevice(jpaDevice);
        //jpaDevice.setEntitiesIndexer(new GPSDeviceIndexer());
        jpaDevice.setEntitiesIndexer(new NewDeviceIndexer());
nch

Posts: 43
Registered: 3/3/08
Re: OutOfMemoryError: Java heap space while indexing 50000 records
Posted: Mar 13, 2008 10:34 AM   in response to: vam in response to: vam
 
  Click to reply to this thread Reply
What I'm currently doing is, through Spring, injecting a
org.springframework.orm.ibatis.SqlMapClientFactoryBean
into a
org.compass.gps.device.ibatis.SqlMapClientGpsDevice
The second one has a property called pageSize which I assumed was meant to control the amount of data in memory. I was playing with that parameter but I could see no difference.

I haven't tried to write my own indexer/pagination mechanism, but I'll have to.

Thank you, vam.
nch

Posts: 43
Registered: 3/3/08
Re: OutOfMemoryError: Java heap space while indexing 50000 records
Posted: Mar 13, 2008 11:11 AM   in response to: vam in response to: vam
 
  Click to reply to this thread Reply
The JpaGpsDevice has an entityIndexer property, but the SqlMapClientGpsDevice doesn't so, at first glance, I can not implement my own indexer.
nch

Posts: 43
Registered: 3/3/08
Re: OutOfMemoryError: Java heap space while indexing 50000 records
Posted: Mar 14, 2008 5:42 AM   in response to: nch in response to: nch
 
  Click to reply to this thread Reply
I can see this is going to be fixed by using queryWithRowHandler instead of queryForPaginatedList in version 2.0.0 M3

http://issues.compass-project.org/browse/ CMP-458
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: OutOfMemoryError: Java heap space while indexing 50000 records
Posted: Mar 14, 2008 8:18 AM   in response to: nch in response to: nch
 
  Click to reply to this thread Reply
Yea, I need to apply this patch for a long time. Can you give it a go yourself and see if it help? If you need help, give me a shout.
nch

Posts: 43
Registered: 3/3/08
Re: OutOfMemoryError: Java heap space while indexing 50000 records
Posted: Mar 15, 2008 10:23 AM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
I tried to download the patch, but I got a "404 Could not find specified attachment".

Tried myself, also, a first approach by overriding SqlMapClientGpsDevice#doIndex. Basically I replaced the queryForList call by:

sqlMapSession.queryWithRowHandler(super.getSelectStatementsIds()[i], parameterObject, new SqlMapClientGpsDeviceRowHandler(session, super.getPageSize()));

The following is the RowHandler:

package org.myorg.compass;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.compass.core.CompassSession;
import com.ibatis.sqlmap.client.event.RowHandler;

public class SqlMapClientGpsDeviceRowHandler implements RowHandler {
private static final Log log = LogFactory.getLog(SqlMapClientGpsDeviceRowHandler.class);
private CompassSession session;
private int pageSize;
private int pageCount= 0;
private int currentItem= 1;

public SqlMapClientGpsDeviceRowHandler(CompassSession session, int pageSize){
this.session= session;
this.pageSize= pageSize;
}

public void handleRow(Object o) {
session.create(o);

if(currentItem== pageSize){
if (log.isDebugEnabled()) {
log.debug("Indexing page number ");
}

session.evictAll();
}

currentItem= (currentItem% pageSize)+ 1;
}
}

This is not working, still running out of heap space.

nch

Posts: 43
Registered: 3/3/08
Re: OutOfMemoryError: Java heap space while indexing 50000 records
Posted: Mar 18, 2008 1:56 PM   in response to: nch in response to: nch
 
  Click to reply to this thread Reply
I subscribed to iBatis user mailing list. One of its members found out this is a MySQL driver issue described at http://lists.mysql.com/java/9137

The MySQL driver will load all the results of the SELECT statement into memory unless fetchSize is set to Integer.MIN_VALUE.

So the solution was, in the corresponding iBatis select statement, the following:
<select id="getAllTestingPojos" resultMap="articleResult" fetchSize="-2147483648">
<![CDATA[
select * from articles
]]>
</select>

I am now indexing the whole 650000 records using a queryWithRowHandler based SqlMapClientGpsDevice.
nch

Posts: 43
Registered: 3/3/08
Re: OutOfMemoryError: Java heap space while indexing 50000 records
Posted: Mar 18, 2008 4:17 PM   in response to: nch in response to: nch
 
  Click to reply to this thread Reply
You can use the "userCursorFetch=true" param on the JDBC URL to get it to adhere to the fetch size.
See http://bugs.mysql.com/bug.php?id=18148
nch

Posts: 43
Registered: 3/3/08
Re: OutOfMemoryError: Java heap space while indexing 50000 records
Posted: Mar 25, 2008 6:12 AM   in response to: nch in response to: nch
 
  Click to reply to this thread Reply
Shay, I just wanted to let you know that indexing is way much faster for me if I use queryWithRowHandler than queryForPaginatedList for the iBatis GPS device.

Cheers.
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: OutOfMemoryError: Java heap space while indexing 50000 records
Posted: Mar 25, 2008 6:09 PM   in response to: nch in response to: nch
 
  Click to reply to this thread Reply
Great!. Would you like to submit a patch for it? You can add it to the same iBatis issue that we already have (which, for some reason) seem to have lost its original patch... .
Legend
Guru: 2001 + pts
Expert: 501 - 2000 pts
Super-star: 101 - 500 pts
Assistant: 51 - 100 pts
Participant: 0 - 50 pts
Helpful Answer (5 pts)
Correct Answer (10 pts)

Point your RSS reader here for a feed of the latest messages in all forums