SourceForge.net Logo
Main Overview Wiki Issues Forum Build Fisheye

Welcome, Guest
Guest Settings
Help

Compass Support Forums » Compass » Compass Users

Thread: GPS, Transactions, and Out of Memory issues

This question is not answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 26 - Pages: 2 [ 1 2 | Next ] - Last Post: Dec 14, 2011 7:28 PM Last Post By: anitas
Ivan

Posts: 39
Registered: 3/10/08
GPS, Transactions, and Out of Memory issues
Posted: Jun 23, 2008 2:26 PM
 
  Click to reply to this thread Reply
Since switching over from pure Lucene to Compass for our search engine, our memory requirements have doubled.

We reindex our database once per day using a custom GPS device. We loop through different content providers in the doIndex() method, restarting a new transaction for each, but I noticed that beginTransaction() is actually joining a current transaction. When doIndex() is called, is there a transaction already started?

Condensed code
public class ContentGpsDeviceLocal extends AbstractGpsDevice implements ActiveMirrorGpsDevice  {
 
	public synchronized void performMirroring() {
		...
	}
 
	@Override
	protected synchronized void doIndex(CompassSession session, IndexPlan indexPlan) {
 
		for (IndexContentProvider contentProvider : contentProviders) {
			CompassTransaction tx = session.beginTransaction();
			
			List<Indexable> indexables = contentProvider.findAll();
			batchUpdate(indexables, session);
			
			tx.commit();
		}
	}
	
	private void batchUpdate(List<Indexable> indexables, CompassSession session) {
        for (Indexable indexable : indexables) {
            updateIndexable(session, indexable);
        }
    }
	
    private void updateIndexable(CompassSession session, Indexable indexable) {        
        if (indexable.isActive()) {
            List<Document> docs = indexable.getSearchDocuments();
            for (Document doc : docs) {
                String resourceType = doc.getField(TYPE_FIELD).stringValue();
                Resource lc = createResource(session, resourceType, doc);
                session.save(lc);
            }
        }
    }	
}


Log
2008-06-23 14:54:55,816 DEBUG LocalTransaction - Starting a new local transcation on thread [pool-1-thread-1] Compass [24889435] Session [27198320]  with isolation [LUCENE]
 
2008-06-23 14:54:55,847  INFO IndexManagerCompassImpl$ContentGpsDeviceLocal - rebuilding: FooIndexProvider
2008-06-23 14:54:55,847 DEBUG LocalTransaction - Joining an existing local transcation on thread [pool-1-thread-1] Compass [24889435] Session [27198320]
2008-06-23 15:00:58,318 DEBUG LocalTransaction - Not committing the transaction since within a local transaction on thread [pool-1-thread-1] Compass [24889435] Session [27198320]
 
2008-06-23 15:00:58,318  INFO IndexManagerCompassImpl$ContentGpsDeviceLocal - rebuilding: BarIndexProvider
2008-06-23 15:00:58,318 DEBUG LocalTransaction - Joining an existing local transcation on thread [pool-1-thread-1] Compass [24889435] Session [27198320]
 
2008-06-23 15:03:02,657 DEBUG LocalTransaction - Not committing the transaction since within a local transaction on thread [pool-1-thread-1] Compass [24889435] Session [27198320]
2008-06-23 15:03:02,657  INFO IndexManagerCompassImpl$ContentGpsDeviceLocal - rebuilding: BarEmailIndexProvider
2008-06-23 15:03:02,657 DEBUG LocalTransaction - Joining an existing local transcation on thread [pool-1-thread-1] Compass [24889435] Session [27198320]
2008-06-23 15:03:09,204 DEBUG LocalTransaction - Not committing the transaction since within a local transaction on thread [pool-1-thread-1] Compass [24889435] Session [27198320]
 
2008-06-23 15:03:09,719 DEBUG LocalTransaction - Committing local transaction on thread [pool-1-thread-1] Compass [24889435] Session [27198320]
 


Are our memory problems transaction related? I have tried to set the transaction logs to be file-based, but the settings have no effect

<prop key="compass.transaction.isolation">read_committed</prop>
<prop key="compass.transaction.readcommitted.translog.connection">file:///C:/Temp/compass</prop>
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: GPS, Transactions, and Out of Memory issues
Posted: Jun 23, 2008 3:14 PM   in response to: Ivan in response to: Ivan
 
  Click to reply to this thread Reply
Can you post how you configure Compass and the Gps devices?
Ivan

Posts: 39
Registered: 3/10/08
Re: GPS, Transactions, and Out of Memory issues
Posted: Jun 23, 2008 3:18 PM   in response to: Ivan in response to: Ivan
 
  Click to reply to this thread Reply
Compass configured in Spring

<bean id="compass" class="org.compass.spring.LocalCompassBean">
	<property name="resourceLocations">
		<list>
			<value>classpath:compass-resource.cpm.xml</value>
		</list>
	</property>
	<property name="postProcessor" ref="compassPostProcessor" />
	<property name="compassSettings">
		<props>
			<prop key="compass.transaction.factory">org.compass.core.transaction.LocalTransactionFactory</prop>
			<prop key="compass.transaction.isolation">read_committed</prop>
			<prop key="compass.transaction.readcommitted.translog.connection">file:///C:/Temp/compass</prop>				
			
			<!-- spell checking properties -->
			<prop key="compass.engine.spellcheck.enable">true</prop>
			<prop key="compass.engine.spellcheck.globablIncludeProperties">content-spellcheck</prop>
			<prop key="compass.engine.analyzer.default.factory">
				com.example.search.SnowballAnalyzerFactory
			</prop>
			<prop key="compass.engine.analyzer.search.factory">
				com.example.search.SnowballAnalyzerFactory
			</prop>
			<prop key="compass.engine.analyzer.spellcheck.type">
				org.apache.lucene.analysis.standard.StandardAnalyzer
			</prop>
			<prop key="compass.engine.spellcheck.schedule">false</prop>
		</props>
	</property>
</bean>
 
<!-- use either a filesystem or ram implementation -->
<bean id="compassPostProcessor" class="com.example.search.DirectoryLocalCompassBeanPostProcessor">
	<property name="directory" ref="fsDirectory" />
</bean>


The gps device in not configured in Spring
private ActiveMirrorGpsDevice gpsDevice;
public void afterPropertiesSet() throws Exception {
	gpsDevice = new ContentGpsDeviceLocal();
}


Forgot to add how the GPS device is used:
CompassGps gps = new SingleCompassGps(compass);
gpsDevice.setName(gpsDevice.getClass().getSimpleName());
gps.addGpsDevice(gpsDevice);
 
gps.start();
gps.index();
gps.stop();


Edited by: Ivan on Jun 23, 2008 4:18 PM
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: GPS, Transactions, and Out of Memory issues
Posted: Jun 23, 2008 3:28 PM   in response to: Ivan in response to: Ivan
 
  Click to reply to this thread Reply
First, another question, which version of Compass are you using? Were you using a different version of Lucene before moving to Compass?

Second, I see you use the SingleCompassGps device. By default, when executing the index operation, it is not executed with read_committed isolation, but with lucene isolation (formally batch_insert). So, the whole transaction log should not be kept in memory, and it should be ok.
Ivan

Posts: 39
Registered: 3/10/08
Re: GPS, Transactions, and Out of Memory issues
Posted: Jun 23, 2008 3:38 PM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
Thanks for the lightning fast replies!

We are using Compass 2.0.0. We used to build from source, but switched over to Maven/Ivy when you finally migrated over. Initially we used Lucene 2.3.0, but have since switched over to 2.3.2

"By default, when executing the index operation, it is not executed with read_committed isolation, but with lucene isolation (formally batch_insert)."

I noticed that from reading the SingleCompassGps source. Are the transaction messages correct in our instance? We wanted commit after every alias type to reduce memory load.

"So, the whole transaction log should not be kept in memory, and it should be ok."

Where are the transaction logs stored?

Thanks.
Ivan

Posts: 39
Registered: 3/10/08
Re: GPS, Transactions, and Out of Memory issues
Posted: Jun 23, 2008 4:01 PM   in response to: Ivan in response to: Ivan
 
  Click to reply to this thread Reply
ivan wrote:
When doIndex() is called, is there a transaction already started?

To answer my own question: yes, the doIndex() method starts a transaction.
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: GPS, Transactions, and Out of Memory issues
Posted: Jun 23, 2008 4:05 PM   in response to: Ivan in response to: Ivan
 
  Click to reply to this thread Reply
If you worked with pure Lucene, then I dive into the details. Basically, the Lucene transaction isolation level uses an IndexWriter and simply adds documents to it. Lucene IndexWriter stores an internal buffer before it flushes information back to the disk. You can control the size of the ram buffer using Compass. (the read committed isolation works a bit differently to ensure scoping).
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: GPS, Transactions, and Out of Memory issues
Posted: Jun 23, 2008 4:07 PM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
Also note that this is preferable to start and close a transaction for each alias, as you will get better indexing performance in this case.
Ivan

Posts: 39
Registered: 3/10/08
Re: GPS, Transactions, and Out of Memory issues
Posted: Jun 24, 2008 9:40 AM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
kimchy wrote:
Also note that this is preferable to start and close a transaction for each alias, as you will get better indexing performance in this case.

I just refactored my code so that only one alias is indexed at a time by the GPS device. However, the memory problems still remain. I switched to read_committed isolation only to solve the memory problems. I doubt controlling the buffer size will have an effect since the default of 16MBs should not cause issues.
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: GPS, Transactions, and Out of Memory issues
Posted: Jun 24, 2008 1:04 PM   in response to: Ivan in response to: Ivan
 
  Click to reply to this thread Reply
This is very strange. Is there a chance that you can profile it and see what is taking so much memory.
Ivan

Posts: 39
Registered: 3/10/08
Re: GPS, Transactions, and Out of Memory issues
Posted: Jun 25, 2008 1:39 PM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
kimchy wrote:
This is very strange. Is there a chance that you can profile it and see what is taking so much memory.

Still trying to pinpoint things. Biggest user of memory appears to be the 589K char[] arrays in the system. Most of them are 32K bytes used for zzBuffer in org.apache.lucene.analysis.standard.StandardTokenizerImpl (private char zzBuffer[] = new char[ZZ_BUFFERSIZE];)

On the Compass side, the DefaultFirstLevelCache has many HashMap$Entry in the table variable.

240K Fields () still lingering (143k bytes) , 18K LuceneResources (151k bytes) ), 9K LuceneMultiResources (152k bytes)

Still looking...

Edited by: Ivan on Jun 25, 2008 3:15 PM
Ivan

Posts: 39
Registered: 3/10/08
Re: GPS, Transactions, and Out of Memory issues
Posted: Jun 26, 2008 11:30 AM   in response to: Ivan in response to: Ivan
 
  Click to reply to this thread Reply
The DefaultFirstLevelCache is definitely the memory hog. Items keep getting set, but nothing seems to be removed.

Creating a duplicate resource in the DefaultMarshallingStrategy doesn't help things either. I understand it is for flexibility among the different types, but with rsem it simply creates a deep copy and both objects are retained in memory.
Ivan

Posts: 39
Registered: 3/10/08
Re: GPS, Transactions, and Out of Memory issues
Posted: Jun 26, 2008 3:32 PM   in response to: Ivan in response to: Ivan
 
  Click to reply to this thread Reply
Setting the isolation level to "lucene" will in turn force the NullFirstLevelCache to be used. My problem should be solved.
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: GPS, Transactions, and Out of Memory issues
Posted: Jun 27, 2008 4:48 AM   in response to: Ivan in response to: Ivan
 
  Click to reply to this thread Reply
By the way, sorry, it is my bad. You should call evictAll on CompassSession while indexing periodically.
discount oakley...

Posts: 35
Registered: 8/20/11
Re: GPS, Transactions, and Out of Memory issues
Posted: Aug 20, 2011 4:16 AM   in response to: Ivan in response to: Ivan
 
  Click to reply to this thread Reply
cheap oakley sunglasses are sold here for all age groups and come in a complete array of shades, models and styles. It does not matter what kind of individual you will be, you can find often oakley sunglasses that is created to suit your needs. At our oakley outlet, you can buy discount oakley sunglasses with high quality, save up to 70%, and these are some colorful oakley sunglasses cheap styles you can choose. Now get top quality fashion oakley sunglasses discount that suit your mood.If you are searching for oakley sunglasses outlet online you'll then surely find there are far too many oakley sunglasses dealers,our oakley sunglasses outlet 100% Quality Guarantee,Low Price,Fast Shipping & No Sales Tax,Fast Service,Safe Transaction,Good Reputation!
Legend
Guru: 2001 + pts
Expert: 501 - 2000 pts
Super-star: 101 - 500 pts
Assistant: 51 - 100 pts
Participant: 0 - 50 pts
Helpful Answer (5 pts)
Correct Answer (10 pts)

Point your RSS reader here for a feed of the latest messages in all forums