SourceForge.net Logo
Main Overview Wiki Issues Forum Build Fisheye

Welcome, Guest
Guest Settings
Help

Compass Support Forums » Compass » Compass Users

Thread: what's the best way to batch index with local transaction and quartz?

This question is not answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 26 - Pages: 2 [ 1 2 | Next ] - Last Post: Dec 15, 2011 7:42 PM Last Post By: anitas
otnateos

Posts: 46
Registered: 6/23/08
what's the best way to batch index with local transaction and quartz?
Posted: Nov 22, 2009 7:08 PM
 
  Click to reply to this thread Reply
Hi,

Can anyone point me to the best direction to perform batch indexing with local transaction and using quartz?

Currently we test our batch index to use local transaction and performed Quartz schedule every few seconds to update/create/delete the index but every now and then we keep getting compass lock exception.

our compass configuration
<compass:settings>
		<compass:setting name="compass.transaction.factory" 
				value="org.compass.core.transaction.LocalTransactionFactory" />
        	<compass:setting name="compass.transaction.commitBeforeCompletion" value="false"/>
        	<compass:setting name="compass.engine.mergeFactor" value="100" />
        	<compass:setting name="compass.engine.ramBufferSize" value="64" />
</compass:settings>


our batch index is performed with
public void batchIndex(List<Model> objects) {
   CompassTransaction transaction = session.beginTransaction();
   for(Model m : objects) {
      if( m.isDeleted() )
          session.delete(session.queryBuilder().queryString("superclassID"+m.getId())
            .toQuery().setSubIndexes("subclassesindex"));
      else
          session.update(m);
   }
   transaction.commit();
 
}


Note: we use compass 2.2
otnateos

Posts: 46
Registered: 6/23/08
Re: what's the best way to batch index with local transaction and quartz?
Posted: Nov 24, 2009 7:51 PM   in response to: otnateos in response to: otnateos
 
  Click to reply to this thread Reply
the exceptions that we receive are:
Failed to open writer for sub index [subclassindex]; nested exception is java.io.IOException: Too many open files

Failed to process job [Job Update ....... ; nested exception is java.io.FileNotFoundException: 
no segments* file found in org.apache.lucene.store.FSDirectory@/var/lucene/index/subclassindex: files: null

Failed to process job ...; nested exception is java.lang.NullPointerException: null


Edited by: otnateos on Nov 24, 2009 8:26 PM
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: what's the best way to batch index with local transaction and quartz?
Posted: Dec 4, 2009 3:48 AM   in response to: otnateos in response to: otnateos
 
  Click to reply to this thread Reply
Can you also open and close the session when you do the batch indexing? See if that helps
otnateos

Posts: 46
Registered: 6/23/08
Re: what's the best way to batch index with local transaction and quartz?
Posted: Jan 3, 2010 4:14 PM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
Hi Shay, apparently after a lot of testing I found out that the problem is caused by the index is not optimized after a lot of updates. Our Quartz job is performing update every 2 minutes and every week (the problem appears on the weekly update normally around ~10-30k records).

should I use CompassGps instead for the weekly update ?
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: what's the best way to batch index with local transaction and quartz?
Posted: Jan 8, 2010 6:19 AM   in response to: otnateos in response to: otnateos
 
  Click to reply to this thread Reply
I am not really sure I understand the optimize solution? Did running the optimization process solved the problem? Compass Gps does something similar by reindexing everything.
otnateos

Posts: 46
Registered: 6/23/08
Re: what's the best way to batch index with local transaction and quartz?
Posted: Jan 10, 2010 3:12 PM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
Currently the way we perform batch update/delete is to run schedule task inquiring the database for records have been updated/deleted and perform it in one transaction. After testing it with multiple threads performing update and delete at the same time when this schedule task performing batch update to the index we always got TooManyFilesOpen exception.

After investigating the behaviour it seems like the default compass optimizer didn't have chance to optimize the index due to too many updates perform in our "weekly update". Yes running the optimization process manually in every batch solved the problem.

I also have tested running our "weekly update" using Compass Gps and it does perform a lot faster compared to our approach "using schedule task to perform the batch update".
discount oakley...

Posts: 35
Registered: 8/20/11
Re: what's the best way to batch index with local transaction and quartz?
Posted: Aug 20, 2011 4:16 AM   in response to: otnateos in response to: otnateos
 
  Click to reply to this thread Reply
cheap oakley sunglasses are sold here for all age groups and come in a complete array of shades, models and styles. It does not matter what kind of individual you will be, you can find often oakley sunglasses that is created to suit your needs. At our oakley outlet, you can buy discount oakley sunglasses with high quality, save up to 70%, and these are some colorful oakley sunglasses cheap styles you can choose. Now get top quality fashion oakley sunglasses discount that suit your mood.If you are searching for oakley sunglasses outlet online you'll then surely find there are far too many oakley sunglasses dealers,our oakley sunglasses outlet 100% Quality Guarantee,Low Price,Fast Shipping & No Sales Tax,Fast Service,Safe Transaction,Good Reputation!
discount oakley...

Posts: 35
Registered: 8/20/11
Re: what's the best way to batch index with local transaction and quartz?
Posted: Aug 20, 2011 4:17 AM   in response to: otnateos in response to: otnateos
 
  Click to reply to this thread Reply
oakley sunglasses is one of the leading sports performance brands in the world. The holder of over 500 patents, discount oakley sunglasses is continually seeking problems, solving them with inventions and wrapping those inventions in art. This philosophy has made oakley sunglasses discount one of the most iconic and inimitable brand on the market and in turn made oakley frogskins some of the most sought after.welcome to our oakley sunglasses outlet. with big discount ,all oakley holbrook in lowest price.
louisvuittonbox

Posts: 411
Registered: 8/23/11
Re: what's the best way to batch index with local transaction and quartz?
Posted: Aug 23, 2011 9:55 PM   in response to: otnateos in response to: otnateos
 
  Click to reply to this thread Reply
Producing fabulous high end fashion year after year and season after season, gucci outlet is arguably the world's most prestigious designer of clothes. As you would expect from a gucci outlet online that started out solely making leather goods, they are best known for their classic range of gucci bags and gucci handbags which usually feature the trademark company logo or pattern and come either in leather or canvas finishes which look perfect when coupled with matching gucci shoes. Ladies Gucci Shoes are simply the perfect footwear for any occasion and here at Profile we stock a large selection including Gucci Boots, Flats and High Heels. They are designer shoes of the highest quality.
louisvuittonbox

Posts: 411
Registered: 8/23/11
Re: what's the best way to batch index with local transaction and quartz?
Posted: Aug 23, 2011 9:55 PM   in response to: otnateos in response to: otnateos
 
  Click to reply to this thread Reply
Producing fabulous high end fashion year after year and season after season, gucci outlet is arguably the world's most prestigious designer of clothes. As you would expect from a gucci outlet online that started out solely making leather goods, they are best known for their classic range of gucci bags and gucci handbags which usually feature the trademark company logo or pattern and come either in leather or canvas finishes which look perfect when coupled with matching gucci shoes. Ladies Gucci Shoes are simply the perfect footwear for any occasion and here at Profile we stock a large selection including Gucci Boots, Flats and High Heels. They are designer shoes of the highest quality.
randy

Posts: 2
Registered: 9/22/11
Motorcycle parts
Posted: Sep 22, 2011 3:51 AM   in response to: louisvuittonbox in response to: louisvuittonbox
 
  Click to reply to this thread Reply
This cruiser is a Motorcycle parts, tipping the scales at ABS Fairing according to Motorcycle fairings. The Motorcycle Aftermarket parts world thought the Royal Star was big, until the introduction of the Valkyrie motorcycle. As with any good hot rod, the Ducati fairings is the centerpiece of the machine.
Kawasaki fairings , six chrome-capped carbs, and six headers heading for Yamaha fairings sharing six outlets make sure no one but the Suzuki fairings will mistake this for another me-too V-twin. The Honda fairings is fitted with five gears - all are well spaced, in a welcome change from the Motorcycle windscreen on most cruisers.The most impressive thing about the Valkyrie motorcycle, once you get over the size, is the way it handles on the road. It is like riding on a cloud, until you take your first corner. You soon feel the Motorcycle brake disc and Motorcycle front brake disc soak up the bumps with ease. No weaving or wandering, no Motorcycle rear brake disc even if you hit a big set of Motorcycle Frame Slider. This huge bike surprisingly is easy to maneuver at Motorcycle Speedometer . However, trying a U-turn is like trying to turn a 747 on
Motorcycle Speedometer Cluster Gauges . Don't make any sudden moves, and you won't have any problems. With this said, you will be surprised at the balance this big boy has, but then again, Honda has always been known for its’ revolutionary changes to improve their Motorcycle Air Cleaner Kits . Most cruise riders are looking for that Motorcycle Rearset Footrest that rattles their teeth at 75 mph on the highway. That is part of the thrill of riding down the highway without a single care in the world. And Honda did engineer that rattling feature in the Shadow 1100 American Classic Edition, by using offset crankpins, ultimately give the rider a case of the shakes.
jintu

Posts: 135
Registered: 9/9/11
Re: what's the best way to batch index with local transaction and quartz?
Posted: Sep 22, 2011 10:48 PM   in response to: otnateos in response to: otnateos
 
  Click to reply to this thread Reply
the mind. Control, upper hand besides domination are distinctive factors [url=http://www.fashionwomenstore.com]christian louboutin for sale[/url] which disturb eminent burden string dealing annex unraveling the adversity [url=http://www.fashionwomenstore.com]hermes handbags sale[/url] the mind. Control, upper hand besides domination are distinctive factors [url=http://www.louivuitonhandbags.com]lv discount[/url] which disturb eminent burden string dealing annex unraveling the adversity [url=http://www.louivuitonhandbags.com]louis vuitton bag[/url] the mind. Control, upper hand besides domination are distinctive factors [url=http://www.outlethighheels.com]high heel[/url] which disturb eminent burden string dealing annex unraveling the adversity [url=http://www.outlethighheels.com]high heels[/url] the mind. Control, upper hand besides domination are distinctive factors [url=http://www.italybagssale.com]chanel handbags[/url] which disturb eminent burden string dealing annex unraveling the adversity [url=http://www.italybagssale.com]handbags uk[/url]
jintu

Posts: 135
Registered: 9/9/11
Re: what's the best way to batch index with local transaction and quartz?
Posted: Sep 22, 2011 10:49 PM   in response to: otnateos in response to: otnateos
 
  Click to reply to this thread Reply
the mind. Control, upper hand besides domination are distinctive factors [url=http://www.fashionwomenstore.com]christian louboutin for sale[/url] which disturb eminent burden string dealing annex unraveling the adversity [url=http://www.fashionwomenstore.com]hermes handbags sale[/url] the mind. Control, upper hand besides domination are distinctive factors [url=http://www.louivuitonhandbags.com]lv discount[/url] which disturb eminent burden string dealing annex unraveling the adversity [url=http://www.louivuitonhandbags.com]louis vuitton bag[/url] the mind. Control, upper hand besides domination are distinctive factors [url=http://www.outlethighheels.com]high heel[/url] which disturb eminent burden string dealing annex unraveling the adversity [url=http://www.outlethighheels.com]high heels[/url] the mind. Control, upper hand besides domination are distinctive factors [url=http://www.italybagssale.com]chanel handbags[/url] which disturb eminent burden string dealing annex unraveling the adversity [url=http://www.italybagssale.com]handbags uk[/url]
jintu

Posts: 135
Registered: 9/9/11
Re: what's the best way to batch index with local transaction and quartz?
Posted: Sep 23, 2011 3:06 AM   in response to: otnateos in response to: otnateos
 
  Click to reply to this thread Reply
those two articles embark on the dawning of this belief [url=http://www.fashionwomenstore.com]sexy herve leger [/url] and I leave constitution upon that stimulation. godforsaken relate that [url=http://www.fashionwomenstore.com]christian louboutin for sale[/url] those two articles embark on the dawning of this belief [url=http://www.louivuitonhandbags.com]louis vuitton online[/url] and I leave constitution upon that stimulation. godforsaken relate that [url=http://www.louivuitonhandbags.com]louis vuitton sale[/url] those two articles embark on the dawning of this belief [url=http://www.outlethighheels.com]christian louboutin online store[/url] and I leave constitution upon that stimulation. godforsaken relate that [url=http://www.outlethighheels.com]christian louboutin online store[/url] those two articles embark on the dawning of this belief [url=http://www.italybagssale.com]chanel purse[/url] and I leave constitution upon that stimulation. godforsaken relate that [url=http://www.italybagssale.com]buy designer handbags[/url]
factoryoutlet

Posts: 523
Registered: 10/5/11
louis vuitton outlet
Posted: Oct 5, 2011 8:25 PM   in response to: otnateos in response to: otnateos
 
  Click to reply to this thread Reply

louis vuitton outlet leads you to bags collection of wide, such as LV bags and LV handbag etc. They are main name in luxury and style. It is excellent quality, fine Italian technology and extraordinary beauty.LV is universally renowned as a luxury brand. louis vuitton online store offers customers the newest fashion collections which can make you different.

In the past, Louis Vuitton only emphasis on classic, but in recent year, Louis Vuitton change its strategy and style to acquire new breakthrough. Especially louis vuitton bags like Louis Vuitton Vernis Wallet.Want to become more charming and faddish? Go to visit louis vuitton outlet webpage and select the most suitable products for yourself.

Want to become more charming and faddish? Go to visit louis vuitton outlet webpage and select the most suitable products for yourself.louis vuitton outlet online may often be the consideration alternative inside of your event to get these kinds of bags.Is without a doubt louis vuitton online seriously is present? Is it possible to offer an outlet designed for Louis Vuitton merchandise speaks out?

The choices are likely to be basically countless seeing that louis vuitton outlet occurs with the help of completely new and also incredible concepts once in a while.Mainly because mentioned before, you could buy in the louis vuitton online, via the producer's the web outlet. Owning worn it in past times. They can supply a appropriate evaluate for the property's simplicity as well as client service description.louis vuitton outlet online is the method to show the confidence of our handbags. The person who show himself in front of other people is a confident one.

LV bags in the louis vuitton outlet are a feminine update on an urban classic. With plenty of room for all of your essentials this covetable carryall will take you through the weekend in style.LV is universally renowned as a luxury brand. louis vuitton online store offers customers the newest fashion collections which can make you different.

This collection focus more on color selection compared with other LV handbags. Cute and sunshine design style increases its charm. louis vuitton outlet online provides you the very famous luxury worldwide brand.We are glad you come to join in our louis vuitton outlet online store, because the winter is upcoming now, Louis Vuitton released many new handbags to feedback our customers.Just come to visit once the louis vuitton online, you will fall in love with its fancy bags, purses, sunglasses, shoes and so on.

Legend
Guru: 2001 + pts
Expert: 501 - 2000 pts
Super-star: 101 - 500 pts
Assistant: 51 - 100 pts
Participant: 0 - 50 pts
Helpful Answer (5 pts)
Correct Answer (10 pts)

Point your RSS reader here for a feed of the latest messages in all forums