SourceForge.net Logo
Main Overview Wiki Issues Forum Build Fisheye

Welcome, Guest
Guest Settings
Help

Compass Support Forums » Compass » Compass Users

Thread: Slow Indexing on Large Amounts of Data - batch_insert without GPS

This question is answered.


Permlink Replies: 8 - Pages: 1 - Last Post: Jan 1, 2012 7:30 PM Last Post By: miyao lin
Raq

Posts: 13
Registered: 1/16/08
Slow Indexing on Large Amounts of Data - batch_insert without GPS
Posted: Feb 26, 2008 8:40 AM
 
  Click to reply to this thread Reply
Hi, its me again :)

I've got the same problem thats stated [url=http://forum.compass-project.org/thread.jspa?threadID=180709]here[/url],

except that I'm not using any GPS devices.

lauri wrote:
I'm indexing approximately million entries through hibernate GPS. The indexing process starts quickly, taking almost 100% of my CPU capacity. Indices appear to the file system quickly as well.
However, the indexing process slows down quite quickly: after 5 hours since the beginning, the CPU load is around 5-10%, and the indexed data appear slower on the file system.

A bit of background into my setup can be found here.
http://forum.compass-project.org/thread.jspa?threadID=215196

Now I've read the tips for optimizing the performance by tweaking [b]compass.engine.maxBufferedDocs[/b] and [b]compass.engine.mergeFactor[/b] on Shays article mentioned here.

http://jroller.com/kimchy/entry/boost_batch_index_performance

[b]compass.engine.maxBufferedDocs[/b] and [b]compass.engine.mergeFactor[/b] only apply to batch_inserts. Having read through the reference document on batch_inserts, it seems to be only applied to designs that are using gps devices.

Can batch_insert be applied to manual iterative inserts? If so, can I please be pointed to an example.

The following is the portion of code that performs a single insert into the index:

CompassSession session = null;
CompassTransaction transaction = null;
 
        try
        {
            session = acompass.openSession();
            transaction = session.beginTransaction();
            session.save(someitem);
            transaction.commit();
            logger.info("Lucene:" + someitem.getPath() + " - has been indexed");
        }
        catch (Exception e)
        {
            logger.error("indexItem(someitem)", e);
            transaction.rollback();
        }
        finally
        {
            session.close();
            session = null;
        }


The above routine will be hit about a million times every night when the indexing operation cascades through our data model.

Should I be saving multiple items to the session (like 1000) before I commit the transaction?

Any advice on how to optimise?
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Slow Indexing on Large Amounts of Data - batch_insert without GPS
Posted: Feb 27, 2008 6:54 AM   in response to: Raq in response to: Raq
Correct
  Click to reply to this thread Reply
Yes, you should batch your save operations and commit only after a set amount. The higher the better.
Raq

Posts: 13
Registered: 1/16/08
Re: Slow Indexing on Large Amounts of Data - batch_insert without GPS
Posted: Feb 27, 2008 8:29 AM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
Time taken to reindex before change = 7.5 Hours

Time taken to reindex after change = Under one minute..

I cant believe my eyes :)

One thing though thats come up on here time and time again..

After its completed reindexing I get [b]Failed to obtain lock on sub-index xyz, will do it next time.[/b] on the Aggressive Optimiser.

and it doesn't go away. its not an exception and does not break anything, but its showing up on the log every couple of seconds.

I've looked at your advice on another topic..

[b]"Stop the optimizer before starting the indexing process"[/b] - Done

[b]"Call optimize each time you close and open a new session/transaction"[/b] - Done

Still get the warning.

[b]"Set the schedule flag to false in your configuration, so no scheduling will happen"[/b]

Tried this, although it gets rid of the warnings.. as no optimisation is happening it has tripled the time to perform queries on the index (200ms to 600+ms).

[b]"You will still need to call optimize periodically."[/b] - Do you mean by creating a seperate job, like a quartz job to call optimize? Or can you suggest a better approach?

Thanks..
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Slow Indexing on Large Amounts of Data - batch_insert without GPS
Posted: Feb 27, 2008 12:17 PM   in response to: Raq in response to: Raq
 
  Click to reply to this thread Reply
First, optimization should be performed at the end of the indexing process.

Second, the lock obtain exception might happen due to two reasons: The first is that somewhere in your code you fail to commit/rollback a transaction. This will cause a lock to remain on the index. The other reason can be that you have frequent index operations, which simple does not allow the optimizer to obtain the lock in order to optimize the index.

If you are in control of batch indexing, I would simply disable the scheduled optimizer, and call optimize after you do your batch indexing. Note, for save operations here and there, don't call optimize on it, since optimize can be a very heavy operation, especially with the AgressiveOptimizer.
louisvuittonbox

Posts: 411
Registered: 8/23/11
Re: Slow Indexing on Large Amounts of Data - batch_insert without GPS
Posted: Aug 23, 2011 9:42 PM   in response to: Raq in response to: Raq
 
  Click to reply to this thread Reply
Producing fabulous high end fashion year after year and season after season, gucci outlet is arguably the world's most prestigious designer of clothes. As you would expect from a gucci outlet online that started out solely making leather goods, they are best known for their classic range of gucci bags and gucci handbags which usually feature the trademark company logo or pattern and come either in leather or canvas finishes which look perfect when coupled with matching gucci shoes. Ladies Gucci Shoes are simply the perfect footwear for any occasion and here at Profile we stock a large selection including Gucci Boots, Flats and High Heels. They are designer shoes of the highest quality.
factoryoutlet

Posts: 523
Registered: 10/5/11
louis vuitton outlet
Posted: Oct 5, 2011 10:09 PM   in response to: Raq in response to: Raq
 
  Click to reply to this thread Reply

louis vuitton outlet leads you to bags collection of wide, such as LV bags and LV handbag etc. They are main name in luxury and style. It is excellent quality, fine Italian technology and extraordinary beauty.LV is universally renowned as a luxury brand. louis vuitton online store offers customers the newest fashion collections which can make you different.

In the past, Louis Vuitton only emphasis on classic, but in recent year, Louis Vuitton change its strategy and style to acquire new breakthrough. Especially louis vuitton bags like Louis Vuitton Vernis Wallet.Want to become more charming and faddish? Go to visit louis vuitton outlet webpage and select the most suitable products for yourself.

Want to become more charming and faddish? Go to visit louis vuitton outlet webpage and select the most suitable products for yourself.louis vuitton outlet online may often be the consideration alternative inside of your event to get these kinds of bags.Is without a doubt louis vuitton online seriously is present? Is it possible to offer an outlet designed for Louis Vuitton merchandise speaks out?

The choices are likely to be basically countless seeing that louis vuitton outlet occurs with the help of completely new and also incredible concepts once in a while.Mainly because mentioned before, you could buy in the louis vuitton online, via the producer's the web outlet. Owning worn it in past times. They can supply a appropriate evaluate for the property's simplicity as well as client service description.louis vuitton outlet online is the method to show the confidence of our handbags. The person who show himself in front of other people is a confident one.

LV bags in the louis vuitton outlet are a feminine update on an urban classic. With plenty of room for all of your essentials this covetable carryall will take you through the weekend in style.LV is universally renowned as a luxury brand. louis vuitton online store offers customers the newest fashion collections which can make you different.

This collection focus more on color selection compared with other LV handbags. Cute and sunshine design style increases its charm. louis vuitton outlet online provides you the very famous luxury worldwide brand.We are glad you come to join in our louis vuitton outlet online store, because the winter is upcoming now, Louis Vuitton released many new handbags to feedback our customers.Just come to visit once the louis vuitton online, you will fall in love with its fancy bags, purses, sunglasses, shoes and so on.

factoryoutlet

Posts: 523
Registered: 10/5/11
coach factory outlet
Posted: Oct 5, 2011 10:22 PM   in response to: Raq in response to: Raq
 
  Click to reply to this thread Reply

coach factory outlet offer excellent, elegant coach products in order to generate room for your new collections.They're purchased more quickly that you may at any time imagine.coach factory store is the greatest destination to buy what type of bags you like. If you need to manage a new Coach carrier and most preferable strategy is to the coach regarding coupon codes.Here's your new way to look sophisticated styles. Carry the hot coach factory outlet store of this summer and be the true trendsetter!in the market you definitely can find various colorways that are designed in as well as the high quality that applied in. For most of you would like to come. So just come to our coach factory outlet online store to choose one.

coach factory outlet is really sizzling kinds of shopping way for you. With the usage of the replica designer coach bags, you can surely be able to change your individual looks in a stunning manner.You know, Coach items are so perfect and fascinating. Now I grow up, and find coach outlet on the Internet offering affordable products with reliable quality.Many fashionable women match with practical Coach Purses which will make the street shopping become relaxed,and make every person can enjoy more diversiform combination in coach factory outlet online.The coach factory outlet store are utilized for formal event in company area that will display your excellent taste.Not only does a briefcase make is easier to carry things.

Almost everyone has identified effectively about this, below, among the best to make sure you will get much more information about coach outlet.In the coach outlet online shop, a collection of high quality and hottest Coach Shoulder Bags is offered to you. They are all at the best prices you will ever find.

You know, Coach items are so perfect and fascinating. Now I grow up, and find coach outlet on the Internet offering affordable products with reliable quality.Once visit, you will be unable to restrain loving it. Warmly welcome all of you to the Summer Sales Promotion on coach factory outlet.coach factory outlet has new coach laptop bags in listing. No matter which style do you want, we can provide it for you.Coach bags enjoy high popularity throughout the world. I would like to share the coach factory outlet online with you. What are you waiting for? Just come to visit.

coach factory outlet supplies the newest and most fashionable brand Coach on sale.The more Coach shoulder bags you order, the more discount you will get.It's wonderful to find out it does not slip in your signature at the bottom of the case and the appearance of brass accents from coach outlet online, Because women who can look stylish this summer.coach outlet has a zippered closure and buckles for extra security. Shiny brass hardware, rounded leather handle, and an interior pocket. It also includes a limited edition Hawaii luggage tag and lock.You can find all kinds of coach bags whatever you chose which one in coach factory outlet online will never be out of fashion. Not only they have many nice and design styles, but also made from the finest leather and fabric.

With the great diversity of styles, patterns and sizes available nowadays, finding the right pieces to highlight your personality is quite important. coach outlet offers exactly what you want.coach factory outlet supplies the newest and most fashionable brand Coach on sale.The more Coach shoulder bags you order, the more discount you will get.Look at this golden Coach bag in the? coach factory outlet online.Those new bags with classic and fashionable design are more and more popular now!After graduating from university, my cousin came back to my hometown from USA last month. She told me that coach outlet store is very popular among her classmate.

coach outlet online own a couple of line, color, or the selected theme, it should be a good choice. It may be the place where offer the bag ladies handbag, you will be able to follow a course of education or training receives for the slim project chest strap wallet.coach outlet will send you a coupon in the post to use in their upon only after you type a achieve. The Coach bags are customarily somewhat many than the ones in the stores.In particular, products from coach factory outlet with leather design are fashionable, handmade,leading the wave of American pop.It with simple,durable and unique style to win consumers.coach factory not only supply with the highest quality crystal at reasonable price but also provide good after-sales service. Companies operating purposes: science and technology in order to survive, credibility and development, strict quality control.

coach outlet handbags's special design and fashion is certainly and so eye-catching, I hope you are able to get much more attention towards the Cheap Coach Crossbody.As long as you open our coach factory outlet webpage, you will view various Coach New Arrivals which are the most popular also the most fashionable in this year.Are you anxious to choose some of the brand bags? Are you searching for reliable sellers? When you come to the coach factory outlet online, these are not problems.

At the coach outlet online you have the largest selection of the day. If you touch the item and like it, keep it in your possession until you make your final decision.Over the years, coach factory online has added a multitude of new handbag shapes, styles and materials to their collection. However, the highest care is taken that every Coach handbags is both aesthetically beautiful and functional.Are you searching for a cool and stylish coach bag? The coach factory outlet online Online is your first priority.

anitas

Posts: 95
Registered: 11/20/11
Re: Slow Indexing on Large Amounts of Data - batch_insert without GPS
Posted: Jan 1, 2012 7:03 PM   in response to: Raq in response to: Raq
 
  Click to reply to this thread Reply
I love this article, it's very well.oakley sunglasses saleFake Oakley Flak Jacket Sunglassesreplica Oakley Polarized SunglassesOakley Batwolf sunglasses outletoakley sunglasses wholesaleReplica Oakley Flak Jacket SunglassesOakley Restless SunglassesOakley Dispatch sunglassesreplica oakley sunglassesReplica Oakley Fuel Cell Sunglassesoakley scalpel sunglassesOakley Flak Jacket sunglassesreplica Oakley Commit SQ SunglassesFake Oakley GascanOakley Split Jacket SunglassesOakley Frogskins sunglasses outletoakley fives squared sunglassesFake Oakley Polarized Sunglassesoakley straight jacket sunglassesOakley Gascan sunglassesOakley Half Jacket SunglassesReplica Oakley Polarized SunglassesOakley Ten SunglassesFake Oakley Hijinx sunglassesreplica Oakley Half Wire SunglassesFake Oakley RadarOakley Zero SunglassesOakley Jawbone sunglassesreplica Oakley Half X SunglassesReplica Oakley sunglassesOakley Juliet sunglassesOakley Hijinx SunglassesCheap oakleys Commit SQ SunglassesOakley Jupiter sunglassesOakley Holbrook SunglassesReplica Oakley Fives SquaredOakley M Frame sunglasses outletoakley jawbone sunglassesReplica Oakley Flak JacketOakley Minute sunglassesoakley juliet sunglassesReplica Oakley FrogskinsOakley Monster Dog sunglassesreplica Oakley Jupiter SunglassesReplica Oakley Fuel CellOakley Oil Rig sunglassesOakley Jury SunglassesReplica Oakley GascanOakley Pit Boss sunglassesoakley m frame sunglassesCheap oakleys Half Wire SunglassesOakley Polarized sunglasses outletoakley minute sunglassesCheap oakleys Half X SunglassesOakley Radar sunglassesreplica oakley minute sunglassesCheap oakleys Hijinx SunglassesOakley Restless sunglassesoakley monster dog sunglassesCheap oakleys Jawbone SunglassesOakley Scalpel sunglassesOakley Necessity SunglassesCheap oakleys Oil Rig SunglassesOakley Straight Jacket sunglassesoakley oil rig sunglassesCheap oakleys Pit Boss SunglassesOakley Ten sunglassesOakley Pit Bull SunglassesCheap oakleys Radar SunglassesOakley Zero sunglassesOakley Plaintiff SunglassesCheap oakleys Straight Jacket Sunglasses
miyao lin

Posts: 97
Registered: 11/20/11
Re: Slow Indexing on Large Amounts of Data - batch_insert without GPS
Posted: Jan 1, 2012 7:30 PM   in response to: Raq in response to: Raq
 
  Click to reply to this thread Reply
It really a good article.cheap oakley sunglassesFake Oakley Flak JacketOakley Half X sunglassesoakley eyewearOakley Antix sunglassesoakley sunglassesFake Oakley Fuel CellOakley Holbrook sunglassesoakley sunglasses discountedOakley Batwolf sunglassesOakley Antix SunglassesFake Oakley Fuel Cell Sunglassesoakley batwolf sunglassesOakley Bottlecap sunglassesOakley Bottlecap SunglassesFake Oakley Gascan SunglassesOakley Commit SQ SunglassesOakley Commit SQ sunglassesoakley five squared sunglassesFake Oakley PolarizedOakley Half X SunglassesOakley Dispatch sunglasses outletOakley Jupiter SunglassesOakley Encounter sunglassescheap oakley minute sunglassesCheap oakleys Bottlecap SunglassesOakley Pit Boss SunglassesCheap oakleys Fives Squared Sunglassesreplica Oakley Pit Boss SunglassesOakley Enduring Pace sunglassesOakley Polarized SunglassesOakley Fives Squared sunglassesoakley crosshair sunglassesCheap oakleys Flak Jacket SunglassesOakley Dangerous SunglassesCheap oakleys Frogskins SunglassesOakley Dart SunglassesFake Oakley Flak Jacket sunglassesoakley dispatch sunglassesOakley Forsake sunglassesOakley Encounter SunglassesCheap oakleys Fuel Cell SunglassesOakley Enduring Pace SunglassesCheap oakleys Gascan SunglassesOakley Fast Jacket SunglassesOakley Frogskins sunglassesOakley Fives SunglassesFake Oakley Gascan sunglassesoakley flak jacket sunglassesFake Oakley DispatchOakley Forsake SunglassesFake Oakley Fast JacketOakley Frogskins SunglassesOakley Hijinx sunglassesoakley fuel cell sunglassesOakley M Frame sunglassesoakley gascan sunglassesFake Oakley Five SquaredOakley Half Wire SunglassesFake Oakley FrogskinsFake Oakley Half XFake Oakley Polarized sunglassesFake Oakley HijinxOakley Polarized sunglassesFake Oakley HolbrookFake Oakley Straight Jacket sunglassesFake Oakley JupiterOakley Fuel Cell sunglassesFake Oakley JuryOakley Half Jacket sunglassesFake Oakley Monster DogOakley Half Wire sunglasses
Legend
Guru: 2001 + pts
Expert: 501 - 2000 pts
Super-star: 101 - 500 pts
Assistant: 51 - 100 pts
Participant: 0 - 50 pts
Helpful Answer (5 pts)
Correct Answer (10 pts)

Point your RSS reader here for a feed of the latest messages in all forums