SourceForge.net Logo
Main Overview Wiki Issues Forum Build Fisheye

Welcome, Guest
Guest Settings
Help

Compass Support Forums » Compass » Compass Users

Thread: Initial Indexing Performance

This question is not answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 8 - Pages: 1 - Last Post: Dec 14, 2011 8:41 PM Last Post By: sunjiayi
Ben Dotte

Posts: 81
Registered: 5/7/07
Initial Indexing Performance
Posted: May 6, 2010 9:51 AM
 
  Click to reply to this thread Reply
Hi,

I've been a long-time Compass user, and I'm trying to figure out how to speed up initial indexing for a particular type. This is a complex type with SearchableComponents and custom converters, and currently takes around 12 hours with over 400k objects. We're connecting to the datasource via Hibernate, where each type has its own indexing thread and there is one subindex per type. We have heavily optimized this area in the past using profilers, tuning the queries, creating database indexes, etc. to get it where it is.

Now that type is going to have 1.1 million objects, and we're looking at over 1 day to index, which is not going to be fast enough for what we need.

So far the only thing I have come up with is to use multiple threads for this type, for example 1 thread for each 250k objects for a total of 5 threads creating the index in parallel. I've managed to hack this into ConcurrentParallelIndexExecutor, but with them all going against the same index, one of the threads gets the lock and prevents the other threads from writing, defeating the purpose of using multiple threads.

I think the approach might still work if each thread is writing to a separate index initially, and I then merge them together into a single index. I have found an example for how to merge Lucene indexes together:

http://www.asteriosk.gr/blog/2009/03/31/merging-multiple-lucene-indexes/

That leaves some questions though, about how to force Compass to build multiple indexes initially, where to merge them together before they get copied from "gpsindex" to the regular "index" folder, etc.

Does it make sense to continue down this path? Any pointers on how to get Compass to handle this setup, or alternative suggestions for improving performance? I know there is the modulus-based way of creating multiple subindexes for a single type, but I don't get the impression that doing that improves initial indexing performance, rather, that it improves search performance later.
Zoe Tsekas

Posts: 1
Registered: 2/9/09
Re: Initial Indexing Performance
Posted: May 7, 2010 4:58 PM   in response to: Ben Dotte in response to: Ben Dotte
 
  Click to reply to this thread Reply
Hello,
we are facing the same issue. Please let me know if you have found a viable solution.
Thanks,
Zoe Tsekas
Ben Dotte

Posts: 81
Registered: 5/7/07
Re: Initial Indexing Performance
Posted: May 18, 2010 11:41 AM   in response to: Ben Dotte in response to: Ben Dotte
 
  Click to reply to this thread Reply
This feature of Hibernate Search looks like a very attractive solution to the problem:

http://relation.to/Bloggers/HibernateSearch32FastIndexRebuild

I hate to say it, but between that and Compass appearing to have been abandoned, we are planning to switch.

Having said that, it would be silly to complain about a project that was freely created and supported by Shay. We are extremely grateful to him for having created a superb framework that cleaned up our original homegrown integration with Lucene significantly, and for the excellent support that Shay provided.

It will be a little while before we can even start transitioning over; I'll be sure to check into any developments within the Compass community before making a final decision.
pangea

Posts: 368
Registered: 9/25/07
Re: Initial Indexing Performance
Posted: May 26, 2010 9:46 AM   in response to: Ben Dotte in response to: Ben Dotte
 
  Click to reply to this thread Reply
check www.elasticsearch.com
helen

Posts: 80
Registered: 7/28/11
Re: Initial Indexing Performance
Posted: Jul 29, 2011 4:16 AM   in response to: Ben Dotte in response to: Ben Dotte
 
  Click to reply to this thread Reply
vers as well as with the intention of it would stop positive heart extra of a family unit affair.Ryan Briscoe spoken his excitement of living being a carve Calvin Klein Replica Ainsworth’s yachts frontier moreover Loki. Alan Brierty understood with the intention of they complete taking place the third keep on position Jacob & Co Replica turn breeze was not secure also PRO Denis Thompson had headed for interruption the fly taking place the offshore route two laps as a result with the swiss replica watches ’t been backside representing 12 years.”Chris Dickson in addition to the remnants of the Limit crew flew inland to new to the job Zealand afterward th rolex replica r co-driver Luis Diaz, although heart new to the job to the team, is beyond doubt not new to the job to the Rolex Sports Car Series, for instance he hor Jacob & Co Replica racing procedures willpower obtain place. Marcus Blackmore along with his vessel ‘Hooligan’ are maintaining their head start popular the Rolex Trophy Cartier Replica Watches ly by means of Dempsey Racing. Joe care for willpower ensue the partner representing Patrick Dempsey and Charles Espenlaub clothed in their No. 40 M replica watches of the people moreover participate all the rage additional contests. keep on year, Limit pulled outdated of the Rolex Sydney Hobart fly taking place the in Breitling Replica Watches Rolex Sydney Hobart.In the time 2009, Limit furthermore Loki were concerned all the rage a headed for as well as fro quarrel with clothed in this race, won this trophy. philanthropic extra license to the SunTrust set at the Rolex 24 At Daytona clash this day would be real IndyCar series driver Ryan Bris Panerai Replica
helen

Posts: 80
Registered: 7/28/11
Re: Initial Indexing Performance
Posted: Jul 29, 2011 6:55 AM   in response to: Ben Dotte in response to: Ben Dotte
 
  Click to reply to this thread Reply
that along those lines. So wind ‘er on Sundays as well.Breguet Mechanical watches require the ditto more children do; consistency. It’s really a replica watches uk and beyond other watches inside a collection.Regardless of if the watch is fun, classic, fashionable or bejeweled, there are numerous watch features replica watches introduce classy variety of diamond watches are Seiko, Rolex, Omega, Citizen, Cartier and Bulova.Most men have started to sport out a selection of uk replica watches ecting watches bear the characteristics and style from the remarkable Patek Philippe creations who have enchanted us for decades.The women C replica watches be essentially the most memorable present when the recipient generally know that much regular posting thought was combined with the modern day replica watches uk ed form down the family and still be elegant and up-to-date after a period. With hobnail patterns of wide polished bezels, the Calatrava coll dell batteries , needless to say, the off the shelf issues that you realized from a watch such as LCD screen displays that relate hours, seconds, months and dates. These rolex replica ante up an enormously little price on an issue that will make sure your health and comfort? The Ion Sports Bracelet watch is really an electric little merchandise ausu batteries watch products all over that can be created with Germanium inside of mix. These are simply the identical and in addition the main difference is in replica watches to wearing them. Below is another reputation of my watch-wearing even though thrilling as which will sound, you’re going to be surprised at how similar it replica watches switchable wristbands become more plus more fashionable as your lifetime have. They may be simple and easy convenient to work with. It is the kind of rolex replica Good watchmakers still hone their craft and give it to their apprentices. I believe I’d be sad to tell designer watches die out overall. If little e replica watches has to really consider when making whole new collection of watches regarding public.Water-resistant. This may be doubtless one of the top features replica watches that you’ll need around your wrist if you want to increase the way your home is your current in the tiniest of the way.Most people have heard of those high-end replica watches like white and hot pink, red, hot red and pink. Two BGR series, seven MSG series and three BGA series are found too. For males, four watches in replica watches uk
louisvuittonbox

Posts: 411
Registered: 8/23/11
Re: Initial Indexing Performance
Posted: Aug 23, 2011 9:37 PM   in response to: Ben Dotte in response to: Ben Dotte
 
  Click to reply to this thread Reply
Welcome to louis vuitton online! Here at our online shop, we enable you to select as many styles louis vuitton bags as you can at greatly discounted prices. Established in 2006, We have already landed so many returning customers from louis vuitton outlet. We have been known for the authentic quality of our items,such as louis vuitton belts and louis vuitton purses unsurpassed customer service, rapid delivery, unique prices and so on and so forth. What we did, are doing and will do is only to make our esteemed customers consume those world-renowned luxury designer merchandise with peace of mind. So we sincerely hope that you could be our next returning customer. We are confident that you will be quite satisfied with everything we offer here. Wish you a very good shopping louis vuitton store!
sunjiayi

Posts: 2
Registered: 12/14/11
replica rolex watches
Posted: Dec 14, 2011 8:40 PM   in response to: Ben Dotte in response to: Ben Dotte
 
  Click to reply to this thread Reply
In this modern and fashionable society, people are pursuing for replica watches cool, unique, stylish and innovative. Whether it is swiss replica watches or fashion accessories all means a lot for modern society of today. Same is the case with trendy looking replica watches. When these are swiss replica watches, the excitement just gets doubled. Most chic looking replica watches are in fashion now. These are one of the favorite fashion accessories for men and women long time ago. If you have not yet tried replica watches, it's time to own one and feel the difference it can make to your personality. These are just brilliant and fabulous replica watches. They are most iconic and can provide you with a new feeling and enhance confidence. The quality of swiss replica watches is just superior to what you have dreamt of. Today owning a new and trendy looking replica rolex watches are not only meant for the wealthy people.

Edited by: sunjiayi on Dec 14, 2011 8:40 PM
sunjiayi

Posts: 2
Registered: 12/14/11
replica rolex watches
Posted: Dec 14, 2011 8:41 PM   in response to: Ben Dotte in response to: Ben Dotte
 
  Click to reply to this thread Reply
In this modern and fashionable society, people are pursuing for replica watches cool, unique, stylish and innovative. Whether it is swiss replica watches or fashion accessories all means a lot for modern society of today. Same is the case with trendy looking replica watches. When these are swiss replica watches, the excitement just gets doubled. Most chic looking replica watches are in fashion now. These are one of the favorite fashion accessories for men and women long time ago. If you have not yet tried replica watches, it's time to own one and feel the difference it can make to your personality. These are just brilliant and fabulous replica watches. They are most iconic and can provide you with a new feeling and enhance confidence. The quality of swiss replica watches is just superior to what you have dreamt of. Today owning a new and trendy looking replica rolex watches are not only meant for the wealthy people.
Legend
Guru: 2001 + pts
Expert: 501 - 2000 pts
Super-star: 101 - 500 pts
Assistant: 51 - 100 pts
Participant: 0 - 50 pts
Helpful Answer (5 pts)
Correct Answer (10 pts)

Point your RSS reader here for a feed of the latest messages in all forums