SourceForge.net Logo
Main Overview Wiki Issues Forum Build Fisheye

Welcome, Guest
Guest Settings
Help

Compass Support Forums » Compass » Compass Users

Thread: Subindices and performance

This question is answered. Helpful answers available: 0. Correct answers available: 1.


Permlink Replies: 11 - Pages: 1 - Last Post: Nov 18, 2011 1:55 AM Last Post By: lorna lee
Yuval Sapir

Posts: 8
Registered: 2/27/08
Subindices and performance
Posted: Feb 27, 2008 7:38 AM
 
  Click to reply to this thread Reply
Hello.
We're starting to start Compass and Lucene in our application and the question of using subindices rose.
Can anyone tell me how performance is affected by the number of subindices?
Is there any other aspect that I should take into account when deciding whether to use subindices and how many?

Thanks,
Yuval
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Subindices and performance
Posted: Feb 27, 2008 12:12 PM   in response to: Yuval Sapir in response to: Yuval Sapir
Helpful
  Click to reply to this thread Reply
When using several sub indexes then the concurrency of your system for dirty operations will be better (since only the relevant sub indexes will be locked), but search will be slower (but usually still very fast).

Also, if you integrate Compass with an ORM, then during the indexing operation, if you have two aliases mapped to the same sub index, they will be indexed one after the other. If they are each mapped to their own respective sub index, then the indexing process will happen in parallel.
lauri

Posts: 37
Registered: 12/4/07
Modulo subindex hashing and parallel indexing
Posted: Feb 28, 2008 1:40 AM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
Is the indexing done in parallel with modulo subindices?

I have an alias which I have configured to use a modulo subindex of size 5. Most of my data is on this alias, and I'd like to take advantage of parallel indexing with this alias.

Any comments on this configuration?
Yuval Sapir

Posts: 8
Registered: 2/27/08
Re: Subindices and performance
Posted: Feb 28, 2008 2:09 AM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
We have a single class (and thus a single alias) and we don't use ORM, so I guess it won't help there.
The reason we considered subindices in the first place is that the indexing takes a very long time and consumes a huge amount of memory (to the point of crashing with OutOfMemory), so we divided the objects so that each (insertion) transaction will only access a single subindex, which eased the problem.
The issue now is that we don't know how will this impact performance when searching (since indexing is done once, I don't really care how long will it take as long as it runs to completion). Before starting doing our own measurements for performance I wanted to know if there's any common knowledge that might save me the time and trouble.
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Modulo subindex hashing and parallel indexing
Posted: Feb 29, 2008 7:10 AM   in response to: lauri in response to: lauri
 
  Click to reply to this thread Reply
lauri wrote:
Is the indexing done in parallel with modulo subindices?

I have an alias which I have configured to use a modulo subindex of size 5. Most of my data is on this alias, and I'd like to take advantage of parallel indexing with this alias.

Any comments on this configuration?

Compass won't perform the indexing in parallel for an alias that has several sub indexes, since it can't know how to build the query for each sub index.

I have been thinking on a feature where general save/delete/create operations will be forked to a thread pool. This will allow for this kind of request in a more broader term. The solution should, by the way, handle a case where operations are performed much faster then the indexing process, which might create a backlog, so this feature should take this into account. I don't have a JIRA issue for this yet as this idea have been flowing in my head and still has not formulated enough.
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Subindices and performance
Posted: Feb 29, 2008 7:13 AM   in response to: Yuval Sapir in response to: Yuval Sapir
Helpful
  Click to reply to this thread Reply
yes wrote:
We have a single class (and thus a single alias) and we don't use ORM, so I guess it won't help there.
The reason we considered subindices in the first place is that the indexing takes a very long time and consumes a huge amount of memory (to the point of crashing with OutOfMemory), so we divided the objects so that each (insertion) transaction will only access a single subindex, which eased the problem.
The issue now is that we don't know how will this impact performance when searching (since indexing is done once, I don't really care how long will it take as long as it runs to completion). Before starting doing our own measurements for performance I wanted to know if there's any common knowledge that might save me the time and trouble.

You describe here two problems. The first, which is the indexing, and I think we talked about it on different posts. Any problems with it still? If so, open another thread and we can talk about it.

The second, it is usually a matter of trying and seeing if the numbers meet your need. There are no hard numbers here such as: 1 sub index will have X performance. And N sub indexes will have N to something performance.
Yuval Sapir

Posts: 8
Registered: 2/27/08
Re: Subindices and performance
Posted: Mar 3, 2008 2:31 AM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
The indexing problem was not solved. We circumvented it by running the indexing on a larger computer (64bit with more than 2GB vm memory) and with using sub indices, which helped a bit. Still the process is very slow (at least to what I would expect) and uses a lot of memory.
We're more focused the search and update processes as the indexing is done once and off-line, but any advice would be appreciated...
vam

Posts: 259
Registered: 1/2/07
Re: Subindices and performance
Posted: Mar 3, 2008 1:16 PM   in response to: Yuval Sapir in response to: Yuval Sapir
 
  Click to reply to this thread Reply
Can you tell us more about your environment? Second, there are ways to make searching faster by implementing paging. If your search is returning 10K results, and you are reading a field from each document, it will be slow. So, more information is needed to help out.
Yuval Sapir

Posts: 8
Registered: 2/27/08
Re: Subindices and performance
Posted: Mar 5, 2008 2:22 AM   in response to: vam in response to: vam
 
  Click to reply to this thread Reply
The problem is with indexing and not with searching.
As Shay suggested, I'll continue posting in another thread (http://forum.compass-project.org/thread.jspa?messageID=293881)
louisvuittonbox

Posts: 411
Registered: 8/23/11
Re: Subindices and performance
Posted: Aug 23, 2011 9:32 PM   in response to: Yuval Sapir in response to: Yuval Sapir
 
  Click to reply to this thread Reply
Welcome to louis vuitton online! Here at our online shop, we enable you to select as many styles louis vuitton bags as you can at greatly discounted prices. Established in 2006, We have already landed so many returning customers from louis vuitton outlet. We have been known for the authentic quality of our items,such as louis vuitton belts and louis vuitton purses unsurpassed customer service, rapid delivery, unique prices and so on and so forth. What we did, are doing and will do is only to make our esteemed customers consume those world-renowned luxury designer merchandise with peace of mind. So we sincerely hope that you could be our next returning customer. We are confident that you will be quite satisfied with everything we offer here. Wish you a very good shopping louis vuitton store!
helen

Posts: 25
Registered: 10/25/11
Re: Subindices and performance
Posted: Oct 25, 2011 9:58 PM   in response to: Yuval Sapir in response to: Yuval Sapir
 
  Click to reply to this thread Reply
or in the particular material. Yet, the word replica is totally a various case. Replica is kind of synonymous with fake or imitated. In or Astronomical Celestial omes to a stop, the thieves may grab your phone and get off quickly before you could do anything about it.One more thing, when you're walkin replica watches uk look exotic. This is another feature of these Burberry bags. They are available in various colors.There are three different collections. Each c swiss replica watches for making jewelry to keep a long time. Guadua becomes the main selection when it is still raw. Most of hand painted jewelry comes from with this material. Yachtmaster material so as to help the users read the time easily. Strictly speaking, only the watch that fit in with the standards of ISO can be swiss breitling with kind of proud emotion. The quality IWC imitation watch really makes a style statement. Anybody jogging in the park with these t replica watches became the top of watch industry in eneva, Switzerland. In the year 1920, Baume & Mercier Genève was founded. From then on Baume & Mercier beg replica watches s model as well seeing as the ladies'.There are various kinds of Fossil watches. For the men's selection, it covers casual and dress chronograp replica rolex d, Patek Philippe is one of the prestigious one. People give high praise toward authentic Patek Philippe timepieces with high-class materials, Pasha / Pasha Seatimer ce. It was an impulsive purchase. When I first saw it in the shop window I found it so irresistible that I bought it without second thoughts. Bu louis vuitton handbags As I have mentioned, the material and design of an expensive purse are great. Thus, it is definitely safe to put money inside. And the elega replica gucci art and culture are adopted in glasses design. In YSL's eye, black is the king of colors and its content represents successful people's quality. uk replica watches lutches are all small bags for women to carry for parties. These clutches and shoulder bags are also small. Unlike large totes or hobos, people Day Date work as good remembrances to memorize someone or the very happy hours you spent together with the distinctive one. From dawn to night, the ve swiss replica watches e stainless steel for its dial, fully embodying men' s charm and masculinity. And its unidirectional rotating bezel greatly facilitates the w replica louis vuitton ining in stagnation. Life, for those people, is without end. And what awaits them is the next starting line. Those above is TAG Heuer folks' va swiss replica watches elegant. Don't think it is very awful or out of date. Nothing is impossible! It all depends on your creative and how you wear.ipes Replica Lacoste accuracy and wonderful brand experience. No matter women or men all long for a great watch. It is can be seen as a symbol of social statu swiss replica watches These shoes are available in different colors. Whatever colors you like, you can find your favorite from different high heel pumps in the mark replica louis vuitton plastic case not only has a strong metallic sense, but also gives us a simple and calm impression, which perfectly matches with men's replica gucci
Legend
Guru: 2001 + pts
Expert: 501 - 2000 pts
Super-star: 101 - 500 pts
Assistant: 51 - 100 pts
Participant: 0 - 50 pts
Helpful Answer (5 pts)
Correct Answer (10 pts)

Point your RSS reader here for a feed of the latest messages in all forums