SourceForge.net Logo
Main Overview Wiki Issues Forum Build Fisheye

Welcome, Guest
Guest Settings
Help

Compass Support Forums » Compass » Compass Users

Thread: Scalability Questions

This question is not answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 13 - Pages: 1 - Last Post: Oct 5, 2011 10:43 PM Last Post By: factoryoutlet
Rodrigo Oliveira

Posts: 5
Registered: 10/24/08
Scalability Questions
Posted: Oct 24, 2008 8:29 PM
 
  Click to reply to this thread Reply
I have the following concerns:

1) Will compass + hibernate + JdbcDirectory work on 2 tomcat servers running the same application using single db server? For example, if server A and B try to make update to the index, which is using Lucene's JdbcDirectory, will there be a problem?

2) Has anybody used compass to index a database with millions of records. I had to write my own indexer because the ones that come with compass were running out of memory. Actually, I think I found a bug with one of them which I can share with whoever is interested. Actually, I testing the one I wrote as I write this and it looks like it will work (first million done, one more to go).

3) Does anybody know if JdbcDirectory will perform better or worse then a regular file system directory? Specifically, if you have a multiple server environment like we do where the index would be queried over the network.
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Scalability Questions
Posted: Oct 25, 2008 10:58 AM   in response to: Rodrigo Oliveira in response to: Rodrigo Oliveira
 
  Click to reply to this thread Reply
Hi,

1. Yes, that is the idea of the Jdbc directory.
2. Can you share the bug you found? I would be happy to get it into Compass if there is a problem.
3. JdbcDirectory will be slower than file system directory. If you can go with shared file system, it is probably better.
Valera

Posts: 9
Registered: 5/6/08
Re: Scalability Questions
Posted: Oct 26, 2008 1:12 AM   in response to: Rodrigo Oliveira in response to: Rodrigo Oliveira
 
  Click to reply to this thread Reply
Concerning your second question.
In my current project we have a requirement to index/search dozens millions of records (each record consists of 5-12 properties-fields, in terms of Lucene). I prepared a small prototype, Compass, RSEM based. No database, all data we store in Lucene indexes. The results were pretty promising, untill we reached 2-5 millions of entries in the index (of course, the number depends on your domain model, hardware, etc.). After that search operations were getting slower and slower - up to 5-10 seconds, what's not acceptable in our case. Profiling the prototype, I found that the obvious bottleneck of the application was disk input-output. The Lucene index was about 1 GB in size. Attempt to utilize index hashing didn't improve performance.
Our next steps will be:
  • reduce the index size (not all properties are searchable, storable)
  • try to use JdbcDirectory, instead of FSDirectory
  • Compass + Terracota (actually already tried and the results are very good)
  • may be Apache SOLR with built-in cluster support may fix the issue????

May somebody share some thoughts, expirience on the scalability issue?

Thanks in advance,
Valera
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Scalability Questions
Posted: Oct 26, 2008 10:58 AM   in response to: Valera in response to: Valera
 
  Click to reply to this thread Reply
Have you made sure the index was optimized when it got big? Search operations (especially on file systems) work much faster with optimized index. There is an optimizer running in the background, but if it never managed to optimizer the index (it needs to obtain a lock in order to perform the optimization), then it might explain the problem.

In your original first point you noted that you found a bug, but I did not understand what it is.
Valera

Posts: 9
Registered: 5/6/08
Re: Scalability Questions
Posted: Oct 26, 2008 11:55 AM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
Yes, I configured optimizer so that it merged different segments of the index. The optimization improved "simple" searches (searches without sorting). It still takes several secs to execute some queries containing sorting on certain properties ( I guess Lucene updates index when such queries executed first time).

It was another post concerning a possible bug in Compass - not mine.
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Scalability Questions
Posted: Oct 26, 2008 12:19 PM   in response to: Valera in response to: Valera
 
  Click to reply to this thread Reply
I see. Note that when using sorting, the first sort might take some time to perform since lucene field cache needs to be initialized.
Rodrigo Oliveira

Posts: 5
Registered: 10/24/08
Re: Scalability Questions
Posted: Oct 27, 2008 1:22 PM   in response to: Shay Banon in response to: Shay Banon
 
  Click to reply to this thread Reply
Regarding the possible bug in PaginationHibernateIndexEntitiesIndexer:

Criteria criteria = entityInfo.getQueryProvider().createCriteria(hibernateSession, entityInfo);
if (criteria != null) {
values = criteria.list();
} else {
Query query = entityInfo.getQueryProvider().createQuery(hibernateSession, entityInfo).setFirstResult(current).setMaxResults(fetchCount);
values = query.list();
}

If the criteria is not null no pagination is applied which causes it to run out of memory on a large table.

Edited by: Rodrigo Oliveira on Oct 27, 2008 1:27 PM

Rodrigo Oliveira

Posts: 5
Registered: 10/24/08
Re: Scalability Questions
Posted: Oct 27, 2008 1:42 PM   in response to: Rodrigo Oliveira in response to: Rodrigo Oliveira
 
  Click to reply to this thread Reply
I am still trying to index a table that has 2 million rows. Now that i fixed the supposed bug mentioned above I now longer run out of memory. However when it gets to the last commit or when it tries to "wrap it up" I get a communication broken error from the database. I can see the it indexed all, or at least up to the last page of records to a temporary table but it did not go as far as moving it to the regular index table.

I am running the index again right and it takes hours. I expect to get the error again and I will added it here if I do.

Just FYI here is what I already tried with no success:

  • Default scrollable indexer: Out Of Memory
  • Pagination indexer: Out Of Memory
  • Modified Pagination Indexer: Communication error with db on "last step".

I am allocating 1024Mb like so: -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=256m

The other thing I noticed is that the process gets slower for each page as it moves forward.

Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Scalability Questions
Posted: Oct 30, 2008 5:39 AM   in response to: Rodrigo Oliveira in response to: Rodrigo Oliveira
 
  Click to reply to this thread Reply
artilheiro wrote:
Regarding the possible bug in PaginationHibernateIndexEntitiesIndexer:

Criteria criteria = entityInfo.getQueryProvider().createCriteria(hibernateSession, entityInfo);
if (criteria != null) {
values = criteria.list();
} else {
Query query = entityInfo.getQueryProvider().createQuery(hibernateSession, entityInfo).setFirstResult(current).setMaxResults(fetchCount);
values = query.list();
}

If the criteria is not null no pagination is applied which causes it to run out of memory on a large table.

Edited by: Rodrigo Oliveira on Oct 27, 2008 1:27 PM


Yes, this is a bug... . I opened CMP-762 and fixed it. Out of curiosity, why don't you use the default indexer which is the scrollable one?
john

Posts: 1
Registered: 1/14/09
Re: Scalability Questions
Posted: Jan 14, 2009 1:05 PM   in response to: Rodrigo Oliveira in response to: Rodrigo Oliveira
 
  Click to reply to this thread Reply
Hi folks,

Anyone have any idea what is the upper limit on the size of the index using jdbcDirectory? I am using mysql and according to the mysql doc that the blob size is 2 gig but practical size is a few hundred megs. I am try to build an index with jdbcDirectory with 175000 documents with 5 fields in each document. Is jdbcDirectory capable of this?

Thanks in advance.
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Scalability Questions
Posted: Jan 14, 2009 2:26 PM   in response to: john in response to: john
 
  Click to reply to this thread Reply
Its really up to the maximum size of the blob the database can store.
louisvuittonbox

Posts: 411
Registered: 8/23/11
Re: Scalability Questions
Posted: Aug 23, 2011 9:19 PM   in response to: Rodrigo Oliveira in response to: Rodrigo Oliveira
 
  Click to reply to this thread Reply
Welcome to visit our louis vuitton outlet store for more series of Louis Vuitton Products. You will find the most complete Louis Vuitton handbags such as Monogram wallets, Louis Vuitton Speedy 30, Damier Canvas. Here, You could choose whatever you want in our louis vuitton store! Louis Vuitton Handbags are one of most popular designer handbags brands throughout the world! our louis vuitton outlet are earnest to offer you the authentic louis vuitton travel which is made of the best material and with exquisite designs. Our coupon 5% is prepare for our dear customers if your order value exceed $80.And you can enjoy the same feeling and weight to the luxury louis vuitton as original one. You can surely get your desired cheap louis vuitton shoes at the lower price here. More Louis Vuitton on sale just in our louis vuitton online!
factoryoutlet

Posts: 523
Registered: 10/5/11
coach factory outlet
Posted: Oct 5, 2011 10:36 PM   in response to: Rodrigo Oliveira in response to: Rodrigo Oliveira
 
  Click to reply to this thread Reply

coach factory outlet offer excellent, elegant coach products in order to generate room for your new collections.They're purchased more quickly that you may at any time imagine.coach factory store is the greatest destination to buy what type of bags you like. If you need to manage a new Coach carrier and most preferable strategy is to the coach regarding coupon codes.Here's your new way to look sophisticated styles. Carry the hot coach factory outlet store of this summer and be the true trendsetter!in the market you definitely can find various colorways that are designed in as well as the high quality that applied in. For most of you would like to come. So just come to our coach factory outlet online store to choose one.

coach factory outlet is really sizzling kinds of shopping way for you. With the usage of the replica designer coach bags, you can surely be able to change your individual looks in a stunning manner.You know, Coach items are so perfect and fascinating. Now I grow up, and find coach outlet on the Internet offering affordable products with reliable quality.Many fashionable women match with practical Coach Purses which will make the street shopping become relaxed,and make every person can enjoy more diversiform combination in coach factory outlet online.The coach factory outlet store are utilized for formal event in company area that will display your excellent taste.Not only does a briefcase make is easier to carry things.

Almost everyone has identified effectively about this, below, among the best to make sure you will get much more information about coach outlet.In the coach outlet online shop, a collection of high quality and hottest Coach Shoulder Bags is offered to you. They are all at the best prices you will ever find.

You know, Coach items are so perfect and fascinating. Now I grow up, and find coach outlet on the Internet offering affordable products with reliable quality.Once visit, you will be unable to restrain loving it. Warmly welcome all of you to the Summer Sales Promotion on coach factory outlet.coach factory outlet has new coach laptop bags in listing. No matter which style do you want, we can provide it for you.Coach bags enjoy high popularity throughout the world. I would like to share the coach factory outlet online with you. What are you waiting for? Just come to visit.

coach factory outlet supplies the newest and most fashionable brand Coach on sale.The more Coach shoulder bags you order, the more discount you will get.It's wonderful to find out it does not slip in your signature at the bottom of the case and the appearance of brass accents from coach outlet online, Because women who can look stylish this summer.coach outlet has a zippered closure and buckles for extra security. Shiny brass hardware, rounded leather handle, and an interior pocket. It also includes a limited edition Hawaii luggage tag and lock.You can find all kinds of coach bags whatever you chose which one in coach factory outlet online will never be out of fashion. Not only they have many nice and design styles, but also made from the finest leather and fabric.

With the great diversity of styles, patterns and sizes available nowadays, finding the right pieces to highlight your personality is quite important. coach outlet offers exactly what you want.coach factory outlet supplies the newest and most fashionable brand Coach on sale.The more Coach shoulder bags you order, the more discount you will get.Look at this golden Coach bag in the? coach factory outlet online.Those new bags with classic and fashionable design are more and more popular now!After graduating from university, my cousin came back to my hometown from USA last month. She told me that coach outlet store is very popular among her classmate.

coach outlet online own a couple of line, color, or the selected theme, it should be a good choice. It may be the place where offer the bag ladies handbag, you will be able to follow a course of education or training receives for the slim project chest strap wallet.coach outlet will send you a coupon in the post to use in their upon only after you type a achieve. The Coach bags are customarily somewhat many than the ones in the stores.In particular, products from coach factory outlet with leather design are fashionable, handmade,leading the wave of American pop.It with simple,durable and unique style to win consumers.coach factory not only supply with the highest quality crystal at reasonable price but also provide good after-sales service. Companies operating purposes: science and technology in order to survive, credibility and development, strict quality control.

coach outlet handbags's special design and fashion is certainly and so eye-catching, I hope you are able to get much more attention towards the Cheap Coach Crossbody.As long as you open our coach factory outlet webpage, you will view various Coach New Arrivals which are the most popular also the most fashionable in this year.Are you anxious to choose some of the brand bags? Are you searching for reliable sellers? When you come to the coach factory outlet online, these are not problems.

At the coach outlet online you have the largest selection of the day. If you touch the item and like it, keep it in your possession until you make your final decision.Over the years, coach factory online has added a multitude of new handbag shapes, styles and materials to their collection. However, the highest care is taken that every Coach handbags is both aesthetically beautiful and functional.Are you searching for a cool and stylish coach bag? The coach factory outlet online Online is your first priority.

factoryoutlet

Posts: 523
Registered: 10/5/11
louis vuitton outlet
Posted: Oct 5, 2011 10:43 PM   in response to: Rodrigo Oliveira in response to: Rodrigo Oliveira
 
  Click to reply to this thread Reply

louis vuitton outlet leads you to bags collection of wide, such as LV bags and LV handbag etc. They are main name in luxury and style. It is excellent quality, fine Italian technology and extraordinary beauty.LV is universally renowned as a luxury brand. louis vuitton online store offers customers the newest fashion collections which can make you different.

In the past, Louis Vuitton only emphasis on classic, but in recent year, Louis Vuitton change its strategy and style to acquire new breakthrough. Especially louis vuitton bags like Louis Vuitton Vernis Wallet.Want to become more charming and faddish? Go to visit louis vuitton outlet webpage and select the most suitable products for yourself.

Want to become more charming and faddish? Go to visit louis vuitton outlet webpage and select the most suitable products for yourself.louis vuitton outlet online may often be the consideration alternative inside of your event to get these kinds of bags.Is without a doubt louis vuitton online seriously is present? Is it possible to offer an outlet designed for Louis Vuitton merchandise speaks out?

The choices are likely to be basically countless seeing that louis vuitton outlet occurs with the help of completely new and also incredible concepts once in a while.Mainly because mentioned before, you could buy in the louis vuitton online, via the producer's the web outlet. Owning worn it in past times. They can supply a appropriate evaluate for the property's simplicity as well as client service description.louis vuitton outlet online is the method to show the confidence of our handbags. The person who show himself in front of other people is a confident one.

LV bags in the louis vuitton outlet are a feminine update on an urban classic. With plenty of room for all of your essentials this covetable carryall will take you through the weekend in style.LV is universally renowned as a luxury brand. louis vuitton online store offers customers the newest fashion collections which can make you different.

This collection focus more on color selection compared with other LV handbags. Cute and sunshine design style increases its charm. louis vuitton outlet online provides you the very famous luxury worldwide brand.We are glad you come to join in our louis vuitton outlet online store, because the winter is upcoming now, Louis Vuitton released many new handbags to feedback our customers.Just come to visit once the louis vuitton online, you will fall in love with its fancy bags, purses, sunglasses, shoes and so on.

Legend
Guru: 2001 + pts
Expert: 501 - 2000 pts
Super-star: 101 - 500 pts
Assistant: 51 - 100 pts
Participant: 0 - 50 pts
Helpful Answer (5 pts)
Correct Answer (10 pts)

Point your RSS reader here for a feed of the latest messages in all forums