SourceForge.net Logo
Main Overview Wiki Issues Forum Build Fisheye

Welcome, Guest
Guest Settings
Help

Compass Support Forums » Compass » Compass Users

Thread: Compass/Lucene and unicode - advice needed

This question is not answered. Helpful answers available: 1. Correct answers available: 1.


Permlink Replies: 10 - Pages: 1 - Last Post: Jan 9, 2012 9:55 PM Last Post By: miyao lin
obender

Posts: 24
Registered: 6/12/09
Compass/Lucene and unicode - advice needed
Posted: Jun 12, 2009 3:55 PM
 
  Click to reply to this thread Reply
I'm new to Compass/Lucene so forgive me if this question was asked before (search returns no results).

I have a database with records in the same table in many different languages (up to 70) it includes all W-European, Arabic, Eastern, CJK, Cyrillic, etc. you name it.
I've looked at what people say about Lucene and it looks like for the most part standard analyzers should do fine with most Unicode languages but there are quite a few exceptions.
Here is some recently updated Lucene Jira thread:
https://issues.apache.org/jira/browse/LUCENE-1488

My question is, what would be the safest bet for me in terms of analyzers/tokenizers?
Do I really have to write my own ones for the bunch of languages that are not supported?
Did anyone already solve the problem similar to mine? I'm sure someone already did :)

And yes, I looked at the Lucene sandbox analyzers. It just adds more confusion. For example why there analyzers for DE and FR? Wouldn't the standard analyzer (which is Unicode complaint as I understood) deal with EU languages just fine?

Thanks in advance for advices :)

Edited by: obender on Jun 12, 2009 3:56 PM
Robert Muir

Posts: 2
Registered: 6/12/09
Re: Compass/Lucene and unicode - advice needed
Posted: Jun 13, 2009 12:04 AM   in response to: obender in response to: obender
 
  Click to reply to this thread Reply
Hi,

The biggest problem you will have is that although lucene supports UCS (Unicode Character Set), it does not support the other parts of Unicode such as text-segmentation (breaking text into words).

For multilingual text, no analyzer or configuration currently available in a release is going to fit the bill for you, but if you want, I can try to recommend a configuration based upon components that are already committed that will work best for you (not LUCENE-1488, its not ready). If you send a note to java-user I will try to help you out, based on the languages you are trying to deal with.

What you want, in my opinion, is just the simple Unicode defaults. I'm extremely biased about all of this though, as I am the one working on LUCENE-1488!
obender

Posts: 24
Registered: 6/12/09
Re: Compass/Lucene and unicode - advice needed
Posted: Jun 15, 2009 8:09 AM   in response to: Robert Muir in response to: Robert Muir
 
  Click to reply to this thread Reply
Sorry, I'm not sure what java-user (group/list?) did you mean, I would definitely like to get some advice on my issue so just let me know where to post my question :)
As for the nature of the text, the structure is such that there will be always one language per record but multiple languages per table. As I mentioned before the number of languages is quite large including some exotic ones.
Robert Muir

Posts: 2
Registered: 6/12/09
Re: Compass/Lucene and unicode - advice needed
Posted: Jun 15, 2009 11:42 AM   in response to: obender in response to: obender
Helpful
  Click to reply to this thread Reply
Sorry, I neglected to provide a link: http://lucene.apache.org/java/docs/mailinglists.html#Java%20User%20List

there are a number of options depending upon the set of languages involved, so there you will get the best feedback I think.
obender

Posts: 24
Registered: 6/12/09
Re: Compass/Lucene and unicode - advice needed
Posted: Jun 15, 2009 12:11 PM   in response to: Robert Muir in response to: Robert Muir
 
  Click to reply to this thread Reply
Thanks! Re-posted my question there.
louisvuittonbox

Posts: 411
Registered: 8/23/11
Re: Compass/Lucene and unicode - advice needed
Posted: Aug 23, 2011 9:41 PM   in response to: obender in response to: obender
 
  Click to reply to this thread Reply
Producing fabulous high end fashion year after year and season after season, gucci outlet is arguably the world's most prestigious designer of clothes. As you would expect from a gucci outlet online that started out solely making leather goods, they are best known for their classic range of gucci bags and gucci handbags which usually feature the trademark company logo or pattern and come either in leather or canvas finishes which look perfect when coupled with matching gucci shoes. Ladies Gucci Shoes are simply the perfect footwear for any occasion and here at Profile we stock a large selection including Gucci Boots, Flats and High Heels. They are designer shoes of the highest quality.
factoryoutlet

Posts: 523
Registered: 10/5/11
coach outlet
Posted: Oct 5, 2011 9:15 PM   in response to: obender in response to: obender
 
  Click to reply to this thread Reply

coach factory outlet offer excellent, elegant coach products in order to generate room for your new collections.They're purchased more quickly that you may at any time imagine.coach factory store is the greatest destination to buy what type of bags you like. If you need to manage a new Coach carrier and most preferable strategy is to the coach regarding coupon codes.Here's your new way to look sophisticated styles. Carry the hot coach factory outlet store of this summer and be the true trendsetter!in the market you definitely can find various colorways that are designed in as well as the high quality that applied in. For most of you would like to come. So just come to our coach factory outlet online store to choose one.

coach factory outlet is really sizzling kinds of shopping way for you. With the usage of the replica designer coach bags, you can surely be able to change your individual looks in a stunning manner.You know, Coach items are so perfect and fascinating. Now I grow up, and find coach outlet on the Internet offering affordable products with reliable quality.Many fashionable women match with practical Coach Purses which will make the street shopping become relaxed,and make every person can enjoy more diversiform combination in coach factory outlet online.The coach factory outlet store are utilized for formal event in company area that will display your excellent taste.Not only does a briefcase make is easier to carry things.

Almost everyone has identified effectively about this, below, among the best to make sure you will get much more information about coach outlet.In the coach outlet online shop, a collection of high quality and hottest Coach Shoulder Bags is offered to you. They are all at the best prices you will ever find.

You know, Coach items are so perfect and fascinating. Now I grow up, and find coach outlet on the Internet offering affordable products with reliable quality.Once visit, you will be unable to restrain loving it. Warmly welcome all of you to the Summer Sales Promotion on coach factory outlet.coach factory outlet has new coach laptop bags in listing. No matter which style do you want, we can provide it for you.Coach bags enjoy high popularity throughout the world. I would like to share the coach factory outlet online with you. What are you waiting for? Just come to visit.

coach factory outlet supplies the newest and most fashionable brand Coach on sale.The more Coach shoulder bags you order, the more discount you will get.It's wonderful to find out it does not slip in your signature at the bottom of the case and the appearance of brass accents from coach outlet online, Because women who can look stylish this summer.coach outlet has a zippered closure and buckles for extra security. Shiny brass hardware, rounded leather handle, and an interior pocket. It also includes a limited edition Hawaii luggage tag and lock.You can find all kinds of coach bags whatever you chose which one in coach factory outlet online will never be out of fashion. Not only they have many nice and design styles, but also made from the finest leather and fabric.

With the great diversity of styles, patterns and sizes available nowadays, finding the right pieces to highlight your personality is quite important. coach outlet offers exactly what you want.coach factory outlet supplies the newest and most fashionable brand Coach on sale.The more Coach shoulder bags you order, the more discount you will get.Look at this golden Coach bag in the? coach factory outlet online.Those new bags with classic and fashionable design are more and more popular now!After graduating from university, my cousin came back to my hometown from USA last month. She told me that coach outlet store is very popular among her classmate.

coach outlet online own a couple of line, color, or the selected theme, it should be a good choice. It may be the place where offer the bag ladies handbag, you will be able to follow a course of education or training receives for the slim project chest strap wallet.coach outlet will send you a coupon in the post to use in their upon only after you type a achieve. The Coach bags are customarily somewhat many than the ones in the stores.In particular, products from coach factory outlet with leather design are fashionable, handmade,leading the wave of American pop.It with simple,durable and unique style to win consumers.coach factory not only supply with the highest quality crystal at reasonable price but also provide good after-sales service. Companies operating purposes: science and technology in order to survive, credibility and development, strict quality control.

coach outlet handbags's special design and fashion is certainly and so eye-catching, I hope you are able to get much more attention towards the Cheap Coach Crossbody.As long as you open our coach factory outlet webpage, you will view various Coach New Arrivals which are the most popular also the most fashionable in this year.Are you anxious to choose some of the brand bags? Are you searching for reliable sellers? When you come to the coach factory outlet online, these are not problems.

At the coach outlet online you have the largest selection of the day. If you touch the item and like it, keep it in your possession until you make your final decision.Over the years, coach factory online has added a multitude of new handbag shapes, styles and materials to their collection. However, the highest care is taken that every Coach handbags is both aesthetically beautiful and functional.Are you searching for a cool and stylish coach bag? The coach factory outlet online Online is your first priority.

factoryoutlet

Posts: 523
Registered: 10/5/11
louis vuitton outlet
Posted: Oct 5, 2011 9:22 PM   in response to: obender in response to: obender
 
  Click to reply to this thread Reply

louis vuitton outlet leads you to bags collection of wide, such as LV bags and LV handbag etc. They are main name in luxury and style. It is excellent quality, fine Italian technology and extraordinary beauty.LV is universally renowned as a luxury brand. louis vuitton online store offers customers the newest fashion collections which can make you different.

In the past, Louis Vuitton only emphasis on classic, but in recent year, Louis Vuitton change its strategy and style to acquire new breakthrough. Especially louis vuitton bags like Louis Vuitton Vernis Wallet.Want to become more charming and faddish? Go to visit louis vuitton outlet webpage and select the most suitable products for yourself.

Want to become more charming and faddish? Go to visit louis vuitton outlet webpage and select the most suitable products for yourself.louis vuitton outlet online may often be the consideration alternative inside of your event to get these kinds of bags.Is without a doubt louis vuitton online seriously is present? Is it possible to offer an outlet designed for Louis Vuitton merchandise speaks out?

The choices are likely to be basically countless seeing that louis vuitton outlet occurs with the help of completely new and also incredible concepts once in a while.Mainly because mentioned before, you could buy in the louis vuitton online, via the producer's the web outlet. Owning worn it in past times. They can supply a appropriate evaluate for the property's simplicity as well as client service description.louis vuitton outlet online is the method to show the confidence of our handbags. The person who show himself in front of other people is a confident one.

LV bags in the louis vuitton outlet are a feminine update on an urban classic. With plenty of room for all of your essentials this covetable carryall will take you through the weekend in style.LV is universally renowned as a luxury brand. louis vuitton online store offers customers the newest fashion collections which can make you different.

This collection focus more on color selection compared with other LV handbags. Cute and sunshine design style increases its charm. louis vuitton outlet online provides you the very famous luxury worldwide brand.We are glad you come to join in our louis vuitton outlet online store, because the winter is upcoming now, Louis Vuitton released many new handbags to feedback our customers.Just come to visit once the louis vuitton online, you will fall in love with its fancy bags, purses, sunglasses, shoes and so on.

longchamp

Posts: 272
Registered: 12/1/11
Longchamp
Posted: Dec 3, 2011 4:45 AM   in response to: obender in response to: obender
 
  Click to reply to this thread Reply
longchamp, il se révèle être and also vital de l'histoire des graphic designers français de cuir, bagages et maroquinerie Longchamp strain devenir l'un des meilleurs quartiers de la marque, et devenir united nations symbole de la haute société. Aujourd'hui, la marque your not seulement la perception limitée de Longchamp et de vendre des reports sac longchamp durante cuir haut de gamme et des bagages, mais à s'impliquer dans la style, accessoires, chaussures, sacs, bijoux, montres, des médias, du vin et d'autres régions du géant des indicateurs de tendance.www.sacslongchamp-france.org
anitas

Posts: 95
Registered: 11/20/11
Re: Compass/Lucene and unicode - advice needed
Posted: Jan 9, 2012 9:25 PM   in response to: obender in response to: obender
 
  Click to reply to this thread Reply
I love this article, it's very well.oakley sunglasses saleFake Oakley Flak Jacket Sunglassesreplica Oakley Polarized SunglassesOakley Batwolf sunglasses outletoakley sunglasses wholesaleReplica Oakley Flak Jacket SunglassesOakley Restless SunglassesOakley Dispatch sunglassesreplica oakley sunglassesReplica Oakley Fuel Cell Sunglassesoakley scalpel sunglassesOakley Flak Jacket sunglassesreplica Oakley Commit SQ SunglassesFake Oakley GascanOakley Split Jacket SunglassesOakley Frogskins sunglasses outletoakley fives squared sunglassesFake Oakley Polarized Sunglassesoakley straight jacket sunglassesOakley Gascan sunglassesOakley Half Jacket SunglassesReplica Oakley Polarized SunglassesOakley Ten SunglassesFake Oakley Hijinx sunglassesreplica Oakley Half Wire SunglassesFake Oakley RadarOakley Zero SunglassesOakley Jawbone sunglassesreplica Oakley Half X SunglassesReplica Oakley sunglassesOakley Juliet sunglassesOakley Hijinx SunglassesCheap oakleys Commit SQ SunglassesOakley Jupiter sunglassesOakley Holbrook SunglassesReplica Oakley Fives SquaredOakley M Frame sunglasses outletoakley jawbone sunglassesReplica Oakley Flak JacketOakley Minute sunglassesoakley juliet sunglassesReplica Oakley FrogskinsOakley Monster Dog sunglassesreplica Oakley Jupiter SunglassesReplica Oakley Fuel CellOakley Oil Rig sunglassesOakley Jury SunglassesReplica Oakley GascanOakley Pit Boss sunglassesoakley m frame sunglassesCheap oakleys Half Wire SunglassesOakley Polarized sunglasses outletoakley minute sunglassesCheap oakleys Half X SunglassesOakley Radar sunglassesreplica oakley minute sunglassesCheap oakleys Hijinx SunglassesOakley Restless sunglassesoakley monster dog sunglassesCheap oakleys Jawbone SunglassesOakley Scalpel sunglassesOakley Necessity SunglassesCheap oakleys Oil Rig SunglassesOakley Straight Jacket sunglassesoakley oil rig sunglassesCheap oakleys Pit Boss SunglassesOakley Ten sunglassesOakley Pit Bull SunglassesCheap oakleys Radar SunglassesOakley Zero sunglassesOakley Plaintiff SunglassesCheap oakleys Straight Jacket Sunglasses
miyao lin

Posts: 97
Registered: 11/20/11
Re: Compass/Lucene and unicode - advice needed
Posted: Jan 9, 2012 9:55 PM   in response to: obender in response to: obender
 
  Click to reply to this thread Reply
It really a good article.cheap oakley sunglassesFake Oakley Flak JacketOakley Half X sunglassesoakley eyewearOakley Antix sunglassesoakley sunglassesFake Oakley Fuel CellOakley Holbrook sunglassesoakley sunglasses discountedOakley Batwolf sunglassesOakley Antix SunglassesFake Oakley Fuel Cell Sunglassesoakley batwolf sunglassesOakley Bottlecap sunglassesOakley Bottlecap SunglassesFake Oakley Gascan SunglassesOakley Commit SQ SunglassesOakley Commit SQ sunglassesoakley five squared sunglassesFake Oakley PolarizedOakley Half X SunglassesOakley Dispatch sunglasses outletOakley Jupiter SunglassesOakley Encounter sunglassescheap oakley minute sunglassesCheap oakleys Bottlecap SunglassesOakley Pit Boss SunglassesCheap oakleys Fives Squared Sunglassesreplica Oakley Pit Boss SunglassesOakley Enduring Pace sunglassesOakley Polarized SunglassesOakley Fives Squared sunglassesoakley crosshair sunglassesCheap oakleys Flak Jacket SunglassesOakley Dangerous SunglassesCheap oakleys Frogskins SunglassesOakley Dart SunglassesFake Oakley Flak Jacket sunglassesoakley dispatch sunglassesOakley Forsake sunglassesOakley Encounter SunglassesCheap oakleys Fuel Cell SunglassesOakley Enduring Pace SunglassesCheap oakleys Gascan SunglassesOakley Fast Jacket SunglassesOakley Frogskins sunglassesOakley Fives SunglassesFake Oakley Gascan sunglassesoakley flak jacket sunglassesFake Oakley DispatchOakley Forsake SunglassesFake Oakley Fast JacketOakley Frogskins SunglassesOakley Hijinx sunglassesoakley fuel cell sunglassesOakley M Frame sunglassesoakley gascan sunglassesFake Oakley Five SquaredOakley Half Wire SunglassesFake Oakley FrogskinsFake Oakley Half XFake Oakley Polarized sunglassesFake Oakley HijinxOakley Polarized sunglassesFake Oakley HolbrookFake Oakley Straight Jacket sunglassesFake Oakley JupiterOakley Fuel Cell sunglassesFake Oakley JuryOakley Half Jacket sunglassesFake Oakley Monster DogOakley Half Wire sunglasses
Legend
Guru: 2001 + pts
Expert: 501 - 2000 pts
Super-star: 101 - 500 pts
Assistant: 51 - 100 pts
Participant: 0 - 50 pts
Helpful Answer (5 pts)
Correct Answer (10 pts)

Point your RSS reader here for a feed of the latest messages in all forums