SourceForge.net Logo
Main Overview Wiki Issues Forum Build Fisheye

Welcome, Guest
Guest Settings
Help

Compass Support Forums » Compass » Compass Developers

Thread: Using Compass & JPA on Google App Engine with a few minor tweaks


Permlink Replies: 38 - Pages: 3 [ 1 2 3 | Next ] - Last Post: Dec 3, 2011 3:47 AM Last Post By: woolrich
Richy

Posts: 6
Registered: 7/23/09
Using Compass & JPA on Google App Engine with a few minor tweaks
Posted: Jul 23, 2009 2:18 AM
  Click to reply to this thread Reply
Hi, I thought I'd let you know that I managed to get Compass (2.3.0-beta1) indexing just fine on GAE using JPA persistence. It did involve a bit more configuration and tinkering than the JDO approach so I'll outline this here in the hope that it may be helpful to someone.

Firstly, for some reason, the GAE JPA implementation requires the use of the fully qualified class name when querying and aliases must be specified. It needs a query string like this in order to work - "select x from com.thingy.whatever.EntityName x"

By default the Compass JPA won't create this as a query when looking for entities to index, so I made the following changes to the Compass source.

1. org/compass/gps/device/jpa/entities/DatanucleusJpaEntitiesLocator.java

65c65
FROM : EntityInformation entityInformation = new EntityInformation(classMapping.getClazz(), entityname, new DefaultJpaQueryProvider("select x from " + classMapping.getClazz().getName()), classMapping.getSubIndexHash().getSubIndexes());
---
TO : EntityInformation entityInformation = new EntityInformation(classMapping.getClazz(), entityname, new DefaultJpaQueryProvider("select x from " + classMapping.getClazz().getName() + " x"), classMapping.getSubIndexHash().getSubIndexes());

2. org/compass/gps/device/jpa/entities/EntityInformation.java

50c50
< FROM : this.name = name;
---
TO : this.name = clazz.getName();

The other change I made was to overwrite the contents of org/compass/gps/device/support/parallel/ConcurrentParallelIndexExecutor.java with the contents of the SameThreadParallelIndexExecutor.java class (well the performIndex method at least). I'm not actually sure if this step is necessary, but it seemed to not pay any attention to me setting the parallel index executor on the device. I'd love to do more testing, but the goal was to get this working quickly.

Anyway, my EMF configuration is as follows.

private static final Compass compass;

private static final CompassGps compassGps;

static {

compass = new CompassConfiguration()
.setConnection("gae://index")
.setSetting(
CompassEnvironment.ExecutorManager.EXECUTOR_MANAGER_TYPE,
"disabled").addScan("com.wherever.entity")
.registerConverter("keyConverter",
com.google.appengine.api.datastore.Key.class,
(Converter) (new KeyConverter())).buildCompass();

compassGps = new SingleCompassGps(compass);

CompassGpsDevice device = new JpaGpsDevice("appengine", emfInstance);

((JpaGpsDevice) device)
.setParallelIndexExecutor(new SameThreadParallelIndexExecutor()); //although this didn't seem to make any difference

compassGps.addGpsDevice(device);

if(compass.getSearchEngineIndexManager().isLocked()){
compass.getSearchEngineIndexManager().releaseLocks();
}

compassGps.start();

//compassGps.index(); - don't index every time the app starts up

}

As our app is using datastore Keys for the Id's of entities I also needed a key converter to allow compass to convert back and forth between keys and strings. Fortunately GAE provides a factory method to do this.

public class KeyConverter extends AbstractBasicConverter<Key> {

@Override
protected Key doFromString(String keyString, ResourcePropertyMapping arg1,
MarshallingContext arg2) throws ConversionException {
return KeyFactory.stringToKey(keyString);
}

@Override
protected String doToString(Key key, ResourcePropertyMapping resourcePropertyMapping, MarshallingContext context){
return KeyFactory.keyToString(key);
}
}

And that's it. I have it running at the moment, indexing thousands of entities and I have to say I'm very impressed. Will definitley be looking at Compass more in the future and I hope this info is of help to somebody.

Oh, the only downside to this approach is that there's no lifecycle listening going on. If you change an entity you will have to update the index yourself, in our case this is fine but YMMV.

Cheers,

Richy
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Using Compass & JPA on Google App Engine with a few minor tweaks
Posted: Jul 24, 2009 12:48 PM   in response to: Richy in response to: Richy
  Click to reply to this thread Reply
Great stuff. I actually started to try and add support for datanucleus as a JPA provider (which I see you started to use). Two things:

1. You started that setting the parallel executor does not work. This is very strange, it should work.
2. Not sure I followed the Entity related changes ... .

Cheers,
Shay
Richy

Posts: 6
Registered: 7/23/09
Re: Using Compass & JPA on Google App Engine with a few minor tweaks
Posted: Jul 27, 2009 3:52 AM   in response to: Shay Banon in response to: Shay Banon
  Click to reply to this thread Reply
Hi Shay,

In answer to your questions :

1. You started that setting the parallel executor does not work. This is very strange, it should work.

Yes, it might, but I'm pretty sure when I tried it I was still getting security access exceptions as the parallel thread executor was being fired. However, this may have been due to a classpath issue that I've since resolved. I'll try and roll back that change and let you know what happens when I get a spare moment.

2. Not sure I followed the Entity related changes ... .

Basically, I just needed to change a few things to ensure that the queries issued against the datastore were of the format "select x from com.fully.qualified.EntityName x"

This just involved using the class.getName() rather than simply the name and appending the trailing 'x' to the query.
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Using Compass & JPA on Google App Engine with a few minor tweaks
Posted: Jul 27, 2009 11:19 AM   in response to: Richy in response to: Richy
  Click to reply to this thread Reply
I see. I need to think how to incorporate this into Compass. Can you open a jira issue for this?
Richy

Posts: 6
Registered: 7/23/09
Re: Using Compass & JPA on Google App Engine with a few minor tweaks
Posted: Jul 27, 2009 11:41 AM   in response to: Shay Banon in response to: Shay Banon
  Click to reply to this thread Reply
Done : http://issues.compass-project.org/browse/ CMP-886

I'll get back to you when I've tested the same thread executor again (may be a few days). In the ticket I mentioned that it must be used, you're probably right however that setting the executor on the device should be sufficient.

Cheers,

Richy
Len T

Posts: 23
Registered: 7/25/09
Re: Using Compass & JPA on Google App Engine with a few minor tweaks
Posted: Jul 27, 2009 6:15 PM   in response to: Richy in response to: Richy
  Click to reply to this thread Reply
Richy, thanks very much for posting the tweaks. It was very helpful since I was running into the same problem.

I am trying to get the mirroing (updating index on modifications) going by using AbstractCompassJpaEntityListener and @PostPersist, @PostRemove and @PostUpdate methods. The mirroring is triggered on modification but I ran into the problem with using entities in more than one entity group in a transaction. Shay had suggested trying to set the transaction parameter on datastore methods (e.g. put) to null in another thread (Link: http://forum.compass-project.org/thread.jspa?messageID=298064&#298064) so I tried doing this (in the GoogleAppEngineDirectory class) but this did not have any effect. It would be great if mirroring could work with transactions.

Len
dn

Posts: 1
Registered: 7/28/09
Re: Using Compass & JPA on Google App Engine with a few minor tweaks
Posted: Jul 28, 2009 3:20 AM   in response to: Richy in response to: Richy
  Click to reply to this thread Reply
the GAE JPA implementation requires the use of the fully qualified class name

No it doesn't. That was fixed in DataNucleus about 4 months ago, and the most recent GAE/J comes with DN1.1.4, which supports the "entity name".
Richy

Posts: 6
Registered: 7/23/09
Re: Using Compass & JPA on Google App Engine with a few minor tweaks
Posted: Jul 28, 2009 11:35 AM   in response to: dn in response to: dn
  Click to reply to this thread Reply
Cool! Well I'll try to update to the latest version of GAE SDK and perhaps try with the compass 2.3.0-beta1 jar unmodified then, if entity name works now then it may just be a question of making sure you use the SameThreadParallelIndexExecutor when setting up the GPS device.

That would be neat.
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Using Compass & JPA on Google App Engine with a few minor tweaks
Posted: Jul 28, 2009 1:39 PM   in response to: Richy in response to: Richy
  Click to reply to this thread Reply
Thanks for the feedback, if someone can test that it would be of great help.
Richy

Posts: 6
Registered: 7/23/09
Re: Using Compass & JPA on Google App Engine with a few minor tweaks
Posted: Jul 29, 2009 8:45 AM   in response to: Shay Banon in response to: Shay Banon
  Click to reply to this thread Reply
Just tried with the latest Google Java SDK which contains DN 1.1.4 and last nights nightly compass build (CMPTRK-NIGHTLY-541).

I get the following error :

SEVERE: {appengine}: Failed to index the database
javax.persistence.PersistenceException: Class name x could not be resolved
at org.datanucleus.jpa.NucleusJPAHelper.getJPAExceptionForNucleusException(NucleusJPAHelper.java:264)

Which would imply that there's still something still not quite right with the queries. My guess would be the query in DatanucleusJpaEntitiesLocator still needs the 'x' appending to the select, as outlined above.

In the meantime, the changes I outlined above do still work against these versions of the google sdk and datanucleus, so I'll continue with those for now. As I'm doing this for a work project I don't get as much time to play as I'd like, but I'll see if just applying the first change to (DatanucleusJpaEntitiesLocator.java) solves this issue when I get a chance.
Shay Banon

Posts: 4,028
Registered: 9/6/05
Re: Using Compass & JPA on Google App Engine with a few minor tweaks
Posted: Jul 29, 2009 12:34 PM   in response to: Richy in response to: Richy
  Click to reply to this thread Reply
I have added the trailing x, and just kicked a nightly build. Can you give it a go?
Richy

Posts: 6
Registered: 7/23/09
Re: Using Compass & JPA on Google App Engine with a few minor tweaks
Posted: Jul 30, 2009 2:22 AM   in response to: Shay Banon in response to: Shay Banon
  Click to reply to this thread Reply
Yep, that seems to have solved the problem :) Nice work Shay. I'm running NIGHTLY 542 without any modifications against a JPA/GAE application and haven't noticed any problems yet (indexing and searching seem fine).

I'll try and spend some time playing with the lifecycle listeners as well, as they'd be a nice to have. But in this state Comass is entirely usable on GAE apps using JPA.
Len T

Posts: 23
Registered: 7/25/09
Re: Using Compass & JPA on Google App Engine with a few minor tweaks
Posted: Dec 9, 2009 4:03 PM   in response to: Len T in response to: Len T
  Click to reply to this thread Reply
I am trying to get the mirroing (updating index on modifications) going by using AbstractCompassJpaEntityListener and @PostPersist, @PostRemove and @PostUpdate methods. The mirroring is triggered on modification but I ran into the problem with using entities in more than one entity group in a transaction. Shay had suggested trying to set the transaction parameter on datastore methods (e.g. put) to null in another thread (Link: http://forum.compass-project.org/thread.jspa?messageID=298064&#298064) so I tried doing this (in the GoogleAppEngineDirectory class) but this did not have any effect. It would be great if mirroring could work with transactions.

I believe that putting the transaction parameter to null in datastore methods (get and put) to null should work. When I tried it previously, I only changed the GoogleAppEngineDirectory class but there are also other classes which call these datastore methods which need changing. Setting the parameter to null does mean that it is not run within a transaction.

Len
infoatdfx

Posts: 3
Registered: 1/3/10
Re: Using Compass & JPA on Google App Engine with a few minor tweaks
Posted: Jan 3, 2010 9:17 AM   in response to: Richy in response to: Richy
  Click to reply to this thread Reply
When I do an indexing at startup with "compassGps.index()", all my objects get indexed. But when I create not objects and make them persistent, none of them get indexed. Does anyone has the same problem?
cometta

Posts: 9
Registered: 7/6/06
Re: Using Compass & JPA on Google App Engine with a few minor tweaks
Posted: Jan 21, 2010 8:15 PM   in response to: Richy in response to: Richy
  Click to reply to this thread Reply
hi, may i know does anyone having issue with javax.naming.namingexception not found when use with spring ?

http://groups.google.com/group/google-appengine-java/browse_thread/thread/ca5deb75bfb1642b/d67ae97de9f9bb69?lnk=gst&q=+javax.naming.NamingException#d67ae97de9f9bb69
Legend
Guru: 2001 + pts
Expert: 501 - 2000 pts
Super-star: 101 - 500 pts
Assistant: 51 - 100 pts
Participant: 0 - 50 pts
Helpful Answer (5 pts)
Correct Answer (10 pts)

Point your RSS reader here for a feed of the latest messages in all forums