Wayne Graham's Blog

Open Source Facebook Application

Tue, 05 Aug 2008 09:50:00 -0500

Over the last year our library has used its Facebook application to let students see where their friends are in the library and expose library resources to users without needing to go to another website. It's been used pretty heavily and there are a lot of folks in the library world who have asked to get the source. This summer Phil spent a couple of weeks rewriting the code to be a little easier to maintain. We reimplemented the code with Smarty and reimplemented the locational device to use FBJS instead of regenerating an image when someone clicked on the map (it's much faster now). If you need a starter app, check it out at http://code.google.com/p/facebook-athenaeum/. If you need a good book on writing Facebook Applications, check out my book, Facebook API Developers Guide.

Rotten Luck

Tue, 17 Jun 2008 06:23:00 -0500

I'm currently at a conference at Lehigh University, and it's given me pause to really reconsider air travel. Every time I travel, seems something goes awry. Coming back from a conference in California about a month ago, the last leg of my flight was canceled because the plane needed a new tire, and all of Charlotte didn't have a tire for that particular aircraft. I could swallow that since the aircraft wasn't safe to fly, but the line to actually reschedule my flight took about 2 hours, and they couldn't get me back to Newport News (where my car was) until almost midnight the next day. Fortunately, my wife came to get me in Norfolk. I was significantly peeved to not choose US Airways for my next flight to Vegas that I was going on in the next couple of weeks. Coming back from Las Vegas (on United), the company had issued some "cost saving measures" that meant that the pilots couldn't have the engines on before we boarded on the flight from Dulles to Richmond. It was hot that day, so they turned the fans on for us, which drained the auxiliary power, which meant the engines couldn't be started. Well, they brought some field units on, but after the first three didn't work, we disembarked the aircraft to wait for maintenance to bring a fourth unit from another unit to get the aircraft started. In my latest trip (unfortunately back on US Airways), I got to Lehigh yesterday afternoon, but my bags only made it as far as Boston. I was first told my bags would be on the 4:00pm flight, but when I called at 6:30 to check, they told me they'd be arriving at 7:00pm. I called at 9:00 to check, and they told me my bag would be delivered by midnight last night. Since I wasn't woken up by my phone last night, I called again this morning. I was informed that the delivery service began at 8:00am, and they had a 4 - 6 hour window for delivery. I'm thinking that the next time I need to travel, Amtrak is looking a lot better...

Thumpers!

Wed, 21 May 2008 09:37:00 -0500

Just received two of Sun's 24TB Sun Fire x4500 Thumper boxes! The plan (at least right now) is to run SMB and NFS shares so the folks in our media studios can actually put their info on something a bit more stable than the local macs. We have a triple-speed network switch between the Macs and our server room (though they I haven't seen any computers here that can actually handle data that fast), but it should be pretty zippy. We'll also be able to consolidate the storage (hopefully) of our institutional repository and staff file storage. We're also getting a 10Gig switch put in this December so we can surf Internet2 at super speed (or at least the folks working in the Integrated Graphics Lab will be able to send their jobs to the super computers faster)!

Computer Graphics

Thu, 17 Apr 2008 20:27:00 -0500

As I mentioned in my last post, I've been silent for a while. I've had a major project in deploying some institutional repository software (DSpace) to collect the research that students here at William and Mary are doing in a systematic way. Generally this is deployed with more than one person...but you do what you need to do to finish a project ;) The other major time consuming project has been my idea that taking computer graphics would be fun. Just a short footnote about me, I'm a former historian who got into computers during graduate school. Anyway, the College hired John Van Rosendale, who among other things, funded NVidia to develop those wonderful drivers for Linux when he was at the NSF. I was pretty geeked about the class, and talked with John before the semester started. He said that while I didn't have the math background (his idea of "basic math" is advanced calculus and linear algebra) or the programming background (he wanted us to use C/C++), that I would just have to learn that stuff. Our first programming assignment (after about a month of linear algebra and vector math review) was to create a wireframe barn in OpenGL. This was the first time I had worked with C/C++ (most of my "real" programming has been in Java), so developing the Vector and Matrix classes to do all the perspective transformations took a little longer than I had expected. After that, generating views using OpenGL line primitives was a snap. All was good until I got to the second part of the assignment...a non-recursive ray tracer that implemented Gouraud and Phong lighting models on a pyramid. After several weeks of messing with the code and getting no where, I wanted to drop the course. John wouldn't let me, and got me through line-triangle intersections using barycentric coordinates. It didn't look like much at the time, really just a square, but it was a great (though very frustrating) introduction to ray-object intersections, coordinate systems, barycentric coordinate calculations, and lighting models. Immediately after this, our first project was to implement from scratch, a recursive ray tracer that handled reflection, refraction, specular hightlights, and shading for a scene of spheres on a checkerboard floor. I got the specular highlights and shadows pretty quickly, but had a hard time with refraction (well, I guess I also had a problem with RGB because I was going from 0 to 255 instead of 0 to 1, but that was just a bug). When all was said and done, I learned a lot in this course and realized how brilliant folks like Ed Catmull (Pixar) are when they were inventing this stuff back in the 1970s! Anyway, here are a couple of samples of the images I produced using my ray tracing engine. Each has has reflective and refractive surfaces with a maximum recursion depth of 5 with a cylinder texture-mapped with a sky image. Each image took just over a minute to render on a dual quad-core Xeon @ 3.2 GHz.

Bit Torrents

Thu, 17 Apr 2008 16:14:00 -0500

I've been offline for a while (and there are a flurry of posts about that coming up), but I had a very strange thing happen yesterday. Earlier this year, I published the Facebook API Developer's Guide. It seems to be selling reasonably well on Amazon...even broke the top 100 computer books for a couple of days. Being it's a book on Facebook, I've had a few folks add me as a friend and shoot me messages and ask some questions. It all seemed pretty cool, until yesterday. I got a note from a person in Tunisia who added me as a friend and asked a question. I sent a response thanking him for purchasing my book, and this is where I was a little taken aback. In fact, this was his exact response: well, to be honest with you i got your book from a torrent site, i would have bought it if i wasnt living in Tunisia, I hope this won't affect anything :) I didn't quite know what to say. He was nice enough to send the link to the torrent site, and apparently there are 41 seeders and 3 leechers. At first, I was thinking of trying to get the site to remove the link, but I've been around long enough to know that doesn't really stop anything. I have say that I'm still scratching my head on this one. I'm not sure if I'm flattered that there are people that think its worth stealing, offended that someone would under the pretense of purchasing my intellectual property ask a question (could have just asked, I'm a pretty nice guy), or mad that he stiffed me a 20% commission on a book. Still trying to figure this one out...

Facebook Developer API

Tue, 18 Mar 2008 08:59:00 -0500

It's been a long while since I posted anything here. Things have been a bit hectic with the launching of a new institutional repository service (using DSpace), taking a computer graphics course (linear algebra, ray tracing, scientific visualization, etc.), and finishing up my book on the Facebook API. This is the first book I've written and I have to admit it was far more of a task than I had first thought it would be. Not that the subject matter was very dense, but there were at least four major changes in the API that required an almost total rewrite of the code base for the examples. There were even a couple of sections that had to get cut because Facebook "fixed" their code because of security and user concerns (plus a lot of folks "abuse" some of the things you could do with the API). There are some great Coldfusion resources for building Facebook applications in ColdFusion. The "un-official" ColdFusion client library for Facebook apps is at nearpersonal. There is also a RIAForge starter project for FBML (FBML is a tag based language for Facebook) named Facebook FBML Starter Kit. So, if you're wanting a short book that highlights some of the more common elements of developing Facebook applications, be sure to check out my book!

SolColdfusion Update

Wed, 31 Oct 2007 13:01:00 -0500

I've been knocked out by a really bad cold the last couple of weeks and I'm just starting to get things back to normal. I did want to send a quick post on the status of the SolColdfusion project. After seeing Ray's Seeker project, it reminded me that I hadn't set up a project at RIAForge yet, so I took care of that last night. You can now access the official project at http://solcoldfusion.riaforge.org/. Since the project site is up-and-running, I also submitted it to the Solr project SOLR-404 (I know it's a coincidence that it's number 404, but it makes me wonder if it's some type of bad omen ;) ...). Anyway, if you think the client should get included in the project, be sure to vote for it! I've also been working on a generic interface, much like Erik Hatcher's Solr Flare. It's been a bit slow coming as I've not had a lot of time to work on these projects, but hopefully things will settle down shortly so I can devote a bit more time to them.

Solr Schema

Fri, 05 Oct 2007 13:27:00 -0500

If you've ever worked on a project that involved Coldfusion's bundled version of Verity, you've no doubt run into the issue of trying to confine your fields into the structure that Verity imposes, and those custom fields are really precious in these instances. About 6 months ago, I ran into an issue with a search project where I had about 125,000 documents to index. Since we also wanted to be able to use the indexes for some other projects, I was a bit nervous to commit almost the entire allotment of indexable objects to one collection. This launched me into writing a custom search engine and indexer using Lucene and slapping Coldfusion around the responses to do things that Verity did. However, once the projects were complete, I never really got around to making it easy to use. It does cool stuff like search across multiple collections, context highlighting, relevancy calculations, term vector calculations, "did you mean", etc. Essentially everything I think all good search engines need to be able to do. Something this system lacked was an easy way to define the fields that you wanted indexed (along with a knowledge of Java to actually make the changes). The ability to create any number of fields to index in different ways (along with faceting) is a real strong point of Solr. Not only can you add fields and choose how that data is analyzed, you can create your own field types that process the information in your index the way you want them to be. This is done in the $SOLR_HOME/config/schema.xml file. The first section (<types>) defines the types of fields that you will be using, and how Solr should process them with Lucene. If you look at some of the fieldtypes, you'll get an idea of what's possible. For instance, the fieldtype for "string" is an untokenized field that doesn't normalize the fields and sorts missing information last.

However, if you need a more robust fieldtype, look at the fieldtype for "text". This uses a whitespace tokenizer (splits words with whitespace) and uses the stopwords defined in the stopwords.txt file. It does some other processing (filters words, converts them to lowercase, runs a porter stemmer, and then removes duplicates). This fieldtype also defines what to do when a query is passed to it (uses the same filters). This is slightly different than the defined "textTight" which does not perform any further analysis on the text when being queried. You'll probably find that most of these work for most instances, but if you need to, you can build your own fieldtype that has very specific indexing and query filters. The next section contains the actual fields you want to use in the aptly named "fields" element. This is where you actually define the fields that will be in your index, the type of analysis to perform on the field, if it should be indexed, stored, have term vectors, or be multivalued (have multiple instances of the same field in the index). Let's say you're wanting to develop an indexing schema for books (hey, I work in a library). At a very basic level, you'd want a field for an id, title, author, reviews, and a set of topics (or tags). Your fields element would contain something along the lines of:

You'll notice that I have a couple of extra fields for title, author, and topic, these are for the faceting info and are just untokenized fields to make the calculations for facets a little more efficient. Now, we're almost done with creating the schema. We just need to declare a unique key, default search field, and default search operator.


id
title

Remember when I made the fields with the "Str" suffix? We can use a really cool feature of Solr called a "copyField" that literally copies the information from one field to another.

It's worth mentioning here that Solr indexes are not databases! While there are some similarities in the way that Solr allows you to add, update, select, store, and delete information from the index, Solr isn't an RDBMS. I've seen a few discussions where there is some confusion as to why Solr can't do the equivalent of a stored procedure, or some other function of a database. Now, your index server is ready to receive documents to search against. The server, in with the above example as the schema, will expect information to be in the following format:



	1
	Solr Rocks!
	Barr, Foo
	This book rocks!
	This book is horrible!
	information retrieval systems
	xml
	search
	apache foundation

Next week when I get some time, I'll write about creating facet queries...

Solr and Coldfusion -- Setting Up

Fri, 05 Oct 2007 12:22:00 -0500

To get up and running with Solr, you'll need some type of Servlet container. Typically when folks start talking about servlet containers, they're talking about Tomcat or Jetty. In fact, Solr comes with Jetty 6.1.3 (they haven't upgraded to 6.1.5 yet in the distribution). You may also hear about Resin, but in my experience, it runs a bit slower than Jetty and Tomcat. As a small note, servlet containers are different than J2EE application servers like JRun, Geronimo, GlassFish, and JBoss (which use servlet containers like Tomcat and Jetty, but also have EJB containers and can handle other types of logic). If you have a J2EE application server running, you can easily use Solr, and if not, consider using Jetty or Tomcat as your container server. Since your environment can be as varied as there are IT departments, I won't try to cover everything. Essentially you need to have at least the Java 1.5 JRE. However, I would strongly suggest the most current Java JDK (and not the JRE) as it has performance enhancements to run in server mode (with -server). If you don't already have this Java version installed on your server (assuming this is the same server running CF), don't worry, ColdFusion will still work if you install the required Java runtime. Essentially the process for deploying Solr, once you have a servlet container up-and-running is to drop the solr.war file into the webapps directory on the server. It won't do anything at this point as you need to set the configuration files for Solr. The easiest way to do this is copy the files from example/solr into a new directory (which I will refer to now as solr_home). You can tell Java about the home directory by setting the solr.solr.home (-Dsolr.sol.home), set the JNDI lookup ("java:comp/env/solr/home"), or just throw it into the JVM's working directory (the default path is ./solr). Now you just need to make sure everything is running. Just point your browser to http://<server>:<port>/solr/admin. You should then see the administration interface (you may need to restart your servlet container to get everything working properly), but it's not an administrative interface like you get in CFAdmin. This is more of an informational administration panel. You can make sure everything is running, that there are documents in your index is set up properly, check out the schema and configuration files, and thread information. Really the only thing you can administer here is the log level. For some more specific notes on intalling Solr in Tomcat and Jetty, check out Solr's wiki. In particular, if you're going to need multiple instances of Solr to run, pay attention to the sections on Multipe Solr apps on those wiki pages.

Coldfusion Solr Client - SolColdfusion

Thu, 04 Oct 2007 15:19:00 -0500

As I hinted at yesterday, I was close to having some code in the pipeline to abstract using Solr. I've finished the initial code with the following built in. Here's a brief setup guide to start playing with the code. First, you're going to need to grab the latest release version of Solr (currently 1.2). The only real requirement to run this software is that you have a JRE of 1.5 or higher. Untar/zip the file somewhere convenient and open a command prompt. Get to the example directory in the apache-solr.1.2.x folder (cd /example). To start up the sample server running Jetty, just issue the following command:


java -jar start.jar

This will start a new instance of the Solr server on your computer on port 8983. You can make sure this is running by navigating to http://localhost:8983/solr (NOTE: this is a link to your computer. If you get an error, it's because your computer isn't running an instance of Solr on port 8983). At this point, it's probably good to send you over to the Solr website to take a look at their tutorial. Go ahead. I'll wait... ... Great, you're back. You've seen some basic inserting, deleting, and querying of Solr index data. You may have also noticed that there are clients for PHP, Ruby, Python, and Java...no ColdFusion. I want to do a little more testing on this before I submit the patch, but I've added the initial code as an encosure here to do updating, deleting, and searching in Coldfusion. The CFC SolColdfusion should be in the path org/apache/client (at least that's where I'm putting in for the purposes of this initial demonstration). The initialization takes one required parameter (the Solr host) and then has two optional parameters (port and path). To set this up, create an instance with

Now, there are a lot of different parameters you can send to Solr to perform different queries. And, since some of these key names can repeat, I chose to implement sending these parameters as an array. So, let's set this up.

These parameters are basically what are the defaults that Solr will return back to you. If you want highlighting, you would need to add two additional row vectors with 'hl = on' and 'hl.fl = '. Searching is straight forward, taking a query, the start row, number of rows to return, and the array of parameters:

This searches all fields and all content and returns back an XML document with the search results in it.

In the result node, you'll see that Solr returns an xmlAttribute of numFound of 0 (assuming you don't have anything in the index). Let's add an example document from the documents that come with Solr.





  F8V7067-APL-KIT
  Belkin Mobile Power Cord for iPod w/ Dock
  Belkin
  electronics
  connector
  car power adapter, white
  4
  19.95
  1
  false

You'll notice I used a commit and optmize statement. Neither of these statements are necessary every time you add a document, but be aware that Solr caches documents and won't flush the new documents to disk unless you either commit the documents or the mergefactor setting you used in your solrconfig.xml file has been reached. Now, let's delete this document...

Don't forget to commit deletions to the index! There'll be more soon (add multiple documents, delete by queries). In the mean time, try it out. If you have any comments, questions, concerns, whatever, let me know.

ColdFusion and Solr

Wed, 03 Oct 2007 15:38:00 -0500

I've spent the last few months working on some projects that didn't really have anything to do with ColdFusion (lots of Java and PHP). One of the projects I've been working with (Vufind.org) uses Solr as it's indexing/search engine. That's starting to get picked up by some pretty big companies (Netflix just relaunched their search using Solr this week). I've been working with Solr in Java for a bit now, and I wanted to start to build an interface for using it as a search engine (my Lucene code is stuck in open source limbo) in Coldfusion. One of the cool things about Solr is that it returns results back through HTTP (in XML, JSON, or ruby). As soon as I get the code finished, I'll post it as a patch in Solr.

JVM Settings

Fri, 31 Aug 2007 08:24:00 -0500

I ran across this great compilation of JVM settings for the different versions of Java. Very useful for Java tuning!

Facebook API

Thu, 14 Jun 2007 10:41:00 -0500

I've been kind of silent here as I've been trying to keep this blog at least somewhat focused on ColdFusion and I've been doing development in some different languages as of late (Java, PHP, and Ruby). However, when Facebook released their API for developers, my summer assistant Phil started hacking away at it and we've released our first app for Facebook: SwemTools. It does a couple of things, provides a search interface for both the online catalog and our website, news feeds, hours, and a bit of a neat application based on Swem Signal that was created by Tom MacWright. We're still working on the code for the Signals part, but the idea is that on a map of the library, you'd be able to see where your friends are. Right now, you can see a dot where your friends are, but not who they are...but we're working on a solution for it. Anyway, if you're on Facebook, check it out...you can add me as a friend too.

Modern Takes on Classic Art

Mon, 11 Jun 2007 15:09:00 -0500

I'm a big fan of Rene Magritte's work (I have La trahison des images hanging in my office). I ran across this shirt, and I'm sure that Magritte would appreciate the update to his work.

Fun with PDFs

Thu, 17 May 2007 17:12:00 -0500

I've been working with a lot of PDF files lately for a few different projects (see The FlatHat and Card Catalog). With our special collections cards, when you got a result back, Acrobat viewer would blow up the image to around 600%, making for a rather ugly image. For the FlatHat, I really wanted to be able to open a PDF and have the search terms highlighted, so I started hunting for ways to actually do this. I've been using PDFBox to extract text from our PDFs to index with Lucene, so I started there and they clued me in to Adobe's PDF Open Parameters. This really killed a few birds with one stone. When I was working on the Flat Hat newspaper, I was originally only returning back the page that the search result was on. I had some misgivings about this (like what if the story was on more than one page), but being able to pass the search query from the engine into the PDF is really nice since the user doesn't have to search through the entire issue to find the the context they are searching for (e.g. whistle bait -- when I saw that term, I cracked up; definitely a different era). Basically, the PDF Open Parameters allow you to pass commands into a PDF to allow you to control how the PDF is opened. They're passed with a "#" after the filename (e.g. filename.pdf#zoom=100). You can string commands together with an ampersand (&) with a few caveats:

only one digit after a decimal is retained
parameters and their values can only be 32 total characters long
you can't use reserved characters (=, #, and &) to escape special characters
if you turn bookmarks off for a PDF that had bookmarks showing, they won't go away until the PDF has been rendered

Anyway, here are some examples of what you can do: