Posts filed under 'Technical'
One of next things we want to do for open shakespeare is provide an open
introduction for to his works. The obvious idea for this was to use the
Shakespeare entry in the 11th ed of the Encyclopaedia Britannica as
detailed in this ticket:
http://p.knowledgeforge.net/shakespeare/trac/ticket/24
We’ve now written code to grab the relevant tiffs off wikimedia:
http://p.knowledgeforge.net/shakespeare/svn/trunk/src/shakespeare/src/eb.py
You can also find them online (28 pages) starting at:
http://upload.wikimedia.org/wikipedia/commons/scans/EB1911_tiff/VOL24%20SAINTE-CLAIRE%20DEVILLE-SHUTTLE/ED4A800.TIF
Next step is to then OCR this stuff (after that we can move on to
proofing whether by ourselves or via http://pgdp.net). When we first had
a stab at this back in April we tried using gocr. Unfortunately the
results were so bad that they were unusable. Recently an old ocr engine
of HP’s has been released as open source under the name of tesseract:
http://code.google.com/p/tesseract-ocr/
We’re going to have a go using this — though if there is anyone out there with access to an alternative system we’d love to hear about it.
August 14th, 2007
After another push over the last few days I’ve got the web annotation system for Open Shakespeare operational (we’ve been hacking on this on and off since back in December).
To see the system in action visit:
http://demo.openshakespeare.org/view?name=phoenix_and_the_turtle_gut&format=annotate
Quite a bit of effort has been made to decouple the annotation system from Open Shakespeare so that it can be easily reused elsewhere. You can find the code for the annotation system (nicknamed annotater) here:
http://p.knowledgeforge.net/shakespeare/svn/annotater/trunk/
There are still some substantial issues with the Open Shakespeare implementation the most obvious of which are:
a) large texts bring the javascript to its knees ((The Phoenix and the Turtle is the shortest of Shakespeare’s works which is why I’m using it).
b) security/user authentication for annotation adding/editing/deleting
But the basic system is working.
April 10th, 2007
One of the main items scheduled for v0.4 of open shakespeare is improvements to the responsiveness of the concordance. Using the v0.3 codebase, using just the sonnets as test material, loading up the list of words for the concordance alone took around 24s on my laptop. This is because even with a single text there are already over 18,000 items in the concordance and we were having to read through all of these to generate the list of words. Some recent commits (e.g. r:72) have gone some way to improving this responsiveness (loading word list is now 3s now compared to 24s) but the result is not entirely satisfactory (printing full statistics is 13s compared to 40s previously). One obvious way to go futher is to use caching — either of individual web pages or of particular key parts such as all the distinct words occurring in the concordance (caching works because the concordance only changes when new texts are added which will usually only happen once — when the system is first initialised).
Relatedly and r:74 is a first step on filtering the concordance — in this case to exclude roman numerals and various non-words. Doing this made me think about whether the concordance should be storing actual words or just stems — for example, it does not seem to make much sense to have different entries for kill, kills, killed etc. Using a stemming algorithm such as the porter stemmer (which I notice has a nice python implementation directly available) we can easily stem each word as we go along. This would have several benefits one of the most prominent being a dramatic reduction in the basic dictionary size (i.e. the number of distinct words in the concordance).
January 3rd, 2007
We intend to add annotation/commentarysupport to the open shakespeare web demo either in this release or next. As a first step we’ve been looking to see what (open-source) web-based annotation systems are already out there. Below is our list of what we’ve been able to find so far (if you know of more please post a comment). After examining several of these in some detail the one we’re going to try our properly is marginalia (if you’re interested our current efforts to do this including writing a python wsgi annotation service backend can be found here in the subversion repository).
stet: javascript annotation system used for gpl v3 comments system
commentary: javascript based wsgi middleware developed by ian bicking
- http://pythonpaste.org/commentary/
- Rather hacked together (apparently he coded it in a week). Had problems getting it working locally and no documentation to help in adaptation. Seems to be unmaintained (demo site is currently down) which is perhaps not surprising given how many other projects Ian has on the go.
- One nice feature is that you don’t seem to have to mess with the underlying web pages you want to add comments to (this only works if you are sitting on top of another wsgi application)
marginalia: javascript library and spec for adding web annotation to pages
annotea: W3C project based on RDF
- http://www.w3.org/2001/Annotea/
- Been around a long time and now seems to be inactive
- Server and client support rather lacking. No simple interface based on, e.g., javascript — you have to write a special client yourself — which is a major drawback
- That said the protocol is well-documented and so writing a client (or a server) shouldn’t be that hard (other than having to mess around with rdf in javascript …)
- The Schema seems reasonable
- xpointer based which according to the marginalia site is a problem
December 18th, 2006