Encyclopaedia Britannica 11th Edition page

Source version (plain text in subversion) can be found at:

http://knowledgeforge.net/shakespeare/svn/trunk/shksprdata/ancillary/britannica-11th.txt

• Wiki page: (sign up here) http://www.okfn.org/wiki/MiniEvents
• When: Saturday 26th of April. Start at 1400 and run until ~ 1900
• How long: Whatever time you can spare. Be it an hour or the whole afternoon.
• How to join in: log in to the irc channel, announce yourself, and then just crack on with one of the work items (see below)
  • irc channel: #okfn on irc.oftc.net
• What: plan and work on Open Shakespeare / Milton
  • trac: http://knowledgeforge.net/shakespeare/trac/roadmap
  • Current tickets: http://knowledgeforge.net/shakespeare/trac/report/1
  • For those not inclined to coding there’s plenty else to do. In particular we need to finish off proof editing Britannica entry, see http://okfn.org/wiki/tmp/BritannicaShakespeare

http://knowledgeforge.net/shakespeare/svn/trunk/shksprdata/ancillary/britannica-11th.txt

What we now need to do is ‘proof’ this to correct the OCR errors. This kind of think is perfect for distributed volunteers so if you’d like to help out just step up and starting correcting with one of the sections. To make it especially easy for people to make edits the text has in a temporary location on the Open Knowledge Foundation wiki (only the first five pages for the time being):

http://okfn.org/wiki/tmp/BritannicaShakespeare

http://p.knowledgeforge.net/shakespeare/trac/ticket/24

We’ve now written code to grab the relevant tiffs off wikimedia:

http://p.knowledgeforge.net/shakespeare/svn/trunk/src/shakespeare/src/eb.py

You can also find them online (28 pages) starting at:

http://upload.wikimedia.org/wikipedia/commons/scans/EB1911_tiff/VOL24%20SAINTE-CLAIRE%20DEVILLE-SHUTTLE/ED4A800.TIF

Next step is to then OCR this stuff (after that we can move on to proofing whether by ourselves or via http://pgdp.net). When we first had a stab at this back in April we tried using gocr. Unfortunately the results were so bad that they were unusable. Recently an old ocr engine of HP’s has been released as open source under the name of tesseract:

http://code.google.com/p/tesseract-ocr/

We’re going to have a go using this — though if there is anyone out there with access to an alternative system we’d love to hear about it.

http://www.openshakespeare.org/code/

Changelog

• Annotation of texts (js-based in browser) (ticket:20, ticket:21) (http://www.openshakespeare.org/2007/04/10/annotation-is-working/)
• Switch to unicode for internal string handling (resolves ticket:23: some texts breaking the viewer)
• Add functional tests for the web interface (ticket:11)
• Substantial improvements to speed of concordance (ticket:22) (http://www.openshakespeare.org/2007/01/03/improvements-to-the-concordance/)
• Switch to genshi templates from kid
• Switch to plain WSGI from cherrypy

Outstanding Issues

• Annotation cannot handle long texts because of javascript performance issues

About Open Shakespeare

A full open set of Shakespeare’s works along with anciallary material, a variety of tools and a python API.

For more information see the about page:

http://www.openshakespeare.org/about/

Get involved: http://www.openshakespeare.org/participate/

Mailing list: http://lists.okfn.org/mailman/listinfo/okfn-discuss/

To see the system in action visit:

http://demo.openshakespeare.org/view?name=phoenix_and_the_turtle_gut&format=annotate

Quite a bit of effort has been made to decouple the annotation system from Open Shakespeare so that it can be easily reused elsewhere. You can find the code for the annotation system (nicknamed annotater) here:

http://p.knowledgeforge.net/shakespeare/svn/annotater/trunk/

There are still some substantial issues with the Open Shakespeare implementation the most obvious of which are:

a) large texts bring the javascript to its knees ((The Phoenix and the Turtle is the shortest of Shakespeare’s works which is why I’m using it).

b) security/user authentication for annotation adding/editing/deleting

But the basic system is working.

The main work in porting annotater was twofold:

1. To create and independent annotation store web application which reproduced the restful web interface needed by the marginalia javascript (we’ve also improved this by giving it a normal human-usable CRUD web interface in addition to the restful one)
2. Plugging this together (aka debugging/hacking around) with the existing marginalia javascript (for example the paste-based WSGI store web app just would not process posts sent using x-www-form-urlencoded!)

Annotater is now fully functioning and we can entirely reproduce the basic demo in the original marginalia though with the major difference that our version has a proper store backend so all creation/deletion updates of annotations get persisted to a real db and aren’t just in memory (to try this out just start the demo wsgi app via $ python annotater.py).

The next step after this is to integrate annotater into open shakespeare along with doing any polishing up of the package that is needed to achieve this.

Relatedly and r:74 is a first step on filtering the concordance — in this case to exclude roman numerals and various non-words. Doing this made me think about whether the concordance should be storing actual words or just stems — for example, it does not seem to make much sense to have different entries for kill, kills, killed etc. Using a stemming algorithm such as the porter stemmer (which I notice has a nice python implementation directly available) we can easily stem each word as we go along. This would have several benefits one of the most prominent being a dramatic reduction in the basic dictionary size (i.e. the number of distinct words in the concordance).

1. stet: javascript annotation system used for gpl v3 comments system

   • http://en.wikipedia.org/wiki/Stet_(software)
   • Bit of a hack at present and did not seem designed for external reuse (when I last looked the README was fairly emphatic that this was very alpha with little documentation)

2. commentary: javascript based wsgi middleware developed by ian bicking

   • http://pythonpaste.org/commentary/
   • Rather hacked together (apparently he coded it in a week). Had problems getting it working locally and no documentation to help in adaptation. Seems to be unmaintained (demo site is currently down) which is perhaps not surprising given how many other projects Ian has on the go.
   • One nice feature is that you don’t seem to have to mess with the underlying web pages you want to add comments to (this only works if you are sitting on top of another wsgi application)

3. marginalia: javascript library and spec for adding web annotation to pages

   • http://www.geof.net/code/annotation/
   • javascript code seems well factored and understandable and docs are good

4. annotea: W3C project based on RDF

   • http://www.w3.org/2001/Annotea/
   • Been around a long time and now seems to be inactive
   • Server and client support rather lacking. No simple interface based on, e.g., javascript — you have to write a special client yourself — which is a major drawback
   • That said the protocol is well-documented and so writing a client (or a server) shouldn’t be that hard (other than having to mess around with rdf in javascript …)
   • The Schema seems reasonable
   • xpointer based which according to the marginalia site is a problem