We rarely get to talk about our Lift work, but I can talk about this one because it's been a Sunday afternoon project for a while.  

I live in Google for my contacts and calendar, and wanted a quick summary of upcoming birthdays and anniversaries.  I'm not alone in wanting something like this.  Reminder Thing itches this scratch by digging into my contacts and calendar each week, and emailing me a single reminder of upcoming birthdays and anniversaries.  

That's it.  Obviously, if you live in Facebook, you don't need this, as they already sensibly send out such an email.  But I've not yet, um, fully embraced Facebook.

But this is mostly a tech blog, so here are the details. It's a Scala app, running on the Lift framework.  It uses SendGrid to send email, and uses Google's own AuthSub and GDATA Java APIs to communicate with Google. The UI is Twitter's Bootstrap.

We build using SBT and deploy using SBT Cloudbees Plugin to a single CloudBees 128M cell.  I believe Cloudbees is currently using JDK 1.6, Tomcat 6 and Nginx, but I neither know for sure and nor do I especially care—if it works, it's fast and I doesn't give me hassle, I'm happy. 

BTW, If you're in the Sydney area and want to know more about how we deploy Scala apps, to CloudBees or AWS, go to ScalaSyd on 26 Oct 2011, where Jono is on a panel discussing Scala in production.

I have two tech observations from writing this code:

First, firing off asynchronous events to Google is a delight with Scala and Lift.  The page where we show spinners while fetching data from Google makes use of the Lift LazyLoad CSS tag (which means it becomes concurrent without me making any code changes) and a little bit of Lift Wiring (8 lines) to make the "Continue" button change colour.

The other comment is that Lift parameterised menus really help you be productive: very worth getting to know.  In Reminder Thing the unsubscribe link plus the ability to undo your unsubscribe uses parameterised menus:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

object Unsubscribe {
  
  /* A menu for Lift's Sitemap which maps /unsubscribe/{uuid} to two
     functions: 1) given a uuid, lookup a Subscription instance
     2) given a Subscription instance, create the link address... */
  val menu = Menu.param[Subscription]("Unsubscribed", "Unsubscribed",
    uuid => Subscription.find(By(Subscription.uuid, uuid)),
    s => s.uuid ) / "unsubscribed" >> Hidden
  
  /* The Loc(ation) part of the menu is what we're going to work with... */
  lazy val loc: Loc[Subscription] = menu.toLoc
  
  /* Turn HTML element with id="undo" into a link, by deleting the current record and
    creating a link that when clicked will re-create the subscription in the database... */
  def render = "#undo" #> loc.currentValue.map { s =>
    s.delete_!
    SHtml.link("/resubscribed", () => s.duplicate, Text("undo") )
  } 

  // the duplicate method copies the fields of s into a new DB row.
    
}

So that's Reminder Thing.  There are a couple of tweets on adding birthdays and anniversaries  from @rmndrthng, which is also the right Twitter account to use to let us know if you like the idea, anything that doesn't work, anything we can do better.  

If you're a Google contacts or calendar user, do give it a go.

Permalink | Leave a comment  »

An external module is one way to share lumps of code with the Lift community.  The word "external" indicates that these are distinct from the modules in the core Lift repository maintained by Lift comitters.  But the pattern to create a module is the same, as Peter outlined on the Lift Wiki and in one or two posts on the Lift mailing list.

Anyone can create one, anywhere they like, and people do. So there's nothing to talk about.

Except... I'd like to have modules build automtically when Lift changes, with reports on problems, and have an easy way to find modules and add them to Lift projects.  If you buy into that as something useful, there are lots of ways to achieve it.  

Are we there yet? Are we there yet? Are we...

Today, we're nowhere near, but we've started by experimenting with a build system and a common repository for external modules. 

We want to track Scala and Lift releases, including Lift milestone releases, and publish these builds so they can be available quickly to anyone wanting to use them via Maven or SBT.  That's the plan.  To explore this we're using CloudBees. Under their open source programme we're given a hosted Jenkins instance and repository for build artifacts.  In other words, we go to the CloudBees web site, create a new Jenkins job, point it at the appropriate GitHub repo and press "Build now". Nice and easy, but all manual.

Currently we're building just a few modules, which you can see on the Jenkins dashboard:

•  Google Analytics support
•  IMAP IDLE, providing push-like email into a Lift app
•  JqPlot module for graphs

...but none of it is automatic yet.  We know we can do better.

The developer experience

This is what the current development process is like for getting a new release out.  Let's say, Lift 2.4-M4 becomes available (which it has; this post has been in draft too long).

The module versioning number I'm using is the Lift version with the module version appended.  So the 0.91 version of the IMAP module for Lift 2.4-M3 is: 2.4-M3-0.91. Sometimes, that's all you need to change, making me wonder if this is a wasteful system.  Anway, let's publish a version of 2.4-M4:

$ cd liftmodules-imap-idle $ git checkout -b 2.4-M4 $ vi build.sbt // modify version, liftVersion, test $ git push origin 2.4-M4 $ sbt > +publish

...then merge back to master:

$ git checkout master $ git merge 2.4-M4 $ git push origin master

Where next?

Are we making a meal out of this?  Is this something worth doing?  Are there better ways?  Feedback is what we need, and if you're part of the Lift community, the place for that is on the mailing list please.

Otherwise we'll be investigating: 

• adding triggers or polling to trigger builds of modules. 
• possibly a directory of modules - but that seems like the easiest part now that we've started to push out notifications to the fantastic implicit.ly space.
• scratching our heads about how to track SNAPSHOP.
• can we automate module changes to track Lift milestone releases?
• automatically publishing builds into the repo.

Or something like that.

If you want your module included in this build process, hassle me or Jono on the Lift mailing list.

Permalink | Leave a comment  »

• Improvements for temporary network failure. We now monitor and disconnect if IDLEing for more than 30 minutes, as a way to catch socket errors hidden by IDLE.
• If disconnected from the IMAP server, we now request all waiting messages on a reconnect.
• Switched to SBT 0.10.1.
• Now also publishing source and javadoc into the repository.

This version is available for Lift 2.4-M3 for Scala 2.8.1 and 2.9.0-1.

ImapIdle.init { m: javax.mail.Message => 
  println("You've got mail: "+EmailUtils.dump(m))
  true // delete the email on the server
}

Permalink | Leave a comment  »

Dean Wampler, 2011, Functional Programming for Java Developers, O'Reilly Media. 88 pages.

Summary

This is the book for you if you code in Java, you've heard of functional and want to know what the fuss is about, but you want it in terms of the constructs of Java that you already know.  "This book explains why functional programming has become an important tool for the challenges of our time" and acknowledges that "much of the literature on functional programming is difficult to understand for people who are new to it."  

In 88 pages there's a limit to what can be covered, and sure I have some reservations, but I don't know of a better introduction for Java developers who aren't comfortable looking at other languages.

Details

The first chapter addresses the "why functional?" question, which is answered primarily in terms of bringing clarity to thinking, more reuse and more concise code, and a better fit to "the unique challenges of our time", such as "working with massive data sets and remaining agile".  But you're not expected to jettison what you know, with later chapters suggesting when mixing in OO is a great idea.

The detailed answer to the "why" question talks in terms of concurrency, big data sets, modularity benefits, keeping code minimally sufficient, reining in complexity—making the argument that functional programming helps you in your job in all the areas, and these are areas you need to be able to handle today.

In describing what functioning programming is (chapter 2), you get to see some Java code simplified via the Functional Java project library.  In this chapter, and rolling into chapter 3, types and data structures are introduced, via an implementation of Option (how to avoid null) and List in Java.  This is all, I think, genuinely useful, especially with the "tips" in the text and the exercises at the end of the chapters.

Reservations

Tackling this subject in Java is a big ask, and around half way through the book it starts to show.  When filter, fold and map are introduced I suspect Java programmers drawn to this text may be scratching their heads asking why they would ever want to write code like this.  

A later chapter introduces pseudo code to illustrate concepts in a Java-like syntax.  This isn't going to compile, and you have to ask yourself if the text has drifted too far from Java. The author puts it well when suggesting the next place to go is "writing real code", pointing in the direction of a variety of languages that have anonymous functions and higher-order functions... which is probably any language except Java.

That said, the chapter on concurrency points to the Java Akka implementation of Actors, which looks like something you might want to use straight away.  It's noted that the Actor model "isn't really a functional approach to concurrency", but it's part of the culture.

Conclusion

A concise and personal look at functional programming from the Java point of view, with some practical and compelling points about the benefits on the large (concurrency) and small (bugs in a class) scale.  If you're drawn to this area, you'll going to find this book valuable.  But you know you're only delaying switching to another language ;-)

Permalink | Leave a comment  »

Pretty much the simplest thing you can do to get started with machine learning in a web app. These are my slides from BathCamp AI night.

Permalink | Leave a comment  »

You know what I'm about to say, right?  I am not a lawyer.

But I do have to keep track of changes in the law that impact us and our clients, and one of those is the Privacy and Electronic Communications Regulations.  It's the one about getting consent to set cookies before you set them.  There's a nice short summary by Simon Weinberg.

We regularly use three groups of cookies in our applications: JSESSIONID, ext_id (Lift's "keep me logged in" functionality), and the handful used by Google Analytics.  My impression is that we can reasonably argue that JSESSIONID is an essential cookie for the use of any of our sites.  The others are going to need consent, and at the very least an update to all of our terms of service and privacy policy documents.

All of the above are assumptions I'm making.  If you want an example of how others are tackling this problem, take a look at the Information Commissioner's Office site.

The long answer

Specifically regarding the ext_id under one usage scenario, I did ask what the guidance is:

Hello

I'm trying to understand if "keep me logged in" functionality falls under the new rules.  By that I mean the use of a cookie to enable a registered user of a site to visit the same site (potentially over many weeks or months) without the need to re-log in to gain access to their account.  This is not used for tracking or marketing. Currently we do not ask the user if they want to set this cookie: we are assuming the user is logged in until they explicitly log out.

Do you have a view on this type of cookie use?

Here's the reply in case you're in a similar position, and I though the repy was really well put together and useful:

Thanks you for your recent enquiry regarding the new Privacy and Electronic Communications Regulations 2003.

It is my understanding that no differentiation between cookies on the basis of their duration is made in the Privacy and Electronic Communications Regulations 2003 (PECR) as amended.  Therefore it is our guidance that cookies need to be consented to irrespective of their temporary nature, unless the requirements of the ‘strictly necessary’ exception are fulfilled.

As there is no exception specifically for ‘login’ cookies, whether or not a login cookie is excepted from the basic consent rule will depend on whether that cookie falls within the ‘strictly necessary’ exception.  There is not a straightforward or catch-all answer to whether a ‘login’ cookie will be ‘strictly necessary’. 

What will be crucial in deciding whether a login cookie fits within the narrow exception of being ‘strictly necessary’ is whether the use of that login cookie is essential for a service requested by the user.  It is essential that both parts of the exception are met – that it is a service the user has requested, and that the cookie itself is strictly necessary to that service.  (Our guidance on this point, as you have no doubt already seen, is available on our website: Read the ICO’s advice to organisations about how to prepare for the new rules on cookies). 

In some circumstances it may be that a login cookie falls within the ‘strictly necessary’ exception – in others it will not.  We have not provided more detailed examples regarding login cookies as it really will depend upon an individual website’s specific circumstances as to whether the requirements of the exception are met.  I can see a situation where it might potentially be argued that a login cookie is strictly necessary if a user has asked to login to a site to access certain information and a cookie is integral to that login or authentication process.  Alternatively, if the login is being imposed on the user for the purposes of the website operator, or if the use of a cookie is not strictly necessary to that login or authentication process then the requisite criteria of the narrow exception are extremely unlikely to be met.

Whether consent will be needed in each case will depend upon whether the login cookie in question is strictly necessary for a service requested by the user.  Where the criteria of the exception are not met, and consent is needed, the nature of that consent will depend in part upon the function(s) which the cookie in question is to carry out.  As you will see from our guidance, the more privacy intrusive a cookie activity is, the more important it will be to obtain meaningful consent. 

The definition of ‘consent’ is given in Directive 95/46/EC – the Data Protection Directive (which you can access online at: http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CELEX:31995L0046:en:HTML, consent is defined in article 2(h)) – as being:

…any freely given specific and informed indication of his wishes…

Whilst the fundamental definition of ‘consent’ will not change, as mentioned above, the level of information provided to obtain consent and the way in which consent is obtained may differ considerably depending on the privacy intrusion represented by the cookie activity in question.  There is therefore very definitely not one ‘catch all’ solution to cookie consent. 

You can find out more about consent on our website at: www.ico.gov.uk/for_organisations/data_protection/the_guide/conditions_for_processing.aspx (scroll down to the heading ‘consent’).

As far as the legislation is drafted, there is nothing to stop one consent being obtained to the setting of all of those cookies simultaneously – that is, when the initial cookie is set/consented to - provided always that appropriate and informed consent is obtained in accordance with the requirements of the new rule.

I hope this information is helpful.

Hell yes: I thought that was helpful and clear.

Permalink | Leave a comment  »

The Functional Brighton meetup for May was a set of short demos and talks on the subject of "What functional programming means to us".  Kit kicked off the evening with an F# show-and-tell of a 3D flocking simulator (complete wth 3D glasses); Eric spoke about why he uses Haskell; Andy showed us some Scheme code and made it progressively more functional; and Tom jumped up with a code tour of a Haskell project of his.  A good evening with plenty of Q&A.

I had every intention of recording what I said about Scala, but failed.  So here goes with a reconstruction of what I said or intended to say.

"What functional programming means to me" has been a fun assignment. I have no background in functional programming—and the funky languages I have used for any period of time were never described as functional.   So the way I've approached this is to report on three things I've noticed during my transition from mostly-Java to Scala, and consequently I'm not sure how much of this is really about functional programming.  

Approach to problem solving

The first of the three things is very general. It's about how I seem to approach problems.  It's also the hardest one for me to explain, and probably the most waffly.

Here (slide 2) is an example of the kind of problem I had to deal with last year, and what we're doing is taking a tweet and adding a hashtag to it if it's not already there.  I'm not sure how I used to solve these problems, but I think it would have dived in with a loop and maybe a string buffer and I'd test and append.

Now what I seem to find myself doing is thinking about what I have, and what I want to get (slide 3).  What do I have? Some tags and a tweet.  What do I want to get back? A tweet. Which really means (slide 4) I have a list of strings and a string, and I want a string back.  By thinking about what types are involved, I find myself asking what kinds of transformations will get me the result I want.

So in code (slide 5) if we have a List[String] and a String, and we want to get a String back, then a fold is a reasonable choice.   This is Scala, we're defining appendTags which takes our tweet which is of type String, and the tags of type List of String, and it returns a String (although I've left that off as it's being inferred).  

AppendOne solves the simple case: it expects a string and a tag and does the obvious thing.  As luck would have it, the two parameters appendOne wants are the parameters that foldLeft gives it for each of the tags and the current tweet. 

(At this point I think we dropped to the REPL and tried this out)

I find these solutions preferable in terms of clarity, LOC, also happiness in not writting the same old loop I've written many times before (slide 6).

That was the first thing: thinking in terms of transformations, and how it sort of suggests solutions in terms of functional programming constructs.

Concurrency

The second thing is about concurrency.  From what I recall, in C it was just plain hard to write concurrent code at all.  In Java, it's easy, but even easier to get it wrong.   It looks to me that the functional world want to address this problem, being both correct and easy to use.  Maybe this quote (slide 7) alludes to why, as concurrent problems boil down to coordinated access to shared mutable state.  As functional tends to point you away from mutable state, part of the battle is over.  

These are facilities I take advantage of today, and it makes it easy to bring not-insubstantial improvements to applications.   I have a code base (slide 8) that makes multiple HTTP requests to web APIs.  In this example the function yahoo, if you call it, hits Yahoo's site and brings back address book entries (let's say).  And the google one does the same to Google's API site.   They don't take very long, but they absolutely can happen concurrently.

I have a list of two functions, the underscore telling us the functions aren't actually being called yet.  I turn them into a parallel collection and then call the apply function on them.  They are executing in parallel and the result will be a List containing two lists: one for the results from each site.

This is one-liner stuff.  There's no barrier to going concurrent.

Actually, that example is from Scala 2.9, and I don't use Scala 2.9 yet, so I do something a little more involved (slide 9).

Useful concepts

The third thing that "functional is" to me, is the set of constructs functional programming assumes.  Facilities like Option, comprehensions, flatMap, pattern matching which, as I've said here (slide 10), are astonishingly useful every day and yet I'd never heard of them before using Scala.

Initially they seem like esoteric, academic, hard to understand.  They're not hard: they're rich. It takes time to appreciate them.  But it's time well spent.

I've blogged about this example (slides 11 and 12), but it's accessing the Google contacts API.  Comparing the Google Java code and the the corresponding use of Option and for comprehensions are much easier to read, and easier to write.  They make a big impact in day-to-day code.

Summary

Those were the three things (slide 13) that functional programming appears to mean to me at the moment.

1. Thinking in terms of transforamtions changed my approach to problem solving for the better.
2. Concurrency without any barriers to use.
3. The facilities in functional programming that let you write better programmes.

Permalink | Leave a comment  »

@wfaler asked:

Doing a presentation soonish on "Why #Scala" - have a truckload of arguments, but what are your arguments for Scala over Java? Give me ammo!

Ok, I'll bite.  Here's one of those every-day things I stop and notice from time-to-time. It's not new, it's not huge, it's just one of the things that makes a difference when you're trying to ship stuff.

The Google Java APIs for accessing Google contacts has you write code like this:

for (int i = 0; i < resultFeed.getEntries().size(); i++) {   ContactEntry entry = resultFeed.getEntries().get(i);   if (entry.hasName()) {     Name name = entry.getName();     if (name.hasFullName()) {     ...
That's how the "retrieving all contacts" example starts.  The basic pattern is: check if the property you want is available, then get it. 

I recently had the need to go through contacts to extract birthdays.  I only care about contacts with a name and a birthday, but I use Scala so don't have to do the if/get dance because Option does the right thing in a for-comprehension:

for { entry <- contacts       name <- Option(entry.getName)       full_name <- Option(name.getFullName)       birthday <- Option(entry.getBirthday)       when = birthday.getWhen }     yield "%s: %s".format(full_name.getValue, when)  

The above evaluates to a List[String] (because contacts starts as a List) containing those contacts with a name and a birthday.  So even when using regular Java libraries, you end with benefits (less code, more readable code, less-likely-to-cock-up-the-logic code) just by using Scala. IMHO.

Permalink | Leave a comment  »

25 Recipes for Getting Started with R, by Paul Teetor. Published O'Reilly Media, Jan 2011.

Summary

Need to "do stats"?  This text is a great way to get a hands-on tour of how to use R, to find out if you want to learn more.

Review

From time to time I need to plot some data, compute correlations... that kind of thing.  Doing anything like that in a spreadsheet is a pain for me, so I'm on the lookout for a stats framework or tool I can live with.  R (http://www.r-project.org/) could be that tool. 

You can work through this 58 page book, trying out the examples, in one session.  You probably won't improve your understanding of statistics at the end of that, but you will get a feeling of what working with R would be like.

For the time I invested in exploring R, I gained a lot of out using this book.  Worth sitting down with if you're in a similar position to me. Recommended.

Permalink | Leave a comment  »

If you want to send email from your Lift app using SendGrid you need to do two things: set an authenticator and enable the javax.mail authentication flag. That second part isn't obvious.

Step by step...

Assuming your Lift props file contains the following:

# Sendgrid.com configuration: mail.smtp.host=smtp.sendgrid.net mail.user=you@example.com mail.password=letme1n

In Boot you need to set the authenticator:

import javax.mail._ def optionalAuth: Box[Authenticator] = for {     user <- Props.get("mail.user")     pass <- Props.get("mail.password")  } yield new Authenticator {    override def getPasswordAuthentication = new PasswordAuthentication(user,pass) } Mailer.authenticator = optionalAuth

And the part that I forgot:
optionalAuth foreach { a => System.setProperty("mail.smtp.auth", "true") }   

Permalink | Leave a comment  »

Graham Tackley spoke at a packed LSUG about how guardian.co.uk implemented their first Scala project, the content API.  It's well worth watching the video to see the timeline of how the team switched their brains from Java to Scala (what the team initially liked, what they liked after a year, grokking flatMap, tools, and more).  

The result after the transition: more fun, better code, more productive and happier developers.  Regarding the cost of switching, the comment from the team during the Q&A was: there's a learning curve to Scala, but providing you have someone to lead the way, the switch doesn't materially effect delivery in the first three months, and post three months the team becomes more productive.

Go watch that video, and if you're close by, sign up to the London Scala User Group.  I had a sneak peek at some of the talks for 2011, and there are some great ones in there.

Related content: 

You might also like David Copeland's Sneaking Scala Into Your Organization video from Scala Days 2010.

Permalink | Leave a comment  »

We build our Lift applications with Hudson Jenkins, and we're in the process of converting some projects from Maven to SBT.  This is mostly straight-forward but there are a couple of non-obvious things you might need to know.

Running SBT

There's the start of a Hudson SBT plugin, but at the moment we're running SBT from Hudson by simply shelling out to execute the SBT command.

To do this, install SBT somewhere (.hudson/tools/sbt for us) and then create a new free-style Hudson job and add a build step of "Execute Shell".  We have:

/usr/lib/jvm/java-6-sun/bin/java -Dsbt.log.noformat=true -jar /home/hudson/.hudson/tools/sbt/sbt-launch-0.7.4.jar clean update test package

That's it. Your project will build.

(The -Dsbt.log.noformat=true part turns off the SBT colour formatting which you want off otherwise your build file will contain weird formatting control characters).

Controlling the WAR name

By default the WAR filename will be something like foo-scala_2.8.1-1.0.war, which is fine, except because of the way we deploy we wanted to drop the Scala version number.  That's easy to do, and you can configure it in your SBT project file:

override def artifactID = "foo"

Adding build information to your MANIFEST.MF

In our apps we like to include the Hudson build number in the footer of every page. We do this by using the fact that Maven Hudson builds include the build number in the META-INF/MANIFEST file.  SBT doesn't.

Fortunately, the fine Scala team at The Guardian have published an SBT plugin to add information to the MANIFEST.  The install details are in the README.  

Hudson sets the build number as an environment (shell) variable.  To pick these up via the plugin we just set them as -D parameters:

/usr/lib/jvm/java-6-sun/bin/java -DBUILD_NUMBER=$BUILD_NUMBER -DGIT_BRANCH=master -DJOB_NAME=$JOB_NAME -Dsbt.log.noformat=true -jar /home/hudson/.hudson/tools/sbt/sbt-launch-0.7.4.jar clean update test package

When that executes:

/usr/lib/jvm/java-6-sun/bin/java -DBUILD_NUMBER=212 -DGIT_BRANCH=master -DJOB_NAME=FooHudsonJob  -Dsbt.log.noformat=true -jar /home/hudson/.hudson/tools/sbt/sbt-launch-0.7.4.jar clean update test package

Our SBT project file picks up these System properties:

def revision = systemOptional[String]("GIT_BRANCH", "DEV").value def build = systemOptional[String]("BUILD_NUMBER", "DEV").value def versionName = systemOptional[String]("JOB_NAME", "DEV").value

The result is a WAR file with a META-INF/MANIFEST.MF file that looks something like this:

Manifest-Version: 1.0 Revision:  Implementation-Title: foo-1.0 Title: foo-1.0 Implementation-Version: FooHudsonJob Branch: master Date: Fri Jan 07 14:41:51 UTC 2011 Build: 212

Test results

With what we have at the moment if a test fails, the build fails.  This is good.

What we don't have is the nice graph of tests.  If you want that, take a look at  Christoph Henkelmann's Blog:  How to build sbt projects with Hudson and integrate test results.

For us, the next step is adding in the Undercover code coverage reporting tool instead. 

Permalink | Leave a comment  »

CouchDB: The Definitive Guide, first edition, by J. Chris Anderson, Jan Lehnardt and Noah Slater.

Published by O'Reilly Media, and available online under a Creative Commons license.

Summary

This first edition is inevitably out-of-date compared to 1.0 of CouchDB, has room for improvement, but is nevertheless a useful introduction to CouchDB and thinking about things in the CouchDB-way.  A pleasure to read.

Review

The sense I get of NoSQL products, such as CouchDB, is that they are good for certain kinds of problem and have potentially great performance characteristics.  I'm nowhere near pushing the capabilities of relational databases, but I was interested in understanding how problems are approached from a NoSQL perspective.  After all, how can I tell if I have a problem suitable for a NoSQL database if the only way of thinking I have is SQL?

I picked up the EPUB version of CouchDB: The Definitive Guide not with the aim of using the product for any particular project, but to get a sense of what the NoSQL movement is about, and how it changes the way you think about problems.  The foreword from Damien Katz, the creator of CouchDB, gave me confidence I had the right text: "The reason for this book is that CouchDB is a very different way of approaching data storage. A way that isn’t inherently better or worse than the ways before—it’s just another tool, another way of thinking about things. It’s missing some features you might be used to, but it’s gained some abilities you’ve maybe never seen. Sometimes it’s an excellent fit for your problems; sometimes it’s terrible".

CouchDB wants you to think in terms of documents.  Real-world, self-contained, messy, documents.  If you're modelling an invoice, the address for the invoice would be in the document, not a reference to another document.   No mobile (cell) number for a particular invoice? Fine, then leave it out of that document rather than being there as a mandatory yet NULL field.  But if you get a number later, the document can evolve to include it.  Storing arbitrary documents sounds like a recipe for disaster, but this is avoided by supplying an optional JavaScript validation function. The CouchDB server will reject documents that fail validation.  

What you're left with is the benefits of simplicity: you can look at a document and it makes sense by itself (or at least has that potential); and the API for storing and querying documents is the REST model of PUT, GET, DELETE.

In practice you run a CouchDB server, and insert documents represented as JSON, making a document a wrapper around the usual JSON data types of string, number, boolean, arrays and hash maps.   There are language-specific APIs for this, but as the authors say "almost all languages have HTTP and JSON libraries".  Many of the examples in the book use cURL to show exactly what's happening when interacting with CouchDB.

Queries

Queries, called views in CouchDB, consist of a map function and an optional reduce function, both implemented in JavaScript.  The map part defines the set of information you're interested in.  It's applied to each document in the database, and it can emit zero, one or more key/value pairs.  Of course, this isn't run each time you make the query.  What you're defining is an index, run once, and updated after document changes, which produces a data set sorted by the key. This can be combined with filters and ranges to select a subset of the results from a view. 

The reduce function allows you to produce aggregate results such as sums.  The weakest part of the book is the explanation of reduce and this could be clearer.    However, if you've dabbled in functional programming, some of this may make you feel at home: defining a mapping function, filtering... it's not far from what you know.

The obvious question from SQL-land ("what about joins?") is answered by way of an example of blog post comments.  The blog post and the blog comments are separate documents, which is great for performance as there are no locks when adding comments.  A query can return all posts and comments, ordered by blog post ID.  You can then select from this view just the post ID you want, which will give you the information you need to show both the post and comments.  

What makes all this fast is the B-Tree implementation, and it's reassuring to see an appendix and parts of the book dedicated to explaining how the magic works under the covers.  I don't need to know this, but it's great to see this in front of you, rather than just hand-wave about what CouchDB is capable of.

Beyond the basics

CouchDB doesn't stop at document storage and retrieval.  There are two other key elements.  The first is that CouchDB can host  whole applications.  You can define an application in terms of JavaScript functions that live inside a CouchDB document (a "design document").  These functions can produce HTML or whatever you need from inside CouchDB, which is conceptually a nice fit with the RESTful and JavaScript underpinnings of the server.  If you have database based on web technologies it's not a stretch to host the whole web app.  It's entirely optional, but I can see the attractions.  

Updating a design document each time you make an edit would be tedious, but CouchDB offers a CouchApp tool to expand your application onto disk for easy editing.  This gives you a layout that won't be too unfamiliar to developers who use Rails or other convention-based frameworks. 

The second key feature is replication.  CouchDB supports replication from one database to another, defined by a URL, meaning it can be local or remote and it makes no difference to CouchDB.  An offline database can use replication to bring itself up-to-date and transmit offline modifications back to the mothership or peers. 

Conflict management is handled by a mandatory revision value on every document. To make a change to a document, you need to provide the revision number you're editing, and the server will reject a change if you're not editing the latest revision.  During replication, however, conflicts are flagged in the document, and an algorithm ensures that all instances will pick the same document as the winning document and make it the latest revision (it's based on the history size and the sorting order of the document ID).   You can then find these documents and apply your own conflict resolution and replace the winner.

You might expect this would make CouchDB great for implementing something like infinite undo, but previous document versions are not guaranteed to be retained.

These two features (documents as applications and replication) gives CouchDB an interesting stance on application deployment.  An application and the data can be replicated to clients, who can then fork the data and application and optionally push it back.  Or you can lock it down.  But an interesting angle, I feel, for app development.

Likes and dislikes

What I'd like to see more of in this book are the idioms or patterns for document thinking.  For example, there's a section describing how to achieve document ordering in a to-do list application. If you want to move an item up your to-do list, rather than change a position field on every document to maintain an order,  you can set your document index to be the floating point number which is the average of the positions above and below your insertion point.  That has it's own problems, but is a nice example of a different approach to problem solving.  There must be more nuggets like that in the CouchDB community.

In a similar vein I'd love to see more on recipes for different applications and an expanded "cookbook for SQL jockeys" section.

I've only touched on the basics: the book covers bulk updates, security, load balancing, clustering, partioning, scalability (yes, the phrase web scale is used a couple of times), the API for watching changes... there's plenty in here.  And throughout the book you'll find hand-annotated code examples, which are a delight and comunicate a lot in a little space.

I do have some gripes.  The practical side is where I hit my first frustration with the book.  You're directed to "appendix D" to install CouchDB, where you're told to avoid installing from source, followed by the instructions for installing by source.  Google for "CouchOne" and install that.   When the authors direct you to run the tests after an install, they fail to mention they only work in Firefox.

There is some lovely writing on occasion, but the rough edges where the sections don't flow or are lacking in introduction really stick out.  In some places you're left feeling a little like a wiki page has been dragged in to the book without enough blending.  The book would benefit from an edit.

The technology is evolving, so there are inevitably mis-matches between CouchDB 1.0 and the version used in the book (a 0.9 version).   In particular, macros are out of favour and the templating format has changed.  In fairness the authors point out it's the readers responsibility to get the right versions of software and applications, but it would have been great to have an easy-to-find download of the exact version of the blog example codebase used in the book to follow along.  But I still learned plenty from figuring out how the latest version of the blog example worked.

But those are small complaints, and overall the book does a good job of introducing the CouchDB way of thinking, and was a pleasure to read.

Permalink | Leave a comment  »

See the full gallery on Posterous

There are a few mentioned on the SBT site, but I realized all I needed was a shell prompt as a view in Eclipse and I'd be happy.  And of course it exists already as "Target Management Terminal", part of the Galileo update site.

1. Eclipse > Install New Software....
2. From "Work with:" select Galileo (or whatever's appropriate for your version of Eclipse).
3. Type "terminal" in the search box, and select Target Management Terminal.
4. Go through the download, restart process.
5. Eclipse > Preferences and search for Terminal if you want to invert the colours (I did).
6. Window > Show View > Other and select Terminal to show the Terminal view in Eclipse.
7. Click the green connect button and set up the type of terminal you want (ssh to localhost in my case, and I'm prompted for a password).
8. Login, you have a shell, do what you want, such as cd into your project and run SBT.

Probably totally obvious, but I missed this trick of having access to a shell from Eclipse.

Permalink | Leave a comment  »

John at the London Scala User Group ran "LSug Workshop: Up-close and personal with SBT" yesterday at Skillsmatter.  It was a kind of mini-unconference, and I contributed an introduction to SBT.  The video above is the first half of a screen recording I made, and it continues into part 2 on Youtube.

I learnt a lot from the evening on how others use SBT, and from Maciej Matyjas talk on giter8 and plugins (including what looks like a wonderful plugin to Posterous). Phil Wills showed us how to create a SBT plugin, and that doesn't look too scary now.

Skillsmatter should post video of the evening at some point, and Chris Tuner live blogged the event.

Permalink | Leave a comment  »

Lift development has moved on from the depressingly long Maven commands to start a new project.  I made the above screencast to show the simple build tool that I'm growing fond of, and an add-on for SBT called Lifty that makes it easy to create Lift projects.

It took me a couple of attempts to make the video, so of course many of the automatically downloaded libraries were locally cached.  It'll take longer for you if you're starting from scratch, but not as long as I'd expected. For some reason (possibly imaginary) the SBT downloads for Lift seemed much smaller, and consequently faster, than I'm used to with Maven. YMMV.

If you want to know more about Lifty, check out Mads Hartmann's talk from the Scala LiftOff London.  There's also an "Up-close and personal with SBT" workshop from the London Scala User Group in the evening on 10 November 2010 (free, just sign up).

Going beyond the screencast

I don't use SBT projects exactly as I've shown in the screencast: I use JRebel and the Scala IDE for Eclipse.

"JRebel maps your project workspace directly to your running application. When a developer makes a change to any class or resource in their IDE the change is immediately reflected in the application, skipping the build and redeploy phases".  And the Scala IDE compiles on save, so the combination gives me "press save in the IDE, hit reload in the browser" development.

I've paid for the enterprise version of JRebel as I have legacy Java apps to live with, but for pure Scala/Lift development you can pick up a free Scala license of JRebel — and you should. 

The configuration is just in two places.  For SBT itself, modify the sbt launch script to include the JRebel magic.  I have:

export REBEL_HOME=/opt/zeroturnaround/jrebel export REBEL_JAR=$REBEL_HOME/jrebel.jar  export REBEL_LIC=/opt/zeroturnaround/javarebel.lic  export WITH_REBEL="-Drebel.license=$REBEL_LIC -noverify -javaagent:$REBEL_JAR" java $JAVA_OPTS -XX:MaxPermSize=256m -Xmx512M $WITH_REBEL -jar `dirname $0`/sbt-launch.jar "$@"

The other change is to prevent Jetty from restarting itself on change.  That's an edit to your project/build/Project.scala file (or whatever you called it):

override def jettyWebappPath = webappPath override def scanDirectories = Nil

That's all, and when you fire up SBT and run the jetty command, you have a pretty efficient working environment.

Oh, to generate the Eclipse project files, there are a few options listed on the SBT wiki, but I'm currently trying SbtEclipsify.

Permalink | Leave a comment  »

All the code that matters is in one file.

Permalink | Leave a comment  »

Permalink | Leave a comment  »

There were a couple of things I was particularly interested in.  First was Ross' description of how Thatcham have moved to Scala and Akka for the services they supply.  There's a write up, but highlights include the stability they have, on just a couple of servers, and the minimal amounts of code they have.  

I loved the module to integrate with Camel, meaning you can do things like wait for a file to be dropped in a directory and have the contents appear as a message to your actor. Or listen for a POST to a URL, meaning the content is delivered to your actor as a message.  Or JMS queue, which turns into...? Yes, a message to your actor.  You get the idea.  Goodbye integration, hello getting stuff done.

Impressive, rich open source product, with commercial support if you need it.

Links:

• The video and slides are online.  
• Actors in Scala by Philipp Haller and Frank Sommers published by Artima.
• http://akkasource.org/

Permalink | Leave a comment  »

See the full gallery on Posterous

I just spent two days at the Scala Lift Off unconference, meeting and chatting with 100+ old and new friends, hearing what everyone is interested in or learning about or doing. Plus the luminaries of the field were there to join in, answer questions, discuss what can be changed and how.  Think about that for a second: it's pretty much the best possible world, and it's really your fault if you can't get something good out of that :-)

I'm not going to give an account of the sessions.  I'm sure others will do a good job of that, and also there's the unconference wiki, photos tagged with scalaliftoffoct78, tweets tagged with #scalalol and videos on the Skillsmatter Vimeo site.    A few small factoids I picked up...

• Happy to see many hands go up in answer to the "who has Scala in production?" question. 
• Scala Solutions are adding their effort to the Scala IDE: making Eclipse+Scala better is seen as a good way to help Java developers over to Scala. 
• Parallel collections to ship in Jan 2011.  Making use of those multicores.
• I should look at Vector and HashMap and not just List, as they are efficient and persistent in 2.8
• sbt and lifty give a very nice way to start and build Lift projects.
• There was lots of interest in Akka.  I didn't hit any of those sessions, but this Wednesday the London Scala User Group is hosting "Simpler Scalability, Fault-tolerance, concurrency and remoting through Actors" at 6:30pm.
• There's going to be a Lift training course in London.

For me the most impressive session was the one where all the scary functional words were listed on a white board ("applicative functor", "monads"...)  and then the group tried to define them and give examples. 

If any of this sounds appealing, keep an eye on the Skillsmatter Scala page, because there's plenty happening you might like.   

See you at Scala Liftoff London 2011 (Oct 6 and 7, registration now open). 

Permalink | Leave a comment  »