We rarely get to talk about our Lift work, but I can talk about this one because it's been a Sunday afternoon project for a while.  

I live in Google for my contacts and calendar, and wanted a quick summary of upcoming birthdays and anniversaries.  I'm not alone in wanting something like this.  Reminder Thing itches this scratch by digging into my contacts and calendar each week, and emailing me a single reminder of upcoming birthdays and anniversaries.  

That's it.  Obviously, if you live in Facebook, you don't need this, as they already sensibly send out such an email.  But I've not yet, um, fully embraced Facebook.

But this is mostly a tech blog, so here are the details. It's a Scala app, running on the Lift framework.  It uses SendGrid to send email, and uses Google's own AuthSub and GDATA Java APIs to communicate with Google. The UI is Twitter's Bootstrap.

We build using SBT and deploy using SBT Cloudbees Plugin to a single CloudBees 128M cell.  I believe Cloudbees is currently using JDK 1.6, Tomcat 6 and Nginx, but I neither know for sure and nor do I especially care—if it works, it's fast and I doesn't give me hassle, I'm happy. 

BTW, If you're in the Sydney area and want to know more about how we deploy Scala apps, to CloudBees or AWS, go to ScalaSyd on 26 Oct 2011, where Jono is on a panel discussing Scala in production.

I have two tech observations from writing this code:

First, firing off asynchronous events to Google is a delight with Scala and Lift.  The page where we show spinners while fetching data from Google makes use of the Lift LazyLoad CSS tag (which means it becomes concurrent without me making any code changes) and a little bit of Lift Wiring (8 lines) to make the "Continue" button change colour.

The other comment is that Lift parameterised menus really help you be productive: very worth getting to know.  In Reminder Thing the unsubscribe link plus the ability to undo your unsubscribe uses parameterised menus:

So that's Reminder Thing.  There are a couple of tweets on adding birthdays and anniversaries  from @rmndrthng, which is also the right Twitter account to use to let us know if you like the idea, anything that doesn't work, anything we can do better.  

If you're a Google contacts or calendar user, do give it a go.

An external module is one way to share lumps of code with the Lift community.  The word "external" indicates that these are distinct from the modules in the core Lift repository maintained by Lift comitters.  But the pattern to create a module is the same, as Peter outlined on the Lift Wiki and in one or two posts on the Lift mailing list.

Anyone can create one, anywhere they like, and people do. So there's nothing to talk about.

Except... I'd like to have modules build automtically when Lift changes, with reports on problems, and have an easy way to find modules and add them to Lift projects.  If you buy into that as something useful, there are lots of ways to achieve it.  

Are we there yet? Are we there yet? Are we...

Today, we're nowhere near, but we've started by experimenting with a build system and a common repository for external modules. 

We want to track Scala and Lift releases, including Lift milestone releases, and publish these builds so they can be available quickly to anyone wanting to use them via Maven or SBT.  That's the plan.  To explore this we're using CloudBees. Under their open source programme we're given a hosted Jenkins instance and repository for build artifacts.  In other words, we go to the CloudBees web site, create a new Jenkins job, point it at the appropriate GitHub repo and press "Build now". Nice and easy, but all manual.

Currently we're building just a few modules, which you can see on the Jenkins dashboard:

•  Google Analytics support
•  IMAP IDLE, providing push-like email into a Lift app
•  JqPlot module for graphs

...but none of it is automatic yet.  We know we can do better.

The developer experience

This is what the current development process is like for getting a new release out.  Let's say, Lift 2.4-M4 becomes available (which it has; this post has been in draft too long).

The module versioning number I'm using is the Lift version with the module version appended.  So the 0.91 version of the IMAP module for Lift 2.4-M3 is: 2.4-M3-0.91. Sometimes, that's all you need to change, making me wonder if this is a wasteful system.  Anway, let's publish a version of 2.4-M4:

$ cd liftmodules-imap-idle
$ git checkout -b 2.4-M4
$ vi build.sbt
// modify version, liftVersion, test
$ git push origin 2.4-M4
$ sbt 
 > +publish   

...then merge back to master:

$ git checkout master
$ git merge 2.4-M4
$ git push origin master   

Where next?

Are we making a meal out of this?  Is this something worth doing?  Are there better ways?  Feedback is what we need, and if you're part of the Lift community, the place for that is on the mailing list please.

Otherwise we'll be investigating: 

• adding triggers or polling to trigger builds of modules. 
• possibly a directory of modules - but that seems like the easiest part now that we've started to push out notifications to the fantastic implicit.ly space.
• scratching our heads about how to track SNAPSHOP.
• can we automate module changes to track Lift milestone releases?
• automatically publishing builds into the repo.

Or something like that.

If you want your module included in this build process, hassle me or Jono on the Lift mailing list.

• Improvements for temporary network failure. We now monitor and disconnect if IDLEing for more than 30 minutes, as a way to catch socket errors hidden by IDLE.
• If disconnected from the IMAP server, we now request all waiting messages on a reconnect.
• Switched to SBT 0.10.1.
• Now also publishing source and javadoc into the repository.

This version is available for Lift 2.4-M3 for Scala 2.8.1 and 2.9.0-1.

ImapIdle.init { m: javax.mail.Message => 
  println("You've got mail: "+EmailUtils.dump(m))
  true // delete the email on the server
}

Dean Wampler, 2011, Functional Programming for Java Developers, O'Reilly Media. 88 pages.

Summary

This is the book for you if you code in Java, you've heard of functional and want to know what the fuss is about, but you want it in terms of the constructs of Java that you already know.  "This book explains why functional programming has become an important tool for the challenges of our time" and acknowledges that "much of the literature on functional programming is difficult to understand for people who are new to it."  

In 88 pages there's a limit to what can be covered, and sure I have some reservations, but I don't know of a better introduction for Java developers who aren't comfortable looking at other languages.

Details

The first chapter addresses the "why functional?" question, which is answered primarily in terms of bringing clarity to thinking, more reuse and more concise code, and a better fit to "the unique challenges of our time", such as "working with massive data sets and remaining agile".  But you're not expected to jettison what you know, with later chapters suggesting when mixing in OO is a great idea.

The detailed answer to the "why" question talks in terms of concurrency, big data sets, modularity benefits, keeping code minimally sufficient, reining in complexity—making the argument that functional programming helps you in your job in all the areas, and these are areas you need to be able to handle today.

In describing what functioning programming is (chapter 2), you get to see some Java code simplified via the Functional Java project library.  In this chapter, and rolling into chapter 3, types and data structures are introduced, via an implementation of Option (how to avoid null) and List in Java.  This is all, I think, genuinely useful, especially with the "tips" in the text and the exercises at the end of the chapters.

Reservations

Tackling this subject in Java is a big ask, and around half way through the book it starts to show.  When filter, fold and map are introduced I suspect Java programmers drawn to this text may be scratching their heads asking why they would ever want to write code like this.  

A later chapter introduces pseudo code to illustrate concepts in a Java-like syntax.  This isn't going to compile, and you have to ask yourself if the text has drifted too far from Java. The author puts it well when suggesting the next place to go is "writing real code", pointing in the direction of a variety of languages that have anonymous functions and higher-order functions... which is probably any language except Java.

That said, the chapter on concurrency points to the Java Akka implementation of Actors, which looks like something you might want to use straight away.  It's noted that the Actor model "isn't really a functional approach to concurrency", but it's part of the culture.

Conclusion

A concise and personal look at functional programming from the Java point of view, with some practical and compelling points about the benefits on the large (concurrency) and small (bugs in a class) scale.  If you're drawn to this area, you'll going to find this book valuable.  But you know you're only delaying switching to another language ;-)

Pretty much the simplest thing you can do to get started with machine learning in a web app. These are my slides from BathCamp AI night.

You know what I'm about to say, right?  I am not a lawyer.

But I do have to keep track of changes in the law that impact us and our clients, and one of those is the Privacy and Electronic Communications Regulations.  It's the one about getting consent to set cookies before you set them.  There's a nice short summary by Simon Weinberg.

We regularly use three groups of cookies in our applications: JSESSIONID, ext_id (Lift's "keep me logged in" functionality), and the handful used by Google Analytics.  My impression is that we can reasonably argue that JSESSIONID is an essential cookie for the use of any of our sites.  The others are going to need consent, and at the very least an update to all of our terms of service and privacy policy documents.

All of the above are assumptions I'm making.  If you want an example of how others are tackling this problem, take a look at the Information Commissioner's Office site.

The long answer

Specifically regarding the ext_id under one usage scenario, I did ask what the guidance is:

Hello

I'm trying to understand if "keep me logged in" functionality falls under the new rules.  By that I mean the use of a cookie to enable a registered user of a site to visit the same site (potentially over many weeks or months) without the need to re-log in to gain access to their account.  This is not used for tracking or marketing. Currently we do not ask the user if they want to set this cookie: we are assuming the user is logged in until they explicitly log out.

Do you have a view on this type of cookie use?

Here's the reply in case you're in a similar position, and I though the repy was really well put together and useful:

Thanks you for your recent enquiry regarding the new Privacy and Electronic Communications Regulations 2003.

It is my understanding that no differentiation between cookies on the basis of their duration is made in the Privacy and Electronic Communications Regulations 2003 (PECR) as amended.  Therefore it is our guidance that cookies need to be consented to irrespective of their temporary nature, unless the requirements of the ‘strictly necessary’ exception are fulfilled.

As there is no exception specifically for ‘login’ cookies, whether or not a login cookie is excepted from the basic consent rule will depend on whether that cookie falls within the ‘strictly necessary’ exception.  There is not a straightforward or catch-all answer to whether a ‘login’ cookie will be ‘strictly necessary’. 

What will be crucial in deciding whether a login cookie fits within the narrow exception of being ‘strictly necessary’ is whether the use of that login cookie is essential for a service requested by the user.  It is essential that both parts of the exception are met – that it is a service the user has requested, and that the cookie itself is strictly necessary to that service.  (Our guidance on this point, as you have no doubt already seen, is available on our website: Read the ICO’s advice to organisations about how to prepare for the new rules on cookies). 

In some circumstances it may be that a login cookie falls within the ‘strictly necessary’ exception – in others it will not.  We have not provided more detailed examples regarding login cookies as it really will depend upon an individual website’s specific circumstances as to whether the requirements of the exception are met.  I can see a situation where it might potentially be argued that a login cookie is strictly necessary if a user has asked to login to a site to access certain information and a cookie is integral to that login or authentication process.  Alternatively, if the login is being imposed on the user for the purposes of the website operator, or if the use of a cookie is not strictly necessary to that login or authentication process then the requisite criteria of the narrow exception are extremely unlikely to be met.

Whether consent will be needed in each case will depend upon whether the login cookie in question is strictly necessary for a service requested by the user.  Where the criteria of the exception are not met, and consent is needed, the nature of that consent will depend in part upon the function(s) which the cookie in question is to carry out.  As you will see from our guidance, the more privacy intrusive a cookie activity is, the more important it will be to obtain meaningful consent. 

The definition of ‘consent’ is given in Directive 95/46/EC – the Data Protection Directive (which you can access online at: http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CELEX:31995L0046:en:HTML, consent is defined in article 2(h)) – as being:

…any freely given specific and informed indication of his wishes…

Whilst the fundamental definition of ‘consent’ will not change, as mentioned above, the level of information provided to obtain consent and the way in which consent is obtained may differ considerably depending on the privacy intrusion represented by the cookie activity in question.  There is therefore very definitely not one ‘catch all’ solution to cookie consent. 

You can find out more about consent on our website at: www.ico.gov.uk/for_organisations/data_protection/the_guide/conditions_for_processing.aspx (scroll down to the heading ‘consent’).

As far as the legislation is drafted, there is nothing to stop one consent being obtained to the setting of all of those cookies simultaneously – that is, when the initial cookie is set/consented to - provided always that appropriate and informed consent is obtained in accordance with the requirements of the new rule.

I hope this information is helpful.

Hell yes: I thought that was helpful and clear.

The Functional Brighton meetup for May was a set of short demos and talks on the subject of "What functional programming means to us".  Kit kicked off the evening with an F# show-and-tell of a 3D flocking simulator (complete wth 3D glasses); Eric spoke about why he uses Haskell; Andy showed us some Scheme code and made it progressively more functional; and Tom jumped up with a code tour of a Haskell project of his.  A good evening with plenty of Q&A.

I had every intention of recording what I said about Scala, but failed.  So here goes with a reconstruction of what I said or intended to say.

"What functional programming means to me" has been a fun assignment. I have no background in functional programming—and the funky languages I have used for any period of time were never described as functional.   So the way I've approached this is to report on three things I've noticed during my transition from mostly-Java to Scala, and consequently I'm not sure how much of this is really about functional programming.  

Approach to problem solving

The first of the three things is very general. It's about how I seem to approach problems.  It's also the hardest one for me to explain, and probably the most waffly.

Here (slide 2) is an example of the kind of problem I had to deal with last year, and what we're doing is taking a tweet and adding a hashtag to it if it's not already there.  I'm not sure how I used to solve these problems, but I think it would have dived in with a loop and maybe a string buffer and I'd test and append.

Now what I seem to find myself doing is thinking about what I have, and what I want to get (slide 3).  What do I have? Some tags and a tweet.  What do I want to get back? A tweet. Which really means (slide 4) I have a list of strings and a string, and I want a string back.  By thinking about what types are involved, I find myself asking what kinds of transformations will get me the result I want.

So in code (slide 5) if we have a List[String] and a String, and we want to get a String back, then a fold is a reasonable choice.   This is Scala, we're defining appendTags which takes our tweet which is of type String, and the tags of type List of String, and it returns a String (although I've left that off as it's being inferred).  

AppendOne solves the simple case: it expects a string and a tag and does the obvious thing.  As luck would have it, the two parameters appendOne wants are the parameters that foldLeft gives it for each of the tags and the current tweet. 

(At this point I think we dropped to the REPL and tried this out)

I find these solutions preferable in terms of clarity, LOC, also happiness in not writting the same old loop I've written many times before (slide 6).

That was the first thing: thinking in terms of transformations, and how it sort of suggests solutions in terms of functional programming constructs.

Concurrency

The second thing is about concurrency.  From what I recall, in C it was just plain hard to write concurrent code at all.  In Java, it's easy, but even easier to get it wrong.   It looks to me that the functional world want to address this problem, being both correct and easy to use.  Maybe this quote (slide 7) alludes to why, as concurrent problems boil down to coordinated access to shared mutable state.  As functional tends to point you away from mutable state, part of the battle is over.  

These are facilities I take advantage of today, and it makes it easy to bring not-insubstantial improvements to applications.   I have a code base (slide 8) that makes multiple HTTP requests to web APIs.  In this example the function yahoo, if you call it, hits Yahoo's site and brings back address book entries (let's say).  And the google one does the same to Google's API site.   They don't take very long, but they absolutely can happen concurrently.

I have a list of two functions, the underscore telling us the functions aren't actually being called yet.  I turn them into a parallel collection and then call the apply function on them.  They are executing in parallel and the result will be a List containing two lists: one for the results from each site.

This is one-liner stuff.  There's no barrier to going concurrent.

Actually, that example is from Scala 2.9, and I don't use Scala 2.9 yet, so I do something a little more involved (slide 9).

Useful concepts

The third thing that "functional is" to me, is the set of constructs functional programming assumes.  Facilities like Option, comprehensions, flatMap, pattern matching which, as I've said here (slide 10), are astonishingly useful every day and yet I'd never heard of them before using Scala.

Initially they seem like esoteric, academic, hard to understand.  They're not hard: they're rich. It takes time to appreciate them.  But it's time well spent.

I've blogged about this example (slides 11 and 12), but it's accessing the Google contacts API.  Comparing the Google Java code and the the corresponding use of Option and for comprehensions are much easier to read, and easier to write.  They make a big impact in day-to-day code.

Summary

Those were the three things (slide 13) that functional programming appears to mean to me at the moment.

1. Thinking in terms of transforamtions changed my approach to problem solving for the better.
2. Concurrency without any barriers to use.
3. The facilities in functional programming that let you write better programmes.

@wfaler asked:

Doing a presentation soonish on "Why #Scala" - have a truckload of arguments, but what are your arguments for Scala over Java? Give me ammo!

Ok, I'll bite.  Here's one of those every-day things I stop and notice from time-to-time. It's not new, it's not huge, it's just one of the things that makes a difference when you're trying to ship stuff.

The Google Java APIs for accessing Google contacts has you write code like this:

for (int i = 0; i < resultFeed.getEntries().size(); i++) {
   ContactEntry entry = resultFeed.getEntries().get(i);
   if (entry.hasName()) {
     Name name = entry.getName();
     if (name.hasFullName()) { 
    ...

I recently had the need to go through contacts to extract birthdays.  I only care about contacts with a name and a birthday, but I use Scala so don't have to do the if/get dance because Option does the right thing in a for-comprehension:

for { entry <- contacts
      name <- Option(entry.getName)
      full_name <- Option(name.getFullName)
      birthday <- Option(entry.getBirthday)
      when = birthday.getWhen 
}
     yield "%s: %s".format(full_name.getValue, when)

The above evaluates to a List[String] (because contacts starts as a List) containing those contacts with a name and a birthday.  So even when using regular Java libraries, you end with benefits (less code, more readable code, less-likely-to-cock-up-the-logic code) just by using Scala. IMHO.

25 Recipes for Getting Started with R, by Paul Teetor. Published O'Reilly Media, Jan 2011.

Summary

Need to "do stats"?  This text is a great way to get a hands-on tour of how to use R, to find out if you want to learn more.

Review

From time to time I need to plot some data, compute correlations... that kind of thing.  Doing anything like that in a spreadsheet is a pain for me, so I'm on the lookout for a stats framework or tool I can live with.  R (http://www.r-project.org/) could be that tool. 

You can work through this 58 page book, trying out the examples, in one session.  You probably won't improve your understanding of statistics at the end of that, but you will get a feeling of what working with R would be like.

For the time I invested in exploring R, I gained a lot of out using this book.  Worth sitting down with if you're in a similar position to me. Recommended.

If you want to send email from your Lift app using SendGrid you need to do two things: set an authenticator and enable the javax.mail authentication flag. That second part isn't obvious.

Step by step...

Assuming your Lift props file contains the following:

# Sendgrid.com configuration: 
mail.smtp.host=smtp.sendgrid.net 
mail.user=you@example.com 
mail.password=letme1n 
 

In Boot you need to set the authenticator:

import javax.mail._ 
def optionalAuth: Box[Authenticator] = for {
    user <- Props.get("mail.user")
    pass <- Props.get("mail.password")
  } yield new Authenticator {
    override def getPasswordAuthentication = new PasswordAuthentication(user,pass)
}
Mailer.authenticator = optionalAuth

And the part that I forgot:

optionalAuth foreach { a => System.setProperty("mail.smtp.auth", "true") }