30 Oct 2009

Nov 28: London Java Unconference

On Sat Nov 28th 2009, I'll be at the London Java Community's Unconference at the IBM south bank building. Not exactly sure what I'll contribute, but my current thinking is something around the area of approaches to adding Scala into existing Java projects.

Check out the details on the Upcoming page or read more on the LJC's blog post. It's £20 to cover costs (any left-over money is put behind the bar :-), but hurry... when I last checked there were only 11 places left out of 50.

17 Sep 2009

The Scala REPL is great for Java developers too

This post is for Java developers who have heard about Scala but, although it might sound interesting, are getting on with what they need to do in Java thankyouverymuch.

If that's you, it's still worth installing Scala because the command-line tool makes noodling with Java a delight. The tool in question is the REPL (read-eval-print-loop, a language shell).

Two quick examples...

Let's say there's some API you're going to use, but you just can't quite remember the format of the return result. The REPL is a great way to quickly find out what you actually get. Here's what I found out looking for the list of all TimeZones:


$ scala
Welcome to Scala version 2.7.5.final [...]
Type in expressions to have them evaluated.
Type :help for more information.

scala> import java.util.TimeZone
import java.util.TimeZone

scala> TimeZone.getAvailableIDs()
res2: Array[java.lang.String] = Array(Etc/GMT+12, Etc/GMT+11,
MIT, Pacific/Apia, Pacific/Midway, Pacific/Niue,
Pacific/Pago_Pago, Pacific/Samoa, US/Samoa, America/Adak,
America/Atka, Etc/GMT+10, HST, Pacific/Fakaofo,
Pacific/Honolulu, Pacific/Johnston, Pacific/Rarotonga,
Pacific/Tahiti, SystemV/HST10, US/Aleutian, US/Hawaii,
Pacific/Marquesas, AST, America/Anchorage, America/Juneau,
A...


Most of the developers I know might do a double take at the results syntax but wouldn't have a problem reading that TimeZone.getAvailabeIDs() gives me back an array of strings like "Pacific/Tahiti". No Scala knowledge required to make use of that tool (ok, I left the line-ending semi-colons out, but you can put them in if you like).

Second example. Quick! Answer this: what does String.split return if there are no matches to the pattern? Not sure? Try it and see:


scala> "Pacific/Tahiti".split("/")
res3: Array[java.lang.String] = Array(Pacific, Tahiti)

scala> "wibble".split("/")
res4: Array[java.lang.String] = Array(wibble)


Hope that's useful. Don't miss that you have command history editing (arrow keys on my keyboard).

If you want to do more take a look at First Steps to Scala.

30 Sep 2008

Tracking down threading issues

One of the projects we've been working on involves kicking off a number of threads to check on availability of certain services. Despite having read the fantastic Java Concurrency in Practice book (which you should buy, possibly from Amazon UK) we'd run into a situation where Tomcat wasn't shutting down. Inevitably it'd be because we'd starting a thread that wasn't stopping. As a reminder to myself here are a bunch of useful tools for figuring out what might be going on....

Step 1: find out what's running with jps -lm:


$ jps -lm
1001 org.apache.catalina.startup.Bootstrap start
1302 sun.tools.jps.Jps -lm
1066 com.j_spaces.core.client.SpaceFinder /./taykt
176

This is a list of the Java process IDs which are "typically, but not necessarily, the operating system's process identifier for the JVM process", according to the JPS manual page. In my case, they were the OS process IDs, although I should point out that I'm running this on Mac OS 10.5 using JDK 5, and the tools do vary by operating system.

What the list is showing me is that the Tomcat process is still running (process 1001). All the other processes I expect to see on my machine, including the mysterious 176 which happens to be IntelliJ.

Step 2: take a look at what threads are running in the process, as described in the J2SE 5.0 Trouble-Shooting and Diagnostic Guide PDF:


$ kill -QUIT 1001

Hmm. No output from that command... of course, because I've signalled to the Tomcat process to give me a thread dump the output will be in the Tomcat logs:

==> catalina.out Full thread dump Java HotSpot(TM) Client VM (1.5.0_16-133
mixed mode, sharing):

"DestroyJavaVM" prio=5 tid=0x010017f0 nid=0xb0801000 waiting
on condition [0x00000000..0xb0800060]

"SelfCleaningTable$Cleaner" daemon prio=5 tid=0x01049030
nid=0x8e7a00 in Object.wait() [0xb171f000..0xb171fd90]
at java.lang.Object.wait(Native Method)
- waiting on (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
- locked (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
at com.j_spaces.obf.eb.run(SourceFile:231)

"pool-1-thread-1" prio=5 tid=0x01021470 nid=0x8f0c00 waiting
on condition [0xb0c8a000..0xb0c8ad90]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport
.park(LockSupport.java:118)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$
ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at java.util.concurrent.LinkedBlockingQueue
.take(LinkedBlockingQueue.java:359)
at java.util.concurrent.ThreadPoolExecutor
.getTask(ThreadPoolExecutor.java:470)
at java.util.concurrent.ThreadPoolExecutor$Worker
.run(ThreadPoolExecutor.java:674)
at java.lang.Thread.run(Thread.java:613)
...


...it goes on for a bit. The parts I was interested in were non-daemon processes (because daemon ones are shutdown when the JVM shutsdown). In my case, the "pool-1-thread-1" process reminded me that we'd put a few things into an Executor and possibly not cleaned up properly. I know it was that thread name, because I'd seen it in the application log files. A quick dip into Java Concurrency in Practice to remind myself about The Rules, and I had a few modifications that resolved the issue. But...

Step 3: try out JConsole. The trick here is that you need to set a flag at Tomcat start up time:


$ export CATALINA_OPTS="-Dcom.sun.management.jmxremote"
$ sh startup.sh

Then a quick jps -lm to find the process number, then fire up JConsole with jconsole pid goes here and you're set. Not only do you get easier access to the thread list, you also have buttons in the MBean tab with tempting names.
14 Jun 2008

Comparing closures in Java, Groovy and Scala

On Paul's return from JavaOne this year, we spoke about Neal Gafter's Closures Cookbook talk. From what I understood, this was a look at the BGGA closures proposal, and contained an example that pushed hard on some of the tougher closure issues for Java. I thought it might be fun to look at the Java example from the talk, and covert it to Scala and Groovy.

Why those languages? Because they are the three JVM languages I'm most interested in. I suppose I could also have compared the closure support in Jython, JRuby or... well, there are a few to choose from, but this blog is going to be plenty long enough with just three.

Let's start with the Java example that was given, remembering that this is a proposed syntax, that may or may not make it to Java 7 or later. As I understood the example it was this: imagine you want to add the ability to time a block of code, and you wanted to do it in a way that would look almost like a new keyword has been added to the language; and you wanted to pass in a parameter to name what you were timing; and the block you're timing returns a result, or might throw an exception. So, quite an involved case.

Here's how the current Java proposal looks:


// Here's a method that uses the time call:
int f() throws MyException {
time("opName", {=>
// some statements that can throw MyException
});
time("opName", {=>
return ...compute result...;
});
}

So we're timing a couple of operations, and we're doing this inside a method, f, that returns an integer. The implementation would be....


interface Block {
R execute() throws X;
}

public R time(
String opName, Block block) throws X {
long startTime = System.nanoTime();
boolean success = true;
try {
return block.execute();
} catch (final Throwable ex) {
success = false;
throw ex;
} finally {
recordTiming(
"opName", System.nanoTime() - startTime, success);
}
}


As you can see, time takes an arbitrary text label and a block of code, runs the block and tells you how long the block took to run and if it succeeded or not.

That's the example that was given at JavaOne. Now for the same thing in Groovy...

To make runnable code for Groovy (and for Scala), I had to decided to time something. So I'm timing a block of code that randomly throws an exception or returns something. And then timing a block of code that just returns a number. In Groovy that would be:


def time(opname, block)
{
long start_time = System.nanoTime()
boolean success = true
try {
return block()
} catch (Throwable ex) {
success = false;
throw ex
} finally {
diff = System.nanoTime() - start_time
println "$opname $diff $success"
}
}

int f() throws Exception {
time("a") {
Random r = new Random()
if (r.nextInt(100) > 50)
throw new IOException("Boom")
else
return 42
}

time("b") {
return 7
}
}

println f()


An example of running the code:


$ groovy time.groovy
a 37116000 true
b 69000 true
7

$ groovy time.groovy
a 39998000 false
Caught: java.io.IOException: Boom
at time$_f_closure1.doCall(time.groovy:21)
at time$_f_closure1.doCall(time.groovy)
at time.time(time.groovy:6)
at time.f(time.groovy:18)
at time.run(time.groovy:31)
at time.main(time.groovy)


Note that the Java example is typed in that it uses a generic type, R, for the return value which gives you some compile-time checks. That is, when you run time and use the result, the compiler will enforce that your declaration of the result has the same type as the return type of the block you're timing.

Although Groovy does support generics, I've not used them here, and as a result the Groovy example doesn't have that type-safety. I think that's the way one would typically write Groovy code.

UPDATE: as was pointed out to me in the comments on the Java Lobby version of this blog post, this isn't the same as the Java version. In the Java version a return in the closure returns out of the enclosing block.

Now a look at the same code in Scala:


def time[R](opname: String)(block: => R) = {
val start_time = System.nanoTime()
var success = true
try {
block
} catch {
case ex: Throwable => {
success = false;
throw ex
}
} finally {
val diff = System.nanoTime() - start_time
println(opname + " " + diff + " "+success)
}
}

def f():Integer = {

val answer = time("a") {
val r = new Random()
if (r.nextInt(100) > 50)
throw new java.io.IOException("Boom")
else
"42"
}

println ("the answer, "+answer+" is of type "+
answer.getClass())

val seven:Integer = time("b") {
7
}

println ("seven is of type "+seven.getClass())

return seven
}

f()


I'm still not using Scala day-to-day, so this might be a little awkward: thank you to the London Scala User Group for helping me clean up my syntax, but all the mistakes are mine.

This code has the same properties as the Java code (type safety via the R generic type), but seems a little shorter and neater. Additionally, the thing I like about the Scala code (and the Groovy code) is that the languages return the value of the last statement in a block, and that the syntax allows a clean time("thing") { ... } format.

One observation: I've used the Integer class, which is deprecated, in order to be able to print out the class of the return type in the function f(). Without the :Integer declaration I was getting weird compile errors. As I said, my understanding of Scala and type inference isn't there yet.

The output from running the code:


a 913000 true
the answer, 42 is of type class java.lang.String
b 13000 true
seven is of type class java.lang.Integer

a 936000 false
java.io.IOException: Boom
at Main$$anonfun$1.apply((virtual file):26)
at Main$$anonfun$1.apply((virtual file):23)
at Main$.time$1((virtual file):8)
at Main$.f$1((virtual file):23)
at Main$.main((virtual file):44)
at Main.main((virtual file))
// rest of stack trace removed

There's no conclusions here. It's just an exercise in comparing closures code in three different languages. I've probably missed some of the nuances of the Java example, but hey... it's a starting point.

13 Apr 2008

Decrypting JetS3t Files

This post is going to be a bit niche. The scenario is that you've used JetS3t to backup data to Amazon S3 via the synchronize tool, and in particular you've used the -c option to encrypt the data. But you've downloaded the file with another tool, such as ForkLift or S3 Browser. How do you decrypt the downloaded file?

The default encryption is PBEWithMD5AndDES, and with that knowledge you may be able to find a tool that can decrypt it for you. I went a different way, and just hooked straight into the encryption utilities inside JetS3t:


// Apache 2 license
public static void main(final String... args)
throws GeneralSecurityException, IOException
{

if (args.length != 4 && args.length != 2)
{
System.err.println(
"Usage: Decrypt [-alg algorithm] KEY FILE");
System.exit(1);
}

final String alg;
final String key;
final File encryptedFile;

if (args.length == 4)
{
alg = args[1];
key = args[2];
encryptedFile = new File(args[3]);
}
else // use defaults for algorithm:
{
alg = "PBEWithMD5AndDES";
key = args[0];
encryptedFile = new File(args[1]);
}

final EncryptionUtil eu = new EncryptionUtil(
key, alg, EncryptionUtil.DEFAULT_VERSION);

CipherInputStream decrypt=null;
BufferedOutputStream outStream=null;
try
{
decrypt = eu.decrypt(
new FileInputStream(encryptedFile));
IOUtils.copy(decrypt, System.out);
}
finally
{
IOUtils.closeQuietly(decrypt);
IOUtils.closeQuietly(System.out);
}

}


So with a bit of fiddling you can copy, paste and compile that (you'll need the JetS3t library and supporting JARs, plus Commons IO). Or you can download jets3t-decrypt-dist.zip, which contains the source and the reqired libraries. Once inside the ZIP you can run:


java -jar Jets3tDecrypt.jar password encrypted.file
> decrypted.file

The libraries and the Jets3tDecrypt.jar was packaged automatically by Netbeans 6.1, which is a lovely touch from an IDE.
16 Mar 2008

QuickTime for Java

A few months ago I was experimenting with QuickTime for Java. It's the binding between Apple's QuickTime "stuff" and the Java language, allowing a Java developer to invoke QuickTime on the Mac (and presumably also on Windows). The reason this appeals is that we do Java, and have accumulated a fair amount of Mac hardware.

To cut a long story short, my limited experienced shows that QuickTime, when compared to the standard Java image libraries, produced smaller images of higher quality, faster. It's unusual to get all three benefits together (smaller, faster, better). Too good to be true, even, which leads me to think I've screwed up someplace, but I've not spotted it yet.

The particular test that interested me was was taking a camera phone photo, resizing it and rotating it. Here's an example original:

Media_httpfarm2static_dkjdl

Here's the rotation using tried-and-trusted AffineTransform plus ImageIO:

Media_httpfarm4static_ctdbz

And here's the same transformation run through QuickTime for Java using the GraphicsImporter and Matrix objects:

Media_httpfarm4static_wfnbh

Now, it's subtle but the QT4J image looks to have sharper colours, and seems to be a better representation of the original input. It's also 20k compared to the 76k using the Java 2D libraries, and the code runs 1.7 times faster. The downside: you need Apple hardware.

A few conclusions: it seems the Java 2D code doesn't just fall through to QuickTime on the Apple platform in any simple sense (which, I suppose I might have naively expected). Second, I suspect there's a lot of tuning that can be done in the Java 2D client usage to improve the quality, but QT4J seems to have good defaults.

(Before you ask, the dog's name is Jack.)

19 May 2007

JavaOne 2007

For the impatient: impressive Seam and JRuby/NetBeans demos; JavaFX doesn't look that crazy after all; performance improvements show promise; quality of the talks were consistently high and better than I recall from previous years; need to look at Groovy and Grails more.

Media_httpfarm3static_atmvf

I don't like to make too much fuss about these events, but when I go into the first keynote on the first day of JavaOne, look around and see the size of the place (space for just 15,000 attendees this year), look at the huge screens on stage, with DJ Anon doing her thing, and then the big numbers come out: 6 million developers, 2 billion mobile devices, 11 million TVs, Java on 91% of desktops. I have to admit to being somewhat awed by the event. They do a good show. It's not Steve Jobs, but it's pretty good.

The big announcement this year was JavaFX. This is a blanket term covering authoring tools for "content professionals", an environment for mobile and a scripting language. The authoring tools don't exist yet. JavaFX Script does, and used to be called F3—a technology I've previously described as "nuts". We first heard about F3 at the London Sun Tech Days, and we couldn't understand why it existed when SVG looked to do the same thing in a similar way.

This time, we had Chris Oliver, the author of the language, on hand to explain it. His talk kicked off by asking why Java applications looked so different to Flash applications. Java has the capability to produce all the effects you see in Flash, but they're harder to get at. Step in JavaFX Script which is component scripting of Java 2D and Swing.

And why not SVG? Well, there are lots of similarities between SVG and JavaFX Script (indeed, there's a converter), but the dynamic stuff in SVG is a pain. That's something I can confirm from experience.

In JavaFX Script you can do the dynamic stuff with Java (naturally), but the language also has a data binding feature (functional reactive programming, apparently). As I understood it, elements of the view are bound to elements of the model. This means there's no listening code to write: the model changes, the view updates. There are demos, downloads and documentation over at openjfx.org.

So maybe not so nuts after all. To get that into the browser, though, needs some work. Right now you need to add about 700k to an applet download. The Java Plugin needs some work to make this better (but see the demos over at swinglabs.org; I must get round to trying out Iris). Chris Oliver wasn't taking questions on this area. In fact, we had the impression that he had perhaps been under a bit of pressure recently: after all, the technology had a different name a month or so ago, was a side demo at London Tech Days, and now was pushed way up the conference agenda.

In related news, there is going to be a consumer JVM, presumably targeting RIA, scheduled for early 2008. The focus is on performance, both runtime and download time. There's also work going on to modularize Java (super packages in development terms; super JARs in deployment terms).

The JavaFX Mobile announcement is that Sun have a complete mobile stack which is being offered to OEMs: it's the SavaJe technology on top of Linux. We saw a demo running on a Neo1973, the SavaJe device, and a Nokia N800. Meh... we'll see what comes of it.

While on the subject of mobile, just two notes: I was disappointed to see only one prototype MSA device (SE Z50i), given that MSA has been talked about for a long time; but I had a look at the Nokia 6110 Navigator, and that has a great deal of promise as a phone for location, although no WLAN and only a single processor.

Anyway, JavaFX Mobile is being placed on the cusp between ME and SE:

Media_httpfarm3static_ugdel

Probably the other headline news was the support for Ruby and Rails via the JRuby implementation. If you're developing in Ruby (and/or using Rails), you'll want to check out the Toy Show Part 2 from the Friday keynote or take a look at one of the online demos. Either way, you should try out NetBeans 6.

Going into the conference I was wondering which scripting language I should be putting effort into: Ruby or Groovy. Both look to be very capable. After attending the Groovy/Grails One Meet-up, and gaining some confidence from hearing the people behind the language, I think that Groovy might be more useful to me. It has the edge on integration with Java right now. What I didn't get was an intro to Grails. Something I'll have to try out.

Another thing that demands attention is GWT. It has a reasonable widget set, fast client development, they've worked on cross-browser and performance issues and browser memory leak issues. Paul blazed the trail, and we managed to hack on a GWT app for a while, getting it all working, including client-to-server serialization. I've not seen anything that puts me off it yet. (Interesting to hear from Google that they think a page load time of 300ms is the threshold for users to be happy. That's the start up time they target.)

Gavin King's Seam demo was just perfect for getting a feel for what Seam is about. The demo nicely contrasted the assumption that EJB is heavy by having a containter that starts up in the IDE in less than a second. He then continued by developing a Seam web application without the need to restart the container once. Another technology worth checking out.

From other sessions...

  • The Hibernate integration with Lucene looks like a no-brainer if you need to add text search to an application.
  • The API for restful web services looks good. I'll take another look when they finish the spec.
  • I was at a session on deploying and scaling massive digital archive repositories. It turned out to be a vendor pitch to go buy a StorageTek 5800, but it was interesting to learn about the APIs they are building into the disk storage, with hints about this becoming an open standard. Forget files. (This could be related to Project Honeycomb.)
  • NASDAQ do 100k transactions/second on the Java platform. They're looking to use Java Real Time. Presumably for predictability.
  • Mark Harriman of Ericsson said they're basing their IMS products on Glassfish. Possibly to be called the Sun Java System Communication Application Server.
  • When running the Glassfish application server, you can write your site in Ruby, Javascript, PHP and Java. Probably other languages too, via the scripting plugin spec.
  • I understand that JChav was mentioned in one session. Fantastic.
  • The only session I was disappointed with was a Jini one. While the language has been changing and going forward, Jini seems (to an outsider like me) relatively unchanged since the first version. Right now to use Jini, you really need to want to use Jini.
  • The Neal Gafter presentation on closures for the languages was great to sit through. Each time I hear about the proposal I understand a little bit more about it.
  • The Effective Java Reloaded session (Josh Bloch again) was superb. Note to self: need to understand wildcard capture in generics, and remember that arrays and generics don't play so well together.

So: quite an emphasis on scripting, language changes and richer UIs.
16 Mar 2007

Sun Tech Days London 2007

Media_httpfarm3static_eyuzr

I caught the middle day of the three-day Sun Tech Days in London. Compared to last year, it was busier and felt to me like there was less in-depth technical content. There was too much "Hello London! Are you ready to rock?" and not enough to get rocking about.

Note to the organizers: yes, maybe we are a tougher crowd than the Japanese, but if your idea of crowd foreplay is just lobbing out a beach ball, you've got to try harder. Also: we need better coffee, make sure all the presentations actually match their titles and descriptions... I could go on... to gauge how unimpressed I was with the whole event I left before the free booze came out. And that's obviously a phrase I never thought I'd have to use.

But I like these events. Forget the coffee, forget the cheerleading, and forgive some of the content. What you get is time to think about the tech with like-minded people, and to look and hear about things you don't normally give yourself the change to think about. It's a good thing, and I'd recommend it to anyone.

For me "London's own" Simon Ritter was the star. I caught his presentation on Java 6 and 7 (PDF) which was useful for the explanation of the bounded wildcard stuff in generics.

His demo on music similarity was the best thing I saw all day. It looks to be the Search inside the Music project and the basic idea is to extract features from music so that you can cluster tracks by similarity. Reading from "A model-based approach to constructing music similarity functions" (which is a good review of the field) it looks like they extract features from MP3 data and then use a learning algorithm to classify a track by genre. What you get out is 10 genre likelihood numbers which you can plot by reducing from 10D to 3D (using MDS, which tries to reduce the dimensions in such as way that points that are close in 10D space are mostly close in 3D space too. I'd expect PCA to work too. In other words, don't ask what the labels on the axis are, because there's no easy way to say). Once plotted you can see the clusters of genre and sub-genre, and then take journeys from one genre to another. Cute.

What else...

  • Netbeans 6 looks like it's getting to be usable day-to-day soon. The profiler looks excellent: so thanks to the guy on the stand who demoed that to us.
  • I learned a little bit about Java DB (a.k.a. Derby, a.k.a Cloudscape): Mostly I learned (PDF) that it's a proper database.
  • I don't think I need to worry about BPEL and JBI.
  • The main features of Java 6 seem to be: Visa L&F; SystemTray stuff; access via Desktop to email, browser; splash screen support; scripting engine; ability to discover available disk space. All dull, but worthy.
  • Java 7 looks to be up for lots of ease-of-development changes: maybe closures, syntax for list and hash creation, properties. Stuff people have been banging on about for ages. Due mid 2008.
  • F3...looks nuts.

The event was held in at Central Hall Westminster. Nice building. It somehow seems right that events like this are held in places of worship.

Media_httpfarm3static_eoqvx
18 Aug 2006

Open Source Java ME

The Rich Green video confirmed that Java ME (CLDC and CDC) are being open sourced. I believe the opening of the Standard Edition was well anticipated, but the inclusion of the Micro Edition is great news. The timing is also impressive: roll out by the end of 2006.

Presumably this will ensure Java is included by more handsets and other devices, including as a default on more Linux-based devices. What happens there is probably determined by the license choice. The intriguing possibly is that ME technology could become common place on device other than phones. Although some will shudder at that idea, remembering back to the original diagrams describing "profiles" and "configurations", it's what the technology was designed for: a configuration, like CLDC, is a set of classes for a range of devices; a profile, such as MIDP, is the collection of APIs for a vertical market.

Apparently "there could be gaps in the Java ME stack that would need to be filled, perhaps by contributions of relevant code from other open-source projects" (IT Week). That's a great opportunity for a small organization like ours to contribute to the source running on the next two billion devices out there.

I assumed that the open sourcing is going to be help during debugging. But then again, maybe not. It depends on who's using Sun's implementations. Symbian, Nokia, IBM, Sony Ericsson and all the other licensees all probably put in a lot of work for the platforms they support. Looking at it the other way round, it's a chance for manufacturers, networks and OS vendors to head-off fragmentation by contributing back to the Sun implementations.

Links:

Richard Dallaway's Posterous

Director at Spiral Arm Ltd. We build stuff using Scala+Lift, offer consulting & create new projects. I live in Brighton, UK.