...and that is why you fail

On the weekend I helped my brother setup a new antenna for his house. Now sure we bought a brand new length of co-ax cable, mast-amp, splitter, connectors and only slightly used (my old) antenna. when I had mounted the antenna and run the first length of cable down from it through the roof, instead of wiring it up straight to the splitter, I first took the whole length, put (temporarily) a connector on the other end and tried it out by hooking it up to his tv. Sure enough only 2 of the 5 channels worked! Lucky I thought, not having yet used the splitter, I now only had 1,2,3, 4!! components to test: antenna, cable, amp, connectors. Yep, even though I thought I was being clever by doing some testing - I was already too late! and infact was trying to test which 1 or more of 4 different components had failed.
To cut a longish story sort, I ended up doing a quick bypass of the masthead amp (which in this case happened to be the most complex component in the system) by virtue of sitting on the roof and holding the 2 pieces of coax together and sure enough the tv started getting all the channels. Now as usual, it was a bit of a painful lesson as it meant I wasted hours and ended up having to climbup onto the roof and ceiling space mutliple times, when of course I could easily have done a unit test of the amp while safely on the ground.
So why didn't I test the amp to start with? Well it was made by someelse who should know what they are doing and I just *assumed* that they tested it thoroughly before giving it to us.
After that, what I ended up doing was testing as much as possible each run of co-ax that I put in, because since even though the components (coax, connectors) were quite simple and not likely to fail, my use of them (I dont do cable runs for a living) was prone to error so deserved testing. But what if I had been unlikly with the cable and had got a batch with a break in it? Well I had my multi-meter with me, so I could easily have done a continuity test as well but of course I didn't since as I said I didn't think about testing all the components individually at the start, but rather half way through, by which time it was even too late to be able to test some components like the cable in isolation.

The moral of this story: version 1.6.1 Helma now comes with the jala unit testing framework out of the box and unless you want to spend a lot of times climbing up and down ladders, I suggest we all start using it.

topic of the week

the need to go cold-turkey for our RDBMS addiction

some think the answer looks a bit like this

I really wish it was as easy as a java lib implementing this.

the downside

I think this guys biggest problem is he has a crap manager, I have a great one and it sure helps!

"Simplicity" aka Maks's guide to Javascript for Java programmers Part I

Javascript (hence forth JS) in its current v1.5 (Ecma262 ) incarnation is a great small, powerful scripting language. A lot of Java developers seem to now be realising this (I'm a convert of about a year now) and given the similarities between the 2 languages might get tripped up.
Also its very easy for new comers to get confuse between what's in the JS language and what is provided by the web browsers that host the JS runtime that most people use JS in (but not all! Actionscript in Flash is a implementation of Ecmascript and you can use JS very nicely on the server-side)
So here is my attempt at slightly damping down the learning curve.

Lets start with the basics:

• Almost everything is an Object.
  
  
• Objects are essentially Maps (i.e. collections of name, value pairs)
  
  
• The name, value pairs are an objects properties.
  
  
• There are NO classes.

How can you have objects without classes? Well do your Java Map objects have a class? No they are just simply a Map, whether you stick an Address, CORBA or String objects in a map its all the same, a Map is a Map and in JS a object is an object.

BUT objects do have a prototype. An objects prototype is really just a special key in the object that points to another object, where the JS interpreter will look for a property requested for the current object but not found in the current object. If this sounds a bit a simple form of inheritance - it is, BUT just don't get carried away and start thinking its the same as inheritance in Java.

• There are only a few built-in data types: Object, Array, Number, String, Function.

If you are paying attention, you would have noticed that the last data type was "Function". That is because functions (ie executing blocks of code) are an actual data type in JS and are in fact objects (remember almost everything is an object). What this means is that you can have functions as stand-alone entities. In Java of course there is not even the concept of a function - the only code you can have is methods which are inextricably bound to a Class and objects of that class.

• Objects have methods
  

Now we're back to more familiar ground. Because you could have an object property that points to a function object. this gives you a function tied closely to an object, aka a method. Again JS methods are simliar to Java ones (they get an implicit "this" reference) but again don't assume they are exactly the same.

• Everything (almost) is an object. Hence there are no "primitives" like Javas int, bool, etc. This means that is if you have a numeric literal, its a literal Number object and you can do stuff like:
  (42.1).toFixed(1)
  


• Objects can have a constructor function.
  

Again this is similar but not ths same as Java.

In fact the truth is that JS is a "new" operator that will create a new "plain" object and then pass it to the function you give it as the argument to "new", calling that function with the newly created object as its context (which just means the newly created object is set to "this" for the function call)
So a line of JS like this:

var me = new Person();

in psuedo code is something like:

make a new object
call a function named "Person" with the "this" reference set to the new object
set the prototype property of the new object to point to the Person function object
assign the new object to the variable "me"

So really there is nothing special about the function Rectangle except that we have written it with a view to using as a constructor but any function object can be called as an argument to the new operator - its just that it only makes sense for those that actually do something useful with their "this" reference - like initialising some properties, eg:

function Rectangle (firstName, lastName) {
this.firstName = firstName;
this.lastName = lastName;
}

• Global variable are really just properties of the Global object

This might sound liek a minor point, but it actually matters alot when we get to discussing scope in a minute because the global object is responsible for the global scope.

• Functions can have local variables.

This might seem like a no brainer but its interesting because local variables are actually properties of a special object called the "call object" which is passed to every function when its called. In fact function parameters are also properties of this call object. Hmm, elegant but nothing special right, well lets get to the next bit...

• Functions have lexical scope.
  

Lexical scope? what the?
Lexical scope just means scope of where functions are defined, not where they are called. Ok that doesn't help much, so maybe this will:

Remember that call object I just talked about? well its actually the start of a "chain" (ie. ordered list) of objects that gets passed to a function to set its scope. So when you try to reference a variable in a function, JS looks through this chain to find the variable you are referring to. So what other objects are in this chain after the call object? Well its all the previous call objects.
So for a function defined in the global scope, its just the Global object.
And for a function called from with another function its just the Global object! That's right, because as I said at the start, lexical scope is about where the function is defined not where its called from. This means that this magic "scope chain" is set-up when the JS interpreter sees the definition of a function and is not changed when a function is actually called somewhere down the track!
Which brings us to:

• Function can be defined within other functions.

Now things can get a bit interesting with lexical scope. Because when you define a function inside another function, the inner function has in it the call object of the containing function. And so now the scop e chain of the inner function has its call object, the containing functions call object and then the global object (and so on if you have mor then 1 level of nesting). And remember this happens at function "definition time" not function execution time.
Well see if this helps:

function a() {
var x = 2;

function b() {
var y = 3;
return y + 2;
}
return b;
}

var func = a();

a = null;

alert("The answer is:"+ func());

Whack that in a html page and what do you think you will get? Yep 5! But stop and think about it for a second - b() is defined inside a() but it uses a property belonging to a when its called in the alert box after a() has already returned and after we set a to null, so it not even defined in the global scope any more !
But if you remember that b()'s scope chain is setup at "definition time" it all makes perfect sense! Since at definition time b() got a()'s call object added to its scope chain (which contains the property/value "x = 2") and it keeps that scope chain forever (or at least til we reload the web page :-)
Now to finish off belabouring this point, keep in mind that the objects in the scope chain (for the most part) are just JS objects. So for instance there's no reason why we can't change the values of their properties, say by using var x as a counter...


function a() {
var x = 0;

function b() {
return x++;
}
return b;
}

var func = a();
a = null;
alert("The first answer is:"+ func());
alert("The second answer is:"+ func());


So essentially var x in the call object of b()'s scope change is a lone "relic" of a()'s existence when the definition of b() was first processed but because all references to a() function object are gone, the x that b() is seeing in its scope chain is its own "private" variable, which brings us to:

• Functions are closures

Which as we now now, simply means functions are not only data, but are "self-contained" keeping the scope they had when they were defined not matter what happens afterwards during the execution of the code.

• Because functions are actually objects they can themselves have properties.

An important example of this is the "special" prototype property I mentioned earlier that all objects get. When a new object is created, its prototype property is set to the prototype property of the "constructor" function object. And the prototype property of a function is usually initialised by JS to a plain, empty object. However you can then set anything you like on it:

function Person() {
...
}

mrJS = new Person("Brendan", "Eich");

Person.fullName = function() { return (this.firstName+" "+this.lastName);}

alert("JS Creator: "+mrJS.fullName());

Yep you read that right, since JS looks up property names it can't find in the current object, in the object assigned to its prototype property (and then if it can't find it there, in its prototype object, and so on) and at the time the property is used, you can assign properties to a prototype and all the objects which reference that object as their prototype will see the new property.

• There are some handy predefined objects, with handy methods: eg. Date, Math

You don't get anything like Java's rich class library in JS, but there are quite a few handy methods in these objects which are defined by default in the global scope.
And since JS is intended to be used as an embedded scripting language its assumed that the host environment (eg, web-browser, web-server) will provide the required objects with lots of useful methods defined.

Well that's about it for quick run through, of course there's lots more and the best place to find it is the JS book, with the big rhino on the front.

(Helma + H2) == "to easy"

So I'd come across H2, the pure java sql rdbms engine written by Thomas Mueller (the original author of hsqldb) and had been menaing to give it a go, but never got past just firing it up and having a few quick clicks in the webbrowser based admin (which was pretty impressive all by itself!)
But then the other day I was working on the windows partition of my laptop and needed a db for a Helma app and couldn't be bothered downloading and installing the current win32 mysql package so decided to see if I could quickly get up and running with H2 instead. I was also thinking it would be good to have a quick and easy DB for a rapid prototyping and in memory db for testing of my helma webapps, so it was all worth spending a little time on and if it didn't work out I could always go through installing mysql.

But boy did it work out. H2 is a fantastic app, with alot of thought put into it. Not only is it easy to get up and running, but its very well documented and has a a lot of nice features such as a batch file that not only starts the dbserver running with web-console enabled, but also puts a little icon in your system tray and opens up a new browser window all ready for you to login!
The experience gets even better once you login with a nice ui that includes sql syntax auto-completion, history AND unbelievably editable tables of results! This puts to shame many of the dedicated java/native sql clients I've used in the past.

And what about using it with helma? (the whole point of the exercise), well that was a piece of cake, just drop the h2.jar into helmas lib/ext folder, restart helma, pop the connection details into my applications db.properties and hey presto!

But you want more? well for just $0 more, you get the full DB *embedded* inside your helma server.
Yep that's right, since H2 has an "embedded" as well as client/server mode, all you need to do is shutdown the H2 server (right click on the systray icon), change the connection URL in your helma apps db.properties from say:

jdbc:h2:tcp://localhost/col

TO:

jdbc:h2:/data/h2/col

and bam! your H2 db is now running in the same JVM process as helma (very handy on a cheap, strapped for RAM webserver for example).

But wait there's more! Yes how would you like to still have that nice web interface into the embedded DB? not possible you say, since now H2 is not running in a separate JVM - but in fact the H2 docs tell you exactly what to do[1], you just need to fire up H2's webserver when your app starts, so in your apps Global folder:
functions.js:

function onStart() {
 writeln("H2 web server started by"+app.name);
 app.data.h2WebServer = Packages.org.h2.tools.Server.createWebServer(null);
 app.data.h2WebServer.start();
}

function onStop() {
 writeln("H2 web server stopped by"+app.name);
 app.data.h2WebServer.stop();
}

Then in your browser, goto http://localhost:8082/ and there's H2's web console waiting for you admin-ing pleasure!

If you've got more then 1 app in your helma server, you won't want to start a new h2 webserver for each one, so in that case just make a new empty helma app, call it maybe h2admin, pop the above code in there and your done[2].

Now that's what I call RAD!
I wonder if the rails crowd will even believe me without a funky screencast demo...

[1] Well actually the H2 docs tell you about how to fire up a DB in a servlet to be accessed by multiple webapps in the same tomcat container, but with a bit of lateral thinking and a quick look at h2.bat I got the idea.

[2] Of course this way H2 is running its own webserver alongside helma's one and I a nice future project would be to 'port' the admin web ui to be a helma app :-)

Update: I'm now confident (crazy?) enough to use this on the test server, but by default the H2 web console won't accept connections for machines other than localhost, so to get this to work you need to use this code instead:

function onStart() {
   writeln("H2 web server started by "+app.name);
   var args = java.lang.reflect.Array.newInstance(java.lang.String, 2);
   args[0] = "-webAllowOthers";
   args[1] = "true";
   app.data.h2WebServer = Packages.org.h2.tools.Server.createWebServer(args);
   app.data.h2WebServer.start();
}

Maks' guide to LDAP for Busy Java developers

So maybe like me, you're a busy java webapp developer (isn't everybody these days?) who just got told/decided for yourself that instead of creating yet another app for which users need to remember a username/password pair, that you will instead do the Right Thing and connect up to your company/organisations central LDAP (or ActiveDirectory) .
And (again like me) you might even need to have cause to setup your own LDAP server either for testing or some other reasons you can't use the existing AD setup in your org.
So instead of replicating the tedious hours of frustration that I've spent - let me give you a quick run down of the pertinent facts you need to get up and running.

First the obligatory intro to LDAP:
its basically a tree of objects. Each object belongs to 1 or more "classes". Classes are just sets of compulsory and optional attributes. Attributes a just name-value pairs (think java properties files).

So getting a bit more specific, if you have a person class that has the compulsory attributes "firstname" and "lastname" and then another class called orgPerson that has a compulsory attribute "email" then you can have an object that has "belongs" to both classes and so will always have a firstname, lastname and email.
eg.

firstname: homer
lastname: simpson
email: hsimpson@springfield.com

Of course the above classes (and attributes) are made up, but are pretty close to the real thing, as in LDAP you can have any class you like, all you need to do is define a schema for it (just like with relational DBs really). Handily, most LDAP server implementations seem to come with a whole bunch of predefined classes (not sure if they are actually a standard or just convention) for users, groups, etc.

Now the whole tree bit is just as easy, its basically just made up of objects (usually belonging to specific 'container' style classes) forming, well, a tree. Now 'container' style classes are ones such as "o" (organisation), "ou" (organizational unit), etc. There is also "dc" which is normally used when creating a tree based on DNS names (eg. www.springfield.com)

Now the only slightly wierd thing about LDAP is the way the naming syntax works, its all about commas. So to say the homer belongs to the staff group at the springfield site of the bruns power company you would write:
uid=homer, ou=users, ou=springfield, o=burnspower

this is assuming that the homer object has am attribute called "uid" that identifies it (its "Relative Distinguished Name" in LDAP speak) and that you've already defined the "users", "springfield" and "burnspower" org units and org respectively. And how would you do that, well you could use a LDAP GUI tool to connect to your LDAP server or write ldif files (which are basically serialised text versions of parts of the LDAP directory tree) to define them,
eg.

dn: o=burnspower
objectClass: top
objectClass: organization
o: burnspower

dn: ou=springfield,o=burnspower
objectClass: top
objectClass: organizationalUnit
ou: springfield

dn: ou=users, ou=springfield,o=burnspower
objectClass: top
objectClass: organizationalUnit
ou: users

As you can see, you just define each object one after the other, and with the "dn" (Distinguished Name") you set the "path" of the object in the tree, just like you would if you were creating a series of nested folders on your PCs filesystem.

Now this brings us nicely to doing some real work, like actually setting up the LDAP server. Basically while there are a few different commercial and open source LDAP servers out there, basically ignore all the rest and use OpenLDAP. Even though it might me slow and chunky, it is the lest painful way to go if you've just got a linux box and not much time or patience.
Often OpenLDAP will be part of the default OS install or just a few clicks/commands away using your package manager of choice.
Once you do have it installed and configured (for OpenLDAP usually just means setting a couple of options in the /etc/openldap/slapd.conf), use something like the above ldif files to set up your basic directory structure and then you can just go start plonking your users in there (again using wither ldif files or via GUI) . The main thing you need to decide is what class(es) you want to use for your user objects - for myself I've found inetOrgPerson had all the attributes I needed.
eg.

dn: uid=homer ou=users, ou=springfield, o=burnspower
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: top
cn: Homer Simpson
displayName: HomerSimpson
givenName: Homer
mail: homer.simpson@burnspower.com
mobile: 122 333 4567
sn: Simpson
uid: homer
userPassword:: adlr0912384nq4o358ndfadasl

Note1: the user has both the inetOrgPerson and organizationalPerson classes in a kinda inheritance way to pickup all the attributes we want to use.

Note2: the above userPassword attribute is how my gui ldap browser exported the ldif file, but I suspect that the correct way to specify them (for OpenLDAP) at least is:
{CRYPT}AF387HED3SDA

Note3: all the above password binaries are just random typing on my part and so dont match :-)

Setting up groups is also pretty straight forward, the 2 handy classes you want are groupOfNames and groupOfUniqueNames. Note that both have attributes which let you specify the members of the group by having multiple instances of that attribute,
eg.

dn: cn= Managers,ou=groups,ou=springfield,o=burnspower
objectClass: groupOfUniqueNames
objectClass: top
cn: Managers
description: the bosses
uniqueMember: uid=monty, ou=users, ou=springfield, o=burnspower
uniqueMember: uid=smithers, ou=users, ou=springfield, o=burnspower

And that about covers it from the LDAP side of things.

Now we can actually get to the java side and setup a single authentication service.
Basically LDAP has lots of different ways to do authentication, most of which seem very complicated and are definitely overkill my mine and most small installs, so I'll just stick to the simple ones:

Method 1.
First of you need to know that in LDAP-speak logging into to an LDAP server is called 'binding'. When you "bind" to the server, you can do it as a particular user as long as you specify the users "DN" as the login.
You also give the password and the LDAP server goes looking for the password attribute for the object given you specified in the DN (eg. uid=homer ou=users, ou=springfield, o=burnspower) and if the bind is successful you know the user is authenticated and hooray, you're done! Once you have bound to the server, you can then also grab all the users details (email, lastname, whatever) using the supplied DN to lookup the user object for yourself.

Method 2.
Another way to do it is to use an "admin" users login to bind to the LDAP server, fetch the user object for the given DN and then compare the password attribute in your app code. You might want to do this for instance if you need to check which groups a user belongs to but the user themselves dont have permission to access the groups part of the directory tree (I forget to mention that you can set permissions for users to read/write (or not) any part of the tree - again just like you do with a filesystem). So you bind to the LDAP server as an 'admin' user which does have permission to go looking up the groups (or whatever) part of the directory tree.

Security Side Note:
One thing should point at this stage is that while the LDAP protocol itself is binary (bit wierd to find that out in this age of internet text protocols - at least its defined in some RFCs) its still all sent in the clear, so the usual modius operandi is to do all the LDAP connections over SSL so that people don't go snooping on your passwords in the net traffic.

Anyway, now that we've covered the connecting to server and authenticating theory, the actual java code (for method 1) is a walk in the park:

 
     public boolean authenticate(String username, String password) {

     String userSuffix = ",ou=users,ou=springfield ,o=burnspower";
     String userRDNPrefix = "uid=";
 
     String hostname = "ldap.burnspower.com";
      String port = "389";

     String userRDN = userRDNPrefix+username;
 

     Hashtable env = new Hashtable();
     env.put(Context.INITIAL_CONTEXT_FACTORY,
         "com.sun.jndi.ldap.LdapCtxFactory");
     env.put(Context.PROVIDER_URL, "ldap://"+hostname+":"+port+"/");

     env.put(Context.SECURITY_AUTHENTICATION, "simple");
     env.put(Context.SECURITY_PRINCIPAL, userRDN+userSuffix);
     env.put(Context.SECURITY_CREDENTIALS, password);

     // Create the initial context
     try {
         DirContext ctx = new InitialDirContext(env);
     } catch (AuthenticationException ae) {
         System.out.println("Invalid Login");
         return false;
     } catch (NamingException e) {
         e.printStackTrace();
         return false;
     }
     return true;
 }

And voila! one less set of passswords for your users to remember.

top 10 reasons I dont use rails

So after a very nice dinner catching up with an old friend, I got asked why don't I use rails for my next webapp project. I replied about how I keep meaning to learn rails (and even bought a book to read) but that I'm more comfortable sticking to helma (the javascript-based app server framework I'm using).
But this got me thinking, apart from my inherent laziness, why aren't I using rails?
Well heres my top 10 reasons after a bit more thinking about it:

1. helma is implemented on top of a standard java servlet container (Jetty) so I get to reuse all my existing j2ee knowledge/experience and techniques (eg. mod_jk connector to apache)
   
2. it uses standard java technologies (eg. JDBC) so again same as above
3. it can directly and very easily use any java library and at this point, its arguable java has surpassed even C as the library lingua-franca, especially for things you do for webdev
4. Using Java libraries means if something goes wrong I get a java exception trace which is helpful (since I mainly use oss java libs which come with source) unlike the less helpful: "Segmentation Fault" that you would get with using C libs with rails
5. Another thing about being able to use Java libs is that its really easy to write my own, wrap up/modify existing ones, etc. which I definitely can't do with C libs (my C coding days are a long way behind me)
   
6. I only need learn 1 new language for both the client and server side programming tasks. Its arguable the Ruby is a nicer/better lang then JS but JSs simplicity really appeals to me (warts & all)
7. While they are hardly alike (except for having a common first name) I find javascript syntax at least close enough to javas' to feel quite comfortable and not get a mental 'jarring' that happens every time I try to read ruby code
   
8. JS is also becoming a very popular language and while ruby isn't exactly hard to learn for java programmers, JS is certainly better for me at my current work where the only other tech people are web designers with only JS or AS (actionscript) knowledge
9. performance with helma is excellent, especially with jetty and the rhino JS engine that uses "under the hood" and the memory footprint is excellant too. Now I've only heard here say about rails (rubys) poor performance but thats enough to put me off, especially since with helma I know I can always fall back on using standard java profiling tools and optimising the frameworks underlying code (got know chance to do that with rails ruby interpreter written in C)
10. Even though its not the "out of the box" templating system, I can use Velocity templates with helma in a fairly nice way (and its such a simple integration that I've taken over maintaining it from the original author and have stated adding some small enhancments of my own). As far as I understand, rails goes down the whole nasty "code in pages" road (ala JSP) and that leads to no place I want to be.
    

well they aren't really in any particular order of importance but for me they are pretty compelling reasons to stick with helma and only read a rails book in my (very limited) spare time.
now there are downside compared wit what I already know about rails:

• no scaffolding
  
• no builtin unit testing
• no builtin support for db migrations

but so far none of these are deal breakser for me and in fact I'm slowly working on adding those features to helma myself.
And another one of the down sides to using helma vs rails is the huge community (comprised of a fair few j2ee refugees) that has sprung up around rails. I certainly don't feel the need for that with helma but I think it would be nice not to be off on my own little island - so I'm finally going to set aside a bit of time and write up a short series of helma "intro" articles here - which if they work out ok, might even form the basis of a nice little book: 'server side js: web development with helma" :-)