More on simple databases

When I think about it some more, the difference between something like MS SQL Server or DB2 vs Hibernate + MySQL with MyISAM, is that the traditional way is for the RDBMS to be a single system, relational management AND static data storage. The persistence layer/db strategy is separating the RMS (relational management system) from the DB (static storage). By separating out the relational management portion, and moving that into the persistence layer, you have the ability to run one processing cluster which serves your application and handles the persistence layer/rms, as opposed to running two clusters, one to serve your application, and one to handle the traditional RDBMS. I would think that combining your servers into one larger cluster to handle more of the computing would be more efficient than splitting it into two clusters, in which case one cluster may hit max load while the other sits idle.

By the way, I know that there are measurable differences between MyISAM, InnoDB, or other storage engines, like row level locking as opposed to table locking on updates.

How I'm learning to love simple databases

Once upon a time I was a total believer in strict database rules. Everything should be fully normalized, schema and diagrams thought out ahead of time, strict data validation at the database level, so on and so forth. A total believer in the primacy of MS SQL Server, DB2, so on and so forth. Within the last 6 months, I've had a change of mind that's been brewing, and only lately have I more fully realized it.

When you're developing a webapp nowadays, chances are you have various frameworks and technologies that you use, each one layered on top of the other. MySQL -> Hibernate -> Java application -> JSP/Servlets -> HTML/markup, as one example. The data that you accept in from your users must pass through each layer to finally get to your storage, whether it's in memory at the application layer, or static storage at the db. Each layer has its own data validation. So if you have a date field that a user fills out, it comes in as a string, which the Servlet can accept in and convert to a valid Java date, which passes it to your application layer, in which you can put that data in an object that checks that it's a valid date (if it's defined as a Date object). Then it goes to your persistence layer, which in our example, Hibernate would check that it's a valid date. It finally gets passed to your database layer, which in our example, MySQL would check that it's a valid date. All that work to verify that it's a date several times. Why? What's the point? If you checked that it's a valid date once, and you don't do any measurable transformation of the data, there is no need to check it again.

If I think about it, when is it exactly, that I care that I check if it's a valid date? When I accept data in from the user, when I do fancy things at the application layer like date/time arithmetic, and when I display data back to the user. None of that touches the database layer. Do I care if the static data is stored in a traditional RDBMS, a bunch of XML files, or a straight up B tree like GT.M? Other than the possible performance gains by storing static data one way or another, it doesn't matter.

I currently use Wicket, and Wicket has easy to use form validation built into the framework itself, which covers all the cases in which I would care to check if the data is appropriate or not. Right now, I do datatype management at each layer because that's how I was taught, but I'm seriously considering switching everything underneath Wicket to string, integer, or float, crossing my fingers, and hoping nothing blows up.

Beyond data validation, what about primary/foreign keys? Many databases don't even have pk/fk relationships built in. Originally, I thought those databases were mad (in a bad way). However, if you think about it, the business logic which says that one table must relate to another table via a field based relationship, that business logic has to exist in your RDBMS, your persistence layer, and your application (because you must pass data with the appropriate relationships to the layer below). Why should each layer check the relational integrity of the data? The only case I can think of, is to protect myself from making a mistake at the application layer. Fine then, I can set those rules in the persistence layer as well as my application, and the database can be simply a static container, nothing more.

I Heart Automator

Every time I wish there were some way I can automate something in OSX, I find out there is a way, through the nifty scripting utility Automator.

I had a particular problem I wanted to solve, which was to add spotlight comments to multiple items. This way I can do dynamic searching of files based on keywords instead of creating an entire folder hierarchy and having to drill down to get to a specified file.

And here's my Automator plugin in action, accessible right from the finder menu. Slick!

MySQL Auto Data Conversion

So I've been researching MySQL, and I have run into something that kinda boggles my mind. Whenever you give MySQL the wrong datatype for a field, MySQL attempts to do autoconversion of your data. This in itself isn't a bad thing, and some of them may make sense, such as doing string to int or int to string. However, if you pass MySQL data that it is out of bounds, MySQL chooses data that is in bounds, and places that data in the field instead. So if you have a field defined as a tinyint, whose bounds are -127 to 127, and you pass it 1,000, MySQL will input 127, and consider it a success. 

You have to write your own MySQL specific data validation in your application BEFORE you pass it to MySQL..... Say it with me..... WHAT??? The whole idea with abstraction and error catching is that each layer of your software stack should be able to pass "something" to another layer and receive back success or failure, at which point the originating layer can decide how to handle it. By having your application layer have to do database validation checks BEFORE passing data to MySQL, you loose the benefits of abstraction, and by definition, you loose the benefits of modern software engineering. 

Somebody out there explain to my how MySQL's handling of out of bounds data is a good idea, cause I don't get it.

Keynote 2008

So I've just done my first presentation with Keynote 2008, and color me impressed. When you compare iWork vs MS Office, I consider Excel better than Numbers, and Word better than Pages. Keynote however, is far better than Powerpoint. Both programs have a practically identical purpose and workflow, but where Keynote excels is in the quality of the default templates, animations, and graphics. Basically those things that taken separately, don't matter much, but taken as a whole, it makes Keynote the superior presentation building application.

I wonder if OSX 10.5.7 will be included in Mac Box Set Family Pack. If so, I may have to pick that up so that I can iWork 2009, iLife 2009, and OSX 10.5.7

Automator for OSX

I've begun using a software VPN utility called Hamachi, owned by the guys at logmein (every family IT dude's friend). Alas, there is a slick Windows client, but the OSX one is pure command line. I'm totally fine with that, but I figure "let's see if I can make it all OSeXy". So I created a bash script to execute all the Hamachi commands, and I combined that with some Applescript and OSX's Automator, which is kind of a wysiwig script editor. I could have done it all in bash or Applescript, but I wanted to put my script on the dock and run it from there. Only Automator "executables" can be moved directly to the dock.

Here's a sample from my automator window. Note how cool it looks to have a wysiwig script editor!

So the end result is I have a clickable "thing" on my dock which will start up or quit the OSX Hamachi client. I also added in a Growl notification for fun, which is shown below.

Evernote

Evernote is one of a long line of notetaking applications, from Microsoft OneNote (which is supposedly pretty good) to Zoho. In my search for a notetaking application to supplement my brain (which has horrible short term memory), I settled on Evernote as the most full featured of the bunch:

• Has a desktop AND web AND iPhone client
• For notebooks that are synced to Evernote's servers, free OCR
• The ability to have synced notebooks (to take advantage of web/iPhone clients and free OCR), or desktop only notebooks (for sensitive info)
• The ability to add pics, pdf, word, excel, numbers, pages, webpages, or free text to a note
• quick searchability

The only thing remaining for me is to put my local Evernote notebooks into my truecrypt partition, then I'd be all set