Gulliver, as the system, washes ashore to be tamed by the custodians.

Warning, this is a an article by a software geek for said geeks. Rapid boredom will probably ensue even for the target audience. Most of us at one time or another have worked on a Big Ball of Mud. For a less seasoned hacker the first tour or two of such a haphazard system may go unnoticed due to a number of circumstances the least of which being naivety. But, over time, the imprints of sloppy, duct tape, spaghetti code jungles come to be recognizable. By no means complete a few standout examples from my career:

1. Code is not stored in an SCM.
2. Bug fires rage 24/7 consuming all resources perpetually.
3. Poor code abounds with total disregard for any standards (e.g., wheel is reinvented regularly, 10 screen shot methods, classes with 100′s of members, etc.).
4. Duplicate data and functionality throughout including “clone” systems.
5. Management, via ignorance, insists things are basal or worse are in complete denial.

Once one can identify a subset of ailments it’s tempting to run from these helter-skelter systems in haste. Knowledge arms us not to repeat them – maybe. Yet Foote and Yoder make clear in BBOM many profitable such systems exist. Profitability aside the fact is they abound! Having designed, developed, and maintained such systems myself the temptation is to disparage. What is it about these hastily crafted boomtown’s that irk us? Is it the simple fact that the architects and custodians are all to comfortable with a level of expediency we are uncomfortable with?

I am poorly equipped to answer from my limited exposure. Both BBOMs I worked on were absolutely driven by a perceived fleeting window of business opportunity. Design marathons, repositories, code standards, pair coding, QC, test harnesses, continuous integration, etc. where all pushed aside in the rush to launch. Once there and stably afloat i.e.,  profitable, did the realization often slowly set in that the system was ineptly fashioned. Once built management forever demands hastily cobbled features to increase ROI almost always at the cost of sane design – the “structure” worsens. Of course, as engineers we fight to avert such quagmires whether through evangelization and/or example.  However, we all have to eat – to deny human nature is foolhardy.

An inverse universe exists where architecture, means and process are king. Laugh if you will but it’s well known that Microsoft runs one of the most process driven engineering shops the world over. For them sheer scale dictates a process rooted approach. However, for the average web business outlet day to day operation is about finding the middle ground between launch mentality and design nirvana.

So, whose world order should prevail in software engineering? The one that works.

Quotes Collection, a WordPress plugin, looked great but after playing with it for five minutes warts appeared. For starters it smartly used a database table, but there were no tools baked in for bulk import.  Secondly, the plugin UI was badly munging anything I bothered to manually insert into the database. However, the plugin provided an easy to use and feature rich PHP function for dropping the quotes into HTML. I decided to keep the plugin for displaying the quotes and hack a perl script to do the bulk database quote insert.

Fortunately, the quotes themselves were easy enough to obtain and due to a lapsed copyright available for the taking. After copy and pasting them into vim and staring at one line for nearly three minutes fatigue settled in. I made a few conceptual mental notes for the morrow and went to bed.

Early today I finished the edit of the quotes file, ttop.dat, in a vim editor. The tedium of manually stripping poor double quote characters, blank lines, and extra spaces raced by with a rested mind. I settled on using ‘—’ as a delimiter between all quotes as most contained multiple lines knowing that perl’s input record separator would permit parsing pliability. Once the quote file edit was complete I hacked up the perl script to slurp and parse the file and insert each quote to the database. After a few trail runs to ferret out the bugs the quotes were neatly tucked away in the database.

#!/usr/bin/perl

use strict;
use warnings;

use DBI;

# insert the new uirs into the db
sub db_insert_quotes {
    my $quote_sref = shift;
    my $dbh = shift;

    my $sql = "INSERT INTO wp_quotescollection (quote, author, source, tags, visible, time_added, time_updated)
               VALUES (?, 'Geoffrey James', 'The Tao Of Programming', 'TTOP', 'yes', NOW(), NOW())";
    my $sth = $$dbh->prepare($sql);
    $sth->bind_param(1, $$quote_sref);
    $sth->execute;
}

# Connect to the database.
my $dbh = DBI->connect("DBI:mysql:database=DATABASE;host=localhost",
                       "USER", "PASSWORD",
                       {'RaiseError' => 1});

$/ = "---\n";
while (<>) {
    chomp;
    chop;
    my $quote = $_;
    #db_insert_quotes(\$quote, \$dbh);
}

exit;

A few months back I vaguely recalled reading a jQuery article with a great simple sliding panel demo. After grabbing a few images and applying a dollop or two of CSS, Javascript, and HTML my panel was functional. Kuler‘s color insights came to the rescue again saving site visitors from my train wreck artistic spasms.

This site enhancement was a joy to work on. I hope visitors are enthused about it as I am.

Set up with a 16G SDHC memory card and a decent B+W lens filter I spent my first day poking around for D90 learning tools and culled the following list:

• Nikon D90 manual
• Nikon D90 Digitutor videos
• Ken Rockwell’s Nikon D90 Users’s Guide

This probably sounds trite but reading parts of the manual, especially the camera body layout and control panel diagrams, taught me a great deal. After shooting some pictures around the apartment I drove to one of my favorite outdoor haunts, Eldorado Canyon State Park, for an afternoon of milling around the trails with an itchy trigger finger.

Fowler Trail Hikers

One dream begins as another ends

Design Patterns Front Cover

The book arrived two days ago and I have been busy poring over it in my spare time. I am using a study guide that groups the patterns by difficulty ranging from easy to advanced. Can’t wait to finish reading and possibly start applying these new nuggets of knowledge to my current AJAX charting application project.

• B = Brown (or Black) Bleeding
• C = Cloudy (white)
• C/K = Cloudy/Clear
• G = Gummy (gluey)
• K = Clear
• L = Lubricative
• P = Pasty (creamy)
• Y = Yellow (even pale yellow)

Each code can be selected once with any other combination of codes.  Some examples of possible code string combinations with dash separator(s):

1. B – C
2. C/K – K – P – Y
3. K – P – Y – C/K
4. L – L – Y Note, not valid due to duplicate L code

Eventually, these code strings are stored in a RDBMS.  The question is how to store them efficiently? We could store them as a string datatype (e.g.,  VARCHAR), but this would be inefficient because on average a code string would take up more bytes then a number datatype (e.g., INT) used to represent the code string.  This implies a look up table of code strings with related numbers and a foreign key on the main storage table(s) referentially constrained to the look up table.

Entity Relationship Diagram

Further, consider the third string code example above. Semantically it’s the same as example two, but is ordered differently (AKA permutation). To properly normalize the data we should alphabetically arrange the permutations so that order does not matter which will vastly reduce the number of possible string code combinations.

Now that we know how the data should be modeled we must populate the string code look up table. The first obvious question is how many combinations are possible when order doesn’t matter? Use the formula below where n is the number of things to choose from, and you choose r of them (No repetition, order doesn’t matter). ! means factorial.

For example, for the eight possible codes if we select four then there are 70 possible code combinations where order doesn’t matter.  However, we need to know the number of non repeating combinations possible for our set of eight codes given we can select between one to eight codes. I’ll leave that simple exercise as reader homework, but excluding the possibility of selecting no codes the answer is 255.

Now the crux of the matter. Insert the 255 non repeating combinations, alphabetically arranged, with a unique number for each into the RDBMS lookup table. Assuming you managed the math part and have made it this far you can now go and gently beat your head against a wall for a few minutes – then return, please.  Perl to the rescue! With it’s ease of extensions via CPAN modules and high level programming abstractions it is the ideal glue language to tackle what looks to be a formidable problem.

Our first order is to google CPAN and see what, if any, perl modules are available to deal with math combinations.  Ah, the Math::Combinatorics module should fit the bill nicely. Second, we need to install the Math::Combinatorics module locally. While we are at we also need to install DBI and DBD::mysql modules to enable perl access to our MySQL RDBMS so we can insert the combinations into our look up table. Most default perl installs come with a handy tool called cpan which gives command line access to the CPAN online module repository to ease searching and installing locally.

joecrotty@macpro:~ $ sudo cpan
Terminal does not support AddHistory.

cpan shell -- CPAN exploration and modules installation (v1.9402)
Enter 'h' for help.

cpan[1]> install Math::Combinatorics
...

cpan[2]> q
Terminal does not support GetHistory.
Lockfile removed.

With the modules in place our finished perl script is below. Note, the actual database inserts are commented out on lines 29 and 30. String_codes.code_id is an AUTO_INCREMENT column.

#!/opt/local/bin/perl

use strict;
use warnings;

use Math::Combinatorics;
use DBI;

$| = 1; # disable buffering
my @codes = ('B', 'C', 'C/K', 'G',
             'K', 'L', 'P', 'Y');

# Connect to the database.
my $dbh = DBI->connect("DBI:mysql:database=DB;host=HOST",
                       "USER", "PASSWD",
                      {'RaiseError' => 1});
my $sql = 'INSERT INTO string_codes (codes)
           VALUES (?)';
my $sth = $dbh->prepare($sql);

for ( 1 .. 8 ) {
    my $combinat = Math::Combinatorics->new(count => $_,
                                             data => [@codes]);

    while(my @combo = $combinat->next_combination){
        @combo = sort @combo;
        my $combi = join('-', @combo);
        print $combi, "\n";
        #$sth->bind_param(1, $combi);
        #$sth->execute;
    }
}

1. ‘Featured Posts’ widget images in the sidebar now link directly to the post.
2. Padding between all the sidebar widgets (e.g., ‘Featured Posts’, ‘Recent Posts’, etc.) is now even.
3. ‘Archives’ sidebar widget is now collapsible. Note, there is a bug in IE6-8 and I am not sure when I’ll have time to fix it. My recommendation is to switch your browser to Firefox, Chrome, or Safari.

Bueno is a simple modern theme that puts the focus squarely on structure and typography that fits well with my general post style. The only theme switch  hassle was resizing all of the larger images which is typical. The only old design element lost was the roll up archives which I’ll make an effort to reinstate at a later date when I tidy up this new theme.  For now we are ready to roll!

1. Header logo and binary code which adds a cleaner more professional feel.  I plan to add a maltese cross, but ran out of time on this effort.
2. Better search box.
3. Subtle post header meta data changes.
4. Archive posts in the right sidebar are rolled up by year and counts listed for each month.

Shot me an email or drop a comment if you find any bugs or simply can’t stand it!

Website Speed

Recently a web client asked me if I could figure out why their page was loading so slowly. Simple enough question on the surface. Like any sensible web engineer I immediately opened Firefox enabled the Firebug plug-in and went to Net Panel. It was clearly evident after poking around for a few minutes that the WordPress site in question, which I built but didn’t do the design and image work for, was sized at just over a megabyte of which ~50% was images. In the scramble to get the site up and running I just took the images the designer handed me and plugged them in without any further thought then “Hey, those do look very good!” I informed the client that the images were hogging up a good bit of bandwidth and they should have a graphics expert put the  images on a diet via PhotoShop.  However, that other 50% of overall slowness that was not related to the images nagged at me.

Within the hour I had mentally resolved to figure out why the rest of the site was slow and apply anything learned to improving current and future site builds. My basic game plan after googling for a few hours and discovering YSlow was as follows:

1. Serve the HTML gzip’ed and cached
2. Serve the CSS/JS gzip’ed and cached
3. Reduce the number of overall requests by combining CSS/JS files where possible
4. Add Expires Headers
5. Move as many images as possible externally

I know YSlow has some other recommendations, but the above are the really low fruit. With WordPress SuperCache enabled the first one is a no brainer.

The second one also seems like a no brainer – just enable apache’s mod_deflate and be done with it.  Ah, but wait a minute.  What if your host doesn’t have mod_deflate enabled?  Even if they do have mod_deflate enabled you could just be pushing your servers CPU and memory to the limit by gziping everything on every request.  Some argue to use a PHP script in conjunction with .htaccess to gzip on the fly.  Again, your just straining the server needlessly all the time. The solution I settled on is to minify and gzip the CSS/JS files on the webserver into psuedo .gz extensions (e.g., *.css = *.css.cgz and *.js = *.js.jgz) with the following script:

#!/bin/sh

# minify and gzip css and js files, Apache .htaccess
# will deliver them to the client encoded

# WP core css files are not minified. All WP core css files and theme files
# are minified and then gziped.  WP core js files are all already minified.  Some of
# the theme files are also minified, but a few are not.  Experiments show that
# minifying theme files caused rendering problems and gave very little
# optimization, so we don't bother to minify them.

yuicomp='yuicompressor-2.4.2'

# gzip all css files
for i in `find . -type f -name "*.css"`; do
echo "minifying and gziping $i"
java -jar $HOME/java/${yuicomp}/build/${yuicomp}.jar --nomunge $i > ${i%.*}.min.css
gzip -c ${i%.*}.min.css > ${i}.cgz
rm -f ${i%.*}.min.css
done

# gzip all js files
for i in `find . -type f -name "*.js"`; do
echo "gziping $i"
gzip -c $i > ${i}.jgz
done

Note, WordPress core JS files are already minified so you only need to gzip.  Further, be careful minifying JS files as there are sometimes unintended consequences.  Given the performance and headache trade-off I only gzip JS and don’t bother to minify.

Then add the following to .htaccess file:

RewriteCond %{HTTP:Accept-Encoding} gzip
 
# serve gzipped css
RewriteCond %{REQUEST_FILENAME}.cgz -f
RewriteRule (.*)\.css$ $1\.css.cgz [L]
AddType "text/css" .css.cgz
AddEncoding gzip .cgz
 
# serve gzipped js
RewriteCond %{REQUEST_FILENAME}.jgz -f
RewriteRule (.*)\.js$ $1\.js.jgz [L]
AddType "text/javascript" .js.jgz
AddEncoding gzip .jgz

Third item turns out to be a real hassle in WordPress.  WordPress is a great CMS that offers a great environment for themes and a diverse selection of plugins. But the price is an architecture where components are spread about a large file tree in a fashion that makes it nearly impossible to combine CSS/JS files without threatening to break when WordPress core, plugin, and theme upgrades roll in which is frequently. I didn’t bother as it just seemed too dangerous to even think about touching with the proverbial ten foot pole.

Number four is another apache related issue.  Either you have, or can have via your host, mod_expire enabled or you don’t.  If you do then it’s just as easy as the following .htaccess line to get the ball rolling:

ExpiresActive On
ExpiresDefault "access plus 1 week"

Remember, though, this assumes your server controls the request.  If you have a large image file on your site that loads frequently and you moved it to Amazon S3 to reduce you load time and bandwidth (IMHO – probably a good move most of time) chances are the external host will not expire the image at all.

Lastly, try to move your big images to an external host like PhotoBucket. Let PhotoBucket deal with the overhead and if you need to pay them for a Pro account. It’s worth a mislay $20/year so that your loyal online readers don’t have to suffer through frequent slow page loads.

Speaking of loyal readers I have applied the lessons from this post to the site and it’s definitively faster!