I purposed we spec a data base (AKA DB) table and code out a change to an upstream process that feed the troubled process so we could start gathering some basic metrics: filename, hostname, size_bytes, and count_msgs.

I hastily began cobbling the idea together and about a day and a half in it dawned on me in a quite moment – the initial design was inept.

For starters the DB table would only store data for one process. But the message data being parsed was part of a broader data pipeline.  The make up of the pipeline was a bit vague like the developing wave below, but a it has the hallmarks of an ETL system.

The solution would need a few tweaks:

1. Three data base tables – note: host table already exists
   1. msg_master
      1. file_id
      2. file_name
      3. file_dt
      4. start_dt
      5. end_dt
      6. created_dt
      7. updated_dt
   2. msg_proc_metrics
      1. proc_id
      2. host_id
      3. file_id
      4. size_bytes
      5. count_msgs
      6. runtime
      7. version
      8. created_dt
   3. msg_procs
      1. proc_id
      2. procname
2. Directory for all processes in the ETL system to write metrics files to.
3. A script, cron scheduled, to load metrics files to the data base.

The new DB model would enable any process to store metrics and normalizes fairly well. Writing metrics to JSON files in a directory insulates  ETL processes from DB latency. I am a big fan of JSON as the data is more portable and readable with a size cost.  However, were only dealing with ~25,000 records/day which is ideal for JSON.  Avro would be a good data serialization solution if the records numbers were a few order of magnitude larger.

With a better functional framework designed  the bit rot front line troops may be getting some better weapons soon.

1. Monitoring
2. Alerting
3. Performance Analysis
4. Trending
5. Quantitative Analysis

1. repo – machine that will hold the master remote repository
2. local – local machine that will hold the checked out repository

devlord is in the git group, but devserf is not. Assuming we already have a code base on local let’s get that checked into a new git repository.

cd /var/www/myproj git init

Now configure repo machine.

su - root<br />adduser git<br />mkdir -p /var/git<br />chmod 775 /var/git<br />su git<br />cd /var/git<br />mkdir -p myproj<br />cd myproj<br />git init --bare --shared=0664<br />

The –bare tells git that it will only hold file deltas and binary information for future checkout.  It’s the master repository not an actual working repository.

Let’s head back to local and finish up getting our myproj code base into the master repository on repo.

cd /var/www/myproj

git remote add origin ssh://devlord@repo/var/git/myproj

git push origin master

I don’t fully understand origin yet, but it’s akin to a local branch and the master is elsewhere – in this case on repo. The second command pushed local myproj over to the master myproj repo.

Now devserf can checkout a copy of myproj on local.

cd /home/devserf

git clone ssh://devserf@repo/var/git/myproj

However, if devserf modifies myproj and wants to push their changes back to master myproj at repo it will fail because they are not in the git group on repo.  Only users in the git group on repo, like devlord, are allowed to make commits back to the master.

Gulliver, as the system, washes ashore to be tamed by the custodians.

Warning, this is a an article by a software geek for said geeks. Rapid boredom will probably ensue even for the target audience. Most of us at one time or another have worked on a Big Ball of Mud. For a less seasoned hacker the first tour or two of such a haphazard system may go unnoticed due to a number of circumstances the least of which being naivety. But, over time, the imprints of sloppy, duct tape, spaghetti code jungles come to be recognizable. By no means complete a few standout examples from my career:

1. Code is not stored in an SCM.
2. Bug fires rage 24/7 consuming all resources perpetually.
3. Poor code abounds with total disregard for any standards (e.g., wheel is reinvented regularly, 10 screen shot methods, classes with 100′s of members, etc.).
4. Duplicate data and functionality throughout including “clone” systems.
5. Management, via ignorance, insists things are basal or worse are in complete denial.

Once one can identify a subset of ailments it’s tempting to run from these helter-skelter systems in haste. Knowledge arms us not to repeat them – maybe. Yet Foote and Yoder make clear in BBOM many profitable such systems exist. Profitability aside the fact is they abound! Having designed, developed, and maintained such systems myself the temptation is to disparage. What is it about these hastily crafted boomtown’s that irk us? Is it the simple fact that the architects and custodians are all to comfortable with a level of expediency we are uncomfortable with?

I am poorly equipped to answer from my limited exposure. Both BBOMs I worked on were absolutely driven by a perceived fleeting window of business opportunity. Design marathons, repositories, code standards, pair coding, QC, test harnesses, continuous integration, etc. where all pushed aside in the rush to launch. Once there and stably afloat i.e.,  profitable, did the realization often slowly set in that the system was ineptly fashioned. Once built management forever demands hastily cobbled features to increase ROI almost always at the cost of sane design – the “structure” worsens. Of course, as engineers we fight to avert such quagmires whether through evangelization and/or example.  However, we all have to eat – to deny human nature is foolhardy.

An inverse universe exists where architecture, means and process are king. Laugh if you will but it’s well known that Microsoft runs one of the most process driven engineering shops the world over. For them sheer scale dictates a process rooted approach. However, for the average web business outlet day to day operation is about finding the middle ground between launch mentality and design nirvana.

So, whose world order should prevail in software engineering? The one that works.

Quotes Collection, a WordPress plugin, looked great but after playing with it for five minutes warts appeared. For starters it smartly used a database table, but there were no tools baked in for bulk import.  Secondly, the plugin UI was badly munging anything I bothered to manually insert into the database. However, the plugin provided an easy to use and feature rich PHP function for dropping the quotes into HTML. I decided to keep the plugin for displaying the quotes and hack a perl script to do the bulk database quote insert.

Fortunately, the quotes themselves were easy enough to obtain and due to a lapsed copyright available for the taking. After copy and pasting them into vim and staring at one line for nearly three minutes fatigue settled in. I made a few conceptual mental notes for the morrow and went to bed.

Early today I finished the edit of the quotes file, ttop.dat, in a vim editor. The tedium of manually stripping poor double quote characters, blank lines, and extra spaces raced by with a rested mind. I settled on using ‘—’ as a delimiter between all quotes as most contained multiple lines knowing that perl’s input record separator would permit parsing pliability. Once the quote file edit was complete I hacked up the perl script to slurp and parse the file and insert each quote to the database. After a few trail runs to ferret out the bugs the quotes were neatly tucked away in the database.

#!/usr/bin/perl

use strict;
use warnings;

use DBI;

# insert the new uirs into the db
sub db_insert_quotes {
    my $quote_sref = shift;
    my $dbh = shift;

    my $sql = "INSERT INTO wp_quotescollection (quote, author, source, tags, visible, time_added, time_updated)
               VALUES (?, 'Geoffrey James', 'The Tao Of Programming', 'TTOP', 'yes', NOW(), NOW())";
    my $sth = $$dbh->prepare($sql);
    $sth->bind_param(1, $$quote_sref);
    $sth->execute;
}

# Connect to the database.
my $dbh = DBI->connect("DBI:mysql:database=DATABASE;host=localhost",
                       "USER", "PASSWORD",
                       {'RaiseError' => 1});

$/ = "---\n";
while (<>) {
    chomp;
    chop;
    my $quote = $_;
    #db_insert_quotes(\$quote, \$dbh);
}

exit;

A few months back I vaguely recalled reading a jQuery article with a great simple sliding panel demo. After grabbing a few images and applying a dollop or two of CSS, Javascript, and HTML my panel was functional. Kuler‘s color insights came to the rescue again saving site visitors from my train wreck artistic spasms.

This site enhancement was a joy to work on. I hope visitors are enthused about it as I am.

Set up with a 16G SDHC memory card and a decent B+W lens filter I spent my first day poking around for D90 learning tools and culled the following list:

• Nikon D90 manual
• Nikon D90 Digitutor videos
• Ken Rockwell’s Nikon D90 Users’s Guide

This probably sounds trite but reading parts of the manual, especially the camera body layout and control panel diagrams, taught me a great deal. After shooting some pictures around the apartment I drove to one of my favorite outdoor haunts, Eldorado Canyon State Park, for an afternoon of milling around the trails with an itchy trigger finger.

Fowler Trail Hikers

One dream begins as another ends

Design Patterns Front Cover

The book arrived two days ago and I have been busy poring over it in my spare time. I am using a study guide that groups the patterns by difficulty ranging from easy to advanced. Can’t wait to finish reading and possibly start applying these new nuggets of knowledge to my current AJAX charting application project.

• B = Brown (or Black) Bleeding
• C = Cloudy (white)
• C/K = Cloudy/Clear
• G = Gummy (gluey)
• K = Clear
• L = Lubricative
• P = Pasty (creamy)
• Y = Yellow (even pale yellow)

Each code can be selected once with any other combination of codes.  Some examples of possible code string combinations with dash separator(s):

1. B – C
2. C/K – K – P – Y
3. K – P – Y – C/K
4. L – L – Y Note, not valid due to duplicate L code

Eventually, these code strings are stored in a RDBMS.  The question is how to store them efficiently? We could store them as a string datatype (e.g.,  VARCHAR), but this would be inefficient because on average a code string would take up more bytes then a number datatype (e.g., INT) used to represent the code string.  This implies a look up table of code strings with related numbers and a foreign key on the main storage table(s) referentially constrained to the look up table.

Entity Relationship Diagram

Further, consider the third string code example above. Semantically it’s the same as example two, but is ordered differently (AKA permutation). To properly normalize the data we should alphabetically arrange the permutations so that order does not matter which will vastly reduce the number of possible string code combinations.

Now that we know how the data should be modeled we must populate the string code look up table. The first obvious question is how many combinations are possible when order doesn’t matter? Use the formula below where n is the number of things to choose from, and you choose r of them (No repetition, order doesn’t matter). ! means factorial.

For example, for the eight possible codes if we select four then there are 70 possible code combinations where order doesn’t matter.  However, we need to know the number of non repeating combinations possible for our set of eight codes given we can select between one to eight codes. I’ll leave that simple exercise as reader homework, but excluding the possibility of selecting no codes the answer is 255.

Now the crux of the matter. Insert the 255 non repeating combinations, alphabetically arranged, with a unique number for each into the RDBMS lookup table. Assuming you managed the math part and have made it this far you can now go and gently beat your head against a wall for a few minutes – then return, please.  Perl to the rescue! With it’s ease of extensions via CPAN modules and high level programming abstractions it is the ideal glue language to tackle what looks to be a formidable problem.

Our first order is to google CPAN and see what, if any, perl modules are available to deal with math combinations.  Ah, the Math::Combinatorics module should fit the bill nicely. Second, we need to install the Math::Combinatorics module locally. While we are at we also need to install DBI and DBD::mysql modules to enable perl access to our MySQL RDBMS so we can insert the combinations into our look up table. Most default perl installs come with a handy tool called cpan which gives command line access to the CPAN online module repository to ease searching and installing locally.

joecrotty@macpro:~ $ sudo cpan
Terminal does not support AddHistory.

cpan shell -- CPAN exploration and modules installation (v1.9402)
Enter 'h' for help.

cpan[1]> install Math::Combinatorics
...

cpan[2]> q
Terminal does not support GetHistory.
Lockfile removed.

With the modules in place our finished perl script is below. Note, the actual database inserts are commented out on lines 29 and 30.

#!/opt/local/bin/perl

use strict;
use warnings;

use Math::Combinatorics;
use DBI;

$| = 1; # disable buffering
my @codes = ('B', 'C', 'C/K', 'G',
             'K', 'L', 'P', 'Y');

# Connect to the database.
my $dbh = DBI->connect("DBI:mysql:database=DB;host=HOST",
                       "USER", "PASSWD",
                      {'RaiseError' => 1});
my $sql = 'INSERT INTO wp_nt_sec_combos (description)
           VALUES (?)';
my $sth = $dbh->prepare($sql);

for ( 1 .. 8 ) {
    my $combinat = Math::Combinatorics->new(count => $_,
                                             data => [@codes]);

    while(my @combo = $combinat->next_combination){
        @combo = sort @combo;
        my $combi = join('-', @combo);
        print $combi, "\n";
        #$sth->bind_param(1, $combi);
        #$sth->execute;
    }
}

1. ‘Featured Posts’ widget images in the sidebar now link directly to the post.
2. Padding between all the sidebar widgets (e.g., ‘Featured Posts’, ‘Recent Posts’, etc.) is now even.
3. ‘Archives’ sidebar widget is now collapsible. Note, there is a bug in IE6-8 and I am not sure when I’ll have time to fix it. My recommendation is to switch your browser to Firefox, Chrome, or Safari.