I purposed we spec a data base (AKA DB) table and code out a change to an upstream process that feed the troubled process so we could start gathering some basic metrics: filename, hostname, size_bytes, and count_msgs.

I hastily began cobbling the idea together and about a day and a half in it dawned on me in a quite moment – the initial design was inept.

For starters the DB table would only store data for one process. But the message data being parsed was part of a broader data pipeline.  The make up of the pipeline was a bit vague like the developing wave below, but a it has the hallmarks of an ETL system.

The solution would need a few tweaks:

1. Three data base tables – note: host table already exists
   1. msg_master
      1. file_id
      2. file_name
      3. file_dt
      4. start_dt
      5. end_dt
      6. created_dt
      7. updated_dt
   2. msg_proc_metrics
      1. proc_id
      2. host_id
      3. file_id
      4. size_bytes
      5. count_msgs
      6. runtime
      7. version
      8. created_dt
   3. msg_procs
      1. proc_id
      2. procname
2. Directory for all processes in the ETL system to write metrics files to.
3. A script, cron scheduled, to load metrics files to the data base.

The new DB model would enable any process to store metrics and normalizes fairly well. Writing metrics to JSON files in a directory insulates  ETL processes from DB latency. I am a big fan of JSON as the data is more portable and readable with a size cost.  However, were only dealing with ~25,000 records/day which is ideal for JSON.  Avro would be a good data serialization solution if the records numbers were a few order of magnitude larger.

With a better functional framework designed  the bit rot front line troops may be getting some better weapons soon.

1. Monitoring
2. Alerting
3. Performance Analysis
4. Trending
5. Quantitative Analysis

Behold this monster linux kernel trap thread where Linus himself weighs in autocratically for 8 column wide hard tab stops which are the code style standard for both the linux kernel and git projects.  At the opposite spectrum Coding Horror comes down in favor of spaces. Notice the spectrum’s advocates hail from the polar worlds of unix and windows respectively – let’s let sleeping dogs lie shall we?

My initial delve into this topic was driven by a desire to correct some files recently migrated from cvs to git at work.  The primary concern was stripping cvs ‘$Id:’ tags lines from the headers of all files and deleting CVS metadata directories.  The scheme occurred to kill two birds with one stone and align the code files on a standard.  However, my idea of an alignment standard may not jive with the teams nor the companies approved standard. I plan to apply the cvs fix pronto and discuss internally which alignment standard to apply.

Advent is full of Christian traditions of all sort. One of the central ones in the West centers on jolly Santa Clause. Saint Nicholas (aka Santa Claus) was a saint and bishop of Myra (modern Turkey) who lived from 270-343.

At the Council of Nicea in 325 Saint Nicholas punched out the heretic Arius who falsely asserted that Jesus was a created being who had not existed from the beginning.  Saint Nicholas was subsequently stripped of the title of bishop and his personal copy of the Gospel and his pallium (i.e., worn vestment akin to a fancy robe) were both confiscated.

Minus the symbols of his former office as bishop he was jailed. That very night Jesus and Mary appeared to Saint Nicholas. Jesus asked him why he was there. Saint Nicholas replied, “Because I love you, my Lord and my God.”

Christ then presented Nicholas with his copy of the Gospels. Next, the Blessed Virgin vested Nicholas with his episcopal pallium, thus restoring him to his rank as bishop. When the Emperor Constatine caught wind of the miracle he reinstated Nicholas as bishop.

Let us pray to Saint Nicholas that God might grace us with a fraction of his faith to fight against denial of the divinity and presence of the Triune God.

1. repo – machine that will hold the master remote repository
2. local – local machine that will hold the checked out repository

devlord is in the git group, but devserf is not. Assuming we already have a code base on local let’s get that checked into a new git repository.

cd /var/www/myproj git init

Now configure repo machine.

su - root<br />adduser git<br />mkdir -p /var/git<br />chmod 775 /var/git<br />su git<br />cd /var/git<br />mkdir -p myproj<br />cd myproj<br />git init --bare --shared=0664<br />

The –bare tells git that it will only hold file deltas and binary information for future checkout.  It’s the master repository not an actual working repository.

Let’s head back to local and finish up getting our myproj code base into the master repository on repo.

cd /var/www/myproj

git remote add origin ssh://devlord@repo/var/git/myproj

git push origin master

I don’t fully understand origin yet, but it’s akin to a local branch and the master is elsewhere – in this case on repo. The second command pushed local myproj over to the master myproj repo.

Now devserf can checkout a copy of myproj on local.

cd /home/devserf

git clone ssh://devserf@repo/var/git/myproj

However, if devserf modifies myproj and wants to push their changes back to master myproj at repo it will fail because they are not in the git group on repo.  Only users in the git group on repo, like devlord, are allowed to make commits back to the master.

It’s been a whirlwind three plus months. My apologies to regular readers for the lack of recent posts. A recap is in order as to what exactly I have been up to. March 15, 2010 or so my wife and I decided to launch a hastily planned attempt to buy our first home.  We went down this same route in 2004 and abandoned ship when it became clear that the housing market was awash with over priced homes. Invigorated by the thought of an $8,000 federal tax tax credit that would apply only if we could get under contract by April 30, 2010 and close by June 30, 2010 we charged forward forcefully. After a lot of looking, researching, showings and a few false starts we found a great house in Broomfield, CO and closed May 13, 2010 and moved in May 15.

To complicate matters I started a new job April 19 at OceansideTen Management, Inc. as a software developer/engineer.  Since the move in it’s been an endless list of to-dos and odd jobs around the house. Last weekend Heather and I drove back to Missouri Valley, Iowa and drove back a U-Haul trailer full of furniture she had been storing at her mom’s house.  Her mom came with and spent the week helping get things in order here. What a blessing! We capped off her visit with a drive up to Estes Park and a hike from Bear Lake to Emerald Lake in Rocky Mountain National Park.

I am looking forward to more outdoor trips this summer now that things are stabilizing at the new home front.

Gulliver, as the system, washes ashore to be tamed by the custodians.

Warning, this is a an article by a software geek for said geeks. Rapid boredom will probably ensue even for the target audience. Most of us at one time or another have worked on a Big Ball of Mud. For a less seasoned hacker the first tour or two of such a haphazard system may go unnoticed due to a number of circumstances the least of which being naivety. But, over time, the imprints of sloppy, duct tape, spaghetti code jungles come to be recognizable. By no means complete a few standout examples from my career:

1. Code is not stored in an SCM.
2. Bug fires rage 24/7 consuming all resources perpetually.
3. Poor code abounds with total disregard for any standards (e.g., wheel is reinvented regularly, 10 screen shot methods, classes with 100′s of members, etc.).
4. Duplicate data and functionality throughout including “clone” systems.
5. Management, via ignorance, insists things are basal or worse are in complete denial.

Once one can identify a subset of ailments it’s tempting to run from these helter-skelter systems in haste. Knowledge arms us not to repeat them – maybe. Yet Foote and Yoder make clear in BBOM many profitable such systems exist. Profitability aside the fact is they abound! Having designed, developed, and maintained such systems myself the temptation is to disparage. What is it about these hastily crafted boomtown’s that irk us? Is it the simple fact that the architects and custodians are all to comfortable with a level of expediency we are uncomfortable with?

I am poorly equipped to answer from my limited exposure. Both BBOMs I worked on were absolutely driven by a perceived fleeting window of business opportunity. Design marathons, repositories, code standards, pair coding, QC, test harnesses, continuous integration, etc. where all pushed aside in the rush to launch. Once there and stably afloat i.e.,  profitable, did the realization often slowly set in that the system was ineptly fashioned. Once built management forever demands hastily cobbled features to increase ROI almost always at the cost of sane design – the “structure” worsens. Of course, as engineers we fight to avert such quagmires whether through evangelization and/or example.  However, we all have to eat – to deny human nature is foolhardy.

An inverse universe exists where architecture, means and process are king. Laugh if you will but it’s well known that Microsoft runs one of the most process driven engineering shops the world over. For them sheer scale dictates a process rooted approach. However, for the average web business outlet day to day operation is about finding the middle ground between launch mentality and design nirvana.

So, whose world order should prevail in software engineering? The one that works.

For the last four years I have volunteered at Holy Trinity Catholic Church & School as a catechist for the religious education program forming young adults. Classes are offered twice a week on Wednesdays evening and Sunday afternoon during the regular school year. My wife and I, Heather, teach the Wednesday class. This year we moved up from teaching second to third grade sacramental kids (i.e., they receive both reconciliation and first communion in the Spring).

The highlight of the year, without a doubt, was seeing these children of God, claimed by him during baptism, receive holy communion for the first time.  I was overjoyed by the visible ceremony, but internally I could sense an extraordinary grace that imbued all of them. They have moved one step further in the natural progression as foot soldiers for Christ. Let us pray their parents remain steady in the fight against evil and deliver each child consistently and without fail to God’s house on Sunday .

Maltese Cross Keys of St. Peter

For a few years now I have been yearning to put a maltese cross styled window sticker on my truck topper. Something that hollers out to a distressed and a generally non believing world that I am Catholic and Proud! I poked around the web and came up with a few ideas but nothing that really stuck. If you want something done right then you have to do it yourself.

The Order of Malta and the associated Maltese Cross are old by almost any standard. Yet,even though the wiki maltese cross image is decent enough they get the geometry wrong. After locating a correct geometric maltese cross layout the next hurdle was designing it such that at least fellow Catholics would get “it.”

I am fairly certain other dyed in the wool Catholics would recognize the maltese cross, but there is uncertainty it would resonate a call to arms in the public square in defense of everything we profess. The other day after dinner at a friends house I noticed they had a small sharp looking Vatican flag bumper sticker on their SUV. In the white half of the flag is the universally recognized crossed Keys of Saint Peter. Obviously, at least to me, a blended design of the Maltese Cross and the Vatican Flag would drive home a clarion call to defend, preserve, and uphold the Church – especially the Pope. We must not flinch in the slightest when the enemies of the Church press in on every side.

When Kinko’s is done printing up the sticker and it’s fixed to the truck topper I’ll post a photo.

I live in Westminster, Colorado a growing suburban area northwest of Denver. It’s a mixed profile of semi-industrial areas, quaint residential neighborhoods, parks, schools, and the the usual sprinkle of hustle and bustle that accompanies suburban life. Having lived here for ten years I know a thing or two about some of the local urban legends. For example, “air guitar master” Mark who makes infrequent rounds on Federal Boulevard in the Federal Heights region in retro ’80′s metal rock garb hammering out licks to his walkman.  I hope to highlight him in a future post, but today’s feature character is the “sign dancer.”

Anyone who has driven through the intersection of 120th Avenue and Sheridan Boulevard and missed the “sign dancer” when present must have been either very busy behind the wheel or blind.  Tim Farnsworth is the name and Professional Sign Shaker is the game. Forget about your typical lifeless fast food employee ordered to a nearby intersection to stir up business. Tim’s sheer energy, movement, drive, costume and general street rapper vibe catapults the banal sign vegetable to new heights. Tim shakes six hours a day five days a week as his main employment for M&E Painting. He almost always brings a smile to my face and a friendly honk!