About a week ago at work I was looking at a support email about a latent process that parses large volumes of email. During the ensuing investigation it dawned on me we had no metrics around the process. I am not a fan of  crap shot guesses.

I purposed we spec a data base (AKA DB) table and code out a change to an upstream process that feed the troubled process so we could start gathering some basic metrics: filename, hostname, size_bytes, and count_msgs.

I hastily began cobbling the idea together and about a day and a half in it dawned on me in a quite moment – the initial design was inept.

For starters the DB table would only store data for one process. But the message data being parsed was part of a broader data pipeline.  The make up of the pipeline was a bit vague like the developing wave below, but a it has the hallmarks of an ETL system.

The solution would need a few tweaks:

1. Three data base tables – note: host table already exists
   1. msg_master
      1. file_id
      2. file_name
      3. file_dt
      4. start_dt
      5. end_dt
      6. created_dt
      7. updated_dt
   2. msg_proc_metrics
      1. proc_id
      2. host_id
      3. file_id
      4. size_bytes
      5. count_msgs
      6. runtime
      7. version
      8. created_dt
   3. msg_procs
      1. proc_id
      2. procname
2. Directory for all processes in the ETL system to write metrics files to.
3. A script, cron scheduled, to load metrics files to the data base.

The new DB model would enable any process to store metrics and normalizes fairly well. Writing metrics to JSON files in a directory insulates  ETL processes from DB latency. I am a big fan of JSON as the data is more portable and readable with a size cost.  However, were only dealing with ~25,000 records/day which is ideal for JSON.  Avro would be a good data serialization solution if the records numbers were a few order of magnitude larger.

With a better functional framework designed  the bit rot front line troops may be getting some better weapons soon.

1. Monitoring
2. Alerting
3. Performance Analysis
4. Trending
5. Quantitative Analysis