Help Perl Project Layout Problem
Over the last few months I have been trying to solve a problem at work in my spare time, fleeting as it is, revolving around how to consistently release Perl code to target environments (dev, tst and prd) with prerequisite modules in tow. A mini hack-a-thon ensued with clear winning tools: Perlbrew, local::lib, Dist::Zilla, Pinto and cpanm.
However, after making some changes to a test projects directory layout I am at a roadblock on how to proceed. A thorny problem made all the more so due to my lack of knowledge about the Perl toolchain and Perl in general. I’ll describe the problem and some possible solutions. Feedback appreciated.
Imagine if you will 100+ small git repositories each with Perl code in them. Mostly .pl files, but occasionally .t and .pm files. Almost all of the Perl scripts are backend jobs of various genre. Most are called by cron from wrapper shell scripts. However, there is more then just Perl scripts. Bash, Java, Python and SQL are sometimes mashed into the same repo. On average each repo has 10 Perl scripts. The repos are released to different machines to a unique location (e.g., /usr/local/foobar). A typical example repo layout follows:
|bin/||Bash, Perl, Ptyon and etc. scripts|
|perl-lib/||New Perl prereqs dir plus tools et al|
|darkpan/||Pinto managed DarkPAN|
|dist/||Task list distributions built with Dist::Zilla.
All prereqs for top level bin/ scripts.
|local/||Target of cpanm installs for all Perl modules.|
|utils/||Git submodule with bash glue scripts|
|bin/||Bash scripts for local::lib env setup,
running Perl module installs, etc.
|t/||Perl tests for top level bin/ Perl scripts.|
To help us get a handle on our Perl release management stack we developed utility scripts adding them to perl-lib/ – new general purpose container (i.e., above in green). Utility scripts in perl-lib/utils/bin lend a helping hand setting local::lib environment variables that drive any Perl toolchain actions:
- Which Perlbrew perl scripts
- Perl scripts search lib dirs via PERL5LIB – points at perl-lib/local
- Dist::Zilla prereq scanning
- Pinto perl-lib/darkpan distribution management
- cpanm Perl distribution installs from darkpan drive by Task module in perl-lib/dist
Some questions I am still struggling with:
- Is perl-lib a silly or otherwise untoward name not in keeping with Perl best practices?
- Do the Perl scripts in top level bin/ still belong there? Should they live elsewhere (e.g., perl-lib/dist) and be installed by cpanm to perl-lib/local? My main concern here is there are hard coded cron scripts pointing at them currently in bin/ and lots of engineers have just gotten used to it “being that way.”
- Where should top level t/ live? It’s location always seemed akward. The tricky part here is that when our continuous integration server, Jenkins, delivers code to integration environments all t/ tests are run with TAP::Formatter::JUnit output displayed by Jenkin. If the answer to #2 above is move the bin/ scripts to perl-lib/dist then I would think t/ goes there to. Currently, perl-lib/dist holds a single distribution that is a Task dist of all the projects Perl prerequisites. We would need some way to only return test results for in house code only and not CPAN modules.
Please, feel free to reply and tell me how lost we are and what insights you have. Much appreciated.
It’s been decided that perl-lib/ a misnomer. Instead we will be using pallet/. Further perl-lib/local was renamed to pallet/perl5.