Tatsuhiko Miyagawa's blog

test_requires all the way

tl;dr Today possibly for the first time ever, CPAN toolchain ecosystem all support test requirements as separate from build requirements. I can’t be happier ever.

Little bit of history. Timeline might be not in the correct order.

Module::Install always had test_requires() with forward compatibility in mind. However META spec v1.4 didn’t support specifying test requirements separate from build requirements. So the most tools merge them as a build requirement, including Module::Install itself. This means most of the Test:: module dependencies, even though they are only used for testing, were installed even when notest options is supplied (with CPAN.pm and cpanminus).

META spec v2.0 came out, and it allows META (and MYMETA) to specify test requirements separately.

cpanm supported parsing test requirements separately, and skips installing test prereqs when —notest is in use. But at that time, most of the tools did not support emitting test requires apart from build requires.

Except Module::Build::Tiny, which just copies META.json to MYMETA.json upon installation, so if you use an authoring tool such as Dist::Zilla or Milla to emit META.json with test requirements apart, it will be skipped.

Meanwhile rjbs added TEST_REQUIRES support to MakeMaker at Paris QA hackathon 2012. 7 months later MakeMaker 6.63_03 was released with the fix, and non-dev version was just out in December.

Same fix was added to Module::Build by tokuhirom and released as 0.40004 today.

Now, what should happen is to update Makefile.PL and Build.PL to make use of these new features - I patched Dist::Zilla so that MakeMaker and ModuleBuild emitters will use test_requires when the installer has the version with the support for it.

I’m trying to fix Module::Install similarly, but as of this writing I’m not that hopeful - if someone wants to take over my work to make it merged that’d be super. But at this point I’d rather give up and stop using Module::Install.

Anyway, now when you have a module that is only used for testing, it will be (and has to be) declared and emitted correctly as a test dependency, and when you run CPAN installers, at least cpanm, with “no test” option, they won’t be installed. This is a very important milestone.

Milla, a Dist::Zilla profile that doesn’t suck (screencast)

Ok, that was a little link bait title.

I’ve been a big fan of Module::Install - it automatically figures out the metadata of my module with just all_from, and does the right thing creating META files. Then it bundles itself in inc/ so that if you install from CPAN you don’t need to have Module::Install pre-installed.

But not everybody likes it, especially contributors who need to install plugins before doing anything, or users who want to grab the latest from git.

The problem is, Module::Install is just a wrapper for MakeMaker, and MakeMaker and Module::Build are not the right tool to create a distribution.

Dist::Zilla is a pioneer in this field, to provide the complete authoring environment for the CPAN distribution, while having the ability to export MakeMaker/Module::Build compatible build files, so that end users don’t need to have that upon installation.

However Dist::Zilla kind of sucks. Well, that was a strong word - it has been coded and maintained actively, so it’s great. But its user experience could have a little improvements.

Milla is my first attempt to address that issue. It is a set of profile - PluginBundle, MintingProfile and command line wrapper, so that you can use it with almost zero-configuration and should do the right thing.

There’s some limitation and assumption that do not work for everybody, but at least for me I hope to replace 95% of my CPAN modules with Milla in the coming weeks.

Here’s a screencast to show you how to use it.

If you already have modules written with MakeMaker/Module::Install + ShipIt, see the Migration document for the details. I might make a new screencast just for the migration.

Hope you enjoy Milla. If you dislike the 195 dependencies it needs, check out our sister project Minilla as well.

Perl versions usage stats (with cpanm)

A week ago I added a neat little feature in cpanm 1.6004 to report its perl versions in User-Agent strings (regardless of whether it uses LWP, curl, wget or HTTP::Tiny) in addition to cpanm’s own version.

One week later, there seems more than 60% of the users seem to have upgraded (including people who runs curl -L cpanmin.us which always run the latest stable), and we get a pretty interesting number.

(In case you’re wondering: the big ones are 5.8.8, 5.10.1, 5.14.2 and 5.16.2. Graph on the link uses google chart API which allows you to hover over the graph to see which portion represents which perl version)

Perl versions used with cpanm is a graph calculated out of 10K recent requests in CPAN Meta DB uniqued by the IP address, updated daily on 1am UTC.

The graph can be viewed using ?date=YYYYMMDD permanent link as well.

Few notes: This does not reflect (yet) the users who use older versions of cpanm, which may or may not be the installation which tend to use older versions of perl. It also doesn’t include cpanm users who uses --mirror-only to work with local CPAN mirrors without hitting CPAN Meta DB nor MetaCPAN. I believe these numbers can be either a) so small that it’s ignorable or b) significant, but spread across many versions in the same way those who don’t.

The number might look low for the recent 10K requests. There are several reasons, but the biggest one is that many requests will end up with pulling 10+ CPAN dependencies, which are (correctly) deduplicated into one.

It’s just one metric, and I don’t argue that it’s accurate, but I believe it’s not that far away, and would be interesting to see how it settles over time.

What is cpanfile and why do I want to use it

So there seems to be a couple of blog posts and even conference talks about cpanfile. It’s great, but just to make sure everyone gets what it is and why they want to (or don’t want to) use it for their stuff, here’s some clarification from the author.

tl;dr for japanese - this slide deck at Hokkaido.pm has a great intro for what cpanfile is.

cpanfile is made specifically to address the two following use cases, while the second one is kinda accidental.

1) “I have an app (web app, command line script) and want to manage its CPAN module dependencies somehow. I created a fake Makefile.PL to describe them, even though I don’t have a plan to release it to CPAN.”

2) “I have a Perl module on my git repository and want my contributors to be able to install from the repo, but they keep saying they can’t load inc/Module/Install.pm, and I don’t want to bump META.yml every time or include inc/ in the repo.

cpanfile is a Module::Install-like DSL to describe CPAN module dependencies. It comes with a module to parse out that file, but isn’t tied to any specific builder nor installers.

It is useful for Perl application developers who want to describe CPAN dependencies, and being able to install them with carton install or cpanm --installdeps .

For CPAN module authors, cpanfile is not something you really need to switch to, although you will (soon) have a benefit of easier installation from git repository if you have one. The reason i wrote “soon” is that cpanm 1.6 as of this writing doesn’t have the ability to bootstrap modules with M::I from git repo - it’s a known issue and I am trying to address that in the sane way before cpanm 1.7. (There’s a workaround out there to specify Module::Install in configure_requires but that is wrong because that’s not necessary once shipped to CPAN)

Standard CPAN installers (CPAN.pm, CPANPLUS, cpanm) work with META files (and then MYMETA after configuration) to detect dependencies. It’s a standard, and cpanfile doesn’t intend to replace them or anything at all. If you decide to use cpanfile to describe deps for your CPAN modules however, you should use Module::Install::CPANfile or Module::Build::Pluggable::CPANfile so that it will transparently work with MYMETA protocol.

FAQ

I am a CPAN author and keep META.yml in the repo. Should I switch to cpanfile?

If your configuration is static and you have no problem remembering to bump META.yml in the repo, that’s just fine. However, putting “Module::Install” (for example) into your configure_requires doesn’t sound right since it’s not necessary once it’s shipped to CPAN. Committing the whole inc/ is another way to go, but it feels like an unnecessary work to me.

I have a web app and want to use cpanfile to deploy to the cloud. Which provider supports cpanfile?

Most PaaS providers (dotcloud, heroku through buildpack, etc.) uses cpanm --installdeps . to install dependencies. Once they upgrade cpanm to 1.6 or later, it will automatically figure out the dependencies out of cpanfile. If they don’t have 1.6 yet, why not requesting them to upgrade it?

I use dzil for my CPAN modules and am not sure what you are talking about.

Great, Forget about this post.

However, contributors to your module may have hard time installing dzil first and then remembering what command to run to test/install because it’s different from other CPAN module convention. cpanfile may be able to help that, but i’m still trying to figure out.

So which phase should I put Module::Install into in cpanfile?

It should most likely be develop, and cpanm will detect the “development environment” i.e not from CPAN tarball, but that’s still being ironed out.

stop shipping MYMETA to CPAN

You’re a new CPAN author (congratulations!) or you have an old distribution that is about to be updated for the first time in years, and you find MYMETA.yml and MYMETA.json in your working directory that git or shipit warns about. You think: What are these files? Should I package these files?

tl;dr - add MYMETA.yml and MYMETA.json to .gitignore and MANIFEST.SKIP (if you have one), and do not include them in the tarball.

MYMETA.yml/json files are a relatively new approach for CPAN installers (CPAN, CPANPLUS, cpanm) and module build tools (MakeMaker, Module::Build) to communicate the configured dependencies on the end-user’s system.

MYMETA is just like META files in terms of the file format and data structure, but whereas META.yml is generated by the distribution author (you), MYMETA.yml is generated by the end user at a configuration time.

Including MYMETA files in MANIFEST and a distribution tarball does not make any sense, and could cause potential problems confusing CPAN installers - do not make them part of your distribution.

(Why this post? Many CPAN authors have been confused or unaware what MYMETA files are and simply include them in the distribution)

Podcast ep3: 2013/02/27 ゲスト: Sadayuki Furuhashi, Kiyoto

bulknews-podcast:

第３回は古橋貞之さん (@frsyuki)、kiyotoさん (@__kiyoto__) をゲストに迎えて MessagePack について話しました。

Subscribe via iTunes | RSS