Prototypes: the good, the bad,and the ugly

This is a reply to chromatic's post The Problem with Prototypes, but his blog didn't allow me to post it there so I post it here.

Pretty much everyone agrees that there are good (such as blocks and sometimes reference) prototypes and bad ones (scalar most of all). Few discuss the third class: the ugly glob prototype.

perlsub describes them as such:

A * allows the subroutine to accept a bareword, constant, scalar expression, typeglob, or a reference to a typeglob in that slot. The value will be available to the subroutine either as a simple scalar, or (in the latter two cases) as a reference to the typeglob.

In other words, they are the same as scalar prototypes, except that they also accept globs and barewords. This is mainly used to pass filehandles, like this:

sub foo (*) {...}

foo STDIN;

but in fact it can be used to pass any bareword to function, as it leaves the interpretation of it to the function.

It's tempting to call this bad, but it offers some API possibilities that would otherwise not be possible, hence I would call it ugly rather than bad per se.

Casting magic against segfaults

The problem

For years, there has been the Sys::Mmap module, however, it has a few issues. For example, let's take this piece of code:

use Sys::Mmap; open my $fd, '+>', 'filename'; mmap my $var, -s $fd, PROT_READ|PROT_WRITE, MAP_SHARED, $fd, 0; $var = 'Foobar'; munmap $var;

First of all, it's simply not user-friendly. mmap takes 6 arguments in a weird order, and uses weird constants. Also munmap shouldn't be necessary: variables should dispose of themselves when they run out of scope.

But more importantly, this program does not do what you think it does, though the only hint of that is an Invalid argument exception when doing munmap. During the assignment, the link between the mapping and the variable is lost, so nothing is written to the file. Worse yet, this can even lead to a segfault in some circumstances.

Ouch!

Tying things up?

The documentation clearly says that you shouldn't do this (or anything else that changes the length of the variable), but IMHO this hole shouldn't be left open in the first place, if only because it is extremely counterintuitive (and thus a maintenance nightmare). Modules should fail more gracefully than this.

Sys::Mmap offers a tied interface as compensation, but this didn't work out. The tied interface indeed is safe, but it creates another problem.

Every time it is read, it copies the whole file into the variable. Every time the variable is modified, it writes the whole new value to the file, even if the change only affects a single byte.

Ouch!

Obviously, that doesn't scale at all. One user of the module reported a 10-20 times slowdown of his program after converting to ties. That's not a workable solution.

The solution

Perl has a powerful but rarely used feature called magic. (It's rare use by module authors is indicated by the fact that the prototypes of the magic virtual table as documented in perlguts aren't even complete: they lack pTHX_'s). They are used by the perl core to implement magic variables such as $! and ties (surprise, surprise). It offers 8 hooks into different stages of handling a variable, the three most important being fetching(svt_get), storing(svt_set) and destruction(svt_free).

In my case, I didn't need get magic, but I did use set and free magic. Freeing the variable is not that interesting (simply unmapping the variable), but setting it is. This function is called just after every write to the variable:

static int mmap_write(pTHX_ SV* var, MAGIC* magic) {     struct mmap_info* info = (struct mmap_info*) magic->mg_ptr;     if (SvPVX(var) != info->address) {         if (ckWARN(WARN_SUBSTR))             warn("Writing directly to a to a memory mapped file is not recommended");         Copy(SvPVX(var), info->address, MIN(SvLEN(var) - 1, info->length), char);         SvPV_free(var);         reset_var(var, info);     }     return 0; }

This function is called after every write to the variable to check if the variable is still linked to the map. If it isn't, it copies the new value into the map, frees the old value and restores the link. As copying is potentially expensive, it will issue a warning if warnings (or actually, 'substr' warnings) is in effect.

There is no perfect solution to this problem, but getting a friendly warning is undeniably better than getting a segmentation fault or data loss.

Anyway, you can find Sys::Mmap::Simple here. It offers more goodies, such as portability to Windows, a greatly simplified interface, and built-in thread synchronization.

Elegance in minimalism

Some time ago, I read this journal entry by Aristotle. I liked it and suspected it could be easily implemented in XS. It turned out to be the most elegant piece of XS I've ever written.

void induce(block, var)     SV* block;     SV* var;     PROTOTYPE: &$     PPCODE:         SAVESPTR(DEFSV);         DEFSV = sv_mortalcopy(var);         while (SvOK(DEFSV)) {             PUSHMARK(SP);             call_sv(block, G_ARRAY);             SPAGAIN;         }

I assume most readers don't know much C, let alone the perl API or XS, so I'll explain it piece by piece.

void induce(block, var)     SV* block;     SV* var;     PROTOTYPE: &$
This declares the xsub. It has two parameters, both scalar values. The function has the prototype &$. So far little surprises.

    PPCODE:
This declares that a piece of code follows. Unlike CODE blocks, PPCODE blocks pop the arguments off the stack at the start. This turns out to be important later on.

        SAVESPTR(DEFSV);         DEFSV = sv_mortalcopy(var);
These lines localizes $_ and initializes it to var.

        while (SvOK(DEFSV)) {
This line is equivalent to while (defined($_)).

Now comes the interesting part:
            PUSHMARK(SP);             call_sv(block, G_ARRAY);             SPAGAIN;
To understand what this block does, you have to know how perl works inside.

If you've read perlxs, you may notice this function does not push any values on the stack. Are careless reader might be mistaken and think this function doesn't return anything: they couldn't be more wrong!

If you've read perlcall, you would notice a lot more is missing. For starters, the function calls SPAGAIN (pretty much equivalent to saying I accept the values you return to me), but it doesn't do anything with them.
Also, you may notive that both ENTER/LEAVE(needed to delocalize $_) and SAVETMPS/FREETMPS (needed to get rid of temporary values) are missing. The function that calls the xsub automatically surrounds it by an ENTER/LEAVE pair, so that one isn't necessary. The lack of SAVETMPS/FREETMPS however is not only deliberate but also essential.

The loop calls the block without arguments (PUSHMARK & call_sv). The xsub accept the return values on the stack and leaves them there. This sequence repeated. This way it assembles the induces values on the stack. PPCODE removing the arguments at the start prevents it from returning those as first two return values. It also adds a trailer that causes all elements that have been pushed on the stack to be recognized as return values of this function. That's why a SAVETMPS/FREETMPS pair would break this code: the values must live after the code returns.

That's the elegance of this function. It doesn't even touch it's return values, it delegates everyting to the block. All the things that are missing make that it does exactly what it should do.