Uploads and files

A file part of a multipart/form-data request arrives as a Punk::Upload:

post '/avatar' => sub {
    my ($c) = @_;
    my $up = $c->upload('avatar') or return $c->text('no file', 400);
    $up->save('/var/lib/app/avatars/' . $c->auth_id);
    $c->json({ name => $up->filename, bytes => $up->size });
};

$c->req->form parses the body once: ordinary fields become parameters, file parts become uploads, reachable through $c->upload($name) and $c->req->uploads. A field uploaded more than once yields an arrayref from uploads; upload gives the first.

filename is the client-supplied name and must never become a path - it is request bytes. It never names the temp file, and it should never name yours: save under an id you minted, or use Blob below and the question goes away.

Large uploads cost a file, not their weight

A part under 64KB is held in memory. A larger one is written to a temp file as it arrives, and the object carries path and fh to it rather than the bytes - so an upload costs a file and a few kilobytes whatever its size. Measured end to end, a 128MB upload holds a worker at 15.5MB of RSS against roughly 275MB when it was copied through memory.

$up->path;      # the temp file, or undef if the part stayed in memory
$up->fh;        # a read handle on it - chunk, hash, stream, never slurp
$up->content;   # the bytes - reads a spilled file whole; small parts only
$up->save($path);

The temp file belongs to the request and is removed when it ends, including when the handler died - use save or fh, do not keep the path. save is a rename when the temp file and the destination share a filesystem, and a chunked copy when they do not, still never through memory.

upload_dir names where a large part is spilled (default TMPDIR, else /tmp):

upload_dir '/var/lib/myapp/incoming';

Worth naming for two reasons that are not obvious. It decides the filesystem, which decides whether save is a rename or another whole copy of a large file. And it decides what shares a filesystem with attacker-controlled bytes. Temp file names owe nothing to the client's filename.

Content-addressed storage

Punk::Plugin::Blob stores bytes by their contents, on Apophis, and hands back an id:

plugin 'Blob' => { root      => '/var/lib/app/blobs',
                   namespace => 'myapp' };

post '/avatar' => sub {
    my ($c) = @_;
    my $id = $c->blob_put($c->upload('file'));
    $c->model('User')->update($c->user->{id}, { avatar => $id });
    $c->json({ id => $id });
};

get '/blob/:id' => sub { $_[0]->blob_send($_[0]->param('id')) };

The address is derived from the contents, so the user's filename is a string in a metadata column and nothing more. Deduplication is free - one file uploaded by a hundred users is stored once - and integrity is checkable, because the name is the hash. A spilled upload is hashed where it lies, in 64KB chunks, then moved into the store: storing a 128MB upload costs about 0.2MB of resident memory, and is a rename when the spill directory and the store share a filesystem - put them on the same one. blob_send serves through send_file, so conditional and Range requests are answered for you.

One thing to read before namespace looks like boilerplate: deduplication crosses its boundary. A store that already holds the bytes returns measurably faster than one that writes, so an account can upload a file, time the response, and learn whether another account holds that exact file. Where possession of a document is itself the confidential fact, give each tenant its own namespace:

namespace => sub { 'tenant-' . $_[0]->tenant_id }

Scanning uploads

Punk::Plugin::ClamAV scans uploads through a running clamd, via ClamAV::Clamd:

plugin 'ClamAV' => { socket => '/run/clamav/clamd.ctl' };

post '/document' => sub {
    my ($c) = @_;
    my $up = $c->upload('file') or return $c->text('no file', 400);
    return $c->text('no thanks', 422) unless $c->upload_ok($up);
    $up->save('/var/lib/app/docs/' . $c->auth_id);
    $c->json({ ok => 1 });
};

$c->upload_ok applies the configured policy; $c->scan_upload returns the verdict itself when "no" is not a good enough answer - is_clean, is_infected, is_unscannable (a password-protected archive, say) and the signature name, so infected, unscannable and scanner-down can each get their own response.

It picks the cheap transport for you: a spilled upload is scanned by descriptor - clamd is handed the open file, not a path, so it needs no permission on your spool directory - and a small upload is sent as bytes rather than written out just to be scanned. The policy fails closed: a scanner that cannot answer is a rejection, not a shrug.