The case was posting 8Gb file on 16Gb Xen instance. While this worked quite nice on real box, on virtual box curl said hello with :
out of memory
Now that seemed quite bizzare. Figurng out that the process actually gets ENOMEM, it was logical to look at curl code and figure out what's going on.
And there it was, power-of-two allocator in the file read loop :
static ParameterError file2memory(char **bufp, size_t *size, FILE *file)
...
char *newbuf;
char *buffer = NULL;
size_t alloc = 512;
size_t nused = 0;
size_t nread;
do {
if(!buffer || (alloc == nused)) {
/* size_t overflow detection for huge files */
if(alloc+1 > ((size_t)-1)/2) {
if(buffer)
free(buffer);
return PARAM_NO_MEM;
}
alloc *= 2;
if((newbuf = realloc(buffer, alloc+1)) == NULL) {
if(buffer)
free(buffer);
return PARAM_NO_MEM;
}
buffer = newbuf;
}
}
Whoa :) - now apparently someone didn't expect some geniuses will try po post XX Gb files with curl - so it's the abusers that are to blame. Stop abusing curl and do your own posts !
However, if you don't have the time to change your app, and still want to post files of the size (N,2N) Gb on a 2N Gb box, a simple hack of given form should do it :
if (alloc < ALLOC_THRESHOLD)
alloc *= 2;
else
alloc = alloc + ALLOC_THRESHOLD;
(Where ALLOC_THRESHOLD would usually be 1Gb)
This should make allocation linear, rather than exponential, once the allocated memory passes given threshold.
Now - what does is all has to do with XEN, you might ask?.
Couple of things, actually. First off, such environment (local or any virtualized cloud platform offering xen instances) usually provide user with something like effective 2^N - penalty memory space (say 15Gb instead of 16Gb) - and that's where the impact of power of two allocator becomes apparent much sooner. Also - memory allocation policies are quite stricter and enomems are dispatched much earlier, oom killer is fast on the trigger, etc :) - so that's why the curl ooms immediately, rather than trying to make that darn realloc() after all.
Moral of the story - don't abuse standard unix tools !
Be nice to curl - do not POST binary data larger than 50% of effective RAM.
Keep it safe !
