Same Old Stuff : a procrastinator's notebook

Web Services Interoperability Soup

2010-03-04T14:30:00.010+01:00

Some notes on .net - j2ee and cros-j2ee web services integrations :

* Compatibility stepping stones : interaction styles, data types, namespace issues.
* wsdl standard - 1.1 vs 2.0 ?

axis2 vs jax-ws 2.0:

1) generated wsdl styles:

- namespace : wsdl-namespace for axis2 vs flat-namespace for jax-ws
- axis2 lacks name property in tag - introduces definitions/documentation tag
- endpoint vs port (though they should be equivalent)
- axis2 - direct schema in wsdl vs external reference for jax-ws
- elementFormDefault="qualified" property in axis2 schema / missing in jax-ws
- for base types, schema seems similar
- separate xsd is often more "natural" setting
- jax-ws misses Action properties for input/output of each operation
- axis2 generates multiple ports - HttpEndpoint / HttpSoap11Endpoint / HttpSoap12Endpoint - depending on the version - 1.1, 1.2, 2.0 - jax-ws assumes default version (1.1 ?)

cURL out of memory on Xen instance (use and abuse series)

2008-12-15T12:23:00.006+01:00

cURL, a part of everyone's favorite UNIX tool subset, got me into a bit of trouble recently, while trying to post a relatively large file, following a common 'just curl it' - logic (so commonplace that a lot of major projects simply incorporate curlin' as a part of standard deploy procedure).

The case was posting 8Gb file on 16Gb Xen instance. While this worked quite nice on real box, on virtual box curl said hello with :

out of memory

Now that seemed quite bizzare. Figurng out that the process actually gets ENOMEM, it was logical to look at curl code and figure out what's going on.

And there it was, power-of-two allocator in the file read loop :

static ParameterError file2memory(char **bufp, size_t *size, FILE *file)
...

char *newbuf;
char *buffer = NULL;
size_t alloc = 512;
size_t nused = 0;
size_t nread;

do {
if(!buffer || (alloc == nused)) {
/* size_t overflow detection for huge files */
if(alloc+1 > ((size_t)-1)/2) {
if(buffer)
free(buffer);
return PARAM_NO_MEM;
}
alloc *= 2;

if((newbuf = realloc(buffer, alloc+1)) == NULL) {
if(buffer)
free(buffer);
return PARAM_NO_MEM;
}
buffer = newbuf;
}
}

Whoa :) - now apparently someone didn't expect some geniuses will try po post XX Gb files with curl - so it's the abusers that are to blame. Stop abusing curl and do your own posts !

However, if you don't have the time to change your app, and still want to post files of the size (N,2N) Gb on a 2N Gb box, a simple hack of given form should do it :

if (alloc < ALLOC_THRESHOLD)
alloc *= 2;
else
alloc = alloc + ALLOC_THRESHOLD;

(Where ALLOC_THRESHOLD would usually be 1Gb)

This should make allocation linear, rather than exponential, once the allocated memory passes given threshold.

Now - what does is all has to do with XEN, you might ask?.

Couple of things, actually. First off, such environment (local or any virtualized cloud platform offering xen instances) usually provide user with something like effective 2^N - penalty memory space (say 15Gb instead of 16Gb) - and that's where the impact of power of two allocator becomes apparent much sooner. Also - memory allocation policies are quite stricter and enomems are dispatched much earlier, oom killer is fast on the trigger, etc :) - so that's why the curl ooms immediately, rather than trying to make that darn realloc() after all.

Moral of the story - don't abuse standard unix tools !
Be nice to curl - do not POST binary data larger than 50% of effective RAM.
Keep it safe !

Instrumenting Java code for fun&profit

2008-07-10T20:14:00.009+02:00

By code instrumentation we assume a proces of adding bytecodes to methods in order to intercept their execution (usually for the profiling purpose)
Some of the purposes include : code tracing (getting the method calls and regenerating the call tree), code profiling (getting the execution times between calls, automatic detection of bottlenecks etc), code monitoring (event/method invocation detection, code state monitoring - getting the structure size etc), and all of the above(for eaxmple, monitoring performance dependent of the code state, size of structures, number of predefined objects, etc)

The java.lang.instrument interface, available since java 1.5, enables us to write a handler,acessible via agent loaded into JVM, to which, all class load requests are passed. This enables us to calculate time,performance, analyze state, or even instrument the classes loaded.

the instrumentation is done via modification of method bytecodes

Creating a instrument agent is pretty straightforward :

- the Premain-Class is defined => which must implement premain method (taking a args string, and a Instrumentation object instance,which is created by jvm and automatically passed to the premain method of Premain-Class). The Instrumentation instance is then passed on, and can be used to monitor list and usage of all loaded/initialized classes, and add/remove transformers and redefine classes).

.....

- what we need to do next in order to reconstruct the full call-tree trace, is the information when which method ends. We cannot get this information via classloader, especially not in execution-time (we might try to get some post-exec via call stack reconstuction). However - we can get the return info the other way -> and that is via mangling the bytecode using the transformation framework and BCEL bytecode engineering library. We are simply altering the bytecode either via creating a wrapper method or by notifying the static method from the added instruction before the final return in the method. All in all this gives us all the necessary ingredients for making a proper runtime-profiler-call-dinamic-executinon-tracing-whatever :) Anyway, once i get it all together and start writing decent posts, i might write a proper post about it :)

References :

http://java.sun.com/j2se/1.5.0/docs/api/java/lang/instrument/package-summary.html
http://www.javalobby.org/java/forums/t19309.html
http://www.ibm.com/developerworks/java/library/j-jtp09196/index.html
http://www.cs.nuim.ie/~jpower/Research/instrument/
http://www-128.ibm.com/developerworks/java/library/j-jip/

Profiling java apps using DTrace

2008-06-25T12:57:00.003+02:00

- monitoring operational performance of java apps
- finding bottlenecks
- examining cache effects
- example : heavily IO-bound app
- OS/X / Solaris

idea :
(a) current java profiling options : (what/where)
(b) java options on system/level profiling (we can debug / but cannot to the strace-like detailed to-syscall-mapping, oprofile-like stuff etc)

dtrace - cool caps :

references :

http://www.devx.com/Java/Article/33943/0/page/3
http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_Java
http://developers.sun.com/solaris/articles/java_on_solaris.html
http://developers.sun.com/solaris/articles/dtrace_ajax.html

Managing shared state in Erlang

2008-03-29T12:57:00.002+01:00

Though a functional language, with no apparent shared state, we can trivially implement state in erlang by (ab)using the single-bounded value concept (each value can be bound only once)

whereis(list_to_atom("portServer" ++ integer_to_list(PORT))).

Bloom filters for profit & fun

2008-03-29T12:55:00.002+01:00

Perfect Hash Functions for fun and profit

2008-03-29T12:54:00.002+01:00

Breakpoint

2008-03-27T17:52:00.002+01:00

Okie, cut here

This is another attempt into actually starting to write something ;)
So, stay tuned for a bunch of stupid posts :) I need something in order to get started. Hopefuly, I might be able to actually post something usefull some day. (in which case i will erase all the posts until that day :) ). So, some value after all .... these posts won't last forever .... :) :) read them while they are still here ... and have mercy :)

Linux buffer cache & how to disable it (and why ?)

2007-12-31T15:36:00.000+01:00

Linux buffer cache provides a excelent mechanism for black-box performance optimization by a modest cost or max 2 memory hits (hit 1 -> page not found -> fetch buffer from disk to buffer cache (free heap space) -> hit 2 -> found in memory -> get block).

However, there are 2 cases when we would want to disable such behaviour :

(1) When 2-hit is too much of price to pay, we might want to think about direct io (O_DIRECT/ madvise()-style), and doing the memory buffer management by hand from userspace - often done for db cache management

(2) When we want to do unbiased benchmarking of heavily IO-dependent software (usually a single shot-benchmark is ok - pages are on the disk), but upont 2nd take , most of the pages remains in buffcache, which results in better performance, so no real metric can be imposed afterwards - so the only way to do the proper benchmark would be to disable buffacahe

Currently, we are interested in (2) , so here are couple of ideas how that could be done :

(a) create a non-trivial file of available memory size and write a simple code that mmap()'s it
(b) tune the swappiness kernel knob (/proc/sys/vm/swappiness) to 0 (proc memory over buffers), and fork() some ~64k dummy (nontrivial) processes :) - this should do the trick
(c) mounting the partition as raw device (no buffering then) - but this is usually highly impractical
(d) seting the O_DIRECT flag for every open() in the source (this is often tedious unless open() is invoked through a wrapper - a nice argument for doing so in such applications). We could write a simple wrapper (if possible for doing so):

int dopen(char *file) {
return open(file, O_DIRECT);
}

or in case of fopen() :

FILE *dfopen(char *file, char *pern) {
int fd = open(file, O_DIRECT | O_RDONLY);
return fdopen(fd,perm);
}

(note theat O_DIRECT is conditionaly defined by _GNU_SOURCE , so don't forget to use -D_GNU_SOURCE flag when building)
(e) reboot the machine (if you are really desperate) :)
(f) allthough one might think that invoking "sync" from command line might do the trick - it actually just flushes *changed* blocks to the disk - which is not actually what we need (flushing the read-buffered blocks)
(g) the dirty way of doing (a) would be simply touching a very big file (for example dd if=/dev/urandom of=file_4GB bs=1024 count=4000000) and doing 'cat file_4GB > /dev/null'
(h) writing a simple code that allocates a huge chunk of memory and locks the allocated pages (using mlock() or similar) - thus reducing the available buffcache to a arbitrary size
(i) doing fcntl() with F_NOCACHE on all file descriptors in the source code - again quite tedious especially if there is no wrapper for open() call in the code
(g) using madvise() mechanism for telling kernel that the pages allocated won't be used in the future (which should result in kernel freeing the allocated pages from buffcache immidiatelly):

size_t dfread(void *ptr,size_t size,size_t nmemb, FILE *stream) {
size_t n;
n = fread(ptr,size,nmemb,stream);
madvise(ptr,size*nmemb,MADV_DONTNEED);
return n;
}

(h) if we're to just flush the entire buffercache, on the 2.6.16+ kernels we could use a a "drop caches" mechanism to free all pages from buffcache :

echo 1 > /proc/sys/vm/drop_caches

Topics in compiler implementation

2006-12-03T00:08:00.000+01:00

Some compiler-related ideas & problems (Appel : Modern compiler implementation in Java):
- Instruction selection : Machine instructions are represented by tree pattern. The optimal code is generated by selecting tree patterns which form the minimum tree. Thus, instruction selection becomes a task of tiling the tree with minimal set of tree patterms. There is a number of approaches to this problem, most notably, the Maximal Munch algorithm and Dynamic Programming.
In pratice, tree pattern - based approach can be a problem when optimizing for cics, rather than risc machine...
- Liveness Analysis : In order to optimize the register allocation, the compiler must analyze which variables are used ("alive") during every segment of program (for example, we can observe every basic block as one segment). Variables which are not used at the same time can be allocated in the same register. We observe the control-flow-graph of the program and determine the "live" variables at every edge of the graph. Then, we can perform graph partitioning in order to divide the program into segments with a number of alive variables which correspond to number of system registers. Again, we should do this in a optimal manner (every segment should allocate all registers, but among segments there should be minimum needed transfer of data between registers and memory).

Notes on automata theory

2006-12-02T14:39:00.000+01:00

Remarks from Motwani, Hopcroft & Ullman : Introduction to Automata Theory, Languages and Computation (during 45min reading) :
Chapter 2 : Finite Automata
- Give an example of practical communication/transaction protocol which can only be analyzed via automata model
- Examples of structural patterns which require formal verification of correctness ?
- Complexity bound for automata which require formal verification?
(even in the case of 2 simple, say 4-state automata, their single interconnection yields a product of automata resulting in 16-state automata )

- Optimal representation of DFA ? (space-efficient Transition Tables). Alternative methods of representation ?
- Algorithm-efficient representations ? (ex. finding all loops in automation by observing transition table -> can we speedup the algoritm by using structure other than table)

A NFA models the ability of input to "guess" the right state. Actually this can be seen as increasing complexity in order to simplify the description. The automata can reduce complexity of the problem by increasing the description. However, there are problems that cannot be treated in this manner.

A NFA is more descriptive than DFA, while there should be a 1-1 mapping NFA <-> DFA (Using a Tompson's algorithm , we can efficiently construct the DFA corresponding to NFA). Still, a NFA of size N corresponds to a DFA of size N+K . How do we prove the 1-1 mapping then ?

- [Offtopic] -> how can we efficiently create a program for automata generation via feeding it with a list of input examples and accept/reject results for each example ?

& stuff.....

2006-12-02T14:36:00.000+01:00

Here it is... I will be typing some random stuff on development, cs & stuff...