I've tried compressing 100+ MB large images in C++ program so I could see better where OpenJpeg fails - it was always some memory allocation function. OpenJpeg uses
a lot of memory for images. For each channel Width*Height*SizeOf(Integer) large buffer is used. So for 100MB 24bit bitmap it's 400MB of buffers. There's also additional memory used to store compressed image. So basically you run out of free memory.
This all should be fixed in OpenJpeg version 2 (unfortunately still looks like far future).
I checked it from OpenJPEG web site and realized that library does not support 32b BMP files. After i convert it to 24b it worked. It was not related with size.
You can store 1bit as well as 1024bit per pixel images in JPEG2000, so 32bits are not a problem for OpenJpeg.
24b image only used less memory.