Me and a coworker found something interesting about Java IO the other day.
We were getting an OutOfMemoryError when trying to write the contents of a big byte array to a file. We all hate OutOfMemoryErrors: you never know what really caused it, specially when it happens in a production environment with tens or hundreds of simultaneous threads serving requests at the same time. The process of fixing it usually involves profiling the memory in search for memory leaks, which can be very time consuming.
This OutOfMemoryError was different though. It happened inside FileOutputStream.write(), which is a native call. I happened to have my personal laptop at the office that day, where I have the OpenJDK source code. I have NetBeans 6.5 setup with the OpenJDK projects, which makes navigating through its C++ code a breeze.
My discovery was interesting: when you write a byte array to a file, Java will copy the contents of the byte array into a native array and pass this native array to the native IO function. If the array size is equal or smaller than 8 Kbytes, it will copy into an area in the stack. If the size is greater than 8Kbytes, it will allocate memory in the native heap and use that area instead.
In our case, the byte array was something around 5 MB. Because of some native libraries that we use, we have to live with 32 bits JVMs, and the heap size of our application servers is set to 1024 MB. Along with a 196 MB PermGen size, plus the memory used by those native libraries, it left us with very little room left for the native code. The JVM failed to allocate the 5 MB native array, which was probably caused by heap fragmentation, and threw an OutOfMemoryError from the native side.
The longer term solution will be to switch to a 64 bit JVM, and the mid term solution will be to review the memory configuration of our application servers. In the short term, however, the solution was very simple: instead of writing the big byte array in one go, all we had to do was to write a loop to write in chunks of 8 Kbytes and the problem was gone!
Having the JDK source code at hand saved us a lot of time of guessing and going in the wrong directions in the dark.
Side note: Heap fragmentation is usually not a problem in Java, because the various JVM garbage collectors put a great deal of effort into compacting the heap (the most advanced GCs don't ensure complete compaction but they do their best). C doesn't have managed pointers, so heap fragmentation can become a real problem for C and C++ programs or libraries.
Tuesday, January 20, 2009
Subscribe to:
Post Comments (Atom)
Posts
10 comentários:
I think there is another solution: the nio MappedByteBuffer. What do you think?
Haven't you tried using a BufferedOutputStream? That way you can write as much bytes as you want, but only be written to the file at 9KB chunks
Meant 8KB, sorry
vrg: it could work in a general case but probably in my situation it would also cause problems because the process address space was almost exhausted, but it is something worth considering!
Pablo: we were actually using it, but the BufferedOutputStream bypasses itself if the data being written is larger than its internal buffer, which kind of makes sense :) If its purpose is to save the number of writes to the underlying stream, why should it make more calls when it receives a large input? :)
"The process of fixing it usually involves profiling the memory in search for memory leaks, which can be very time consuming."
Not all out memory errors are related to memory leaks. In fact a significantly large portion are just memory capacity issues due to a relative high number of concurrent memory hungry (alloc) requests.
William
By the way I still do not understand what exactly the cause is. Even if the byte array is relatively large (5M) , which is a problem (waste) in itself, every IO class allows one to specify a range in the writing.
Kind regards,
William
William: about the majority of the OOM errors being caused by memory capacity issues: you are totally right :)
William: the problem was exactly that we weren't writing the array in chunks. Since we had the whole array at hand, it was simpler to just call out.write(array) only once, which caused the memory error. The solution was to do exactly as you suggested.
This is really useful, thanks :-)
I'm interested in why the JVM copies the byte array, instead of using the original one.
since the IO call doesn't return till all of the data was written to the IO pipe, what the copy is needed for? Another thread might tamper with the byte array? that's the responsibility of the user to avoid from such acts. Any idea?
Post a Comment