Encountered while running 50-terminal TPCC on dev01:
Exception in thread "Thread-21" java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:659)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:113)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:305)
at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:75)
at sun.nio.ch.IOUtil.write(IOUtil.java:87)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:689)
at com.persistit.MediatedFileChannel.write(MediatedFileChannel.java:238)
at com.persistit.JournalManager.flush(JournalManager.java:1236)
at com.persistit.JournalManager.prepareWriteBuffer(JournalManager.java:1318)
at com.persistit.JournalManager.writeTransactionToJournal(JournalManager.java:1063)
at com.persistit.Transaction.flushTransactionBuffer(Transaction.java:1018)
at com.persistit.Transaction.commit(Transaction.java:723)
at com.persistit.Transaction.commit(Transaction.java:642)
at com.persistit.tpcc.PersistitTransactionEngine.newOrderTransaction(PersistitTransactionEngine.java:632)
at com.persistit.tpcc.jTPCCTerminal.executeTransaction(jTPCCTerminal.java:147)
at com.persistit.tpcc.jTPCCTerminal.executeTransactions(jTPCCTerminal.java:93)
at com.persistit.tpcc.jTPCCTerminal.run(jTPCCTerminal.java:50)
at java.lang.Thread.run(Thread.java:679)
Exception in thread "Thread-32" java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:659)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:113)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:305)
at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:75)
at sun.nio.ch.IOUtil.write(IOUtil.java:87)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:689)
at com.persistit.MediatedFileChannel.write(MediatedFileChannel.java:238)
at com.persistit.JournalManager.flush(JournalManager.java:1236)
at com.persistit.JournalManager.prepareWriteBuffer(JournalManager.java:1318)
at com.persistit.JournalManager.writeTransactionToJournal(JournalManager.java:1063)
at com.persistit.Transaction.flushTransactionBuffer(Transaction.java:1018)
at com.persistit.Transaction.commit(Transaction.java:723)
at com.persistit.Transaction.commit(Transaction.java:642)
at com.persistit.tpcc.PersistitTransactionEngine.paymentTransaction(PersistitTransactionEngine.java:1116)
at com.persistit.tpcc.jTPCCTerminal.executeTransaction(jTPCCTerminal.java:185)
at com.persistit.tpcc.jTPCCTerminal.executeTransactions(jTPCCTerminal.java:81)
at com.persistit.tpcc.jTPCCTerminal.run(jTPCCTerminal.java:50)
at java.lang.Thread.run(Thread.java:679)
Need to determine whether this is merely due to a configuration issue or is a genuine bug. I have never seen this before and have used this configuration on dev01 many times.
Upon further investigation, it appears that IOUtil.java potentially allocates and caches a direct ByteBuffer for each thread. The cached buffers are head in SoftReferences, but I assume multiple threads could be backed up waiting to write to the journal, and therefore multiple threads could temporarily hold strong references to their DirectByteBuffers; this could exhaust the address space available for direct buffer allocation?
Bottom line is that the JournalManager's _writeBuffer field should be allocated as a direct buffer to avoid all the extra allocation and copying. There is a reference to the array() method in readFully that will need to change; the arraycopy operation can be done instead by a carefully executed call to put(bb) where bb is the destination ByteBuffer.
This change could improve performance, due primarily to gc and to a lesser extend, avoided array copying.