ISAM Database - IO and NIO

[zparticle]

I often have a need for a small easy to use database so I decided to whip up a quick library for ISAM like database access. I have the basics working now and in a couple of days will be adding indexing using single and concatenated keys. Might also write a little database explorer in swing.  A friend wants me to add network access to it as well.

I'm also using this project to learn a bit about NIO. The first version I'm working on now is all regular io package stuff. I will be converting it to NIO after it has all the features I want.

If anyone else has need for this sort of thing let me know and I'll post the library up when it is finished.

If you haven't used ISAM before setting up a database is very easy, mine is a little bit different from ISAM but has the same outcome:

//---------------------------------
// create the record format and add the coulmns you need
//---------------------------------
FixedRecordFormat format = new FixedRecordFormat();
format.addIntegerField("ID");
format.addStringField("First Name",30);
format.addStringField("Last Name",30);

//---------------------------------
// create the empty database files
//---------------------------------
String location = "\temp\mydatabase";
FixedRecordFormatFile database = new FixedRecordFormatFile(location);
database.create();
database.close();

//---------------------------------
// add a few records
//---------------------------------
database.open(); // notice that once a database has been created you don't have to set up the record format

Record rec = new Record(database.getFormat());
rec.setIntegerFieldValue("ID",1);
rec.setStringFieldValue("First Name","Scott");
rec.setStringFieldValue("Last Name","Shaver");

database.addRecord(rec);
rec.setIntegerFieldValue("ID",2);
rec.setStringFieldValue("First Name","Bert");
rec.setStringFieldValue("Last Name","Shaver");
database.addRecord(rec);

//---------------------------------
// other commands
//---------------------------------
database.rewind(); // set the current record position to the first record
database.setCurrentRecord(recIndex); // set a specific record position as the current record
database.nextRecord(); // move to the next record position
database.previousRecord(); // move to the previous record position
database.deleteRecord(); // delete the record at the current record position
database.writeRecord(recordToWrite); // write a recrod at the current record position
database.compact(); // remove any empty records from the database file
database.getRecordCount(); // find the total current capacity of the database file

[zparticle]

Okay so I have fiished the regular IO version of the database and converted it to NIO. However I'm having difficulty with a couple of items.

First, there seems to be no way to increase the size of a file that is memory mapped. This means that in the nio version every time I create a new record at the end of the file I have to:

a> close the file
b> open it with regular io and extend it
c> open the file with nio again
d> write the new record into the empty spot at the end.

This is obviously extreemly slow for lots of inserts at the end of the file. Any clue how to get around this?

Second, there is an option to compact the database file removing all empty/deleted records. So I move all the non-empty records to the top of the file and then attempt to truncate the file. This causes a problem just like the first one above. However this time when I try to do the close, truncate, open flow I get an IOException with a windows system error message:

<b>The requested operation cannot be performed on a file with a user-mapped section open.<b>

This happens on the call to truncate the file even though I have previously closed the RandomAccessFile and  FileChannel objects and set the MappedByteBuffer to null. Help?

[blahblahblahh]

This is not a helpful response, I'm afraid, but may be useful anyway . (EDIT: sorry but I have v.little time at the moment, so not enough time to go look up my own refs and past experiences; hopefully what I just spouted is at least slightly helpful)

Memory-mapping in 1.4.x has sufficiently many serious bugs as to be considered unusable by many people . It's a bit like Socket NIO in 1.4.0 and 1.4.1 - there were so many bugs that it COULD NOT be used in real applications, unless you were willing to make huge concessions (one of which I recall was "never run on windows" in several not uncommon scenarios).

So, first port of call is the JDC Bug Parade and an hour of searching for outstanding mem-mapping bugs I'm afraid . Pay particular attention to closed and wontfix bugs - IIRC there's a couple of those that smell more of engineers trying to avoid being the Sun employee who opens the can of worms ... than of things that are genuinely not bugs.

Finally, mem-mapping is often NOT better than using vanilla methods...so much so that IIRC the API docs currently say "we suggest you only use mem-mapping where you've experimentally determined that it definitely makes a large positive difference to your particular application".

i.e. mem-mapping is really still a "beta" technology in the JDK. From micro benchmarks I've seen and done, it can be hard to find situations where mem-mapping brings significant performance boosts. At a guess I'd say this is because it's one of those areas of the JDK which so far has had very few staff and so Sun haven't had time to do much more than "bare bones" implementations that are slow inefficient and buggy. (based upon extensive experience with the general rubbishness of the nio.* and socket nio.channels.*, and it's rapid evolution through successive 1.4.x point releases...)

Games published by our own members! Check 'em out!

[Mark Thornton]

As the submitter of several memory mapping bugs, I have to agree with blah³.

This one is number 11 on the top 25 RFE list
http://developer.java.sun.com/developer/bugParade/bugs/4724038.html

[zparticle]

Thanks guys. I guess I'm screwed for now.

On the up side I do see huge gains from NIO on the reading side. So for now I suppose I could create static databases using the IO version and read them with NIO. For more dynamic databases I'll simply only use the IO version of my lib.