We have been considering introducing the use of an archival file format such as zip or cab into the CDB in order to reduce the file count. There are many issues to consider in selecting a solution that simultaneously meets all of our requirements:
a) We want to minimize file access overhead, hence a scheme that reduces the overall file count for the same informational content is a desirable solution, e.g. bigger files
b) We want to deploy CDB databases as efficiently as possible, hence a scheme that reduces the overall file count for the same informational content is a desirable solution, e.g. bigger files.
c) We want to keep the granularity of the CDB informational content as fine as possible; this keeps the memory footprint of client-devices and DB tools memory footprints as small as possible, e.g smaller files or archival files formats which allow direct access to portions of the file without a priori loading the entire file in memory
d) We want to maintain/improve performance of CDB editing tools; this again implies that the granularity of the CDB informational content should be as fine as possible; furthermore, it should be possible to insert content within the file without having to re-shuffle the entire contents of the file. So the solution is either smaller files, or an archival file formats which are easily editable through internal manipulation of pointers.
e) We want to support "Dynamic Terrain" capabilities whereby one or more simulations (weaponry simulation, weather simulation, computer generated forces simulations, etc...) are simultaneoulsy modifying the CDB on-the-fly. This again implies that the granularity of the CDB informational content should be as fine as possible; furthermore, it should be possible to insert content within the file without having to re-shuffle the entire contents of the file. So the solution is either smaller files, or an archival file formats which are easily editable through internal manipulation of pointers.
As you can see, there are many conflicting needs to satisfy simultaneously. We are currently leaning towards an archival file format that would allow direct access to portions of the file without a priori loading the entire file in memory. We are not aware of an archival file format that address items d) and e), and this is the main reason why we have held off in making any changes to the CDB in this respect.
Any suggestions anyone...
Craig
Hi there!
I attended to the CDB User group meeting in I/ITSEC in Orlando. I must say it was a very constructive session and I want to congratulate to all speakers for their contribution.
Reviewing the information you gave us about CDB API and new releases, an issue arises to me.
You told us that for CDB 4.0 release you plan to group files in bigger ones or maybe in zipped files and also that you plan for CDB API 1.1 to enable a new feature that allow to lock files while they are being read by some client. This locking system is a good solution when trying to enable multithread and don't want multiple accesses to the same file but when files become larger and larger it has a problem.
When flying a database is almost a fact that several clients are in the same geographic area. The bigger are the files, the more probable is that two or more clients ask for the same file, provoking a request queue that can seriously affect the performance.
Did you already think a solution for this or it is an open issue?