In addition to the correct answer by Espen Harlinn:
Breaking a file into chunks will hardly help you, unless those chunks are of different natures (different formats, representing different data structures), so they were put in one file without proper justification.
In other cases, it's good to use the big file and keep it open. There are cases when you need to split the file in two pieces. This is just the basic idea; see below.
So, I would assume that the file is big just because it represent a collection of object of the same type or few different types. If all the items are of the same size (in file storage units), addressing then is trivial: you simply need to multiply the size by required index of the item, to get a position parameter for
Stream.Seek
. So, the only non-trivial case is when you have a collection of items of different size. If this is the case, you should
index the file and build the
index table. The index table will consist of the units of the same size, which is typically the list/array of file positions per index. Due to this fact, addressing to the index table can be done by index (shift), as described above, and then you read position of the "big" file, move file position there and read data.
You will have 2 options: 1) keep index table in memory; you can recalculate it each time; but it's better to do it once (cache) and to keep it in some file, the same or a separate one; 2) to have it in a file and read this file at required position. This way, you will have to seek the position in the file(s) in two steps. In principle, this method will allow you to access files of any size (limited only by
System.Uint64.MaximumValue
).
After you position in a stream of a "big" file, you can read a single item. You can use
serialization for this purpose. Please see:
http://en.wikipedia.org/wiki/Serialization#.NET_Framework[
^],
http://msdn.microsoft.com/en-us/library/vstudio/ms233843.aspx[
^],
http://msdn.microsoft.com/en-us/library/system.runtime.serialization.formatters.binary.binaryformatter.aspx[
^].
A fancy way of implementing all the solutions with index table would be encapsulating it all in the class with
indexed property.
—SA