C / C++ / MFC

1-Jun-05 5:55

Hi guys

I apologize for reposting this question but probably I didn't reach enough people last night and only got one response that didn't help me much (sorry Chris but thank you).

I'm still strugling with the problem of reading a huge file in the most efficient way. I was suggested last week to "read the file into memory as one big chunk". I think that the advice refered to reading the data in binary format instead of text. Cool... so first I have to translate my text files into binary and then read them... Problem is that all examples that I've found refer to either read a single number or things such as a struct with phone directory...

How about reading a 2 dimensional array of doubles as text, save it as binary and then load it into another array after reading the binary file???

examples???

Thanks a million.

David Crow1-Jun-05 7:07

David Crow

1-Jun-05 7:07

A file is neither binary nor text as it resides on disk. It all depends on how the file is processed that determines whether it is binary or text. You can open the file and read all of it into memory with something like:

HANDLE hFile = CreateFile("myfile", GENERIC_READ, ...);
LPBYTE lpBuffer = new BYTE[10240];
DWORD dwRead;
ReadFile(hFile, lpBuffer, sizeof(BYTE) * 10240, &dwRead, NULL);
CloseHandle(hFile);

Now you can do whatever you want with lpBuffer. You might also benefit from a memory-mapped file.

"Ideas are a dime a dozen. People who put them into action are priceless." - Unknown

knapak1-Jun-05 8:21

1-Jun-05 8:21

Hi
Sorry if this sounds stupid, please bear with me since I'm a self instructed amateur.

If files are neither text or binary, what's the point of using something like:

To write a binary file

int MyNumber = 10;<br />
ofstream FileOut(FileName,ios::binary);<br />
FileOut.write((char*) &MyNumber, sizeof MyNumber);<br />
fout.close();

and to read

ifstream FileIn(FileName, ios::binary)<br />
FileIn.read((char*) &MyNumber,sizeof MyNumber);<br />
FileIn.close();<br />

Thanks a million you for your help

S. Senthil Kumar1-Jun-05 8:33

S. Senthil Kumar

1-Jun-05 8:33

ios::binary is used to tell the ifstream and ofstream classes to treat the data as binary. If you don't specify ios::binary and try to read a binary file and if the binary file happens to have a non-text byte value, it'll stop there or throw an exception.

You can open any file in binary mode though, because ifstream and ofstream will not try to interpret them as text characters.

Regards
Senthil
_____________________________
My Blog | My Articles | WinMacro

knapak1-Jun-05 9:23

1-Jun-05 9:23

OK... so I wrote the following code and it writes and reads the file with a map. It compiles, runs and gives the requested output, but apparently there's a runtime problem since after execution a window pops indicating that there was a problem (one of those windows to send information to the creator of the program)... I can't see what's wrong but I know where the problem is since I commented different sections of the code and narrowed it down to the marked fragment below.

Here's the code:

int main()<br />
{<br />
	typedef map<int, double> IMAP;<br />
<br />
	IMAP Grid, NewGrid;<br />
<br />
	int IntValue1, rows = 3;<br />
	double DouValue2;<br />
<br />
	for(int i=0; i < rows; i++)<br />
	{<br />
		IntValue1 = i + 1;<br />
		DouValue2 = i * 2;<br />
		Grid.insert(IMAP::value_type(IntValue1, DouValue2));<br />
	}<br />
<br />
	IMAP::const_iterator IteratorG = Grid.begin();<br />
<br />
	cout << "Original Map" << endl;<br />
	while (IteratorG != Grid.end() )<br />
	{<br />
		cout << IteratorG->first << " " << IteratorG->second << endl;<br />
		IteratorG ++;<br />
	}<br />
<br />
	ofstream FileOut("C:/MyBinary.bin" , ios::binary);<br />
	FileOut.write((char*) &Grid, sizeof Grid);<br />
	FileOut.close();<br />

THE ERROR IS FROM HERE

ifstream FileIn("C:/MyBinary.bin", ios::binary);<br />
FileIn.read((char*) &NewGrid,sizeof NewGrid);<br />
FileIn.close();<br />

TO HERE

	IMAP::const_iterator NewIteratorG = NewGrid.begin();<br />
<br />
	cout << " " << endl;<br />
	cout << "New Map" << endl;<br />
	while (NewIteratorG != NewGrid.end() )<br />
	{<br />
		cout << NewIteratorG->first << " " << NewIteratorG->second << endl;<br />
		NewIteratorG ++;<br />
	}<br />
	return 0;<br />
}<br />

Thanks a million!

S. Senthil Kumar2-Jun-05 1:15

S. Senthil Kumar

2-Jun-05 1:15

Writing out a std::map is not a good idea. It internally uses pointers and dynamically allocated memory, so writing the map will only write out the pointers and not their values. For eg

class Test
{
public:
   int *p;
};

int main()
{
   Test test;
   test.p = new int();
   *(test.p) = 100;
}

Writing out test to a file will write out p, which is just a address (and which will change for every program run). Because you're reading it back in the same program, you're alteast able to read back the values. Try reading the file after shutting down the program that wrote to it, you'll find that iteration itself crashes.

I'd suggest writing out the values to file, manually and then reading them back, (again manually).

Regards
Senthil
_____________________________
My Blog | My Articles | WinMacro

knapak2-Jun-05 11:48

using library in different solution

2-Jun-05 11:48

Hi Senthil

Thanks a lot for your response. I've actually found that you are partly right. I can't send the whole map at once for the reasons that you point out. However, if you send the map element by element it does work like a charm. For me, the reason to work with this containers is because I need to read a humongous file and then find a record or a series of records associated to a tag or tags (in which case I'd use a multimap). These containers have algorithms that make the search process incredibly fast and easy. If you are interested in the new version of the code (the one that works) I'll be happy to post it.

Cheers

Carlos

_indra_m1-Jun-05 5:49

_indra_m

1-Jun-05 5:49

I try to make umanaged C++ class and managed C++ library.
I put it in C:\MathLibrary\Arithmetic.h , \MathLibrary.lib and \ManagedMathLibrary.dll.

But when I try to use it in my project such as put the dll in reference and change the project properties to tell the compiler the right path -> for unmanaged class and unmanaged lib, it doesn't work. What is the problem? Smile | :)

Multiple files

Andrew Admire1-Jun-05 4:35

1-Jun-05 4:35

How would I be able to restrict the size of a text file then create another file to hold the over flow?

M.Mehrdad.M1-Jun-05 5:48

M.Mehrdad.M

1-Jun-05 5:48

Simply, You Can Count The Number Of Bytes In the First File, !!!! Laugh | :laugh:

Andrew Admire1-Jun-05 6:32

1-Jun-05 6:32

ha, ha. No, really how do you create multiple files?

David Crow1-Jun-05 7:32

David Crow

1-Jun-05 7:32

Right before writing to the file, check to see if the file's current size plus the amount of text to be written would exceed some threshold. If it would, close the file, open/create the next file, write the text.

"Ideas are a dime a dozen. People who put them into action are priceless." - Unknown

Andrew Admire1-Jun-05 8:07

1-Jun-05 8:07

Am I going in the right direction? or what am I missing?

ifstream inClientFile ("diction.txt", ios::binary);
if(inClientFile != sizeof (10000));
{
ofstream outClientFile ("diction.txt", ios::app);
{
for (int a=0; a<=10; a++)
{
for (int aa=0; aa<10; aa++)
outClientFile<

David Crow1-Jun-05 8:16

David Crow

1-Jun-05 8:16

Andrew Admire wrote:
if(inClientFile != sizeof (10000));

This is not the correct way to check a file's size. I'm not familar with istream/ostream/etc but I would look into the tellg() or tellp() methods.

"Ideas are a dime a dozen. People who put them into action are priceless." - Unknown

S. Senthil Kumar1-Jun-05 8:36

S. Senthil Kumar

1-Jun-05 8:36

sizeof(10000) will only give you the size of an integer (typically 4 bytes). For finding the size of the file opened by ifstream, I've used this trick

inClientFile.seekg(ios::end);
int size = inClientFile.tellg();

Not sure about the parameter to seekg, but it used to work. Don't know if there are any better methods though.

Regards
Senthil
_____________________________
My Blog | My Articles | WinMacro

Andrew Admire1-Jun-05 9:49