|
I am debugging a Visual C++ program, and my job is to speed up the file access. Currently the program is reading in a 600 MB ascii-text file using fopen and fscanf, etc., in text mode. Is there an easy way to speed this up? I have tried using streams and they aren't defined. Would changing it to binary make it faster?
Thanks so much for your help!
Jennifer
|
|
|
|
|
JenniferLeonard522 wrote:
I have tried using streams and they aren't defined.
How do you mean, they aren't defined ? Regardless of speed issues, moving your file handling code to c++ is a positive step.
Christian Graus - Microsoft MVP - C++
|
|
|
|
|
When I use ifstream and cin they aren't defined. the code I am debugging currently uses fopen, fscanf, fprintf, etc.
Thanks!
Jen
|
|
|
|
|
Anonymous wrote:
When I use ifstream and cin they aren't defined.
Did you include them ? ifstream is in fstrem and cin is in iostream. You need to scope them in std, or pull them into the global namespace with using statements as well.
Anonymous wrote:
the code I am debugging currently uses fopen, fscanf, fprintf, etc.
Yeah, the world is full of crappy code that uses those instead of C++. It's a common problem.
Christian Graus - Microsoft MVP - C++
|
|
|
|
|
For reading files of this size, I don't recommend using C++ stream classes.
Instead, I recommend you use file mapping API functions, which will greatly increase the speed of your code.
For more info, look at your help files for the following API functions:
MapViewOfFile
CreateFileMapping
UnmapViewOfFile
For example code, check out the following links:
http://code.axter.com/MapFileToMemory.h
and
http://code.axter.com/mapfile2mem.cpp
The C++ stream classes are reliable, but not very speedy.
If you have a large file, and you need speed, they're not the best choice.
Top ten member of C++ Expert Exchange.
http://www.experts-exchange.com/Cplusplus
|
|
|
|
|
Hello.
One way to speed up file handling (while maintaining ths stream I/O) could be to use fread() and fwrite(), using a buffer size that match the disk sector size of the file. Since the normal sector size is 512 bytes, read and write blocks that is an even multiplier of 512.
(I guess the most effective disk I/O would be to read/write the size of a complete disk cluster at once, but I'm not sure).
This means that you have to use your own code to extract/create the data (text lines) in the buffer, but that job has to take place anyhow. If it´s done in the runtime library or in your own code dosen't really matter (provided that your own code is efficially written).
Another way could be to use the native Win32-API and to use overlapped I/O. But it works (logically) in the same way as fread/fwrite, which means you still have to write your own code to handle your buffer.
|
|
|
|
|
I rated your answer as 5 (but I do not know if it took).
You are correct about reading and writing a cluster at a time (or a multiple there of). The problems may start to appear with how the memory required is managed, which the question ignores (I,m not going into that).
The simpilest solution (if they are using MFC), is to use a CMemFile (which side steps the issue entirely and allows MFC to handle it).
INTP
Every thing is relative...
|
|
|
|
|
Thank you for all the suggestions. I looked into CMemFile, but it said "
Because CMemFile doesn't use a disk file, the data member CFile::m_hFile is not used and has no meaning." I read through the description and didn't see how to use it to read a large data file from disk.
I also looked into writing the file a cluster at a time, but the problem is that my file is constantly growing (every 10 minutes it gets more information). So right now it is 600MB, but in a couple months it will be 800MB. What would the quickest way be to read a large file like this?
Thanks!
Jen
|
|
|
|
|
The first statement that the m_hFile is not used and has no meaning is essentially true. That is the file handle was only needed long enough to load the file into memory. I (incorrectly) assumed that was not the case (blast it).
The fasted way to read a file is a cluster at a time. You do not need to actualy care about this detail, since you are only reading the file as a whole.
Look into sharing file read access. This will not speed up the actual read time, but will reduce the preceived time (if used properly). What this means, is that (if) the file is only changed by adding to it, then you just need to read the additional information added to the file.
What I mensioned above is also true if you do not have file sharing read access. What I mean is that (assuming the other application closes the file, after writing to it) you can check if the file size has changed on a regular basis, and just read the changes.
What all that bull boils down to is this: Only read what has changed and nothing more.
I hope that helps, because I cann't explane all the ideas that have poped into my head.
INTP
Every thing is relative...
|
|
|
|
|
I am doing a project that requires two monitors, one standard display and one 3D. The video card is cloning the display from one to another (not a dual monitor situation where the display is split).
This is fine but here's my problem:
I need to make one monitor sleep while the 3D monitor shows something 3D Upon the mouse movement, make the 3D monitor sleep while the other monitor is powered-up to show a form, then back to the 3D monitor when the form is submitted.
I have found this code:
// Turn off monitor
Sleep(500); // Eliminate user's interaction for 500 ms
SendMessage(HWND_BROADCAST, WM_SYSCOMMAND, SC_MONITORPOWER, (LPARAM) 2);
// Turn on monitor
SendMessage(HWND_BROADCAST, WM_SYSCOMMAND, SC_MONITORPOWER, (LPARAM) -1);
// Low power monitor
Sleep(500); // Eliminate user's interaction for 500 ms
SendMessage(HWND_BROADCAST, WM_SYSCOMMAND, SC_MONITORPOWER, (LPARAM) 1);
But I only need to make one of the monitors sleep at a time. Can this even be done?
|
|
|
|
|
Use dual View Mode on your video card to have to separate display, and then you can create Direct3D surface on the second monitor, and to sleep it...
NG
|
|
|
|
|
I am working in VC++ 6 under Windows NT/2000. I need to access the OS environment variables in code similar to the following in VB:
m_strAllUsersProfile = Environ("ALLUSERSPROFILE")
How do I do this?
Thanks...
|
|
|
|
|
How about getenv() ?
"One must learn from the bite of the fire to leave it alone." - Native American Proverb
|
|
|
|
|
I am missing something to get it to work; can you supply more info/working example with any headers, whatever? for instance, accessing ALLUSERSPROFILE or other?
thanks
|
|
|
|
|
Kirk Woller wrote:
can you supply more info/working example
See here[^]
|
|
|
|
|
Hello there guys.
I have something to ask about how can I customize the closing function of an application. Using MFC/VC++, a created application is run, and with that application window, there are some link that will display in a new IE window.
I wanted that if I close the main application, all IE window displayed will also close. Please tell me how I can do that, I don't have much idea about it.
Thanks.
With great power comes a great responsibility.
|
|
|
|
|
Hi,
maybe you can customize your virtual int CWinApp::ExitInstance(); method. I don't know the code you need, but if I knew, I would put it there.
Regards
Achim Klein
We can do no great things, only small things with great love. - Mother Theresa
|
|
|
|
|
Thanks for your reply.
I'll try to look at it and maybe try the ::FindWindow() and SendMessage() function calling WM_CLOSE.
Thanks again.
With great power comes a great responsibility.
|
|
|
|
|
I'm using the same source files on windows and on linux
<br />
#ifdef WIN32<br />
fd = open (*it, _O_RDONLY | _O_BINARY);<br />
#endif<br />
<br />
#ifdef __unix<br />
fd = open (*it, O_RDONLY);<br />
<br />
#endif<br />
if (fd == -1) {<br />
return 1;<br />
}<br />
<br />
Then I do stuff
and finally:
#ifdef WIN32<br />
_close(fd);<br />
#endif<br />
<br />
#ifdef __unix<br />
close(fd);<br />
#endif
Its fine on windows, crashes with an invalid pointer on linux at the close. I dont know what this means, but if instead of _close, I use close() on windows, it crashes on windows too.
How do I close my files on linux?
Thanks,
sb
-- modified at 15:58 Tuesday 20th September, 2005
|
|
|
|
|
Is it possible that __unix and WIN32 exist at the same time?
bonosa wrote:
How do I close my files on linux?
What close-type functions exist in the io.h file on that platform?
"One must learn from the bite of the fire to leave it alone." - Native American Proverb
|
|
|
|
|
No. The flags are distinct and working ok.
As for io.h on linux the one i found in /usr/include/asm/ didnt have the word close in it. The unix man for close() seems to indicate that its what I want (in unistd.h). But it crashes.
Theres a lot of code written that uses _open and _close, so I dont want to change over to fopen, fclose. I might have to though
Thanks,
sb
|
|
|
|
|
I just stumbled on to this:
http://www.awprofessional.com/articles/article.asp?p=99706&seqNum=5&rl=1[^]
says:
When a process terminates, all open files are automatically closed by the kernel. Many programs take advantage of this fact and don't explicitly close open files
This would explain why I get the invalid pointer crash on linux.
However, if I open too many files in my loop it chokes. Interestingly on windows strerror(errno) is 24(too many open) but on linux its 0 --- Success.
So looks like its not getting set.
Most puzzling.
Thanks for helping,
sb
How
|
|
|
|
|
bonosa wrote:
This would explain why I get the invalid pointer crash on linux.
I disagree, but if that is indeed the case, it is a very poor design on someone's part. Windows has the same sort of functionality in that it will close file handles and free up memory when an application terminates. However, if that application has already done its own house cleaning, Windows just moves on to something else. It's not going to try and close a handle that is not open or free memory that is not still allocated.
"One must learn from the bite of the fire to leave it alone." - Native American Proverb
|
|
|
|
|
Hi all,
I have three questions about changing an edit box BG color:
I am trying to change the background color of a multi line edit box, I use the WM_CTLCOLOR message and it works but:
First:
initially, when the color changes, the area that changes is only the first line of the edit box (the first line has the desired background color, while the other lines have the white(default) background color), and as I add lines (Enter Key) the following line changes color and so on.
how to solve this situation? I want the whole control to be painted with the desired bg color.
Second, Even when I change the color of a single line edit box, the color changes, but there still small rectangles of the default bg color in the bottom and the right side of the edit box, it really doesnt exactly have the same effect as writing in VB:
Text1.BGColor=vbBlack
Why?!!!
Thrid and last,
When I use the WM_CTLCOLOR message to change my edit box bg color, if the Locked state of the edit box is set, the color is not changed at all, I dont think its wrong to have a locked edit box with a black bg color.
how to solve this?
I hope I will find some help here.
Regards,
Mohammad Gdeisat
|
|
|
|
|
You should be using WM_CTLCOLOREDIT. Also you should return the handle of the brush that Windows should use to paint the background of the control. If you just use SetBkColor and don't return a brush handle then I seem to recall that you get the results you describe.
Hope that's of some help.
|
|
|
|