C / C++ / MFC

dazinith7-Mar-03 10:10

dazinith

7-Mar-03 10:10

i am using ifstream because it is all i am currently familiar with, i am looking to learn any quicker methods.. is there any way that i can handle the file like notepad is? once i have all of my lines loaded into memory the search is very very quick, the whole problem is loading the whole file into memory (which isnt a requirement, its just how im currently doing it, as im deleting all objects once i find the text anyways)

still a newb.. cut me some slack :P
-dz

Neville Franks7-Mar-03 10:22

Neville Franks

7-Mar-03 10:22

I don't understand why this is taking so long. I'd get rid of the progress bar updates. Also see if you can increase the size of the buffer ifstream uses. Why do you need to copy the string into a CString. Just use strchr() on szBuffer2.

As others have said Memory Mapped Files are great for this, but using streams should also be more than fast enough. There is something or several things fundamentally wrong with your implementation.

Also look at profiling the code, which will show you where the bottlenecks are. Look at Glowcode www.glowcode.com[^]

In ED (see sig) loading a 30K line is instantaneous. As is finding something at the end of such a file.

Neville Franks, Author of ED for Windows. www.getsoft.com
Make money with our new Affilate program

Scott H. Settlemier7-Mar-03 10:14

7-Mar-03 10:14

Map the file into memory and use
good guesses about where a certain
line may be. Then make your way from
there to where you need to read.
That way you should only have to
read a small part of the file.
(hopefully)

dazinith7-Mar-03 10:23

dazinith

7-Mar-03 10:23

can you give me some keywords on where to start to map the file into memory? i don't know how to do that. im searching cp for 'memory file', and other things, but im not having any success.. is there a keyword associated with mapping a file to memorY?

thanks!

still a newb.. cut me some slack :P
-dz

Scott H. Settlemier7-Mar-03 11:21

7-Mar-03 11:21

Bare bones:

HANDLE hFile=::CreateFile(Filename,GENERIC_READ,FILE_SHARE_READ,0,OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL,NULL);
HANDLE hMap=::CreateFileMapping(hFile,0,PAGE_READONLY,0,0,0);
const char* pFile=(char*)::MapViewOfFile(hMap,FILE_MAP_READ,0,0,0);

; // work on file

::UnmapViewOfFile(pFile);
::CloseHandle(hMap);
::CloseHandle(hFile);

Tim Smith7-Mar-03 10:25

Tim Smith

7-Mar-03 10:25

First, don't use strTemp. It is a total waste of time. All of those left operations are creating temp objects that just aren't needed.

if (szBuffer2 [0] != '.')
{
    CNumberAndName *pEntry = new CNumberAndName ();
    szBuffer [9] = 0;
    pEntry ->m_strRoutingNumber = szBuffer; 
    szBuffer [11+36] = 0;
    pEntry ->m_strBankName = &szBuffer [11];
    paBankArray ->Add (pEntry);
}

Tim Smith

I'm going to patent thought. I have yet to see any prior art.

dazinith7-Mar-03 11:00

dazinith

7-Mar-03 11:00

thanks a bunch.. seems that i need to learn how char arrays work, i always use cstrings because they are easy for me to use.. thanks for the code.. now all i have to do is figure out what you are doing with the char arrays Smile | :)

thanks again for everyones help!

still a newb.. cut me some slack :P
-dz

Tim Smith7-Mar-03 11:44

Tim Smith

7-Mar-03 11:44

The "szBuffer [0] == '.'" is just checking to see if the first character is a '.'.

The szBuffer [9] = 0; is splitting the string into two parts. The first part starts at szBuffer [0] and is 9 characters long. The second part starts at szBuffer [10] and contains the rest of the string. This allows you to copy the first part of the string into your destination.

The szBuffer [11+36] = 0; is doing the same thing. But now we have three strings.

What this does is remove the need to allocate a new buffer to hold the string, copy that string into the new buffer, and then extra elements from the string piece by piece. By directly operating on the original string, we avoid a fair amount of overhead.

Tim Smith

I'm going to patent thought. I have yet to see any prior art.

DShow-implementing CBaseControlWindow

Ted Ferenc8-Mar-03 2:02

Ted Ferenc

8-Mar-03 2:02

Have a look for grep.c on the net, it is a UNIX tool, did a quick test on one of my files using it, 545,000 lines searching for a non existing string took 8 seconds on my 450mhz dual celeron. Oh I forgot to mention the file is c. 100MB

If I have seen further it is by standing on the shoulders of Giants. - Isaac Newton 1676

Jakub Misek7-Mar-03 9:36

Jakub Misek

7-Mar-03 9:36

do you know how to implement CBaseControlWindow interface in IGraphBuilder ? plz, thnx u

Compliers and File IO buffer efficiency?

phyzics7-Mar-03 9:12

phyzics

7-Mar-03 9:12

I was experimenting with file IO times and discovered some peculiar results that I do not understand and thought mabye sombody could explain it to me.

I was testing the hypothesis that in file IO processing it is better to use a bigger buffer than a smaller one (I was thinking that the fastest way to perform large file IO was to read a disk sector at a time). To test this I performed the following test.

Test:
Copy one file on one drive to another file on another drive. To perform the copy I used VisualC++ 6.0 on a box running Win2k. I wrote a simple console app that just used fread() and fwrite() to and from the files on the different drives.

Results:
The results I got were shocking to me. I ran the code in debug mode first and discovered that on my hardware and a 120Mb file copy I obtainted the following times (in seconds) for each read/write buffer size:

1 byte = 128(s)
2 bytes = 112(s)
4 bytes = 104(s)
8 bytes = 100(s)
16 bytes = 99(s)
...
512K bytes = 99(s)

So there appeared to be a threashold at around 8 bytes above which buffer size makes no difference.

When I re-ran the code in Release mode with the optimizer set to maximize speed the results were much different:

1 byte = 120(s)
2 bytes = 120(s)
...
512K bytes = 120(s)

The "plateau" region went away and the curve flattened out. However the large buffer read times went up! This makes no sense to me!

Does anybody out there know what is going on here? I would appreciate any enlightenment.

Thanks!

Re: Compliers and File IO buffer efficiency?

Tim Smith7-Mar-03 9:22

Tim Smith

7-Mar-03 9:22

First problem is that you are using fread/fwrite that include their own buffering. To better test raw performance, use ReadFile and WriteFile.

Tim Smith

I'm going to patent thought. I have yet to see any prior art.

Help With Threads...PLEASE!

dantan987-Mar-03 7:57

dantan98

7-Mar-03 7:57

I am having major thread problems in my application.

I am trying to set up our program so the threads don't run into each other (access the same file at the same time) and so they don't wait on each other forever.

I have been working on this for 2 weeks now and am nearly suicidal because it isn't working. I need a solution to replace what I have now. Please read everything below and respond if you are able to help out.

This is in a Doc/View structure. Basically, I need to know where to create the threads, what I should use for syncrhonization, where to wait, and where to set the threads/events/etc. as signalled. Thank you so much for your help!!

--Dan

/\/\/\

Right now, we have a thread for Action-S, for Action-A, and for Action-C.

Action-S accesses a file in memory and then updates a physical file.

Action-A accesses the same file in memory and updates a local file.

Action-C accesses the same physical file as Action-S and then locks down the main thread until a process is finished (the lockdown works fine - I don't need to worry about that).

All Action-S's and Action-A's only care about the actions for a particular file name. If the file "xyz" is open, Action-A for "xyz" does not care about what is happening for file abc (such as an Action-A for "abc").

For one file only, please refer to this diagram...

L P
\ / \
A\ S/ C\
\ / \
M W

"A" - Action-A (performs M then L)
"S" - Action-S (performs M then P)
"C" - Action-C (performs P then W)
"L" - Local File
"M" - File in Memory
"P" - Physical File
"W" - Process to Wait On.

"\" - a Thread
"/" - a Thread

dantan987-Mar-03 7:59

dantan98

7-Mar-03 7:59

The diagram again...

L       P
 \     / \
 A\  S/  C\
   \ /     \
    M       W

AlexO7-Mar-03 8:16

AlexO

7-Mar-03 8:16

dantan987-Mar-03 8:34

dantan98

7-Mar-03 8:34

Action-S and Action-A only wait for each other when accessing the file in memory. Then (if there are only these two threads), they can go about their business independently.

AlexO7-Mar-03 9:28

AlexO

7-Mar-03 9:28

then use something like

HANDLE hEvent = CreateEvent(0, TRUE, TRUE, "abc");//note that we use filename to name the event

//just in case
if(WaitForSingleObject(hEvent, INFINITE) == WAIT_OBJECT_0 && ResetEvent(hEvent))//lock event on your behave
{
//... do your staff

SetEvent(hEvent);//let somebody else to continue
}

Scott H. Settlemier7-Mar-03 9:59

7-Mar-03 9:59

If you're just going to lock access,
you really want a mutex there though.

HANDLE hMutex=CreateMutex(0,FALSE,Filename);
WaitForSingleObject(hMutex,INFINITE);
// Do stuff
ReleaseMutex(hMutex);

Neville Franks7-Mar-03 10:26

Neville Franks

7-Mar-03 10:26

Unless you are using multiple processes you want to use a critical section, not a mutex. cs are much faster.

Neville Franks, Author of ED for Windows. www.getsoft.com
Make money with our new Affilate program

Scott H. Settlemier7-Mar-03 11:27

7-Mar-03 11:27

Yeah, completely correct. As further overkill, the example
used a named object. I don't know if it really matters as
much as I am reluctant to create named objects when an
unnamed one will do, but I always prefer to limit my
exposure.

(oh yeah, which I did not do in that ex. Blush | :O

)