|
Does anyone know how to use ifstream to read binary files over 4 GB in size?
Environment: Microsoft Visual C++ 6.0.
We have a simulator that can generate huge amounts of time-series data. We store this data in binary files. We then read that data as needed for charts, map views, etc. To read a particular item, we use the known layout of the binary files to compute the offset from the beginning of the file:
m_iFstream.open(strFName, ios::in | ios::binary);
int totalOffset = timeStepOffset + ObjectOffset + attributeOffset;
char* buf = NULL;
buf = new char[nLength+1];
m_iFstream.seekg(totalOffset, ios::beg);
m_iFstream.read((char*)buf,(sizeof(char)) * nLength);
buf[nLength] = '\0';
We then copy the read buffer into float.
This works great until the offset exceeds the MAX_INT value. To overcome that, I tried a series of relative offsets as needed, using 64-bit integers to provide a little "head room":
m_iFstream.open(strFName, ios::in | ios::binary);
__int64 llTotalOffset = llTimeStepOffset + objectOffset + attributeOffset;
char* buf = NULL;
buf = new char[nLength+1];
int nRemainingOffset;
if (llTotalOffset < INT_MAX)
{
// Single, direct seek is possible.
//m_iFstream.seekg(0);//seek to begining of file
nRemainingOffset = llTotalOffset;
m_iFstream.seekg(nRemainingOffset,ios::beg);//seek from the beginning of the file
}
else
{
// The offset exceeds what an int can represent.
// Must iteratively seek to the location.
int nMaxSeek = INT_MAX;
m_iFstream.seekg(nMaxSeek, ios::beg); // first seek is from the beginning
llTotalOffset -= nMaxSeek;
while(llTotalOffset >= nMaxSeek)
{
m_iFstream.seekg(nMaxSeek, ios::cur);
llTotalOffset -= nMaxSeek;
}
nRemainingOffset = llTotalOffset;
if (nRemainingOffset > 0)
m_iFstream.seekg(nRemainingOffset, ios::cur);
}
m_iFstream.read((char*)buf,(sizeof(char)) * nLength);
buf[nLength] = '\0';
In theory, this should work. In practice, it doesn’t work consistently. I think the failure must be due to the ifstream class not itself being able to store current file offset (ie same issue we had with variable ‘totalOffset’).
Does anyone know how I might be able to use ifstream in this context?
Note that I did look into declaring a specialized char_traits for the stream. I’m no expert with STL file I/O, but it appears to me that the ‘off_type’ and ‘pos_type’ have to correspond to the ‘char_type’ – so I’m hesitant to try this since we are using the file with standard ‘char’ character type.
Thanks in advance for your advice
Ron Hinthorn
|
|
|
|
|
Hi,
I need to load a large text file into my MFC app and later save changes to disk. What is the best way to do this?
Many thanks
|
|
|
|
|
How about CFile and its Read() and Write() methods?
"The pointy end goes in the other man." - Antonio Banderas (Zorro, 1998)
|
|
|
|
|
Will all of the magical and fantastic container classes MFC seems to have brought, I thought there might be a way to read it into some kind of buffer class or something?
Cheers
|
|
|
|
|
ummmmmmm not really
a file has to be read from the disk and put into memory
whether u do it in lil bits or one big a**ed lump
its all the same
"there is no spoon" biz stuff about me
|
|
|
|
|
|
Jamie Kenyon wrote:
I thought there might be a way to read it into some kind of buffer class or something?
The first parameter to CFile::Read() is the buffer.
"The pointy end goes in the other man." - Antonio Banderas (Zorro, 1998)
|
|
|
|
|
Hi.
Is is possible to set up another location for the suo and ncb file? I have a temp-directory in my project directory where all the obj files are put. Before I backup I delete the tmp directory. I want that the suo and ncb files are also put in the temp directory.
|
|
|
|
|
Or are there tools that only backup the files that are *really* required?
|
|
|
|
|
I have a batch file that runs nightly in all of my development folders. It looks like:
del /s /q *.pch
del /s /q *.sbr
del /s /q *.aps
del /s /q *.pdb
del /s /q *.ncb
del /s /q *.opt
del /s /q *.bsc
del /s /q *.ilk
del /s /q *.clw
This keeps the larger files from taking up space on the backup tape unnecessarily.
"The pointy end goes in the other man." - Antonio Banderas (Zorro, 1998)
|
|
|
|
|
Oh yeah. That rules!
Many thanks David
|
|
|
|
|
Hi, I'm new in Visual C++, and I have a problem with a thread and the diferents values of the variables.
My thread is constantly (while(1){.....}) working with a variables and y would like to print their values on the main windows, but I can't use UpdateData(FALSE), because me thread is a global rutine and UpdateData is undefined.
So my question is someone could help my amd explin my how can I refresh the main windows with the news values.
I don't know if I have been enough cleared, but I hope you could understand my problem.
Thanks
|
|
|
|
|
bryanbryan wrote:
My thread is constantly (while(1){.....}) working with a variables and y would like to print their values on the main windows, but I can't use UpdateData(FALSE)...
There's hardly, if ever, a need to use UpdateData() . In your worker thread, simply post a message back to the primary thread (the one that owns the UI) with the values that have been updated.
To get a better understanding of what threads can and cannot do, I suggest:
http://flounder.com/workerthreads.htm
http://flounder.com/uithreads.htm
"The pointy end goes in the other man." - Antonio Banderas (Zorro, 1998)
|
|
|
|
|
Hi,
I would like to use the Panel component which is on the "Windows Forms" tab in the toolbox of Visual C++.NET
Unfortunately, the icon which representes this component looks like disabled.
How can i do to make it enabled ?
thanks.
Maileen
|
|
|
|
|
Im creating threads and I want to pass a pointer to the threads, Is is possible to cast a pointer to a class into a DWORD, if so, how is it done???
Here is the code Im working with, its taken straight from the MSDN
void main(void){<br />
<br />
DWORD loadPerThrdID;<br />
HANDLE loadPerThrd;<br />
TCHAR szMsg[80];<br />
<br />
DWORD dwThrdParam; <br />
<br />
static WmiWrapper Wmi(NULL, NULL, NULL);
<br />
loadPerThrd = CreateThread(<br />
NULL, 0, <br />
loadPerThrdFunc,<br />
&dwThrdParam,<br />
0, &loadPerThrdID);<br />
<br />
if(loadPerThrd == NULL){<br />
wsprintf(szMsg, (TEXT("Create Thread failed for LoadPercentage")));<br />
MessageBox(NULL, szMsg, NULL, MB_OK);<br />
}<br />
else{<br />
_getch();<br />
CloseHandle(loadPerThrd);<br />
}<br />
}<br />
<br />
DWORD WINAPI loadPerThrdFunc (WmiWrapper Wmi)<br />
{<br />
................<br />
}
What can I do to get this to work, or am I completly out??????
|
|
|
|
|
roadragedave wrote:
Is is possible to cast a pointer to a class into a DWORD, if so, how is it done???
Both a pointer and a DWORD are 32-bit values.
DWORD WINAPI loadPerThrdFunc( LPVOID param )
{
DWORD *p = (DWORD *) param;
TCHAR s[32];
wsprintf(s, "The thread value is %lu\n", *p)
MessageBox(s);
}
"The pointy end goes in the other man." - Antonio Banderas (Zorro, 1998)
|
|
|
|
|
Im still lost,
You said that pointers and DWORD are 32 bits, but when I build the app, the following line warnss me that there is a possible mismatch
DWORD dwThrdParam = UINT_PTR(&Wmi);
gives me:
warning C4244: 'initializing' : conversion from '__w64 unsigned int' to 'DWORD', possible loss of data
but otherwise it passes fine, but how do I turn the parameter LPVOID param back into the object or at least a pointer to the object????
We have a mathematician, a different kind of mathematician, and a statistician!
|
|
|
|
|
roadragedave wrote:
You said that pointers and DWORD are 32 bits...
Most of the time they are. You are apparently doing some 64-bit work.
DWORD dwThrdParam = (DWORD) UINT_PTR(&Wmi); // potential loss of data
or
__w64 dwThrdParam = UINT_PTR(&Wmi);
"The pointy end goes in the other man." - Antonio Banderas (Zorro, 1998)
|
|
|
|
|
Excellent stuff, that got rid of the warning, but again how, do I change it back inside the thread function
We have a mathematician, a different kind of mathematician, and a statistician!
|
|
|
|
|
Was the example I provided earlier incorrect? If so, how is CreateThread() currently being called, and what does your thread function currently look like?
"The pointy end goes in the other man." - Antonio Banderas (Zorro, 1998)
|
|
|
|
|
But how, by using this:
DWORD WINAPI loadPerThrdFunc( LPVOID param )<br />
{ DWORD *p = (DWORD *) param; <br />
TCHAR s[32]; <br />
wsprintf(s, "The thread value is %lu\n", *p) <br />
MessageBox(s);<br />
}
can I get to call the methods of the WmiWrapper class Wmi
Im calling the CreateThread method like so:
DWORD loadPerThrd = CreateThread(<br />
NULL, 0, <br />
loadPerThrdFunc,<br />
&dwThrdParam,<br />
0, &loadPerThrdID);
We have a mathematician, a different kind of mathematician, and a statistician!
|
|
|
|
|
DWORD loadPerThrd = CreateThread(
NULL, 0,
loadPerThrdFunc,
&Wmi,
0, &loadPerThrdID);
DWORD WINAPI loadPerThrdFunc( LPVOID param )
{
WmiWrapper *p = (WmiWrapper *) param;
p->...
}
"The pointy end goes in the other man." - Antonio Banderas (Zorro, 1998)
|
|
|
|
|
Fantastic!!!!
We have a mathematician, a different kind of mathematician, and a statistician!
|
|
|
|
|
CreateThread() has a param just for this purpose: void* lpParameter . Set that to the address of the Wmi object. loadPerThrdFunc takes one parameter, again a void* , which you cast back to a WmiWrapper*
Note that you should be using _beginthreadex() and not CreateThread() if you link with the C runtime (which you probably are)
--Mike--
Personal stuff:: Ericahist | Homepage
Shareware stuff:: 1ClickPicGrabber | RightClick-Encrypt
CP stuff:: CP SearchBar v2.0.2 | C++ Forum FAQ
----
Windows troubleshooting: Reboot first, ask questions later.
|
|
|
|
|
Did some modification in your existing code. This would help you.
For the passing parameter to thread, first cast it to DWORD and receive it in callback handler of thread,cast it back to its original type,now use it as you want.
void main(void){
DWORD loadPerThrdID;
HANDLE loadPerThrd;
TCHAR szMsg[80];
DWORD dwThrdParam;
static WmiWrapper Wmi(NULL, NULL, NULL); //Creating object
loadPerThrd = CreateThread(
NULL, 0,
loadPerThrdFunc,
//&dwThrdParam,
(DWORD) &Wmi, //Pass the object pointer as a thread parameter.
0, &loadPerThrdID);
if(loadPerThrd == NULL){
wsprintf(szMsg, (TEXT("Create Thread failed for LoadPercentage")));
MessageBox(NULL, szMsg, NULL, MB_OK);
}
else{
_getch();
CloseHandle(loadPerThrd);
}
}
//Threads function
DWORD WINAPI loadPerThrdFunc (DWORD param1)
{
WmiWrapper* wmi = (WmiWrapper*) param1;
//Now you can use this wmi class pointer.
................
}
Jitendra
|
|
|
|
|