|
Hi, I am trying to write/read a large vector of vector of doubles to comma delimited text file using std::ofstream. The problem is that it doesn't seem to be very fast, taking a few minutes at a time.
The matrix to be written/read is of the size N x M, where N is approx. 2,000 and M is between 1,000 and 40,000. That means a file size of up to approx. 800Mb for M == 40,000.
Should I consider serialization as something that might be faster?
At the moment I am not doing anything complicated, below is my code for writing the 'spreads' matrix to file (removed conditional error checks, etc):
ofstream write_file;
write_file.open(fileName.c_str(), ios_base::out | ios_base::trunc );
int numRows = spreads.size();
int numCols = spreads[0].size();
for (int row = 0; row < numRows; row++)
{
for (int col = 0; col < numCols; col++)
{
write_file << "," << spreads[row][col];
}
write_file << endl;
}
write_file.close();
When reading it in I do much the same thing, reading in the file line by line and using a split string algorithm to extract each row of doubles, building up the matrix of doubles.
The CPU is at 100% at all times when this is run, so it's not slow file I/O that's the problem.
Any suggestions would be appreciated.
Thanks,
Christopher
The bomb lives only as it is falling
|
|
|
|
|
One solution is the STL's copy() algorithm and ostream_iterator.
Kuphryn
|
|
|
|
|
copy() works, but i still need to do it row by row. it turns out to be only marginally faster for 1900 rows and 10000 columns. It takes 91 seconds to write this matrix to file using copy() and 95 seconds using a raw loop.
was considering writing the raw byte in memory representation of the matrix to file using the Win32 API WriteFile(). i don't know how wise that would be, or how to do it properly though.
christopher
The bomb lives only as it is falling
|
|
|
|
|
I had a similar problem, when I read a large vector from ifstream via STL.
It's a sad disadvantage of STL - it's too generic to be efficient. For example the stl::copy of vectors is implemented as a cycle, which applies copy constructor to each member of the vector. If you have a vector of complex objects with nontrivial copy constructors, this approach is fair. But if you have for a vector of int 's or char 's, the STL implementation is too much slower than a simple memcpy .
I think, that std::vector is internally stored as a continuous array of objects, so the iterators to the first and last members should be pointers to begin and end of a memory block, where the array is stored. Thus using of CFile::Write is simple.
Reading a vector is more complex, I think, that you should write your own allocator, which builds a vector from a memory block. But it's only my opinion, I've never write any allocator
The general solution would be to write a STL extension with containers optimized for special member types. Do anybody know, whether such a library exists?
Robert-Antonio
Love, truth and electric traction must gain victory over hate, lie
and diesel traction.
|
|
|
|
|
I have some code that retrieves simple URL's as follows,
DWORD nStatus = NULL;
CHttpFile* pFile = NULL;
CHttpConnection* pConn = NULL;
CInternetSession ISess;
pConn = ISess.GetHttpConnection(sz_Server);
pFile = pConn->OpenRequest(CHttpConnection::HTTP_VERB_GET, sz_ServiceURL);
pFile->SendRequest();
pFile->QueryInfoStatusCode(nStatus);
if ( nStatus == HTTP_STATUS_OK )
{
CString strBuff;
CString strRead;
while ( pFile->ReadString(strRead) )
{
strBuff += strRead;
}
cout << (LPCTSTR)strBuff << endl;
}
I would like to remove the looping reading and replace it with a single read. But I'm not able to figure out how big a buffer to allocate. Is there some method of the CHttpFile class that I could use that would give me the amount of data to expect. I've tried the GetLength method, but this is not returning the correct value. Thanks.
Chris Meech
We're more like a hobbiest in a Home Depot drooling at all the shiny power tools, rather than a craftsman that makes the chair to an exacting level of comfort by measuring the customer's butt. Marc Clifton
VB is like a toolbox, in the hands of a craftsman, you can end up with some amazing stuff, but without the skills to use it right you end up with Homer Simpson's attempt at building a barbeque or his attempt at a Spice rack. Michael P. Butler
|
|
|
|
|
I have an application developed on Win2K. The problem that I am experiencing with my application has to do with the non-client area(title bar). My application changes the color of the background of the title bar. So, I have to redraw the min/max/close buttons. On Win2K, everything seems to work ok, but I have noticed that when I run my application on Windows XP(with theme enabled), my min/max/close buttons are off. The buttons are bigger, but the hit tests are off(obviously). How do I detect themes enabled for an application developed on Win2k(without uxtheme.dll/.lib/.h etc)?
The CaptionPainter class that I am using is from the following link:
www.codeproject.com/dialog/gradienttitlebar.asp
|
|
|
|
|
|
Mike,
Thanks for the reply, but I am constrained to Win2k development. SetWindowTheme doesn't come with Studio 6.0 libs. Maybe platform SDKs?
|
|
|
|
|
It doesn't matter what OS you use for development, you can call any API you want. BUt yes, you'll need the Platform SDK to get the lastest header files.
--Mike--
Personal stuff:: Ericahist | Homepage
Shareware stuff:: 1ClickPicGrabber | RightClick-Encrypt
CP stuff:: CP SearchBar v2.0.2 | C++ Forum FAQ
----
Pinky, are you pondering what I'm pondering?
I think so Brain, but if we shaved our heads, we'd look like weasels!
|
|
|
|
|
Mike,
Hey thanks for the reply - I have since brought in the header file for UxTheme.h, but am getting a link error now. I have tried bringing in UxTheme.dll and linking against that, but it has a dependency on the newest Ntdll.dll. The Ntdll.dll has other dependencies as well. So, what else do I need to bring into my Win2K Studio environment to get this to compile?
Thanks,
Bob
|
|
|
|
|
|
Mike,
I am developing on Win2k and have the Platform SDKs installed. I have included c:\Program Files\Microsoft SDK\include and c:\Program Files\Microsoft SDK\lib (to get uxtheme.lib). However, when the compile finishes, I get an error dialog that says "Uxtheme.dll" cannot be found. Doesn't this mean I have to have Uxtheme.dll installed on Win2K? BTW, my application works nice on XP, but I have to make it work on Win2k and NT as well.
Thanks for all of your help!
Bob
|
|
|
|
|
uxtheme.dll isn't present on 2K (or earlier). You can't link with uxtheme.lib since, as you've seen, that makes the app require that DLL to load.
You have two choices:
1. use the /delayload:uxtheme.dll linker switch
2. use LoadLibrary/GetProcAddress at runtime
Either way, you must add OS checks to your code so that you don't call the functions in uxtheme unless the app is running on XP or later.
--Mike--
Personal stuff:: Ericahist | Homepage
Shareware stuff:: 1ClickPicGrabber | RightClick-Encrypt
CP stuff:: CP SearchBar v2.0.2 | C++ Forum FAQ
----
If my rhyme was a drug, I'd sell it by the gram.
|
|
|
|
|
I've used C++ 6's built-in macro to print all open files, only to end up having it try to print out the whole mfc. I wanted to know is there any freeware C++ source code printing programs out there that also format the code in a nice-looking way.
thanks
bdiamond
If it's broken, I probably did it
|
|
|
|
|
bdiamond wrote:
only to end up having it try to print out the whole mfc
How did you do that? That has never happened to me. It should only print the files that you have currently open in devstudio...
John
|
|
|
|
|
I know that it SHOULD do that, but it only seems to do it right for me once out of ten times. any other time I seem to have to deal with a few hundred pages of MFC source code. I've been using textpad lately, but I have to go and open them all up from TextPad, then print them.
If it's broken, I probably did it
|
|
|
|
|
Are you sure that these MFC files were not open in developer studio when you hit print? The macro that prints is a very simple macro that gets a list of the open documents and prints each one.
Here is the code for the macro:
Sub PrintAllOpenDocuments ()
'DESCRIPTION: Prints all open, active documents.
'Small, quick macro, but it can be usefull.
for each doc in Application.Documents
Doc.PrintOut
next
End Sub
Unless devstudio somehow thinks more files are open than you actually have open I do not see how this can fail...
John
|
|
|
|
|
I also thought about that and that may be the case (developer studio thinking I have more files open that I actually do). But since then I've gotten a new machine and reloaded it. I haven't tried that way in so long now because of the problems I was experiencing before, but I still wanna find a way with better formatting capabilities if possible.
If it's broken, I probably did it
|
|
|
|
|
I have used source code printer applications in the past but since my projects are now way too big to print I just print a few pages at a time using developer studio. I have one link to a couple of printer programs but I do not remember the one that I used in the past. I may still have a copy on my hd though... Here is the one link: http://www.softpanorama.org/Tools/beautifiers.shtml#C++[^]
John
|
|
|
|
|
|
If I create a multi threaded application how do I pass data from my main thread to my spawned thread.
Thanks
Tom Wright
tawright915@yahoo.com
|
|
|
|
|
Messaging will be the best bet i guess, create a user defined message and use postthreadmessage api to post the message and appropriate data along with it.
|
|
|
|
|
Quick question. Is there a limit to the amount of data that can be passed? Or could I pass a char pointer to the spawned thread that have it look there for the data?
Thanks
PS I know this is a totally different subject but if I created a variable like CString holdBuf is this a char pointer or is char holdbuf[] a char pointer. I'm not sure how I would say that I want to pass a pointer to CString holdBuf.
Thanks again
Tom Wright
tawright915@yahoo.com
|
|
|
|
|
Passing strings from threads to the main app can get tricky esp if your thread exits before all the data can be passed etc. A few months back I found a really good article here on codeproject on the subject.. I believe the classes are called CMessageString and CMessageStringManager. You should look into using the classes or at least read the article it will give you some great info. (sorry I dont have the direct link to the article)
Rob
Whoever said nothing's impossible never tried slamming a revolving door!
|
|
|
|
|
Tom Wright wrote:
PS I know this is a totally different subject but if I created a variable like CString holdBuf is this a char pointer or is char holdbuf[] a char pointer. I'm not sure how I would say that I want to pass a pointer to CString holdBuf .
CString has a operator(LPCTSTR) that is able to transparently cast the CString into a Long-Pointer-to-Constant-T_STRing, which is what LPCTSTR means. In non-unicode environments this is indeed a const char* .
When you need a non-const char*, you have to use CStrings GetBuffer()-function (and don't forget to call ReleaseBuffer() after use!), which gives you a char*.
"We trained hard, but it seemed that every time we were beginning to form up into teams we would be reorganised. I was to learn later in life that we tend to meet any new situation by reorganising: and a wonderful method it can be for creating the illusion of progress, while producing confusion, inefficiency and demoralisation."
-- Caius Petronius, Roman Consul, 66 A.D.
|
|
|
|