|
I need to improve the way I am reading from a .csv file. The files could get very large > 2MB. When new information needs to be stored, it needs to be placed at the end of the file. The problem is that I need to check if I am placing duplicated information in.
The information that cannot be duplicated is always in the second to last, and last positions of each line. For example:
154,56.0,12,57,409956,10976634
55,408.5,505,246,409957,10976637
708,8894.6,13,8074,8945,409958,10976639
The last two numbers on each line are the numbers I need to get. The line lengths are not going to be the same, so I can't read backward the same amount to get to the next previous line.
There are some tricks I can do to prevent me from reading back through the entire file, but I don't want to flood you with details.
Is there a better way to read backward - line by line? I am currently reading one byte at a time until I see the \n character. Once I'm there, I keep reading until I see 2 comma characters.
I did some testing, and some lines are taking up to 250ms each! This will never work for what I need this for.
Thank you for any help you may be able to provide.
|
|
|
|
|
Dave_ wrote:
Is there a better way to read backward - line by line? I am currently reading one byte at a time until I see the \n character. Once I'm there, I keep reading until I see 2 comma characters.
You need to use buffering, then reading will be block by block, then searching for the last \n will be faster, but it will still be slow. See the tac unix implementation (hey, it's GPL'ed it should be easy to find it) for a sample good performing implementation. If you don't know, cat is the unix command like the "type" on DOS, and tac is the command that do it backwards.
Approach 2: Since (it seems) you are only appending to the file it's easy to keep a (binary) index file with the position of the lines.
I see dumb people
|
|
|
|
|
Without knowing the details of what you are doing, here a few things that you might be able to put together to achieve your goal:
1) Use a memory-mapped file, which lets the OS control the blocking and reading and lets you access the file contents as a large array. See http://www.codeproject.com/win32/cmemmap.asp[^]
1a) You could then try the reverse string/char search (strrchr?) in the array looking for the '\n' end-of-lines.
2) Use strtok to break a given line into tokens, to isolate the last two numbers, using the comma as the field delimeter.
3) When reading from files (without the memory-mapping), you can read an entire line by specifying in the format string to 'read until I find a specified character'. This is done by specifying "%[^'\n']s" which is the "%s" format with the allowable characters in square brackets, but specified "everything until" by using the caret and the single char "newline" in single quotes.
The memory-mapping seems to me to be your best bet.
Dave
"You can say that again." -- Dept. of Redundancy Dept.
|
|
|
|
|
This is probably a very newbieish question:
In a MFC application, if CString sPathname is the full path for a valid file (i.e. sPathname = "C:\My Documents\image.bmp"), what is the simplest way to find out the size of that file?
|
|
|
|
|
|
CreateFile() and GetFileSize() if you use Win32 API or _stat if you're using CRT API.
--
There's a new game we like to play you see. A game with added reality. You treat me like a dog, get me down on my knees.
We call it master and servant.
|
|
|
|
|
CFileFind ff;
DWORD dwSize = 0;
if (ff.FindFile(_T("C:\My Documents\image.bmp")))
{
ff.FindNextFile();
dwSize = ff.GetLength();
}
|
|
|
|
|
Thanks, Bin! That was exactly what I was looking for.
|
|
|
|
|
I´m searching the Internet für Icons and Bitmaps which I can use in the Toolbar (VC++ MFC). Do you now a good link?
|
|
|
|
|
strizi wrote:
I´m searching the Internet für Icons and Bitmaps which I can use in the Toolbar (VC++ MFC). Do you now a good link?
Me Too!!!
- Nitron
"Those that say a task is impossible shouldn't interrupt the ones who are doing it." - Chinese Proverb
|
|
|
|
|
Lokk in the Internet for Glyphs!
|
|
|
|
|
This:
template <typename T>
struct deref {
typedef void type;
};
template <typename T>
struct deref<T*> {
typedef T type;
}; doesn't work in VS.NET. Got any ideas to work around the compiler?
What I want is something like this: typeid(deref<int*>::type) == typeid(int)
--
There's a new game we like to play you see. A game with added reality. You treat me like a dog, get me down on my knees.
We call it master and servant.
|
|
|
|
|
I don't think it's possible. boost/type_traits has such facilities, but the case you're looking for requires partial template specialization (hey, if the Boost guys weren't able to make it in VS.NET chances are it is damn hard). If you can get by with info about whether a given type is a pointer or not, this can be done in VS.NET (if you're interested I can send you some code detached from Boost for easier comsumption).
Joaquín M López Muñoz
Telefónica, Investigación y Desarrollo
|
|
|
|
|
Joaquín M López Muñoz wrote:
hey, if the Boost guys weren't able to make it in VS.NET chances are it is damn hard
True!
Joaquín M López Muñoz wrote:
If you can get by with info about whether a given type is a pointer or not, this can be done in VS.NET
For my particular problem that won't help me much anyway.
My problem is basically this:
template <typename T>
struct deleter {
void operator()(T* pObj) const { delete pObj; }
};
typedef std::vector<AClass*> AClassVector;
AClassVector vec;
vec.push_back(...);
...
std::for_each(vec.begin(), vec.end(), deleter<AClassVector::value_type>()); This doesn't work well since my deleter template works on a different level of abstraction. operator()(T* pObj) becomes operator()(AClass** pObj) which delete won't handle. So, I basically wanted to do this:
std::for_each(vec.begin(), vec.end(), deleter<deref<AClassVector::value_type> >()); I know I can pass AClass as template parameter to deleter<> (which I did later ). But I think deref<AClassVector::value_type> would have been a more elegant solution.
--
There's a new game we like to play you see. A game with added reality. You treat me like a dog, get me down on my knees.
We call it master and servant.
|
|
|
|
|
Can anyone explain me what I have to do with this error?
0xC0000005:Acces Violation
The debugger stops at:
//perform specific initializations
if (!pThread->InitInstance())
{
if(pThread->m_pMainWnd !=NULL)
{ TRACE0("Warning: Destroying non-NULL m_pMainWnd\n");
pThread->m_pMainWnd->DestroyWindow();
}
nReturnCode=pThread->ExitInstance();
goto InitFailure;
}
nReturnCode = pThread ->Run();
The debugger stops at the line; pThread->m_pMainWnd->DestroyWindow();
Thanx,
kind regards,
Geert
|
|
|
|
|
put a breakpoint before the line and see what it is returning..and what u were expecting..so you will know why it is happening.. I guess you are doing it wrong way..so it doesn't know anything such as window to be destroyed..try m_pmainWnd->destroywindow(); i hope this will work..;)
cheers..
Himanshu
|
|
|
|
|
Just a quick stab in the dark, but the fact that your
UI Thread failed to initialised :
pThread->InitInstance() // returns false
.. would indicate that the handle to the thread's main window,
pThread->m_pMainWnd, may not have been initialised.
This would explain the memory violation when attempting to do
pThread->m_pMainWnd->DestroyWindow(); .
The fact that m_pMainWnd is not NULL is not an indication that it is a valid handle.
..
...
<shockhorror>
and my golly god, did I see a goto statement at the end of your code?
|
|
|
|
|
Or you have not waited until it HAS initialised
If I have seen further it is by standing on the shoulders of Giants. - Isaac Newton 1676
|
|
|
|
|
damn, thats a bold 'goto' statement. to hell with efficiency, long live 'goto' lol! !
todo....
:: insert inpirational text here ::
|
|
|
|
|
Mail having attachment type "octet-stream"? and encoding base64. how to decode this?
thanks in advance
Murali
|
|
|
|
|
An octet is a byte. An "octet-stream" mime type is just an unspecified binary file, which could be a .EXE, an MP3, a ZIP, or anything you may want.
A base64 encoded text can be decoded by a base64 decoder.
Sorry, I know the last explanation sucks, but base64 is a very simple encoding and there are lots of encoders/decoders freely available on the web, and, IIRC, even on CP.
The .NET framework also has a base64 decoder included, so if you are doing MC++ this would be easy too.
I see dumb people
|
|
|
|
|
What is cross posting ?
Christian
No offense, but I don't really want to encourage the creation of another VB developer.
- Larry Antram 22 Oct 2002
C# will attract all comers, where VB is for IT Journalists and managers - Michael
P Butler 05-12-2002
Again, you can screw up a C/C++ program just as easily as a VB program. OK, maybe not
as easily, but it's certainly doable. - Jamie Nordmeyer - 15-Nov-2002
|
|
|
|
|
Christian Graus wrote:
What is cross posting ?
It's the second most common break of netiquette rules on CP. The first is programming questions on The Lounge.
I see dumb people
|
|
|
|
|
Hi,
can I create a button which not get focus if it is clicked? (a focusless button like a button in a toolbar)
The focus should stay with the current control and not be "stolen" by button click. Possibly it can be done by overriding WM_LBUTTONDOWN ...? It would be great if I can avoid ownerdraw (to keep XP style button, if code is running on XP). What I found on Google mentions it has to be a onwerdraw button rebuild from scratch.
At the moment I simply set the focus back after button is clicked, but that's not that great.
Thx for help, Moak
|
|
|
|
|
For example:
CButton* bt = (CButton*)GetDlgItem(IDC_BUTTON1);
void XDlg::OnButton1()
{
bt->EnableWindow(FALSE);//For Focusless
}
void XDlg::OnRbuttonDown()
{
bt->EnableWindow(TRUE);//For back to Focus;
}
Is this helpful to you.
|
|
|
|