How to optimize fgets file read time - C programming

Question

0.00/5 (No votes)

See more:

Hello,

I am reading a notepad file(contains matrix value datas) using following C programming

statement

fgets( cpReadString, 6144, fStream )

but due to '6144' count, it is taking three

seconds to read. I want to reduce the time to zero seconds, can somebody help me out

to solve this issue.

Thanks in advance.

Posted 5-Dec-12 22:21pm

Ajain A K

Add a Solution

Comments

CPallini 6-Dec-12 4:35am

Why are you using fgets? Cannot you use fread instead?

Ajain A K 7-Dec-12 6:26am

CPallini,

Thanks a lot friend

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Jochen Arndt · Accepted Answer · 2012-12-05T22:52:00

Solution 1

fgets() reads char wise from the file until it reads a new line character, the buffer size is reached, or at the end of the file. But even with a buffer size of 6144 bytes and a line that is so long this should not take three seconds. Maybe the time is consumed elsewhere in your code.

To improbe reading, you may read the complete file content into an allocated buffer and parse this to get your data:

C++

long lFileSize;
char *pBuf;
// Get length of file
lFileSize = _filelength(_fileno(fStream));
// Allocate buffer, add one byte for terminating NULL char
pBuf = (char *)malloc(lFileSize + 1);
// Read data from file into buffer
fread(pBuf, 1, lFileSize, fStream);
// Append terminating NULL char
pBuf[lFileSize] = '\0';
// Parse buffer here
//
// Free allocated memory
free(pBuf);

Posted 5-Dec-12 22:52pm

Jochen Arndt

Comments

Ajain A K 6-Dec-12 6:55am

Dear Jochen Arndt,

Thanks very much for responding me quickly and the answer is very helpful. I am trying to debugg the following code but the break point taking 3 seconds to exit this while loop so I subtracted the 6144 and tested then I could find the time reducing thats why I put the point like that in my question, can you look at this code and consult me a good solution (quickly!!!!!!!)

short ReadData( const char * cpFileName, int nRows, int nColumns, struct MG1G1_ *stpData )
{

short stReturn = 0 ;
short stFileOppend = 0 ;
short int nRow = 0 ;
short int nColumn = 0 ;
float fValue=0;

FILE * fStream = NULL;

while ( NULL != fgets( cpReadString, 6144, fStream ) && nRow < nRows )

{

stpData ->dbpMG1G1_data[nRow] = ( double * ) malloc ( sizeof ( double ) * nColumns ) ;

nColumn = 0 ;

cpSubString = strtok(cpReadString, "," ) ;

while ( NULL != cpSubString && nColumn < 2052)

{

fValue = atof( Trim( cpSubString )) ;

stpData ->dbpMG1G1_data[nRow][nColumn] = fValue ;

++nColumn ;

cpSubString = strtok( NULL, "," ) ;

}

++nRow ;

}

free ( cpReadString ) ;

stReturn = 1 ;

}

while ( 0 ) ;

}

if( NULL != fStream && 1 == stFileOppend )
{

if ( 0 != fclose( fStream ) ) perror( "Cannot close file after read : " ) ;

}

fclose(fp);

return stReturn ;

}

Note:

cpFileName = "c:/Matrixdata.txt" its size is 8260KB

Thanks in advance

Jochen Arndt 6-Dec-12 8:41am

The code looks basically OK. Just three notes:

1. while (NULL != cpSubString && nColumn < 2052)
Should be: nColumn < nColumns

2. fValue = atof(Trim(cpSubString));
Trim() should be not necessary. atof() ignores leading spaces and conversion stops when a non-digit occurs.

3. Your file is quite large. The size of cpReadString must be big enough to read in the longest line. If not, you are adding items from the actual row to the next one. You can check this by comparing cpReadString[strlen(cpReadString)-1] with the line feed character '\n'.

With each line you are allocating an array of double. With many lines, this consumes some time. Using a flat array with malloc(sizeof(double) * nRows * nColumns) may reduce the time but requires calculation of element indexes (value[nRow][nColumn] = flat_array[nRow * nColumns + nColumn]).

Reading the file at once into a large 8 MB buffer and parsing is still an option. For parsing use strchr() with '\n' for end of line and ',' for next value (untested code):
char *pEOL = strchr(cpReadString, '\n');
char *pValue = cpReadString;
do {
nColumn = 0;
while (pValue < pEOL)
{
fValue = atof(pValue);
pValue = strchr(pValue, ',');
if (pValue && pValue < pEOL)
{
++pValue;
++nColumn;
}
}
pValue = pEOL + 1;
pEOL = strchr(pValue, '\n');
if (NULL == pEOL)
pEOL = pValue + strlen(pValue);
++nRow;
} while (pValue);

Ajain A K 7-Dec-12 6:25am

Dear Jochen Arndt,

Thanks a lot for the very useful answer, it is very helpful.

Jochen Arndt 7-Dec-12 6:33am

Thank you for your feedback and accepting the answer.