Flat file parsing using .NET C# ?

Or perhaps:

List<string> uniqueElements = (from w in System.IO.File.ReadAllLines("File.txt") 
                          select w).Distinct().ToList(); 
</string>

Regards
Espen Harlinn

I would suggest to parse the file line by line in a separate thread so that the your application still responsive and you can cancel the operation at any time you want. Also its not good idea to load the whole file in memory since 1 million line could eat some memory based on how much data the file contains.

you may try

TextReader or StreamReader ReadLine method.

varieties of read methods also there for your choice.
Read more TextReader.ReadLine in MSDN[^]

FileStream fileStream = new FileStream(unit.FileName, FileMode.Open, FileAccess.Read, FileShare.Read, 256,FileOptions.SequentialScan);
StreamReader streamReader = new StreamReader(fileStream);

while(true)
{
 string line = streamReader.ReadLine(); // reads a line and moves the file pointer to next line
 if(line == null)
 { 
   break; you reached the end of line.
 } 
 //parse the line
}
streamReader .Close()
fileStream .Close();

"FileOptions.SequentialScan" will speedup your file reading if you are reading from the top to bottom