|
Hello,
Thanks for the great parser and for keeping it up-to-date!
I have a small 80GB file the I'm working on but the header record is wrapped in quotes which generates an error. Here is the header record and first record:
"Household_Member1","Address","City_State_Zip","Area_Code_and_Phone","Phone_Area_Code","Phone_Number_without_Area_Code","City","State","ZIP","ZIP_+_4","ZIP_+_4_+_DPBC","Delivery_Point_Bar_Code","Carrier_Route","County","MSA","Household_Member_First1","Household_Member_Last1","Household_Member_Gender1","Household_Member_Age1","Household_Member2","Household_Member_First2","Household_Member_Last2","Household_Member_Gender2","Household_Member_Age2","Household_Member3","Household_Member_First3","Household_Member_Last3","Household_Member_Gender3","Household_Member_Age3","Household_Member4","Household_Member_First4","Household_Member_Last4","Household_Member_Gender4","Household_Member_Age4","Household_Member5","Household_Member_First5","Household_Member_Last5","Household_Member_Gender5","Household_Member_Age5","Household_Member6","Household_Member_First6","Household_Member_Last6","Household_Member_Gender6","Household_Member_Age6","Household_Member7","Household_Member_First7","Household_Member_Last7","Household_Member_Gender7","Household_Member_Age7","Household_Member8","Household_Member_First8","Household_Member_Last8","Household_Member_Gender8","Household_Member_Age8","Household_Member9","Household_Member_First9","Household_Member_Last9","Household_Member_Gender9","Household_Member_Age9","Household_Member10","Household_Member_First10","Household_Member_Last10","Household_Member_Gender10","Household_Member_Age10","Residence_Type","Home_Age","Est_Home_Value","Est_Income","Own/Rent","Length_of_Residence","Mortgage_Age","Mortgage_Count","Mortgage_Finance_Type","Mortgage_Loan_to_Value_Ratio","Mortgage_Loan_Type","Personal_Details1","Personal_Details2","Personal_Details3","Personal_Details4","Personal_Details5","Mail_Order_Buyer1","Mail_Order_Buyer2","Mail_Order_Buyer3","Mail_Order_Buyer4","Mail_Order_Buyer5","Mail_Order_Buyer6","Mail_Order_Buyer7","Mail_Order_Buyer8","Mail_Order_Buyer9","Mail_Order_Buyer10","Mail_Order_Buyer11","Mail_Order_Buyer12","Mail_Order_Buyer13","Mail_Order_Buyer14","Mail_Order_Buyer15","Mail_Order_Buyer16","Mail_Order_Buyer17","Mail_Order_Buyer18","Mail_Order_Buyer19","Mail_Order_Buyer20","Mail_Order_Buyer21","Mail_Order_Buyer22","Mail_Order_Buyer23","Mail_Order_Buyer24","Donor_Type1","Donor_Type2","Donor_Type3","Donor_Type4","Donor_Type5","timezone","latitude","longitude"
"Rose M Modarelli",""2822 Bears Den CT","Youngstown, OH 44511-1214","","","","Youngstown","OH","44511","44511-1214","44511-1214-223","223","C090","Mahoning","Youngstown-Warren, OH","Rose","Modarelli","F","65 and older","Dina M Modarelli","Dina","Modarelli","F","25 to 34","Dominic G Modarelli","Dominic","Modarelli","M","25 to 34","Frank J Modarelli","Frank","Modarelli","M","45 to 54","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","Single family dwelling","","","","Home Owner","26 to 30 years","26 to 30 years","Unknown","Unknown","0.01 to 0.09","Unknown","Pool","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","EST",80.696899999999999,41.069099999999999
Works fine if I take the quotes off the header record. I get a lot of these files in so if I can make the parser handle it that would be better than appending a new header to these file since they are so large.
I know you are a busy guy so if you can point me in the general direction I'll try to make the change myself. Thanks again for your time.
Chad
|
|
|
|
|
Thanks for your comment
First of all, knowing what is the exact error would help me ... That said, I can see there is an extra " in ""2822 Bears Den CT" which may cause your error. Hope this help!
|
|
|
|
|
Sebastien,
Thanks for pointing that out. That's what I get for programming so late. Things blend together at that hour! Work fine. Thanks again!
Off to parser 124 million records (90 fields). Your product is solid, I've use this before on 200 million+ with way more fields. No problems.
Great piece of code! Keep up the good work!
Chad
|
|
|
|
|
That reader was meant for multi-gigabytes files. Glad to hear I am not the only one that is dealing with these monsters :p
|
|
|
|
|
Whew I finally figured it out.. thanks for this nice CSVReader..
|
|
|
|
|
I mean id GridView not GridPanel.. sorry
|
|
|
|
|
Im only getting this items in my gridpanel.
System.String[];
how do i use this CSVReader properly using ASP.Net2.0?
Thankyou!
|
|
|
|
|
Do you have any example code/xaml of binding this in wpf to a control such as ListView or ContentControl?
|
|
|
|
|
Sorry for late reply, but for some reason, I did not get notified via email of your post.
Using the class CachedCsvReader, you can use standard binding mechanisms, be it WinForms or WPF. If you do not know how to bind in WPF, take a look at the doc on the Binding class.
|
|
|
|
|
HI there,
I think this is a very usefull great tool!
But I experienced problems, when the first row of a csv file has less columns than the following rows. In this case it only parses as many values per row as given in the first row.
is there a way of setting the number of columns to be read?
I didn't find any...
Thanks to all
ans most of all to the author!!!
Anja
There are only 10 types of people in the world: Those who understand binary, and those who don't...
|
|
|
|
|
Sorry for late reply, but for some reason, I did not get notified via email of your post.
I am sorry, this is one of the few limitations of this CSV reader. I built it unaware that it is so hard for a multi-billion company to follow a spec of 5 lines.
|
|
|
|
|
Hi I am importing a CSV wich has a field in this way:
"Monitor TFT 19" and 17" size"
when I read that file the csv object return a corrupt data error. I guess that using " (double quote) chars inside a field where the string delimiter is defined as " is not admited. Is there any way to fix this problem by pre-processing the line (changing the " char by the escape equivalent).
Or just another idea... would it be possible to make the method to keep on reading till it finds the next field separator even the string is not well delimited/formed?
Thanks in advance.
Jero.
modified on Wednesday, January 02, 2008 11:51:03 AM
|
|
|
|
|
Sorry for late reply, but for some reason, I did not get notified via email of your post.
You can use " inside a field delimited by ", but you need to escape it, ie use "" or \" (see EscapeCharacter property).
As for supporting badly formatted CSV, I am afraid the support goes as far as notifying you it is badly formatted via an event or exception. You can preprocess your file or keep track of problematic records and process them in some other way later.
|
|
|
|
|
I downloaded the program and wanted to see the demo. Upon attempting a build I get the error "The type or namespace name 'NUnit' could not be found (are you missing a using directive or an assembly reference?)"
I receive the same error in all four files that are part of LumenWorks.Framework.Tests.Unit and within the Csv folder.
Please help me in seeing what I am doing wrong or did I miss a file in the download.
Thanks
|
|
|
|
|
Get the files from http://www.nunit.org
|
|
|
|
|
Something on the server happened and now it is a very old version of my article which is online. Please wait until I post a message saying it is back to normal. Thank you!
|
|
|
|
|
Ok, the article has now been correctly restored by the CP team. Enjoy
|
|
|
|
|
The parser cannot read ñ character.
"We don't see things how they are. We see things as we are." -Talmud
|
|
|
|
|
Please read the comment just below yours ....
|
|
|
|
|
I tried specifying Unicode as encoding but the ReadNextRecord failed.
using (CsvReader csv = new CsvReader(new StreamReader("c:\\test.csv",Encoding.Unicode), true))
{
int fieldCount = csv.FieldCount; <-- returns only 1 instead of 5 (file has 5 columns)
string[] headers = csv.GetFieldHeaders();
while (csv.ReadNextRecord()) <-- returns false on first call
{
}
}
"We don't see things how they are. We see things as we are." -Talmud
|
|
|
|
|
Well, maybe your file is not in unicode format... Have you tried anything else? And please stop reading the Talmud and be nice to people instead. Like say thank you or something.
|
|
|
|
|
Hi, do you support Unicode? Because letter like ÄÖÜ are not recodginzed while reading!
regards
gocez
|
|
|
|
|
You need to specify the encoding to use when creating a StreamReader.
|
|
|
|
|
|
Sebastien,
After I read a file, I need to replace certain values. Is this possible. For example, I have read the file with-
Dim csv As New CachedCsvReader(New StreamReader(Server.MapPath("/path/") & FileUpload1.FileName), True)
Do While csv.ReadNextRecord
If (csv(0) = lbCompanies.SelectedValue) Then
'csv(0).Replace(csv(0), lbCompaniesAll.SelectedValue)
'Over here, I need to replace the value in csv(0) with a value a user selected from a listbox (for example)
End If
Loop
Any ideas? I guess I can load csv into a dataset and handle my operations over there.
|
|
|
|
|