|
Ok, thanks to some help, I think I have what I need to extract the data I need from a HUGE (900Mb-2Gb) text file.
Now I need to store the huge volume of data so that I can process it. Its only 5 fields, but lots of entries. I will need to group the data based upon various fields so I can bin and plot it. I will also need to do things like MIN/Max/Average of one field for groups of another field.
So the question is what type of data structure do I put it into? Any suggestions?
David Wilkes
|
|
|
|
|
amatbrewer wrote: So the question is what type of data structure do I put it into? Any suggestions?
A database. It is designed specifically to hold and process large volumes of data like this.
|
|
|
|
|
and with several gigs of data, that DB should be something other than access. 
--
Rules of thumb should not be taken for the whole hand.
|
|
|
|
|
Ie every time your app runs the data is different I wouldn't recommend a database. When parsing the national do not call registry I wrote a simple binary search to traverse the massive mulit-gig file to return entries by area code.
Putting it into SQL Server took 2 days. Based on the need and my knowledge of algorithms no db was a better answer.
Of course depending on how complicated you get will depend on when a db becomes a valid choice.
File Not Found
|
|
|
|
|
Ennis Ray Lynch, Jr. wrote: Ie every time your app runs the data is different I wouldn't recommend a database.
Thanks!!! I had not thought of that part. The data will be 100% new each time. I basically need short term data storage that I can build and access quickly, and that is gone once I am done.
We are only talkign about 6 fields (all numbers with one being in Octial format). But the quantity of data is Huge!
I am working on creating an object that will read the file, parse out the data I need, and store it so I can perform functions on the data.
This is an entirely new area for me and I am open to any ideas.
David Wilkes
|
|
|
|
|
Use streams and only persist the results to permanent storage. You may have to fiddle if you havent taken a good Data Structures and Algorithms class but it can be done reasonably fast.
File Not Found
|
|
|
|
|
I am all getting confused. I searched on the net but got more. Basically, when and where do we use class as an Interface, abstract class and static Class or methods. What are the advantages.
I know that Interface doesn't have any implementation it just have methods declaration.
Static class doesn't get instationed.
Thanks
|
|
|
|
|
If you want to define multiple items that must be implemented in a class to guarantee an operation, then use an interface. This means that you can refer to the class that instantiates it without needing to know what the class is. As an example, suppose that you have developed a database agnostic application and you want to read a DataReader, then you could do this:
public class AgnosticReaderImplementation
{
public void Fill(IDataReader dr)
{
while (dr.HasRows)
{
...
}
}
}
This way, the code outside your class could pass in a SqlDataReader, or an OracleDataReader and so on.
An abstract class is normally the base for something else, so it defines the basics (or common options) for your implementations. It is a relatively high-level abstraction that you must inherit from to do something useful. Consider this very trivial example.
public abstract IdBase
{
private string _id = string.Empty;
protected object lockObject = new Object();
private bool _isDirty = false;
public Id
{
get
{
lock (lockObject)
{
return _id;
}
}
set
{
lock (lockObject)
{
Changed(_id, value);
_id = value;
}
}
}
private void Changed(object old, object change)
{
if (old != null && !old.Equals(change))
isDirty = true;
}
public bool IsDirty
{
get {
lock (lockObject)
{
return _isDirty ;
}
set {
lock (lockObject)
{
_isDirty = true;
}
}
}
}
public class DatabaseHandler : IdBase
{
}
This means that in the class DatabaseHandler there certain items that have already been applied by virtue of being in the class IdBase. You can't instantiate IdBase directly, because it isn't really very useful on its own, but it does give you a good base for using in other classes.
Implementing things as static means that you don't have to instantiate a class to call a method. This is useful, for instance, when you want to do something like writing to a log. Suppose that you have a class called Utility which implements a static method WriteToLog, then all you have to do in your code is call Utility.WriteToLog to call it.
the last thing I want to see is some pasty-faced geek with skin so pale that it's almost translucent trying to bump parts with a partner - John Simmons / outlaw programmer
Deja View - the feeling that you've seen this post before.
|
|
|
|
|
An interface is a contract that a class has to fullfil to implement it. You can write code that uses an interface without first having a class that implements it. This is for instance used in the framework to allow for custom comparers.
An abstract class is a class that you can not instantiate, only inherit from to create other classes.
A static class is just a container for static members. You can use it as a container for things like global settings or factory methods that only create objects of other classes.
---
Year happy = new Year(2007);
|
|
|
|
|
I was wondering. I see how complicated the messages on this board are. That I am wondering where I can turn for my insignificant questions as a programmer who has less than 1,000 lines of code in experience.
I find Visual C# 2005 Express Edition very nice, I got a book about it. And that in my second attempt to make a program to multiply matrices and vectors by operators I got a problem which is so weird because according to the book I read this shouldn't be a problem at all. To write a method to create an instance of a class you use a method with the same name as the class, simple, but I got the error message that it wasn't allowed because it caused a name problem.
Wouldn't it be a good idea to add to this board a group or category for beginners?
Ranger
Beginner.
|
|
|
|
|
Well, chances are you would get mostly beginner's replies too. 
Luc Pattyn
|
|
|
|
|
Luc Pattyn wrote: Well, chances are you would get mostly beginner's replies too.
Luc Pattyn
That wouldn't be a problem if they knew a solution to your problem...
But that if my program won't work because Visual C# 2005 Express Edition cannot handle it, then you really have a problem as a beginner!
I remember how some two years ago I tried Java and that my programming project at the time didn't work, and that the problem was Java and not my code.
I just installed the SP1 for V C# 2005 EE, that is the least I could do. So now I have just one error left during compiling. Still my program won't run!
What do I do now?
Ranger.
beginner
|
|
|
|
|
Normally, your first port of call when you have an error is to google for it. Get the text of your error and see if others have had the same issue. There will be some false alarms, and some blind alleys here, but this is a fantastic way to learn things because very often somebody will have written something else along the way that you will want to explore.
the last thing I want to see is some pasty-faced geek with skin so pale that it's almost translucent trying to bump parts with a partner - John Simmons / outlaw programmer
Deja View - the feeling that you've seen this post before.
|
|
|
|
|
As long as the compiler generates errors, your code will not run.
I prefer a short edit/build/run cycle, so I dont add lots of code at once.
That normally avoids adding lots of errors at once too.
If you cant get rid of compile errors, my advice is to disable part of the new code,
this should allow you to run the code, and see how it performs.
To disable some code (that the compiler does not yet like), the easiest way is
to use "#if false" and "#endif" statements (be careful to respect all nestings,
e.g. in the remaining code { and } must still be matched.
You could then:
- either use the debugger capabilities (such as single stepping, watching variables, etc)
- include "log" statements that show intermediate values (either in a Console window,
a listbox, or a file).
Any time you are satisfied with the working code, add some code and watch it in operation.
etc.
Luc Pattyn
|
|
|
|
|
Ranger49 wrote: That wouldn't be a problem if they knew a solution to your problem...
Well... that depends on your point of view. I've seen a beginner ask a question here and another beginner answer it. The answer can range from a little bit off target to so utterly wrong you wonder what on earth they were thinking.
If you have a good mix of people of different skill levels a more experienced person can at least step in and correct the additional problem.
For example: I frequently correct answers on questions relating to databases because the answer, while appearing to solve the immediate problem, open up such a huge can of worms that isn't immediately apparent that I feel I have to step if (if only to protect my self from any application they write that I end up having to interact with)
Ranger49 wrote: I just installed the SP1 for V C# 2005 EE, that is the least I could do. So now I have just one error left during compiling. Still my program won't run!
What do I do now?
Google the error message can work. That is the first thing I do when I don't understand an error (Actually, I use search.live.com[^] these days)
If you still have difficulty then ask on a forum like this. Post the relevant snippet of code and the error message. Remember to include anything that the line is dependent on.
For example: If you have an exception being thrown on a line that reads
myCommand.ExecuteNonQuery(); then that line on its own doesn't mean anything to anyone. You have to include the lines of code that show how the myCommand object was built up so we know what kind of query, what parameters were used and so on. It depends on the actual line of code what you need to include. Also, remember to include the real lines of code. There is no point including something that is similar to the line of code (especially for a syntax error) because then we can't see what is really going on.
I hope this helps you make better use of the forums. We are, on the whole, a pretty good bunch of people that like to help others. But we are more keen to help those that show they are at least willing to try to help themselves first by showing what they've tried, any giving the relevant information. Don't worry if you don't know what is relevant yet, if there is pertinent information missing we'll let you know.
|
|
|
|
|
Colin Angus Mackay wrote: If you have a good mix of people of different skill levels a more experienced person can at least step in and correct the additional problem.
The advice I got here is basically not to try to take too big steps in your learning process.
I appreciate it a lot if someone who has a lot of programming experience will answer my question. But that I am a little bit embarrassed to run into problems which are so simple if you know how to solve it.
It is like riding your first bicycle, you first need to be able to balance yourself and that this takes time and only adults can keep you from falling. So until you reach a certain level of expertise you will run into problems of which you don't have a clue as to what you are doing wrong.
Thanks for everybody who sent me some advice! I got it right now, even though I haven't run this program yet.
Ranger.
Beginner
|
|
|
|
|
Don't be embarrassed. We all started off as beginners. I just wish there had been sites like this when I started (or even the Internet for that matter).
People will sometimes come down hard on somebody who passes themself off as a professional, but who asks dumba*s questions. If you are honest about your level of experience, then people are more inclined to be helpful and will tend to give you the benefit of the doubt.
The issue that you posted certainly wasn't one that I think anybody would be too upset about answering. You posted it, accepted that you were new to the game and listened with respect. With an attitude like that, you could end up going far in this game. I would rather hire one beginner who was willing to learn than 50 idiots who won't listen.
Keep it up, and I look forward to seeing you on the boards more.
the last thing I want to see is some pasty-faced geek with skin so pale that it's almost translucent trying to bump parts with a partner - John Simmons / outlaw programmer
Deja View - the feeling that you've seen this post before.
|
|
|
|
|
I concur.
And I would like to add, we are all beginners, maybe at different levels; but the more you
learn the more you get aware there is more to learn (someone has said that before, in
better words I believe).
Luc Pattyn
|
|
|
|
|
Ranger49 wrote: The advice I got here is basically not to try to take too big steps in your learning process.
Absolutely right - But it is also important not to learn bad habits because it takes longer to unlearn them that it did to learn them in the first place.
Ranger49 wrote: I am a little bit embarrassed to run into problems which are so simple if you know how to solve it.
To reiterate what Pete said, don't be embarrased. If you feel too embarrassed and don't want your friends or colleagues to see what you've posted set up a second account and post under a psudonym if you prefer.
|
|
|
|
|
Ranger49 wrote: But that if my program won't work because Visual C# 2005 Express Edition cannot handle it, then you really have a problem as a beginner!
The express edition can run any C# code. That is not the problem
Ranger49 wrote: So now I have just one error left during compiling. Still my program won't run!
So, you post your code with a statement explaining the error, and what you're trying to achieve.
Christian Graus - C++ MVP
'Why don't we jump on a fad that hasn't already been widely discredited ?' - Dilbert
|
|
|
|
|
Christian Graus wrote: The express edition can run any C# code
That is really nice to hear! And lets face it, express edition wouldn't be any good if it didn't.
It is all a matter of me being a newbe.
About the error message that the compiler handed me, I cut and pasted it into the search/how do I? help line in express edition, and I got the impression that this particular error wasn't listed in the help database. If it had been I probably wouldn't have needed to post in this forum...
Ranger.
Beginner
|
|
|
|
|
All questions are answered here as long as you show that you have made an effort to find a solution yourself. Posts that include existing code fragments are often answered.
Post that as people to do the work for you are usually ignored.
|
|
|
|
|
using System;
using System.Collections.Generic;
using System.Text;
namespace Operator3
{
class Matrix
{
double a11;
double a12;
double a13;
double a14;
double a21;
double a22;
double a23;
double a24;
double a31;
double a32;
double a33;
double a34;
double a41;
double a42;
double a43;
double a44;
public Matrix(double Ina11, double Ina12, double Ina13, double Ina14, double Ina21, double Ina22, double Ina23, double Ina24, double Ina31, double Ina32, double Ina33, double Ina34, double Ina41, double Ina42, double Ina43, double Ina44)
//this method should make an instance of the class Matrix, instead I get the error message that the name Matrix is the same as the name of the class so that it isn't allowed. I haven't got a clue why this doesn't compile!
{
this.a11 = Ina11;
this.a12 = Ina12;
this.a13 = Ina13;
this.a14 = Ina14;
this.a21 = Ina21;
this.a22 = Ina22;
this.a23 = Ina23;
this.a24 = Ina24;
this.a31 = Ina31;
this.a32 = Ina32;
this.a33 = Ina33;
this.a34 = Ina34;
this.a41 = Ina41;
this.a42 = Ina42;
this.a43 = Ina43;
this.a44 = Ina44;
}
public static Matrix operator *(Matrix M1, Matrix M2)
{
// the compiler doesn't seem to have a problem here
return new Matrix(
//kolom 1
M1.a11 * M2.a11 + M1.a21 * M2.a12 + M1.a31 * M2.a13 + M1.a14 * M2.a14,
M1.a12 * M2.a11 + M1.a22 * M2.a12 + M1.a32 * M2.a13 + M1.a24 * M2.a14,
M1.a13 * M2.a11 + M1.a23 * M2.a12 + M1.a33 * M2.a13 + M1.a34 * M2.a14,
M1.a14 * M2.a11 + M1.a24 * M2.a12 + M1.a34 * M2.a13 + M1.a44 * M2.a14,
//kolom 2
M1.a11 * M2.a21 + M1.a21 * M2.a22 + M1.a31 * M2.a23 + M1.a14 * M2.a24,
M1.a12 * M2.a21 + M1.a22 * M2.a22 + M1.a32 * M2.a23 + M1.a24 * M2.a24,
M1.a13 * M2.a21 + M1.a23 * M2.a22 + M1.a33 * M2.a23 + M1.a34 * M2.a24,
M1.a14 * M2.a21 + M1.a24 * M2.a22 + M1.a34 * M2.a23 + M1.a44 * M2.a24,
//kolom 3
M1.a11 * M2.a31 + M1.a21 * M2.a32 + M1.a31 * M2.a33 + M1.a14 * M2.a34,
M1.a12 * M2.a31 + M1.a22 * M2.a32 + M1.a32 * M2.a33 + M1.a24 * M2.a34,
M1.a13 * M2.a31 + M1.a23 * M2.a32 + M1.a33 * M2.a33 + M1.a34 * M2.a34,
M1.a14 * M2.a31 + M1.a24 * M2.a32 + M1.a34 * M2.a33 + M1.a44 * M2.a34,
//kolom 4
M1.a11 * M2.a41 + M1.a21 * M2.a42 + M1.a31 * M2.a43 + M1.a14 * M2.a44,
M1.a12 * M2.a41 + M1.a22 * M2.a42 + M1.a32 * M2.a43 + M1.a24 * M2.a44,
M1.a13 * M2.a41 + M1.a23 * M2.a42 + M1.a33 * M2.a43 + M1.a34 * M2.a44,
M1.a14 * M2.a41 + M1.a24 * M2.a42 + M1.a34 * M2.a43 + M1.a44 * M2.a44);
}
}
public class Vector
{
double v1, v2, v3, v4;
public Vector(double v1, double v2, double v3, double v4)
//This won't compile, why won't this method just create an instance of the class Vector?
{
this.v1 = v1;
this.v2 = v2;
this.v3 = v3;
this.v4 = v4;
}
public static Vector operator *(Matrix M, Vector V)
{
return new Vector(
M.a11 * V.v1 + M.a12 * V.v2 + M.a13 * V.v3 + M.a14 * V.v4,
M.a21 * V.v1 + M.a22 * V.v2 + M.a23 * V.v3 + M.a24 * V.v4,
M.a31 * V.v1 + M.a23 * V.v2 + M.a33 * V.v3 + M.a34 * V.v4,
M.a41 * V.v1 + M.a24 * V.v3 + M.a43 * V.v3 + M.a44 * V.v4);
}
}
class Program
{
static void Main(string[] args)
{
Matrix M = new Matrix(1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0);
Vector V = new Vector(1.0, 0.0, 0.0, 0.0);
}
}
}
//This version is really my 2nd attempt of making a class that defines and multiplies matrices and vectors. So I am a newbe. Please give me some advice.
Beginner.
|
|
|
|
|
Your code compiles just fine for me, using Studio 7.1
Luc Pattyn
|
|
|
|
|
Hi there,
Your code compiles for me on VS2005 Professional. Note, I had to make two changes:
1. "class Matrix" should be "public class Matrix" since the Vector class is public, and the * operator is public and returns a Matrix. The default protection level of a class, if none is specified, is "protected". This resulted in an inconsistent protection level - the Vector * operator was public, but the object it returned is protected.
2. I had to make all the doubles in the Matrix class public, so that the Vector class could access them to do the multiplication.
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
