|
As a first time poster, you may not be aware, the etiquette on this site is to post questions in English. By using English, you get the most chance that your question will be answered. If you'd care to edit your question, we may be able to help.
|
|
|
|
|
hello
I am javad andamani from iran.
the question is type FARSI(persian).
tank you
|
|
|
|
|
The question type mught well be Farsi, but the only language you should be posting in (except the dedicated Indian and Chinese Forums) is English. Please read an FAQ.
|
|
|
|
|
Well whoop-do-frakkin-do. That's no good to me - I don't read Farsi, so how in the name of St Steve Jobs do you expect me to be able to answer it? For all I know, you could be posting a recipe for your mother's delicious home made fudge. As I said before - rephrase your question in English. I'd already figured it was Farsi.
|
|
|
|
|
Fudge would be nice... With a few almonds in a german chocolate fudge... Now I'm drooling.... Again!!!
I wasn't, now I am, then I won't be anymore.
|
|
|
|
|
|
Never post your email address in any forum, unless you really like spam! If anyone replies to you, you will receive an email (like this one) to let you know.
For your own protection, delete your reply!
Real men don't use instructions. They are only the manufacturers opinion on how to put the thing together.
|
|
|
|
|
Right next to the Reply link is an Email link. If you click on that it should let you send an email to the member whose message you are replying to without posting it publicly in the forum.
Jack of all trades ~ Master of none.
|
|
|
|
|
I have an app that reads data from a sensor, displays the last 60 seconds or so on as a graph and has some other events that need to happen at specific timings. Currently I'm only storing the last 60 seconds of data at any one time in order to stop it from consuming increasing amounts of memory.
However, I now need to think about persisting the data to a file as it's collected and I'm looking for any recommendations on best practice for this. It is very important that the writing to file doesn't interfere with reading data from the sensor and the timing of other events (which are trigger using Theading.Timer) as much as possible. In other words, I don't want disk access to delay processing of data coming from the sensor or to delay running the Timer's delegate method when the timer is up. Or in other, other words, the writing to disk should be very low priority.
The data coming from the sensor isn't particularly fast, it's about 7Hz and what I need to store is the time since the start of recording (in seconds, stored as a double) and the value from the sensor (also a double). I could easily see an hour of data being collected at one time which I (very) roughly estimate as being about 1.5 Mb of raw data. Ultimately this would need to be saved in a human readable format (or at least an Excel importable format) maybe tab delimited text, so the file size on disk would be several times bigger.
So any thoughts? Recommendations? Am I better off trying to store as much as possible in memory and then only write to disk when recording ends? Or is it better to try and write pieces as it's collected?
[edit: typo in topic title]
modified on Wednesday, January 5, 2011 3:39 PM
|
|
|
|
|
With the low volume you have there should be no problem at all (assuming you want to stream it to disk).
1.
Yes you could keep it in memory and output it all when done.
2.
You could also store it in a file; if you keep the stream open, it will not slow down over time, all that will happen is it will periodically be written to disk in chunks of one or a few sectors. For maximum performance, you could store binary data; if ASCII data is needed a post-processing step could be considered. (For your throughput, you could write ASCII text right away).
3.
And finally you could store the data in a database (on the same or a different machine), again in the appropriate numeric format. You could then load it into Excel faster than through a CSV file.
I have one advice on the data capture: whatever means you use for timing it all, in my experience it is wise to store measured time (maybe just DateTime.Now), not calculated time. That way, even when something would go wrong for a while (your system suddenly being very busy at something else) you would still store sensor values with their matching time values.
|
|
|
|
|
Thanks for sharing your thoughts on this.
|
|
|
|
|
You're welcome.
|
|
|
|
|
Would this be a good place for a Synchronized Queue[^]? You could enqueue with a higher-priority thread, and output (dequeue) with a lower one.
Is the response from a Queue fast enough to make that feasible? (I assume so.)
|
|
|
|
|
1.
yes, I guess so. I haven't used it yet, I expect it is a regular queue plus a lock, which would be fine. If the app is adding one item (sensor value + time value) to the queue every few seconds, then obviously anything goes.
2.
don't get me started on thread priorities. IMO they aren't worth much in Windows (I have a background in embedded real-time systems). If your producer needs to have higher priotity, then the consumer might get flooded with data, and if that is acceptable, you basically are admitting you could store everything in memory before flushing it to disk...
|
|
|
|
|
Re: #2
Sure it could be flooded, but this allows for the near-time graph display (mentioned in original post, unless I am misunderstanding it); besides, storing it all in memory seemed acceptable when you proposed it earlier. I don't disagree, this may just be a way to have your cake and eat it too, so to speak. 
|
|
|
|
|
if the graph is to show all the measurements (as opposed to say the last 300 samples), then all data is in memory anyway; and I'll eat the cake anyway.
|
|
|
|
|
Actually the graph show the last 60 seconds worth of data. It's about 400 odd points - I actually store 450 in a circular array so it over writes itself. I change the max and min of the time axis as each data point is added so nobody sees the missing past data. I'm surprised it works as well as it does, it gives it a cool chart recorder vibe! Maybe I should try and write an article about it.
|
|
|
|
|
I did exactly that once to experiment and check a flicker free chart would result; it looked great indeed, whereas lots of people seem to believe it can't be done in WinForms at all.
|
|
|
|
|
Mine WPF using the WPF toolkit from CodePlex.
|
|
|
|
|
Thanks for the suggestion. I'll look into it.
|
|
|
|
|
I had to do something very similar at a previous job. I set it up so that each data source (your sensor) had it's own reader thread. This thread parsed out the data and made note of other things like timing, etc. Each reader had a collection of data sinks, which was just a generic term for a consumer of the data. The reader would pass on each chunk of data to each data sink, which in turn would do whatever it wanted with the data. If you try something similar, you could have both your real-time graph and the file be data sinks. The real-time graph data sink could have it's own thread and 60-second circular queue. The file data sink could use its own queue or write the data directly to the file. Personally, I'd have the data file use another thread and queue and write data to the file in chunks of say 5-10 seconds or so. It's fast enough that you shouldn't lose too many data points if something goes wrong, and it's infrequent enough that things should not get bogged down.
|
|
|
|
|
Interesting. I like your set up with the idea of data sinks. One other problem that I didn't really elaborate on in my original post is that I need to trigger other things based on the data coming in, so, for example, when the data crossing a threshold I need another device to be triggered. So I really need to have at least one sink be "high-priority" because it has to identify certain conditions in the data and react as quickly as possible.
|
|
|
|
|
Wjousts wrote: The data coming from the sensor isn't particularly fast, it's about 7Hz and what I need to store is the time since the start of recording (in seconds, stored as a double) and the value from the sensor (also a double). I could easily see an hour of data being collected at one time which I (very) roughly estimate as being about 1.5 Mb of raw data. Ultimately this would need to be saved in a human readable format (or at least an Excel importable format) maybe tab delimited text, so the file size on disk would be several times bigger.
So any thoughts? Recommendations? Am I better off trying to store as much as possible in memory and then only write to disk when recording ends? Or is it better to try and write pieces as it's collected?
Where 7 Hz = 7 inputs per second?
In a 24 hour period that gives you 600,000+ rows. Exclusive of finding one value at a specific time or a range in small interval that isn't going to be human readable regardless of what you do.
Storage as text if you store only the time would be probably 12 meg for each day.
You could do one csv file per day.
If you really want to do time since start rather than time of day then you should plan on putting the time the app started either in the file name or at the beginning of the file. The date of course would be in the file name.
|
|
|
|
|
Okay, human readable is the wrong phrase, importable into Excel or something similar is what I meant. We wouldn't collect 24 hours worth of data at a time anyway. Probably not more than about 1 hour tops, and what we need to be able to do is graph that data and see how well it matches up with data coming from somewhere else.
Putting the time in the filename is a nice idea though. It eliminates the mess of letting users decide what the file should be called as well.
|
|
|
|
|
I also recommend looking at the performance of your program too. I think you're not storing huge amounts of continuous data. I use a profiler http://www.red-gate.com/products/dotnet-development/[^] to help me spot nastiness. When collecting data from devices, it pays to make sure that your program runs smoothly and doesn't accumulate extra memory.
I know it's a little off topic, but doing performance analysis has helped me quite a bit.
"Simplicity carried to the extreme becomes elegance."
-Jon Franklin
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
