|
When the data length can vary, there are a couple of ways to cope:
1.
prefix the data with the total length; i.e. first insert an (or a long!) value that specifies total payload length. At the receiver, read the length first, then loop your payload read operations.
2.
alternatively, implement some more elaborate protocol (maybe data packets of a fixed size with a there-is-more flag. This is slightly harder but pays off when you don't know the length in advance.
Your receiver accidentally failing because the payload length was a multiple of 1024 could be very likely up to very unlikely depending on the size of a Common instance; if it is 1024 (or a multiple) then you have the problem for sure; if it is 512B, then you have 50% chance; etc.
But all this does not explain your different list/array behavior...
|
|
|
|
|
Well let me first try to correct the way I am getting the data. I'm going to have to lookup what you are talking about though since I'm not sure how exactly I need to do that since I am just sending and receiving Serialized data and not just text or whatever.
I did leave my laptop at work so I will have to try this tomorrow morning 
|
|
|
|
|
two ways: have the length field go through the MemoryStream (may be difficult as you don't know the length yet), or write it to the socket separately. So replace s.Send(ms.ToArray()); by
byte[] bytes=ms.ToArray();
int len=bytes.Length;
s.Send(BitConverter.GetBytes(len));
s.Send(bytes);
plus some similar changes at the receiver of course.
|
|
|
|
|
Luc,
Thank you for the replies. So I do understand how to send the bytes that contain the length of how many bytes are in my custom object. What I am trying to understand is how I will work this into my program.
How will I know to try to deserialize or just convert the bytes to integer?
On the server the first receive would always be the integer. Once I received all that data I would then call another method to perform another receive but this time deserializing it?
Example:
private void FirstReadCallback(IAsyncResult iar)
{
StateObject so = iar as StateObject;
Socket handler = so.workSocket;
int bytesRead = handler.EndReceive(iar);
if (bytesRead > 0)
{
int myLength = BitConverter.ToInt32(state.buffer);
state.buffer = new byte[myLength];
handler.BeginReceive(state.buffer, 0, state.buffer.Length, 0, new AsyncCallback(SecondReadCallback), state);
}
}
private void SecondReadCallback(IAsyncResult iar)
{
StateObject state = iar.AsyncState as StateObject;
Socket handler = state.worker;
int bytesRead = handler.EndReceive(iar);
if (bytesRead > 0)
{
state.ms.Write(state.buffer, 0, bytesRead);
state.ms.Seek(0, SeekOrigin.Begin);
IFormatter formatter = new BinaryFormatter();
object receivedObject = null;
receivedObject = formatter.Deserialize(state.ms);
}
}
|
|
|
|
|
1.
you would need a Length data member in your class, initially zero.
2.
I guess you have a BeginReceive() somewhere outside the callback; if so, just make it ask for 4 bytes. That is the easiest way.
3.
You can do all EndReceives in a single callback; the first time (when Length is still zero), you need an EndReceive that matches step #2, hope for 4 bytes, and decode them to an integer, to be stored in Length. Use BitConverter.ToInt32(). Later on (when Length isn't zero) do an EndReceive for a full load of data (the maximum of Length and your buffer size), and decrement Length. Keep doing till Length is zero, or timeout or other error occurs.
|
|
|
|
|
Luc,
Sorry for sounding dumb.. but what is with the 4 bytes? I'm searching google some more for example code of this but what I find is people doing things differently than what I'm trying to do. I see most of the code using Encoding to get the string until a EOF was passed or something. I guess I could do it that way. Pass the bytes as a string with a <eof> and cut off the EOF and convert to integer. I do want to try to understand your method, but right now I'm not fully understanding it.
Would you mind providing a short example?
[Edit]
I'm guessing the 4 bytes would be the BitConverter.GetBytes(<ms length="">)..... but how certain is 4 bytes would contain the entire length?
I did some test and even with doing like a: byte[] b = new byte[4958454] the array length is still only 4.
So let me do some test tomorrow and I'll post what I come up with. Thanks for your help
modified on Wednesday, December 1, 2010 11:29 PM
|
|
|
|
|
If the length fits an int at the client (i.e. when it is less than 2 GB), all it takes to communicate it is the four bytes that hold the binary representation of that length value. So it will always be exactly four bytes.
One could pass that as a string (with variable length) but why do so? it only complicates matters, as then one does not know how many bytes/characters to receive, and furthermore the payload is binary data anyway, as it is using binary serialization. I probably would pass length as a string if all the data were text, but absolutely not for binary data.
Here is the receiver in pseudo-code (that looks like C# but isn't):
int length;
byte[] buffer;
int bufsize;
int lengthRequested;
startReceiver() {
length=0;
bufsize=1024;
buffer=new byte[bufsize];
installReceiveHandler(receiveHandler);
startRead(buffer, 0, 4);
}
receiveHandler() {
if (length==0) {
int len=endRead();
length=BitConverter.ToInt32(buffer);
} else {
int len=endRead();
length-=len;
...
}
if (length<=0) {
...
} else {
lengthRequested=length;
if (lengthRequested>bufsize) lengthRequested=bufsize;
startRead(buffer, 0, lengthRequested);
}
}
Warning: not included are error handling, timeouts, etc.
|
|
|
|
|
Ok I fixed it but still have a little problem:
Agent Log:
Sent notification to server that we were about to send 20230 bytes
Transferred the following actions to server: SERVICES,
Server Log:
[12/2/2010 9:48:56 AM]: -- 192.168.1.163:21064 has connected to the server
[12/2/2010 9:48:56 AM]: 192.168.1.163:21064 is about to send 20230 bytes
[12/2/2010 9:48:56 AM]: Finished receiving 8760 bytes from 192.168.1.163:21064
[12/2/2010 9:48:56 AM]: Agent 192.168.1.163:21064 sent back null
[12/2/2010 9:48:56 AM]: Finished sending 118 bytes to 192.168.1.163:21064
So as you can see the agent is sending all the data, but the server is only getting 8760 bytes? Why is the server not reading all of the data?
State Object:
public Socket worker = null;
public int BufferSize = 4;
public byte[] buffer;
public MemoryStream ms = new MemoryStream();
Server:
int bytesRead = handler.EndReceive(iar);
if (bytesRead > 0)
{
if (state.BufferSize == 4)
{
state.BufferSize = BitConverter.ToInt32(state.buffer, 0);
state.buffer = new byte[state.BufferSize];
Logging.Log(handler.RemoteEndPoint.ToString() + " is about to send " + state.BufferSize.ToString() + " bytes", true);
handler.BeginReceive(state.buffer, 0, state.BufferSize, 0,
new AsyncCallback(ReadCallback), state);
}
else
{
state.ms.Write(state.buffer, 0, bytesRead);
Logging.Log("Finished receiving " + state.ms.Length.ToString() + " bytes from " +
handler.RemoteEndPoint.ToString(), true);
state.ms.Seek(0, SeekOrigin.Begin);
IFormatter formatter = new BinaryFormatter();
object receivedObject = null;
try
{
receivedObject = formatter.Deserialize(state.ms);
}
catch (SerializationException se)
{
Logging.Log("Object passed was not capable of being deserialized. Error: " + se.ToString(), false);
}
Client:
MemoryStream ms = new MemoryStream();
IFormatter formatter = new BinaryFormatter();
formatter.Serialize(ms, commons);
byte[] buffer = ms.ToArray();
int len = buffer.Length;
s.Send(BitConverter.GetBytes(len));
Logging.Log("Sent notification to server that we were about to send " + len.ToString() + " bytes", true);
s.Send(buffer);
Logging.Log("Transferred the following actions to server: "+ Tasks.GetActions(), true);
ms.Dispose();
I hope I took your suggestions and something like I should have.
|
|
|
|
|
your client code seems OK; I do have a few comments on the server side:
1.
you never have shown the code that launches the first read, i.e. the BeginReceive that sits outside any callback.
2.
I suggested you first receive the length (4 bytes), then one or more large amounts of data; my pseudo-code was using the same buffer for all of these, so it had to set length to 4 initially, and to the actual buffer size later on. I don't see your code do that.
3.
you could always (temporarily) add more logging to better grasp what is going on.
4.
I suggest your first test is with a very small data set, so it fits in one receive buffer. Only when that works you should test larger amounts of data.
|
|
|
|
|
Ahhhhh ok I see what you are doing now.. but if I'm calling BeginReceive again within the Callback then I don't think that will work. Because the first thing you are doing is checking if Length == 0. There are two times this will happen... for the first BeginReceive and also when we read all the bytes (because we know we are done reading when Length == 0).
I get the idea.. unless I'm missing smoething.. changing it now
|
|
|
|
|
you kick off the receiving actions outside the callback once, then continue the process by issuing another receive inside the callback until you had enough. And length holds the amount still to be received, so once you got it all, you don't issue a new beginreceive, so the callback isn't called any more, the only time it is called with length zero is at the start of a conversation.
|
|
|
|
|
Ah yeah I got it now. Luc I just want to thank you for your help. Here is what I have now and it seems to be working well:
int bytesRead = handler.EndReceive(iar);
if (bytesRead > 0)
{
if (state.Length == 0)
{
state.Length = BitConverter.ToInt32(state.buffer, 0);
Logging.Log(handler.RemoteEndPoint.ToString() + " is about to send " + state.Length.ToString() + " bytes", true);
handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReadCallback), state);
}
else
{
state.Length -= bytesRead;
state.ms.Write(state.buffer, 0, bytesRead);
if (state.Length > 0)
handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReadCallback), state);
else
{
Logging.Log("Finished receiving " + state.ms.Length.ToString() + " bytes from " +
handler.RemoteEndPoint.ToString(), true);
state.ms.Seek(0, SeekOrigin.Begin);
IFormatter formatter = new BinaryFormatter();
object receivedObject = null;
try
{
receivedObject = formatter.Deserialize(state.ms);
|
|
|
|
|
Hi Jacob,
you're welcome. Glad you got it working reliably.
BTW: You could still reduce your code by sharing the read-more stuff, look again at my pseudo-code[^].
|
|
|
|
|
I had posted a question a while ago that asked how I could destroy an object in a windows program (the Flash player) and recreate it because there's a memory leak in the player. It's not huge but if you leave it around long enough it eventually will kill the system.
I got a response that said to put the player in a different application domain. Could somebody explain that please? I've read up on application domains and don't see how I would be able to use it.
What about putting a window in a different process and then have it display the flash. When it's done, it could close the window and kill itself. (at least I think I could do that.) Would that solve the memory leak?
TIA - Jeff.
|
|
|
|
|
why didn't you ask in the thread where you got the advice, so that person (my bet is on Pete) could get a notification e-mail and reply right away?
|
|
|
|
|
I went searching for it and couldn't find it. I am not sure if I asked it here or in the Microsoft forums.
J.
|
|
|
|
|
|
jbradshaw wrote: So if you are going to respond ...
I won't, however I will try and learn from any replies you'll get.
|
|
|
|
|
Luc Pattyn wrote: so that person (my bet is on Pete)
How did you guess?
|
|
|
|
|
I have a memory problem, and it isn't the more common one. I did remember the original thread well, and was hoping for a rather detailed reply. I think an article is in order...
|
|
|
|
|
Luc Pattyn wrote: I did remember the original thread well, and was hoping for a rather detailed reply. I think an article is in order...
OK - I get the hint. I'll see what I can do for you.
|
|
|
|
|
Thanks. I'm looking forward already.
|
|
|
|
|
I'll be looking out for it too! 
|
|
|
|
|
jbradshaw wrote: What about putting a window in a different process and then have it display the flash. When it's done, it could close the window and kill itself. (at least I think I could do that.) Would that solve the memory leak?
Exactly. That was what I was talking about. Basically you'd have a sentinel application that existed purely to create these app domains which host the application.
|
|
|
|
|
Jeff - Luc's persuaded me to write an article on this (it wouldn't be the first time), so I'll pull one together and post it here on CP. Hopefully this will demonstrate what I've been suggesting.
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
