|
There is no need to update List of arrays back.
As I understood you suggested:
List<MyObject> myObjects;
byte[][] data = new byte[myObjects.Count][];
for (i = 0; i < myObjects.Count; i++)
data[i][] = myObjects[i].Data;
Thus byte[][] data array contains variable length rows.
Will there be memory access violation if managed memory is not distributed equally, the array of pointers?
void function(byte** data, int N)
{
byte* pRow = data[0];
pRow = data[1];
pRow = data[2];
pRow = data[N - 1];
}
Чесноков
|
|
|
|
|
That should work fine as the data is laid out correctly in memory if required when PInvoked (I think! I would run an overnight test to repeat this millions of times with varying size arrays to be sure ), with arrays anyway. Luc and I conducted some tests several months ago to see if pinning of array data was required and we reached the conclusion it wasn't as the marshaller handles all the clever array stuff automatically, I would assume this is the case here.
An alternative method may be to use C++ to create a dll that has the native stuff built in and exposes a public method(s() with overloads that can take byte , byte[] , byte[][] , IList<byte> , params byte data etc... This will give you the flexibility you seem to need and may be easier to deal with.
|
|
|
|
|
That is great, I was going to create that solution today to test managed/unmanaged interaction with PInvoke declaration using byte[][] etc... as signatures to confirm fixed is not needed and memory is not moved.
It may need to test it with low free memory or garbage collection somehow to make sure memory will be fixed.
You get my 5 then, great hint. But if those changes are dangerous for access violation then ... Ж8-О
Чесноков
|
|
|
|
|
There is exception in calling byte[][] signature
"Cannot marshal 'parameter #1': There is no marshaling support for nested arrays." System.Exception System.Runtime.InteropServices.MarshalDirectiveException<br />
unsafe static void Print()
{
Random random = new Random();
List<int[]> list = new List<int[]>();
int nRows = random.Next(1, 25);
for (int i = 0; i < nRows; i++)
{
int nCols = random.Next(1, 25);
int[] row = new int[nCols];
row[0] = nCols;
list.Add(row);
}
int[][] data = new int[list.Count][];
for (int i = 0; i < list.Count; i++)
data[i] = list[i];
print(data, nRows);
}
[DllImport("native.dll", CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Unicode)]
static extern unsafe int print(int[][] data, int nRows);
NATIVE_API void print(int** data, int nRows)
{
_tprintf(_T("data** size: %d\n"), sizeof(data));
_tprintf(_T(" data* size: %d\n"), sizeof(data[0]));
for (int i = 0; i < nRows; i++) {
int nCols = data[i][0];
for (int j = 0; j < nCols; j++)
_tprintf(_T("%3d "), data[i][j]);
_tprintf(_T("\n"));
}
}
Are you sure you tried that approach without exceptions?
Also if using byte[,] instead of jagged byte[][] the pointer passed to unmanaged functions turned out to be of 0x00000000 value when I stepped in.
}8-0 I'm going to change my votes ....
Чесноков
|
|
|
|
|
I hadn't tried it usting byte[,] or byte[][], just trying to help as in theory it should work. If it can't be handled by PInvoke then you may have to rethink how you're storing the bytes in the first place.
Chesnokov Yuriy wrote: I'm going to change my votes ....
As is your right. I will play around with this a little this evening regardless as I find the issue quite intersting.
|
|
|
|
|
All right I trust in you, hope you will be able to conjure the right solution
Чесноков
|
|
|
|
|
LOL
Try my latest post here[^]
|
|
|
|
|
Try this...
Change the PInvoke method signature to take a System.IntPtr.
The following method isn't very efficient as it takes two iterations: 1 to determine the memory size required; 2 to copy the data to unmanaged memory, but it should work.
public static IntPtr GetPointerToListOfByteArrays(List<byte[]> byteArrayList)
{
int size = 0;
foreach (byte[] byteArray in byteArrayList)
size += byteArray.Length;
IntPtr pointer = Marshal.AllocHGlobal(size);
IntPtr counter = pointer;
foreach (byte[] byteArray in byteArrayList)
{
Marshal.Copy(byteArray, 0, counter, byteArray.Length);
counter += byteArray.Length;
}
return pointer;
}
It's important that the allocated memory is freed when done so either call Marshal.FreeHGlobal on the pointer returned above or add this helper method and call it passing the pointer.
public static void FreePointerToListOfByteArrays(IntPtr pointer)
{
Marshal.FreeHGlobal(pointer);
}
Edit: Just realized this will layout the bytes themselves in memory, not pointers to the arrays - will fix and repost!
See next post.
Dave
Binging is like googling, it just feels dirtier.
Please take your VB.NET out of our nice case sensitive forum.
Astonish us. Be exceptional. (Pete O'Hanlon)
BTW, in software, hope and pray is not a viable strategy. (Luc Pattyn)
modified on Wednesday, January 19, 2011 3:23 PM
|
|
|
|
|
OK, next try!
Change the PInvoke signature to IntPtr[] and use this:
public static IntPtr[] Allocate(List<byte[]> byteArrayList)
{
IntPtr[] allocatedPointers = new IntPtr[byteArrayList.Count];
int counter = 0;
foreach (byte[] byteArray in byteArrayList)
{
IntPtr pointerToByteArray = Marshal.AllocHGlobal(byteArray.Length);
Marshal.Copy(byteArray, 0, pointerToByteArray, byteArray.Length);
allocatedPointers[counter] = pointerToByteArray;
counter++;
}
return allocatedPointers;
}
public static void Free(IntPtr[] allocatedPointers)
{
foreach (IntPtr allocatedPointer in allocatedPointers)
Marshal.FreeHGlobal(allocatedPointer);
}
|
|
|
|
|
Thanks for your efforts.
I think we are closing the original proposition I remembered before.
Allocate with stackalloc array of pointers and pin references. But I do not remeber how to do it.
That is faster and memory efficient as it needs to allocate only array of list.Count elements.
You solution should work but it leads to superfluous memory operations.
Consider the total memory in byte[] arrays of hundreds Mb.
Чесноков
|
|
|
|
|
The bigest problem I see is - how does the unmanaged side know the size of each byte array as that information isn't being passed (they are variable size arrays right?) It would be possible with either an additional array of ints (the same length as the pointer array) or an array of struct that has an IntPtr (pointer) field and int (size) field. Otherwise, each byte array will have to have a predetermined fixed size.
|
|
|
|
|
That is not the problem
The data is some serialized object which contains the size and other information which C++ application uses to read it, e.g. consider compressed binary file in zip, rar, cab etc... formats.
IntPtr[] data = new IntPtr[list.Count];
for (int i = 0; i < list.Count; i++)
{
row = list[i];
GCHandle handle = GCHandle.Alloc(row, GCHandleType.Pinned);
data[i] = handle.AddrOfPinnedObject();
}
print(data, nRows);
for (int i = 0; i < list.Count; i++)
{
GCHandle handle = GCHandle.FromIntPtr(data[i]);
handle.Free();
}
That works but GCHandle.FromIntPtr() throws exception, event if you omit print call.
I think it needs to keep list of handles to free them
Managed Debugging Assistant 'FatalExecutionEngineError' has detected a problem in 'C:\projs\native\Debug\NativeTest.exe'.
Additional Information: The runtime has encountered a fatal error. The address of the error was at 0x655229e9, on thread 0xebc. The error code is 0xc0000005. This error may be a bug in the CLR or in the unsafe or non-verifiable portions of user code. Common sources of this bug include user marshaling errors for COM-interop or PInvoke, which may corrupt the stack.
Чесноков
|
|
|
|
|
On which iteration does it fail? If it fails on the second time then you may need to run the freeing for loop in reverse to unwind the data.
Just a thought as I can't see why it would fail otherwise.
for (int i = list.Count - 1; i >= 0; i--)
{
GCHandle handle = GCHandle.FromIntPtr(data[i]);
handle.Free();
}
|
|
|
|
|
It fails on the very first iteration. Reverse order does not solve it.
Чесноков
|
|
|
|
|
The List<T> class has an inner private field (_items) which is an array of T. You might retrieve it using reflection, but there is a problem: you do not have control over the size of this array unless you specify its capacity. If you want to pass the arrays without making any copy of it, your best choice is to develop a new class for this, using a two dimensional array, I mean, a byte[,] instead of a byte[][], as a field to hold the data, and maybe wrap the P/Invoke call within this class. This way you can use the reference without making any copy and, also, you will be sure that everything in memory has the size it should. Something like:
public class ByteMatrix
{
byte[,] _matrix;
int _n;
int _m;
public ByteMatrix(int n, int m)
{
_matrix = new byte[n, m];
_n = n;
_m = m;
}
[DllImport ... whatever]
private static extern int unmanagedFunction(byte[,] array, int n, int m);
public int CallUnmanaged()
{
return unmanagedFunction(_matrix, _n, _m);
}
}
|
|
|
|
|
|
Well, the msdn documentation for this is a little mess, but I remember some time ago, after some hours of research, I read that we had to pass a two dimensional array in these cases, I mean, we have to pass int[,] for an int** parameter in the unmanaged version, though I don't remember exactly if we could pass the reference itself. I still think there is a way to avoid copying data from one place to another. This time we should import the unmanaged function, setting IntPtr as the type for the fisrt parameter. The class I told you before would change as follows:
public class ByteMatrix
{
byte[,] _matrix;
int _n;
int _m;
public ByteMatrix(int n, int m)
{
_matrix = new byte[n, m];
_n = n;
_m = m;
}
[DllImport ... whatever]
private static extern int unmanagedFunction(IntPtr array, int n, int m);
public int CallUnmanaged()
{
GCHandle handle = GCHandle.Alloc(_matrix, GCHandleType.Pinned);
int ret = unmanagedFunction(handle.AddrOfPinnedObject(), _n, _m);
handle.Free();
return ret;
}
|
|
|
|
|
1) byte[] vectors are not of equal length, it needs to use jagged array byte[][] instead but that does not work with your solution
2) byte[,] array is not C byte** array, which is array of pointers to rows, the memory is different, in unmanaged function data[0] is 0x0000000 address
It needs array of handles and pin each byte[] from the list
Чесноков
|
|
|
|
|
I've been lucky and have found it again. Here is a link[^] to the msdn documentation for this. See how they import the TestMatrixOfInts function.
I don't have time to make tests, but I am trying to help you. However, I won't be able to do so if you discard my suggestions before giving them a try. I know it may seem wrong at a first glance, but anyway a .NET bidimensional array might be internally implemented with an array of arrays.
Anyway, up to this moment I had understood that all of your byte[] arrays were the same size, but now I am not sure if that premise is correct. Is it?
Edit:
I wrote this before reading the part of the conversation with Dave about this topic. Now I have a better picture of it. The byte[] arrays are not of the same size, so a bidimensional array would not be helpful.
You have suggested stackalloc, but that is not what I recommend you. It allocates the memory in the stack, not in the heap, and that memory will be released when the method ends. Dave gave you a good advise with the IntPtr[] array, but instead of making Marshal.AllocHGlobal and Marshal.Copy you can use GCHandle, so you would not have to make a copy of the arrays into the global heap. Something like this:
int CallUnmanagedMethod(List<byte[]> lst)
{
GCHandle[] handleArray = new GCHandle[lst.Count];
for (int i=0; i<lst.Count; i++)
handleArray[i] = GCHandle.Alloc(lst[i], GCHandleType.Pinned);
IntPtr[] pointers = (from GCHandle handle in handleArray select
handle.GetAddrOfPinnedObject()).ToArray();
int ret = unmanagedMethod(pointers, ...
foreach (GCHandle handle in handleArray)
handle.Free();
return ret;
}
This method would just pin into the managed heap the byte[] arrays you have in the list, without making any copy of them, then, using GCHandle, you build an IntPtr[] array to hold the address of each one and you can make the P/Invoke call.
|
|
|
|
|
_Erik_ wrote: I don't have time to make tests, but I am trying to help you. However, I won't be able to do so if you discard my suggestions before giving them a try. I
I tried to pass either byte[][] or byte[,] directly. First is not valid to be passed.
Second does not contain pointers to its rows and in native function data[0] results in the value hold in first element in the managed array.
MSDN link you posted is useful.
I gathered from your and Dave ideas the following solution before your post
That one atually works.
http://www.codeproject.com/Messages/3740534/Re-How-to-pin-List-to-unmanaged-byte.aspx[^]
But I wanted to keep only IntPtr[] array and restore handles with GCHandle.FromIntPtr() which results in .NET message box of incorrect operation in debugger, my next post.
Чесноков
|
|
|
|
|
Yes, GCHandle is a pretty delicate thing. When a GCHandle is created it adds a new entry to the AppDomain's handle table, and this entry is kept there until Free method is invoked. The problems may arise when we create more GCHandle objects to the same target and in the same method. Remember GCHandle is also a struc, I mean, a value type, so any instance of it will be a different copy into the stack. Even if you made this:
GCHandle handle2 = handle1;
Since GCHandle is a struct, freeing handle2 would not free handle1, and this might take us to a strange situation becouse handle1's target might become a null reference, and this handle would still be in the AppDomain's handle table. This is the reason why I always try to avoid making several copies of a GCHandle.
Summarizing, it is much better to maintain the array of GCHandle objects than trying to recreate them.
|
|
|
|
|
I presume byte[][] or byte[,] layout is different.
It is not array of pointers and it is possible to avoid fixed with byte[] array
Чесноков
|
|
|
|
|
First thing(s) thats important to know is:
1) who creates the byte[]? obviously the C# side is the caller, but what I mean is, is it calling the C++ code to say "you build the buffer and give it back in this pointer", or is it saying "heres a buffer, fill it"?
2) does the C++ code need to grow or modify the array? or just read it?
3) who owns the buffer after the call? Does the C# side still own it, or does the C++ side now own it?
4) do you have control to modify the C++ side?
|
|
|
|
|
1) C# application creates those byte[] array buffers and fills them with data
2) there is no need to modify, only read it
3) C# owns them since that is the application created the
4) yes but the prefered format is array of pointers byte**
As C# application creates those buffers they are managed resources.
As they are passed as pointers C++ application can also write to them, but that is no needed.
Чесноков
|
|
|
|
|
Have you looked at Marshall.AllocHGlobal and Marshall.Copy?
|
|
|
|
|