|
Can you show us the signature of the unmanaged function?
|
|
|
|
|
I forgot how to handle that situation in C#.
It needs to allocate with static alloc array of pointers then pin byte[] from List to that array.
void function(byte** data, unsigned int N, unsigned int M)
{
for (int i = 0; i < N; i++)
{
for (int j = 0; j < M; j++)
printf("%c ", data[i][j]);
printf("\n");
}
}
The functions takes array of pointers to byte* arrays.
In C# those byte[] arrays are stored in the List<byte[]> collection.
Чесноков
|
|
|
|
|
I would try calling ToArray() on the List<byte[]> and passing the result.
It should work as an array is a reference type so it should be a pointer to an array of pointers.
|
|
|
|
|
That is not the interaction between 2 managed modules.
What will you do with byte[][] to pass it to unmanaged function taking byte** as argument?
Чесноков
|
|
|
|
|
Ok, I have not tested it, but if the lst object is a List<byte[]> , this is the easiest way:
byte[][] arr = lst.ToArray<byte[]>();
And then pass the arr object to the function if you have imported it setting byte[][] as the type for the first parameter, and you have to make sure that each byte[] array into the List<byte[]> has the same length.
As I said, I have not tested it. Tell us if it does not work, becouse there is another way, though it is a little more tricky.
|
|
|
|
|
.NET will never allow to pass byte[][] to byte** parameter
Чесноков
|
|
|
|
|
Errrr, I think I've got lost somewhere... I was speaking about P/Invoke. What are you speaking about?
|
|
|
|
|
PInvoke.
It is not possible to pass managed byte[][] to unmanaged byte**.
Even if it was possible managed memory is not fixed.
I have List<byte[]> and I need to pass it to byte**.
byte[][] conversion is out of scope as arrays may be of different size.
It needs to allocate with static alloc array to hold pinned pointers. Then to pin them.
I do not remeber how to do it.
Чесноков
|
|
|
|
|
Ok. Let's see.
Chesnokov Yuriy wrote: It is not possible to pass managed byte[][] to unmanaged byte**.
This possibility depends on how you import the unmanaged function. When using P/Invoke the parameters in the managed version just need to have the same size of the target unmanaged version. Since byte** is a pointer, and byte[][] is a reference, both of them have the same size becouse both of them are pointers.
Chesnokov Yuriy wrote: Even if it was possible managed memory is not fixed.
Yes, that's right, but when you make a P/Invoke call, the CLR automatically pins the references passed to the unmanaged function into the managed heap, so the references will not move until the call is finished so, if you use references (such as byte[]), there is no need to manually pin them.
Chesnokov Yuriy wrote: I have List and I need to pass it to byte**.
Try importing the unmanaged function as I told you before, I mean, use byte[][] and not byte** as the type for the first parameter. Belive me, this is the easiest way, it's worth to try. We can try the tricky version later if this does not work.
Chesnokov Yuriy wrote: byte[][] conversion is out of scope as arrays may be of different size.
And casting to byte** is a warranty for that?
Chesnokov Yuriy wrote: It needs to allocate with static alloc array to hold pinned pointers. Then to pin them.
To pin references into the managed heap you have to use GCHandle struct.
|
|
|
|
|
_Erik_ wrote: Yes, that's right, but when you make a P/Invoke call, the CLR automatically pins the references passed to the unmanaged function into the managed heap, so the references will not move until the call is finished so, if you use references (such as byte[]), there is no need to manually pin them.
_Erik_ wrote: Try importing the unmanaged function as I told you before, I mean, use byte[][] and not byte** as the type for the first parameter. Belive me, this is the easiest way, it's worth to try. We can try the tricky version later if this does not work.
That is very intresting.
So far declaring DllImport for PInvoke stated to use native pointers and data types and use fixed every time. That actually compiled. It is a great relief to use managed types in PInvoke declarations instead of unmanaged with fixed statements.
However ToArray() produces the copy of the list. I do not want to spawn copies of the data.
Also is there guarantee that byte[][] array layout in managed memory is plain and there will be no access violation?
I need the ability to quickly pass byte[] objects to byte** array of pointers.
Those byte[] arrays can be stored in a List items or each byte[] array can be a Field in another object thus it is not possible to have ToArray on the list of those objects with those byte[] array items.
Чесноков
|
|
|
|
|
There is more going on inside a List<T> than just a wrapped array which is why it isn't exposed.
private T[] _items;
Havoc could be caused if the array were to be altered outside of the class.
I would suggest calling ToArray and then recreate the list from the array after the PInvoke call if it is likely to change to keep the list in sync.
As for passing the occaisional individual byte[] field - a simple function like this will quickly create an array of array<T>
public static T[][] ArrayToArray<T>(T[] element)
{
return new T[1][]{ element };
}
|
|
|
|
|
byte[] arrays can be of variable length. it will not be possible to create byte[][] from them.
there is no need to modify contents of those byte[] arrays in unmanaged application, but there will be no trouble if you alter the bytes of the array in unmanaged side.
they are passed as pointers, thus unmanaged application can modify them.
Чесноков
|
|
|
|
|
Chesnokov Yuriy wrote: byte[] arrays can be of variable length. it will not be possible to create byte[][] from them.
you can create a byte[1][] from a byte[] as the method I posted shows, regardless of the Length of byte[].
Chesnokov Yuriy wrote: but there will be no trouble if you alter the bytes of the array in unmanaged side.
they are passed as pointers, thus unmanaged application can modify them.
Yes, exactly my point. IIRC arrays are copied to unmanaged memory when passed using PInvoke and then managed memory is updated when the function returns, hence my suggestion to recreate the List when the function returns. All instances of the List will then reflect the changed date.
|
|
|
|
|
There is no need to update List of arrays back.
As I understood you suggested:
List<MyObject> myObjects;
byte[][] data = new byte[myObjects.Count][];
for (i = 0; i < myObjects.Count; i++)
data[i][] = myObjects[i].Data;
Thus byte[][] data array contains variable length rows.
Will there be memory access violation if managed memory is not distributed equally, the array of pointers?
void function(byte** data, int N)
{
byte* pRow = data[0];
pRow = data[1];
pRow = data[2];
pRow = data[N - 1];
}
Чесноков
|
|
|
|
|
That should work fine as the data is laid out correctly in memory if required when PInvoked (I think! I would run an overnight test to repeat this millions of times with varying size arrays to be sure ), with arrays anyway. Luc and I conducted some tests several months ago to see if pinning of array data was required and we reached the conclusion it wasn't as the marshaller handles all the clever array stuff automatically, I would assume this is the case here.
An alternative method may be to use C++ to create a dll that has the native stuff built in and exposes a public method(s() with overloads that can take byte , byte[] , byte[][] , IList<byte> , params byte data etc... This will give you the flexibility you seem to need and may be easier to deal with.
|
|
|
|
|
That is great, I was going to create that solution today to test managed/unmanaged interaction with PInvoke declaration using byte[][] etc... as signatures to confirm fixed is not needed and memory is not moved.
It may need to test it with low free memory or garbage collection somehow to make sure memory will be fixed.
You get my 5 then, great hint. But if those changes are dangerous for access violation then ... Ж8-О
Чесноков
|
|
|
|
|
There is exception in calling byte[][] signature
"Cannot marshal 'parameter #1': There is no marshaling support for nested arrays." System.Exception System.Runtime.InteropServices.MarshalDirectiveException<br />
unsafe static void Print()
{
Random random = new Random();
List<int[]> list = new List<int[]>();
int nRows = random.Next(1, 25);
for (int i = 0; i < nRows; i++)
{
int nCols = random.Next(1, 25);
int[] row = new int[nCols];
row[0] = nCols;
list.Add(row);
}
int[][] data = new int[list.Count][];
for (int i = 0; i < list.Count; i++)
data[i] = list[i];
print(data, nRows);
}
[DllImport("native.dll", CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Unicode)]
static extern unsafe int print(int[][] data, int nRows);
NATIVE_API void print(int** data, int nRows)
{
_tprintf(_T("data** size: %d\n"), sizeof(data));
_tprintf(_T(" data* size: %d\n"), sizeof(data[0]));
for (int i = 0; i < nRows; i++) {
int nCols = data[i][0];
for (int j = 0; j < nCols; j++)
_tprintf(_T("%3d "), data[i][j]);
_tprintf(_T("\n"));
}
}
Are you sure you tried that approach without exceptions?
Also if using byte[,] instead of jagged byte[][] the pointer passed to unmanaged functions turned out to be of 0x00000000 value when I stepped in.
}8-0 I'm going to change my votes ....
Чесноков
|
|
|
|
|
I hadn't tried it usting byte[,] or byte[][], just trying to help as in theory it should work. If it can't be handled by PInvoke then you may have to rethink how you're storing the bytes in the first place.
Chesnokov Yuriy wrote: I'm going to change my votes ....
As is your right. I will play around with this a little this evening regardless as I find the issue quite intersting.
|
|
|
|
|
All right I trust in you, hope you will be able to conjure the right solution
Чесноков
|
|
|
|
|
LOL
Try my latest post here[^]
|
|
|
|
|
Try this...
Change the PInvoke method signature to take a System.IntPtr.
The following method isn't very efficient as it takes two iterations: 1 to determine the memory size required; 2 to copy the data to unmanaged memory, but it should work.
public static IntPtr GetPointerToListOfByteArrays(List<byte[]> byteArrayList)
{
int size = 0;
foreach (byte[] byteArray in byteArrayList)
size += byteArray.Length;
IntPtr pointer = Marshal.AllocHGlobal(size);
IntPtr counter = pointer;
foreach (byte[] byteArray in byteArrayList)
{
Marshal.Copy(byteArray, 0, counter, byteArray.Length);
counter += byteArray.Length;
}
return pointer;
}
It's important that the allocated memory is freed when done so either call Marshal.FreeHGlobal on the pointer returned above or add this helper method and call it passing the pointer.
public static void FreePointerToListOfByteArrays(IntPtr pointer)
{
Marshal.FreeHGlobal(pointer);
}
Edit: Just realized this will layout the bytes themselves in memory, not pointers to the arrays - will fix and repost!
See next post.
Dave
Binging is like googling, it just feels dirtier.
Please take your VB.NET out of our nice case sensitive forum.
Astonish us. Be exceptional. (Pete O'Hanlon)
BTW, in software, hope and pray is not a viable strategy. (Luc Pattyn)
modified on Wednesday, January 19, 2011 3:23 PM
|
|
|
|
|
OK, next try!
Change the PInvoke signature to IntPtr[] and use this:
public static IntPtr[] Allocate(List<byte[]> byteArrayList)
{
IntPtr[] allocatedPointers = new IntPtr[byteArrayList.Count];
int counter = 0;
foreach (byte[] byteArray in byteArrayList)
{
IntPtr pointerToByteArray = Marshal.AllocHGlobal(byteArray.Length);
Marshal.Copy(byteArray, 0, pointerToByteArray, byteArray.Length);
allocatedPointers[counter] = pointerToByteArray;
counter++;
}
return allocatedPointers;
}
public static void Free(IntPtr[] allocatedPointers)
{
foreach (IntPtr allocatedPointer in allocatedPointers)
Marshal.FreeHGlobal(allocatedPointer);
}
|
|
|
|
|
Thanks for your efforts.
I think we are closing the original proposition I remembered before.
Allocate with stackalloc array of pointers and pin references. But I do not remeber how to do it.
That is faster and memory efficient as it needs to allocate only array of list.Count elements.
You solution should work but it leads to superfluous memory operations.
Consider the total memory in byte[] arrays of hundreds Mb.
Чесноков
|
|
|
|
|
The bigest problem I see is - how does the unmanaged side know the size of each byte array as that information isn't being passed (they are variable size arrays right?) It would be possible with either an additional array of ints (the same length as the pointer array) or an array of struct that has an IntPtr (pointer) field and int (size) field. Otherwise, each byte array will have to have a predetermined fixed size.
|
|
|
|
|
That is not the problem
The data is some serialized object which contains the size and other information which C++ application uses to read it, e.g. consider compressed binary file in zip, rar, cab etc... formats.
IntPtr[] data = new IntPtr[list.Count];
for (int i = 0; i < list.Count; i++)
{
row = list[i];
GCHandle handle = GCHandle.Alloc(row, GCHandleType.Pinned);
data[i] = handle.AddrOfPinnedObject();
}
print(data, nRows);
for (int i = 0; i < list.Count; i++)
{
GCHandle handle = GCHandle.FromIntPtr(data[i]);
handle.Free();
}
That works but GCHandle.FromIntPtr() throws exception, event if you omit print call.
I think it needs to keep list of handles to free them
Managed Debugging Assistant 'FatalExecutionEngineError' has detected a problem in 'C:\projs\native\Debug\NativeTest.exe'.
Additional Information: The runtime has encountered a fatal error. The address of the error was at 0x655229e9, on thread 0xebc. The error code is 0xc0000005. This error may be a bug in the CLR or in the unsafe or non-verifiable portions of user code. Common sources of this bug include user marshaling errors for COM-interop or PInvoke, which may corrupt the stack.
Чесноков
|
|
|
|
|