You don't say how you instantiate the COM Server, but if it's an out-of-process server, or a component that you access from a remote machine, and you are a client of that server, then the COM library uses the RPC protocol to transfer the data back and forth between the two machines. You might refer to:
The Network Protocol[
^], a chapter from the book "Inside COM+", which describes the methods COM uses to communicate with a DCOM component on a remote machine. Also, from the same book,
Standard Marshaling[
^], explains the actual COM Library mechanisms involved in managing the proxy and stub architecture.
If you are accessing an out-of-process server, then that is definitely your bottleneck, and not the code that you posted in the C# Forum.