[Download Library From Github]
Introduction
Lately, I needed to deal with (i.e, read/write) files from any Android Phone/Tablet/iPhone/iPad connected via USB, programmatically.
Should be easy... But these devices don't show up as drives when you enumerate DriveInfo.GetDrives, so there's no way to acess the files/folders on them.
That's what my Lib solves:
drive_root.inst.parse_folder("[a0]:/*/dcim/camera").files.First()
.copy_sync("C:/my_camera");
Background
The library I created wants to deal with any external drive - anything that can be connected/disconnected via just a plug. One of the issues I will deal with (all the subject of another article), is the fact that you might connect an external drive, and it might end up being "drive F:", but at a later time, the same drive will end up being "drive H:". Wouldn't you like to have a way to uniquely identify a file/folder name to a specific external drive/sd card/phone/tablet?
After implementing the code, and dealing with lots of Windows idiosyncrasies, I decided to make this freely available.
Dealing with USB Drives
If you play around a bit with external devices, you'll notice something interesting: external drives, sticks, CD/DVDs, SD cards will auto-mount: as soon as you plug them in, they will be recognized as a new drive. Just enumerate the new drives, and then see what files/folders are on them.
foreach ( var drive in DriveInfo.GetDrives())
Console.WriteLine(drive.Name);
All, but the Portable-connected USB devices, such as your Phone or Tablet...
They will show up in "My Computer" (Win-E), as a sort of virtual drive, without a drive letter (thus, not accessible via the regular DriveInfo API).
After quite a bit of googling, it finally hit me: use the shell32.dll's API to access virtual folders - the USB drive will show up there. It'll be a bumpy ride, but all well worth it!
Portable vs Android vs Phone/Tablet
The library gives you get access to any Portable device that basically implements the shell32.dll's Windows Shell Interface that allows access to its "drive". This can be any of:
- an Android device
- an Android Phone / iPhone
- an Android Tablet / iPad
- a Camera connected via USB
- anything else that "registers" itself as a child of "My Computer", not as a drive, but as a Virtual Folder (that can have Virtual Sub Folders and/or Files)
From now on, I will refer more to "Android", or "Phone/Tablet", than Portable (which is a rather vague term).
But know that if it shows here, under "Devices and Drives", you can access it with my library:
Sync thy Phones
Before showing you the API, lets delve into a bit of client code...
Say you're tasked to allow your client to Synchronize all his Phones and Tablets, and cache the last 100 images taken with the camera (so he can access them even if the device is not connected).
By that, I meant - as soon as he plugs in the device, as long as its unlocked, the Synchronize process should happen automatically.
static void cache_last_files_thread() {
var cache_root = "c:/john/buff";
Dictionary<string, DateTime> needs_cache = new Dictionary<string, DateTime>();
while (true) {
Thread.Sleep(1000);
foreach (var ad in drive_root.inst.drives.Where(d => d.type.is_android())) {
var already_cached = needs_cache.ContainsKey(ad.unique_id)
&& needs_cache[ad.unique_id] > DateTime.Now;
if (!already_cached) {
var valid_until = DateTime.Now.AddHours(1);
if ( !needs_cache.ContainsKey(ad.unique_id))
needs_cache.Add( ad.unique_id, DateTime.MinValue);
needs_cache[ad.unique_id] = valid_until;
cache_now(ad, cache_root);
}
}
}
}
static void cache_now(IDrive ad, string root) {
var files = ad.parse_folder("*/dcim/camera").files.OrderBy(f => f.last_write_time).ToList();
if (files.Count > 100) files = files.GetRange(files.Count - 100, 100);
var local_root = root + "/" + ad.unique_id;
foreach (var f in files)
if ( !File.Exists(local_root + "/" + f.name))
f.copy_sync(local_root);
}
Here are the highlights:
- In the
cache_last_files_thread, every second, we enumerate all external Android drives. If it's a new drive, do cache_now. - Every drive has a
.unique_id (explained in more detail later). Think of the unique ID as a serial number that uniquely identifies this device at all times. This is how we identify each device. - When caching, we have a local cache directory. Each external device gets a subdirectory, named by its .unique_id . In that sub-directory, we hold the cache for that device. This makes it easy to see if we have already cached a file.
Not bad, for 25 lines of code...
The drive_root
The interface is meant to be super easy to use:
class drive_root {
public static drive_root inst { get; } = new drive_root();
IFile parse_file(string path) ;
IFile try_parse_file(string path) ;
IFolder parse_folder(string path) ;
IFolder try_parse_folder(string path) ;
IFolder new_folder(string path) ;
IReadOnlyList<IDrive> drives ;
};
Basically, just allow easy access to any file or folder on any Portable drive:
Console.WriteLine(
string.Join(",",drive_root.inst.parse_folder("[a0]:/*/dcim/camera").files.Select(f => f.name)));
Or, just print a file's size:
Console.WriteLine(drive_root.inst.parse_file("[a0]:/*/dcim/camera/20171005_121557.jpg").size);
The drive_root allows you to easily get access to a folder or file, either from an external device (such as a USB Android device) or an internal HDD. From there on, you can enumerate a folder's files or deal with a specific file.
parse_file and parse_folder will return a file/folder respectively, or throw if it does not exist. Their try_ counterparts will return null if the file/folder does not exist.
The reason I allow access to all drives (not just the external ones) is simple: you may want to copy a file from your Phone to your hard-disk, but you could want to copy from your hard-disk to your phone:
drive_root.inst.parse_file("[a0]:/*/dcim/camera/my_photo.jpg").copy("c:/john/buff");
drive_root.inst.parse_file("c:/john/buff").copy("[a0]:/*/dcim/camera");
As a final note, you can use "/" and "\" interchangeably - as file/folder separators. I prefer to use /, since using \ will always force me to escape it as '\\'.
The drive(s)
In .[try_]parse_folder and .[try_]parse_file, you can access a drive via several ways:
- its drive letter. Like,
drive_root.inst.parse_folder("c:/windows"). I encourage you to do this only for internal HDD drives. Any external drive, when you plug it in, it can get a new drive letter. Such as, a Toshiba external HDD, it can be mounted as E:, and a few days later, be mounted as H:. - the Unique ID. This is perfect for any Portable device. Each device is uniquely identified by some sort of Serial Number. This allows you to uniquely identify a file/folder to a speicific device. Once a new device is connected, even if it has the same folder, you know it's got different contents. Example:
drive_root.inst.parse_folder("{930002527dd2345ab}:/phone/dcim"); - the Portable-index. This is really useful for testing, or to easily identify the first, second, third connected device. Since usually users will have one or maximum two USB-connected devices, it's sooo easy to just get to them. To get to the first connected Portable device, use:
drive_root.inst.parse_folder("[p0]:/phone/dcim"); To get to the third connected Portable device, use drive_root.inst.parse_folder("[p2]:/phone/dcim"); - the Android-index. This is similar to the Portable-index, only that it refers only to Android devices. To get to the first connected Android device, use:
drive_root.inst.parse_folder("[a0]:/phone/dcim"); - the Device-index. This is the index of the device, within the device root. Note that always the internal HDDs are first, and USB-connected devices are last. For instance,
drive_root.inst.parse_folder("[d0]:/windows"); (very likely, equivalent to "c:/windows"), or drive_root.inst.parse_folder("[d1]:/pics"); (very likely, equivalent to "d:/pics"). Use this when you know the device order, very likely for testing.
While you're free to use any of the above, for external drives, I encourage you to use the Unique ID. First of all, it's sooo easy to know the Unique ID of the Android drives:
var drives = drive_root.inst.drives.Where(d => d.type.is_android());
foreach ( var d in drives) Console.WriteLine(d.unique_id);
Any file or folder, when they return the full path, they will use the Unique ID as their prefix. This clearly specifies where the file/folder is coming from (it's bound to a specific device).
Assume you want to cache some files that come from an Android device. You will want to know the exact device the cahed files relate to. You don't want to plug in another device that happens to have some files with the exact same names (with completely different contents), and end up using the cached files, which are nothing like the ones in the new device.
Therefore, each file or folder, when you ask for their full path, they are prefixed by the drive's Unique ID:
Console.WriteLine( drive_root.inst.parse_folder("[a0]:/*/dcim").full_path);
As a final goodie, for .[try_]parse_folder / .[try_]parse_file, if a Portable drive's path starts with *, it means the following: if the drive root contains a single folder, replace * by that folder name.
Usually, each Phone/Table maker will have the root folder of their drive called as they please, such as "Phone", "Tablet", "P8_Lite", or whatever. If you want to get to the camera's photos, on your first Android device, here's how:
var photos = drive_root.inst.parse_folder("[a0]:/*/dcim/camera").files.;
The IDrive
You can get to a drive several ways:
drive_root.inst.drives - get all drives, internal and external- any file's
.drive property - any folder's
.drive property drive_root.inst.[try_]get_drive(drive_name) - gets the given drive by name (try_ will return null if it doesn't exist; the get_drive will throw if drive does not exist).
Usually you'll want to focus more on folders and files, rather than on drives. Therefore, the IDrive's interface is really simple:
public interface IDrive {
bool is_connected();
drive_type type { get; }
string root_name { get; }
string unique_id { get; }
string friendly_name { get; }
IEnumerable<IFolder> folders { get; }
IEnumerable<IFile> files { get; }
IFile parse_file(string path);
IFolder parse_folder(string path);
IFile try_parse_file(string path);
IFolder try_parse_folder(string path);
IFolder create_folder(string folder);
}
It's all pretty straightforward. Just a few things to notice:
is_connected():
- Guaranteed not to to throw an exception
- Returns
true for all internal hdds. - For portable drives, returns
true if the drive is connected via USB and it's connected for "Transferring media files via USB". In other words, if in "My Computer", you can click on this drive, and it's got folders and files, then is_connected() returns true. - As soon as you unplug the portable device, this will return
false. - When the portable device is unplugged, that drive will not show up in
drive_root.inst.drives
type: returns the drive type. It's an enumeration. You can also ask if is_portable, is_adroid, is_iOSunique_id: returns the unique ID of the drive. I've explained this in details above. Once again, think of it as the drive's serial number.friendly_name: a friendly name for this drive. This is given by your Portable maker, and it can be something like "Galaxy S6", "P8 Lite", etc.folders and files: returns the root folders and files in the root. For most devices, folders will return no files, and a single root folder (which has subfolders and files)parse_file and parse_folder will return a file/folder respectively, or throw if it does not exist. Their try_ counterparts will return null if the file/folder does not exist.
Here's how to find out the unique id, friendly name, and how many photos you have on your Camera, on each of the connected Android drives:
foreach ( var ad in drive_root.inst.drives.Where(d => d.type.is_android()))
Console.Write(ad.unique_id + " - " + ad.friendly_name
+ " , photos: " + ad.parse_folder("*/dcim/camera").files.Count());
The IFolder
Once you get a folder, here's what you can do with it
public interface IFolder {
string name { get; }
bool exists { get; }
string full_path { get; }
IDrive drive { get; }
IFolder parent { get; }
IEnumerable<IFile> files { get; }
IEnumerable<IFolder> child_folders { get; }
void delete_async();
void delete_sync();
}
Once again, this should look familiar. A few notes:
name: returns the folder's name, guaranteed not to throwexists: retusn true if the folder exists. Note that if you get to an IFolder, it always exists. However, if you unplug its drive, at that point, exists will return false.full_path: returns the folder's full path. For Portable devices, the drive name is always {unique_id}. So, a folder's full path can look like {2388752ea37882}:\Phone\DCIM\Cameradelete_sync and delete_async: deletes the folder synchronously, respectively, asynchronously- no
copy: I've decided not to implement copy for a full folder for now. You can always implement this by copying each file from a folder
Here's how to find out all the Albums you created on your Phone:
var dcim = drive_root.inst.parse_folder("[a0]:/*/dcim");
foreach (var f in dcim.child_folders)
Console.WriteLine(f.full_path);
The IFile
When you have a file, here's what you can do wit it:
public interface IFile {
string name { get; }
bool exists { get; }
IFolder folder { get; }
string full_path { get; }
IDrive drive { get; }
long size { get; }
DateTime last_write_time { get; }
void copy_sync(string dest_path);
void copy_async(string dest_path);
void delete_async();
void delete_sync();
}
I've kept the interface to a minimum:
name: the file's nameexists: whether the file exists. This is exactly the same as for IFolder - it will return false when its drive gets unplugged- folder: returns the folder this file belongs to
- full_path: returns the file's full path. For Portable devices, the drive name is always
{unique_id}. So, a file's full path can look like {2388752ea37882}:\Phone\DCIM\Camera\20171010_1384292.jpg size: returns the file's size, in byteslast_write_time: returns the file's Modify Date (when it was last written to).copy_sync and copy_async: copies the file to another path, synchronously or asynchronously. Very important: in case the destination file exists, it will automatically override it (or throw, if it's not possible)delete_sync and delete_async: deletes a file, synchronously or asynchronously- no
move: I've decided not to implement move for now. Each of the copy/delete operations have proven difficult to implement, so I chose to forgo it, for now. You can always implement move() as a copy() + delete()
Here's how to copy the last taken photo, to its parent folder:
var camera = drive_root.inst.parse_folder("[a0]:/phone/dcim/camera");
var last_file = camera.files.OrderBy(f => -f.last_write_time.Ticks).First();
last_file.copy_sync(camera.parent.full_path);
Don't use on your UI Thread
Before continuing, a word of caution. A lot of these functions block the current thread, sometimes for quite a while, I may say. This is the way shell32.dll is implemented, and not much we can do about that (ok, there's nothing we can do about that :) ).
Expect the following:
IDrive.folders, IDrive.files, IFolder.child_folders, IFolder.files can be very time consuming, especially for folders that contain a lot of files.IFile.size, IFile.last_write_time can be consuming, even though you can often not worry about them
The library is quite straightforward to use, so you can easily invoke actions async:
Task.Run(() => foreach(var f in drive_root.inst.parse_folder("[a0]:/*/dcim/camera").files)
f.copy_sync(somewhere));
Bulk Copy
The dreadful implementation details are that the way Windows implements the Copy (and Delete, for that matter), when using the API that I'm using internally, is "sometimes synchronously, sometimes asynchronously". You can't really rely on a Copy or Delete to be synchronous. There are clear cases when you'd want the copy to be sycnhronous. Therefore, I've implented IFile.copy_sync. When copy_sync returns, it guarantees that the file has been fully copied. This means that sometimes, we'll wait a bit longer even after the file has been copied, to be "notified" of that. Or to put it bluntly, there is no notification, I'm just constantly monitoring the file's size - which isn't always up-to-date.
Having said all the above, that's why I've implemented bulk copy. There are several optimizations I was able to do, and copying files in bulk can be about 10% faster than copying each file separately.
The API that I'm using can actually sometimes copy files in bulk (when that's possible, I'm using that), or otherwise, I copy all the files asynchronously, and then wait for them to be fully copied:
public static class bulk {
static void bulk_copy_sync(string src_folder, string dest_folder
, Action<string,int,int> copy_complete_callback = null);
static void bulk_copy_async(string src_folder, string dest_folder
, Action<string,int,int> copy_complete_callback = null);
static void bulk_copy_sync(IEnumerable<IFile> src_files, string dest_folder
, Action<string,int,int> copy_complete_callback = null);
static void bulk_copy_async(IEnumerable<IFile> src_files, string dest_folder
, Action<string,int,int> copy_complete_callback = null);
}
To copy every even file from a folder:
var i = 0;
var files = drive_root.inst.parse_folder("d:/cool_pics").files.Where(f => i++ % 2 == 0);
bulk.bulk_copy_sync(files, "[a0]:/*/dcim/cool_pics");
drive_root.inst.create_folder("[a0]:/*/dcim/cool_pics");
foreach ( var f in files)
f.copy_sync("[a0]:/*/dcim/cool_pics");
You can use callbacks, that are called after each complete file copy:
- The callback arguments are : source file name, the file index, the file count (the full number of files)
For instance, say you want to show some sort of copy progress:
var camera = drive_root.inst.try_parse_folder("[a0]:/*/dcim/camera");
var temp = new_temp_path();
Console.WriteLine("Copying to " + temp);
bulk.bulk_copy_sync(camera.files.ToList(), temp, (f, i, c) => {
Console.WriteLine(f + " to " + temp + "(" + (i+1) + " of " + c + ")");
});
Console.WriteLine("Copying to " + temp + " - complete, took "
+ (int)(DateTime.Now - start).TotalMilliseconds + " ms" );
Using the Library
Well, that's it. I hope you'll find the library easy to use, and definitely useful. Get it from Github. I've written a few examples to get you started (see console_test project).
I wrote it because I could not find something that would abstract away all the complications of dealing with external drives. And the information about the API that I've used (shell32.dll's FolderItem/Folder objects) is really scarce, so I had to do quite a bit of digging.
I can tell you that it hides quite a few really ugly Windows details about what happens under the hood. If you don't care about those details, just use the Lib and have fun.
Otherwise, join me for a few laughs, below!
shell32: the Fun stuff
The API I'm using internally is shell32.dll's Folder/FolderItem shell objects. I get access to "My Computer" object, enumerate its child folders, and then ignore those that reside on the internal HDD. And I'm left with the Portable drives.
The API That Uses Dialogs
Microsoft probably hasn't thought about the shell32.dll library very hard and it tightly coupled it with Windows Explorer (or "My Computer", on Win7+, or "This PC" on Win10). The first culprit you'll see, is that sometimes, when using the API, you'll see dialogs or progress dialogs - like, the progress dialog you usually see when copying a large folder, within Windows Explorer.
But then I said, Folder.CopyHere and Folder.MoveHere (what I use internally), they have an extra argument options, which I can use to turn off the UI. Brilliant! But unfortunately, it's completely ignored... Bummer...
So, what I'm left to do is: create a thread which constantly monitors for Dialog windows coming from our Process (thankfully, the Dialogs and Progress Dialogs are created within our process). As soon as I find one, i move it offscreen. Initially, I wanted to hide it, but the dialog blisfully ignores hiding.
Just in case you don't want that, just do drive_root.inst.auto_close_win_dialogs = false;. At this point, you'll notice the progress dialogs, when shown.
Another UI you would notice, when copying/moving a file, and the file already exists in the destination, is this baby:
In order to avoid it, my library checks if the file in the destination exists, and if so, automatically deletes it, then performs the copy.
The Awesome Delete
This also involves a dialog, but it's so funny, it deserves its own section...
For reasons I cannot begin to fathom, when using the FolderItem.Delete API, no matter what you delete, you'll be met with:
There's simply no way to turn it off. Most answers on google simply involved sending the Enter key as soon as the dialog is shown. After almost giving up on this, I found a very nice workaround: instead of FolderItem.Delete, just move the file from the Portable drive to the HDD, and then System.IO.File.Delete it there. When moving, there's no annoying dialog - case closed.
The [A]Synchronous Flexibility
When using Folder.CopyHere or Folder.MoveHere, sometimes the API can be synchronous, and sometimes asynchronous. In my tests, it seems that Portable-to-HDD is synchronous. Portable-to-Portable and HDD-to-Portable are asynchronous.
Either way, it's clearly we can't rely on them to know when an operation has completed. So, the way I monitor (on copy_sync), is by checking the destination file's size, until it matches the source file.
The FolderItems "Collection"
When doing a bulk copy operation, it would clearly help if I could tell the shell32 API to copy everything in one go.
The Folder.CopyHere's first argument can be a FolderItem (a single file or folder), or a FolderItems (which is a collection of files and/or folders).
The only way to get a FolderItems is by enumerating a Folder - therefore, it will return all its files and all its sub-folders. Sadly, you can't add or remove anything. In other words, either you Copy all the Folder's children at once, or you Copy file-by-file.
There is a somewhat solution to this... or so I thought. You can convert the FolderItems object you get from a Folder into the FolderItems3 COM interface. Which, has a Filter function, that can basically perform a filter on all the items it contains.
The filter's second argument is a find-spec, and you can use ";" to specify several files to copy. So you could say:
items.Filter(flags, "file1;file2;file3...")
This would allow us to copy a specific set of files from a folder in one go, thus, allowing for easy Bulk Copy. Sadly, this works only if the source Folder is from HDD. If the source Folder is from a Portable device, any Filter you apply to the items object will return 0 items (except "*.*", which returns all original items).
Hoping that Microsoft will sometimes fix this, my library tries to use items.Filter. If that doesn't work (items.Filter returns 0), it will do a file-by-file Copy.
Troubleshooting
Hopefully, you won't need to get here, but just in case something doesn't go right... This basically means that we don't recognize your Portable device, even though you see it in "My Computer".
Unrecognized Drive or Bad/Invalid unique ID
This basically means that we could not parse the root path of your drive. Long story short, it should look more or less like:
::{20D04FE0-3AFA-1069-A2D8-06002B30309D}\\\\\\?\\usb#vid_13d2&pid_0307#557a9de7#{6ad27878-a6fa-4155-ba85-f38f491d4f33}
Some devices may have a slightly different root path, such as:
::{20D04FE0-3AEA-1069-A2D8-08002B30309D}\\\\\\\\\\\\?\\\\activesyncwpdenumerator#umb#2&306b293b&2&aceecamez1500windowsmobile5#{6ac27878-a6fa-4155-ba85-f98f491d4f33}
If that is the case, I just need to update the code, and parse that path correctly.
Please run the following code, in console_test project:
foreach ( var p in usb_util.get_all_portable_paths())
Console.WriteLine(p);
foreach ( var p in usb_util.get_all_usb_pnp_device_ids())
Console.WriteLine(p);
foreach ( var p in usb_util.get_all_usb_dependent_ids())
Console.WriteLine(p);
and write a comment here with the results. I should be able to parse the device root path correctly, and make it work for your specific device.
Can't match a unique ID to a USB Device ID
Sometimes (for older WinCE devices, for instance), it's not straightforward to match a USB Device ID to a Unique ID. The reason we need this, is so that we know when a drive gets unplugged (thus, disconnected).
In this case, please run the following code in console_test project:
usb_util.monitor_usb_devices("Win32_USBHub");
Then, plug, wait a few seconds, and unplug the device. If anything gets written on the screen, please write a comment with the results.
If nothing gets written, please modify the above code into this:
usb_util.monitor_usb_devices("Win32_USBControllerDevice");
... and do the same. Thanks!
iPhones
iPhones only allow access to the photos you have on your device. Which is pretty much what you'd probably want anyway. Note that you won't be able to parse folders the easy Android way, where in "[a0]:/*/dcim", you have all your Albums, and the photos you've taken are in "[a0]:/*/dcim/camera".
Luckily, you can simply enumerate all folders from "[i0]:/*/dcim". They would likely be 100Apple, 101Apple, and so on, but this is not always the case.
From those folders, just enumerate all files. You can easily sort them by date, and voila - you have all the photos you took with your iPhone!
An Alternative Implementation
I was pointed recently by someone on reddit that there's another way to implement dealing with Portable Devices - namely, WPD. Well, that whole interface is COM, and in order to convert it to C#, you'd need to import 2 COM DLLs. That is not the biggest issue though. There seem to be some marshalling issues, so using this implementation is far from easy (just look at the comments). There is already an implementation of a WPD .net wrapper, and you can find it here. It does suffer from the marshalling issues described above.
Personally, at this time, I'm not using WPD - seems this would bring quite a few issues itself. I may reimplement the portable_* classes to use WPD, but so far, I do not see any decent reason to do it.
History
- 1.0, 5 Nov 2017, Initial version
- 1.1, 7 Nov 2017, all_drives -> drives, removed external_drives, small additions
- 1.1b, 8 Nov 2017, fix issue found by Dirk Bahle, added easy way to test Portable Paths and PNPDeviceIDs
- 1.2, 11 Nov 2017, added examples, per Dirk Bahle's suggestion. Added IDrive.is_available() . Added try_parse_file, try_parse_folder
- 1.2.4, 18 Nov 2017, added/tested iPhone recognition, added bulk_copy callbacks
- 1.2.5, 26 Nov 2017,
- handle shell32's progress dialog, so that the user can't cancel a file copy.
- Added "Can't match a unique ID to a USB Device ID" section
- added "Don't use on your UI Thread" section
- 1.2.5b, 30 Nov 2017 - added "An Alternate Implementation" section
