Introduction
The Smart Partition File Exchange is a Windows utility that automatizes process of swapping the data between two different partitions in the file system. It is particularly useful for automatization of file exchange process between partitions that are very close to full in capacity – in fact, this was the main goal of this project in the first place.
Normally, in situations where the free space on the partitions is enough to just copy all the files from partition A to partition B and then vice versa – this would require two steps and is done manually quite easily. But given a more frequent scenario when there is not enough space on the partitions to transfer all the files in one go, you would have to transfer the files in smaller batches, A --> B then B --> A then again A --> B and so on. The next figure explains the concept:
This utility will automatically calculate all the necessary steps and do all of them in one click.
Using the Code
Usage
The user first selects the paths between which he/she wants to make the exchange of file/folder structure. These can be either written inside the textboxes on the top side, or they can be chosen by using the Browse buttons.
As soon as a path is selected / entered, its contents will be shown in the Webbrowser control under the textbox, and the labels will show the partition's current usage info. NOTE: The size will, in some cases, take a bit longer to calculate (in case of large number of files). In those cases, Size label will say "(… calculating …)" and the transfer start will be disabled until the calculation is finished.
After both the paths have been chosen and their sizes calculated, the user can press the "START TRANSFER" button. This will trigger the calculation and subsequently execution of steps.
NOTE: There is also a "Simulation" checkbox, which is currently checked as default. When this checkbox is checked, there will be no actual transfer of folders and files, only the steps will be calculated and simulated. With "Transfer speed" textbox, you can enter the desired speed of simulated transfer (if this is left empty or zero, maximum speed will be used).
After the transfer starts, an additional button will appear above the "START TRANSFER" button – the "Cancel" button. By clicking on this button, you get a choice to revert all the changes, or just cancel the transfer completely and leave the already transferred files in their new location.
Algorithm
There are several stages of execution after the Transfer process is started.
- Preparation – Here, the files for both locations are enumerated and their size is calculated.
- * Collect steps – create folders – The first thing that needs to be done is folder creation, because the directory structure needs to exist before the files are moved, otherwise, the move will cause an exception if the destination folder does not exist. In this part of the code, the subfolders are enumerated from both paths and the base path strings in them are switched.
i.e.,
(If path1 = X:\ and path2 = Y:\)
X:\dir\subdir => [CREATE] Y:\dir\subdir
Y:\another_dir\another_subdir => [CREATE] X:\another_dir\another_subdir
- * Collect steps – move files – After the proper directory structure has been created, the files can be moved. In this step, the files are enumerated for both paths and in their path strings, those base paths are switched. Method
CalculateIterations is called, and it attempts to calculate the minimum number of steps to transfer all the files.
i.e.,
(If path1 = X:\ and path2 = Y:\)
X:\dir\subdir\file1.txt => [MOVE TO] Y:\dir\subdir\file1.txt
Y:\another_dir\another_subdir\file2.txt =>
[MOVE TO] X:\another_dir\another_subdir\file2.txt
- * Collect steps – Rename files – At this point, there is a possibility that some of the destination filenames match the names of the files that already exist in the destination path. For that purpose, for each of those files, a new temporary destination name is created, and an additional step is also created for that temporary filename to be renamed back to the original filename.
i.e.,
(If path1 = X:\ and path2 = Y:\)
X:\file1.txt => [MOVE TO] Y:\file1.txt !!! Y:\file1.txt ALREADY EXISTS !!!
X:\file1.txt => [MOVE TO] Y:\file1.txt_somespecialidentifier.tmp
Additional step (in the end):
Y:\ file1.txt_somespecialidentifier.tmp => [RENAME TO] Y:\file1.txt
- * Collect steps – Delete folders – After the files have been moved, it's time to clean up! In this stage, the source folders are enumerated on both paths, and they are marked to be deleted after all the files are moved (because these folders will at this point be empty).
- Write out the steps into an XML (for debugging)
- * Execution – The steps are executed in the given order; they are looped through a
for loop and the loop also handles any errors that may happen during the execution + if the cancel button should be pressed, the user will be asked if the revert back is necessary, and if so, the loop will be reversed and the steps will be executed in reverse, from the break point downward all the way to the first step.
The stages marked with * will be explained more thoroughly in further text.
Step Class
The most important data structure used is the Step class.
The Step object has the following attributes:
public PROGRESS_STAGE ProgressStage;
public string FolderName;
public string SourceFile;
public string DestinationFile;
public long FileSize;
public int Iteration;
public bool StepDone = false;
private bool folderExists = false;
ProgressStage attribute marks the „type“ of the Step object. PROGRESS_STAGE is an enum with the following members:
enum PROGRESS_STAGE
{ PREP_CALC, PREP_CALC_DONE, CREATE_FOLDERS, MOVE_FILES, RENAME_FILES, DELETE_FOLDERS };
FolderName is the folder path – used in CREATE_FOLDERS and DELETE_FOLDERS steps.
SourceFile and DestinationFile are file paths used in MOVE_FILES and RENAME_FILES steps.
FileSize is the size of the file being transferred (used also in MOVE_FILES and RENAME_FILES steps).
Iteration marks the iteration of the step. Only MOVE_FILES steps will have an iteration assigned.
StepDone is marked true once the step gets executed. The revert execution also resets this attribute.
Finally, folderExists is a private attribute that will control whether a directory will be created or not (if, in fact, it already exists on target).
Step class also defines the following methods:
public void DoStep();
public void RevertStep();
public string PrintStepXml(ref FileStream stream);
Methods DoStep() and RevertStep() define how a step is executed depending on its ProgressStage attribute, in normal or reverse direction. For example, DoStep() will move SourceFile to DestinationFile, while RevertStep() will move DestinationFile to SourceFile. DoStep() will set the StepDone to true, while the RevertStep() will set StepDone to false.
The method PrintStepXml(ref FileStream stream) returns the Step object's attributes in XML format string, and writes it to the provided FileStream. This is used for debugging purposes.
The steps shall be collected in dictionaries declared such as this:
private Dictionary<int, Step> _steps;
(The _steps Dictionary is a global variable and the main dictionary where all the steps will be gathered and executed from. There are several other temporary Dictionary objects like this one used during the calculation process.)
File/Folder and Step Enumerations – Using LINQ to Objects
The Step objects are being created during several calculation stages (already described in the chapter Algorithm). For most of the Step collections, LINQ to Objects is used to get the enumerations.
CREATE_FOLDERS steps
_steps = _steps.Concat(
DirectoryAlternative.EnumerateDirectories(_path1, "*",
SearchOption.AllDirectories).Select(x => x.Replace
(_drive1, _drive2))
.Concat(DirectoryAlternative.EnumerateDirectories
(_path2, "*", SearchOption.AllDirectories).Select
(x => x.Replace(_drive2, _drive1)))
.Where(x => !x.Contains(RECYCLE_BIN) &&
!x.Contains(SYSTEM_VOLUME_INFORMATION))
.OrderBy(x => x)
.ToDictionary(k => step_nr++, v => new Step
(PROGRESS_STAGE.CREATE_FOLDERS, v))
).ToDictionary(k => k.Key, v => v.Value);
First, all the directories are enumerated from both paths, but their base path is switched; path1 to path2 and vice versa. Then, the system folders are excluded. The folders are sorted in ascending order to make sure that the parent folders are created before their children (i.e., X:\folder before X:\folder\subfolder). Finally, Step objects are created and the enumeration is cast into a Dictionary.
MOVE_FILES Steps
The MOVE_FILES calculation is a bit more complicated, as it attempts to simulate file transfer between the two paths, and tries to do the job in as few iterations as possible. This is done in a method called CalculateIterations.
private Dictionary<int, Step> CalculateIterations(TRANSFER_DIRECTION direction);
The method takes TRANSFER_DIRECTION as a parameter, which is an enum with 2 choices – LEFT2RIGHT and RIGHT2LEFT – marking if the transfer process imitation should begin from path1 (left) to path2 (right) or the other way around. The method will be called twice in separate threads, one thread for each option. For this purpose, Task objects were used, that are run asynchronously, to check both options simultaneously; the caller method is paused until the first (any) of the Task objects is done.
Task[] tasks = new Task[2];
Dictionary<int, Step> _steps1 = new Dictionary<int, Step>();
Dictionary<int, Step> _steps2 = new Dictionary<int, Step>();
Dictionary<int, Step> _steps_move = new Dictionary<int, Step>();
tasks[0] = Task.Factory.StartNew(() => _steps1 =
CalculateIterations(TRANSFER_DIRECTION.LEFT2RIGHT));
tasks[1] = Task.Factory.StartNew(() => _steps2 =
CalculateIterations(TRANSFER_DIRECTION.RIGHT2LEFT));
Task.Factory.ContinueWhenAny(tasks, x =>
{
…
}
RENAME_FILES Steps
These steps are calculated in a for loop that runs through all the MOVE_FILES steps, and checks if the destination file exists. If it does, it adds a unique suffix to the destination filename, and creates an additional step to be done after the MOVE_FILES steps are finished – to replace the temporary filename with the original one.
string suffix;
int step_nr_max = _steps_move.Max(x => x.Key);
int step_nr_min = _steps_move.Min(x => x.Key);
int i = 1;
Dictionary<int, Step> _steps_rename_back = new Dictionary<int, Step>();
for (step_nr = step_nr_min; _steps_move.ContainsKey(step_nr) &&
step_nr <= step_nr_max; step_nr++)
{
if (File.Exists(_steps_move[step_nr].DestinationFile))
{
suffix = "_" + DateTime.Now.ToString("yyyyMMddHHmmssfffffff") + ".tmp";
_steps_rename_back.Add(
i++,
new Step(PROGRESS_STAGE.RENAME_FILES, null,
_steps_move[step_nr].DestinationFile + suffix, _steps_move[step_nr].DestinationFile)
);
_steps_move[step_nr].DestinationFile += suffix;
Thread.Sleep(1);
}
}
DELETE_FOLDERS Steps
_steps = _steps.Concat(
DirectoryAlternative.EnumerateDirectories
(_path1, "*", SearchOption.AllDirectories)
.Concat(DirectoryAlternative.EnumerateDirectories
(_path2, "*", SearchOption.AllDirectories))
.Except(_steps.Values.Where(x => x.ProgressStage ==
PROGRESS_STAGE.CREATE_FOLDERS).Select(x => x.FolderName))
.Where(x => !x.Contains(RECYCLE_BIN) &&
!x.Contains(SYSTEM_VOLUME_INFORMATION))
.OrderByDescending(x => x)
.ToDictionary(k => step_nr++,
v => new Step(PROGRESS_STAGE.DELETE_FOLDERS, v))
).ToDictionary(k => k.Key, v => v.Value);
First, all the folders from path1 and path2 are enumerated. Then the folders that are to be created are removed (i.e., if there is the same folder name on both path1 and path2 – then this folder should be kept in both locations). Then the system folders are excluded. The descending order is used so that the children get to be deleted before their parents. The Step Dictionary is finally created.
Browsers
For the purpose of visually showing the contents of the source/destination filesystem locations, and the transfer progress itself, two WebBrowser controls are used. They are being constantly refreshed during the Transfer by a BackgroundWorker object called backgroundWorkerRefresh, along with the size and free space labels above them.
There are two buttons above each WebBrowser:
- Reset the WebBrowser to show the original path
- Browse up
Backgroundworkers
There are four BackgroundWorker objects used in total in this project. Three of them are used for various refresh operations, while the main one, backgroundWorkerFileTransfer is handling the actual work. All of them together ensure that during the operation, the main thread does not, at any point, get unresponsive.
The initialization of the BackgroundWorker objects is done in the default constructor of the form:
this.backgroundWorkerCalculate1.DoWork += BackgroundWorkerCalculate_DoWork;
this.backgroundWorkerCalculate1.RunWorkerCompleted +=
BackgroundWorkerCalculate_RunWorkerCompleted;
this.backgroundWorkerCalculate2.DoWork += BackgroundWorkerCalculate_DoWork;
this.backgroundWorkerCalculate2.RunWorkerCompleted +=
BackgroundWorkerCalculate_RunWorkerCompleted;
this.backgroundWorkerFileTransfer.DoWork += BackgroundWorkerFileTransfer_DoWork;
this.backgroundWorkerFileTransfer.ProgressChanged +=
BackgroundWorkerFileTransfer_ProgressChanged;
this.backgroundWorkerFileTransfer.RunWorkerCompleted +=
BackgroundWorkerFileTransfer_RunWorkerCompleted;
this.backgroundWorkerRefresh.DoWork += BackgroundWorkerRefresh_DoWork;
this.backgroundWorkerRefresh.ProgressChanged += BackgroundWorkerRefresh_ProgressChanged;
this.backgroundWorkerRefresh.RunWorkerCompleted += BackgroundWorkerRefresh_RunWorkerCompleted;
BackgroundWorkerCalculate1 and BackgroundWorkerCalculate2
These two BW objects share the same methods for DoWork and RunWorkerCompleted event handling, because they basically do the same thing, except the first is used for calculations on path1, and the latter is used for calculations on path2.
They are both run each time the method RefreshSizeFreeSpace is called, which is every time any of the locations changes (or technically, any time when one of the path textboxes loses focus). They calculate used and free space, and enumerate files, each for their respective path.
BackgroundWorkerRefresh
This BW object refreshes periodically the size and freespace labels, and also refreshes the WebBrowser objects during the File Transfer operation. It is also responsible for the calculation of the time remaining. The time remaining is calculated as:
TIME_REMAINING = SIZE_OF_REMAINING_FILES / (SIZE_OF_TRANSFERRED_FILES / TIME_ELAPSED)
BackgroundWorkerRefresh is started simultaneously with the backgroundWorkerFileTransfer, and is cancelled (stopped) when the file transfer is finished.
BackgroundWorkerFileTransfer
The backgroundWorkerFileTransfer is the central object of the whole project. It does all the important work.
It is triggered by buttonStart.Click event and it performs all the steps of the algorithm described earlier.
During the Preparation phase, it calls the method CalculateFilesAndSize for each path, which enumerates files and calculates disk usage and free space. If this method fails, an error is reported.
In the next few phases, the BW gathers the necessary steps for the proper execution of the file transfer. It uses LINQ to Object methods to populate the _steps Dictionary, all of which were described earlier.
After all the steps are enumerated, the method will use the WriteStepXml method of Step objects to write out all the steps into an XML file – for debugging purposes.
After this comes the execution part.
The steps execution is done in a for loop which loops through all the Step objects in the _steps Dictionary one by one, in the designated order. If the _revertback = true, it will execute the RevertStep() method of the Step object, otherwise it will execute the DoStep() method. If an exception occurs, it will offer the user 3 choices – abort/retry/ignore – and depending on the DialogResult, it will either retry the last step, ignore and continue with the next step or, in case of abort, ask the user if it should revert back all the steps – and if yes, it will set the _revertback = true, and revert the counter so the for loop now counts backwards – and all the steps that were already done shall be reverted by calling the RevertStep() method.
This BW also refreshes the size and freespace global variables for size and freespace for both paths, refreshes labelIterations, and writes out UI and log messages using the method GetMessage.
Pain Points
Enumerating Files
Normally, one would use System.IO.Directory .NET library methods to enumerate system entries on Windows system. However, there is a flaw in the standard Enumerate methods in .NET, which causes the methods to throw an Exception whenever they hit a system file or folder; for example, if you are enumerating files and/or folders on a partition root (i.e., D:\), then the Enumerate method will eventually run into $RECYCLEBIN folder, and will break, returning only a partial enumeration of filesystem entries.
To overcome this issue, I have created an alternative library to System.IO.Directory, named System.IO.DirectoryAlternative, which uses the same WinAPI functions as the original library, but without the described flaw, and it also runs faster then the .NET version. You can find the detailed description in the following article:
Single-Threading vs. Multi-Threading
In single-threaded operation, GUI typically becomes unresponsive while processor-demanding operations are performed. This is why several Backgroundworker objects are used in this project, which work each in a separate thread, to maintain a stable and responsive GUI. These BW objects are described in the previous chapters.
Difference in Calculation of Free Space
Since the actual free space of the volume to which the files are being moved may differ slightly than the calculated free space in the app, a buffer of 10 MB (can also be set differently) is used during the calculation process (i.e., there is always at least 10 MB left of free space on each volume).
const long BUFFER_SIZE = 10485760;
Destination Folder Already Exists
If there is a folder with the same name on both source and destination path, then it doesn't need to be created, but we also need to make sure that it doesn't get deleted if we are doing a revert back. Therefore, a private attribute folderExists in the Step class was introduced; for a folder that already exists on the destination, a CREATE_FOLDER step will still be created, but the attribute folderExists shall be set to true and so the folder will not actually be created.
App Freezing Because of Too Many Progress Changes
If there is a very large number of files to be moved, then for each file, the progress would be reported by the backgroundWorkerFileTransfer. This, however, caused the app to become unresponsive, and the message queue was congested and no messages were coming through to the main thread until the entire process was over.
This was solved by moving the label message setting commands to the DoWork event handler by using cross-threading. This ensured that the messages were synchronous to the operations, and no messages were sent to the queue.
this.Invoke(new Action(() => labelIterations.Text = _steps[step_nr].Iteration.ToString() +
" out of " + _num_iterations.ToString()));
Only the progressbar was left to be updated by the ProgressChanged method, and it was made only in discreet time slots because, anyhow, in most cases, anything below 1% change is unobservable in the ProgressBar.
if ((_revertback ? _revert_timer : _timer).ElapsedMilliseconds - tick >= PROGRESS_RATE)
{
backgroundWorkerFileTransfer.ReportProgress(progress, msg.Split('\n')[0]);
tick = (_revertback ? _revert_timer : _timer).ElapsedMilliseconds;
}
History
- 22nd December, 2019: Initial version
