Introduction
In my previous article I posted my observations regarding one specific area which I haven't been able to find much examples for and that was
the implementation of a file system watcher on Linux using C++. Now I will try to summarize my findings regarding
my next topic - deleting to and restoring files from Trash on Linux with
a brief overview of similar Windows functionality regarding Recycle Bin.
When I started working on extending the basic functionalities provided by
the boost::filesystem namespace I was aware that it will be difficult to find sources for things like
a .NET-like portable file system watcher which I discussed in part 1 but was really surprised that areas like working with Linux Trash are totally neglected as well and with almost no usable answers or documentation regarding C++ implementations. Even on Windows, although sources to follow can be found, I haven't been able to find code samples for managing Recycle Bin with C++ which worked without problems. I am planning to dedicate
the bigger part of this article to the Linux implementation, but before that I will try to give some of my observations for
the Windows implementation I have provided in the code you can download. Again, to compile this code, look for instructions available in
the Part 1 article.
Addition: I added new project ready to compile with Visual Studio 2012, so now you have that option too. For additional instructions look at Part 1 article also.
Working with Windows Recycle Bin Using C++
I have to say that listing Recycle Bin contents and deleting files/folders to Recycle Bin isn't that hard once you get to know
a couple of not-so obvious things, specially for people who don't have much previous experience with Windows
Shell API like in my case. Without this knowledge, you can still create code samples that will work in some occasions but "mysteriously" fail in others. At least, that is what happened to me. For listing contents of Recycle Bin look for my implementation of
the FileSystemController::ListRecycleBin() function. I have seen that a lot of people had problems when compiling code like I provided there using
the MinGW compiler. To be specific, if you are using the CoTaskMemFree() API call, you'll likely get
a linker error stating something like 'undefined reference to CoTaskMemFree@4'. After further research I found that
the CoTaskMemFree() function implementation should be in ole32.dll and I tried to add it to libraries. After that, everything compiled without problems.
The second set of problems occurred while implementing deletion to Recycle Bin. My implementation looks like this:
OpResults moveToRecycleBin(const STRING_TYPE& source)
{
OpResults opres;
SHFILEOPSTRUCTW operation;
operation.wFunc = FO_DELETE;
unique_ptr<wchar_t[]> name(createDoubleNullTermString(source));
operation.pFrom = (LPCWSTR)name.get(); operation.fFlags = FOF_ALLOWUNDO | FOF_SILENT | FOF_NOCONFIRMATION;
operation.hNameMappings = NULL;
operation.lpszProgressTitle = NULL;
operation.fAnyOperationsAborted = FALSE;
operation.hwnd = NULL;
operation.pTo = NULL;
int result = SHFileOperationW(&operation);
if(result != 0)
opres.AddResult(OperationResult(false, source, TO_STRING_TYPE("Return Error value:") +
Helpers::ToString<int, STR_SET>(result)));
return opres;
}
Basically, we are providing a path to the file system item we want to be moved to Recycle Bin and return
the operation outcome inside the specialized container class I implemented (OpResults). This is more or less something common to most functions I
have provided inside the FileSystemController class which you can find inside
the Qt Creator project for download. The first thing I wasn't aware of at first is that you need to provide
an item path to SHFILEOPSTRUCTW in the form of a double null-terminated string. If you don't do that, you will get partial success with this operation, or to be more precise, sometimes it will work, sometimes it won't. To get
a double null terminated string from the standard std:: wstring, you can use
the function below:
wchar_t* createDoubleNullTermString(const wstring& ws)
{
wchar_t* buff = new wchar_t[MAX_PATH + 2];
wcscpy(buff, ws.c_str());
memcpy(buff + ws.length() + 1, L"\0\0", 2);
return buff;
} Please notice that this way we are creating a wchar_t* array on
the heap which needs to be deleted later on. If you are using a unique_ptr smart pointer from
the standard library like in my code, make sure to use "wchar_t[]" as
the provided type of pointer to indicate that a specialized array delete should be used.
the next problem was that my implementation didn't work on Windows XP. After extensive research I found while browsing the internet, a piece of code from
a totally unrelated project with a small comment pointing out that a SHFileOperation() call could crash without setting values to
SHFILEOPSTRUCTW fields like hNameMappings, lpszProgressTitle, fAnyOperationsAborted,
hwnd, and pTo. After assigning values like in the provided sample,
the XP problem was gone.
Finally, before I start discussing the Linux Trash implementation, I have to point out that there seems to be no easy way to restore
a file from Recycle Bin on Windows (at least I haven't been successful in finding one) using C++. I have tried
a couple of different approaches including the ShellExecute() API call which needed
a specific "verb" which I tried to find in the Registry but without success. This is the only functionality I wanted but failed to provide in
the FileSystemController class.
Manipulating Trash on Linux using C++
As I mentioned before, when I started looking for available information on various internet forums, I ended up with almost no usable data to continue with regarding this topic. The only usable source I found is
the freedesktop.org Trash specification. By saying usable I don't want to give you the wrong impression that it should be followed by the letter. You should test each and every feature described there to make sure that the actual behavior on
the distribution you intend to use actually is implemented as described. For example, according to freedesktop.org Trash specification,
the real home Trash location should be available over an environment variable as $XDG_DATA_HOME/Trash. This isn't the case with any major distributions I tested. I must say that I'm by no means
a Linux expert so maybe I missed something in the process but it seems that we can expect two possible Trash locations on most popular distributions.
The first is the so called home trash location which can usually be found in the /home/<user>/.local/share/Trash hidden folder and the behavior of this one is implemented mostly as described in
the previously mentioned specification. The other(s) can be found on attachable medias like USB sticks and similar, and is located on
the media's root as a .Trash-$uid hidden folder where $uid stands for current user id.
All these locations contain
two sub-folders: files and info. The first one (files subdirectory) is used for storing actual data (files/folders) that
are moved to trash. The other one (info subdirectory) has
the corresponding info files with extension .trashinfo and contain URI encoded original item path which is used for restore operation, and
a datetime value which indicates when this item was deleted. Although the corresponding files and info entries build up their names from
the original item name which is moved to Trash, the specification states that these names should not be used as
the source for a restore operation. In the Trash folder one additional file can occasionally be found. That would be
the metadata file which holds
the current Trash size. As I have found out, this file was/is a major source of
the various unexpected behaviors when using Trash (this is from personal experience
in my previous Open SUSE 12.1 OS). First of all, if you are experiencing behavior like
the Trash limit is reached when you are certain that it actually isn't, you should look for
the problem source in this place. It seems that the value stored in metadata is unsigned (which makes sense) and
the actual correction provided by the OS sometimes happens to be wrong causing it to underflow with
a negative value. This will cause a really big value to be written in the metadata file and thus
the Trash appears as if the size limit has been reached. I also tried to create
a mechanism for updating the trash size value inside the metadata file but after some time
I gave up on this since I couldn't figure out why and how the system performs some calculations for
a new size value, specially in cases when we are moving symbolic links to Trash. It doesn't seem to add neither
the symlink size nor the actual item size that it points to. This doesn't seem to be the bug, just something I haven't been able to track in
the documentation so I solved this the “easy” way - by deleting the metadata file each time some kind of change is made to
the Trash content with the
FileSystemController class. In this case the environment will rebuild
the metadata file with the recalculated value inside, which just happens to be a nice convenience allowing us to pass this task away from us.
One more thing seems to be unclear regarding trash locations
in Linux to many people – if you haven't still used Trash, chances are that the previously described actual Trash folders don't exist yet. They are usually created on first usage. If we intend to mimic Linux trash operations, we should implement the ability to create these paths if they are not present at the time we are accessing Trash.
To establish moving to and restoring from functionalities for Trash locations, I needed
a number of different helper functions that would be responsible for each part of the job:
inline UINT getUserID()
{
return getuid();
}
Getting the ID of the physical storage device:
inline UINT getDeviceID(const string& source)
{
struct stat _info;
const char* orgPath = source.c_str();
if(stat(orgPath, &_info) == 0)
{
return _info.st_dev;
}
else
return 0; }
Getting all the available mount points on the system and associating them with
the ID of the actual device they are on:
std::map<UINT, std::string> getMountPoints()
{
std::map<UINT, std::string> mount_points;
struct mntent* ent;
FILE* aFile;
aFile = setmntent("/proc/mounts", "r");
if (aFile != NULL)
{
while (NULL != (ent = getmntent(aFile)))
{
mount_points.insert(std::make_pair(getDeviceID(ent->mnt_dir), ent->mnt_dir));
}
}
endmntent(aFile);
return mount_points;
}
Functions for locating actual Trash path(s). As mentioned above, if we don't find
the Trash location where it should be, we should create it.
This is what the system would do for us in the background if we are using standard built-in ways to access Trash:
OperationResult createTrash(const std::string& trash_path)
{
OperationResult opOK = createOKOpResult(trash_path);
system::error_code ec;
bfs::create_directory(trash_path, ec);
if(ec.value() != 0)
return OperationResult(false, trash_path, ec.message());
bfs::path trashInfoPath = trash_path;
bfs::path trashfilesPath = trash_path;
trashInfoPath /= "info";
trashfilesPath /= "files";
bfs::create_directory(trashInfoPath, ec);
if(ec.value() != 0)
{
deleteItem(trash_path);
return OperationResult(false, trashInfoPath.string(), ec.message());
}
bfs::create_directory(trashfilesPath, ec);
if(ec.value() != 0)
{
deleteItem(trash_path);
return OperationResult(false, trashfilesPath.string(), ec.message());
}
return opOK;
}
inline std::string getHomeTrashPath()
{
bfs::path trash;
std::string trashPath;
if((trashPath = freedesktop_org_HomeDataPath()).empty())
{
trash = homePath();
trash /= ".local/share/Trash";
}
else
{
trash = trashPath;
trash /= "Trash";
}
if(!bfs::exists(trash))
{
OperationResult _result = createTrash(trash.string());
if(!_result.result)
trash = bfs::path(EMPTYSTR);
}
return trash.string();
}
inline std::string getTrashPath(UINT device_id = UINT_MAX, bool fallBackToHomeTrash = true)
{
if(device_id == UINT_MAX)
{
return getHomeTrashPath();
}
std::map<UINT, std::string> mount_points(getMountPoints());
std::string trashOnMount(mount_points[device_id]);
bfs::path trash;
if(!trashOnMount.empty())
{
UINT usrID = getUserID();
std::string trashName(".Trash-");
trashName += Helpers::ToString(usrID);
trash = trashOnMount;
trash /= trashName;
if(!bfs::exists(trash))
{
std::string strTrash = trash.string();
if((strTrash.find("/media/") == 0 && strTrash.length() > 7) ||
(strTrash.find("/run/media/") == 0 && strTrash.length() > 10))
{
OperationResult _result = createTrash(trash.string());
if(!_result.result)
trash = bfs::path(EMPTYSTR);
}
else if(fallBackToHomeTrash)
{
trash = getHomeTrashPath();
}
}
}
return trash.string();
}
std::vector<std::string> getAvailableTrashPaths()
{
std::map<UINT, std::string> mount_points(getMountPoints());
vector<std::string> trashPaths;
trashPaths.push_back(getHomeTrashPath());
for_each(mount_points.begin(), mount_points.end(), [&trashPaths](const std::pair<UINT, std::string> p)
{
std::string possibleTrashPath(getTrashPath(p.first, false));
if(bfs::exists(possibleTrashPath))
trashPaths.push_back(possibleTrashPath);
});
return trashPaths;
}
For listing all trash contents we need to assemble all entries which can be found in each available Trash path on the system. During the process we also need to get
the actual and restore paths for each item obtained in the previous search. To do that, reading the corresponding .trashinfo file in the info
Trash subfolder is necessary. Finally, we would need to URI decode the obtained restore path. Here is my FileSystemController::ListTrash()
function:
fInfosPtr FileSystemController::ListTrash(OpResults* resultOutcomes)
{
vector<std::string> availableTrashPaths(getAvailableTrashPaths());
fInfosPtr returnEntries = new vector<F_INFO>;
std::string homeTrash(getHomeTrashPath());
for(auto& trashStr : availableTrashPaths)
{
unique_ptr<vector<F_INFO> > entries(getEntries(getTrash_Files(trashStr), true));
unique_ptr<vector<F_INFO> > infoEntries(getEntries(getTrash_Info(trashStr), true)); for_each(entries->begin(), entries->end(), [&infoEntries, &resultOutcomes](F_INFO& entry)
{
for(auto beg=infoEntries->begin(), end = infoEntries->end(); beg != end; ++beg)
{
const F_INFO& infoEntry = *beg;
string::size_type pos = infoEntry.name.find(".trashinfo");
if(pos != string::npos)
{
string baseInfoName = infoEntry.name.substr(0, pos);
if(entry.name == baseInfoName)
{
try
{
entry.orgTrashPath = entry.path;
entry.path = getOriginalPathFromInfoTrashFile(infoEntry.path);
entry.name = boost::filesystem::path(entry.path).filename().string();
resultOutcomes->AddResult(createOKOpResult(entry.path));
}
catch(const string& errorMsg)
{
resultOutcomes->AddResult(OperationResult(false, infoEntry.path, errorMsg));
}
break;
}
}
};
});
Helpers::AddVectors(*returnEntries, *entries);
}
return returnEntries;
}Each .trashinfo file has a structure which is well described (and followed to the letter on every distribution I have tested) in
the freedesktop.org Trash specification:
[Trash Info]
Path=<URI encoded restore path>
DeletionDate=<deletion date and time>
The name of this file with the exception of the .trashinfo extension is the same as the one present in
the files sub-folder in the Trash location belonging to the actual deleted content and this is the main criteria for their matching when file restoration needs to be done. To read the content that fills
the .trashinfo file, you could use
the function:
std::string getOriginalPathFromInfoTrashFile(const string& infoFilePath) throw(std::string)
{
std::string prefix;
auto pref_pos = infoFilePath.find(".Trash-");
if(pref_pos != std::string::npos)
prefix = infoFilePath.substr(0, pref_pos);
std::string text = Helpers::textFileRead(infoFilePath);
std::string::size_type pos = text.find("Path=");
std::string::size_type posEnd = text.find("\nDeletionDate=", pos);
if(pos == std::string::npos || posEnd == std::string::npos)
throw(std::string("Incorrect info file structure in ") + infoFilePath);
pos += 5;
std::string orgPath = text.substr(pos, posEnd - pos);
return prefix + Helpers::URIDec(orgPath);
}There is also a special case concerning names of corresponding files in
the files and info sub-folders of Trash. When you have already deleted item in Trash which is named with
a name that is identical to the item we want to move to trash, the system does the check and adds
a suffix like "name 1", "name 2", "name 3"...etc. (depending on how many files with the same name you already have stored in trash; I believe that you could follow
a different naming pattern than this as long as you make sure that each file stored in Trash has
a unique name and that the names of entries in the files and info sub-folders match (with
the exception of the .trashinfo extension). I personally followed the previously described behavior which I observed
in my main development system -Open SUSE Linux. Once we want to manage Trash through our own code, we need to implement this ability to have unique names for each deleted item in Trash
(bfs in the next function is an alias to boost::filesystem):
inline std::string correctTrashFileName(bfs::path& filePathToCorrect)
{
typedef std::string::size_type POS_TYPE;
string filePath;
string parent_path = filePathToCorrect.parent_path().string();
string ext = filePathToCorrect.extension().string();
string fName = filePathToCorrect.filename().string();
fName = fName.substr(0, fName.find_last_of('.'));
POS_TYPE pos = fName.find_last_of(' ');
if(pos != string::npos)
{
string substr = fName.substr(++pos);
int num = atoi(substr.c_str());
if(num != 0)
{
++num;
substr = fName.substr(0, pos);
fName = substr + Helpers::ToString(num) + ext;
filePath = parent_path + "/" + fName;
filePathToCorrect = bfs::path(filePath);
}
else
{
if(bfs::exists(filePathToCorrect))
{
fName += " ";
fName += "1";
fName += ext;
filePath = parent_path + "/" + fName;
filePathToCorrect = bfs::path(filePath);
}
else
fName += ext;
}
}
else
{
if(bfs::exists(filePathToCorrect))
{
fName += " ";
fName += "1";
fName += ext;
filePath = parent_path + "/" + fName;
filePathToCorrect = bfs::path(filePath);
}
else
fName += ext;
}
if(bfs::exists(filePathToCorrect))
fName = correctTrashFileName(filePathToCorrect);
return fName;
}Prior to calling this function we already have to have constructed the complete path to where we intend to move
the current file/folder item (trash path) and check if it exists as such in Trash.
The previously presented function then checks which suffix to add to the file name we already have to make it unique. It means checking if there already is
a trashed file with the suffix, and if so, parsing that suffix as a number and incrementing it.
The final name for Trash would then be something like "name 1" or "name x" where x stands for the smallest increment which isn't already taken by
another trashed item.
For actual functions that provide Trash functionality you can look up
the Qt Creator project in the download link. These functions are FileSystemController::ListTrash() already presented here, and
the functions moveToTrash()
located in FSC_Helpers_Linux.hpp and FileSystemController::RestoreFile().
The function moveToTrash() will do the actual "deleting" of
the file system item to Trash:
OpResults moveToTrash(const string &source)
{
OpResults results;
bfs::path file(source);
if(!bfs::exists(file))
{
results.AddResult(OperationResult(false, source, "source path does not exist"));
return results;
}
string fName = file.filename().string();
std::string trashPath;
if(goesInHomeTrash(source))
trashPath = getHomeTrashPath();
else trashPath = getTrashPath(getDeviceID(source));
if(trashPath.empty())
{
results.AddResult(OperationResult(false, source, "Trash cannot be found!"));
return results;
}
bool isLocalTrash = trashPath.find(".Trash-") != std::string::npos ? true : false;
std::string trashFiles(getTrash_Files(trashPath));
std::string trashInfo(getTrash_Info(trashPath));
bfs::path newFilePath(trashFiles);
newFilePath /= fName;
fName = correctTrashFileName(newFilePath);
bfs::path trashInfoFilePath(trashInfo);
trashInfoFilePath /= (fName + ".trashinfo");
std::string infoName;
if(!isLocalTrash)
{
infoName = TrashRecoveryName(source);
}
else
{
infoName = LocalTrashRecoveryName(source);
}
std::string infoDate = getFileDeletionTime();
try
{
Helpers::FileWriter<> _writer(trashInfoFilePath.string());
_writer << string("[Trash Info]");
_writer << (string("Path=") + infoName);
_writer << (string("DeletionDate=") + infoDate);
}
catch(const string& str)
{
results.AddResult(OperationResult(false, trashInfoFilePath.string(), str));
return results;
}
catch(...)
{
results.AddResult(OperationResult(false, trashInfoFilePath.string(),
"Undefined error ocurred while trying to write info trash file"));
return results;
}
results = moveItem(source, newFilePath.string(), true);
if(results.hasErrors())
{
if(!bfs::exists(source))
results += moveItem(newFilePath.string(), source, true);
return results;
}
if(bfs::exists(infoName))
results += deleteItem(infoName);
if(results.hasErrors())
{
if(!bfs::exists(source))
results += moveItem(newFilePath.string(), source, true);
return results;
}
std::string metadataPath(getTrashMetadataPath(trashPath));
if(bfs::exists(metadataPath))
deleteItem(metadataPath);
return results;
}The function moveToTrash() first needs to examine if the item path for deletion exists and get
the appropriate Trash path where to move it. With the help of the previously described helper functions it should (re)build
the Trash directory if it doesn't exist in the process. The next thing it should do is construct
a .trashinfo file for the item we want to delete. This needs to be done before
the actual movement of the selected item to the files subdirectory of Trash. Otherwise when using standard file managers like KDE Dolphin, we could experience that
the Trash content isn't detected correctly. This happens mostly in cases when we need name "correction" for
a new Trash item since there is already an item with the same name there. After that, the procedure is easy to understand -
the function writes the .trashinfo file to the info subdirectory and moves
the corresponding item to the files subdirectory and deletes the metadata file if existent.
Finally,
the function FileSystemController::RestoreFile() can be used to do the actual restore from the Trash location. To do that, it needs to go through
the contents of the files and info subdirectories of the appropriate Trash location and find the matching pair of entries in both directories, get the original restore path from
the corresponding .trashinfo file, delete the .trashinfo file in
the info subdirectory of Trash, and finally restore
the Trash item by moving it from the Trash files subdirectory to its original location. Since we are making changes to
the Trash size by doing this, the function deletes the metadata Trash file which holds
the current Trash size, forcing the system to create a new one (if used) with
the recalculated value.
Conclusion
I know that this article is quite lengthy, but I felt like this area has a lot of unknowns since working with Linux Trash proved to be similar to walking in the dark. To determine almost each step, I needed to observe (or assume) what
the system is doing and to try to mimic that behavior through my code. I feel like my Trash implementation became usable to the point which makes standard managing tasks possible on major Linux distributions but it still has some flaws - not respecting
the Trash size limit for one. This is because I wasn't able to find a way to get this limit information. There are probably other flaws as well but considering
the amount of time I had for this task I don't feel like I could have done a much better job.
To summarize, if you intend to use my implementation in your code, feel free to do so but I suggest you test it first. The easiest way for this would be to download
a compiled Linux "executable" of the FSHelper terminal application and try to use it for trash management (if you use x86 Linux, you're probably out of luck and you will need to recompile
the provided Qt project). If it works OK, chances are that the underlying
FileSystemController class implementation will work as well without problems.
For instructions about code available from download links, please refer to
the Part 1 article under the section "Using the code".
