Update
Please note that as of 13th January 2020, the application described in the article DOES NOT WORK as Google has removed file links from the takeout JSON files.
You may want to refer to the source code if you want to develop any progress bar based application.
Introduction
Google provides a way for users to download data from Google photos through Google TakeOut. User can download a zip file containing all (all really?) photos. The problem is that Google selectively replaces a Json file for an image file inside the zip. This json file contains the link where the actual image is located.
The utility described in this article takes an extracted takeput folder as an input and downloads all the real images in the output folder.
Background
Some days ago, my wife's Google account happened to stop working. On investigating, I found out that she had used all the 15GB data. The main culprit was the image files. Her Android phone was syncing every image on the phone to Google Photos.
It left me with two options:
- Opt for additional paid storage from Google.
- Download the images and free up the Google storage to unlock the account.
I chose the second option.
I downloaded the Google photos zip through Google TakeOut under the impression that I would be done once I downloaded the zip. But on extracting the zip, what I found was, Google had zipped json files (which contains links to the actual image file) and not all image files.
To tackle this, I developed a utility which reads all these json files and downloads the corresponding images.
Using the Application
Prior to using this application, you will need to:
- Download and extract the takeout zip from Google takeout.
More information can be found here.
- Once you extract the zip file, open the demo application.
- Choose the extracted directory as an input directory.
- Choose an Output directory as where you need to download your images.
- Click on "Start Download".
Now sit back and relax. The application will download all the images to your output folder.
About the Source Code
There are only three classes in the demo application added:
DownloadManager
: Which does the actual downloading task. IDownloadProgressNotifier
: An interface to provide download progress information back to the UI. Form
class: for the UI to be shown. It implements IDownloadProgressNotifier
interface.
Download Manager
Download manager does the actual work of:
- Reading the json file from the extracted folder for image link
- Download the image file from the link in the json file
The fields of our interest in the json file are:
url
: It is a link where the actual image is located. title
: Title of the image. We will save the image locally with this name.
Fragment of the json looks like this:
{
"title": "IMG_20110711_192757772.jpg",
"description": "",
"url": "https://lh3.googleusercontent.com/-JHYN2OSYh21s/Vaxpt7adEp2I/
AAAAAAAABCu0/hIlcgO9TzwwkEJm2eQ9PcBu2rL1kPOqZWqwCLABGAYYCw/
s0-d/IMG_20110711_192757772.jpg",
"imageViews": "0",
Using Download Manager
To use Download manager, you will need to create the object of download manager by providing the following parameters to its constructor.
- Input directory
- Output directory
- Flag to tell whether to continue on error or not
- Download progress notifier handler
DownloadManager downloader = new DownloadManager(srcPath, dstPath, true, false, this);
Now call StartDownload()
. This will start the download progress. The download progress will be notified to the notifier handler which can be used to update the UI.
downloader.StartDownload();
Inside Download Manager
Download manager has only one public
function.
public void StartDownload()
StartDownload() Function
This function will be a starting point for all processing. It does the following things:
- It calls the
PreDownload
function to notify the UI that download process is about to start.
if (Notifier != null)
Notifier.PreDownload(TotalFilesCount);
- Calls
CreateDirectory
to create output root directory.
CreateDirectory(new DirectoryInfo(Path.GetDirectoryName(inputRootFolder)));
- Calls
TraverseFolderTree
to go through all files in the inputFolder and its subfolder.
TraverseFolderTree(inputRootFolder);
- Function
TraverseFolderTree
returns when it completes downloading all files. On completion, the UI is being notified by calling the notifier function OnDownloadFinish()
.
if (Notifier != null)
Notifier.OnDownloadFinish();
TraverseFolderTree Function
This function goes through the input directory tree structure. It traverses the input directory structure recursively to check for all files with the extension "*.jpg.json". We are interested in these files only.
First, we need to create outputfolder path. The directory structure of the inputfolder and outputfolder will remain the same.
Example: If user selects the following directories as an input and output directory.
- Input directory: C:\Users\test\Desktop\TakeoutProject\Input
- Output directory: C:\Users\test\Desktop\TakeoutProject\Output
If the current input processing folder is C:\Users\test\Desktop\TakeoutProject\Input\2015-07-17, then the output directory name is created by replacing the input root directory prefix with the output root directory.
The current Output
directory will be C:\Users\test\Desktop\TakeoutProject\Output\2015-07-17.
String sCurrentOutputFolder = sSeedFolder.Replace(inputRootFolder, outputRootFolder) ;
The output directory is created where the image files will be downloaded.
Directory.CreateDirectory(sCurrentOutputFolder);
Now enumerate through all the files with "*.jpg.json" extension in the currentInputDirectory
. These files are our candidates for download.
foreach (String strFile in Directory.EnumerateFiles
(sSeedFolder, JSON_JPEG_FILE_FILTER))
{
....
....
....
}
Each file path is passed to the RetrieveImage
function to read the file and download the actual image file.
RetrieveImage(strFile, sCurrentOutputFolder);
The UI is notified by calling the notifier function OnCurrentFileProgress
. In case of the error, the UI is notified by calling the notifier function OnError
.
The final loop looks like this:
foreach (String strFile in Directory.EnumerateFiles
(sSeedFolder, JSON_JPEG_FILE_FILTER))
{
try
{
if (Notifier != null)
Notifier.OnCurrentFileProgress
(strFile, ProcessedFilesCount, TotalFilesCount);
RetrieveImage(strFile, sCurrentOutputFolder);
ProcessedFilesCount++;
}
catch (Exception exc)
{
if (Notifier != null)
Notifier.OnError(strFile, exc.Message);
if (stopOnError)
{
throw exc;
}
}
}
Now traverse through all the remaining directories/subdirectories recursively.
foreach (String strDirectory in Directory.EnumerateDirectories(sSeedFolder))
{
TraverseFolderTree(strDirectory);
}
RetrieveImage Function
This function downloads the .jpeg.json file and reads it for actual image file location.
private void RetrieveImage(String sJsonFilePath, String sImageFilePathOutputFolder)
{
JsonValue fullJson = JsonValue.Parse(System.IO.File.ReadAllText(sJsonFilePath));
String Title = fullJson["title"];
String url = fullJson["url"];
using (WebClient client = new WebClient())
{
client.DownloadFile(url, sImageFilePathOutputFolder +
Path.DirectorySeparatorChar + Title);
}
}
As already stated, we are interested only in "title
" and "url
" field.
The "url
" represents the actual web location of the file. WebClient
class is used to download the image file from this URL. The output filename will be the same as that of the "title
".
using (WebClient client = new WebClient())
{
client.DownloadFile(url, sImageFilePathOutputFolder + Path.DirectorySeparatorChar + Title);
}
IDownloadProgressNotifier Interface
This interface
declares the following functions. Downloader manager uses this interface to notify the downloading status.
Interface Functions
OnCurrentFileProgress(String sCurrentFileName, int processedFilesCount, int totalFileCount)
This function notifies the handler about the current file being processed along with the total progress.
PreDownload(int TotalFilesCount)
This function will be called just before start of the download.
OnDownloadFinish();
This function will be called after download finishes.
OnError(String sFileName, String sErrorInformation);
This function will be called in case any error occurs. It passes the name of the file for which error occurred as well as the error information.
Form1 Class
This class is a simple C# form class which owns the UI. Along with the simple UI elements shown in the picture, it implements the IDownloadProgressNotifier
interface. The reference to this class is sent to the Download Manager as a handler. The implementation of the IDownloadProgressNotifier
interface functions is used to update the UI with progress status.
IDownloadProgressNotifier Interface Implementation
OnCurrentFileProgress(String sCurrentFileName, int processedFilesCount, int totalFileCount)
This function notifies the handler about the current file being processed along with the total progress. This callback function increments the progressbar
by one. This makes progressbar
to move forward by one unit.
public void OnCurrentFileProgress
(string sCurrentFileName, int processedFilesCount, int totalFileCount)
{
pbTotalProgress.Increment(1);
}
PreDownload(int TotalFilesCount)
This callback function is used to initialize the progress bar on the UI.
public void PreDownload(int TotalFilesCount)
{
pbTotalProgress.Minimum = 0;
pbTotalProgress.Maximum = TotalFilesCount;
pbTotalProgress.Step = 1;
}
This function will be called just before start of the download.
OnDownloadFinish();
This function will be called after download finishes. This function is used to show the process completion message box.
public void OnDownloadFinish()
{
pbTotalProgress.Value = pbTotalProgress.Maximum;
MessageBox.Show("Process finished.", "Success");
}
OnError(String sFileName, String sErrorInformation);
This callback function is used to show the error occurred.
public void OnError(string sFileName, string sErrorInformation)
{
MessageBox.Show("Process interrupted with an error
while processing file \n File Name :" +
sFileName +"\n ErrorInformation" + sErrorInformation, "Error" );
}
History
- First version(1)
- Second version: Added code walk through