Introduction
The SprightlySoft S3 Sync application allows you to take a folder on your computer and upload it to Amazon S3. You can make additions, deletions, and changes to your local files, and the next time you run the application, it will detect these changes and apply them to S3. This program allows you to create a mirror of a local folder on S3 and always keep it up to date.
Background
Amazon Simple Storage Service (Amazon S3) is a service that allows you to store files in Amazon's cloud computing environment. When your files are in Amazon's system, you can retrieve the files from anywhere on the web. You can also use Amazon's CloudFront service in conjunction with S3 to distribute your files to millions of people. Amazon S3 is the same highly scalable, reliable, secure, fast, inexpensive infrastructure that Amazon uses to run its own global network of web sites. Best of all, Amazon S3 is free for the first 5 GB of storage.
SprightlySoft S3 Sync uses the free SprightlySoft AWS component for .NET to interact with Amazon S3. In the code, you will see an example of uploading files to S3, deleting files from S3, listing files, and getting the properties of files.
Using the code
The SprightlySoft S3 Sync program works by listing your files on S3, listing your files locally, and comparing the differences between the two lists. Here is the logic of the program:
- The code starts by getting a list of all user settings. These include your Amazon AWS credentials, the S3 bucket you are syncing to, and the local folder you are syncing from.
- The next major call is the
PopulateS3HashTable
function. Here, all the files in your Amazon S3 bucket are listed. The code calls the ListBucket
function from the SprighztlySoft AWS component. This function returns an ArrayList
of objects that represent each item in S3. Properties of each object include S3 key name, size, date modified, and ETag. These objects are added to a HashTable
so they can be used later on.
private static Boolean PopulateS3HashTable(String AWSAccessKeyId,
String AWSSecretAccessKey, Boolean UseSSL, String RequestEndpoint,
String BucketName, String UploadPrefix,
ref System.Collections.Hashtable S3HashTable)
{
Boolean RetBool;
SprightlySoftAWS.S3.ListBucket MyListBucket =
new SprightlySoftAWS.S3.ListBucket();
RetBool = MyListBucket.ListBucket(UseSSL, RequestEndpoint, BucketName,
"", UploadPrefix, AWSAccessKeyId, AWSSecretAccessKey);
if (RetBool == true)
{
foreach (SprightlySoftAWS.S3.ListBucket.BucketItemObject
MyBucketItemObject in MyListBucket.BucketItemsArrayList)
{
WriteToLog(" Item added to S3 list. KeyName=" +
MyBucketItemObject.KeyName, 2);
S3HashTable.Add(MyBucketItemObject.KeyName, MyBucketItemObject);
}
}
else
{
WriteToLog(" Error listing files on S3. ErrorDescription=" +
MyListBucket.ErrorDescription);
WriteToLog(MyListBucket.LogData, 3);
}
return RetBool;
}
- The next call is the
PopulateLocalArrayList
. This function lists all local files and folders you want to synchronize. Each file and folder is added to an ArrayList
so it can be used later on. The function has the option to only include certain files or exclude certain files. The function is recursive. That means it calls itself for each sub folder.
private static void PopulateLocalArrayList(String BaseFolder,
String CurrentFolder, Boolean IncludeSubFolders,
System.Collections.ArrayList ExcludeFolderslArrayList,
String IncludeOnlyFilesRegularExpression,
String ExcludeFilesRegularExpression,
ref System.Collections.ArrayList LocalArrayList)
{
System.IO.DirectoryInfo CurrentDirectoryInfo;
CurrentDirectoryInfo = new System.IO.DirectoryInfo(CurrentFolder);
String FolderName;
foreach (System.IO.FileInfo MyFileInfo in CurrentDirectoryInfo.GetFiles())
{
if (ExcludeFilesRegularExpression != "")
{
if (System.Text.RegularExpressions.Regex.IsMatch(
MyFileInfo.FullName, ExcludeFilesRegularExpression,
System.Text.RegularExpressions.RegexOptions.IgnoreCase) == true)
{
WriteToLog(" Local file excluded by " +
"ExcludeFilesRegularExpression. Name=" +
MyFileInfo.FullName, 2);
}
else
{
if (IncludeOnlyFilesRegularExpression != "")
{
if (System.Text.RegularExpressions.Regex.IsMatch(
MyFileInfo.FullName, IncludeOnlyFilesRegularExpression,
System.Text.RegularExpressions.RegexOptions.IgnoreCase) == true)
{
WriteToLog(" File added to local " +
"list. Name=" + MyFileInfo.FullName, 2);
LocalArrayList.Add(MyFileInfo.FullName);
}
else
{
WriteToLog(" Local file not included by " +
"IncludeOnlyFilesRegularExpression. Name=" +
MyFileInfo.FullName, 2);
}
}
else
{
WriteToLog(" File added to local list. Name=" +
MyFileInfo.FullName, 2);
LocalArrayList.Add(MyFileInfo.FullName);
}
}
}
else
{
if (IncludeOnlyFilesRegularExpression != "")
{
if (System.Text.RegularExpressions.Regex.IsMatch(
MyFileInfo.FullName, IncludeOnlyFilesRegularExpression,
System.Text.RegularExpressions.RegexOptions.IgnoreCase) == true)
{
WriteToLog(" File added to list. Name=" +
MyFileInfo.FullName, 2);
LocalArrayList.Add(MyFileInfo.FullName);
}
else
{
WriteToLog(" Local file not included by " +
"IncludeOnlyFilesRegularExpression. Name=" +
MyFileInfo.FullName, 2);
}
}
else
{
WriteToLog(" File added to local list. Name=" +
MyFileInfo.FullName, 2);
LocalArrayList.Add(MyFileInfo.FullName);
}
}
}
if (IncludeSubFolders == true)
{
foreach (System.IO.DirectoryInfo SubDirectoryInfo
in CurrentDirectoryInfo.GetDirectories())
{
if (ExcludeFolderslArrayList.Contains(
SubDirectoryInfo.FullName.ToLower()) == true)
{
WriteToLog(" Local folder excluded by " +
"ExcludeFolders. Name=" +
SubDirectoryInfo.FullName, 2);
}
else
{
FolderName = SubDirectoryInfo.FullName;
if (FolderName.EndsWith("\\") == false)
{
FolderName += "\\";
}
WriteToLog(" Folder added to local list. Name=" +
FolderName, 2);
LocalArrayList.Add(FolderName);
PopulateLocalArrayList(BaseFolder, SubDirectoryInfo.FullName,
IncludeSubFolders,
ExcludeFolderslArrayList,
IncludeOnlyFilesRegularExpression,
ExcludeFilesRegularExpression,
ref LocalArrayList);
}
}
}
else
{
WriteToLog(" Sub folders excluded by IncludeSubFolders.", 2);
}
}
- Next, the program creates a list of files that should be deleted on Amazon S3. This is done through the
PopulateDeleteS3ArrayList
function. This function goes through each item in the list of files on S3 and checks if they exist in the list of local items. If the S3 item does not exist locally, the item is added to DeleteS3ArrayList
.
private static void PopulateDeleteS3ArrayList(ref System.Collections.Hashtable
S3HashTable, ref System.Collections.ArrayList LocalArrayList,
String UploadPrefix, String S3FolderDelimiter, String SoureFolder,
ref System.Collections.ArrayList DeleteS3ArrayList)
{
String KeyName;
String LocalPath;
foreach (System.Collections.DictionaryEntry MyDictionaryEntry in S3HashTable)
{
KeyName = MyDictionaryEntry.Key.ToString();
KeyName = KeyName.Substring(UploadPrefix.Length);
LocalPath = System.IO.Path.Combine(SoureFolder,
KeyName.Replace(S3FolderDelimiter, "\\"));
if (LocalArrayList.Contains(LocalPath) == false)
{
DeleteS3ArrayList.Add(MyDictionaryEntry.Key);
}
}
}
- Next, the program finds which local files do not exist on S3. This is done through the
PopulateUploadDictionary
function. This function goes through the local ArrayList
and checks if each item exists in the S3 HashTable
. The item may exist on S3 and locally but the local file may have different content. To determine if a file is the same between S3 and locally, the program has an option to compare files by ETag. An ETag is an identifier based on the content of a file. If the file changes, the ETag changes. Amazon stores the Etag of each file you upload. When you list files on S3, the ETag for each file is returned. If you choose to compare by ETag, the program will calculate the ETag of the local file and check if it matches the ETag returned by Amazon. Any file that doesn't match is added to a Dictionary
of items that need to be uploaded to S3.
private static void PopulateUploadDictionary(ref System.Collections.Hashtable
S3HashTable, ref System.Collections.ArrayList LocalArrayList,
String UploadPrefix, String S3FolderDelimiter,
String SoureFolder, String CompareFilesBy,
ref Dictionary<string, > UploadDictionary)
{
String LocalPathAsKey;
SprightlySoftAWS.S3.CalculateHash MyCalculateHash =
new SprightlySoftAWS.S3.CalculateHash();
String LocalETag;
System.IO.FileInfo MyFileInfo;
CompareFilesBy = CompareFilesBy.ToLower();
foreach (String LocalPath in LocalArrayList)
{
LocalPathAsKey = LocalPath;
LocalPathAsKey = LocalPathAsKey.Substring(SoureFolder.Length);
LocalPathAsKey = LocalPathAsKey.Replace("\\", S3FolderDelimiter);
LocalPathAsKey = System.IO.Path.Combine(UploadPrefix, LocalPathAsKey);
if (S3HashTable.ContainsKey(LocalPathAsKey) == true)
{
if (LocalPath.EndsWith("\\") == false)
{
SprightlySoftAWS.S3.ListBucket.BucketItemObject MyBucketItemObject;
MyBucketItemObject = S3HashTable[LocalPathAsKey]
as SprightlySoftAWS.S3.ListBucket.BucketItemObject;
if (CompareFilesBy == "etag")
{
LocalETag = MyCalculateHash.CalculateETagFromFile(LocalPath);
if (LocalETag == MyBucketItemObject.ETag.Replace("\"", ""))
{
}
else
{
UploadDictionary.Add(LocalPath, LocalETag);
}
}
else if (CompareFilesBy == "size")
{
MyFileInfo = new System.IO.FileInfo(LocalPath);
if (MyFileInfo.Length == MyBucketItemObject.Size)
{
}
else
{
UploadDictionary.Add(LocalPath, "");
}
}
else
{
}
}
}
else
{
UploadDictionary.Add(LocalPath, "");
}
}
}
Now we have a list of files that need to be deleted on S3 and a list of files that need to be uploaded to S3. The program has an option to list these changes, or go ahead and apply these changes to S3.
If we are deleting files on S3, the DeleteExtraOnS3
function is called. This function goes through the DeleteS3ArrayList
and deletes each file in it. This is done by calling the MakeS3Request
function. This function uses the SprightlySoft AWS component to send the appropriate command to S3. For more information about using the AWS component, see the documentation included with the source code and read the Amazon S3 API Reference documentation from Amazon. The function has the ability to retry the command if it fails.
private static void DeleteExtraOnS3(String AWSAccessKeyId,
String AWSSecretAccessKey, Boolean UseSSL, String RequestEndpoint,
String BucketName, ref System.Collections.ArrayList DeleteS3ArrayList,
int S3ErrorRetries)
{
Boolean RetBool;
int ErrorNumber = 0;
String ErrorDescription = "";
String LogData = "";
int ResponseStatusCode = 0;
String ResponseStatusDescription = "";
Dictionary<string, > ResponseHeaders = new Dictionary<string, >();
String ResponseString = "";
int DeleteCount = 0;
foreach (String DeleteItem in DeleteS3ArrayList)
{
RetBool = MakeS3Request(AWSAccessKeyId, AWSSecretAccessKey,
UseSSL, RequestEndpoint, BucketName, DeleteItem,
"", "DELETE", null, "", S3ErrorRetries,
ref ErrorNumber, ref ErrorDescription, ref LogData, ref ResponseStatusCode,
ref ResponseStatusDescription, ref ResponseHeaders, ref ResponseString);
if (RetBool == true)
{
WriteToLog(" Delete S3 file successful. S3KeyName=" + DeleteItem);
DeleteCount += 1;
}
else
{
WriteToLog(" Delete S3 file failed. S3KeyName=" +
DeleteItem + " ErrorNumber=" + ErrorNumber +
" ErrorDescription=" + ErrorDescription +
" ResponseString=" + ResponseString);
WriteToLog(LogData, 2);
WriteToLog(" Canceling deletion of extra files on S3.");
ExitCode = 1;
break;
}
}
WriteToLog(" Number of items deleted: " + DeleteCount);
}
private static Boolean MakeS3Request(String AWSAccessKeyId, String AWSSecretAccessKey,
Boolean UseSSL, String RequestEndpoint, String BucketName,
String KeyName, String QueryString, String RequestMethod,
Dictionary<string, > ExtraHeaders, String SendData, int RetryTimes,
ref int ErrorNumber, ref String ErrorDescription, ref String LogData,
ref int ResponseStatusCode, ref String ResponseStatusDescription,
ref Dictionary<string, > ResponseHeaders, ref String ResponseString)
{
SprightlySoftAWS.REST MyREST = new SprightlySoftAWS.REST();
String RequestURL;
Dictionary<string, > ExtraRequestHeaders;
String AuthorizationValue;
Boolean RetBool = true;
LogData = "";
for (int i = 0; i == RetryTimes; i++)
{
RequestURL = MyREST.BuildS3RequestURL(UseSSL, RequestEndpoint,
BucketName, KeyName, QueryString);
ExtraRequestHeaders = new Dictionary<string, >();
if (ExtraHeaders != null)
{
foreach (KeyValuePair<string, > MyKeyValuePair in ExtraHeaders)
{
ExtraRequestHeaders.Add(MyKeyValuePair.Key,
MyKeyValuePair.Value);
}
}
ExtraRequestHeaders.Add("x-amz-date",
DateTime.UtcNow.ToString("r"));
AuthorizationValue = MyREST.GetS3AuthorizationValue(RequestURL,
RequestMethod, ExtraRequestHeaders, AWSAccessKeyId, AWSSecretAccessKey);
ExtraRequestHeaders.Add("Authorization", AuthorizationValue);
RetBool = MyREST.MakeRequest(RequestURL, RequestMethod,
ExtraRequestHeaders, SendData);
ErrorNumber = MyREST.ErrorNumber;
ErrorDescription = MyREST.ErrorDescription;
LogData += MyREST.LogData;
ResponseStatusCode = MyREST.ResponseStatusCode;
ResponseStatusDescription = MyREST.ResponseStatusDescription;
ResponseHeaders = MyREST.ResponseHeaders;
ResponseString = MyREST.ResponseString;
if (RetBool == true)
{
break;
}
else
{
if (MyREST.ResponseStatusCode == 503)
{
System.Threading.Thread.Sleep(1000 * i * i);
}
else if (MyREST.ErrorNumber == 1003)
{
System.Threading.Thread.Sleep(1000 * i * i);
}
else
{
break;
}
}
}
return RetBool;
}
Finally, the program calls the UploadMissingToS3
function to upload files from UploadDictionary
to S3. Here, the program sets header information such as Content-MD5, Content-Type, and metadata to store the local file's timestamp in S3. It then calls the UploadFileToS3
function, which is very similar to the MakeS3Request
function. The difference is, the UploadFileToS3
function has parameters that are only relevant to uploading a file. The program uses a variable called MyUpload
which is a SprightlySoft AWS component object. This object raises an event whenever the progress on an upload changes. The program hooks into this progress event and shows the progress of the upload while it is taking place. This is done in the MyUpload_ProgressChangedEvent
function.
private static void UploadMissingToS3(String AWSAccessKeyId,
String AWSSecretAccessKey, Boolean UseSSL,
String RequestEndpoint, String BucketName,
String UploadPrefix, String S3FolderDelimiter,
String SoureFolder, Dictionary<string, > UserRequestHeaders,
Dictionary<string, > UserContentTypes,
Boolean CalculateMD5ForUpload,
ref Dictionary<string, > UploadDictionary,
Boolean SaveTimestampsInMetadata, Single UploadSpeedLimitKBps,
Boolean ShowUploadProgress, int S3ErrorRetries)
{
Boolean RetBool;
int ErrorNumber = 0;
String ErrorDescription = "";
String LogData = "";
int ResponseStatusCode = 0;
String ResponseStatusDescription = "";
Dictionary<string, > ResponseHeaders = new Dictionary<string, >();
String ResponseString = "";
Dictionary<string, > ExtraHeaders = new Dictionary<string, >();
String DiffName;
String LocalMD5Hash;
String MyExtension;
String KeyName;
System.IO.FileInfo MyFileInfo;
System.IO.DirectoryInfo MyDirectoryInfo;
int UploadCount = 0;
SprightlySoftAWS.S3.CalculateHash MyCalculateHash =
new SprightlySoftAWS.S3.CalculateHash();
SprightlySoftAWS.S3.Helper MyS3Helper = new SprightlySoftAWS.S3.Helper();
Dictionary<string, > ContentTypesDictionary;
ContentTypesDictionary = MyS3Helper.GetContentTypesDictionary();
foreach (KeyValuePair<string, > UploadKeyValuePair in UploadDictionary)
{
DiffName = UploadKeyValuePair.Key.Substring(SoureFolder.Length,
UploadKeyValuePair.Key.Length - SoureFolder.Length);
DiffName = DiffName.Replace("\\", "/");
KeyName = UploadPrefix + DiffName;
if (UploadKeyValuePair.Key.EndsWith("\\") == true)
{
ExtraHeaders = new Dictionary<string, >();
if (SaveTimestampsInMetadata == true)
{
MyDirectoryInfo = new System.IO.DirectoryInfo(UploadKeyValuePair.Key);
ExtraHeaders.Add("x-amz-meta-local-date-created",
MyDirectoryInfo.CreationTime.ToFileTimeUtc().ToString());
}
RetBool = MakeS3Request(AWSAccessKeyId, AWSSecretAccessKey,
UseSSL, RequestEndpoint, BucketName, KeyName, "",
"PUT", ExtraHeaders, "", S3ErrorRetries,
ref ErrorNumber, ref ErrorDescription, ref LogData,
ref ResponseStatusCode, ref ResponseStatusDescription,
ref ResponseHeaders, ref ResponseString);
if (RetBool == true)
{
WriteToLog(" Create S3 folder successful. S3KeyName=" + KeyName);
UploadCount += 1;
}
else
{
WriteToLog(" Create S3 folder failed. S3KeyName=" +
KeyName + " ErrorNumber=" + ErrorNumber +
" ErrorDescription=" + ErrorDescription +
" ResponseString=" + ResponseString);
WriteToLog(LogData, 2);
WriteToLog(" Canceling upload of missing files to S3.");
ExitCode = 1;
break;
}
}
else
{
ExtraHeaders = new Dictionary<string, >();
if (CalculateMD5ForUpload == true)
{
if (UploadKeyValuePair.Value == "")
{
LocalMD5Hash =
MyCalculateHash.CalculateMD5FromFile(UploadKeyValuePair.Key);
}
else
{
LocalMD5Hash = MyS3Helper.ConvertETagToMD5(UploadKeyValuePair.Value);
}
ExtraHeaders.Add("Content-MD5", LocalMD5Hash);
}
MyExtension = System.IO.Path.GetExtension(UploadKeyValuePair.Key).ToLower();
if (UserContentTypes.ContainsKey(MyExtension) == true)
{
ExtraHeaders.Add("Content-Type", UserContentTypes[MyExtension]);
}
else if (ContentTypesDictionary.ContainsKey(MyExtension) == true)
{
ExtraHeaders.Add("Content-Type",
ContentTypesDictionary[MyExtension]);
}
if (SaveTimestampsInMetadata == true)
{
MyFileInfo = new System.IO.FileInfo(UploadKeyValuePair.Key);
ExtraHeaders.Add("x-amz-meta-local-date-modified",
MyFileInfo.LastWriteTimeUtc.ToFileTimeUtc().ToString());
ExtraHeaders.Add("x-amz-meta-local-date-created",
MyFileInfo.CreationTime.ToFileTimeUtc().ToString());
}
foreach (KeyValuePair<string, > MyKeyValuePair in UserRequestHeaders)
{
ExtraHeaders.Add(MyKeyValuePair.Key, MyKeyValuePair.Value);
}
WriteToLog(" Uploading file. S3KeyName=" + KeyName);
RetBool = UploadFileToS3(AWSAccessKeyId, AWSSecretAccessKey, UseSSL,
RequestEndpoint, BucketName, KeyName, "PUT",
ExtraHeaders, UploadKeyValuePair.Key, UploadSpeedLimitKBps,
S3ErrorRetries, ref ErrorNumber, ref ErrorDescription,
ref LogData, ref ResponseStatusCode,
ref ResponseStatusDescription, ref ResponseHeaders,
ref ResponseString);
if (RetBool == true)
{
WriteToLog(" Upload file to S3 was successful.");
UploadCount += 1;
}
else
{
WriteToLog(" Upload file to S3 failed. ErrorNumber=" +
ErrorNumber + " ErrorDescription=" +
ErrorDescription + " ResponseString=" +
ResponseString);
WriteToLog(LogData, 2);
WriteToLog(" Canceling upload of missing files to S3.");
ExitCode = 1;
break;
}
}
}
WriteToLog(" Number of items uploaded: " + UploadCount);
}
private static Boolean UploadFileToS3(String AWSAccessKeyId,
String AWSSecretAccessKey, Boolean UseSSL, String RequestEndpoint,
String BucketName, String KeyName, String RequestMethod,
Dictionary<string, > ExtraHeaders, String LocalFileName,
Single UploadSpeedLimitKBps, int RetryTimes, ref int ErrorNumber,
ref String ErrorDescription, ref String LogData,
ref int ResponseStatusCode, ref String ResponseStatusDescription,
ref Dictionary<string, > ResponseHeaders, ref String ResponseString)
{
String RequestURL;
Dictionary<string, > ExtraRequestHeaders;
String AuthorizationValue;
Boolean RetBool = true;
LogData = "";
if (UploadSpeedLimitKBps > 0)
{
MyUpload.LimitKBpsSpeed = UploadSpeedLimitKBps;
}
for (int i = 0; i == RetryTimes; i++)
{
RequestURL = MyUpload.BuildS3RequestURL(UseSSL, RequestEndpoint,
BucketName, KeyName, "");
ExtraRequestHeaders = new Dictionary<string, >();
if (ExtraHeaders != null)
{
foreach (KeyValuePair<string, > MyKeyValuePair in ExtraHeaders)
{
ExtraRequestHeaders.Add(MyKeyValuePair.Key,
MyKeyValuePair.Value);
}
}
ExtraRequestHeaders.Add("x-amz-date",
DateTime.UtcNow.ToString("r"));
AuthorizationValue = MyUpload.GetS3AuthorizationValue(RequestURL,
RequestMethod, ExtraRequestHeaders,
AWSAccessKeyId, AWSSecretAccessKey);
ExtraRequestHeaders.Add("Authorization", AuthorizationValue);
RetBool = MyUpload.UploadFile(RequestURL, RequestMethod,
ExtraRequestHeaders, LocalFileName);
ErrorNumber = MyUpload.ErrorNumber;
ErrorDescription = MyUpload.ErrorDescription;
LogData += MyUpload.LogData;
ResponseStatusCode = MyUpload.ResponseStatusCode;
ResponseStatusDescription = MyUpload.ResponseStatusDescription;
ResponseHeaders = MyUpload.ResponseHeaders;
ResponseString = MyUpload.ResponseString;
if (RetBool == true)
{
break;
}
else
{
if (MyUpload.ResponseStatusCode == 503)
{
System.Threading.Thread.Sleep(1000 * i * i);
}
else if (MyUpload.ErrorNumber == 1003)
{
System.Threading.Thread.Sleep(1000 * i * i);
}
else
{
break;
}
}
}
return RetBool;
}
When the program completes, it has an option to send log information through an email. This is useful if you run the program as a scheduled task and you want to be notified if there is a failure.
About SprightlySoft
SprightlySoft develops tools and techniques for Microsoft developers to more easily work with Amazon Web Services. Visit http://sprightlysoft.com/ for more information.
History