AWS: Simple Storage Service (S3) Helper Class - TransferUtility - C#

In my previous post, we learnt about Amazon S3 and upload / download / copy / delete the files(or objects) of an S3 bucket using C#. The methods given in the previous post uses a Single GET / PUT operations, so that it will upload / download upto 5 GB (Refer here) and we need to use Multipart Upload API. So for implementing the Multipart Upload API concept .NET uses TransferUtility class.

TransferUtility

TransferUtility is a high level utility for managing transfers to and from Amazon S3. It provides a simple API for uploading  and downloading content to/from Amazon S3. It uses Amazon S3 multipart upload API, so you can upload large objects, up to 5 TB. 

It uses multiple threads to upload multiple parts of a single file at once which increase throughput, when dealing with large content sizes and high bandwidth.

Configure the TransferUtility

There are three optional properties that you can configure:

ConcurrentServiceRequests 

Determines how many active threads or the number of concurrent asynchronous web requests will be used to upload/download the file. The default value is 10.

MinSizeBeforePartUpload

Gets or sets the minimum part size for upload parts in bytes. The default is 16 MB. Decreasing the minimum part size causes multipart uploads to be split into a larger number of smaller parts. Setting this value too low has a negative effect on transfer speeds, causing extra latency and network communication for each part.

NumberOfUploadThreads 

Gets or sets the number of executing threads. This property determines how many active threads will be used to upload the file. The default value is 10 threads.

Following is the C#.NET helper method for Upload / Download files using TransferUtility 

public class AmazonS3TransferHelper
{
 AmazonS3Client client;

 private static readonly ILog _logger = LogManager.GetLogger(typeof(AmazonS3TransferHelper));
 private readonly string accessKeyId, secretKey, serviceUrl;
 public AmazonS3TransferHelper(string accessKeyId, string secretKey, string serviceUrl)
 {
  this.accessKeyId = accessKeyId;
  this.secretKey = secretKey;
  this.serviceUrl = serviceUrl;
  client = GetClient();
 }

 /// <summary>
 /// Initializes and returns the AmazonS3 object
 /// </summary>
 /// <returns></returns>
 private AmazonS3Client GetClient()
 {
  if (client == null)
  {
   try
   {
    // S3 config object
    AmazonS3Config clientConfig = new AmazonS3Config
    {
     // Set the endpoint URL
     ServiceURL = serviceUrl
    };
    client = new AmazonS3Client(accessKeyId, secretKey, clientConfig);
   }
   catch (AmazonS3Exception ex)
   { _logger.Error($"Error (AmazonS3Exception) creating S3 client", ex); }
   catch (AmazonServiceException ex)
   { _logger.Error($"Error (AmazonServiceException) creating S3 client", ex); }
   catch (Exception ex)
   { _logger.Error($"Error creating AWS S3 client", ex); }
  }
  return client;
 }

 private TransferUtility GetTransferUtility()
 {
  var config = new TransferUtilityConfig()
  {
   ConcurrentServiceRequests = 10,
   MinSizeBeforePartUpload = 16 * 1024 * 1024
  };

  return new TransferUtility(GetClient(), config);
 }

 /// <summary>
 /// Uploads the file to the S3 bucket. 
 /// </summary>
 /// <param name="bucketNameWithPath">S3 bucket name along with the subfolders. Ex. If you are using a 'dev' folder under bucket 'myBucket', this value should be myBucket/dev</param>
 /// <param name="fileNameInS3">File name used to store the content in the bucket</param>
 /// <param name="fileContent">String content which needs to be stored in the file</param>
 /// <returns></returns>
 public async Task Upload(string bucketNameWithPath, string fileNameInS3, string fileContent)
 {
  _logger.Info("Entering AmazonS3TransferHelper.Upload");
  try
  {
   byte[] byteArray = Encoding.ASCII.GetBytes(fileContent);
   MemoryStream stream = new MemoryStream(byteArray);

   var tranferUtility = GetTransferUtility();
   var transferUploadRequest = new TransferUtilityUploadRequest
   {
    BucketName = bucketNameWithPath,
    Key = fileNameInS3,
    InputStream = stream
   };

   await tranferUtility.UploadAsync(transferUploadRequest); //commensing the transfer  
  }
  catch (Exception ex)
  {
   _logger.Error("Error in uploading file to s3 bucket", ex);
  }

  _logger.Info("Leaving AmazonS3TransferHelper.Upload");
 }

 /// <summary>
 /// Downloads the file from S3 bucket
 /// </summary>
 /// <param name="bucketNameWithPath">S3 bucket name along with the subfolders. Ex. If you are using a 'dev' folder under bucket 'myBucket', this value should be myBucket/dev</param>
 /// <param name="fileNameInS3">File which needs to be get from the bucket</param>
 /// <param name="localFilePath">Local folder path to store the file downloaded from S3</param>
 /// <returns></returns>
 public async Task Download(string bucketNameWithPath, string fileNameInS3, string localFilePath)
 {
  _logger.Info("Entering AmazonS3TransferHelper.Download");
  try
  {
   var tranferUtility = GetTransferUtility();
   var transferDownloadRequest = new TransferUtilityDownloadRequest
   {
    BucketName = bucketNameWithPath,
    Key = fileNameInS3,
    FilePath = localFilePath + "/" + fileNameInS3
   };

   await tranferUtility.DownloadAsync(transferDownloadRequest);
  }
  catch (Exception ex)
  {
   _logger.Error("Error in downloading file to s3 bucket", ex);
  }

  _logger.Info("Leaving AmazonS3TransferHelper.Download");
 }
}

Usage:

Create the helper class object

AmazonS3TransferHelper transferHelper = new AmazonS3TransferHelper(<accesskey>, <secret key>, <AWS S3 endpoint url>);

Upload:

Here, I am uploading S3.txt file to the dev folder in my S3 bucket named gopiBucket with name "S3Sample_1.txt".

string fileContent = File.ReadAllText("S3.txt");
await transferHelper.Upload("gopiBucket/dev", "S3Sample_1.txt", fileContent);

Download a file:

Downloading the file from dev folder to the "Downloads" folder in my solution. If the "Downloads" folder not exits, it will create new one.

await transferHelper.Download("gopiBucket/dev", "S3Sample_1.txt", "Downloads");

Happy Coding  😊!!

Read More»

AWS: Simple Storage Service (S3) Helper Class - C#

As per Amazon, Amazon Simple Storage Service (S3) is storage for the Internet. It is designed to make web-scale computing easier for developers. It has a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly salable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers.

Following is the helper class for working with the objects stored in Amazon S3 buckets using C#.NET. AWSSDK.S3 (https://www.nuget.org/packages/AWSSDK.S3/) is the NuGet package required to work with AWS S3.

public class AmazonS3Helper
{
 AmazonS3Client client;

 private static readonly ILog _logger = LogManager.GetLogger(typeof(AmazonS3Helper));
 private readonly string accessKeyId, secretKey, serviceUrl;
 public AmazonS3Helper(string accessKeyId, string secretKey, string serviceUrl)
 {
  this.accessKeyId = accessKeyId;
  this.secretKey = secretKey;
  this.serviceUrl = serviceUrl;
  client = GetClient();
 }

 /// <summary>
 /// Initializes and returns the AmazonS3 object
 /// </summary>
 /// <returns></returns>
 private AmazonS3Client GetClient()
 {
  if (client == null)
  {
   try
   {
    // S3 config object
    AmazonS3Config clientConfig = new AmazonS3Config
    {
     // Set the endpoint URL
     ServiceURL = serviceUrl
    };
    client = new AmazonS3Client(accessKeyId, secretKey, clientConfig);
   }
   catch (AmazonS3Exception ex)
   { _logger.Error($"Error (AmazonS3Exception) creating S3 client", ex); }
   catch (AmazonServiceException ex)
   { _logger.Error($"Error (AmazonServiceException) creating S3 client", ex); }
   catch (Exception ex)
   { _logger.Error($"Error creating AWS S3 client", ex); }
  }
  return client;
 }

 /// <summary>
 /// Uploads the file to the S3 bucket. 
 /// </summary>
 /// <param name="bucketNameWithPath">S3 bucket name along with the subfolders. Ex. If you are using a 'dev' folder under bucket 'myBucket', this value should be myBucket/dev</param>
 /// <param name="fileNameInS3">File name used to store the content in the bucket</param>
 /// <param name="fileContent">String content which needs to be stored in the file</param>
 /// <returns></returns>
 public async Task Upload(string bucketNameWithPath, string fileNameInS3, string fileContent)
 {
  _logger.Info("Entering AmazonS3Helper.Upload");
  try
  {
   byte[] byteArray = Encoding.ASCII.GetBytes(fileContent);
   MemoryStream stream = new MemoryStream(byteArray);

   var putRequest = new PutObjectRequest
   {
    BucketName = bucketNameWithPath,
    Key = fileNameInS3,
    InputStream = stream
   };
   await client.PutObjectAsync(putRequest);
  }
  catch (Exception ex)
  {
   _logger.Error("Error in uploading file to s3 bucket", ex);
  }

  _logger.Info("Leaving AmazonS3Helper.Upload");
 }


 /// <summary>
 /// Downloads the file from S3 bucket
 /// </summary>
 /// <param name="bucketNameWithPath">S3 bucket name along with the subfolders. Ex. If you are using a 'dev' folder under bucket 'myBucket', this value should be myBucket/dev</param>
 /// <param name="fileNameInS3">File which needs to be get from the bucket</param>
 /// <param name="downloadFolder">Local folder path to store the file downloaded from S3</param>
 /// <returns></returns>
 public async Task Download(string bucketNameWithPath, string fileNameInS3, string downloadFolder)
 {
  _logger.Info("Entering AmazonS3Helper.Download");
  try
  {
   var getRequest = new GetObjectRequest
   {
    BucketName = bucketNameWithPath,
    Key = fileNameInS3
   };
   using (GetObjectResponse response = await client.GetObjectAsync(getRequest))
   using (StreamReader reader = new StreamReader(response.ResponseStream))
   {
    var fileBody = reader.ReadToEnd(); // Now you process the response body.
    if (!Directory.Exists(downloadFolder))
     Directory.CreateDirectory(downloadFolder);
    // Append filename
    downloadFolder = downloadFolder + "/" + fileNameInS3;

    File.WriteAllText(downloadFolder, fileBody);
   }
  }
  catch (Exception ex)
  {
   _logger.Error("Error in downloading file to s3 bucket", ex);
  }

  _logger.Info("Leaving AmazonS3Helper.Download");
 }

 /// <summary>
 /// Copies a file from one bucket to another bucket
 /// </summary>
 /// <param name="sourceBucketWithPath">Source bucket name along with the subfolders. Ex. If you are using a 'dev' folder under bucket 'myBucket', this value should be myBucket/dev</param>
 /// <param name="sourceFileName">File name in source bucket</param>
 /// <param name="destinationBucketWithPath">Destination bucket name along with the subfolders. Ex. If you are using a 'dev' folder under bucket 'myBucket', this value should be myBucket/dev</param>
 /// <param name="destinationFileName">File name in destination bucket</param>
 /// <returns></returns>
 public async Task Copy(string sourceBucketWithPath, string sourceFileName, string destinationBucketWithPath, string destinationFileName)
 {
  _logger.Info("Entering AmazonS3Helper.Copy");
  try
  {
   var copyRequest = new CopyObjectRequest
   {
    SourceBucket = sourceBucketWithPath,
    SourceKey = sourceFileName,
    DestinationBucket = destinationBucketWithPath,
    DestinationKey = destinationFileName
   };
   await client.CopyObjectAsync(copyRequest);
  }
  catch (Exception ex)
  {
   _logger.Error("Error in copying file in S3", ex);
  }

  _logger.Info("Leaving AmazonS3Helper.Copy");
 }

 /// <summary>
 /// Lists all the files in the given bucket
 /// </summary>
 /// <param name="bucketName">S3 bucket name </param>
 /// <param name="path">Folder path in the bucket</param>
 /// <returns></returns>
 public async Task<List<S3Object>> ListFiles(string bucketName, string path = "")
 {
  List<S3Object> files = new List<S3Object>();
  _logger.Info("Entering AmazonS3Helper.ListFiles");
  try
  {
   var listRequest = new ListObjectsV2Request
   {
    BucketName = bucketName,
    MaxKeys = 10
   };

   if (!string.IsNullOrEmpty(path))
   {
    if (path[path.Length - 1] != '/')
     path += "/";
    listRequest.Prefix = path;
   }

   ListObjectsV2Response response;
   do
   {
    response = await client.ListObjectsV2Async(listRequest);
    files.AddRange(response.S3Objects);
    listRequest.ContinuationToken = response.NextContinuationToken;
   } while (response.IsTruncated);


  }
  catch (Exception ex)
  {
   _logger.Error("Error while listing files in S3", ex);
  }
  _logger.Info("Leaving AmazonS3Helper.ListFiles");
  return files;
 }

 /// <summary>
 /// Deletes the file from S3 bucket
 /// </summary>
 /// <param name="bucketNameWithPath">S3 bucket name along with the subfolders. Ex. If you are using a 'dev' folder under bucket 'myBucket', this value should be myBucket/dev</param>
 /// <param name="fileNameInS3">File which needs to be delete from the bucket</param>
 /// <returns></returns>
 public async Task Delete(string bucketNameWithPath, string fileNameInS3)
 {
  _logger.Info("Entering AmazonS3Helper.Delete");
  try
  {
   var deleteObjectRequest = new DeleteObjectRequest
   {
    BucketName = bucketNameWithPath,
    Key = fileNameInS3
   };

   await client.DeleteObjectAsync(deleteObjectRequest);
  }
  catch (Exception ex)
  {
   _logger.Error("Error in deleting file from s3 bucket", ex);
  }

  _logger.Info("Leaving AmazonS3Helper.Delete");
 }

 /// <summary>
 /// Deletes multiple files from S3 bucket
 /// </summary>
 /// <param name="bucketName">S3 bucket name</param>
 /// <param name="path">Folder path in the bucket</param>
 /// <returns></returns>
 public async Task<int> DeleteFolder(string bucketName, string path)
 {
  var fileDeleted = 0;
  _logger.Info("Entering AmazonS3Helper.MultipleDelete");
  try
  {
   var files = await ListFiles(bucketName, path);

   // Converting filesnames as KeyVersions List
   List<KeyVersion> keyVersions = files.Select(x => new KeyVersion() { Key = x.Key }).ToList();

   var deleteObjectsRequest = new DeleteObjectsRequest
   {
    BucketName = bucketName,
    Objects = keyVersions
   };
   DeleteObjectsResponse response = await client.DeleteObjectsAsync(deleteObjectsRequest);
   fileDeleted = response.DeletedObjects.Count;
  }
  catch (DeleteObjectsException e)
  {
   _logger.Error("Error in deleting files from s3 bucket");
   // Catch the successfully deleted files count
   fileDeleted = e.Response.DeletedObjects.Count;
   foreach (DeleteError deleteError in e.Response.DeleteErrors)
   {
    _logger.Error($"Object Key: {deleteError.Key}\t{deleteError.Code}\t{deleteError.Message}");
   }
  }
  catch (Exception ex)
  {
   _logger.Error("Error in deleting files from s3 bucket", ex);
  }
  _logger.Info("Leaving AmazonS3Helper.MultipleDelete");
  return fileDeleted;
 }

 /// <summary>
 /// Deletes multiple files from S3 bucket
 /// </summary>
 /// <param name="bucketName">S3 bucket name</param>
 /// <param name="path">Folder path in the bucket</param>
 /// <param name="fileNamesInS3">Files which needs to be delete from the bucket</param>
 /// <returns></returns>
 public async Task<int> Delete(string bucketName,  string[] fileNamesInS3, string path="")
 {
  var fileDeleted = 0;
  _logger.Info("Entering AmazonS3Helper.MultipleDelete");
  try
  {
   if (!string.IsNullOrEmpty(path))
   {
    if (path[path.Length - 1] != '/')
     path += "/";
   }
   // Converting filesnames as KeyVersions List
   List<KeyVersion> keyVersions = fileNamesInS3.Select(x => new KeyVersion() { Key = path + x }).ToList();

   var deleteObjectsRequest = new DeleteObjectsRequest
   {
    BucketName = bucketName,
    Objects = keyVersions
   };
   DeleteObjectsResponse response = await client.DeleteObjectsAsync(deleteObjectsRequest);
   fileDeleted = response.DeletedObjects.Count;
  }
  catch (DeleteObjectsException e)
  {
   _logger.Error("Error in deleting files from s3 bucket");
   // Catch the successfully deleted files count
   fileDeleted = e.Response.DeletedObjects.Count;
   foreach (DeleteError deleteError in e.Response.DeleteErrors)
   {
    _logger.Error($"Object Key: {deleteError.Key}\t{deleteError.Code}\t{deleteError.Message}");
   }
  }
  catch (Exception ex)
  {
   _logger.Error("Error in deleting files from s3 bucket", ex);
  }
  _logger.Info("Leaving AmazonS3Helper.MultipleDelete");
  return fileDeleted;
 }
}

Usage:

Create the helper class object

AmazonS3Helper amazonS3Helper = new AmazonS3Helper(<accesskey>, <secret key>, <AWS S3 endpoint url>);

Upload a file
Here, I am uploading S3.txt file to the dev folder in my S3 bucket named gopiBucket with name "S3Sample_1.txt"

string fileContent = File.ReadAllText("S3.txt");
await amazonS3Helper.Upload("gopiBucket/dev", "S3Sample_1.txt", fileContent);

Copy File:
Copy file method copies the file from one bucket to another bucket or one folder to another folder in a bucket. Here, I am copying my file from dev folder to prod folder in same bucket

await amazonS3Helper.Copy("gopiBucket/dev", "S3Sample_1.txt", "gopiBucket/prod", "S3Sample_1_prod.txt");

List files:
Here I am listing all the files in prod folder.

var files = await amazonS3Helper.ListFiles("gopiBucket", "prod");
foreach (var file in files)
{
 Console.WriteLine($"{file.Key}\t{file.Size}");
}

Download a file:
Donwloading the file from dev folder to the "Downloads" folder in my solution. If the "Downloads" folder not exits, it will create new one.

await amazonS3Helper.Download("gopiBucket/dev", "S3Sample_1.txt", "Downloads");

Delete a file:

await amazonS3Helper.Delete("gopiBucket/prod", "S3Sample_1_prod.txt");

Delete multiple files:

string[] filesToDelete = new string[] { "S3Sample_1_prod.txt", "S3Sample_2_prod.txt", "S3Sample_3_prod.txt" };
await amazonS3Helper.Delete("gopiBucket", filesToDelete, "prod");

Delete a completed folder:

await amazonS3Helper.DeleteFolder("gopiBucket", "prod/S1")

Note: The above mentioned Upload and Download operation methods uploads / downloads the files using a single GET / PUT operation. So you can upload / download objects up to 5 GB in size. If you want to store / retrieve more than 5GB, you need to use Multipart Upload API. I will give the Multipart Upload API details in my next post.

Happy Coding 😊 !!

Read More»