AWS: Simple Storage Service (S3) Helper Class - C#

As per Amazon, Amazon Simple Storage Service (S3) is storage for the Internet. It is designed to make web-scale computing easier for developers. It has a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly salable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers.

Following is the helper class for working with the objects stored in Amazon S3 buckets using C#.NET. AWSSDK.S3 (https://www.nuget.org/packages/AWSSDK.S3/) is the NuGet package required to work with AWS S3.

public class AmazonS3Helper
{
 AmazonS3Client client;

 private static readonly ILog _logger = LogManager.GetLogger(typeof(AmazonS3Helper));
 private readonly string accessKeyId, secretKey, serviceUrl;
 public AmazonS3Helper(string accessKeyId, string secretKey, string serviceUrl)
 {
  this.accessKeyId = accessKeyId;
  this.secretKey = secretKey;
  this.serviceUrl = serviceUrl;
  client = GetClient();
 }

 /// <summary>
 /// Initializes and returns the AmazonS3 object
 /// </summary>
 /// <returns></returns>
 private AmazonS3Client GetClient()
 {
  if (client == null)
  {
   try
   {
    // S3 config object
    AmazonS3Config clientConfig = new AmazonS3Config
    {
     // Set the endpoint URL
     ServiceURL = serviceUrl
    };
    client = new AmazonS3Client(accessKeyId, secretKey, clientConfig);
   }
   catch (AmazonS3Exception ex)
   { _logger.Error($"Error (AmazonS3Exception) creating S3 client", ex); }
   catch (AmazonServiceException ex)
   { _logger.Error($"Error (AmazonServiceException) creating S3 client", ex); }
   catch (Exception ex)
   { _logger.Error($"Error creating AWS S3 client", ex); }
  }
  return client;
 }

 /// <summary>
 /// Uploads the file to the S3 bucket. 
 /// </summary>
 /// <param name="bucketNameWithPath">S3 bucket name along with the subfolders. Ex. If you are using a 'dev' folder under bucket 'myBucket', this value should be myBucket/dev</param>
 /// <param name="fileNameInS3">File name used to store the content in the bucket</param>
 /// <param name="fileContent">String content which needs to be stored in the file</param>
 /// <returns></returns>
 public async Task Upload(string bucketNameWithPath, string fileNameInS3, string fileContent)
 {
  _logger.Info("Entering AmazonS3Helper.Upload");
  try
  {
   byte[] byteArray = Encoding.ASCII.GetBytes(fileContent);
   MemoryStream stream = new MemoryStream(byteArray);

   var putRequest = new PutObjectRequest
   {
    BucketName = bucketNameWithPath,
    Key = fileNameInS3,
    InputStream = stream
   };
   await client.PutObjectAsync(putRequest);
  }
  catch (Exception ex)
  {
   _logger.Error("Error in uploading file to s3 bucket", ex);
  }

  _logger.Info("Leaving AmazonS3Helper.Upload");
 }


 /// <summary>
 /// Downloads the file from S3 bucket
 /// </summary>
 /// <param name="bucketNameWithPath">S3 bucket name along with the subfolders. Ex. If you are using a 'dev' folder under bucket 'myBucket', this value should be myBucket/dev</param>
 /// <param name="fileNameInS3">File which needs to be get from the bucket</param>
 /// <param name="downloadFolder">Local folder path to store the file downloaded from S3</param>
 /// <returns></returns>
 public async Task Download(string bucketNameWithPath, string fileNameInS3, string downloadFolder)
 {
  _logger.Info("Entering AmazonS3Helper.Download");
  try
  {
   var getRequest = new GetObjectRequest
   {
    BucketName = bucketNameWithPath,
    Key = fileNameInS3
   };
   using (GetObjectResponse response = await client.GetObjectAsync(getRequest))
   using (StreamReader reader = new StreamReader(response.ResponseStream))
   {
    var fileBody = reader.ReadToEnd(); // Now you process the response body.
    if (!Directory.Exists(downloadFolder))
     Directory.CreateDirectory(downloadFolder);
    // Append filename
    downloadFolder = downloadFolder + "/" + fileNameInS3;

    File.WriteAllText(downloadFolder, fileBody);
   }
  }
  catch (Exception ex)
  {
   _logger.Error("Error in downloading file to s3 bucket", ex);
  }

  _logger.Info("Leaving AmazonS3Helper.Download");
 }

 /// <summary>
 /// Copies a file from one bucket to another bucket
 /// </summary>
 /// <param name="sourceBucketWithPath">Source bucket name along with the subfolders. Ex. If you are using a 'dev' folder under bucket 'myBucket', this value should be myBucket/dev</param>
 /// <param name="sourceFileName">File name in source bucket</param>
 /// <param name="destinationBucketWithPath">Destination bucket name along with the subfolders. Ex. If you are using a 'dev' folder under bucket 'myBucket', this value should be myBucket/dev</param>
 /// <param name="destinationFileName">File name in destination bucket</param>
 /// <returns></returns>
 public async Task Copy(string sourceBucketWithPath, string sourceFileName, string destinationBucketWithPath, string destinationFileName)
 {
  _logger.Info("Entering AmazonS3Helper.Copy");
  try
  {
   var copyRequest = new CopyObjectRequest
   {
    SourceBucket = sourceBucketWithPath,
    SourceKey = sourceFileName,
    DestinationBucket = destinationBucketWithPath,
    DestinationKey = destinationFileName
   };
   await client.CopyObjectAsync(copyRequest);
  }
  catch (Exception ex)
  {
   _logger.Error("Error in copying file in S3", ex);
  }

  _logger.Info("Leaving AmazonS3Helper.Copy");
 }

 /// <summary>
 /// Lists all the files in the given bucket
 /// </summary>
 /// <param name="bucketName">S3 bucket name </param>
 /// <param name="path">Folder path in the bucket</param>
 /// <returns></returns>
 public async Task<List<S3Object>> ListFiles(string bucketName, string path = "")
 {
  List<S3Object> files = new List<S3Object>();
  _logger.Info("Entering AmazonS3Helper.ListFiles");
  try
  {
   var listRequest = new ListObjectsV2Request
   {
    BucketName = bucketName,
    MaxKeys = 10
   };

   if (!string.IsNullOrEmpty(path))
   {
    if (path[path.Length - 1] != '/')
     path += "/";
    listRequest.Prefix = path;
   }

   ListObjectsV2Response response;
   do
   {
    response = await client.ListObjectsV2Async(listRequest);
    files.AddRange(response.S3Objects);
    listRequest.ContinuationToken = response.NextContinuationToken;
   } while (response.IsTruncated);


  }
  catch (Exception ex)
  {
   _logger.Error("Error while listing files in S3", ex);
  }
  _logger.Info("Leaving AmazonS3Helper.ListFiles");
  return files;
 }

 /// <summary>
 /// Deletes the file from S3 bucket
 /// </summary>
 /// <param name="bucketNameWithPath">S3 bucket name along with the subfolders. Ex. If you are using a 'dev' folder under bucket 'myBucket', this value should be myBucket/dev</param>
 /// <param name="fileNameInS3">File which needs to be delete from the bucket</param>
 /// <returns></returns>
 public async Task Delete(string bucketNameWithPath, string fileNameInS3)
 {
  _logger.Info("Entering AmazonS3Helper.Delete");
  try
  {
   var deleteObjectRequest = new DeleteObjectRequest
   {
    BucketName = bucketNameWithPath,
    Key = fileNameInS3
   };

   await client.DeleteObjectAsync(deleteObjectRequest);
  }
  catch (Exception ex)
  {
   _logger.Error("Error in deleting file from s3 bucket", ex);
  }

  _logger.Info("Leaving AmazonS3Helper.Delete");
 }

 /// <summary>
 /// Deletes multiple files from S3 bucket
 /// </summary>
 /// <param name="bucketName">S3 bucket name</param>
 /// <param name="path">Folder path in the bucket</param>
 /// <returns></returns>
 public async Task<int> DeleteFolder(string bucketName, string path)
 {
  var fileDeleted = 0;
  _logger.Info("Entering AmazonS3Helper.MultipleDelete");
  try
  {
   var files = await ListFiles(bucketName, path);

   // Converting filesnames as KeyVersions List
   List<KeyVersion> keyVersions = files.Select(x => new KeyVersion() { Key = x.Key }).ToList();

   var deleteObjectsRequest = new DeleteObjectsRequest
   {
    BucketName = bucketName,
    Objects = keyVersions
   };
   DeleteObjectsResponse response = await client.DeleteObjectsAsync(deleteObjectsRequest);
   fileDeleted = response.DeletedObjects.Count;
  }
  catch (DeleteObjectsException e)
  {
   _logger.Error("Error in deleting files from s3 bucket");
   // Catch the successfully deleted files count
   fileDeleted = e.Response.DeletedObjects.Count;
   foreach (DeleteError deleteError in e.Response.DeleteErrors)
   {
    _logger.Error($"Object Key: {deleteError.Key}\t{deleteError.Code}\t{deleteError.Message}");
   }
  }
  catch (Exception ex)
  {
   _logger.Error("Error in deleting files from s3 bucket", ex);
  }
  _logger.Info("Leaving AmazonS3Helper.MultipleDelete");
  return fileDeleted;
 }

 /// <summary>
 /// Deletes multiple files from S3 bucket
 /// </summary>
 /// <param name="bucketName">S3 bucket name</param>
 /// <param name="path">Folder path in the bucket</param>
 /// <param name="fileNamesInS3">Files which needs to be delete from the bucket</param>
 /// <returns></returns>
 public async Task<int> Delete(string bucketName,  string[] fileNamesInS3, string path="")
 {
  var fileDeleted = 0;
  _logger.Info("Entering AmazonS3Helper.MultipleDelete");
  try
  {
   if (!string.IsNullOrEmpty(path))
   {
    if (path[path.Length - 1] != '/')
     path += "/";
   }
   // Converting filesnames as KeyVersions List
   List<KeyVersion> keyVersions = fileNamesInS3.Select(x => new KeyVersion() { Key = path + x }).ToList();

   var deleteObjectsRequest = new DeleteObjectsRequest
   {
    BucketName = bucketName,
    Objects = keyVersions
   };
   DeleteObjectsResponse response = await client.DeleteObjectsAsync(deleteObjectsRequest);
   fileDeleted = response.DeletedObjects.Count;
  }
  catch (DeleteObjectsException e)
  {
   _logger.Error("Error in deleting files from s3 bucket");
   // Catch the successfully deleted files count
   fileDeleted = e.Response.DeletedObjects.Count;
   foreach (DeleteError deleteError in e.Response.DeleteErrors)
   {
    _logger.Error($"Object Key: {deleteError.Key}\t{deleteError.Code}\t{deleteError.Message}");
   }
  }
  catch (Exception ex)
  {
   _logger.Error("Error in deleting files from s3 bucket", ex);
  }
  _logger.Info("Leaving AmazonS3Helper.MultipleDelete");
  return fileDeleted;
 }
}

Usage:

Create the helper class object

AmazonS3Helper amazonS3Helper = new AmazonS3Helper(<accesskey>, <secret key>, <AWS S3 endpoint url>);

Upload a file
Here, I am uploading S3.txt file to the dev folder in my S3 bucket named gopiBucket with name "S3Sample_1.txt"

string fileContent = File.ReadAllText("S3.txt");
await amazonS3Helper.Upload("gopiBucket/dev", "S3Sample_1.txt", fileContent);

Copy File:
Copy file method copies the file from one bucket to another bucket or one folder to another folder in a bucket. Here, I am copying my file from dev folder to prod folder in same bucket

await amazonS3Helper.Copy("gopiBucket/dev", "S3Sample_1.txt", "gopiBucket/prod", "S3Sample_1_prod.txt");

List files:
Here I am listing all the files in prod folder.

var files = await amazonS3Helper.ListFiles("gopiBucket", "prod");
foreach (var file in files)
{
 Console.WriteLine($"{file.Key}\t{file.Size}");
}

Download a file:
Donwloading the file from dev folder to the "Downloads" folder in my solution. If the "Downloads" folder not exits, it will create new one.

await amazonS3Helper.Download("gopiBucket/dev", "S3Sample_1.txt", "Downloads");

Delete a file:

await amazonS3Helper.Delete("gopiBucket/prod", "S3Sample_1_prod.txt");

Delete multiple files:

string[] filesToDelete = new string[] { "S3Sample_1_prod.txt", "S3Sample_2_prod.txt", "S3Sample_3_prod.txt" };
await amazonS3Helper.Delete("gopiBucket", filesToDelete, "prod");

Delete a completed folder:

await amazonS3Helper.DeleteFolder("gopiBucket", "prod/S1")

Note: The above mentioned Upload and Download operation methods uploads / downloads the files using a single GET / PUT operation. So you can upload / download objects up to 5 GB in size. If you want to store / retrieve more than 5GB, you need to use Multipart Upload API. I will give the Multipart Upload API details in my next post.

Happy Coding 😊 !!

Gopikrishna