Skip to content

AWS S3 Storage Provider ​

Prerequisites: Storage Providers Overview

The NPipeline.StorageProviders.S3.Aws package implements IStorageProvider for Amazon S3. Supports the default AWS credential chain, explicit credentials, multipart uploads, virtual-hosted and path-style addressing, and per-URI region overrides.

Installation ​

bash
dotnet add package NPipeline.StorageProviders.S3.Aws

Dependencies: AWSSDK.S3 4.x, AWSSDK.Core 4.x

Quick Start ​

csharp
var options = new AwsS3StorageProviderOptions
{
    DefaultRegion = RegionEndpoint.USEast1
};
var factory = new AwsS3ClientFactory(options);
var provider = new AwsS3StorageProvider(factory, options);

var stream = await provider.OpenReadAsync(
    StorageUri.Parse("s3://my-bucket/data/orders.csv"));

URI Format ​

s3://bucket-name/key/path?region=ap-southeast-2
ComponentDescription
bucket-nameS3 bucket (URI host)
key/pathObject key (URI path)
regionOptional - override DefaultRegion for this request

Authentication ​

Credentials are resolved in this order:

  1. Explicit credentials - set DefaultCredentials to any AWSCredentials instance
  2. Default credential chain (default) - environment variables → shared credentials file → EC2 instance profile → ECS task role
csharp
// Explicit credentials (development only)
var options = new AwsS3StorageProviderOptions
{
    DefaultCredentials = new BasicAWSCredentials("AKIA...", "secret"),
    DefaultRegion = RegionEndpoint.USEast1
};

// Default chain (recommended for production)
var options = new AwsS3StorageProviderOptions
{
    UseDefaultCredentialChain = true,    // default
    DefaultRegion = RegionEndpoint.USEast1
};

Configuration ​

PropertyTypeDefaultDescription
DefaultRegionRegionEndpoint?nullAWS region for S3 API calls. A region URI parameter overrides it. When neither is set, the AWS SDK resolves the region (AWS_REGION, the shared profile, or instance metadata). With ServiceUrl, the region is used to sign requests.
DefaultCredentialsAWSCredentials?nullExplicit AWS credentials
UseDefaultCredentialChainbooltrueFall back to the default credential chain
ServiceUrlUri?nullCustom S3 endpoint (Floci, MinIO)
ForcePathStyleboolfalseUse path-style URLs instead of virtual-hosted
MultipartUploadThresholdByteslong64 MBSwitch to multipart upload above this size

Dependency Injection ​

csharp
// Default options
services.AddAwsS3StorageProvider();

// Configure inline
services.AddAwsS3StorageProvider(options =>
{
    options.DefaultRegion = RegionEndpoint.EUWest1;
});

// Pre-built options
services.AddAwsS3StorageProvider(new AwsS3StorageProviderOptions
{
    DefaultRegion = RegionEndpoint.USEast1
});

Registers: IStorageProvider, IStorageProviderMetadataProvider

Features ​

  • Multipart uploads - files above MultipartUploadThresholdBytes are uploaded using the S3 multipart API
  • Client caching - S3 clients are cached and reused per region/endpoint
  • Virtual-hosted addressing - default; set ForcePathStyle = true for Floci or older S3-compatible services
  • Metadata - implements IStorageProviderMetadataProvider for Size, LastModified, ContentType, ETag

URI Parameters ​

ParameterDescriptionExample
regionAWS region nameregion=ap-southeast-2
accessKeyAWS access key (dev only)accessKey=AKIAIOSFODNN7EXAMPLE
secretKeyAWS secret key (dev only)secretKey=wJalrXUtnFEMI/...
serviceUrlCustom S3 endpointserviceUrl=http://localhost:9000
pathStyleForce path-style (true/false)pathStyle=true
contentTypeContent type on writecontentType=application/json
csharp
// With region override
var uri = StorageUri.Parse("s3://my-bucket/data/input.csv?region=us-west-2");

// With custom endpoint (Floci)
var uri = StorageUri.Parse("s3://local-bucket/data/file.csv?serviceUrl=http://localhost:4566&pathStyle=true");

Security: Avoid credentials in URIs for production - URIs may be logged. Use the credential chain or DI.

Configuration Examples ​

Floci (Testing) ​

csharp
services.AddAwsS3StorageProvider(options =>
{
    options.ServiceUrl = new Uri("http://localhost:4566");
    options.ForcePathStyle = true;
    options.DefaultRegion = RegionEndpoint.USEast1;
});

MinIO (via AWS Provider) ​

Recommended: Use the S3-Compatible Provider instead for non-AWS endpoints.

csharp
services.AddAwsS3StorageProvider(options =>
{
    options.ServiceUrl = new Uri("http://localhost:9000");
    options.ForcePathStyle = true;
    options.DefaultRegion = RegionEndpoint.USEast1;
    options.DefaultCredentials = new BasicAWSCredentials("minioadmin", "minioadmin");
    options.UseDefaultCredentialChain = false;
});

Examples ​

Reading ​

csharp
var uri = StorageUri.Parse("s3://my-bucket/data.csv");

using var stream = await provider.OpenReadAsync(uri);
using var reader = new StreamReader(stream);
var content = await reader.ReadToEndAsync();

Writing ​

csharp
var uri = StorageUri.Parse("s3://my-bucket/output.csv");

using var stream = await provider.OpenWriteAsync(uri);
using var writer = new StreamWriter(stream);
await writer.WriteLineAsync("id,name,value");
await writer.WriteLineAsync("1,Item A,100");

Listing ​

csharp
var prefix = StorageUri.Parse("s3://my-bucket/data/");

await foreach (var item in provider.ListAsync(prefix, recursive: true))
{
    Console.WriteLine($"{item.Uri} - {item.Size} bytes");
}

Metadata ​

csharp
var metadata = await provider.GetMetadataAsync(uri);
if (metadata is not null)
{
    Console.WriteLine($"Size: {metadata.Size}, ContentType: {metadata.ContentType}");
    Console.WriteLine($"Last Modified: {metadata.LastModified}, ETag: {metadata.ETag}");
}

Error Handling ​

S3 Error Code.NET ExceptionDescription
AccessDenied, InvalidAccessKeyId, SignatureDoesNotMatchUnauthorizedAccessExceptionAuth or permission failure
InvalidBucketName, InvalidKeyArgumentExceptionInvalid bucket/key
NoSuchBucket, NotFoundFileNotFoundExceptionBucket or object missing
Other AmazonS3ExceptionIOExceptionGeneral S3 failure

IAM Permissions ​

OperationRequired Permission
OpenReadAsyncs3:GetObject
OpenWriteAsyncs3:PutObject
ListAsyncs3:ListBucket
ExistsAsyncs3:GetObject
GetMetadataAsyncs3:GetObject

Minimal Read/Write Policy ​

json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket"],
      "Resource": ["arn:aws:s3:::my-bucket", "arn:aws:s3:::my-bucket/*"]
    }
  ]
}

Limitations ​

  • Flat storage - S3 has no directories; prefixes simulate hierarchy
  • Multipart uploads - files above threshold use multipart API; ensure sufficient memory and bandwidth
  • Eventually consistent - S3 standard now provides strong read-after-write consistency for PUT, but LIST operations may lag

Next Steps ​

Released under the MIT License.