Skip to content

Azure Blob Storage Provider ​

Prerequisites: Storage Providers Overview

The NPipeline.StorageProviders.Azure package implements IStorageProvider for Azure Blob Storage. Supports DefaultAzureCredential, connection strings, chunked block blob uploads, client caching with LRU eviction, and the Azurite emulator.

Installation ​

bash
dotnet add package NPipeline.StorageProviders.Azure

Dependencies: Azure.Storage.Blobs 12.x, Azure.Identity 1.x

Quick Start ​

csharp
var options = new AzureBlobStorageProviderOptions
{
    DefaultConnectionString = "DefaultEndpointsProtocol=https;AccountName=mystorageaccount;AccountKey=..."
};
var factory = new AzureBlobClientFactory(options);
var provider = new AzureBlobStorageProvider(factory, options);

var stream = await provider.OpenReadAsync(
    StorageUri.Parse("azure://my-container/data/orders.csv"));

URI Format ​

azure://container-name/blob/path
ComponentDescription
container-nameBlob container (URI host)
blob/pathBlob name (URI path)

Authentication ​

Credentials are resolved in this order:

  1. Connection string - a connectionString URI parameter or DefaultConnectionString. It takes precedence over the options below. A connection string names its own endpoint, so combining it with ServiceUrl or a serviceUrl parameter throws ArgumentException.
  2. Explicit credential - DefaultCredential (any TokenCredential)
  3. Default credential chain - DefaultAzureCredential (environment → managed identity → Azure CLI)
csharp
// Connection string
var options = new AzureBlobStorageProviderOptions
{
    DefaultConnectionString = "DefaultEndpointsProtocol=https;AccountName=...;AccountKey=..."
};

// Managed identity (recommended for production)
var options = new AzureBlobStorageProviderOptions
{
    UseDefaultCredentialChain = true    // default
};

// Explicit credential
var options = new AzureBlobStorageProviderOptions
{
    DefaultCredential = new ManagedIdentityCredential()
};

Configuration ​

PropertyTypeDefaultDescription
DefaultConnectionStringstring?nullAzure Storage connection string
DefaultCredentialTokenCredential?nullAzure TokenCredential
UseDefaultCredentialChainbooltrueUse DefaultAzureCredential
AllowAnonymousAccessboolfalseConnect without credentials when none are configured, for public containers. When false, a missing credential throws InvalidOperationException.
ServiceUrlUri?nullCustom Blob service URL (Azurite)
ServiceVersionBlobClientOptions.ServiceVersion?nullAPI version override
BlockBlobUploadThresholdByteslong64 MBSwitch to staged upload above this size
UploadMaximumConcurrencyint?nullParallel upload threads
UploadMaximumTransferSizeBytesint?nullBlock size for staged uploads
ClientCacheSizeLimitint100Max cached BlobContainerClient instances

Dependency Injection ​

csharp
// Default options
services.AddAzureBlobStorageProvider();

// Configure inline
services.AddAzureBlobStorageProvider(options =>
{
    options.DefaultConnectionString = "...";
});

Registers: IStorageProvider, IStorageProviderMetadataProvider

Features ​

  • Chunked uploads - files above BlockBlobUploadThresholdBytes use the staged block upload API
  • Client caching - BlobContainerClient instances are cached per container; LRU eviction when ClientCacheSizeLimit is reached
  • Azurite emulator - set ServiceUrl for local development
  • Metadata - implements IStorageProviderMetadataProvider for Size, LastModified, ContentType, ETag

URI Parameters ​

ParameterDescriptionExample
accountNameStorage account nameaccountName=mystorageaccount
accountKeyAccount key (dev only)accountKey=mykey
sasTokenSAS token (URL-encoded)sasToken=sp%3Dr%26st%3D2023-01-01
connectionStringFull connection stringconnectionString=DefaultEndpointsProtocol=...
serviceUrlCustom service URL (Azurite)serviceUrl=http://localhost:10000/devstoreaccount1
contentTypeContent type on writecontentType=application/json
csharp
// With SAS token
var uri = StorageUri.Parse("azure://my-container/data/output.json?sasToken=sp%3Dr%26st%3D2023-01-01");

// Azurite endpoint
var uri = StorageUri.Parse("azure://my-container/data/file.csv?serviceUrl=http://localhost:10000/devstoreaccount1");

Security: Avoid credentials in URIs for production. Use the credential chain or connection strings via DI.

Configuration Examples ​

Azurite (Local Development) ​

csharp
services.AddAzureBlobStorageProvider(options =>
{
    options.ServiceUrl = new Uri("http://localhost:10000/devstoreaccount1");
    options.DefaultConnectionString = "UseDevelopmentStorage=true";
});

Custom Upload Settings ​

csharp
services.AddAzureBlobStorageProvider(options =>
{
    options.BlockBlobUploadThresholdBytes = 128 * 1024 * 1024; // 128 MB
    options.UploadMaximumConcurrency = 8;
    options.UploadMaximumTransferSizeBytes = 8 * 1024 * 1024;  // 8 MB blocks
});

Examples ​

Reading ​

csharp
var uri = StorageUri.Parse("azure://my-container/data.csv");

using var stream = await provider.OpenReadAsync(uri);
using var reader = new StreamReader(stream);
var content = await reader.ReadToEndAsync();

Writing ​

csharp
var uri = StorageUri.Parse("azure://my-container/output.csv");

using var stream = await provider.OpenWriteAsync(uri);
using var writer = new StreamWriter(stream);
await writer.WriteLineAsync("id,name,value");

Listing ​

csharp
var prefix = StorageUri.Parse("azure://my-container/data/");

await foreach (var item in provider.ListAsync(prefix, recursive: true))
{
    Console.WriteLine($"{item.Uri} - {item.Size} bytes");
}

Metadata ​

csharp
var metadata = await provider.GetMetadataAsync(uri);
if (metadata is not null)
{
    Console.WriteLine($"Size: {metadata.Size}, ContentType: {metadata.ContentType}");
    Console.WriteLine($"ETag: {metadata.ETag}");
}

Error Handling ​

Azure ErrorHTTP Status.NET Exception
AuthenticationFailed401UnauthorizedAccessException
AuthorizationFailed403UnauthorizedAccessException
ContainerNotFound, BlobNotFound404FileNotFoundException
InvalidResourceName400ArgumentException
Other RequestFailedExceptionVariousIOException

Azure Permissions ​

OperationRequired Permission
OpenReadAsyncStorage Blob Data Reader
OpenWriteAsyncStorage Blob Data Contributor
ListAsyncStorage Blob Data Reader
ExistsAsyncStorage Blob Data Reader

Assign Storage Blob Data Contributor for full read/write/delete access. For read-only pipelines, Storage Blob Data Reader is sufficient.

Limitations ​

  • Flat storage - Blob Storage has no directories; prefix-based hierarchy is simulated
  • Concurrent writes - writing to the same blob from multiple threads may race; use locking or versioning
  • Block blob size - maximum 190.7 TiB per blob; each block max 4,000 MiB

Next Steps ​

Released under the MIT License.