C# — work with Azure Data Lake storage files

The last post was a short introduction, how to work with blobs in Azure Blob Storage. C# — work with Azure BLOB storage files

Today, I will provide a few examples, how to perform simple operations on files in a Data Lake Storage.

As usual, we need some libraries.

• Azure.Storage.Files.DataLake
• Azure.Identity

using Azure;
using Azure.Storage.Files.DataLake;
using Azure.Storage.Files.DataLake.Models;
using Azure.Identity;

// dotnet add package Azure.Storage.Files.DataLake
// dotnet add package Azure.Identity

As in the previous post, we need to create a service client and a file system client. The File System is a container reference in this case.

// Create a dataLakeServiceClient
var credential = new ClientSecretCredential(AzureVariables.tenantId, AzureVariables.clientId, AzureVariables.clientSecret);
Uri dataLakeEndpoint = new Uri(AzureVariables.adlsEndpoint);
DataLakeServiceClient dataLakeServiceClient = new DataLakeServiceClient(dataLakeEndpoint, credential);

// Create a dataLakeFileSystemClient
string containerName = "iowa-json";
DataLakeFileSystemClient dataLakeFileSystemClient = dataLakeServiceClient.GetFileSystemClient(containerName);

Here is the way, how to create a new folder.

// Create a new folder
DataLakeDirectoryClient dataLakeDirectoryClient00 = await dataLakeFileSystemClient.CreateDirectoryAsync("adls_json_upload");

Rename the folder.

// Rename a folder
DataLakeDirectoryClient dataLakeDirectoryClient01 = dataLakeFileSystemClient.GetDirectoryClient("adls_json_upload");
DataLakeDirectoryClient dataLakeDirectoryClient02 = await dataLakeDirectoryClient01.RenameAsync("adls-json_upload-01");

Of course, we are able to delete this folder.

// Delete the directory
await dataLakeDirectoryClient02.DeleteAsync();

Assuming we didn’t remove the folder, we can upload a file there.

// Upload a file
string fileName = "09.json";
string fileToUpload = Path.Combine("json_files", fileName);

DataLakeFileClient fileToUploadClient = await dataLakeDirectoryClient02.CreateFileAsync(fileName);

FileStream fileStream00 = File.OpenRead(fileToUpload);
await fileToUploadClient.AppendAsync(fileStream00, offset: 0);
await fileToUploadClient.FlushAsync(position: fileStream00.Length);

The easy way of listing files in the folder.

// List files in a directory
await foreach (PathItem pathItem in dataLakeDirectoryClient02.GetPathsAsync(recursive: true))
{
    System.Console.WriteLine($"File: {pathItem.Name}");
}

This example shows how to download a file to a local storage.

// Download a file
DataLakeFileClient fileToDownload = dataLakeDirectoryClient02.GetFileClient("09.json");
Response<FileDownloadInfo> downloadResponse  = await fileToDownload.ReadAsync();
BinaryReader reader = new BinaryReader(downloadResponse.Value.Content);
FileStream fileStream01 = File.OpenWrite("json_files/09-01.json");

byte[] buffer = new byte[4096];
int count;

while((count = reader.Read(buffer, 0, buffer.Length)) != 0)
{
    fileStream01.Write(buffer, 0, count);
}

await fileStream01.FlushAsync();
fileStream01.Close();

Rename the same file. You can rename only a file or do it along with the folder. In this case we change both. A destination folder must exist.

// Rename a file and move to another folder - the new folder must exist
await fileToDownload.RenameAsync("adls-json_upload-02/09-03.json");

And delete.

// Delete a file
await dataLakeFileSystemClient.DeleteFileAsync("adls-json_upload-01/09.json");

Here is the code on GitHub.