Simplifying .NET Application Logging: Seamless Integration with Azure Log Analytics and CI/CD Automation

Introduction

In this article, we introduce the CustomCloudLogger, a versatile logging client for .NET Standard 2.0 that simplifies the process of sending logs to Azure Log Analytics Workspace. This NuGet package is particularly useful for developers working with various .NET platforms, including .NET Core, .NET Framework, Xamarin, and Tizen. The CustomCloudLogger allows you to create custom log objects and send them directly to Azure, making logging straightforward and efficient. This solution is inspired by the LogAnalytics.Client  (.NET Core/5/6 only) package, providing a more flexible and comprehensive logging mechanism than the most commonly used packages. This is because properties are automatically transformed into columns. 

Background

Logging is an essential aspect of application development, allowing developers to track and diagnose issues, monitor application performance, and gain insights into user behavior. Azure Log Analytics is a powerful tool that aggregates and analyzes log data from various sources. The CustomCloudLogger project aims to bridge the gap between .NET applications and Azure Log Analytics by offering a seamless way to send logs to the Azure workspace, regardless of what columns you want to see, what .NET environment you use, what .NET version you use or what operating system you work on.

Using the code

The CustomCloudLogger package is available on NuGet and GitHub and can be easily installed into your .NET project. Here are the steps to get started:

Step 1: Install the Package

dotnet add package ConnectingApps.CustomCloudLogger

Step 2: Initialize the LogAnalyticsClient

Create an instance of the LogAnalyticsClient using your Azure workspace ID and shared key:

LogAnalyticsClient _client = new(WorkSpaceId, SharedKey);

Step 3: Create and Send Log Entries

You can create custom log entries and send them to Azure Log Analytics as follows:

var logEntries = new TryData[]
{
    new()
    {
        X = $"Test1 {Environment.MachineName}",
        Y = 1,
    },
    new()
    {
        X = $"Test2 {Environment.MachineName}",
        Y = 2,
    }
};
await _client.LogEntryAsync(logEntries.First(), "TuplesLog");
await _client.LogEntriesAsync(logEntries, "TuplesLog");

In this code:

1. An instance of LogAnalyticsClient is created.
2. A single log entry is sent using LogEntryAsync.
3. Multiple log entries are sent using LogEntriesAsync.

Remember to dispose of the client properly in production code to free up resources.

Step 4: Query Logs in Azure

Once your logs are sent, you can query them in Azure Log Analytics using Kusto Query Language (KQL). For example:

TuplesLog_CL
| where X_s contains "Test"

Step 5: Analyze and Act on Log Data

After executing your query, you will see the results displayed in the Azure Log Analytics interface. You can take various actions based on these results, such as setting up new alert rules or refining your queries for more precise log analysis. Below is an example of how such a result looks like:

In the screenshot, you can see the log entries that match the query criteria. You can use this data to monitor application behavior, detect anomalies, and respond to critical events in real-time. For instance, you might set up an alert rule to notify you when certain conditions are met, ensuring you can address issues promptly.

Key Methods Explained

The CustomCloudLogger relies on two key methods to function effectively: SendLogEntriesPrivateAsync and ValidatePropertyTypes.

SendLogEntriesPrivateAsync

This method handles the core functionality of sending log entries to Azure. Here's how it works:

private async Task SendLogEntriesPrivateAsync<t>(IReadOnlyList<t> entities, string logType, string? resourceId = null, string? timeGeneratedCustomFieldName = null)
{
    foreach (var entity in entities)
    {
        ValidatePropertyTypes(entity);
    }

    var dateTimeNow = DateTime.UtcNow.ToString("r", System.Globalization.CultureInfo.InvariantCulture);
    var entityAsJson = JsonSerializer.Serialize(entities, SerializeOptions);
    var authSignature = GetAuthSignature(entityAsJson, dateTimeNow);

    var headers = new Dictionary<string, string="">
    {
        {"Authorization", authSignature},
        {"Log-Type", logType},
        {"x-ms-date", dateTimeNow},
        {"time-generated-field", timeGeneratedCustomFieldName},
        {"x-ms-AzureResourceId", resourceId}
    };
    
    using var request = new HttpRequestMessage(HttpMethod.Post, this._requestBaseUrl);
    foreach (var header in headers.Where(h => !string.IsNullOrEmpty(h.Value)))
    {
        request.Headers.Add(header.Key, header.Value);
    }

    HttpContent httpContent = new StringContent(entityAsJson, Encoding.UTF8);
    httpContent.Headers.ContentType = new MediaTypeHeaderValue("application/json");

    request.Content = httpContent;
    var response = await _httpClient.SendAsync(request).ConfigureAwait(false);
    response.EnsureSuccessStatusCode();
}

In this method:

1. It first validates the types of properties within each entity using the ValidatePropertyTypes method.
2. It then prepares the data to be sent by serializing the entities to JSON and generating an authorization signature.
3. Headers are set up, including authorization, log type, and date.
4. An HTTP POST request is made to send the log data to Azure.
5. It ensures the request is successful and throws an exception if not.

ValidatePropertyTypes

This method ensures that the properties of the entities being logged are of allowed types:

private void ValidatePropertyTypes<t>(T entity)
{
    // Retrieve all properties of the entity type using reflection
    var properties = entity!.GetType().GetProperties();

    // Validate each property's type
    foreach (var propertyInfo in properties)
    {
        if (!AllowedTypes.Contains(propertyInfo.PropertyType))
        {
            var typedString = propertyInfo.PropertyType.ToString();
            // Check for .NET 6+ types
            if (typedString.EndsWith("System.DateOnly") || typedString.EndsWith("System.DateOnly]"))
            {
                continue;
            }
            throw new ArgumentOutOfRangeException(
                $"Property '{propertyInfo.Name}' of entity with type '{entity.GetType()}' " +
                $"is not one of the valid properties:" +
                $" String, Boolean, Double, Integer, DateTime, DateOnly and Guid.");
        }
    }
}

This method:

1. Retrieves all properties of the entity type using reflection.
2. Checks each property to ensure it is one of the allowed types (e.g., string, bool, double, int, DateTime, DateOnly, Guid). DateOnly is checked as a string since this type is not directly supported in .NET Standard. It is a relatively new type introduced in .NET 6. 
3. Throws an exception if a property is not of an allowed type.

These methods are critical for ensuring that the data being sent to Azure Log Analytics is valid and properly formatted, thereby maintaining the integrity and reliability of the logging process.

GitHub Actions Pipeline Explained

The CustomCloudLogger project uses GitHub Actions to automate the build and test process. Below is the GitHub Actions pipeline configuration and an explanation of each step:

name: .NET CI

on:
  push:
    branches:
      - main
      - release
      - develop
      - feature/**
      - bugfix/**

jobs:
  build_and_test:
    name: Build and Test
    runs-on: ubuntu-22.04

    env:  # Setting environment variable at the job level
      WORKSPACE_ID: ${{ secrets.WORKSPACE_ID }}
      SHARED_KEY: ${{ secrets.SHARED_KEY }} 

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup .NET SDK
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: '8.0.x' 

      - name: Restore dependencies
        run: dotnet restore CustomCloudLogger.sln

      - name: Build Solution
        run: dotnet build CustomCloudLogger.sln --configuration Release --no-restore

      - name: Run Tests
        run: python -c "import os; os.system('dotnet test CustomCloudLogger.sln --configuration Release --no-build --verbosity normal  --logger trx');"  

      - name: Publish Test Results
        uses: dorny/test-reporter@v1
        with:
          name: 'Test Results'
          path: '**/TestResults/**/*.

trx'
          reporter: 'dotnet-trx'

      - name: Upload NuGet packages as artifacts
        uses: actions/upload-artifact@v4
        with:
          name: nuget-packages
          path: '**/*.nupkg'  

      - name: Publish to NuGet
        if: github.ref == 'refs/heads/main'
        run: dotnet nuget push "ConnectingApps.CustomCloudLogger/bin/Release/*.nupkg" --api-key ${{ secrets.NUGET_KEY }} --source https://api.nuget.org/v3/index.json       

This pipeline is triggered by pushes to specific branches (main, release, develop, feature/*, bugfix/*). It consists of the following steps:

1. Checkout code: This step uses the actions/checkout@v4 action to check out the code from the repository.
2. Setup .NET SDK: This step uses the actions/setup-dotnet@v4 action to install the specified version of the .NET SDK (8.0.x).
3. Restore dependencies: This step runs the dotnet restore command to restore the project's dependencies.
4. Build Solution: This step runs the dotnet build command to build the solution in Release configuration without restoring dependencies again.
5. Run Tests: This step runs the dotnet test command to execute the tests. It uses a Python script to run the command and generates a trx file for the test results. The reason for using python instead of direct commandline execution is that the step should not fail in case one of the tests fails. This would prevent the next step from being executed.
6. Publish Test Results: This step uses the dorny/test-reporter@v1 action to publish the test results. Failure of one of the tests means that this step is the last one.
7. Upload NuGet packages as artifacts: This step uses the actions/upload-artifact@v4 action to upload the generated NuGet packages as build artifacts.
8. Publish to NuGet: This step runs the dotnet nuget push command to publish the package to NuGet.org if the branch is main.

The pipeline uses secrets to securely manage sensitive information:

• WORKSPACE_ID: The ID of the Azure Log Analytics workspace.
• SHARED_KEY: The shared key for authenticating with the Azure Log Analytics workspace.
• NUGET_KEY: The API key for publishing packages to NuGet.org.

These secrets are stored securely in the GitHub repository's settings and are referenced in the pipeline using the ${{ secrets.[SECRET_NAME] }} syntax.

Integration Tests

The CustomCloudLogger project includes integration tests to ensure that the logging functionality works as expected with Azure Log Analytics. The GitHub Actions pipeline sets up the necessary environment variables, which are then used in the integration tests. Below is the relevant source code for the integration tests:

private static readonly string WorkSpaceId;
private static readonly string SharedKey;
private readonly LogAnalyticsClient _client = new(WorkSpaceId, SharedKey);

static LogAnalyticsClientTest()
{
    WorkSpaceId = Environment.GetEnvironmentVariable("WORKSPACE_ID")!;
    SharedKey = Environment.GetEnvironmentVariable("SHARED_KEY")!;
    WorkSpaceId.Should().NotBeNullOrEmpty();
    SharedKey.Should().NotBeNullOrEmpty();
}

[Fact]
public async Task SendMessageTest()
{
    var logEntry = new 
    {
        Date = DateTime.UtcNow,
        Message = $"Test log message System Text from {Environment.MachineName}",
        Severity = "Info"
    };

    await _client.LogEntryAsync(logEntry, "TestLog");
}

In this test:

1. The WorkSpaceId and SharedKey are retrieved from the environment variables set by the GitHub Actions pipeline.
2. An instance of LogAnalyticsClient is created using these environment variables.
3. The SendMessageTest method creates a log entry with the current date, a test message, and a severity level.
4. The log entry is sent to the Azure Log Analytics workspace using the LogEntryAsync method.

These integration tests ensure that the logging client can successfully communicate with Azure Log Analytics, providing confidence that the logging functionality will work correctly in production environments.

Points of Interest

One interesting aspect of developing with CustomCloudLogger is its support for various .NET versions and platforms. The inclusion of the DateOnly data type for .NET 6 and higher is particularly noteworthy, offering modern date handling capabilities. Additionally, integrating with Azure's powerful log analytics tools provides developers with a robust logging solution that scales with their applications. 

History

This is the initial release of the CustomCloudLogger article. Future updates will include more detailed examples, performance benchmarks, and best practices for large-scale logging in enterprise applications.