Azure App Logging
Overview
Azure Monitor is our primary source of information on our cloud environment, which includes data on application logs, user interactions, and overall performance. Since many of our services rely on APIs or serve APIs themselves, the Azure App Insights service is what we use for tracking. It is simple to attach to a cloud resource and have it collect information about all interactions involving it. All resources from our resource group should be tied to the same Application Insights service so that their logs fall into the same Log Analytics workspace.
Export Options
For the purposes of analyzing these logs, it is useful to export them to a more suitable location. One reason is that logs may be purged past a certain date, which affects our governance and accountability. Another reason is that the analysis tools offered natively in Azure are relatively limited, necessitating another platform. Databricks is our choice for this, due to the ability to create dashboards and write custom code. Databricks allows us to import data directly from Azure cloud storage locations, which falls in line with our need to keep logs in longer-term storage.
This means we need to rely on Log Analytics workspace data exports. As long as the exported logs fall into an Azure cloud storage location (specifically an Azure Data Lake Storage or ADLS), we will be able to import them into Databricks. So here is what it looks like at a high level:
graph LR
A[App 1] & B[App 2] --(App Insights)--> C[Azure Monitor]
C --> D[Log Analytics Workspace]
D --(Data Export)--> E["Storage Account (Data Lake Storage)"]
E --(Unity Catalog)--> F[Databricks]Log Analytics to Storage Account
The guide for putting data from the Log Analytics workspace into an Azure Storage volume indicates that we should select specific tables to be exported as part of an export rule. There is a limit of 10 rules per Log Analytics workspace, but each can include multiple tables. Each rule must have a different Storage Account associated with it. A list of tables that are possible to export follows. Pricing is $0.10/GB.
| Table name | Solution | Category | Resource type |
|---|---|---|---|
| Alert | LogManagement | Azure Monitor | |
| AppAvailabilityResults | LogManagement | Applications | Application Insights |
| AppBrowserTimings | LogManagement | Applications | Application Insights |
| AppCenterError | LogManagement | ||
| AppDependencies | LogManagement | Applications | Application Insights |
| AppExtensionConsoleLogs | LogManagement | Audit, Azure Resources | Container Apps |
| AppExtensionFileLogs | LogManagement | Audit, Azure Resources | Container Apps |
| AppExtensionPerfCounters | LogManagement | Audit, Azure Resources | Container Apps |
| AppExtensionSnapshotConsoleLogs | LogManagement | Audit, Azure Resources | Container Apps |
| AppEvents | LogManagement | Applications | Application Insights |
| AppExceptions | LogManagement | Applications | Application Insights |
| AppPageViews | LogManagement | Applications | Application Insights |
| AppPerformanceCounters | LogManagement | Applications | Application Insights |
| Apps | LogManagement | Applications | Application Insights |
| AppRequests | LogManagement | Applications | Application Insights |
| AppSystemEvents | LogManagement | Applications | Application Insights |
| AppTraces | LogManagement | Applications | Application Insights |
| Availability | LogManagement | Azure Resources, Audit, Security | Container Apps, Application Insights |
| ComputerGroup | LogManagement | Azure Monitor, Virtual Machines, IT & Management Tools | |
| ContainerInventory | LogManagement | Audit, Azure Resources | Container Apps |
| ContainerPerf | LogManagement | Audit, Azure Resources | Container Apps |
| InsightMetrics | LogManagement | Virtual Machines, Containers, Azure Resources | |
| Operation | LogManagement | Azure Monitor | |
| Usage | LogManagement | Azure Monitor |
Depending on the application and its logging strategy, some or none of these tables may be relevant for analysis. For example, for Beacon, the AppTraces table contains conversational traces that log requests made to the chat app and responses out. This is different from DES, which relies on ContainerAppConsoleLogs to record app activity as most activity is recorded via print statements.
Note
There are alternatives to putting logs into a storage account. We choose not to use them for simplicity. Options include:
- Direct logging to a file or table: This would require additional setup per application rather than relying on App Insights.
- Blob Storage: Unity Catalog (see below) is not compatible with Azure Blob Storage. In addition, the method for querying blob storage requires Key Vault access and has additional complexity.
- Event Hubs: We don't have a need for event-based triggers yet and this has additional complexity.
- Splunk: We don't currently have access to Splunk and its analysis tools are also somewhat limited.
Storage Account to Databricks
The recommended strategy for moving logs from Azure Data Lake Storage (ADLS) to Databricks is to use Unity Catalog. It is compatible with only ADLS, so generic Blob Storage will not work. It is also necessary to use the ADLS as an external storage location as Unity Catalog will attempt to govern the storage location itself otherwise, complicating the ability to write to the location.
To attach ADLS as an "external location" in Unity Catalog, you must provide the full path to the cloud storage location and the credentials needed for access. Unity Catalog will use those values to create a "storage credential", which it will use to read data when requested. For ADLS, this takes the form of a service principal (or Azure Managed Identity). It is important not to abuse this connection within Databricks to make edits to these tables as that will cause them to become out of sync and require repairs. Please see the instructions on how to perform table repairs.
The specific process to create a storage credential for ADLS requires admin privileges. It must then be assigned to the ADLS as an external location. Please contact the EDP team for assistance in this process.
Summary
Overall, the steps to connect Azure Monitor to Databricks can be summarized as follows:
- Create the ADLS container as part of a storage account with a hierarchical namespace.
- Set up a Log Analytics workspace export rule pointing to that ADLS container.
- Provision an Azure Managed Identity or service principal with permissions over the ADLS container.
- Create a storage credential with the help of the EDP team.
- (Optional) Give the credential "read-only" access.
- Assign the storage credential to the ADLS container as an external location with the help of the EDP team.
Priorities for Logging AI-based Applications
The following list of potential log information is not exhaustive, but contains a general list of content that could be logged for governance and future analysis. How logging should answer these questions is up to the app author, but these are good to consider.
- Requestor:
- Who performed the action?
- What is their identity or group?
- Did they have the right permissions?
- Where did the request originate?
- When did the query come in?
- Request content
- What was the query?
- What was attached to the request?
- Was the query properly formatted and sanitized?
- App activity
- What actions did the app take?
- What did the app report during execution?
- What information would be useful to know for debugging?
- Were there any errors or exceptions encountered?
- How long did execution take?
- What seems to be bottle-necking the program?
- Response content
- What did the app return to the user or program?
- Did the user provide any feedback?
- When was this response provided?
- To where did this response go?