Azure Databricks log runtime errors to Application insights

Using open census library to push error logs to Azure monitor

Prerequisite

• Azure account
• Azure Databricks
• Azure Storage

Steps

• Create a Databricks cluster
• install library

opencensus-ext-azure

• Create a notebook

Code

• Choose python as language

import logging
from opencensus.ext.azure.log_exporter import AzureLogHandler
logger = logging.getLogger(__name__)
# TODO: replace the all-zero GUID with your instrumentation key.
logger.addHandler(AzureLogHandler(
    connection_string='InstrumentationKey=xxxxx-xxxxxx-xxxxxx-xxxxxxx')
)

• now log some sample logs

logger.warning("Sample from open census test 01")
logger.error("Sample from open census test 02")

• Now lets log an exception

from opencensus.ext.azure.trace_exporter import AzureExporter
from opencensus.trace.samplers import ProbabilitySampler
from opencensus.trace.tracer import Tracer
properties = {'custom_dimensions': {'key_1': 'value_1', 'key_2': 'value_2'}}
# Use properties in exception logs
try:
    result = 1 / 0  # generate a ZeroDivisionError
except Exception:
    logger.exception('Captured an exception.', extra=properties)

• Log into Application insights and go to logs

• the above test for done from — https://docs.microsoft.com/en-us/azure/azure-monitor/app/opencensus-python#configure-azure-monitor-exporters

Originally published at https://github.com.