MuleSoft Mastery: From Zero to Hero | Ch.13

You’ve secured your integrations in Chapter 12; now you have to operate them. When an order pipeline starts failing at 2 a.m., you need three things fast: a signal that something broke, the log line that explains it, and a trace that shows exactly which hop in a chain of services went dark. That’s the three pillars of observability — metrics, logs, and traces — and MuleSoft gives you a built-in stack plus open paths to every major platform.

By the end of this chapter you will:

• Use Anypoint Monitoring dashboards and alerts, and enable distributed tracing
• Send business metrics with the Custom Metrics connector — the correct custom-metrics:send operation
• Forward logs to Splunk, ELK, and Azure with asynchronous Log4j2 appenders
• Correlate everything with MDC — correlationId, traceId, spanId
• Export OpenTelemetry traces (and logs) via the Direct Telemetry Stream and an OTel Collector
• Diagnose a real incident by triangulating across all three pillars

01 · Three Pillars — Metrics, Logs & Traces

Observability isn’t one tool — it’s three complementary signals. Each answers a different question, and you need all three to move from “something’s wrong” to “here’s the fix” without guessing.

02 · Anypoint Monitoring — Built-in Dashboards & Alerts

For apps on CloudHub 2.0 and Runtime Fabric, Anypoint Monitoring collects metrics, logs, and traces automatically — reachable via Runtime Manager → your app → Monitoring. Out of the box you get an Application panel (throughput, error rates, response times, JVM heap/GC/threads) and an API panel (requests by method, response codes, latency percentiles).

2.1 · Alerts that page you

Alerts fire when a metric crosses a threshold for a sustained window. Create them under Alerts → Create Alert in Runtime Manager. A well-tuned alert names a resource, a metric, a condition, a duration (to avoid flapping), and a notification channel.

03 · Custom Metrics — Tracking Business KPIs

Platform metrics tell you the runtime is healthy; custom metrics tell you the business is healthy — orders per minute, payment success rate, processing time by region. These come from the Custom Metrics connector, which sends data to Anypoint Monitoring. It requires the Anypoint Monitoring Agent and a supported runtime.

<dependency>
  <groupId>com.mule.modules</groupId>
  <artifactId>mule-custom-metrics-extension</artifactId>
  <version>2.3.2</version>
  <classifier>mule-plugin</classifier>
</dependency>

<custom-metrics:send config-ref="Custom_Metrics"
                     metricName="order.processing">
  <!-- dimensions: LOW cardinality labels you slice by -->
  <custom-metrics:dimensions>#[{
    status:       "success",
    region:       payload.region,
    customerTier: payload.tier
  }]</custom-metrics:dimensions>
  <!-- facts: NUMERIC values only -->
  <custom-metrics:facts>#[{
    count:            1,
    orderValue:       payload.totalAmount,
    processingTimeMs: now() - vars.startTime
  }]</custom-metrics:facts>
</custom-metrics:send>

04 · Logs Everywhere — Splunk, ELK & Azure via Log4j2

Mule logs through Log4j2, so forwarding to an external platform is a matter of adding the right appender to src/main/resources/log4j2.xml. Two non-negotiables: appenders must be asynchronous (a slow log backend must never block your message thread), and secrets must come from placeholders, never hardcoded.

<Configuration>
  <Appenders>
    <SplunkHttp name="splunk"
        url="https://http-inputs.splunk.com:443"
        token="${sys:splunk.token}"          <!-- never hardcode -->
        index="mule" source="mule-app">
      <PatternLayout pattern="%d %p [%X{correlationId}] %c - %m%n"/>
    </SplunkHttp>
    <!-- wrap ANY appender to make it non-blocking -->
    <Async name="asyncSplunk">
      <AppenderRef ref="splunk"/>
    </Async>
  </Appenders>
  <Loggers>
    <AsyncRoot level="INFO">
      <AppenderRef ref="asyncSplunk"/>
    </AsyncRoot>
  </Loggers>
</Configuration>

05 · OpenTelemetry — Vendor-Neutral Distributed Tracing

OpenTelemetry (OTel) is the open standard for telemetry, and Mule runtime speaks it natively. With the Direct Telemetry Stream (Mule 4.11.0+), the runtime exports traces and logs in OTLP format to any compatible backend — Jaeger, Datadog, Dynatrace, Splunk, Azure. You enable it in the deployment settings alongside “Enable tracing.”

When you run the community OpenTelemetry module (or the SDK in autoconfigure mode), you point the exporter at your collector through standard OTel properties — set as system properties or environment variables:

# export traces and logs over OTLP to the collector
otel.traces.exporter=otlp
otel.logs.exporter=otlp
otel.exporter.otlp.endpoint=http://otel-collector:4318
otel.exporter.otlp.protocol=http/protobuf
otel.service.name=customer-api
otel.resource.attributes=environment=production,region=us-east-1

# sample to control volume in production
otel.traces.sampler=parentbased_traceidratio
otel.traces.sampler.arg=0.1   # 10% of traces

5.1 · Correlating with Azure Application Insights

Azure keys distributed traces off the W3C traceparent header, mapping it to operation_Id. With tracing enabled, Mule’s HTTP Listener and Requester propagate traceparent automatically — so the fix when correlation breaks is almost always a component (or proxy) overwriting or stripping that header. Verify nothing in the chain rewrites traceparent, and the same operation_Id will line up across every span.

06 · Case Study — Diagnosing Payment-Gateway Failures

An e-commerce platform hits intermittent order failures. The on-call engineer uses all three pillars in sequence — and finds the root cause in minutes, not hours.

index=main sourcetype=mule-app errorType="HTTP:CONNECTIVITY"
| timechart count by region

07 · Best Practices — Observability Rules That Hold Up

08 · Recap — What You Now Know