Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
ON THIS PAGE
 

Pitrace Binary Tracing Overview

PiTrace (Pixel Tracing) provides a low-overhead binary tracing infrastructure that continuously records protocol operational events and routing workflow activity in compact trace files for post-incident troubleshooting and root-cause analysis. Unlike traditional verbose text-based tracing, PiTrace minimizes operational overhead, allowing it to remain active during normal system operation without significantly affecting scale or performance.

PiTrace stores trace records in memory and periodically writes them to binary trace files under the /var/log/pitrace directory. The system flushes trace data to disk when memory thresholds are reached or when users initiate manual flush operations. To preserve recent operational activity for later analysis, PiTrace uses configurable trace-file retention limits to maintain a rolling trace history.

The tracing framework supports multithreaded and multicore environments, enabling efficient event collection with minimal execution overhead. Users can simultaneously run binary tracing alongside legacy text-based logging.

PiTrace enables tracing for IS-IS protocol events, BGP and routing protocol daemon (rpd) high-CPU conditions, and historical operational data for RSVP and MPLS. Administrators can selectively disable IS-IS pixel tracing for individual routing or protocol instances.

Benefits of Always-on PiTrace Binary Tracing

  • Preserves recent protocol activity to support faster post-incident troubleshooting and root-cause analysis.

  • Reduces operational overhead compared to verbose text-based tracing while maintaining diagnostic visibility.

  • Improves troubleshooting of intermittent and transient issues that are difficult to reproduce.

  • Provides visibility into routing protocol and rpd high-CPU activity during sustained CPU spikes.

  • Extends troubleshooting visibility for RSVP and MPLS by preserving historical operational command data.

  • Minimizes trace noise and reduces storage usage through selective trace filtering for IS-IS instances

  • Enables correlation of protocol transitions, operational workflows, and system events over time.

How PiTrace Works

When enabled, PiTrace continuously generates compact binary trace records from routing protocol and operational workflows. Protocols write trace information into in-memory buffers through the PiTrace infrastructure

A writer thread collects buffered trace records and writes them to PiTrace files stored under the /var/log/pitrace directory. Trace data is flushed to disk when memory thresholds are reached, or through manual flush operations.

The tracing framework maintains a rolling operational history that can be analyzed after an incident occurs. This approach supports troubleshooting of intermittent or transient issues that might otherwise be missed using reactive debugging methods.

Before collecting logs, you can manually flush in-memory trace buffers to disk to ensure that the latest protocol events are included in the trace files.

Supported Protocol and Operational Contexts

IS-IS

Records binary trace events for IS-IS control-plane transitions and operational workflows, including events such as LSP change processing, SPF execution, route download activity, interface state transitions, overload conditions, and error events.

Multiple IS-IS instancs can contribute to the same tracing stream unless selective trace filtering is configured. Tracing can also be disabled selectively for individual routing or protocol instances.

BGP and rpd High-CPU Events

Captures job-level execution details during sustained high-CPU conditions in the routing protocol daemon (rpd). Trace data helps correlate CPU spikes with routing activity and scheduled protocol jobs, including high CPU-consuming BGP operations.

RSVP and MPLS

Preserves historical RSVP and MPLS operational command data in PiTrace files to extend troubleshooting visibility beyond limited in-memory operational history buffers.

Supported operational command history includes RSVP session, neighbor, and interface operational data, along with MPLS LSP operational state information.

Trace Configuration

PiTrace tracing levels can be configured under the system pixel-tracing hierarchy.

Supported tracing level include:

  • any

  • critical

  • debug

  • info

  • none

In 64-bit RPD environments, pixel tracing for Info and Critical events is enabled by default across all modules. No explicit configuration is required to enable these levels.

You can also configure the maximum number of retained PiTrace dump files per tracing context. By default, up to 20 PiTrace files are retained for each context.

IS-IS Trace Control

You can selectively disable PiTrace tracing for specific IS-IS protocol or routing instances.

Disabling tracing for one IS-IS instance does not affect tracing for other instances.

To disable pixel tracing, use the following command:

Manual Trace Flush

Use the following command to flush in-memory trace information to PiTrace files on disk:

This operation ensures that the latest protocol events are written to trace files before log collection or analysis.