Tutorial on how to add a low level driver to the HPC Event Processor

Source Code

HPC source code is included with Phoenix in a git repository. The simpc-build-instructions explains how to clone the source code. For VxWorks, the source code is located in the components directory.

The root directory of the Phoenix source code repository is referred to here as PH_ROOT.

PH_ROOT/hpc/src: HPC source code
PH_ROOT/hpc/include/hpc: HPC public include files
PH_ROOT/platforms/vxworks/release/osconfig/vxworks/cdf: CDFs
PH_ROOT/platforms/vxworks/release/osconfig/vxworks/src: Configlettes

HPC Overview

Cafe instruments are able to process Hardware Performance Counter (HPC) events from the HPC root Event Processor (EP). The HPC EP includes a driver framework to configure and control low-level hardware performance counter resources. This tutorial describes how to add a driver to this framework. The diagram below shows how the performance counters for the MPC8540 CPU device fit into this framework.

+-----------+                  +------------------+
| hpc_ep.c  |<-register--------| mpc8540EvtCfg.c  |
|           |                  | driver event     |
| HPC Event |                  | configuration    |
| Processor |                  +------------------+
|           |
|           |                  +------------------+
|           |<-register--------| e500CoreEvgCfg.c |
|           |                  | driver event     |
|           |                  | configuration    |
|           |                  +------------------+
|           |
|           |                  +------------------+
|           |----get settings->| e500DrvSet.c     |
|           |    (mpc8540)     | driver event     |
|           |----get settings->| settings         |
|           |    (e500 core)   |                  |
|           |                  +------------------+
|           |
|           |                  +-------------------------+
|           |                  | e500Drv.c               |
|           |                  | HPC low-level driver    |--+
|           |----------------->| hpcHwDriverInit()       |  |
|           |----------------->| hpcHwCreateCtrForEvent()|  |
|           |----------------->| hpcHwCtrStart()         |  |
|           |----------------->| hpcHwCtrStop()          |  |
|           |----------------->| hpcHwDeleteCtrForEvent()|  |
|           |----------------->| hpcHwDriverFinit()      |  |
|           |                  |                         |  |
|           |  perf            |                         |  |
|           |<-interrupt-------|                         |  |
|           |  callback        |                         |  |
|           |                  |    e500 core instance   |  |
+-----------+                  +-------------------------+  |
                                 | mpc8540 device instance  |
                                 +--------------------------+

The MPC8540 has two sources for performance counters: the core and the device. The hardware resource for each one uses the same register model, the only difference being the number of counters and the register access method. Registers for the core counters are Special Purpose Registers and are accessed using mtspr and mfspr instructions, whilst registers for the device counters are memory mapped. Because of this similarity, a common driver can be used, namely e500Drv.c. Two instances for the driver are used, one for the e500 core performance monitor and one for the mpc8540 device performance monitor. Each instance has its own driver event configuration which are both registered with the HPC EP.

At initialization the driver event configurations are registered with the HPC EP. Once registered, instruments can then request hpc events from the HPC EP. This results in the HPC EP configuring the requested event by calling the appropriate driver instance and driver instance routines. The low-level hardware counter for the event will start counting and at some point in time will cause a performance counter overflow interrupt. The interrupt handler in the driver will make a call to the interrupt handler callback in the HPC EP which will send the event to other EPs in the instrument chain.

Driver and Event Configuration

The driver event source must create a struct hpcDriverConfig, which is used to register the driver with the HPC EP (refer to hpcDrvEvtCfg.h). The registration is done using function hpc_hw_driver_register(). Refer to mpc8540EvtCfg.c and e500CoreEvtCfg.c for two event configurations. Note that hpc_driver_register() is called for each driver configuration.

Define the Number of CPU Cores: numCpuCores

The HPC EP needs to know if the performance counters are per core or per device. For example, on the MPC8540, the core counters are per core (so numCpuCores is set to the number of E500 cores) and the device counters are per device (so numCpuCores is set to 1). This allows the HPC EP to control which events are counted on each core.

Define the Events: pEventDesc

This structure member is a pointer to a list of all the events. Note that there are common named events, like hpc.total_cycles. The common named events must have the same value type string across all CPUs. Not all CPUs will support all of the common event names. The array also lists the hardware specific event names, which follow a naming convention: hpc.[CPU name]_[vendor event name]. For example: hpc.e500_load_micro_ops_completed and hpc.mpc_L2_allocates_from_any_source.

Define the Low-Level Driver Functions: pFuncs

This structure member points to the low-level driver functions that the HPC EP uses to configure and control the CPUs hardware performance registers. The functions are defined in the low-level driver header include/hpc/hpcHwDrv.h. For the e500 driver, this is in file e500Drv.c.

Define the Low-Level Driver Instance Data: pDrvInst

The driver configuration can also define a pointer to a low-level driver instance data area. The e500 driver defines struct driverInstE500 for this. A pointer to this data structure is passed to the low-level driver API as parameter void *pDrvInst. The low-level driver uses this data structure for anything that it wants.

Define the Get Settings Function: pEventSettingGet

The driver must define a function to decode the event settings. For the e500 driver, this function is in file e500DrvSet.c.

Lowest Level: The HPC Hardware Driver

The low-level HPC hardware driver is described by the file include/hpc/hpcHwDrv.h, it defines struct hpcHwDriverFuncs. This structure contains the function pointer bindings used by the HPC EP to control the hardware. They are used in the following way. In this example the driver is configured to support two cores. The idea below is to show in what order the API functions are called, so most of the parameters are missing for clarity. HCD is the Hardware Counter Data handle. The performance counter interrupt callback is also not shown.

/* driver init */
hpcHwDriverInit()
hpcHwPerCoreInit()  /* called on core 1 */
hpcHwPerCoreInit()  /* called on core 2 */

    /* create and configure two events */
    hpcHwCreateCtrForEvent(HCD1)    /* called on core 1 */
    hpcHwCreateCtrForEvent(HCD1)    /* called on core 2 */
    hpcHwCreateCtrForEvent(HCD2)    /* called on core 1 */
    hpcHwCreateCtrForEvent(HCD2)    /* called on core 2 */

        /* Start both events counters */
        hpcHwCtrStart(HCD1) /* called on core 1 */
        hpcHwCtrStart(HCD1) /* called on core 2 */
        hpcHwCtrStart(HCD2) /* called on core 1 */
        hpcHwCtrStart(HCD2) /* called on core 2 */

        /* Read both event counters */
        hpcHwCtrRead(HCD1)  /* called on core 1 */
        hpcHwCtrRead(HCD1)  /* called on core 2 */
        hpcHwCtrRead(HCD2)  /* called on core 1 */
        hpcHwCtrRead(HCD2)  /* called on core 2 */

        /* Stop both event counters */
        hpcHwCtrStop(HCD1)  /* called on core 1 */
        hpcHwCtrStop(HCD1)  /* called on core 2 */
        hpcHwCtrStop(HCD2)  /* called on core 1 */
        hpcHwCtrStop(HCD2)  /* called on core 2 */

    hpcHwDeleteCtrForEvent (HCD1)   /* called on core 1 */
    hpcHwDeleteCtrForEvent (HCD1)   /* called on core 2 */
    hpcHwDeleteCtrForEvent (HCD2)   /* called on core 1 */
    hpcHwDeleteCtrForEvent (HCD2)   /* called on core 2 */

hpcHwDriverFinit()
hpcHwPerCoreFinit()     /* called on core 1 */
hpcHwPerCoreFinit()     /* called on core 2 */

VxWorks HPC Components

INCLUDE_HPC_E500_CORE: Enables E500 core perf counters.

INCLUDE_HPC_MPC8540_DEVICE: Enables MPC8540 device perf counters.

INCLUDE_HPC_FSL_P2020_DEVICE: Enables FSL P2020 device perf counters.

INCLUDE_HPC_I86_CORE2: Enables Intel Core2 perf counters.

And for testing:

INCLUDE_ANALYSIS_HPC_API_TEST_HW: Enables driver API Test Suite.

INCLUDE_ANALYSIS_HPC_TEST_HW: Enables HPC EP Driver Configuration Test Suite.

Testing

The HPC EP and driver framework (and all of Cafe project code) has been developed using a Linux machine with native compilation. This allows the code to be tested using Valgrind, which serves as a very useful debugging tool for catching memory corruption. Obviously, the code also compiles for VxWorks and the same set of tests execute in both environments.

To execute the tests on a Linux host struct hpcHwDrvTestFuncs must be defined for the driver configuration. For the E500 driver refer to e500DrvTest.c. These functions allow the test code to increment counters and generate the performance counter interrupt callback. In order to do this the targets hardware performance counter register model is simulated (refer to the code in the *_stub.c files).

On Linux native builds, all tests can be run as follows:

$ cd PH_ROOT
$ mkdir unix-build
$ cd unix-build

for release builds:
$ cmake ..

or for debug builds:
$ cmake -DCMAKE_BUILD_TYPE=Debug ..

$ make all test

The individual HPC tests can be run as follows:

$ ./bin/cafe_hpc_drv_config_unit_test_suite
$ ./bin/cafe_hpc_drv_unit_test_suite
$ ./bin/cafe_hpc_unit_test_suite

PH_ROOT/hpc/tests_driver (cafe_hpc_drv_unit_test_suite)

This is the low-level HPC driver API test suite. The driver API test suite does not require the driver event configuration as it solely tests the driver low-level API.

On Linux Native Builds

For Linux, the tests are compiled to the executable bin/cafe_hpc_drv_unit_test_suite. New drivers are added to hpc_test_drv_main.c.

On VxWorks

On VxWorks, the test entry function is called hpcHwDrvUnitTestSuiteHw(). New drivers are added to the test harness by adding a file like hpc_test_drv_hw_mpc8540.c. Initialization for the test is required in the hpcInit.c configlette. To include the test in the VxWorks VIP, component INCLUDE_ANALYSIS_HPC_API_TEST_HW is required. The tests can be run from the VxWorks target shell. For example:

-> sp hpcHwDrvUnitTestSuiteHw
Task spawned: id = 0x48bf9d0, name = t1
value = 76282320 = 0x48bf9d0
hpc_driver_utils_suite: tests-run:  1, assertions:     5, passes:   1, failures:   0
Start of big loop...
Counter 0 20348574274
Start of big loop...
Counter 0 125677
    hpc_driver_api_suite: tests-run:  1, assertions:    23, passes:   1, failures:   0
Start of big loop...
Counter 0 20352534983
Counter 1 13217135768
Counter 2 13218170409
Start of big loop...
Counter 0 1019f5
Counter 1 5193f
Counter 2 60458
    hpc_driver_api_suite: tests-run:  1, assertions:    41, passes:   1, failures:   0
         Overall Results: tests-run:  3, assertions:    78, passes:   3, failures:   0
value = 0 = 0x0
->

PH_ROOT/hpc/tests (cafe_hpc_drv_config_unit_test_suite)

This is the HPC EP driver configuration test suite. The HPC EP test suite tests the driver and event configuration using the HPC EP. The test suite requires a new TEST_*_BOARD to be added to hpc_test.h. It also requires support code adding to hpc_test_init.c.

On Linux Native Builds

For Linux, the tests are compiled to the executable bin/cafe_hpc_unit_test_suite.

On VxWorks

On VxWorks, the name of the test suite is hpc_unit_test_hw. A new driver will require a new initialization function, like hpc_init_mpc_8540_tests() to be added to hpc_main2.c. This function is called from the hpcInit.c configlette. To include the test in the VxWorks VIP, component INCLUDE_ANALYSIS_HPC_TEST_HW is required. The tests can be run from the VxWorks target shell. For example:

-> sp hpc_unit_test_hw
Task spawned: id = 0x48bf9d0, name = t1
value = 76282320 = 0x48bf9d0
TEST_VXW_HPC_HW_DRV
HPC ERROR:364 unable to alloc memory for virtual counters 89612
HPC ERROR:496 can't free vc as some are still in use
HPC ERROR:364 unable to alloc memory for virtual counters 2012
HPC ERROR:398 couldn't allocate virtual counters 2
HPC ERROR:434 given invalid vc to free 0x45a0010
HPC ERROR:422 free_vc error, called with NULL
HPC ERROR:675 can't cleanup ep_inst_list as some are still in the list
HPC ERROR:622 Can't allocate memory for configuration 20
HPC ERROR:650 given invalid hpc_inst to free 0x4598dcc
HPC ERROR:638 free_ep_inst error, called with NULL
Start of big loop...
P[hpc.x86archpm1_BRANCH_MISSES_RETIRED]:                        21822422
P[hpc.x86archpm1_BRANCH_INSTRUCTION_RETIRED]:                   1954151521
P[hpc.x86archpm2_CPU_CLK_UNHALTED__CORE]:                       8694985060
P[hpc.x86archpm2_CPU_CLK_UNHALTED__REF]:                        11985777042
P[hpc.total_instructions]:                      11987033195
         hpc_suite: tests-run: 15, assertions:   707, passes:  15, failures:   0
   Overall Results: tests-run: 15, assertions:   707, passes:  15, failures:   0

Note that the errors above are not actual errors, the unit test suite is testing the error path of the code.

Filename Convention

The following naming convention has been used:

  1. *Drv.c: HPC low-level driver.
  2. *EvtCfg.c: HPC Driver and Event configuration.
  3. *DrvSet.c: HPC Driver Event Settings.
  4. *DrvTest.c: HPC Driver Test Support Code.