HPC source code is included with Phoenix in a git repository. The simpc-build-instructions explains how to clone the source code. For VxWorks, the source code is located in the components directory.
The root directory of the Phoenix source code repository is referred
to here as PH_ROOT
.
PH_ROOT/hpc/src: HPC source code
PH_ROOT/hpc/include/hpc: HPC public include files
PH_ROOT/platforms/vxworks/release/osconfig/vxworks/cdf: CDFs
PH_ROOT/platforms/vxworks/release/osconfig/vxworks/src: Configlettes
Cafe instruments are able to process Hardware Performance Counter (HPC) events from the HPC root Event Processor (EP). The HPC EP includes a driver framework to configure and control low-level hardware performance counter resources. This tutorial describes how to add a driver to this framework. The diagram below shows how the performance counters for the MPC8540 CPU device fit into this framework.
+-----------+ +------------------+
| hpc_ep.c |<-register--------| mpc8540EvtCfg.c |
| | | driver event |
| HPC Event | | configuration |
| Processor | +------------------+
| |
| | +------------------+
| |<-register--------| e500CoreEvgCfg.c |
| | | driver event |
| | | configuration |
| | +------------------+
| |
| | +------------------+
| |----get settings->| e500DrvSet.c |
| | (mpc8540) | driver event |
| |----get settings->| settings |
| | (e500 core) | |
| | +------------------+
| |
| | +-------------------------+
| | | e500Drv.c |
| | | HPC low-level driver |--+
| |----------------->| hpcHwDriverInit() | |
| |----------------->| hpcHwCreateCtrForEvent()| |
| |----------------->| hpcHwCtrStart() | |
| |----------------->| hpcHwCtrStop() | |
| |----------------->| hpcHwDeleteCtrForEvent()| |
| |----------------->| hpcHwDriverFinit() | |
| | | | |
| | perf | | |
| |<-interrupt-------| | |
| | callback | | |
| | | e500 core instance | |
+-----------+ +-------------------------+ |
| mpc8540 device instance |
+--------------------------+
The MPC8540 has two sources for performance counters: the core and the
device. The hardware resource for each one uses the same register
model, the only difference being the number of counters and the
register access method. Registers for the core counters are Special
Purpose Registers and are accessed using mtspr and mfspr instructions,
whilst registers for the device counters are memory mapped. Because of
this similarity, a common driver can be used, namely e500Drv.c
. Two
instances for the driver are used, one for the e500 core performance
monitor and one for the mpc8540 device performance monitor. Each
instance has its own driver event configuration which are both
registered with the HPC EP.
At initialization the driver event configurations are registered with the HPC EP. Once registered, instruments can then request hpc events from the HPC EP. This results in the HPC EP configuring the requested event by calling the appropriate driver instance and driver instance routines. The low-level hardware counter for the event will start counting and at some point in time will cause a performance counter overflow interrupt. The interrupt handler in the driver will make a call to the interrupt handler callback in the HPC EP which will send the event to other EPs in the instrument chain.
The driver event source must create a struct hpcDriverConfig
, which
is used to register the driver with the HPC EP (refer to
hpcDrvEvtCfg.h
). The registration is done using function
hpc_hw_driver_register()
. Refer to mpc8540EvtCfg.c
and
e500CoreEvtCfg.c
for two event configurations. Note that
hpc_driver_register()
is called for each driver configuration.
numCpuCores
The HPC EP needs to know if the performance counters are per core or
per device. For example, on the MPC8540, the core counters are per
core (so numCpuCores
is set to the number of E500 cores) and the
device counters are per device (so numCpuCores
is set to 1). This
allows the HPC EP to control which events are counted on each core.
pEventDesc
This structure member is a pointer to a list of all the events. Note that there are common named events, like hpc.total_cycles. The common named events must have the same value type string across all CPUs. Not all CPUs will support all of the common event names. The array also lists the hardware specific event names, which follow a naming convention: hpc.[CPU name]_[vendor event name]. For example: hpc.e500_load_micro_ops_completed and hpc.mpc_L2_allocates_from_any_source.
pFuncs
This structure member points to the low-level driver functions that
the HPC EP uses to configure and control the CPUs hardware performance
registers. The functions are defined in the low-level driver header
include/hpc/hpcHwDrv.h
. For the e500 driver, this is in file
e500Drv.c
.
pDrvInst
The driver configuration can also define a pointer to a low-level
driver instance data area. The e500 driver defines struct
driverInstE500
for this. A pointer to this data structure is passed to
the low-level driver API as parameter void *pDrvInst
. The low-level
driver uses this data structure for anything that it wants.
pEventSettingGet
The driver must define a function to decode the event settings. For
the e500 driver, this function is in file e500DrvSet.c
.
The low-level HPC hardware driver is described by the file
include/hpc/hpcHwDrv.h
, it defines struct hpcHwDriverFuncs
. This
structure contains the function pointer bindings used by the HPC EP to
control the hardware. They are used in the following way. In this
example the driver is configured to support two cores. The idea
below is to show in what order the API functions are called, so most
of the parameters are missing for clarity. HCD is the
Hardware Counter Data handle. The performance counter interrupt
callback is also not shown.
/* driver init */
hpcHwDriverInit()
hpcHwPerCoreInit() /* called on core 1 */
hpcHwPerCoreInit() /* called on core 2 */
/* create and configure two events */
hpcHwCreateCtrForEvent(HCD1) /* called on core 1 */
hpcHwCreateCtrForEvent(HCD1) /* called on core 2 */
hpcHwCreateCtrForEvent(HCD2) /* called on core 1 */
hpcHwCreateCtrForEvent(HCD2) /* called on core 2 */
/* Start both events counters */
hpcHwCtrStart(HCD1) /* called on core 1 */
hpcHwCtrStart(HCD1) /* called on core 2 */
hpcHwCtrStart(HCD2) /* called on core 1 */
hpcHwCtrStart(HCD2) /* called on core 2 */
/* Read both event counters */
hpcHwCtrRead(HCD1) /* called on core 1 */
hpcHwCtrRead(HCD1) /* called on core 2 */
hpcHwCtrRead(HCD2) /* called on core 1 */
hpcHwCtrRead(HCD2) /* called on core 2 */
/* Stop both event counters */
hpcHwCtrStop(HCD1) /* called on core 1 */
hpcHwCtrStop(HCD1) /* called on core 2 */
hpcHwCtrStop(HCD2) /* called on core 1 */
hpcHwCtrStop(HCD2) /* called on core 2 */
hpcHwDeleteCtrForEvent (HCD1) /* called on core 1 */
hpcHwDeleteCtrForEvent (HCD1) /* called on core 2 */
hpcHwDeleteCtrForEvent (HCD2) /* called on core 1 */
hpcHwDeleteCtrForEvent (HCD2) /* called on core 2 */
hpcHwDriverFinit()
hpcHwPerCoreFinit() /* called on core 1 */
hpcHwPerCoreFinit() /* called on core 2 */
INCLUDE_HPC_E500_CORE
: Enables E500 core perf counters.
INCLUDE_HPC_MPC8540_DEVICE
: Enables MPC8540 device perf counters.
INCLUDE_HPC_FSL_P2020_DEVICE
: Enables FSL P2020 device perf counters.
INCLUDE_HPC_I86_CORE2
: Enables Intel Core2 perf counters.
And for testing:
INCLUDE_ANALYSIS_HPC_API_TEST_HW
: Enables driver API Test Suite.
INCLUDE_ANALYSIS_HPC_TEST_HW
: Enables HPC EP Driver Configuration Test Suite.
The HPC EP and driver framework (and all of Cafe project code) has been developed using a Linux machine with native compilation. This allows the code to be tested using Valgrind, which serves as a very useful debugging tool for catching memory corruption. Obviously, the code also compiles for VxWorks and the same set of tests execute in both environments.
To execute the tests on a Linux host struct hpcHwDrvTestFuncs
must be
defined for the driver configuration. For the E500 driver refer to
e500DrvTest.c
. These functions allow the test code to increment counters
and generate the performance counter interrupt callback. In order to
do this the targets hardware performance counter register model is
simulated (refer to the code in the *_stub.c
files).
On Linux native builds, all tests can be run as follows:
$ cd PH_ROOT
$ mkdir unix-build
$ cd unix-build
for release builds:
$ cmake ..
or for debug builds:
$ cmake -DCMAKE_BUILD_TYPE=Debug ..
$ make all test
The individual HPC tests can be run as follows:
$ ./bin/cafe_hpc_drv_config_unit_test_suite
$ ./bin/cafe_hpc_drv_unit_test_suite
$ ./bin/cafe_hpc_unit_test_suite
cafe_hpc_drv_unit_test_suite
)This is the low-level HPC driver API test suite. The driver API test suite does not require the driver event configuration as it solely tests the driver low-level API.
For Linux, the tests are compiled to the executable
bin/cafe_hpc_drv_unit_test_suite
. New drivers are added to
hpc_test_drv_main.c
.
On VxWorks, the test entry function is called
hpcHwDrvUnitTestSuiteHw()
. New drivers are added to the test harness
by adding a file like hpc_test_drv_hw_mpc8540.c
. Initialization for
the test is required in the hpcInit.c
configlette. To include the
test in the VxWorks VIP, component INCLUDE_ANALYSIS_HPC_API_TEST_HW
is
required. The tests can be run from the VxWorks target shell. For
example:
-> sp hpcHwDrvUnitTestSuiteHw
Task spawned: id = 0x48bf9d0, name = t1
value = 76282320 = 0x48bf9d0
hpc_driver_utils_suite: tests-run: 1, assertions: 5, passes: 1, failures: 0
Start of big loop...
Counter 0 20348574274
Start of big loop...
Counter 0 125677
hpc_driver_api_suite: tests-run: 1, assertions: 23, passes: 1, failures: 0
Start of big loop...
Counter 0 20352534983
Counter 1 13217135768
Counter 2 13218170409
Start of big loop...
Counter 0 1019f5
Counter 1 5193f
Counter 2 60458
hpc_driver_api_suite: tests-run: 1, assertions: 41, passes: 1, failures: 0
Overall Results: tests-run: 3, assertions: 78, passes: 3, failures: 0
value = 0 = 0x0
->
cafe_hpc_drv_config_unit_test_suite
)This is the HPC EP driver configuration test suite. The HPC EP test
suite tests the driver and event configuration using the HPC EP. The
test suite requires a new TEST_*_BOARD
to be added to
hpc_test.h
. It also requires support code adding to
hpc_test_init.c
.
For Linux, the tests are compiled to the executable
bin/cafe_hpc_unit_test_suite
.
On VxWorks, the name of the test suite is hpc_unit_test_hw
. A new
driver will require a new initialization function, like
hpc_init_mpc_8540_tests()
to be added to hpc_main2.c
. This
function is called from the hpcInit.c
configlette. To include the
test in the VxWorks VIP, component INCLUDE_ANALYSIS_HPC_TEST_HW
is
required. The tests can be run from the VxWorks target shell. For
example:
-> sp hpc_unit_test_hw
Task spawned: id = 0x48bf9d0, name = t1
value = 76282320 = 0x48bf9d0
TEST_VXW_HPC_HW_DRV
HPC ERROR:364 unable to alloc memory for virtual counters 89612
HPC ERROR:496 can't free vc as some are still in use
HPC ERROR:364 unable to alloc memory for virtual counters 2012
HPC ERROR:398 couldn't allocate virtual counters 2
HPC ERROR:434 given invalid vc to free 0x45a0010
HPC ERROR:422 free_vc error, called with NULL
HPC ERROR:675 can't cleanup ep_inst_list as some are still in the list
HPC ERROR:622 Can't allocate memory for configuration 20
HPC ERROR:650 given invalid hpc_inst to free 0x4598dcc
HPC ERROR:638 free_ep_inst error, called with NULL
Start of big loop...
P[hpc.x86archpm1_BRANCH_MISSES_RETIRED]: 21822422
P[hpc.x86archpm1_BRANCH_INSTRUCTION_RETIRED]: 1954151521
P[hpc.x86archpm2_CPU_CLK_UNHALTED__CORE]: 8694985060
P[hpc.x86archpm2_CPU_CLK_UNHALTED__REF]: 11985777042
P[hpc.total_instructions]: 11987033195
hpc_suite: tests-run: 15, assertions: 707, passes: 15, failures: 0
Overall Results: tests-run: 15, assertions: 707, passes: 15, failures: 0
Note that the errors above are not actual errors, the unit test suite is testing the error path of the code.
The following naming convention has been used:
*Drv.c
: HPC low-level driver.*EvtCfg.c
: HPC Driver and Event configuration.*DrvSet.c
: HPC Driver Event Settings.*DrvTest.c
: HPC Driver Test Support Code.