Using the Services SDK Data Handling Functions

The data handling functions allow applications running on the data CPU to send and receive packets.

The system saves packets as messages in a FIFO queue that you can (but do not have to) access.

Overview of the Services SDK Data Functionality

The data component of an application running on the Multiservices PIC runs a packet loop to send and receive data; this is also known as a data loop.

A data loop is an abstraction for the following combination:

These components are handled automatically when you call the functions. You do not need to interact with them directly except in special cases.

The FIFO queue is a "First In First Out" data structure, used to send and receive messages. A message can be a network packet or any other type of data. A type attribute in the FIFO functions allows you to differentiate among different types of messages.

Overview of the Services SDK Functions

The Services SDK functions fall into these categories:

A typical application will use a subset of the functions. In particular, the msp_fifo, msp_env, and mp_process functions are provided for special cases where customization or performance tuning are needed.

Using the msp_data and msp_fifo Functions

To send or receive a packet using the Services SDK library, the first step is to create the data loop. You have three options for doing this; which option you choose depends on the degree of customization you need.

If you are using the default round-robin packet distribution (see Flow Affinity), there is no reason to use the other options, unless the default FIFO size is not sufficient for your needs (you can set it using the third option).

Both the second and third options allow customizations per CPU and FIFO. This allows your code to differentiate between threads and data loops. For example, if you can predict that certain types of traffic will be handled by a particular thread, you can customize that thread for that particular traffic. While this determination is nontrivial, it is possible.

Creating a Data Loop on All CPUs

Creating a data loop on all CPUs is the simplest approach. The msp_data_create_loops() function call creates three data loops per data core and provides the maximum parallelism. All the loops are started with the same parameters, and they all use the same start function.

To create a data loop on all CPUs, you first define a packet-processing function and then create a loop that calls the function. To pass application-specific data to the loop, you set the app_data field in the dloop_params_t structure, declared in the Services SDK header file.

For example, this code sends the gateway application data stored in the app_cb structure and calls the function jnx_gw_data_process_packet() in the loop:

bzero(&dloop_params, sizeof(msp_dataloop_params_t));

dloop_params.app_data = app_cb;

if (msp_data_create_loops((msp_data_loop_t )jnx_gw_data_process_packet,
 &dloop_params) != msp_OK)
    {
      jnx_gw_log(LOG_INFO, "Data Agent packet processing loop create fail");
      goto cleanup_threads;
    }
...

Creating a Data Loop on a Specific CPU

Creating a data loop on a specific CPU allows you to specify a data CPU number, a start function, and optional parameters. You can create from one to three data loops per data core. This method gives you control over the degree of parallelism and performance. Using this method, your code must iterate over the data CPUs.

The sample code shown next calls the packet-processing function packet_loop(), creates the loop on the next available data CPU, and passes user data to the loop:

#if (CREATE_LOOP_OPT == OPT_CREATE_LOOP_ON_CPU)
static int loop_user_data[MSP_MAX_CPUS]; // user data passed to data loop
#endif

...

static int
create_loops(void)
{
    int cpu_num;
    msp_dataloop_params_t loop_option;
    msp_dataloop_result_t loop_result;
    int ret;

    bzero(&loop_option, sizeof(loop_option));
    bzero(&loop_result, sizeof(loop_result));

    cpu_num = MSP_NEXT_NONE;
    while(1) {
        cpu_num = msp_env_get_next_data_cpu(cpu_num);
        if(cpu_num == MSP_NEXT_END) {
            break;
        }

        loop_user_data[cpu_num] = cpu_num;
        loop_option.app_data = &loop_user_data[cpu_num];

        /* Create loop with user data. */
        ret = msp_data_create_loop_on_cpu(cpu_num, packet_loop,
                &loop_option, &loop_result);
        loop_handle[cpu_num] = loop_result.dhandle;
        if(ret != MSP_OK) {
            syslog(LOG_ERR,
                    "%s: Create loop on CPU %d FAILED! err: %d",
                    __FUNCTION__, cpu_num, ret);
        }
    }
    return MSP_OK;
}

Creating a Data Loop as a Thread

The most flexible approach, as well as the approach that requires the most setup code, is to create the data loop and FIFO manually. As in the second option, you must iterate over the data CPUs. You can specify a data CPU number, a start function, and optional parameters. You can create from one to three data loops per data core. This method also gives you control over the degree of parallelism and performance.

The msp_fifo_create_fifo() function used here allows you to control the depth of each FIFO. The default FIFO depth is 1023. The pthread_attr_setcpuaffinity_np() function attaches a pthread to a hardware thread and gives you control over other thread attributes. (The pthread functions are part of the FreeBSD libraries included in JUNOS.)

To create a thread on a CPU, you first find an available CPU and then create a FIFO for the CPU you retrieved. The following code from a packet loop uses the Services SDK functions to do this. It references the constants MSP_NEXT_NONE (initial value) and MSP_NEXT_END (final value) that your code can use to iterate on the next available CPU or core. These and other constants are defined in the header file /src/junos/lib/libmp-sdk/h/jnx/mpsdk.h.

 cpu_num = MSP_NEXT_NONE;
         while(1) {
             /* Retrieve the next data CPU number. */
             cpu_num = msp_env_get_next_data_cpu(cpu_num);
             if(cpu_num == MSP_NEXT_END) {
                 break;
             }

Next, within the same loop, the code calls its own function to create the FIFO:

             /* Create FIFO for data CPU. */
             bzero(&fifo_create, sizeof(fifo_create));
             fifo_create.fifo_depth = MSP_FIFO_DEFAULT_DEPTH;
             ret = msp_fifo_create_fifo(cpu_num, &fifo_create);

...

The application also creates a thread. It uses the packet_thread_arg_t structure, which it declares as follows (msp_fifo_handle_t is the msp FIFO identifier):

typedef struct packet_thread_arg_s {
         msp_fifo_handle_t handle;
         int cpu_num;
     } packet_thread_arg_t;

The following code creates the thread. The msp_fifo_create_t structure is declared in the mpsdk.h header file and used in this code:

typedef struct msp_fifo_create_s {
         msp_fifo_handle_t fhandle;     /* FIFO identifier                */
         int               fifo_depth;  /* Maximum messages FIFO can hold */
    } msp_fifo_create_t;

The following code calls setup functions to create the pthread and finally calls the msp_fifo_register_pthread() function to register the thread with the FIFO.

             pthread_attr_t attr;
             pthread_t tid;
             packet_thread_arg_t *thread_arg = NULL;
...
           /* Create thread parameter. */
             thread_arg = malloc(sizeof(*thread_arg));
...
             thread_arg->handle = fifo_create.fhandle;
             thread_arg->cpu_num = cpu_num;

             
            pthread_attr_init(&attr);

            if(pthread_attr_setscope(&attr, PTHREAD_SCOPE_SYSTEM)) {
               syslog(LOG_ERR,
               "%s: pthread_attr_setscope() on CPU %d FAILED!",
               __FUNCTION__, cpu_num);
              }
             else if(pthread_attr_setcpuaffinity_np(&attr, cpu_num)) {
               syslog(LOG_ERR,
               "%s: pthread_attr_setcpuaffinity_np() on CPU %d FAILED!",
                __FUNCTION__, cpu_num);
              }
             else if (pthread_create(&tid, &attr,
              (void *)&packet_thread_loop, thread_arg)) {
               syslog(LOG_ERR,
               "%s: pthread_create() on CPU %d FAILED!",
               __FUNCTION__, cpu_num);
            }
           else if(msp_fifo_register_pthread(fifo_create.fhandle, tid)) {
              syslog(LOG_ERR,
             "%s: msp_fifo_register_pthread() on CPU %d FAILED!",
             __FUNCTION__, cpu_num);
}

Additional Notes

Some additional suggestions for coding your data loop handler are as follows:


© 2007-2009 Juniper Networks, Inc. All rights reserved. The information contained herein is confidential information of Juniper Networks, Inc., and may not be used, disclosed, distributed, modified, or copied without the prior written consent of Juniper Networks, Inc. in an express license. This information is subject to change by Juniper Networks, Inc. Juniper Networks, the Juniper Networks logo, and JUNOS are registered trademarks of Juniper Networks, Inc. in the United States and other countries. All other trademarks, service marks, registered trademarks, or registered service marks are the property of their respective owners.
Generated on Sun May 30 20:26:47 2010 for Juniper Networks Partner Solution Development Platform JUNOS SDK 10.2R1 by Doxygen 1.4.5