Locking

Performance can be affected when a process creates, writes, reads, or destroys data from different threads, due to the potential requirement for locking to serialize access to the data.

When the data is accessed from real-time priority (rtprio) threads (that is, threads bound to user CPUs) or from data threads (threads bound to data CPUs) which also run in real time, you can use spinlocks to access your data with maximum efficiency.

Spinlocks are lightweight and do not cause the thread acquiring the lock to sleep when it is already locked by another thread. Instead, spinlocks make the thread spin in a loop, trying to acquire the lock until the lock becomes available. This is ideal for real-time priority threads that cannot be preempted because they always have exclusive access to a CPU.

Handling Threads on Control CPUs

If the same data needs to be accessed by a thread that is not running in real time (that is, by a thread on a control CPU) using a spinlock could pose a problem. Such threads share the control CPUs and are scheduled by the operating system. If a spinlock is acquired by such a thread, there is no guarantee that the operating system will not preempt it, putting it to sleep temporarily while another thread can use the CPU. If this preemption happened while holding a spinlock, this could lead to an inefficient situation where other real-time threads could be waiting for the lock held by a thread that is not even running. For real-time threads such as data threads that poll and process packets, this long of a wait for a lock could cause the packet FIFOs to become excessively backed up, and the system could eventually drop packets. This is only one example of an undesired situation stemming from the use of spinlocks by non-real-time threads.

To deal with this problem, threads on control CPUs can use a user-level critical section to overcome the problem of getting preempted. While in a critical section, the operating system raises the priority of the thread high enough so that it will not be preempted by the scheduler. The JUNOs SDK includes functions to allow a thread on a control CPU to simultaneously enter a critical section and acquire a spinlock. These represent the only proper way to acquire a spinlock from a non-real-time thread.

To use the user-level critical section functions, the thread that typically runs over all control CPUs must be bound to run on a single control CPU. This does not guarantee exclusive access to that control CPU, but it guarantees that the given thread cannot be scheduled to run on any other control CPUs, as is normally the case. Critical section usage must be for short, infrequent usage. (If this is violated, your code could get a SIGCPU or be killed.)

The functions are at sandbox/src/junos/libmp-sdk/h/jnx/msp_locks.h and sandbox/src/junos/lib/libmp-sdk/h/jnx/msp_ucrit.h, and are documented in the Library Reference. For sample code, see the programming task Working with Spinlocks and User-Level Critical Sections.

Note:

To bind a thread to a CPU, use the functions described in Flow, Thread, and Session Affinities. Any thread you create automatically runs as a control thread across the control CPUs unless you use specific flow (CPU) affinity functions.

Using rtprio Threads on User CPUs

Aside from data threads (a thread attached to a data CPU), which are always real-time threads, a real-time priority (rtprio) thread is tied to one of the user CPUs on the Multiservices PIC.

You can manually bind a thread to a user CPU, or create a thread and assign it to a user CPU by calling pthread_attr_setcpuaffinity_np(). There are SDK functions to facilitate the latter as well.

When you call the msp_rtprio_thread_create() function to create an rtprio thread, the system creates a pthread, bound to a user hardware thread (CPU), to run your code. The system also sets the scheduling policy so the thread runs without preemption.

There can be no more than one rtprio thread attached to the same user CPU.

The rtprio thread functions are in sandbox/src/junos/lib/libmp-sdk/h/jnx/msp_rtprio_thread.h in your backing sandbox, and are documented in the SDK Library Reference.

---