Per-CPU Data

Per CPU data. More...

Collaboration diagram for Per-CPU Data:

Detailed Description

Per CPU data.

In the x86 architecture the gs and fs segment registers can be used to access data relative to the address stored in the MSR_GS_BASE or MSR_FS_BASE MSRs. In AT&T assembly this would look like this:

mov %gs:0x10, %rax ; Load the value at address in `MSR_GS_BASE` + 0x10 into rax

This means that, since each cpu has its own MSR_GS_BASE, we can store the address of each CPU's structure in its own MSR_GS_BASE and then access data within that structure using offsets.

Allocating a percpu variable then becomes as simple as allocating an offset within the percpu buffer in the CPU structure, and accessing it using the gs segment register.

Note

Its important to be aware of the distinction that the gs register does not store an address directly, rather it allows us to access memory relative to the address stored in the MSR_GS_BASE MSR. This is why we define Per-CPU variables as offsets within the CPU structure rather than absolute addresses.

Defining Per-CPU Variables

To define a Per-CPU variable use the PERCPU_DEFINE() macro. This will add a percpu_def_t entry to the ._percpu section. The PERCPU_INIT()macro can be used to allocate and initialize all Per-CPU variables defined in the module's._percpu` section, potentially invoking any needed constructors.

Note

All percpu variables should use the pcpu_ prefix for clarity.

Constructors and Destructors

All Per-CPU variables can optionally have constructors (ctor) and destructors (dtor) defined. These will be called on each CPU either during boot, when the CPU is initialized, or via a call to percpu_update().

By default, all variables are zero-initialized when allocated.

Data Structures
struct	percpu_def_t
	Structure to define a percpu variable. More...

Macros
#define	PERCPU_ALIGNMENT   64
	The PERCPU_ALIGNMENT constant defines the alignment for per-CPU variables.
#define	PERCPU   __seg_gs*
	Attribute specifying that the variable is an offset into the GS segment register.
#define	SELF   ((cpu_t PERCPU)0)
	Macro to access data in the current cpu.
#define	SELF_PTR(ptr)   ((void*)((uintptr_t)(SELF->self->percpu) + ((uintptr_t)(ptr) - offsetof(cpu_t, percpu))))
	Macro to get a pointer to a percpu variable on the current CPU.
#define	CPU_PTR(id, ptr)    ((void*)((uintptr_t)(cpu_get_by_id(id)->percpu) + ((uintptr_t)(ptr) - offsetof(cpu_t, percpu))))
	Macro to get a pointer to a percpu variable on a specific CPU.
#define	PERCPU_DEFINE(type, name, ...)
	Macro to define a percpu variable.
#define	PERCPU_DEFINE_CTOR(type, name)
	Macro to define a percpu variable with a constructor.
#define	PERCPU_DEFINE_DTOR(type, name)
	Macro to define a percpu variable with a destructor.
#define	PERCPU_INIT()
	Initialize all percpu variables within the current modules .percpu section.
#define	PERCPU_DEINIT()
	Deinitialize all percpu variables within the current modules .percpu section.

Typedefs
typedef size_t	percpu_t
	The type that the compiler uses to store per-CPU variables.

Functions
void	percpu_init (cpu_t *cpu)
	Initialize the percpu system.
percpu_t	percpu_alloc (size_t size)
	Allocates a percpu variable.
void	percpu_free (percpu_t ptr, size_t size)
	Frees a percpu variable.
void	percpu_update (void)
	Update percpu sections on the current CPU.
void	percpu_section_init (percpu_def_t *start, percpu_def_t *end)
	Register a percpu section and run constructors.
void	percpu_section_deinit (percpu_def_t *start, percpu_def_t *end)
	Unregister a percpu section and run destructors.

Macro Definition Documentation

PERCPU_ALIGNMENT

#define PERCPU_ALIGNMENT   64

The PERCPU_ALIGNMENT constant defines the alignment for per-CPU variables.

Note

This value should be a power of 2.

Definition at line 52 of file percpu.h.

PERCPU

#define PERCPU   __seg_gs*

Attribute specifying that the variable is an offset into the GS segment register.

Definition at line 75 of file percpu.h.

SELF

#define SELF   ((cpu_t PERCPU)0)

Macro to access data in the current cpu.

Intended to be used as a pointer to the current cpu structure.

Warning

The value of this macro is not the address of the current cpu structure, to actually retrieve the address use SELF->self.

Definition at line 85 of file percpu.h.

SELF_PTR

#define SELF_PTR

(

ptr

)

((void*)((uintptr_t)(SELF->self->percpu) + ((uintptr_t)(ptr) - offsetof(cpu_t, percpu))))

Macro to get a pointer to a percpu variable on the current CPU.

Parameters

ptr	The percpu variable pointer.

Returns

A pointer to the percpu variable for the current CPU.

Definition at line 93 of file percpu.h.

CPU_PTR

#define CPU_PTR	(		id,
			ptr
	)		((void*)((uintptr_t)(cpu_get_by_id(id)->percpu) + ((uintptr_t)(ptr) - offsetof(cpu_t, percpu))))

Macro to get a pointer to a percpu variable on a specific CPU.

Parameters

id	The ID of the CPU.
ptr	The percpu variable pointer.

Returns

A pointer to the percpu variable for the specified CPU.

Definition at line 102 of file percpu.h.

PERCPU_DEFINE

#define PERCPU_DEFINE	(		type,
			name,
			...
	)

Value:

    type PERCPU name; \
    static const percpu_def_t __attribute__((used, section("._percpu"))) _percpu##name = {.ptr = (percpu_t*)&(name), \
        .size = sizeof(typeof(*name)), \
        ##__VA_ARGS__}

Macro to define a percpu variable.

This macro defines a percpu variable and registers it in the ._percpu section, so that it can be initialized by PERCPU_INIT().

Parameters

type	The type of the percpu variable.
name	The name of the percpu variable.
...	Optional constructor and destructor functions for the percpu variable.

Definition at line 115 of file percpu.h.

PERCPU_DEFINE_CTOR

#define PERCPU_DEFINE_CTOR	(		type,
			name
	)

Value:

    static void name##_ctor(void); \
    PERCPU_DEFINE(type, name, .ctor = name##_ctor, .dtor = NULL); \
    static void name##_ctor(void)

Macro to define a percpu variable with a constructor.

This macro defines a percpu variable with a constructor and registers it in the ._percpu section, so that it can be initialized by PERCPU_INIT().

Parameters

type	The type of the percpu variable.
name	The name of the percpu variable.

Definition at line 130 of file percpu.h.

PERCPU_DEFINE_DTOR

#define PERCPU_DEFINE_DTOR	(		type,
			name
	)

Value:

    static void name##_dtor(void); \
    PERCPU_DEFINE(type, name, .ctor = NULL, .dtor = name##_dtor); \
    static void name##_dtor(void)

Macro to define a percpu variable with a destructor.

This macro defines a percpu variable with a destructor and registers it in the ._percpu section, so that it can be deinitialized by PERCPU_FINIT().

Parameters

type	The type of the percpu variable.
name	The name of the percpu variable.

Definition at line 144 of file percpu.h.

PERCPU_INIT

#define PERCPU_INIT

(

)

Value:

    do \
    { \
        extern percpu_def_t _percpu_start; \
        extern percpu_def_t _percpu_end; \
        percpu_section_init(&_percpu_start, &_percpu_end); \
    } while (0)

Initialize all percpu variables within the current modules .percpu section.

Definition at line 194 of file percpu.h.

PERCPU_DEINIT

#define PERCPU_DEINIT

(

)

Value:

    do \
    { \
        extern percpu_def_t _percpu_start; \
        extern percpu_def_t _percpu_end; \
        percpu_section_deinit(&_percpu_start, &_percpu_end); \
    } while (0)

Deinitialize all percpu variables within the current modules .percpu section.

Definition at line 205 of file percpu.h.

Typedef Documentation

percpu_t

typedef size_t percpu_t

The type that the compiler uses to store per-CPU variables.

Definition at line 57 of file percpu.h.

Function Documentation

percpu_init()

void percpu_init

(

cpu_t *

cpu

)

Initialize the percpu system.

This will setup the gs segment register to point to the CPU structure.

Parameters

cpu	The CPU to initialize percpu for, should be the current CPU.

Definition at line 33 of file percpu.c.

Here is the call graph for this function:

Here is the caller graph for this function:

percpu_alloc()

percpu_t percpu_alloc

(

size_t

size

)

Allocates a percpu variable.

Parameters

size	The size of the percpu variable.

Returns

The offset into the GS segment register, or ERR on failure.

Definition at line 39 of file percpu.c.

Here is the call graph for this function:

Here is the caller graph for this function:

percpu_free()

void percpu_free	(	percpu_t	ptr,
		size_t	size
	)

Frees a percpu variable.

Parameters

ptr	The offset into the GS segment register.
size	The size of the percpu variable.

Definition at line 63 of file percpu.c.

Here is the call graph for this function:

Here is the caller graph for this function:

percpu_update()

void percpu_update

(

void

)

Update percpu sections on the current CPU.

This will run any pending constructors or destructors for percpu sections.

Definition at line 99 of file percpu.c.

Here is the call graph for this function:

Here is the caller graph for this function:

percpu_section_init()

void percpu_section_init	(	percpu_def_t *	start,
		percpu_def_t *	end
	)

Register a percpu section and run constructors.

Definition at line 130 of file percpu.c.

Here is the call graph for this function:

percpu_section_deinit()

void percpu_section_deinit	(	percpu_def_t *	start,
		percpu_def_t *	end
	)

Unregister a percpu section and run destructors.

Definition at line 159 of file percpu.c.

Here is the call graph for this function: