Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Overview

PCIem is a Linux framework that enables software developers to write PCIe card drivers on the host to target unexisting PCIe cards on the bus.

This helps model a driver if the developers don’t have a physical PCIe card to test it on, effectively enabling pre-silicon engineering on the software side of things for the prototype card.

Initially, PCIem was thought after enabling communications from/to a QEMU instance; but has ever since been evolving to support a wide variety of use-cases.

You can think of the framework as a MITM (Man-in-the-Middle) that sits between the untouched, production drivers (Which are, unaware of PCIem’s existence) and the Linux kernel.

%%{init: {'themeVariables': {'fontSize': '18px'}}}%%
graph TB
    subgraph Kernel ["Host Linux Kernel"]
        direction TB
        RealDriver["Real PCIe Driver"]
        subgraph Framework ["PCIem Framework"]
            direction TB
            Config["PCI Config Space"]
            BARs["BARs"]
            IRQ["Interrupts"]
            DMA["DMA / IOMMU"]
        end
    end
    Interface(("/dev/pciem"))
    subgraph User ["Linux Userspace"]
        direction TB
        Shim["Device Emulation"]
    end
    Framework <==> Interface
    Interface <==> Shim

Getting started

Cloning the repository

In order to use PCIem, we’ll first clone the repository:

git clone https://github.com/cakehonolulu/pciem

With the repository already cloned, we’ll enter it:

cd pciem/

Compiling the code

To compile PCIem, it should be enough to have make, a C compiler & the Linux headers for your currently-running kernel.

Depending on the amount of features your userspace-PCI shim has, you may need further requirements (Think of, displaying a framebuffer the driver writes on with SDL3 or alike); but for a simple one, the aforementioned ones will suffice.

Issuing a compilation is as simple as:

make all

Using PCIem

After we compile everything, we’ll end up with a bunch of object files, an executable and a kernel module.

In short, the way this framework works is by loading the pciem.ko kernel driver (With a certain set of parameters that’ll be discussed below), then loading the userspace-shim that actually creates the device using the exported functionalities of PCIem, and finally; you load the actual (Untouched!) PCIe driver you want to test.

┌───────┐     ┌──────────┐     ┌────────────┐
│ Linux ├─────► pciem.ko ├─────► User-space │
└───────┘     └──────────┘     │    shim    │
                               └────────────┘

PCIem parameters

pciem_phys_region module argument

Let’s start with the preface that, we’re obviously going to use insmod to load pciem.ko, but the way we do it changes the behaviour of the framework.

Starting from PCIem 0.1, one can specify kernel module arguments to alter/instruct certain logic on the code.

pciem_phys_region: This basically specifies what physically-contiguous memory region is reserved and free to use by PCIem.

Reserving memory with memmap command line argument

This is attained by passing the memmap= argument to Linux’s cmdline.

As per kernel.org’s memmap:

memmap=nn[KMG]$ss[KMG]
                        [KNL,ACPI,EARLY] Mark specific memory as reserved.
                        Region of memory to be reserved is from ss to ss+nn.
                        Example: Exclude memory from 0x18690000-0x1869ffff
                                 memmap=64K$0x18690000
                                 or
                                 memmap=0x10000$0x18690000
                        Some bootloaders may need an escape character before '$',
                        like Grub2, otherwise '$' and the following number
                        will be eaten.

This effectively means that, if we append the following to the cmdline:

memmap=128M$0x1bf000000

Linux is going to carve 128M starting from physical address 0x1bf000000 out of the System RAM and mark it as Reserved so PCIem can use it.

With that exact setup, we’d then load PCIem as follows:

sudo insmod kernel/pciem.ko pciem_phys_region="0x1bf000000:0x8000000"

What this does is, tell PCIem that it basically has 128MBs to play starting from physical 0x1bf000000:

Note: On some AArch64 platforms (such as the Raspberry Pi 4B), memmap= may not be honoured by the bootloader or may not be available at all. In those cases, the preferred alternative is to reserve memory via a Device Tree overlay instead.

Reserving memory with a custom device tree overlay

Create a file named reserve-mem.dts with the following contents, adjusting the base address and size to match your desired reservation:

/dts-v1/;
/plugin/;
/ {
    compatible = "brcm,bcm2711";
    fragment@0 {
        target-path = "/";
        __overlay__ {
            #address-cells = <2>;
            #size-cells = <1>;
            reserved-memory {
                #address-cells = <2>;
                #size-cells = <1>;
                ranges;
                pciem_reservation@50000000 {
                    reg = <0x0 0x50000000 0x10000000>;
                    no-map;
                    status = "okay";
                };
            };
        };
    };
};

In this example, 0x50000000 is the base address and 0x10000000 is the size (256MB). Adjust these to fit your target platform’s memory layout.

Compile the overlay and install it:

dtc -@ -I dts -O dtb -o reserve-mem.dtbo reserve-mem.dts
sudo cp reserve-mem.dtbo /boot/firmware/overlays/

Then enable it by appending the following line to /boot/firmware/config.txt:

dtoverlay=reserve-mem

Note: For RPi 4B (Or similar) boards you should add the line within the [all] node.

Reboot for the reservation to take effect. You can verify it was applied by checking:

cat /proc/iomem | grep -i reserved

Once confirmed, load PCIem pointing at the same region specified in the overlay:

sudo insmod kernel/pciem.ko pciem_phys_region="0x50000000:0x10000000"

Note: The method written above has been tested on a Raspberry Pi 4B with the stock Raspbian distribution (64-bit version). Some aarch64 kernels/boards may be fine with memmap= or even mem= (Even though the latest hasn’t been tested) but it’s best to try and decide which one works.

p2p_regions

The p2p_regions argument is a bit more complex; think of it as a whitelist for DMA accesses within PCIem.

The framework supports Peer-to-Peer DMA, but in order to add a little bit of security; one has to manually whitelist the target BAR region of the device we want to P2P from/to.

To know what to “share”, you need to obtain the device’s desired bar start and end address:

$ lspci -vvv
...
f147:00:00.0 System peripheral: Red Hat, Inc. Virtio 1.0 file system (rev 01)
        Subsystem: Red Hat, Inc. Device 0040
        Physical Slot: 2113372925
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64
        NUMA node: 0
        Region 0: Memory at e00000000 (64-bit, non-prefetchable) [size=4K]
        Region 2: Memory at e00001000 (64-bit, non-prefetchable) [size=4K]
        Region 4: Memory at c00000000 (64-bit, non-prefetchable) [size=8G]
        Capabilities: <access denied>
        Kernel driver in use: virtio-pci

If we were to share BAR0 of this particular device, we’d instruct PCIem as follows:

sudo insmod kernel/pciem.ko p2p_regions="0xe00000000:0x1000"

This will, in turn, give PCIem access to that BAR so PCIe shims can do P2P DMA.

General functionality

PCIem exposes an interface at /dev/pciem which you can freely call into to construct your PCIe shim from within the userspace.

One can see the documented interface on the API page.

Dummy Device Walkthrough

This walkthrough demonstrates how to create a minimal PCIe device using PCIem.

We’ll build a simple “clock/counting device” with basic MMIO registers and MSI interrupt support.

The dummy PCIe

Our dummy PCIe has:

  • One BAR (BAR0, 4KB) with three 32-bit registers:
    • REG_CONTROL (0x00): Write 1 to increment counter
    • REG_STATUS (0x04): Status flags (IRQ pending bit)
    • REG_COUNTER (0x08): Current counter value (read-only)
  • MSI: One interrupt vector
  • Interrupts: Every 10 counts, an MSI is fired

Prerequisites

Before starting, make sure you have:

  1. PCIem kernel module loaded (see Getting Started)
  2. A C compiler and kernel headers installed
  3. Basic understanding of PCIe concepts (BARs, config space, interrupts)

The Code

Userspace Device Emulator

The userspace program creates the PCIe device, establishes the event channel, and handles MMIO operations.

/*
 * DummyClockPCIe userspace shim
 */

#include <errno.h>
#include <fcntl.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <sys/eventfd.h>
#include <unistd.h>
#include <stdatomic.h>
#include <poll.h>

#include "pciem_userspace.h"

/* Defines for the BAR0 register offsets */
#define REG_CONTROL  0x00
#define REG_STATUS   0x04
#define REG_COUNTER  0x08

#define STATUS_IRQ_PENDING (1 << 0)

/* DummyClockPCIe state */
struct counter_device {
    int fd;
    int instance_fd;
    uint32_t counter;
    uint32_t status;
    struct pciem_shared_ring *ring;
};

1. Opening the PCIem Device

fd = open("/dev/pciem", O_RDWR);
if (fd < 0) {
    perror("Failed to open /dev/pciem");
    return 1;
}

This opens the PCIem control device. All configuration happens on this file descriptor.

2. Creating the Device

struct pciem_create_device create = {0};
ret = ioctl(fd, PCIEM_IOCTL_CREATE_DEVICE, &create);
if (ret < 0) {
    perror("Failed to create device");
    return 1;
}

This tells PCIem we’re starting to configure a new virtual PCIe device.

3. Adding BAR0

struct pciem_bar_config bar = {
    .bar_index = 0,
    .size = 4096,
    .flags = PCI_BASE_ADDRESS_SPACE_MEMORY |
             PCI_BASE_ADDRESS_MEM_TYPE_32,
};
ret = ioctl(fd, PCIEM_IOCTL_ADD_BAR, &bar);

This creates a 4KB memory BAR at index 0.

4. Adding MSI Capability

struct pciem_cap_msi_userspace msi = {
    .num_vectors_log2 = 0,  /* 2^0 = 1 vector */
    .has_64bit = 1,
    .has_masking = 0,
};
struct pciem_cap_config cap = {
    .cap_type = PCIEM_CAP_MSI,
    .cap_size = sizeof(msi),
};
memcpy(cap.cap_data, &msi, sizeof(msi));

ret = ioctl(fd, PCIEM_IOCTL_ADD_CAPABILITY, &cap);

5. Setting Config Space

struct pciem_config_space cfg = {
    .vendor_id = 0x1234,
    .device_id = 0x5678,
    .subsys_vendor_id = 0x1234,
    .subsys_device_id = 0x5678,
    .revision = 0x01,
    .class_code = {0x00, 0x00, 0xFF},
    .header_type = 0x00,
};
ret = ioctl(fd, PCIEM_IOCTL_SET_CONFIG, &cfg);

This sets how the device identifies itself, one part important later is:

  • Vendor/Device ID pair: Used by the kernel to look for drivers

6. Setting Up Watchpoints

Watchpoints allow you to get immediate hardware-level notifications when the driver accesses specific registers:

struct pciem_watchpoint_config wp = {
    .bar_index = 0,
    .offset = REG_CONTROL,  /* 0x00                         */
    .width = 4,             /* 4 bytes (32-bit register)    */
    .flags = PCIEM_WP_FLAG_BAR_KPROBES,
};

ret = ioctl(fd, PCIEM_IOCTL_SET_WATCHPOINT, &wp);
if (ret < 0) {
    perror("Failed to set watchpoint");
}

Watchpoint flags:

PCIEM_WP_FLAG_BAR_KPROBES: PCIem automatically locates the BAR mapping (recommended). – PCIEM_WP_FLAG_BAR_MANUAL: You provide the virtual address manually (advanced usage).

Note: Watchpoints use hardware debug registers, so you can only have a limited number active (Don’t assume there’s loads…).

7. Registering the Device

ret = ioctl(fd, PCIEM_IOCTL_REGISTER, 0);
if (ret < 0) {
    perror("Failed to register device");
    return 1;
}
printf("Device registered! Instance FD: %d\n", ret);

This makes the device visible to the kernel. It returns a new instance_fd which represents the synthetic device on the bus.

8. Setting up the Event Ring

To receive events efficiently, we map a shared atomic ring buffer and set up an eventfd for notifications. This avoids busy polling.

int efd = eventfd(0, EFD_CLOEXEC | EFD_NONBLOCK);
struct pciem_eventfd_config efd_cfg = { .eventfd = efd };

if (ioctl(fd, PCIEM_IOCTL_SET_EVENTFD, &efd_cfg) < 0) {
    perror("Failed to set eventfd");
    return 1;
}

struct pciem_shared_ring *ring = mmap(NULL, sizeof(struct pciem_shared_ring),
                                      PROT_READ | PROT_WRITE, MAP_SHARED,
                                      fd, 0);
uint32_t head = atomic_load(&ring->head);

struct pciem_bar_info_query bar0_info = { .bar_index = 0 };
ioctl(fd, PCIEM_IOCTL_GET_BAR_INFO, &bar0_info);

volatile uint32_t *bar0 = mmap(NULL, bar0_info.size,
                               PROT_READ | PROT_WRITE,
                               MAP_SHARED, instance_fd, 0);

9. The Event Loop

Now we enter the main loop. If the ring is empty, we poll() on the eventfd to sleep until the kernel wakes us up.

struct pollfd pfd = {
    .fd = efd,
    .events = POLLIN
};

struct counter_device dev = { .fd = fd, .counter = 0 };

while (1) {
    uint32_t tail = atomic_load(&ring->tail);

    if (head == tail) {
        if (poll(&pfd, 1, -1) > 0) {
            uint64_t count;
            read(efd, &count, sizeof(count));
        }
        continue;
    }

    while (head != tail) {
        atomic_thread_fence(memory_order_acquire);

        struct pciem_event *evt = &ring->events[head];

        switch (evt->type) {
            case PCIEM_EVENT_MMIO_WRITE:
                handle_mmio_write(fd, evt, &dev, bar0);
                break;

            case PCIEM_EVENT_MMIO_READ:
                printf("Driver read offset 0x%lx\n", evt->offset);
                break;
        }

        head = (head + 1) % PCIEM_RING_SIZE;
        atomic_store(&ring->head, head);
    }
}

10. Processing Events

Since PCIem uses a shared memory model, you do not send a response to the kernel for events.

  • For Reads: The driver reads directly from the BAR memory you mapped.
  • For Writes: The driver writes to memory and posts a notification event. You update your internal state and the BAR memory immediately.
static void handle_mmio_write(int fd, struct pciem_event *evt,
                              struct counter_device *dev,
                              volatile uint32_t *bar0)
{
    if (evt->bar == 0 && evt->offset == REG_CONTROL) {
        if (evt->data & 1) {
            dev->counter++;

            bar0[REG_COUNTER / 4] = dev->counter;

            if (dev->counter % 10 == 0) {
                struct pciem_irq_inject irq = { .vector = 0 };
                ioctl(fd, PCIEM_IOCTL_INJECT_IRQ, &irq);
            }
        }
    }
}

11. Injecting Interrupts

As shown in the handler above, when you want to signal the driver asynchronously (like when the counter reaches a threshold), you use the IRQ injection IOCTL:

struct pciem_irq_inject irq = { .vector = 0 };
ioctl(fd, PCIEM_IOCTL_INJECT_IRQ, &irq);

API Reference

This page is auto-generated from the PCIem kernel headers. Do not edit it by hand — run make docs to regenerate.

PCIem exposes a userspace-facing API through /dev/pciem that lets developers configure and emulate PCIe devices entirely from userspace.

Contents

Constants

PCIEM_CAP_*

NameValueDescription
PCIEM_CAP_MSI0
PCIEM_CAP_MSIX1
PCIEM_CAP_PASID5
PCIEM_CAP_PCIE3
PCIEM_CAP_PM2
PCIEM_CAP_VSEC4

PCIEM_EVENT_*

NameValueDescription
PCIEM_EVENT_CONFIG_READ3
PCIEM_EVENT_CONFIG_WRITE4
PCIEM_EVENT_MMIO_READ1
PCIEM_EVENT_MMIO_WRITE2
PCIEM_EVENT_MSI_ACK5
PCIEM_EVENT_RESET6

PCIEM_TRACE_*

NameValueDescription
PCIEM_TRACE_READS(1 << 0)
PCIEM_TRACE_STOP_WRITES(1 << 2)
PCIEM_TRACE_WRITES(1 << 1)

PCIEM_ATOMIC_*

NameValueDescription
PCIEM_ATOMIC_CAS4
PCIEM_ATOMIC_FETCH_ADD1
PCIEM_ATOMIC_FETCH_AND5
PCIEM_ATOMIC_FETCH_OR6
PCIEM_ATOMIC_FETCH_SUB2
PCIEM_ATOMIC_FETCH_XOR7
PCIEM_ATOMIC_SWAP3

PCIEM_DMA_FLAG_*

NameValueDescription
PCIEM_DMA_FLAG_READ0x1
PCIEM_DMA_FLAG_WRITE0x2

PCIEM_IRQFD_FLAG_*

NameValueDescription
PCIEM_IRQFD_FLAG_DEASSERT(1 << 1)
PCIEM_IRQFD_FLAG_LEVEL(1 << 0)

PCIEM_WP_FLAG_BAR_*

NameValueDescription
PCIEM_WP_FLAG_BAR_KPROBES(1 << 0)
PCIEM_WP_FLAG_BAR_MANUAL(1 << 1)

PCIEM_CREATE_FLAG_BUS_MODE_*

NameValueDescription
PCIEM_CREATE_FLAG_BUS_MODE_ATTACH0x00000001Attach to an existing physical PCIe bus.
PCIEM_CREATE_FLAG_BUS_MODE_MASK0x00000003Mask to extract the bus mode bits from the flags field.
PCIEM_CREATE_FLAG_BUS_MODE_VIRTUAL0x00000000Create a virtual PCIe bus owned entirely by PCIem.

Miscellaneous

NameValueDescription
PCIEM_MAX_IRQFDS32
PCIEM_RING_SIZE256

Structures

Defined in pciem_api.h

pciem_create_device

struct pciem_create_device
{
    uint32_t flags;
};
FieldTypeDescription
flagsuint32_t

Used by: PCIEM_IOCTL_CREATE_DEVICE

pciem_bar_config

struct pciem_bar_config
{
    uint32_t bar_index;
    uint32_t flags;
    uint64_t size;
    uint32_t reserved;
};
FieldTypeDescription
bar_indexuint32_t
flagsuint32_t
sizeuint64_t
reserveduint32_t

Used by: PCIEM_IOCTL_ADD_BAR

pciem_cap_msi_userspace

struct pciem_cap_msi_userspace
{
    uint8_t num_vectors_log2;
    uint8_t has_64bit;
    uint8_t has_masking;
    uint8_t reserved;
};
FieldTypeDescription
num_vectors_log2uint8_t
has_64bituint8_t
has_maskinguint8_t
reserveduint8_t

pciem_cap_msix_userspace

struct pciem_cap_msix_userspace
{
    uint8_t bar_index;
    uint8_t reserved[3];
    uint32_t table_offset;
    uint32_t pba_offset;
    uint16_t table_size;
    uint16_t reserved2;
};
FieldTypeDescription
bar_indexuint8_t
reserveduint8_t[3]
table_offsetuint32_t
pba_offsetuint32_t
table_sizeuint16_t
reserved2uint16_t

pciem_cap_pasid_userspace

struct pciem_cap_pasid_userspace
{
    uint8_t max_pasid_width;
    uint8_t execute_permission;
    uint8_t privileged_mode;
    uint8_t reserved;
};
FieldTypeDescription
max_pasid_widthuint8_t
execute_permissionuint8_t
privileged_modeuint8_t
reserveduint8_t

pciem_cap_config

struct pciem_cap_config
{
    uint32_t cap_type;
    struct pciem_cap_msi_userspace msi;
    struct pciem_cap_msix_userspace msix;
    struct pciem_cap_pasid_userspace pasid;
};

Used by: PCIEM_IOCTL_ADD_CAPABILITY

pciem_config_space

struct pciem_config_space
{
    uint16_t vendor_id;
    uint16_t device_id;
    uint16_t subsys_vendor_id;
    uint16_t subsys_device_id;
    uint8_t revision;
    uint8_t class_code[3];
    uint8_t header_type;
    uint8_t reserved[7];
};
FieldTypeDescription
vendor_iduint16_t
device_iduint16_t
subsys_vendor_iduint16_t
subsys_device_iduint16_t
revisionuint8_t
class_codeuint8_t[3]
header_typeuint8_t
reserveduint8_t[7]

Used by: PCIEM_IOCTL_SET_CONFIG

pciem_event

struct pciem_event
{
    uint64_t seq;
    uint32_t type;
    uint32_t bar;
    uint64_t offset;
    uint32_t size;
    uint32_t reserved;
    uint64_t data;
    uint64_t timestamp;
};
FieldTypeDescription
sequint64_t
typeuint32_t
baruint32_t
offsetuint64_t
sizeuint32_t
reserveduint32_t
datauint64_t
timestampuint64_t

pciem_response

struct pciem_response
{
    uint64_t seq;
    uint64_t data;
    int32_t status;
    uint32_t reserved;
};
FieldTypeDescription
sequint64_t
datauint64_t
statusint32_t
reserveduint32_t

pciem_irq_inject

Parameters for PCIEM_IOCTL_INJECT_IRQ.

struct pciem_irq_inject
{
    uint32_t vector;
    uint32_t reserved;
};
FieldTypeDescription
vectoruint32_tMSI/MSI-X vector number to inject into the guest.
reserveduint32_tMust be zero.

Used by: PCIEM_IOCTL_INJECT_IRQ

pciem_dma_op

struct pciem_dma_op
{
    uint64_t guest_iova;
    uint64_t user_addr;
    uint32_t length;
    uint32_t pasid;
    uint32_t flags;
    uint32_t reserved;
};
FieldTypeDescription
guest_iovauint64_t
user_addruint64_t
lengthuint32_t
pasiduint32_t
flagsuint32_t
reserveduint32_t

Used by: PCIEM_IOCTL_DMA

pciem_dma_atomic

struct pciem_dma_atomic
{
    uint64_t guest_iova;
    uint64_t operand;
    uint64_t compare;
    uint32_t op_type;
    uint32_t pasid;
    uint64_t result;
};
FieldTypeDescription
guest_iovauint64_t
operanduint64_t
compareuint64_t
op_typeuint32_t
pasiduint32_t
resultuint64_t

Used by: PCIEM_IOCTL_DMA_ATOMIC

pciem_p2p_op_user

struct pciem_p2p_op_user
{
    uint64_t target_phys_addr;
    uint64_t user_addr;
    uint32_t length;
    uint32_t flags;
};
FieldTypeDescription
target_phys_addruint64_t
user_addruint64_t
lengthuint32_t
flagsuint32_t

Used by: PCIEM_IOCTL_P2P

pciem_bar_info_query

struct pciem_bar_info_query
{
    uint32_t bar_index;
    uint64_t phys_addr;
    uint64_t size;
    uint32_t flags;
};
FieldTypeDescription
bar_indexuint32_t
phys_addruint64_t
sizeuint64_t
flagsuint32_t

Used by: PCIEM_IOCTL_GET_BAR_INFO

pciem_eventfd_config

struct pciem_eventfd_config
{
    int32_t eventfd;
    uint32_t reserved;
};
FieldTypeDescription
eventfdint32_t
reserveduint32_t

Used by: PCIEM_IOCTL_SET_EVENTFD

pciem_irqfd_config

struct pciem_irqfd_config
{
    int32_t eventfd;
    uint32_t vector;
    uint32_t flags;
    uint32_t reserved;
};
FieldTypeDescription
eventfdint32_t
vectoruint32_t
flagsuint32_t
reserveduint32_t

Used by: PCIEM_IOCTL_SET_IRQFD

pciem_dma_indirect

struct pciem_dma_indirect
{
    uint64_t prp1;
    uint64_t prp2;
    uint64_t user_addr;
    uint32_t length;
    uint32_t page_size;
    uint32_t pasid;
    uint32_t flags;
    uint32_t reserved;
};
FieldTypeDescription
prp1uint64_t
prp2uint64_t
user_addruint64_t
lengthuint32_t
page_sizeuint32_t
pasiduint32_t
flagsuint32_t
reserveduint32_t

Used by: PCIEM_IOCTL_DMA_INDIRECT

pciem_trace_bar

struct pciem_trace_bar
{
    uint32_t bar_index;
    uint32_t flags;
};
FieldTypeDescription
bar_indexuint32_t
flagsuint32_t

Used by: PCIEM_IOCTL_TRACE_BAR

pciem_shared_ring

Lock-free single-producer/single-consumer event ring shared between the kernel and userspace.

The kernel writes events by advancing @head; userspace consumes them by advancing @tail. Each counter is cache-line padded. The ring is mapped read-only into userspace via mmap on the PCIem fd.

@head from @tail.

@tail from the event array.

struct pciem_shared_ring
{
    atomic_t head;
    char _pad1[60];
    atomic_t tail;
    char _pad2[60];
    struct pciem_event events[PCIEM_RING_SIZE];
};
FieldTypeDescription
headatomic_tWrite index, owned by the kernel. Incremented atomically after each event is committed.
_pad1char[60]Cache-line padding to isolate
tailatomic_tRead index, owned by userspace. Incremented after each event is consumed.
_pad2char[60]Cache-line padding to isolate
eventsstruct pciem_event[PCIEM_RING_SIZE]Circular buffer of PCIEM_RING_SIZE events.

IOCTLs

All ioctls are issued on the /dev/pciem file descriptor unless noted otherwise.

PCIEM_IOCTL_CREATE_DEVICE

#define PCIEM_IOCTL_CREATE_DEVICE _IOWR(PCIEM_IOCTL_MAGIC, 10, struct pciem_create_device)

Direction: read/write (both directions)

Parameter struct: pciem_create_device

struct pciem_create_device arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_CREATE_DEVICE, &arg);

PCIEM_IOCTL_ADD_BAR

#define PCIEM_IOCTL_ADD_BAR _IOW(PCIEM_IOCTL_MAGIC, 11, struct pciem_bar_config)

Direction: write (userspace → kernel)

Parameter struct: pciem_bar_config

struct pciem_bar_config arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_ADD_BAR, &arg);

PCIEM_IOCTL_ADD_CAPABILITY

#define PCIEM_IOCTL_ADD_CAPABILITY _IOW(PCIEM_IOCTL_MAGIC, 12, struct pciem_cap_config)

Direction: write (userspace → kernel)

Parameter struct: pciem_cap_config

struct pciem_cap_config arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_ADD_CAPABILITY, &arg);

PCIEM_IOCTL_SET_CONFIG

#define PCIEM_IOCTL_SET_CONFIG _IOW(PCIEM_IOCTL_MAGIC, 13, struct pciem_config_space)

Direction: write (userspace → kernel)

Parameter struct: pciem_config_space

struct pciem_config_space arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_SET_CONFIG, &arg);

PCIEM_IOCTL_REGISTER

#define PCIEM_IOCTL_REGISTER _IO(PCIEM_IOCTL_MAGIC, 14)

Direction: none (no data transfer)

int ret = ioctl(fd, PCIEM_IOCTL_REGISTER);

PCIEM_IOCTL_INJECT_IRQ

#define PCIEM_IOCTL_INJECT_IRQ _IOW(PCIEM_IOCTL_MAGIC, 15, struct pciem_irq_inject)

Direction: write (userspace → kernel)

Parameter struct: pciem_irq_inject

struct pciem_irq_inject arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_INJECT_IRQ, &arg);

PCIEM_IOCTL_DMA

#define PCIEM_IOCTL_DMA _IOWR(PCIEM_IOCTL_MAGIC, 16, struct pciem_dma_op)

Direction: read/write (both directions)

Parameter struct: pciem_dma_op

struct pciem_dma_op arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_DMA, &arg);

PCIEM_IOCTL_DMA_ATOMIC

#define PCIEM_IOCTL_DMA_ATOMIC _IOWR(PCIEM_IOCTL_MAGIC, 17, struct pciem_dma_atomic)

Direction: read/write (both directions)

Parameter struct: pciem_dma_atomic

struct pciem_dma_atomic arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_DMA_ATOMIC, &arg);

PCIEM_IOCTL_P2P

#define PCIEM_IOCTL_P2P _IOWR(PCIEM_IOCTL_MAGIC, 18, struct pciem_p2p_op_user)

Direction: read/write (both directions)

Parameter struct: pciem_p2p_op_user

struct pciem_p2p_op_user arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_P2P, &arg);

PCIEM_IOCTL_GET_BAR_INFO

#define PCIEM_IOCTL_GET_BAR_INFO _IOWR(PCIEM_IOCTL_MAGIC, 19, struct pciem_bar_info_query)

Direction: read/write (both directions)

Parameter struct: pciem_bar_info_query

struct pciem_bar_info_query arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_GET_BAR_INFO, &arg);

PCIEM_IOCTL_SET_EVENTFD

#define PCIEM_IOCTL_SET_EVENTFD _IOW(PCIEM_IOCTL_MAGIC, 21, struct pciem_eventfd_config)

Direction: write (userspace → kernel)

Parameter struct: pciem_eventfd_config

struct pciem_eventfd_config arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_SET_EVENTFD, &arg);

PCIEM_IOCTL_SET_IRQFD

#define PCIEM_IOCTL_SET_IRQFD _IOW(PCIEM_IOCTL_MAGIC, 22, struct pciem_irqfd_config)

Direction: write (userspace → kernel)

Parameter struct: pciem_irqfd_config

struct pciem_irqfd_config arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_SET_IRQFD, &arg);

PCIEM_IOCTL_DMA_INDIRECT

#define PCIEM_IOCTL_DMA_INDIRECT _IOWR(PCIEM_IOCTL_MAGIC, 24, struct pciem_dma_indirect)

Direction: read/write (both directions)

Parameter struct: pciem_dma_indirect

struct pciem_dma_indirect arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_DMA_INDIRECT, &arg);

PCIEM_IOCTL_TRACE_BAR

#define PCIEM_IOCTL_TRACE_BAR _IOWR(PCIEM_IOCTL_MAGIC, 25, struct pciem_trace_bar)

Direction: read/write (both directions)

Parameter struct: pciem_trace_bar

struct pciem_trace_bar arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_TRACE_BAR, &arg);

PCIEM_IOCTL_START

#define PCIEM_IOCTL_START _IO(PCIEM_IOCTL_MAGIC, 26)

Direction: none (no data transfer)

int ret = ioctl(fd, PCIEM_IOCTL_START);