Overview
PCIem is a Linux framework that enables software developers to write PCIe card drivers on the host to target unexisting PCIe cards on the bus.
This helps model a driver if the developers don’t have a physical PCIe card to test it on, effectively enabling pre-silicon engineering on the software side of things for the prototype card.
Initially, PCIem was thought after enabling communications from/to a QEMU instance; but has ever since been evolving to support a wide variety of use-cases.
You can think of the framework as a MITM (Man-in-the-Middle) that sits between the untouched, production drivers (Which are, unaware of PCIem’s existence) and the Linux kernel.
%%{init: {'themeVariables': {'fontSize': '18px'}}}%%
graph TB
subgraph Kernel ["Host Linux Kernel"]
direction TB
RealDriver["Real PCIe Driver"]
subgraph Framework ["PCIem Framework"]
direction TB
Config["PCI Config Space"]
BARs["BARs"]
IRQ["Interrupts"]
DMA["DMA / IOMMU"]
end
end
Interface(("/dev/pciem"))
subgraph User ["Linux Userspace"]
direction TB
Shim["Device Emulation"]
end
Framework <==> Interface
Interface <==> Shim
Getting started
Cloning the repository
In order to use PCIem, we’ll first clone the repository:
git clone https://github.com/cakehonolulu/pciem
With the repository already cloned, we’ll enter it:
cd pciem/
Compiling the code
To compile PCIem, it should be enough to have make, a C compiler & the Linux headers for your currently-running kernel.
Depending on the amount of features your userspace-PCI shim has, you may need further requirements (Think of, displaying a framebuffer the driver writes on with SDL3 or alike); but for a simple one, the aforementioned ones will suffice.
Issuing a compilation is as simple as:
make all
Using PCIem
After we compile everything, we’ll end up with a bunch of object files, an executable and a kernel module.
In short, the way this framework works is by loading the pciem.ko kernel driver (With a certain set of parameters that’ll be discussed below), then loading the userspace-shim that actually creates the device using the exported functionalities of PCIem, and finally; you load the actual (Untouched!) PCIe driver you want to test.
┌───────┐ ┌──────────┐ ┌────────────┐
│ Linux ├─────► pciem.ko ├─────► User-space │
└───────┘ └──────────┘ │ shim │
└────────────┘
PCIem parameters
pciem_phys_region module argument
Let’s start with the preface that, we’re obviously going to use insmod to load pciem.ko, but the way we do it changes the behaviour of the framework.
Starting from PCIem 0.1, one can specify kernel module arguments to alter/instruct certain logic on the code.
pciem_phys_region: This basically specifies what physically-contiguous memory region is reserved and free to use by PCIem.
Reserving memory with memmap command line argument
This is attained by passing the memmap= argument to Linux’s cmdline.
As per kernel.org’s memmap:
memmap=nn[KMG]$ss[KMG]
[KNL,ACPI,EARLY] Mark specific memory as reserved.
Region of memory to be reserved is from ss to ss+nn.
Example: Exclude memory from 0x18690000-0x1869ffff
memmap=64K$0x18690000
or
memmap=0x10000$0x18690000
Some bootloaders may need an escape character before '$',
like Grub2, otherwise '$' and the following number
will be eaten.
This effectively means that, if we append the following to the cmdline:
memmap=128M$0x1bf000000
Linux is going to carve 128M starting from physical address 0x1bf000000 out of the System RAM and mark it as Reserved so PCIem can use it.
With that exact setup, we’d then load PCIem as follows:
sudo insmod kernel/pciem.ko pciem_phys_region="0x1bf000000:0x8000000"
What this does is, tell PCIem that it basically has 128MBs to play starting from physical 0x1bf000000:
Note: On some AArch64 platforms (such as the Raspberry Pi 4B),
memmap=may not be honoured by the bootloader or may not be available at all. In those cases, the preferred alternative is to reserve memory via a Device Tree overlay instead.
Reserving memory with a custom device tree overlay
Create a file named reserve-mem.dts with the following contents, adjusting the base address and size to match your desired reservation:
/dts-v1/;
/plugin/;
/ {
compatible = "brcm,bcm2711";
fragment@0 {
target-path = "/";
__overlay__ {
#address-cells = <2>;
#size-cells = <1>;
reserved-memory {
#address-cells = <2>;
#size-cells = <1>;
ranges;
pciem_reservation@50000000 {
reg = <0x0 0x50000000 0x10000000>;
no-map;
status = "okay";
};
};
};
};
};
In this example, 0x50000000 is the base address and 0x10000000 is the size (256MB). Adjust these to fit your target platform’s memory layout.
Compile the overlay and install it:
dtc -@ -I dts -O dtb -o reserve-mem.dtbo reserve-mem.dts
sudo cp reserve-mem.dtbo /boot/firmware/overlays/
Then enable it by appending the following line to /boot/firmware/config.txt:
dtoverlay=reserve-mem
Note: For RPi 4B (Or similar) boards you should add the line within the
[all]node.
Reboot for the reservation to take effect. You can verify it was applied by checking:
cat /proc/iomem | grep -i reserved
Once confirmed, load PCIem pointing at the same region specified in the overlay:
sudo insmod kernel/pciem.ko pciem_phys_region="0x50000000:0x10000000"
Note: The method written above has been tested on a Raspberry Pi 4B with the stock Raspbian distribution (64-bit version). Some aarch64 kernels/boards may be fine with
memmap=or evenmem=(Even though the latest hasn’t been tested) but it’s best to try and decide which one works.
p2p_regions
The p2p_regions argument is a bit more complex; think of it as a whitelist for DMA accesses within PCIem.
The framework supports Peer-to-Peer DMA, but in order to add a little bit of security; one has to manually whitelist the target BAR region of the device we want to P2P from/to.
To know what to “share”, you need to obtain the device’s desired bar start and end address:
$ lspci -vvv
...
f147:00:00.0 System peripheral: Red Hat, Inc. Virtio 1.0 file system (rev 01)
Subsystem: Red Hat, Inc. Device 0040
Physical Slot: 2113372925
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64
NUMA node: 0
Region 0: Memory at e00000000 (64-bit, non-prefetchable) [size=4K]
Region 2: Memory at e00001000 (64-bit, non-prefetchable) [size=4K]
Region 4: Memory at c00000000 (64-bit, non-prefetchable) [size=8G]
Capabilities: <access denied>
Kernel driver in use: virtio-pci
If we were to share BAR0 of this particular device, we’d instruct PCIem as follows:
sudo insmod kernel/pciem.ko p2p_regions="0xe00000000:0x1000"
This will, in turn, give PCIem access to that BAR so PCIe shims can do P2P DMA.
General functionality
PCIem exposes an interface at /dev/pciem which you can freely call into to construct your PCIe shim from within the userspace.
One can see the documented interface on the API page.
Dummy Device Walkthrough
This walkthrough demonstrates how to create a minimal PCIe device using PCIem.
We’ll build a simple “clock/counting device” with basic MMIO registers and MSI interrupt support.
The dummy PCIe
Our dummy PCIe has:
- One BAR (BAR0, 4KB) with three 32-bit registers:
REG_CONTROL(0x00): Write 1 to increment counterREG_STATUS(0x04): Status flags (IRQ pending bit)REG_COUNTER(0x08): Current counter value (read-only)
- MSI: One interrupt vector
- Interrupts: Every 10 counts, an MSI is fired
Prerequisites
Before starting, make sure you have:
- PCIem kernel module loaded (see Getting Started)
- A C compiler and kernel headers installed
- Basic understanding of PCIe concepts (BARs, config space, interrupts)
The Code
Userspace Device Emulator
The userspace program creates the PCIe device, establishes the event channel, and handles MMIO operations.
/*
* DummyClockPCIe userspace shim
*/
#include <errno.h>
#include <fcntl.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <sys/eventfd.h>
#include <unistd.h>
#include <stdatomic.h>
#include <poll.h>
#include "pciem_userspace.h"
/* Defines for the BAR0 register offsets */
#define REG_CONTROL 0x00
#define REG_STATUS 0x04
#define REG_COUNTER 0x08
#define STATUS_IRQ_PENDING (1 << 0)
/* DummyClockPCIe state */
struct counter_device {
int fd;
int instance_fd;
uint32_t counter;
uint32_t status;
struct pciem_shared_ring *ring;
};
1. Opening the PCIem Device
fd = open("/dev/pciem", O_RDWR);
if (fd < 0) {
perror("Failed to open /dev/pciem");
return 1;
}
This opens the PCIem control device. All configuration happens on this file descriptor.
2. Creating the Device
struct pciem_create_device create = {0};
ret = ioctl(fd, PCIEM_IOCTL_CREATE_DEVICE, &create);
if (ret < 0) {
perror("Failed to create device");
return 1;
}
This tells PCIem we’re starting to configure a new virtual PCIe device.
3. Adding BAR0
struct pciem_bar_config bar = {
.bar_index = 0,
.size = 4096,
.flags = PCI_BASE_ADDRESS_SPACE_MEMORY |
PCI_BASE_ADDRESS_MEM_TYPE_32,
};
ret = ioctl(fd, PCIEM_IOCTL_ADD_BAR, &bar);
This creates a 4KB memory BAR at index 0.
4. Adding MSI Capability
struct pciem_cap_msi_userspace msi = {
.num_vectors_log2 = 0, /* 2^0 = 1 vector */
.has_64bit = 1,
.has_masking = 0,
};
struct pciem_cap_config cap = {
.cap_type = PCIEM_CAP_MSI,
.cap_size = sizeof(msi),
};
memcpy(cap.cap_data, &msi, sizeof(msi));
ret = ioctl(fd, PCIEM_IOCTL_ADD_CAPABILITY, &cap);
5. Setting Config Space
struct pciem_config_space cfg = {
.vendor_id = 0x1234,
.device_id = 0x5678,
.subsys_vendor_id = 0x1234,
.subsys_device_id = 0x5678,
.revision = 0x01,
.class_code = {0x00, 0x00, 0xFF},
.header_type = 0x00,
};
ret = ioctl(fd, PCIEM_IOCTL_SET_CONFIG, &cfg);
This sets how the device identifies itself, one part important later is:
- Vendor/Device ID pair: Used by the kernel to look for drivers
6. Setting Up Watchpoints
Watchpoints allow you to get immediate hardware-level notifications when the driver accesses specific registers:
struct pciem_watchpoint_config wp = {
.bar_index = 0,
.offset = REG_CONTROL, /* 0x00 */
.width = 4, /* 4 bytes (32-bit register) */
.flags = PCIEM_WP_FLAG_BAR_KPROBES,
};
ret = ioctl(fd, PCIEM_IOCTL_SET_WATCHPOINT, &wp);
if (ret < 0) {
perror("Failed to set watchpoint");
}
Watchpoint flags:
– PCIEM_WP_FLAG_BAR_KPROBES: PCIem automatically locates the BAR mapping (recommended).
– PCIEM_WP_FLAG_BAR_MANUAL: You provide the virtual address manually (advanced usage).
Note: Watchpoints use hardware debug registers, so you can only have a limited number active (Don’t assume there’s loads…).
7. Registering the Device
ret = ioctl(fd, PCIEM_IOCTL_REGISTER, 0);
if (ret < 0) {
perror("Failed to register device");
return 1;
}
printf("Device registered! Instance FD: %d\n", ret);
This makes the device visible to the kernel. It returns a new instance_fd which represents the synthetic device on the bus.
8. Setting up the Event Ring
To receive events efficiently, we map a shared atomic ring buffer and set up an eventfd for notifications. This avoids busy polling.
int efd = eventfd(0, EFD_CLOEXEC | EFD_NONBLOCK);
struct pciem_eventfd_config efd_cfg = { .eventfd = efd };
if (ioctl(fd, PCIEM_IOCTL_SET_EVENTFD, &efd_cfg) < 0) {
perror("Failed to set eventfd");
return 1;
}
struct pciem_shared_ring *ring = mmap(NULL, sizeof(struct pciem_shared_ring),
PROT_READ | PROT_WRITE, MAP_SHARED,
fd, 0);
uint32_t head = atomic_load(&ring->head);
struct pciem_bar_info_query bar0_info = { .bar_index = 0 };
ioctl(fd, PCIEM_IOCTL_GET_BAR_INFO, &bar0_info);
volatile uint32_t *bar0 = mmap(NULL, bar0_info.size,
PROT_READ | PROT_WRITE,
MAP_SHARED, instance_fd, 0);
9. The Event Loop
Now we enter the main loop. If the ring is empty, we poll() on the eventfd to sleep until the kernel wakes us up.
struct pollfd pfd = {
.fd = efd,
.events = POLLIN
};
struct counter_device dev = { .fd = fd, .counter = 0 };
while (1) {
uint32_t tail = atomic_load(&ring->tail);
if (head == tail) {
if (poll(&pfd, 1, -1) > 0) {
uint64_t count;
read(efd, &count, sizeof(count));
}
continue;
}
while (head != tail) {
atomic_thread_fence(memory_order_acquire);
struct pciem_event *evt = &ring->events[head];
switch (evt->type) {
case PCIEM_EVENT_MMIO_WRITE:
handle_mmio_write(fd, evt, &dev, bar0);
break;
case PCIEM_EVENT_MMIO_READ:
printf("Driver read offset 0x%lx\n", evt->offset);
break;
}
head = (head + 1) % PCIEM_RING_SIZE;
atomic_store(&ring->head, head);
}
}
10. Processing Events
Since PCIem uses a shared memory model, you do not send a response to the kernel for events.
- For Reads: The driver reads directly from the BAR memory you mapped.
- For Writes: The driver writes to memory and posts a notification event. You update your internal state and the BAR memory immediately.
static void handle_mmio_write(int fd, struct pciem_event *evt,
struct counter_device *dev,
volatile uint32_t *bar0)
{
if (evt->bar == 0 && evt->offset == REG_CONTROL) {
if (evt->data & 1) {
dev->counter++;
bar0[REG_COUNTER / 4] = dev->counter;
if (dev->counter % 10 == 0) {
struct pciem_irq_inject irq = { .vector = 0 };
ioctl(fd, PCIEM_IOCTL_INJECT_IRQ, &irq);
}
}
}
}
11. Injecting Interrupts
As shown in the handler above, when you want to signal the driver asynchronously (like when the counter reaches a threshold), you use the IRQ injection IOCTL:
struct pciem_irq_inject irq = { .vector = 0 };
ioctl(fd, PCIEM_IOCTL_INJECT_IRQ, &irq);
API Reference
This page is auto-generated from the PCIem kernel headers. Do not edit it by hand — run make docs to regenerate.
PCIem exposes a userspace-facing API through /dev/pciem that lets developers configure and emulate PCIe devices entirely from userspace.
Contents
- Constants
- Structures
pciem_create_devicepciem_bar_configpciem_cap_msi_userspacepciem_cap_msix_userspacepciem_cap_pasid_userspacepciem_cap_configpciem_config_spacepciem_eventpciem_responsepciem_irq_injectpciem_dma_oppciem_dma_atomicpciem_p2p_op_userpciem_bar_info_querypciem_eventfd_configpciem_irqfd_configpciem_dma_indirectpciem_trace_barpciem_shared_ring
- IOCTLs
PCIEM_IOCTL_CREATE_DEVICEPCIEM_IOCTL_ADD_BARPCIEM_IOCTL_ADD_CAPABILITYPCIEM_IOCTL_SET_CONFIGPCIEM_IOCTL_REGISTERPCIEM_IOCTL_INJECT_IRQPCIEM_IOCTL_DMAPCIEM_IOCTL_DMA_ATOMICPCIEM_IOCTL_P2PPCIEM_IOCTL_GET_BAR_INFOPCIEM_IOCTL_SET_EVENTFDPCIEM_IOCTL_SET_IRQFDPCIEM_IOCTL_DMA_INDIRECTPCIEM_IOCTL_TRACE_BARPCIEM_IOCTL_START
Constants
PCIEM_CAP_*
| Name | Value | Description |
|---|---|---|
PCIEM_CAP_MSI | 0 | |
PCIEM_CAP_MSIX | 1 | |
PCIEM_CAP_PASID | 5 | |
PCIEM_CAP_PCIE | 3 | |
PCIEM_CAP_PM | 2 | |
PCIEM_CAP_VSEC | 4 |
PCIEM_EVENT_*
| Name | Value | Description |
|---|---|---|
PCIEM_EVENT_CONFIG_READ | 3 | |
PCIEM_EVENT_CONFIG_WRITE | 4 | |
PCIEM_EVENT_MMIO_READ | 1 | |
PCIEM_EVENT_MMIO_WRITE | 2 | |
PCIEM_EVENT_MSI_ACK | 5 | |
PCIEM_EVENT_RESET | 6 |
PCIEM_TRACE_*
| Name | Value | Description |
|---|---|---|
PCIEM_TRACE_READS | (1 << 0) | |
PCIEM_TRACE_STOP_WRITES | (1 << 2) | |
PCIEM_TRACE_WRITES | (1 << 1) |
PCIEM_ATOMIC_*
| Name | Value | Description |
|---|---|---|
PCIEM_ATOMIC_CAS | 4 | |
PCIEM_ATOMIC_FETCH_ADD | 1 | |
PCIEM_ATOMIC_FETCH_AND | 5 | |
PCIEM_ATOMIC_FETCH_OR | 6 | |
PCIEM_ATOMIC_FETCH_SUB | 2 | |
PCIEM_ATOMIC_FETCH_XOR | 7 | |
PCIEM_ATOMIC_SWAP | 3 |
PCIEM_DMA_FLAG_*
| Name | Value | Description |
|---|---|---|
PCIEM_DMA_FLAG_READ | 0x1 | |
PCIEM_DMA_FLAG_WRITE | 0x2 |
PCIEM_IRQFD_FLAG_*
| Name | Value | Description |
|---|---|---|
PCIEM_IRQFD_FLAG_DEASSERT | (1 << 1) | |
PCIEM_IRQFD_FLAG_LEVEL | (1 << 0) |
PCIEM_WP_FLAG_BAR_*
| Name | Value | Description |
|---|---|---|
PCIEM_WP_FLAG_BAR_KPROBES | (1 << 0) | |
PCIEM_WP_FLAG_BAR_MANUAL | (1 << 1) |
PCIEM_CREATE_FLAG_BUS_MODE_*
| Name | Value | Description |
|---|---|---|
PCIEM_CREATE_FLAG_BUS_MODE_ATTACH | 0x00000001 | Attach to an existing physical PCIe bus. |
PCIEM_CREATE_FLAG_BUS_MODE_MASK | 0x00000003 | Mask to extract the bus mode bits from the flags field. |
PCIEM_CREATE_FLAG_BUS_MODE_VIRTUAL | 0x00000000 | Create a virtual PCIe bus owned entirely by PCIem. |
Miscellaneous
| Name | Value | Description |
|---|---|---|
PCIEM_MAX_IRQFDS | 32 | |
PCIEM_RING_SIZE | 256 |
Structures
Defined in
pciem_api.h
pciem_create_device
struct pciem_create_device
{
uint32_t flags;
};
| Field | Type | Description |
|---|---|---|
flags | uint32_t |
Used by: PCIEM_IOCTL_CREATE_DEVICE
pciem_bar_config
struct pciem_bar_config
{
uint32_t bar_index;
uint32_t flags;
uint64_t size;
uint32_t reserved;
};
| Field | Type | Description |
|---|---|---|
bar_index | uint32_t | |
flags | uint32_t | |
size | uint64_t | |
reserved | uint32_t |
Used by: PCIEM_IOCTL_ADD_BAR
pciem_cap_msi_userspace
struct pciem_cap_msi_userspace
{
uint8_t num_vectors_log2;
uint8_t has_64bit;
uint8_t has_masking;
uint8_t reserved;
};
| Field | Type | Description |
|---|---|---|
num_vectors_log2 | uint8_t | |
has_64bit | uint8_t | |
has_masking | uint8_t | |
reserved | uint8_t |
pciem_cap_msix_userspace
struct pciem_cap_msix_userspace
{
uint8_t bar_index;
uint8_t reserved[3];
uint32_t table_offset;
uint32_t pba_offset;
uint16_t table_size;
uint16_t reserved2;
};
| Field | Type | Description |
|---|---|---|
bar_index | uint8_t | |
reserved | uint8_t[3] | |
table_offset | uint32_t | |
pba_offset | uint32_t | |
table_size | uint16_t | |
reserved2 | uint16_t |
pciem_cap_pasid_userspace
struct pciem_cap_pasid_userspace
{
uint8_t max_pasid_width;
uint8_t execute_permission;
uint8_t privileged_mode;
uint8_t reserved;
};
| Field | Type | Description |
|---|---|---|
max_pasid_width | uint8_t | |
execute_permission | uint8_t | |
privileged_mode | uint8_t | |
reserved | uint8_t |
pciem_cap_config
struct pciem_cap_config
{
uint32_t cap_type;
struct pciem_cap_msi_userspace msi;
struct pciem_cap_msix_userspace msix;
struct pciem_cap_pasid_userspace pasid;
};
| Field | Type | Description |
|---|---|---|
cap_type | uint32_t | |
msi | struct pciem_cap_msi_userspace | |
msix | struct pciem_cap_msix_userspace | |
pasid | struct pciem_cap_pasid_userspace |
Used by: PCIEM_IOCTL_ADD_CAPABILITY
pciem_config_space
struct pciem_config_space
{
uint16_t vendor_id;
uint16_t device_id;
uint16_t subsys_vendor_id;
uint16_t subsys_device_id;
uint8_t revision;
uint8_t class_code[3];
uint8_t header_type;
uint8_t reserved[7];
};
| Field | Type | Description |
|---|---|---|
vendor_id | uint16_t | |
device_id | uint16_t | |
subsys_vendor_id | uint16_t | |
subsys_device_id | uint16_t | |
revision | uint8_t | |
class_code | uint8_t[3] | |
header_type | uint8_t | |
reserved | uint8_t[7] |
Used by: PCIEM_IOCTL_SET_CONFIG
pciem_event
struct pciem_event
{
uint64_t seq;
uint32_t type;
uint32_t bar;
uint64_t offset;
uint32_t size;
uint32_t reserved;
uint64_t data;
uint64_t timestamp;
};
| Field | Type | Description |
|---|---|---|
seq | uint64_t | |
type | uint32_t | |
bar | uint32_t | |
offset | uint64_t | |
size | uint32_t | |
reserved | uint32_t | |
data | uint64_t | |
timestamp | uint64_t |
pciem_response
struct pciem_response
{
uint64_t seq;
uint64_t data;
int32_t status;
uint32_t reserved;
};
| Field | Type | Description |
|---|---|---|
seq | uint64_t | |
data | uint64_t | |
status | int32_t | |
reserved | uint32_t |
pciem_irq_inject
Parameters for PCIEM_IOCTL_INJECT_IRQ.
struct pciem_irq_inject
{
uint32_t vector;
uint32_t reserved;
};
| Field | Type | Description |
|---|---|---|
vector | uint32_t | MSI/MSI-X vector number to inject into the guest. |
reserved | uint32_t | Must be zero. |
Used by: PCIEM_IOCTL_INJECT_IRQ
pciem_dma_op
struct pciem_dma_op
{
uint64_t guest_iova;
uint64_t user_addr;
uint32_t length;
uint32_t pasid;
uint32_t flags;
uint32_t reserved;
};
| Field | Type | Description |
|---|---|---|
guest_iova | uint64_t | |
user_addr | uint64_t | |
length | uint32_t | |
pasid | uint32_t | |
flags | uint32_t | |
reserved | uint32_t |
Used by: PCIEM_IOCTL_DMA
pciem_dma_atomic
struct pciem_dma_atomic
{
uint64_t guest_iova;
uint64_t operand;
uint64_t compare;
uint32_t op_type;
uint32_t pasid;
uint64_t result;
};
| Field | Type | Description |
|---|---|---|
guest_iova | uint64_t | |
operand | uint64_t | |
compare | uint64_t | |
op_type | uint32_t | |
pasid | uint32_t | |
result | uint64_t |
Used by: PCIEM_IOCTL_DMA_ATOMIC
pciem_p2p_op_user
struct pciem_p2p_op_user
{
uint64_t target_phys_addr;
uint64_t user_addr;
uint32_t length;
uint32_t flags;
};
| Field | Type | Description |
|---|---|---|
target_phys_addr | uint64_t | |
user_addr | uint64_t | |
length | uint32_t | |
flags | uint32_t |
Used by: PCIEM_IOCTL_P2P
pciem_bar_info_query
struct pciem_bar_info_query
{
uint32_t bar_index;
uint64_t phys_addr;
uint64_t size;
uint32_t flags;
};
| Field | Type | Description |
|---|---|---|
bar_index | uint32_t | |
phys_addr | uint64_t | |
size | uint64_t | |
flags | uint32_t |
Used by: PCIEM_IOCTL_GET_BAR_INFO
pciem_eventfd_config
struct pciem_eventfd_config
{
int32_t eventfd;
uint32_t reserved;
};
| Field | Type | Description |
|---|---|---|
eventfd | int32_t | |
reserved | uint32_t |
Used by: PCIEM_IOCTL_SET_EVENTFD
pciem_irqfd_config
struct pciem_irqfd_config
{
int32_t eventfd;
uint32_t vector;
uint32_t flags;
uint32_t reserved;
};
| Field | Type | Description |
|---|---|---|
eventfd | int32_t | |
vector | uint32_t | |
flags | uint32_t | |
reserved | uint32_t |
Used by: PCIEM_IOCTL_SET_IRQFD
pciem_dma_indirect
struct pciem_dma_indirect
{
uint64_t prp1;
uint64_t prp2;
uint64_t user_addr;
uint32_t length;
uint32_t page_size;
uint32_t pasid;
uint32_t flags;
uint32_t reserved;
};
| Field | Type | Description |
|---|---|---|
prp1 | uint64_t | |
prp2 | uint64_t | |
user_addr | uint64_t | |
length | uint32_t | |
page_size | uint32_t | |
pasid | uint32_t | |
flags | uint32_t | |
reserved | uint32_t |
Used by: PCIEM_IOCTL_DMA_INDIRECT
pciem_trace_bar
struct pciem_trace_bar
{
uint32_t bar_index;
uint32_t flags;
};
| Field | Type | Description |
|---|---|---|
bar_index | uint32_t | |
flags | uint32_t |
Used by: PCIEM_IOCTL_TRACE_BAR
pciem_shared_ring
Lock-free single-producer/single-consumer event ring shared between the kernel and userspace.
The kernel writes events by advancing @head; userspace consumes them by advancing @tail. Each counter is cache-line padded. The ring is mapped read-only into userspace via mmap on the PCIem fd.
@head from @tail.
@tail from the event array.
struct pciem_shared_ring
{
atomic_t head;
char _pad1[60];
atomic_t tail;
char _pad2[60];
struct pciem_event events[PCIEM_RING_SIZE];
};
| Field | Type | Description |
|---|---|---|
head | atomic_t | Write index, owned by the kernel. Incremented atomically after each event is committed. |
_pad1 | char[60] | Cache-line padding to isolate |
tail | atomic_t | Read index, owned by userspace. Incremented after each event is consumed. |
_pad2 | char[60] | Cache-line padding to isolate |
events | struct pciem_event[PCIEM_RING_SIZE] | Circular buffer of PCIEM_RING_SIZE events. |
IOCTLs
All ioctls are issued on the
/dev/pciemfile descriptor unless noted otherwise.
PCIEM_IOCTL_CREATE_DEVICE
#define PCIEM_IOCTL_CREATE_DEVICE _IOWR(PCIEM_IOCTL_MAGIC, 10, struct pciem_create_device)
Direction: read/write (both directions)
Parameter struct: pciem_create_device
struct pciem_create_device arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_CREATE_DEVICE, &arg);
PCIEM_IOCTL_ADD_BAR
#define PCIEM_IOCTL_ADD_BAR _IOW(PCIEM_IOCTL_MAGIC, 11, struct pciem_bar_config)
Direction: write (userspace → kernel)
Parameter struct: pciem_bar_config
struct pciem_bar_config arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_ADD_BAR, &arg);
PCIEM_IOCTL_ADD_CAPABILITY
#define PCIEM_IOCTL_ADD_CAPABILITY _IOW(PCIEM_IOCTL_MAGIC, 12, struct pciem_cap_config)
Direction: write (userspace → kernel)
Parameter struct: pciem_cap_config
struct pciem_cap_config arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_ADD_CAPABILITY, &arg);
PCIEM_IOCTL_SET_CONFIG
#define PCIEM_IOCTL_SET_CONFIG _IOW(PCIEM_IOCTL_MAGIC, 13, struct pciem_config_space)
Direction: write (userspace → kernel)
Parameter struct: pciem_config_space
struct pciem_config_space arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_SET_CONFIG, &arg);
PCIEM_IOCTL_REGISTER
#define PCIEM_IOCTL_REGISTER _IO(PCIEM_IOCTL_MAGIC, 14)
Direction: none (no data transfer)
int ret = ioctl(fd, PCIEM_IOCTL_REGISTER);
PCIEM_IOCTL_INJECT_IRQ
#define PCIEM_IOCTL_INJECT_IRQ _IOW(PCIEM_IOCTL_MAGIC, 15, struct pciem_irq_inject)
Direction: write (userspace → kernel)
Parameter struct: pciem_irq_inject
struct pciem_irq_inject arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_INJECT_IRQ, &arg);
PCIEM_IOCTL_DMA
#define PCIEM_IOCTL_DMA _IOWR(PCIEM_IOCTL_MAGIC, 16, struct pciem_dma_op)
Direction: read/write (both directions)
Parameter struct: pciem_dma_op
struct pciem_dma_op arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_DMA, &arg);
PCIEM_IOCTL_DMA_ATOMIC
#define PCIEM_IOCTL_DMA_ATOMIC _IOWR(PCIEM_IOCTL_MAGIC, 17, struct pciem_dma_atomic)
Direction: read/write (both directions)
Parameter struct: pciem_dma_atomic
struct pciem_dma_atomic arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_DMA_ATOMIC, &arg);
PCIEM_IOCTL_P2P
#define PCIEM_IOCTL_P2P _IOWR(PCIEM_IOCTL_MAGIC, 18, struct pciem_p2p_op_user)
Direction: read/write (both directions)
Parameter struct: pciem_p2p_op_user
struct pciem_p2p_op_user arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_P2P, &arg);
PCIEM_IOCTL_GET_BAR_INFO
#define PCIEM_IOCTL_GET_BAR_INFO _IOWR(PCIEM_IOCTL_MAGIC, 19, struct pciem_bar_info_query)
Direction: read/write (both directions)
Parameter struct: pciem_bar_info_query
struct pciem_bar_info_query arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_GET_BAR_INFO, &arg);
PCIEM_IOCTL_SET_EVENTFD
#define PCIEM_IOCTL_SET_EVENTFD _IOW(PCIEM_IOCTL_MAGIC, 21, struct pciem_eventfd_config)
Direction: write (userspace → kernel)
Parameter struct: pciem_eventfd_config
struct pciem_eventfd_config arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_SET_EVENTFD, &arg);
PCIEM_IOCTL_SET_IRQFD
#define PCIEM_IOCTL_SET_IRQFD _IOW(PCIEM_IOCTL_MAGIC, 22, struct pciem_irqfd_config)
Direction: write (userspace → kernel)
Parameter struct: pciem_irqfd_config
struct pciem_irqfd_config arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_SET_IRQFD, &arg);
PCIEM_IOCTL_DMA_INDIRECT
#define PCIEM_IOCTL_DMA_INDIRECT _IOWR(PCIEM_IOCTL_MAGIC, 24, struct pciem_dma_indirect)
Direction: read/write (both directions)
Parameter struct: pciem_dma_indirect
struct pciem_dma_indirect arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_DMA_INDIRECT, &arg);
PCIEM_IOCTL_TRACE_BAR
#define PCIEM_IOCTL_TRACE_BAR _IOWR(PCIEM_IOCTL_MAGIC, 25, struct pciem_trace_bar)
Direction: read/write (both directions)
Parameter struct: pciem_trace_bar
struct pciem_trace_bar arg = { /* ... */ };
int ret = ioctl(fd, PCIEM_IOCTL_TRACE_BAR, &arg);
PCIEM_IOCTL_START
#define PCIEM_IOCTL_START _IO(PCIEM_IOCTL_MAGIC, 26)
Direction: none (no data transfer)
int ret = ioctl(fd, PCIEM_IOCTL_START);