In-class: PCI Enumeration

This assignment explores the 32-bit PCI bus, creating a utility in xv6 to list PCI devices and their parameters.

Submit your solutions before the beginning of the next lecture to the CS7680 submission web site.

Please collaborate with others on these exercises.

PCI bus overview

The PCI bus (see wiki.osdev.org/PCI provides a standard architecture for accessing I/O devices on most modern computers, supporting discovery and configuration of attached devices as well as both memory-mapped and IO-mapped (e.g. via inb, outb instructions) access. In particular PCI provides a mechanism where each card or device specifies both its identity and the resources needed (memory-mapped or I/O space, interrupts) and the BIOS and/or OS can discover all devices in the system and allocate interrupts, physical memory space, and I/O space to each device.

PCI has three levels of structure - bus, device, and function. Buses are structured as a tree, with bus 0 as the root and additional buses connected by PCI bridges. An individual PCI card (or chip on the motherboard) that attaches to a PCI bus is a device; note that PCI bridges are themselves devices. Simple devices implement a single function (function 0); more complex devices (e.g. a multi-port ethernet card) may contain multiple functions, which operate (for the most part) like independent PCI devices.

The PCI bus itself is implemented by the motherboard chipset (actually now it's typically integrated onto the CPU itself), which handles tasks such as bus arbitration - ensuring that only one device can transmit on the bus at any time. In parallel PCI the data lines are shared between all devices on a bus, and arbitration is handled by separate bus request lines from each slot to the PCI controller, and corresponding bus grant lines from the controller back to each slot.

The PCI controller is also responsible for providing software access to the configuration space for each device, a 256-byte set of registers which identifies the device and allows configuration of device properties. This is performed via a separate signal to each card or device, so the CPU can detect and configure all devices on the bus without any prior knowledge of e.g. what address they are mapped at.

lspci and examples

The lspci command in Linux lists the devices connected to the PCI bus; here is an example from one of the machines in the first-floor computer lab:

pjd@prosoccer:~$ lspci
00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
00:16.3 Serial controller: Intel Corporation 6 Series/C200 Series Chipset Family KT Controller (rev 04)
00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 04)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b4)
00:1c.2 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 3 (rev b4)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a4)
00:1f.0 ISA bridge: Intel Corporation Q67 Express Chipset Family LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller (rev 04)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 04)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series]
03:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 03)

The configuration space stores 16-bit vendor and device IDs, rather than these descriptive names; lspci performs a table lookup to report these names. The actual numeric values on the same machine are as follows. (note that Intel is vendor id 0x8086)

pjd@prosoccer:~$ lspci -n
00:00.0 0600: 8086:0100 (rev 09)
00:01.0 0604: 8086:0101 (rev 09)
00:16.0 0780: 8086:1c3a (rev 04)
00:16.3 0700: 8086:1c3d (rev 04)
00:19.0 0200: 8086:1502 (rev 04)
00:1a.0 0c03: 8086:1c2d (rev 04)
00:1b.0 0403: 8086:1c20 (rev 04)
00:1c.0 0604: 8086:1c10 (rev b4)
00:1c.2 0604: 8086:1c14 (rev b4)
00:1d.0 0c03: 8086:1c26 (rev 04)
00:1e.0 0604: 8086:244e (rev a4)
00:1f.0 0601: 8086:1c4e (rev 04)
00:1f.2 0106: 8086:1c02 (rev 04)
00:1f.3 0c05: 8086:1c22 (rev 04)
01:00.0 0300: 1002:68f9
03:00.0 0c03: 1033:0194 (rev 03)

[note - tables and some descriptions adapted from http://wiki.osdev.org/PCI]

Configuration space access

BIOS and OS code must be able to access these configuration registers in order to discover and configure PCI devices. On the PC platform this is done via two 32-bit IO ports, CONFIG_ADDRESS (0xCF8) and CONFIG_DATA (0xCFC). An address specifying bus, device, function, and a 4-byte-aligned byte offset is written to CONFIG_ADDRESS:

31	30 - 24	23 - 16	15 - 11	10 - 8	7 - 0
Enable Bit	Reserved	Bus Number	Device Number	Function Number	Offset

and then a 32-bit value is read from or written to CONFIG_DATA. If no device is present, the result will be 0xFFFFFFFF. (note that 0xFFFF is an invalid vendor code, so the absence of a device can be detected by reading the first 32-bit word of the corresponding configuration space)

Reading configuration space on XV6

For this assignment you will add a user-space utility to XV6 to enumerate the PCI configuration space.

IO port access

First you'll need access to I/O instructions within a user process. The Intel architecture provides two mechanisms which allow access to in and out instructions while in user space:

TSS bitmap - a bitmap can be specified at the end of the TSS which identifies which I/O ports may be accessed from user mode (ring 3).
EFLAGS - bits 12 and 13 of the EFLAGS register indicate the priority level necessary to access any I/O port; by default these bits are zero.

The TSS bitmap hasn't been implemented in XV6, and we might not want to extend it far enough to cover ports 0xCF8 and 0xCFC anyway. (that would take over 400 bytes per process) Instead you'll implement a new system call, iopl, which takes a single argument in the range 0-3 and sets the I/O privilege level bits in EFLAGS to that value; your readpci utility can then call iopl(3) and will be able to access I/O ports directly.

Hint - where does EFLAGS come from when you return to user space? Do you actually have to modify the current EFLAGS register in the iopl system call?

Create a user program named readpci (add it to UPROGS in your makefile) and start out with the following code to test iopl:

#include "types.h"
#include "user.h"
#include "x86.h"

int main(int argc, char **argv)
{
    iopl(3);  // allow ring 3 to access IO ports
    unsigned char c = inb(0x3bc);
    printf(1, "%x\n", c);
    exit();
}

If iopl was implemented correctly it should run, printing FF, while it will die with a fault otherwise.

PCI register access

To access CONFIG_ADDR and CONFIG_DATA you'll need 32-bit versions of the in/out instructions, which aren't provided in XV6:

unsigned int inl(int port)
{
        unsigned int data;
        __asm __volatile("inl %w1,%0" : "=a" (data) : "d" (port));
        return data;
}
void outl(int port, unsigned int data)
{
  __asm __volatile("outl %0,%w1" : : "a" (data), "d" (port));
}

Modify readpci to take 3 numeric arguments - bus, device, and function - and read and print the corresponding configuration space in hex. If you want the printout to look nice like this you'll have to modify printf.c to support formats like "%02x" :-)

xv6...
cpu0: starting
init: starting sh
$ readpci 0 1 0
00: 86 80 00 70 03 00 00 02 00 00 01 06 00 00 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 F4 1A 00 11
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
$

Configuration space structure

All PCI configuration spaces share a common header:

register	bits 31-24	bits 23-16	bits 15-8	bits 7-0
00	Device ID		Vendor ID
04	Status		Command
08	Class code	Subclass	Prog IF	Revision ID
0C	BIST	Header type	Latency Timer	Cache Line Size

The fields are as follows, ignoring ones that we don't need:

Device ID: Identifies the particular device. Where valid IDs are allocated by the vendor.
Vendor ID: Identifies the manufacturer of the device. Where valid IDs are allocated by PCI-SIG to ensure uniqueness and 0xFFFF is an invalid value that will be returned on read accesses to Configuration Space registers of non-existent devices.
Status, Command: for handling various error cases
Class Code: A read-only register that specifies the type of function the device performs.
Subclass: A read-only register that specifies the specific function the device performs.
Prog IF: A read-only register that specifies a register-level programming interface the device has, if it has any at all.
Revision ID: So a vendor can fix something without allocating a new device ID.
BIST: built-in self-test stuff.
Header Type: bit 7 (0x80) indicates whether it is a multi-function device, while interesting values of the remaining bits are: 00 = general device, 01 = PCI-to-PCI bridge.
Latency Timer, Cache Line Size: very hardware-specific stuff. We'll let the BIOS set it properly and ignore it.

Note that all 16 and 32-bit values are in little-endian order. (not surprisingly, as it was originally developed by Intel) This means that you can access them directly on a PC without needing to translate byte order.

For a general-purpose device the remainder of the configuration space is structured as follows:

register	bits 31-24	bits 23-16	bits 15-8	bits 7-0
10	Base address #0 (BAR0)
14	Base address #1 (BAR1)
18	Base address #2 (BAR2)
1C	Base address #3 (BAR3)
20	Base address #4 (BAR4)
24	Base address #5 (BAR5)
28	Cardbus CIS Pointer
2C	Subsystem ID		Subsystem Vendor ID
30	Expansion ROM base address
34	Reserved			Capabilities Pointer
38	Reserved
3C	Max latency	Min Grant	Interrupt PIN	Interrupt Line

The fields (other than base addresses) are:

CardBus CIS Pointer: obsolete
Interrupt Line: PIC IRQ 0-15, or FF (no IRQ)
Interrupt Pin: PCI-specific, or for use with IOAPIC.
Max Latency: hardware-specific
Min Grant: hardware-specific
Capabilities Pointer: never mind...

We'll ignore all of these values for now, and talk about them during class.

PCI bridges

In a PCI bridge this portion of the configuration data has the following structure:

register	bits 31-24	bits 23-16	bits 15-8	bits 7-0
10	Base address #0 (BAR0)
14	Base address #1 (BAR1)
18	Secondary Latency Timer	Subordinate Bus Number	Secondary Bus Number	Primary Bus Number
1C	Secondary Status		I/O Limit	I/O Base
20	Memory Limit		Memory Base
24	Prefetchable Memory Limit		Prefetchable Memory Base
28	Prefetchable Base Upper 32 Bits
2C	Prefetchable Limit Upper 32 Bits
30	I/O Limit Upper 16 Bits		I/O Base Upper 16 Bits
34	Reserved			Capability Pointer
38	Expansion ROM base address
3C	Bridge Control		Interrupt PIN	Interrupt Line

The main field of interest is the Secondary Bus Number, which identifies the "child" bus in the tree. For example, since the root PCI bus is always 0, if there is a bus 1 there must be a PCI bridge on bus 0 with secondary bus number 1.

You can recursively enumerate devices on the PCI bus by scanning bus 0, and whenever you detect a PCI bridge recursively scanning its secondary bus. In doing this keep in mind:

There can be up to 32 devices on a bus. Each device number corresponds to a slot, so some may be missing in the middle. (e.g. empty PCI slots)
There can be up to 8 functions in a multi-function device; however you can stop when you get to the first missing one.
The first PCI bridge is going to be the host bridge, with a configuration record indicating that it bridges from bus 0 to bus 0. Don't enumerate it recursively.

You may find the file pci-cfg.h useful, as it contains C structure definitions for PCI device and bridge configurations.

Modify readpci to recursively enumerate PCI buses. (e.g. if invoked as readpci -scan instead of e.g. readpci 0 1 0)

To test recursive enumeration you will need a newer version of QEMU - you can download the latest version for Ubuntu 12.04.3 (i.e. the virtual machine image most of you are using) as follows:

mkdir -p ~/bin
cd ~/bin
wget http://www.ccs.neu.edu/~pjd/cs7680/qemu-system-i386
chmod +x qemu-system-i386

In your xv6 directory you should then be able to run QEMU as follows:

~/bin/qemu-system-i386 -nographic -hdb fs.img xv6.img -smp 1 -m 512 -bios /usr/share/qemu/bios.bin \
    -device i82801b11-bridge-device rtl8139,bus=pci.1 -device i82801b11-bridge -device ne2k_pci,bus=pci.2

This is the equivalent of running make qemu-nox - if you want the graphical window, eliminate the -nographic argument. It should print several warning messages and then start up, and if you enumerate the PCI bus correctly you'll see devices on buses 1 and 2.

Submit: Your working readpci.c file.

Optional

With the following code modification you can compile readpci.c for Linux with the command gcc -DHOST_VERSION readpci.c -o readpci; you will also need to change all your printf calls from printf(1, "..."); to cprintf("...");. You should then be able to run your readpci command on any Linux system where you have root access. (e.g. your class virtual machine image)

#ifdef HOST_VERSION
#include 
#include 
#include 
typedef unsigned char uchar;
#define cprintf printf
#else
#include "types.h"
#include "user.h"
#include "x86.h"
#define cprintf(...) printf(1, __VA_ARGS__)
#endif

Last modified: Mon Mar 17 23:43:06 EDT 2014