---

Overview

This document describes how to use Arria V !SoC with PCIe Root Port design example release package. The FPGA design is based on the Golden System Reference Design(GSRD). Newly added modules include: PCIe !RootPort (RP) IP, MSI-toGIC generator IP, MSGDMA and throughput measurement modules. The design demonstrates the Altera PCIe HIP Root Port ability to enumerate a Gen2x4 PCIe Endpoint and measure the link throughput. The Endpoint consists of an Intel® Gigabit CT Desktop Adapter or Cyclone V GT FPGA with PCIe HIP. This example design is provided as a starting point for PCIe system designs. It consists of both hardware designs and software packages.

Figure 1: Altera !SoC !RootPort to Endpoint connectivity through a PCIe Link.

GSRD

The hardware and software designs are based on the GSRD release. Keeping hardware and software design releases in sync is important to ensure compatibility and functionality. This example design is based on Arria V !SoC GSRD, below are the links for information and kit installation.

GSRD: http://www.rocketboards.org/foswiki/Documentation/GSRD131

---

Hardware and Software Packages

Getting Started

This example design is based on Arria V !SoC GSRD. Keeping hardware and software design releases in sync is important to ensure compatibility and functionality.

The example design is built and tested with Quartus II 13.1.

More information for Arria V !SoC development kit:

http://www.altera.com/products/devkits/altera/kit-arria-v-soc.html

The following is a link to the "GSRD User Manual".

http://www.rocketboards.org/foswiki/Documentation/GSRD131

• Step through "Prerequisites", "GHRD Overview" and "Getting Started Guide"
• Understand the "Development Flow Overview"

Release Contents

The PCIe RP example design sources and prebuilt binaries can be downloaded from here.

Folder	File	Desciption
bin	linux-socfpga-pcierp-13.1-av-bin.tar.gz	Arria V binaries archive. All the prebuilt images needed for demo and building the SD Image. sd_image.bin.tar.gz is a 2GB SD card image tar file contains all the require components for demo, including Preloader, U-boot, rootfs, kernel image and device tree blob.
hw	cv_ep_ram_gt_design.tar.gz	DMA End Point device
^	av_soc_rp_full_design.tar.gz	This design contains an additional modular SGDMA in the Root Port design. This DMA machine connects directly from on chip memory to the PCIe Txs data path. This connection is used to perform PCIe throughput measurements. Logic usage is about 22,355 ALMs.
src	altera_pcie_yocto.patch	Patch to yocto build extracted from GSRD kit installation linux-socfpga-gsrd-13.1-src.bsx
^	altera_pcierp_driver.patch	Patch if building kernel with branch “socfpga-3.9-rel”
^	boot.script	U-boot script

Hardware

Hardware require for this example design

• Root Port: Arria V !SoC Development Kit (5ASTFD5)
• Endpoint: Cyclone V GT FPGA Development Kit (5CGTFD9)
• Endpoint: Intel® Gigabit CT Desktop Adapter (Intel® 82574L Gigabit Ethernet Controller)
• 4GB microSDHC flash card

Software

Software required for this example design

• sd_image.bin to be programmed into SD card

Back to Top

---

Setting up the example design platform

Hardware jumper, USB-Blaster, Serial terminal, etc should be set the same as GSRD, please refer to GSRD Getting Started Guide.

Prepare SD card

The release package includes an sd_image.bin, which combined all the software require to boot and demo the example design. There are number of ways to program the SD card. For Windows, “Win32 Disk Imager” (http://sourceforge.net/projects/win32diskimager/) can be used. Follow the GSRD User Manual, untar sd_image.bin.tar.gz and program sd_image.bin into SD card.For Linux, dd utility is used.

Demo: Setup board and boot

Once you have SD card ready, connect the platform as shown in figures below. Again, the base setup must follow GSRD Getting Started Guides.

Insert the SD card programed from Section Prepare SD card. Power up the system. If setting up the Cyclone V GT FPGA Endpoint (Figure 3), configure the end point FPGA image (pcie_ep_ram_5cgtfd9.sof) with Quartus Programmer. Start a serial terminal (!PuTTy or minicom) on the host PC to communicate with the Linux target. Press the warm reset button. Observe Preloader → U-Boot → Linux booting on the serial terminal. Note, U-Boot will auto program the Arria V <nop>SoC Rootport example design (soc_system.rbf in SD card). At the Linux Kernel,login as root: socfpga login: root Execute the following commands: # modprobe altera_rpde # modprobe altera_epde # /home/root/altera/dmaxfer

Note: If just performing the Arria V Soc PCIe !RootPort demo, sections Hardware Development Flow and Software Development Flow can be skipped. Both section will step through the Hardware and Software development flows. These flows list how to test the hardware as well as building the software kernel.

Figure 2:Intel® Gigabit CT Desktop Adapter

Figure 3: Cyclone V GT FPGA Development Kit (5CGTFD9) as End Point device

Back to Top

---

Hardware Development Flow

Note: Unless the Qsys design is required to be modified, please use the prebuilt image in release package. The following steps are only required if modification to the Qsys design is needed.

Altera Complete Design Suite (ACDS) flow is used for developing the hardware design of PCIe RootPort-EndPoint example, from Qsys generation to Quartus full compilation. However, some components, such as the Modular SGDMA and Interrupt Capture Module, are not currently available in the Qsys 13.1 Component Library. These components are included in this example design’s, IP folder. Another point to note is the difference between the Cyclone V & Arria V PCIe Root Port design with MSI is the fact that the datawidth is at 128bit versus the one found in cyclone V which is 64bit wide to accomodate the increased bandwidth required to transport data at PCIe Gen 2 speeds.

The example design package zip file, av_soc_rp_full_design.tar.gz, consists of the following:

• Qsys design files
• Quartus files
• RTL files (including PCIe IP patch)
• Qsys component library files
• RBF binary

Qsys Design and Generation

pcie_rp_ed_5astfd5.qsys is the Qsys top level design file. The design consists of a HPS subsystem, PCIe HIP, Modular SGDMA subsystem, and some peripherals designed for PCIe RP example.

The user is only required to perform Qsys generation for pcie_rp_ed_5astfd5.qsys. Generated RTL will be contained in pcie_rp_ed_5astfd5 folder.

Quartus Compilation

First, you will need to invoke the Quartus GUI, to achieve that, you have to double click on the pcie_rp_ed_5astfd5.qpf icon in the folder which contains your project. After the GUI has successfully initialized, you have to kick-off the following procedures :

1. Run Analysis & Synthesis by clicking on the "play" button under the Task panel.
   • If you prefer to use a clean qsf file versus the one that comes along with the project you have to invoke the hps_sdram_p0_pin_assignments.tcl script so that your new qsf file will be populated by the pin assignments needed by the SDRAM controller residing in the HPS. Skip the steps below altogether if you use the .qsf file that ships together with the tarball.
   • The following screen shot illustrate how to use !QuartusII to select the TCL script to be run.
   • 
   • Figure 4: Location of Tcl Scripts option in Quartus II
   • 
   • Figure 5: Selecting the tcl script to run
2. Kick - off the place and route engine by clicking on the "Fitter (Place & Route)" button.
3. Once fitter has sucessfully completed its task, you have to initiate the sof file generation flow by clicking on the "Assembler (Generate Programming files)" button.Once you have the sof file, you will need to convert it to a raw binary file format by referring to the documents pointed by the link below ;

Convert .sof to .rbf

Please refer to GSRD User Manual to convert FPGA image from .sof to .rbf.

Back to Top

If you happen to run into timing failures when you are compiling the example design, you may refer to Arria V Hard IP for PCI Express User Guide which contains handy information to help you meet timing in your design.

---

Software Development Flow

The complete software flow is similar to GSRD flow, except to patch kernel and yocto before build binaries. This section describes the additional flow require to build binaries.

Note: Unless Qsys design and/or Software are required to be modified, please use the prebuilt image in the release package. Steps below are required only if the Qsys design and/or Software modification is needed.

Figure 8: Overview Diagram

Environment setup based on GSRD

Please follow the setup document in the link below, as these steps are required before starting this section.

http://www.rocketboards.org/foswiki/Documentation/GSRD131GettingStartedYocto

Get Linux BSP of

• Linux kernel
• u-boot
• The tool chain

Export variable

• export CROSS_COMPILE=arm-linux-gnueabihf-
• export ARCH=arm
• PATH=<u-boot directory>/tools/:$PATH

Yocto build

This is a complete build using yocto recipes from the GSRD kit installation and patch that build with PCIe RP components. There is no need for step Individual components build if following steps Building components with yocto and Creating SD Image.

Building components with yocto

Download BSX source from Rocketboards. $ chmod +x linux-socfpga-gsrd-13.1-src.bsx $ ./linux-socfpga-gsrd-13.1-src.bsx ./altera-linux $ ./altera-linux/bin/install_altera_socfpga_src.sh ./yocto Copy altera_pcie_yocto.patch to the Yocto directory $ patch -p1 -i altera_pcie_yocto.patch Change "uImage" to "zImage" in <yocto-path>/meta-altera/conf/machine/socfpga_arria5.conf: $ sed -i s/KERNEL_IMAGETYPE\ \=\ \"uImage\"/KERNEL_IMAGETYPE\ \=\ \"zImage\"/g meta-altera/conf/machine/socfpga_arria5.conf $ source ./altera-init ./build Run command below to change default socfpga_cyclone5 to socfpga_arria in conf/local.conf: $ sed -i s/MACHINE\ \?\?\=\ \"socfpga_cyclone5\"/MACHINE\ \?\?\=\ \"socfpga_arria5\"/g conf/local.conf $ bitbake virtual/kernel (generate zImage) $ bitbake virtual/bootloader (generate u-boot-socfpga_arria5.img) $ bitbake altera-pcie-image (generate altera_rpde.ko, altera_epde.ko, dmaxfer and altera-pcie-image-socfpga_arria5.tar.gz) Build preloader image file as described in GSRD Replace all required components into SD Card, or build and replace the whole image as described in Creating SD Image

Creating SD Image

For more information about creating SD card, please refer GSRD User Manual - SD Card.

$ sudo tar xzf ./yocto/build/tmp/deploy/images/altera-pcie-image-socfpga_arria5.tar.gz -C /tmp/rootfs $ sudo sh ./altera-linux/bin/make_sdimage.sh -k zImage,socfpga.dtb,soc_system.rbf,u-boot.scr -p preloader-mkpimage.bin -b u-boot-socfpga_arria5.img -r /tmp/rootfs -o /tmp/sd_image.bin -g 2G Follow section Prepare SD card to program SD card

Individual components build

Note: Steps for quickly performing a component build for testing and debugging.

Quickly building and debugging individual components is preferable in order to save time. Steps below allow components to be built individually.

Building kernel from Rocketboards.org

Download linux-socfpga.git from rocketboards.org branch “socfpga-3.9-rel” http://rocketboards.org/foswiki/Documentation/GitWeb $ git clone http://git.rocketboards.org/linux-socfpga.git $ cd linux-socfpga $ git checkout -b socfpga-3.9-rel-pcie origin/socfpga-3.9-rel Copy altera_pcierp_driver.patch the kernel root source tree Check if patch can be applied $ git apply --check altera_pcierp_driver.patch $ git am altera_pcierp_driver.patch $ make socfpga_defconfig $ make zImage Copy and replace zImage into SD Card

Building Device Tree Blob

For more details about device tree generation, please refer to GSRD User Manual - Generating the Device Tree.

$ sopc2dts --input <pcie_design>.sopcinfo --output socfpga.dts --board <pcie_design>_board_info.xml --board hps_clock_info.xml --bridge-removal all You can found *.sopcinfo and *.xml files from PCIe example design hardware directory. $ dtc -I dts -O dtb -o socfpga.dtb socfpga.dts Copy and replace socfpga.dtb into SD card.

Building Host system driver and End point device driver

Both device drivers are built as Loadable Kernel Module (LKM), and load into Kernel at run time after system booted.

After applying altera_pcie_yocto.patch in section Building components with yocto. Go to <Yocto installation directory>/meta-altera/recipes-pcie/pcie-ed/files/ $ make KERNEL_SRC=<path-to-your-Linux-kernel-directory> Copy altera_rpde.ko and altera_epde.ko into SD card. Use insmod command to load these 2 kernel modules on SOCFPGA terminal.

Building Throughput Linux application

The application is compiled with ARM cross-compiler.

$ arm-linux-gnueabihf-gcc -o dmaxfer dmaxfer.c

Kernel Configuration

Disable MSI

The following steps illustrate how to disable MSI.

Go to Linux kernel top directory and type “make menuconfig“ Disable Bus support→Message Signaled Interrupts (MSI and MSI-X) Save and exit Recompile kernel

Enable MSI

The following steps illustrate how to enable MSI.

Go to Linux kernel top directory and type “make menuconfig“ Enable Bus support→Message Signaled Interrupts (MSI and MSI-X) Enable Bus support→PCI host controller drivers→Altera PCIe controller→Altera MSI-to-GIC support Save and exit Recompile kernel

PCI Related Linux Commands

Here is a list of PCI related Linux commands.

lspci is the standard tool to query the devices connected to any PCI compatible bus. Type “lspci --help” for more details. $ lspci –v $ lspci -x To list vendor and device IDs $ cat /proc/bus/pci/devices To access the PCI resources on platform $ cat /sys/devices/pci0000:00

Back to Top

---

Open Issues

FB 118697 – PCIe RP Subsystem Vendor and Device ID do not show up in PCI config

Problem: RP Subsystem Vendor and Device ID does not show up in PCI configuration space.

Implication: Customer/OEM designed RP system will not able to load driver that come with Subsystem Vendor and Device ID.

Workaround: Alternatively use Vendor and Device ID only

Status: Quartus II 14.0

Back to Top

---

References

1. Arria V Hard IP for PCI Express User Guide
2. GSRD User Manual
3. Cyclone V PCIe Root Port with MSI Example Design

Back to Top