***Please refer to Stratix 10 SoC Design Example for 10Gbe with IEEE1588 PTP Capability - Sustaining - User Manual for sustaining design***

Introduction

IEEE 1588 v2 is a protocol that enables precise synchronization of all real time-of-day (ToD) clocks in a network to a master clock. Diagram below illustrates how PTP synchronization can be achieved by exchanging of timestamp packets between master and slave nodes. The master node will initiate the handshaking by sending a Sync packet. When all 4 t1, t2, t3, t4 timestamps are gathered, slave node can calculate and adjust its time-of-day to achieve synchronization with master node. This handshake process is repeated at specified interval to ensure that slave clock is always synchronized to master clock.

Pre-requisites

This design example is a derivative from Stratix 10 SoC GSRD and it is recommended to familiarize with the development flow of Golden System Reference Design prior attending to this design example.

Overview

This design example presents an example of IEEE 1588v2 2-step FPGA implementation in Quartus Pro v21.2 using Stratix 10 SoC, Low Latency Ethernet 10G MAC with Multi-rate PHY and Linux kernel v5.4 software stack.

This design supports ordinary clock, both PTP Master and Slave mode. PTP timestamping is handled by the hardware while the HPS and software stack handles packet generation and slave ToD adjustment.

Hardware Architecture

Main design components of the hardware design are Low Latency Ethernet 10G MAC, Multi-rate Ethernet PHY, ToD and synchronizer, DMA and HPS system:

• MAC and PHY transmit and receive ethernet packets including PTP packets with timestamp;

• ToD module generates local time-of-day for TX and RX MAC;

• HPS will run Ethernet and PTP stack, include updating ToD. HPS sends and receives packets via DMA.

When PTP is initiated (see section Verify PTP capability using ptp4l tool), the HPS will be sending PTP packets (Sync, Follow-up, etc.). DMA modules will act as data mover between HPS and Ethernet MAC IP. All TX egress timestamps and RX ingress timestamps will be collected.

This design can support both PTP master and PTP slave. When it is configured as PTP master, the HPS will initiate timestamp packet exchange at a fixed interval e.g. once every second. When the design is configured as PTP slave, the HPS will calculate the offset between master and slave based on t1, t2, t3, t4 timestamps collected. It will then perform time-of-day adjustments by writing to its ToD module. The ToD module can support seconds, nanoseconds and fractional nanoseconds adjustment.

Note: In v21.2, standard MSGDMA Prefetcher IP is adopted instead of modified Prefetcher IP used in earlier design version.

Clocking Scheme

Reset Scheme

Software Architecture

Intel SoCFPGA Ethernet driver

1. QSE (MAC driver): intel_fpga_qse_ll_main.c

• Initialization/uninitialization of PHY and MAC, setup of Tx & Rx operations, interrupt routing
• Dynamic SFP module is used to support 10GbE
• PTP support - IOCTLs from application software to enable/disable/configure supported modes of timestamp operations

2. mSGDMA Prefetcher: altera_msgdma_prefetcher.c

• Implements mSGDMA Prefetcher IP core features such as setting up Tx & Rx descriptors, handles DMA transfers

3. ToD: intel_fpga_tod.c

• Registers PTP clock driver to kernel
• Calculates the ToD of clock offset adjustments

Release Content

Folder	File	Description
bin	Image socfpga_stratix10_socdk.dtb ghrd_1sx280lu2f50e2vg_hps.sof	Prebuild binaries for this project. Replace the Image file in a default GSRD Image - kernel Image which enabled QSE driver socfpga_stratix10_socdk.dtb - device tree blob ghrd_1sx280lu2f50e2vg_hps.sof – hardware sof integrated with FSBL
hw	s10_soc_devkit_ghrd_10gbe_1588_21p2.tar.gz	Hardware design example archive, consist of necessary hardware system file to generate .sof image
src	s10_1588_devkit_defconfig	Device config file to enable QSE driver. Config requre for 10G QSE as below: CONFIG_INTEL_FPGA_QSE_LL=y CONFIG_ALTERA_QSE_LL=y CONFIG_PHYLINK=y CONFIG_NET_ALTERA_ETH=y CONFIG_SFP=y
	u-boot.scr	Default GSRD u-boot script to enable F2S bridge

System Requirements

Hardware Requirements

• PTP Master/Slave hardware:
  • 2 Stratix 10 SoC Development Kits (L-Tile) with:
    • HiLo DDR4 memory card (4 GB)
    • HPS peripheral daughter cards (OOBE)
• 2 SFP+ module and fiber optic cable
• 2 USB type A to mini USB type B cables

Software Requirements

This design example is build based on Stratix 10 SoC GSRD (Golden System Reference Design) and tested with Quartus Prime Pro version 21.2 (This reference design will not work on Quartus Prime Standard). It is recommended for user to setup and go through below materials before getting started with this design example.

• Quartus Prime Pro 21.2 Release Notes
• Stratix 10 GSRD
• Linux host PC with Ubuntu 16.04 LTS (or any newer version)
• Embedded software version used in this design example
  • arm-trusted-firmware: socfpga_2_4.1
  • uboot-socfpga: socfpga_v2020.01
  • linux-socfpga: socfpga-5.4.114-lts

Quick Start Guide

This design example leverage on default S10 SoC GSRD, with changes to Kernel Image and Device tree to enable QSE driver. There is no changes in U-boot image. User is required to download and update the Kernel Image and DTB in SD card.

Flash SD Card

1. Create a SD Card image with default S10 GSRD Image.

2. Download release content bin files and replace the following files in SD card:

• Image
• socfpga_stratix10_socdk.dtb
• u-boot.scr (only copy if it is not available in SD card content - required to enable bridge during boot-up)

For more information on how to write SD image to the SD card, please refer to for boot from SD Card, please refer to section "3.Creating SD Card" in Booting Linux with SD Card Image.

Setup hardware connection

1. Connect two Stratix 10 SoC Development Kits using a fiber optic cable via SFP+ port (J7 port). One of the boards will act as PTP Master while the other one as PTP Slave.

Optionally, the development kit can also be paired with X710 network adapter card to connect to a Linux machine.

2. The mini-USB port (J52) on the Stratix 10 SoC Development Kits are connected to a host machine with USB type A to mini-USB type B cables.

Hardware bring up

1. Configure the S10 SoC Development Kit as follows

• SW2: 1:ON 2:ON 3:ON 4: ON (set MSEL to JTAG configuration mode)
• SW4: 1:ON 2:OFF 3:OFF 4:ON

2. After powering up the development kit, launch Clock Controller application, which is part of development kit package installer, to reconfigure FPGA Transceiver Reference Clocks U33. In the application, click on Si5341 tab and change the frequency of OUT2 from 135MHz to 125MHz.Then click on "Set" to configure the clock.

Note: you have to repeat this step for every board power cycle

3. Configure FPGA on the development kit with ghrd_1sx280lu2f50e2vg_hps.sof.

Executing the Reference Design

Verify link is up

1. Linux operating system running on the Stratix 10 Soc Development Kit can be accessed using Serial Communication program such as Putty. Modify the serial line ID based on the COM port connected to the host.

2. Perform step 2-3 of Hardware bring up section (if you haven't already).

3. Type root as the login name when requested.

4. Repeat steps 1-3 for the other Stratix10 SoC Development Kit.

5. When both development kits are powered up and connected, link up status will be shown at console printout:

Assign IP address and verify connection with ping

1. Assign IP address for both boards:

S10 #1> SOCFPGA_STRATIX10 # ifconfig eth0 192.168.0.1
S10 #2> SOCFPGA_STRATIX10 # ifconfig eth0 192.168.0.2

2. Run ifconfig command to verify if the IP address is successfully assigned for each board.

3. Use Ping command to verify connection between the two boards

S10 #1> SOCFPGA_STRATIX10 # ping 192.168.0.2

4. Press Ctrl + C to stop ping packet transmission.

Verify PTP capability using ptp4l tool

In this reference design, ptp4l in LinuxPTP package is used to implement PTP master and slave clock.

1. Enable hardware flow control for both boards with the following command:

SOCFPGA_STRATIX10 # ethtool -A eth0 rx on tx on

Flow control is used to prevent packet drop during high traffic.

2. Setup S10 #1 as master and S10 #2 as slave.

S10 #1> SOCFPGA_STRATIX10 # ptp4l -i eth0 -m
S10 #2> SOCFPGA_STRATIX10 # ptp4l -i eth0 -m -s

3. As the ToD of PTP slave is synchronized to the ToD of PTP master, a convergence of master offset from large to smaller number can be observed in the slave console as shown in the figure below.

PTP master:

PTP slave:

4. Press Ctrl + C on PTP slave board to stop the PTP measurements. Press Ctrl + C on PTP master board to terminate the PTP master.

Verify bandwidth using iperf3 tool

1. Setup S10 #1 as iperf3 master and S10 #2 as iperf3 slave.

S10 #1> SOCFPGA_STRATIX10 # iperf3 -s
S10 #2> SOCFPGA_STRATIX10 # iperf3 -c 192.168.0.1

S10#1 as iperf master

S10#2 as iperf slave

Note:

• -A is used in iperf3 to specify the CPU affinity to improve performance
• Ethernet bandwidth has dependency on CPU maximum performance and NAPI (New API) driver on Ethernet packet handling
• For performance test reference:
  • Ethernet Design Example Demo Testing
  • iperf3 command
  • NAPI performance

Building Hardware Design

IP settings

Tables below show the IP settings for Low Latency Ethernet 10G MAC IP and 1G/2.5G/5G/10G Multi-rate Ethernet PHY IP.

Component	Description
LL 10Gbe MAC	Speed: 1G/2.5G/10G Datapath options: TX & RX Enable ECC on memory blocks: Not selected Enable supplementary address: Selected Enable statistics collection: Selected Statistics counters: Memory-based TX and RX datapath Reset/Default To Enable: Selected Enable time stamping: Selected Enable PTP one-step clock support: Not selected Timestamp fingerprint width: 8 Time Of Day format*: Enable both 96b Time of Day Format only Use legacy Ethernet 10G MAC XGMII interface: selected Use legacy Ethernet 10G MAC Avalon Memory-Mapped interface: Not selected Use legacy Ethernet 10G MAC Avalon Streaming interface: selected
1G/2.5G/5G/10G Multi-rate Ethernet PHY	Connect to MGBASE-T PHY: Selected Connect to NBASE-T PHY: Not selected Speed: 1G/2.5G/10G Enable SGMII bridge: Not selected Enabled IEEE 1588 Precision Time Protocol: Selected PHY ID (32 bit): 0x00000000 VCCR_GXB and VCC_GXB supply voltage for the Transceiver: 1_0V Reference clock frequency for 10GbE (MHz): 644.53125 Selected TX PMA local clock division factor for 1 GbE : 1 Selected TX PMA local clock division factor for 2.5 GbE : 1 Enable Native PHY Debug Master Endpoint: Not selected Enable capability registers: Not selected Enable control and status registers: Not selected Enable PRBS soft accumulators: Not selected

Quartus Project Compilation

1. Retrieve hardware project from s10_soc_devkit_ghrd_10gbe_1588_21p2.tar.gz (refer to Release Content).

2. In Quartus, click Open Project, select ghrd_1sx280lu2f50e2vg.qpf and click Open.

3. Click Processing → Start Compilation and run full compilation to generate .sof file.

Building Software Binaries

This design example leverage on default S10 SoC GSRD, with changes to Kernel Image and Device tree to enable QSE driver. There is no changes in U-boot. For a detail flow of how to generate software binaries, please refer to Stratix 10 SoC GSRD.

Building ATF/U-Boot

For a detail flow of ATF abd U-boot binaries generation, please refer to Starix 10 SoC - Boot from SD Card - Build U-Boot.

Integrating U-boot FSBL to .sof file

1. During full compilation, the Intel Quartus Prime Pro Edition generates the data in the HPS FSBL handoff binary based on the settings inside the Platform Designer. The file contains information on:

• HPS pin multiplexing configuration, including HPS dedicated I/O settings and export to FPGA settings
• HPS clock configuration
• HPS SSBL boot source

2. Run below command to merge both .sof and FSBL:

• $ quartus_pfg -c -o hps_path=<hex_file> <input_sof> <output_sof>

$ quartus_pfg -c -o hps_path=software/hps_debug/u-boot-spl-dtb.hex output_files/ghrd_1sx280lu2f50e2vg.sof output_files/ghrd_1sx280lu2f50e2vg_hps.sof

Building Linux

1. Setup build toolchain - compilation of the software relies on downloading the build toolchain and setting up the CROSS_COMPILE and ARCH environment variables accordingly:

$ cd $TOP_FOLDER
$ wget https://developer.arm.com/-/media/Files/downloads/gnu-a/10.2-2020.11/binrel/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz
$ tar xf gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz
$ rm gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz
$ export CROSS_COMPILE=`pwd`/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-
$ export ARCH=arm64

2. The following instructions are used to build Linux.

$ cd $TOP_FOLDER
$ rm -rf linux-socfpga
$ git clone https://github.com/altera-opensource/linux-socfpga 
$ cd linux-socfpga

2. Checkout socfpga-5.4.114-lts branch after git clone.

$ git checkout -b s10_10g_212 -t origin/socfpga-5.4.114-lts

3. Download and copy the s10_1588_devkit_defconfig file into ../linux-socfpga/arch/arm64/configs/

• s10_1588_devkit_defconfig enable the following driver:
  • CONFIG_INTEL_FPGA_QSE_LL=y
  • CONFIG_ALTERA_QSE_LL=y
  • CONFIG_PHYLINK=y
  • CONFIG_NET_ALTERA_ETH=y
  • CONFIG_SFP=y

4. Rebuild kernel image and DTB with the following command:

$ make clean && make mrproper
$ make s10_1588_devkit_defconfig 
$ make -j8

5.The following files are generated:

File name	Location	Description
Image	arch/arm64/boot/	Linux Image
socfpga_stratix10_socdk.dtb	arch/arm64/boot/dts/altera/	Device Tree Blob

6. Replace the default S10 GSRD image content in the SD card with the newly build kernel image (Image) and DTB (socfpga_stratix10_socdk.dtb).