***Please refer to Stratix 10 SoC Design Example for 10Gbe with IEEE1588 PTP Capability - Sustaining - User Manual for sustaining design***

Introduction

IEEE 1588 v2 is a protocol that enables precise synchronization of all real time-of-day (ToD) clocks in a network to a master clock. Diagram below illustrates how PTP synchronization can be achieved by exchanging of timestamp packets between master and slave nodes. The master node will initiate the handshaking by sending a Sync packet. When all 4 t1, t2, t3, t4 timestamps are gathered, slave node can calculate and adjust its time-of-day to achieve synchronization with master node. This handshake process is repeated at specified interval to ensure that slave clock is always synchronized to master clock.

Pre-requisites

This design example is a derivative from Stratix 10 SoC GSRD and it is recommended to familiarize with the development flow of Golden System Reference Design prior attending to this design example.

Overview

This design example presents an example of IEEE 1588v2 2-step FPGA implementation in Quartus Pro v20.1 using Stratix 10 SoC, Low Latency Ethernet 10G MAC with multi-rate PHY and Linux kernel v5.4 software stack.

This design supports ordinary clock, both PTP Master and Slave mode. PTP timestamping is handled by the hardware while the HPS and software stack handles packet generation and slave ToD adjustment.

Hardware Architecture:

Main design components of the hardware design are Low Latency Ethernet 10G MAC, Multi-rate Ethernet PHY, ToD and synchronizer, DMA and HPS system:

• MAC and PHY transmit and receive ethernet packets including PTP packets with timestamp;

• ToD module generates local time-of-day for TX and RX MAC;

• HPS will run Ethernet and PTP stack, include updating ToD. HPS sends and receives packets via DMA.

When PTP is initiated (see section Verify PTP capability using ptp4l tool), the HPS will be sending PTP packets (Sync, Follow-up, etc.). DMA modules will act as data mover between HPS and Ethernet MAC IP. All TX egress timestamps and RX ingress timestamps will be collected.

This design can support both PTP master and PTP slave. When it is configured as PTP master, the HPS will initiate timestamp packet exchange at a fixed interval e.g. once every second. When the design is configured as PTP slave, the HPS will calculate the offset between master and slave based on t1, t2, t3, t4 timestamps collected. It will then perform time-of-day adjustments by writing to its ToD module. The ToD module can support seconds, nanoseconds and fractional nanoseconds adjustment.

Software Architecture:

Intel SoCFPGA Ethernet driver

1. QSE (MAC driver): intel_fpga_qse_ll_main.c

• Initialization/uninitialization of PHY and MAC, setup of Tx & Rx operations, interrupt routing
• Dynamic SFP module is used to support 10GbE
• PTP support - IOCTLs from application software to enable/disable/configure supported modes of timestamp operations

2. mSGDMA Prefetcher: altera_msgdma_prefetcher.c

• Implements mSGDMA Prefetcher IP core features such as setting up Tx & Rx descriptors, handles DMA transfers

3. ToD: intel_fpga_tod.c

• Registers PTP clock driver to kernel
• Calculates the ToD of clock offset adjustments

Release Content

Folder	File	Description
bin	s10_1588_sdimage.tar.gz	Prebuild binaries archive for this design example. It contains all the required components for quick start purpose: sdimage.img - SD card image file that comes with u-boot, rootfs, kernel source, kernel image and Device Tree Blob ghrd_1sx280lu2f50e2vg_hps.sof – hardware sof integrated with FSBL
hw	s10_soc_devkit_ghrd_qse_10gbe_1588.tar.gz	[Updated 09/03/2020] Hardware design example archive, consist of necessary hardware system file to generate .sof image
src	qse_1588_patch.tar.gz	Linux patches archive which consist of 10Gbe QSE 1588 driver, DTS and defconfig file.
	u-boot.scr	U-boot script to enable bridge settings during boot up

System Requirements

Hardware Requirements

• PTP Master/Slave hardware:
  • 2 Stratix 10 SoC Development Kits (H-Tile / L-Tile) with:
    • HiLo DDR4 memory card (4 GB)
    • HPS peripheral daughter cards (OOBE)
• 2 SFP+ module and fiber optic cable
• 2 USB type A to mini USB type B cables
• Optional slave hardware:
  • Intel Ethernet Network Adapter X710 card + Linux host PC

Software Requirements

This design example is build based on Stratix 10 SoC GSRD (Golden System Reference Design) and tested with Quartus Prime Pro version 20.1 (This reference design will not work on Quartus Prime Standard). It is recommended for user to setup and go through below materials before getting started with this design example.

• Quartus Prime Pro 20.1 Release Notes
• Stratix 10 GSRD User Manual
• Stratix 10 GSRD 20.1
• Linux host PC with Ubuntu 16.04 LTS (or any newer version)

Quick Start Guide

Flash SD Card

The SD card image file is provided in "s10_1588_sdimage.tar.gz" package (refer to Release Content).

Follow the instructions under "Create SD Card" section in "Booting Linux with SD Card Image" to create a boot-able SD card with this image file.

Setup hardware connection

1. Connect two Stratix 10 SoC Development Kits using a fiber optic cable via SFP+ port (J7 port). One of the boards will act as PTP Master while the other one as PTP Slave.

Optionally, the development kit can also be paired with X710 network adapter card to connect to a Linux machine.

2. The mini-USB port (J52) on the Stratix 10 SoC Development Kits are connected to a host machine with USB type A to mini-USB type B cables.

Hardware bring up

1. Configure the S10 SoC Development Kit as follows

• SW2: 1:ON 2:ON 3:ON 4: ON (set MSEL to JTAG configuration mode)
• SW4: 1:ON 2:OFF 3:OFF 4:ON

2. After powering up the development kit, launch Clock Controller application, which is part of development kit package installer, to reconfigure FPGA Transceiver Reference Clocks U33. In the application, click on Si5341 tab and change the frequency of OUT2 from 135MHz to 125MHz.Then click on "Set" to configure the clock.

Note: you have to repeat this step for every board power cycle

3. Configure FPGA on the development kit with ghrd_1sx280lu2f50e2vg_hps.sof.

Executing the Reference Design

Verify link is up

1. Linux operating system running on the Stratix 10 Soc Development Kit can be accessed using Serial Communication program such as Putty. Modify the serial line ID based on the COM port connected to the host.

2. Perform step 2-3 of Hardware bring up section (if you haven't already).

3. Type root as the login name when requested.

4. Repeat steps 1-3 for the other Stratix10 SoC Development Kit.

5. When both development kits are powered up and connected, link up status will be shown at console printout:

Assign IP address and verify connection with ping

1. Assign IP address for both boards:

S10 #1> SOCFPGA_STRATIX10 # ifconfig eth0 192.168.0.1
S10 #2> SOCFPGA_STRATIX10 # ifconfig eth0 192.168.0.2

2. Run ifconfig command to verify if the IP address is successfully assigned for each board.

3. Use Ping command to verify connection between the two boards

S10 #1> SOCFPGA_STRATIX10 # ping 192.168.0.2

4. Press Ctrl + C to stop ping packet transmission.

Verify PTP capability using ptp4l tool

In this reference design, ptp4l in LinuxPTP package is used to implement PTP master and slave clock.

1. Enable hardware flow control for both boards with the following command:

SOCFPGA_STRATIX10 # ethtool -A eth0 rx on tx on

Flow control is used to prevent packet drop during high traffic.

2. Setup S10 #1 as master and S10 #2 as slave.

S10 #1> SOCFPGA_STRATIX10 # ptp4l -i eth0 -m
S10 #2> SOCFPGA_STRATIX10 # ptp4l -i eth0 -m -s

3. As the ToD of PTP slave is synchronized to the ToD of PTP master, a convergence of master offset from large to smaller number can be observed in the slave console as shown in the figure below.

PTP master:

PTP slave:

4. Press Ctrl + C on PTP slave board to stop the PTP measurements. Press Ctrl + C on PTP master board to terminate the PTP master.

Verify bandwidth using iperf3 tool

1. Setup S10 #1 as iperf3 master and S10 #2 as iperf3 slave.

S10 #1> SOCFPGA_STRATIX10 # iperf3 -s
S10 #2> SOCFPGA_STRATIX10 # iperf3 -c 192.168.0.1

S10#1 as iperf master

S10#2 as iperf slave

Note:

• -A is used in iperf3 to specify the CPU affinity to improve performance
• Ethernet bandwidth has dependency on CPU maximum performance and NAPI (New API) driver on Ethernet packet handling
• For performance test reference:
  • Ethernet Design Example Demo Testing
  • iperf3 command
  • NAPI performance

Building Hardware Design

IP settings

Tables below show the IP settings for Low Latency Ethernet 10G MAC IP and 1G/2.5G/5G/10G Multi-rate Ethernet PHY IP.

Component	Description
LL 10Gbe MAC	Speed: 1G/2.5G/10G Datapath options: TX & RX Enable ECC on memory blocks: Not selected Enable supplementary address: Selected Enable statistics collection: Selected Statistics counters: Memory-based TX and RX datapath Reset/Default To Enable: Selected Enable time stamping: Selected Enable PTP one-step clock support: Not selected Timestamp fingerprint width: 8 Time Of Day format*: Enable both 96b Time of Day Format only Use legacy Ethernet 10G MAC XGMII interface: selected Use legacy Ethernet 10G MAC Avalon Memory-Mapped interface: Not selected Use legacy Ethernet 10G MAC Avalon Streaming interface: selected
1G/2.5G/5G/10G Multi-rate Ethernet PHY	Connect to MGBASE-T PHY: Selected Connect to NBASE-T PHY: Not selected Speed: 1G/2.5G/10G Enable SGMII bridge: Not selected Enabled IEEE 1588 Precision Time Protocol: Selected PHY ID (32 bit): 0x00000000 VCCR_GXB and VCC_GXB supply voltage for the Transceiver: 1_0V Reference clock frequency for 10GbE (MHz): 644.53125 Selected TX PMA local clock division factor for 1 GbE : 1 Selected TX PMA local clock division factor for 2.5 GbE : 1 Enable Native PHY Debug Master Endpoint: Not selected Enable capability registers: Not selected Enable control and status registers: Not selected Enable PRBS soft accumulators: Not selected

Quartus Project Compilation

1. Retrieve hardware project from s10_soc_devkit_ghrd_qse_10gbe_1588.tar.gz (refer to Release Content).

2. In Quartus, restore the Quartus project by clicking Open Project, select ghrd_1sx280lu2f50e2vg.qpf.

3. Click Processing → Start Compilation and run full compilation to generate .sof file.

Building Software Binaries

For a detail flow of how to generate software binaries, please refer to Stratix 10 SoC GSRD.

Building U-Boot

Please refer to S10 GSRD User Manual: Compile U-Boot FSBL and SSBL to generate First Stage Boot Loader (FSBL) to integrate with hardware .sof file.

Integrating u-boot FSBL to .sof file

1. For detailed steps, please refer to S10 GSRD User Manual: Compiling the Hardware Design and Integrating U-boot FSBL

2. During full compilation, the Intel Quartus Prime Pro Edition generates the data in the HPS FSBL handoff binary based on the settings inside the Platform Designer. The file contains information on:

• HPS pin multiplexing configuration, including HPS dedicated I/O settings and export to FPGA settings
• HPS clock configuration
• HPS SSBL boot source

3. Run below command to merge both .sof and FSBL:

• $ quartus_pfg -c -o hps_path=<hex_file> <input_sof> <output_sof>

$ quartus_pfg -c -o hps_path=software/u-boot/spl/u-boot-spl-dtb.ihex output_files/ghrd_1sx280lu2f50e2vg.sof ghrd_1sx280lu2f50e2vg_hps.sof

Building Linux

1. Please refer to S10 GSRD User Manual: Compiling Linux for Stratix 10 on tool chain setup and cloning git trees to build Linux kernel.

2. Checkout socfpga-5.4-lts branch after git clone. Please refer to S10 SoC GSRD 20.1 for tagging information.

$ git checkout ACDS20.1_REL_GSRD_PR

3. Extract qse_1588_patch.tar.gz and copy the patch folder into linux-socfpga directory. The patch folder contains 22 patches. Apply these patches to the kernel source with the following command:

$ git am all/*.patch

4. Rebuild kernel image and DTB with the following command:

$ make S10_1588_devkit_defconfig 
$ make -j8

5.The following files are generated:

File name	Location	Description
Image	arch/arm64/boot/	Linux Image
socfpga_stratix10_socdk.dtb	arch/arm64/boot/dts/altera/	Device Tree Blob

6. Replace the default S10 GSRD image content in the SD card with the newly build kernel image (Image) and DTB (socfpga_stratix10_socdk.dtb).