Embedded Linux Beginners Guide

Beginners guide to using embedded Linux on Altera SoC devices

Introduction

This guide will walk you through every step of the process to go from a custom design for an Altera SoC to a shiny new embedded Linux device. This is meant for engineers who are both new to working with embedded Linux on Altera SoC’s, as well as those who are new to embedded Linux in general. No matter your current skill level, every step of the process is explained in detail, and for those parts that need more explanation, links to other resources are available.

Prerequisites

• This lab will be using an Atlas/DE0-Nano-Soc development kit (henceforth, just “Atlas board”) although most of the material in this lab applies to any Altera SoC product. The main differences between dev boards are in the GHRD (number of LEDs, pinout, etc.) and the device tree. Those sections will need modifications when you get to them (although, your dev kit manufacturer should have been nice enough to give you a custom GHRD and device tree board XML file)
• Basic experience with Quartus/Qsys as well as how to use Linux
• A blank microSD card and microSD card USB adapter (or a laptop with microSD card support).
• A Linux PC with two USB ports and sudo support for the following commands:
  • mount, umount, fdisk, losetup, partprobe, and dd
• The required Altera software (Quartus and SoC EDS). You don’t need DS-5 installed: we’ll be writing our makefiles by hand. This guide was completed with v15.0 of the software
• Access to GitHub
• A terminal emulator program

Agenda

1. Overview of the hardware (Quartus/Qsys) design
2. Core concepts and boot flow
3. Generate the preloader
4. Configure and compile U-Boot
5. Generate and compile the device tree
6. Create an SD card and test the design so far
7. Configure and compile the Linux kernel
8. Generate a root filesystem
9. Test the final hardware system
10. Create a user-space software application
11. Create a kernel-space device driver

Don’t worry if you don’t understand all of the words used above (device tree, U-Boot, etc.). All will be explained in due time young padawan.

Set the MSEL Switches on the Atlas Board

This is important! If you don’t set these correctly, the FPGA will not get programmed!

To be able to program the FPGA from the processor (which is commonly how embedded Linux systems on SoCs work), the MSEL switches need to all be placed into the “ON” (aka, 0) position. This will tell the processor to program the FPGA in “Fast Passive Parallel x16” mode as described in the Atlas User Manual (page 13). This is the same mode used to generate the .rbf file (described at the end of the next section). An .rbf file is the raw binary file bitstream that is used to configure an FPGA.

If the modes don’t match, the bootloader will throw an error then continue with booting Linux (which will most likely boot all the way through, making the user oblivious to the fact that their FPGA never got programmed). If you create a compressed .rbf file (or use one given to you from a GHRD, which are usually compressed) you will need to change the MSEL switches to “ON-OFF-ON-OFF-ON-ON”.

Hardware Design Overview

The hardware design that will be used in this guide is a slightly modified version of the Golden Hardware Reference Design (GHRD) for the Atlas board (view the attached files below, it's atlas_linux_ghrd.tar.gz). If you’re new to Altera SoC’s, the GHRD is a hardware design that contains many of the basic components needed to build an SoC design (including an already parameterized Hard Processor System IP). This GHRD is usually provided by the development board manufacturer and is a good starting point for many designs. In our case, Terasic, the makers of the Atlas board, have the unmodified GHRD in the “System CD” that can be downloaded from their website (in the Demonstrations/SoC_FPGA/DE0_NANO_SOC_GHRD folder).

The only modification that was made to the GHRD was the PIO (Parallel I/O) that drove the LEDs was removed and replaced with a single custom component. This is to simulate the beginning process of any user’s design: start off with the GHRD, remove what you don’t need, and add in your own custom components. Throughout the rest of this guide we’ll learn how to integrate and use this custom component throughout every step of the embedded Linux development process.

I encourage you to open up the Quartus (soc_system.qpf) and Qsys (soc_system.qsys) projects and examine exactly what’s going on. The custom LED component placed inside the Qsys system is shown below:

The Custom Component

The component that was introduced into this system is a simple Avalon Memory-Mapped slave device that allows the HPS to read and write a single 8-bit register. That 8-bit register is then exported from the Qsys system so it can be connected to in the top-level Verilog file. In the top-level file, this exported “leds” signal is connected to the eight LED pins on the Atlas board. In other words, we’ve created a very simple output-only PIO (Parallel I/O, aka, GPIO) module. Later on, we will write Linux software to write to this register and change the state of the LEDs.

For more resources on how to build your own custom Qsys components and how Avalon-MM interfaces work, check out the links at the bottom of this page. The source code is shown below:

Generate the Raw Binary File Needed to Program the FPGA

When you finish compiling a design in Quartus, a .sof file (SRAM Object File) is created. This can then be used to program the FPGA portion of the SoC with a JTAG programmer. When booting embedded Linux, a common boot scheme is to have the HPS (Hard Processor System) boot up first and program the FPGA. In our case specifically, the bootloader (described in a future section) will program the FPGA for us before Linux starts up. To do that, it needs an .rbf file instead of a .sof file.

The .rbf file has been created for you in this design! Only do the following if you really want to.

If you are working on a custom design and want to know how to generate an .rbf file, follow the steps below:

1. Within Quartus, open up the Convert Programming Files window (File→Convert Programming Files).
2. In “Programming file type” select “Raw Binary File (.rbf)”
3. In “Mode” select “Passive Parallel x16” (this is the default mode for programming from Linux).
4. Choose a name for the resulting file (I named mine soc_system.rbf).
5. Click on “SOF Data” in the “Input files to convert” section and then click the “Add File…” button and browse for your .sof file.
6. Your window should like the one below:

   

7. Click Generate and your .rbf file should be created

Resources

• Terasic’s Atlas-SoC Download Page
  If you want to download all of the datasheets, user manuals, unmodified GHRDs, etc. for your Atlas board, then this is the place to go.
• Creating Qsys Components
  This is the guide to building a custom Qsys component. Contains nearly all the information you’d need to know on the subject.
• Creating Custom Qsys Components with the Qsys Component Editor
  A short YouTube video on creating custom components.
• Avalon Interface Specifications
  The seminal resource on how Avalon works.

Core Concepts and Boot Flow

Every Altera SoC-based embedded Linux design is going to follow a similar process to boot up. Coincidentally, it’s also a good idea to have a small understanding of what each of the steps in the process are before digging down into the details of each one. This is the “Big Picture” if you will. A slide from the second Altera SoC Workshop sums it up pretty nicely:

As can be seen above, the first piece of software that gets run is the Boot ROM, which then loads up the Preloader, which then loads up U-boot, which then loads up Linux. This chain of software is called the “boot flow”, and for most embedded Linux designs on Cyclone V and Arria V SoC’s, there shouldn’t be much deviation from this pattern.

The binaries for the preloader, u-boot, and Linux can be placed on many different types of non-volatile memory devices (QSPI flash, SD Card, etc.). To know where the preloader (the second stage in the boot process) is located, the board designer will need to set the BSEL (Boot Select) pins. On the Atlas board, these pins have been hardcoded into a configuration that tells the Boot ROM to look on the SD Card to find the preloader. If you’re using a different board, consult the Cyclone V Technical Reference Manual for information on what to set the BSEL pins to. Later on in this guide, you will learn how to partition a microSD Card so the Boot ROM can navigate it.

Simplified Linux Kernel Boot Flow

And although that graphic does a good job at explaining the lower level details of getting Linux started, you might be wondering about what happens after Linux starts up? You might be thinking to yourself “Surely there has to be more to it than a couple of arrows pointing at sparsely colored boxes?” And to that, I respond “Yes, you’re right!”

When the Linux kernel boots up, it starts off by performing a lot of low-level architecture specific initialization sequences (setting up the processors registers, memory management unit, interrupt controller, etc.). And as early as possible, loads up a serial driver so it can output debug messages to help developers realize how they screwed up the previous four steps in the boot flow.

After that, it starts initializing all of the kernel subsystems and drivers that were compiled into the kernel (when you compile the kernel, you get to choose what goes in and what doesn’t). Lastly, it attempts to mount the “root filesystem.” This is the filesystem that contains your shell and all of the programs you want to run.

If the kernel was able to mount your root filesystem (you did remember to compile in the driver for the filesystem you’re using, right?) it will then attempt to run the “/sbin/init” program. This is the first program to run in user-space. This is also the program that will start all of your other programs. There are many different init programs out there (systemd, upstart, SysVinit, etc.), although in this guide we will use an init system provided by an awesome collection of software called BusyBox that is geared for embedded systems (this is all explained in the “Generate a root filesystem” section).

The following slide from Free Electrons’ Embedded Linux development course sums this up nicely:

Once the Linux system has fully booted and you’re placed at a login prompt, you can start tailoring your embedded Linux system to fit your specific application’s needs. If you need networking support, then you should probably include some network management software, and write a script that runs at startup to initialize everything. Want graphics? Maybe throw in a framebuffer device as well as some libraries to make creating graphics applications easier.

By the end of this guide you will have a fully functioning—albeit, minimal—embedded Linux distribution that you can then customize to fit your specific application’s needs. Continue on to the next section to learn how to create the preloader.

Resources

• Free Electron’s course on Embedded Linux Development
  Free Electrons does an amazing job at explaining various aspects of embedded Linux development in this training. Definitely recommended reading for learning more about general embedded Linux concepts.
• Altera SoC Workshop #2
  This workshop does a good job at explaining many of the SoC specific aspects of embedded Linux development. If you don’t know much about your SoC, start by flipping through these slides.
• Altera Application note on the HPS Boot Flow
  Another official resource on learning about the different ways the chip can boot up.

Generating the Preloader

The Preloader—or Secondary Program Loader (SPL)—is a piece of software that gets called from the Boot ROM with the sole purpose of setting up the system to a point that the actual bootloader (U-Boot) can be run. This involves setting up the PLLs and clocks, muxing the pins, initializing the SDRAM, and more. Since the configuration for the system can change every time you modify your Qsys system (the preloader is dependent on many of the HPS’s configuration options), you have to re-generate the preloader every time you change your hardware design. Quartus will give you a helpful (annoying?) warning whenever you need to regenerate your preloader.

Luckily, generating a preloader isn’t as complicated as it might seem (especially once you’ve set up a Board Support Package settings file). There’s a handy GUI tool called the BSP Editor (the terms ‘BSP’ and ‘Preloader’ are used synonymously) that handles most of the heavy lifting. This tool takes in all of the hardware/software handoff files (these are files generated by Qsys/Quartus that tell the BSP Editor how to configure the hardware for the user’s configuration) and any configuration options that you want to set, and outputs a set of source code files that can be compiled into a proper preloader image.

The preloader is actually based off of the U-Boot secondary program loader—which is a chunk of code that knows exactly how to start up U-Boot—with some modifications done by Altera to make it work on their SoC products. That’s why the BSP editor only bothers to generate the code specific to the user’s hardware configuration, and then copies and pastes the rest of the source code from the SoCEDS and ARM Development Studio directory at compile time (“U-Boot Source Code Archive” in the above graphic).

This final set of source code is compiled with a simple call to make (a makefile will be generated by the BSP Editor).This will produce a preloader image that the Boot ROM can load up. Later on we will take this preloader image and burn it to an SD card to actually test our system.

As an aside, most of the following information I have received from various parts of the Rocketboards website. The resources list at the bottom of this page points to more places you can get help on this subject.

Starting up the BSP Editor

If you haven’t done so already, download the atlas_linux_ghrd.tar.gz file and uncompress it into a folder of your choosing.

tar –xzvf atlas_linux_ghrd.tar.gz

Next, start up the embedded command shell. This shell sets up many of the environment variables and paths needed to run the tools used later in this guide. If you close the shell, remember to re-open it before continuing with the guide.

<path-to-soceds-tools>/embedded/embedded_command_shell.sh

Change into the atlas_linux_ghrd folder you just uncompressed.

cd atlas_linux_ghrd

Now start up the BSP Editor

bsp-editor &

The BSP Editor GUI should appear. If you’ve ever used the BSP Editor for the Nios II Build Tools, you’ll notice it looks awfully similar to this BSP Editor. So if you’re used to that BSP Editor before, you’ll be right at home (but if not, don’t worry, the BSP Editor isn’t complicated to use).

In the BSP Editor window, select File→New HPS BSP... to create a new BSP project for your board’s preloader. The following window should appear.

Click on the “…” button next to “Preloader settings directory”, browse to “atlas_linux_ghrd/hps_isw_handoff/soc_system_hps_0” and click Open. This tells the BSP Editor where our handoff files are located.

The window should now have all of its settings automatically populated. Click OK to accept the default settings.

Configuring the Preloader

In the window that appears we can configure the code that will be generated by the BSP Editor. In reality, most of the default settings are exactly what we want, but let's step through a few of the more important settings to get a feel for what we can customize.

Under “Settings”, in the tree view on the left, click on “Common.” This will show you the most important settings: boot options that tell the preloader where to find the bootloader. Just like we had to tell the Boot ROM where to find the preloader with the BSEL pins, these options tell the preloader where to locate the bootloader. As can be seen, you can place the bootloader in QSPI Flash, NAND Flash, RAM, or on an SD Card. Ensure BOOT_FROM_SDMMC is enabled since we’ll be putting U-Boot on our SD Card.

We’ll also be placing U-Boot on a FAT partition on said SD Card, so make sure to enable FAT_SUPPORT and that the FAT_BOOT_PARTITION is set to “1” (when we format our SD Card, we’ll make sure the partition we put U-Boot on, will be partition 1). Lastly, ensure that FAT_LOAD_PAYLOAD_NAME is set to “u-boot.img”. As can be inferred, this is the name of the U-Boot image the preloader will look for.

If we didn’t enable FAT_SUPPORT, we’d have to place the bootloader at address 0x40000 on the SD Card for the system to find the bootloader successfully. In the tree view on the left, select “Advanced→spl→reset_assert”. If any of these checkboxes are enabled, the preloader will keep that part of the HPS in reset. We don’t need to reset anything, so keep these boxes unchecked.

Under “Advanced→spl→boot” you can find options for enabling the watchdog timer, checksum checking, and SDRAM scrubbing. If you’re development board has RAM with ECC storage, you should enable SDRAM scrubbing here (the Atlas board doesn’t so I’ll keep this feature off).

Lastly, under “Advanced→spl→performance” notice how the “SERIAL_SUPPORT” box is enabled. This means that the preloader will print out status messages over the UART as its running. I encourage you to read through the rest of the available options (there aren’t too many) to get a feel for what you can do with the preloader. To get help information for any option, expand the tree view on the left until you can see the individual options, and then click on the one you want to see help for.

Click on Generate in the bottom right corner, and when that’s done click Exit.

Compiling the Preloader

Go back to your terminal and cd into the new “software/spl_bsp” directory and ls to see the new files.

cd software/spl_bsp ; ls

Of particular importance are the “generated” folder, and the “Makefile” and “settings.bsp” files. The “generated” folder contains all of the generated source code from the BSP Editor. This code will be merged with code from U-Boot stored in the SoCEDS and ARM Development Studio directory when we compile. The “settings.bsp” file contains all of the options that you just set within the BSP Editor GUI. And lastly, we’ll use the Makefile to generate the preloader image. (There also a couple of files that work with DS-5, but we’ll ignore those).

make

After compilation, you’ll find a new “uboot-socfpga” directory which contains a subfolder called “spl” that contains the code for the secondary program loader that the generated code was merged with. There’s also a new “preloader-mkpimage.bin” file that contains both the preloader image and a “Boot ROM Header” that is required for the Boot ROM to recognize this as an actual preloader image. We will burn this file to an SD card later on.

Continue onto the next section to learn how to compile U-Boot.

Resources

• Altera SoC Workshop #2 – Lab 1
  This lab is the source of quite a bit of the material on this page. With that said, I tried to add explanations that I felt were missing. If my explanations didn’t stick with you, definitely try this lab.
• Preloader and U-Boot Customization
  This is the premier guide for anyone who wants to understand the preloader at a much lower level. If you want to modify the preloader to support your own custom board, this is also the place to learn how. The graphic at the bottom of this page gives a very detailed description of how the preloader works; I’d recommend taking a glance.

Configure and Compile U-Boot

At this stage in the boot flow, the low-level aspects of our SoC (clocks, pins, and SDRAM) have been initialized to the point where the next boot stage can begin. That boot stage being the bootloader of course. In terms of bootloaders, U-Boot is vastly the most popular bootloader used in embedded Linux designs (occasionally you’ll see a vendor push a different bootloader for their SoC, but most have moved over to U-Boot).

U-Boot’s sole purpose is to get the system up and running to the point where the Linux Kernel can be started up. This is akin to how the preloader’s sole purpose was to get the system initialized enough to run the bootloader. Each stage in the boot flow enables only the hardware needed to run the next stage. This process allows for a leaner, faster boot flow.

In our specific case, U-Boot will need to initialize the SD/MMC hardware at a minimum because our kernel will be on an SD Card. U-Boot contains the drivers needed to initialize the SD/MMC hardware as well as to read the FAT filesystem on the SD Card. U-Boot is also responsible for passing a series of arguments to the kernel called “boot arguments” that tell the kernel low-level details of how to boot up (what serial port to print debug messages, where to find the root filesystem, etc.).

U-Boot also performs another task very important to Altera SoC devices: programming the FPGA fabric. This isn’t an ability built into the mainline U-Boot code so we’ll be using Altera’s special blend of U-Boot. And although you can write software to program the programmable logic fabric from either the preloader or from Linux, the recommended method is to have U-Boot load up the bitstream and program the FPGA before the Linux kernel begins. This way, as the kernel is booting up it can initialize the drivers needed for your custom logic before any applications are run that may depend on it. This will be shown later on in this section.

With that said, U-Boot doesn’t magically know where everything is located on the SD Card and in exactly what order it should load things up. To manage this, U-Boot provides a shell at startup where you can type in commands to tell it what to do. The U-Boot website contains the full list of commands (although Altera has added some custom commands in their version of U-Boot as will be seen below). And just like a regular Linux shell, this shell is scriptable. Later on in this section we will write a script that tells U-Boot to program the FPGA, load up the device tree and kernel, and start running the kernel.

Getting the Source

Now we will obtain the source for, configure, and compile U-Boot. First off, cd back into the “software” directory that was created by the preloader (assuming you’re in the “spl_bsp” subdirectory).

cd ..

We will be using the standard Linaro GCC toolchain for the ARMv7 instruction set to compile U-Boot (and later, the software in the root filesystem) so we need to download and extract that.

wget http://releases.linaro.org/14.09/components/toolchain/binaries/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux.tar.xz
tar -xvf gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux.tar.xz

Next we need to tell U-Boot (and other makefiles) to compile using this toolchain by setting up the CROSS_COMPILE environment variable. If you ever close your shell, remember to re-export this variable.

export CROSS_COMPILE=$PWD/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/arm-linux-gnueabihf-

Now that the toolchain is setup, we need to download Altera’s version of U-Boot from GitHub.

git clone https://github.com/altera-opensource/u-boot-socfpga.git
cd u-boot-socfpga

Altera verifies complete releases of the Golden System Reference Design (GSRD) User Manuals (U-Boot, Kernel, and GHRD included) for every release of Quartus and the SoCEDS and ARM Development Studio. With that said, instead of using the absolute bleeding edge code from the master branch (also known as the “development” or “possibly buggy” branch), we’ll use one of these regression tested releases. Run the following commands to see a list of all of the releases tagged by date or Quartus/SoCEDS version respectively.

git tag -l rel*
git tag -l ACDS*

If you are unfamiliar with the how the Git version control system works, now is a very good time to stop what you are doing and go read the tutorial on the Git website. This will walk you through everything you’ll need to know about tagging, branching, and how to download source code. Many open source projects (including U-Boot and the Linux kernel) use git to version control their code, and knowing how to navigate a repository is an absolutely vital skill.

Once you’ve read up on git and know what you’re doing, run the following command to switch to the branch containing the September 2015 release of Altera’s U-Boot. If there’s a newer release out by the time you read this, I encourage you to try it out; although I can’t guarantee the following steps in this guide will still be functional.

git checkout rel_socfpga_v2013.01.01_15.09.01_pr

If you ever want to make changes to this branch and commit them, you will need to create a new branch.

Compile U-Boot

Before compiling, clean the U-Boot directory.

make mrproper

Understanding the Default Configuration

We will be using the default Cyclone V configuration file in this compilation. Before we compile U-Boot let’s take a brief detour and look at this default configuration file. Use the following command to view the file in vim, or use your favorite text editor.

vim include/configs/socfpga_cyclone5.h

By looking at this file, the first thing that should become apparent is just how sparse it is. Besides specifying that our command prompt in U-Boot shall be “SOCFPGA_CYCLONE5 #”, it doesn’t do much. As can be seen in the header file includes, this file is including another file called “socfpga_common.h”. That file is where most of the configuration is happening.

Open “include/configs/socfpga_common.h” in your favorite text editor. Take a moment to browse this file and look at all of the settings that it configures for you. If you’re going to be designing your own board you might need to change a few of these settings. Of particular usefulness is the CONFIG_BOOTDELAY option that is set to ‘5’ on line 154. This means that U-Boot will wait five seconds for you to press a button to stop the auto-boot process before continuing. Set this to zero on a production system to have U-Boot immediately run your boot script without waiting.

Another important area is the big list of environment variables defined at line 181. These environment variables will be utilized in our boot script to provide some shortcuts to loading data from the SD Card and booting the kernel. We will discuss these environment variables in more detail in the boot script section.

Lastly, starting from about line 349 and down, there are configuration options for many of the drivers that are supported by U-Boot. Feel free to browse through these to get a feel for what U-Boot can do.

Actually Compiling U-Boot

Now that you know exactly what the default configuration is doing (a very important thing to know when your device fails to boot), configure U-Boot to use this configuration and then compile.

make socfpga_cyclone5_config
make

A shiny new U-Boot image called “u-boot.img” should be located in the u-boot-socfpga directory. We will put this on the SD Card to boot from in a later part of the guide.

Writing the Boot Script

As described earlier, the boot script is a list of U-Boot commands that will automatically be run whenever the device boots up (and you don’t hit any keys to go into the command shell). This script will program the FPGA, load up the device tree, load up the kernel, and run the kernel with a set of boot arguments.

Create a new file called “boot.script” file in the software directory and copy in the following:

echo -- Programming FPGA --
fatload mmc 0:1 $fpgadata soc_system.rbf;
fpga load 0 $fpgadata $filesize;
run bridge_enable_handoff;

echo -- Setting Env Variables --
setenv fdtimage soc_system.dtb;
setenv mmcroot /dev/mmcblk0p2;
setenv mmcload 'mmc rescan;${mmcloadcmd} mmc 0:${mmcloadpart} ${loadaddr} ${bootimage};${mmcloadcmd} mmc 0:${mmcloadpart} ${fdtaddr} ${fdtimage};';
setenv mmcboot 'setenv bootargs console=ttyS0,115200 root=${mmcroot} rw rootwait; bootz ${loadaddr} - ${fdtaddr}';

run mmcload;
run mmcboot;

Understanding the Boot Script

This script starts off by loading up the FPGA configuration bitstream from the first partition on the first SD Card (also called ‘mmc’) it finds and places it at address $fpgadata in RAM (which if you look back in the configuration header file, this value is equivalent to 0x2000000). It then programs the FPGA and enables the HPS-to-FPGA bridges using some Altera-specific commands/scripts.

Next, it sets the name of the device tree binary (which will be created and explained in the next section) as well as sets a variable that will tell the kernel where to find our root filesystem (also explained in a later section). The interesting part of this script comes into play with the two ‘shortcut’ commands we create: mmcload and mmcboot.

The first command will load up the kernel image and device tree binary from the SD Card and place it into RAM at pre-determined addresses (once again, defined in the config file). To do this, it uses the fatload command (aliased by the ${mmcloadcmd} variable) which takes in what device (zero-th SD Card) and partition (first partition, which will be our FAT partition) the file is located on, and then loads that file into ${loadaddr} and ${fdtaddr} respectively (once again, pre-defined in the config file).

For more information regarding the “mmc” and “fatload” commands used in this script, use the help system built into the U-Boot shell (once you get it booted up in a later section). For instance, to learn the exact syntax for fatload, type the following into the U-Boot shell:

help fatload

The second command is what actually starts up the Linux kernel. It begins by setting the “boot arguments” for the kernel. These are parameters that tell the kernel low level details about how it should boot up. These are settings that don’t get configured at compile time and can possibly change over time. In our case, all we’re doing is setting up the debug console to be the first serial device (which on the Atlas board is connected to a USB-to-UART converter chip) and telling the kernel where to find the root filesystem. The filesystem will be placed on /dev/mmcblk0p2, or in English, the second partition on the first SD Card. When we create our SD Card that is where we’ll be creating the root filesystem.

The “rw” argument specifies that our root filesystem should be mounted as read/write, while “rootwait” tells the kernel to wait for the SD Card to initialize and become visible to the kernel before continuing. Descriptions for all of these kernel parameters and more can be found in the Documentation/kernel-parameters.txt file located in the kernel source code that we’ll download later on (or you can just view it on GitHub with the previous link).

After the boot arguments are setup, the bootz command is used to start the kernel. This command expects the addresses to a flattened device tree binary and compressed kernel image to be passed in. Once this command gets run, control passes over to the kernel.

Compiling the Boot Script

U-Boot requires that the boot script be compiled into a binary file. Run the following to compile the boot script.

mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n "Boot Script Name" -d boot.script u-boot.scr

We need to name this file “u-boot.scr” as described in the command above for U-Boot to find it. This is once again a cause of the configuration file. In the big list of environment variables set within the default configuration file, one of them is “scriptfile=u-boot.scr” and since we’re using the defaults, we need to conform to this name.

And with that, U-Boot is compiled and ready to go! Continue on to the next section to learn about the Device Tree.

Resources

• Official Git documentation
• Altera’s U-Boot GitHub repository
• Linux kernel boot argument documentation
• The official U-Boot documentation
• The Rocketboards’ explanation on how to program the FPGA from U-Boot
• Guide on how to add a new board design to U-Boot
  If you’re creating your own custom board, I highly recommend reading this to learn how to set up U-Boot to initialize the critical pieces of hardware on your board.

Generating and Compiling the Device Tree

Device trees are a data structure that define the hardware inside of a system (this includes both SoC-level hardware and board-level hardware). The device tree is a separate file that can be changed individually from U-Boot and the kernel. This means that as along as those two pieces of software are compiled with support for the device you’re adding, all you have to do is modify your device tree to say that you have that hardware, and the kernel will handle the rest of the process (loading and running the driver).

In the time before device trees, every SoC and board manufacturer needed to write initialization code in the kernel to tell the kernel what their specific platform supported. This way the kernel would only load up drivers for the exact hardware in the system. These SoC’s or boards had a “machine ID” tied to them and U-Boot would pass the machine ID to the kernel for the machine it’s booting on. This let the kernel know which initialization routines to run.

As the proliferation of ARM-based SoC devices started to skyrocket, this became a maintenance nightmare for the Linux kernel maintainers. Many of the SoC devices had very similar initialization code or peripherals built into them, but every chip had its own memory map and interrupt vectors. As more SoC devices were added it became more difficult to share code between the devices as each manufacturers had to hardcode the addresses, registers, and other device specific information into the kernel code itself. Eventually enough kernel developers got fed up with dealing with all of this to make the decision to adopt the Flattened Device Tree (FDT) interface used by the Open Firmware project to describe the hardware in a system.

Now, instead of writing extra code for functionally equivalent hardware that just happens to have a different memory map, the peripheral configuration details can be dynamically determined at runtime. The device tree doesn’t store only the memory map, it also contains details about the interrupts used (if any), the clocks, and literally any other piece of information a driver might need to understand how to use a device. With the device tree, a single Linux image may be able to support many variations of hardware (which is obviously important when your hardware is programmable).

The device tree is written in a human-readable (this is arguable) text format called the device tree source (or dts) and compiled into a device tree binary (dtb). This device tree binary is loaded up from the SD Card by U-Boot and placed into RAM. The address at which the device tree binary was loaded is then passed to the kernel in the bootz command. When the kernel starts, it loads up the device tree, parses it, and determines which drivers to load to accommodate your platform’s hardware.

The Linux kernel uses the Device Tree data structure for three major purposes:

• Platform identification (which SoC/board it’s running on)
• Runtime configuration (another method of passing boot arguments)
• Device population (load up drivers for any peripherals or devices on the SoC/board)

Understanding the Device Tree Source

As the name would imply, device trees have a hierarchical nature to them. There’s a single root node that consists of multiple child nodes. Each child node can have other child nodes and each node can have many different properties (some standard, some driver-specific). The device tree source usually contains the following information:

• Number of CPUs
• Size and location of various RAMs
• Buses and bridges
• Peripheral device connections
• Interrupt controllers and IRQ line connections
• Specific device driver configuration, such as:
  • Ethernet MAC address
  • Peripheral’s input clock

Below is an image from Altera’s SoC Workshop series that breaks down the source for a single node:

And that’s just one specific type of node (a Synopsis I2C controller in this case). Different devices will require different sets of properties, or bindings. Each driver generally has its own specific set of properties that it looks for when reading information from the device tree. The “bindings” for all of the drivers in the Linux kernel are located in “Documentation/devicetree/bindings” in the kernel source code. When creating your device tree source, make sure to read up on the exact bindings for the drivers that your device will use.

But how does the kernel know how to match the devices in the device tree with which drivers? That comes from the “compatible” string in each device tree node. That string shows up in most nodes in the device tree, as well as any drivers that support device tree devices (say that ten times fast). Each driver hardcodes in a list of “compatibility strings” that define which devices they support. When the kernel loads in the device tree, it reads the compatibility strings of each of the nodes and tries to find drivers with the same compatibility string. If it finds a match, it loads up that driver and calls the driver’s “probe” function to inform it that a new device was added (this will be explained in more detail in the driver section of this guide).

STOP! Before going any further, I highly recommend reading through the “Device Tree for Dummies” guide that was put out by Free Electrons. There’s also a video to go along with it if you’d rather watch a presentation.

If you’re still confused about what the device tree does and how it works, skip to the resources section at the bottom of this page for more places to learn from. The resources section also contains links for understanding the device tree source’s somewhat confusing syntax.

The Device Tree Generator

With the invention of re-programmable SoC’s, the prospect of writing a device tree once for a SoC and never touching it diminishes. Since the hardware within an Altera SoC can change, the device tree needs to be re-created every time the hardware changes (this makes sure the kernel loads up the correct drivers and initialization routines). Luckily, Altera provides a tool that makes this process relatively simple.

The Device Tree Generator tool takes in the .sopcinfo file outputted by Qsys as well as one or more board XML files that describe the devices on the board (anything not within the chip/Qsys system). It will then auto-magically generate the device tree source for you. After that, all you have to do is compile the device tree source into a binary, and you should be good to go. And although you can have the device tree generate the binary for you (as shown in the diagram below), I would recommend only generating the source and compiling later. That way you have a chance to review the source that was generated and validate that it did a good job.

Board XML Files

These XML files describe the devices and peripherals on the PCB the SoC is connected to. Since these external devices don’t show up in Qsys, there’s no way that the generator can know about them and correctly include them in the design without you explicitly saying what’s there. These files take on an Altera-created syntax that appends new properties and new nodes to nodes that already exist within the source (aka, nodes that were generated from the .sopcinfo file).

The Device Tree Generator page on Rocketboards describes the syntax in more detail. It also describes required board information for various peripherals. Of particular importance to us are the “Required SDMMC Peripheral Properties” which are needed when booting from an SD Card. Read through the rest of that page to understand what properties are needed for some of the commonly used peripherals.

Most GHRD’s will come with two board XML ones: one common to any design used on that board, and one specific for a user’s design. The common XML file used in our design is “hps_common_board_info.xml” located in the project’s root directory. If you look at this file, you’ll notice that it defines properties for the SD Card, Ethernet, SPI and other peripherals. The xml file specific to our design “soc_system_board_info.xml” only contains some compatibility strings that let the kernel know we’re creating a Cyclone V SoC design. If we were going to connect a custom I2C or SPI device to the board, we would define it here.

Actually Generating the Device Tree

Generating the device tree source file is quite simple once you’ve set up the board XML files (or are given them, as in this case). Run the following command in the root “atlas_linux_ghrd” folder to create the device tree source.

sopc2dts --input soc_system.sopcinfo\ 
  --output soc_system.dts\ 
  --type dts\ 
  --board soc_system_board_info.xml\ 
  --board hps_common_board_info.xml\ 
  --bridge-removal all\ 
  --clocks

Open the resulting “soc_system.dts” file and browse through it to see how it describes the hardware. Also notice how it auto-generated a device node for the “Custom LEDs” IP that was added to the Qsys system. How this was done, is explained in the next section.

Now that we know what the device tree generator created, compile the source.

dtc -I dts -O dtb -o soc_system.dtb soc_system.dts

A device tree binary called “soc_system.dtb” should have been created. This binary will be placed on the SD Card in the next part of this guide. That name should look familiar as you have already saw how U-Boot was configured to load up a device tree binary with that exact name (clearly not a mere coincidence).

Adding Device Tree Node Generation to Custom IP

Information can be added to custom Qsys components that tell the Device Tree Generator how to create the device tree node for that component. These settings are placed in the custom component’s _hw.tcl file (look in the resources section of the hardware design overview to learn more about creating custom Qsys components).

The above tcl commands (found inside of the “custom_leds_hw.tcl” file) generated the following node within the device tree source:

The above compatibility strings will be used when we create a custom driver for this module. Our module will register with the kernel that it can support any device with the “dev,custom-leds” compatibility string. Once the kernel knows that, it will tell our driver that it found just such a device while reading through the device tree. After that, our driver will have access to any of the information placed into the device tree for that specific instantiation of the “Custom LEDs” module (in this case, the register map and clock). If there were more instantiations of that custom IP, then the driver will be called once for each instantiation.

Please visit the Device Tree Generator documentation for more information about adding device tree generation support to a custom IP block.

And that should be everything you’ll ever need to know about device trees! In the next section, we’ll finally test our system and make sure we didn’t break something in the past three steps.

Resources

• Device Tree for Dummies slides A great explanation of what device trees are and how they work.
• The video for the above slides
• The Linux kernel’s own Device Tree usage documentation If the previous slides didn’t clear up any confusion, this document goes into great detail about what the device tree is and how it fits into the Linux driver model.
• A Device Tree Usage walkthrough Explains the device tree source syntax in quite a bit of detail.
• The official Device Tree specification (it’s a part of the ePAPR spec) It’s about as exciting and easy to read as you’d imagine. Needless to say, it makes for some great bathroom reading.
• Part 1 and Part 2 of a series Linux Weekly News did on the Device Tree
• Rocketboards documentation for the Device Tree Generator Anything you should ever need to know about the device tree generator all in one place.
• How to add Device Tree Generation support to a custom IP Block If you have custom IP you’re integrating into a Qsys system, then this guide will tell you how to make your IP play nice with the Device Tree Generator.

Testing the System

In this section we’ll finally create an SD Card image and attempt to boot everything up. We first need to create a blank image file for the SD Card and create the partitions that all of the files we’ve generated will be placed on. The image below describes how our device will be partitioned.

The first partition will be a FAT partition that will hold our U-Boot image, U-Boot boot script, device tree binary, and kernel image. Partition two will hold the root filesystem (all of our files and programs) that we’ll create in a later section. In this section we’ll install the FAT and EXT4 filesystems on these two partitions respectively. The last partition is a raw partition (no filesystem) that will contain the preloader image. The Boot ROM will look for a partition with the “A2” type when scanning the SD Card. Once found, it will run the preloader image located on it.

If you aren’t familiar with the differences between disks, partitions, and filesystems, the OpenSuse wiki has a good description. IBM also has very detailed descriptions of partitions and filesystems. Go read those pages, then come back.

Create and Partition the SD Card image

We’re going to create a 512MiB image where the FAT partition will take up 256MiB, the root filesystem will use 254MiB, and the raw A2 partition will use 1MiB (the last 1MiB is used by the Master Boot Record). Every embedded Linux device will have different size requirements. If you find that you need more space for your root filesystem, feel free to come back to this section later and recreate the SD Card with larger partitions.

CAUTION: You must be absolutely certain that you use the correct device name, as “dd” to the wrong device can destroy your host system.

sudo dd if=/dev/zero of=sdcard.img bs=512M count=1

We’ll utilize a loopback device to manipulate the SD image as if it was a disk. The following command will return back the name of the loop device associated with the image file (in this case /dev/loop0). Remember this device name for later. If you don’t know what a loopback device is, check out the Wikipedia article for it.

sudo losetup –-show –f sdcard.img
/dev/loop0

We now need to create the three previously mentioned partitions. We’ll start off with the creating the A2 partition. We’ll use the fdisk utility to do this (remember to replace /dev/loop0 with whatever the previous command outputted).

sudo fdisk /dev/loop0
-- fdisk welcome message –

Command (m for help):

Verify that there are no partitions currently on the device with the “p” command.

Next, we’ll create the A2 (Altera custom) partition for the preloader. Each preloader image (as generated in the preloader section) will contain four redundant 64KB preloader images (in case of flash corruption) so this partition will need to be at least be 256KB. We’ll make it 1MiB to be safe (also coincidentally the smallest partition that fdisk will create). If you would like to store U-Boot in this partition instead of the FAT partition, the 1MB size should be large enough to do so as well.

We’ll use the “n” command to create a new partition. Create this as a primary partition, partition number 3, default first sector, and 1MiB size by entering the command listed below when prompted.

By default, fdisk creates every partition with the “Linux” type. We need to change this to the A2 type using the “t” command. When prompted for the partition type use “a2” (without quotes) as shown below. This isn’t a standard partition type, so it will come up as “unknown”.

At this point, the Master Boot Record (MBR) and A2 partition will have used up 2MiB. Now, we’ll add the 254MiB partition for the root filesystem. Create partition 2 as a 254MiB primary partition with the default starting sector of 4096.

The fdisk utility sets the partition type to “Linux” by default, so we don’t need to change the partition type for the root filesystem (we will still need to actually create the filesystem later).

Lastly, create the FAT partition that will store our boot files. This will be a primary partition, partition 1, and accept the defaults for both the first and last sectors (this will use up the rest of the SD card image). Once you know exactly how much space your boot images will take up, feel free to reduce the size of this partition and increase the size of the root filesystem partition (partition 2).

Now, change partition 1 to be a FAT partition (this doesn’t create the filesystem, it just informs devices reading the SD card that there will be a FAT filesystem on this partition).

Print out the partition table, and verify that it looks like the following.

The last thing we need to do is write out our changes to the sd card image using the “w” command.

As fdisk informs before it exits, the kernel doesn’t know about the new partitions we just added. We need to use the “partprobe” command to let the kernel know what has changed. Remember to change “/dev/loop0” with whatever your loopback device is called.

sudo partprobe /dev/loop0

Create the filesystems

Now that the partitions have been successfully created, let’s create the filesystems and start copying files over. We’ll start off by copying the preloader image into the A2 partition (partition 3). Once again, replace /dev/loop0p3 with your loopback device.

sudo dd if=software/spl_bsp/preloader-mkpimage.bin of=/dev/loop0p3 bs=64k seek=0

Next, create the FAT filesystem on partition 1 (the partition we set to be the “FAT” type). Replace /dev/loop0p1 with whatever your loop device is.

sudo mkfs –t vfat /dev/loop0p1

If you get “unable to get drive geometry, using default 255/63”, don’t panic, that’s expected. Next, create the EXT4 filesystem on the Linux partition we created. This will be used as our root filesystem later on. (Do I even need to tell you about the loop device anymore?)

sudo mkfs.ext4 /dev/loop0p2

Now we’ll mount the FAT partition and copy over all of the boot files. Start off by creating a temporary directory to mount the filesystem to, and then mount it.

mkdir temp_mount
sudo mount /dev/loop0p1 ./temp_mount

Copy over all of the boot files (U-Boot image, boot script, device tree binary, and FPGA bitstream).

sudo cp software/u-boot-socfpga/u-boot.img software/u-boot.scr soc_system.dtb soc_system.rbf temp_mount

After the files are copied, synchronize (make sure the files are written to the image) and unmount the FAT partition.

sync
sudo umount temp_mount

Burn the SD Card

Lastly, we need to take the image we just created and burn it to an SD Card. Place your SD card into either a USB adapter, or if you’re on a laptop, directly into your computer (possibly through a microSD-SD adapter). Before you run the next command, you need to know the device name (/dev/XXX) for the SD Card. Use the lsblk command to display a list of block devices on your computer. Look for a block device with the same size as your SD card. Verify that this is indeed the correct block device by unplugging your SD Card and re-running lsblk. If the device disappears then you picked the right one. Now that you’re sure you won’t overwrite your hard drive, run the following command.

sudo dd if=sdcard.img of=/dev/XXX bs=2048
sync

This might take a couple minutes to burn depending on the speed of your SD adapter.

Actually Testing the System

Once the image is done burning, it’s time to actually test the system. Connect a mini-USB cable into the mini-USB port on the right side of the board. This port is connected to a USB-to-UART chip that is connected to the serial port on the SoC. This converter chip will translate the serial data coming from the SoC into USB packets so we can connect this up to a modern computer that doesn’t have a serial port (the drivers for this chip are built-in to Linux).

To view the status information coming from the SoC, you’ll need a serial terminal installed on your system. Two popular ones are the command line tool picocom and the GUI tool putty. Use the preferred package installation system for your distribution to download one of these (or your favorite) serial terminal. After that, power up the Atlas board and ls the /dev directory as shown.

ls /dev/ttyUSB*

Whichever device appears is the one that you’ll need to use when telling your serial terminal which serial port to read from. As for the rest of the settings, you should have a baud rate of 115200, 8 data bits, 1 stop bit, and no parity.

Note: You will need to either be a part of the “dialout” group or running your serial terminal as root to get access to serial devices. Look up online how to add your user to a group if you don’t know how.

Once your terminal is up and running, power cycle (unplug and replug the power cable) the Atlas board and watch as status messages fill up your screen. The output coming from the preloader should look the same as the following.

Notice how the preloader printed out status messages telling you it initialized the clocks and SDRAM before loading up U-Boot. The output coming from U-Boot should look like the following.

If your output contains a lot of error messages about a “zImage”, then everything was set up correctly! The “zImage” is the compressed kernel image that we’ll generate in the next section (which is why U-Boot can’t find it). You will also get warnings about a bad CRC which you can safely ignore as stated on the U-Boot website. That message appears because we never saved our environment variables to flash (we only put them in the boot script). U-Boot will also fail to assign an Ethernet address because we didn’t fully setup the MAC address in the hardware system (we won’t be using networking in this example).

Ensure that your output only has errors about the missing “zImage” and nothing else. The other files (U-Boot script, FPGA bitstream, and device tree binary) should have loaded up correctly. If U-Boot can’t find one of the files, make sure that you copied that file over onto the correct partition as outlined previously. And lastly, if you’re getting an error message “altera_load: Failed with error code -4” make sure that you correctly set up the MSEL switches as outlined at the beginning of this guide. If the MSEL switches aren’t setup correctly, then the FPGA won’t be able to get programmed with the .rbf file that was generated.

At this point, feel free to pat yourself on the back because you’re halfway to a functioning embedded Linux system! Continue onwards to the next section to learn how to configure and compile the Linux kernel.

Resources

• OpenSuse wiki’s description of partitions and filesystems
  If you’re not aware of the differences between disk partitions and filesystems, this documentation gives a good overview.
• IBM’s description of partitions
• IBM’s description of filesystems

Configuring and Compiling the Linux Kernel

And now for the moment we’ve all been waiting for: the Linux kernel! And as huge of a piece this will be in our embedded Linux system, it’s surprisingly easy to configure and compile. The Linux kernel has hundreds of options available to the user that let you change exactly what features get compiled into your kernel image. For most embedded Linux distributions, many of these options aren’t needed and can be disabled to save space on the SD Card (less enabled options = less features = smaller kernel).

Most of the options can either be enabled (I want the feature) or disabled (I don’t want the feature). When it comes to drivers, there’s a third option available: set as loadable module. When a driver is enabled, it will get compiled into the kernel image. When a driver is set as a loadable module, it won’t get compiled into the kernel image, but instead be loaded at runtime from the root filesystem. Only loading up the drivers needed to start the system, and then loading in the rest of the needed drivers after the system is fully booted can greatly speed up boot times. In our system, we’re trying to keep things simple, so we’ll compile everything into the kernel.

Another thing to keep in mind is that every processor architecture the kernel supports will have its own special set of options that need to be set. The options that you see when configuring a kernel for ARM will be different than if you were configuring for x86. If you want to learn more about configuring a Linux kernel for a desktop PC, I would start off by looking at your distribution’s website. Most distributions will provide default kernel configurations and guides on how to compile the kernel to be compatible with the distribution.

Getting the Source Code

Before we can compile the kernel, we need to get the source code. And just like with U-Boot, Altera has their very own special blend of the kernel that contains drivers and configuration for Altera SoC devices. Altera makes effort to upstream all of their major changes. With that said, it’s still best to get the latest kernel from Altera directly to make sure you have all of the latest bug fixes and features for all of the Altera-created code.

Clone Altera’s Linux repository:

cd software
git clone https://github.com/altera-opensource/linux-socfpga.git

This will take a few minutes to download depending on your internet speed. Once it’s done downloading, check out what releases are available by issuing the following command.

cd linux-socfpga
git tag –l rel*

In this guide we’ll use the September 2015 release of the 4.1 kernel. If there is a newer version of the kernel available (I guarantee there will be) then feel free to try out the newer kernel. However, if you do try out a newer kernel, I cannot guarantee that the following steps will be the same or work. Newer versions of the kernel will also add, remove, or modify configuration options. If you can’t find an option that I say you need to set, then you’ll have to do some research and find out what happened to that option and whether it’s still needed.

git checkout rel_socfpga-4.1_15.09.01_pr

Configuring the Linux Kernel

Considering there are over a couple thousand options, we’re lucky to have a default configuration that will set up the most important options for us. This default configuration file is located in “/arch/arm/configs/socfpga_defconfig”. I encourage you to open that file and browse through the options it sets. Google is always your friend when trying to find out what different options do (many of these options will be explained later as well).

If you’ve closed your shell since you compiled U-Boot, make sure to open up the embedded command shell and to export the CROSS_COMPILE variable as described in the U-Boot section. Run the following command to set the default configuration. This will create a “.config” file with the same settings that were in the default configuration file.

make ARCH=arm socfpga_defconfig

The recommended way of setting options is to use one of the kernel configuration utilities that come with the kernel. There are GUI utilities available, but we’ll use an ncurses based command line tool (it requires less dependencies than the GUI tools). The GUI tools are either based off of GTK or Qt, and will require dependencies from one of those frameworks depending on the GUI tool you want to use. The dependencies for the Qt-based tool on a Debian-based system are the libqt4-dev and pkg-config packages. To follow along with this guide, install the ncurses library. On a Debian-based system:

sudo apt-get install libncurses5-dev

And then open up the kernel configuration tool:

make ARCH=arm menuconfig

You will be presented with a window at looks like the following. Read over the help information at the top of the window before continuing.

Go into the “General Setup” menu. Uncheck “Automatically append version information to the version string”. This will prevent the kernel from adding extra “version” information to the kernel. Whenever we try to dynamically load a driver (also called kernel modules, as discussed in a later section) the kernel will check to see if the driver was built with the same version of the source code as itself. If it isn’t, it will reject to load that driver. For development, it’s useful to disable these options to make it easier to test out different versions of drivers. In a production system however, it’s recommend to keep this option enabled and only use drivers that were compiled with the correct version of the kernel.

That’s one of the two options that need to be changed from the defaults, but in the coming paragraphs, we’ll explore some of the more important options that are enabled in our system for knowledge’s sake.

I encourage you to peruse the options in the General Setup menu and see what’s available to you (hitting “?” to view the help info for the highlighted option). Of particular importance to us is the “Embedded System” option (turns on advanced features) and the type of SLAB allocator used (determines how memory will be dynamically allocated in the kernel). If you want to use an initial ram disk or ram filesystem that would be enabled here as well (these will be explained in the next section).

Hit the escape key twice to go back to the main menu. Notice how the “Enable loadable module support” is checked. This means we’ll be able to dynamically load drivers after the kernel has booted up. This will be useful when developing our custom driver (this way, we won’t have to re-compile every time we want to test the driver). “Enable block layer” is also enabled. This allows us to use “block devices” (hard drives, SSDs, SD Cards, USB Flash drives, and many other storage devices) within our system. If we accidentally disabled this, the kernel wouldn’t be able to read from our SD Card. (As an aside, flash devices are technically different than block devices, but many flash devices--like SD Cards and flash drives--have a microcontroller on them that mimic block devices).

Enter the “Enable the block layer” menu option and enable the “Support for large (2TB+) block devices and files” option. Although the chances of you actually having 2TB+ files on your filesystem are small, if you look at the help for this option (press “?”) you’ll notice that this option is required to be enabled if you’re using the EXT4 filesystem (which we are). If you forget to enable this option, the kernel will mount your filesystem in read-only mode and print out a helpful message reminding you to come back and enable this if you want full read/write support.

Under the “System Type” menu, scroll down until you see the “Altera SOCFPGA family” option. This is enabled to let the kernel know what type of SoC it will be running on.

Under “Kernel Features” the “Symmetric Multi-Processing” option should be enabled. This tells the kernel to use both of the ARM Cortex-A9 cores available on the SoC. The “Memory split” and “High Memory Support” options are useful in determining how much of the address space is dedicated to the kernel and how much physical memory can be accessed from inside of the system respectively. These are some complex topics and if you plan on using 1GiB of memory or more, I suggest reading through this Linux Weekly News article to give you a basic idea of how virtual memory works within Linux.

Under the “Device Drivers” heading browse through and see what drivers will be getting compiled into the kernel. A couple of options to note (since we’re using an Altera SoC) are the “Device Drivers→Misc Devices→FPGA Bridges→FPGA Bridge Drivers” and “Device Drivers→FPGA devices→FPGA Framework” option. These drivers are required to configure and interact with the FPGA from Linux. MMC/SD/SDIO card support is also enabled in this menu. Without that enabled, Linux wouldn’t be able to access the SD Card.

Hit the escape key twice to go back to the main menu. Under “File systems” make sure that at least the “The Extended 4 (ext4) filesystem” option is enabled. Considering we’re using this filesystem for our root filesystem, the kernel won’t be able to mount the root filesystem without support from this driver.

Under “Miscellaneous filesystems”, note how the “Jounalling Flash File System v2” option is enabled. This is a very popular filesystem used on flash devices (like QSPI or NAND flash). If you ever want to put Linux on a flash device (not just a block device), then make sure this option is enabled (assuming you want to use this filesystem).

Go back to the main menu, then head to “Kernel Hacking→Compile-time checks and compiler options”. The “Compile the kernel with debug info” option should be enabled. This will compile debug symbols into the kernel making it possible to analyze crash dumps and use kernel debugging tools. This also makes the kernel larger and should be disabled on production systems. Other debug options are enabled in the “Kernel Hacking” menu that could possibly be disabled to reduce the kernel size.

Remember to browse through the rest of the kernel options and try to figure out what they do. The built-in help system is very useful (just press “?” when hovering over an option) as well as Google.

When you’re done looking at the available options, hit the right arrow key to select the “Save” option at the bottom of the window and press enter. When asked for a filename, leave it at the default (“.config”) and hit enter. Hit enter again, then exit the configuration tool.

Compiling the Kernel

Once we’re done setting up the configuration for the kernel, it’s as simple as running the make command to compile.

make ARCH=arm LOCALVERSION= zImage

(NOTE: I HAD TO ADD "CROSS_COMPILE=arm-linux-gnueabihf-" before LOCALVERSION for it to compile properly. MikeK)

The “LOCALVERSION=” part of the command once again makes sure that no extra versioning information will be added to the kernel. If we had made modifications to the kernel’s source code (and were using git), the kernel will automatically append a “+” to the end of the kernel version. By setting the “LOCALVERSION” option to nothing (that is why there’s nothing after the equal sign) no plus symbol will be added even if we modify things.

The “zImage” option specifies that we want a compressed version of the Linux kernel image that is self-extracting. Another commonly used option is a “uImage” which is the uncompressed Linux kernel image file along with a U-Boot wrapper that includes the OS type and loader information. Older versions of U-Boot only supported the “bootm” command which had to take in a uImage. Modern versions of U-Boot (like the one we compiled) have the “bootz” command that can boot a zImage.

This compilation process could take up to an hour depending on how fast your machine is (my Ivy Bridge Core i5 laptop takes about 15 minutes for comparison). After the kernel is done compiling, the resulting image files will be located in “arch/arm/boot”.

If you loaded the kernel image onto the SD Card and started the board up, you’d notice that the kernel crashes saying it can’t find “init” (you don’t need to do this if you don’t want to). Head on over to the next section to learn about “init” and how to generate a root filesystem.

Resources

• Article on configuring the Linux kernel
  Just note that most of the articles online that talk about configuring the Linux kernel are usually using an x86 system where the boot process and options are a little different.
• A good overview of kernel configuration
  Gives a description of each of the menu options found in the configuration tool (for x86, so a few of the menu items will differ).
• Google
  If the help explanation for an option isn’t clear enough, don’t hesitate to google the option to find out more information about it. Forgetting to enable a single option can be enough to crash a system.

Generating a Root Filesystem

A root filesystem is the filesystem that contains all of the files that are necessary for booting the system up to a point where other filesystems can be mounted (if need be). The exact contents of a root filesystem will vary depending on your application, but there are a few pieces that are common for most:

• An init system. This program will be the first application to run after the kernel is done booting up. It has the responsible of setting up user-space and kicking off all of the other user applications (like a login prompt and shell).
• A “/dev” directory to hold device nodes. In the dark days (before the 2.6 kernel) every device that had a device file associated with it, needed to have that device file manually created using the mknod command. Nowadays, this process is automated for us through one of a variety of systems (we’ll choose one). (A device file is a file that represents a device connected to the kernel. It’s just one way that user-space can interact and pass data to/from kernel-space. Check out this page for more information).
• Any loadable kernel modules. If you set any drivers to be a “loadable module” while configuring the kernel, then they’ll need to be placed on this root filesystem. Typically, these are placed in “/lib/modules//”.

The Linux Information Project contains a nice summary of what a root filesystem is that you should definitely read through before continuing.

Initial Ram Disk (initrd) and Initial Ram Filesystems (initramfs)

Before explaining how to create a root filesystem, I’d like to take a moment to talk about another approach to booting up a system. This approach involves loading a filesystem into ram (instead of reading it from a block device) and running the init program off of that. This is essentially an “early userspace” root filesystem that can be used to get the minimum needed functionality loaded in order to continue the boot process.

This is especially useful on desktop Linux systems where the hardware one user might have can be wildly different than another user. The issue of knowing which drivers to have compiled into the kernel and which ones to keep as loadable modules becomes difficult. If you don’t know what device (hard drive, SSD, flash drive, etc.) and filesystem (ext2/3/4, FAT, NTFS, etc.) that somebody is booting off of, how do you know which driver to compile in? One solution is to have a basic ram disk/filesystem (I’ll explain the difference in a moment) that runs before the regular root filesystem is mounted. This ram disk/filesystem will determine which drivers need to be loaded to continue the boot process. This keeps both boot times and kernel image sizes low. It also allows for more complex boot processes (network booting, booting an encrypted root filesystem, etc.).

An initial ram disk is a chunk of ram that simulates a block device. This lets the kernel mount a filesystem on it. Considering the kernel is mounting a filesystem on this ram disk device, the driver for that filesystem needs to be compiled into the kernel (ext2 is a common choice). An initial ram filesystem on the other hand, is a cpio (copy in, copy out) archive that is compiled into the kernel. Because it’s compiled into the kernel, there’s no need for a filesystem driver to be installed at the time this filesystem is mounted. This is the newer and preferred ram disk method to use if you need to initialize hardware early in a system, have an encrypted root filesystem, or want to dynamically determine which drivers to load before mounting the actual root filesystem. Network booting and other complex setups might also require the use of an initial ram filesystem. The image below from Free Electron’s Embedded Linux Development course describes the boot process with an initramfs:

On most embedded systems, it makes more sense to just compile into the kernel whatever we need because the hardware we’re using is known at compile time (which is why we never enabled initramfs support in the kernel). With that said, using an initramfs as the actual root filesystem can make for a small single application system with no persistent data. This could be useful for devices with small amounts of non-volatile storage that are running applications that don’t need permanent storage.

This has only been a very brief overview of initrd/ramfs and the differences between them. I highly recommend checking out the resources section for links to websites where you can learn more about initial ram disks/filesystems.

Introduction to Buildroot and BusyBox

Trying to build up a root filesystem (both a regular or ram filesystem) can be a painstakingly laborious process. You need to manually set up the folder structure (usually following the Filesystem Hierarchy Standard), compile and load up all of the needed applications, and either use a system that will populate “/dev” for you or manually create all of the device files. Luckily, there are tools and software packages available that can automate this process.

The first important piece of software is used in most hand-made embedded Linux distributions: BusyBox. BusyBox is a highly-configurable collection of software commonly used on Linux systems. This includes things like the init program, the shell, and common UNIX utilities like ls, cd, mkdir, and more. This software package lets us build up a root filesystem with the basic utilities and programs that we’re used to using on a desktop computer. By far the best part of BusyBox is its configurability. Every program can be enabled, disabled, and configured to only include the functionality that you need. This makes BusyBox very space efficient.

In reality though, most systems are going to need more than BusyBox can provide. Extra libraries and more complex applications (BusyBox only has the smaller shell utilities) are usually needed to build up a complete system. There are usually three ways to build up a root filesystem:

• Use a pre-built binary distribution like Debian or Fedora (not very flexible and usually only supports certain popular pieces of hardware).
• Build all of the components manually (very flexible but painful and inefficient).
• Use an automated build system that builds up the root filesystem for you (Buildroot, OpenEmbedded, and Yocto are three of the most popular build systems).

As can be guessed, we’ll be using the third approach to build our root filesystem (we’ll use Buildroot in this guide). Most automated build systems will compile not only a root filesystem but every other part of the build process (creating a cross-compilation toolchain, compiling the bootloader, compiling the kernel, etc.) but in our case, we’ll use Buildroot to only create the root filesystem. After you’ve completed this guide, I highly recommend you fully learn how to use an automated build system. Once learned, these systems can greatly speed up the development process. Free Electrons has great courses on both Buildroot and Yocto, and Rocketboards has information on using Yocto when working with an Altera SoC.

Configuring Buildroot

If you’ve closed your shell since we compiled the kernel, make sure to open up the embedded command shell and to export the CROSS_COMPILE variable as described in the U-Boot section. We’re going to start off by downloading Buildroot’s source (from their git repository) into the software directory.

cd software
git clone http://git.buildroot.net/git/buildroot.git

This will download Buildroot into an aptly named “buildroot” folder. Once Buildroot has finished downloading, we can begin configuring it using the following command. Since we’re using a toolchain we’ve already downloaded, we’ll pass in the directory to that toolchain. You can do this from within the configuration utility as well, but while we’re still in the shell, we have access to the “pwd” command which makes life easier.

---

NOTE

You will be on the master branch of the current buildroot repository, which is most likely not compatible with the Linaro 2014.09 toolchain used in the instructions below. We had to do the following before continuing:

cd buildroot
git checkout 2015.08.x
cd ..

-- SteveScott - 24 Jan 2017 - 16:02

---

make -C buildroot ARCH=ARM BR2_TOOLCHAIN_EXTERNAL_PATH=$(pwd)/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux nconfig

This will open up a window that looks like the following:

Press F1 and read through the help to learn how to navigate this utility (use the arrow keys to scroll). Once done, hit enter to get back to the configuration and go to “Target options→Target Architecture” and select “ARM (little endian)”.

• Under “Target Architecture Variant” select “cortex-A9”.
• Under Target ABI select “EABIhf” (this enables support for hard floating point processing).
• Enable “NEON SIMD extension support” (NEON is a single instruction/multiple data engine found in many ARM cores).
• Lastly, set the “Floating point strategy” to use the “NEON” engine we just enabled. Leave “Target Binary Format” and “ARM Instruction set” at their defaults.

Hit escape to go back to the main menu, and dive into the “Toolchain” menu. Since we already have a toolchain downloaded, we’ll need to tell Buildroot not to create one for us. Usually we would have to build up a toolchain profile to tell Buildroot exactly what our toolchain supports, but luckily Buildroot has support for Linaro 2014.09 built-in (check out the Buildroot manual for more info on setting up a custom toolchain).

• Under “Toolchain type” select “External toolchain”.
• Make sure “Toolchain” is set to “Linaro ARM 2014.09”. If you’re using a newer version of Buildroot that doesn’t automatically support the Linaro toolchain, then you’ll have to select “custom toolchain” in this menu and manually fill out extra options.
• Under “Toolchain origin” select “Pre-installed toolchain”.
• Ignore the toolchain path (we’re passing this option from the command line which means we don’t need to set it here).
• Enable “copy gdb server to the Target”. This will allow us to debug applications running on our embedded Linux system.
• Leave everything else at the defaults.

Hit Escape and go into the “System configuration” menu. This menu contains options on how our embedded Linux system will operate (init system, root password, login prompts).

• Feel free to change the hostname (name of the computer) and System banner (message that gets displayed when you login) to whatever you like.
• Notice that the “Init system” is set as “BusyBox”. This will use the small init program that comes with BusyBox. You could also select the systemV init system (the one used on many desktop computers before systemd became popular) or systemd (although, it needs a specific standard C library and version of the kernel). BusyBox’s init system should be suitable for most embedded Linux needs. The Buildroot manual explains the differences very well.
• “/dev management” should be set to “Dynamic using devtmpfs only”. This means the kernel will automatically add device files to the “/dev” folder when it discovers them. You can also use more powerful systems that run in a combination of user-space and kernel-space. “eudev” is a system based off udev (the one that is most likely running on your desktop computer) but without all of the systemd integrations. “mdev” is an alternative to (e)udev that was created when people got pissed off that udev became a part of systemd. The Buildroot manual explains the differences very well. Considering we’re not doing anything complex, using devtmpfs only should be sufficient.
• Remember to set the root password to whatever you like.
• Leave the shell to be BusyBox’s default shell (which is ‘ash’, a lightweight version of the Bourne shell). You can set this to other shells that Buildroot supports if you set the “BUSYBOX_SHOW_OTHERS” option.
• Notice that “Run a getty (login prompt) after boot” is enabled. For development purposes, this can be very useful because we can log in and get to a shell. For a production application that does need a concept of “users”, you can disable this option and just write programs that will auto-run at startup (this will be demonstrated later in this section).
• Look over the rest of the default options and press F2 to see help for any that look confusing.

Go back to the main menu and dive into the “Kernel” menu. Leave the “Linux Kernel” option unchecked. Buildroot can automate the process of compiling the kernel (you still have to pass it a .config file that you’ve configured) but in our case, we’ll use the kernel image we’ve already created. Once you’ve finished this guide, I suggest coming back and trying to automate the kernel compilation using Buildroot (or another build system depending on your needs).

Hit escape to go back to the main menu then go into the “Target packages” menu. This is where most of the root filesystem generation will occur. All of the sub-menus in this menu let you choose different applications and libraries to bundle up into your root filesystem. To demonstrate how this works, we’ll install valgrind (a popular profiling and debugging tool) on our root filesystem. The only other thing that will be installed is everything that comes with BusyBox (which we’ll configure in a moment).

• Under “Debugging, profiling and benchmark”, scroll to the bottom and enable “valgrind”.

In reality, the only package we really need to make sure our system is “usable” is BusyBox (and all of the software that comes with it), but feel free to peruse these settings and figure out if any of these packages are something you might want in the future. Remember you can always press F2 to learn more about a package. Once you’re done setting up any packages you may want, go back to the main menu.

From within the “Filesystem images” menu, we can set up how we want the root filesystem to be generated. We have a lot of options available, but we’ll go with the simple one (also the default) and just output a tar archive of the files. Later on we’ll just unarchive this tar file onto the root filesystem partition on the SD Card that we created a couple sections back. Before leaving this menu, notice that there’s an option to create a cpio archive. If you are trying to generate an initramfs, this would be the archive that you want to generate (this cpio would then get loaded into ram and passed to the kernel by the bootloader).

Buildroot also has the ability to compile U-Boot (or another bootloader) for you as well (located in the Bootloaders menu). It works the same way as the kernel in that you pass in the configuration file that you want to use and it will perform the compilation. Buildroot can also just download the latest version of U-Boot from the official git repository or from a custom git repository (like Altera’s U-Boot repository).

Lastly, Buildroot can generate some tools for your desktop Linux system that can be useful. Check the “Host utilities” menu for more information.

Once you’re done, hit F6 to save the configuration and then hit enter to accept the default name. Exit the configuration utility (either Escape or F6).

Configure BusyBox

Now that Buildroot is setup correctly, we’ll configure BusyBox to contain the software that will be useful for a minimal development environment. Keep in mind that you don’t have to configure everything exactly how I’m doing it (aka, leaving it at the defaults). Every application will have different requirements and learning what packages your system needs is just one step in the development process.

Start the BusyBox configuration utility:

make –C buildroot busybox-menuconfig

That command will open up the following window.

For BusyBox to be functional, none of these settings need to be changed. The default settings will provide nearly all of the functionality you’ll need. If you want to try and slim BusyBox down, then I recommend going through and removing applications that you know you won’t need. I won’t walk through every section in this configuration utility because most of it is fairly self-explanatory. If you’re unsure what an option does, just remember this utility works exactly like the kernel’s configuration: hitting “?” will bring up help information.

When you’re done configuring packages, use the arrow keys to move the menu at the bottom to the “Exit” item, then hit enter. Hit enter again to save your changes.

Actually Generating the Root Filesystem

Now that everything is configured, it’s time to actually compile Buildroot (which will generate our root filesystem tar archive).

make -C buildroot BR2_TOOLCHAIN_EXTERNAL_PATH=$(pwd)/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux all

This process can take nearly as long as compiling the kernel, so be patient. Once the compilation is done, the root filesystem will be located in “buildroot/output/images/rootfs.tar”. Continue onto the next section to test our system and make sure we can get to a command prompt!

Resources

• A good description of /dev and device files
• Linux Information Project’s definition of a root filesystem
• A very detailed article from the Linux foundation on Initrd/Initramfs
• Initramfs kernel documentation
• Another good description of initrd vs. initramfs
• Free Electron’s Buildroot Course
• Free Electron’s Yocto/OpenEmbedded Course
• The Buildroot User Manual
  If you have trouble with any of the steps in this guide, check out the official manual for more information.

Testing the System (Part 2)

Now that we’ve compiled both the kernel and root filesystem, we can test our system to see if it will boot all the way up to a login prompt (spoiler alert: it should). This section will be relatively short. All we need to do is copy and paste the kernel image onto the SD Card’s FAT partition and then unarchive the rootfs.tar file onto the root filesystem we created in the previous “Testing the System” section.

Insert the microSD card into your computer (either through a USB adapter or microSD-SD adapter if your computer has a media card reader) and determine which device it is (/dev/XXX). Use lsblk to get a list of block devices connected to your computer and look for one with the same size as your SD card as well with three partitions of sizes 256M, 254M, and 1M. If you somehow manage to have another block device connected to your computer with three partitions of those exact sizes, I’d be very impressed. Remember this device name.

Where your computer mounts each of these partitions is going to vary depending on your distribution and /etc/fstab file. By default on my somewhat-customized Debian system, the FAT partition gets placed into /media/usb0 while the ext4 partition gets mounted to /media/<my username>/<long string of numbers and letters, probably the uuid>. The lsblk command will tell you where things are mounted.

From within the software directory, copy over the kernel image.

sudo cp linux-socfpga/arch/arm/boot/zImage <mount point that lsblk tells you, remember the FAT partition is the 256M one>

Next decompress the root filesystem’s tar archive onto the correct partition and synchronize (make sure everything actually gets written to disk).

sudo tar –xvf buildroot/output/images/rootfs.tar –C <mount point of your ext4 partition, it’s the 254M one>
sync

Next, unplug your SD card and put it back into your Atlas board and make sure your terminal is set up as explained in the previous “Testing Your System” section. When you’re ready to see the magic known as Linux, power on your board. If all went well, you should be presented with something that looks like the following (your “banner” message will probably be different, but if it isn’t, you have good taste).

Login to your new system with the “root” login and whatever password you setup in the BusyBox configuration. Once logged in, explore around the filesystem using the commands you’re used to (cd, ls, etc.). Make sure to try and run valgrind to verify that it was indeed installed (valgrind --version).

Next, run the following command:

ls -lha /bin

Notice how literally every single application is a symbolic link to BusyBox (except BusyBox itself). BusyBox is compiled as a single executable and then uses the symlinks to make each command reference back to that one executable. If you’re curious, you can also ls into /sbin and /usr/bin to find out where init and valgrind are located respectively. If you would like to write your own custom script that runs at startup, you would place it within the /etc/init.d folder. This folder holds each of the startup scripts with each script having a unique name that denotes the order to run it in (S<number><name>). You’ll also notice “rcS” and “rcK” scripts in this folder. These get run on startup and shutdown/restart respectively. All they do is run the other scripts in this directory in the correct order for startup, and reverse order for shutdown (it also passes a variable to the script to let it know whether it’s being started up or shut down).

The lower the number, the earlier it will run. If you created a script called “S60script” it would run after the rest of the other scripts. If you want an interesting exercise, use the built-in vi editor (it’s a default package for BusyBox) to create a basic shell script called “S60script” then restart your board and see if it runs (have it print a string to the console for instance). This script should run after the other scripts, but before the login prompt is displayed.

The shell script should use the same syntax as regular bash scripting, but you can read the man page on ‘dash’ for more information (ash is based off dash). Also, remember to give your script execute permissions before restarting your board.

chmod +x <scriptname>

Congrats, your new system is fully up and running! Move on to the last two sections to learn how to write both user-space and kernel-space software that interacts with our custom LED hardware.

Writing a User-space Application

Now that our embedded Linux system is completely up and running, it’s time to start to developing applications for it that utilize our custom hardware (the simple LED module). The first application we’ll write will access the hardware’s registers from user-space without the use of a driver. To do this, we’ll utilize a device file called “/dev/mem”. This file is a device that represents the entirety of the 4GB address space. In other words, if we open this file up and try to write data at certain addresses, that data will actually get sent to those addresses within the chip. Since the custom LED module has its registers located at a specific address, all we need to do is open up “/dev/mem” and write values to an address that gets mapped to our custom hardware.

Unfortunately, if I tried to explain the entirety of Linux programming in this guide, this would become a complex programming book (instead of a relatively short guide to embedded Linux). Instead, I’ll link you to resources that can be useful in learning more about Linux programming. For instance, if you would like to know more about operating systems in general (which is always good knowledge to have), then I would highly recommend this free online textbook: Operating Systems: Three Easy Pieces.

Address Spaces

To fully understand the program we’ll be writing in this section, you need to understand a little bit about process address spaces. If you never took an operating systems course in college, this topic might be new to you. This also ties into the topic of virtual memory which is useful to understand as well. Please read through the links below before continuing if you don’t already understand address spaces.

• Columbia University presentation on Processes and Address Spaces
• Anatomy of a Program in Memory (highly recommended introductory article)
• A detailed (and complex) look at how Linux implements processes/address spaces

Memory Mapping

Using the mmap system call to map files into a process’s address space is another topic you need to understand to be able to write the following program. If you’ve never used the mmap system call, look at the links below to learn more about it.

• Man page documentation for the mmap system call
• Wikipedia article on memory mapped files
• Book chapter on memory mapping (meant for kernel developers, but provides a good overview)

Creating a System Header File

Even though we’ll be writing our application for user space, you’ll still need to know at what address to find the registers associated with our custom module (Altera SoC’s follow a memory-mapped I/O architecture). To make this easier, Altera has created a tool that will take in our Qsys files, and generate a header file with all of the address offsets for the peripherals in the FPGA. This tool is aptly named “sopc-create-header-files”. Review the help information for this command before continuing (if you closed the command shell, remember to re-open it and export the CROSS_COMPILE variable).

sopc-create-header-files --help

In a nutshell, if we give it an .sopcinfo file and the name of a module in that Qsys file that has an Avalon memory mapped master in it, it will create a header file with the addresses of the peripherals connected to that master interface. From within the “software” directory, create a new folder to contain our program and then create the system header file.

mkdir userspace ; cd userspace
sopc-create-header-files ../../soc_system.sopcinfo --single hps_0.h --module hps_0

This will create a new “hps_0.h” header file in the current directory. We’ll use this header file to get the address offset at which our custom LED module’s registers are located.

HPS Address Map

The header file that we generated above will only tell us the offsets at which the peripherals are located. Offset from what, you might ask? The answer is the base address of the bridge that peripheral is connected to. There are two bridges that a peripheral can be connected to if it wants to pass data from HPS into FPGA: HPS-to-FPGA and HPS-to-FPGA-Lightweight bridges. The lightweight bridge is a smaller (as in how much of the address space it takes up) bridge that is meant to connect the control and status interfaces of peripherals to the HPS. The “full” bridge is meant for passing large chunks of data between the HPS and FPGA.

Both of these bridges have their own base address as shown in the table below.

Both of the above images were taken from the first chapter of the Cyclone V Hard Processor System Technical Reference Manual. That chapter contains more information about the HPS’s address space that I highly recommend you peruse through before continuing. So, we now have the base address of the lightweight bridge (0xFF200000) and we know the offsets of our peripherals (from the generated header file). That’s everything we need to actual modify the register in our custom LED module. Time to start coding!

The Source Code

Below you will find the source code for an application that will take in a number from the command line, and using our custom module, blink the LEDs that number of times. To do this, it will map the address space for the lightweight bridge into the address space for the current process. It will then access the registers within the lightweight bridge using the offsets from our generated header file. The way the custom LED module works, is that the first 8-bits that get written to the module's address will determine which LEDs get turned on (‘1’) and which LEDs get turned off (‘0’). A hex value of 0xFF means every LED is on (the Atlas board has eight LEDs) and a hex value of 0x00 will turn off every LED. Analyze the source code and make sure you understand it before continuing on.

Type this into a file called “devmem_demo.c” in the “userspace” folder (this file can also be found in the atlas_linux_ghrd.tar.gz archive)

#include <stdio.h>
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
#include <error.h>
#include <stdint.h>
#include <sys/mman.h>

#include "hps_0.h"

// The start address and length of the Lightweight bridge
#define HPS_TO_FPGA_LW_BASE 0xFF200000
#define HPS_TO_FPGA_LW_SPAN 0x0020000

int main(int argc, char ** argv)
{
    void * lw_bridge_map = 0;
    uint32_t * custom_led_map = 0;
    int devmem_fd = 0;
    int result = 0;
    int blink_times = 0;

    // Check to make sure they entered a valid input value
    if(argc != 2)
    {
        printf("Please enter only one argument that specifies the number of times to blink the LEDs\n");
        exit(EXIT_FAILURE);
    }

    // Get the number of times to blink the LEDS from the passed in arguments
    blink_times = atoi(argv[1]);

    // Open up the /dev/mem device (aka, RAM)
    devmem_fd = open("/dev/mem", O_RDWR | O_SYNC);
    if(devmem_fd < 0) {
        perror("devmem open");
        exit(EXIT_FAILURE);
    }

    // mmap() the entire address space of the Lightweight bridge so we can access our custom module 
    lw_bridge_map = (uint32_t*)mmap(NULL, HPS_TO_FPGA_LW_SPAN, PROT_READ|PROT_WRITE, MAP_SHARED, devmem_fd, HPS_TO_FPGA_LW_BASE); 
    if(lw_bridge_map == MAP_FAILED) {
        perror("devmem mmap");
        close(devmem_fd);
        exit(EXIT_FAILURE);
    }

    // Set the custom_led_map to the correct offset within the RAM (CUSTOM_LEDS_0_BASE is from "hps_0.h")
    custom_led_map = (uint32_t*)(lw_bridge_map + CUSTOM_LEDS_0_BASE);

    // Blink the LED ten times
    printf("Blinking LEDs %d times...\n", blink_times);
    for(int i = 0; i < blink_times; ++i) {
        // Turn all LEDs on
        *custom_led_map = 0xFF;

        // Wait half a second
        usleep(500000);

        // Turn all the LEDS off
        *custom_led_map = 0x00;

        // Wait half a second
        usleep(500000); 
    }

    printf("Done!\n");

    // Unmap everything and close the /dev/mem file descriptor
    result = munmap(lw_bridge_map, HPS_TO_FPGA_LW_SPAN); 
    if(result < 0) {
        perror("devmem munmap");
        close(devmem_fd);
        exit(EXIT_FAILURE);
    }

    close(devmem_fd);
    exit(EXIT_SUCCESS);
}

The makefile used to compile the code above is shown below. If you’re new to the makefile syntax, please check out this tutorial. This makefile also makes the assumption that you set the CROSS_COMPILE variable, so make sure to do that if you haven’t already. There are also plenty of other resources online as well. When you type this into a file, name the file “Makefile” (no file extension).

TARGET = devmem_demo

CFLAGS = -static -g -Wall -std=gnu99
LDFLAGS = -g -Wall
CC = $(CROSS_COMPILE)gcc
ARCH = arm

build: $(TARGET)
$(TARGET): $(TARGET).o
    $(CC) $(LDFLAGS) $^ -o $@

%.o: %.c
    $(CC) $(CFLAGS) -c $< -o $@

.PHONY: clean
clean:
    rm -f $(TARGET) *.a *.o *.~

Once you’ve saved those two files, all you need to do is invoke the make command to compile the program (from inside of the “userspace” directory). After the program has been compiled, plug the microSD card into your computer, and copy and paste the “devmem_demo” executable into the “/root” directory in the root filesystem like so.

sudo cp devmem_demo <path-to-rootfilesystem-partition>/root
sync

After that, put the microSD card back into the board and boot the device up like usual. After logging in, run the program with the following command.

./devmem_demo 10

In this case, the LEDs should blink ten times then stop. If you put it in a different number, the LEDs should blink that number of times.

Congratulations, you’ve written your first piece of embedded Linux software! Move onto the next section to learn how to build a simple driver that interacts with the custom LED hardware.

Resources

• Operating Systems: Three Easy Pieces
  An awesome free operating systems textbook.
• Columbia University presentation on Processes and Address Spaces
• Anatomy of a Program in Memory (highly recommended introductory article)
• A detailed (and complex) look at how Linux implements processes/address spaces
• Man page documentation for the mmap system call
• Wikipedia article on memory mapped files
• Book chapter on memory mapping (meant for kernel developers, but provides a good overview)
• Introduction to Makefiles
• Cyclone V Hard Processor System Technical Reference Manual

Writing a Simple Driver

The next step in developing software for our embedded Linux distribution is to create a driver. Unfortunately, a detailed explanation of Linux driver development could take an entire book (in fact, it has) so this section will be more of an introduction to driver development. The resources section at the end of this chapter has links to more information if you would like to learn more about Linux driver development.

Kernel Modules

Loadable Kernel Modules (LKM) are chunks of code that can be dynamically inserted at runtime into the Linux kernel. In this section we’ll be creating a driver that is also a kernel module (meaning, we didn’t compile the driver into the kernel and will instead install it at runtime). Every kernel module needs at least two functions associated with it: initialization and exit functions. These are the functions that are called when the module is installed and removed respectively (using the insmod and rmmod commands shown later). Derek Molloy wrote a nice introductory tutorial to kernel module development that I recommend reading over before continuing.

Device Driver Model

At the end of the day, all drivers get connected with three things: busses, devices, and kernel frameworks. Busses can be described as entities that other devices or busses connect to. PCI Express, USB, and SPI are examples of busses. They usually provide a mechanism for “enumerating” the devices on the bus (discovering what devices are connected) but if they don’t, there’s usually a way to list what devices are connected with the bus (in our case that would be through the device tree).

Devices are the things that are connected to the busses. A flash drive or webcam could be examples of devices that would connect to the USB bus. The bus driver enumerates all of the devices that are currently connected (or reads the device tree to find out which devices are connected) and it will try and match each device with an available driver (in the device tree case, using the “compatibility” property). If an available driver is found for the device, the driver’s “probe” function is called once for every device that it can handle. This will give the driver time to initialize the device and setup any device-specific memory (for instance, each device could have a different memory map and that information will need to be saved somewhere).

Kernel frameworks are different areas of the kernel that handle logical sets of devices. For instance, there are frameworks for networking, mass storage devices (e.g. hard/flash drives), input (e.g. mice, keyboard, etc.) and more. When a driver registers itself with a certain bus, it will usually register itself with a framework as well. By registering with a framework, a standard interface is exposed to user-space for accessing that device. This way, even if you have the coolest mouse in the world with its own custom driver, as long as it registers with the input subsystem, it will look like any other mouse to user-space programs.

A good example that ties all of this together is a USB Wifi adapter. The driver for this device would register itself with the USB bus and the networking framework. When the USB bus notices that one of these devices was connected, it will call the “probe” function of the driver to initialize the device. After that, the driver will handle receiving data packets from the networking framework and translating them into the USB protocol, and vice versa.

Platform Bus

All of this talk about devices being connected to busses poses a problem for most SoC chips: how do we handle devices that aren’t connected to a bus (e.g. internal hardware modules)? The answer is the platform bus. The platform bus is a “fake” bus that handles all of the devices that aren’t connected to actual busses. Here’s how the kernel documentation describes platform devices (Documentation/driver-model/platform.txt):

Platform devices are devices that typically appear as autonomous entities in the system. This includes legacy port-based devices and host bridges to peripheral buses, and most controllers integrated into system-on-chip platforms. What they usually have in common is direct addressing from a CPU bus. Rarely, a platform_device will be connected through a segment of some other kind of bus; but its registers will still be directly addressable.

Most of the devices that you create in the FPGA portion of a programmable SoC will fall under the veil of the platform bus. As such, when you develop a driver for one of your custom peripherals, you will have to make sure to register your driver with the platform bus (passing on a list of compatibility strings your driver supports). When you register your driver with the platform bus, the bus will read through the device tree and try to match any of the device’s compatibility strings with the list that you provide. If it finds any compatible devices, it will call the “probe” function in your driver. Starting at Slide 145, Free Electron’s Linux Kernel Development slides describe platform drivers in very good detail (go read that then come back).

Misc Framework

The Miscellaneous framework provides a simple layer above character files for devices that don’t have any other framework to connect to. Registering with the Misc subsystem simplifies the creation of a character file. In the “Custom LEDs IP” case, all we want to do is expose a character file (/dev/custom_leds) that when written to, will change which LEDs are currently turned on. Slide 282 of Free Electron’s Linux Kernel Development slides describes how to create a driver that utilizes the miscellaneous subsystem. It would be smart to read through those before continuing.

When creating a character file, a file operations structure is needed to define which functions to call when a user opens, closes, reads, and writes to the file. This structure will be passed to the Miscellaneous subsystem (shortened to Misc subsystem) when you register a device with it. One thing to note is that when you use the misc subsystem, it will automatically handle the “open” function for you. Inside the automatically created “open” function, it will tie the “miscdevice” structure that you create to the private data field for the file that’s being opened. This is useful, so in your write/read functions you can get access to the miscdevice structure, which will let you get access to the registers and other custom values for this specific device. You’ll see this come into play in the source code.

Instead of connecting to the misc subsystem, we could have connected with the GPIO subsystem which has mechanisms for handling LEDs, or we could have created a character file by hand. I chose to use the misc subsystem because it would simplify both interacting with the device in user-space as well as programming. If you want to learn about GPIO drivers, read the documentation at “Documentation/gpio.txt”. If you want to learn how to create character files by hand, read Chapter 3 of Linux Device Drivers, 3rd Edition.

Accessing Memory-Mapped Devices

Once a driver has registered itself with both a bus and framework (platform bus and misc subsystem respectively in our driver’s case) it will most likely need to access the registers (addresses) that the device exposes so it can initialize and interact with the device. The first step to accessing a memory-mapped device is to request the memory region that the memory-mapped device is in. This information can usually be found within the device tree binding for the device (the “reg” property). The platform bus has functions available to get access to device tree information. Requesting the memory region will make sure that other drivers don’t try to use that memory region.

After the memory region is claimed, you still won’t be able to access it. Kernel drivers deal with virtual addresses, so you need to map the physical address range that represents the device’s memory map into the kernel’s address space. Once the memory has been mapped into the kernel’s address space, it will finally be available for access. Instead of dereferencing pointers to access memory (which is probably your first instinct), you should access memory using the iowrite/ioread suite of functions. This is because some architectures handle accessing memory in strange ways, but if you use these functions, all of that is abstracted away (making the driver more portable). Using the iowrite32 function to write a 32-bit value into memory is demonstrated in the source code later in this section.

The Source Code

And for the moment you’ve all been waiting for: the actual driver code! Try to read through every line and understand what’s going on. If you see a function or type that you haven't seen before (which will probably be most of them), I highly recommend looking it up on the Linux Cross Reference (graciously hosted by Free Electrons). The best way to learn about writing kernel code is to read kernel code. With that in mind, use the cross reference website to dig into the implementations of the functions that are called in the code below and read whatever documentation you can find in the source. Even if you don't fully understand how each function works, getting a general idea of where different parts of the kernel are located in source is always a valuable skill.

Put the following code into a file called “custom_leds.c”:

#include <linux/module.h>
#include <linux/platform_device.h>
#include <linux/io.h>
#include <linux/miscdevice.h>
#include <linux/fs.h>
#include <linux/types.h>
#include <linux/uaccess.h>

// Prototypes
static int leds_probe(struct platform_device *pdev);
static int leds_remove(struct platform_device *pdev);
static ssize_t leds_read(struct file *file, char *buffer, size_t len, loff_t *offset);
static ssize_t leds_write(struct file *file, const char *buffer, size_t len, loff_t *offset);

// An instance of this structure will be created for every custom_led IP in the system
struct custom_leds_dev {
    struct miscdevice miscdev;
    void __iomem *regs;
    u8 leds_value;
};

// Specify which device tree devices this driver supports
static struct of_device_id custom_leds_dt_ids[] = {
    {
        .compatible = "dev,custom-leds"
    },
    { /* end of table */ }
};

// Inform the kernel about the devices this driver supports
MODULE_DEVICE_TABLE(of, custom_leds_dt_ids);

// Data structure that links the probe and remove functions with our driver
static struct platform_driver leds_platform = {
    .probe = leds_probe,
    .remove = leds_remove,
    .driver = {
        .name = "Custom LEDs Driver",
        .owner = THIS_MODULE,
        .of_match_table = custom_leds_dt_ids
    }
};

// The file operations that can be performed on the custom_leds character file
static const struct file_operations custom_leds_fops = {
    .owner = THIS_MODULE,
    .read = leds_read,
    .write = leds_write
};

// Called when the driver is installed
static int leds_init(void)
{
    int ret_val = 0;
    pr_info("Initializing the Custom LEDs module\n");

    // Register our driver with the "Platform Driver" bus
    ret_val = platform_driver_register(&leds_platform);
    if(ret_val != 0) {
        pr_err("platform_driver_register returned %d\n", ret_val);
        return ret_val;
    }

    pr_info("Custom LEDs module successfully initialized!\n");

    return 0;
}

// Called whenever the kernel finds a new device that our driver can handle
// (In our case, this should only get called for the one instantiation of the Custom LEDs module)
static int leds_probe(struct platform_device *pdev)
{
    int ret_val = -EBUSY;
    struct custom_leds_dev *dev;
    struct resource *r = 0;

    pr_info("leds_probe enter\n");

    // Get the memory resources for this LED device
    r = platform_get_resource(pdev, IORESOURCE_MEM, 0);
    if(r == NULL) {
        pr_err("IORESOURCE_MEM (register space) does not exist\n");
        goto bad_exit_return;
    }

    // Create structure to hold device-specific information (like the registers)
    dev = devm_kzalloc(&pdev→dev, sizeof(struct custom_leds_dev), GFP_KERNEL);

    // Both request and ioremap a memory region
    // This makes sure nobody else can grab this memory region
    // as well as moving it into our address space so we can actually use it
    dev→regs = devm_ioremap_resource(&pdev→dev, r);
    if(IS_ERR(dev→regs))
        goto bad_ioremap;

    // Turn the LEDs on (access the 0th register in the custom LEDs module)
    dev→leds_value = 0xFF;
    iowrite32(dev→leds_value, dev→regs);

    // Initialize the misc device (this is used to create a character file in userspace)
    dev→miscdev.minor = MISC_DYNAMIC_MINOR;    // Dynamically choose a minor number
    dev→miscdev.name = "custom_leds";
    dev→miscdev.fops = &custom_leds_fops;

    ret_val = misc_register(&dev→miscdev);
    if(ret_val != 0) {
        pr_info("Couldn't register misc device :(");
        goto bad_exit_return;
    }

    // Give a pointer to the instance-specific data to the generic platform_device structure
    // so we can access this data later on (for instance, in the read and write functions)
    platform_set_drvdata(pdev, (void*)dev);

    pr_info("leds_probe exit\n");

    return 0;

bad_ioremap:
   ret_val = PTR_ERR(dev→regs); 
bad_exit_return:
    pr_info("leds_probe bad exit :(\n");
    return ret_val;
}

// This function gets called whenever a read operation occurs on one of the character files
static ssize_t leds_read(struct file *file, char *buffer, size_t len, loff_t *offset)
{
    int success = 0;

    /* 
    * Get the custom_leds_dev structure out of the miscdevice structure.
    *
    * Remember, the Misc subsystem has a default "open" function that will set
    * "file"s private data to the appropriate miscdevice structure. We then use the
    * container_of macro to get the structure that miscdevice is stored inside of (which
    * is our custom_leds_dev structure that has the current led value).
    * 
    * For more info on how container_of works, check out:
    * http://linuxwell.com/2012/11/10/magical-container_of-macro/
    */
    struct custom_leds_dev *dev = container_of(file→private_data, struct custom_leds_dev, miscdev);

    // Give the user the current led value
    success = copy_to_user(buffer, &dev→leds_value, sizeof(dev→leds_value));

    // If we failed to copy the value to userspace, display an error message
    if(success != 0) {
        pr_info("Failed to return current led value to userspace\n");
        return -EFAULT; // Bad address error value. It's likely that "buffer" doesn't point to a good address
    }

    return 0; // "0" indicates End of File, aka, it tells the user process to stop reading
}

// This function gets called whenever a write operation occurs on one of the character files
static ssize_t leds_write(struct file *file, const char *buffer, size_t len, loff_t *offset)
{
    int success = 0;

    /* 
    * Get the custom_leds_dev structure out of the miscdevice structure.
    *
    * Remember, the Misc subsystem has a default "open" function that will set
    * "file"s private data to the appropriate miscdevice structure. We then use the
    * container_of macro to get the structure that miscdevice is stored inside of (which
    * is our custom_leds_dev structure that has the current led value).
    * 
    * For more info on how container_of works, check out:
    * http://linuxwell.com/2012/11/10/magical-container_of-macro/
    */
    struct custom_leds_dev *dev = container_of(file→private_data, struct custom_leds_dev, miscdev);

    // Get the new led value (this is just the first byte of the given data)
    success = copy_from_user(&dev→leds_value, buffer, sizeof(dev→leds_value));

    // If we failed to copy the value from userspace, display an error message
    if(success != 0) {
        pr_info("Failed to read led value from userspace\n");
        return -EFAULT; // Bad address error value. It's likely that "buffer" doesn't point to a good address
    } else {
        // We read the data correctly, so update the LEDs
        iowrite32(dev→leds_value, dev→regs);
    }

    // Tell the user process that we wrote every byte they sent 
    // (even if we only wrote the first value, this will ensure they don't try to re-write their data)
    return len;
}

// Gets called whenever a device this driver handles is removed.
// This will also get called for each device being handled when 
// our driver gets removed from the system (using the rmmod command).
static int leds_remove(struct platform_device *pdev)
{
    // Grab the instance-specific information out of the platform device
    struct custom_leds_dev *dev = (struct custom_leds_dev*)platform_get_drvdata(pdev);

    pr_info("leds_remove enter\n");

    // Turn the LEDs off
    iowrite32(0x00, dev->regs);

    // Unregister the character file (remove it from /dev)
    misc_deregister(&dev→miscdev);

    pr_info("leds_remove exit\n");

    return 0;
}

// Called when the driver is removed
static void leds_exit(void)
{
    pr_info("Custom LEDs module exit\n");

    // Unregister our driver from the "Platform Driver" bus
    // This will cause "leds_remove" to be called for each connected device
    platform_driver_unregister(&leds_platform);

    pr_info("Custom LEDs module successfully unregistered\n");
}

// Tell the kernel which functions are the initialization and exit functions
module_init(leds_init);
module_exit(leds_exit);

// Define information about this kernel module
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Devon Andrade <devon.andrade@oit.edu>");
MODULE_DESCRIPTION("Exposes a character device to user space that lets users turn LEDs on and off");
MODULE_VERSION("1.0");

The Kernel Build System

To actually compile the driver, you need to link your kernel module against the build system included with the Linux kernel. Before continuing, read the documentation at Documentation/kbuild/makefiles.txt to learn about the Kbuild (kernel build) system in general, and the documentation at Documentation/kbuild/modules.txt for information on specifically building dynamically loadable modules.

The Makefile used to build the above driver is shown below (replace “../linux-socfpga” with the path to the kernel if you placed yours in a different directory than what I specified):

KDIR ?= ../linux-socfpga

default:
    $(MAKE) -C $(KDIR) ARCH=arm M=$(CURDIR)

clean:
    $(MAKE) -C $(KDIR) ARCH=arm M=$(CURDIR) clean

help:
    $(MAKE) -C $(KDIR) ARCH=arm M=$(CURDIR) help

The Kbuild file:

obj-m := custom_leds.o

Assuming you set the CROSS_COMPILE variable as described in earlier sections, all you need to do is run make in the same directory as the above files to compile the driver. The result will be a “custom_leds.ko” file (ko stands for kernel object).

Testing the Driver

Insert the SD card into your computer and copy over the “custom_leds.ko” into the /root directory (this process was described in the previous chapter). After the file is copied, put the SD Card back into the board and boot it up. Once booted up, run the following to install the driver:

insmod custom_leds.ko

All of the LEDS should turn on (you can find the code that does this within the “leds_probe” function). A character file was also created at /dev/custom_leds. You can write values to that file to change which LEDs are turned on. Now run the following command.

echo “9” > /dev/custom_leds

The ASCII character “9” is represented in binary as “00111001” (or hex 39). After you run the above command, the LEDs should change to reflect this binary value. Feel free to write other values to this file to see how the LEDs change. When you’re done, remove the driver using the rmmod command (all of the LEDs should turn off, as described in the “leds_remove” function).

rmmod custom_leds

Congratulations, you’ve written and tested your first driver! And beyond that, you’ve finished this guide (which is a feat in and of itself). If you find yourself fuzzy on any of the discussed topics (which is to be expected, the subject of Embedded Linux is large and confusing) I recommend reading through the resources section at the end of each chapter as well as perusing the actual kernel documentation and source code.

Good luck with your next Embedded Linux project!

Resources

• Linux Device Drivers, 3rd Edition
  This is the seminal book on Linux driver development (it’s also free)! If you’re serious about wanting to learn Linux driver development, I highly recommend this book.
• Free Electron’s presentation on Linux Kernel Development
  This is probably one of the most up to date pieces of information available for learning driver development. If you can make it through all of the slides and labs, you will come out with a far better grasp of how various drivers are built.
• Derek Molloy’s Kernel Module Tutorial