Build a Real-Time Linux Kernel for Robotics in 90 Minutes

Configure and compile PREEMPT_RT kernel patches for deterministic robot control with sub-millisecond latency on Ubuntu 24.04.
·
6 min read
·
Mark
·
Robotics Linux

Problem: Robot Control Loops Miss Deadlines

Your robot's control loop runs at 1kHz but occasionally misses cycles, causing jitter in motor commands and unstable motion. Standard Linux kernels can't guarantee timing for hard real-time tasks.

You'll learn:

  • How to patch and build a PREEMPT_RT kernel
  • Verify real-time performance with cyclictest
  • Configure system for deterministic latency under load

Time: 90 min | Level: Advanced

Why This Happens

Standard Linux is a general-purpose OS optimized for throughput, not latency. The kernel can delay your control loop for 10-100ms while handling interrupts or swapping memory.

Common symptoms:

  • Control loop jitter exceeding 1ms
  • Motors stuttering under CPU load
  • ROS2 executor missing deadlines
  • Inconsistent sensor sampling rates

Real-time requirements:

  • Soft real-time: Late responses degrade performance (video streaming)
  • Hard real-time: Missed deadlines cause system failure (motor control)

Robotics needs hard real-time guarantees: 1kHz control = 1ms deadline every cycle.

Solution

Step 1: Verify Hardware Support

# Check CPU supports constant TSC (required for stable timing)
grep -E "constant_tsc|nonstop_tsc" /proc/cpuinfo

# Verify no SMI interrupts (BIOS power management)
sudo modprobe msr
sudo rdmsr -a 0x1a0

Expected: Both commands return results. If rdmsr fails, disable CPU power management in BIOS.

Supported hardware:

  • CPUs: Intel Core 4th gen+, AMD Ryzen 2nd gen+, ARM Cortex-A53+
  • Avoid: Atom/Celeron (high SMI latency), virtualized environments

Step 2: Download Kernel and RT Patches

# Create workspace
mkdir -p ~/rt-kernel && cd ~/rt-kernel

# Download kernel 6.6 LTS (matches current PREEMPT_RT support)
wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.6.15.tar.xz
tar -xf linux-6.6.15.tar.xz
cd linux-6.6.15

# Download matching RT patch
wget https://cdn.kernel.org/pub/linux/kernel/projects/rt/6.6/patch-6.6.15-rt22.patch.xz
xzcat ../patch-6.6.15-rt22.patch.xz | patch -p1

Why 6.6 LTS: Stable RT patches, supported until Dec 2026. Avoid 6.7+ until RT patches mature.

If it fails:

Step 3: Configure Kernel

# Start from current system config
cp /boot/config-$(uname -r) .config
make olddefconfig

# Open configuration menu
make menuconfig

Critical settings to enable:

Navigate to General setup:

  • Enable: Preemption Model → Fully Preemptible Kernel (Real-Time)
  • Enable: Timer subsystem → High Resolution Timer Support

Navigate to Processor type and features:

  • Disable: Transparent Hugepage Support (causes latency spikes)
  • Set: Timer frequency → 1000 HZ

Navigate to Power management:

  • Disable: CPU Frequency scaling
  • Disable: ACPI Processor Aggregator

Why these matter:

  • Fully Preemptible: Kernel can be interrupted for RT tasks
  • 1000 HZ: Matches typical robot control loop frequency
  • Disable power mgmt: Prevents CPU from changing frequency mid-task

Save and exit.

Step 4: Compile Kernel

# Install build dependencies
sudo apt update
sudo apt install -y build-essential libncurses-dev bison flex \
  libssl-dev libelf-dev bc dwarves

# Compile (use all CPU cores)
make -j$(nproc) deb-pkg LOCALVERSION=-rtrobot

Expected: Compilation takes 20-40 minutes. Creates .deb files in parent directory.

If it fails:

  • "No rule to make target 'debian/certs/'": Run scripts/config --disable SYSTEM_TRUSTED_KEYS --disable SYSTEM_REVOCATION_KEYS
  • Out of memory: Reduce threads with -j4 instead of -j$(nproc)

Step 5: Install RT Kernel

cd ..
sudo dpkg -i linux-image-6.6.15-rtrobot_*.deb \
              linux-headers-6.6.15-rtrobot_*.deb

# Update bootloader
sudo update-grub

# Reboot into RT kernel
sudo reboot

After reboot:

# Verify RT kernel loaded
uname -a
# Should show: "6.6.15-rtrobot SMP PREEMPT_RT"

# Check preemption model
cat /sys/kernel/realtime
# Should return: 1

Step 6: Configure System for RT

# Add user to realtime group
sudo groupadd realtime
sudo usermod -aG realtime $USER

# Set RT scheduling limits
sudo tee /etc/security/limits.d/99-realtime.conf << EOF
@realtime soft rtprio 99
@realtime soft priority 99
@realtime soft memlock unlimited
@realtime hard rtprio 99
@realtime hard priority 99
@realtime hard memlock unlimited
EOF

# Disable CPU frequency scaling
sudo systemctl disable ondemand
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Isolate CPU cores for RT tasks (reserve cores 2-3)
sudo tee -a /etc/default/grub << EOF
GRUB_CMDLINE_LINUX="isolcpus=2,3 nohz_full=2,3 rcu_nocbs=2,3"
EOF
sudo update-grub

# Reboot to apply
sudo reboot

Why isolate cores: Prevents kernel from scheduling non-RT tasks on dedicated RT cores, reducing interference.

Verification

Test 1: Measure Latency Under Load

# Install RT testing tools
sudo apt install rt-tests stress-ng

# Run cyclictest on isolated core (10 min test)
sudo cyclictest -p 95 -a 2 -t 1 -n -m -D 10m &

# Stress system in parallel
stress-ng --cpu 4 --io 4 --vm 2 --vm-bytes 1G --timeout 600s

You should see:

T: 0 ( 1234) P:95 I:1000 C: 600000 Min:   2 Act:   3 Avg:   4 Max:  18

Good RT performance:

  • Max latency < 50μs: Excellent (suitable for <1kHz control)
  • Max latency < 100μs: Good (suitable for 1kHz control)
  • Max latency > 200μs: Poor (investigate SMI, disable more services)

Test 2: Compare with Standard Kernel

Boot into your old kernel (select in GRUB menu) and run same test. You'll see:

  • Standard kernel: Max latency 5,000-50,000μs
  • RT kernel: Max latency <100μs

If latency is still high:

  • Check SMI: sudo cat /sys/firmware/acpi/interrupts/gpe* | grep enabled - disable ACPI GPE in BIOS
  • Check IRQ threading: ps aux | grep irq/ - all IRQs should show [irq/XX-...]
  • Disable hyperthreading: In BIOS, or add nosmt to kernel cmdline

What You Learned

  • PREEMPT_RT patches make Linux preemptible for hard real-time tasks
  • Kernel configuration and system tuning are equally critical
  • CPU isolation prevents interference from non-RT workloads
  • Cyclictest validates deterministic latency under stress

Limitations:

  • RT kernel has ~5-10% lower throughput than standard kernel
  • Some drivers (NVIDIA proprietary) incompatible with RT
  • Requires dedicated hardware (no VM/cloud support)

When NOT to use:

  • Soft real-time needs: ROS2 with 10Hz loops (standard kernel sufficient)
  • Cloud robotics: Use specialized RT OS (VxWorks, QNX)
  • Simulation: GAZEBO/Isaac Sim don't need RT

Production Checklist

Before deploying to robots:

  • Test with actual robot hardware (motors, sensors)
  • Verify RT performance holds during typical operations
  • Document kernel version and config for fleet reproducibility
  • Set up monitoring for latency spikes in production
  • Train team on RT debugging (ftrace, latency histograms)

Fleet management:

  • Pin kernel version (prevent automatic updates)
  • Use same kernel across robot fleet
  • Keep RT patches updated for security fixes
  • Maintain fallback to standard kernel in GRUB

Next Steps

Integrate with ROS2:

// Use RT-safe memory allocation
#include <tlsf_cpp/tlsf.hpp>
auto allocator = std::make_shared<tlsf_heap_allocator>();

// Set thread priority
sched_param param;
param.sched_priority = 90;
pthread_setschedparam(pthread_self(), SCHED_FIFO, &param);

Resources:

Troubleshooting

Problem: Latency spikes every 60 seconds

  • Cause: Cron jobs or systemd timers
  • Fix: Disable non-essential timers with systemctl list-timers and systemctl disable

Problem: USB devices cause latency

  • Cause: USB interrupts not threaded
  • Fix: Verify CONFIG_USB=y and CONFIG_USB_EHCI_HCD=m (module, not built-in)

Problem: Network traffic causes jitter

  • Cause: Network IRQs on same core as RT tasks
  • Fix: Pin network IRQ to non-isolated core with echo 1 > /proc/irq/XX/smp_affinity

Tested on Ubuntu 24.04, Kernel 6.6.15-rt22, Intel Core i7-12700, Raspberry Pi 5 Hardware: Universal Robots UR5e, Franka Emika Panda, Custom motor controllers