Much of FlightAware is implemented in a scripting language called Tcl, which has served us well since our inception nearly two decades ago. But looking around at today’s software ecosystem, it’s difficult to claim that Tcl will continue to be the best choice for our needs in the future.

For example, we routinely need to maintain our own Tcl bindings and implementations for commonly used software such as Kafka, Prometheus, and PostgreSQL. In most other language ecosystems that are prevalent today, there are widely used packages that provide this functionality which don’t require us to build them in-house.

A few years ago, FlightAware made the decision to move away from Tcl and embrace several other language ecosystems that are a stronger fit for our work by defining our “First-Class Languages”: Go, Rust, and Python (along with application-specific languages such as Typescript and Swift as appropriate).

When I joined FlightAware on the Flight Tracking team in March 2023, several projects using these new language selections were already underway. However, one system that was still firmly rooted in Tcl was Hyperfeed.

Hyperfeed is FlightAware’s core flight tracking engine. It’s responsible for fusing incoming flight information from disparate sources and producing a single, consistent data stream of live aircraft activity worldwide. The resulting feed is consumed by the many downstream services that show flights on our website and mobile apps, use ML models to predict departure and arrival times, detect aircraft flying in holding patterns, provide data to customers through our REST API and our streaming API, Firehose, send alert emails and push notifications, and much more. If Hyperfeed isn’t running, time is frozen; no aircraft move on our maps, no alerts get sent, and the data shown on our website and in our APIs gets stale.

As you might imagine, we approach changes to this critical system with care. Hyperfeed encodes subtleties learned from years of experience tracking flights, and there are plenty of examples throughout software engineering's history that demonstrate the risks of attempting to rewrite such a large and complex system from scratch.

Instead of rewriting Hyperfeed from scratch in a new language, we have chosen a path of incremental improvements — leveraging our first-class languages — that will morph the system over time into one that no longer depends on Tcl.

In particular, we’ve chosen to use Rust because it’s one of the languages several members of the team are already familiar with and it has good Foreign Function Interface (FFI) support, which is crucial for integration with Tcl.

The goal is that by gradually factoring out sensible abstractions into Rust, we can both strengthen the structure of Hyperfeed and immediately reap the performance benefits of a compiled language. The type safety that Rust brings to the table will also be a welcome addition to the codebase.

The heavy lifting is being done by the tcl crate (crates are the packaging mechanism for the Rust ecosystem). To help manage the interactions between all of the native code involved, we are using Nix (which is used heavily across FlightAware).

There are a few layers to this, and you can follow along with the complete source code here. First, we have the pure Rust library implementation in src/greeter.rs:

use std::fmt::Debug;

#[derive(Debug)]
pub enum Language {
    English,
    French,
    Spanish,
}

/// Greet returns a greeting message in the preferred language for the provided recipient.
pub fn greet(who: &str, lang: Language) -> String {
    match lang {
        Language::English => format!("Hello, {who}!"),
        Language::French => format!("Bonjour, {who}!"),
        Language::Spanish => format!("Hola, {who}!"),
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_english_greeting() {
        assert_eq!(
            greet("Alice", Language::English),
            "Hello, Alice!".to_string()
        );
        assert_eq!(
            greet("Alice", Language::French),
            "Bonjour, Alice!".to_string()
        );
        assert_eq!(
            greet("Alice", Language::Spanish),
            "Hola, Alice!".to_string()
        );
    }
}

Notice that this is where we’re putting our tests for the library logic, and there are no mentions of Tcl whatsoever. With this in place, we can build a Tcl wrapper using the tcl crate in src/lib.rs:

use tcl::reexport_clib::{Tcl_Interp, TCL_OK};
use tcl::*;
use tcl_derive::proc;
use version::version;

mod greeter;

#[derive(thiserror::Error, Debug)]
enum Error {
    #[error("unsupported language")]
    UnsupportedLanguage,
}

/// Initialize the Tcl module.
///
/// # Safety
///
/// This function uses unsafe calls to Tcl's C library.
#[no_mangle]
pub unsafe extern "C" fn Greeter_Init(interp: *mut Tcl_Interp) -> u32 {
    let interp = Interp::from_raw(interp).expect("No interpreter");
    interp.def_proc("::greeter::greet", greet as ObjCmdProc);
    interp.package_provide("greeter", version!());
    TCL_OK
}

#[proc]
fn greet(who: String, lang: String) -> Result<String, Box<dyn std::error::Error>> {
    let lang = match lang.as_str() {
        "en" => greeter::Language::English,
        "fr" => greeter::Language::French,
        "es" => greeter::Language::Spanish,
        _ => return Err(Box::new(Error::UnsupportedLanguage)),
    };
    Ok(greeter::greet(&who, lang))
}

#[cfg(test)]
mod tests {
    use super::*;
    use tcl::Interpreter;

    fn setup_interpreter() -> Interpreter {
        let interp = Interpreter::new().expect("Could not create interpreter");
        unsafe {
            Greeter_Init(interp.as_ptr());
        }
        interp
    }

    #[test]
    fn test_greeting_english() {
        let interp = setup_interpreter();
        let code = "
            ::greeter::greet Alice en
        ";
        assert_eq!("Hello, Alice!", interp.eval(code).unwrap().get_string());
    }

    #[test]
    fn test_greeting_unknown() {
        let interp = setup_interpreter();
        let code = "
            ::greeter::greet Alice xx
        ";
        assert!(interp.eval(code).is_err());
    }
}

The primary objective of this layer is to handle the translation between Tcl and Rust. For example, the greet function annotated with the #[proc] macro converts language names passed as strings from the Tcl interpreter into Language enum values. Another interesting thing to note here is our ability to test the interface between Tcl and Rust by evaluating some Tcl code.

Next, we can write a small Tcl program to test out our new library:

package require greeter

foreach lang {en fr es} {
    puts [::greeter::greet "world" $lang]
}

And finally, we’ll use Nix to enter a shell with our compiled Rust library and the exact version of Tcl that it was built against, and run our sample program:

We’ve already used this approach to implement a Rust version of one of our internal libraries. Since Hyperfeed’s deployment process has historically been focused around Tcl, our release process didn’t really have a “build” step where Rust code could be compiled. With some updates to our release tooling, we are now able to build a single release artifact that contains Tcl scripts along with compiled Rust libraries and push it to each of the Hyperfeed servers. As a result, we can now begin to introduce Rust implementations into Hyperfeed in production.