A Flight by Any Other Identifier… Not So Sweet

What’s in A Flight? While a rose by any other name may smell just as sweet, the name - or unique identifier - given to an incoming data point at FlightAware has significant implications.

As we’ve noted before, flights do not have a single identifier, and the identifiers are not unique. Most flights have a tail number, but some flights do not, and of course a given aircraft might fly multiple times in a day. Many flights have a callsign or flight number, but callsigns come with no promise of uniqueness; Most major airlines tie callsigns to routes, so you can have multiple flights with the same callsign in a single day. Certain airspaces promise to assign every flight a Globally Unique Flight Identifier (GUFI), but that only covers a small portion of all flights we track.

Since there are no unique external identifiers upon which we can fully rely, we have to create our own. Every flight we track gets a unique ID, which follows it from the first moment we hear about it until the end of time. That lets us keep our records straight, but we still have to figure out how to map the real world down to our easily-distinguishable records. We must be able to answer the question: what constitutes a flight? For commercial airlines, it’s reasonable to say “whatever aircraft a group of ticketed passengers ultimately ride to their destination, whenever that happens, constitutes a flight”. That is generally what our users who setup flight alerts want to see. But when flights get delayed, diverted, or cancelled, we don’t get detailed information on what each ticket-holder is doing, so that method is out.

And besides, we track all flights, not just airline flights. Imagine a chartered jet was going to fly to New York City departing at 1pm, and instead at 1:10pm they fly to Scranton, PA. Did they cancel the original flight and fly a new one, or is it “the same flight” with the destination changed? Reasonable flight trackers can disagree.

Alas, Poor Flight. I Barely Knew It.

For every new message we process, we must decide: is this message describing a change to an existing flight already in our database, or is it an entirely new flight that just happens to be similar to another? Or, does the message describes an update to an existing flight, but the information is just plain wrong and should be ignored? FlightAware stitches together data from dozens of sources to create our comprehensive picture of past, current, and future flights, and 0% of those sources are 100% accurate. Whether due to technical glitches in remote sensors or human error typing information on a keyboard, every data point we receive must be treated with skepticism.

Every day, FlightAware tracks hundreds of thousands of flights. Most of these flights are well behaved - multiple sources corroborate that some aircraft will fly under some callsign from point A to point B, and any disagreements on exact departure and arrival times are small. But some flights are not well behaved, and must be treated like the fabled Ship of Theseus: things can change and change, but at some point we must decide that they’ve changed too much, and we’re really talking about a completely different flight than we originally thought. When parsing a new message describing a flight, and comparing to existing flights already in our database, how does one determine if the incoming message is describing a change to an existing record or a brand new flight? It is the jewel of the Flight Tracking crew, our software system known as HyperFeed®, which decides where to draw the line. Before we go over how HyperFeed makes decisions about the incoming data, let's look at some real world examples of where things get difficult. No matter how many edge cases you think you’ve thought of, I promise you: there are more.

Here are some strange cases FlightAware has had to handle recently:

• An airline plans for plane #1 to operate flight ABC123. They roll plane #1 out to the gate to begin boarding. Then, for a reason unknown but which we might guess was a mechanical issue, plane #1 returns to the hangar, and is replaced with plane #2. But alas, someone forgets to turn off the ADS-B transponder on plane #1, and it continues broadcasting. Our incoming data then shows two planes - plane #1 and plane #2 - which each claim to be flight ABC123. Which one is real, and which one is the imposter?
• An airline overseas plans to operate two flights: flight 123 heading to the western coast of the United States, and flight 456 heading to the east coast. We’re told in advance that flight 123 will be flown by tail ABC, and flight 456 will be flown by tail XYZ. Then, before the flight, the airline quietly decides to swap which plane is flying each route. We see tail ABC announce that it is flight 456, and tail XYZ announce that it is flight 123. Is this a mistake? Did one (or two) pilots mis-key their transponders? Or is it a genuine change?
• An aircraft with tail ABC and callsign 123 takes off and starts flying its route. Part way through the flight, we stop seeing that particular callsign in our data. Around the same time we start see flight 456 appear, which was supposed to take off around this time. But curiously this flight 456 has tail ABC, and happens to be a decent distance from its origin. Tail numbers are supposed to be unique, but nothing in this world is guaranteed. There’s probably only one aircraft at play here, but we can’t be totally certain. And if it is only one aircraft, that means its callsign changed mid-flight for some reason. Was it wrong before, or is it wrong now?

Consistency Is All I Ask! Score Us This, Our Daily Mask

HyperFeed makes rapid decisions about how to associate the incoming messages with flight records using a variety of real-time and historic data, combined with a myriad of rules and heuristics built up over decades of live flight tracking.

We open our analysis with HyperFeed’s scoring engine. As the first step of processing each new message, HyperFeed performs a broad search against its database for flights with which the incoming data could plausibly be associated. HyperFeed then scores each candidate flightplan based on how well the new message makes sense in the context of what we already know about the flight. The candidate gets scored based on things like how close the incoming departure and arrival times are to the candidate's departure and arrival times, whether the origins match and the destinations match, whether the reported position of the aircraft is near previously observed positions for the candidate, etc. Further consideration adjusts these scores up and down: the penalty for a mismatch in some dimension might be increased if the value has been corroborated by multiple sources, or decreased if we’ve observed it oscillate back and forth between different values.

HyperFeed decides which flight, if any, to match with the incoming data using a score-threshold system. If a candidate scores below the threshold, it’s rejected no matter what scores the other candidates received. If any candidates reach the threshold, it’s usually only one, and HyperFeed proceeds on the assumption that the incoming message is talking about that flight. If none of the candidate flights score high enough, HyperFeed concludes the incoming message must be describing a flight about which we haven’t heard, and creates a new flight record to track. Occasionally more than one candidate will exceed the threshold, in which case they enter a run-off. HyperFeed takes a closer (and more computationally expensive) look at the flights to pick a winner. If we decide two records we thought referred to different flights are actually talking about the same flight, we merge them.

The scoring system gives FlightAware coarse- and fine-grained knobs to tweak to adjust HyperFeed’s heuristics. As Air Navigation Service Providers like the FAA and Eurocontrol change their rules or how strictly they enforce them, as airlines make operational changes, and as technology improves and our sensor coverage expands, we can adjust those knobs to encode into HyperFeed our empiric knowledge about How The World Really Works.

After scoring comes attribute-by-attribute validations, with more heuristic-based processing to determine which attributes are inconsistent, and in what way. If a message moves the estimated arrival time of a not-yet-departed flight up by some amount, does it make more sense to adjust the estimated departure time, the estimated travel time, or to assume the new arrival time is incorrect and keep what we have? If a sensor observed an aircraft at a certain place and time, do we believe the aircraft could have gotten there by that time based on where we previously knew it to be? Or was our previous assumption about the aircraft’s position incorrect? Relevant bits of data are kept and used to update our comprehensive picture of the sky; unbelievable bits are discarded.

For Flights are a Tricky Thing, and This Is My Conclusion

FlightAware processes millions of messages per day across hundreds of thousands of flights. HyperFeed is the system that separates fact from fiction and decides when Theseus' ship has had so many boards replaced that it can no longer be the same ship. Not only must it make these decisions carefully, it must make them quickly. FlightAware provides real-time updates about current and future flights across the entire globe - we must maintain high throughput to cover the world, and low latency to keep up, both of which impose limits on the depth to which HyperFeed can analyze each incoming message, and which forces us to be clever and judicious in our analysis.

But there is increased computational headroom on the horizon! As we’ve talked about a bit before, the Flight Tracking crew is moving more of the codebase to Rust. The increased speed will give us the opportunity to explore more advanced analysis techniques that consider greater quantities of data, improving latency and accuracy at once! It’s an exciting time to be in flight tracking.