AeroAPI is FlightAware's query-based API, allowing customers access to FlightAware's vast array of flight and related data. It is utilized by thousands of customers for myriad purposes, some directly related to aviation and others tangentially so. It has evolved over the years, and the latest iteration offers a RESTful API with comprehensive documentation, better tooling for customers, and an ever-expanding aviation data offering.
A Brief History of the API
The API was launched back in 2006 as DirectFlight (v1), offering a SOAP interface and implemented with the TclSOAP library. At the time (and really for most of FlightAware’s history), Tcl was the language of choice for all backend services, so it was utilized for the DirectFlight implementation. This early version charged users based on their query count and the class of the query, and it was FlightAware's first API offering which attracted a number of customers. The query interfaces mirrored how FlightAware stored its flight data and presented fields and operations matching those internal representations.
DirectFlight was followed up by FlightXML (v2) in 2010 as an expanded API offering. With the new version, there were several new endpoints offered, expanding flight data availability for customers. Behind the scenes, a new Tcl library, tclws, was utilized to implement the API; its endpoints were hosted within Apache utilizing Apache Rivet and made available on the same servers hosting FlightAware's website. The original FlightXML release offered the same SOAP interface as DirectFlight, but in 2011 a JSON implementation was added, allowing for more flexible customer integrations. This version continued the same usage-based monetization model used by DirectFlight and continued to grow the API customer base.
Fast forward to 2017 and a new version of FlightXML, v3, was released as a beta. The new version maintained the same underlying architecture with a few enhancements to the tclws library but offered more comprehensive data in an individual call. This was driven by user feedback that the number of calls required to gather the full data for a given flight was onerous in FlightXML v2, so richer endpoints were produced to address this concern. As an example of the difference: with earlier versions of FlightXML, to get information about a particular aircraft, you would call the FlightInfoEx endpoint which would give you standard data on origin, destination, and runway departure and arrival times. That endpoint would return a list of flights, each with a FlightAware flight ID (a unique identifier for each flight), which would be used to query AirlineFlightInfo to get the gate departure and arrival times, information about codeshares, baggage claim, and terminal information. If you wanted that detailed information for 10 flights, you would make 1 call to FlightInfoEx followed by 10 calls to AirlineFlightInfo.
With the updated version, all this data was made available in a single call. The other major change in v3 was the monetization model utilized. In the v3 version, users would select a tier where they prepaid for calls in advance, so assuming they stayed within the allotted call volume, they would have a predictable spend each month.
Following the release of the FlightXML v3 beta, the team pivoted to work on enhancing and expanding FlightAware's streaming API: Firehose. There were still several endpoints that would need to be implemented to make v3 a full product offering, but it would be a few years before resources and priorities aligned for that work.
The Motivation for AeroAPI (v4)
In the Summer of 2020, after adjusting to a new world where we all worked remotely and were dealing with COVID, our product manager for APIs and I began talking about the future of FlightXML. We had achieved quite a bit of success with Firehose, but there were use cases where it didn't make sense and FlightXML was starting to look pretty long in the tooth. As we began brainstorming about its future, our PM proposed examining some commercial offerings, in particular certain API Gateway and Management solutions. I wasn't familiar with the various offerings from Mulesoft, Apigee, IBM, etc., and didn't see the value add at first blush.
During my first three and a half years at FlightAware, we had continued to embrace Tcl and home grow many of our solutions and product offerings. As the company was maturing, though, we recognized that we spent a non-trivial amount of engineering time designing libraries and other tooling for Tcl that would be freely available to us with other, more widely adopted languages. By 2020, we were beginning to adopt new third-party solutions and had integrated different languages into our technology stack. As an example, our Predict team utilized Python, Scala, and Spark in their tech stack and AWS for their model training. This was one of FlightAware's first uses of cloud technologies and a cloud compute provider. Historically, all of FlightAware's hardware was housed within leased data centers on our own servers and related equipment. This all led to some reluctance on my part at first to seriously consider any cloud hosted, third-party solutions for the future of FlightXML.
To guide our API's evolution, we had conducted surveys of our existing FlightXML users and there were a few key points that they wanted to see addressed. First was an improvement to the documentation and resources for developers using FlightXML. Second, they wanted us to offer a truly RESTful interface for the API. And finally, they wanted to see the functionality in v3 expanded to encompass the full suite of flight data endpoints. We realized that we were not going to be able to easily address all these concerns with another iteration of our existing API infrastructure, using the tclws library, without major investment in that library. We did not consider continued investment in that library tenable, and instead were excited about utilizing RESTful tooling available in most other languages. We also wanted to use OpenAPI to define the next interface of FlightXML, which would give us access to better documentation and tooling right out of the box. As we considered these requirements, it became clear that rewriting our API in a new language would be a monumental undertaking given all the ancillary work that would be required to make that a reality. We wanted to deliver this new version on an accelerated timeline and realized we could make use of an API gateway to leverage our existing endpoints to expose a new RESTful interface, combining v2 and v3 functionality.
We still needed to determine what API management features would be of interest to us and what monetization model our sales and product teams wanted to employ in the next version. That would require both more discovery for the engineering team and some legwork for myself and the product manager to wrangle the commercial details.
Building the Case for AeroAPI v4
Going into late 2020, FlightXML was rebranded as AeroAPI. We also determined that we wanted the first new version of AeroAPI to be a RESTful interface, defined by an OpenAPI spec, and implemented using an API Gateway. We had our baseline requirements together, but we knew we would have to do some internal selling to build consensus for our proposed approach, and we would need to examine the market of API Gateway/Management providers to determine which solution would fit our needs. We started this internal promotion campaign with the sales and product teams. During our meetings with them, we presented the findings from our survey of users, discussed what we wanted the next version of the API to be, and demoed some OpenAPI features to start building consensus around our approach. At this point we discussed how we were evaluating open source and commercial API Gateways and Management solutions with these stakeholders in a non-technical way, speaking to the feedback we had received from customers and how our proposed solution and approach would solve those problems.
We also began discussions with the groups around the monetization model they wanted to employ. This had historically been a bit of a pain point for us, as our billing systems for the APIs were all bespoke and had a limited set of variability designed into them. So, if the sales team had a particular deal with a customer in mind that didn't fit into one of our billing arrangements, there was a considerable amount of engineering time or manual work required to accommodate that sale. Ideally, with our next generation platform we wanted to address their initial needs but have the flexibility to support variations in the billing model for custom deals. The sales group determined that they wanted to drive charges largely based on usage versus the prepaid plan in place for v3, but that for certain features they wanted customers to commit to some minimum monthly spend.
In parallel to the selling and exploring phase with the sales and product groups, we started examining a list of solution providers. We started with a pretty big mix of both commercial and open source offerings and looked at our basic requirements to begin narrowing the field. Our initial batch of candidates was on the order of 10 to 15 offerings, but through an examination of their published feature sets and our initial requirements, we were able to narrow that list down to the 6 top contenders. Those included two of the top API Management providers (Apigee, Mulesoft), two cloud solution providers (Azure API Management / AWS API Gateway), and two open source solutions.
In order to evaluate each solution, we created a draft OpenAPI spec for the updated API. We would use this as a baseline for our POC work with the various solutions and to begin the discussions around endpoint pricing with product and sales. Moving into the first part of 2021, we met with each of our potential solution providers, attended workshops, set up trials, and created POCs of the API with our various platform choices. Based on those evaluations, we were able to establish some metrics around the effort required to fully develop and integrate the new platform into the API. We also looked at what, if any, monetization options the platforms supported and examined the various bundled developer portals that were offered. This data was compiled into a feature comparison chart, and we began to assign dollar values to the integration development efforts required and licensing costs for each option.
Arriving at a Solution
As we narrowed down our preferred solution providers in the beginning of 2021, it was time to start engaging with the executive team to pitch our solutions and see what kind of budget we could have allocated for the project. During our evaluations, we had been impressed with the offering from Apigee, and found their monetization package's flexibility attractive. Apigee was one of the most expensive options from a licensing perspective, but we believed it would be one of the quickest to fully implement and integrate, minimizing internal development costs.
We prepared presentations on the options we had evaluated, the objectives we saw for the new API, and some proposed timelines for delivering the new product. Our product manager and I did several dress rehearsals on our presentation with our director to ensure we delivered our message succinctly and in the most impactful way. When the day came and we presented our plans to the team, we were encouraged to find that the CEO and others on the executive team recognized that while the more mature options – in particular, Apigee – had higher licensing costs than some of the cloud providers or open source solutions, there were significant development costs associated with implementing those solutions and that time to market was an essential variable as well. During this presentation, we did not make a final recommendation on platform choice but discussed our top three options and sought guidance on budget to arrive at a final recommendation within their guidelines. We were given approval to complete the exploration of our final options and embarked on that last endeavor.
The final technical explorations focused on our top three contenders. While we were excited about some of the features in our preferred open source solution, there were various limitations in their out-of-the-box offering that would have required significant development efforts to fully realize, and we eliminated that option first. This left us with Apigee and one of the cloud provider's API management solutions. The cloud provider had a very simplistic billing option allowing you to connect to a third-party payment solution and setup a hook for managing your own monetization, while Apigee’s monetization model was very flexible and could support FlightAware’s billing model. Given the pains we have had with our bespoke billing implementations, Apigee's monetization solution was particularly attractive.
I mentioned previously that we were also evaluating the possibility of using the API management solution's developer portal as an out-of-the-box solution to enhance our customer's experience. Both of our final options offered these developer portals, but the cloud solution's portal was written in TypeScript while Apigee's relied on Drupal and a custom Drupal module. Neither one would be a perfect solution for our needs, so we would need to invest development efforts in either case. In the Apigee case, Drupal is a PHP CRM solution and FlightAware's web stack contains no PHP, nor do we have PHP developers on staff. Had we proposed deploying a modified Drupal supported by FlightAware engineers, that proposal would have been rejected as that is not a direction we want to embrace for our web engineers. Therefore, if we were to select Apigee, we decided we would build our own developer portal backed by Apigee's API in React and host it on FlightAware's website.
This left us with a choice between Apigee, where we would need to leverage our web team's efforts to build a developer portal, or the cloud solution which would require some modification of their developer portal as well as building out a full billing system to accommodate our monetization model. We now updated our more detailed features comparison chart, compared the development effort and resource availability, and weighed our choices.
We ultimately decided that Apigee best fit our needs, as we were better prepared to develop a new web portal than another generation of billing code. The executive team agreed with our final recommendation, and we were given the approval to proceed with final contract negotiation with Apigee and development of the AeroAPI v4+ API. That process was undertaken during the Summer of 2021, and we launched the new API in the first week of October. There were, of course, surprises along the way during the implementation, but no major roadblocks during that implementation. An upcoming companion post, written by our tech lead, Chris Roberts, will dive into the details of the technical implementation and discuss lessons learned during that process.
The Future of AeroAPI
With AeroAPI now out the door, we've been delivering on the promise of adding more capabilities to the API on a faster release cadence. The product and engineering teams have been working with sales and marketing to determine what new features and changes need to be made to the API to address more customer needs, and they have delivered several new features this year with a host of future enhancements in the backlog. You may recall that one of our reasons for utilizing an API gateway was that it would allow us to leverage our existing endpoints in a new product. That's only part of the story, though. We are increasingly moving away from the use of Tcl for our backend infrastructure, and our API gateway is allowing us to begin replacing legacy backends with new solutions in an iterative way. One of the first examples of that will likely be our alerts configuration backend. As the API team enhances our alerting capabilities, they're going to replace that backend; we will then make use of that new backend not only for the API but also for our web and mobile products, allowing us to multiply the benefits of one team's efforts.
It is an exciting and challenging time for us at FlightAware. This project and our efforts in predictive flight data have been some of our the company's first uses of cloud offerings and third party solutions, but they have paid off significantly and paved the way for future use cases. We are in the process of replacing a number of our legacy billing solutions with a third-party platform, and are excited about how all of these changes will accelerate our ability to deliver more value to customers in the future.
