At FlightAware, one of our key priorities is to consistently deliver high-quality software. We are committed to maintaining and continuously improving product quality, benefiting both our customers and engineering teams. Test automation is crucial in achieving this goal. FlightAware crews and the QA wing constantly strive to maximize our test automation coverage, ensuring faster software delivery without bugs and regressions.

For our testing strategy, we try to evaluate code from all possible angles by implementing comprehensive test coverage for unit, integration, UI tests, and performance tests. We also remain cognizant of the test turnaround time (yes, we try to prevent test builds from taking several hours to finish)

Last year, when I joined FlightAware, I became part of an exciting crew responsible for a technical transformation effort called WebNXT, with a goal to reimagine FlightAware’s web stack utilizing modern languages, frameworks, and tools. As part of this effort, our crew began migrating legacy FlightAware maps and pages and adopted a monorepository structure for newly developed applications and libraries.

When selecting a test automation framework, there were several factors we considered for decision-making, like how tests can provide faster feedback to developers and how can we achieve extensive coverage, includes running our tests on cross browser, cross-platform, and mobile web. We evaluated options like Cypress, Selenium, and Playwright. After a few proof of concepts, and carefully weighing the pros, cons and efficiency, we decided to adopt Playwright for building and maintaining our end-to-end testing suite.

In this blog post, we'll delve into why we chose Playwright as our test automation framework, discussing the decision factors and benefits we’ve gained from it. I'll also showcase how integrating key features of Playwright has positively influenced our development and testing lifecycle.

Cross-browser

Cross-language

Mobile view emulation

Cross domain testing

Network interception and mocking

Integrating Playwright in a Monorepo Environment

Configuring Playwright in Monorepo application was quite simple. It took me less than 30 minutes to setup our app and run our first mock test. Quick, right? You can refer to Nx’s readme on Playwright on how you can set it up for your app. Once our app was migrated to the monorepo, here’s how I created an end-to-end project for it.

This command created an e2e tests folder adjacent to existing app-1. Since I wanted to run my tests against the local development server, I passed the webServerCommand and webServerAddress optional arguments.

playwright.config.ts

Did you notice something on line 2 in the code snippet above? Well, that’s the root-level baseConfig I created for all my apps. This type of configuration abstraction helps us maintain common settings leveraged by e2e tests and avoid writing repetitive boilerplate configurations across different apps. Pretty Neat!

Example, base-config playwright.config.ts

For reference, here’s What your project would look like…

Write your first test script in the e2e folder and simply run the command nx e2e app-1-e2e , and you will have your first successful test run.

Network Interception and Data Mocking for Enhanced Testing

One of the challenges of testing applications like FlightAware is validating the canvas map and related objects like data tables (like our newly designed flight cards), which are dynamic and updated based upon real-time flight data. We addressed this challenge by integrating Playwright’s network interception capabilities and incorporating mocking in our test workflow.

For instance, the following code intercepts a graphql request for flightBox operation and injects mock data in real-time to generate an expected flight within the flight card. Depending upon the operation, we may also use it to render an expected flight path on the map.

This workflow addresses two challenges:

1. We can capture a screenshot of such a flight path and compare it with the baseline image through visual testing.
2. We can validate the mock data in the tables to make sure they are correctly rendered.

You can utilize network interception in test automation to essentially block or mock any request to meet your goals. For instance, one can use it to block ads on the webpage if testing ads isn’t a primary test goal.

This is one of the great features that Playwright offers compared to its counterparts. During our proof of concept with Cypress, we faced challenges intercepting and mocking requests in Firefox and WebKit (Safari's rendering engine). While Cypress allows you to disable chromeWebSecurity in Chrome, there’s no simple way to do this in Firefox or Safari, which makes intercepting cross-origin requests more challenging. Although Cypress primarily supports Chrome, it can be used with other browsers, but there may be limitations.

Insights Unleashed with enhanced Reporting

Test reporting enhances transparency and visibility into overall quality and coverage for stakeholders. We at FlightAware try to keep our testing cycles concise and effective by running tests at different layers, and timely reporting of the test results is an important objective. This enables our crew and QA wing engineers to track any test failures, facilitate rapid resolution in case of a bug, and mitigate false positives quickly.

We’ve integrated Slack reporting into our Playwright tests to achieve this goal. Integration is straight-forward - you can install playwright-slack-report using a simple npm command - npm install playwright-slack-report -D.

Hook this package into your playwright.config.ts

There are several customizable features available for reporting. For example, we tag the relevant engineer or team alias whenever a test build fails, ensuring a quick, direct response which results in minimum delays in identifying and resolving issues.

Optimize Efficiency with Performance Testing

Have you ever had an opportunity to visit our newly built surface maps ? Take a look at Atlanta Airport and you’ll notice a high volume of flights displayed on the map, and they are all updating in real-time. As part of our overall testing strategy, we aim to ensure that our releases do not negatively impact performance.

We’re taking advantage of Playwright’s performance monitoring capabilities, which leverage CDP (Chrome DevTool Protocol) in capturing the performance metrics during the test runs. Our primary focus is to monitor and record JSHeapUsedSize and CPUUsage metrics of a webpage over an extended period of time while simulating various user actions. This kind of test helps us identify potential memory leaks and page crashes due to code bugs.

We collect these metrics during our test runs and then use the data collected to plot a chart to demonstrate the JSHeapUsedSize over the duration of the test.

The direct integration with Chrome DevTools Protocol (CDP) gives Playwright a significant advantage over other testing frameworks like Selenium and Cypress. While Cypress offers some basic performance testing features, such as measuring page load times and intercepting network requests, Playwright excels with its CDP integration. This allows for more detailed performance monitoring and the ability to collect comprehensive performance traces.

Advanced Playwright Testing: Managing Browser Contexts

Browser context in Playwright is an isolated environment within a single browser instance, allowing multiple tests or sessions to run concurrently without interference. Each context behaves like a separate browser, with its own: Cookies, Cache, Session storage, etc.

This feature enables us to simulate multiple users or sessions, allowing us to test various scenarios, such as concurrent logins or different application states for the same user. For example, session management is a key functionality that can be automated using this capability. You’ve probably come across functionality (such as the one on FlightAware’s website) that lets users manage all their active sessions in one place—viewing, deleting individual sessions, or clearing all sessions at once. With Playwright’s ability to create multiple contexts, we can simulate and manage multiple user sessions for the same user within a single test.

Here’s a very simple example demonstrating how to use browser contexts in Playwright. This example creates multiple browser contexts within a single test and verifies that the sessions and cookies are distinct for each page.

The ability to use browser contexts in Playwright offers distinct advantages over other frameworks. Cypress lacks built-in support for multiple contexts, making it challenging to manage separate user sessions. While Selenium does allow for multiple browser instances, it does not provide the same level of context isolation. This means we often need to manage cookies and local storage manually across sessions, adding complexity to our testing.

Some helpful Tips & Tricks

Following are the tricks that have been super useful for me; sharing them here in case they help you

• If you want to run tests on a specific platform, such as mobile web, then just include the keyword “mobile” in the test suite description and use the following example config.

• You can use environment variables to trigger condition based behavior. For instance, if you intend to send slack reports only for scheduled job runs, refer to the example code snippet in the Reporting section above.
• If your test is verifying numerous hyperlinks by navigating back and forth, and experiencing stability issues, consider using native JS commands instead of Playwright options.

• Configuring global setup and teardown files can be useful for performing certain operations at the beginning of your test suite execution. For instance, you might want to run a proxy server before any tests begin and stop it after all tests have completed. While hooks like before, beforeEach, after, and afterEach are available in different testing frameworks, they may not be efficient for parallel test execution. Additionally, many frameworks require extra boilerplate code for global setup and teardown. In contrast, Playwright streamlines this process by allowing you to configure a single shared environment for all tests, making resource setup more efficient by only requiring it once instead of multiple times.

Wrap-Up Thoughts

We are continually exploring and integrating various features of Playwright into our testing strategy. For example, we are currently utilizing multiple browser context capabilities to write test that simulates multiple user or sessions. We are also enhancing our testing framework by incorporating parallel testing features. Overall, we’ve found Playwright to be highly effective in strengthening our testing strategy.

Whether you're just starting with test automation for a new application or aiming to optimize your current setup, the insights shared in this post may help you leverage some exciting features that Playwright offers. We encourage you to experiment with the capabilities discussed and see how they can transform your development and testing lifecycle.

Happy automating! 😄