Fixing a flaky Playwright test with a loop

The problem

At work, I ran into a problem when trying to write a Playwright test for a user flow that contained a Headless UI radio group component.

Occasionally the test would fail because we didn't successfully click the radio button.

The actionability checks would pass.
The Playwright test would try to click it.
No errors would be thrown by Playwright during the click attempt.

But the test would fail, because an assertion after this click depended on that radio button being selected.

When I inspected the trace, I could see red validation text under the radio group, which only happens when no selection has been made and the user tries to proceed to the next page.

Experimenting with different solutions

Finding the root cause was the easy part.

How do you fix a flaky click that never fails actionability checks?

Isn't this why we have actionability checks in the first place?

I tried a few things to get the click to work:

• Using different locator strategies aside from getByTestId

• Targeting different HTML elements within the radio button's clickable area

• Clicking 20 times in a row

• Waiting for specific conditions like the correct page URL before attempting the click. Basically, adding intelligent waitFor statements that didn't duplicate the actionability checks.

None of this worked very well, or at all.

So I rolled up my "LeetCoder" sleeves and tried something I hadn't used since learning algorithm scripting on freeCodeCamp: a while loop.

Designing the while loop

First, I needed to analyze the user flow to determine how a "failed click" and a "successful click" affect the UI.

This sounds like common sense, but we don't often think about exactly which UI elements show up in a failure or success scenario.

I discovered that if I tried to proceed without clicking the radio button, validation text would appear under the radio group.

If I was successful in clicking the radio button, the test would click the "Next" button on the form and navigate to a new page.

But we would know even before clicking the "Next" button because an isChecked() matcher would tell us if the radio button was selected.

This "Next" button is what I call the "error text trigger," because if the radio button click did not succeed, clicking the "Next" button would "trigger" the error text.

With this information, I was able to create a conditional statement to determine if the radio button click succeeded or failed:

let buttonWasClicked = await radioButton.isChecked();

if (buttonWasClicked) {
    await errorTextTrigger.click();
    errorTextIsVisible = await page.getByTestId("ErrorText").isVisible();
}

But this wasn't good enough.

We needed to be able to retry this condition multiple times if we didn't have a successful click on the first try.

That's why I chose to use a while loop.

I could say "if we see the failure scenario, try again".

I could also break out of the loop if I saw the success scenario, resulting in the least possible amount of retries.

I also had to design the while loop in a way that it would not become an infinite loop.

So I used a maximum of 5 click attempts.

This was the minimum number of iterations needed to ensure this test was no longer flaky.

while (errorTextIsVisible || !buttonWasClicked) {
    if (iterationCounter === 5) {
        break;
    } else {
        iterationCounter++;
    }
...

But that was the last thing I did, after I saw the loop was solving the problem.

I had started out with infinite iterations because an infinite loop would tell me the loop wasn't solving the problem.

When the loop started to succeed, I set the counter to 20. Then 10. Then 5.

I still ran it locally and in CI a few times on multiple browsers to make sure the solution was airtight.

The final solution

You'll notice the final solution contains a waitForURL check.

This is meant to give the page enough time to load and to make sure we're not trying to interact with the wrong page.

We probably don't need it, but it's an added layer of assurance.

If this check fails, we won't even enter the loop, which allows us to get faster feedback.

export async function radioButtonGroupClick(
    page: Page,
    urlToWaitFor: string, // The URL where the radio button is
    radioButton: Locator,
    errorTextTrigger: Locator // A button you can click that will throw error text if the radio button wasn't clicked
) {
    await page.waitForURL(urlToWaitFor);

    let iterationCounter = 0;
    let errorTextIsVisible = await page.getByTestId("ErrorText").isVisible();
    let buttonWasClicked = await radioButton.isChecked();

    while (errorTextIsVisible || !buttonWasClicked) {
        if (iterationCounter === 5) {
            break;
        } else {
            iterationCounter++;
        }

        await radioButton.click();
        buttonWasClicked = await radioButton.isChecked();

        if (buttonWasClicked) {
            await errorTextTrigger.click();
            errorTextIsVisible = await page.getByTestId("ErrorText").isVisible();
        }
    }
}

Some final thoughts

This isn't the first time I've encountered issues with Playwright tests that interact with HTML generated by 3rd-party UI libraries.

For example you can't use Playwright's selectOption on a React Select component.

Instead, you have to click on the dropdown element and find an option by its label text.

While components borrowed from these libraries have significant productivity advantages for developers, they may result in productivity losses due to flaky tests.

We should take note of flaky tests that we can trace back to one of these libraries.

After enough time, patterns might begin to emerge and point to 1 library in particular.

There are many options on the market for UI components.

If one isn't providing a seamless testing experience, we can probably swap it out for another that does.

This is one way QA can "shift left" within the team.

Next time you have a flaky test and you think it could be the UI implementation, look at the code for that UI component.

Find the import statement. Note down the library used for that UI component somewhere.

If you're feeling spicy, do an experiment: swap the problematic component for the same component from a different UI library.

You might be able to reduce the time your team spends on flaky tests.

UPDATE: My solution kinda sucks actually

A reader by the name of Thananjayan Rajasekaran proposed the most sensible alternative in a comment on my LinkedIn post:

I had completely forgotten about Playwright's expect.toPass() feature!

I couldn't help myself. I was too curious to know if this could solve my problem.

So I tried it. And lo and behold...

    await expect(async () => {
        await radioButton.click(); // locator defined outside this block
        await expect(radioButton).toBeChecked();
    }).toPass();

4 frickin' lines of code, man.

What was I doing?

This expect.toPass() approach also fixed another flaky test I was dealing with.

(For that one, I passed some intervals to it to dictate how often to retry and how many retries to perform)

Anyway, hope you liked my over-engineered solution, too.

It was fun developing it, even if I've ripped it out of the codebase already.

Cheers, Thananjayan!