TLDR; We can only test to the level supported by our Ability, and the degree to which we are supported by tooling to Observe, Interrogate and Manipulate the System.

Video

Watch on YouTube

• 00:00 Introduction
• 00:26 Application Under Test
• 02:51 Live Testing
• 03:41 Testing vs Automating
• 06:03 Testability vs Automatability
• 09:25 Workarounds
• 10:55 Technical Knowledge
• 15:44 Test Approach — JavaScript
• 17:02 Exercise — Test Approach
• 17:24 Automated Execution Approach
• 18:34 Exercise — Code Review
• 20:35 Exercise — Your Project
• 20:59 Server Side
• 24:20 Exercise — Test Approach — API
• 24:52 API Tooling
• 29:14 Exercise — API Testing
• 29:34 Automating the API
• 31:54 API Interacting with UI
• 37:09 End Notes

Introduction

A customer has reported that one of your pages doesn’t work. But your automated execution coverage tells you that it does. What went wrong?

That’s what we’re going to explore now.

• difference between testing and automating,
• testability vs automatability,
• we’ll think about test coverage from a technology perspective,
• and we’ll look at some tools that can help you test.

Here’s the Application Under Test.

Infinite Scroll Challenge on the TestPages

Many sites have infinite scroll. You scroll to the bottom and it loads new items.

That functionality works. We have have automated coverage. That passes.

What’s the issue?

Oh yeah, sorry I forgot to mention. I’m going to encourage you to think. Ideally we would do this as a workshop, as hands on training, but it’s a video and text workbook.So all I can do is prompt you into thinking.

Contact me if you are interested in live training for your team or organization.

So what’s the issue?

Well, I can’t click the button. The page refreshes and scrolls too quickly and the button isn’t enabled for long enough when it is visible.

This happens with a lot of infinite scroll sites, there are footers that can’t be clicked, stuff at the bottom of the screen that you can’t see.

This did work at some point, because we tested it, but I guess someone changed the time out value and the automated execution assertions didn’t flag this as an issue.

So let’s start thinking about this from a Testing perspective.

Live

If this was a live project we could just stop here and raise a defect.

Obviously the timeout is wrong. Raise the defect. Move on to the next issue.

But wait.

We have automated coverage, and it didn’t highlight this issue.

We should think about the difference between Testing and Automating.

Testing vs Automating

Testing is what we do. As humans, we interact with the software and we observe, we build models, we learn, we compare the actual interaction with the models. We report information derived from the difference between our models and the observations.

When we investigate an issue and expand our models we interrogate the system more deeply to learn what’s going on.

• Testing is the human activity of interacting, observing, thinking, learning, experimenting.
• Automating, is the human activity of making the interaction and observation of the system an automated process. And that includes the comparing of observed results with the expected results.

Automating is a human activity which results in the output of an automated execution process. The only involvement with the human after the activity has been automated is to investigate the failure reports and maintain it when it fails.

So both Automating and Testing are human processes. But testing leads to more human processes, automating leads to an automated process that only involves humans when it fails.

NOTE: AI might change how I view the process of automating. Certainly I’ll need more distinctions in how I describe nuances around Automating, but for now, all the automating I do is Human Initiated, or Directed, and results in some automated execution process.

Testability vs Automatability

And let’s just quickly look at Testability and Automatability because these words are used badly when describing software.

Is this system testable?

Yes. I can access it in the browser, I can see it, I can interact with it. The button toggles too quickly but I can test the application.

But when we talk about testability we often talk about, adding id s to the elements, make it it more observable, etc.

Well the id s, are really for automatability, not testability.

In the browser, the browser tooling makes it observable because of the technology used, not the system.

Most of the time when we are talking about testability we are really talking about automatability.

This application has been built with automatability in mind.

It has id s, there are classes, the JavaScript source is visible, it is easy to change the state and configuration variables, this thing is so easy to automate and observe. But none of that was required to help me test it.

Testability is not the same as automatability. Keep that in mind as we continue through this process, particularly when we move to the server side interaction.

With the JavaScript Infinite Scroll system, the main thing that impacts my Testability, is the bug, preventing me from testing the button functionality.

Workarounds

Let’s quickly consider workarounds.

Your ability to find workarounds will impact your ability to test the system.

So your ability impacts the Testability.

Testability is as much about your ability to test the application as it is the application supporting you in testing it.

Depending on how deep you want to go. You can only test the application to the limit that you can observe, manipulate and interrogate the application.

Your ability to test the application is often impacted by the usability of the application, and that is true here. The application is not usable for one main function so it is hard to test that function.

But, we can work around that, for this application, with more Technical Knowledge.

Technical Knowledge

When working with the web, we need to understand our tool capabilities.

My tool at this point is the browser.

What can I do with it?

I can view the page source.

I need to be able to understand HTML. Some CSS. Some JavaScript.

Having the tooling ability to look at the source doesn’t help me unless I understand what I’m looking at.

So if you’re testing web applications, you probably want to understand:

• How a browser works,
• How HTML works,
• What is CSS and how we use it,
• What is JavaScript and how it works.

At the very least to have a reading ability of these technology artifacts.

So at this level, at the source, with the technical knowledge I have… I can see in the JavaScript some variables, I can see they are amendable.

But, I don’t have the tooling to amend it yet.

So, what else do I have available?

I can look at the dev tools.

I can see the DOM view shows me much the same as the source. But… it can be different.

• The source is what we gave the browser to work with.
• The DOM is what the browser created after interpreting and executing the source.

In the DOM I can see the JavaScript as well.

I also have the ability now, to interact and manipulate the JavaScript variables that I observed in the source.

There are a few timeout and millisecond variables there.

Perhaps the bug is the scrollAfterMillis variable?

Let me change that to 3000.

And… now it just takes longer to refresh, but I still can’t click the button.

Let me try the preLoadTimeout:

Now, the button stays active for 2 seconds before it loads the next set of data. That’s my workaround to make this application testable.

It also means that I can go a little deeper in the bug report and say that the root cause is the preLoadTimeout being too small a value so the user doesn't have time to click the button.

You might be interested in this video and blog post covering more details of Chrome Dev Tools for Testing.

Test Approach

So what is my test approach for this?

• load the page
• scroll down
• see that it refreshes and adds the data I expect
• click the button
• see that it doesn’t refresh when I scroll

Is that it?

• Do I need to reload the page and check that it starts auto refreshing again?
• How long do I need to wait after stopping it, and trying to scroll to make sure it doesn’t start auto-refreshing again?
• What else?

Exercise — What is your test approach?

After this video I encourage you to think through what your Test Approach for this application would be.

Test Approach vs Automated Execution Approach

Once you’ve figure out how to test it, how would you automate it?

I’ll show you what we have here.

And this test passes.

@Test  
public void scrollToStopLoadingAndClick(){  
WebDriverWait(driver, Duration.ofSeconds(10)).until(  
        ExpectedConditions.elementToBeClickable(page.getStopLoadingButton()  
    ));  
 
    page.getStopLoadingButton().click();  
  
    new WebDriverWait(driver, Duration.ofSeconds(10)).until(  
        ExpectedConditions.textToBePresentInElement(  
            driver.findElement(By.id("statusMessage")),"Clicked")  
    );  
}

Is that good enough?

It works. It passes.

But it didn’t highlight the fact that the halting functionality is unusable for a human.

This is one of the issues when we automate something.

We automate the functionality.

We don’t automate the experience.

If you are interested in more details about WebDriver with Java then have a look at this video Masterclass on the basics of WebDriver with Java

Exercise — Critique the code

So after the video, or now, pause the video.

Critique this code:

• Understand what it does
• What does it not do?
• I can see it doesn’t try to re-scroll the page to make sure that clicking the button stopped the auto-scroll.
• What else does it not do?
• What conditions does it not check?
• Should it match the user experience?
• How could it?
• What would you change to match the user experience?

Much automated execution coverage does not match the user experience. When it doesn’t then there is a risk that the automated execution passes, but the user experience doesn’t.

And there is a risk we don’t notice.

Testing will hopefully identify those issues. But automated execution often does not.

Exercise — Consider Your Project

Is there a risk on the projects you work with that the automated execution covers the functionality, but not the user experience of that functionality?

Server Side

OK, so that was the JavaScript version of the Infinite Scroll.

It offered us some scope for thinking like the tester.

But let’s push this a little further.

We also have the server side infinite scroll.

https://testpages.eviltester.com/challenges/synchronization/infinite-scroll/server-side/

It looks the same. It has the same bug, the button is too fast. And my automated execution passes.

But this page, when it needs new items to display it calls the server to get the information back.

Do you trust that statement?

How do you know it calls the server?

Visibly, in the browser, when I test it and observe the output. It looks the same.

We need to be able to be able to interact with the application from a technical perspective.

What if I just think it is connecting to the server, but I accidentally release the wrong version? What if I just renamed the file?

So we need to check.

We need to observe the application at multiple technical levels.

So let’s look in the dev tools again. And this time we’ll look in the network tab.

Filter by Fetch/XHR to see the requests made by JavaScript and I can observe a call to moreitems

Let’s interrogate that request

I can see a JSON response:

[
    {
        "id": 1,
        "text": "This is content item number 1. Scroll down
         to automatically load more content when the
          \"Click to Stop\" button becomes visible."
    },
    {
        "id": 2,
        "text": "This is content item number 2. Scroll down
         to automatically load more content when the
          \"Click to Stop\" button becomes visible."
    },
    ...
]

Great.

So I have tooling to increase my ability to test this.

But I need to know about:

• HTTP Requests
• Fetch and XML Http Request
• JSON

My ability to test this application will be limited if I do not have that technical knowledge and if I do not know how to use the Dev tools network tab.

But at least I know now that it is making server requests.

I didn’t have to trust anyone. I could verify that this was true.

So now, our testing scope just expanded.

Exercise — Server Side Test Approach

Now, what do you have to test?

Is it good enough to just test the front end now but scrolling up and down and clicking the button?

After watching this video, take some time to think through what you want to test to cover the Server Side infinite scroll.

• Do we also have to test that HTTP call?
• How much do we have to test it?

Server Side Test Tooling

And… do you know how to do that?

How can you amend the HTTP requests?

We can do some of that from Chrome.

I could:

• copy as cURL
• copy as fetch

with cURL I can use the command line.

Or fetch I can amend it in the console.

I could take the cURL and paste it into an API tool.

I can do that in Bruno by creating a new request from cURL.

Or in postman I can paste in the cURL command into the new request.

I can experiment with it in these tools.

It is important to test the API on its own.

For example, when I built this, it was only when I was automating and testing the API, that I realised that I really need a limit on the count. If I let people make a request and ask for 6 million items back. That could easily bring down my server.

NOTE: you can find a list of API Tools on the API Challenges site.

Exercise — API Testing

As an exercise, think through what conditions you would want to test on the API, then you can use any of the tooling approaches mentioned to experiment.

Automating The API

I would probably want to automate the API as well.

This does remove us from the user experience because the responses from this API are normally handled by the UI, so just because we see something working in the API we can’t assume that it works with the API interacting with the UI.

I used RestAssured to automate the API.

The coverage runs at the same time as the web UI coverage.

And I have a lot more coverage at the API level, than I do at the UI level.

Think about what coverage you would add for the API.

Then try to automate it. I used Java, with RestAssured, but you could use any library or programming language you want.

Or you could even have a set of canned requests in Postman or Bruno or any of the other API tools.

For production API automating, I primarily use code and HTTP or API libraries. I don’t tend to use tools like Bruno or Postman.

But for production API Testing, I do use Bruno and other tools. Because I like the flexibility and the ability to see the requests being made.

Testing and Automating often sound like the same thing. But they have different aims and are supported by different tooling.

API Interacting with UI

And now I’m in the position where I’m testing the API in isolation of the UI.

Is that a risk?

For example, I don’t know how the front end handles a 500 error response. I can see it is the same JSON format but it is a different status code, does that make a difference?

[
    {
        "id"=0,
        "text"="For input string: '-1.02'"
    }
]

Would you test that?

Do you know how to test that?

One way to do that is to use a Proxy.

Intercept the request, amend it to be one that triggers an error and play it through to the front end.

And then I can see if the system handles error responses or not.

I would either use Zap or BurpSuite.

For this exercise I would use ZAP.

• open a session using Edge
• create a new context for testpages.eviltester.com
• filter the history to “Show only URLs in Scope”
• Set the breakpoint on all requests and responses
• trigger a scroll through the UI
• amend the request
• see the result in the UI

I can use tooling to increase my ability to observe the system and manipulate the system at different technology levels.

NOTE: A list of recommended HTTP Proxy Tools is available on the API Challenges site

End Notes

It is often surprising how much depth we have to test and automate when testing even simple pieces of functionality.

The more that we extend our technical ability to cover the multiple levels of the application, and we learn how to use tooling to help us observe, interrogate and manipulate at those different technology levels, the more we can expand the coverage of our testing.

Automating is not the same as Testing. Both are human processes, but the output of automating is not the same as the output of testing.

But yes… we can be testing, as we are automating, we may well learn things during the process of automating the application. But we should not confuse the continued execution of the output from automating with testing.

Testing can miss things because we are human, or we may not have covered the conditions, or we may not have gone deep enough into the system.

Automated execution can miss things because we humans forgot to add the conditions, or we didn’t assert enough. But automated execution can also miss the human user experience and tell us things are working, when they clearly are not.

I hope that you do now go off and do the exercises yourself. You may not have all the skills to do this yet, you may not have tried all the tools. That just means you can revisit this exercise and application multiple times as you grow your knowledge and skill set. And repeat it when you want to evaluate or learn new tools.

That’s why I created the Test Pages, and that’s also why this is a fairly high level overview.

Remember to work through the exercises.

Exercises

The Applications Under Test

• https://testpages.eviltester.com/challenges/synchronization/infinite-scroll/
• https://testpages.eviltester.com/challenges/synchronization/infinite-scroll/server-side/

Exercise — What is your test approach for JavaScript Infinite Scroll?

• think through what your Test Approach for the JavaScript Infinite Scroll application would be.
• Do you need to reload the page and check that it starts auto refreshing again?
• How long do you need to wait after stopping it, and trying to scroll to make sure it doesn’t start auto-refreshing again?
• What else?
• what conditions would you cover?
• how would you approach the testing?

Exercise — Critique the code

• Understand what it does
• What does it not do? — I can see it doesn’t try to re-scroll the page to make sure that clicking the button stopped the auto-scroll. — What else does it not do? — What conditions does it not check?
• Should it match the user experience? — How could it? — What would you change to match the user experience?

Critique this code:

public class InfiniteScrollTest {  
  
    static WebDriver driver;  
    static InfiniteScrollPage page;  
  
    @BeforeAll  
    static void setupWebDriver(){  
        driver = DriverFactory.getNew();  
        page = new InfiniteScrollPage(driver);  
    }  
  
    @BeforeEach  
    public void reload(){  
        page.open();  
    }  
  
    @Test  
    public void scrollToStopLoadingAndClick(){  
            ExpectedConditions.elementToBeClickable(page.getStopLoadingButton()  
        ));  
  
        page.getStopLoadingButton().click();  
  
        new WebDriverWait(driver, Duration.ofSeconds(10)).until(  
            ExpectedConditions.textToBePresentInElement(  
                driver.findElement(By.id("statusMessage")),"Clicked")  
        );  
    }  
  
    @AfterAll  
    public static void closeDriver(){  
        driver.close();  
    }  
  
}

Supporting Abstractions:

public class DriverFactory {  
    public static WebDriver getNew() {  
  
        WebDriver driver;  
        ChromeOptions options = new ChromeOptions();  
		options.addArguments("--disable-smooth-scrolling");   
        driver = new ChromeDriver(options);   
        return driver;  
    }  
}

public class InfiniteScrollPage {  
  
    private final WebDriver driver;  
  
    public InfiniteScrollPage(WebDriver driver) {  
        this.driver = driver;  
    }  
  
    public void open() {  
        String url = SiteConfig.SITE_DOMAIN +
         "/challenges/synchronization/infinite-scroll/";  
        driver.get(url);  
    }  
  
    public WebElement getStopLoadingButton() {  
        return driver.findElement(By.id("loadMoreBtn"));  
    }  
}

public class SiteConfig {  
  
    public static final String SITE_DOMAIN = "https://testpages.eviltester.com";  
   
}

Exercise — Consider Your Project

• Is there a risk on the projects you work with, that the automated execution covers the functionality, but not the user experience of that functionality?

Because if that’s a risk, you might want to revisit your test approach and your automated execution approach.

• How might your approach need to change?

Exercise — Server Side Test Approach

What do you have to test now that the Server Side calls are involved?

• Is it good enough to just test the front end now but scrolling up and down and clicking the button?
• Do we also have to test that HTTP call?
• How much do we have to test it?

Exercise — API Testing

As an exercise, think through what conditions you would want to test on the API, then you can use any of the tooling approaches mentioned to experiment.

Suggested Tools:

• Browser Dev Tools — fetch requests in console
• Browser Dev Tools — generate cURL and use from CLI
• Bruno
• Postman

A List of API Tools is available on API Challenges site

Exercise — UI and API Interactive Testing

• Use a Proxy to allow you to observe and interrogate the traffic from the Web Site to the Internal API backend.
• Amend the Request to trigger an error response and see how the front end handles it.
• Try testing the API from within the proxy.

A list of recommended HTTP Proxy Tools is available on the API Challenges site

Join our Patreon from as little as $1 a month for early access to videos, ad-free videos, free e-books and courses, and lots of exclusive content.

Originally published at https://eviltester.com on February 6, 2026.

TLDR; When testing web apps we need to test the interaction between the browser implemented controls and our system. We don’t test the browser.

Testing HTML Form Fields

Watch on YouTube

Testing Web Technology

I like to use the term Technical Testing.

Many people don’t. I think other people view the term as implying that some people are technical and other people are not.

I use the term to cover “Testing which is informed by Technical knowledge”.

So if you’re testing a web application, you need to understand web technology, you need technical understanding of how browsers and the web works, otherwise you can’t test it effectively.

In this video I try to explore some of the reasons why this is important.

If I’m testing a web application and it has a form, and in the form is a date field which has a date control calendar pop up which lets me set a date.

What should I test?

What should I test?

Should I test that…

• the calendar popup works on mobile and every web browser?
• the calendar control works with keyboard?
• the calendar control correctly switches years and months when I click the control buttons?

Well, no. Not if the HTML looks like this:

<input id="datetime-local-input" type="datetime-local"
 name="datetime-local">

All of the complex UI interaction is implemented by the browser.

All our application does is use an off-the-shelf browser provided control.

Do I not test the functionality?

When we are using off-the-shelf controls like this we have to test the interaction between the control and our application.

If we added JavaScript event listeners to the control to perform extra validation or functionality then… yes we have to test that.

If we use the date entered in the control by the user, then we have to test that our application can handle the data supplied by the value of that control.

If we are not using the date then we should question why we have the control on our application UI.

Learn to spot technology risks

I can see from the HTML above that there might be an accessibility risk, because I don’t see any ARIA attributes associated with that control.

So we might not have configured the control in the page properly to implement all the accessibility requirements we might want.

If we had created a custom control, or it is specialised React control, then we do want to test it because we just introduced a technology risk that might impact mobile and different browsers. We might also have functional bugs.

But if we are using built in browser controls we don’t need to test as much functionality related to the control. We need to test the functional interaction with our application and custom code.

Test the Domain Configuration

We shouldn’t spend a long time testing validation messages from custom controls, but if a control has been configured then we need to test that the configuration is correct.

<input id="password-input" type="password" name="password"
 pattern="^[A-Za-z0-9_]{4,12}$" maxlength="14" required="true">

I don’t need to test that the password control above shows dots instead of letters and hides the input. But I do need to check that the configuration we supplied for the validation pattern and length are correct. And I need to make sure that the password is actually required (because we configured that it is).

NOTE: eagle-eyed readers, who have a technical understanding of how the configuration works will probably have spotted a bug in the above configuration. Yes. We should find that bug in our testing. And we know to look for it because we understand that the control has been configured with our application domain rules, and that’s what we are testing.

Don’t Trust the Client

The greater the technical understanding we have of the web technology, the less we will trust any data that the server receives.

We can edit the DOM to bypass domain configuration.

We can use JavaScript to enter data that bypasses browser validation.

We can bypass the browser altogether and just communicate directly with the server.

A greater technical understanding allows us to understand the risks of the technology more effectively

Testing Web Apps

All of the examples above are from:

• testpages.eviltester.com

I also have a guide to testing HTML elements to describe some of the risks and issues that we might have when using or configuring off-the-shelf HTML fields and elements.

• Testing Input Elements — Notes and Exercises

I had to learn web technologies when testing web applications.

I have seen people functionally test the browser implemented controls because they didn’t learn web technologies.

This means they wasted time, because they didn’t understand the technology they were testing.

Join our Patreon from as little as $1 a month for early access to videos, ad-free videos, free e-books and courses, and lots of exclusive content.

Originally published at https://eviltester.com on January 16, 2026.

I released a new version of my CounterString generation Chrome Extension with new features and a new UI.

CounterString Extension

Watch on YouTube

Source and Download Links

I have a tool page for the CounterString extension with more details.

• https://www.eviltester.com/page/tools/counterstringjs/

But you can find the tool:

• source
• on the chrome store

Overview

I first released the CounterString Extension in 2019 (I think), and at that point it was a simple input field that created a string and added it as a value to the input field.

Now it can:

• Generate CounterStrings
• Configurable length and delimiters
• Generate Random Data From Regex
• Generate Character Ranges
• Repeat text, characters and regex
• Add generated data as value, with input event, to an input field
• Trigger Key events to ‘type’ the data with configurable speed
• Binary Chop Calculator for field length exploration

Some of this was a result of migrating code from my Java Test Tool Hub (which I think I worked on about 12 years ago):

• https://github.com/eviltester/testtoolhub

From the Test Tool Hub I took the:

• Robot — I had to find a different way of triggering events, but this allows ‘typing’ the data into the field rather than just amending the value.
• Increased customization of the CounterStrings
• Ranges of Data
• Repeated Data
• Binary Chop Range Calculation

And the Data from Regex was repurposing some code from my AnyWayData.com test data generator which uses the same JavaScript library to generate data from a Regex — RandExp

CounterStrings

I wrote about CounterString Algorithms in an earlier blog post.

This extension uses both reverse generation and forward generation.

I initially learned about CounterStrings from James Bach. James has a tool called PerlClip which is a command line tool to generate data directly into the clipboard. This uses a reverse algorithm where the full string is generated and then reversed to paste into a field.

A forward algorithm allows ‘typing’ the String character by character without generating the full string first.

While I don’t generate the full String first, I do precompute a Schema which I then follow to generate the String, so it takes about the same processing time, but uses less memory and allows streaming of the result.

I created a different algorithm this time, which is much simpler to understand, but probably takes longer to execute. I’ll need to update the algorithm page at some point, but you can see the algorithm implemented in the source code.

• github.com/eviltester/counterstringjs/blob/master/extension/js/generateSchema.js

Other CounterString Tools

A quick hunt around the web revealed a few more CounterString tools that I don’t think I had found all of these before. So these might be interesting for anyone thinking of creating their own implementation.

• PERL: https://www.satisfice.com/download/perlclip
• TYPESCRIPT: https://github.com/j19sch/counterstring/
• RUBY: https://github.com/jamesmartin/counterstring
• JAVA: https://github.com/eviltester/testtoolhub
• PYTHON: https://github.com/deefex/pyclip
• RUST: https://github.com/thomaschaplin/rust-counter-strings

Another Chrome Extension:

• https://github.com/Pawel-Albert/utilities-for-testing-extension
• and some associated Test Pages that I haven’t seen before
• https://pawel-albert.github.io/utilities-for-testing-extension/

Join our Patreon from as little as $1 a month for early access to videos, ad-free videos, free e-books and courses, and lots of exclusive content.

Originally published at https://eviltester.com on January 13, 2026.

Should you have an online portfolio showcasing your skills and abilities to help get a job?

It really depends on the recruitment process. But… if I’m recruiting, and you have a profile then I will have looked at it. So it better be good.

Welcome to The Evil Tester Show! Covering the common question of “Should we have portfolio projects?” Alan shares his experience as both a creator and reviewer of portfolios, covering everything from the dos and don’ts of GitHub repos, to which skills actually benefit from a showcase, and how a well-crafted readme can make a difference. Whether you’re considering building a portfolio, looking to upgrade your current one, or wondering where to focus your efforts, this episode has practical advice to help you stand out as a top 1% applicants and avoid creating a portfolio that could hurt more than help.

audio and subscribe links here

Episode Summary

A practical look at online portfolios for developers and testers. Covering both trying to find your next role and also looking to switch careers. Most people don’t have public portfolios so that means you can really stand out if you build a good one.

We explore what actually matters in a portfolio: the difference between learning repos, personal projects, and true showcase projects designed for job hunting. We break down the content that impresses, and the stuff that doesn’t. We emphasize the need for simplicity and quality. Expect to find tips about managing forks on GitHub, writing an effective README, documenting your learning process, and promoting your work across LinkedIn and blogs to maximize visibility.

Tailor your projects to communicate exactly what you want and make your portfolio a focused sales tool rather than a catch-all for everything you’ve ever coded. Plus, advice on how to keep your repo clean, how to handle learning projects when switching roles or languages, and why you should never be afraid to show that you’re still learning.

Most recruiters won’t look at your portfolio, but when they do, you’ll want to be in the top 1%. If you do decide to build a portfolio, make sure it represents your best work.

Outline

00:00 Portfolio Value

The episode starts by discussing the real-world value of online portfolios. Most recruiters don’t look at them, but if the right person does, yours better be impressive. The host shares personal experiences with portfolios and offers perspective on how useful these tools can really be.

02:59 Stand Out Skills

We dig into what actually makes you, and your portfolio, stand out. Typical skills tested during interviews like defect writing, test cases, or strategies do not need to be shown off in a repo. Instead, focus on things that make you unique, such as personal opinions, in-depth case studies, or examples of thought leadership. This is often best done through blogs or articles, linked to your GitHub on your LinkedIn profile.

09:19 Project Types

A breakdown of project types:

• Learning Projects: Messy, experimental, used to learn new tools, languages, or skills. It’s okay if they’re not polished, but document what you learned.
• Personal Projects: Hobbies or interests that show personality and initiative. These don’t need to be perfect but should reflect genuine enthusiasm and curiosity.
• Portfolio Projects (Showcase Projects): Your best, most complete work. These should be polished, well-documented, and focused on the skills you’re trying to sell.

Making these distinctions helps recruiters judge your work appropriately. Always clarify the purpose in your README files.

12:27 Showcase Projects

Showcase projects are your sales pitch-they need to be as close to perfect as possible. The host shares advice on what to include in a GitHub repo:

• Use your profile README to introduce yourself professionally.
• Only include original repos, not forks, to keep things clean.
• Don’t commit IDE or compiled files; use .gitignore.
• Use static analysis tools and add unit tests to show you care about quality.
• Readme documentation should cover project intent, installation instructions, known limitations, and choices made during development.
• Whenever possible, leverage GitHub Actions to show your code works.

Recency and maintenance matter when looking for work. Keep your portfolio projects updated and relevant.

19:39 Promoting Yourself

Once you’ve created a portfolio project, get the word out. Add it to your LinkedIn profile, feature it as a project, and share articles about your work. Updating and promoting your portfolio regularly keeps it visible and demonstrates your ongoing growth.

21:44 Final Advice

Most professionals don’t bother with portfolios, a missed opportunity for many, but if you go the extra mile to create one you can stand out. Make sure it’s good, you don’t want to stand out because it’s bad. Even if recruiters don’t look at it most of the time, the right one will, and that’s when it really helps. Sharing your learning and even your mistakes is crucial for personal development, especially as you gain more experience.

Key Takeaways

• Most developers and testers don’t have a portfolio, so building one sets you apart when it’s high quality.
• A strong online portfolio acts as a sales tool, focusing on your best, most relevant skills for the job you want.
• Differentiate between learning projects, personal hobby repos, and polished, focused showcase repositories.
• Keeping documentation clear, and up-to-date is just as important as good code in your portfolio.
• Promoting your portfolio on platforms like LinkedIn and your blog can drive the right attention.

Quotes

“Should you have an online portfolio showcasing your skills and abilities and will that help you get a job? Well, I got an online portfolio, I’ve had one for years. But I also know that people don’t look at it.”

“If you’ve got a portfolio and it’s bad, it might put interviewers off. If you’ve got a learning project and you’ve been copying someone on YouTube… When people look at that, they’re gonna go, ‘This person doesn’t know what they’re doing.’”

“What you want to showcase is something that makes you different, a reason why someone would pick you up as opposed to someone else.”

“Portfolio project says, ‘I know how to do this, therefore judge me on the work.’ A learning project is, ‘I learned from these other people.’”

Join our Patreon from as little as $1 a month for early access to videos, ad-free videos, free e-books and courses, and lots of exclusive content.

Originally published at https://eviltester.com on December 24, 2025.

Welcome to Episode 23 of The Evil Tester Show. This episode of dives deep into the realities of tech recruitment, job search strategies, and career planning for Software Developer and Testers — with expert recruitment consultant Jack Cole from WEDOTech.uk — Whether you’re an experienced Test manager, expert Tester or just starting out, Jack’s decades of industry know-how will give you the tips and tricks you need to understand what works in today’s competitive market.

Are you trying to figure out how to break into the software testing job market or make your next big move?. In this packed hour-long conversation, we cover everything from market trends, LinkedIn networking, and the recruitment pipeline, to building a career roadmap and even the AI hype machine. Grab your notebook, settle in, and get ready for real insights you can use — plus a few stories from the trenches and actionable tips for every step of your job hunt.

We keep things down-to-earth and practical — no stuffy jargon, just the honest, practical advice that will help you stand out, get noticed, and map your next steps with confidence.

audio and subscribe links are here

Special Guest: Jack Cole

Find Jack @ wedotech.uk

Follow Jack on LinkedIn for jobs and job hunting tricks.

• Over 14 years of experience in software testing and development recruitment
• Organizer of industry leadership forums and community events
• Well-known on LinkedIn for candid recruitment advice and market insights
• Recently working on new career resources and a community project called “build your edge”

Key Takeaways

• Job market for testers is saturated and highly competitive; you need to stand out.
• Networking beats blind applications. Real conversations and tailored outreach are what make the difference.
• Treat your job search like a sales process: research prospects, follow up, and show clear business value.
• Testimonials and recommendations on LinkedIn really do help — don’t be shy about asking!
• Plan your career with a long-term view: the next job is only one stepping stone; know where you want to go in two or three moves.

Current State of the Software Testing Recruitment Market

• Market Imbalance and Challenges: The job market for software testers is highly saturated, with a significant disparity between job openings and candidates. Testers are often first impacted by redundancies during downturns and experience slower recovery in hiring.
• Competition and Differentiation: With many skilled professionals seeking roles, standing out requires candidates to showcase unique, niche, or highly technical skills, as well as experience solving specific business problems.

Application and Recruitment Process Insights

• Volume of Applicants and Recruiter Constraints: Recruiters and talent managers face overwhelming application volumes, leading to reliance on trusted networks and referrals rather than job board applicants. Initial and final applicants often get more attention than those in the middle.
• Being Proactive and Strategic: Candidates must do more than submit applications. They should connect directly with hiring stakeholders, use tailored messaging, and highlight relevant achievements and problem-solving experiences.

Effective Networking and LinkedIn Strategies

• Optimizing LinkedIn Presence: Candidates are encouraged to actively engage on LinkedIn, use the ‘Open to Work’ banner strategically, request testimonials, and maintain consistency between their CV and online profiles.
• Building Relationships: Networking is not just about asking for jobs but consistently reminding people of your expertise and availability. Following target companies and key individuals, and using features like LinkedIn’s notification bell, helps candidates stay informed and responsive.

Treating the Job Search as a Sales and Marketing Process

• Sales Mindset for Job Seekers: Successful candidates adopt sales strategies-prospecting potential employers, following leads, and crafting messages that focus on business value and solved challenges rather than just technical skills.
• Nurturing Connections: Building and leveraging past relationships with colleagues and recruiters can provide priority access to opportunities, similar to VIP entry compared to standing in the general job queue.

Working with Recruiters vs. Direct Applications

• Recruiter Partnerships: Engaged recruiters advocate for their candidates and can streamline processes, coach applicants, and accelerate decision-making if the fit is strong.
• Direct Company Outreach: When recruiter communication is lacking or unclear, candidates may consider bypassing recruiters, but if a recruiter is actively involved, direct outreach can sometimes be counterproductive.

Salary Trends and In-Demand Skills

• Market Salary Ranges: Salaries for Tester roles in the UK range from £55k-£140k depending on level and specialization, with London showing higher averages. Technical, niche, and SRE/DevOps-adjacent skills command a premium.
• Comparison with Development Roles: Tester salaries are relatively aligned with developer roles, especially as more technical and AI-adjacent skills become increasingly sought after.

Career Planning and Long-Term Strategy

• Forward Planning: Candidates are advised to plan two to three steps ahead in their careers, considering potential transitions into product management, leadership, SRE, security, or other specialized fields.
• Leveraging Transferable Skills: Test professionals possess broad exposure to software lifecycles, equipping them with unique skills highly valued in adjacent roles, particularly in product and leadership paths.

Interview Preparation and Process Trends

• Interview Structure: Interviews are progressively structured, starting with informal conversations to assess skills and chemistry, moving to technical/architecture tasks, and finally focusing on cultural fit and conflict management.
• Recruitment Influence: Recruiters can sometimes influence hiring processes, helping candidates by providing detailed context, interview preparation resources, and feedback to improve chances of success.

Impact of AI on QA Roles and Future Skills

• AI-Driven Demand: There is emerging demand for testers with experience in AI and data science, particularly for use cases such as chatbots and data-driven testing functions.
• Adapting Skills: The future job market requires a blend of traditional Test expertise with familiarity in AI, infrastructure-as-code, monitoring, and DevSecOps, depending on candidates’ long-term goals.

Community, Events, and Personal Branding

• Building Credibility: Engaging in community events, seeking and giving testimonials, and maintaining visibility help candidates bolster their professional brand and trustworthiness.

Final Recommendations

• Candidates should think beyond immediate job needs, network proactively, invest in their brand, and adopt a persistent, sales-oriented approach.

Episode Summary

In this episode, Alan sits down with Jack Cole — a recruitment expert who knows the software testing world inside and out — for a practical, straight-talking look at the realities of getting hired today. Jack breaks down the state of the UK (and wider) hiring market for testers and Software Development professionals: he’s open about just how crowded and tough things have gotten, especially with recent layoffs, market volatility, and a shift in what employers are looking for.

Jack doesn’t sugarcoat it: landing a job takes more than just hitting “apply” on LinkedIn. He walks us through why networking is more important than ever, and how to make sure your online presence works for you — including tips on using the “Open to Work” banner effectively, following companies and managers, and writing those all-important tailored messages.

A big part of the discussion focuses on treating your job search like a sales process — prospecting for leads (jobs and companies), pitching yourself with clear value (“here’s the business problem I solve”), and using testimonials for social proof. Jack also digs into how recruiters actually operate behind the scenes: how they filter candidates, the difference between recruiter and direct applications, and why getting in early or late can change your odds.

We also dig into salary expectations at different levels (with London/non-London comparisons), the increasing demand for niche and technical skill sets (like infrastructure as code, and AI), and how to think long-term about your career — planning two or three moves ahead, not just the next job.

Notable Quotes and Examples

“To stand out in today’s market, you’ve got to have something unique or special about you — something niche or super technical. Clients want people who’ve solved real problems in the world, not just ticked skills boxes.”

“I would actually say be one of the first or be one of the last to apply — otherwise you might just get lost in the middle. But even more, you have to do more now to get recognized and noticed.”

“Treat your job search as a sales process. You’re not just applying; you’re prospecting, researching, following up, showcasing value, and backing it with social proof.”

“If you just apply for jobs, you’re basically standing in a queue to get into a nightclub — and it’s probably already full. Make friends with the manager or the owner, and you get right in. That’s the power of networking.”

“Don’t just think about your next role — think about the next two or three roles you want, and plan your skills and connections that way.”

Take Action:

• “Standing out means more than having the skills. It’s about solving real business problems and showing you can do it.”
• “If you’re just applying for jobs, you’re in a long queue. It’s the relationships that get you in the door.”
• “Think beyond just the next gig — where do you want to be two, three moves from now?”
• “Don’t be shy — reach out for testimonials and make your value visible on LinkedIn.”
• “Most interviewers want to see outcomes, not just tech buzzwords. Prep your stories!”

Detailed Episode Breakdown

00:00 — Welcome & Intro

Alan introduces Jack Cole, giving listeners a quick summary of Jack’s long history in tech recruitment, community events, and active presence on LinkedIn.

02:03 — Market Overview

Jack immediately lays out the current reality: the QA/testing job market is crowded, with way more candidates than positions, especially at the leadership level. He shares hard data from LinkedIn Recruiter:

• ~14,000 active QA management profiles in the UK, 400k open to work
• Only 1,400 jobs posted for those roles This creates a real “buyer’s market” where employers can pick and choose. Redundancies have hit hard, and experienced people are often the first out and slowest to get rehired.

10:33 — Networking Secrets

Forget just applying to jobs — Jack emphasizes the importance of maintaining visibility and relationships. He covers:

• Effectiveness of LinkedIn “Open to Work” banner
• How recruiters get lost in a flood of CVs and rely on their trusted network
• The importance of being top-of-mind when recruiters get a relevant position
• Why internal recruiters are swamped and why timing your applications matters

16:00 — Sales Approach

Jack provides actionable advice:

• Treat each job application like a sales pitch:
• Research the company, identify managers, and connect thoughtfully
• Message those contacts with specific examples of relevant problems you’ve solved
• Always focus on business outcomes, not just tech skills
• Use LinkedIn testimonials as “social proof”
• Don’t be afraid to ask directly for recommendations
• Keep “outreach” messages to under 120 words for best response

32:01 — Salary Insights

Jack breaks down salary expectations for QAs:

• UK average Tester: £55k (maxing out ~£70k for mid-level, up to £80k for senior, £85-£140k for highly specialized leads)
• London rates are higher: up to £100k for seniors, niche skills up to £140k
• Niche skills in demand: automation, infrastructure as code, operational acceptance, non-functional testing He also compares Test with Dev salaries and explains how recruiters educate clients on setting realistic budgets.

38:49 — Career Progression

Jack encourages listeners to plan beyond the immediate job hunt. Advice includes:

• Map out 2–3 moves ahead: do you want to become a product manager, security engineer, or Test leader?
• Connect with people who have already made those transitions; ask them what worked
• Capitalize on breadth as a Test professional — more options open than you think He notes a trend of Testers moving into product roles, thanks to their cross-team communication and customer focus.

41:20 — Interview Strategies

When it comes to interviews:

• Always treat them as a two-way conversation — it’s not just you being evaluated
• Be ready to talk not just about skills, but about specific outcomes and problem-solving
• For technical roles, expect more system architecture, real-life scenario questions, code reviews, or pairing exercises, not just coding tests
• Do research on interviewers and company context Jack emphasizes: preparedness and rapport-building really matter.

47:47 — AI Impact

AI is an emerging focus area:

• AI Test roles are already appearing (e.g., chatbot testing, LLM understanding)
• Companies are investing, but the field is new; few people have direct experience
• Jack suggests: if you want to ride the AI wave, start experimenting and learning — but choice should fit your desired career direction (AI, SRE, DevSecOps, etc.)

52:06 — Jack’s Resources

Jack teases new resources:

• “Build Your Edge” — a new community for learning, events, career coaching, and CV support
• More actionable resources and maybe a whitepaper on career transitions in QA coming soon
• Advice: Keep following for updates, and use company/leader “bell” notifications on LinkedIn

Originally published at https://eviltester.com on December 23, 2025.

Rather than retrospect on the year in general. This podcasts looks at how I’ve been approaching the learning of AI. Where the industry has gone wrong and what to look forward to with AI.

audio links and subscribe here

AI Optimism or Pessimism

I sat down to think what am I optimistic about, and what am I pessimistic about with the Software Development industry’s use of AI.

I see a lot of positive benefits for AI use in the Software Development Process.

There are many risks involved with adding AI into products and we should rightly be concerned about those and try to identify how to test AI enabled applications properly.

Many of the statements concerning how Software Testing has to evolve concern me. e.g. more technical skills, more focus on risk, point out flaws in process, identify architecture risk. I thought Software Testing already did this. So perhaps one good thing from AI is an evaluation of what part Software Testing plays in the Development Process and what are the fundamental skills and knowledge we need in order to Test software effectively.

AI Productivity

I’m incredibly optimistic about what I can personally do with AI. It’s massively increased what I can get done, especially in programming, content creation, and marketing.

But companies are cutting jobs and blaming AI. But companies have always looked for scapegoats when trimming staff AI is just the latest excuse.

If leadership truly saw their people as the reason for their growth, they’d use AI to amplify productivity and expand the team’s capabilities, not use it as a reason to shrink down.

“If companies viewed the staff as a core engine for growth, they wouldn’t want to get rid of people when AI comes in. They’d want to use AI to make the staff more effective and grow the company. Clearly they don’t.”

Many management teams see staff as a cost center, not as an engine for growth. Sales gets viewed as the engine of growth and is often paid on commission, so it seems cheap. The product is a ‘thing to be sold’. And management… are always essential.

Some organizations don’t think “how do we help our people do more?” They take the position: “how can we do the same (or less) with fewer people?”

AI is the scapegoat. Not the cause.

Short-Term Pessimism

AI is overhyped. Leadership decisions are being made based on hope, not the technology’s actual capabilities.

In the short term, I’m a bit pessimistic, not about AI itself, but about how people (management) are choosing to react to it. Too many seem to want to use AI to restrict and constrain, instead of to grow.

But our work is already changing. New tools exist. We can’t afford to be complacent. We need to experiment and figure out where AI makes sense.

How Should We Use AI?

Not every product needs AI enablement and features. Too much risk, particularly if you do not know how to test the results.

• I do NOT use AI for direct human-to-human communication.
• If you get an email from me, I wrote it. If I get one from you, I read it, not an AI.
• I would not use AI for company procedures or performance reviews.
• Adding AI to these human, recognition-based workflows is a mistake.
• I won’t use AI as a buffer between people.
• Don’t use AI to hide. Engage.

I do use AI to summarize public podcasts, blog posts, etc, Stuff I choose to consume, but not personal, direct communication.

Management is about people, and putting an AI buffer between you and your team leads to bad communication and alienation.

“Your boss should already know what you’re doing. They should be helping you explain the value that you’ve added. This is a human process of recognizing value. It’s not something that we add AI into the middle of.”

How I learned AI

When I started playing with AI, I created a podcast summarizer. My process for learning anything new is to find a project and use the new technology to build something.

I wanted to consume more podcasts but couldn’t keep up. So, I wrote a tool using:

• Hugging Face libraries
• Whisper AI for transcription
• Different LLM models for summarization
• Prompt engineering
• Ollama for running models locally
• OpenRouter for connecting to cloud-based models

I did everything locally as much as possible, partly to learn the limits before relying on cloud or paid models. Only after learning the limits did I see real value in some of the paid services.

Don’t believe in ‘magic’ from big models until you know what smaller, free models can actually do.

Step By Step Adoption

Step 1: Chat Interfaces

At first, I used AI the same way most people, from a chat interface, like ChatGPT or Claude. It worked but had limits. I wanted something embedded into my IDE.

Step 2: Embedding into the IDE

Now, I use a plugin called Continue, which works in both IntelliJ and Visual Studio Code.

• Initially connected to Ollama (local code models)
• Later I Switched to OpenRouter for bigger cloud models

So now, I can:

• Jump into a chat interface via my IDE
• Still use chatbots like Claude/ChatGPT for generic things
• Never look at Stack Overflow anymore, everything’s in the IDE chat

Step 3: Agentic Coding

I started using agentic tools, like OpenCode, on the command line. They can scan my whole codebase, not just what’s open in the IDE.

I use it to:

• Create a page object model for a URL
• Write a test for that model
• Generate K6 Scripts
• Create swagger files
• Write scripts
• Write API Abstractions
• … so much

Now instead of:

1. Writing the test
2. Building abstractions and models manually
3. Iteratively improving

I let the AI generate code, then I come in and change or adapt what I need.

“I’m using it as a tool to help me create things faster. Then I come in and use my knowledge and experience to review the code or expand it.”

Step 4: OpenSpec for Better Requirements

OpenSpec helps generate and maintain evolving and up to date requirements documentation.

• Specs are continuously updated as requirements change
• OpenCode has documentation to work from, in addition to my prompts

When I use OpenSpec, I’m letting the Agentic Coding Assistant write even more code, than when I use it iteratively at the CLI.

This has helped me:

• write a new application faster
• Keep requirements and documentation in sync
• Automate coverage incrementally

I’m not sure how well it would scale to a full team, but it works well for a solo dev.

But…

• AI sometimes makes strange design choices
• If you don’t review and refactor, your codebase can fall apart fast
• Human review is still necessary

AI saves time, but still requires fundamental core skills to evaluate their output and fix them when they go wrong.

I use AI for programming all the time, but not for actual testing.

Agentic AI in Testing

I’m beginning to look into Agentic AI tools specifically designed for testing.

I’m currently experimenting with AQE Fleet from Dragan Spiridonov

• https://github.com/proffesor-for-testing/agentic-qe
• https://forge-quality.dev/

Pros and Cons

Right now, I am optimistic about AI empowering individual development team members. It makes us faster, and the tools are getting better.

But I am pessimistic about the way companies choose to adapt to AI. Focus on headcount reduction instead of increased effectiveness and growth.

What is Testing?

“I keep seeing posts that testers need to evolve into quality engineers and risk analysts and customer experience advocates because we’ll be reviewing the systems for risk more than we’re and testing them… but I thought we already did those things.”

People keep saying testers now need to be risk analysts, customer experience advocates, technical experts, etc. But that’s what good testers have always done. It almost feels like AI is forcing companies to rediscover what software testing really is, and always has been.

Join our Patreon from as little as $1 a month for early access to videos, ad-free videos, free e-books and courses, and lots of exclusive content.

Originally published at https://dev.to on December 22, 2025.

TLDR; Record and Playback is a notoriously bad way to automate. But what if we record, then have AI write code from the recording? Answer: it is a little better.

I performed an experiment to try tand demonstrate how to replicate the AI-powered test automation features commonly found in commercial SaaS testing tools using open source solutions. The process involved recording user interactions in an application with Chrome DevTools Recorder, exporting the recording as JSON, and then using an AI tool to convert the JSON into JUnit tests with WebDriver and page objects.

Overview Video

Watch on YouTube

In this video I:

• record some web actions using Chrome Dev Tools recorder
• fail to play them back because record and playback is notoriously bad
• feed the recorded JSON file into OpenCode with a prompt
• check the results of the AI process writing Automated Execution using WebDriver and Java

We do get working automated coverage out the back. I would need to edit it a little, and if I wanted to do this longer term I would need to amend the prompt to meet my coding preferences.

process

I used the triangle application on Test Pages:

• https://testpages.eviltester.com/apps/triangle/

Test Pages has a deliberately hard to automate side bar. The easy automation is through the top level menu.

But I recorded the Triangle application by:

• clicking on the side menu bar to open the Triangle application
• and the instructions
• reading the instructions
• clicking back on the triangle application
• interacting with the app to generate invalid and valid triangles

When I tried to playback the recording in the Chrome Dev Tools, it failed.

Most of us experience failure with record and playback tooling.

AI tooling

I have OpenCode, configured to use Chrome Dev Tools MCP, and working with Kat Code Pro Free in OpenRouter, but could just as easily have used any of the free coding LLMS on Open Code Zen or with Ollama locally.

• https://opencode.ai/
• https://github.com/ChromeDevTools/chrome-devtools-mcp/
• https://openrouter.ai/
• https://openrouter.ai/kwaipilot/kat-coder-pro:free

I used a basic prompt

“given the json attached, convert the json into a set of junit tests using webdriver and page objects to recreate the flows recorded by the user. The json file is a recording made in Chrome Dev Tools recorder”.

The process took about 5 minutes to run.

• It opened the web page a few times,
• created some Page Objects.
• Found that the menu interaction was hard.
• Decided to jump straight to the triangle page and automate that way
• It did replicate the functional interaction included in the recording
• it did generate assertions for the results
• I could have used the test code generate with some small amendments.

Lessons Learned

SAAS tools which take this approach will use Agents, which are essentially longer prompts. I should have added into the prompt more conditions:

“given the json attached, convert the json into a set of junit tests using webdriver and page objects to recreate the flows recorded by the user. The json file is a recording made in Chrome Dev Tools recorder. In the test code, do not write any findElement code, always abstract the location and interaction into a page object. Use existing page objects where possible and try to re-use existing methods. Make sure that any page object methods you use are used in a test so we have coverage of the interactions.”

Over time, if I wanted to continue to automate using this process, I would refine the prompt further until I got the results I wanted, and possibly encode this prompt as an agent.

Would I do this?

Unlikely that I would do this in the real world.

This seems like the replication of a record and playback approach.

I do create Page Objects and have created initial execution coverage using AI.

I have some empathy for AI tooling scanning applications and building flows through the application to automatically interact and generate a wide variety of data, to increase the scope of coverage across a few simple flows. I can see potential in augmenting a team to allow the team to go off and do more detailed stuff. Provided the AI tooling doesn’t get noisy and keep distracting the team.

For any functionality as simple as I experimented with here, I hope that when people evaluate SAAS tools, that they also compare the paid tools to the capabilities of Open Source tools.

Nothing wrong with paid tools. They are incredibly valuable when they make hard things simple, and add value when they make hard things easier.

But, when the activity is simple and the functionality differences between Open Source and paid are small, I hope companies consider “what if we upskilled our staff a little such that we could use the Open Source tools”.

Join our Patreon from as little as $1 a month for early access to videos, ad-free videos, free e-books and courses, and lots of exclusive content.

Originally published at https://eviltester.com on December 19, 2025.

Welcome to Episode 22 of The Evil Tester Show, where we’re diving into the fascinating world of practice with the renowned James Lyndsay. In this conversation, your host Alan Richardson chats with James about the essence of practice in software testing, exploring how exercises and real-world scenarios can enrich our skills. James shares insights on his weekly online practice sessions and the interactive Test Lab concept, offering a dynamic playground for testers.

This was released to Patreon supporters early, with transcript and no ads.

Discover how practice blends with rehearsal and learning, and delve into the intriguing intersection of testing and development. With firsthand experiences in software experiments, fencing, and scientific investigation, James and Alan discuss the art of modeling and exploring software systems. Whether you’re refining your testing techniques or embracing new perspectives with AI, this episode offers a wealth of wisdom for testers at all levels.

Join us as we learn, laugh, and leap into the realm of testing practice. Tune in, engage with new ideas, and maybe even find inspiration for your own practice sessions. Don’t forget to check out James’s resources at workroom-productions.com for more testing challenges and exercises.

Special Guest: James Lyndsay

Find James @ workroom-productions.com

Follow James on LinkedIn to learn about up-coming practice sessions.

• About James
• Blog Posts and Articles
• exercises
• puzzles

Key Takeaways

• The Importance of Practice: The episode focuses on the theme of practicing, emphasizing how James Lyndsay uses practice exercises, tools, and software to improve testing skills.
• Different Testing Styles: Alan and James discuss how different testing styles can lead to varied approaches in testing exercises, improving both technique and understanding.
• Online Practice Sessions: James Lyndsay mentions his online practice sessions, which are focused on interactive exercises rather than lectures, aiming to foster learning through play and discussion.
• Test Lab Concept: The Test Lab is introduced as a space at conferences where people can test real software with freedom and creativity, highlighting the benefits of hands-on collaborative learning.
• Solo Practice vs. Group Rehearsal: James distinguishes between solo practice, which is more about experimentation, and group rehearsal, which focuses on coordination and collective practice.
• Exploratory Testing: The episode highlights the value of exploratory testing, modeling, and understanding how systems work through testing rather than following predefined rules or scripts.
• Role of AI in Practice: James describes an experience where an AI helped him understand a programming issue, suggesting that AI could change how we approach practice and learning.
• Developing Models: Both speakers emphasize the need to build models while testing, to direct further exploration and identify gaps in understanding.
• Interactive Learning: They also discuss how learning with others in a test lab can be more beneficial than isolated practice, as it offers diverse perspectives and shared learning experiences.
• Mindset in Testing: The importance of having an open mindset that welcomes failure as a learning tool and not being constrained by the need to know what software is ‘for’ before engaging with it is reinforced.

Support our conteent. Join our Patreon for only $1

Originally published at https://dev.to on December 18, 2025.

This episode explores how to navigate context in testing environments, adapt our approaches, and effectively challenge and evolve systems. Discover the importance of context-driven testing in software development, exploring models, adaptability, and useful practices.

This was released to Patreon supporters early, with full transcript and no ads.

Episode

Watch:

Listen & Subscribe:

Show notes page with listen links

Notes

These were the notes I used to create the podcast from. They are not a transcript.

What is Context?

There is no such thing as Context Driven Testing but it is a useful phrase when thinking about our approach to testing, so in that sense you could call your approach Contextually Driven.

There is no such ‘thing’ as context, it is a model.

Context is a relationship between our model of the real world and our mental model of what is going on.

Context changes over time.

Our being part of the context, changes the context.

What is context driven testing?

Context-Driven-Testing.com

In the olden days, the world of testing was trying to show that not all testing is the same, not all testing can be standardised, and standardised testing holds people and the craft of testing back.

https://context-driven-testing.com/

The website lists the creation of the concept as initially developed by

James Bach, Brian Marick, Bret Pettichord, Cem Kaner

I used to think about this in terms of Systems within Systems — System of Business, inside is a System of Development, inside a System of Programming and Testing and Analysis and Design, and then projects and programs and individuals have their own System.

Context Driven Testing seemed like a fine set of words to describe this.

I didn’t sign up to the school or manifesto I didn’t like the wording of the 7 basic principles, and I didn’t think it needed principles and why are their only 7?

This is an old site and an old list and everyone who was involved initially has moved on and created their own ways of talking about it.

But the reason for mentioning it is… what site current ranks top for the phrase “Context Driven Testing”? Yes, context-driven-testing.com

• What are we doing?
• Is it helping more than hindering?
• What does better look like?
• How can we do it better?
• Span systems — People systems, Role systems, Project Systems, computer systems etc.
• I do not impose methodologies or processes and call it testing.

Context Driven Testing takes into account: stakeholders, process, communication, individuals, aims, beliefs, mandates, etc.

Ideology

• context, avoiding ideology
• explore everything can work

If your beliefs about something can be predicted then you might be in the grip of an ideology.

Context requires beliefs that open up options rather than shutting them down.

As an exercise you can use Chat GPT to provide answers about a topic and if you find yourself agreeing with it all then you’re really agreeing with an average position, so think of a situation that breaks the advice or description in chat gpt. Identify a context which exposes the ‘wisdom of the crowds average’ as an ideology.

We want every individual to be unique.

Context is the same, applied to environments, projects, companies. Context is about allowing environments to be unique and to change and to grow.

Introducing ideology into an environment can restrict the context.

Ideologies:

• tool X is better than tool Y
• we need more automated execution at the Unit level than the UI level
• Unit tests don’t interact with anything else
• In a Unit test every interaction is mocked
• Page Objects are bad, use Screenplay
• Screenplay is bad, use Page Objects
• etc.

Important to note that ‘everything can work’

Agile Testing

There is no such thing as Agile Testing, there is testing within an Agile Context where the Agile Context might be different for each team and project within the company.

Could, Should, Would

Really what we can look at is:

• capability
• context
• contextual fit

Capability comes down to “could” you do it, or “can” you do it.

Context covers “should” you do it. What’s the impact?

Contextual fit covers “would” you do it. Will you commit to doing it? How will you do it? What changes do you have to make?

e.g. Forcing/Teaching testers to code so that automating our systems speeds up our release process

I think people read the statement in terms of capability. i.e. is it possible for people to learn to program if they are testers? If they do learn to program can they become good at it? The statement suggests… probably not. But the answer to both of the questions in my experience is Yes.

But it depends on the context and contextual fit.

If they are being ‘forced’ then you need to have really good training, really good experiential training, really safe ways to learn, and really good ‘this is what good looks like’.

People need to be motivated to learn, and if they are being ‘forced’ then they might not.

If the context that is forcing people doesn’t know what ‘good’ is then they might stop at the point where they have given people the most basic skills and that can translate into ‘bad’.

Sometimes the ‘forced’ might be — If I don’t learn this then I won’t get a job. So the context is survival. This can be quite motivating. But… if you don’t have the right training, and don’t have a safe place or time to learn then you might not get over the initial learning hump and might not take the time to get good.

Contextual fit also covers is the change happening for the right reasons? Because people might be trying to achieve unrealistic goals with the wrong approach.

Adaptation

I’d rather adopting a more scientific approach of:

• Observing what we do,
• Modeling (rather than standardising) to understand,
• Investigating our inefficiencies (often caused by lack of understanding, or forgetting because we don’t do the same thing the same way multiple times every day)
• Experimenting (to improve against our existing model and observed approaches)
• Letting and experiment settle so we are evaluating it after developing competency in it, and not trying to perform too many experiments at once
• Repeating to evolve based on the needs of the people and the environment

We have to be prepared to challenge the approaches and even interpretation of results.

If someone claims something is a success, then that can often prevent learning from all the mistakes that happened to create that success.

Contextual error patterns

Contextually, looking at errors that have happened on applications in your own environment, generalising the error, then looking for that error in other applications in your environment, is often an easier way to find issues that are likely in your own environment.

From an observation work back through the application and discover the design issue that triggered the problem.

This now becomes a higher likelihood error pattern in my model of the legacy apps and is something I look for more, or treat as a higher risk.

Follow on

Have a look at James bach’s site for the Context Driven Methodology post

https://www.satisfice.com/blog/archives/74

There is no such ‘thing’ as context, it is a model.

Context is a relationship between our model of the real world and our mental model of what is going on.

Context changes over time.

Our being part of the context, changes the context.

There is no one true way. We make decisions and we need to be able to justify those decisions

Episode Summary

In Episode 21, we explore the boundaries of “context-driven testing,” considering it less as a fixed methodology and more as a model to understand dynamic testing environments. Our journey begins with a look at the historical context of the term and how it was coined. We unravel the layered systems approach, examining how interlocking systems within a business shape the context and how testers must adapt to these systems.

Moving into the heart of testing, the discussion opens up on changing contexts and the resulting dynamic processes. As contexts within testing environments are ever-evolving, testers are encouraged to think adaptively and engage with the environment in a way that embraces its complexity and unpredictability.

We also consider the importance of capabilities-what can be done, what should be done, and how context influences these decisions. We stress the importance of the ability to adapt, how we challenge assumptions and interactions within the project. Additionally the importance of deliberate error detection and close with thoughts on models and reality, stressing the necessity of adaptability and openness to change.

Key Takeaways

• Context-driven testing isn’t fixed; it’s an adaptable approach to evolving testing environments.
• Understanding the interplay between systems and context is essential for effective testing.
• Testing isn’t about following best practices but evaluating what’s right for the current context.
• Constant adaptation to change-both personal and environmental-is crucial.
• Identifying and learning from error patterns in context-specific situations enhances problem detection.
• Testing is about decision making and persona responsibility.
• We are part of the context.
• Our actions shape the context.
• The context can push back.

Quotes and Examples

“There’s no such thing as context; we create models that interact with real-world situations.” [01:10]

“Projects unfold unpredictably, so our testing approach must comfortably handle these variations.” [17:11]

“It’s not just the practice; it’s how people view the practice and how the results are perceived.” [07:04]

“Our beliefs should expand options and facilitate experimentation, not limit them.” [14:39]

“Deliberate error detection in context can uncover unique issues on your project.” [22:48]

• “Testing isn’t about best practices; it’s about what’s suitable for the context.”
• “Adaptation is key as the environment and our understanding continuously evolve.”
• “Context is an ongoing relationship between personal and external models.”
• “Error patterns particular to a context can illuminate unseen issues.”
• “Never let ideologies limit your testing explorations and experiments.”

Discussion Questions and Exercises

• Defining Context: How do you define ‘context’ in relation to testing?
• Context Changes: How does being part of a project team influence the context? Can you provide an example from your own experience where team dynamics changed the context of a project?
• How have you deliberately tried to change the context?
• How has your presence on a project changed the context?
• Context-Driven Testing Principles: Have you read the context-driven principles? Do you agree with them? Do you agree with the wording of them? How would you state them in your own words?
• Systems within Systems: The podcast considering testing in terms of “systems within systems.” Do you view systems like this? How could this perspective influence your approach to testing?
• Adaptive Approaches: What does an “adaptive and evolving approach” mean with respect to testing? How does this differ from traditional or process-driven methods?
• Avoiding Ideology: Is avoiding rigid ideology in testing important? How can one ensure their beliefs about testing remain flexible and context-aware?
• Role of Experimentation: The podcast describes the necessity of experimentation in context-driven testing. Can you think of a situation where experimenting led to a better understanding or improvement in your testing approach?
• Agile and Context Driven: How does context-driven testing align with or differ from Agile testing? Have you encountered any challenges in melding these approaches?
• Capability, Context, and Fit: What are the differences between capability, context, and contextual fit as described by Alan? Have you analysed your approach in these terms? Contextual Error Patterns: How can identifying contextual error patterns improve your testing process? Can you identify any recurring errors in your current or past projects that might benefit from this approach?
• Have you tried the chat GPT exercise? Ask Chat GPT to describe part of your testing approach or testing definitions. Do you agree with it? Find the differences and gaps between your description and the Chat GPT description.

Originally published at https://eviltester.com/show on Jan 4th, 2025.

The answers given during an AMA session held on Discord on the 11th of December 2025, following a live LinkedIn video stream. The session focused on “Mastering Automatability for Test Automation”. The main theme is the concept of Automatability, which I view as the ability to automate, this personal skill is more critical than reliance on specific tools. The discussion covers various topics, including how to separate automation problems from application design issues, dealing with slow UIs and non-automation friendly third-party widgets, evaluating automation readiness, and addressing common architectural failings related to large-scale UI automation.

Audio versions of the above video can be found here.

This was a Browserstack hosted event. The initial Q&A session started on Linkedin with a conversation between Alan Richardson and David Burns.

Recording for initial session can be found here:

• LinkedIn Event

The session then moved to Discord. The Browserstack discord has many AMA and interviews so is worth signing up to have a look at.

• the Browserstack Knowledge-hub forum on Discord
• Alan Richardson AMA

Other AMA sessions include:

• Jason Huggins
• Gil Zilberfeld
• Testers’s Day Panel with Vikas Mittal, Jenna Charlton and Manish Saini

Join the BrowserStack Community on Discord and discover many more sessions, videos and conversations.

• https://www.browserstack.com/community

Q&A Session Questions and Summaries

I’ve listed the questions and summary answers. Full answers can be found in the podcast, audio or video, or on the Discord AMA chat.

Q&A 1: Understanding Automatability for a First Automation Framework

Question: If I’m building my first test automation framework, what’s the one thing about automatability I should understand?

Summary of Answer:

The most important thing to understand is that automatability refers to your ability to automate. By having a strong ability to automate, you become less dependent on specific tools, making it easier to create workarounds and choose from multiple tools. Developing experience in how to automate allows you to succeed more often, and means you are not reliant on a tool interacting with your system, which makes workarounds harder. Automating is fundamentally about your understanding of what and how to automate, and practicing the application of that ability.

Q&A 2: Separating Automation Problems from Application Design Problems

Question: How do you separate automation problems from application design problems?

Summary of Answer:

If an issue causes problems when you are automating, I would call it an automation problem. While this problem might be triggered by an application design problem (such as a state-based system that is hard to track, or features that are harder to automate), the issue itself remains distinct. If the team cannot change the application design, they must figure out how to automate the application as it is. This might involve absorbing the issue, figuring out how to automate it at a different level (not end-to-end), or handling it through testing processes using observability tools like DataDog.

Q&A 3: Slow UIs and Testability/Automatability

Question: When dealing with slow UIs is the slowness a testability issue an automatability issue or both?

Summary of Answer:

Slowness is likely both, and more, because it is also a usability/user experience issue. If the slow UI impacts the user experience, it is more likely to be addressed than if it only impacts testing or automation. In cybernetics terms, testers or automators must possess the “requisite variety” to handle the variety (slowness) in the system being tested, which means knowing how to synchronize or potentially cleaning the environment to improve speed. The focus would be on the impact of the slowness, rather than the slowness itself, and whether the team or its tools can absorb the variation in response times.

Question: How do you handle third party widgets like payment gateways that are inherently not automation friendly?

Summary of Answer:

If a third-party widget is “not automation friendly” for one tactic (e.g., UI automation), it might become easier to automate by adopting different tactics, such as issuing HTTP requests using cookies collected from the UI. Teams may not need to automate the full flow of the widget, but instead focus on ensuring the widget is wired up correctly within their own application. This can involve only testing partway through the flow, or using a mock or stub in the environment so that the full widget flow doesn’t need to be tested constantly.

Q&A 5: Evaluating Automation Readiness and Consultancy Frameworks

Question: How do you evaluate an application’s automation readiness during consulting? Do you follow a framework?

Summary of Answer:

I do not use a formal consulting frameworks. The closest methodology used is the meta model from Neuro-Linguistic Programming, which involves asking questions to build a model of the client’s environment and processes, comparing it to the reality they face.

Regarding automation readiness, an application is considered ready to automate “as soon as someone wants to automate it”. Readiness is judged by whether the client is prepared to commit to whatever it takes to automate the application at that specific point in time to achieve their desired outcomes, regardless of the application’s current state.

Q&A 6: Architectural Patterns Failing in Large-Scale UI Automation

Question: What architectural patterns do you see repeatedly failing when it comes to large scale UI automation?

Summary of Answer:

Recurring issues often stem from the team lacking the ability to automate and consequently blaming the tool for problems, rather than creating necessary workarounds. Common process anti-patterns, not strictly architectural patterns, include deploying differently into test and production environments (not using the same install process).

A major failure point is test data maintenance, especially trying to use production data or any data that the team does not control. Automating against specific data conditions without control over that data causes random test failures. This can be worked around by hardcoding tests against data conditions instead of specific data, and dynamically selecting the required data during execution.

Q&A 7: Prioritizing Testability and Automation in Sprint Planning

Question: If testability improves debugging and automation improves scale, how do we prioritize them during sprint planning?

Summary of Answer:

Prioritization can be based on what the team wants to achieve (the outcomes) by the end of the sprint, specifically focusing on the expected coverage from testing and automation. It is beneficial to plan for features that need extensive testing to be delivered early in the sprint. Ideally, testing and automating occur in parallel, and teams automate at lower levels (like unit level) to reduce the necessary coverage at the higher UI level. Issues often arise when teams are divided into isolated roles, creating process problems that hinder effective interaction and prioritization.

Q&A 8: Playwright and the Illusion of Reduced Automatability Design Needs

Question: Do modern frameworks like Playwright reduce the need for high automatability design or is that an illusion?

Summary of Answer:

It is an illusion. Frameworks like Playwright are designed to absorb application variability through features like retry mechanisms (for synchronization) and locator strategies (like visible text), which reduces the need for constant notification when minor changes occur. This absorption capability makes Playwright effective for agent-based automation where the goal is checking an end-to-end path and the final result.

However, this absorption can hide issues that a team might want exposed. Even when using Playwright, developers must still understand how to automate and structure their code using abstraction layers (like page objects, domain objects) to ensure long-term maintainability and efficiency.

Q&A 9: Explaining Automatability as an Investment to Leadership

Question: How do I explain to leadership that improving automatability is an investment not a delay?

Summary of Answer:

The explanation depends on what is being improved. If improving automatability means increasing the team’s ability to automate, it can be presented as an investment in staff training. If it involves adding technical aids (like IDs in the UI), leadership might perceive it as a delay because they may not value UI execution coverage or may already be confident in unit-level automation. To convince leadership, the team could demonstrate the return on investment by showing the alternative world. This involves comparing the current reality to a scenario where improved automatability allows the team to do beneficial things they otherwise couldn’t, thereby highlighting the value gained.

Q&A 10: AI Agents Dealing with Dynamic Elements

Question: We are exploring AI agents for our teams and I want to know how does the AI agent deal with dynamic elements like rotating banner banners, third party widgets or AB tests?

Summary of Answer:

How the agent deals with dynamic elements depends on how it works (e.g., building high-level BDD scripts or generating code). Agents often operate on first principles. If an agent uses a BDD approach, it works from a runtime specification and handles dynamic elements because it works from scratch for each execution, constantly aiming to fulfill the objective. For example, if an unexpected pop-up appears, the agent clears it and continues.

If the agent writes code, it uses what is often called “autohealing”. This process automatically amends the script based on the current application state, prioritizing the achievement of the objective regardless of whether the change is “right”.

Q&A 11: Early Signals of Flaky Features

Question: What early signals tell you that a feature will become flaky once automated?

Summary of Answer:

Early signals involve understanding the synchronization points of the page. A feature is likely to be flaky if the application is populated or amended over time by JavaScript and the automation tool is not synchronizing properly on the DOM buildup. If a page is being constantly updated in the background without clear visual indicators (like spinners, which are easy to synchronize on), flakiness is more likely.

The signal is the update process itself, particularly when it is non-deterministic (e.g., how totals are updated in a shopping cart). The core question is whether synchronization is required to prevent flakiness, and if it is difficult to synchronize, that is a strong signal. If necessary, automatability might be enhanced by adding an extra flag to the DOM to signal when the update is complete.

Q&A 12: Layers to Focus on in Microservices for Automatability

Question: In microservices setups which layers should teams focus on first to increase overall automatability?

Summary of Answer:

The foundational layer to focus on is the human understanding of the architecture and the requirements for automation. In microservices specifically, teams would typically focus on the interface layer and their ability to automate it while keeping the interface standard.

If microservices are communicating via HTTP messages compliant with a version standard, automation can be relatively easy. If interfaces are internal and change randomly, issues may arise, requiring attention to managing event-based queues if applicable. Strategies include using versioned interfaces or having a process to update automated coverage and abstraction layers when microservice interfaces change. It is crucial to avoid replicating interface objects (like payload objects) directly into test code, as this can prevent tests from spotting issues when new or removed fields occur in the application interface.

Q&A 13: Budget-Limited Automatability Fixes for Fastest ROI

Question: For a team with limited budget which problems around automatability should we fix first to get the fastest return on investment?

Summary of Answer:

The fastest return on investment comes from enhancing the team’s ability to automate. This improvement allows teams to develop workarounds, find alternative solutions, and identify when to use different techniques. It is not about purchasing multiple or expensive tools. Instead, investment could be placed in training, practicing, exploring the capabilities of existing tools, and eliminating fundamental issues like test flakiness by fixing the root causes.

Q&A 14: Collaborating Earlier to Avoid Automatability Rework

Question: How can developers and testers collaborate earlier to avoid expensive rework on automatability issues?

Summary of Answer:

Collaboration is achieved by removing the barriers that cause people to be isolated into silos, such as separate programming, testing, or test automation teams. The core issue is often that a “development team” is defined only as a programming team, instead of encompassing responsibility for design, programming, product suitability, testing, and production.

Practical steps include involving the programming team in the automated execution maintenance. When programmers contribute, they often ensure data IDs are present, which removes many hard-to-automate issues typically found during end-to-end testing. Sharing the responsibility for maintenance ensures people understand and resolve related issues earlier.

Q&A 15: Automatability in Continuous Delivery and Trunk-Based Development

Question: How should teams think about automatability when shifting towards continuous delivery and trunk-based development?

Summary of Answer:

This environment requires high automated execution coverage that runs quickly, often with features being merged multiple times a day. Automatability is achieved by ensuring the person or pair responsible for creating a feature is also responsible for adding automated execution coverage (unit tests). These tests demonstrate that the feature has been tested and provide future checks against accidental changes impacting the functionality.

Teams could focus on structuring unit tests at the domain level (e.g., focusing on users or orders) rather than strictly class level. This approach results in a degree of internal end-to-end flow tests without needing extensive external system testing. Furthermore, application architecture can be designed so that interfaces (like HTTP interfaces) can be tested primarily at the domain level, reducing the need for numerous actual HTTP calls.

Originally published at https://www.eviltester.com.