From Spreadsheets to Checklists for Manual Testing

Delete the Sheet

Customers are always awesome at finding ways to use a product for things you don’t initially envisage. It’s nice when we discover that a problem that a customer is looking to solve with our platform is one we have too!

Recently a customer moved their large manual test case suite into our platform. Before they’d used a combination of Google Sheets and Forms to try and wrangle external testers to perform weekly tasks against the latest builds.

They were having some real problems making this scale. Quite often the labor in managing the spreadsheets and coordinating with the external parties was taking longer than the test schedule itself.

After a successful migration into the system they reduced their weekly setup burden to minutes. Clever Share allowed them to get the individual tests out to the third parties with little friction.

This made us take a step back and look at our own manual testing procedures and when we thought about it we realized we doing this to. We’d let some “one time” spreadsheets become difficult to manage before realising there was a better way close at hand.

So always looking to eat more of “our own dog food” we thought about how using Clever Checklist could make this better for our testing.

We love spreadsheets. But we love to delete spreadsheets more!

So why manually test at all?

You still manually test!? Shouldn’t we be automating all the things?

Every company doing automated testing still does some amount of manual testing. We are not ready and won’t ever be ready to replace all the humans.

An industry survey revealed 75 percent of all functional testing is still performed manually.

You shouldn't automate when you don't know if what you are doing provides the result you are actually after. Once you’ve nailed the effectiveness (definition) of a process - then you can safely work on its efficiency (automation)

Some common reasons why manual testing still happens.

• Time: Teams may not be allocated time to examine alternate testing options, figure out how to utilize testing frameworks, or create and maintain supporting code.
• Application Complexity: Applications may be too complex and not suitable for automated testing and effective mocking.
• Skillset: Some teams lack skills needed to be successful with test automation tools.
• Cost: Organizations may not own automated testing tools and may lack the budget to invest in them.

When is Manual Testing a Good idea?

While automation should ultimately save the most time in the long run there are still scenarios where manual testing makes more sense.

• Subjective Validation: Functions that must be validated subjectively by humans such as usability or look-and-feel.
• New/Changing Functionality: New application functions that are still being developed and changing frequently.
• Strategic Development: Strategic application functions that you want testers to pay specific attention to.
• Complex Functionality: For application functions that are complex, test automation may be a major challenge (time & cost investment outweighs the benefit)

Common pitfalls with a spreadsheet based process.

So when manually testing makes sense how do we keep track? The goto tool for manual testing is the trusty spreadsheet. It was natural for us to pick this up first as well.

However very quickly you can end up with something like this:

In a white paper produced by IBM entitled "The Risks of Using Spreadsheets for Statistical Analysis" they shared the scary fact:

More than 90% of mission critical spreadsheets contained errors

The biggest hurdle for growing teams with spreadsheets is no practical accountability - but there are many more:

• As the organization grows, more people need access.
• The more complex the business process gets, the harder the spreadsheet is to use.
• Remote workers or those working from different infrastructure may be excluded from the workflow.
• There is no concept of access control.

and more than a few hard questions:

How do you make sure everyone’s on the same page?
How do you make sure what needs to get done gets done in time?
How do you collect, compile, and review all of those results?

How Clever Checklist helped our customers (and us) overcome these scenarios.

So like our customer we started moving these manual testing scenarios into our platform and found it a very natural fit.

Templates :
Templates give us the consistency and comfort that the steps in the Checklist will be the same. They are quick to start new jobs from and we don’t have to worry about things not being copy / pasted and how to name and store multiple versions of files. If test cases needed to be added or deleted from the template, the merging functionality allows the process owner to update mid run Checklists to ensure consistency.

Checklists:
Checklists that have started from templates give many different workflow options for the people performing the test. Every tester can have their own open copy of the test cases. Alternatively multiple testers can work concurrently on the same open Checklist with all progress being kept in sync across the users.

Item Details:
Item Details lets us show, not tell. In more complicated scenarios there might be a lot of detail you need to communicate to someone to get them to replicate your actions. Item details is ideal for this as it lets us capture long form instructions for all the intermediate steps and only require a checkbox for the overall use case. This reduces a tremendous amount of friction for the person doing the work as they don’t need to swap between multiple documents and only need to tick a single item that covers multiple steps.

Copy/Paste:
Copy/Paste blocks in Item Details gives us the further benefit of allowing testers to uniformly and quickly copy data out of test cases and enter it into target systems. There is only one button to click and we know the data is right.

Evidence :
Just marking pass or fail is often not enough to help a developer locate and understand a problem. To know the job has been done right, we use Evidence to capture additional detail relating to an item that helps us be comfortable everything worked as expected. To know the job has been done right, we use Evidence to capture additional details to ensure item accuracy. This can be in the form of a screenshot that is attached to a step. Or asking for some subjective feedback through a question or list of dropdown options.

Comments :
Comments leave room for the unexpected. Items can be commented on to communicate any problems back to the developers. Collaboration between the testers and developers can occur within the testing document to allow efficient retesting.

Clever Share :
Delegating the work both inside and outside the organization is easy with Clever Share. Just enter in the email of your outsourced partner and they will get a copy of the Checklist with all changes tracked against their email address. You can watch the progress of checklists in real time and receive notifications when work is complete and ready for review.

An example of how you can structure testing in Clever Checklist to minimize friction.

Lets look at a simple example of a test case you want to ensure is performed correctly every time.

The steps are (in Martin Fowlers Given When Then format):

Feature: Fill tool
  Scenario: User wants to color a circle in red that is overlapping a square that is already blue (default color)
    Given I have a Blue Square on the canvas
       And I have an empty Circle on the canvas
       And the shapes are overlapping
       And the circle is the upper most layer

    When I select the Fill Tool
       And select the Red colour
       And click within any area of the circle

     Then I should have the Circle become filled with Red
      And I should have the visible portion of the Square remain blue
      And the background remains the original color

A traditional step by step Checklist may look like this:

This is OK but the user has to click through 6 steps.

With Item Details instead of 6 steps that needs to be signed off this can be one like this:

Conclusion

Initially spreadsheets are a natural place to accumulate processes. Once it is clear that it will be needed repeatedly this quickly becomes a liability more than an asset. In testing where the ultimate goal is consistency and quality for your product, don’t manage the process using a system that is prone to adding its own errors into the mix.

Consider deleting the spreadsheet today and move your process into Clever Checklist.