Delete the Sheet
Customers are always awesome at finding ways to use a product for things you don’t initially envisage. It’s nice when we discover that a problem that a customer is looking to solve with our platform is one we have too!
Recently a customer moved their large manual test case suite into our platform.
Before they’d used a combination of Google Sheets and Forms to try and wrangle
external testers to perform weekly tasks against the latest builds.
They were having some real problems making this scale. Quite often the labor
in managing the spreadsheets and coordinating with the external parties was
taking longer than the test schedule itself.
After a successful migration into the system they reduced their weekly setup
burden to minutes. Clever Share allowed them to get the individual tests out to
the third parties with little friction.
This made us take a step back and look at our own manual testing procedures and
when we thought about it we realized we doing this to. We’d let some “one time”
spreadsheets become difficult to manage before realising there was a better way
close at hand.
So always looking to eat more of “our own dog food” we thought about how
using Clever Checklist could make this better for our testing.
We love spreadsheets. But we love to delete spreadsheets more!
So why manually test at all?
You still manually test!? Shouldn’t we be automating all the things?
Every company doing automated testing still does some amount of manual testing.
We are not ready and won’t ever be ready to replace all the humans.
An industry survey revealed 75 percent of all functional testing is still
You shouldn't automate when you don't know if what you are doing provides the
result you are actually after. Once you’ve nailed the effectiveness
(definition) of a process - then you can safely work on its efficiency
Some common reasons why manual testing still happens.
- Time: Teams may not be allocated time to examine alternate testing options,
figure out how to utilize testing frameworks, or create and maintain supporting
- Application Complexity: Applications may be too complex and not suitable for
automated testing and effective mocking.
- Skillset: Some teams lack skills needed to be successful with test automation tools.
- Cost: Organizations may not own automated testing tools and may lack the budget to invest in them.
When is Manual Testing a Good idea?
While automation should ultimately save the most time in the long run there are
still scenarios where manual testing makes more sense.
- Subjective Validation: Functions that must be validated subjectively by humans
such as usability or look-and-feel.
- New/Changing Functionality: New application
functions that are still being developed and changing frequently.
- Strategic Development: Strategic application functions that you want testers to pay
specific attention to.
- Complex Functionality: For application functions that are
complex, test automation may be a major challenge (time & cost investment
outweighs the benefit)
Common pitfalls with a spreadsheet based process.
So when manually testing makes sense how do we keep track? The goto tool for
manual testing is the trusty spreadsheet. It was natural for us to pick this up
first as well.
However very quickly you can end up with something like this:
In a white paper produced by IBM entitled "The Risks of Using Spreadsheets for
Statistical Analysis" they shared the scary fact:
More than 90% of mission critical spreadsheets contained errors
The biggest hurdle for growing teams with spreadsheets is no practical
accountability - but there are many more:
- As the organization grows, more people need access.
- The more complex the business process gets, the harder the spreadsheet is to use.
- Remote workers or those working from different infrastructure may be excluded from the workflow.
- There is no concept of access control.
and more than a few hard questions:
How do you make sure everyone’s on the same page?
How do you make sure what needs to get done gets done in time?
How do you collect, compile, and review all of those results?
How Clever Checklist helped our customers (and us) overcome these scenarios.
So like our customer we started moving these manual testing scenarios into our
platform and found it a very natural fit.
Templates give us the consistency and comfort that the steps in the Checklist
will be the same. They are quick to start new jobs from and we don’t have to
worry about things not being copy / pasted and how to name and store multiple
versions of files. If test cases needed to be added or deleted from the
template, the merging functionality allows the process owner to update mid run
Checklists to ensure consistency.
Checklists that have started from templates give many different
workflow options for the people performing the test. Every tester can have their
own open copy of the test cases. Alternatively multiple testers can work
concurrently on the same open Checklist with all progress being kept in sync
across the users.
Item Details lets us show, not tell. In more complicated
scenarios there might be a lot of detail you need to communicate to someone to
get them to replicate your actions. Item details is ideal for this as it lets us
capture long form instructions for all the intermediate steps and only require a
checkbox for the overall use case. This reduces a tremendous amount of friction
for the person doing the work as they don’t need to swap between multiple
documents and only need to tick a single item that covers multiple steps.
Copy/Paste blocks in Item Details gives us the further benefit of
allowing testers to uniformly and quickly copy data out of test cases and enter
it into target systems. There is only one button to click and we know the data
Just marking pass or fail is often not enough to help a developer locate and
understand a problem. To know the job has been done right, we use Evidence to
capture additional detail relating to an item that helps us be comfortable
everything worked as expected. To know the job has been done right, we use
Evidence to capture additional details to ensure item accuracy. This can be in
the form of a screenshot that is attached to a step. Or asking for some
subjective feedback through a question or list of dropdown options.
Comments leave room for the unexpected. Items can be commented on to
communicate any problems back to the developers. Collaboration between the
testers and developers can occur within the testing document to allow efficient
Clever Share :
Delegating the work both inside and outside the organization is easy with
Clever Share. Just enter in the email of your outsourced partner and they will
get a copy of the Checklist with all changes tracked against their email
address. You can watch the progress of checklists in real time and receive
notifications when work is complete and ready for review.
An example of how you can structure testing in Clever Checklist to minimize friction.
Lets look at a simple example of a test case you want to ensure is performed
correctly every time.
The steps are (in Martin Fowlers Given When Then
Feature: Fill tool
Scenario: User wants to color a circle in red that is overlapping a square that is already blue (default color)
Given I have a Blue Square on the canvas
And I have an empty Circle on the canvas
And the shapes are overlapping
And the circle is the upper most layer
When I select the Fill Tool
And select the Red colour
And click within any area of the circle
Then I should have the Circle become filled with Red
And I should have the visible portion of the Square remain blue
And the background remains the original color
A traditional step by step Checklist may look like this:
This is OK but the user has to click through 6 steps.
With Item Details instead of 6 steps that needs to be signed off this can be
one like this:
Initially spreadsheets are a natural place to accumulate processes. Once it is
clear that it will be needed repeatedly this quickly becomes a liability more
than an asset. In testing where the ultimate goal is consistency and quality for
your product, don’t manage the process using a system that is prone to adding
its own errors into the mix.
Consider deleting the spreadsheet today and move your process into Clever