Human review reads. The respondent navigates.
A person checks the questionnaire in Word, but the respondent sees the programmed version. They are two different objects, and the error lives in the difference.
Automated survey testing
Your team still tests the questionnaire by clicking through it screen by screen.
Survey QA walks the questionnaire's paths on desktop and mobile, the way a respondent would, and returns the verdict screen by screen, with the screenshot of every screen and the rule behind every finding.
Test roundRound 1
Route 1 of 6·Age filter
Desktopreading
Q1
How old are you?
Next
Numeric with range
Mobile
Q1
How old are you?
Next
- Q1reading
- Q2queued
- Q3queued
- Q4queued
- Q5queued
- Q6queued
- Q7queued
- Q8queued
- Q9queued
- Q10queued
- Q11queued
- Q12queued
- Q13queued
- Q14queued
- Q15queued
- Q16queued
- Q17queued
- Q18queued
0 screens analyzed0 findings
01The work
The questionnaire is programmed. Someone has to test the script before fieldwork.
Every research firm runs this test. It takes days from the people who know the most, and the error that slips through shows up in tabulation, when fixing it already costs a wave.
Topic
Testing the questionnaire script before fieldwork
The questionnaire was programmed on the data collection platform and needs approval before it goes to field. Every filter, every quota and every consistency check opens a different path through the script.
How it is done today
An analyst walks through everything by hand
An analyst opens the link and walks by hand through the consistency checks, the filters, the quotas, the completes, the layout and the grammar, then redoes the route changing the answers.
The pain
Days of a senior analyst on checking work
And the error that slips through is expensive later. One field has already been saved by this test: the questionnaire went back to programming and the team had to contact the clients again.
Where Cassi.ai comes in
The robot walks, the person approves
Survey QA walks the paths on desktop and mobile and returns the verdict screen by screen. Your analyst still signs the approval, now with the list of what was verified.
First the list of paths. Then the robot. Every route exists because it proves a rule of the questionnaire.
02Desktop and mobile, together
The robot answers the same screen on desktop and on mobile.
Every screen is navigated on desktop and mobile at the same time. Whatever breaks on only one of them becomes a finding with the screen and the rule: screenshot, measurement and the rule that failed.
Test roundScene 1 of 18
Single choiceDesktopreading
Q1
Which of these brands did you buy last?
Next
Single choice
Mobile
Q1
Which of these brands did you buy last?
Next
The question types the robot answers
0 screens analyzed0 findings
Seven checks on every mobile screen
- Horizontal scroll
- Question text off screen
- Truncated text
- Overflowing grid
- Next button off screen
- Small tap target
- Small font
03The error that slips through
Programming errors reach fieldwork every day.
A filter that does not terminate, a grid that disappears on mobile, a brand left off the list. The error passes human review, enters the field and the cost shows up in tabulation: discarded interviews, a redone wave, a missed deadline.
In field · day 900:56
1.180respondents have already answered
- Q1Ageok
- Q2Regionok
- Q3Usagenobody saw it
- Q4Brandsok
- Q5Attributesok
- Q6Recommendationok
Most people answer on mobile.
A wide grid, a next button off screen and truncated text only show up on mobile. Nobody checks every screen at two sizes by hand.
The later, the more expensive.
Before fieldwork it is a fix that takes minutes. During fieldwork it is lost interviews and a wave that does not compare with the previous one.
04The turn
What if every screen were tested before the first respondent?
The robot goes in before fieldwork. It navigates every screen, proves every rule, measures every layout on desktop and mobile. With a screenshot of every screen.
03How it works
Five steps, one truth.
From uploading the documents to the link the client opens. A person approves each step before the robot runs.
Scroll to advance the steps
Step 1 of 5
layer time≈ 1 min
Sign in
One account per research firm, with analyst, director and platform administrator roles.
05What Survey QA answers
The questions the analyst asks before approving the release to field.
Every operational question has a module that answers it with the screen and the rule. The list comes from the approved questionnaire, and the robot only reports what it measured.
Is everyone who does not qualify terminated where the questionnaire says?
- ModuleFilters and terminations
- Every conditional route is walked once, with the answer that should terminate and with the one that should continue. The robot records on which screen the programming stopped.
Are incompatible answers across questions blocked?
- ModuleConsistency checks
- Required fields, ranges, sums and combinations across questions. The robot tries to break each rule and records what the programming did, including the combination that only breaks on the third try.
Does a closed quota terminate and an open quota run to the end?
- ModuleQuotas
- Cell by cell: the closed one has to terminate, the open one has to reach a complete. When a filter sends the respondent to a block the quota has already closed, that becomes a finding with the screen and the rule.
Are brands and attributes in the right order and rotation?
- ModuleOrder and rotation
- The JobSpec list is checked item by item, in the programmed order and the expected rotation, on desktop and mobile.
Is the text on screen the text of the approved questionnaire?
- ModuleTexts and options
- Question text, options and instructions compared with the document, screen by screen. The instruction inherited from the previous study shows up here, even when only a small slice of the sample would see it.
Does the screen fit on a phone?
- ModuleMobile layout
- Seven checks on mobile on every screen, next to desktop: horizontal scroll, question text off screen, truncated text, overflowing grid, next button off screen, tap target and font size.
Does the new wave still compare with the previous one?
- ModuleTracking waves
- Waves compared one by one: what broke, what was fixed and what changed in the text from one wave to the next.
What exactly was verified?
- ModuleScreen statement
- Every session records the screens it walked, the route, the objective and the screenshot of every screen. The statement is readable by the client without a translator, and no route is repeated.
06Method
Six stages, in the order they happen.
A person approves each stage before the robot runs. Judgment is by rule, and every finding comes with the screen and the rule.
- 01
Compile the questionnaire, the JobSpec and the script into a single truth
Word, Excel, MDD or a Dimensions script become one contract: questions, options, rules, brands and attributes. A conflict between sources blocks approval until the corrected document is uploaded.
input: .docx .xlsx .xlsm .mdd
- 02
Teach the robot with a person
Someone answers the survey once, with desktop and mobile side by side. The recording teaches the path, the selectors and the way each platform advances.
co-pilot: 1 human pass
- 03
Plan the routes, one per objective
The plan comes from the contract: every filter, quota, consistency check and list gets its own route. Repeated routes are forbidden, and the person approves the plan before running.
plan: 1 route = 1 objective
- 04
Walk it on desktop and mobile
The robot navigates every route at both sizes at the same time, screen by screen, with a screenshot and a measurement at every step.
desktop and mobile, side by side
- 05
Judge screen by screen, by rule
Every rule in the contract becomes a check: the robot tries to break the rule and records what the programming did. The verdict comes from rules, with no language model.
judgment by rule
- 06
Return the verdict screen by screen
One sentence, four numbers and the screen-by-screen ledger with the screenshot of every screen open. A client link in three languages, a one-page PDF and a QCF spreadsheet per screen and per variable.
output: link, PDF, QCF
07Proof
Four errors that only show up when someone walks the whole script.
These are the errors manual testing rarely reaches, because they live in a combination of answers or in a small slice of the sample.
- 01
A filter that closes on its own
The profile question sends the respondent to a block the quota has already closed. They see no error. They see the end of the questionnaire, and enter the database as a dropout.
- 02
A consistency check that only breaks on the third combination
Age, region and purchase frequency pass separately. Together, they send the respondent to another target's block. Manual testing rarely reaches that combination.
- 03
A return route that erases the open answer
The respondent gets the format wrong, corrects it, and lands on the screen before the one they were on. Fieldwork goes on, and the open answer from that screen is gone.
- 04
Text inherited from the previous study
The instruction of an open question kept the text from the previous study. It only appears to whoever falls into that filter, always a small slice of the sample.
Numbers of the method
- 7
- layout checks on every mobile screen
- rule of every session
- 2
- sizes per screen: desktop and mobile
- rule of every session
- 0
- AI calls in judgment
- rule of the method
- 1
- route per objective, none repeated
- rule for every session
Rules that hold in every session, with the source named and no client names.
Read the article on the Cassi.ai blog08What the client receives
One screen, one verdict, the screenshot of every finding.
The result fits in one sentence, with four numbers below it and the screen-by-screen ledger for anyone who wants to check.
Along with the result
- Client link in Portuguese, English and Spanish
- One-page PDF
- QCF spreadsheet per screen and per variable
Verdict
Ready for fieldwork after 2 fixes.
- 42
- Screens analyzed
- 2
- Findings
- 31
- Rules proven
- 3
- Routes walked
Ledger
- Q5 · attribute grid overflows mobileopen screenshot
- Q1 · age filter does not terminate at 17open screenshot
- Q7 · quota termination checkedopen screenshot
- Q3 · usage, required, checkedopen screenshot
- Q4 · JobSpec brands in the right orderopen screenshot
0 false positives
09Limits
What Survey QA never does.
A test that promises everything cannot be checked. These are the limits that hold in every study.
- 01
Never approves the release to field in your place. Your analyst still signs.
- 02
Never forces a step when the questionnaire refuses to advance. The refusal becomes a finding, with the screen.
- 03
Never uses a client's questionnaire, answers or screenshots without authorization.
- 04
Never shows "passed" without the list of what was verified.
- 05
Never claims coverage it did not measure. Every coverage sentence states its scope.
- 06
Never works around anti-bot barriers or CAPTCHA. It detects, reports and calls a person.
- 07
Never repeats a route already charged. Every session has its own objective.
10Platforms
The robot recognizes the platform on the first screen.
The robot does not depend on a catalog of platforms. On the first screen it discovers how the survey advances, how it marks an option and how it records a termination, and goes on from there.
- WebRaptor
- Sawtooth
- IBM Dimensions
- Qualtrics
- Confirmit/Decipher/Forsta
- Alchemer
- QuestionPro
Your platform
Send the test link. If the first screen answers, the rest of the questionnaire answers too.
11Who it is for
For whoever answers for the field.
Research firms
- The final test with the same standard in every study, regardless of who reviewed it.
Fieldwork agencies
- The link arrives tested. Fewer discarded interviews and less rework in the middle of collection.
Insights teams
- A readable verdict to approve fieldwork with the list of what was verified.
Questionnaire programmers
- The screen, the rule and the measurement of every finding, without back and forth by email.
12Questions and answers
What people ask before requesting access.
What is Survey QA?
Survey QA is a Cassi.ai product for automated survey testing. A robot walks the survey's paths on desktop and mobile, the way a respondent would, and returns a verdict screen by screen with the screenshot and the rule for every finding.
What does the test check?
Filters and terminations, consistency checks across questions, quotas, order and rotation of brands and attributes, texts and options against the approved questionnaire, mobile layout in seven checks and the comparison between waves of a tracking study.
Which platforms does it work on?
WebRaptor, Sawtooth, IBM Dimensions, Qualtrics, Confirmit/Decipher/Forsta, Alchemer and QuestionPro. The robot recognizes the platform on the first screen and discovers how to advance. Your platform goes through the same path: send the link.
Does the robot use artificial intelligence?
Not in judgment. Every rule of the questionnaire becomes a deterministic check, and the verdict comes from rules, with zero AI calls. The reading of the documents is checked by a person before the robot runs.
What does the client receive at the end?
One screen with the verdict in one sentence, four numbers and the screen-by-screen ledger with the screenshot of every screen open. Along with it go the client link in Portuguese, English and Spanish, a one-page PDF and the QCF spreadsheet per screen and per variable.
How much of the questionnaire is covered?
What the approved plan enumerates, and only that. Every coverage sentence states its scope: routes planned, routes walked, screens reached. The robot never claims coverage it did not measure.
What happens when the questionnaire has a CAPTCHA or an anti-bot barrier?
The robot detects it, reports it and stops. It never works around the barrier. The finding comes out with the screen, and a person decides how to proceed, usually with the test link without the barrier.
Who approves the release to field?
Your analyst. Survey QA delivers the list of what was verified, with the screenshot of every screen, and never approves fieldwork in place of whoever signs.
How do I get access?
Through the access request on this page. Products available via SaaS by credit, by project, or as a platform purchase. Talk to us, we have a format that fits your budget.
Every questionnaire that goes to field has paths nobody walked.
Whoever enumerates those paths first signs the approval with a different peace of mind.
Take the last questionnaire your team approved and count the hours of manual testing. We run the same script through Survey QA and show you the verdict.
Products available via SaaS by credit, by project, or as a platform purchase. Talk to us, we have a format that fits your budget.
Cassi.ai is a software engineering company specialized in the pains of research, innovation and insights. Twenty years in market research, former ESOMAR Brazil, with talks at IIEX, ESOMAR and ABEP.