Rendered at 21:01:05 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
ottoflux 1 days ago [-]
100% and I’m a software developer and have been for ~30 years. Good QA people know how to find regression and bugs _that you didn’t think about_ which is the whole reason why it shouldn’t be under “engineering” and that it should exist. One of the QA people I work with currently is one of my favorite people. They don’t always make me happy (in the moment) with their bugs or with how they decide to break the software, but in the end it makes a better, more resilient product.
_doctor_love 1 days ago [-]
Couldn't agree more strongly. The best QA people have the old-school “test to break” mentality. They do weird things, like pulling the network cable out of their machine mid-transaction, or kicking off a series of performance scripts and then powering off servers in a distributed system just to see what happens.
When I got started in software, QA was already in a heavy decline. A mentor who had been a QA manager at Apple told me that being a QA person (in the industry) was once seen as a high-trust position. He was sad at what it became.
ffsm8 13 hours ago [-]
When I got into software that employer was pretty small (50 people overall I think).
Their approach to QA was to have this be a optional thing the service people could do.
It worked surprisingly well, with the caveat that they never created regression tests.
The employer eventually switched someone from that team to establish these regression tests full time, but they had no programming experience and by the time I left no real progress was done in around 6 month. No idea what came of that, and a few years later they fired a lot of the team
cindyllm 1 days ago [-]
[dead]
jeremyjh 1 days ago [-]
Agreed. QA specialists are there to think about what the engineer didn't think about. Unless the engineer is incompetent or the organization is broken, the engineer has already written tests for everything they could think of, but they can't think of everything.
More importantly, it is almost impossible for engineers to be as well incentivized to spend extra time exploring edge cases in something they already believe to work than to ship a feature on time.
Like everything else though, its contextual. Complexity of domain, surface area and age of product, depth of experience on team and consequences of failure are all so variable that there cannot be only one answer.
I have done it both ways for many years. I have worked on teams where QA is a frustrating nuisance, and teams where they were critical to success. I have worked on teams that did pretty good without them, and probably those were the highest throughput, most productive teams because the engineers were forced to own all the consequences - every bug they shipped was a production issue they were immediately forced to track down and resolve.
But those were very small teams, and eventually I was the only founding engineer left on the team and far too many mistakes by other people made it to my desk because I was the only person who could find them in review or track them down quickly in production. That was when I started hiring QA people.
MoreQARespect 1 days ago [-]
Ive almost never worked on a project where there was the right number of QAs who were doing the right thing.
Usually there either arent any in which case bugs get missed or there are 5 very cheap ones running mindless scripts who are standing in for the devs' inability or unwillingness to write decent automated tests but dont catch the really deep level thorny stuff.
ghywertelling 19 hours ago [-]
QA has a reputation problem. Because it is considered as unimportant role, good people don't get attracted to it. The average people who do don't do a good job.
locknitpicker 9 hours ago [-]
> QA has a reputation problem. Because it is considered as unimportant role, good people don't get attracted to it.
Hard disagree. QAs' main tasks are acceptance testing and V&V. This means their work is to act as independent parties whose role is to verify that the results are indeed being delivered. Their importance is up there next to Product Owners.
The problem is that some QA roles are relegated to running the same old test scripts and do some bullshit manual exploratory tests that catch and assert nothing. It's the ultimate bullshit job as it's performative. This is clear in the way that LLM agents are seen as threatening the very existence of the role.
ricardobayes 1 days ago [-]
Interesting username
anothereng 1 days ago [-]
i know right hahah
9wzYQbTYsAIc 1 days ago [-]
> More importantly, it is almost impossible for engineers to be as well incentivized to spend extra time exploring edge cases in something they already believe to work than to ship a feature on time.
Personal liability and professional insurance works for all the actual “professions” in the US, to some extent, right?
It might be time to start the considerations for professional licensing for platform scale or commercially published software.
rfrey 23 hours ago [-]
Real engineering disciplines also have dedicated QA and test engineers.
ivan_gammel 1 days ago [-]
More like certified products. New ISO standard may require professional liability for software products, which will be adopted as requirement by big consumers and will pull the industry into certification loop, because insurers will ask for it. This will obviously put a high entry barrier to many product categories, slowing down innovation.
9wzYQbTYsAIc 1 days ago [-]
Yes, but slowing down to avoid hazards is sometimes important.
Medical devices and such are the only places I’d expect to see the need for certified products. By extension, in the new era, we really ought only expect certified software where we expect a duty to care from the software system (or any other assigned duty)
ivan_gammel 1 days ago [-]
In development of medical devices existing quality controls are already working well, right?
9wzYQbTYsAIc 1 days ago [-]
My point exactly, embedded devices are the closest software gets to actually being built by licensed engineers. The expectation can often be that you are an electrical engineer by training, where licensure is a viable path, unlike in software engineering.
canpan 1 days ago [-]
Yes, QA is important. My code will always "work" in that everything I tested is bug free. But having someone other test, especially someone who knows the service is gold.
But there is also bad QA: The most worthless QA I was forced to work with, was an external company, where I, as developer, had to write the test sheet and they just tested that. Obviously they could not find bugs as I tested everything on the sheet.
My most impressive QA experience where when I helped out a famous Japanese gaming company. They tested things like press multiple buttons in the same frame and see my code crash.
darth_aardvark 1 days ago [-]
> But there is also bad QA: The most worthless QA I was forced to work with, was an external company, where I, as developer, had to write the test sheet and they just tested that. Obviously they could not find bugs as I tested everything on the sheet.
This was my sole experience at the one place I worked with an internal QA team. They absolutely could never find bugs that devs missed, often mis-marked ones that didn't exist, and failed to find obvious edge cases that did exist.
Multiple devs fired because the CEO believed the QA over the engineering team; if they marked a bug as present, it was the engineer's fault for writing it. If they didn't catch a bug that made it to prod, it was the engineer's fault for not including it in the test plan. They represented nothing but red tape and provided no value.
Good QA sounds great! I'd love to know what that's like someday! It'd be great to have someone testing my code and finding breakages I missed! I'm only slightly (incredibly) bitter about my bad experience with its implementation.
Ekaros 1 days ago [-]
I do think the type of testing where QA just follows pre-generated script has place. But it is about long term regression. The first round absolutely should not find anything. But with complex system it also should find nothing in a year or three or five years... Offloading this to dedicate resource could be useful in certain industries.
canpan 1 days ago [-]
I did not think of that. Maybe for some industries, it might make sense. But if I want a regression test, I would probably set it up as automated test. In the case I mentioned above it was the only test beside my own for a new service.
philk10 1 days ago [-]
Not really that impressive, that's Testing Quick Attacks 101
jghn 1 days ago [-]
The tension is that QA is important. But most QA practitioners are not good. The world is filled with QA people who couldn't make it as an SWE and now are button pushers. But high quality QA people are amazing. These are the ones who understand how to break apart a system, push it to its limits, and engineer the quality plan.
This is an area where I expect AI to create a bimodal future. The smaller group of high quality QA people will now be able to offload the activity to agents instead of the QA drones. They'll still be worth their weight in gold, whereas the drones will be redundant.
malfist 1 days ago [-]
I've worked with a lot of QA folks that just repackage up the unit tests the dev already writes. And I've met a few that strikes out on their own and comes up with real tests.
The latter is much more high touch, but they're often worth their weight in gold. The former is kinda pointless.
jghn 1 days ago [-]
Exactly. I think AI tooling will make that good group even more effective. And it will make the bad group even more pointless.
z500 24 hours ago [-]
We don't have dedicated QA at my job, but personally I'm thrilled when a coworker finds a bug in my code. It's much more preferable to the customer finding it
D-Coder 21 hours ago [-]
Yes! I had a QA person apologize once for finding a bug in my code. I thanked him for not letting it make me look like an idiot.
LatencyKills 1 days ago [-]
Exactly. I spent 20 years split between MS and Apple. Some of the best people I ever worked with were in QA. One guy in particular was an extremely talented engineer who simply didn't enjoy the canonical "coding" role; what he did enjoy was finding bugs and breaking things. ;-)
MoreQARespect 1 days ago [-]
Really? The best people I worked with were never QA.
Moreover, the best QAs would almost always try to be not QA - to shift into a better respected and better paid field.
I wish it werent so (hence my username) but there is a definite class divide between devs and QA and it shows up not just in terms of the pay packets but also who gets the boot in down times and who gets listened to. This definitely affects the quality of people.
I think it's overdue an overhaul much like the sysadmin->devops transition.
LatencyKills 1 days ago [-]
We have differing experiences, which shouldn't be surprising. My example explicitly referred to someone who was a good engineer who enjoyed the QA role.
This might have been an Apple/MS thing, but we always had very technical QA people on the dev tools team. For example, the QA lead for the C++ compiler had written their own compiler from scratch and was an amazing contributor.
stiglitz 1 days ago [-]
In the Windows team (back before the test org was decimated) I saw the described "class divide". Anybody who was good enough would switch from SDET to SDE [disclaimer: obviously there were some isolated exceptions]. The test team produced reams of crappy test frameworks, each of which seemed like a "proving project" for its creators to show they could be competent SDEs. After the Great Decimation my dev team took ownership of many such frameworks and it was a total boondoggle; we wasted years trying (and mostly failing) to sort through the crappy test code.
This was all unfortunate, and I agree in principle with having a separate test org, but in Windows the culture unfortunately seemed to be built around testers as second-class software developers.
LatencyKills 1 days ago [-]
I spent most of my time working on Visual Studio (in the Boston time frame) so we got to interact with pretty much every team. I absolutely hated interacting with the Windows team. Everything was a fight for no reason.
As I said above, everyone has their own experiences but the QA folks I worked with at MS were fantastic.
Not sure if you're aware but Dave Plumber now has a really good YT channel [0] where he talks about MS back in those days. It's a fun walk down memory lane.
I mean the people that come up thru QA may be the best while getting enough time in the company to go to a position that pays.
But yea, so many companies cheap their QA and then wonders why their QA sucks.
saghm 1 days ago [-]
> Really? The best people I worked with were never QA.
> Moreover, the best QAs would almost always try to be not QA - to shift into a better respected and better paid field.
That sort of seems circular. If they're not respected or paid well, of course most of the talented people would not want to remain in QA, and eventually you'd just have mediocre QA. That doesn't really give you any insight into whether high quality QA would be useful though.
(edit: I see now that's basically the point you're trying to make, so I guess we're in agreement)
dotancohen 19 hours ago [-]
> They don’t always make me happy (in the moment) with their bugs or with how they decide to break the software, but in the end it makes a better, more resilient product.
Did you try with 2 QA people? 65536 QA people? 0 QA people? NULL QA people?
efsavage 24 hours ago [-]
Great QA people are rarer than great developers, and potentially even more valuable.
pphysch 24 hours ago [-]
Yes, QA and test engineers tend to have a better ability to specify the correct behavior of a system than anyone else. It's literally their job.
This is a big asset in the current paradigm where a LLM agent can effectively implement behavior only when given a robust test suite to iterate against.
Perepiska 12 hours ago [-]
> with their bugs
This is funny.
stingraycharles 1 days ago [-]
> Good QA people know how to find regression and bugs _that you didn’t think about_ which is the whole reason why it shouldn’t be under “engineering”
I don’t understand the reasoning here why QA shouldn’t be engineering.
9wzYQbTYsAIc 1 days ago [-]
> I don’t understand the reasoning here why QA shouldn’t be engineering.
Who watches the watcher, right?
That aside, the core idea is the same as the principles of independent audit, peer review, or even simply just specialization.
Red team / Blue team?
stingraycharles 1 days ago [-]
Yes but both the red team and blue team would still be engineering.
9wzYQbTYsAIc 1 days ago [-]
Yes, but police and military are both law enforcement, on one level, but each are very different from the other.
Even the military have police, right?
edit: ultimately, it comes down to the importance of independent audit, the builders and the breaker/fixers are very different groups in engineering.
IAmBroom 1 days ago [-]
The red team and blue team should not share supervisors.
Nor, in the case of QA, should the audit team be engineers trained to act and think like the ones who wrote the software. A fresh perspective is useful.
But in the long run, supervisory independence is the real deal. I know of a QA manager who shut down an entire factory's output until a major safety issue (that had been kicked down the road several times) was addressed. It took chutzpah, and serious power, to do that. The Dir. of Engrg. would NEVER have allowed it.
liampulles 1 days ago [-]
Frankly, calling software development engineering is quite debatable. We should be calling less things engineering that aren't actually engineering qualifications.
bigfatkitten 24 hours ago [-]
Being a branch of engineering implies a certain level of professionalism and accountability that the software development community actively resists.
rfrey 23 hours ago [-]
Engineering like the guy in the booth at a show is a sound engineer. Talented: check; challenging work: check; valuable: check; creative: check. "Engineering" like designing a building, bridge, or power line? Nope.
It's not a protected term in the US so it's jarring to those of us living where it is.
mememememememo 21 hours ago [-]
It should be under engineering but as a seperate skill/role. To ship fast you want QA aligned with milestones, sprints etc. You want QA to feel the deadlines, and then like an Eng push up if things aren't realistic.
What I have seen before with QA is a queue like system. Hand to QA. get back in 3 days. Not the QA persons fault. It is the setup if the org that is wrong.
Imagine if you had a code review role and code review team and you had to wait to get your code OKed. We wouldn't put up with it.
QA teams seperate from Eng worked when the software was burned to CD rather than uploaded to s3.
locknitpicker 10 hours ago [-]
> Good QA people know how to find regression and bugs _that you didn’t think about_ which is the whole reason why it shouldn’t be under “engineering” and that it should exist.
I think the core of the issue is the weasel word "Good QA" instead of just "QA". You're underlining the importance of having someone in a team who has good product understanding and has good ownership skills. How many QAs fit that description? Not many, unfortunately. Some FANGs outright eliminated the role and replaced it with a mix of product owners and team ownership, and some QAs are just doing their 9-to-5 job going through their manual test scripts to verify it something meets a definition of done. What happens when you can have a LLM agent doing the same instantly as a step in a random CICD pipeline?
popularonion 20 hours ago [-]
From many years of first hand experience:
- QA is always the first thing companies outsource, with predictable results
- Companies either go the route or “separate QA org with separate management chain” or “have QA engineers report to dev managers”. I’ve seen serious misaligned incentives and toxic outcomes with both
- Frequent Slack messages at 4:15 PM on Friday - “hey they just merged the PR, we really need it tested before Monday stand up”
- QA becomes a de facto dumping ground for glue work that other teams don’t want to do. Senior QA ends up morphing into a de facto “responsibility without authority” project manager role
- There is zero internet “community” around QA the way there is for developers or designers. There is no Slashdot or Hacker News for QA and there never will be. Just a bazaar of book authors and consultants promoting themselves on LinkedIn
IMO the only thing that makes sense anymore is having good SDETs embedded in engineering teams.
darkwater 1 days ago [-]
"Engineers sometimes exhibit an arrogance that they can do everyone else’s job,"
This rings so many bells that it feels like some Buddhist festival.
Apply the same approach to QA, Operations, and anything outside the actual product development: when this arrogance was shared between bosses and developers, all good on their side.
Now with the AI, the arrogance is staying only on the bosses' side, and we have developers freaking out.
ozlikethewizard 1 days ago [-]
Its also a fucking annoying sentiment for us as engineers. I dont want to do everyone else's job
steveBK123 1 days ago [-]
A good QA actually does the work that product managers should but don't, in terms of understanding actual users and checking the feature delivered matches actual user ask.
But I've worked at places with a whole spectrum of coverage in roles spanning Product Mgrs, Project Mgrs, BAs, QAs, production support level 1, production support level 2, etc. The one constant is whatever is missing or understaffed just ends up getting done by engineers.
Testing, on-call, Jira managing, requirements gathering with users, analysis, etc... all falls on to engineering. Then management gets even more wound up about dev productivity/velocity, etc.
jagged-chisel 1 days ago [-]
But, as someone who’s agile and adaptable, I can do any job. That doesn’t mean I can do them all simultaneously. It doesn’t mean I can be the full-time loan officer and the full-time app developer.
Can I do your job? Yep. Can I also, at the same time, be the engineer that optimizes the IT systems? No - one of these jobs will suffer.
Give me the chance to understand your job, and I’ll replace as much of it as possible with code to do the same thing. But what it won’t do is have good judgement. It will make decisions on actual data - accurate data, erroneous data, it doesn’t care.
I think this is an interesting place to put “AI” - can it take input in the form of data and historical decisions, and come to a new decision from recent data? The same decision a human would?
IAmBroom 1 days ago [-]
Thank you for embodying and exemplifying the point, albeit ironically.
jagged-chisel 1 days ago [-]
Aside: it has become an interesting personal experiment to stop being obviously ironic and see how people read what I’ve written. The voting is telling.
rfrey 23 hours ago [-]
I didn't "vote" but there aren't many clues in your comment suggesting it's satire. It is a mainstream presentation of a mainstream point of view, how would anybody distinguish between you believing it earnestly and you mocking it?
1 days ago [-]
ivan_gammel 1 days ago [-]
There are two very important ideas in this article, which I fully agree with: QA are not the only people responsible for quality - entire team is. QA act as experts and drivers of quality management process, but they should not and are not acting alone. They should have adversarial approach which is helpful on every stage of SDLC. Thus, few more items from my list why QA is useful in every engineering organization and why every team I hire has at least one QA starting from 4-5 people:
1. Quality management is a continuous process that starts with product discovery and business requirements. Developers often assume that requirements are clear and move on to building the happy path. QA often explore requirements in depth and ask a lot of good questions.
2. QA usually have the best knowledge of the product and help product managers to understand its current behavior, when new requirements suggest to change it.
3. The same applies to product design. Good designer never leaves the team with a few annotated screens, supporting developers until the product is shipped. Design QA - the verification of conformance of implementation to design specs - can be done with QA team, which can assist with automation of design-specific tests.
4. Customer support - QA people are natural partners of customer support organization, with their knowledge of the product, existing bugs and workarounds.
And just a story: on one of my previous jobs recently hired QA engineer spotted number error in an All Hands presentation. That was an immediate buy-in from founders. :)
megadopechos 1 days ago [-]
Absolutely QA "should" exist. Our QAs are the most knowledgeable people on our product, often informing devs and product alike of requirements, missing requirements, weird configuration outliers, how to actually use the damned app, etc. Without QA we would be developing and testing for brittle requirements to get code into an MVP state, not a functional, user-friendly state.
fragmede 1 days ago [-]
"Should" is one of those words. QA absolutely should not exist. Developers should not write buggy code to begin with! But they do, so QA (in some form) must exist for software to be any good.
ryanmcbride 20 hours ago [-]
It's not just finding bugs, QA people are the people who actually USE the product, every part of it, every single day. They find pain points and things to improve and things that don't make sense constantly. But yes they're also really good at finding bugs.
bluGill 23 hours ago [-]
QA should find not just bugs but where 'works as speced' is wrong because it doesn't make sense to do it that way.
ryanmcbride 20 hours ago [-]
This is something I've put a lot of thought into the past couple of years, and a few little soundbites I've come up with during my imaginary shower interviews are:
1. If you don't have Quality Assurance, then you have Quality Uncertainty.
2. QA is a full time job. If you offload the responsibility of QA to the engineers, then you're giving them 2 full time jobs. So unless they're working 16 hours days (even if they are tbh), you aren't assuring quality, you're compromising it.
mixmastamyk 19 hours ago [-]
Yes, and paying more for the same work.
ninjagoo 1 days ago [-]
So many of these articles talk about why a particular role or type of role within an org should be there or not, but they fail to touch on the 'theory' of why or why not. This article has that same lack of foundation, and so meanders around a bit, IMHO.
Any process in an organization of size will have indicators that measure output. Those indicators should typically be paired with indicators that measure the quality of the output, to ensure product or service levels. That's the theory, and the genesis of 'quality management': whether you're measuring output code or breakfasts [1] or chemicals or widgets or medicine, you need to measure the quality of the output if there are any client specifications or expectations around the output. And there are very few cases where your customers will not have any specs or expectations around your product or service.
How you manage quality follows from those basics; it matters where you measure quality but it is so process dependent - earlier in the process lowers costs, but may not suffice to guarantee final quality - that quality management has to be designed around the specific process; balancing cost with benefit and requirements. How deep or specialized quality management becomes depends on the needs of the org, the size of the org, and the needs of the particular process.
This is why I'm skeptical about whether broad articles like this are beneficial overall. Why and how matter, and where's the foundational discussion behind why and how? Do folks not think at the organizational/business level? Maybe not everyone is a Sheryl Sandberg :-)
Except I worked at a company with a QA department made up of entirely "Automated Verification Engineers" ... over a decade ago. And the head of the department had taught at a local QA school (so presumably other QA engineers learned that style of work from her also).
Good QA departments switched to this mode long before AI was even a thing! Maybe 90+% of QA departments didn't work that way pre-AI, but there certainly were ones that did!
namelosw 1 days ago [-]
I worked with an excellent QA once, and that changed my perspective completely as a dev.
A great QA can understand the features of a product quickly, turn those concepts into some sort of grid or matrix in their mind, then pull a bunch of paths and scenarios with estimated priorities and probabilities at a fast and efficient pace, all with great coverage. They can also identify features contradicting each other more quickly than product people.
I think a good QA is capable of being a great vibe coder nowadays, too. If you can write great test suites (write names only), agents nowadays are able to turn those specs into decent codebases. Comparatively, I know a lot of decent dev having not very good taste in testing, who often write overlapping tests or missing important paths.
l72 8 hours ago [-]
In my opinion, the first mistake this article makes is assuming QA should be a completely separate team. My experience has shown that QA works best when it is an integral part of the engineering team. When you silo QA, you create a "throw it over the wall" dynamic where testing becomes a bottleneck rather than a collaborative process. Instead, they should be managed alongside developers and attend the same meetings.
What I find works best is when the Product Manager, Developer Lead, and QA Lead collaborate on requirements. The Product Manager designs the feature, the Developer focuses on technical design and architecture, and the QA Lead maps out user scenarios, integration points, and edge cases. Together, they define the requirements that constitute a successful feature.
When QA is involved early in the process, they understand and drive exactly how the feature should behave. This ensures that critical cases are considered and potential risks are handled before a single line of code is written.
kabdib 1 days ago [-]
I miss my Q/A partner. He was a engineer who gave me feedback on designs and code, helped debug, and I could run the tests he wrote.
It's terrifying without that support. The beancounter level mgt does not Get It that it's cheaper to have this kind of support now than much later in the product cycle.
acrophiliac 1 days ago [-]
The article seems to equate QA with testing, which is a short-sighted view. QA also includes things like standards, and importantly, design and code reviews, which are actually the best way to improve quality.
yabones 1 days ago [-]
A good QA team will be the definitive experts on a piece of software. They have more complete and in-depth experience than sales, engineering, or anybody else. A single software engineer might know their specific piece, but your QA tester will know everything.
itmitica 1 days ago [-]
Is this the real life? Is this just fantasy?
If engineering owns quality, then engineering own all, up the chain. No need for anything and anybody.
Which is the AI pipe dream, really.
You are, in fact, with using AI, QA or coding or otherwise, externalizing services in hope the services will improve and costs will drop.
Let me know how that goes without HITLs.
9wzYQbTYsAIc 1 days ago [-]
If engineering owns quality, it ought to also own the liability, too.
That would put the damper on the pipe dream pretty quick. Probably more healthily than any data center ban could ever do.
If engineers were licensed, bonded, and liable, things would go very differently.
* speaking as having been a practicing software “engineer” for a decade
itmitica 1 days ago [-]
Applies to making bread or brain surgery. Sometimes things go wrong and people are accounted for.
But since CEOs, or any other bosses, need to make a living, they will eat the liability in exchange for wealth, and leave engineering in the dust.
9wzYQbTYsAIc 1 days ago [-]
Somehow hospital CEOs and doctors/nurses have managed to both eat the liability and make a living.
I think the same patterns can be applied here.
ninjagoo 1 days ago [-]
> Is this the real life? Is this just fantasy?
Caught in a landslide, no escape from reality ...
:-)
GarnetFloride 18 hours ago [-]
Yes, QA should exist, and should be managed by Operations.
I've been places where devs have no idea what the product-as a whole-does. They just work on the feature of the sprint and throw the code over the wall. Their testing consists of if: it compiled==it passed. They have no idea how to even start actually testing if it's not on the happy path.
I been in places where the code accomplished the spec, but in the most lazy way possible so it appeared to work but was useless outside of what the tests looked for.
I knew one QA guy that was amazing but was so overloaded because management kept hiring "cheap" QA that were actively making his life worse.
I'm a tech writer right now at a tech company and a dev just sent over an LLM generated "doc" that's referring to things that don't exist.
Neither management nor dev has learned anything from Therac-25. QA is hard.
Zeik 16 hours ago [-]
I agreed 100% with beginning but I got lost with the automatic verification engineers. Especially with the part starting with “ You also need to level up your skills in the testing and deployment lifecycle….” How do you know what skills the qa person possesses? This sounds exactly like the things I might hear from developer explaining how I should do my work and what tools I should use.
Test automation is not same coding that developers do. I don’t mean from quality point but they have completely different set of things that matter.
Unlike writer here may think while it is important to have tests that pass or fail fast the performance is not one of the most important characteristics. You see tests are run less often than the code and not all tests need to run every time or even every day.
One of the most important characteristics of a good test is something developers easily overlook. It is maintainability. When code is written then it’s tested and then it sits in codebase until refactored. Tests don’t get this. Tests work on environment that is under constant change (especially in shift left) that makes maintainability and abstraction layers much more important with test automation code than with developing a feature.
Something to note that some developers do not understand testing at all. They may think it’s all about getting tests passed or getting software broken. It’s not either of those. It’s just checking that software works like expected in different scenarios.
Because of those two things: code maintainability and (mis)understanding testing in deeper level, some developers do great tests and enjoy good qa as resulting product gets better. And then there are developers that struggle with all of that and usually with their developing as well.
BlueOrigin50 16 hours ago [-]
A big reason companies constantly cycle between "developers should own QA" and "we need a dedicated QA team" is the E2E maintenance cliff.
It's easy for developers to own 50 E2E tests. But from what we see with our users, it's a nightmare when they scale to thousands of tests, and between releases, 10 features might have changed, suddenly causing 300+ tests to fail simultaneously in CI. No developer wants to spend three days debugging and updating the tests, so the test suite rots.
The bottleneck for large-scale E2E testing isn't creation, it's maintenance. We built Stably (https://stably.ai) specifically for this—we have a scalable cloud infrastructure that can process hundreds of concurrent Playwright test failures, understand them by stepping through the failure traces to debug the UI drift, and auto-fix the test scripts in minutes. Developer-led QA only works if you completely remove that massive maintenance tax.
eviks 11 hours ago [-]
> Note this isn’t because QA is finding issues — it’s because of the back and forth.
But it isn't? Finding and fixing are the major time sinks?
> You’re creating bad behavior for your engineers by not having them responsible for their quality.
What prevents you from maintaining responsibility if QA finds the bug? The thing you're throwing over is the investigation/finding, it doesn't follow that you should absolve the throwers
liampulles 1 days ago [-]
I work with someone who does great QA work. They know how to rip something apart, they understand the user's non-technical perspective and approach, and they understand what edge cases to look out and they have the actual equipment to test on different physical devices (and so on).
Most importantly, they have the diligence and patience to methodically test subtly different cases, which I frankly don't have.
On the question of whether QA slows things down, I have to ask: slows down what? Slows down releasing something broken? Why is that something to optimize for? We should always be asking how long it takes to release the right thing (indeed I'm most productive when I can close a ticket after concluding nothing is needed).
0x696C6961 1 days ago [-]
If all/most QA people were like this then no one would be complaining.
liampulles 1 days ago [-]
Sure, but this issue is not specific to QA. Any roles which you depend on with incompetent people occupying them will lead to issues and frustration.
monster_truck 19 hours ago [-]
A good QA engineer is worth many multiples of their weight in gold.
I often find that devs who think otherwise have never had to ship anything against a drop dead date, or definitively prove that a supplier has not met a contractual obligation (after the normal engineers have found it to be their problem in the first place).
Neywiny 19 hours ago [-]
I think I agree with your first sentence but opposite to the second. People I've met who don't prioritize QA (or test, reviews, etc) often only shipped with a mad dash towards a schedule is king deadline. Only on times where we can slow down and do things right do they appreciate the investment and its resulting dividends.
kevinbaiv 15 hours ago [-]
Feels like AI is flipping this question.
If code generation becomes cheap, then verification becomes the bottleneck.
In that world, QA isn’t going away — it becomes the core engineering function.
vdnkh 1 days ago [-]
I ship a very visible product which, when it breaks, generates a lot of social media angst (it's in the gaming adjacent space). So we try not to break things to the best of our ability. We have very few QA people and have whittled down that team over the past few years (DevOps was eliminated during the first round of layoffs ~2023).
This was painful at first but I do think it's the way to go. We found that too much manual QA incentivizes devs writing features to throw it over the fence - why should they test more if someone else is paid to do it? Devs need to feel the pain of writing tests if their code is hard to test, and they need to be held accountable when their code blows up in production. This feedback loop is valuable in the long run.
Same thing for test automation. Previously we shipped this over to our in-team DevOps people and they built complicated CI/CD setups. Losing them meant we needed to simplify our stack. Took a while and it slowed down feature development, but it was worth it. Of course you need leadership who understands this and dedicates time to building this out.
In defense of DevOps, I think the landscape for automation was poor a few years back. Jenkins and Teamcity are way too complex. Github Actions (for all its warts, and there are many) is much simpler. Our pipelines are also in their own CI/CD (CDK, CodeBuild) - infrastructure as code is the key to scaling.
We still have manual QA people to test things we can't automate. Usually this is for weird surfaces such as smart TVs, or for perceptual tests. I don't see this going away any time soon, but high levels of automation elsewhere drive down the need for "catch-all" manual testing.
crush_robo_1536 1 days ago [-]
I built products at Stripe as an engineer and never worked with an explicit QA team. Each team was responsible for the quality to a large extent.
That being said, QA is definitely an important aspect of software development - regardless of who owns this work. Imo, instead of having a QA engineer per team or a few teams, you should have a QA shape role (similar to AVE you mentioned) that oversees a large area like an Org and pushes hard to make sure quality standards are held high across teams.
It's also trivial for engineers these days to have great e2e automated test coverage with AI. We're actually building getlark.ai that helps engineers with this.
rfrey 23 hours ago [-]
The idea that test coverage equates to well-tested software is one of the unfortunate consequences of MBAs killing QA departments.
damn I'm legit not AI bro I think we're all just getting too skeptical these days. Just wanted to express my viewpoint and I mentioned what we're building because it seemed relevant, this isn't a sales motivated post.
donatj 1 days ago [-]
Yes. Without a doubt.
I worked with a QA team for the last fifteen years until last year when they laid them all off.
QA is a discrete skill in and of itself. I have never met a dev truly qualified to do QA. If you don't think this you have never worked with a good QA person. A good QA persons super power is finding weird broken interactions between features and layers where they meet. Things you would never think of in a million years. Any dingbat can test input validation, but it takes a truly talented person to ask "what if I did X in one tab, Y in another, and then Z, all with this exact timing so events overlap". I have been truly stunned at some of the issues QA has found in the past.
As for time, they saved us so much time! Unless your goal is to not test at all and push slop, they are taking so much work off your plate!
Beyond feature testing, when a customer defect would come in they would use their expertise to validate it, reproduce it, document the parameters and boundaries of the issue before it ever got passed on to dev. Now all that work is on us.
baal80spam 1 days ago [-]
> "what if I did X in one tab, Y in another, and then Z, all with this exact timing so events overlap"
As a QA: this bug will get downprioritised by PM to oblivion.
donatj 1 days ago [-]
If anyone should not exist, it's PMs.
I kid a little, I worked with some very good PMs when we did client work who made my life much easier. Working on a SaaS though, I find them generally less than useful.
bluGill 1 days ago [-]
Not anyplace that cares about quality.
where I work it is normally easier to fix things than deprioritize to oblivion. I can fix an issue, but priority puts a dozen people in a meeting.
brazzy 1 days ago [-]
Depends on what happens in that case, no?
If it messes up the UI until you refresh, yeah, I understand deprioritizing that.
If it causes catastrophic data corruption or leaks admin credentials, any sane PM would want that fixed ASAP.
gorgoiler 1 days ago [-]
The best QA isn’t just about finding bugs. It’s about bringing quality to the codebase: typing, better static analysis, linters, and useful libraries. In the other direction, it’s also about integrating into the release process by using integrating the what-goes-on / what-stays-in-beta decisions into quality’s approach to giving signal over any other part of the codebase.
Anything that involves gating bits of code, basically, and deciding whether to gate bits of code or not.
coffeebeqn 1 days ago [-]
The best QA is linting, CI and language features? Sure and automated testing at different levels too but this sounds like basic things the engineering org does
jbmsf 22 hours ago [-]
QA should exist.
QA should not be forced into an engineering or automation track because the incentives are wrong. You end up with test code becoming the goal and then it usually rots due to most QA not having the experience to create a codebase that scales.
I don't think the industry today understands how to treat QA and I think that leads to a lot of assumptions that it's not useful.
try-working 17 hours ago [-]
You need to scale up QA. You need a way for them to share their work with whichever harness you're using. QA will improve the quality of your code base immensely, and they will increase speed of development, and reduce costs.
zahlman 20 hours ago [-]
> Perhaps the strongest argument nowadays for QA is that with AI, automated verification is a leverage maximizer.
That sure is a sequence of words.
It comes across to me like the emphasized part is arguing against what it's supposed to be arguing for.
Or is the premise that the developers somehow can't run the AI verifier?
mobilene 1 days ago [-]
I was in testing for 17 years before moving back into Engineering. I have spent my time in Engineering leading teams to push quality left. But I think it's better to say "quality is a system" than to say "engineers own quality." What are you building into your SDLC that makes sure quality happens? Testing is just a part of that, and not even the biggest one.
issamG 1 hours ago [-]
Quality mindset should always exist, so should quality checkpoints.
To mature your system and practices, your code generator, code reviewer and quality checker, shouldn't be the same (people or agents)
spc476 22 hours ago [-]
At a previous job, the team I was on [1] had a dedicated QA engineer (unofficially---he was the only QA engineer that ever worked with our team). Before we got bought, we worked closely with the QA engineer. He had access to the source repo, could compile and test our stuff, and we constantly told him of new features we were working on to give him a heads up. During this time, we were our customer's (yes, we only worked with one customer, who was paying us seven figures per month for service) favorite vendor. Over the 10 years or so of this time, we had like two regressions hit production, and those were found during deployment and we could roll back immediately.
We then got bought out and new management put in. They siloed QA and made it impossible for us to even talk to them about what we were doing. Within a year, we had one deployment fail four times in a row and went from favorite vendor to "utter trash vendor we can't get rid of." Our QA engineer quit, as well as the rest of the team (I was the last to leave). I'm still surprised they still have that customer.
[1] We were the only team having to deal with SS7. It wasn't easy hiring programmers for it, and I think at the highest head count, we had like five members (including the manager when we had one [2], but not including QA, which was "officially" never a part of our team).
[2] If it was tough hiring programmers to deal with SS7, it was even harder to hire managers to deal with programmers dealing with SS7. I think for half the time I was there (over 10 years) I had no official manager and reported to a director or higher in the company.
james-clef 1 days ago [-]
Taking the shift left thing to it's logical conclusion, you have like everything automated and we risk forgetting we are building for people. At some point someone needs to use the thing with fresh eyes, and validate it is as advertised.
alsetmusic 17 hours ago [-]
Fuck off. I was a QA engineer and I helped prevent all kinds of unusual bugs that tests would never find because they required specific workflows to uncover them. Not that I was some genius who knew all the best workflows to test… but when I was randomly poking around and a crash or other unwanted behavior presented, I was methodical about working out the exact steps that led to the repro and I wrote actionable bug reports and collected useable environment data.
Also, I worked tech support for a number of years and I've watched all kinds of unexpected ways that people would interact with their devices. I always did all the weird stuff that most of us wouldn't consider when operating software:
"Why would someone drag every file to the desktop before they opened them?"
"I don't know, but I saw someone who always did that so I tried it."
didip 20 hours ago [-]
Yes, but should be supercharged with AI.
I’d like them to record their steps in details in videos and have AI dissected the steps into text descriptor that’s relevant to the code.
pragmatic 1 days ago [-]
Should Accounting exist?
Should Legal exist?
Should Facilities exist?
Surely your average employee could own each of these functions.
rglover 1 days ago [-]
If you have to ask, the answer is "yes."
9wzYQbTYsAIc 1 days ago [-]
> Have engineering own quality
The moment that happens it will either be re-outsourced to QA anyways or quality will become a question of licensing and bonding of professional engineers
returnInfinity 1 days ago [-]
I think social media companies don't need that
Enterprise software companies selling definitely need it. Customers ask was this tested? where is the test report?
stingraycharles 1 days ago [-]
It’s not even optional as soon as you’re getting close to any type of standards or compliance framework like SOC2 and the likes.
ChrisMarshallNY 19 hours ago [-]
Go ahead, and do without.
What could possibly go wrong?
chrysoprace 1 days ago [-]
Background: I was a software tester for 6.5 years; currently a software engineer, having worked with dedicated testers for about 5 years.
"QA" should exist regardless of whether you think dedicated software testing staff fit into your org. The whole team is responsible for assuring quality.
Dedicated software testers verify that the solution actually does what it's meant to do, and good software testers become deeply knowledgeable about the product and how features interact. They are ultimately a second pair of eyes, and should have a direct line to product owners or customers.
This can't be automated. The ongoing tests for verifying existing features continue to work without regression can and should be automated (throughout the dev process), adding generative AI to the human verification step is a recipe for disaster.
sz4kerto 1 days ago [-]
"Before I weigh in further, I’d like to make sure you’re familiar with the testing pyramid."
The testing pyramid is a par excellance SWE kool-aid. Someone wrote a logically-sounding blogpost about it many years ago and then people started regurgitating it without any empirical evidence behind it.
Many of us have realised that you need a "testing hourglass", not a "testing pyramid". Unit tests are universally considered useful, there's not much debate about it (also they're cheap). Integration tests are expensive and, in most cases, have very limited use. UI and API tests are extremely useful because they are testing whether the system behaves as we expect it to behave.
E.g. for a specific system of ours we have ~30k unit tests and ~10k UI/API tests. UI and API tests are effectively the living, valid documentation of how the system behaves. Those tests are what prevent the system becoming 'legacy'. UI and API tests are what enable large-scale refactors without breaking stuff.
Isolated QA should not exist because anything a QA engineer can do manually can be automated.
budro 5 hours ago [-]
> Isolated QA should not exist because anything a QA engineer can do manually can be automated.
I almost entirely agree. You'd certainly want a majority of QA responsibilities to fall to devs since that keeps the feedback loop tight. Also with the amount of compute we have you can trivially outpace a typical QA team with a single dev working on fuzz tests and property based tests. If you really still feel the need to outsource then there are cool tools like Antithesis [1] which let you trade money for compute in their complicated fuzz testing suite.
Where that falls apart imo is when your software interacts with hardware in any non-trivial way. Fuzz testing doesn't really work when your system is spinning big and heavy things for example. Real hardware that interacts with the world will always create this gap where the devs don't know how to fully test it. Your systems engineers are best equipped to handle this and ensure certain controls follow the right curves, but they typically don't have the bandwidth or are very inexperienced with software. I think a QA organization _can_ fill this gap and deliver value. But I'd almost always prefer hiring another 1 or 2 system engineers who can work towards this problem full time instead. It's much easier to train someone in software than it is to train them in motor control or sealed systems.
the vertical axis is not test type. It is would you run the test. At the bottom are deterministic fast tests for something completely unrelated to what you are working on - but they are so easy/fast you run them anyway 'just in case'. As you move up you get tests that you more and more want to aviod running. Tests that take a long time, tests that randomly fail when nothing is wrong, tests that need some settup, tests that need some expensive license (i can't think of more now but I'm sure there are).
You want to drive everything down as far as possible, but there is value in tests that are higher so you won't get rid of it. Just remember as soon you get to the 'make would run this test but I'm skipping it for now because it is annoying' line you need a seperate process to ensure the test is eventually run - you are trading off speed now for the risk that the test will find something and it is 10x harder to fix when you get there - when a test is run all the time you know what caused the failure and can go right there, while later means you did several things and have forgotten details. 10x is an estimate, depending where in your process you put it it could be 100 or even 1000 times harder.
weinzierl 1 days ago [-]
The testing pyramid comes from a time of desktop apps with no API and when UI tests were extremely expensive. I made 100% sense in that context, it never did in other contexts. Despite its omnipresence it had not made any sense for the vast majority of us in the past 25 years.
throwaway74628 1 days ago [-]
I’ve never encountered an initiative to “shift left” that wasn’t directly motivated by clunky, slow, unreliable and unmaintainable E2E tests. Failing earlier, especially pre-deployment, with targeted integration and contract testing is fabulous but it can’t replace rubber hitting road.
I’ve had quite a bit of success in helping my dev teams to own quality, devising and writing their own test cases, maintaining test pipelines, running bug hunts, etc. 90% of this can be attributed to treating developers as my customer, for whom I build software products which allow them to be more productive.
ivan_gammel 1 days ago [-]
> because anything a QA engineer can do manually can be automated.
Looks like you never worked with a decent QA team and do not understand the full scope of quality management. They have plenty of creative tasks not aligned with other roles.
fhd2 1 days ago [-]
> Isolated QA should not exist because anything a QA engineer can do manually can be automated.
Well, sort of maybe, but it's not always economical. For a normal web app - yeah I guess. Depends on the complexity of the software and the environment / inputs it deals with.
And then there's explorative testing, where I always found a good QA invaluable. Sure, you can also automate that to some degree. But someone who knows the software well and tries to find ways to get it to behave in unexpected ways, also valuable.
I would agree that solid development practices can handle 80% of the overall QA though, mainly regression testing. But those last 20%, well I think about those differently.
sz4kerto 1 days ago [-]
> And then there's explorative testing, where I always found a good QA invaluable.
Yes, I agree. We do this too. Findings are followed by a post-mortem-like process: - fix the problem
- produce an automated test
- evaluate why the feature wasn't autotested properly
sublinear 1 days ago [-]
> it's not always economical. For a normal web app - yeah I guess
What do you define as "normal"? I can't think of anything harder to test than a web app.
Even a seemingly trivial static HTML site with some CSS on it will already have inconsistencies across every browser and device. Even if you fix all of that (unlikely), you still haven't done your WCAG compliance, SEO, etc.
The web is probably the best example case for needing a QA team.
jamiemallers 1 days ago [-]
[dead]
philk10 1 days ago [-]
QA? Or testers?
winrid 22 hours ago [-]
QA should always exist. The question is just do you want to pay for them. Usually the preferred gaslighting is "without QA devs will do better testing", but it's always about money.
wavemode 1 days ago [-]
I don't see this as a "modern" vs "back in the day" thing.
The real reason many software orgs nowadays don't have QA is for the simple reason that it's slow. Everything in the consumer tech space is about rapid growth, and moving as fast as possible. Nobody cares very much that the software has bugs, what matters is whether it has users.
But outside of consumer tech, QA is a lot more common, since it matters a lot more that the software's logic is correct. (Speaking personally - I used to work for a genetics lab, and we had QA.) There are just different economic incentives involved.
thanksgiving 1 days ago [-]
Of course, anyone would agree that if wishes were fishes, QAs should not exist. We would all use agile with cross-functional teams. Every single team member can do any work that may be needed. All team members can take time off any time they need to because we have full coverage and the world is a beautiful place.
Of course, none of this is true in the real world.
For example, just last week we had a QA essentially bring down our web application on staging environment always reproducible with a sequence of four clicks. Follow the sequence with about the proper timing and boom, exception.
Should this have been caught before a single line of code was written? Yes, it should have been caught before any code was written. However, the reality is that it did not. Should this have been caught by some unit test? Integration test? End to end test? Code review? I'd argue as we barrel down a world of AI slop, we need to slow down more. We need QA more than ever.
adshotco 1 days ago [-]
[flagged]
blindriver 22 hours ago [-]
As someone who believed firmly that QA is a dying/dead profession, after moving over to AI coding over the last 6 months, I think coding is dead and QA and code reviews are what will remain after the aftermath of AI coding. Being able to test the output of AI to make sure it is doing what you want it to do is most of my job now.
sanex 20 hours ago [-]
I have been having the same thought. Since I've stepped back into code in the past month I have written maybe 3 or 4 lines. Deleted more than that for sure. Most of the time has been spent babysitting and manual testing.
andrewstuart 1 days ago [-]
Hard to believe people are asking this question in 2026.
Quality is something that takes dedicated focus and lots of work. Therefore it’s a job, not an afterthought or latest priority for someone whose primary focus is not quality.
oybng 1 days ago [-]
Like the majority of slop articles, the author is advertising his services.
darkerside 1 days ago [-]
So why would you outsource it to another team that isn't doing the actual work?
Ekaros 1 days ago [-]
Team doing the work should do QA so they only produce quality.
But on other hand those people can not often be trusted. As such you need a team that does checks again. Or alternatively they might have misunderstood something and thus produced incorrect system. Or there is some other fault in their thought process or reality. And system operates differently in more real scenario.
Juliate 1 days ago [-]
QA is actual work. Building the thing is actual work. Each is not "the" work, which is the task of the whole company.
QA perspective and focus is just different from the one of the team building the thing. It's precisely because of their detached perspective that they can do their work properly.
splatter9859 1 days ago [-]
As someone who holds a degree in Medicine besides his degree in Computer Science and works in medical devices, can I just please emphasise that QA should exist in this industry and is well deserved?
Sorry. No 'blue screens' or stack traces in my pacemaker or insulin pump, please.
stronglikedan 1 days ago [-]
> No 'blue screens' ... in my pacemaker or insulin pump
they have screens?!? /s
harlequinetcie 22 hours ago [-]
Yes
esafak 24 hours ago [-]
After AI reviewers, continuous AI QA is a possibility. If anyone is doing this I would like to hear your experience.
grzes 21 hours ago [-]
should cats drink milk?
haar 6 hours ago [-]
No, they're lactose intollerant - or do you mean kittens?
lightedman 1 days ago [-]
I've done software dev and manufacturing and engineering- QA is ESSENTIAL EVERYWHERE - ALWAYS.
I would ask anyone that thinks QA is unnecessary to spend a few days on an actual aerospace production line. Software and hardware get EXCESSIVE QA. For good reason.
fatbird 1 days ago [-]
There's a basic loop that goes on regardless:
1. define a requirement
2. implement the requirement
3. verify that the requirement was implemented
TDD was built around the idea that 1 and 3 could be unified in automated testing, and that's certainly true for a large part of it. But QA as a discrete role needs to exist because, beyond verifying that 2 was done correctly, they expose higher level bugs in 1, the requirements themselves.
It's virtually impossible to define requirements completely and without second order interactions that cause problems. QA is as effective at exposing assumptions and handwaving by the people who created the wireframes or the visual design as by the developers failing to test their own work.
And ideally, this leads to the cycle being virtuous: higher quality starts at the requirements phase, not the implentation phase. It's not just that QA should work closely with the engineers--the engineers need to work closely with UX and VD to ensure they fully understand the requirements. The incentives are aligned among all parties.
pluc 1 days ago [-]
In the age of AI of course not, AI is your QA
almostdeadguy 1 days ago [-]
I think the unstated (but highly prevalent) view among executives in large swathes of this industry is that they don't really care to spend any time or money on user testing or quality assurance, and if this role exists at companies it is usually under-compensated and straddles both these functions to have some party be accountable. It is sometimes a check on product teams and vision-driven executive teams who don't prototype/test their ideas (or empower their teams to do so), and sometimes a check on engineers and engineering managers who don't want to be accountable to gaps in quality.
Juliate 1 days ago [-]
If you care about: the consistency of your output, what you're selling to your customers, you have not much of a choice (than to control what you are shipping).
Not even mentioning the potential regulatory/market and legal consequences if you don't.
sys_64738 1 days ago [-]
Developers are liars. Why would I trust them to test their software?
Ekaros 1 days ago [-]
Has there been enough times it doesn't even compile on their machine? Which might be the absolute bare minimum...
salawat 1 days ago [-]
Or even just running their own tests...
somewhereoutth 1 days ago [-]
If your product is used by humans, then it needs to be tested by humans - this cannot be automated. Those humans can be your QA people, or your customers. Perhaps your customers are happy to be testers, perhaps not.
Unit tests are very expensive and return little value. Conversely, a (manual?) 'smoke test' is very cheap and returns great value - the first thing you do when updating a server for example is to check it still responds (and nothing has gone wrong in the deployment process), takes 2 seconds to do, prevents highly embarrassing downtime due to
a misconfigured docker pull or whatever.
baal80spam 1 days ago [-]
> Unit tests are very expensive and return little value
Why are unit tests very expensive? This goes against everything I know.
somewhereoutth 1 days ago [-]
Unit tests very roughly double the amount of effort required to make any meaningful change to your codebase. They are also require maintenance same as ordinary code - but the customer does not care in the slightest whether or not they pass. On the other side, they can only really tell you about low level bugs that you already expected, they cannot surface system level bugs - the actual hard bugs that cause problems for you and your customers.
Then there is the danger of thinking that green=all good, an example of 'automation bias' where we learn to trust the automation even as things go wrong.
As makers, it is also tempting to believe that [all] problems can be solved by making something (i.e. code), but actually many problems are not of that nature, and cannot be solved in that way.
baal80spam 1 days ago [-]
Thank you, that makes sense. What I meant was that today all unit tests are basically written by an AI so the "cost" is almost zero. Am I wrong?
somewhereoutth 9 hours ago [-]
Sorry yes. If an LLM written unit test fails, then it has to be determined whether the test was wrong or the code was wrong. This is an expense in human oversight, unless of course we believe that LLMs will get it right at a high enough rate that they can be left to code everything themselves completely automatically.
ivan_gammel 1 days ago [-]
Automated tests are no longer expensive to write and nowadays less expensive to execute.
Pannoniae 15 hours ago [-]
Sad to see you being downvoted, but you're exactly right. Well, almost - if you can afford to invest in a good integration test suite, that can catch many errors without requiring a human to regression-test every time.
At the same time, many quality attributes can't really be automatically tested, so automation shouldn't try to replace manual testing, it should be used to augment it.
hrmtst93837 23 hours ago [-]
If you think smoke testing is cheap, try doing it across concurrent releases and multiple envs, then see how fast the "just check it manually" step gets dropped when nobody owns it. Manual QA finds weird edge cases, but if that's all you have, regression will eat you alive.
Smoke tests and unit tests do different jobs, and calling unit tests low value is backwards unless your code almost never changes. If you skip automated tests because they are "expensive", you are betting release quality on hope and a lot of repeat work.
TJ_FLEET 1 days ago [-]
not really . but distill his knowledge into an OpenClaw.
When I got started in software, QA was already in a heavy decline. A mentor who had been a QA manager at Apple told me that being a QA person (in the industry) was once seen as a high-trust position. He was sad at what it became.
Their approach to QA was to have this be a optional thing the service people could do.
It worked surprisingly well, with the caveat that they never created regression tests.
The employer eventually switched someone from that team to establish these regression tests full time, but they had no programming experience and by the time I left no real progress was done in around 6 month. No idea what came of that, and a few years later they fired a lot of the team
More importantly, it is almost impossible for engineers to be as well incentivized to spend extra time exploring edge cases in something they already believe to work than to ship a feature on time.
Like everything else though, its contextual. Complexity of domain, surface area and age of product, depth of experience on team and consequences of failure are all so variable that there cannot be only one answer.
I have done it both ways for many years. I have worked on teams where QA is a frustrating nuisance, and teams where they were critical to success. I have worked on teams that did pretty good without them, and probably those were the highest throughput, most productive teams because the engineers were forced to own all the consequences - every bug they shipped was a production issue they were immediately forced to track down and resolve.
But those were very small teams, and eventually I was the only founding engineer left on the team and far too many mistakes by other people made it to my desk because I was the only person who could find them in review or track them down quickly in production. That was when I started hiring QA people.
Usually there either arent any in which case bugs get missed or there are 5 very cheap ones running mindless scripts who are standing in for the devs' inability or unwillingness to write decent automated tests but dont catch the really deep level thorny stuff.
Hard disagree. QAs' main tasks are acceptance testing and V&V. This means their work is to act as independent parties whose role is to verify that the results are indeed being delivered. Their importance is up there next to Product Owners.
The problem is that some QA roles are relegated to running the same old test scripts and do some bullshit manual exploratory tests that catch and assert nothing. It's the ultimate bullshit job as it's performative. This is clear in the way that LLM agents are seen as threatening the very existence of the role.
Personal liability and professional insurance works for all the actual “professions” in the US, to some extent, right?
It might be time to start the considerations for professional licensing for platform scale or commercially published software.
Medical devices and such are the only places I’d expect to see the need for certified products. By extension, in the new era, we really ought only expect certified software where we expect a duty to care from the software system (or any other assigned duty)
But there is also bad QA: The most worthless QA I was forced to work with, was an external company, where I, as developer, had to write the test sheet and they just tested that. Obviously they could not find bugs as I tested everything on the sheet.
My most impressive QA experience where when I helped out a famous Japanese gaming company. They tested things like press multiple buttons in the same frame and see my code crash.
This was my sole experience at the one place I worked with an internal QA team. They absolutely could never find bugs that devs missed, often mis-marked ones that didn't exist, and failed to find obvious edge cases that did exist.
Multiple devs fired because the CEO believed the QA over the engineering team; if they marked a bug as present, it was the engineer's fault for writing it. If they didn't catch a bug that made it to prod, it was the engineer's fault for not including it in the test plan. They represented nothing but red tape and provided no value.
Good QA sounds great! I'd love to know what that's like someday! It'd be great to have someone testing my code and finding breakages I missed! I'm only slightly (incredibly) bitter about my bad experience with its implementation.
This is an area where I expect AI to create a bimodal future. The smaller group of high quality QA people will now be able to offload the activity to agents instead of the QA drones. They'll still be worth their weight in gold, whereas the drones will be redundant.
The latter is much more high touch, but they're often worth their weight in gold. The former is kinda pointless.
Moreover, the best QAs would almost always try to be not QA - to shift into a better respected and better paid field.
I wish it werent so (hence my username) but there is a definite class divide between devs and QA and it shows up not just in terms of the pay packets but also who gets the boot in down times and who gets listened to. This definitely affects the quality of people.
I think it's overdue an overhaul much like the sysadmin->devops transition.
This might have been an Apple/MS thing, but we always had very technical QA people on the dev tools team. For example, the QA lead for the C++ compiler had written their own compiler from scratch and was an amazing contributor.
This was all unfortunate, and I agree in principle with having a separate test org, but in Windows the culture unfortunately seemed to be built around testers as second-class software developers.
As I said above, everyone has their own experiences but the QA folks I worked with at MS were fantastic.
Not sure if you're aware but Dave Plumber now has a really good YT channel [0] where he talks about MS back in those days. It's a fun walk down memory lane.
[0]: https://www.youtube.com/@DavesGarage
But yea, so many companies cheap their QA and then wonders why their QA sucks.
> Moreover, the best QAs would almost always try to be not QA - to shift into a better respected and better paid field.
That sort of seems circular. If they're not respected or paid well, of course most of the talented people would not want to remain in QA, and eventually you'd just have mediocre QA. That doesn't really give you any insight into whether high quality QA would be useful though.
(edit: I see now that's basically the point you're trying to make, so I guess we're in agreement)
This is a big asset in the current paradigm where a LLM agent can effectively implement behavior only when given a robust test suite to iterate against.
This is funny.
I don’t understand the reasoning here why QA shouldn’t be engineering.
Who watches the watcher, right?
That aside, the core idea is the same as the principles of independent audit, peer review, or even simply just specialization.
Red team / Blue team?
Even the military have police, right?
edit: ultimately, it comes down to the importance of independent audit, the builders and the breaker/fixers are very different groups in engineering.
Nor, in the case of QA, should the audit team be engineers trained to act and think like the ones who wrote the software. A fresh perspective is useful.
But in the long run, supervisory independence is the real deal. I know of a QA manager who shut down an entire factory's output until a major safety issue (that had been kicked down the road several times) was addressed. It took chutzpah, and serious power, to do that. The Dir. of Engrg. would NEVER have allowed it.
It's not a protected term in the US so it's jarring to those of us living where it is.
What I have seen before with QA is a queue like system. Hand to QA. get back in 3 days. Not the QA persons fault. It is the setup if the org that is wrong.
Imagine if you had a code review role and code review team and you had to wait to get your code OKed. We wouldn't put up with it.
QA teams seperate from Eng worked when the software was burned to CD rather than uploaded to s3.
I think the core of the issue is the weasel word "Good QA" instead of just "QA". You're underlining the importance of having someone in a team who has good product understanding and has good ownership skills. How many QAs fit that description? Not many, unfortunately. Some FANGs outright eliminated the role and replaced it with a mix of product owners and team ownership, and some QAs are just doing their 9-to-5 job going through their manual test scripts to verify it something meets a definition of done. What happens when you can have a LLM agent doing the same instantly as a step in a random CICD pipeline?
- QA is always the first thing companies outsource, with predictable results
- Companies either go the route or “separate QA org with separate management chain” or “have QA engineers report to dev managers”. I’ve seen serious misaligned incentives and toxic outcomes with both
- Frequent Slack messages at 4:15 PM on Friday - “hey they just merged the PR, we really need it tested before Monday stand up”
- QA becomes a de facto dumping ground for glue work that other teams don’t want to do. Senior QA ends up morphing into a de facto “responsibility without authority” project manager role
- There is zero internet “community” around QA the way there is for developers or designers. There is no Slashdot or Hacker News for QA and there never will be. Just a bazaar of book authors and consultants promoting themselves on LinkedIn
IMO the only thing that makes sense anymore is having good SDETs embedded in engineering teams.
This rings so many bells that it feels like some Buddhist festival. Apply the same approach to QA, Operations, and anything outside the actual product development: when this arrogance was shared between bosses and developers, all good on their side. Now with the AI, the arrogance is staying only on the bosses' side, and we have developers freaking out.
But I've worked at places with a whole spectrum of coverage in roles spanning Product Mgrs, Project Mgrs, BAs, QAs, production support level 1, production support level 2, etc. The one constant is whatever is missing or understaffed just ends up getting done by engineers.
Testing, on-call, Jira managing, requirements gathering with users, analysis, etc... all falls on to engineering. Then management gets even more wound up about dev productivity/velocity, etc.
Can I do your job? Yep. Can I also, at the same time, be the engineer that optimizes the IT systems? No - one of these jobs will suffer.
Give me the chance to understand your job, and I’ll replace as much of it as possible with code to do the same thing. But what it won’t do is have good judgement. It will make decisions on actual data - accurate data, erroneous data, it doesn’t care.
I think this is an interesting place to put “AI” - can it take input in the form of data and historical decisions, and come to a new decision from recent data? The same decision a human would?
1. Quality management is a continuous process that starts with product discovery and business requirements. Developers often assume that requirements are clear and move on to building the happy path. QA often explore requirements in depth and ask a lot of good questions.
2. QA usually have the best knowledge of the product and help product managers to understand its current behavior, when new requirements suggest to change it.
3. The same applies to product design. Good designer never leaves the team with a few annotated screens, supporting developers until the product is shipped. Design QA - the verification of conformance of implementation to design specs - can be done with QA team, which can assist with automation of design-specific tests.
4. Customer support - QA people are natural partners of customer support organization, with their knowledge of the product, existing bugs and workarounds.
And just a story: on one of my previous jobs recently hired QA engineer spotted number error in an All Hands presentation. That was an immediate buy-in from founders. :)
1. If you don't have Quality Assurance, then you have Quality Uncertainty.
2. QA is a full time job. If you offload the responsibility of QA to the engineers, then you're giving them 2 full time jobs. So unless they're working 16 hours days (even if they are tbh), you aren't assuring quality, you're compromising it.
Any process in an organization of size will have indicators that measure output. Those indicators should typically be paired with indicators that measure the quality of the output, to ensure product or service levels. That's the theory, and the genesis of 'quality management': whether you're measuring output code or breakfasts [1] or chemicals or widgets or medicine, you need to measure the quality of the output if there are any client specifications or expectations around the output. And there are very few cases where your customers will not have any specs or expectations around your product or service.
How you manage quality follows from those basics; it matters where you measure quality but it is so process dependent - earlier in the process lowers costs, but may not suffice to guarantee final quality - that quality management has to be designed around the specific process; balancing cost with benefit and requirements. How deep or specialized quality management becomes depends on the needs of the org, the size of the org, and the needs of the particular process.
This is why I'm skeptical about whether broad articles like this are beneficial overall. Why and how matter, and where's the foundational discussion behind why and how? Do folks not think at the organizational/business level? Maybe not everyone is a Sheryl Sandberg :-)
[1] https://en.wikipedia.org/wiki/High_Output_Management
Except I worked at a company with a QA department made up of entirely "Automated Verification Engineers" ... over a decade ago. And the head of the department had taught at a local QA school (so presumably other QA engineers learned that style of work from her also).
Good QA departments switched to this mode long before AI was even a thing! Maybe 90+% of QA departments didn't work that way pre-AI, but there certainly were ones that did!
A great QA can understand the features of a product quickly, turn those concepts into some sort of grid or matrix in their mind, then pull a bunch of paths and scenarios with estimated priorities and probabilities at a fast and efficient pace, all with great coverage. They can also identify features contradicting each other more quickly than product people.
I think a good QA is capable of being a great vibe coder nowadays, too. If you can write great test suites (write names only), agents nowadays are able to turn those specs into decent codebases. Comparatively, I know a lot of decent dev having not very good taste in testing, who often write overlapping tests or missing important paths.
What I find works best is when the Product Manager, Developer Lead, and QA Lead collaborate on requirements. The Product Manager designs the feature, the Developer focuses on technical design and architecture, and the QA Lead maps out user scenarios, integration points, and edge cases. Together, they define the requirements that constitute a successful feature.
When QA is involved early in the process, they understand and drive exactly how the feature should behave. This ensures that critical cases are considered and potential risks are handled before a single line of code is written.
It's terrifying without that support. The beancounter level mgt does not Get It that it's cheaper to have this kind of support now than much later in the product cycle.
If engineering owns quality, then engineering own all, up the chain. No need for anything and anybody.
Which is the AI pipe dream, really.
You are, in fact, with using AI, QA or coding or otherwise, externalizing services in hope the services will improve and costs will drop.
Let me know how that goes without HITLs.
That would put the damper on the pipe dream pretty quick. Probably more healthily than any data center ban could ever do.
If engineers were licensed, bonded, and liable, things would go very differently.
* speaking as having been a practicing software “engineer” for a decade
But since CEOs, or any other bosses, need to make a living, they will eat the liability in exchange for wealth, and leave engineering in the dust.
I think the same patterns can be applied here.
Caught in a landslide, no escape from reality ...
:-)
I've been places where devs have no idea what the product-as a whole-does. They just work on the feature of the sprint and throw the code over the wall. Their testing consists of if: it compiled==it passed. They have no idea how to even start actually testing if it's not on the happy path.
I been in places where the code accomplished the spec, but in the most lazy way possible so it appeared to work but was useless outside of what the tests looked for.
I knew one QA guy that was amazing but was so overloaded because management kept hiring "cheap" QA that were actively making his life worse.
I'm a tech writer right now at a tech company and a dev just sent over an LLM generated "doc" that's referring to things that don't exist.
Neither management nor dev has learned anything from Therac-25. QA is hard.
Test automation is not same coding that developers do. I don’t mean from quality point but they have completely different set of things that matter. Unlike writer here may think while it is important to have tests that pass or fail fast the performance is not one of the most important characteristics. You see tests are run less often than the code and not all tests need to run every time or even every day. One of the most important characteristics of a good test is something developers easily overlook. It is maintainability. When code is written then it’s tested and then it sits in codebase until refactored. Tests don’t get this. Tests work on environment that is under constant change (especially in shift left) that makes maintainability and abstraction layers much more important with test automation code than with developing a feature.
Something to note that some developers do not understand testing at all. They may think it’s all about getting tests passed or getting software broken. It’s not either of those. It’s just checking that software works like expected in different scenarios.
Because of those two things: code maintainability and (mis)understanding testing in deeper level, some developers do great tests and enjoy good qa as resulting product gets better. And then there are developers that struggle with all of that and usually with their developing as well.
It's easy for developers to own 50 E2E tests. But from what we see with our users, it's a nightmare when they scale to thousands of tests, and between releases, 10 features might have changed, suddenly causing 300+ tests to fail simultaneously in CI. No developer wants to spend three days debugging and updating the tests, so the test suite rots.
The bottleneck for large-scale E2E testing isn't creation, it's maintenance. We built Stably (https://stably.ai) specifically for this—we have a scalable cloud infrastructure that can process hundreds of concurrent Playwright test failures, understand them by stepping through the failure traces to debug the UI drift, and auto-fix the test scripts in minutes. Developer-led QA only works if you completely remove that massive maintenance tax.
But it isn't? Finding and fixing are the major time sinks?
> You’re creating bad behavior for your engineers by not having them responsible for their quality.
What prevents you from maintaining responsibility if QA finds the bug? The thing you're throwing over is the investigation/finding, it doesn't follow that you should absolve the throwers
Most importantly, they have the diligence and patience to methodically test subtly different cases, which I frankly don't have.
On the question of whether QA slows things down, I have to ask: slows down what? Slows down releasing something broken? Why is that something to optimize for? We should always be asking how long it takes to release the right thing (indeed I'm most productive when I can close a ticket after concluding nothing is needed).
I often find that devs who think otherwise have never had to ship anything against a drop dead date, or definitively prove that a supplier has not met a contractual obligation (after the normal engineers have found it to be their problem in the first place).
If code generation becomes cheap, then verification becomes the bottleneck.
In that world, QA isn’t going away — it becomes the core engineering function.
This was painful at first but I do think it's the way to go. We found that too much manual QA incentivizes devs writing features to throw it over the fence - why should they test more if someone else is paid to do it? Devs need to feel the pain of writing tests if their code is hard to test, and they need to be held accountable when their code blows up in production. This feedback loop is valuable in the long run.
Same thing for test automation. Previously we shipped this over to our in-team DevOps people and they built complicated CI/CD setups. Losing them meant we needed to simplify our stack. Took a while and it slowed down feature development, but it was worth it. Of course you need leadership who understands this and dedicates time to building this out.
In defense of DevOps, I think the landscape for automation was poor a few years back. Jenkins and Teamcity are way too complex. Github Actions (for all its warts, and there are many) is much simpler. Our pipelines are also in their own CI/CD (CDK, CodeBuild) - infrastructure as code is the key to scaling.
We still have manual QA people to test things we can't automate. Usually this is for weird surfaces such as smart TVs, or for perceptual tests. I don't see this going away any time soon, but high levels of automation elsewhere drive down the need for "catch-all" manual testing.
That being said, QA is definitely an important aspect of software development - regardless of who owns this work. Imo, instead of having a QA engineer per team or a few teams, you should have a QA shape role (similar to AVE you mentioned) that oversees a large area like an Org and pushes hard to make sure quality standards are held high across teams.
It's also trivial for engineers these days to have great e2e automated test coverage with AI. We're actually building getlark.ai that helps engineers with this.
I worked with a QA team for the last fifteen years until last year when they laid them all off.
QA is a discrete skill in and of itself. I have never met a dev truly qualified to do QA. If you don't think this you have never worked with a good QA person. A good QA persons super power is finding weird broken interactions between features and layers where they meet. Things you would never think of in a million years. Any dingbat can test input validation, but it takes a truly talented person to ask "what if I did X in one tab, Y in another, and then Z, all with this exact timing so events overlap". I have been truly stunned at some of the issues QA has found in the past.
As for time, they saved us so much time! Unless your goal is to not test at all and push slop, they are taking so much work off your plate!
Beyond feature testing, when a customer defect would come in they would use their expertise to validate it, reproduce it, document the parameters and boundaries of the issue before it ever got passed on to dev. Now all that work is on us.
As a QA: this bug will get downprioritised by PM to oblivion.
I kid a little, I worked with some very good PMs when we did client work who made my life much easier. Working on a SaaS though, I find them generally less than useful.
where I work it is normally easier to fix things than deprioritize to oblivion. I can fix an issue, but priority puts a dozen people in a meeting.
If it messes up the UI until you refresh, yeah, I understand deprioritizing that.
If it causes catastrophic data corruption or leaks admin credentials, any sane PM would want that fixed ASAP.
Anything that involves gating bits of code, basically, and deciding whether to gate bits of code or not.
QA should not be forced into an engineering or automation track because the incentives are wrong. You end up with test code becoming the goal and then it usually rots due to most QA not having the experience to create a codebase that scales.
I don't think the industry today understands how to treat QA and I think that leads to a lot of assumptions that it's not useful.
That sure is a sequence of words.
It comes across to me like the emphasized part is arguing against what it's supposed to be arguing for.
Or is the premise that the developers somehow can't run the AI verifier?
We then got bought out and new management put in. They siloed QA and made it impossible for us to even talk to them about what we were doing. Within a year, we had one deployment fail four times in a row and went from favorite vendor to "utter trash vendor we can't get rid of." Our QA engineer quit, as well as the rest of the team (I was the last to leave). I'm still surprised they still have that customer.
[1] We were the only team having to deal with SS7. It wasn't easy hiring programmers for it, and I think at the highest head count, we had like five members (including the manager when we had one [2], but not including QA, which was "officially" never a part of our team).
[2] If it was tough hiring programmers to deal with SS7, it was even harder to hire managers to deal with programmers dealing with SS7. I think for half the time I was there (over 10 years) I had no official manager and reported to a director or higher in the company.
Also, I worked tech support for a number of years and I've watched all kinds of unexpected ways that people would interact with their devices. I always did all the weird stuff that most of us wouldn't consider when operating software:
"Why would someone drag every file to the desktop before they opened them?"
"I don't know, but I saw someone who always did that so I tried it."
I’d like them to record their steps in details in videos and have AI dissected the steps into text descriptor that’s relevant to the code.
Should Legal exist?
Should Facilities exist?
Surely your average employee could own each of these functions.
The moment that happens it will either be re-outsourced to QA anyways or quality will become a question of licensing and bonding of professional engineers
Enterprise software companies selling definitely need it. Customers ask was this tested? where is the test report?
What could possibly go wrong?
"QA" should exist regardless of whether you think dedicated software testing staff fit into your org. The whole team is responsible for assuring quality.
Dedicated software testers verify that the solution actually does what it's meant to do, and good software testers become deeply knowledgeable about the product and how features interact. They are ultimately a second pair of eyes, and should have a direct line to product owners or customers.
This can't be automated. The ongoing tests for verifying existing features continue to work without regression can and should be automated (throughout the dev process), adding generative AI to the human verification step is a recipe for disaster.
The testing pyramid is a par excellance SWE kool-aid. Someone wrote a logically-sounding blogpost about it many years ago and then people started regurgitating it without any empirical evidence behind it.
Many of us have realised that you need a "testing hourglass", not a "testing pyramid". Unit tests are universally considered useful, there's not much debate about it (also they're cheap). Integration tests are expensive and, in most cases, have very limited use. UI and API tests are extremely useful because they are testing whether the system behaves as we expect it to behave.
E.g. for a specific system of ours we have ~30k unit tests and ~10k UI/API tests. UI and API tests are effectively the living, valid documentation of how the system behaves. Those tests are what prevent the system becoming 'legacy'. UI and API tests are what enable large-scale refactors without breaking stuff.
Isolated QA should not exist because anything a QA engineer can do manually can be automated.
I almost entirely agree. You'd certainly want a majority of QA responsibilities to fall to devs since that keeps the feedback loop tight. Also with the amount of compute we have you can trivially outpace a typical QA team with a single dev working on fuzz tests and property based tests. If you really still feel the need to outsource then there are cool tools like Antithesis [1] which let you trade money for compute in their complicated fuzz testing suite.
Where that falls apart imo is when your software interacts with hardware in any non-trivial way. Fuzz testing doesn't really work when your system is spinning big and heavy things for example. Real hardware that interacts with the world will always create this gap where the devs don't know how to fully test it. Your systems engineers are best equipped to handle this and ensure certain controls follow the right curves, but they typically don't have the bandwidth or are very inexperienced with software. I think a QA organization _can_ fill this gap and deliver value. But I'd almost always prefer hiring another 1 or 2 system engineers who can work towards this problem full time instead. It's much easier to train someone in software than it is to train them in motor control or sealed systems.
[1] https://antithesis.com/
the vertical axis is not test type. It is would you run the test. At the bottom are deterministic fast tests for something completely unrelated to what you are working on - but they are so easy/fast you run them anyway 'just in case'. As you move up you get tests that you more and more want to aviod running. Tests that take a long time, tests that randomly fail when nothing is wrong, tests that need some settup, tests that need some expensive license (i can't think of more now but I'm sure there are).
You want to drive everything down as far as possible, but there is value in tests that are higher so you won't get rid of it. Just remember as soon you get to the 'make would run this test but I'm skipping it for now because it is annoying' line you need a seperate process to ensure the test is eventually run - you are trading off speed now for the risk that the test will find something and it is 10x harder to fix when you get there - when a test is run all the time you know what caused the failure and can go right there, while later means you did several things and have forgotten details. 10x is an estimate, depending where in your process you put it it could be 100 or even 1000 times harder.
I’ve had quite a bit of success in helping my dev teams to own quality, devising and writing their own test cases, maintaining test pipelines, running bug hunts, etc. 90% of this can be attributed to treating developers as my customer, for whom I build software products which allow them to be more productive.
Looks like you never worked with a decent QA team and do not understand the full scope of quality management. They have plenty of creative tasks not aligned with other roles.
Well, sort of maybe, but it's not always economical. For a normal web app - yeah I guess. Depends on the complexity of the software and the environment / inputs it deals with.
And then there's explorative testing, where I always found a good QA invaluable. Sure, you can also automate that to some degree. But someone who knows the software well and tries to find ways to get it to behave in unexpected ways, also valuable.
I would agree that solid development practices can handle 80% of the overall QA though, mainly regression testing. But those last 20%, well I think about those differently.
Yes, I agree. We do this too. Findings are followed by a post-mortem-like process: - fix the problem - produce an automated test - evaluate why the feature wasn't autotested properly
What do you define as "normal"? I can't think of anything harder to test than a web app.
Even a seemingly trivial static HTML site with some CSS on it will already have inconsistencies across every browser and device. Even if you fix all of that (unlikely), you still haven't done your WCAG compliance, SEO, etc.
The web is probably the best example case for needing a QA team.
The real reason many software orgs nowadays don't have QA is for the simple reason that it's slow. Everything in the consumer tech space is about rapid growth, and moving as fast as possible. Nobody cares very much that the software has bugs, what matters is whether it has users.
But outside of consumer tech, QA is a lot more common, since it matters a lot more that the software's logic is correct. (Speaking personally - I used to work for a genetics lab, and we had QA.) There are just different economic incentives involved.
Of course, none of this is true in the real world.
For example, just last week we had a QA essentially bring down our web application on staging environment always reproducible with a sequence of four clicks. Follow the sequence with about the proper timing and boom, exception.
Should this have been caught before a single line of code was written? Yes, it should have been caught before any code was written. However, the reality is that it did not. Should this have been caught by some unit test? Integration test? End to end test? Code review? I'd argue as we barrel down a world of AI slop, we need to slow down more. We need QA more than ever.
Quality is something that takes dedicated focus and lots of work. Therefore it’s a job, not an afterthought or latest priority for someone whose primary focus is not quality.
But on other hand those people can not often be trusted. As such you need a team that does checks again. Or alternatively they might have misunderstood something and thus produced incorrect system. Or there is some other fault in their thought process or reality. And system operates differently in more real scenario.
QA perspective and focus is just different from the one of the team building the thing. It's precisely because of their detached perspective that they can do their work properly.
Sorry. No 'blue screens' or stack traces in my pacemaker or insulin pump, please.
they have screens?!? /s
I would ask anyone that thinks QA is unnecessary to spend a few days on an actual aerospace production line. Software and hardware get EXCESSIVE QA. For good reason.
1. define a requirement 2. implement the requirement 3. verify that the requirement was implemented
TDD was built around the idea that 1 and 3 could be unified in automated testing, and that's certainly true for a large part of it. But QA as a discrete role needs to exist because, beyond verifying that 2 was done correctly, they expose higher level bugs in 1, the requirements themselves.
It's virtually impossible to define requirements completely and without second order interactions that cause problems. QA is as effective at exposing assumptions and handwaving by the people who created the wireframes or the visual design as by the developers failing to test their own work.
And ideally, this leads to the cycle being virtuous: higher quality starts at the requirements phase, not the implentation phase. It's not just that QA should work closely with the engineers--the engineers need to work closely with UX and VD to ensure they fully understand the requirements. The incentives are aligned among all parties.
Not even mentioning the potential regulatory/market and legal consequences if you don't.
Unit tests are very expensive and return little value. Conversely, a (manual?) 'smoke test' is very cheap and returns great value - the first thing you do when updating a server for example is to check it still responds (and nothing has gone wrong in the deployment process), takes 2 seconds to do, prevents highly embarrassing downtime due to a misconfigured docker pull or whatever.
Why are unit tests very expensive? This goes against everything I know.
Then there is the danger of thinking that green=all good, an example of 'automation bias' where we learn to trust the automation even as things go wrong.
As makers, it is also tempting to believe that [all] problems can be solved by making something (i.e. code), but actually many problems are not of that nature, and cannot be solved in that way.
At the same time, many quality attributes can't really be automatically tested, so automation shouldn't try to replace manual testing, it should be used to augment it.
Smoke tests and unit tests do different jobs, and calling unit tests low value is backwards unless your code almost never changes. If you skip automated tests because they are "expensive", you are betting release quality on hope and a lot of repeat work.