It’s not that testing is bad, it’s that you’re starting to test it too late.

Oh I agree, but the OP, and many here who lament "big business" are aghast that there is use of, god forbid, an actual test plan (test plan 9 from outer space, presumably).

I realise that sometimes people think that I'm a bit brash, or loud (almost 20 years in Scotland - I can't help it, this is just how I talk haha) or opinionated. I probably am all of those things. But in the last 3 years, watching WordPress's explosion, it pains/worries me that so many take stock from their singular viewpoint.

What started out as Guidelines, suddenly became Standards, before morphing into Evangelism, and now close to Rhetoric (e.g. Thou shalt upgrade the very second a new WordPress version is release without question or business case); and those who still view and act upon these things as Guidelines, as after all it's open source software, are viewed upon with distain and distrust.

1-click-upgrade is wonderful, truly bloody awesome, but it's also had a negative effect, where a greater number of people put their faith in the testing thats already been done by the community, and just click-it-and-see. There is always a balance between making it easy for someone, and making them lazy.

Which has added another problem. The original thought process was that there were people out there who didn't know how to upgrade, and now they can. Well now we have people who know how to upgrade, but are fed up of doing so. People choosing not to upgrade simply because "meh, there'll be another one in a week or so anyway". Security wise (just to bring it back to Jeffro's post), we're making people very numb to updates, something that is not good at all.

If you wait until a “release” before you begin testing, then of course you’re going to have problems. Realistically, you should be under a continuous cycle of testing and development at the same time.

why is your “dev” site not running trunk? You’d get at least a month or two head start on any testing and new development for upcoming changes, and by the time release rolled around, you’d be already tested and up and running.

At the risk of pouring salt into old wounds, I have a simple answer to that: Capital_P_Dangit()
(plus the clusterf**k that was 2 years of bbPress and BackPress of-course).

Faith in the Alpha > Beta > RC > release system no longer presides in my company, or my clients. When the core team (hi Matt) are able to add in features after the RC to rewrite file paths on a marketing whim, then what's the point of Beta's for people like us? When did "feature freeze" mean "feature freeze, unless we want to add some"

I mean, the current Beta, not alpha or nightly but Beta for 3.3's admin section cant be navigated in IE7. Not a small feature, but the Admin doesn't work, at all, in IE7. How in the blazes of all that is Joss Whedon did that hit Beta? If only there was a test plan (9 from outer space). Just to clarify, every single person on the core team and who works on the releases for Automattic and the dev team, every single one of them didn't test it on IE7, or even IE8 and hit compatible mode.

To quote Wikipedia here:

Beta... It generally begins when the software is feature complete. The focus of beta testing is reducing impacts to users, often incorporating usability testing. The process of delivering a beta version to the users is called beta release and this is typically the first time that the software is available outside of the organization that developed it.

And yet we've got a Beta version in 3.3 that NO features work ing IE7, and no-one tested it. Not in-depth testing btw, but no-one even loaded it once or they would have seen the menu not working. Bluntly, with Beta's like this, and releases with the likes of Capital_P_Dangit(), running trunk on our dev site sometimes feels like farting against thunder - something must smell terrible for us to even take notice!

Put frankly, and with all due respect to someone who I've admired for a long time Otto, the state of WordPress beta's over the last year (and 4 releases) have been so far from what I'd call a beta, that we only use it to give us a heads up of anything truly huge thats changed.

I've been saying it for years now, and people really ridicule me for it, but maybe we need a slightly better ratio of Project Managers to Happyness Gardeners, cos right now for the WP core it's 6 developers, 0 Project Managers and 1 community manager employed by WP.org (who tested the Beta with 12 people on macs & 8 on PCs, but everyone used FF or Chrome). Frankly, if thats a good test plan, then can the last person here switch off the lights please?