Anachronic wrote:I'd be interested in any examples or lessons you have about Agile going wrong. I took an Agile class right at the end of getting an associate's certificate in software dev from BCIT, and it was the funnest class of the whole program. My previous classes had touched on Agile a little bit but seemed more rooted in Waterfall, so it was kinda sad to be introduced to something pitched as a far superior system right at the end. Knowing what I know now, if I could do it again, I would probably have taken that class first.
This is a topic I would absolutely love to discuss in tons of specifics, but I don't know if I want to start writing too thorough a reply, since I suspect that I'll be asleep soon.
I've worked in 3 outfits that were self-proclaimed "agile" shops. What these lead engineers really meant when they said they adhered to an agile process, though, was that they had no process at all, and everything was just wackyshack. Any agile methodology done correctly is not everyone doing whatever they want whenever and however they want to do it. It's not code whatever you feel like and jam it into production whenever. The problem I've seen most is that many developers think that this free-form madness is
agile, and the term "agile" is just used as a code word for undisciplined and incompetent.
The whole point of agile processes is to ease some of onerous constraints imposed by heavyweight methodologies, and to shorten the amount of time between releases, so an organization can better adapt to changing business needs. It's about reasonable shortcuts that can make for more efficient operations on certain types of projects and with certain types of teams. As such, even when done right, agile processes need experienced, talented personnel to work properly.
What most software teams don't realize is that agile actually requires much more
discipline, much more
experience, and much more
talent to pull off than heavyweight methodologies. Responsibility for doing things right is shifted away from compliance with formal mandates and is placed in the hands of individual developers. If any of your individual developers are idiots, screw-ups, or just inexperienced, it's not going to work, because they're not going to handle things correctly themselves, and they're going to produce poor results. Heavyweight processes are actually far better for teams with junior personnel or less-than-elite talent bases, because they impose the discipline. They keep everything on track. They provide the framework that rank and file developers need
to be productive. When that framework is removed, only the best kinds of software engineers can keep their heads above water.
Furthermore, lots of "things" that people call "agile development methodolgies" are not, I repeat, absolutely are not
software development methodologies. Scrum is a prime example of this. If you ask a chief engineer about their software engineering methodology and they start talking about scrum, you've got a problem. Scrum is a project management framework. It's a mechanism for managing personnel, keeping track of what they're doing, and facilitating communication on teams - it's content agnostic, and has nothing to do with building software. Employing scrum is not imposing any kind of order or framework for building software. It's tangentially related, but that's as close as it comes. Scrum has nothing to do with determining technical designs or articulating business requirements or QA'ing new releases.
I worked in precisely such a place, where management would say, "We use scrum. We're agile software developers!" Meanwhile, there's no method for scoping sets of deliverables, I never saw a single formal, business requirement, no one designs anything beyond thinking about it as they code it, and you generally just had a disorganized gaggle of coders hanging around their cubicles, coding whatever, moving it to production whenever, without any practical oversight of engineering framework to govern what was going on. Needless to say, it was a dumpster fire. The codebase was an embarrassment, virtually nothing worked, every minor feature took ten times longer than it should've to implement, the architecture was a byzantine, incomprehensible nightmare-mess. It was a really difficult environment to navigate.
But we were having our daily standup meetings! We were spending hours on Fridays delivering our weekly work presentations! We were checking all the scrum boxes, doing an A-OK great job! No, we weren't, because the end product was a disaster, everyone was fighting everyone all the time, the environment was bitterly toxic (because of nothing working, leading to negligible sales, leading to management freak-outs, leading to inordinate stress, leading to more childish behavior) - we were doing scrum, alright, but that has nothing to do with building good software.
Management, of course, was up in arms at all times about the problems that they were experiencing, but when a consultant would try to discuss the fundamental nature of the problems the organization had (the lack of any engineering control), it would be me with a statement like, "We are using scrum. That is never going to change." I'm not talking about abandoning scrum for project management - I'm talking about imposing engineering rigor. I'm talking about basic code reviews. I'm talking about formalized designs. I'm talking about no more single developers doing completely unchecked releases on their own.
The whole point of this example is just to convey that "agile", when it's used as a code word for no engineering rigor at all, is a recipe for disaster. Whenever you have a middle manager that just attends a scrum course and is then put in charge of a software shop, this is going to be the result. And this is what a lot of agile is, out in the world.
I find it really funny that waterfall is used as a dirty word, among educators especially, when it comes to software engineering. Whether a team embraces it or not, you cannot build software without waterfall. You can't write code without figuring out a design. You can't figure out a design without understanding how the system needs to work. You can't figure out how a system needs to work without understanding its requirements. You can't understand its requirements without having a general idea of what you're aiming to make.
As such, nearly all methodologies are, themselves, running through all the waterfall steps! The only difference is that they're doing it in smaller iterations. I kinda' hate the term agile, because it's such a corporate-y sort of buzz word. It's inherently good, because of the word's general meaning. Of course you want to be agile! Why the hell wouldn't you? What kind of idiot wouldn't want to be agile? Any argument against agile immediately sounds dumb, and the middle manager that doesn't know anything other than what his 3 week agile seminar told him retorts with a variety of straw man arguments about team cohesion and other stupid lunacy.
Nearly all practical implementations of agile should just be called "iterative waterfall with less formality", because that's what they really are. If we called them that, instead of agile, it'd be much easier to have frank and sensible conversations about the merits of agile and BDUF approaches on different projects and different environments. Sometimes agile is the right way to go! Sometimes BDUF is. Every experienced engineer I've worked with in a BDUF environments acknowledges that, for some sorts of projects, agile approaches yield benefits. But the standard bearers on the agile side present their approach as infallible and universally applicable. It's just because they can sell more seminar sessions that way.
So, that's my nutshell take on real world experiences involving agile shops. So many of the people who are talking agile vs. heavyweight don't even understand what the two terms mean with respect to software engineering, and these people are often the senior management making the decisions, which is a recipe for disaster. Ultimately, organizations are about personnel. They need talented personnel making decisions they understand, and they need senior VPs who, despite their titles and salaries, can sometimes acknowledge that they don't know about a domain and can delegate decision making authority to someone who does. If that basic criterion isn't satisfied, little else really matters.
Now, I don't want this to sound bitter, here! Quite the opposite. The kinds of situations described above are what software engineering consultants are made for, and these are the environments where the biggest, most positive impacts can be made! And you only need two things to turn the worst disaster into a technical success -
1. An organization that's willing to improve.
2. The right consultant.
So I got into some territory that sits conceptually above the agile vs. heavyweight discussion, just to highlight what an organization needs to understand this decision, and how many outfits go off the rails. Above all else, when considering agile vs. heavyweight for a project, understand what it is you're deciding on, because you'd be shocked how many industry professionals with 6 figure salaries really have no idea what agile means with respect to software development.