W.

Why Software Projects Miss Deadlines: The Known and the Unknown

By In Technology

Software development deadlines. They haunt our dreams, ruin our weekends, and yet, somehow, we’re always shocked when we miss them. From tiny web projects to colossal AAA games, the software industry has an infamously bad reputation for blowing past deadlines and budgets. But how do software engineers manage to be worse than, say, the folks who bid on highway systems?

The Known Unknowns: We Sort of Know What We’re Dealing With

Before we tackle the real villain, let’s touch on the “known unknowns.” These are the things we anticipate will pose questions or challenges. When a project kicks off, developers huddle with domain experts to map out the grand plan. This early phase is rife with questions:

  1. Architectural Trade-offs: “Should we use Framework X for rapid development? Will it scale if our user base explodes?”
  2. Technical Feasibility: “Is there a library that does what we need? Do we have to build it from scratch?”
  3. Project Management Triangle: “Do we write quick-and-dirty code now, knowing we’ll refactor later, or aim for scalability and security from the get-go?”

These questions are predictable and somewhat manageable. We know they’ll pop up and can plan (read: guesstimate) for them. But then come the…

Unknown Unknowns

Unknown unknowns are the sneaky, unpredictable gremlins that can throw a project into chaos. They are the reason even the most detailed plans can blow up spectacularly.

Examples of Unknown Unknowns

So, what do these elusive unknown unknowns look like in practice? Here are a few examples:

  • The Market/Product Changes: This is the most common. When you’re building software you are building it for an end user with a purpose in mind. But you can never know FOR SURE that you’re building the right thing. You will only really know what needs to change when you first watch a user play with it. That’s why it’s so important to get it into the hands of real users as quickly as possible.
  • Emerging Technology Changes: A new version of a critical framework or library is released midway through your project, rendering your current approach obsolete.
  • Unforeseen Integration Issues: An external API you depend on changes or is deprecated without warning, requiring significant rework. This literally happened to me this week, and I spent two days fixing a problem that worked the first time with the old API. And this was a tiny python script and a Make.com workflow… not some giant HIPAA compliant stack that has five engineers working on it. That kind of change could mess up a project by months.
  • Unexpected User Behavior: During testing, users interact with your software in completely unanticipated ways, uncovering bugs and usability issues that require substantial fixes. Think of yourself as an explorer here; learning new ways your system can be used, and how users can f*@&k it up.
  • Regulatory Changes: New laws or industry regulations come into effect, necessitating changes to your software to remain compliant.
  • Team Changes: Key team members leave the project, taking with them critical knowledge and slowing down progress as new members are onboarded.

The Exploratory Nature of Software Development

Software development often feels like your are Indiana Jones navigating a jungle with a half-baked map. Sure, you have a rough idea of where you’re going, but the terrain can change unexpectedly. When developers estimate how long a task will take, they’re picturing a perfect world scenario. It’s tough to convey to managers or clients—who often see programming as a kind of magic—why one feature is a cakewalk and another is a nightmare.

Agile vs. Waterfall: Why Flexibility Matters

This is why traditional waterfall methodologies, which demand every detail be nailed down upfront, often fail in software development. Designing everything on paper without accounting for unforeseen problems is a recipe for disaster. Agile methodologies, with their iterative and flexible nature, help mitigate the chaos of unknown unknowns by allowing for continuous reassessment and adaptation.

Agile is by no means perfect though. Agile projects fail for different reasons: scope creep, running out of budget, not being disciplined in prioritizing features, not launching quick enough, over-engineering, and plain old building bad software.

The Myth of Big Design Up Front

Big design upfront (BDUF) robs software of its most powerful trait: adaptability. The ability to iterate, update, and improve on the fly is crucial. Agile approaches embrace this, understanding that no one can predict every challenge. The flexibility to pivot based on real-world feedback and emerging obstacles is what keeps projects on track—relatively speaking.

Making Software Development More Predictable

While we can’t eliminate all the unknowns, there are strategies to make software development more predictable. The problem is that most of this stuff makes you slow down. There is always a trade off. So enjoy!

  1. LAUCH EARLY AND OFTEN: Get your system into the hands of real users as soon as possible. If you’re not embarrassed to let users play with it, you’ve launched too late. I just wrote about this.
  2. Adopt Agile Methodologies: Agile frameworks like Scrum and Kanban promote iterative development, continuous feedback, and flexibility, making it easier to adapt to changes and unforeseen issues.
  3. Frequent Check-ins and Reviews: Regularly scheduled reviews, stand-ups, and retrospectives help catch issues early, before they spiral out of control.
  4. Automated Testing: Implement a robust suite of automated tests to catch regressions and integration issues early in the development process.
  5. Continuous Integration/Continuous Deployment (CI/CD): Use CI/CD pipelines to automate the deployment process, ensuring that new code integrates smoothly with existing systems.
  6. Risk Management: Proactively identify potential risks and develop contingency plans to address them if they arise.
  7. Maintain a Flexible Architecture: Design systems with modularity and scalability in mind, allowing easier adjustments as requirements evolve.
  8. Clear Documentation: Ensure comprehensive and up-to-date documentation so that new team members can quickly get up to speed and existing members can easily reference past decisions.
  9. Cross-Training Team Members: Promote knowledge sharing and cross-training within the team to prevent project delays if someone leaves or is unavailable.

Embrace the SOME of the Chaos

Software development is messy, unpredictable, and often maddening. But by recognizing the difference between known unknowns and unknown unknowns, and embracing methodologies that allow for flexibility, we can navigate the chaos more effectively. Next time a deadline whooshes by, remember: it’s not (just) about poor planning. It’s about the nature of the beast itself.

In the end, the key to handling software project deadlines is accepting that some things will always be out of our control. So, let’s embrace SOME of the chaos, adapt on the fly, and maybe, just maybe, we’ll get a bit closer to hitting that elusive target. And if not, there’s always more tea, whiskey and a good sense of humor to see us through. Although that will never make your boss our your client happy.

Previous ArticleNext Article
I help companies turn their technical ideas into reality. CEO @Sourcetoad and @OnDeck. Author of Herding Cats and Coders. Fan of squash, whiskey, aggressive inline, and temperamental British sports cars.

Leave a Reply

The Near Impossible Task of Defining an MVP

In Technology

We set out with the best intentions: we want to be lean and agile, and we are going to put something out into the world that we are “slightly embarrassed by” and we’re going to pivot and iterate from there. Right? That’s how it’s supposed to work.

But in reality, stakeholders never, ever seem to remember what that agreed MVP was. They fixate on a roadmapping session from six months ago or have a visual design comp stuck in their head that shows three features that were agreed to be post-MVP items.

Why Does This Happen?

  1. Memory is Fallible: We can remember talking about something, and knowing it was important, but not remember that we agreed it was out of scope or budget.
  2. Stakeholder Tactics: Some stakeholders might want to squeeze as much as they can out of development teams. They might agree to a smaller scope initially, hoping developers don’t remember the negotiation, or they might be unethically disingenuous.
  3. Lack of Experience: Stakeholders might not have enough experience in product management or development processes. They see software as infinitely changeable and might not grasp why certain features were deferred.

How Can We Stop It?

  1. Do Everything Waterfall: Agile just might not be the best approach for first-time founders, especially non-technical ones. Having clear visual comps of EVERYTHING can ease them into the fast-paced world of tech businesses, though this has many disadvantages.
  2. Hyper-Organization Mode: Developers and PMs can track every decision meticulously. A “Git for decisions” tool would be ideal, where every choice branches out with roadmaps and budgets, showing stakeholders how their decisions impact the project. Without such a tool, a detailed decision journal could work as a good alternative.
  3. Selective Client Engagement: Developers might choose to avoid working with first-time, non-technical founders. For products like platforms, where the technology is integral, having a technically savvy founder is crucial. For tech-enabled services, like an app to schedule plumbing appointments, the tech isn’t the core value, so non-tech founders might manage better.

What Can We Replace “MVPs” With?

I’m not sure. We have to launch with something, and that should have some level of definition, but I have three ideas:

Idea 1. Time Based MVPs

Maybe MVPs should be time-based, like launching something after a set period, regardless of what we have. Launch dates could drive everything in this way. Imagine a 90 day launch window after design/discovery has been performed. Then you would have weekly sprints, where the stakeholders are hyper engaged for a short period of time. The theory here is that this focus would cut out all the need for the “who said what when” arguments and everyone would only work on what was needed.

Idea 2. The 5-feature Rule

This is something I’m making up (desperately) on the spot. Perhaps the way we should be working to MVP should have some over arching philosophy about what an MVP even is.

  1. Basic User Management: Ensuring the system can handle user registration, login, and basic user profile management.
  2. Payment Handling: If the application involves transactions, it must handle payments from day one. Keep it simple and avoid novel payment systems.
  3. Performance: The application must perform well on its main targeted platform, device, or browser.
  4. Core Value: It should offer one really interesting, killer, or useful feature to the user. That’s it. Just one.
  5. Baseline Analytics: Include basic analytics from day one to track user engagement and system performance.

Idea 3: The Walking Skeleton Approach

Another approach to replace the traditional MVP is the “Walking Skeleton.” This method focuses on creating a fully clickable interface, essentially a complete UI, built with code, before any underlying logic is implemented. Acting as a pre-MVP, the Walking Skeleton allows stakeholders to interact with the prototype, providing a tangible sense of the final product without the complexity of backend functionality. This method helps stakeholders learn agility by iterating on this version, refining the user experience, and understanding the project’s scope more clearly. Once the interface is polished and aligns with the stakeholders’ vision, features can be prioritized and integrated based on what needs to launch first. Building out the application this way could be a lower stakes method for learning collaborative development, but it also doesn’t throw the baby (or code) out with the bath water.

Conclusion

By simplifying the MVP to these core elements, maybe we can mitigate the confusion and ensure everyone is on the same page about what the initial product will deliver. I prefer to end my posts the same way ChatGPT does, with a positive conclusion. But today, I’m less sure I can do that. I’ve read countless books, articles, and posts on how to define an MVP. I’ve helped launch hundreds of tech companies, and I’ve launched a couple of my own as well. So why is so difficult to launch when working with well-intentioned, intelligent, well-funded, people who are going through this process for the first time? I’m going to at least try Time Based MVPs and my new 5-Features rule on my next two projects and see if that helps. Wish me luck.