I remember working on this one major project - an online music store - and one of the execs hired his very lovely wife to project manage a team of developers. She would create these fantastic charts in Microsoft Project and every day try to somehow - somehow - get the developers to either (a) give her timescales for each component, or (b) explain why previous component X took twice as long as they had scheduled.

I've been writing code for 40 years and I still get caught out myself. I try to give realistic deadlines and then some fucking bug in the compiler or other such insanity will come along and make me look like a fool.

Call it immaturity, or perhaps just not being a "team player", but I've been coding for 15 years and stopped giving time estimates about 5 years ago.

I simply refuse, and instead, explain that I would just be making something up, and I'm not interested in taking responsibility for an arbitrary guess, since I know from experience how it can come back to haunt me.

I try to suggest saving such speculation for a time, hopefully in the near future, where the initial research has been completed and _some_ progress has been made on the foundation of the requested work, so that my speculation can have some basis in reality.

If all else fails, I give a wild estimate that I don't expect anyone to agree to, such as 2 years for work that might not even take 2 or 3 months.

Things still go off the rails of course making an an imperfect solution, but it ends up being more right than any other method I’ve tried.

*If there are any unknown unknowns I might start multiplying by the number of big ones.

Nah, it can happen through sheer chance. I remember hitting a compiler bug when I was young (the only time I hit a compiler bug). I was certainly not particularly smart, just unlucky.

The interesting thing is that it had been drilled into me that "it's never a compiler bug" and so I banged my head against that wall for a couple of weeks before a more experienced programmer took interest and showed that it was actually a compiler bug.

Definitely not in my case -- sometimes we just stumble over these things. As an inexperienced programmer (around 1989 or 1990?) I found a bug in the beta of microsoft's first C++ compiler, shipped on a dozen 5.25" floppy discs. All the other developers around me said "Don't be stupid. It's your code, it's not the compiler!" but I distilled it down to a minimal test case and proved it. The bug?

i=i+i;

That's it. It would give random results based on prior instructions unrelated to i.

I reported it and got a nice note back from the MSFT engineering team. Something like "Wow! I've no idea how we missed this. Good catch!"

It's been a while so I don't remember all the details but I remember an ancient version of gcc refusing to accept perfectly valid zero initialisation of structs. Not terribly exciting but you don't have to be too clever to hit compiler bugs.

The planning fallacy is something like "Humans are hopelessly optimistic. Even if you think you're planning for the worst case, unexpected things will happen, and your estimate is more like the best case scenario. This is true even if you try to compensate for the planning fallacy by being more pessimistic".

So you can't fix it by padding more weeks, trying to think of all the things that might happen, etc.

The only way to compensate for this is detailed in Kahneman's "Thinking Fast and Slow" (Must-read for many reasons). That is to start with prior evidence. Rather than making your estimate, start with "How long did it ACTUALLY take the last time someone did something like this?" and only then modifying the estimate for circumstances.

There is a huge gap between these two methods:

1. "How long does it take an average team to create a SAS product? Maybe a year?" Then modify for "but our team is extra smart, so maybe I'll shorten that to 10 months"

vs

2. "How long will this take? It's not a complicated product, and our team is smart: maybe 4 weeks for feature X, 6 for feature Y, 2 to deploy. I'll even throw in an extra 4 weeks of padding for unknowns, so 16 weeks".

The first is that you can predict that there will be unforeseen problems, but not how many of them nor how long each will take to resolve. The result is high variance. Then if the average such project takes 16 months, and you predict each such project will take 16 months, then some take 7 months and some take 17 months and some take 25 months and people get mad at you for underestimating some of them. But if you predict at the upper end of the range then people will get mad at you for giving 25 month estimates for projects that only take 16 months on average.

The second is that people know what the estimate is and work expands to fill all available time. When you know you have 16 months, you set a pace that will have the project finished in 16 months. Then in month 14 you encounter some unexpected problem that takes six months to resolve. If you'd found it in month 2 you'd have been fine, because you'd have cut some less critical features and spent the time on that, but now that's in the past and your project is five months late.

Project Manager/Scheduler/Sales/Customer doing one/any/all of: not believing the input of the SME (you) when they ask for input developing the schedule, overpromising, expecting the world, or so many other problems.

Too many times in my own field (heavy construction) the scheduler has asked me how long it will take to do a given task, and it is actually reasonably quantifiable to say "It will take my crew X days to move Y volume of material at Z/shift" only to be told "Well our proposal only allocated X-5 days and budgeted a Crew of <your chosen size - 4> for that task. Figure it out."

Haha, I came to that same conclusion decades ago - I don't know if it would actually produce meaningful estimates or not, though, because every time I try to estimate that way, somebody more "important" than me comes along and throws out that estimate for one that fits the timeline he already had in mind.

This requires to have prior evidence within your team to start with as not all teams have the same velocity.

Agile, for example, suggests that decreasing the cost of change at the expense of certainty of cost will increase the total chances of delivery within cost and date. Outsources are (implicitly) arguing that the decrease of total costs are worth the other friction in the organization.

This all goes back to return on investment. In the most simplistic case, if I have ten people working on this task now, and this software will allow me to reduce that to six, then that's four people I don't have to pay for that task. If those four people are making $100,000 a year, and the software lasts 10 years before having to be replaced, that's 4 million dollars. The software plus maintenance needs to cost less than 4 million dollars to make sense. We all know the simplistic case.

But if I can trade $100,000 dollars in guaranteed cost for a reduction of $500,000 in risk in making that goal, I can trade cost for certainty and that's going to be a good trade. Now, as developers we know that these numbers are often fudged, but everyone realizes this is inexact and just the best we all can do. We need to know how to think about these in terms of arguing against bad investments and steer toward good investments by knowing how the tradeoffs works.

Yes, that's the promise. I've seen it tried as earnestly as possible over the past 20 or so years since the term became popular, but I've never seen it deliver on its promises either. I suspect that a lot of that is psychological - risk-averse management hears "on time and on budget" and consciously or unconsciously starts steering the development process back to a more predictable seeming waterfall style that negates the supposed benefits of "agile" development.

I assert, however, based on 30 years of observation, that no matter how carefully you follow an agile delivery framework, whatever that may be, actual delivery and cost will have no relationship whatsoever on your estimated cost and date.

But I also predict that you and I will continue to smile and nod when they insist that it must be finished by such and such date.

The real problem is that management think they can directly constrain those kinds of variables but in reality they are not inputs they're outputs. The real inputs are things you can actually control like what you're building and which of your team you assign to build which parts of it at which times. That might come in on time and under budget or it might not but shortening a timeline or decreasing a budget won't change how long it takes or what it costs.

Of course that sucks for managers who want to make informed decisions like whether to go ahead with a project that will require an uncertain amount of investment to complete but needs to generate more return than some threshold to be worth doing. Now the managers have to find a way to guess what the important outputs like cost and timescales will be as some sort of probability curve so they can make a rational decision based on the best but still incomplete information they have available. And even then a lot of managers insist on trying to be absolute about things like time and money when the actionable decisions they need to make are more often whether to start some project at all or which ongoing projects currently have the greatest need for any additional resources that are available.

1. When doing estimates, estimate hierarchically and break down every task until the leaf nodes are 4 hours or less. If the leaf nodes are a week of effort, that means that you don't yet understand fully what the task entails. If you don't know what the task entails, it is very difficult to get an accurate estimate.

2. Maintain a database of various projects and features, and use them for comparison purposes. If you know how long something actually took in the past, you can much more accurately how long it will take to do something similar again.

3. Include specific time for various types of "overhead": meetings, code review, writing tests, debugging, making presentations, etc.

4. Derisk ahead of time things that are truly unpredictable. Doing something in a brand new programing language or framework, training an ML model on a new type of data, working on datasets of unknown quality, etc. For those you need to estimate out a project to determine feasibility and generate an estimate.

The main thing is that this is clearly not impossible. Years ago I got really interested in the Apollo program as an example of bringing in perhaps the most complex and risk-laden project ever attempted, on time. One of the things that clearly stands out about that project was that there was huge derisking happening first. At one level you had the Mercury and Gemini programs that were figuring out human spaceflight generally, and docking with Gemini, and also things like the first stage engines of the Saturn V were already in development in the mid fifties.

How long did that huge derisking take? What was the budget for the time spent doing the derisking? I'm willing to bet the "huge derisking" ended up taking longer than the actual project (and was itself open ended and unpredictable).

AFAICT it's actually worse - the exploratory phase always seems to get bogged down by theoretical discussions and imaginary scenarios that don't end up being very important (and putting off how long it takes to find the scenarios that actually do).

Where it makes sense is from a sociocultural perspective. Under Taylorist management philosophy, management is to do the thinking, and workers are to do the doing. Structuring a project where managers take on the exploratory phase fits well within the set of assumptions taught in project management programs.

If you’re doing that kind of work, your employer will be happy to replace you with an AI system.

Or a couple of Excel macros.

You also need someone with 1) strong technical and/or product skills, 2) good communication skills and decisiveness, and 3) explicit executive authority. That's the only way to avoid scope creep and bikeshedding.

When you started working on a project without knowing what you're building, or end up delivering a very different project than what was originally planned, it's hard to even say "that project was late." No it wasn't: you just never built whatever you had promised to build by the original deadline.

For example:

If the job involves dealing with anyone from Samsung or LG, it will take 2-3 times as long as you think.

If the job involves adding in a commercial third-party component which has a website about it, and that website doesn't clearly explain what the benefit of the thing actually is, the job will take 10-20 times as long as it should.

If the job involves a new feature Amazon or Google have just launched on one of their devices, and you are being CC'd on email chains involving representatives of Amazon or Google who are claiming implementation is very easy, the job will take a minimum of 6 months and may never be completed because the feature will be dropped or replaced with V2 before V1 ever actually works.

But your CTO will think it will take 10-20 times less time than it should, because that's what the vendor promised.

This is why they are not only inevitable but unending.

Most real world problems are so large and messy that they do not have closed form solutions. In this context "bug" more often means a deficiency, a reflections of changing or competing and incompatible requirements, than abstract mistakes.

In this context, I've learned to appreciate the choice becomes "which bugs", because "fixing" such bugs is not necessarily a net positive, especially when the fix is not a natural fit for the design or architecture of the overarching solution. The mindless, incessant fixing of such bugs on a case-by-case basis can result in accumulation of so much complexity and LoC it becomes self perpetuating in bug fixing alone. When you are conscious of this, and have enough respect and autonomy, it's possible to decide whether it's worth this intangible cost, or whether to put it into the pile of "wont fix", this kind of decision requires cross domain collaboration. But that pile is not worthless, you can periodically revisit it and see opportunities to a fundamentally different approach that may tick more boxes (or more preferable boxes) without being ad-hock.

> Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law.