Software development effort estimation is the process of predicting the most realistic use of effort required to develop or maintain software based on incomplete, uncertain and/or noisy input. Effort estimates may be used (and are often used) as input to many other business processes (project plans, iteration plans, budgets, investment analyses, pricing processes and bidding rounds).
Published surveys on estimation practice suggest that expert estimation is the dominant strategy when estimating software development effort. With this method typically, effort estimates are over-optimistic and there is a strong over-confidence in their accuracy.
Overconfidence: Why You Suck At Making Development Time Estimates posting recommends to read a a lengthy post Coding, Fast and Slow: Developers and the Psychology of Overconfidence about why software developers often struggle to estimate the time required to implement their projects. Drawing on lessons from a book called Thinking Fast and Slow by Dan Kahneman, the writer explains how overconfidence frequently leads to underestimations of a project’s complexity. Unfortunately, the nature of overconfidence makes it tough to compensate.
Coding, Fast and Slow: Developers and the Psychology of Overconfidence article gives reasons why we’re so bad at making estimates:
The first is the sort of irreducible one: writing software involves figuring out something in such incredibly precise detail that you can tell a computer how to do it. And the problem is that, hidden in the parts you don’t fully understand when you start, there are often these problems that will explode and just utterly screw you. And this is genuinely irreducible.
The second problem is overconfidence. Specifically, in many, many situations, the following three things hold true:
1- “Expert” predictions about some future event are so completely unreliable as to be basically meaningless
2- Nonetheless, the experts in question are extremely confident about the accuracy of their predictions
3- And, best of all: absolutely nothing seems to be able to diminish the confidence that experts feel
Alright, So Let’s Stop Being So Overconfident! The promise of the “full specification before we start coding” approach is that you can get more accurate estimations. Congratulations, you’ve just invented Waterfall. But that totally fails. Like, always.
Using short sprints in development are one way on trying to handle the bad estimation problems: they place a hard limit on the cost of a horrifically bad estimate.
13 Comments
Tomi Engdahl says:
Sorry, College Grads, I Probably Won’t Hire You
If you’re at all interested in media, technology or related fields, please learn a little computer programming.
http://online.wsj.com/article/SB10001424127887323744604578470900844821388.html
The problem is that the right skills are very hard to find. And I’m sorry to say it, dear graduates, but you probably don’t have them.
According to one recent report, in the next decade American colleges will mint 40,000 graduates with a bachelor’s degree in computer science, though the U.S. economy is slated to create 120,000 computing jobs that require such degrees.
I don’t mean that you need to become genius programmers
Coding at such a level is a very particular and rare skill
What we nonexperts do possess is the ability to know enough about how these information systems work that we can be useful discussing them with others. Consider this example: Suppose you’re sitting in a meeting with clients, and someone asks you how long a certain digital project is slated to take.
Unless you understand the fundamentals of what engineers and programmers do, unless you’re familiar enough with the principles and machinations of coding to know how the back end of the business works, any answer you give is a guess and therefore probably wrong. Even if your dream job is in marketing or sales or another department seemingly unrelated to programming, I’m not going to hire you unless you can at least understand the basic way my company works. And I’m not alone.
If you want a job in media, technology or a related field, make learning basic computer language your goal this summer.
derma rollers says:
Attractive portion of content. I just stumbled upon your site and in accession capital to claim that I get in fact enjoyed account your weblog posts. Any way I will be subscribing in your feeds and even I achievement you get entry to consistently quickly.
dermaroller says:
hi!,I love your writing very much! proportion we keep up a correspondence more about your post on AOL? I require an expert on this space to unravel my problem. Maybe that’s you! Having a look forward to look you.
ROCKPORT ロックポート says:
In his system there is a minimum of seven lines. He wants his goalkeeper to be part of the play, then the centre backs, then what he calls the ‘controller’ (a deep-lying playmaker), then the full-backs pushed on, the two attacking midfielders, the wingers and then the centre-forward. That allows you to draw seven horizontal lines across the pitch.
ROCKPORT ロックポート http://www.japanbrandsale.com/ROCKPORT%20%20%E3%83%AD%E3%83%83%E3%82%AF%E3%83%9D%E3%83%BC%E3%83%88-20832/
Chante Lewis says:
Competitive Edge Writer…
I merely wanted to thank you once more for the amazing blog you have made here. It really is full of ideas for those who are definitely interested in that subject, especially this very post….
Tomi Engdahl says:
Does your calendar make you to run to death?
The organization’s heroes are not those who promise a lot, but they are getting a lot for the benefit of customers already.
Striking in a hurry
Hurry is basically due to the fact that our work is not as regular and predictable as we would like to believe. It only takes one illness, a busy queue going to work, wrong assessment of the amount of work or an unexpected difficulty in doing the work, and the whole house of cards falls apart.
Entire project offices may be put in crisis meetings, to manage flood of e-mail, and time and time again to organize the work detail again. When the false promises are to be kept, will be taken to compromise the quality of the high risks and overloading the workers. This creates a bad spiral of productivity will decline from round to round, and in a hurry to get worse.
Do not block your calendar
Multi-use inherent in the very small to-do lists to keep the jobs in memory. When the calendar tool to help lead us to the Lutheran work ethic ahnehtimisen temptation. So we fill our calendar 110 per cent, and the problems can start.
Success in principle is simple: “Take one small task at a time and make it ready.” If the job is big, it is worth it to split, that can be done with one spell of work completed. These sizes less than a day to keep prioritize tasks so that the benefit to the customer is good.
Team work is a distressing look at the work to stop because someone has booked a job and then do it. The bottleneck is often a boss or a specialist, even if the job is quite capable of making someone else. Excessive specialization and co-ordination of decision delimitation will lead to chaos, which will be unbearable, especially if the team is not in the same room. This is reflected in waiting, when the work performed by others are not ready when I have the time available to continue the work.
In creative work such as the design of the software is needed to focus on and work in peace. Task switches, redundant work of moving one person to another, waiting, interruptions, participation in multiple projects at the same time, the correction of errors and their prevention rather than inefficient meetings are a waste, the reduction is worth investing.
Performance management does not work
A very common method is to set a strict schedule jobs without a realistic plan or the ability to achieve these goals. Deming’s 14 point quality of the program has been abandoned quantitative targets, as they reduce the quality. The fact that the work may not be completed on schedule by heart, not because of laziness, but employees work into the process, including a natural variation.
Parkinson’s Law, according to which the work meets the allotted period of time, to keep its place in, if the work is done the motivation is external reward or avoiding punishment. It’s easy to do the same thing as the required task, but also other less important and less urgent matters. Realistic goals and milestones help you focus and work in simple tasks, such as running a marathon. Creative challenges in complex team work situation is quite different.
The problem may also be the only working in the budgeting and investing your calendar. This is particularly evident in those meetings where the norm is to use the entire time allotted, even if the work has already been done.
Trust in the fact that the employees are doing their best, based on openness. Guests receive a high-quality supplies at frequent intervals – in the software industry every few weeks. The customer and the supplier know each other well and make the day to day co-operation. Leadership is the presence of the encouragement
Lean, Kanban, Scrum and agile
Simple laws of life and the practice of products to help their dissemination and implementation. A large part of the IT sector employees is Lean, Scrum and agile certification.
Preparation and tests are, of course, only a first step towards better ways of working and corporate culture change towards the standards of this millennium.
Source: http://www.tietoviikko.fi/viisaat/tieturi/juoksuttaako+kalenterisi+sinut+kuoliaaksi/a932581
www.tele-dyktafon.pl says:
All Shingle style should be as it is known as – shingles deal with everything
on the exterior. include a higher prospect of risk to
be the project.
CieA Solutions,CieA, Software Development,Customized Software, Software Developers Malappuram, Malappuram, Software development Malappuram, Wandoor, Customised Software Wandoor, Mobile Application Development, Web Development, Website design Wandoor, Web says:
Simply want to say your article is as amazing. The clarity on your put up is simply spectacular and that i could assume you’re knowledgeable in this subject. Well together with your permission allow me to snatch your feed to stay up to date with drawing close post. Thanks 1,000,000 and please keep up the gratifying work.
Tomi Engdahl says:
This Is Why You’re Late All The Time (And What To Do About It)
http://www.huffingtonpost.com/2013/11/07/psychology-lateness_n_4229057.html?ref=topbar
“Lateness is really a commonly misunderstood problem,” says Diana DeLonzor, author of Never Be Late Again, who has conducted her own research on the perpetually tardy. “Yes, it’s a rude act, but I’ve interviewed hundreds of people and the vast majority of late people really dislike being late, they try to be on time, but this is something that has plagued them throughout their lives. Telling a chronic late person to be on time is like telling a dieter, ‘Don’t eat so much.’”
The issue might have something to do with fundamental differences in the way we think, according to DeLonzor. In her own research, she’s found that late-arrivers tend to actually perceive time differently than their punctual peers.
Psychological components can also contribute to chronic lateness.
“One of the things I found is that some people were subconsciously drawn to the adrenaline rush of that last minute sprint to the finish line … They have a hard time motivating themselves without that looming deadline, without that crisis on the horizon,” she says. “Then, as they realize there’s no way they’re going to make it on time, that positive feeling turns to dread. Then they start beating themselves up.”
derma roller says:
Heya i’m for the first time here. I came across this board and I in finding It truly useful & it helped me out much. I am hoping to present one thing again and help others such as you aided me.
Tomi Engdahl says:
Why Are So Many IT Projects Failing?
http://www.cio.com/article/744168/Why_Are_So_Many_IT_Projects_Failing_
A recent study reports that 50 percent of companies had an IT project fail in the last 12 months. Business leaders who blame IT are missing the real project management issues.
Fifty percent of businesses had an IT project fail during the last year, according to a survey by cloud portfolio management provider Innotas. The primary reason, according to 74 percent of respondents, was a lack of resources to meet project demands.
Where have all the project managers gone? Is the IT industry suffering a shortage of employees with these skills?
Not necessarily, says Dice.com’s president, Shravan Goli. Both supply and demand for project managers has remained consistent, with the number of available positions currently available on Dice.com staying at about 3,200, he says.
What has changed is the qualitative side, Goli says, as project managers’ roles shift, they are expected to take on additional responsibilities above and beyond the fundamental scope of managing each IT project.
“The role has evolved over time, and there are a few trends that may be infringing on the project manager’s core job description,” Goli says.
“The fundamental, core job description remains management and monitoring of project scope, communication between groups, motivating teams to drive delivery,” he says. “But the emergence of the Agile development methodology means that project managers must also take on the role of development lead.”
For companies that adopt Agile, many do see an increased need for project managers to drive delivery of an increasing number of software-based technology solutions and applications. But instead of adding staff and resources, firms instead are delegating these roles to their existing project managers, Goli says.
The shortage of resources could be one reason why many projects fail, says Kern, but there’s also a pervasive mindset that IT is the problem, not the solution.
As organizations move to a more application-centric focus, the number of IT projects increases, and IT departments have trouble saying “no,” Kern says, because of the risk of being seen not as a valuable business partner, but as an expensive cost center.
“We’ve moved away from an era of hardware and operating systems and it’s all about applications. Nobody on the business side cares how solutions are delivered, they care about the value of the application,” Kern says.
This shift puts an increasing burden on IT departments to deliver these valuable applications, even if they’re overworked, understaffed, and have no way to prioritize projects, he says.
“The shift to an application-centric approach means there’s no shortage of demand for project managers to handle these projects. But what IT might say ‘yes’ to today could be irrelevant in six months, so they need to better prioritize their pipeline. Unfortunately, because there’s always this dark cloud of ‘What value do you have? You’re a cost center,’ they feel they have to agree to everything, even if they can’t possibly get it done,” he says.
“Consolidation and cost-cutting have really taken their toll, and there’s not a lot of incentive for businesses to get beyond cost-cutting.”
Tomi Engdahl says:
Ask Slashdot: Are Accurate Software Development Time Predictions a Myth?
https://ask.slashdot.org/story/17/04/26/2040212/ask-slashdot-are-accurate-software-development-time-predictions-a-myth
Any attempts we make to estimate the amount of time software development tasks will take inevitably end in folly. Do you find you can make accurate estimates, or is it really the case, as the author, DuroSoft Technologies’ CTO/Co-CEO Sam Johnson, suggests via Hacker Noon, that “writing and maintaining code can be seen as a fundamentally chaotic activity, subject to sudden, unpredictable gotchas that take up an inordinate amount of time” and that therefore attempting to make predictions in the first place is itself a waste of our valuable time?
The myth of software time estimations
https://hackernoon.com/the-myth-of-software-time-estimations-576a7466d91a
I like to think of software development as navigating through a dark, perilous wilderness. You have no map. If there is a map, it is because someone has already made your idea, so there is no map. The terrain is always uncharted, and always dangerous.
Even if we completely ignore the “human” element at play in the development cycle, the notion that we could come up with accurate software development time estimations is the quintessential unrealistic expectation — a fool’s errand. Its intractability comes not from incompetency or from a lack of discipline, but from a deep-seated, fundamental limitation imposed on our reality and codified in the proof of the Church-Turing thesis:
Given the initial conditions (inputs) and source code of an arbitrary program, and given access to unlimited processing and memory resources, it is impossible to always predict whether such a program will run forever or eventually halt its execution after some amount of runtime.
While there are some obvious exceptions in terms of formal methods and programs that are provably correct, the chances are that you haven’t had the time or found it practical to implement formal methods and mathematically prove that your program does exactly what it was designed to do in every possible execution state.
In software development, this chaos is normalcy, and wandering around in this unknown, unproven territory where many things could go wrong at any moment simply is the perpetual state of programming.
Why do we keep up this charade?
There are a number of reasons why perfectly reasonable people might subject themselves to the irrational constraints brought on by development time estimations. First off, it is a fad right now, so there’s that. Another reason is that business requirements, and the business/sales-types who make decisions based on their perception of these requirements like to make promises about the future (especially to investors and potential users), which inevitably involves becoming entangled in some set of unachievable development goals and/or deadlines that when unmet, reflect poorly on the organization as a whole, often more so than if no time-sensitive promises had been made at all. This is especially true at companies where non-technical individuals hold a disproportionate amount of power or influence over the development process, and perhaps don’t fully understand that software development is a fundamentally more difficult, unpredictable activity than are other activities you encounter in the business world.
At the end of the day, even the simplest, easiest tasks may contain latent, unpredictable gotchas that will take up weeks of development time.
Time estimations considered harmful
Development time estimations lead us to cut corners, to design painfully simple software that lacks the architectural complexity it needs to one day tackle the fully-realized version of the problem we want to solve. Since business requirements inevitably undercut even the most brazenly optimistic development timelines, the things we would do given even a little bit more time end up not getting done or even planned. I like to call this complexity debt
But at organization X the estimations are typically accurate!
There is a world where building software is not like wandering through a vast unknown wilderness, but is instead more like a short stroll through a field of daises — but it is a doomed world. I would argue that projects like this, where the estimations are consistently right, and it feels like there is an actual end to development in sight, do in fact exist, but that these aspects are symptomatic of something worrisome and broken on a fundamental level.
If time estimations are so bad, what should we use?
In general, you should consider using pure Kanban without time estimations, but first, you should take the time to write out the following sentence, maybe hang it on your office wall, add it to your mission statement, set it as your development Slack room’s status, and tattoo it to your CTO’s forehead:
As a company policy, we do not provide or deal in software development time estimations of any kind, and we view such a practice as harmful to the overall development process, and to the industry as a whole.
The key is not to provide these types of estimations in the first place. Good investors don’t need to know when a feature will be done — rather they just want to know that it is a priority and that it is being worked on — and if they are smart, they will respect you when you highlight the reasons why providing concrete estimations will ultimately decrease the value of their investment (e.g. complexity debt, etc.).
What you should do
the fact that estimations are just that: fantastical theories, that are subject to potentially dramatic changes as more of the surrounding landscape comes into view, and are in no way a commitment to a particular completion date. This is why tentative difficulty estimations, completely devoid of any time commitment whatsoever, are the way to go.
Tomi Engdahl says:
Don’t Keep Making the Same Embedded Development Mistakes
https://www.designnews.com/electronics-test/don-t-keep-making-same-embedded-development-mistakes/61798276956654?cid=nl.x.dn14.edt.aud.dn.20170427.tst004t
Because the study of software programming seldom includes case studies, many engineers are doomed to keep repeating the same mistakes.
Because the study of software programming seldom includes case studies, many engineers are doomed to keep repeating the same mistakes, an embedded development expert will tell attendees at the upcoming Embedded Systems Conference (ESC) in Boston.
David Nadler, founder of Nadler & Associates , says that in his role as a consultant, he has seen those repetitive mistakes end up costing money. “If you don’t plan correctly, it can cause considerable excitement,” he recently told Design News. “You have missed deadlines, lots of yelling, and a very unpleasant time for all.”
Still, most engineers are never exposed to the benefits of case studies, which is why the mistakes keep happening. “If you go to the Harvard Business School, you see case studies of companies that went off the rails,” he told Design News recently. “If you go to medical school, you look at case studies of diagnostic problems. But in software development, we don’t study cases of software development gone bad, and that’s to our detriment. So we end up making the same mistakes over and over again.”
“We’ll walk through the techniques used to get the projects back on track,” he told us. “Then we’ll talk about how developers might want to use these techniques at the beginning of a project – before it goes off the rails.” Mostly, he said he will examine the problem of failing to plan for product test and verification, which may be the most common of all.
Nadler contends that all embedded developers benefit from an examination of case studies, especially those who’ve “picked up” software during their professional careers. “In the embedded world, a lot of people weren’t trained in software at all,” he said. “They pick it up after getting an electrical engineering degree, or a mechanical engineering degree, or a physics degree. But case studies also help people who are trained in software, because most curricula don’t teach that.”