Adnan Masood's Weblog! عدنان مسعود

Tipping point’s author Malcom Gladwell discusses in great detail the power of context and its implications in epidemiology. However there is another kind which could adversely effect context driven expansions in quite interesting ways. In the following screenshots, the power of context is apparently being ignored by the full text search and substitution algorithms which are considering the Amazon jungles as Amazon the company. The mistake being replicated by both Google Finance and Forbes websites.

We definitely need something better than the following pseudo code

String.Replace (String.IndexOf(<companyName>), <companyname>.StockSymbol) 

A text based context sensitive substitution which actually recognizes when word “Amazon” as in the jungle and Apple as in the fruit.

Skyscrapr is your window into the architectural perspective. Solution, Infrastructure, Industry, and Strategic Architects create visions, develop plans, and work with developers to implement solutions.

Read More

Déjà Vu? It's mostly a dreaded question. Do you remember the last time your project manager asked you to estimate the time frame for a project you knew very little about, had very little written specifications on and the stake holders kept changing their minds about feature set? How did you go about estimating the time frame? Was it a ball park estimate, an educated guess or a WAG? How close was it to reality when the software actually got developed? I’ll discuss these questions and share some of my experiences in this writing.

In the January 2007 MSDN magazine, James Waletzky discussed the estimation from a SWAG (Silly Wild Ass Guess) prospect and then talked about Wideband Delphi for estimation. Rapid estimations are a reality of today’s fast paced development environments especially in the vertical markets where software development is means for a tool and not the core business of the organization, What would be the most appropriate method to estimate the time frame which is more efficient than guessing M&M’s in a jar at a Rubio’s and less time consuming than a function point estimation via Matson, Barnett, and Mellichamp Model for E = 585.7 + 15.12 FP?. Let’s see what popular methodologies have to offer.

As a principle, all software estimation methods share the Gödel's incompleteness theorems as their fundamental axiom i.e. the modeling of system is essentially not perfect. Traditional software engineering approaches to estimation include decomposition techniques like LOC (lines of code) based estimations, function point based estimates, process-based estimates, use case oriented estimates etc. These techniques essentially quantify the task breakdown via different methodologies and then estimate the time based on how long a single entity would take. Sounds like common sense, right? The empirical estimation models include function oriented metrics, COCOMO model and its variations. Aside from these heavy weight processes, Agile methodology describes the estimation process as follows.

• Each user scenario is considered separately for estimation purposes
• The scenario is decomposed into the set of functions and engineering tasks required to develop them.
• Each task is estimated separately, based on historical data, an empirical model, or “experience”
• Estimates for each task are summed to obtain an estimate for the scenario.
• The effort estimates for all scenarios are summed for a given increment to obtain the increment estimate

This sounds more realistic, five step process and you are done with it. As we already know as Moore et al described “Our preliminary results suggest that complicated methods may not necessarily yield a more accurate estimate, particularly when developers can incorporate their own intuition into the estimate”, or simply that making fancy estimations won’t make you more accurate.

In order to determine the rough order of magnitude (ROM), i.e. a rough estimate of the number of person-days to complete the task, I prefer the agile way. A typical workflow would be as follows.

Identify: Go through the specs (purpose statement/business requirement document or any other document) you have which identifies what needs to be done. Identify how you would do it in the scope and boundaries of current system.

Divide and Factor: With a rough sketch of identified tasks, factor out the similarities and what components can be shared across the board. Split the tasks into smaller pieces until it reaches ‘sane atomicity’. Try to think in service context so the responsibility is centralized instead of shared across the components, it helps the bottom line.

Raise the Unknown: Always identify unknown as risk, it helps. If you don’t know that the new mail server can handle bounce back processing for your application or you have not evaluated the ISO format file parser library performance yet, please make note of it.

Estimate:  Take an educated guess of how long it would take you to complete the tasks. If you are working as a team, knowing their respective skillets, ask fellow developers for their estimates. Most likely they would be the ones working on some of the components too. Find out the median, pad it to include the process factors (documentation, release notes, collaboration delays) and here your ROM guesstimate.

Repeat: As you identify the tasks more and more clearly, re-iterate through the steps for accuracy and effectiveness.

This process does not take too long, depending on the size of project it could be less than an hour and its much better than “hmmm, there are 1000 M&M’s in this bottle”. This is more like “In one square inch of this jar, I can see almost 20 chocolate goodies. Providing the curve and integrating below it, I think it would have…”, you got the idea. You can re-calibrate it with future iterations as it would be a good learning exercise in estimation. This method will get more and more concrete proportional to your specifications as they say, requirements are like water. They're easier to build estimate when they're frozen.

“I love deadlines. I like the whooshing sound they make as they fly by.”
-- Douglas Adams

References and Further Readings

• MIT Blog on Software Cost Modelling
• Agile Software Development for the Entire Project
  Granville Miller, Microsoft
• The Classic Story of the Pig and Chicken
• Bayesian Analysis of Empirical Software Engineering Cost Models
• An Empirical Study Using Task Assignment Patterns to Improve the Accuracy of Software Effort Estimation
• Modern Empirical Cost and Schedule Estimation Tools
• Empirically-Guided Software Effort Guesstimation
  Philip M. Johnson Carleton A. Moore
  Joseph A. Dane Robert S. Brewer
  Collaborative Software Development Laboratory
  Information & Computer Sciences Department
  University of Hawaii

An interesting letter by Dr. Lotfi Zadeh on Determinism vs. Chance published on UAI. He also referneces his GTU paper for migration from random fuzzy sets to granule-valued distributions;

Dear Hung,

   Thank you for your illuminating analyses of the Valentina
example,and your comments regarding the theory of random fuzzy sets.
Your high expertise in both probability theory and fuzzy logic is in
evidence.

   Regarding the Bayesian approach outlined by Aleks Jakulin, I should
like to add the following. First, the conditioning information is
perception-based and not quantifiable. Specifically, how would wrinkles,
for example, be dealt with? Furthermore, if my perception is that
Valentina is young, how would the Bayesian approach apply? If a
subjective probability distribution is associated with young, what would
be the probability distribution associated with not young? We could
apply random sets to this example but fuzzy logic would be much simpler
to use.

   Clearly, the power of probabilistic methods is enhanced when we move
from point-valued discrete probability distributions to set-valued
discrete probability distributions, that is, to random sets. The power
is enhanced further when we move from random sets to random fuzzy sets,
as you do. Please note that in my 1979 paper "Fuzzy Sets and Information
Granularity," Advances in Fuzzy Set Theory and Applications, M. Gupta,
R. Ragade and R. Yager (eds.), 3-18. Amsterdam: North-Holland Publishing
Co., 1979 (available upon request), I employed random fuzzy sets to
generalize the Dempster-Shafer framework. However, moving from random
sets to random fuzzy sets is not sufficient. What has to be done is
moving from random fuzzy sets to granule-valued distributions, as
described in my paper "Generalized Theory of Uncertainty
(GTU)--Principal Concepts and Ideas," in Computational Statistics & Data
Analysis 51, 15-46, 2006. Downloadable: http://www.sciencedirect.com/ or
available upon request. The concept of a granule is more general than
the concept of a fuzzy set. A granule is characterized by a generalized
constraint. In my view, this level of generalization is needed to
enhance the power of probability theory to a point where it can deal
with the examples given in my messages. Try the following: Most Swedes
are much taller than most Italians. What is the difference in the
average height of Swedes and the average height of Italians? A solution
is given in my GTU paper.

   Theory of random fuzzy sets enriches probability theory but not to a
point where it can be said, as you do, that randomness and fuzziness can
coexist in the framework of probability theory. Many other problems
remain. One of them, as is pointed out in my JSPI paper, which is cited
in my previous message, is that in dealing with imprecise probabilities
we have to deal in addition with imprecise events, imprecise functions,
imprecise relations and other imprecise dependencies. More importantly,
a basic problem is that almost all concepts in probability theory are
bivalent, e.g, events A and B are either independent or not independent,
a process is either stationary or nonstationary, an event either occurs
or does not occur, with no shades of truth allowed. In reality, these
and other concepts are not bivalent--they are a matter of degree. It may
take a long time for this to happen, but I have no doubt that eventually
it will be recognized that bivalent logic is not the right kind of logic
to serve as a foundation for probability theory.

   You have made and are continuing to make important contributions to
both probability theory and fuzzy logic, and building bridges between
them. Please continue to do so.

            With my warm regards,

                  Lotfi
--
Lotfi A. Zadeh
Professor in the Graduate School
Director, Berkeley Initiative in Soft Computing (BISC)

----------------------------------------

Soft Computing Applications in Business

*Final Call for book Chapters*

*(deadline extended)*

The following two books are planned to be published by Springer-Verlag
during 2007/2008:
(1). Soft Computing Applications in Business
(2). Soft Computing Applications in Industry

We invite book chapter contributions. Please see the website:
http://www.bhanuprasad.org/chapters.html for more information.

Joe White's Blog states Randy Miller's MSF for agile salient points some of which are inspired from context driven school. Interesting ideas and quite true in their regards

1. The value of any practice depends on its context.

2. There are good practices in context, but there are no best practices.

3. People, working together, are the most important part of any project's context.
   

4. Projects unfold over time in ways that are often not predictable.

5. The product is a solution. If the problem isn't solved, the product doesn't work.

6. Good software testing is a challenging intellectual process.

7. Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.

Reference: Seven Basic Principles of Context Driven School