Monday, March 29, 2010

Feynman Cargo Cult Science:
"All experiments in psychology are not of this type, however. For example, there have been many experiments running rats through all kinds of mazes, and so on--with little clear result. But in 1937 a man named Young did a very interesting one. He had a long corridor with doors all along one side where the rats came in, and doors along the other side where the food was. He wanted to see if he could train the rats to go in at the third door down from wherever he started them off. No. The rats went immediately to the door where the food had been the time before.

The question was, how did the rats know, because the corridor was so beautifully built and so uniform, that this was the same door as before? Obviously there was something about the door that was different from the other doors. So he painted the doors very carefully, arranging the textures on the faces of the doors exactly the same. Still the rats could tell. Then he thought maybe the rats were smelling the food, so he used chemicals to change the smell after each run. Still the rats could tell. Then he realized the rats might be able to tell by seeing the lights and the arrangement in the laboratory like any commonsense person. So he covered the corridor, and still the rats could tell.

He finally found that they could tell by the way the floor sounded when they ran over it. And he could only fix that by putting his corridor in sand. So he covered one after another of all possible clues and finally was able to fool the rats so that they had to learn to go in the third door. If he relaxed any of his conditions, the rats could tell.

Now, from a scientific standpoint, that is an A-number-one experiment. That is the experiment that makes rat-running experiments sensible, because it uncovers the clues that the rat is really using--not what you think it's using. And that is the experiment that tells exactly what conditions you have to use in order to be careful and control everything in an experiment with rat-running.

I looked into the subsequent history of this research. The next experiment, and the one after that, never referred to Mr. Young. They never used any of his criteria of putting the corridor on sand, or being very careful. They just went right on running rats in the same old way, and paid no attention to the great discoveries of Mr. Young, and his papers are not referred to, because he didn't discover anything about the rats. In fact, he discovered all the things you have to do to discover something about rats. But not paying attention to experiments like that is a characteristic of
cargo cult science."

Why it is important to record and report every auxiliary and decision made

The key ability of others to interpret and replicate experiments relies on the replication of the exact same decisions to result in the exact same auxiliary hypotheses. If these minute details are not reported, and they matter, then replication will fail and resources will be wasted searching for the proper design to replicate the previous experiment. Furthermore, as the research field builds, a standard set of decisions/auxiliaries builds and the decision-making process becomes easier and easier.
If we want to know the probability of our result replicating, we can use Prep. (Killeen, 2008; In Goode, 2008). Prep is P(d2|d1), however, it requires that study 2 is exactly identical to study 1. Thus, it is impossible for anyone to expect to replicate a study with probability= Prep if they do not know the exact details involved in the study. For example, “Many of the current conventions for sample size may be credited to Jacob Cohen’s (1988, 1992) tireless efforts to improve the quality of social science research. His voice is joined by a large group of methodologists deploring the failure of researchers to fully report the details needed to put the findings in context with the rest of the field. Therefore, an empirical study based on a limited sample is further weakened by superficial reporting that will not allow fellow scientists to replicate the complete study or even duplicate the calculations. This is a pointed cautionary note for novice researchers who have not yet grasped the nuances of the decisions to be made.” (137) (Peterson, 2008; in Goode, 2008).

Thursday, March 18, 2010

Proofs and refutations

"Delta, I am flabbergasted. You say nothing? Can't you define this new counterexample out of existence? I thought there was no hypothesis in the world which you could not save from falsification with a suitable linguistic trick."

Al Jazeera English - Europe - Electric torture show shocks France

Al Jazeera English - Europe - Electric torture show shocks France

Tuesday, March 16, 2010

An algorithm for existential abduction

The first part of existential abduction involves generating ill-defined elements of a space of possible explanations of a phenomenon. One should proceed as follows: Generate theories that seem most plausible. Keep generating theories until some function of the marginal plausibility of the next theory and the marginal cost to generating the next theory (cost is in units of time) are equal. This assumes that plausibility can be monetized (as time can easily be monetized) and that people are able to think of theories in descending order of likelihood.

Existential abduction

My thesis will cover two of the most fundamental points of science: 1) How we should and do develop structural causal models of the world (existential abduction; Haig, 2005) and 2) how we should and do test these models and evaluate the uncertainty associated with them (induction). Search algorithms (also Teddy’s mention of when a model should have an additional parameter from Jevons?) can help elucidate the first question, and Bayesian meta-analysis, the second.
Exploratory factor analysis is one of the only tools for existential abduction (Haig, 2005). I want to propose new tools and procedures. As we abduct and enumerate possible causal explanations for an effect (phenomenon), we assign probabilities to each one; these are our priors. How we create a well-defined, mutually exclusive and exhaustive set of abductive inferences (hypotheses) about a phenomenon is currently unknown, and this dissertation hopes to illuminate. Each element—well-definedness, exclusivity, and exhaustiveness—are extremely difficult to achieve on their own. Thus existential abduction is difficult.