Comments on: Self Experimentation is Credible Science??

Self Experimentation is Credible Science??

peacay — Thu, 07 Apr 2005 11:11:37 -0800

"Chance favours the prepared mind" (Pasteur) but can a science of n = 1 be credible? Seth Roberts is a UCBerkeley Psychology Professor who is into generating novel scientific ideas from self-experimentation. He has written a very serious journal article (abstract) in Behaviour and Brain Science in which he alleges: Seeing faces in the morning on television decreased mood in the evening and improved mood the next day . . . Standing 8 hours per day reduced early awakening and made sleep more restorative . . . Drinking unflavored fructose water caused a large weight loss that has lasted more than 1 year.. among other things. The entire paper was published along with formal peer reviews and a response from Roberts [warning: 63page .pdf] (Peers came down about 50:50 in support/dissenting) A short review/discussion of the article and followup and a short followup Roberts paper with experimental replications (pdf) via

By: peacay

peacay — Thu, 07 Apr 2005 11:27:00 -0800

Some tangentially related stuff

By: anapestic

anapestic — Thu, 07 Apr 2005 11:37:32 -0800

It sounds to me like the guy wants to make a career out of studying himself. It should come as news to no one that you learn things about yourself and about how various stimuli affect you by paying attention to yourself. Raising it to the level of science seems silly, however.

By: gurple

gurple — Thu, 07 Apr 2005 11:43:59 -0800

The first real descriptions of the effects of cocaine were written by a doctor who tested it on himself subdermally. He upped the dosage gradually and was pretty much raving by the time he eschewed "scientific study" in favor of "tragic addiction". Go self-study!

By: ZenMasterThis

ZenMasterThis — Thu, 07 Apr 2005 11:50:30 -0800

To paraphrase Persig, the magic of science is in the formulation of the hypotheses. Roberts pitches his self-experimentation as "a source of new ideas." I would call his results an interesting beginning, worthy of further investigation, rather than an end. ...drinking a lot of fructose at once can cause diarrhea. Yep.

By: koeselitz

koeselitz — Thu, 07 Apr 2005 12:13:23 -0800

This is really interesting. I don't imagine this guy understands what he's getting into. So far, science, as we've formulated it for the last few hundred years, involves multiple tests of observable results to make sure they can be duplicated. The results aren't exactly observable, and the moments in a person's life can't exactly be duplicated. Really, he's leaving off science in favor of another kind of knowing. The "brain-and-consciousness" people might respond to him that his efforts are superfluous, as, in a generation or so, we'll be able to hook wires up to your brain and tell you what makes you happy, et cetera. But I think they're wrong: who can say what makes you happy, what makes you sad, what helps your mood, what helps you awaken? The facts are so complex as to require a great deal of attention paid to small, daily things. Each of us can only do this for ourselves. Freud, who tried to understand the human experience in concrete and unbending metaphysical terms, failed, I think, because he didn't do what this guy is doing: thinking about existence one moment at a time, and trying to evaluate it. The idea of doing "little experiments" was formulated a few years ago by Mr. Friedrich Nietzsche in one of his best and, sadly, least-remembered books, Daybreak.

By: anapestic

anapestic — Thu, 07 Apr 2005 12:24:49 -0800

Roberts pitches his self-experimentation as "a source of new ideas." I would call his results an interesting beginning, worthy of further investigation, rather than an end. I don't see how it's a beginning. He's merely doing what people have always done. His results boil down to: this is what works for me. Anyone who's ever kept a diet journal or a sleep journal, or almost any kind of journal has been doing self-experimentation. He's just dressing it up and getting funding for it.

By: ontic

ontic — Thu, 07 Apr 2005 12:26:29 -0800

Sure it's science. It's just not guaranteed to be useful science. I could test how I generally spend on gasoline every month. Again, scientific if done in the right way but not particularly useful. Socrates would be proud.

By: koeselitz

koeselitz — Thu, 07 Apr 2005 12:35:35 -0800

ontic: "Sure it's science. It's just not guaranteed to be useful science. I could test how I generally spend on gasoline every month. Again, scientific if done in the right way but not particularly useful." I don't understand. Isn't "does eating breakfast help me wake up?" an infinitely more useful experiment than, say, seeing how 100 people respond to certain stimuli in a laboratory? I guess maybe you mean something different by "useful." But when I say the word "useful," I mean "useful to me." Sure, it has less to do figuring out how to save the world, but my experience is that knowing my own body and my own reactions to various small daily variations is one of the major keys to finding happiness.

By: andrew cooke

andrew cooke — Thu, 07 Apr 2005 12:49:35 -0800

people are reading too much into this (both here and in some of the negative formal comments). his paper is about new ways to find ideas. he's not claiming that he's got strong evidence for stuff only that this is a way of exploring things that might be interesting. so he's not rewriting science and he's not breaking rules. he's just showing what he considers a good approach to the creative part of scientific work. so, for example, his paper is arguing that there is something interesting to look at about breakfasts. but no-one is saying that it's "hard science" until a large study is done. the two approaches complement each other. disclaimer - i've still not read the whole paper, but the abstract and some other links.

By: BlackLeotardFront

BlackLeotardFront — Thu, 07 Apr 2005 12:57:27 -0800

I don't think it's science.. it's experimentation, sure, but there is no advance of knowledge because we cannot generalize any of his conclusions. This guy is clearly a smart researcher and is serious about it, but he is the subject, experimenter, and statistician of it all and we can't just assume there is no bias. The scientific method is based in objectivity - there is none here. Isn't "does eating breakfast help me wake up?" an infinitely more useful experiment than, say, seeing how 100 people respond to certain stimuli in a laboratory? Koeselitz, I don't think so. Seeing 100 people doing things establishes actual trends within populations. From these trends we can make more hypotheses, look at the differences between the groups, and get to the bottom of the issue, which was the issue in the first place. If you are trying to figure out how to get up earlier, well, you don't need a study from a neuroscience journal to tell you how or why. But if you are trying to figure out which kinds of people get up earlier naturally, what it is that makes them do that, and so on, you need a large, double-blind experiment or even several. On preview: andrew cooke, I think you're right. It's certainly worth a damn if you look at it that way - a smart, experienced researcher noting trends in his everyday life and suggesting there may be material there for more serious, systematic research. That's definitely a perfectly all right way to think about it, I'm just cautioning that his results are strictly anecdotal and cannot be extended beyond himself.

By: googly

googly — Thu, 07 Apr 2005 12:57:41 -0800

Very interesting post, peacay. Thanks! I think that the comments so far are missing the point here. Roberts is certainly doing something scientific here - he is conducting an observational experiment, recording the results, and using his data to generate new hypotheses about the causal mechanisms involved. The reason it may not seem "scientific" or "useful" is that, because his sample size is so small (n=1), his results have a very low external validity - that is, they are very unlikely to be generalizable. But this problem pertains to almost all experimental research. Generalizing from experiments on 5, 15, or 50 people runs into the same problems. They may be more generalizable, but its a difference of degree rather than kind. This is why he's careful to say that his experiments are a source of "new ideas" - e.g., hypotheses that can be validated and generalized through future experiments. Also keep in mind that he is not reasoning from anecdote, as we generally do in daily life. He is not saying "gee, last week I drank a lot of water and my weight seemed to diminish, i guess that means drinking more water leads to weight loss." Instead he is carefully tracking his water intake and weight, controlling for other factors such as diet, and analyzing the results to see if there is a correlation.

By: fixedgear

fixedgear — Thu, 07 Apr 2005 13:10:56 -0800

He needs to talk to a few nutritionists with respect to his somewhat unorthodox ideas about weight gain and loss.

By: koeselitz

koeselitz — Thu, 07 Apr 2005 13:12:19 -0800

BlackLeotardFront: "Seeing 100 people doing things establishes actual trends within populations. From these trends we can make more hypotheses, look at the differences between the groups, and get to the bottom of the issue, which was the issue in the first place. If you are trying to figure out how to get up earlier, well, you don't need a study from a neuroscience journal to tell you how or why. But if you are trying to figure out which kinds of people get up earlier naturally, what it is that makes them do that, and so on, you need a large, double-blind experiment or even several." I still disagree. First, because knowing what kinds of people get up for what reasons when isn't as useful to me as knowing why I get up when I do; second, because even observation of people doesn't really yield very good results, because it can't possibly be clear observation of the cause. People are so complex that it requires incredibly close study to see the reasons why they do things; general studies aren't very helpful, I'm afraid, unless they're as simply physical, i. e. medical. It's remarkable, furthermore, how few people actually do such experiments for themselves, no matter how much it may seem like common sense. In past ages, some people had incredibly detailed knowledge of the human body and what kept it healthy; this was gleaned from self-observation. For centuries, doctors were only people who had unusually acute powers of self-observation, and who used those powers to help others. Nowadays, people are afraid to think anything about their own bodies if they aren't told to by a guy in a white coat. That worries me. This reminds me, however, of a joke: What did one behavioral scientist say to the other after sex? "It was good for you; was it good for me?" heh heh heh

By: forforf

forforf — Thu, 07 Apr 2005 13:51:47 -0800

I'm an engineer so my comment will have that bias in it. I agree with googly and others. The guy is making a contribution to science. He is observing a sample of 1, he is in effect creating a single data point, granted for lots of different things. A single data point isn't necessarily that useful to generalize from, but it does provide some very significant use. First, it proves that something is possible. Second, all it takes is others to replicate his experiments, and you will have statistical samples to draw from. This is much more likely to be done in a formal way now that he's done it and published it. The science isn't done, but it looks like its a start to me.

By: ontic

ontic — Thu, 07 Apr 2005 14:15:38 -0800

koeselitz: I was using "useful" to mean widely useful there, you're right. His results may be useless to you and me, but were we to try his method, we would get individually useful results. BlackLeotardFront: But knowledge has little to do with generalization -- perhaps the advance of knowledge in general does. But if knowledge had to be somewhat general, you couldn't scientifically study the last dodo bird, last panda, last anything in particular. You couldn't scientifically study any one-member classes. And that can't be right. If we find one dead alien, surely we can scietifically study it. As for objectivity, why doesn't repetition and variance under different experimental conditions count towards objectivity? Isn't this exactly what Sanctorius (Santorio? Wikipedia seems to have failed me on this one??) was doing with his early experiments on metabolism?

By: OmieWise

OmieWise — Thu, 07 Apr 2005 14:24:42 -0800

See the last link for a few other data points that replicate some of his results. Still not particularly generalizable, but more than anecdotal.

By: _sirmissalot_

_sirmissalot_ — Thu, 07 Apr 2005 14:50:10 -0800

I'm totally going to try that sugar water diet.

By: quadog

quadog — Thu, 07 Apr 2005 14:53:12 -0800

Doesn't this guy have any respect. What would happen if we ever take experimentation out of the the hands of accredited scientists!

By: davejay

davejay — Thu, 07 Apr 2005 14:54:49 -0800

He's just dressing it up and getting funding for it. And is, therefore, a genius. Just not necessarily a scientific one.

By: daver

daver — Thu, 07 Apr 2005 16:49:04 -0800

Hmm. If the only real problem here is sample size, why not recruit volunteers, seti@home style. As _sirmissalot_ demonstrates, it may not be that hard to recruit subjects. A site could facilitate experimental design and review, recruiting, instructions, experimental notes/data gathering, and summaries. Issues would probably be around the reliability of subjects and falsification of data (intentional or not), but they may be possible to deal with. experiments would also need to be designed appropriately and possibly reviewed. Folksperiments to go along with folksonomies. Obviously this applies to the experimental part of his paper, not to the idea generation part. I don't see why the idea generation part wouldn't benefit as well though... On the otherhand, it sounds like it would be a nightmare to administer. Matt, you need another site to run?

By: anapestic

anapestic — Thu, 07 Apr 2005 17:36:30 -0800

He doesn't want to increase his sample size. The whole point is that the sample size is only one. I have trouble believing that people are taking this so seriously. Are academics that hard up for work?

By: painquale

painquale — Thu, 07 Apr 2005 23:36:37 -0800

When an experiment (in, say, physics) takes a great deal of time and cost to perform, and scientists are happy with the results, then it often just gets performed once. Why run it again? This reminds me of what this guy is doing. He's making his own little private tests that would be difficult/costly/too offbeat to apply to large groups, and checking out what happens. If he discovers some weird or unexpected phenomenon, then it's time to draw up a battery of tests. This "n=1" stuff goes on in science all the time... it just looks really weird in psychology. Then again, some of his tests could, I think, show a lot of observer bias and placebo effects. If he wants a certain effect badly enough, then his body might start manifesting it just because of his desire. Stuff like that happens.

By: BlackLeotardFront

BlackLeotardFront — Fri, 08 Apr 2005 01:58:08 -0800

Koeselitz, I totally see what you're saying. What I mean is only that his knowledge of himself is only applicable to himself - aside from the single data-point idea that forforf mentions. Psychology is about establishing trends in populations and if possible determining their origins - but anything more detailed than that has to be studied on an individual level as this fellow has done. Certainly both have their merits, but I was speaking of his work in the context of the "advancement of knowledge in general".. whatever that really means to you and me. Ontic, certainly those conventions of experimentation give a nod towards objectivity, but if you are the one observing yourself there is no way to check for bias. He's being a sort of intellectual Judge Dredd - Judge, jury, and executioner ... of knowledge! Anyway it's still cool stuff and not really worth quibbling over its perceived contribution to "science."

By: klangklangston

klangklangston — Fri, 08 Apr 2005 06:03:48 -0800

To echo BLF: While I think this is incredibly cool in an OCD way, the reason why this isn't necessarily useful is that the only thing coming out of it that could be used by more than Seth Roberts is the idea of doing incredibly disciplined experiments and research collection by individuals for effects in their lives. But it's not useful for sleep studies or obesity without increasing the sample size (something he seems weirdly dismissive of) because it's a "that works for me" sort of thing. If he finds that he's happier only when he eats Strawberry Poptarts, that doesn't tell us anything about anyone else, except that they should keep track of what they eat like he did. It's scientific narcisissism. And at best, it's a method for individualized results, but his argument that somehow ideas applicable to more than just Seth Roberts could result from his Seth Roberts experiments strikes me as kind of hollow.

By: donfactor

donfactor — Fri, 08 Apr 2005 11:15:04 -0800

What this debate adds up to in my mind (sample of 1) is that we have yet to ask, Where do no ideas come from?

By: peacay

peacay — Fri, 08 Apr 2005 15:02:50 -0800

donfactor....do you mean 'new ideas'? I like Robert's tenacity and rigorous adherence, as far as possible under the circumstances, to rational scientific experimental principles. That is a very kosher write-up, leaving aside the bigger issues of statistical aberrances and personal bias invalidation. Mostly I was interested in Roberts overall concept of trying to find plausible new ways of thinking scientifically - a kind of a new angle on the usual thing of merely standing on the shoulders of those that go before you. That he openly admits that he's searching for the ideas rather than experimental conclusions is to his credit. Yes, n does = 1, but I don't think that detracts from the philosophical aim. His unique science to me is a wake up call or an urging for people to look at the normal things with a different viewpoint - that was my take from the journal article and I think that's a commendable aim.