Dear Bayesians,

A question from Dennis Lindley prompts me to consult this list in search of answers.

We've heard a lot about the Higgs boson. The news reports say that the LHC needed convincing evidence before they would announce that a particle had been found that looks like (in the sense of having some of the right characteristics of) the elusive Higgs boson. Specifically, the news referred to a confidence interval with 5-sigma limits.

Now this appears to correspond to a frequentist significance test with an extreme significance level. Five standard deviations, assuming normality, means a p-value of around 0.0000005. A number of questions spring to mind.

1. Why such an extreme evidence requirement? We know from a Bayesian perspective that this only makes sense if (a) the existence of the Higgs boson (or some other particle sharing some of its properties) has extremely small prior probability and/or (b) the consequences of erroneously announcing its discovery are dire in the extreme. Neither seems to be the case, so why 5-sigma?

2. Rather than ad hoc justification of a p-value, it is of course better to do a proper Bayesian analysis. Are the particle physics community completely wedded to frequentist analysis? If so, has anyone tried to explain what bad science that is?

3. We know that given enough data it is nearly always possible for a significance test to reject the null hypothesis at arbitrarily low p-values, simply because the parameter will never be exactly equal to its null value. And apparently the LNC has accumulated a very large quantity of data. So could even this extreme p-value be illusory?

If anyone has any answers to these or related questions, I'd be interested to know and will be sure to pass them on to Dennis.

Regards,

Tony

--

--

Professor A O'Hagan Email:

a.ohagan@sheffield.ac.uk
Department of Probability and Statistics

University of Sheffield Phone: +44 114 222 3773

Hicks Building

Sheffield S3 7RH, UK Fax: +44 114 222 3759

-----------

http://www.tonyohagan.co.uk/ ------------

************************************************************************

ISBA now maintains the bayes-news and Valencia email lists; you are

receiving this message as you have opted-in to receive email from

bayes-news, the Valencia list or the ISBA forums. To opt out or change

your email subscriptions please login at http://bayesian.org/user. Go

to the My Account menu and select Forum Email Integration. Check the

forums to which you wish to subscribe, then save your settings. You

may view past and present content at any time by visiting

http://bayesian.org/forum

If someone can get these data to do a Bayesian analysis, I would be interested in seeing the results (and evaluating the underlying assumptions!)

Jeff Morris

--

From: ISBA Webmaster [mailto:hans@stat.duke.edu]

Sent: Tuesday, July 10, 2012 8:47 PM

To: news@bayesian.org

Subject: Higgs boson

Dear Bayesians,

A question from Dennis Lindley prompts me to consult this list in search of answers.

We've heard a lot about the Higgs boson. The news reports say that the LHC needed convincing evidence before they would announce that a particle had been found that looks like (in the sense of having some of the right characteristics of) the elusive Higgs boson. Specifically, the news referred to a confidence interval with 5-sigma limits.

Now this appears to correspond to a frequentist significance test with an extreme significance level. Five standard deviations, assuming normality, means a p-value of around 0.0000005. A number of questions spring to mind.

1. Why such an extreme evidence requirement? We know from a Bayesian perspective that this only makes sense if (a) the existence of the Higgs boson (or some other particle sharing some of its properties) has extremely small prior probability and/or (b) the consequences of erroneously announcing its discovery are dire in the extreme. Neither seems to be the case, so why 5-sigma?

2. Rather than ad hoc justification of a p-value, it is of course better to do a proper Bayesian analysis. Are the particle physics community completely wedded to frequentist analysis? If so, has anyone tried to explain what bad science that is?

3. We know that given enough data it is nearly always possible for a significance test to reject the null hypothesis at arbitrarily low p-values,

simply because the parameter will never be exactly equal to its null value.

And apparently the LNC has accumulated a very large quantity of data. So could even this extreme p-value be illusory?

If anyone has any answers to these or related questions, I'd be interested to know and will be sure to pass them on to Dennis.

Regards,

Tony

----

Professor A O'Hagan Email: a.ohagan@sheffield.ac.uk

Department of Probability and Statistics

University of Sheffield Phone: +44 114 222 3773

Hicks Building

Sheffield S3 7RH, UK Fax: +44 114 222 3759

----------- http://www.tonyohagan.co.uk/ ------------

************************************************************************

ISBA now maintains the bayes-news and Valencia email lists; you are receiving this message as you have opted-in to receive email from bayes-news, the Valencia list or the ISBA forums. To opt out or change your email subscriptions please login at http://bayesian.org/user. Go to the My Account menu and select Forum Email Integration. Check the forums to which you wish to subscribe, then save your settings. You may view past and present content at any time by visiting http://bayesian.org/forum

I'd offer the following partial answers:

1. A 5-sigma rule has become a tradition in particle physics, but it is not clear why.

2. Yes, there have been Bayesian monographs written by and for physicists. Enrico Fermi seems to have recognized the issue. I cite two publications at http://for-sci-law-now.blogspot.com/2012/07/probability-that-higgs-boson...

3. Given the confluence of theory and other, less compelling experiments, it looks like there is a particle with the predicted mass producing the observed decay products. At any rate, that is the consensus of the physics community.

Cavaet: I am no physicist. I am just synthesizing what I have read.

David Kaye

http://www.personal.psu.edu/dhk3

________________________________________

From: ISBA Webmaster [hans@stat.duke.edu]

Sent: Tuesday, July 10, 2012 9:46 PM

To: news@bayesian.org

Subject: Higgs boson

Dear Bayesians,

A question from Dennis Lindley prompts me to consult this list in search of

answers.

We've heard a lot about the Higgs boson. The news reports say that the LHC

needed convincing evidence before they would announce that a particle had

been found that looks like (in the sense of having some of the right

characteristics of) the elusive Higgs boson. Specifically, the news referred

to a confidence interval with 5-sigma limits.

Now this appears to correspond to a frequentist significance test with an

extreme significance level. Five standard deviations, assuming normality,

means a p-value of around 0.0000005. A number of questions spring to mind.

1. Why such an extreme evidence requirement? We know from a Bayesian

perspective that this only makes sense if (a) the existence of the Higgs

boson (or some other particle sharing some of its properties) has extremely

small prior probability and/or (b) the consequences of erroneously announcing

its discovery are dire in the extreme. Neither seems to be the case, so why

5-sigma?

2. Rather than ad hoc justification of a p-value, it is of course better to

do a proper Bayesian analysis. Are the particle physics community completely

wedded to frequentist analysis? If so, has anyone tried to explain what bad

science that is?

3. We know that given enough data it is nearly always possible for a

significance test to reject the null hypothesis at arbitrarily low p-values,

simply because the parameter will never be exactly equal to its null value.

And apparently the LNC has accumulated a very large quantity of data. So

could even this extreme p-value be illusory?

If anyone has any answers to these or related questions, I'd be interested to

know and will be sure to pass them on to Dennis.

Regards,

Tony

--

Professor A O'Hagan Email: a.ohagan@sheffield.ac.uk

Department of Probability and Statistics

University of Sheffield Phone: +44 114 222 3773

Hicks Building

Sheffield S3 7RH, UK Fax: +44 114 222 3759

----------- http://www.tonyohagan.co.uk/ ------------

************************************************************************

ISBA now maintains the bayes-news and Valencia email lists; you are

receiving this message as you have opted-in to receive email from

bayes-news, the Valencia list or the ISBA forums. To opt out or change

your email subscriptions please login at http://bayesian.org/user. Go

to the My Account menu and select Forum Email Integration. Check the

forums to which you wish to subscribe, then save your settings. You

may view past and present content at any time by visiting

http://bayesian.org/forum

[Moderator's note: this message is from

Harrison B. Prosper

Kirby W. Kemper Professor of Physics

Distinguished Research Professor

Florida State University

harry@hep.fsu.edu

]

Dear Tony,

First some general remarks, then I'll try to answer your questions.

I am in an interesting position regarding the "Higgs" boson discovery: I am thrilled to be an insider with respect to the discovery and I happen also to be one of the relatively few particle physicists who actually regard Bayesian reasoning as "exactly what is needed" to make sense of what we do. The vast majority of my colleagues believe that p-values are objective and therefore "scientific". Therefore, many of my colleagues move mountains, or at any rate consume prodigious amounts of computing power, to check that some (typically ad hoc) procedure covers.

For your edification I've attached a (PUBLIC!) plot [Moderator's note: available at http://bayesian.org/webfm_send/274 ] of (slightly massaged) binned data from my collaboration (CMS) that shows a spectrum arising from proton-proton collisions that resulted in the creation of a pair of photons (gammas in high energy argot). The Standard Model predicts that the Higgs boson should decay (that is break up) into a pair of photons. (The Higgs is predicted to decay in other ways too, such as a pair of Z bosons.) The bump in the plot at around 125 GeV is evidence for the existence of some particle of a definite mass that decays into a pair of photons. That something, as far as we've been able to ascertain, is likely to be the Higgs boson. These data, along with data in which proton-proton collisions yield two Z bosons are the basis of our 5-sigma claim.

These data can be modeled with the function

f(x) = exp(a0 + a1*x + a2*x^2) + s * Gaussian(x|m, w)

where "x" is the mass the di-photon (pair of photons), and the first term describes the smoothly falling (background) spectrum, while the second term models the bump. "s" is the total expected signal, "m" is the mass of the new particle and "w" is the width of the bump. The total background, that is, the "noise" is just the integral of the first term. "a0", "a1", "a2" are nuisance parameters. This is therefore a 6-parameter problem for which we have no prior information (or choose to act as if this is so) for the six parameters. The analysis of this spectrum has caused a lot of angst about the "look-elsewhere-effect" (multiple hypothesis testing), which I think is a red herring in this context.

Now for your questions. See below.

Harrison

Quoting ISBA Webmaster <hans@stat.duke.edu>:

Dear Bayesians,

A question from Dennis Lindley prompts me to consult this list in search of

answers.

We've heard a lot about the Higgs boson. The news reports say that the LHC

needed convincing evidence before they would announce that a particle had

been found that looks like (in the sense of having some of the right

characteristics of) the elusive Higgs boson. Specifically, the news referred

to a confidence interval with 5-sigma limits.

Now this appears to correspond to a frequentist significance test with an

extreme significance level. Five standard deviations, assuming normality,

means a p-value of around 0.0000005. A number of questions spring to mind.

1. Why such an extreme evidence requirement? We know from a Bayesian

perspective that this only makes sense if (a) the existence of the Higgs

boson (or some other particle sharing some of its properties) has extremely

small prior probability and/or (b) the consequences of erroneously announcing

its discovery are dire in the extreme. Neither seems to be the case, so why

5-sigma?

The "5-sigma" (p-value = 3.0e-7) is an historical artifact. Over the past several decades, we have made many a "discovery" that turned out not to be so. As a consequence, we gradually settled on a p-value thought to be small enough to reduce the chance that we are fooling ourselves. In fact, we do have high standards because in our view we are trying to arrive at "true" statements about the world in the pragmatic sense that these statements yield predictions that turn out to be correct. Given that the search for the Higgs took some 45 years, tens of thousands of scientists and engineers, billions of dollars, not to mention numerous divorces, huge amounts of sleep deprivation, tens of thousands of bad airline meals, etc., etc., we want to be sure as is humanly possible that this is real.

2. Rather than ad hoc justification of a p-value, it is of course better to

do a proper Bayesian analysis. Are the particle physics community completely

wedded to frequentist analysis? If so, has anyone tried to explain what bad

science that is?

I for one would be delighted to see a Bayesian analysis of these data from you guys! Unfortunately, however, I am forbidden from e-mailing you the 50,000 "x"s I have right here on my laptop...very frustrating...

3. We know that given enough data it is nearly always possible for a

significance test to reject the null hypothesis at arbitrarily low p-values,

simply because the parameter will never be exactly equal to its null value.

And apparently the LNC has accumulated a very large quantity of data. So

could even this extreme p-value be illusory?

As noted above, small p-value-based "discoveries" have come and gone.

However, the reason I am convinced this is real is not because of the p-value, nor frankly because of the (pseudo)-frequentist method of analysis showcased on July 4th during the announcement at CERN, a method that I have repeatedly criticized within my collaboration. Rather it is because when I study the profile likelihood in the variables "s", "m", and "w" for the 2011 dataset and for the 2012 dataset, I find visually convincing structures in the profile likelihood at m ~ 125 GeV in both independent datasets, obtained at different proton beam energies (7 TeV and 8 TeV). Of course, I would rather have preferred to have done a Bayesian analysis, marginalizing over "a0", "a1", "a2", and "w", and to study the posterior density in the variables "s" and "m", but constructing a non-evidence-based prior in 4-dimensions that would pass muster seems quite a chore. Any advice from You would be welcome. (I favor the recursive reference prior algorithm of Bernardo, but this would have to be done numerically and I have not yet figured out how to do so efficiently, while taking into account the "nested compact sets". The whole thing seems rather daunting.)

If anyone has any answers to these or related questions, I'd be interested to

know and will be sure to pass them on to Dennis.

Regards,

Tony

--

Professor A O'Hagan Email: a.ohagan@sheffield.ac.uk

Department of Probability and Statistics

University of Sheffield Phone: +44 114 222 3773

Hicks Building

Sheffield S3 7RH, UK Fax: +44 114 222 3759

----------- http://www.tonyohagan.co.uk/ ------------

************************************************************************

ISBA now maintains the bayes-news and Valencia email lists; you are

receiving this message as you have opted-in to receive email from

bayes-news, the Valencia list or the ISBA forums. To opt out or change

your email subscriptions please login at http://bayesian.org/user. Go

to the My Account menu and select Forum Email Integration. Check the

forums to which you wish to subscribe, then save your settings. You

may view past and present content at any time by visiting

http://bayesian.org/forum

--

This message has been scanned for viruses and

dangerous content by MailScanner, and is

believed to be clean.

--

Harrison B. Prosper

Kirby W. Kemper Professor of Physics

Distinguished Research Professor

Florida State University

harry@hep.fsu.edu

Tel: 850 644 6760

Fax: 850 644 6735

In the best character there is an element of pride - not the sort of

pride that despises others, but the sort that will not be deflected

from what it thinks good by outside pressure.

"The Value of Free Thought", Bertrand Russell

----------------------------------------------------------------

Hi Tony,

There is a group of high energy physicists that think very hard of source detection, model comparison, and statistical foundations. They host periodic workshops (https://plone4.fnal.gov:4430/P0/phystat/ and most recently http://www-conf.slac.stanford.edu/statisticalissues2012/) and regularly invite prominent statistician (Bayesian or otherwise) to participate. These meetings are great fun because of the active and informed dialogue on exactly the sorts of issues that you mention. Some physicists in this group remain skeptical about Bayesian procedures while others advocate for Bayes.

Unfortunately, this group does not get the final word on the statistical procedures that are actually used. The search for the Higgs boson in both incredibly high profile and incredibly expensive. There is a bureaucracy that determines in advance the procedures (statistical and otherwise) that will be implemented. Like most bureaucracies, this one is rather conservative and is unlikely to be far out-of-step with the general consensus of its scientific community, a community that is much larger than group that participates in Phystat workshops. So far, the official statistical methods have not been Bayesian.

As for why high-energy physicists require "5 sigma" to claim a discovery, there have been high profile cases where seemingly highly significant detections have had to be recalled. The response of editors at the leading journals in the field has been to require "more sigma". My personal opinion on the matter is that the problem is not with the significance level of the tests, but that there are statistical effects that are nor accounted for properly by the procedures (e.g., model misspecification or "systematic errors", unaccounted for effects of uncertainty in instrument calibration, and the so-called "look elsewhere effect", a sort of multiple testing).

If you are interested in reading more, there are a couple of good places to start:

1. An overview of statistical issues in particle physics<http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aoas/1223908045> by Louis Lyons published in The Annals of Applied Statistics (Vol 2, pp. 887-915).

2. An older discussion pape<http://www2.imperial.ac.uk/~dvandyk/presentations-astrostat.php>r in Statistical Science by Mark Mandelkern (Vol 17, pp. 149-172)

3. I discussed the question of "why 5 sigma" at a talk in a 2010 BIRS Workshop: Talk #15 at http://www2.imperial.ac.uk/~dvandyk/presentations-astrostat.php<http://w...

David

David A. van Dyk

Chair in Statistics

Department of Mathematics

Imperial College London SW7 2AZ

URL: http://www2.imperial.ac.uk/~dvandyk/

phone: +44 20 7594 8574

office: 535 Huxley Building

On Jul 11, 2012, at 2:46 AM, ISBA Webmaster wrote:

Dear Bayesians,

A question from Dennis Lindley prompts me to consult this list in search of

answers.

needed convincing evidence before they would announce that a particle had

been found that looks like (in the sense of having some of the right

characteristics of) the elusive Higgs boson. Specifically, the news referred

to a confidence interval with 5-sigma limits.

extreme significance level. Five standard deviations, assuming normality,

means a p-value of around 0.0000005. A number of questions spring to mind.

perspective that this only makes sense if (a) the existence of the Higgs

boson (or some other particle sharing some of its properties) has extremely

small prior probability and/or (b) the consequences of erroneously announcing

its discovery are dire in the extreme. Neither seems to be the case, so why

5-sigma?

do a proper Bayesian analysis. Are the particle physics community completely

wedded to frequentist analysis? If so, has anyone tried to explain what bad

science that is?

significance test to reject the null hypothesis at arbitrarily low p-values,

simply because the parameter will never be exactly equal to its null value.

And apparently the LNC has accumulated a very large quantity of data. So

could even this extreme p-value be illusory?

know and will be sure to pass them on to Dennis.

Regards,

Tony

--

Professor A O'Hagan Email: a.ohagan@sheffield.ac.uk<mailto:a.ohagan@sheffield.ac.uk>

Department of Probability and Statistics

University of Sheffield Phone: +44 114 222 3773

Hicks Building

Sheffield S3 7RH, UK Fax: +44 114 222 3759

----------- http://www.tonyohagan.co.uk/ ------------

ISBA now maintains the bayes-news and Valencia email lists; you are

receiving this message as you have opted-in to receive email from

bayes-news, the Valencia list or the ISBA forums. To opt out or change

your email subscriptions please login at http://bayesian.org/user. Go

to the My Account menu and select Forum Email Integration. Check the

forums to which you wish to subscribe, then save your settings. You

may view past and present content at any time by visiting

http://bayesian.org/forum

Dear Tony, dear all,

this paper provides some hints

http://arxiv.org/abs/1112.3620

(see also http://www.roma1.infn.it/~dagos/badmath/index.html#added )

Moreover

- The "higgs p-values" does not seem to be what a professional

(frequentistic) statistician would mean by that term:

-> there is no serious null hypothesis without Higgs, because

a Standard Model without Higgs mechanism loses completely meaning.

-> what has the meaning of a p-value depending on the mass?

(a number, calculated in the hypothesis that the Higgs does not

exist, reported in function of its mass...)

- Also the "95% CL exclusion" regions have dubious meaning

because they are derived by "prescriptions" that do not

provide a quantitative statement of how much we should

be confidence on something.

Regards, and best greetings to Dennis,

Giulio

On Tue, 10 Jul 2012, ISBA Webmaster wrote:

Hello Bayesians,

Below are a few answers. Thnaks for the interest

Louis Lyons (organiser of PHYSTAT series of meetings, and member of CMS Collaboration at CERN.)

________________________________________

From: ISBA Webmaster [hans@stat.duke.edu]

Sent: 11 July 2012 02:46

To: news@bayesian.org

Subject: Higgs boson

Dear Bayesians,

A question from Dennis Lindley prompts me to consult this list in search of

answers.

We've heard a lot about the Higgs boson. The news reports say that the LHC

needed convincing evidence before they would announce that a particle had

been found that looks like (in the sense of having some of the right

characteristics of) the elusive Higgs boson.

************ The test statistic we use for looking at p-values is basically the likelihood

ratio for the two hypotheses (H_0 = Standard Model (S. M.) of Particle Physics, but no Higgs;

H_1 = S.M with Higgs). A small p_0 (and a reasonable p_1) then implies that H_1 is a better

description of the data than H_0. This of course does not prove that H_1 is correct, but

maybe Nature corresponds to some H_2, which is more like H_1 than it is like H_0. Indeed

in principle data will never prove a theory is true, but the more experimental tests it survives,

the happier we are to use it - e.g. Newtonian mechanics was fine for centuries till the arrival

of Relativity.

************* In the case of the Higgs, it can decay to different sets of particles, and these rates

are defined by the S.M. We measure these ratios, but with large uncertainties with the present data.

They are consistent with the S.M. predictions, but it could be much more convincing with more

data. Hence the caution about saying we have discovered the Higgs of the S. M..

Specifically, the news referred

to a confidence interval with 5-sigma limits.

************** 5-sigma really refers to p_0

extreme significance level. Five standard deviations, assuming normality,

means a p-value of around 0.0000005. A number of questions spring to mind.

1. Why such an extreme evidence requirement? We know from a Bayesian

perspective that this only makes sense if (a) the existence of the Higgs

boson (or some other particle sharing some of its properties) has extremely

small prior probability and/or (b) the consequences of erroneously announcing

its discovery are dire in the extreme. Neither seems to be the case, so why

5-sigma?

********************** This is an unfortunate tradition, that is used more readily

by journal editors than by Particle Physicists. Reasons are

a) Historically we have had 3 and 4 sigma effects that have gone away

b) The 'Look Elsewhere Effect' (LEE). We are worried about the chance of a

statistical fluctuation mimicking our observation, not only at the given mass

of 125 GeV but anywhere in the spectrum. The quoted p-values are

'local' i.e. the chance of a fluctuation at the observed mass. Unfortunately

the LEE correction factor is not very precisely defined, because of ambiguities about

what is meant by 'elsewhere'

c) The possibility of some systematic effect (characterised by a nuisance parameter)

being more important than allowed for in the analysis, or even overlooked - see the

recent experiment at CERN which claimed that neutrinos travelled faster than the speed of

light.

d) A subconscious use of Bayes Theorem to turn p-values into probabilities about the

hypotheses.

All the above vary from experiment to experiment, so we realise that it is a bit unfair to

use the same standard for discovery for all analyses. We prefer just to quote the p-values

(or whatever).

2. Rather than ad hoc justification of a p-value, it is of course better to

do a proper Bayesian analysis. Are the particle physics community completely

wedded to frequentist analysis?

************** No we are not anti-Bayesian, and indeed our test statistics is a likelihood ratio.

If you like, you can regard our p-values as an attempt to calibrate the meaning of a

particular value of the likelihood ratio.

************** We actually recommend that for parameter determination at the LHC, it is

useful to compare Bayesian and Frequentist methods. But for comparing hypotheses

(e.g. an experimental distribution is fitted by H_0 = a smooth distribution; or by H_1 = a smooth

distribution plus a localised peak), we are worried about what priors to use for the extra

parameters that occur in the alternative hypothesis. ******* We would welcome advice.**********

If so, has anyone tried to explain what bad

science that is?

*************** Comment ignored

3. We know that given enough data it is nearly always possible for a

significance test to reject the null hypothesis at arbitrarily low p-values,

simply because the parameter will never be exactly equal to its null value.

And apparently the LHC has accumulated a very large quantity of data. So

could even this extreme p-value be illusory?

************** We are aware of this. But in fact, although the LHC has accumulated enormous

amounts of data, the Higgs search is like looking for a needle in a haystack. The final samples

of events that are used to look for the Higgs contain only tens to thousands of events.

If anyone has any answers to these or related questions,

***************** These and related issues are discussed to some extent in my

article "Open statistical issues in Particle Physics", Ann. Appl. Stat. Volume 2, Number 3 (2008),

887-915. It is supposed to be statistician-friendly

I'd be interested to know and will be sure to pass them on to Dennis.

Regards,

Tony

--

Professor A O'Hagan Email: a.ohagan@sheffield.ac.uk

Department of Probability and Statistics

University of Sheffield Phone: +44 114 222 3773

Hicks Building

Sheffield S3 7RH, UK Fax: +44 114 222 3759

----------- http://www.tonyohagan.co.uk/ ------------

ISBA now maintains the bayes-news and Valencia email lists; you are

receiving this message as you have opted-in to receive email from

bayes-news, the Valencia list or the ISBA forums. To opt out or change

your email subscriptions please login at http://bayesian.org/user. Go

to the My Account menu and select Forum Email Integration. Check the

forums to which you wish to subscribe, then save your settings. You

may view past and present content at any time by visiting

http://bayesian.org/forum

Dear Tony

I have written a bit about the explanation of the P-value here

http://understandinguncertainty.org/explaining-5-sigma-higgs-how-well-di...

The CERN teams' reports also discuss what they would expect were the

Higgs there, so there seems a real possibility of a likelihood ratio

being computed, which would be a start. Not sure why they don't do this.

d

On 11/07/2012 02:46, ISBA Webmaster wrote:

--

David Spiegelhalter

Winton Professor for the Public Understanding of Risk

Statistical Laboratory

Centre for Mathematical Sciences

Wilberforce Road

Cambridge

CB3 0WB

UK

Tel: +44 (0)1223 337945

Fax: +44 (0)1223 337956

www.statslab.cam.ac.uk/Dept/People/Spiegelhalter/davids.html

www.understandinguncertainty.org

Hi Tony

I was on the statistics committee for one of the detectors

at Fermi-Lab for a year. These physicists think about these things pretty

carefully.

The 5-sigma rule is for two-reasons; first, the stakes are very high

and protecting against a false positive is very important. Billions of

dollars

are at stake not to mention Nobel prizes. Also, the entire future direction

of particle physics research depends on getting it right.

Second, there is much more to the analysis than just a single

hypothesis test. Using a small p-value cut-off protects from

implicit multiple testing problems.

They are aware of Bayesian methods. There are some particle physicists

who write about them. But they mainly use frequentist methods for good

reasons: they are objective. (Niether you or Dennis will agree with that

but I do.)

The p-value is not illusory. This is not social science where the

null is always false and it is only a matter fo time until we reject.

In this case, the physics defines the null and alternatively clearly.

I hope this helps

Larry

On Tue, Jul 10, 2012 at 9:46 PM, ISBA Webmaster <hans@stat.duke.edu> wrote:

--

Larry Wasserman

Professor, Department of Statistics

and Machine Learning Department

Carnegie Mellon University

larry@stat.cmu.edu

www.stat.cmu.edu/~larry <http://www.stat.cmu.edu/%7Elarry>

-------

Please DO NOT SEND Microsoft Word attachments to me. Word is a

secret proprietary format, so I cannot read it. If you send me pdf,

plain text or HTML, then I can read it.

See

www.gnu.org/philosophy/no-word-attachments.html

for more information.

Is there any possibility of getting a Savage (et al.) - Berger (et al.) type of bound for a realistic version of this problem? It would be to the point and "objective".

Best to you all,

John

________________________________________

From: ISBA Webmaster [hans@stat.duke.edu]

Sent: Tuesday, July 10, 2012 9:46 PM

To: news@bayesian.org

Subject: Higgs boson

Dear Bayesians,

A question from Dennis Lindley prompts me to consult this list in search of

answers.

needed convincing evidence before they would announce that a particle had

been found that looks like (in the sense of having some of the right

characteristics of) the elusive Higgs boson. Specifically, the news referred

to a confidence interval with 5-sigma limits.

extreme significance level. Five standard deviations, assuming normality,

means a p-value of around 0.0000005. A number of questions spring to mind.

perspective that this only makes sense if (a) the existence of the Higgs

boson (or some other particle sharing some of its properties) has extremely

small prior probability and/or (b) the consequences of erroneously announcing

its discovery are dire in the extreme. Neither seems to be the case, so why

5-sigma?

do a proper Bayesian analysis. Are the particle physics community completely

wedded to frequentist analysis? If so, has anyone tried to explain what bad

science that is?

significance test to reject the null hypothesis at arbitrarily low p-values,

simply because the parameter will never be exactly equal to its null value.

And apparently the LNC has accumulated a very large quantity of data. So

could even this extreme p-value be illusory?

know and will be sure to pass them on to Dennis.

Regards,

Tony

--

Professor A O'Hagan Email: a.ohagan@sheffield.ac.uk

Department of Probability and Statistics

University of Sheffield Phone: +44 114 222 3773

Hicks Building

Sheffield S3 7RH, UK Fax: +44 114 222 3759

----------- http://www.tonyohagan.co.uk/ ------------

ISBA now maintains the bayes-news and Valencia email lists; you are

receiving this message as you have opted-in to receive email from

bayes-news, the Valencia list or the ISBA forums. To opt out or change

your email subscriptions please login at http://bayesian.org/user. Go

to the My Account menu and select Forum Email Integration. Check the

forums to which you wish to subscribe, then save your settings. You

may view past and present content at any time by visiting

http://bayesian.org/forum

[Moderator's note: this message is from

Bill Jefferys

bill@astro.as.utexas.edu

Department of Astronomy

The University of Texas at Austin

Department of Mathematics and Statistics

The University of Vermont

]

Dear Tony,

A CERN physicist wrote the following blog entry, which directly

addresses your third point:

http://www.science20.com/quantum_diaries_survivor/blog/keeplooking_bias

I am not a high-energy physicist, although I subscribe to a list that is

inhabited by a lot of them and where these issues are discussed almost

every day. As others have noted, and from my discussions with these

high-energy physicists, the reason for 5 sigmas is just that they have

been burned too often using less stringent criteria. And yes, it is a

problem in this expensive field when you announce results that you may

have to withdraw later. Better to be cautious seems to be the attitude.

It does seem that there is a preference in this field for

frequentist/likelihoodist statistics, though there are some who do use

Bayesian methods; the journals want frequentist statistics, however.

[Moderator's note: the following link was supplied by Bill in a follow-up comment:

http://www.technologyreview.com/view/428428/higgs-boson-may-be-an-impost...

This is one of the reasons for the very cautious words that you heard in

the press conference!

]

Bill Jefferys

Department of Astronomy

The University of Texas at Austin

Department of Mathematics and Statistics

The University of Vermont

--

On 7/10/12 9:46 PM, ISBA Webmaster wrote:

[Moderator's note: the following message was sent from

hrubin@stat.purdue.edu

and references a message sent by:

JPRATT@HBS.EDU (John below)

]

On Wed, 11 Jul 2012, ISBA Webmaster wrote:

I do not agree with the pseudo-objective Berger approach, but I consider

the proper use of statistics to be in the balancing of risks. In most

cases, I do not believe it can be put in the form of a Bayes factor,

and alas, this is one of them.

In the assumed null model, there is a distribution of certain types

of events. If, and here are several ifs, there is a particle in a

fairly small mass range, which has the presumed properties of the

Higgs boson, the distribution due to it will be of a slightly different

form. So here we can get the likelihood ratio, or a good approximation

thereof, as a function of the frequency of Higgs decays, which we seem

not to have a good idea about; if we had, we could just use a likelihood

ratio test and get the Bayes factor.

So we are again in the situation of testing a point null versus

a non-null effect. Here we have a one-sided test, but I have

found (not yet written up) that the asymptotic results in my

paper with Sethuraman (Sankhya 1965) does work even for fairly

small samples. What counts is the density under the alternative

relative to the mass of the null in the units used. If one

does not wish to make this constant locally, but have it behave

like a power or something else, that change should be made.

If the Type II risk is constant, the result calculates the

posterior probability of the alternative; the prior probability

is unimportant, as large values of the alternative parameter are

going to reject with very high probability. These large values

definitely have not been found. And while the calculations in

our paper technically have an infinite prior integral for the

alternative, that is irrerlevant, again, even for moderate

sized samples.

[Moderator's note: the following message was sent by

Peter Diggle

P.DIGGLE@LANCASTER.AC.UK

]

Well, the extremity of the physicists' p-values pales into insignificance (sorry!) when compared with p-values routinely quoted in genetics papers. I'm sure Nature has published p-values as small as 10^(-31).

Peter Diggle

--

From: ISBA Webmaster [mailto:hans@stat.duke.edu]

Sent: 11 July 2012 02:47

To: news@bayesian.org

Subject: Higgs boson

Dear Bayesians,

A question from Dennis Lindley prompts me to consult this list in search of answers.

simply because the parameter will never be exactly equal to its null value.

And apparently the LNC has accumulated a very large quantity of data. So could even this extreme p-value be illusory?

Regards,

Tony

----

Professor A O'Hagan Email: a.ohagan@sheffield.ac.uk

Department of Probability and Statistics

University of Sheffield Phone: +44 114 222 3773

Hicks Building

Sheffield S3 7RH, UK Fax: +44 114 222 3759

----------- http://www.tonyohagan.co.uk/ ------------

ISBA now maintains the bayes-news and Valencia email lists; you are receiving this message as you have opted-in to receive email from bayes-news, the Valencia list or the ISBA forums. To opt out or change your email subscriptions please login at http://bayesian.org/user. Go to the My Account menu and select Forum Email Integration. Check the forums to which you wish to subscribe, then save your settings. You may view past and present content at any time by visiting http://bayesian.org/forum