Bayesian Bloggers

The winds of Winter [Bayesian prediction]

Xian's Og - Mon, 2014-10-06 18:14

A surprising entry on arXiv this morning: Richard Vale (from Christchurch, NZ) has posted a paper about the characters appearing in the yet hypothetical next volume of George R.R. Martin’s Song of ice and fire series, The winds of Winter [not even put for pre-sale on amazon!]. Using the previous five books in the series and the frequency of occurrence of characters’ point of view [each chapter being told as from the point of view of one single character], Vale proceeds to model the number of occurrences in a given book by a truncated Poisson model,

in order to account for [most] characters dying at some point in the series. All parameters are endowed with prior distributions, including the terrible “large” hyperpriors familiar to BUGS users… Despite the code being written in R by the author. The modelling does not use anything but the frequencies of the previous books, so knowledge that characters like Eddard Stark had died is not exploited. (Nonetheless, the prediction gives zero chapter to this character in the coming volumes.) Interestingly, a character who seemingly died at the end of the last book is still given a 60% probability of having at least one chapter in  The winds of Winter [no spoiler here, but many in the paper itself!]. As pointed out by the author, the model as such does not allow for prediction of new-character chapters, which remains likely given Martin’s storytelling style! Vale still predicts 11 new-character chapters, which seems high if considering the series should be over in two more books [and an unpredictable number of years!].

As an aside, this paper makes use of the truncnorm R package, which I did not know and which is based on John Geweke’s accept-reject algorithm for truncated normals that I (independently) proposed a few years later.

Filed under: Books, Kids, R, Statistics, University life Tagged: A Song of Ice and Fire, arXiv, Bayesian predictive, Game of Thrones, George Martin, heroic fantasy, John Geweke, R, The Winds of Winter, truncated normal, truncnorm
Categories: Bayesian Bloggers

hypothesis testing for MCMC

Xian's Og - Mon, 2014-10-06 08:18

A recent arXival by Benjamin Gyori and Daniel Paulin considers sequential testing based on MCMC simulation. The test is about an expectation under the target and stationary distribution of the Markov chain (i.e., the posterior in a Bayesian setting). Hence testing whether or not the posterior expectation is below a certain bound is not directly relevant from a Bayesian perspective. One would test instead whether or not the parameter itself is below the bound… The paper is then more a study of sequential tests when the data is a Markov chain than in any clear connection with MCMC topics. Despite the paper including an example of a Metropolis-Hastings scheme for approximating the posterior on the parameters of an ODE. I am a bit puzzled by the purpose of the test, as I was rather expecting tests connected with the convergence of the Markov chain or of the empirical mean. (But, given the current hour, I may also have missed a crucial point!)

Filed under: Books, Statistics, University life Tagged: Markov chain Monte Carlo algorithm, Markov chains, MCMC, ODEs, sequential testing
Categories: Bayesian Bloggers

Monte Carlo simulation and resampling methods for social science [book review]

Xian's Og - Sun, 2014-10-05 18:14

Monte Carlo simulation and resampling methods for social science is a short paperback written by Thomas Carsey and Jeffrey Harden on the use of Monte Carlo simulation to evaluate the adequacy of a model and the impact of assumptions behind this model. I picked it in the library the other day and browse through the chapters during one of my métro rides. Definitely not an in-depth reading, so be warned!

Overall, I think the book is doing a good job of advocating the use of simulation to evaluate the pros and cons of a given model (rephrased as data generating process) when faced with data. And doing it in R. After some rudiments in probability theory and in R programming, it briefly explains the use of resident random generators if not of how to handle new distributions and then spend a large part of the book on simulation around generalised and regular linear models. For instance, in the linear model, the authors test the impact of heterocedasticity, multicollinearity, measurement error, omitted variable(s), serial correlation, clustered data, and heavy-tailed errors. While this is a perfect way of exploring those semi-hidden hypotheses behind the linear model, I wonder at the impact on students of this exploration. On the one hand, they will perceive the importance of those assumptions and hopefully remember them. On the other hand, and this is a very recurrent criticism of mine, this implies a lot of maturity from the students, i.e., they have to distinguish the data, the model [maybe] behind the data, the finite if large number of hypotheses one can test, and the interpretation of the outcome of a simulation test… Given that they were introduced to basic probability just a few chapters before, this expectation [from the students] may prove unrealistic. (And a similar criticism applies to the following chapters, from GLM to jackknife and bootstrap.)

At the end of the book, the authors ask the question as to how could a reader use the information in this book towards one’s work. Drafting a generic protocol for this reader, who is supposed to consider “alterations to the data generating process” (p.272) and to “identify a possible problem or assumption violation” (p.271). Thus requiring a readership “who has some training in quantitative methods” (p.1). And then some more. But I definitely sympathise with the goal of confronting models and theory with the harsh reality of simulation output!

Filed under: Books, Kids, R, Statistics, University life Tagged: bootstrap, data generating process, jacknife, Monte Carlo methods, Monte Carlo simulation and resampling methods for social science, resampling, simulation
Categories: Bayesian Bloggers

misty dawn

Xian's Og - Sun, 2014-10-05 08:18
Categories: Bayesian Bloggers

Argentan half-marathon [1 25' 02" - 29/503 - V2: 3/104 - 19°C]

Xian's Og - Sat, 2014-10-04 18:14

A rather comparable race in Argentan with last year, with exactly the same time, a similar weather (a bit too hot and too windy), and a lack of pack for half of the race that again saw me running by myself the whole second half. This was my iXXth Argentan half-marathon. I started a bit more slowly than last year, wary of the headwind, then accelerated in the forest and felt really well till the 17th kilometer, where I slowed down a bit, with a group of three runners passing me on the 18th and alas one of them V2. I tried to keep up till the 20th, but they ended up 20 seconds faster. This year I had left my traditional Insee Paris Club red tank top, to reduce the chances to be spotted but my opponent from last year was apparently not there. The number of runners seems to be going down steadily from one year to the next, maybe more so this year due to another race on Pont de Normandie tomorrow. This helped with keeping my chances of reaching a podium higher than expected, as I was not at all thinking I could make it again this year… [Thanks again and again to the photographs of Normandiecourseapied for their free pictures!]

Filed under: pictures, Running Tagged: Argentan, half-marathon, Normandy, race, veteran (V2)
Categories: Bayesian Bloggers

Rogue Male [book review]

Xian's Og - Fri, 2014-10-03 18:14

When I was about to leave a library in Birmingham, I spotted a “buy one get one half-price” book on a pile next to the cashier. Despite a rather weird title, Geoffrey Household’s Rogue Male looked classic enough to rank with Graham Green’s Confidential Agent or Erskine Childers’ Riddle of the Sands or yet John Buchan’s 39 Steps… Not mentioning the early Eric Ambler novels. I mean, a classic British thriller with political ramifications and a central character exposed with shortcomings and doubts.  After reading the book last week, I am glad I impulsively bought it. Rogue Male is not a Greene’s novel and this for several reason: (a) it is much more nationalistic, to the point of refusing to contact English authorities for fear of exposing some official backup of the attempted assassination, while Greene seemed to lean more to the Left, (b) it is both less and more psychological, in that it (i) superbly describes the process of getting rogue, i.e. of being hunted and of cutting or trying to cut [some] human feelings to rely on animal instincts for survival but (ii) leaves the overall motivation for Hitler’s attempted assassination and for the hunt by Nazi secret agents mostly unspecified  (c) it involves a very limited number of characters, all of them men, (d) it leaves so much of the action at the periphery that this appears as a weakness of the book… Still, there are some features also found in Greene’s Confidential Agent like the character failing in his attempt and being nearly captured or killed in the ensuing hunt, or the inner doubts about the (un)ethical nature of the fight… (Actually, both Greene and Household worked for the British secret services.) The overall story behind Rogue Male is a wee bit shallow and often too allusive to make sense but the underground part with the final psychological battle is superb. Truly a classic!

Filed under: Books Tagged: book review, Geoffrey Household, Graham Greene, Rogue Male, secret services, thriller

Categories: Bayesian Bloggers

echo vulnerable

Xian's Og - Fri, 2014-10-03 08:18

Even though most people are now aware of the Shellshock security problem on the bash shell, here is a test to check whether your Unix system is at risk:

env x='() { :;}; echo vulnerable' bash -c 'echo hello'

if the prompt returns vulnerable, it means the system is vulnerable and needs to be upgraded with the proper security patch… For instance running

sudo apt-get update && sudo apt-get install --only-upgrade bash

for Debian/Ubuntu versions. Check Apple support page for Apple OS.

Filed under: Linux Tagged: Linux, Shellshock, ubuntu, unix
Categories: Bayesian Bloggers

Kirchen in Wien

Xian's Og - Thu, 2014-10-02 18:14
Categories: Bayesian Bloggers

plenty of new arXivals!

Xian's Og - Thu, 2014-10-02 07:28

Here are some entries I spotted in the past days as of potential interest, for which I will have not enough time to comment:

  • arXiv:1410.0163: Instrumental Variables: An Econometrician’s Perspective by Guido Imbens
  • arXiv:1410.0123: Deep Tempering by Guillaume Desjardins, Heng Luo, Aaron Courville, Yoshua Bengio
  • arXiv:1410.0255: Variance reduction for irreversible Langevin samplers and diffusion on graphs by Luc Rey-Bellet, Konstantinos Spiliopoulos
  • arXiv:1409.8502: Combining Particle MCMC with Rao-Blackwellized Monte Carlo Data Association for Parameter Estimation in Multiple Target Tracking by Juho Kokkala, Simo Särkkä
  • arXiv:1409.8185: Adaptive Low-Complexity Sequential Inference for Dirichlet Process Mixture Models by Theodoros Tsiligkaridis, Keith W. Forsythe
  • arXiv:1409.7986: Hypothesis testing for Markov chain Monte Carlo by Benjamin M. Gyori, Daniel Paulin
  • arXiv:1409.7672: Order-invariant prior specification in Bayesian factor analysis by Dennis Leung, Mathias Drton
  • arXiv:1409.7458: Beyond Maximum Likelihood: from Theory to Practice by Jiantao Jiao, Kartik Venkat, Yanjun Han, Tsachy Weissman
  • arXiv:1409.7419: Identifying the number of clusters in discrete mixture models by Cláudia Silvestre, Margarida G. M. S. Cardoso, Mário A. T. Figueiredo
  • arXiv:1409.7287: Identification of jump Markov linear models using particle filters by Andreas Svensson, Thomas B. Schön, Fredrik Lindsten
  • arXiv:1409.7074: Variational Pseudolikelihood for Regularized Ising Inference by Charles K. Fisher

Filed under: Statistics, University life Tagged: arXiv, Bayesian Analysis, Ising, Monte Carlo methods, simulation, Statistics
Categories: Bayesian Bloggers

Approximate Bayesian Computation in state space models

Xian's Og - Wed, 2014-10-01 18:14

While it took quite a while (!), with several visits by three of us to our respective antipodes, incl. my exciting trip to Melbourne and Monash University two years ago, our paper on ABC for state space models was arXived yesterday! Thanks to my coauthors, Gael Martin, Brendan McCabe, and  Worapree Maneesoonthorn,  I am very glad of this outcome and of the new perspective on ABC it produces.  For one thing, it concentrates on the selection of summary statistics from a more econometrics than usual point of view, defining asymptotic sufficiency in this context and demonstrated that both asymptotic sufficiency and Bayes consistency can be achieved when using maximum likelihood estimators of the parameters of an auxiliary model as summary statistics. In addition, the proximity to (asymptotic) sufficiency yielded by the MLE is replicated by the score vector. Using the score instead of the MLE as a summary statistics allows for huge gains in terms of speed. The method is then applied to a continuous time state space model, using as auxiliary model an augmented unscented Kalman filter. We also found in the various state space models tested therein that the ABC approach based on the marginal [likelihood] score was performing quite well, including wrt Fearnhead’s and Prangle’s (2012) approach… I like the idea of using such a generic object as the unscented Kalman filter for state space models, even when it is not a particularly accurate representation of the true model. Another appealing feature of the paper is in the connections made with indirect inference.

Filed under: Statistics, Travel, University life Tagged: ABC, indirect inference, Kalman filter, marginal likelihood, Melbourne, Monash University, score function, state space model
Categories: Bayesian Bloggers

ABC model choice via random forests [expanded]

Xian's Og - Tue, 2014-09-30 18:14

Today, we arXived a second version of our paper on ABC model choice with random forests. Or maybe [A]BC model choice with random forests. Since the random forest is built on a simulation from the prior predictive and no further approximation is used in the process. Except for the computation of the posterior [predictive] error rate. The update wrt the earlier version is that we ran massive simulations throughout the summer, on existing and new datasets. In particular, we have included a Human dataset extracted from the 1000 Genomes Project. Made of 51,250 SNP loci. While this dataset is not used to test new evolution scenarios, we compared six out-of-Africa scenarios, with a possible admixture for Americans of African ancestry. The scenario selected by a random forest procedure posits a single out-of-Africa colonization event with a secondary split into a European and an East Asian population lineages, and a recent genetic admixture between African and European lineages, for Americans of African origin. The procedure reported a high level of confidence since the estimated posterior error rate is equal to zero. The SNP loci were carefully selected using the following criteria: (i) all individuals have a genotype characterized by a quality score (GQ)>10, (ii) polymorphism is present in at least one of the individuals in order to fit the SNP simulation algorithm of Hudson (2002) used in DIYABC V2 (Cornuet et al., 2014), (iii) the minimum distance between two consecutive SNPs is 1 kb in order to minimize linkage disequilibrium between SNP, and (iv) SNP loci showing significant deviation from Hardy-Weinberg equilibrium at a 1% threshold in at least one of the four populations have been removed.

In terms of random forests, we optimised the size of the bootstrap subsamples for all of our datasets. While this optimisation requires extra computing time, it is negligible when compared with the enormous time taken by a logistic regression, which is [yet] the standard ABC model choice approach. Now the data has been gathered, it is only a matter of days before we can send the paper to a journal

Filed under: Statistics, University life Tagged: 1000 Genomes Project, ABC, ABC model choice, admixture, bootstrap, DIYABC, Human evolution, logistic regression, out-of -Africa scenario, posterior predictive, prior predictive, random forests
Categories: Bayesian Bloggers

The chocolate factory gone up in smoke

Xian's Og - Tue, 2014-09-30 10:52

There was a major fire near my house yesterday with many fire-engines rushing by and a wet smoke smell lingering by the whole night. As I found out during my early morning run, the nearby chocolate factory had completely burned. Actually, sixteen  hours after the beginning of the fire, the building was still smouldering, with a dozen fire-engines yet on site and huge hoses running on adjacent streets. A fireman told me the fire had started from an electric spark and that the entire  reserves had been destroyed. This is quite sad, as hitting a local business and a great chocolate maker, Patrick Roger. I do not know whether or not the company will survive this disaster, but if you happen to come by one of the shops in Paris or Brussels, drop in and buy some chocolates! For the taste of it and as a support.

Filed under: Kids, Running Tagged: chocolate, fire, Patrick Roger, Sceaux
Categories: Bayesian Bloggers

The Unimaginable Mathematics of Borges’ Library of Babel [book review]

Xian's Og - Mon, 2014-09-29 18:14

This is a book I carried away from JSM in Boston as the Oxford University Press representative kindly provided my with a copy at the end of the meeting. After I asked for it, as I was quite excited to see a book linking Jorge Luis Borges’ great Library of Babel short story with mathematical concepts. Even though many other short stories by Borges have a mathematical flavour and are bound to fascinate mathematicians, the Library of Babel is particularly prone to mathemati-sation as it deals with the notions of infinite, periodicity, permutation, randomness… As it happens, William Goldbloom Bloch [a patronym that would surely have inspired Borges!], professor of mathematics at Wheaton College, Mass., published the unimaginable mathematics of Borges’ Library of Babel in 2008, so this is not a recent publication. But I had managed to miss through the several conferences where I stopped at OUP exhibit booth. (Interestingly William Bloch has also published a mathematical paper on Neil Stephenson’s Cryptonomicon.)

Now, what is unimaginable in the maths behind Borges’ great Library of Babel??? The obvious line of entry to the mathematical aspects of the book is combinatorics: how many different books are there in total? [Ans. 10¹⁸³⁴⁰⁹⁷...] how many hexagons are needed to shelf that many books? [Ans. 10⁶⁸¹⁵³¹...] how long would it take to visit all those hexagons? how many librarians are needed for a Library containing all volumes once and only once? how many different libraries are there [Ans. 1010⁶...] Then the book embarks upon some cohomology, Cavalieri’s infinitesimals (mentioned by Borges in a footnote), Zeno’s paradox, topology (with Klein’s bottle), graph theory (and the important question as to whether or not each hexagon has one or two stairs), information theory, Turing’s machine. The concluding chapters are comments about other mathematical analysis of Borges’ Grand Œuvre and a discussion on how much maths Borges knew.

So a nice escapade through some mathematical landscapes with more or less connection with the original masterpiece. I am not convinced it brings any further dimension or insight about it, or even that one should try to dissect it that way, because it kills the poetry in the story, especially the play around the notion(s) of infinite. The fact that the short story is incomplete [and short on details] makes its beauty: if one starts wondering at the possibility of the Library or at the daily life of the librarians [like, what do they eat? why are they there? where are the readers? what happens when they die? &tc.] the intrusion of realism closes the enchantment! Nonetheless, the unimaginable mathematics of Borges’ Library of Babel provides a pleasant entry into some mathematical concepts and as such may initiate a layperson not too shy of maths formulas to the beauty of mathematics.

Filed under: Books, Statistics, Travel, University life Tagged: book review, Boston, cohomology, combinatorics, infinity, information theory, Jorge Luis Borges, JSM 2014, Library of Babel, Oxford University Press, Turing's machine
Categories: Bayesian Bloggers

future of computational statistics

Xian's Og - Sun, 2014-09-28 18:14

I am currently preparing a survey paper on the present state of computational statistics, reflecting on the massive evolution of the field since my early Monte Carlo simulations on an Apple //e, which would take a few days to return a curve of approximate expected squared error losses… It seems to me that MCMC is attracting more attention nowadays than in the past decade, both because of methodological advances linked with better theoretical tools, as for instance in the handling of stochastic processes, and because of new forays in accelerated computing via parallel and cloud computing, The breadth and quality of talks at MCMski IV is testimony to this. A second trend that is not unrelated to the first one is the development of new and the rehabilitation of older techniques to handle complex models by approximations, witness ABC, Expectation-Propagation, variational Bayes, &tc. With a corollary being an healthy questioning of the models themselves. As illustrated for instance in Chris Holmes’ talk last week. While those simplifications are inevitable when faced with hardly imaginable levels of complexity, I still remain confident about the “inevitability” of turning statistics into an “optimize+penalize” tunnel vision…  A third characteristic is the emergence of new languages and meta-languages intended to handle complexity both of problems and of solutions towards a wider audience of users. STAN obviously comes to mind. And JAGS. But it may be that another scale of language is now required…

If you have any suggestion of novel directions in computational statistics or instead of dead ends, I would be most interested in hearing them! So please do comment or send emails to my gmail address bayesianstatistics…

Filed under: Books, pictures, R, Statistics, University life Tagged: ABC, Apple II, approximation, BUGS, computational statistics, expectation-propagation, JAGS, MCMC, MCMSki IV, Monte Carlo, optimisation, STAN, statistical computing, sunset, variational Bayes methods
Categories: Bayesian Bloggers


Xian's Og - Sat, 2014-09-27 18:14

“For the first nine years of its existence, aside from being appointed the flagship, there was nothing particularly special about it, from a statistical point of view.”

A book I grabbed at the last minute in a bookstore, downtown Birmingham. Maybe I should have waited this extra minute… Or picked the other Scalzi’s on the shelf, Lock In that just came out! (I already ordered that one for my incomiing lecture in Gainesville. Along with the not final volume of Patrick Rothfuss’ masterpiece, The Slow Regard of Silent Things, which will just be out by then! It is only a side story within the same universe, as pointed out by Dan…)

“What you’re trying to do is impose causality on random events, just like everyone else here has been doing.”

What amazes most me is that Scalzi’s redshirts got the 2013 Hugo Award. I mean, The Hugo Award?! While I definitely liked the Old Man Wars saga, this novel is more like a light writing experiment and a byproduct of writing a TV series. Enjoyable at a higher conceptual level, but not as a story. Although this is somewhat of a spoiler (!), the title refers to the characters wearing red shirts in Star Trek, who have a statistically significant tendency to die on the next mission. [Not that I knew this when I bought the book! Maybe it would have warned me against the book.] And redshirts is about those characters reflecting about how unlikely their fate is (or rather the fate of the characters before them) and rebelling against the series writer. Ensues games with the paradoxes of space travel and doubles. Then games within games. The book is well-written and, once again, enjoyable at some level, with alternative writing styles used in different parts (or coda) of the novel. It still remains a purely intellectual perspective, with no psychological involvement towards those characters. I just cannot relate to the story. Maybe because of the pastiche aspect or of the mostly comic turn. redshirts certainly feels very different from those Philip K. Dick stories (e.g., Ubik) where virtual realities abounded without a definitive conclusion on which was which.

Filed under: Books, pictures, Travel Tagged: Birmingham, England, Hugo Awards, John Scalzi, Patrick Rothfuss, redshirts, Star Trek
Categories: Bayesian Bloggers

métro static

Xian's Og - Sat, 2014-09-27 08:18

“Mon premier marathon je le fais en courant.” [I will do my first marathon running.]

Filed under: pictures, Running Tagged: Badwater Ultramarathon, Florida, Marathon FL, métro, métro static, Paris, sea, sunset
Categories: Bayesian Bloggers

all models are wrong

Xian's Og - Fri, 2014-09-26 18:14

“Using ABC to evaluate competing models has various hazards and comes with recommended precautions (Robert et al. 2011), and unsurprisingly, many if not most researchers have a healthy scepticism as these tools continue to mature.”

Michael Hickerson just published an open-access letter with the above title in Molecular Ecology. (As in several earlier papers, incl. the (in)famous ones by Templeton, Hickerson confuses running an ABC algorithm with conducting Bayesian model comparison, but this is not the main point of this post.)

“Rather than using ABC with weighted model averaging to obtain the three corresponding posterior model probabilities while allowing for the handful of model parameters (θ, τ, γ, Μ) to be estimated under each model conditioned on each model’s posterior probability, these three models are sliced up into 143 ‘submodels’ according to various parameter ranges.”

The letter is in fact a supporting argument for the earlier paper of Pelletier and Carstens (2014, Molecular Ecology) which conducted the above splitting experiment. I could not read this paper so cannot judge of the relevance of splitting this way the parameter range. From what I understand it amounts to using mutually exclusive priors by using different supports.

“Specifically, they demonstrate that as greater numbers of the 143 sub-models are evaluated, the inference from their ABC model choice procedure becomes increasingly.”

An interestingly cut sentence. Increasingly unreliable? mediocre? weak?

“…with greater numbers of models being compared, the most probable models are assigned diminishing levels of posterior probability. This is an expected result…”

True, if the number of models under consideration increases, under a uniform prior over model indices, the posterior probability of a given model mechanically decreases. But the pairwise Bayes factors should not be impacted by the number of models under comparison and the letter by Hickerson states that Pelletier and Carstens found the opposite:

“…pairwise Bayes factor[s] will always be more conservative except in cases when the posterior probabilities are equal for all models that are less probable than the most probable model.”

Which means that the “Bayes factor” in this study is computed as the ratio of a marginal likelihood and of a compound (or super-marginal) likelihood, averaged over all models and hence incorporating the prior probabilities of the model indices as well. I had never encountered such a proposal before. Contrary to the letter’s claim:

“…using the Bayes factor, incorporating all models is perhaps more consistent with the Bayesian approach of incorporating all uncertainty associated with the ABC model choice procedure.”

Besides the needless inclusion of ABC in this sentence, a somewhat confusing sentence, as Bayes factors are not, stricto sensu, Bayesian procedures since they remove the prior probabilities from the picture.

“Although the outcome of model comparison with ABC or other similar likelihood-based methods will always be dependent on the composition of the model set, and parameter estimates will only be as good as the models that are used, model-based inference provides a number of benefits.”

All models are wrong but the very fact that they are models allows for producing pseudo-data from those models and for checking if the pseudo-data is similar enough to the observed data. In components that matters the most for the experimenter. Hence a loss function of sorts…

Filed under: Statistics, University life Tagged: ABC, Bayes factor, Bayesian model choice, George Box, model posterior probabilities, Molecular Ecology, phylogenetic model, phylogeography
Categories: Bayesian Bloggers

two, three, five, …, a million standard deviations!

Xian's Og - Thu, 2014-09-25 18:14

I first spotted Peter Coles’ great post title “Frequentism: the art of probably answering the wrong question” (a very sensible piece by the way!, and mentioning a physicist’s view on the Jeffreys-Lindley paradox I had intended to comment) and from there the following site jumping occured:

“I confess that in my early in my career as a physicist I was rather cynical about sophisticated statistical tools, being of the opinion that “if any of this makes a difference, just get more data”. That is, if you do enough experiments, the confidence level will be so high that the exact statistical treatment you use to evaluate it is irrelevant.” John Butterworth, Sept. 15, 2014

After Val Johnson‘s suggestion to move the significant level from .05 down to .005, hence roughly from 2σ up to 3σ, John Butterworth, a physicist whose book Smashing Physics just came out, discusses in The Guardian the practice of using 5σ in Physics. It is actually induced by Louis Lyons’ arXival of a recent talk with the following points (discussed below):

  1. Should we insist on the 5 sigma criterion for discovery claims?
  2. The probability of A, given B, is not the same as the probability of B, given A.
  3. The meaning of p-values.
  4. What is Wilks Theorem and when does it not apply?
  5. How should we deal with the `Look Elsewhere Effect’?
  6. Dealing with systematics such as background parametrisation.
  7. Coverage: What is it and does my method have the correct coverage?
  8. The use of p0 versus p1 plots.

Butterworth’s conclusion is worth reproducing:

“…there’s a need to be clear-eyed about the limitations and advantages of the statistical treatment, wonder what is the “elsewhere” you are looking at, and accept that your level of certainty may never feasibly be 5σ. In fact, if the claims being made aren’t extraordinary, a one-in-2million chance of a mistake may indeed be overkill, as well being unobtainable. And you have to factor in the consequences of acting, or failing to act, based on the best evidence available – evidence that should include a good statistical treatment of the data.” John Butterworth, Sept. 15, 2014

esp. the part about the “consequences of acting”, which I interpret as incorporating a loss function in the picture.

Louis’s paper-ised talk 1. [somewhat] argues in favour of the 5σ because 2σ and 3σ are not necessarily significant on larger datasets. I figure the same could be said of 5σ, no?! He also mentions (a) “systematics”, which I do not understand. Even though this is not the first time I encounter the notion in Physics. And (b) “subconscious Bayes factors”, which means that the likelihood ratio [considered here as a transform of the p-value] is moderated by the ratio of the prior probabilities, even when people do not follow a Bayesian procedure. But this does not explain why a fixed deviation from the mean should be adopted. 2. and 3. The following two points are about the common confusion in the use of the p-value, found in most statistics textbooks. Even though the defence of the p-value against the remark that it is wrong half the time (as in Val’s PNAS paper) misses the point. 4. Wilk’s theorem is a warning that the χ² approximation only operates under some assumptions. 5. Looking elsewhere is the translation of multiple testing or cherry-picking. 6. Systematics is explained here as a form of model misspecification. One suggestion is to use a Bayesian modelling of this misspecification, another non-parametrics (why not both together?!). 7. Coverage is somewhat disjunct from the other points as it explains the [frequentist] meaning of the coverage of a confidence interval. Which hence does not apply to the actual data. 8. p0 versus p1 plots is a sketchy part referring to a recent proposal by the author. So in the end a rather anticlimactic coverage of standard remarks, surprisingly giving birth to a sequence of posts (incl. this one!)…

Filed under: Books, Statistics, University life Tagged: Bayesian modeling, five sigma, John Butterworth, likelihood ratio, Louis Lyons, p-values, PNAS, The Guardian, Valen Johnson
Categories: Bayesian Bloggers

interesting mis-quote

Xian's Og - Wed, 2014-09-24 18:14

At a recent conference on Big Data, one speaker mentioned this quote from Peter Norvig, the director of research at Google:

“All models are wrong, and increasingly you can succeed without them.”

quote that I found rather shocking, esp. when considering the amount of modelling behind Google tools. And coming from someone citing Kernel Methods for Pattern Analysis by Shawe-Taylor and Christianini as one of his favourite books and Bayesian Data Analysis as another one… Or displaying Bayes [or his alleged portrait] and Turing in his book cover. So I went searching on the Web for more information about this surprising quote. And found the explanation, as given by Peter Norvig himself:

“To set the record straight: That’s a silly statement, I didn’t say it, and I disagree with it.”

Which means that weird quotes have a high probability of being misquotes. And used by others to (obviously) support their own agenda. In the current case, Chris Anderson and his End of Theory paradigm. Briefly and mildly discussed by Andrew a few years ago.

Filed under: Books, pictures, Statistics, Travel, University life Tagged: Alan Turing, all models are wrong, artificial intelligence, George Box, misquote, Peter Norvig, statistical modelling, The End of Theory, Thomas Bayes
Categories: Bayesian Bloggers