Xian's Og

Syndicate content Xi'an's Og
an attempt at bloggin, nothing more...
Updated: 10 hours 43 min ago

understanding the Hastings algorithm

Mon, 2014-08-25 18:14

David Minh and Paul Minh [who wrote a 2001 Applied Probability Models] have recently arXived a paper on “understanding the Hastings algorithm”. They revert to the form of the acceptance probability suggested by Hastings (1970):

where s(x,y) is a symmetric function keeping the above between 0 and 1, and q is the proposal. This obviously includes the standard Metropolis-Hastings form of the ratio, as well as Barker’s (1965):

which is known to be less efficient by accepting less often (see, e.g., Antonietta Mira’s PhD thesis). The authors also consider the alternative

which I had not seen earlier. It is a rather intriguing quantity in that it can be interpreted as (a) a simulation of y from the cutoff target corrected by reweighing the previous x into a simulation from q(x|y); (b) a sequence of two acceptance-rejection steps, each concerned with a correspondence between target and proposal for x or y. There is an obvious caveat in this representation when the target is unnormalised since the ratio may then be arbitrarily small… Yet another alternative could be proposed in this framework, namely the delayed acceptance probability of our paper with Marco and Clara, one special case being


is an arbitrary decomposition of the target. An interesting remark in the paper is that any Hastings representation can alternatively be written as

where k(x,y) is a (positive) symmetric function. Hence every single Metropolis-Hastings is also a delayed acceptance in the sense that it can be interpreted as a two-stage decision.

The second part of the paper considers an extension of the accept-reject algorithm where a value y proposed from a density q(y) is accepted with probability

and else the current x is repeated, where M is an arbitrary constant (incl. of course the case where it is a proper constant for the original accept-reject algorithm). Curiouser and curiouser, as Alice would say! While I think I have read some similar proposal in the past, I am a wee intrigued at the appear of using only the proposed quantity y to decide about acceptance, since it does not provide the benefit of avoiding generations that are rejected. In this sense, it appears as the opposite of our vanilla Rao-Blackwellisation. (The paper however considers the symmetric version called the independent Markovian minorizing algorithm that only depends on the current x.) In the extension to proposals that depend on the current value x, the authors establish that this Markovian AR is in fine equivalent to the generic Hastings algorithm, hence providing an interpretation of the “mysterious” s(x,y) through a local maximising “constant” M(x,y). A possibly missing section in the paper is the comparison of the alternatives, albeit the authors mention Peskun’s (1973) result that exhibits the Metropolis-Hastings form as the optimum.

Filed under: Books, Statistics Tagged: accept-reject algorithm, acceptance probability, Barker's algorithm, delayed acceptance, Metropolis-Hastings algorithms, vanilla Rao-Blackwellisation
Categories: Bayesian Bloggers

NIPS workshops (Dec. 12-13, 2014, Montréal)

Sun, 2014-08-24 18:14

Following a proposal put forward by Ted Meeds, Max Welling,  Richard Wilkinson, Neil Lawrence and myself, our ABC in Montréal workshop has been accepted by the NIPS 2014 committee and will thus take place on either Friday, Dec. 11, or Saturday, Dec. 12, at the end of the main NIPS meeting (Dec. 8-10). (Despite the title, this workshop is not part of the ABC in … series I started five years ago. It will only last a single day with a few invited talks and no poster. And no free wine & cheese party.) On top of this workshop, our colleagues Vikash K Mansinghka, Daniel M Roy, Josh Tenenbaum, Thomas Dietterich, and Stuart J Russell have also been successful in their bid for the 3rd NIPS Workshop on Probabilistic Programming which will presumably be held on the opposite day to ours, as Vikash is speaking at our workshop, while I am speaking in this workshop. I am yet undecided as to whether or not to attend the main conference, given that I am already travelling a lot this semester and have to teach two courses, incl. a large undergraduate statistics inference course… Obviously, I will try to attend if our joint paper is accepted by the editorial board! Even though Marco will then be the speaker.

Filed under: Kids, Statistics, Travel, University life Tagged: ABC, ABC in Montréal, Canada, delayed acceptance, machine learning, Montréal, NIPS 2014, prefetching, Québec
Categories: Bayesian Bloggers

Drake Island beach (#2)

Sun, 2014-08-24 08:18
Categories: Bayesian Bloggers

the intelligent-life lottery

Sat, 2014-08-23 18:14

In a theme connected with one argument in Dawkins’ The God Delusion, The New York Time just published a piece on the 20th anniversary of the debate between Carl Sagan and Ernst Mayr about the likelihood of the apparition of intelligent life. While 20 years ago, there was very little evidence if any of the existence of Earth-like planets, the current estimate is about 40 billions… The argument against the high likelihood of other inhabited planets is that the appearance of life on Earth is an accumulation of unlikely events. This is where the paper goes off-road and into the ditch, in my opinion, as it makes the comparison of the emergence of intelligent (at the level of human) life to be “as likely as if a Powerball winner kept buying tickets and — round after round — hit a bigger jackpot each time”. The later having a very clearly defined probability of occurring. Since “the chance of winning the grand prize is about one in 175 million”. The paper does not tell where the assessment of this probability can be found for the emergence of human life and I very much doubt it can be justified. Given the myriad of different species found throughout the history of evolution on Earth, some of which evolved and many more which vanished, I indeed find it hard to believe that evolution towards higher intelligence is the result of a basically zero probability event. As to conceive that similar levels of intelligence do exist on other planets, it also seems more likely than not that life took on average the same span to appear and to evolve and thus that other inhabited planets are equally missing means to communicate across galaxies. Or that the signals they managed to send earlier than us have yet to reach us. Or Earth a long time after the last form of intelligent life will have vanished…

Filed under: Books, Kids Tagged: Carl Sagan, cosmology, Ernst Mayr, evolution, exoplanet, Powerball, The God Delusion, The New York Times
Categories: Bayesian Bloggers

Drake Island beach (#1)

Sat, 2014-08-23 08:18
Categories: Bayesian Bloggers

summer reads

Fri, 2014-08-22 18:14

I had planned my summer read long in advance to have an Amazon shipment sent to my friend Natesh out of my Amazon associate slush funds. While in Boston and Maine, I read Richard Dawkins’ The God delusion, the fourth Kelly McCullough’s Fallen Blade novel, Blade reforged, the second Ancient Blades novel, unrelated to the above, A thief in the night, by David Chandler, and also the second Tad Williams’ Bobby Dollar novel, Happy Hour in HellThe God delusion is commented on another post.

Blade reforged is not a major novel, unsurprisingly for a fourth entry, but pleasant nonetheless, especially when reading in the shade of a pavilion on Revere Beach! The characters are mostly the same as previously and it could be that the story has (hopefully) come to an end, with (spoilers!) the evil ruler replaced by the hero’s significant other and his mystical weapons returned to him. A few loose ends and a central sword fight with a more than surprising victory, but a good summer read. Checking on Kelly McCullough’s website, I notice that two more novels are in the making….

Tad Williams’ second novel Happy Hour in Hell is much less enjoyable as the author was unable to keep up with the pace and tone of the highly imaginative first novel, full of witty and hard-boiled exchanges. The first novel introduced the (after-)life of a guardian angel in California, Doloriel (a.k.a. Bobby Dollar), with enough levels of political intrigue between Heaven and Hell and Earth and plots, pursuits, assassination attempts, etc., to make it a page-turner. This second novel sends Doloriel on a suicide mission to Hell… and the reader to a Hell of sorts where the damnation is one of eternal boredom! What made the first novel so original, namely the juxtaposition of the purpose of a guardian with his every-day terrestrial life, is lost. All we have there is a fantastic creature (from Heaven) transposed in another fantastic environment (Hell) and trying to survive without a proper guide book. The representation of Hell is not particularly enticing (!), even with acknowledged copies from Dante’s Inferno and Hieronymus Bosch’s paintings. There is a very low tolerance level to my reading of damned souls being tortured, dismembered, eaten or resuscitated, even when it gets to the hero’s turn. Add to that a continuation of the first book’s search for a particular feather. And an amazing amount of space dedicated to the characters’ meals. This makes for a very boring book. Even for a rainy day on a Maine lake! The depiction of the levels and inhabitants of Hell reminded me of another endless book by Tad Williams, Shadowmarch, where some characters end up in a subterranean semi-industrial structure, with a horde of demon-like creatures and no fun [for the reader!]. Ironically, the funniest part of reading Happy Hour in Hell was to do it after Dawkins’ as some reflections of the angel about the roles of Heaven and Hell (and religion) could have fitted well into The God delusion! (Too bad my Maine rental had Monty Python’s Holy Grail instead of The Life of Brian, as it would have made a perfect trilogy!)

Most sadly, David Chandler’s A thief in the night had exactly the same shortcomings as another book  I had previously read and maybe reviewed, even though I cannot trace the review or even remember the title of the book (!), and somewhat those of Tad Williams’ Happy Hour in Hell as well, that is, once again a subterranean adventure in a deserted mythical mega-structure that ends up being not deserted at all and even less plausible. I really had to be stuck on a beach or in an airport lounge to finish it! The points noted about Den of Thieves apply even more forcibly here, that is, very charicaturesque characters and a weak and predictable plot. With the addition of the unbearable underground hidden world… I think I should have re-read my own review before ordering this book.

Filed under: Books, Travel Tagged: amazon associates, Ancient Blades, blade reforged, Boston, David Chandler, Kelly McCullough, Maine, Richard Dawkins, Tad Williams, The God Delusion, Wells
Categories: Bayesian Bloggers

a week at the lake (#6)

Fri, 2014-08-22 13:18

Filed under: pictures, Travel Tagged: lake, Maine, moon, moonlight, Newfield, vacations
Categories: Bayesian Bloggers

the god delusion [statistically speaking]

Thu, 2014-08-21 18:14

While in Bangalore, I spotted Richard Dawkins’ The God delusion in the [fantastic if chaotic] campus bookstore and bought the Indian edition for a nominal amount.  I read most of it during my week in Boston. And finished by the lake in Maine. While I agree with most of the points made in Dawkins’ book about the irrationality of religions, and of their overall negative impact on human societies, I found the first part rather boring in that I see little appeal in dissecting so minutely the [infinitely many] incoherences of religious myths and beliefs, as this will likely miss the intended target [i.e., literal believers]. Similarly, the chapter on evolution versus intelligent design made valuable points, albeit I had already seen them before. Nothing wrong with repeating those, in particular that evolution has little to do with chance, but again unlikely to convince the [fundamentalist] masses. Overall, the book mostly focus on the Judeo-Christian-Muslim branch of religions, which may reflect on the author’s own culture and upbringing but also misses the recent attempts of Buddhism to incorporate science into their picture.

“A universe in which we are alone except for other slowly evolved intelligences is a very different universe from one with an original guiding agent whose intelligent design is responsible for its very existence.” (p.85)

What is most interesting in the book (for me) is when Dawkins tries to set the God hypothesis as a scientific hypothesis and to apply scientific methods to validate or invalidate this hypothesis. Even though there is no p-value or quantitative answer at the end. Despite the highly frequent use of “statistical” and “statistically improbable” in the corresponding chapter. What’s even more fascinating is Dawkins’ take at Bayesian arguments! Either because it is associated with a reverent or because it relies on subjective prior assessments, Bayesian statistics does not fit as a proper approach. Funny enough, Dawkins himself relies on subjective prior probabilities when discussing the likelihood of find a planet such as Earth. Now, into the details [with the Devil1] in a rather haphazard order or lack thereof:

“A universe in which we are alone except for other slowly evolved intelligences is a very different universe from one with an original guiding agent whose intelligent design is responsible for its very existence.” (p.85)

I appreciated Dawkins’ debunking of the non-overlapping magisteria (NOMA) principle, namely that scientists should leave religion questions to philosophers and theologians. Restating that the “God Hypothesis” truly is a scientific question that must be addressed from a scientific perspective and hence by scientific means. Mentioning Francis Galton, “the first to analyse scientifically whether praying for the royal family is efficacious (…) [he] looked into it, and found no statistical difference.” (p.85)

“Arguments for God’s existence falls into two main categories, the a priori and the a posteriori.” (p.103)

Another interesting (if not particularly novel) remark is about the strong secularism of the “Founding Fathers” like Jefferson, Adams (John, not Douglas!) or Madison, who were most likely atheists. Which conflicts with the equally strong influence of religion in 21st-century America.

“Huxley, in his concentration upon the absolute impossibility of proving or disproving God, seems to have been ignoring the shading of probability.” (p.72)

While I do not feel the book is particularly strong from a philosophical viewpoint, I appreciated the section on agnosticism: the apparently reasonable sceptic attitude of refusing to commit for or against something that cannot be demonstrated does not stand once one starts piling creeds and beliefs like Russell’s teapot, against the absence of God. In Dawkins’ word, a 50 percent probability in favour of each hypothesis does not hold: “the odds in favour of the teapot are not equal to the odds against” (p.76). (Note that he does not fall into Templeton’s fallacy of reducing the probabilities of hypotheses by multiplying those hypotheses!) Connected funny remarks: as monotheism considers itself superior of polytheism (despite the myriads of characters in the Catholic pantheon), why not move one step further and subtract the remaining god? And the fact than an atheist is simply disbelieving in one god more (or less) than any monotheist.

“Bayes’ Theorem (…) is a mathematical engine for combining many estimated likelihoods and coming up with a final verdict, which bears its own quantitative estimate of likelihood. But of course that final estimate can only be as goof as the original numbers fed in. These are usually subjectively judged, with all the doubts that inevitably flow from that.” (p.133)

Dawkins thus spends five pages on some Bayesian arguments for God’s existence.  Surprisingly, he is missing the historical facts that Bayes and Pierce set the theorem into motion to answer Hume’s probabilistic arguments against the existence of miracles. And that one of Bayes’ few existing papers is about the existence of God. Dawkins does not aim at the whole Bayesian perspective when he starts from Stephen Uwin’s Probability of God. However, his [Dawkins'] criticism would equally apply to any Bayesian argument. He seems to see any subjective input or personal probability as a-scientific and akin to faith: “who cares about subjective judgements anyway?” (p.135) A pedantic remark is about Dawkins’ unusual meaning of likelihood, which he seems to understand as probability (of an hypothesis?) and even as posterior probability if I get the “estimated” right.

“The book is better seen as a `How To’ manual, a sort of Bayes’ Theorem for Dummies, using the existence of God as a semi-facetious case study (…) Uwin’s quixotic attempt to put a number on the probability is quite agreeably funny.” (p.132)

The above quote and the one introducing Bayes’ Theorem are surprising in that they reveal Dawkins’ perception of probability as a-numeric (and hence a-Bayesian). However, in a later chapter, he does not mind putting a (prior?) probability on “the odds of life originating spontaneously on a planet [as] a billion to one against” (p.165). And to argue that “any probability statement is made in the context of a certain level of ignorance” (p.166), a point followed by a description of what ends up being Bayesian updating!

“The trouble is that (to repeat) the six weightings are not measured quantities but simply Stephen Unwin’s own personal judgements, turned into numbers for the exercise.” (p.133)

Obviously, I cannot say much about Uwin’s arguments to reach 0.67 as the posterior probability of existence of God for a prior (agnostic) probability of 0.5 and six contributing facts. It seems rather weird to mix facts (e.g., Hitler existed) and beliefs (there might be miracles) selected out of the myriad possible statements about God’s existence, and use them naïvely into Bayes’ Theorem. I thus share Dawkins’ scepticism at this level. If not at the overall issue of putting personal and subjective assessments on non-reproducible events.

“When challenged by a zealous Popperian to say how evolution could ever be falsified, J.B.S. Haldane famously growled: `Fossil rabbits in the Precambrian’.” (p.154)

I found the part about cosmology and the anthropic principle less convincing than the seamless analysis of Darwin and the principle of biological evolution. While the very existence of our Universe is stupendous, the fact that it does exist does not imply in any way the deed of an higher being or the simultaneous existence of parallel universes. Or, even less so, a Darwinian sort of natural selection among multiverses (p.175). These are interesting intellectual constructs, but there is no evidence about them. At least within the book. To argue that “people who think [there is no difference with the God hypothesis] have not had their conscience raised by natural selection” (p.175) is not a scientific argument. Nor is the following line that the relative simplicity of the multiverse implies a higher “statistical improbability”. I also remain quite sceptical about the lyrical passages on the meme pool, as I am not convinced there is anything provable [or falsifiable!] in the concept. (Meme is a “unit of cultural inheritance”, p.222.)

Overall, this is a clearly needed book, which sweeps away common arguments about the societal benefits of religion(s). As written above, I concur with most of Dawkins’ points and while I think his arguments cannot reach fundamentalists who in the first place would not approach the book with anything but a flame-thrower, it may enlighten agnostics and doubters who hover at the fringe of [a] religion and who need a gentle push to jump to the other side of the border.

Filed under: Books, Kids, pictures, Statistics, Travel, University life Tagged: atheism, Bangalore, India, Philosophy of religions, Philosophy of Science, Richard Dawkins
Categories: Bayesian Bloggers

a week at the lake (#5)

Thu, 2014-08-21 13:18

Filed under: Kids, Mountains, pictures, Running, Travel
Categories: Bayesian Bloggers

To Susie [by Kerrie Mengersen]

Wed, 2014-08-20 18:14

[Here is a poem written by my friend Kerrie for the last ISBA cabaret in Cancun, to Susie who could not make it to a Valencia meeting for the first time... Along with a picture of Susie, Alicia and Jennifer taking part in another ISBA cabaret in Knossos, Crete, in 2000.]

This is a parody of a classic Australian bush poem, ‘The Man from Snowy River’, that talks of an amazing horseman in the rugged mountain bush of Australia, who out-performed the ‘cracks’ and became a legend. That’s how I think of Susie, so this very bad poem comes with a big thanks for being such an inspiration, a great colleague and a good friend.

There was movement in the stats world as the emails caught alight
For the cult from Reverend Bayes had got away
And had joined the ‘ISBA’ forces, and were calling for a fight
So all the cracks had gathered to the fray.

All the noted statisticians from the countries near and far
Had flown into Cancun overnight
For the Bayesians love their meetings where the sandy beaches are
And the Fishers snuffed the battle with delight.

There were Jim and Ed and Robert, who were ‘fathers of the Bayes’
They were known as the whiskey drinking crowd
But they’d invented all the theory in those Valencia days
Yes, they were smart, but oh boy were they loud!

And Jose M Bernardo came down to lend a hand
A finer Bayesian never wrote a prior
And Mike West, Duke of Bayesians, also joined the band
And brought down all the graduates he could hire

Sonia and Maria strapped their laptops to the cause
And Anto, Chris and Peter ran – in thongs!
Sirs Adrian and David came with armour and a horse
While Brad and Gareth murdered battle songs

And one was there, a Spaniard, blonde and fierce and proud
With a passion for statistics and for fun
She’d been there with the founders of the nouveau Bayesian crowd
And kept those Fisher stats folk on the run

But Jim’s subjective prior made him doubt her power to fight
Mike Goldstein said, ‘That girl will never do,
In the heat of battle, deary, you just don’t have the might
This stoush will be too rough for such as you.’

 But Berger and Bernardo came to Susie’s side
We think we ought to let her in, they said
For we warrant she’ll be with us when the blood has fairly dried
For Susie is Valencia born and bred.

She did her Bayesian training in the classic Spanish way
Where the stats is twice as hard and twice as rough
And she knows nonparametrics, which is useful in a fray
She’s soft outside, but inside, man she’s tough!

She went. They found those Fisher stats folk sunning on the beach
And as they grabbed their laptops from the sand
Jim Berger muttered fiercely, ‘right, twist any head you reach
We cannot let those Fish get out of hand.’

Alicia, grab a Dirichlet and break them with a stick
Chris, it’s easy, just like ABC
And Sylvia, a mixture model ought to do the trick
But just you leave that Ronnie up to me.

Jose battled them with inference and curdled Neyman’s blood
And Ed told jokes that made them shake their head
And posteriors lined like beaches like sandbags for a flood
And Jim threw whiskey bottles as they fled.

And when the Bayesians and the Fishers were washed up on the sand
The fight was almost judged to be a tie
But it was Susie who kept going, who led the final charge
For she didn’t want objective Bayes to die

She sent the beach on fire as she galloped through the fray
Hurling P and F tests through the foam
‘til the Fishers raised surrender and called the fight a day
And shut their laptops down and sailed for home.

And now at ISBA meetings where the Bayesians spend their days
To laugh and learn and share a drink or two
A glass is always toasted: to Susie, Queen of Bayes
And the cheering echoes loudly round the crew.

She will be remembered for setting Bayesian stats on fire
For her contributions to the field are long
And her passion and her laughter will continue to inspire
The Bayesian from Valencia lives on!

Filed under: pictures, Statistics, University life Tagged: Cancún, Crete, ISBA, ISBA 2014, Kerrie Mengersen, Knossos, Susie Bayarri, València
Categories: Bayesian Bloggers

Mas de Martin Ecce Vino

Wed, 2014-08-20 14:20

Filed under: Wines
Categories: Bayesian Bloggers

hasta luego, Susie!

Tue, 2014-08-19 18:14

I just heard that our dear, dear friend Susie Bayarri passed away early this morning, on August 19, in Valencià, Spain… I had known Susie for many, many years, our first meeting being in Purdue in 1987, and we shared many, many great times during simultaneous visits to Purdue University and Cornell University in the 1990’s. During a workshop in Cornell organised by George Casella (to become the unforgettable Camp Casella!), we shared a flat together and our common breakfasts led her to make fun of my abnormal consumption of cereals  forever after, a recurrent joke each time we met! Another time, we were coming from the movie theatre in Lafayette in Susie’ s car when we got stopped for going through a red light. Although she tried very hard, her humour and Spanish verve were for once insufficient to convince her interlocutor.

Susie was a great Bayesian, contributing to the foundations of Bayesian testing in her numerous papers and through the direction of deep PhD theses in Valencia. As well as to queuing systems and computer models. She was also incredibly active in ISBA, from the very start of the Bayesian society, and was one of the first ISBA presidents. She also definitely contributed to the Objective Bayes section of ISBA, especially in the construction of the O’Bayes meetings. She gave a great tutorial on Bayes factors at the last O’Bayes conference in Duke last December, full of jokes and passion, despite being already weak from her cancer…

So, hasta luego, Susie!, from all your friends. I know we shared the same attitude about our Catholic education and our first names heavily laden with religious meaning, but I’d still like to believe that your rich and contagious laugh now resonates throughout the cosmos. So, hasta luego, Susie, and un abrazo to all of us missing her.

Filed under: Statistics, University life Tagged: Bayesian foundations, Cornell University, George Casella, ISBA, O'Bayes, Purdue University, Spain, Susie Bayarri, València
Categories: Bayesian Bloggers

a week at the lake (#4)

Tue, 2014-08-19 08:18
Categories: Bayesian Bloggers

on intelligent design…

Mon, 2014-08-18 18:11

In connection with Dawkins’ The God delusion, which review is soon to appear on the ‘Og, a poster at an exhibit on evolution in the Harvard Museum of Natural History, which illustrates one of Dawkins’ points on scientific agosticism. Namely, that refusing to take a stand on the logical and philosophical opposition between science and religion(s) is not a scientific position. The last sentence in the poster is thus worse than unnecessary…

Filed under: Books, Kids, Travel Tagged: "intelligent" design, agnosticism, atheism, Cambridge, creationism, evolution, Harvard, Harvard Museum of Natural History
Categories: Bayesian Bloggers

a week at the lake (#3)

Sun, 2014-08-17 18:14
Categories: Bayesian Bloggers

Munbai map [recycled art]

Sat, 2014-08-16 18:14

While my transfer in Munbai from Terminal 1 to Terminal 2 was a bit hectic, with a security luggage scan before boarding a shuttle that would drive us to outside the terminal [next to slums that seemed to have direct access to the runways of the airport!), the brand new Terminal 2 was impressive as well as efficient as I went through security and passport controls quite quickly. I eventually reached the museum part, as this terminal holds an unbelievable collection of artefacts, sculptures and paintings. From the lounge, I could admire the above map of the region of Mumbai, made of recycled computer boards, by Akshay Rajpurkar, which I found deeply moving. If I ever travel past Mumbai T2 terminal, I will make sure to tour the rest of the collection!

Filed under: pictures, Travel Tagged: Akshay Rajpurkar, art brut, chips, India, lounge, motherboard, Mumbai, Mumbai airport, Terminal 2
Categories: Bayesian Bloggers