Xian's Og

Syndicate content Xi'an's Og
an attempt at bloggin, nothing more...
Updated: 26 min 41 sec ago

AISTATS 2014 (day #1)

Tue, 2014-04-22 18:14

First day at AISTATS 2014! After three Icelandic vacations days driving (a lot) and hinkg (too little) around South- and West-Iceland, I joined close to 300 attendees for this edition of the AISTATS conference series. I was quite happy to be there, if only because I had missed the conference last year (in Phoenix) and did not want this to become a tradition… Second, the mix of statistics, artificial intelligence and machine learning that characterises this conference is quite exciting, if challenging at time. What I most appreciated in this discovery of the conference is the central importance of the poster session, most talks being actually introductions to or oral presentations of posters! I find this feature terrific enough (is there such a notion as “terrific enough”?!) worth adopting in future conferences I am involved in. I just wish I had managed to tour the whole collection of posters today… The (first and) plenary lecture was delivered by Peter Bühlman, who spoke about a compelling if unusual (for me) version of causal inference. This was followed by sessions on Gaussian processes, graphical models, and mixed data sources. One highlight talk was the one by Marc Deisenroth, who showed impressive robotic fast learning based on Gaussian processes. At the end of this full day, I also attended an Amazon mixer where I learned about Amazon‘s entry on the local market, where it seems the company is getting a better picture of the current and future state of the U.S. economy than governmental services, thanks to a very fine analysis of the sales and entries on Amazon‘s entry. Then it was time to bike “home” on my rental bike, in the setting sun…


Filed under: Mountains, pictures, Running, Statistics, Travel, University life Tagged: AISTATS 2014, Amazon, Þingvellir, Iceland, machine learning, Phoenix
Categories: Bayesian Bloggers

a day for comments

Sun, 2014-04-20 18:14

As I was flying over Skye (with [maybe] a first if hazy perspective on the Cuillin ridge!) to Iceland, three long sets of replies to some of my posts appeared on the ‘Og:

Thanks to them for taking the time to answer my musings…

 


Filed under: Mountains, Statistics, Travel, University life Tagged: AISTATS 2014, Bayesian variable selection, Brad Carlin, Cuillin ridge, Gaussian mixture, Gibbs sampler, hierarchical models, Iceland, ICML, Langevin MCMC algorithm, MCMC, Metropolis-Hastings algorithms, mixtures, model complexity, penalisation, reference priors, Reykjavik, RJMCMC, Russian doll, Scotland, sequential Monte Carlo, Sid Chib, Skye, speedup, spike-and-slab prior, variable dimension models
Categories: Bayesian Bloggers

Turley zinfandel

Sun, 2014-04-20 14:20


Filed under: Wines Tagged: California, Napa Valley, Turley, zinfandel
Categories: Bayesian Bloggers

Den of thieves [book review]

Sat, 2014-04-19 18:14

Last month, I ordered several books on amazon,  taking advantage of my amazon associate gains, and some of them were suggested by amazon algorithms based on my recent history. As I had recently read books involving thieves (like Giant Thief, or Broken Blade and the subsequent books), a lot of titles involved thieves or thievery related names… I picked Den of Thieves mainly for its cover as I did not know the author and the story sounded rather common. When I started reading the book, the story got more and more common, pertaining more to an extended Dungeons & Dragons scenario than to a genuine book! The theme of a bright young thief emerging from the gritty underworld of a close city has been over and over exploited in the fantasy literature, the best (?) example being The lies of Locke Lamora. (Whose third volume, The Republic of Thieves, is in my bag for Reykjavik!) This time, the thief does not appear particularly bright, except at times when he starts philosophy-sing with extremely dangerous enemies!, and the way he eventually overcomes insanely unbalanced odds is just too much. Most characters in the novel are not particularly engaging and way too much caricaturesque from the terribly evil sorcerer cavorting with she-demons to the rigid knight sticking to an idealistic vision of the world where ‘honour” and the code of chivalry is the solution to all problems. It is not even in the slightest sarcastic or tongue-in-cheek as the many novels by David Eddings and the main characters are mostly humourless. I wonder why the book did not get better edited as the weaknesses are very easy to spot! A good example where amazon software failed to make a worthy recommendation!


Filed under: Books, Kids, pictures Tagged: Amazon, David Chandler, David Eddings, Den of Thieves
Categories: Bayesian Bloggers

beer factory

Sat, 2014-04-19 07:54
Categories: Bayesian Bloggers

走ることについて語るときに僕の語ること [book review]

Fri, 2014-04-18 18:14

The English title of this 2007 book of Murakami is “What I talk about when I talk about running”. Which is a parody of Raymond Carver’s collection of [superb] short stories, “What we talk about when we talk about love”. (Murakami translated the complete œuvres of Raymond Carver in Japanese.) It is a sort of diary about Murakami’s running practice and the reasons why he is running. It definitely is not a novel and the style is quite loose or lazy, but this is not a drawback as the way the book is written somehow translates the way thoughts drift away and suddenly switch topics when one is running. At least during low-intensity practice, when I often realise I have been running for minutes without paying any attention to my route. Or when I cannot recall what I was thinking about for the past minutes. During races, the mind concentration is at a different level, first focussing on keeping the right pace, refraining from the deadly rush during the first km, then trying to merge with the right batch of runners, then fighting wind, slope, and eventually fatigue. While the book includes more general autobiographical entries than those related with Murakami’s runner’s life, there are many points most long-distance runners would relate with. From the righteous  feeling of sticking to a strict training and diet, to the almost present depression catching us in the final kms of a race, to the very flimsy balance between under-training and over-training, to the strangely accurate control over one’s pace at the end of a training season, and, for us old runners, to the irremediable decline in one’s performances as years pass by… On a more personal basis, I also shared the pain of hitting one of the slopes in Central Park and the lack of nice long route along Boston’s Charles river. And shared the special pleasure of running near a river or seafront (which is completely uncorrelated with the fact it is flat, I believe!) Overall, what I think this book demonstrates is that there is no rational reason to run, which makes the title more than a parody, as fighting weight, age, health problems, depression, &tc. and seeking solitude, quiet, exhaustion, challenge, performances, zen, &tc. are only partial explanations. Maybe the reason stated in the book that I can relate the most with is this feeling of having an orderly structure one entirely controls (provided the body does not rebel!) at least once a day.  Thus, I am not certain the book appeals to non-runners. And contrary to some reviews of the book, it certainly is not a training manual for novice runners. (Murakami clearly is a strong runner so some of his training practice could be harmful to weaker runners…)


Filed under: Books, Running Tagged: Boston, Central Park, Charles river, depression, Haruki Murakami, Hawai, Japan, marathon, New York City Marathon, running, training, ultra-marathon
Categories: Bayesian Bloggers

AI and Statistics 2014

Fri, 2014-04-18 08:12

Today, I am leaving Paris for a 8 day stay in Iceland! This is quite exciting, for many reasons: first, I missed the AISTATS 2013 last year as I was still in the hospital;  second, I am giving a short short tutorial on ABC methods which will be more like a long (two hours)  talk; third, it gives me the fantastic opportunity to visit Iceland for a few days, a place that was top of my wish list of countries to visit. The weather forecast is rather bleak but I am carrying enough waterproof layers to withstand a wee bit of snow and rain… The conference proper starts next Tuesday, April 22, with the tutorials taking place next Friday, April 25. Hence leaving me three completely free days for exploring the area near Reykjavik.


Filed under: Kids, Mountains, Statistics, Travel, University life Tagged: ABC, AISTATS 2014, Iceland, Reykjavik, tutorial, vacations
Categories: Bayesian Bloggers

AI and Statistics 2014

Fri, 2014-04-18 08:12

Today, I am leaving Paris for a 8 day stay in Iceland! This is quite exciting, for many reasons: first, I missed the AISTATS 2013 last year as I was still in the hospital;  second, I am giving a short short tutorial on ABC methods which will be more like a long (two hours)  talk; third, it gives me the fantastic opportunity to visit Iceland for a few days, a place that was top of my wish list of countries to visit. The weather forecast is rather bleak but I am bring enough waterproof layers to withstand a wee bit of snow and rain… The conference proper starts next Tuesday, April 22, with the tutorials taking place next Friday, April 25. Hence leaving me three completely free days for exploring the area near Reykjavik.


Filed under: Kids, Mountains, Statistics, Travel, University life Tagged: ABC, AISTATS 2014, Iceland, Reykjavik, tutorial, vacations
Categories: Bayesian Bloggers

Dan Simpson’s seminar at CREST

Thu, 2014-04-17 18:14

Daniel Simpson gave a seminar at CREST yesterday on his recently arXived paper, “Penalising model component complexity: A principled, practical  approach to constructing priors” written with Thiago Martins, Andrea Riebler, Håvard Rue, and Sigrunn Sørbye. Paper that he should also have given in Banff last month had he not lost his passport in København airport…  I have already commented at length on this exciting paper, hopefully to become a discussion paper in a top journal!, so I am just pointing out two things that came to my mind during the energetic talk delivered by Dan to our group. The first thing is that those penalised complexity (PC) priors of theirs rely on some choices in the ordering of the relevance, complexity, nuisance level, &tc. of the parameters, just like reference priors. While Dan already wrote a paper on Russian roulette, there is also a Russian doll principle at work behind (or within) PC priors. Each shell of the Russian doll corresponds to a further level of complexity whose order need be decided by the modeller… Not very realistic in a hierarchical model with several types of parameters having only local meaning.

My second point is that the construction of those “politically correct” (PC) priors reflects another Russian doll structure, namely one of embedded models, hence would and should lead to a natural multiple testing methodology. Except that Dan rejected this notion during his talk, by being opposed to testing per se. (A good topic for one of my summer projects, if nothing more, then!)


Filed under: Kids, Mountains, Statistics, Travel, University life Tagged: Banff, BiPS, CREST, hierarchical models, model complexity, Paris, penalisation, reference priors, Russian doll, Russian roulette
Categories: Bayesian Bloggers

Dan Simpson’s seminar at CREST

Thu, 2014-04-17 18:14

Daniel Simpson gave a seminar at CREST yesterday on his recently arXived paper, “Penalising model component complexity: A principled, practical  approach to constructing priors” written with Thiago Martins, Andrea Riebler, Håvard Rue, and Sigrunn Sørbye. Paper that he should also have given in Banff last month had he not lost his passport in København airport…  I have already commented at length on this exciting paper, hopefully to become a discussion paper in a top journal!, so I am just pointing out two things that came to my mind during the energetic talk delivered by Dan to our group. The first thing is that those penalised complexity (PC) priors of theirs rely on some choices in the ordering of the relevance, complexity, nuisance level, &tc. of the parameters, just like reference priors. While Dan already wrote a paper on Russian roulette, there is also a Russian doll principle at work behind (or within) PC priors. Each shell of the Russian doll corresponds to a further level of complexity whose order need be decided by the modeller… Not very realistic in a hierarchical model with several types of parameters having only local meaning.

My second point is that the construction of those “politically correct” (PC) priors reflects another Russian doll structure, namely one of embedded models, hence would and should lead to a natural multiple testing methodology. Except that Dan rejected this notion during his talk, by being opposed to testing per se. (A good topic for one of my summer projects, if nothing more, then!)


Filed under: Kids, Mountains, Statistics, Travel, University life Tagged: Banff, BiPS, CREST, hierarchical models, model complexity, Paris, penalisation, reference priors, Russian doll, Russian roulette
Categories: Bayesian Bloggers

MCMC for sampling from mixture models

Wed, 2014-04-16 18:14

Randal Douc, Florian Maire, and Jimmy Olsson recently arXived a paper on the use of Markov chain Monte Carlo methods for the sampling of mixture models, which contains the recourse to Carlin and Chib (1995) pseudo-priors to simulate from a mixture distribution (and not from the posterior distribution associated with a mixture sampling model). As reported earlier, I was in the thesis defence of Florian Maire and this approach had already puzzled me at the time. In short, a mixture structure

gives rises to as many auxiliary variables as there are components, minus one: namely, if a simulation z is generated from a given component i of the mixture, one can create pseudo-simulations u from all the other components, using pseudo-priors à la Carlin and Chib. A Gibbs sampler based on this augmented state-space can then be implemented:  (a) simulate a new component index m given (z,u);  (b) simulate a new value of (z,u) given m. One version (MCC) of the algorithm simulates z given m from the proper conditional posterior by a Metropolis step, while another one (FCC) only simulate the u‘s. The paper shows that MCC has a smaller asymptotic variance than FCC. I however fail to understand why a Carlin and Chib is necessary in a mixture context: it seems (from the introduction) that the motivation is that a regular Gibbs sampler [simulating z by a Metropolis-Hastings proposal then m] has difficulties moving between components when those components are well-separated. This is correct but slightly moot, as each component of the mixture can be simulated separately and in advance in z, which leads to a natural construction of (a) the pseudo-priors used in the paper, (b) approximations to the weights of the mixture, and (c) a global mixture independent proposal, which can be used in an independent Metropolis-Hastings mixture proposal that [seems to me to] alleviate(s) the need to simulate the component index m. Both examples used in the paper, a toy two-component two-dimensional Gaussian mixture and another toy two-component one-dimensional Gaussian mixture observed with noise (and in absolute value), do not help in perceiving the definitive need for this Carlin and Chib version. Especially when considering the construction of the pseudo-priors.


Filed under: Kids, Statistics, University life Tagged: Carlin, Gaussian mixture, mixtures
Categories: Bayesian Bloggers

MCMC for sampling from mixture models

Wed, 2014-04-16 18:14

Randal Douc, Florian Maire, and Jimmy Olsson recently arXived a paper on the use of Markov chain Monte Carlo methods for the sampling of mixture models, which contains the recourse to Carlin and Chib (1995) pseudo-priors to simulate from a mixture distribution (and not from the posterior distribution associated with a mixture sampling model). As reported earlier, I was in the thesis defence of Florian Maire and this approach had already puzzled me at the time. In short, a mixture structure

gives rises to as many auxiliary variables as there are components, minus one: namely, if a simulation z is generated from a given component i of the mixture, one can create pseudo-simulations u from all the other components, using pseudo-priors à la Carlin and Chib. A Gibbs sampler based on this augmented state-space can then be implemented:  (a) simulate a new component index m given (z,u);  (b) simulate a new value of (z,u) given m. One version (MCC) of the algorithm simulates z given m from the proper conditional posterior by a Metropolis step, while another one (FCC) only simulate the u‘s. The paper shows that MCC has a smaller asymptotic variance than FCC. I however fail to understand why a Carlin and Chib is necessary in a mixture context: it seems (from the introduction) that the motivation is that a regular Gibbs sampler [simulating z by a Metropolis-Hastings proposal then m] has difficulties moving between components when those components are well-separated. This is correct but slightly moot, as each component of the mixture can be simulated separately and in advance in z, which leads to a natural construction of (a) the pseudo-priors used in the paper, (b) approximations to the weights of the mixture, and (c) a global mixture independent proposal, which can be used in an independent Metropolis-Hastings mixture proposal that [seems to me to] alleviate(s) the need to simulate the component index m. Both examples used in the paper, a toy two-component two-dimensional Gaussian mixture and another toy two-component one-dimensional Gaussian mixture observed with noise (and in absolute value), do not help in perceiving the definitive need for this Carlin and Chib version. Especially when considering the construction of the pseudo-priors.


Filed under: Kids, Statistics, University life Tagged: Carlin, Gaussian mixture, mixtures
Categories: Bayesian Bloggers

Journées MAS2014, Toulouse, Aug. 27-29

Wed, 2014-04-16 10:57

For those interested in visiting Toulouse at the end of the summer for a French speaking conference in Probability and Statistics, the Modélisation-Aléatoire-Statistique branch of SMAI (the French version of SIAM) is holding its yearly conference. The main theme this year is “High dimension phenomena”, but a large panel of the French research in Probability and Statistics will be represented. The program contains in particular:

  • Six plenary conferences and 3 talks by the recent winners of the “Prix Jacques Neveu” award [including Pierre Jacob!],
  • 22 parallel sessions, from probability theory to applied statistics and machine learning,
  • Posters session for students

More detail is available on the conference website (in French).  (The organizing committee is made of Aurélien Garivier, Sébastien Gerchinovitz, Aldéric Joulin, Clément Pellegrini, and Laurent Risser.)


Filed under: Kids, pictures, Travel, University life, Wines Tagged: France, Jacques Neveu, MAS, MAS2014, Pierre Jacob, SMAI, Statistics conference, Toulouse
Categories: Bayesian Bloggers

Journées MAS2014, Toulouse, Aug. 27-29

Wed, 2014-04-16 10:57

For those interested in visiting Toulouse at the end of the summer for a French speaking conference in Probability and Statistics, the Modélisation-Aléatoire-Statistique branch of SMAI (the French version of SIAM) is holding its yearly conference. The main theme this year is “High dimension phenomena”, but a large panel of the French research in Probability and Statistics will be represented. The program contains in particular:

  • Six plenary conferences and 3 talks by the recent winners of the “Prix Jacques Neveu” award [including Pierre Jacob!],
  • 22 parallel sessions, from probability theory to applied statistics and machine learning,
  • Posters session for students

More detail is available on the conference website (in French).  (The organizing committee is made of Aurélien Garivier, Sébastien Gerchinovitz, Aldéric Joulin, Clément Pellegrini, and Laurent Risser.)


Filed under: Kids, pictures, Travel, University life, Wines Tagged: France, Jacques Neveu, MAS, MAS2014, Pierre Jacob, SMAI, Statistics conference, Toulouse
Categories: Bayesian Bloggers

MCqMC 2014 [closup]

Tue, 2014-04-15 18:14

As mentioned earlier, this was my very first MCqMC conference and I really enjoyed it, even though (or because) there were many topics that did not fall within my areas of interest. (By comparison, WSC is a serie of conferences too remote from those areas for my taste, as I realised in Berlin where we hardly attended any talk and hardly anyone attended my session!) Here I appreciated the exposure to different mathematical visions on Monte Carlo, without being swamped by applications as at WSC… Obviously, our own Bayesian computational community was much less represented than at, say, MCMSki! Nonetheless, I learned a lot during this conference for instance from Peter Glynn‘s fantastic talk, and I came back home with new problems and useful references [as well as a two-hour delay in the train ride from Brussels]. I also obviously enjoyed the college-town atmosphere of Leuven, the many historical landmarks  and the easily-found running routes out of the town. I am thus quite eager to attend the next MCqMC 2016 meeting (in Stanford, an added bonus!) and even vaguely toying with the idea of organising MCqMC 2018 in Monaco (depending on the return for ISBA 2016 and ISBA 2018). In any case, thanks to the scientific committee for the invitation to give a plenary lecture in Leuven and to the local committee for a perfect organisation of the meeting.


Filed under: pictures, Running, Statistics, Travel, University life, Wines Tagged: Berlin, Brussels, ISBA 2016, Leuven, MCMSki IV, MCQMC2014, train, WSC 2012
Categories: Bayesian Bloggers

MCqMC 2014 [closup]

Tue, 2014-04-15 18:14

As mentioned earlier, this was my very first MCqMC conference and I really enjoyed it, even though (or because) there were many topics that did not fall within my areas of interest. (By comparison, WSC is a serie of conferences too remote from those areas for my taste, as I realised in Berlin where we hardly attended any talk and hardly anyone attended my session!) Here I appreciated the exposure to different mathematical visions on Monte Carlo, without being swamped by applications as at WSC… Obviously, our own Bayesian computational community was much less represented than at, say, MCMSki! Nonetheless, I learned a lot during this conference for instance from Peter Glynn‘s fantastic talk, and I came back home with new problems and useful references [as well as a two-hour delay in the train ride from Brussels]. I also obviously enjoyed the college-town atmosphere of Leuven, the many historical landmarks  and the easily-found running routes out of the town. I am thus quite eager to attend the next MCqMC 2016 meeting (in Stanford, an added bonus!) and even vaguely toying with the idea of organising MCqMC 2018 in Monaco (depending on the return for ISBA 2016 and ISBA 2018). In any case, thanks to the scientific committee for the invitation to give a plenary lecture in Leuven and to the local committee for a perfect organisation of the meeting.


Filed under: pictures, Running, Statistics, Travel, University life, Wines Tagged: Berlin, Brussels, ISBA 2016, Leuven, MCMSki IV, MCQMC2014, train, WSC 2012
Categories: Bayesian Bloggers

adaptive subsampling for MCMC

Mon, 2014-04-14 18:14

“At equilibrium, we thus should not expect gains of several orders of magnitude.”

As was signaled to me several times during the MCqMC conference in Leuven, Rémi Bardenet, Arnaud Doucet and Chris Holmes (all from Oxford) just wrote a short paper for the proceedings of ICML on a way to speed up Metropolis-Hastings by reducing the number of terms one computes in the likelihood ratio involved in the acceptance probability, i.e.

The observations appearing in this likelihood ratio are a random subsample from the original sample. Even though this leads to an unbiased estimator of the true log-likelihood sum, this approach is not justified on a pseudo-marginal basis à la Andrieu-Roberts (2009). (Writing this in the train back to Paris, I am not convinced this approach is in fact applicable to this proposal as the likelihood itself is not estimated in an unbiased manner…)

In the paper, the quality of the approximation is evaluated by Hoeffding’s like inequalities, which serves as the basis for a stopping rule on the number of terms eventually evaluated in the random subsample. In fine, the method uses a sequential procedure to determine if enough terms are used to take the decision and the probability to take the same decision as with the whole sample is bounded from below. The sequential nature of the algorithm requires to either recompute the vector of likelihood terms for the previous value of the parameter or to store all of them for deriving the partial ratios. While the authors adress the issue of self-evaluating whether or not this complication is worth the effort, I wonder (from my train seat) why they focus so much on recovering the same decision as with the complete likelihood ratio and the same uniform. It would suffice to get the same distribution for the decision (an alternative that is easier to propose than to create of course). I also (idly) wonder if a Gibbs version would be manageable, i.e. by changing only some terms in the likelihood ratio at each iteration, in which case the method could be exact… (I found the above quote quite relevant as, in an alternative technique we are constructing with Marco Banterle, the speedup is particularly visible in the warmup stage.) Hence another direction in this recent flow of papers attempting to speed up MCMC methods against the incoming tsunami of “Big Data” problems.


Filed under: pictures, Statistics, Travel Tagged: acceptance rate, Brussels, Gibbs sampler, Hoeffding, ICML, Leuven, MCMC, MCQMC2014, Metropolis-Hastings algorithms, Paris, sequential Monte Carlo, speedup
Categories: Bayesian Bloggers

adaptive subsampling for MCMC

Mon, 2014-04-14 18:14

“At equilibrium, we thus should not expect gains of several orders of magnitude.”

As was signaled to me several times during the MCqMC conference in Leuven, Rémi Bardenet, Arnaud Doucet and Chris Holmes (all from Oxford) just wrote a short paper for the proceedings of ICML on a way to speed up Metropolis-Hastings by reducing the number of terms one computes in the likelihood ratio involved in the acceptance probability, i.e.

The observations appearing in this likelihood ratio are a random subsample from the original sample. Even though this leads to an unbiased estimator of the true log-likelihood sum, this approach is not justified on a pseudo-marginal basis à la Andrieu-Roberts (2009). (Writing this in the train back to Paris, I am not convinced this approach is in fact applicable to this proposal as the likelihood itself is not estimated in an unbiased manner…)

In the paper, the quality of the approximation is evaluated by Hoeffding’s like inequalities, which serves as the basis for a stopping rule on the number of terms eventually evaluated in the random subsample. In fine, the method uses a sequential procedure to determine if enough terms are used to take the decision and the probability to take the same decision as with the whole sample is bounded from below. The sequential nature of the algorithm requires to either recompute the vector of likelihood terms for the previous value of the parameter or to store all of them for deriving the partial ratios. While the authors adress the issue of self-evaluating whether or not this complication is worth the effort, I wonder (from my train seat) why they focus so much on recovering the same decision as with the complete likelihood ratio and the same uniform. It would suffice to get the same distribution for the decision (an alternative that is easier to propose than to create of course). I also (idly) wonder if a Gibbs version would be manageable, i.e. by changing only some terms in the likelihood ratio at each iteration, in which case the method could be exact… (I found the above quote quite relevant as, in an alternative technique we are constructing with Marco Banterle, the speedup is particularly visible in the warmup stage.) Hence another direction in this recent flow of papers attempting to speed up MCMC methods against the incoming tsunami of “Big Data” problems.


Filed under: pictures, Statistics, Travel Tagged: acceptance rate, Brussels, Gibbs sampler, Hoeffding, ICML, Leuven, MCMC, MCQMC2014, Metropolis-Hastings algorithms, Paris, sequential Monte Carlo, speedup
Categories: Bayesian Bloggers