TheorySpace

A lab for theoretical plurality

Hold more than one theory at a time.

Almost any dataset can be explained by several different theories. The usual practice is to pick one, fit it and report the answer as if the others had never been in the running.

I think we're entering a period where that stops being good enough, and where the people who do well are the ones who can keep several theories in play and weigh them properly. TheorySpace is where I work on the ideas and the tools for doing that.

Three theories fitted to one datasetTwelve observations rise from left to right. Three curves all pass through them. Past the last observation one keeps rising, one levels off and one turns down.Theory A · 0.46Theory B · 0.33Theory C · 0.21ObservedNot yet observed
Three theories, one dataset. They agree where there's data and part ways where there isn't, which is usually where the decision sits. The numbers are each theory's share of the belief. Illustrative.

The argument

  1. Plurality is the normal case

    A dataset is what's left behind after some process has run, and there's nearly always more than one process that could have left it. The candidate theories agree where you have data. They part ways where you don't, and that's usually the place you need an answer.

  2. Bayes is how you keep the books

    Bayesian inference was built for this. You put belief on the theories as well as on their parameters, let the data move that belief around, and carry whatever doubt is left into the prediction. If you aren't sure which theory is right, the forecast should say so.

  3. Why it matters now

    A new theory used to be expensive: one analyst, one model, weeks of work. An AI system can now draft ten candidate models in an afternoon, each with working code and a confident write-up. Producing theories is no longer the hard part. Comparing them, weighting them and deciding what to do when they disagree is.

  4. It isn't easy

    The true theory usually isn't on your list. Priors over theories are hard to defend, and the weights can swing on choices nobody looked at. The computation is heavy, and the person waiting for the result wants one number. I don't have clean answers to any of this, which is why I'm calling it a lab.

What this space is for

Ideas

Notes on what it means to work with several theories at once, and where the approach breaks down.

Tools

Methods tried on public data with the code posted, including the parts that didn't work.

Tutorials

Worked examples you can run yourself, starting with hierarchical models.

I'm Jay Sheldon. I trained in mathematics and I build Bayesian models for a living. This site is where I do the research side of that in the open.

Where I'm applying it

More detail

Bayesian models of choice and preference

Hierarchical Bayes for conjoint and MaxDiff, and the demand simulators built on top of them. This is my day job, so it's where I have the most opinions.

Analytic agents and their evaluation

Agents that pull evidence together, suggest hypotheses and run the analysis. What I mostly want to know is how you'd check their work.

Causal inference

Effects from observational data, with the assumptions written down and a look at how hard the answer leans on each one.

Adaptive experiments

Bandit and sequential designs next to plain balanced ones. Adaptive designs save sample, and I'd like to know what that costs you when it's time to estimate something.

Latest work

All notes

Applied work

I also do this for a living. I take on consulting projects, model reviews, agent builds and research collaborations, plus the occasional contract or fractional role. Paid work runs through Significance.ai, my consulting practice.