One Incontrovertible, Unequivocal, Undeniable Fact Which Refutes the Diary

[Iconoclast]

Jeff,

I think it's only fair that you get a proper briefing this time.

Here's the scenario: James Maybrick dies on May 11, 1889. On March 9, 1992, his study floorboards are known to have been raised on the record for the first time in the intervening 37,557 days (including March 9, 1992, obviously). Now, if you knew that during those 37,557 days, someone had 'phoned a literary agent claiming to have the diary of Jack the Ripper where James Maybrick turned out to be the purported author, what were the odds that those two events would happen on the same day out of the 37,557 by sheer chance alone?

I hope that you and I can agree that - at the very simplest level of probability analysis - those odds are:

Number of times the floorboards were raised (on the record) divided by the number of days which the floorboards could have come up.

There's some debate about how big our numerator should be, but - as we're talking on the record events - I think you'll agree that that is 1 over 37,557?

(As I've said, I'd accept '2' if you want to include the unknown day in the 1960s that Maybrick's old office was demolished.)

Now, you and I know that we would never ever ever ever ever bet on anything where the odds were 1/37,557 (or 2/37,557) because we are statisticians and we know that those odds are truly awful.

Now, take a deep breath, Jeff. Those two events did happen during those 37,557 days (they obviously didn't have to) and - lo and behold - they happened on the same day. Yes, the very first time the floorboards were raised on the record after an agonising wait of 37,557 days, the 'phone call went in to the literary agent.

Cut to the chase: the chances of this happening by chance alone were 1/37,557 (or 2/37,557) - agree or disagree?

Additional information you may want to cogitate over:

1) The call could have been made from anywhere in the entire world, but the guy who made that call came from Liverpool, where the floorboards were raised that morning; and
2) Although Maybrick's house was EIGHT MILES away from where 'Phonecall Guy lived, a member of the electrical team just happened to drink in the same pub as he.

Please don't attempt to influence the initial calculation with that information. Please just confirm that you agree the odds of that first coincidence happening by chance alone were either 1/37,557 or 2/37,557 (please stick to on the record events for your numerator otherwise you aren't going to be answering the question you've been asked).

Ike

[rjpalmer]

are you actually claiming that probability for known events happening simultaneously by chance alone cannot ever be calculated? [B]

Known events happening simultaneously and not proven to be causality connected cannot be analyzed and given a meaningful probability.

Yes, that is correct. The 'connection' might not be actual; it may simply be part of a human 'story,' and human stories are created at an alarmingly probable rate.

Obviously, if someone pulls the trigger of a gun, and the gun fires, we know the two events are causality related, and so we can calculate how many times the gun would normally go off (100% of the time, give or take a few, correcting for misfires, sudden paralysis, blank cartridges, etc.) We also know if a coin is flipped it will land either heads or tails, or, in few hundred thousand throws, on its edge. It's a law of physics. Such things can be calculated using statistics.

But you cannot take two events that happened simultaneously, but you do not know were causality connected (the diary and the floorboards) and pretend to calculate the odds. And certainly not by assuming the floorboards are 100% relevant, and then start rambling on about 37,585 days since they were last lifted, etc.

This sort of statistical analysis is confined to the corridors of Colney Hatch.

This has been explained too many times already, Ike. You'll have to get over it.

So, instead of offering further resistance, let me just dedicate the following song to you, Ike. It might help. The moment you are stuck in is 3 May 1889. You must find a way out of it. You must muster your wits and your courage and realize it is not a legitimate part of your calculations. Good luck.

[Iconoclast]

But you cannot take two events that happened simultaneously, but you do not know were causality connected (the diary and the floorboards) and pretend to calculate the odds. And certainly not by assuming the floorboards are 100% relevant, and then start rambling on about 37,585 days since they were last lifted, etc.

OMG RJ, what on earth do you think we've been debating for the last few weeks??????????

The probability calculation is based upon the two events NOT being 'causality connected'!!! Two Maybrick-related events - one fairly prosaic but evidently very very rare (lifting of floorboards), the other very obscure and almost certainly bound to be unique (the call to the literary agent) happened on the same day after 37,557 days, and we have been debating the probability that they are unrelated (that is, resulting purely and simply from fickle chance that day).

What on earth have you been thinking we were talking about, man? If we know they were 'causality connected', we would not need to resort to probability theory because we would know that chance played no role that day (by definition!). If we do not know if they were 'causality connected', we would need to resort to simple probability theory to show how likely it was that those two events would occur on the same day by chance alone after 37,557 days when one or both could have happened.

This has been explained too many times already, Ike. You'll have to get over it.

You have done no more than 'explain' how badly you have understood the debate we've been having, RJ.

Ike

[The Baron]

Thanks Dave

You have to be grateful there is Lord Orsam in the field to educate you.

Your post is full of hate, the wounds are screaming inside you.

Don't ever accuse me of being Orsam again.



The Baron

[erobitha]

You have to be grateful there is Lord Orsam in the field to educate you.

Your post is full of hate, the wounds are screaming inside you.

Don't ever accuse me of being Orsam again.



The Baron

Bit melodramatic.

[JeffHamm]

Ok, maybe if I try it this way the flaw in the approach will be more apparent.

The idea being argued is that two events known to have happened are considered improbable events, so the combination of them must be so improbable that they must therefore be connected. This is evident from Ike's post #6706, particularly the point "...we have been debating the probability that they are unrelated (that is, resulting purely and simply from fickle chance that day." The implication being that because they are improbable by chance, they must be related.

What I'm trying to point out is that known events are no longer improbable, they happened. We know the floor boards were lifted on that day. We know Mike called the publishers that day. You cannot infer a causal link between them, which is the opposite of being "unrelated", by this kind of probability calculation.

All events that happen are extremely improbable events when you do that. Look, I exist because a particular sperm combined with a particular egg at my conception. The probability of that particular sperm and that particular egg combining is, if we apply this same approach, so improbable that I should not exist unless there was some causal design between that selection process. The same, of course, applies to each and every one of us, so the "chance" probability of any of us conversing right now, albeit through posts, is so infinitesimally small, that it really cannot be happening. Add those probabilities to the characters involved in the finding and exchanging of the book, and the chance combination of the floorboards and phone call pale in comparison with out improbable those two people should event exit, let alone interact on this particular day.

It's will-o-the-wisps, and meaningless calculations that have nothing at all to do with whether or not there is a connection between the two events in question.

The more specific you look at any set of events that happen in life, the more these kinds of calculations will demonstrate that even normal everyday life is just a sequence of incredibly unlikely events. That doesn't make them connected. Things happen. Look, the other day I was outside on my doorstep, and a single car drove past, and the licence plate had for the 3 letter portion my initials. That 3 letter combination only has a 1 in 17,576 probability of occurring by chance, so was it a sign? Did I somehow "cause" that combination?

When I walk into work, I pass people, buses, and so forth, in a particular order, and the chance probability of me passing those people, in that order, is infinitesimally small. It doesn't mean there's a connection, or meaning, involved.

So while we could quibble about what the probability of the floorboards coming up are, such as it increases over time, and so forth, none of that matters one whit. This sort of probability calculation is of the type that misleads us, because it is irrelevant to the question we are interested in.

Given we know those two events happened, we then have to determine if there are consequences of them being "linked" events. Does the diary show any evidence of having been under the floorboards? Was there dust, or dirt, or whatever, found on the diary that can be traced to that storage area? Was that area damp, had it ever flooded, etc, and if so, does the diary show corresponding water damage? Were there insects under the floorboards? Does the diary show signs of similar insects? Does the electrician have a known history for stealing items from job sites? If not, what's the probability he would suddenly engage in theft this time? How long do people who do come across artifacts normally take to decide on what to do with it (i.e. publish it, auction it, TV rights, movie rights? etc), allowing us to estimate the probability of Mike deciding to contact a book publisher on the first day (so within hours)? Is that probable? I doubt it, but I don't know because that would require doing research into similar things.

What I'm trying to make clear is that having two known events and a bit of creativity makes it entirely probable that a false story could be created to connect them. What we want to know is if the story we have is just one of those false stories, or instead, a true one. And that cannot be determined by post-hoc evaluation of the probabilities of the known events - because those are always improbable - all life events are, when you calculate them, improbable, but events in life happen. These ones happened.

In other words, the entire starting premise of the calculations is wrong. Worrying about the "chance probability co-occurrence" of these two events that are know to have happened will lead to nothing other than the realization of how improbable life is. Deciding if the diary really did come from under the floorboards, and was handed to Mike in the pub, can only be assessed by demonstrating that story leads us to information that cannot, or should not, occur if that story is false. If it does that (leads us to evidence that should not otherwise occur I mean), then we have evidence that our story, which is only one of an infinite number of possible stories we could invent from those two known events, might be the one true one.

But all the focus on calculating the probabilities of the floorboards coming up on that day, whether we spread the probability out equally over all 37000+ days or if we go with a more complicated model where the daily probabilities increase over time, is a complete red herring with regards to whether or not there is a connection between the diary, the floorboards, and Mike.

I really don't know how to make it more clear than that. To be honest, I'm only responding because statistics are so badly misused to nefarious purposes (I'm referring to everyday life, not commenting on things here) that a lot of people have a blanket mistrust of them. Yes, when misused they can be highly abused, but when used properly, they are also highly informative; they're a good tool often badly used. But to use them properly one has to avoid "common sense", because "common sense" is what we resort to when we don't actually know how things work.

- Jeff

[MrBarnett]

Ok, maybe if I try it this way the flaw in the approach will be more apparent.

The idea being argued is that two events known to have happened are considered improbable events, so the combination of them must be so improbable that they must therefore be connected. This is evident from Ike's post #6706, particularly the point "...we have been debating the probability that they are unrelated (that is, resulting purely and simply from fickle chance that day." The implication being that because they are improbable by chance, they must be related.

What I'm trying to point out is that known events are no longer improbable, they happened. We know the floor boards were lifted on that day. We know Mike called the publishers that day. You cannot infer a causal link between them, which is the opposite of being "unrelated", by this kind of probability calculation.

All events that happen are extremely improbable events when you do that. Look, I exist because a particular sperm combined with a particular egg at my conception. The probability of that particular sperm and that particular egg combining is, if we apply this same approach, so improbable that I should not exist unless there was some causal design between that selection process. The same, of course, applies to each and every one of us, so the "chance" probability of any of us conversing right now, albeit through posts, is so infinitesimally small, that it really cannot be happening. Add those probabilities to the characters involved in the finding and exchanging of the book, and the chance combination of the floorboards and phone call pale in comparison with out improbable those two people should event exit, let alone interact on this particular day.

It's will-o-the-wisps, and meaningless calculations that have nothing at all to do with whether or not there is a connection between the two events in question.

The more specific you look at any set of events that happen in life, the more these kinds of calculations will demonstrate that even normal everyday life is just a sequence of incredibly unlikely events. That doesn't make them connected. Things happen. Look, the other day I was outside on my doorstep, and a single car drove past, and the licence plate had for the 3 letter portion my initials. That 3 letter combination only has a 1 in 17,576 probability of occurring by chance, so was it a sign? Did I somehow "cause" that combination?

When I walk into work, I pass people, buses, and so forth, in a particular order, and the chance probability of me passing those people, in that order, is infinitesimally small. It doesn't mean there's a connection, or meaning, involved.

So while we could quibble about what the probability of the floorboards coming up are, such as it increases over time, and so forth, none of that matters one whit. This sort of probability calculation is of the type that misleads us, because it is irrelevant to the question we are interested in.

Given we know those two events happened, we then have to determine if there are consequences of them being "linked" events. Does the diary show any evidence of having been under the floorboards? Was there dust, or dirt, or whatever, found on the diary that can be traced to that storage area? Was that area damp, had it ever flooded, etc, and if so, does the diary show corresponding water damage? Were there insects under the floorboards? Does the diary show signs of similar insects? Does the electrician have a known history for stealing items from job sites? If not, what's the probability he would suddenly engage in theft this time? How long do people who do come across artifacts normally take to decide on what to do with it (i.e. publish it, auction it, TV rights, movie rights? etc), allowing us to estimate the probability of Mike deciding to contact a book publisher on the first day (so within hours)? Is that probable? I doubt it, but I don't know because that would require doing research into similar things.

What I'm trying to make clear is that having two known events and a bit of creativity makes it entirely probable that a false story could be created to connect them. What we want to know is if the story we have is just one of those false stories, or instead, a true one. And that cannot be determined by post-hoc evaluation of the probabilities of the known events - because those are always improbable - all life events are, when you calculate them, improbable, but events in life happen. These ones happened.

In other words, the entire starting premise of the calculations is wrong. Worrying about the "chance probability co-occurrence" of these two events that are know to have happened will lead to nothing other than the realization of how improbable life is. Deciding if the diary really did come from under the floorboards, and was handed to Mike in the pub, can only be assessed by demonstrating that story leads us to information that cannot, or should not, occur if that story is false. If it does that (leads us to evidence that should not otherwise occur I mean), then we have evidence that our story, which is only one of an infinite number of possible stories we could invent from those two known events, might be the one true one.

But all the focus on calculating the probabilities of the floorboards coming up on that day, whether we spread the probability out equally over all 37000+ days or if we go with a more complicated model where the daily probabilities increase over time, is a complete red herring with regards to whether or not there is a connection between the diary, the floorboards, and Mike.

I really don't know how to make it more clear than that. To be honest, I'm only responding because statistics are so badly misused to nefarious purposes (I'm referring to everyday life, not commenting on things here) that a lot of people have a blanket mistrust of them. Yes, when misused they can be highly abused, but when used properly, they are also highly informative; they're a good tool often badly used. But to use them properly one has to avoid "common sense", because "common sense" is what we resort to when we don't actually know how things work.

- Jeff

All way above my head, but I’m assuming that, counter-intuitively, 366/254 could be equally as nefarious as 37,000.

‘O’ dear!

[JeffHamm]

All way above my head, but I’m assuming that, counter-intuitively, 366/254 could be equally as nefarious as 37,000.

‘O’ dear!

Equally irrelevant with regards to the question of interest might be the way to phrase it. And, to be clear, I'm not suggesting people here are acting with nefarious intention (that was more a comment upon the world in general), but I do think people are being led astray by the will-o-the-wisp of irrelevant probabilities. Anyway, I'm not sure if I've been able to get the critical point across, but there are right ways to use statistics and there are wrong ways.

The short version, I guess, is that while one could do the arithmetic for the calculations being discussed here correctly doesn't change the fact that this is simply the wrong arithmetic to be doing if one is interested in making claims about there being a relationship between the two events.

- Jeff

[rjpalmer]

I don't think Orsam was seriously suggesting that the odds of Dodd's floorboards coinciding with Barrett calling a literary agent was 18 to 1.

People chose to interpret it that way, but I think all he was pointing out is that it isn't all that strange for a man to have work done on his house. We know of 14 days in 1992 when this happened at 'Battlecrease.' And since the diary doesn't specifically mention floorboards, any work at Battlecrease could have triggered a provenance tale.

But yeah, I think we can now agree that statistics don't really belong here.

I just noticed that the Israeli Prime Minister Menachem Begin died on 9 March 1992, at 3:30 a.m., Tel Aviv time. The same day that Barrett called a literary agent.

He was 85, and theoretically could have died any time over the previous 28,695 days.

But this is meaningless information, unless we attach meaning to it.

Which we won't.

[Iconoclast]

Ok, maybe if I try it this way the flaw in the approach will be more apparent.

Jeff, maybe try it this way - try it the way I asked without feeling the need to complicate it with the addition of inference. We'll do inference next, just to keep everyone happy, but most of our dear readers will know that I have asked a basic probability question which did not specify any amount of inference whatsoever. I won't pose the question again. Suffice it to say that the probability of those two events happening by chance after 37,557 days when either or both could have happened on any previous day is very very very very very obviously 1/37,557 (let's skip Knowsley Buildings on this occasion).

So no inference involved. No causality implied. There we have our stat. It's the same process as what is the probability of someone winning the lottery (someone, not a specific person) on any given day the lottery runs. Those odds are NOT 1/13.7m (in the UK lottery) - that is the probability of any one combination of balls being the winning combination that day. The odds of someone winning the lottery are:

How many possible combinations have been sold, over
How many combinations there are

It's the same with the March 9, 1992 event. How many times have either event happened on the record over how many days have passed since both could have happened. Note: No need for inference and factoring-in a multitude of unknowables at this stage.

If you can get over this inference you're doing, you should be able to agree that we are dealing with odds of 1/37,557 that those two events (if they happened at all) would happen on the same day by March 9, 1992 if chance alone were the cause of it.

We should stop there because that's the simple calculation done. I didn't ask what could be inferred from this probability (that is for a different conversation).

The idea being argued is that two events known to have happened are considered improbable events, so the combination of them must be so improbable that they must therefore be connected. This is evident from Ike's post #6706, particularly the point "...we have been debating the probability that they are unrelated (that is, resulting purely and simply from fickle chance that day." The implication being that because they are improbable by chance, they must be related.

This is all focused on inference, Jeff, and therefore you are asking a different question. We can ask ourselves the inferential questions as a consequence of answering our simple probability question, but we can't make progress if you keep 'answering' the simple probability question with answers laden-down with huge doses of inference. We answer the simple probability question first, and then we say "Does that probability give us grounds to start inferring causality?". If it does, the detective in us starts to test for causality between the two or more events. In the same way, the experimenter in us starts to test for causality between two or more events.

I awoke to rather tragic news this morning: a friend of my sister-in-law was involved in a car crash in Ballinger, Texas, yesterday and died. The driver of the other car was my sister-in-law's cousin. Now, that seems incredibly unlikely, but neither I nor Mrs Iconoclast thought it was causally linked. We assumed it was just chance, though Mrs Iconoclast did point out that Ballinger is not a particularly big town so perhaps the odds of it happening by chance were much greater than if the event had happened in Houston. Nevertheless, if my sister-in-law were the local sheriff, I would expect her to at least consider the possibility - however unlikely - that there was a causal link between these two events. I would expect her to use inference to just sense check whether the act was not random at all based on the knowledge that the two people knew one another.

So inference is a good thing. It gives us licence to explore whether apparently random events were actually linked based upon what forms of causal linking seem reasonable to investigate. But none of that is required at Stage 1. At Stage 1, we're just checking the simple probability of two events happening by chance alone.

What I'm trying to point out is that known events are no longer improbable, they happened.

Careful, Jeff, because you are again falling into RJ's trap of implying that known events cease to have probabilities once they occur. When I toss a coin, the odds of throwing a heads is 50% before I throw it. After I throw that heads, the probability that I threw it remains 50%. In your model (if you just follow through with your logic quoted above), because we now know that that throw came up heads means that it's probability (50%) vanishes and is replaced by certainty (100%). You use the word 'improbable' but that's a very subjective term. Objectively, an event with a probability of one-in-million was not certain to happen simply because it did on this occasion happen. The fact that your egg and sperm met was - on average - a one-in-350million chance. When we knew it happened, the odds of it having happened were not majestically risen to certainty (100%, or 1). They remained a one-in-350million chance that day. The fact that you are here to pose the probability question is simply because the sperm that formed you was the pick of the bunch that day. If it hadn't been and some other sperm had, then a different human being might be posting in your place today. Assuming that a human sperm is guaranteed to fuse with a human egg (which is obviously not guaranteed but bear with me here) means that the result was 100% going to be a human being. There was, however, only a 1-in-350million chance that that human being would become Jeff Hamm. And yet 'Jeff Hamm' happened! So there was 100% chance of a human being occurring, but only a 1-in-350million chance of that human being being Jeff Hamm.

We know the floor boards were lifted on that day. We know Mike called the publishers that day. You cannot infer a causal link between them ...

And if you could just get that stick of inference from out of your rear, you would see that the question I have posed requires - at this basic stage - no inference at all.

... which is the opposite of being "unrelated", by this kind of probability calculation.

I'm pretty sure this is incorrect. If two events which coincide are not causally related, they must be chance related. I think I'm right in insisting that this is a simple binary set of alternatives.

All events that happen are extremely improbable events when you do that. Look, I exist because a particular sperm combined with a particular egg at my conception. The probability of that particular sperm and that particular egg combining is, if we apply this same approach, so improbable that I should not exist unless there was some causal design between that selection process.

Dear, dear. The probability that 'Jeff Hamm' would be the product of that moment of coitus was very improbable so if someone had been able to predict it, that would be amazing, but the fact that a sperm and egg met was definitely not remarkable. I'd have happily laid a tenner on that happening, but definitely would not have bet on which sperm would step up to the plate that day. As explained above, the probability of a human being being the result was 100% (if we assume that fusion of sperm and egg would happen that day). The chances of you being the end result was indeed highly improbable, but don't worry about that mate because your mum and dad would have been happy with whichever one of the 350million possibilities they got. I imagine they didn't predict beforehand, "I hope this shag produces a boy who we can call 'Jeff' and who will grow up to be a statistics lecturer and poster on Casebook: Jack the Spratt McVitie". If they had, then what they got really would have been a statistical miracle rather than what it was, a rather routine statistical certainty (all other things being equal).

The same, of course, applies to each and every one of us, so the "chance" probability of any of us conversing right now, albeit through posts, is so infinitesimally small, that it really cannot be happening.

Dear, dear. If someone had been prescient enough to have predicted a year ago we'd be having this conversation, then those odds might be so small that one might state it could never happen. The fact that it has happened should give us cause to look back and wonder if there was any causal link which led to this improbable event becoming manifest. Well, can we infer a causal link? I know I can.

Add those probabilities to the characters involved in the finding and exchanging of the book, and the chance combination of the floorboards and phone call pale in comparison with out improbable those two people should event exit, let alone interact on this particular day.

You're adding in extraneous detail again, Jeff - unquantifiable 'what-ifs' which contribute nothing to the debate. The characters did not have to be the characters, but history shows us that they were.

It's will-o-the-wisps, and meaningless calculations that have nothing at all to do with whether or not there is a connection between the two events in question.

That's right, Jeff, these kind of events are happening constantly throughout the world. My bookshelf is simply heaving with diaries of Jack the Ripper purportedly written by the likes of Donald Swanson, Robert Louis Stevenson, Bismarck, oh and James Maybrick.

The more specific you look at any set of events that happen in life, the more these kinds of calculations will demonstrate that even normal everyday life is just a sequence of incredibly unlikely events. That doesn't make them connected. Things happen. Look, the other day I was outside on my doorstep, and a single car drove past, and the licence plate had for the 3 letter portion my initials. That 3 letter combination only has a 1 in 17,576 probability of occurring by chance, so was it a sign? Did I somehow "cause" that combination?

No, you presumably had no foreknowledge (unless it was your own car, of course) nor had any hand in it being there in that moment you looked outside your door (again, unless it was your own car, of course) therefore there is no reason to think there is any kind of causal link between the two events. But your example is not in reality in the slightest bit improbable. If a monkey typed randomly for all eternity, it would eventually produce the complete works of Shakespeare by sheer chance alone. In the same way, if billions of people look outside their doors numerous times each day, eventually someone somewhere will see a car go past with their initials on the registration plate. If someone had predicted that that would happen to you that day, now that would be a statistical miracle and well worth investigating for possible causality.

So you can't compare apples with flares, Jeff. Only one set of Maybrick's study floorboards came up during those 37,557 halcyon days before any of this debate started, not billions. Only one. Only you looked out of your door, once, in 37,557 days, and in that very moment that you did one car went past your house and it bore your initials. If that was the case, I'd definitely be inferring something very weird had just happened and I would jolly well investigate it further.

When I walk into work, I pass people, buses, and so forth, in a particular order, and the chance probability of me passing those people, in that order, is infinitesimally small. It doesn't mean there's a connection, or meaning, involved.

If you predicted in advance that you would pass those buses, say at a specific time, I'd be impressed. The fact that you passed the ones you did are like the sperm and the egg. Millions of alternatives were possible, but only one event happened. No need to look for causality. It would be quite transparent that there was almost certainly no causality whatsoever. Doesn't mean the events which happened shouldn't have happened. Chance just has things happen at the same time. It is only us who subsequently investigate for causality when chance alone seems too unreliable a guide for why something observed was observed.

Given we know those two events happened, we then have to determine if there are consequences of them being "linked" events. Does the diary show any evidence of having been under the floorboards? Was there dust, or dirt, or whatever, found on the diary that can be traced to that storage area? Was that area damp, had it ever flooded, etc, and if so, does the diary show corresponding water damage? Were there insects under the floorboards? Does the diary show signs of similar insects? Does the electrician have a known history for stealing items from job sites? If not, what's the probability he would suddenly engage in theft this time? How long do people who do come across artifacts normally take to decide on what to do with it (i.e. publish it, auction it, TV rights, movie rights? etc), allowing us to estimate the probability of Mike deciding to contact a book publisher on the first day (so within hours)? Is that probable? I doubt it, but I don't know because that would require doing research into similar things.

You're at Stage 2 now Jeff. No argument with that, except that I asked you about the odds of being at Stage 1 which is a simple probability calculation based upon the known facts.

What I'm trying to make clear is that having two known events and a bit of creativity makes it entirely probable that a false story could be created to connect them.

For creativity, read 'Stage 2 inference'. A false story could be inferred. Oh - guess what? - a correct one could be inferred also!

What we want to know is if the story we have is just one of those false stories, or instead, a true one. And that cannot be determined by post-hoc evaluation of the probabilities of the known events - because those are always improbable - all life events are, when you calculate them, improbable, but events in life happen. These ones happened.

We agree at last! Stage 1, we can calculate the probability of known events happening by chance. Stage 2, if we are uncomfortable with the odds we get, we can move to Stage 2 and start to infer causality. And then we can move to Stage 3 when we start to investigate for causality. Et cetera.

In other words, the entire starting premise of the calculations is wrong. Worrying about the "chance probability co-occurrence" of these two events that are know to have happened will lead to nothing other than the realization of how improbable life is.

No, Jeff, predicting reasonably unlikely events in advance is improbable. Once we look back at what has definitely happened, we can calculate the probability that chance alone produced what we got. If we are uncomfortable with the odds we get, we can investigate for causality.

Deciding if the diary really did come from under the floorboards, and was handed to Mike in the pub, can only be assessed by demonstrating that story leads us to information that cannot, or should not, occur if that story is false. If it does that (leads us to evidence that should not otherwise occur I mean), then we have evidence that our story, which is only one of an infinite number of possible stories we could invent from those two known events, might be the one true one.

Okay, so - for example - we know those two events occurred on March 9, 1992, so following your advice we would seek bits of evidence that should not have occurred if chance alone were the reason they happened at the same time. So we might ask ourselves where the 'phone call was made. If it was made in Australia, it probably (but not definitely) favours chance. If the call was made from Liverpool, it probably (but not definitely) favours causality. If there is no link between the electrical team and 'Phonecall Guy, that obviously favours chance. If a member of the team drank in the same pub EIGHT MILES AWAY FROM MAYBRICK'S HOUSE as 'Phonecall Guy, that obviously favours causality.

But, remember, our 1/37,557 odds remain the same - if it was all down to chance, it was a 1/37,557 chance regardless of what causality we may infer.

But all the focus on calculating the probabilities of the floorboards coming up on that day, whether we spread the probability out equally over all 37000+ days or if we go with a more complicated model where the daily probabilities increase over time, is a complete red herring with regards to whether or not there is a connection between the diary, the floorboards, and Mike.

On the contrary, the lower the probability that those two events would happen on March 9, 1992 by chance alone, the higher the possibility that we might be able to find a causal link through investigation.

I really don't know how to make it more clear than that.

Well if you were addressing the question I asked, you wouldn't need to.

To be honest, I'm only responding because statistics are so badly misused to nefarious purposes (I'm referring to everyday life, not commenting on things here) that a lot of people have a blanket mistrust of them. Yes, when misused they can be highly abused, but when used properly, they are also highly informative; they're a good tool often badly used. But to use them properly one has to avoid "common sense", because "common sense" is what we resort to when we don't actually know how things work.

So I love the inference that they are being used badly by me!

Okay, job jobbed. Jeff, I thank you for taking the time out to post your thoughts. We obviously differ on so many points, but it would have been genuinely interesting if we could have addressed my Stage 1 probability before delving into inference.

Cheers,

Ike

[Iconoclast]

All way above my head, but I’m assuming that, counter-intuitively, 366/254 could be equally as nefarious as 37,000.

‘O’ dear!

I think you got your numbers wrong there, MrBarnett, but that doesn't matter - you make a very good point which is that if my calculations are entirely erroneous than logically so must Lord Orsam's also.

Now, we all know that RJ and Lord Orsam cannot ever ever ever ever ever be wrong so watch out for the Backtrack Dance any second now ...

[Iconoclast]

I don't think Orsam was seriously suggesting that the odds of Dodd's floorboards coinciding with Barrett calling a literary agent was 18 to 1.

And there it is!

[Iconoclast]

I just noticed that the Israeli Prime Minister Menachem Begin died on 9 March 1992, at 3:30 a.m., Tel Aviv time. The same day that Barrett called a literary agent.

He was 85, and theoretically could have died any time over the previous 28,695 days.

But this is meaningless information, unless we attach meaning to it.

Which we won't.

Of course we won't, RJ, because we have no grounds to infer a causal link between two or more events (because we only have one event - the fact he died that day). Nevertheless, the odds of him dying on March 9, 1992 (in the absence of any other known facts) is 1/28,695 (the number of days he died over the number of days he could have died). No need to infer causality between his death and something else because at this point we only know he died that day. C. S. Lewis (The Chronicles of Narnia) and Aldous Huxley (Brave New World) both died on November 22, 1963 but I don't think anyone thinks they were shot by Lee Harvey Oswald.

However ... [drum roll]

If it transpired that a bomb had gone off in Begin's office that day, one could say "What were the simple stats of that happening on the same day by chance alone?" and the answer would be (if you have calculated it correctly) 1/28,695 (assuming that he only died once in his lifetime and that only once in his lifetime did a bomb in his office go off). So if they both happened on March 9, 1992, I suspect that even Detective Hamm would put down his cwoffee and donuts and investigate to see if the two things were linked.

So Maybrick's floorboards coming up eventually are not particularly remarkable. Someone contacting a literary agent with a diary of Jack purportedly written by James Maybrick was arguably very unlikely indeed. The specific odds of these two events happening separately are irrelevant to our present discussion, however. But if both events did happen (not guaranteed, of course) the odds of the two events happening for the first time on the same day after 37,557 days had passed since Maybrick died are 1/37,557 - and those odds are more than improbable enough for Detective Hamm and the rest of us to start investigating for causality. Personally, I'd finish my cwoffee and donuts first, but maybe that's because I'm a better statistician than I am a detective?

Same with Begin and his bomb had the two also coincided on March 9, 1992, by the way.

Iconoclast of the Yard

[MrBarnett]

I think you got your numbers wrong there, MrBarnett, but that doesn't matter - you make a very good point which is that if my calculations are entirely erroneous than logically so must Lord Orsam's also.

Now, we all know that RJ and Lord Orsam cannot ever ever ever ever ever be wrong so watch out for the Backtrack Dance any second now ...

Nefarious, Ike, not merely erroneous.

I wonder whether Lord O was joking when he said there isn’t a single use of the expression ‘one off’ recorded in the 19th century. Of course, there is, so was he being jocular, erroneous or nefarious?

[JeffHamm]

...
We agree at last! Stage 1, we can calculate the probability of known events happening by chance. Stage 2, if we are uncomfortable with the odds we get, we can move to Stage 2 and start to infer causality. And then we can move to Stage 3 when we start to investigate for causality. Et cetera.
...
Ike

Yes, we agree that life events are improbable, that was one of the fundamental points I was making.

Stage 2, though, shows you didn't get the only other point I was making. No, you cannot infer causality just because life's events are improbable. You can, however, formulate a hypothesis if you wish. But that hypothesis is just one of an infinite number of possible hypotheses, so it is considered as nothing more than a guess, even though your specific life events seem improbable. Because all life events are improbable, so just because these ones are too is irrelevant and cannot be used to infer causality, even if you're uncomfortable with them. Your comfort has nothing to do with it.


If you want to investigate a hypothesis, that's a really good idea, so actually we agree on "phase 3" too. But to turn a hypothesis into anything other than one of an infinite number of possible stories one could come up with involving floor boards and calls to publishers (including ones without any causal link by the way), you have to then get actual evidence. Not more day-to-day events that seem improbable but do not directly test the hypothesis.

If the diary was under the floorboards, then in the binding, between the pages, on the cover, somewhere there would be debris that connects the diary with that space. However, with all of the tests done on it, I have never heard of anything that has suggested it was stored in such a location. Maybe I've missed it and that analysis exists, I don't know, I don't follow the diary that closely and as I say I'm well out of date. Maybe someone in the pub did see Mike and the electrician talking and exchanging something. I've never heard of such a person being located or any suggestion that has been uncovered when following up on this hypothesis. That would be a test of it though. I've not even heard if there is any evidence Mike and the electrician ever actually spoke to each other? I know they were both regulars at the same pub, and apparently both there on the same day, but I don't know if it's been shown they were there at the same time on the same day? Maybe they were, but even that doesn't prove they spoke, it would just shows they had the opportunity, not that the opportunity was taken. But if they weren't there at the same time, then that would make it impossible for them to exchange words, or physical objects. That would reduce the probability to zero, but now we're testing based upon predictions derived from the hypothesis and then going out to test them. Now probabilities come into it, and from those we can start to make inferences about the truth value of our hypothesis.

Theoretical chance probabilities of events known to have happened that generate the initial hypothesis are meaningless, and do not allow you to even lean towards causality. You are free to disagree, but that will not make you correct to do so. It is wrong to suggest you can in any way infer causality just because real life is improbable. Where probability comes in is when you make the prediction first. I predict, for example, that if the diary was not under the floorboards that the chance of finding dust/debris that is similar to that found under the floorboards is so low that if you did find it then my prediction would have to have been wrong, and since that prediction is tied to the idea of the diary not having been under the floorboards, it would weigh heavily against that hypothesis. Or, I could say "I predict that if the diary was under the floorboards the probability of finding physical traces consistent with the space under the floor is so high that if there is not such evidence then my prediction fails, and likewise my hypothesis". That's when probabilities matter. What we've been discussing up to now is smoke and mirrors.

- Jeff