Agreed! While these results are very promising, there's still a lot to explore in this space.
In addition to the "prompt consistency" and "thought-control" ideas mentioned in the post, I'm definitely curious how the performance is on more complex structured data (things like codegen).
It's too bad that reality doesn't really care if you have clever accounting tricks to show that "things aren't really as bad as they look!"
The only number that matters at all is global green house gas emissions. We need to get that to zero, and it has continued to go up.
Imagine you have a drinking problem and your doctor tells you you need to stop drinking alcohol or you will suffer liver failure. You can argue all day about what percentage of your drinks are alcoholic. "Doctor, I drink 3 cans of coke and only 12 G&Ts now, not 1 can of coke and 24 beers like I used to, I went from 4% of my drinks being non-alcoholic to 20%!" If your daily total alcohol consumed keep rising you have a problem.
That's the situation we're in with co2 emissions. There are many clever ways to make it look like everything is fine, but reality doesn't care. If our global CO2 emissions continue to rise we have absolutely no hope.
Totally right that what matters is greenhouse gases in the atmosphere. Tackling them, however, is going to take all our muscles. Many climate deniers have used the 80% of primary energy from fossil fuel figure as a way to claim it's too hard to do anything about the problem or to say there's been little progress. That's misleading and I wrote the article to explain to correct that narrative.
>That's misleading and I wrote the article to explain to correct that narrative.
There is your first mistake, you can't reason someone out of a position they didn't reason themselves in to. There are literally people denying covid is real/serious while they or their love ones drown in their fluids.
I don't think this is a useful way to think about it:
1. You don't need to convince literally everyone. There are still some people who believe the Earth is flat and probably always will be, but it is not super-relevant to collective actions because they are a tiny minority
2. People do in fact change their minds on things. It usually just happens on larger time scales so while one article isn't moving the needle, the accretion of many articles and arguments can change population-wide beliefs.
3. The population of people with an opinion on any given topic is not static. Even if any individual never changes their mind, the composition of the population can change if new people are forming opinions in one way more often. Science progresses one funeral at a time and all that....
The harsh reality is a facebook "meme" about how climate change is a hoax created by "globalist" to take away your rights is way more effective than a nuance article from the "lamestream" media trying to explain the energy efficiency of coal power plants.
If the goal is to convert science deniers or correct their disinformation this was a complete and utter failure.
It’s funny, I’m not a claimant change denier, but I don’t view emissions as a problem. An insanely large portion of the CO2 is simply us and our livestock breathing. There’s also volcanos, etc.
I’m a lot more worried about chemicals outside of CO2. Such as lead or pesticides. CO2 fluctuates massively throughout the year. Presumably the world can much more quickly (due to vegetation) remove CO2 compared to those other chemicals. CO2 is also far less damaging to the body.
Not saying it’s not important, just I think we’d be better off focusing on issues such as pesticide contamination
Others haven't already pointed out that you're mistaken regarding "an insanely large portion of the CO2 is simply us and our livestock breathing." But also, consider this:
The carbon in the CO2 that you breath out comes from the carbon that you consume. That carbon - by virtue of its state - is part of an equilibrium carbon cycle that makes our planet habitable. I.e. the apple tree consumed CO2 from the atmosphere to create the apple that you ate, and you returned back to the atmosphere the carbon that the tree previously consumed.
Now consider the carbon that has been isolated from the cycle over billions of years as the planet became habitable to creatures such as ourselves and our non-human friends. That carbon is in the form of coal, oil, etc. If we leave that stuff in the ground we retain the current carbon equilibrium. If we burn it, we change the equilibrium by taking sequestered carbon from the ground that was "locked away" and add it to the atmospheric carbon cycle.
What exactly do you think the definition of a "climate change denier" is? - It's someone that doesn't think Earth's climate is changing. It's NOT someone that doesn't think CO2/cows/etc actually has a role in it.
Climate is indeed changing. What he's trying to argue is the role of CO2 in all of it.
100% right about this:"The important part is recognizing why".
So far no raw data has been made fully available to see see what, how, when and why. At least I was not able to find the data those graphs that you keep popping up are based on. I also don't remember seeing any of these graphs taking account solar activity either since that is our heat source after all.
That is a literal interpretation of the words "climate change denier."
It's meaning in the cultural zeitgeist, however, has evolved to include people who don't deny that the climate does change over time, or even is currently changing; but, who deny the science supporting anthropogenic climate change, specifically the clear impacts to atmospheric CO2 starting during the industrial revolution and driven primarily by fossil fuel emissions.
Essentially, I think most people would agree at this point that the common usage also includes people who deny we can/should do anything about climate change, or that the current situation vis-a-vis greatly accelerated climate change is our "fault."
The records re: CO2 are not hard to find, they come from ice cores and date back ~800,000 years.
> An insanely large portion of the CO2 is simply us and our livestock breathing
Then explain why China, despite having more than four times as many people as the US, only has twice the CO2 emissions?
Can you even link a single study that puts anything but fossil fuel use as the primary contributor (by a wide margin) to atmospheric CO2? Because I couldn't in several minutes of googling.
> Presumably the world can much more quickly (due to vegetation) remove CO2 compared to those other chemicals
If that were true, climatologists wouldn't be so concerned about the problem, they'd just be telling us to plant some more trees.
I am trusting the climate scientists who produced this data because they are the people who've spent their schooling and careers studying this sort of thing and their work has been vetted by the scientific community.
If you have a better standard of judgement then by all means present contradictory data.
perhaps they are confusing CO2 for methane emitted by large fields of livestock (at the cost of their consumption of vegetation no longer available to help capture CO2) which is a non-trivial portion of greenhouse gas emission.
What if the police asked for the names of everyone who discussed using cannabis in their email exchanges? Would you really argue that in states where cannabis use was illegal that it would be perfectly okay to do this, collect just the names of people who had used mentioned cannabis in email and then visit their homes to question them further about this?
I'm pretty surprised how many people on HN seem to find this acceptable when this is clearly a form of dragnet surveillance.
For a more rigorous, scientific and yet still thoroughly digestible talk I strongly recommend watching Dr. Richard Alley's talk "The Biggest Control Knob: Carbon Dioxide in Earth's Climate History" [0]
As a Bayesian I particularly enjoy Alley's running theme that: while there certainly could be alternative explanations for what is happening, we simply cannot find anything that explains the data better than CO2.
I see many skeptics pointing out tiny holes in the main AGW hypotheses, but the real Bayesian test is "how much better does one hypothesis explain the observed data than then other?". When you put all the pieces of the atmospheric CO2 argument together it seem to explain what we're observing dramatically better than a thousand "...but what about?" that don't fit together into a coherent counter hypothesis.
As an example: Suppose I come home and see my front window broken, my door open and my laptop missing. I assume I have been robbed based on this evidence. You could say "but couldn't the window have been broken by some kid throwing rocks?", "maybe you left the door unlocked and the wind blew it open", "are you absolutely sure you didn't leave your laptop at work?"
While individually each of these counter hypotheses may explain a single event just as well, together they don't work:
P(window broke, door open, laptop missing | robbed) x P(robbed) >> P(window broke, door open, laptop missing | neighbor threw rock, left door unlocked and left laptop at work) x P(neighbor threw rock & left door unlocked & left laptop at work).
Thank you for the link. This is a fantastic lecture that presents a very accessible synthesis of geologic and atmospheric science. Starts off a bit slow, but I suggest sticking with it.
It addresses a number of the arguments presented in this thread.
I just watched it: It has a lot of interesting stuff on time scales of some millions or billions of years. Yup, in particular, there have been some huge swings in temperature and CO2 concentrations and big effects in weathering rocks, the pH of the oceans, what plants have to do differently in their breathing in response to big CO2 concentration changes, etc.
He wants to claim "The Biggest Control Knob: Carbon Dioxide in Earth's Climate History".
It does appear that he argued that CO2 is a big indicator of temperature changes. He brings in some surprising ideas that with big changes in temperature from whatever cause, CO2 turns out to be nearly necessarily an indicator. Okay.
But, I'm not seeing that his interesting stuff says that CO2 is a control.
He discusses
two big causes of big temperature changes, and neither is CO2:
His first cause is that some many millions of years ago, the sun gave the Earth about 30% less solar energy because the sun was cooler because it was burning more hydrogen, thus, converting it to helium and later got warmer because it had more helium. He does confess that we have poor data on how much cooler the sun was. But he goes on to say that it was clear that then the Earth had liquid water, and for this it needed to have the greenhouse effect of CO2 to keep the water from freezing. So, if believe that some many millions (some billions) of years ago the sun was significantly cooler, then maybe a CO2 greenhouse effect kept some of the water from freezing. Other ways to keep the water from freezing? Maybe volcanoes, radioactive decay of rocks, asteroid strikes, dust, water vapor, methane in the atmosphere, maybe more? But, okay, maybe in an extreme situation of a few billion years ago with an Earth atmosphere with a lot of CO2 (didn't yet have plants converting CO2 to O2), maybe also methane and water vapor, maybe clouds of dust, etc. the CO2 did help warm the Earth. But here the control was the cooler sun, not the CO2.
His second cause was some effects of the Earth's orbit or some such. So, that effect caused several periods of global cooling that can be seen in the ice core samples going back 800,000 years. He states clearly that CO2 did not cause that cooling, that as he mentioned the orbit did. But then he goes on to explain how with the cooling there got to be more CO2 and, then, some warming he attributes to the extra CO2. But, again, the control was the orbit, not the CO2.
His arguments are interesting, but his case is big swings in temperature and CO2 concentrations
over millions of years with lots of big effects from the sun, the Earth's orbit, volcanoes, geology (weathering of rocks), pH of the oceans, huge changes in the oceans (e.g., lots of hydrogen sulfide), etc.
He's not really talking about some significant effect of Joe's gasoline powered lawnmower on the temperature in NYC in year 2050 thus justifying carbon taxes on Joe's gasoline.
> He's not really talking about some significant effect of Joe's gasoline powered lawnmower on the temperature in NYC in year 2050 thus justifying carbon taxes on Joe's gasoline.
Of course not. First "on the temperature in NYC" is wrong. The topic is the whole Earth. Second it's not "Joe's gasoline powered lawnmower" that we worry about, but the amount of CO2 pushed by the whole Earth, all actions of all humans, measured in billions of tonnes of carbon per year! Third the scientists don't do anything about the "carbon taxes" (at least not the climate scientists) that's what the politicians (and economists) claim "has sense." The scientists just say "don't push that much CO2, the results can be very bad."
I suspect that we are mostly in agreement but are not reading English the same way. What I said about Joe, a generic guy, his lawnmower, a generic emitter of CO2, gasoline, a generic source of carbon to burn, and NYC, a generic city highly interested in CO2 and pushing carbon taxes, were all generic examples; looks like I should have had a prefix of "such as".
But my main point was that the lecture was talking about wildly different scales than what we are seeing on Earth now or likely for, say, the rest of this century. Or, the lecture was talking about really extreme, very different, conditions -- due to a cooler sun and the earth's orbit -- over 800,000 years to many millions of years. Heavily he was discussing geology, that is, a science of really slow changes, from volcanoes, huge swings in the chemistry of the air and water of earth, etc.
For your "billions of tonnes of carbon per year", I don't know what to think about that, that is, I don't know what to compare it with so don't know if that amount is, for all of the earth and sources of CO2, methane, etc. just a tiny, insignificant, nearly invisible drop in the total bucket or a major change. E.g., I have yet to see good data on how much CO2 and/or methane comes from volcanoes, on land and in the oceans, year by year. E.g., how much CO2 is dissolved in the upper, say, 2000 feet of the oceans, and if the temperature of that water is increased by, say, 1 degree C, how much of that CO2 will escape into the atmosphere? My point here is, I'm short on information to evaluate the quantities involved here, e.g., your "billions".
It was an interesting talk, e.g., with lots of really cute chemistry -- I wish my chemistry courses had had that much depth. I can see how geologists, e.g., in the AGU, could get really involved. Or, the lecture should be really popular on some Web site like HN but GN for Geology News!
But I see very little relevance in that lecture for the current intense discussions leading to carbon taxes, etc., which seem to be of high concern on HN.
E.g., if the sun gets 30% cooler, then we might want to review the lecture, if anyone is still here! Or, if the earth has an orbit situation such as in the several cases of global cooling that show in the ice cores from 800,000 or so years ago, then maybe we should rush to add all the CO2 to the atmosphere we could and otherwise risk big ice sheets in Kansas, Texas, Panama, or some such?
But these cases, along with the big asteroid hit of 65 million years ago, the huge volcanoes of the Russian traps, when India moved north, hit South Asia and pushed up the Himalayas, even when Yellowstone blew and put, IIRC, ash 1000 feet deep 1000 miles down wind, are super major biggie causes of disaster, and, of course, if a gamma ray burst popped off somewhere in our part of our galaxy, as far as I can tell, that make everything humans are doing now just an invisible, trivial nit.
From all I've seen, if we want to talk about human sources of CO2 and their contributions to significant global warming for the rest of this century, then we have a lot of really hard work to do.
Why?
(1) A lot of work was done trying to predict what the realistic CO2 levels would do to global temperatures, and now we can see that nearly all the temperature predictions from that work were way, way too high.
(2) In a sense, we know too much: E.g., we know that the fluid flows, both in the atmosphere and the oceans, satisfy the Navier-Stokes equations of fluid flow. So, first cut, we would have to solve those equations for the surface of the earth, on both land and the oceans. And we would have to include the effects of all the solar and infrared radiation involved. Then, even to get started, we would need the current initial conditions, e.g., what the volcanoes were doing, and we are a long way from knowing those. Then, we would be very short on a suitable computer.
Sure, we'd like to take an approach easier than (2), but as in (1) that easier approach was nearly all wildly inaccurate. With (2), we are stuck, or stuck-o.
But the lecture had some amazing geology and chemistry!
> A lot of work was done trying to predict what the realistic CO2 levels would do to global temperatures, and now we can see that nearly all the temperature predictions from that work were way, way too high.
You're wrong. The predictions are very good, and your claims are based on... some wrong sources obviously.
The rest of your post is also wrong. Specifically:
> I don't know what to compare it with so don't know if that amount is, for all of the earth and sources of CO2, methane, etc. just a tiny, insignificant, nearly invisible drop in the total bucket or a major change.
It's irrelevant that you don't know. Your "ignorance" won't make global warming disappear. Ditto for your claiming that the scientific methods aren't to your taste. No "denier" was able to make anything scientifically relevant up to now. That's the fact.
that for dozens of climate models plots the predicted values of global temperature along with the observed values. Nearly all the predicted values are wildly too high. So, if we believe the graph, then in science we essentially junk nearly all the models.
As I tried to outline, the model builders were on a long walk on a short pier: Nearly all their modeling efforts, with whatever approximations, short cuts, etc. they took, failed.
Then, really, to return to first principles, they had to solve the Navier-Stokes equations for the surface of the earth, the oceans and the atmosphere. Doing a good job with that would be essentially minute by minute weather prediction, and as we know we can't do that accurately for more than two weeks or so.
That's what the model builders ran into -- the Navier-Stokes equations. IIRC we've had some good progress with those equations for, say, some boat hulls and some aircraft wings. The whole earth? Not a chance.
> It's irrelevant that you don't know.
Sure it is relevant if accept a little diplomatic circumlocution: I've tried to follow some of the global warming issue for years, and so far I have not seen the data. Without the circumlocution and diplomacy, the more literal and blunt remark is that nearly all of us are short on such data either because it has not been collected very well or the people writing the main arguments don't include it. Or, the situation is like I was a customer at a fast food restaurant, got a hamburger, opened it up, and asked "I don't know where the beef is".
> No "denier" was able to make anything scientifically relevant up to now. That's the fact.
Well, in the graph in the link I gave above, a few of the models, maybe only one, were fairly accurate -- they predicted nearly no increase in global temperatures. So, maybe the author of that model was a "denier" in which case at least one "denier" did something "scientifically relevant".
For more on "scientifically relevant" from deniers, there is the argument that at least for the past few thousand years, we know the cause of the temperature changes, and CO2 had nothing to do with the changes. Really, basically the greenhouse effect -- from CO2, methane, water vapor, etc. -- didn't either.
Instead the cause of global cooling was more clouds -- they have a net cooling effect (net, not a greenhouse effect). The clouds were from some water molecules in the atmosphere collected into water droplets from the action of cosmic rays. Of course, the cosmic rays come at essentially constant rates over time but maybe with some variation with direction. So, when there was fewer cosmic rays, there were fewer clouds and some global warming. Why fewer cosmic rays? Well, when the sun has sun spots, there is more solar wind -- think really 3D all around our solar system, not just in the plane of the orbit of the earth, and reducing the rate of cosmic rays from all directions including way out and high above the plane of the earth's orbit.
So, when the sun has more sun spots, there is more solar wind, more cosmic rays blocked from hitting the earth, fewer clouds, and global warming. When the sun is quieter, there is some global cooling.
IIRC, over the past few thousand years or so, and maybe longer, the data on variations in sun spots do well fitting the variations in global temperatures while data on CO2 concentrations does not.
So, people with the sun spot explanation are "deniers" and may have found the real cause of global warming/cooling for the last few thousand years or so at least and, thus, been "scientifically relevant".
More generally, science needs to be subject to review, and a reviewer, even a "denier", who does find actual errors is also "scientifically relevant".
Along the lines of reviewing, I would note that so far it appears that over the past 800,000 years, the cases of global cooling were not preceded by lower concentrations of CO2. E.g., the Little Ice Age was not preceded by lower concentrations of CO2. Neither was the cooling from 1940 to 1970.
For more, yes, sure, CO2 is a greenhouse gas. Fine. We agree on that. So, CO2 can warm the earth -- we agree. But we can also agree that lighting a match will warm the earth. So, for CO2 warming the earth, the question is how much? Hate to say: (A) Just looking at the historical record from the past few thousand years, maybe the past 800,000 years, it appears that more CO2 has no detectable warming effect. (B) For some solid science, e.g., evaluating the effect of CO2 just from first principles of mathematical physics, including the Navier-Stokes equations, we can't do the calculations and, thus, for this approach, are stuck-o. Here maybe (A) is a review and "scientifically relevant". For (B), that's also "scientifically relevant" but so obvious that it is a trivial contribution.
There are a lot of research problems without good solutions -- some of the details of the big bang, dark energy, dark matter, the origin of the highest energy cosmic rays, the origin of life on earth, quantum mechanics and relativity inside a black hole, a cure for cancer, how good natural intelligence works, etc. Well, one more is prediction of the temperature of earth for the next 100 years. Sorry 'bout that. While I've solved some problems in applied math and published the results, I can't apologize for not solving all outstanding problems. Sorry I don't know how to predict the temperature of earth for the next 100 years, but maybe I should say that I can't see that anyone else knows how either. In that case, my view is that carbon taxes are not justified.
> Sorry I don't know how to predict the temperature of earth for the next 100 years, but maybe I should say that I can't see that anyone else knows how either. In that case, my view is that carbon taxes are not justified.
That summarizes how you see the issue. If the scientists can't convince you that they can "predict" the values exactly in the shortest time period ("100 years") the "carbon taxes are not justified."
Everything else is trying to rationalize that, taking the info from these who cherry-pick their argument of the day that nothing important is going on.
In reality the global warming issue is not about meteorologically "predicting" "100 years" at all. It's about recognizing the impact, which will happen no matter how much cherry-picking origin-adjusting short-term charts could be produced.
Yes, "the models" are not giving meteorological predictions. I don't know how old are you, but if you're going to live even just the next 20 years, you'll see for yourself that in the 20 years even if "the models" would wiggle between, the Earth will be much warmer on average, just like 2016 broke all the records since we measure. It will happen again and again. And if you plan to have children or have them already, please write one single letter to them with your "predictions" of how they are going experience Earth warming. You probably won't live enough to see that, but they will be able to see how much you were wrong, that's I'm sure.
First, the scientific evidence for your claims is in one word, JUNK. For proof of the junk status, their predictions have already failed badly.
Again, once again, over again, yet again, one more time, this time just for you, in science, when predictions are proven badly wrong, then we junk the science. That's much of why we do look to science for solid answers.
As in the graphs in the link I gave, nearly all the models have been proven badly wrong; just click on the link and look at the picture; download the picture and use a graphics program to zoom in on the details.
So, due to the bad predictions, we junk the models.
Now we are very short on scientific evidence.
The long dark ages of the ascent of man show that non-scientific evidence is ugly, disgusting, dangerous stuff -- snake oil, witches' brews, toxic swill, leach bleeding, burning girls alive at the stake, horrible human suffering, e.g., get the Black Death, kill about a third of the population and have dead bodies piled up in the streets from the bacteria from the fleas from the rats because some idiots killed off the cats that were eating the rats.
Second, you are concentrating on CO2, and there is next to no solid evidence that CO2 is having any effect now on the temperature of the earth at all. For evidence, over at least the last few thousand years, higher temperatures were not preceded by higher CO2 concentrations (except for pulling out of the Little Ice Age clearly caused by fewer sun spots so that the pulling out was likely caused by normal sun spot activity and not the CO2 from the industrial revolution -- also note the cooling of 1940 to 1970 when if anything CO2 concentrations were higher)
and lower temperatures were not preceded by lower CO2 concentrations.
Third, some of the best evidence is that the temperature variations in the earth over the past few thousand years at least up to the present are due to variations in sun spot activity with CO2 just irrelevant. The sun spot data actually fits the temperature data, and the CO2 data very much does not. A biggie is the Little Ice Age -- lower sun spot activity but not lower CO2.
Your claims that it's so hot now are from data cooked by NOAA and NASA due to what Obama and the global warming alarmists wanted. Or, really, how much warmer is it, since when, in degrees C, measured how, by whom, when, and published where? At best you are talking changes too small to be at all sure about. More likely you are being fooled by deliberately fraudulent data.
For a little more, as of 2006, the US National Academy of Sciences report, right, still at
quite carefully showed (A) that the temperature of the earth in 2006 was essentially the same as in year 1000 before the Little Ice Age, and the increase in temperature in the 100 years before year 2006 was very much like the increase in temperature in the 100 years before year 1000. Just see the graph early in the report. The time series statistics were from David Brillinger at Berkeley, a student of John Tukey at Princeton and Bell Labs, and Brillinger is a good candidate for the best time series guy on the planet. It's a serious report. Clearly the increase in temperature before year 1000 was not caused by an increase in human emissions of CO2. So, CO2 is not the only cause of temperature increases or changes. So there must be other causes. So, even if you do all you want against CO2, the other causes might remain and, really, make your CO2 efforts a total waste.
Or, again, yet again, once again, over again, one more time, this time maybe you will pay attention: The Little Ice Age was real, e.g., was the cause of the ice on the Delaware River in the famous painting of Washington crossing that river and of the extreme cold when Napoleon left Moscow. Well, the Little Ice Age certainly was not caused by lower CO2 concentrations. So, why, oh why do you continue to believe that changes in CO2 concentrations are the only cause, or now even a significant cause at all, of changes in global temperature? Nothing could be more clear: (A) At the start of the Little Ice Age, temperature went down, but lower CO2 was not the cause. (B) CO2 changes are not the only cause of temperature changes. You are fully able to see this.
Warmer? Heck there was some significant warming: In the Middle Ages, they were growing grapes in England! That's warmer than now! And that warming was not from higher CO2 concentrations.
And, for more, there was nothing very unusual about the temperature in year 2006. And since year 2006, the change in temperature has been too small to be at all sure the change was not actually zero -- there's been no significant change since year 2006. Net, we're not warmer than in year 1000 when CO2 was irrelevant, and, again, it is not clear that CO2 now is having any effects at all.
Due to sun spots or whatever, it really might be warmer in 20 years as you fear. No one really knows. The IPCC doesn't know; I don't know; no one knows what the sun spots might do; no one knows what the effect of realistic amounts of CO2 might be; there is no good evidence; the evidence that CO2 is a big danger is junk science seriously debunked by the bad predictions I referenced; and you don't know.
You are back to the thinking of the Mayans who killed people to pour their blood on a rock to keep the sun moving across the sky. You want stop humans from emitting CO2 to keep from warming the planet, and your evidence is, net, no better than that of the Mayans. Your solution is similar: Sacrifice.
Or, you have in mind a classic trilogy: (A) Human transgression, sin, this time against Mother Earth, (B) retribution from an angry god, Mother Earth, with global warming that hurts our children, (C) redemption for our sins from sacrifice, like the Mayans did, that causes us to suffer, e.g., shut down much of our economy. You are going for the attractions of an old pagan religion. You are being manipulated by people who want money from subsidies, etc. No thanks. Wise up.
We don't know what the climate will do. We can't cure all cancers, either. Relax. For CO2 and the climate, f'get about them. Do something else.
And, no carbon taxes. No higher electric bills. No attacking our cars, trucks, and airplanes. No shooting our economy in the gut. No thanks.
What you call "the graphs in the link I gave" is not the most recent graph from the (denialist) source that you'd like to use, it's from 2013, and since then we have had the warmest year ever recorded (2016). Here's the discussion of the more recent graph from your sources, which your sources used in their 2016 testimony (and that graph still doesn't include the results of 2016 and it's still "cooked"). Note that the baseline was "cooked". Here the discussion:
To be precise, in the graphs, the prediction of the models is compared to the satellite data which don't measure surface temperatures and which are confirmed to have slower warming trend at the moment. Are the measured data below the range of the models in some years? Sure. Will the measurements move in the direction of the model once 2016 data are there? Yes. Does the surface temperature data (which you avoided by picking only the satellite ones) match the models significantly better? Yes, and that isn't on these graphs at all. Here's how it looks when we use all the data we have and compare with the models:
In short, what you believe to see on that graph is not what most of the scientists see (I'm not counting among the "most" John Christy and Roy Spencer, who produce or promote these cooked versions and ignore all the evidence they don't like). The scientists see that the measured satellite data did go below the lines of what the model would have expected, but they don't consider that the models are invalidated with these measurements, and even less so once 2016 is included.
So, by supporting what you support, you're still destroying the future of your children and grandchildren, if you ever plan to have them, and you obviously do it only for the very egoistic goals of not even trying to make any change because from your perspective you're doing fine. (your words: "no carbon taxes. No higher electric bills. No attacking our cars, trucks, and airplanes. No shooting our economy in the gut. No thanks.")
Like I've said, it will be obvious in only 20 years, but then it will be even harder to do anything -- and the effects will last for hundreds of years. And if you write the letter to your children and grandchildren (or the kids of the relatives if you don't plan to have children) and they survive to read it, they will be able to see who and how influenced their future, so please do that. If you're right, they will celebrate you as a hero. If you aren't right, and unless you are already planning to die sooner than in 20 years, you'll see it for yourself, then I'm sure they'll use some other words for you, reading your message.
It may be that we are at significant risk of a big disaster.
So, we want to be correct.
For one, we can conclude that there is little or no danger, that we need do nothing, do nothing, be wrong, and have a big disaster.
For another, we
can put on big time carbon taxes, cobber the economy for ourselves and our children, but again might be wrong, that is, have clobbered the economy for no good reason.
So we want a good answer.
Finally, just applying my judgment and what I can smell with my nose, I come down on the side that the crucial part of this issue is not science but politics and money.
E.g., today it did finally sink in to me that, with all the different model results in the graph I linked to, at least one of the results had to be close to reality and, thus, an accurate prediction. Luck would have it that the model with the most accurate prediction was the one predicting essentially no change -- whew, that was a close one! A lot of people could have been fooled with that one!
It's fascinating how you ignore all the measurements that don't match your, as you say, "political" view, if you followed the links, that is. Because you can't say then you didn't understand how much are of them disproving your claims, and what they mean, and still claim you earned a Ph.D. Do you really think it's the whole world conspiracy? Because the "political" claim has no sense unless you believe that world consists only of the US of A, it's most of the scientists all around the world. If you simply say that the short-term politics is more important for you than the future of the world, I can at least understand.
Anyway, "the greatest shortcoming of the human race is our inability to understand the exponential function":
I had a look at the Navier-Stokes stuff and what I found was incredible complicated maths. I'd like it if you'd linked out to some discussions on this topic but in any case I was able to find this [0] which takes the position that many climate scientists have invoked Navier-Stokes as a theoretical basis but mathematicians in general consider the finer details of NS behaviour at this scale to be (practically) unknowable.
My personal insight from an information science pov is that NS would be the wrong tool to use. It looks like an exhaustive model used at a fine-grained scale that would be simply too complex at macro level.
I followed your link on the Navier-Stokes equations. Nice.
They mentioned C. Fefferman: I'd forgotten that he wrote the problem description for the Navier-Stokes equations for Clay. Fefferman is one of the most respected mathematicians going.
When I was applying to grad school for applied math, one place I applied was the Division of Applied Math at Brown University. I had a business trip to the small jet engine shop of GE in Lynn, MA, so went down to Brown to meet some profs. Had lunch. Talked. I got one piece of advice: "Stay away from the Navier-Stokes equations."
Yup, good advice.
The Navier-Stokes equations are old stuff now. Basically they are just Newton's second law, the gas law, etc. -- just basic physics. If you want to calculate the flows of fluids -- liquids and/or gasses -- from first principles of the basic physics, then you are stuck with the Navier-Stokes equations. It's like calculating the trajectory of a home run baseball -- need Newton's second law, the law of gravity, and at least something first cut on air resistance (right, in fine detail, the air resistance would again be the Navier-Stokes equations but good enough for a baseball or artillery shell, there are some good enough, simple approximations).
Or for calculating what fluids do, from the basic physics, that is, from what we DO know about the basic physics, we just can't avoid the Navier-Stokes equations anymore than for the baseball we can avoid Newton's second law force = mass times acceleration. Yup, this is a case were we know too much: We DO know that, for calculating from a solid basis, that is, from first principles, for the math we need, we DO have the equations and they are just the Navier-Stokes equations. Sorry 'bout that.
You mentioned that you noticed that the Navier-Stokes equations are complicated math -- right!
As I mentioned, maybe we can do well with the Navier-Stokes equations for some boat hull or some airplane wing. Okay. Early in my career, I got started on the Navier-Stokes equations at the US Naval Ship R&D Center at Carderock, MD -- right, with the big towing tank used for designing ship hulls. That was before the guy at Brown told me "stay away from the" equations. He was right!
So, you are correct: Trying to solve the Navier-Stokes equations for all the oceans and all the atmosphere of the earth is a wild thigh slapper, absurd, out of the question. We don't have such a computer. Even if we did, we don't have even the required initial conditions -- that is, the current state of the flows in the oceans and the atmosphere. Or, we are stuck-o.
We will also want to know that the Navier-Stokes equations are nicely stable, that is, that a butterfly flapping its wings in NY will not cause rain instead of sunshine in Japan (to borrow from various movies). IIRC, Richard Bellman wrote his Princeton dissertation on the stability of ordinary differential equations -- IIRC the stability of the Navier-Stokes equations is a challenging topic, e.g., appears to be part of one of the Clay Math problems along with P versus NP, etc.
And the real problem is still worse: E.g., likely we would have to handle turbulence, known to be difficult. When I was at Carderock, there was a guy working on turblence; he had been for years; maybe he is still there still working on turbulence; maybe in another 100 years he will have some good progress! In principle, we are talking about handling winds blowing through trees (would need the details on all the leaves of all the trees!!!) and the resulting turbulence. Would need some good details on associated biology. Would need .... And after have all of that, as the climate started to change, we would need good details on how the biology would change, and we don't have any equations from first principles for that.
So, why do we need to do the fluid flow calculations? Well, we're talking about CO2. For that, we want to know where it goes, e.g., into the water, out of the water, into/out of seashells into the upper atmosphere, close to the ground, as it warms, as in the greenhouse effect, where it goes, sucked up by the plants, reacts with rocks, etc.
So, what people have done is use various assumptions, simplifications, and approximations. It's a little like in freshman physics where we assume a block slides down a plane, and the plane has no friction, or a ball rolls down a plane and we ignore the moment of inertia of the ball.
So, people tried such approximations, etc. We DO know the basic physics, and a big part of that is the Navier-Stokes equations. And we have more physics on the black body radiation that is the source of the infrared radiation that is the source of the warming of the CO2 that is the warming of the greenhouse effect. We have a lot of the basic physics and chemistry. And that basic science just does not tell us that there are some nice, easy approximations that will let us predict the climate.
E.g., suppose some day during the years we are predicting, it rains. It might! Then after the rains, where is the CO2? Do we have partially carbonated rain water? In principle, we will want to know where the CO2 goes. So, part of our calculation will be to predict when it rains. Hmm ....
Yes, as is often the case in physics, for some purposes we can just use the law of conservation of energy, calculate energy into the earth from the sun and energy out of the earth from radiation to space and get the balance and temperature change. Okay. But the details, if we want them, of the energy in and energy out will take us back to the Navier-Stokes equations. So, maybe we can make some simplifying assumptions. Apparently then ... we come up with the models that predicted much higher temperatures by now.
Then, in all of this, we have an assumption that is now looking like week old dead fish: It's all about the greenhouse effect and CO2 and not something else. What "something else"?
Apparently an argument can be made that, really, at anything like currently realistic levels of CO2, CO2 and the greenhouse effect are essentially irrelevant and the main cause is just clouds from water droplets from cosmic rays blocked or not by the solar wind from sun spots. In that case, we can calculate all we want with the Navier-Stokes equations, make more bad predictions, and accumulate some big computer bills. So, for accurate predictions, we'd be into predicting the sun spot activity of the sun. Hmm .... Is that at all promising?
The lecture on geology and CO2 was really interesting: It got into the orbit of the earth, the power (energy per unit time) to the earth from solar radiation, some geology, and some tricky chemistry but not sun spots!
Net, so far, it's tough to predict either the weather or the climate. Sorry 'bout that! It's also tough to cure cancer, explain dark matter and dark energy, ..., etc.
"IAU, international astronomical organisation that brings together more than 10 000 professional astronomers from almost 100 countries:
The results, presented at the IAU XXIX General Assembly in Honolulu, Hawai`i, today, make it difficult to explain the observed changes in the climate that started in the 18th century and extended through the industrial revolution to the 20th century as being significantly influenced by natural solar trends.
The sunspot number is the only direct record of the evolution of the solar cycle over multiple centuries and is the longest scientific experiment still ongoing.
The apparent upward trend of solar activity between the 18th century and the late 20th century has now been identified as a major calibration error in the Group Sunspot Number. Now that this error has been corrected, solar activity appears to have remained relatively stable since the 1700s [3]."
So, there were measurements of larger, easier to see sun spots, all sun spots including small ones, and sun spot clusters. There have been some old records and also some data from some tricky chemistry on earth in some rocks or some such of effects of sun spots, data that goes way back, maybe thousands of years, maybe more. So, someone applied Kelley's Variable Constant and Fink's Finagle Factor with their thumb on the scale and corrected all the data and got, presto, bingo, wonder of wonders, a grant for CO2 research from the old Obama Administration?????
Even taking that article at face value, for discussing global warming, CO2, and sun spots, the article
doesn't look nearly as relevant as we would want. Actually it looks like it is knocking down arguments I was not making, picking arguments it could knock down, setting up straw men just to knock them down, and not very directly addressing global warming.
E.g., the article knocks down an argument about a long term, increasing "trend" of sun spots. Okay. I was never aware that anyone claimed that there was such a "trend".
E.g.,
apparently there have been claims of a recent "maximum" of sun spots, and the article claimed to knock down that claim, also. Gee, the article was the first I'd heard of any such "maximum" -- I was not arguing for such a "maximum".
It remains that the Little Ice Age was darned cold, and it was in force when Washington crossed the Delaware and Napoleon returned from Moscow. IIRC from the last time I looked up The Little Ice Age on Wikipedia, the LIttle Ice Age lasted much longer than the link's relatively short interval for the Maunder Minimum.
Q. 1. So, what the heck caused the fall in temperature at the beginning of the Little Ice Age?
A. 1. Apparently no one is arguing, or has data on, lower CO2 concentrations as the cause. So, CO2 is not the only cause of global cooling.
Then, sure, cancel that cause, i.e., suppose whatever it was it goes away, and we should see some global warming without considering CO2 unless there was lower CO2 at the beginning of the cooling.
Q. 2. What got us out of the Little Ice Age?
A. 2. Maybe CO2 from the Industrial Revolution? Or maybe the earth just returned to what it was doing before it got cold, whatever the cause of that warmer globe was, and not CO2.
that's really sad stuff; I'm sorry to see the UCS push that stuff: So, they argue that to the best they know how, just "natural" can't explain the temperature variations. Then in their models, they put in what they believe would be the effects of CO2 and, presto, bingo, wonder of wonders, and I can believe after some appropiate debugging and grant from the Obama Administration, the model fits old history. That's weaker than over cooked pasta.
Didn't one of those links mention how warm it is now? I don't believe it is especially warm now: From what I've seen, there's been no significant increase in temperature for the past 16 or so years. So, in particular the temperature is essentially the same as in year 2006 when the NAS report I referenced claimed that the temperature in 2006 was essentially the same as in year 1000 before any influence from human sources of CO2.
I can't be very sure about the sun spot explanation, but to me it looks much better than the CO2 explanation.
Net, I don't see work that lets us predict the temperature 100 years from now other than just guess "no change".
Also, in just simple terms from lots of just simple temperature evidence, I see no reason for alarm.
Indeed in your circumlocutive manner you have described a task akin to building a sand castle one grain of sand at a time. You're getting into the territory of pascals demon. If you could produce such a sophisticated model why would you even bother with the weather since you'd be able to produce so much more than than that!
What you describe sounds quixotic. Tilting at windmills. I would be very disappointed indeed to discover that climate science floundered in the face of trying to do the impossible.
The history of innovation is littered with stories of impasse quickly followed by the discovery of a short cut and then progress. It's all about short cuts. When I play chess I don't evaluate all possible moves - when I play poker I don't count the cards. Yet - I can play both quite effectively. How is this? Because dispite my incapability to grasp all the variables I can develop quite sophisticated models with effective predictive power.
Nobody expects the sciences to be exhaustive down to the atomic level. Not all sciences are physics. All that is required is a framework into which all extant knowledge can be integrated; methodology for gathering new data and a means to extrapolate predictions.
As a software engineer I'm never expected to write x86 assembly language to implement a functioning web site. All I need to do is read a book on HTML look at a few examples online and then off I go. I can probably do a better web site than the guy writing the ASM code for a living.
That is what using your NS equations is like for computing your boat hull. Far better off reviewing other people's accounts of what works and putting them together - testing - and seeing if it works. Yes there are a great number of edge cases you may never be able to account for but there are only so many hours in a day and you need to produce something that works for the case in point - not all possible situations.
And so it goes with climate science - you can't possibly predict everything to this minutist detail - you can only observe and measure, observe and measure, catalogue, and integrate and continue as infinitum continuously refining your models. Some stuff won't integrate like other stuff but eventually you'll get to a point where you start to have some predictable capacity. By definition this approach can never be "complete" but as we discuss "completeness" isn't possible.
But what we do have from our catalogued observations and measurements is a huuuuge body of knowledge about what "actually does happen". Nothing is ever that wildly out of the blue that our existing knowledge and modelling can't have some premonition of it. There are exceptions of course but like we say without doing the impossible it isn't possible to foresee these ...
But what we can do is, to within a certain degree of certainty make predictions about what is likely to happen. Like if you're swimming in a lake in Africa or something and you see a shimmering movement on the water do you wait to compute that it's just an agglomeration of leaves? Do you wait to see if it moves with the characteristic vortices of a crocodile or so you just GET THE FUCK OUT OF THE WATER straight away?
I know which is be doing. Even if it means all my friends laughing at me because sure I'll feel a little bit embarrassed but in the alternative reality where it was a crocodile but I gave up trying to exhaustively model what it was I have died a horrible and brutal death.
So as much as I appreciate your educational pedigree and thank you for sharing your valuable time with me I do remain sceptical of your scepticism. Though you have conclusively demonstrated what we cannot know you have said little to explain what we do and our corroborating experiences heretofore. Hope this doesn't come across as too much of a circumlocutive gish gallop I've been trying to type this on my phone on the train without recourse to any notes or any editing capabilities. Sorry 'bout that ;-)
Your view of a lot of the methodology in science, and more so in engineering, is correct or nearly so, and I agree with it nearly so.
BUT!!!! So far in saying what human sources of CO2 will do to the temperature of the earth, as in the graph at the link I gave, we're in deep, fuming, reeking, brown sticky stuff -- the predictions from nearly all the models were badly wrong.
So, one approach to something better is to return to first principles of physics. History shows that that approach has been known to work. And IIRC by now the Navier-Stokes equations have been quite good for boat hulls and airplane wings. But as I outlined and you noticed, using first principles to say what the temperature of earth will be in 100 years is too difficult, absurdly too difficult, even if ignore turbulence around leaves, too darned difficult -- and it might even be unstable. So, that approach won't work, either.
So, as you outlined, we need to find a way that does work. In such a case we will need to check it, test it, validate it, etc. And at this point, from the graphs, no one but 10 year old sweet, cute, darling, adorable but gullible and naive Virginia will believe any testing based on predicting old data and, instead, will want a prediction of the NEXT 20 years and the WAIT 20 years to take the work seriously.
Soooo, right, we fall back to some back of the envelope approaches.
Here's one such I've already explained here:
(1) For CO2, for nearly everything in the record for the past 800,000 years
(the geologic time intervals in the lecture seem to be different) shows that temperature went up without preceding CO2 concentrations going up and temperature went down without preceding CO2 concentrations going down. So, net, CO2 is not nearly the only cause of temperature changes. And, since we are now willing to do first cut approximations, we conclude in practical terms that, under anything like current conditions, CO2 is irrelevant. Bluntly, the variations in CO2 concentrations don't match the variations in temperature worth a darn.
(2) The stuff I outlined about sun spots fit the temperature data much better. The example that sticks out like sore thumb is the Little Ice Age -- that is, the Maunder Minimum after guy Maunder who noticed that there were nearly no sun spots during the Little Ice Age.
Otherwise, so far we don't know even as much as dip squat about the change in temperature, from CO2 or anything else, in the next 100 years. Clearly the first cut, safe bet is that there won't be any change.
And, it helps a lot that currently the temperature and the changes in temperature don't look at all unusual. I.e., it was hotter in the Medieval times when people were growing grapes in England.
Or, worry about if going to get cancer and f'get about CO2. Sure as heck f'get about carbon taxes.
The problem I'm having, and I have a masters in statistics, is that this data is generated after the fact. If you compare the predictions of the 2000-2010 period with the "after-the-fact" predictions of the 2000-2010 period ... There's a huge difference. According to the prediction EVERY year between 2000 and 2010 was between a 3 and 5-sigma event. In the after-the-fact model this is simply ignored, and the data follows the graph much better and ... nothing special happened apparently.
Sorry no. This is an indication that your predictions are worthless. And of course, after such a massive fuckup, it's going to take a bit of work to convince people otherwise. But none of that is happening, I wonder why not.
The worst of it all is that I used to make a similar argument, first about 1990-2000, then about 1995-2005. They all had the same result. Suddenly, and without explanation they were far, far cooler (compared to the prediction). Not quite 5-sigma years, but 3 sigma years were in there. 2000-2010 is the worst of the lot, but there isn't any decade-long period where one can say ... well climate scientists ... well done. You were right. The IPCC AR's, predicted temperature anomaly versus observed temperature anomaly.
Given these circumstances, how can we give any credence to the idea that the IPCC predictions about the future will match reality ? The joke is that in order for their 95% prediction to actually be true in 95% of cases they now have to be correct for 20.000 years. They don't even predict out that far. It's a joke, but it's also true. You'd think it would give people pause.
And all this is still ignoring the many fundamental theoretical problems climate science has. When they say "it's worse than ever before" ... well they're right, in terms of many variables. Ok ... but that means their methods are invalid. Statistics can fill in expected values in an observed population of values. And yet their claim is that the observed populations are VERY different from the ones seen before. Ok, fine claim. You can even prove that to some extent with statistics. And they've done that. After that, of course, you cannot use statistics to predict these values anymore ! This alone could explain why their predictions are so horribly bad.
I get the utilitarian argument. We don't have any better way of making predictions about the climate. That's great, but that doesn't make our best prediction adequate, nor is it even value-free. In some ways, it's just a political argument : "this looks worrying, let's fix it". It should be treated as such.
You claim to have a "masters in statistics" but that doesn't really help me in understanding the main points of your argument. In a nutshell what you are saying is that a lot of the statistical models developed have changed over the last 20 years or so to accommodate the data that has actually emerged. That each of these models if they were applied today would diverge significantly in their predictions from the actual situation we're seeing today? I find this a stunning indictment of mainstream scientific consensus, especially considering many of the actors in that field probably have Masters in statistics and other fields, PhDs and professorships.
But, we do see this kind of skew in other fields, in particular economics comes to mind, where an overarching consensus on what is and is not good seems to promote the viewpoints selected for discussion, while neglecting counter-theses.
> I find this a stunning indictment of mainstream scientific consensus
How exactly ? Climate science is thoroughly in the humanities at this point, very much not in exact science, and this is hardly the only wrong-but-we-can't-do-better consensus that exists. People believe this and it provides them with things to do, with the "ability to predict". Except, it's not correct use of math, with the predictable result that the outcome doesn't match reality. We "believe" a lot of things for political reasons. For instance, starting in 1995 or so papers suddenly started showing that all races are equal in everything, from iq to running speed. It's becoming controversial to state that Africans need supplemental UV at high latitudes, despite the medical consequences of not doing so. Looking at differences in their performance is systematically removed from all textbooks. Now look at reality. You have watched the olympic games recently ? I assure you the impression you get from those races is more or less correct.
As for this specific one, you can simply look this up. That's supposed to be the beauty of science, except, of course where there's "scientific consensus", whatever that means. And you know, worst case is you learn something.
For the lazy, google it (this looks reasonable): http://clivebest.com/blog/?p=2208 (but obviously download the IPCC AR0 and check the data in a table as well)
Obviously there is scientific consensus in the statistics community too that you can't do what climate models do without destroying your credibility, but coming out and saying that ... isn't happening. And sadly, I know why.
Even if you disregard results, you know how the IPCC arrives at their 95% certainty interval ? They take 20 highly non-linear climate models, assume normal distribution of the results per-year (think about that for 5 seconds), and come up with confidence intervals. To claim this won't provide accurate results ... does that really require data ? At what point can you just call bullshit without further study ? One statistics paper humorously provided a proof that with this process, you can create any outcome you want.
And for the conspiracy nuts : the most recent IPCC AR leaves out actual numbers entirely. Because apparently (yes really) they're too easy to criticize.
> They take 20 highly non-linear climate models, assume normal distribution of the results per-year (think about that for 5 seconds), and come up with confidence intervals.
Like I say, "stunning". Thank you for following up - I'll be sure to follow up your links etc. at the next possible opportunity.
I'm still wedded to the notion that climate change, global warming etc. "is" real, and caused by human activity, but I'm very much open to scepticism about how science is deployed day to day.
EDIT
I don't think it's fair to describe climate science as being "in the humanities" just because it can't have the same precision as physics. It is still concerned with the empirical gathering and analysis of information. Perhaps it is going beyond its capabilities, similarly to economics (hence my comparison there) but like economics it does still does possess a great degree of predictive power and pedagogical value.
At a high level, compilers just translate one programming language to another. A key part of this translation is the Abstract Syntax Tree (AST), which represents the programming language transformed into a tree of computation independent of the syntax of the language. Once you have the AST, you can then step through it and translate the tree to anther language like java bytecode, ASM, CIL, etc.
When compiling the AST sits in the middle of the whole process. The first part deals with parsing your language into the AST the second part deals with transforming the AST to the target language (including possible optimizations of the tree).
Lisp is effectively the raw AST, which is where its power comes from. The use of parentheses is the cleanest way to directly represent a raw tree that you can interact with. This means that you can use Lisp to cut the language design process in half from either direction:
On the one hand, if you are worried about parsing your language, then you can simply transform it into lisp, which is virtually identical to creating the AST, and then you can use a lisp interpreter/compiler for the second half.
On the other hand, if you're interested in writing the interpreter/compiler for a language and don't want to stress about parsing, you can write it for lisp and not have to worry about parsing a complex language. If you follow the Norvig code you can extend that example to a language devoid of parenthesis by writing the parser for it.
Even if you want to do both it's not a terrible idea to prototype both halves using Lisp and then perform the minimal work pull out the Lisp code and replace it with some other implementation of the AST.
> Given the current state of machine learning, could an AST be generated to accommodate a given language?
No machine learning required. It is certainly possible to automatically generate an AST based on a grammar.
The more interesting question is whether you can automatically generate a grammar based on samples of the language - https://en.wikipedia.org/wiki/Grammar_induction - although it is unclear why you'd bother trying for a programming language (as opposed to natural language).
> And more importantly, can the form of an AST cause a difference in performance of the overall program?
If you are compiling to machine code, or to bytecode for some VM, or transpiling to another source language (e.g. JavaScript): not really. The form of an AST could make a difference to compilation speed or the ease of implementing later stages of the compiler, but I can't see how it would directly make any difference to the performance of the compiled program.
However, if you are doing tree-based interpretation: Yes, differences in choice of AST can make a significant difference to runtime performance.
Part II of Skiena's wonderful "The Algorithm Design Manual" is basically a 300 page, high-level overview of a wide range of advanced data structures/algorithms, practical insight into when to use them, and advice on which implementations to use.
Even if you already have and are happy with CLRS, Skiena's book is a great addition to your library.
If you really want them to write "well-tested and modular algorithmic code" then you should teach them how to write packages in RStudio. Then require everything submitted to be in RMarkdown.
I mentor learning data scientists and my advice is always to start using RMarkdown as soon they're remotely comfortable with RStudio. Not only does it avoid issues with an easily polluted global namespace, but more importantly encourages literate programming from the early stages. In stats/data science literate programming is vital to having any idea what you were working on a few months ago. It also makes writing reports much, much easier.
RStudio makes it pretty easy to put together R packages, and the package structure for R does a great job of enforcing proper documentation and testing. Sourcing R files should primarily be used to quickly play around with ideas, or for exploratory data analysis that doesn't fit well inside an RMarkdown document. Any code you intended on reusing between projects should end up in a local package.
I do think it's a problem that R has no intermediate method of organizing code like simple modules in Python. But this means if you're serious about writing clean R, you just have to bite the bullet and teach students to write packages.
I worked at BBN (Bolt, Beranek and Newman) from 2006-2008 and was fortunate enough to have a one-on-one phone call with Leo while I worked there. At that time he was in his early 90s, and was still amazingly sharp and full of energy. Even in a brief phone call his curiosity and kindness left a long lasting impact.
BBN was (and presumably still is, though now part of Raytheon) an amazing company to work for even 50 years after its heyday as one of the original contractors on the ARPANET project. I remember getting to meet Ray Tomlinson (who sadly passed this March) and a wide range of others who were instrumental in the early days of the internet. Seeing what was happening in CS research at the time was pivotal for me changing my career towards computer science.
One of my favorite anecdotes about the early years of the company: As mentioned in another comment, BBN started as an acoustics firm. While JCR Licklider was there briefly in the early 60s he got the company to purchase a computer. This was expensive and somewhat out of the scope of the company, when asked why purchase such the thing the response was “this company is full of smart people, they’ll figure something out”. A few years later Licklider was a PM at ARPA in charge of the ARPANET project which BBN would soon become the lead contractor on.
Leo definitely lived to a ripe old age, but he will be missed nonetheless.
For those who've not seen the film "Computer Networks - The Herald of Resource Sharing" from 1972, I think you'll find it fascinating how many things they got right 40+ years ago and how many ideas became our reality. JCR Licklider is featured prominently in the film.
This statement is a misunderstanding of the Bayesian approach:
"Take the proposition that the Earth goes round the Sun. It either does or it doesn’t, so it’s hard to see how we could pick a probability for this statement."
Bayes Factor, the Bayesian alternative to a NHST, is quite a bit different than simply creating the Bayesian equivalent of a t-test. Bayes factor asks "How many times better is my Hypothesis at explaining the data than an alternate Hypothesis". So the Bayesian statement would first pit one model of the Earth's orbit against another. The Bayesian statement of the question of the Earth's orbit would be:
"How much more likely is the astronomical data we've observed to have happened given that the Earth revolves around the sun than it is if the sun revolved around the Earth."
For a more concrete example let's suppose that we have a coin. I think the coin has only heads and you think it is a fair coin, with a 50/50 chance of getting heads or tails. We observe three heads in a row. My hypothesis says that the probability of getting 3 heads in a row given a trick coin is 1. Your hypothesis says that the probability of getting 3 heads in a row given a fair coin is 0.5 x 0.5 x 0.5 = 0.125. My hypothesis explains the data 1/0.125 = 8 times better than your hypothesis. Now suppose the next flip is a tail. The probability of HHHT in my model is 0 and yours is 0.5 x 0.5 x 0.5 x 0.5 = 0.0625. You're hypothesis explains the model infinitely better than mine!
Now we can say that our new Hypothesis is that the coin is fair. Suppose another friend comes along and claims that they thought the coin had a 75% chance of getting heads and only a 25% chance of tails. We flip the coin 5 more times and get HHTTH. Your hypothesis says 0.5^5 = 0.03125, and the friend's says 0.75^3 x 0.25^2 = 0.0263... You're hypothesis explains the data only 1.2 times better than theirs. Clearly, we need more data to feel really confident in one hypothesis over the other.
If you want an even longer example, I wrote a post awhile back about "Bayesian Reasoning in the Twilight Zone" that goes into more detail (including priors)[0]
So in the real world, are there developed techniques to build relatively complex models that account for covariates and sources of variability (e.g. random effects) and repeated measures?
For anyone interested, I noticed that Dyalog made their proprietary implementation of APL free for non-commercial use just recently [0]. I suspect this change might have been in reaction to an uptick in traffic to their site from this other APL HN post[1] a bit more than a month ago (there's some good discussion their as well).
They also have Mastering Dyalog APL by Bernard Legrand as a free PDF download. I've just starting reading it, seems to be a very gentle introduction to APL: http://www.dyalog.com/mastering-dyalog-apl.htm
In addition to the "prompt consistency" and "thought-control" ideas mentioned in the post, I'm definitely curious how the performance is on more complex structured data (things like codegen).