Tag Archives: ChatGPT

AI at the crossroads: can ChatGPT turn you into a statistical Robert Johnson?

Leave a reply

When it comes to the statistical analysis of data, I know my limits. Maths was never my strong point at school or university, and my approach has always been to keep analyses as simple and straightforward as possible*, or to rely on colleagues with fancier statistical chops to do the heavy lifting. I wish that were not the case – I wish I had a brain that was more number-focused than it is. But I don’t and I’ve learned to live with it, to play to my actual strengths as a scientist, and to collaborate with others who can bring different skills to the party.

In theory, the development of the R platform was supposed to make life easier for those of us who wanted to analyse complex data sets. But actually having to script, from scratch, the code to carry out even simple analyses always seemed to me to be a step backwards from the push-button days of SPSS or Minitab. Yes, I get that R is incredibly powerful and flexible and blah blah. But it still involves a heavy time commitment and an aptitude for writing code that many of us struggle with.

Recently, however, things have changed. I find myself carrying out complex statistical analyses that would have stumped me 12 months ago. Not only that, but I now understand those tests on a much deeper level than I ever did before. I also feel much more confident in the interpretation of the outputs from the tests I’m running, and their limitations.

Why the over night change? ChatGPT.

More precisely, I’m using ChatGPT to help me decide which analytical approaches are best for the data that I have, getting it to help me to write the R script to carry out the tests, and then (crucially) it’s advising me on the interpretation of the statistical output and suggesting future steps.

Let me give you an example. I’ve just submitted a manuscript to a journal which describes the results from an experiment that had confounded me for years, which is why I’d not published the work previously. Following discussions with some colleagues in China I realised that my framing of the work was wrong (by coincidence, a topic that Jeremy Fox has recently discussed over on the Dynamic Ecology blog). However, there was still a contradiction in two of the sets of results that I could not resolve: they should have been telling me the same thing but they were not. When I queried ChaptGPT on this it suggested that I model the data taking into account the fact that I had missing data – missingness in statistical jargon. When I did – bingo! – the results made sense: the absence of some data in my experimental treatments had systematically biased the results. It all made perfect sense.

Now, I could have talked this over with a statistician or a more statistically-minded ecologist colleague. But scientists are busy people and I did not want to impose on someone’s limited time. Or rather multiple someone’s limited times, because I know from past experience that when you ask folks these sorts of questions you can get different advice depending upon their own experiences, training, or preferred flavours of statistical analysis. By treating ChatGPT as a collaborator I can get an objective answer to my data questions, written in a way that I can understand. That last point is key because for all of us with specific expertise it’s sometimes difficult to translate our knowledge into broadly interpretable language.

How can I know that ChatGPT is giving me reliable statistical advice? It certainly didn’t give me accurate information about Erasmus Darwin a couple of years ago (a story, incidentally, that I included in my recent book Birds & Flowers: An Intimate 50 Million Year Relationship). But since then, the reliability and accuracy of ChatGPT has improved considerably and when I’ve checked the information it’s given about analyses it was usually accurate as far as I can gauge. In one case, however, it completely missed the point of what I was trying to do with another set of data. But of course advice from human collaborators can also be inaccurate – everyone is fallible. So including human (my!) oversight in all of this is important.

I’m certainly not the only one using ChatGPT and other AI platforms in this way – here’s a small sample of some online articles I’ve found on the topic:

I especially like this quote from that last article:

“If I hired a consultant to write the code when I told them what I needed, would that be a problem? Then, what’s the problem in doing stats with an (AI) consultant?”

I can only agree, and again, I emphasise that we need to think of AI as a collaborator as much as a tool.

By now you might be wondering what any of this has got to do with blues musician Robert Johnson**. Well, according to legend, Johnson was initially a fairly mediocre guitar player. His overnight transformation into a brilliant and highly influential guitarist was attributed to him meeting the Devil at a local crossroads and trading his soul for the mastery of his instrument.

Of course the story is nonsense, and Johnson clearly worked hard at his craft, but it feels like a nice metaphor of where we are at the moment: at a crossroads when it comes to how we use AI, with the opportunity for it to turn us, overnight, into experts. Will that come at a profound cost? Are we selling our souls to some kind of digital Devil? AI naysayers claim it will and we are. But as with all technological advancements, from the wheel to the internet, there are both good and bad aspects to it. Only time will tell whether the former outweighs the latter. In the meantime, let’s try to discover how best to use this amazing technology for our own and society’s benefits.

——————————————————————————————————————-

*In fact one of my recent data papers involved no statistical analysis at all because the results were so clear cut: to paraphrase the abstract, when East African lowland honey bees encountered other insects on flower heads, those other insects were displaced 100% of the time. That’s not a result that requires any kind of test.

**Blues is one of my favourite genres of music and this is not the first time it’s featured on the blog. When Karin and I returned to the UK in March 2024 one of the first things that I got involved with was helping to kick-start a regular blues evening featuring professional British and international players. The not-for-profit initiative is called Harborough Blues – check out the website for upcoming gigs.

Image credit: ChatGPT of course, under instruction from me

How reliable is ChatGPT? A weird encounter with Erasmus Darwin

22 Replies

There’s currently a lot of interest in ChatGPT and over the past week, like many others, I have been playing around with this artificial intelligence (AI) system and testing its knowledge and creative powers. The latter are really very impressive: I asked it to write a couple of limericks about Karin and my professions, and about our move to Denmark, and it produced a decent stab at those.

I also probed its knowledge about topics with which I’m familiar, mainly pollinators and pollination of course, and it produced some good answers. When I highlighted some errors it had made, ChatGPT went back and revised its answers, and politely thanked me in the process. If I were a student I would not rely on it to write accurate essays, but so far so good(ish).

This morning, however, I had a very odd experience. I was trying to track down something attributed to Erasmus Darwin (Charles’s grandfather) and I asked ChatGPT: “What did Erasmus Darwin say about birds visiting flowers?”

This is the answer that came back:

Erasmus Darwin was a naturalist, physician, and poet who wrote about a wide range of topics, including birds and flowers. In his 1794 poem “The Loves of the Plants,” he describes how different species of birds visit flowers in search of nectar:

“Birds of each hue, from the gay parrot’s beak
To the goldfinch, painted in tiny streak,
Sip nectar from the cups that bloom and bow
With every breeze, and nod and dance below.”

Now, the first part of this, about Erasmus’s life, is true. It’s also true that he wrote a poem with that title, though it was published in 1791, not 1794.

But, when I double-checked those attributed lines against the published version of the poem that’s on Project Gutenberg, I discovered that they did not exist. ChatGPT had made them up!

As I said, this AI is pretty creative and will certainly write a poem for you on any topic you care to request. But I didn’t ask it for that, I had made a factual query. And ChatGPT had just invented some poetry and claimed it as fact, unless of course those really are Erasmus’s lines and I’ve missed their source (please correct me if I have).

Now as a scientist and writer I try to be rigorous when quoting and citing sources, and if I hadn’t double-checked this I might have taken it at face value. And one could argue that my initial question – “What did Erasmus Darwin say…?” – was a little ambiguous. But nonetheless, I was not expecting an AI to try to pass off something it had created as fact.

Has anyone else experienced anything like this when using ChatGPT? It’s a fun, creative tool. But as far as I can tell, it also has a very human ability to lie.