How I test an AI chatbot’s coding ability – and you can, too
David Gewirtz/ZDNETSince ChatGPT and generative artificial intelligence (AI) hit the public consciousness in 2022, I’ve been exploring how well AI chatbots can write code. At first, the technology was a novelty, akin to encouraging a puppy to perform a new trick. But since seeing how AI chatbots can be effective productivity tools and programming partners, I’ve been subjecting the tools to more in-depth testing. Over time, I’ve compiled a set of four real-world tests that we’ve used to evaluate the performance of the main AI large language models (LLMs). So far, I’ve tested 10 LLMs. You can see the comprehensive results of all ten in this summary article:This article is intended to be a living document, where you can see my tests and even copy them to run your own. I’ll continue my series of individual tests, along with the articles that describe their performance. But now, you can dig in and play along at home (or wherever you have a good internet connection). If I update or add tests, I’ll also update this article, so feel free to check back in over time. More