Another Java Question LLMs Can’t Answer

March 3rd, 2025

This time I asked Gemini, “How does Maven version comparison differ from semver version comparison?”, and while this time its answer didn’t say anything patently false, it did fail to find the most important distinction. It’s operating at the level of an undergraduate with better than average writing skills but who still doesn’t really understand the subject matter. It didn’t find anything new, although in this case there was something new to be found. (I found it myself a few weeks ago, but haven’t gotten around to publishing it on the Web yet.)

Read the rest of this entry »

Gemini Doesn’t Understand Namespaces Either

February 25th, 2025

Last week Gemini gave me a very wrong answer about exception handling in Java. This week let’s see if it can handle an even less controversial but often misunderstood point about XML namespaces. (Spoiler: it can’t.)

The question is:

Should XML namespace URLs include a version number?

The correct answer is no.1, 2 Gemini’s answer is the opposite, and the reasons it gives show why it’s wrong.
Read the rest of this entry »

Gemini doesn’t understand exceptions. Then again neither do you.

February 21st, 2025

Some days I feel like code review is an exercise in educating developers. Nowhere is that clearer than in Java exception handling. Almost nobody understands this. When I’m interviewing software engineers, I pretty much never ask about exceptions because it’s an almost 100% guaranteed fail. If I’m being interviewed and I’m asked about exceptions, I have to play the meta game where I assume that my interviewer believes things that aren’t true, and I tailor my response to fit their misconceptions. Mostly I do that by focusing on the syntax and behavior without covering the semantics and proper use of exceptions. Effective Java comes closer than most to correctly explaining which exceptions you should use when, but even Bloch doesn’t quite stick the landing.

Today after yet another round of back-and-forth code review comments where I once again had to explain to an otherwise experienced developer how exception handling is supposed to work, I got the bright idea to ask Gemini, Google’s machine learning chatbot, what it thought. It gave me a better answer to the question than most programmers do; but it still made two fundamental mistakes, one of commission (saying something that isn’t true) and one of omission (leaving out an absolutely critical point). See if you can spot the mistakes.

Read the rest of this entry »

Bleeding

February 19th, 2025

For many centuries, bleeding patients was a standard treatment for many diseases. Cancer? Bleed the patient. Headache? Bleed the patient. Fever? Bleed the patient. Pneumonia? Bleed the patient. Bleeding was accepted medical wisdom.

Perhaps surprisingly to modern patients, bleeding worked, at least some of the time. The patient would get better. Of course, a lot of the time if the doctor did nothing, the patient still got better. No one bothered to ask whether it was the bleeding that caused the patient to get better or not. Few people even knew how to phrase the question.
Read the rest of this entry »

Stack Traces Considered Harmful

February 13th, 2025

I’m trying to build an open source project in a language I’m unfamiliar with and hit this problem:

(base) $ ruby bin/setup
== Installing dependencies ==

A new release of RubyGems is available: 3.6.2 ? 3.6.3!
Run `gem update --system 3.6.3` to update your installation.

Bundler 2.6.2 is running, but your lockfile was generated with 2.3.9. Installing Bundler 2.3.9 and restarting using that version.
Fetching gem metadata from https://rubygems.org/.
Fetching bundler 2.3.9
Installing bundler 2.3.9
Your Ruby version is 3.4.1, but your Gemfile specified ~> 3.0.4
bin/setup:16:in 'Kernel#system': Command failed with exit 18: bundle (RuntimeError)
	from bin/setup:16:in 'block in 
' from /opt/homebrew/Cellar/ruby/3.4.1/lib/ruby/3.4.0/fileutils.rb:241:in 'Dir.chdir' from /opt/homebrew/Cellar/ruby/3.4.1/lib/ruby/3.4.0/fileutils.rb:241:in 'FileUtils#cd' from bin/setup:10:in '<main>'

This illustrates a common antipattern in error handling. This is a Ruby program, but I’ve encountered it often in Python programs too, including the Google Cloud SDK. It also happens in Java, though less frequently. The most common place it appears in Java is when JUnit tests fail. Do you see it?
Read the rest of this entry »