Counting on the order of files to support multiple versions of jars was never a good idea. Java does have multiple version ("release") jar files for your use case since java 9. See <https://docs.oracle.com/en/java/javase/24/docs/api/java.base...>.
Since duplicates on the classpath don't cause problems, a quick & dirty fix is to manually list versioned jars first, in order, then the jars/* argument.
> there was a client library that needed a Bouncy Castle “provider” with a version “jdk15”+ as the client initialization used specific properties from a class, and those properties were only available in “jdk15”+.
> up until the node image update, we “fortunately” had node images with directory hash seeds ordering “jdk15” or “jdk18” before “jdk14”.
So the actual bug is that something needing jdk15+ should either retry or be deterministically fed a valid file, right? And this whole article is figuring out why the filesystem coincidentally masked it by accidentally always happening to hand it a file with what it needed?
It does matter for performance.
If you read files in the same order they are on disk (often, the order in which they were written, which readdir on modern filesystems should choose to produce), I/O is much faster.
Build tools supporting duplicate class detection have existed for… well a long time. Ignore them at your own peril.
The orange site discusses the article in the first footnote here: https://news.ycombinator.com/item?id=43573507
Always fun when code relies on the order of iterating over a dir (which is in general clearly not defined to have any order, even iterating the same dir 2th consecutively might not yield the same order depending on "stuff" (e.g. exact file system used)).
So if order matters, always sort.
(Luckily in most situations where dir iter order matters, the performance impact from sorting is acceptable or even outright irrelevant.)
By the way, max hardlink count for ext4 seems configured ridiculously low for modern standards, at least on Ubuntu.
"ls" can take ages on a large folder. Is there a way to make it more immediate, i.e. streaming output without sorting?
Could they have avoided that issue by specifying the classpath without the star?
So -cp /jars/ instead of -cp /jars/*?
So what was the production fix? Surely you're not hex-editing the image until the end of time?
great article
I'm feeling like an old man now but who the hell calls a tool "buildah"? Especially with its ugly dog logo. You can almost assume the dog wants to say "builder" but the extra flaps of skin makes the sound distorted
ext4 has no checksums, integrity checks, etc. it will silently corrupt your data and you wouldn't even know about it. switch to btrfs, it's way better
fun read, now i want to learn about overlays
> the actual argument value the JVM receives is "/jars/*", and in turn decides to be helpful, and expand the wildcard anyway
Whenever I see such things, I immediately think "whatever the resulting order is, it had better not matter"; and if it does, which is definitely true for Java classpaths, I consider it a bug that needs to be fixed ASAP, before it causes what happened in the article.