JDK-8275703: System.loadLibrary fails on Big Sur for libraries hidden from filesystem #6127

mlchung · 2021-10-26T22:51:29Z

On, macOS 11.x, system libraries are loaded from dynamic linker cache. The libraries are no longer present on the filesystem. NativeLibraries::loadLibrary checks for the file existence before calling JVM_LoadLibrary. Such check no longer applies on Big Sur. This proposes that on macOS >= 11, it will skip the file existence check and attempt to load a library for each path from java.library.path and system library path.

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8275703: System.loadLibrary fails on Big Sur for libraries hidden from filesystem

Reviewers

David Holmes (@dholmes-ora - Reviewer) ⚠️ Review applies to e034029
Alan Bateman (@AlanBateman - Reviewer) ⚠️ Review applies to e034029
Maurizio Cimadamore (@mcimadamore - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/6127/head:pull/6127
$ git checkout pull/6127

Update a local copy of the PR:
$ git checkout pull/6127
$ git pull https://git.openjdk.java.net/jdk pull/6127/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 6127

View PR using the GUI difftool:
$ git pr show -t 6127

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/6127.diff

… from filesystem

bridgekeeper · 2021-10-26T22:52:14Z

👋 Welcome back mchung! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-10-26T22:56:37Z

@mlchung The following labels will be automatically applied to this pull request:

core-libs
hotspot

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

mlbridge · 2021-10-27T02:32:29Z

Webrevs

01: Full - Incremental (710925b5)
00: Full (e034029f)

dholmes-ora

Hi Mandy,

Hotspot changes are fine.

Thanks,
David

openjdk · 2021-10-27T02:42:01Z

@mlchung This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8275703: System.loadLibrary fails on Big Sur for libraries hidden from filesystem

Reviewed-by: dholmes, alanb, mcimadamore

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 15 new commits pushed to the master branch:

9a3e954: 8274879: Replace uses of StringBuffer with StringBuilder within java.base classes
e6fa5fa: 8276063: ProblemList gtest dll_address_to_function_and_library_name on macosx-generic
809488b: 8276046: codestrings.validate_vm gtest fails on ppc64, s390
93be099: 4718400: Many quantities are held as signed that should be unsigned
168081e: 8270490: Charset.forName() taking fallback default value
a292733: 8275975: Remove dead code in ciInstanceKlass
4060602: 8275800: Redefinition leaks MethodData::_extra_data_lock
485d658: 8275851: Deproblemlist open/test/jdk/javax/swing/JComponent/6683775/bug6683775.java
b3f45f8: 8275689: [TESTBUG] Use color tolerance only for XRender in BlitRotateClippedArea test
6c05cc9: 8275863: Use encodeASCII for ASCII-compatible DoubleByte encodings
... and 5 more: https://git.openjdk.java.net/jdk/compare/7addcd7cfb73652841c65c54e84b6ebffcbd664e...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

jaikiran · 2021-10-27T02:51:24Z

src/java.base/macosx/classes/jdk/internal/loader/ClassLoaderHelper.java

+                major = Integer.parseInt(osVersion.substring(0, i));
+            } catch (NumberFormatException e) {}
+        }
+        hasDynamicLoaderCache = major >= 11;


Hello Mandy,
I'm not too familiar with MacOS versioning schemes. However, in this specific logic, if the os.version value doesn't contain a dot character, then the major is initialized to 11, which would then evaluate this hasDynamicLoaderCache to true. That would mean if the os.version is (for example) 10, then hasDynamicLoaderCache will be incorrectly set to true here, isn't it?

macOS product version contains 3 parts: major, minor, and patch version. But it does not hurt to handle the case where no dot exists in the string.

int major = 11; int i = osVersion.indexOf('.'); try { major = Integer.parseInt(i < 0 ? osVersion : osVersion.substring(0, i)); } catch (NumberFormatException e) {}

src/java.base/share/classes/jdk/internal/loader/NativeLibraries.java

AlanBateman

The change is unfortunate but looks okay.

mlbridge · 2021-10-27T18:25:40Z

Mailing list message from Alan Snyder on core-libs-dev:

On Oct 27, 2021, at 9:28 AM, Mandy Chung <mchung at openjdk.java.net> wrote:

On, macOS 11.x, system libraries are loaded from dynamic linker cache. The libraries are no longer present on the filesystem. `NativeLibraries::loadLibrary` checks for the file existence before calling `JVM_LoadLibrary`. Such check no longer applies on Big Sur. This proposes that on macOS >= 11, it will skip the file existence check and attempt to load a library for each path from java.library.path and system library path.

I?m curious about this issue.

I never load system libraries directly. I load my own libraries (that support JNI entry points) and the system loader loads the necessary system frameworks that they were linked against.

What?s different in this case that motivates loading system libraries directly from Java?

Alan

mcimadamore

This came up on panama-dev a bunch of time now. In fact, in the panama use case this would make sense for other systems as well. For instance on linux systems, there's a bunch of known library names in gnu/lib-names.h which programs can depend on, but for which System::load/loadLibrary cannot be used - as they are neither library names, nor paths - for instance "libm.so.6".

mlchung · 2021-10-27T19:02:31Z

I never load system libraries directly. I load my own libraries (that support JNI entry points) and the system loader loads the necessary system frameworks that they were linked against.

What's different in this case that motivates loading system libraries directly from Java

Panama would be the right place to provide the support of loading system libraries directly from Java and enhance the interconnection between JVM and native code.

The bug report is an example showing that developers want to load a system library for their native code to use. It's not surprise that they use System::load/loadLibrary even it's designed for JNI without Panama since it works. So this patch is for compatibility for existing applications to continue to run on Big Sur.

mlchung · 2021-10-28T15:26:15Z

/integrate

openjdk · 2021-10-28T15:27:38Z

Going to push as commit 309acbf.
Since your change was applied there have been 25 commits pushed to the master branch:

abe52ae: 8275518: accessibility issue in Inet6Address docs
85d8cd8: 8276100: Remove G1SegmentedArray constructor name parameter
a343fa8: 8275865: Print deoptimization statistics in product builds
bec977c: 8275917: Some locks shouldn't allow_vm_block
7c996d5: 8269401: Merge "Exceptions" and "Errors" into "Exception Classes"
d88b89f: 8276067: ZGC: Remove unused function alloc_high_address_at_most()
593401f: 8276042: Remove unused local variables in java.naming
a2f2d8f: 8276057: Update JMH devkit to 1.33
1750a6e: 8276055: ZGC: Defragment address space
d9b0138: 8275704: Metaspace::contains() should be threadsafe
... and 15 more: https://git.openjdk.java.net/jdk/compare/7addcd7cfb73652841c65c54e84b6ebffcbd664e...master

Your commit was automatically rebased without conflicts.

openjdk · 2021-10-28T15:27:51Z

@mlchung Pushed as commit 309acbf.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

mlchung added 2 commits October 26, 2021 15:42

JDK-8275703: System.loadLibrary fails on Big Sur for libraries hidden…

9f983e5

… from filesystem

fix typo

8e081d3

add copyright header

94b9e13

openjdk bot added hotspot core-libs labels Oct 26, 2021

trim whitespaces

e034029

openjdk bot added the rfr label Oct 27, 2021

dholmes-ora approved these changes Oct 27, 2021

View reviewed changes

openjdk bot added the ready label Oct 27, 2021

jaikiran reviewed Oct 27, 2021

View reviewed changes

src/java.base/share/classes/jdk/internal/loader/NativeLibraries.java Show resolved Hide resolved

AlanBateman approved these changes Oct 27, 2021

View reviewed changes

mlchung added 2 commits October 27, 2021 09:11

Exclude building exeLibraryCache.c on other platforms except macOS

04f1df6

Adjust parsing os.version to handle no dot version in case it's allowed

710925b

mcimadamore approved these changes Oct 27, 2021

View reviewed changes

openjdk bot closed this Oct 28, 2021

openjdk bot added integrated and removed ready rfr labels Oct 28, 2021

mlchung deleted the loadLibrary branch November 4, 2021 19:26

tstuefe mentioned this pull request Sep 17, 2022

JDK-8293659: Improve UnsatisfiedLinkError error message to include dlopen error details #10286

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JDK-8275703: System.loadLibrary fails on Big Sur for libraries hidden from filesystem #6127

JDK-8275703: System.loadLibrary fails on Big Sur for libraries hidden from filesystem #6127

mlchung commented Oct 26, 2021 •

edited by openjdk bot

Loading

bridgekeeper bot commented Oct 26, 2021

openjdk bot commented Oct 26, 2021

mlbridge bot commented Oct 27, 2021 •

edited

Loading

dholmes-ora left a comment

openjdk bot commented Oct 27, 2021 •

edited

Loading

jaikiran Oct 27, 2021

mlchung Oct 27, 2021

AlanBateman left a comment

mlbridge bot commented Oct 27, 2021

mcimadamore left a comment

mlchung commented Oct 27, 2021

mlchung commented Oct 28, 2021

openjdk bot commented Oct 28, 2021

openjdk bot commented Oct 28, 2021

JDK-8275703: System.loadLibrary fails on Big Sur for libraries hidden from filesystem #6127

JDK-8275703: System.loadLibrary fails on Big Sur for libraries hidden from filesystem #6127

Conversation

mlchung commented Oct 26, 2021 • edited by openjdk bot Loading

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented Oct 26, 2021

openjdk bot commented Oct 26, 2021

mlbridge bot commented Oct 27, 2021 • edited Loading

Webrevs

dholmes-ora left a comment

Choose a reason for hiding this comment

openjdk bot commented Oct 27, 2021 • edited Loading

jaikiran Oct 27, 2021

Choose a reason for hiding this comment

mlchung Oct 27, 2021

Choose a reason for hiding this comment

AlanBateman left a comment

Choose a reason for hiding this comment

mlbridge bot commented Oct 27, 2021

mcimadamore left a comment

Choose a reason for hiding this comment

mlchung commented Oct 27, 2021

mlchung commented Oct 28, 2021

openjdk bot commented Oct 28, 2021

openjdk bot commented Oct 28, 2021

mlchung commented Oct 26, 2021 •

edited by openjdk bot

Loading

mlbridge bot commented Oct 27, 2021 •

edited

Loading

openjdk bot commented Oct 27, 2021 •

edited

Loading