8280861: Robot color picker broken on Linux with scaling above 100% #7425

mkartashev · 2022-02-10T14:08:48Z

The primary API for image capture on modern Linuxes is gdk_pixbuf_get_from_window() that expects both coordinates and the size unscaled (i.e. not multiplied by the current desktop scale). At the same time, gtk3_interface.c:gtk3_get_drawable_data() gets the coordinates from Robot that pre-scales them (multiplying by the scale known to Java, however, not necessarily the current desktop scale). The problem with the size had been partly taken care of in JDK-8225118, but coordinates are still passed to gdk_pixbuf_get_from_window() pre-scaled.

The idea of the fix is to capture a possibly larger area that is guaranteed to contain the one that is interesting to the caller and then only copy the interesting pixels to the output image. As a positive side effect, the size of the captured area cannot be less than 1x1 (provided the correct input, of course). This solves the problem of zero size passed to (*fp_gdk_pixbuf_get_from_drawable) when the desktop scale is 3 and we're asked to capture just one pixel. In that case, the previous formula (width / (float) win_scale + 0.5) would have yielded 0.

A related issue would be that tests written specifically for this general area (java/awt/Robot) didn't catch this problem and only somewhat unrelated tests (javax/swing/...) were affected.

This one is solved by adding pixel-sized areas to the test image in HiDPIRobotScreenCaptureTest.java, which precise colors are then verified to match the expected ones. In addition to that, instead of verifying just the color of the center of a large area, 4 more pixels close to the area's border are checked. This helps to make sure that captured area coordinates aren't way off.

The tests under javax/swing/... that originally helped to identify this problem were modified to provide more screen space for painting their components as extreme desktop scaling (300%) on Ubuntu doesn't leave much room due to a humongous title bar.

The fix was tested by running all the modified tests on Ubuntu with desktop scaling set to 100%, 200%, and 300%. The Robot tests were also executed on Windows (300% scaling) and MacOS.

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8280861: Robot color picker broken on Linux with scaling above 100%

Reviewers

Sergey Bylokhov (@mrserb - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/7425/head:pull/7425
$ git checkout pull/7425

Update a local copy of the PR:
$ git checkout pull/7425
$ git pull https://git.openjdk.java.net/jdk pull/7425/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 7425

View PR using the GUI difftool:
$ git pr show -t 7425

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/7425.diff

bridgekeeper · 2022-02-10T14:09:00Z

👋 Welcome back mkartashev! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2022-02-10T14:11:30Z

@mkartashev The following label will be automatically applied to this pull request:

client

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2022-02-10T14:15:25Z

Webrevs

mrserb · 2022-02-10T18:04:37Z

src/java.desktop/unix/native/libawt_xawt/awt/gtk3_interface.c

+inline static int scale_down_to_minus_inf(int what, int scale) {
+    return (int)floorf(what / (float)scale);
+}
+
 static gboolean gtk3_get_drawable_data(JNIEnv *env, jintArray pixelArray,


Don't we need to do the same for the gtk2? I suggest to update one of the test below to cover gtk_robot on/off for both gtk2/gtk3.

Maybe not as the previous fix was mode for gtk3 only, but I can double-check. Do you know if MATE desktop environment is a good candidate for finding gtk2 or are there easier options on Ubuntu?

I have no idea what MATE is but if you are saying you have a recent Ubuntu and you don't have GTK2, I am at least 95% sure GTK2 is still available from Ubuntu package manager for all current releases.

MATE desktop continues the "traditional" Gnome2 desktop development, so I thought it would be a good test for Gtk2 interfaces. FWIW, I ran the tests on such a system with DPI set to 200 (there's no other way to scale the interface under MATE AFAIK) and they all pass. However, the test executes the same code path as before going by if (gtk3_version_3_10) branch.

@prrace

GTK2 is still available from Ubuntu package manager for all current releases.

Right, but as long as gtk3 is also available, that's what we are going to use as AWT backend, see the code in gtk_interface.c.
I hacked around to forbid gtk3 from loading and the tests still passed (this time, without executing my changes).
I think gtk2 has no real notion of scale so the coordinates and size are always in screen pixels. See, for example, the documentation for gdk_pixbuf_get_from_drawable().

You do not need to hack the code, just add a "jdk.gtk.version" option to enable specific gtk version, this is what I suggested above. to disable the gtk usage in the robot the "awt.robot.gtk" can be used. So you can add that to some test to check that code paths works.

Thanks for the pointers, I'll set up a separate test for that.

mrserb · 2022-02-11T07:28:28Z

src/java.desktop/unix/native/libawt_xawt/awt/gtk3_interface.c

+inline static int scale_down_to_minus_inf(int what, int scale) {
+    return (int)floorf(what / (float)scale);
+}
+
 static gboolean gtk3_get_drawable_data(JNIEnv *env, jintArray pixelArray,
     int x, jint y, jint width, jint height, jint jwidth, int dx, int dy,
                                                                   jint scale) {


Is this scale parameter is simply ignored? If the passed parameters are always in the device space and the array is allocated properly then we should not care about this scale(especially in case of gtk2)?

Is this scale parameter is simply ignored?

Looks like it. I can only find one call site of get_drawable_data() (in Java_sun_awt_X11_XRobotPeer_getRGBPixelsImpl()) and the scale parameter is always 1. I think I can drop it from the interface if you prefer.

Yes please, this "scale" just make the code complicated w/o reason.

That's done; please, take a look at the updated code.

effectively unused.

prrace · 2022-02-15T00:43:42Z

src/java.desktop/unix/native/libawt_xawt/awt/gtk3_interface.c

@@ -2861,35 +2863,48 @@ static void transform_detail_string (const gchar *detail,
    }
 }

+inline static int scale_down_to_plus_inf(int what, int scale) {
+    return (int)ceilf(what / (float)scale);


I find the name of this and the companion function confusing.
what is "inf" short for ? I see that and I think plus_infinity.
scaled_ceiling and scaled_floor might be better names based on what I see.

Right, "inf" is for infinity; this is sort of how the rounding modes are described in the IEEE754 standard. But since full length words make the function name too long and short versions are confusing, I will rename as you suggested.

@prrace The change has been made, please, take a look.

openjdk · 2022-02-18T04:02:12Z

@mkartashev This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8280861: Robot color picker broken on Linux with scaling above 100%

Reviewed-by: serb

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 96 new commits pushed to the master branch:

c928958: 8281936: compiler/arguments/TestCodeEntryAlignment.java fails on AVX512 machines
a22f422: 8037573: Typo in DefaultTreeModel docs: askAllowsChildren instead of asksAllowsChildren
fdce35f: 8282025: assert(ctrl != __null) failed: control out is assumed to be unique after JDK-8281732
f830cbe: 8188073: Add Capstone as backend for hsdis
69fc273: 8282075: ProblemList 3 compiler/whitebox tests on macosx-x64
1292776: 8281317: CompactNumberFormat displays 4-digit values when rounding to a new range
cd9a3cf: 8282017: sun/net/www/protocol/https/HttpsURLConnection/B6216082.java fails with "SocketException: Unexpected end of file from server"
a6f8a38: 8281000: ClassLoader::registerAsParallelCapable throws NPE if caller is null
4c7f8b4: 8268250: Class.arrayType() for a 255-d array throws undocumented IllegalArgumentException
d0e1180: 8282019: Unused static fields DEGREES_TO_RADIANS, RADIANS_TO_DEGREES in StrictMath
... and 86 more: https://git.openjdk.java.net/jdk/compare/039313d65d47dc85cb8c91d3e1d2752d365f70f9...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@mrserb) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

mkartashev · 2022-02-18T05:10:51Z

/integrate

openjdk · 2022-02-18T05:12:15Z

@mkartashev
Your change (at version 4a8c20a) is now ready to be sponsored by a Committer.

forantar · 2022-02-21T11:38:09Z

/sponsor

openjdk · 2022-02-21T11:39:41Z

Going to push as commit cc7cf81.
Since your change was applied there have been 118 commits pushed to the master branch:

d7a706a: 8253757: Add LLVM-based backend for hsdis
bdae1d8: 8282147: [TESTBUG] waitForIdle after creating frame in JSpinnerMouseAndKeyPressTest.java
51f4420: 8282130: (bf) Remove unused ARRAY_BASE_OFFSET, ARRAY_INDEX_SCALE from read-only Heap Buffers
34aae32: 8282166: JDK-8282158 changed ECParameters' package by accident
c5d9142: 8282096: G1: Remove redundant checks in G1CardSet::free_mem_object
52a85d8: 8282158: ECParameters InvalidParameterSpecException messages missed ECKeySizeParameterSpec
4e0b81c: 8281544: assert(VM_Version::supports_avx512bw()) failed for Tests jdk/incubator/vector/
8563d86: 8282085: The REGISTER_DEFINITION macro is useless after JDK-8269122
d28b048: 8281815: x86: Use short jumps in TIG::generate_slow_signature_handler
d7f31d0: 8282077: PKCS11 provider C_sign() impl should handle CKR_BUFFER_TOO_SMALL error
... and 108 more: https://git.openjdk.java.net/jdk/compare/039313d65d47dc85cb8c91d3e1d2752d365f70f9...master

Your commit was automatically rebased without conflicts.

openjdk · 2022-02-21T11:39:57Z

@forantar @mkartashev Pushed as commit cc7cf81.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

8280861: Robot color picker broken on Linux with scaling above 100%

acb5637

openjdk bot added the rfr label Feb 10, 2022

openjdk bot added the client label Feb 10, 2022

mrserb reviewed Feb 10, 2022

View reviewed changes

mrserb reviewed Feb 11, 2022

View reviewed changes

mkartashev added 3 commits February 11, 2022 05:14

Dropped the 'scale' parameter from gtk*_get_drawable_data() as it was

d26291f

effectively unused.

Added a separate test with different GTK versions

a00cdc4

Updated copyright date in TestJProgressBarHighlightColor.java

fc651ad

prrace reviewed Feb 15, 2022

View reviewed changes

Renamed scaling functions and updated copyright.

4a8c20a

mrserb approved these changes Feb 18, 2022

View reviewed changes

openjdk bot added the ready label Feb 18, 2022

openjdk bot added the sponsor label Feb 18, 2022

openjdk bot added the integrated label Feb 21, 2022

openjdk bot closed this Feb 21, 2022

openjdk bot removed ready rfr sponsor labels Feb 21, 2022

aivanov-jdk mentioned this pull request Mar 15, 2022

8282270: java/awt/Robot Screen Capture tests fail after 8280861 #7613

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8280861: Robot color picker broken on Linux with scaling above 100% #7425

8280861: Robot color picker broken on Linux with scaling above 100% #7425

mkartashev commented Feb 10, 2022 •

edited by openjdk bot

Loading

bridgekeeper bot commented Feb 10, 2022

openjdk bot commented Feb 10, 2022

mlbridge bot commented Feb 10, 2022 •

edited

Loading

mrserb Feb 10, 2022

mkartashev Feb 10, 2022

prrace Feb 10, 2022

mkartashev Feb 11, 2022

mrserb Feb 11, 2022

mkartashev Feb 11, 2022

mrserb Feb 11, 2022

mkartashev Feb 11, 2022

mrserb Feb 14, 2022

mkartashev Feb 15, 2022

prrace Feb 15, 2022

mkartashev Feb 15, 2022

mkartashev Feb 15, 2022

openjdk bot commented Feb 18, 2022 •

edited

Loading

mkartashev commented Feb 18, 2022

openjdk bot commented Feb 18, 2022

forantar commented Feb 21, 2022

openjdk bot commented Feb 21, 2022

openjdk bot commented Feb 21, 2022

8280861: Robot color picker broken on Linux with scaling above 100% #7425

8280861: Robot color picker broken on Linux with scaling above 100% #7425

Conversation

mkartashev commented Feb 10, 2022 • edited by openjdk bot Loading

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented Feb 10, 2022

openjdk bot commented Feb 10, 2022

mlbridge bot commented Feb 10, 2022 • edited Loading

Webrevs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openjdk bot commented Feb 18, 2022 • edited Loading

mkartashev commented Feb 18, 2022

openjdk bot commented Feb 18, 2022

forantar commented Feb 21, 2022

openjdk bot commented Feb 21, 2022

openjdk bot commented Feb 21, 2022

mkartashev commented Feb 10, 2022 •

edited by openjdk bot

Loading

mlbridge bot commented Feb 10, 2022 •

edited

Loading

openjdk bot commented Feb 18, 2022 •

edited

Loading