Skip to content

Commit 7b5b114

Browse files
fabOnReactfacebook-github-bot
authored andcommitted
Making links independently focusable by Talkback (#33215)
Summary: This issue fixes [32004][23]. The Pull Request was previously published by [blavalla][10] with [31757][24]. >This is a follow-up on [D23553222 (https://github.com/facebook/react-native/commit/b352e2da8137452f66717cf1cecb2e72abd727d7)][18], which made links functional by using [Talkback's Links menu][1]. We don't often use this as the sole access point for links due to it being more difficult for users to navigate to and easy for users to miss if they don't listen to the full description, including the hint text that announces that links are available. The Implementation of the functionality consists of: Retrieving the accessibility links and triggering the TalkBack Focus over the Text 1. nested Text components with accessibilityRole link are saved as [ReactClickableSpan][17] instances in Android native [TextView][20] ([more info][19]) 1. If the TextView contains any [ClickableSpans][15] (which are [nested Text][14] components with role link), set a view tag and reset the accessibility delegate. 3. Obtain each link description, start, end, and position relative to the parent Text (id) from the Span as an [AccessibilityLink][16] 4. Use the [AccessibilityLink][16] to display TalkBack focus over the link with the `getVirtualViewAt` method (more [info][13]) Implementing ExploreByTouchHelper to detect touches over links and to display TalkBack rectangle around them. 1. ReactAccessibilityDelegate inherits from [ExploreByTouchHelper][12] 2. If the [ReactTextView][21] has an accessibility delegate, trigger ExploreByTouchHelper method [dispatchHoverEvent][22] 3. Implements the methods `getVirtualViewAt` and `onPopulateBoundsForVirtualView`. The two methods implements the following functionalities (more [info][13]): * detecting the TalkBack onPress/focus on nested Text with accessibilityRole="link" * displaying TalkBack rectangle around nested Text with accessibilityRole="link" ## Changelog [Android] [Added] - Make links independently focusable by Talkback Pull Request resolved: #33215 Test Plan: [1]. User Interacts with links through TalkBack default accessibility menu ([link][1]) [2]. The nested link becomes the next focusable element after the parent element that contains it. ([link][2]) [3]. Testing accessibility examples in pr branch ([link][3]) [4]. Testing accessibility android examples in pr branch ([link][4]) [7]. TalkBack focus moves through links in the correct order from top to bottom (PR Branch with [link.id][25]) ([link to video test][7]) ([discussion][26]) [8]. TalkBack focus does not move through links in the correct order from top to bottom (PR Branch without [link.id][25]) ([link to video test][8]) ([discussion][26]) Test on main branch [5]. Testing accessibility examples in main branch ([link][5]) [6]. Testing accessibility android examples in main branch ([link][6]) [1]: fabOnReact/react-native-notes#9 (comment) [2]: fabOnReact/react-native-notes#9 (comment) [3]: fabOnReact/react-native-notes#9 (comment) [4]: fabOnReact/react-native-notes#9 (comment) [5]: fabOnReact/react-native-notes#9 (comment) [6]: fabOnReact/react-native-notes#9 (comment) [7]: fabOnReact/react-native-notes#9 (comment) [8]: fabOnReact/react-native-notes#9 (comment) [10]: https://github.com/blavalla "blavalla github profile" [12]: https://github.com/aosp-mirror/platform_frameworks_base/blob/1ac46f932ef88a8f96d652580d8105e361ffc842/core/java/com/android/internal/widget/ExploreByTouchHelper.java#L48 "com/android/internal/widget/ExploreByTouchHelper.java#L48" [13]: fabOnReact/react-native-notes#9 (comment) "explanation of getVirtualViewAt and onPopulateBoundsForVirtualView" [14]: https://github.com/aosp-mirror/platform_frameworks_base/blob/1ac46f932ef88a8f96d652580d8105e361ffc842/core/java/android/text/Spannable.java#L3 "core/java/android/text/Spannable.java#L3" [15]: https://github.com/fabriziobertoglio1987/react-native/blob/561266fc180b96d6337d6c6c5c3323522d66cc44/ReactAndroid/src/main/java/com/facebook/react/views/text/ReactTextViewManager.java#L70-L71 "react/views/text/ReactTextViewManager.java#L70-L71" [16]: https://github.com/fabriziobertoglio1987/react-native/blob/561266fc180b96d6337d6c6c5c3323522d66cc44/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java#L680-L685 "react/uimanager/ReactAccessibilityDelegate.java#L680-L685" [17]: https://github.com/facebook/react-native/blob/561266fc180b96d6337d6c6c5c3323522d66cc44/ReactAndroid/src/main/java/com/facebook/react/views/text/TextLayoutManager.java#L126-L129 "react/views/text/TextLayoutManager.java#L126-L129" [18]: b352e2d [19]: #30375 (comment) "explanation on how nested Text are converted to Android Spans" [20]: https://github.com/aosp-mirror/platform_frameworks_base/blob/1ac46f932ef88a8f96d652580d8105e361ffc842/core/java/android/widget/TextView.java#L214-L220 "core/java/android/widget/TextView.java#L214-L220" [21]: https://github.com/facebook/react-native/blob/485cf6118b0ab0b59e078b96701b69ae64c4dfb7/ReactAndroid/src/main/java/com/facebook/react/views/text/ReactTextView.java#L577 "dispatchHoverEvent in ReactTextView" [22]: https://github.com/aosp-mirror/platform_frameworks_base/blob/1ac46f932ef88a8f96d652580d8105e361ffc842/core/java/com/android/internal/widget/ExploreByTouchHelper.java#L120-L138 "dispatchHoverEvent in ExploreByTouchHelper" [23]: #32004 [24]: #31757 [25]: https://github.com/fabriziobertoglio1987/react-native/blob/485cf6118b0ab0b59e078b96701b69ae64c4dfb7/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java#L648 "setting link.id in the AccessibilityLink constructor" [26]: https://github.com/facebook/react-native/pull/33215/files/485cf6118b0ab0b59e078b96701b69ae64c4dfb7#r820014411 "comment on role of link.id" Reviewed By: blavalla Differential Revision: D34687371 Pulled By: philIip fbshipit-source-id: 8e63c70e9318ad8d27317bd68497705e595dea0f
1 parent 6ee70a9 commit 7b5b114

16 files changed

+423
-51
lines changed

Libraries/Text/Text.js

+1
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,7 @@ const Text: React.AbstractComponent<
129129
onResponderTerminate(event);
130130
}
131131
},
132+
onClick: eventHandlers.onClick,
132133
onResponderTerminationRequest:
133134
eventHandlers.onResponderTerminationRequest,
134135
onStartShouldSetResponder: eventHandlers.onStartShouldSetResponder,

Libraries/Text/TextNativeComponent.js

+2
Original file line numberDiff line numberDiff line change
@@ -14,11 +14,13 @@ import {type HostComponent} from '../Renderer/shims/ReactNativeTypes';
1414
import createReactNativeComponentClass from '../Renderer/shims/createReactNativeComponentClass';
1515
import {type ProcessedColorValue} from '../StyleSheet/processColor';
1616
import {type TextProps} from './TextProps';
17+
import {type PressEvent} from '../Types/CoreEventTypes';
1718

1819
type NativeTextProps = $ReadOnly<{
1920
...TextProps,
2021
isHighlighted?: ?boolean,
2122
selectionColor?: ?ProcessedColorValue,
23+
onClick?: ?(event: PressEvent) => mixed,
2224
// This is only needed for platforms that optimize text hit testing, e.g.,
2325
// react-native-windows. It can be used to only hit test virtual text spans
2426
// that have pressable events attached to them.

ReactAndroid/src/main/java/com/facebook/react/uimanager/BaseViewManager.java

+2-1
Original file line numberDiff line numberDiff line change
@@ -433,7 +433,8 @@ private static void resetTransformProperty(@NonNull View view) {
433433
}
434434

435435
private void updateViewAccessibility(@NonNull T view) {
436-
ReactAccessibilityDelegate.setDelegate(view);
436+
ReactAccessibilityDelegate.setDelegate(
437+
view, view.isFocusable(), view.getImportantForAccessibility());
437438
}
438439

439440
@Override

ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java

+248-21
Original file line numberDiff line numberDiff line change
@@ -8,19 +8,26 @@
88
package com.facebook.react.uimanager;
99

1010
import android.content.Context;
11+
import android.graphics.Paint;
12+
import android.graphics.Rect;
1113
import android.os.Bundle;
1214
import android.os.Handler;
1315
import android.os.Message;
14-
import android.text.SpannableString;
15-
import android.text.style.URLSpan;
16+
import android.text.Layout;
17+
import android.text.Spannable;
18+
import android.text.Spanned;
19+
import android.text.style.AbsoluteSizeSpan;
20+
import android.text.style.ClickableSpan;
1621
import android.view.View;
1722
import android.view.accessibility.AccessibilityEvent;
23+
import android.widget.TextView;
24+
import androidx.annotation.NonNull;
1825
import androidx.annotation.Nullable;
19-
import androidx.core.view.AccessibilityDelegateCompat;
2026
import androidx.core.view.ViewCompat;
2127
import androidx.core.view.accessibility.AccessibilityNodeInfoCompat;
2228
import androidx.core.view.accessibility.AccessibilityNodeInfoCompat.AccessibilityActionCompat;
2329
import androidx.core.view.accessibility.AccessibilityNodeInfoCompat.RangeInfoCompat;
30+
import androidx.customview.widget.ExploreByTouchHelper;
2431
import com.facebook.react.R;
2532
import com.facebook.react.bridge.Arguments;
2633
import com.facebook.react.bridge.Dynamic;
@@ -36,13 +43,15 @@
3643
import com.facebook.react.uimanager.events.Event;
3744
import com.facebook.react.uimanager.events.EventDispatcher;
3845
import com.facebook.react.uimanager.util.ReactFindViewUtil;
46+
import java.util.ArrayList;
3947
import java.util.HashMap;
48+
import java.util.List;
4049

4150
/**
4251
* Utility class that handles the addition of a "role" for accessibility to either a View or
4352
* AccessibilityNodeInfo.
4453
*/
45-
public class ReactAccessibilityDelegate extends AccessibilityDelegateCompat {
54+
public class ReactAccessibilityDelegate extends ExploreByTouchHelper {
4655

4756
private static final String TAG = "ReactAccessibilityDelegate";
4857
public static final String TOP_ACCESSIBILITY_ACTION_EVENT = "topAccessibilityAction";
@@ -59,6 +68,9 @@ public class ReactAccessibilityDelegate extends AccessibilityDelegateCompat {
5968
sActionIdMap.put("decrement", AccessibilityActionCompat.ACTION_SCROLL_BACKWARD.getId());
6069
}
6170

71+
private final View mView;
72+
private final AccessibilityLinks mAccessibilityLinks;
73+
6274
private Handler mHandler;
6375

6476
/**
@@ -179,8 +191,10 @@ public static AccessibilityRole fromValue(@Nullable String value) {
179191
private static final String STATE_SELECTED = "selected";
180192
private static final String STATE_CHECKED = "checked";
181193

182-
public ReactAccessibilityDelegate() {
183-
super();
194+
public ReactAccessibilityDelegate(
195+
final View view, boolean originalFocus, int originalImportantForAccessibility) {
196+
super(view);
197+
mView = view;
184198
mAccessibilityActionsMap = new HashMap<Integer, String>();
185199
mHandler =
186200
new Handler() {
@@ -190,6 +204,14 @@ public void handleMessage(Message msg) {
190204
host.sendAccessibilityEvent(AccessibilityEvent.TYPE_VIEW_SELECTED);
191205
}
192206
};
207+
208+
// We need to reset these two properties, as ExploreByTouchHelper sets focusable to "true" and
209+
// importantForAccessibility to "Yes" (if it is Auto). If we don't reset these it would force
210+
// every element that has this delegate attached to be focusable, and not allow for
211+
// announcement coalescing.
212+
mView.setFocusable(originalFocus);
213+
ViewCompat.setImportantForAccessibility(mView, originalImportantForAccessibility);
214+
mAccessibilityLinks = (AccessibilityLinks) mView.getTag(R.id.accessibility_links);
193215
}
194216

195217
@Nullable View mAccessibilityLabelledBy;
@@ -388,18 +410,6 @@ public static void setRole(
388410
nodeInfo.setClassName(AccessibilityRole.getValue(role));
389411
if (role.equals(AccessibilityRole.LINK)) {
390412
nodeInfo.setRoleDescription(context.getString(R.string.link_description));
391-
392-
if (nodeInfo.getContentDescription() != null) {
393-
SpannableString spannable = new SpannableString(nodeInfo.getContentDescription());
394-
spannable.setSpan(new URLSpan(""), 0, spannable.length(), 0);
395-
nodeInfo.setContentDescription(spannable);
396-
}
397-
398-
if (nodeInfo.getText() != null) {
399-
SpannableString spannable = new SpannableString(nodeInfo.getText());
400-
spannable.setSpan(new URLSpan(""), 0, spannable.length(), 0);
401-
nodeInfo.setText(spannable);
402-
}
403413
} else if (role.equals(AccessibilityRole.IMAGE)) {
404414
nodeInfo.setRoleDescription(context.getString(R.string.image_description));
405415
} else if (role.equals(AccessibilityRole.IMAGEBUTTON)) {
@@ -445,16 +455,233 @@ public static void setRole(
445455
}
446456
}
447457

448-
public static void setDelegate(final View view) {
458+
public static void setDelegate(
459+
final View view, boolean originalFocus, int originalImportantForAccessibility) {
449460
// if a view already has an accessibility delegate, replacing it could cause
450461
// problems,
451462
// so leave it alone.
452463
if (!ViewCompat.hasAccessibilityDelegate(view)
453464
&& (view.getTag(R.id.accessibility_role) != null
454465
|| view.getTag(R.id.accessibility_state) != null
455466
|| view.getTag(R.id.accessibility_actions) != null
456-
|| view.getTag(R.id.react_test_id) != null)) {
457-
ViewCompat.setAccessibilityDelegate(view, new ReactAccessibilityDelegate());
467+
|| view.getTag(R.id.react_test_id) != null
468+
|| view.getTag(R.id.accessibility_links) != null)) {
469+
ViewCompat.setAccessibilityDelegate(
470+
view,
471+
new ReactAccessibilityDelegate(view, originalFocus, originalImportantForAccessibility));
472+
}
473+
}
474+
475+
// Explicitly re-set the delegate, even if one has already been set.
476+
public static void resetDelegate(
477+
final View view, boolean originalFocus, int originalImportantForAccessibility) {
478+
ViewCompat.setAccessibilityDelegate(
479+
view,
480+
new ReactAccessibilityDelegate(view, originalFocus, originalImportantForAccessibility));
481+
}
482+
483+
@Override
484+
protected int getVirtualViewAt(float x, float y) {
485+
if (mAccessibilityLinks == null
486+
|| mAccessibilityLinks.size() == 0
487+
|| !(mView instanceof TextView)) {
488+
return INVALID_ID;
489+
}
490+
491+
TextView textView = (TextView) mView;
492+
if (!(textView.getText() instanceof Spanned)) {
493+
return INVALID_ID;
494+
}
495+
496+
Layout layout = textView.getLayout();
497+
if (layout == null) {
498+
return INVALID_ID;
499+
}
500+
501+
x -= textView.getTotalPaddingLeft();
502+
y -= textView.getTotalPaddingTop();
503+
x += textView.getScrollX();
504+
y += textView.getScrollY();
505+
506+
int line = layout.getLineForVertical((int) y);
507+
int charOffset = layout.getOffsetForHorizontal(line, x);
508+
509+
ClickableSpan clickableSpan = getFirstSpan(charOffset, charOffset, ClickableSpan.class);
510+
if (clickableSpan == null) {
511+
return INVALID_ID;
512+
}
513+
514+
Spanned spanned = (Spanned) textView.getText();
515+
int start = spanned.getSpanStart(clickableSpan);
516+
int end = spanned.getSpanEnd(clickableSpan);
517+
518+
final AccessibilityLinks.AccessibleLink link = mAccessibilityLinks.getLinkBySpanPos(start, end);
519+
return link != null ? link.id : INVALID_ID;
520+
}
521+
522+
@Override
523+
protected void getVisibleVirtualViews(List<Integer> virtualViewIds) {
524+
if (mAccessibilityLinks == null) {
525+
return;
526+
}
527+
528+
for (int i = 0; i < mAccessibilityLinks.size(); i++) {
529+
virtualViewIds.add(i);
530+
}
531+
}
532+
533+
@Override
534+
protected void onPopulateNodeForVirtualView(
535+
int virtualViewId, @NonNull AccessibilityNodeInfoCompat node) {
536+
// If we get an invalid virtualViewId for some reason (which is known to happen in API 19 and
537+
// below), return an "empty" node to prevent from crashing. This will never be presented to
538+
// the user, as Talkback filters out nodes with no content to announce.
539+
if (mAccessibilityLinks == null) {
540+
node.setContentDescription("");
541+
node.setBoundsInParent(new Rect(0, 0, 1, 1));
542+
return;
543+
}
544+
545+
final AccessibilityLinks.AccessibleLink accessibleTextSpan =
546+
mAccessibilityLinks.getLinkById(virtualViewId);
547+
if (accessibleTextSpan == null) {
548+
node.setContentDescription("");
549+
node.setBoundsInParent(new Rect(0, 0, 1, 1));
550+
return;
551+
}
552+
553+
node.setContentDescription(accessibleTextSpan.description);
554+
node.addAction(AccessibilityNodeInfoCompat.ACTION_CLICK);
555+
node.setBoundsInParent(getBoundsInParent(accessibleTextSpan));
556+
node.setRoleDescription(mView.getResources().getString(R.string.link_description));
557+
node.setClassName(AccessibilityRole.getValue(AccessibilityRole.BUTTON));
558+
}
559+
560+
private Rect getBoundsInParent(AccessibilityLinks.AccessibleLink accessibleLink) {
561+
// This view is not a text view, so return the entire views bounds.
562+
if (!(mView instanceof TextView)) {
563+
return new Rect(0, 0, mView.getWidth(), mView.getHeight());
564+
}
565+
566+
TextView textView = (TextView) mView;
567+
Layout textViewLayout = textView.getLayout();
568+
if (textViewLayout == null) {
569+
return new Rect(0, 0, textView.getWidth(), textView.getHeight());
570+
}
571+
572+
Rect rootRect = new Rect();
573+
574+
double startOffset = accessibleLink.start;
575+
double endOffset = accessibleLink.end;
576+
double startXCoordinates = textViewLayout.getPrimaryHorizontal((int) startOffset);
577+
578+
final Paint paint = new Paint();
579+
AbsoluteSizeSpan sizeSpan =
580+
getFirstSpan(accessibleLink.start, accessibleLink.end, AbsoluteSizeSpan.class);
581+
float textSize = sizeSpan != null ? sizeSpan.getSize() : textView.getTextSize();
582+
paint.setTextSize(textSize);
583+
int textWidth = (int) Math.ceil(paint.measureText(accessibleLink.description));
584+
585+
int startOffsetLineNumber = textViewLayout.getLineForOffset((int) startOffset);
586+
int endOffsetLineNumber = textViewLayout.getLineForOffset((int) endOffset);
587+
boolean isMultiline = startOffsetLineNumber != endOffsetLineNumber;
588+
textViewLayout.getLineBounds(startOffsetLineNumber, rootRect);
589+
590+
int verticalOffset = textView.getScrollY() + textView.getTotalPaddingTop();
591+
rootRect.top += verticalOffset;
592+
rootRect.bottom += verticalOffset;
593+
rootRect.left += startXCoordinates + textView.getTotalPaddingLeft() - textView.getScrollX();
594+
595+
// The bounds for multi-line strings should *only* include the first line. This is because for
596+
// API 25 and below, Talkback's click is triggered at the center point of these bounds, and if
597+
// that center point is outside the spannable, it will click on something else. There is no
598+
// harm in not outlining the wrapped part of the string, as the text for the whole string will
599+
// be read regardless of the bounding box.
600+
if (isMultiline) {
601+
return new Rect(rootRect.left, rootRect.top, rootRect.right, rootRect.bottom);
602+
}
603+
604+
return new Rect(rootRect.left, rootRect.top, rootRect.left + textWidth, rootRect.bottom);
605+
}
606+
607+
@Override
608+
protected boolean onPerformActionForVirtualView(
609+
int virtualViewId, int action, @Nullable Bundle arguments) {
610+
return false;
611+
}
612+
613+
protected @Nullable <T> T getFirstSpan(int start, int end, Class<T> classType) {
614+
if (!(mView instanceof TextView) || !(((TextView) mView).getText() instanceof Spanned)) {
615+
return null;
616+
}
617+
618+
Spanned spanned = (Spanned) ((TextView) mView).getText();
619+
T[] spans = spanned.getSpans(start, end, classType);
620+
return spans.length > 0 ? spans[0] : null;
621+
}
622+
623+
public static class AccessibilityLinks {
624+
private final List<AccessibleLink> mLinks;
625+
626+
public AccessibilityLinks(ClickableSpan[] spans, Spannable text) {
627+
ArrayList<AccessibleLink> links = new ArrayList<>();
628+
for (int i = 0; i < spans.length; i++) {
629+
ClickableSpan span = spans[i];
630+
int start = text.getSpanStart(span);
631+
int end = text.getSpanEnd(span);
632+
// zero length spans, and out of range spans should not be included.
633+
if (start == end || start < 0 || end < 0 || start > text.length() || end > text.length()) {
634+
continue;
635+
}
636+
637+
final AccessibleLink link = new AccessibleLink();
638+
link.description = text.subSequence(start, end).toString();
639+
link.start = start;
640+
link.end = end;
641+
642+
// ID is the reverse of what is expected, since the ClickableSpans are returned in reverse
643+
// order due to being added in reverse order. If we don't do this, focus will move to the
644+
// last link first and move backwards.
645+
//
646+
// If this approach becomes unreliable, we should instead look at their start position and
647+
// order them manually.
648+
link.id = spans.length - 1 - i;
649+
links.add(link);
650+
}
651+
mLinks = links;
652+
}
653+
654+
@Nullable
655+
public AccessibleLink getLinkById(int id) {
656+
for (AccessibleLink link : mLinks) {
657+
if (link.id == id) {
658+
return link;
659+
}
660+
}
661+
662+
return null;
663+
}
664+
665+
@Nullable
666+
public AccessibleLink getLinkBySpanPos(int start, int end) {
667+
for (AccessibleLink link : mLinks) {
668+
if (link.start == start && link.end == end) {
669+
return link;
670+
}
671+
}
672+
673+
return null;
674+
}
675+
676+
public int size() {
677+
return mLinks.size();
678+
}
679+
680+
private static class AccessibleLink {
681+
public String description;
682+
public int start;
683+
public int end;
684+
public int id;
458685
}
459686
}
460687
}

0 commit comments

Comments
 (0)