Compose Multiplatform on iOS doesn't have the proper accessibility hierarchy

[GuilhE]

Is your feature request related to a problem? Please describe.
Adding support for Compose Multiplatform would be a valuable enhancement, especially considering JetBrains's significant investment in it.

Describe the solution you'd like
Same functionality we have when using Jetpack Compose and Swift UI:

• Faster evaluation time
• Possibility to distinguish widgets on Maestro Studio
• Stability

Additional context
You can test it by running my sample and compare the time it takes to evaluate for instance:

• assertVisible: Stand by
• tapOn: point: 50%,81% (the simulator gets the command but the Studio does not stream)

using Maestro Studio on Swift UI and Compose screen with an iOS Simulator.
I'm also experiencing random crashs when navigating between screens when Maestro Studio is connected with the iOS Simulator:

Navigate to http://localhost:9999 in your browser to open Maestro Studio. Ctrl-C to exit.
io.ktor.util.cio.ChannelWriteException: Cannot write to a channel
        at io.ktor.server.netty.cio.NettyHttpResponsePipeline.respondWithFailure(NettyHttpResponsePipeline.kt:102)
        at io.ktor.server.netty.cio.NettyHttpResponsePipeline.respondWithBodyAndTrailerMessage(NettyHttpResponsePipeline.kt:252)
        at io.ktor.server.netty.cio.NettyHttpResponsePipeline.access$respondWithBodyAndTrailerMessage(NettyHttpResponsePipeline.kt:26)
        at io.ktor.server.netty.cio.NettyHttpResponsePipeline$respondWithBodyAndTrailerMessage$1.invokeSuspend(NettyHttpResponsePipeline.kt)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.ktor.server.netty.EventLoopGroupProxy$Companion.create$lambda$1$lambda$0(NettyApplicationEngine.kt:291)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.netty.channel.StacklessClosedChannelException
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object, ChannelPromise)(Unknown Source)
        Error: Request for viewHierarchy failed, because of app crash, body: {"errorMessage":"Error getting main window kAXErrorCannotComplete","code":"internal"}
maestro.MaestroException$AppCrash: App crashed or stopped while executing flow, please check diagnostic logs: ~/Library/Logs/DiagnosticReports directory
        at maestro.drivers.IOSDriver.runDeviceCall(IOSDriver.kt:482)
        at maestro.drivers.IOSDriver.contentDescriptor(IOSDriver.kt:142)
        at maestro.ViewHierarchy$Companion.from-c1iYVAs(ViewHierarchy.kt:29)
        at maestro.Maestro.viewHierarchy-prqvCes(Maestro.kt:405)
        at maestro.studio.DeviceService.getDeviceScreen(DeviceService.kt:172)
        at maestro.studio.DeviceService.access$getDeviceScreen(DeviceService.kt:43)
        at maestro.studio.DeviceService$routes$3$1.invokeSuspend(DeviceService.kt:82)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:42)
        at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:95)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)

To replicate:

1. Tap on COMPOSE
2. Tap on Back
3. repeat until it crashes

When the Maestro Studio is not connected it will not crash.

[Alaksion]

@GuilhE Have you managed to get maestro test working on IOS devices? I'm trying to write a workflow to run my CMP app on IOS but it looks like the composition content is not accessible

[Alaksion]

Looks like the accessibility indexes can be interacted with when accessibilitySyncOptions is set to Always inside the MainViewController composable. But even then I cannot interact with the elements using actions

[bartekpacia]

Hello @GuilhE! Thanks for creating this issue.

What's exactly the problem here? Maestro works by querying the accessibility hierarchy, so as long as your app does this correctly (and afaik, Compose does a good job at it), it should work.

possibly related: #1746

[GuilhE]

Hello @bartekpacia
What I mean by support for Compose Multiplatform (CMP) I was hopping that my compose elements would be selectable like the ones in SwiftUI are. Here's what I mean:

demo.mp4

If I run maestro studio on Android simulator I can select them. If you want to try use this.

[bartekpacia]

Thanks for sharing this video, it's very helpful. I see the problem now. It seems that Compose is for some reason not doing good job at accessibility.

Does the same problem occur on Android?

[GuilhE]

On Android, same composables, are selectable.

[bartekpacia]

Thanks. Labeling it as iOS-only.

If you're up to that, I'd also like to ask you to report this issue to the Compose repo. This looks like a problem on their side. You can attach output of maestro hierarchy on both Android and iOS and say that you expect them to be the same, but they're not.

I see in your repo that you're using the latest dev build of Compose: v1.7.0-dev1721. So accessibility should work fine, but it doesn't.

[GuilhE]

How is the view hierarchy implemented on the Maestro side? I'm unsure if this should be considered a bug from Maestro or CMP, as I expect the output to differ across platforms. On Android, there's the View system, which isn't present on iOS, web and desktop. Should Maestro perhaps adjust its approach regarding layout scanning? 🤔

Maestro also fails to detect Web Apps made in CMP (Wasm):

I consider this a feature request an not a bug, but perhaps I'm wrong. Compose Multiplatform outside Android is not detected on Maestro.

[bartekpacia]

How is the view hierarchy implemented on the Maestro side?

It's using UIAutomator for Android, and XCUITest for iOS. It shouldn't matter if your app is built with native iOS/Android framework, or Compose Multiplatform, or anything different. As long as your app has accessibility, Maestro will (or should) query that.

I think the video you showed here is a great demo of the problem and would make a good issue on JetBrains issue tracker for Compose Multiplatform.

[GuilhE]

I can do it, no problem 😊, but it might be more effective if you did it, since you have more knowledge about the intricacies of Maestro to better answer incoming questions. You can use the resources I've shared here.

[bartekpacia]

I'd be so grateful if you did it! There's a lot of stuff going on in our issue tracker, every help is much appreciated:)

[GuilhE]

There you go: https://youtrack.jetbrains.com/issue/CMP-5635/Compose-Multiplatform-iOS-accessibility-tree

[GuilhE]

The take way is, to use Maestro to inspect Compose Multiplatform on iOS, we have 2 options:

1 - if using physical devices, turn on VoiceOver.
2 - if using simulators add the accessibilitySyncOptions = Always(null) in the iOS view controller file: composeApp/src/iosMain/kotlin/MainViewController.kt (or similar):

import androidx.compose.runtime.ExperimentalComposeApi
import androidx.compose.ui.platform.AccessibilitySyncOptions
import androidx.compose.ui.window.ComposeUIViewController
import platform.UIKit.UIViewController
import com.example.example.YourApp

@OptIn(ExperimentalComposeApi::class)
@Suppress("FunctionNaming")
fun MainViewController(): UIViewController {
    return ComposeUIViewController(
        configure = { accessibilitySyncOptions = AccessibilitySyncOptions.Always(null) },
        content = { YourApp() }
    )
}

demo.mp4

[GuilhE]

In version 1.8.0 we don't need to add those accessibilitySyncOptions:

AccessibilitySyncOptions removed. The accessibility tree is built on demand

Remove AccessibilitySyncOptions. Make the accessibility tree load lazily. by ASalavei · Pull Request #1780 · JetBrains/compose-multiplatform-core
Make the accessibility tree load lazily. The tree will load completely after the first request from the iOS accessibility engine, and dispose when the screen reader no longer interacts with it.