Unable to Access Certain UI Elements on Windows 11

[Jeomon]

I am currently using the uiautomation library on Windows 11 to automate tasks and interact with UI elements. However, I am facing significant limitations when attempting to access certain UI elements. Specifically:

1. Start Menu:

   • The library cannot detect or interact with the icons in the Start Menu, including those under "Pinned" or "Recommended".

2. Dropdown Menus:

   • In applications like Notepad, when clicking on a menu item (e.g., "File"), the elements within the dropdown menu are not recognized or accessible.

I am exploring alternatives and came across WinAppDriver. Before diving deeper, I wanted to confirm:

• Does WinAppDriver provide access to all interactive UI elements on the Windows desktop, including Start Menu items and dropdown menus in applications like Notepad?
• Are there any specific configurations or workarounds required to achieve this functionality on Windows 11?

Environment Details:

• OS: Windows 11
• Library: uiautomation (current solution)
• Target Applications: Start Menu, Notepad, and potentially other system apps with similar UI accessibility issues.

I would appreciate any insights or suggestions about whether WinAppDriver can take care of these limitations or if there are other ways to access these elements.

[anunay1]

Are the elements visible in inspect.exe ?

[Jeomon]

I used Accessibility Insights but here the thing is when I click on the start button the menu appears but they don't show the icons inside I mean it is empty but when I click on the Pinned or the Recommended it will show the list of icons inside those sections similar for the notepad case.

[anunay1]

Check with inspect.exe

[Jeomon]

I tried Inspect.exe and same like Accessibility Insights when I press the start button at that I can't see nothing inside the start menu in the inspect.exe but when I click on near regions of pinned or recommended then those icons will show in the inspect.exe

[anunay1]

That is how inspect works.

[Jeomon]

My use case is to create an accessibility tree. When we click on the Start menu, we need to get all the icons inside the Start menu. Can we do that with WinAppDriver?

Similarly, if I am automating the Notepad workflow when I click on the file menu, I need to get the buttons in that dropdown to automate.

Thank you for replying so far. 😊

[anunay1]

yes its possible just use the xpath locator with * or get it in a list with classname.

ele = driver.findelementbyname("filemenu")
buttons = ele.findelementsbyclassname("buttons").tolist()

[Jeomon]

If you don't mind could you please guide me how to use the driver if possible could you plz share the code to load the driver
I know about launching the server but don't find any videos to write python code to use the driver and start creating the accessibility tree

[anunay1]

Check

https://github.com/anunay1/SampleWADPython

[Jeomon]

from appium import webdriver

Desired capabilities for Windows desktop automation

desired_caps = {
"platformName": "Windows", # Platform is Windows
"deviceName": "WindowsPC", # You can give a custom device name here
"app": "Root" # The "Root" app launches the Windows desktop
}

Initialize the WebDriver with the desired capabilities

driver = webdriver.Remote(
command_executor='http://127.0.0.1:4723/wd/hub', # Make sure Appium server is running at this address
desired_capabilities=desired_caps
)

Get and print window handles

print(driver.window_handles)

Quit the driver after use

driver.quit()

this is a simple version which I used for now but I have an error it saying
TypeError: WebDriver.init() got an unexpected keyword argument 'desired_capabilities'

[Jeomon]

Could you please tell me which version of appium windows client is used