Crashes on Windows on ARM #304
Comments
|
Now using the x64 version, emulated for ARM, drop-in replacement and no crash so far. |
|
Sounds similar to what has been discussed on #218 (comment). Probably it makes sense to have a distinct issue for that so I'm not closing this as duplicate. Considering I don't have aarch64 hardware to reproduce this I'm limited in how I can help here. It makes probably sense if you debug this further in your own but I can of course try to assist. Maybe it makes a difference that the ARM builds use libc++ instead of libstdc++. Another difference is that the ARM build uses QProcess instead of Boost.Process. |
|
Without a stacktrace this is hard to pin-down. So I recommend you create a debug build using MSYS2's mingw-w64 packaging and reproduce/debug the crash with gdb. You can find build instructions on the README of c++utilities (another repo on my GitHub page). If you paste a stacktrace here I can also help with the further investigation. |
|
I was able to compile the latest version of Boost for Windows on ARM with the Boost.Process library using the patch from msys2/MINGW-packages@da4f4c3#diff-35ebcd8b8f3e30bb84ddbfad90b6dd1ed6f063ae4b2197b55464081b736f5a28R85. I was also able to compile the latest Syncthing Tray release against it using Boost.Process. The builds that are supposed to be statically linked are unfortunately dynamically linked so one can't test the binaries easily at the moment. (The linking problem has been fixed in c++utilities and thus will be fixed on the next release. Maybe I can provide development builds in the meantime. I also came up with a change in my release scripts to prevent uploading binaries that depend on DLLs not provided by Windows itself.) So on the next release the ARM build will be closer to the x86_64 build; maybe that's already enough. (Somehow it suspect it is the use of libc++, though.) |
|
Currently running the arm build myself and have both VS devtools and mingw installed-I can try giving stuff a debug compile when i get the chance later. |
|
okay for whatever its worth it appears to have survived multiple sleep wake cycles w/ boost-process will let you know if/when it dies |
|
yeah i think that was it |
|
So it works with Boost.Process? That's good news. I only hope that my cross-compiled builds using static libraries will work as well. |
|
That it does. Feel free to shoot over a test build it you'd like. |
|
....that one crashed on first wake. for whatever its worth its not dying on debug attach so i should be able to get a proper trace. |
|
|
Probably my builds are done with more security hardening flags than your builds so a problem with the code leads to this hard crash. I have no idea what the problem with my code could be, though. The stracktrace doesn't point to a concrete function of my code or Qt. I'm not even sure where the thread comes from. Syncthing Tray has actually code that runs explicitly after the system has been resumed ( ) but I cannot see a problem with it and the stacktrace also doesn't point to this handler. The Qt documentation also says that this kind of event filter will be executed on the main thread so if it was the culprit the crash would have happened on the main thread.Those are the compile flags I used to compile Syncthing Tray and all of its dependencies: https://github.com/Martchus/PKGBUILDs/blob/master/environment/mingw-w64-clang/mingw-env.sh |
|
want me to rebuild on my end w/ your flags? |
|
You could try; I'm not sure whether it'll make a difference when just compiling Syncthing Tray itself with different flags. Maybe the compilation of Qt, other dependencies and mingw-w64 itself is also relevant. |
|
It looks like the use of (This page contains more information about possibly relevant flags: https://gist.github.com/alvinhochun/a65e4177e2b34d551d7ecb02b55a4b0a) |
it won't even let me 😅 |
|
This stack trace doesn't even leave windows DLLs. Unless it is corrupted main wasn't even called. You might want to report it to one of the Microsoft forums, not sure which one. |
|
Maybe |
|
Seems a bit weird then that it would cause this error. |
|
Nevermind you're right. RtlpHandleInvalidUserCallTarget handles a call to an invalid function pointer. |
|
If you could read the registers it would be nice. The function pointer should still be there. |
|
I suppose the relevant WinAPI function where one would pass the callback is PowerRegisterSuspendResumeNotification (https://learn.microsoft.com/en-us/windows/win32/api/powerbase/nf-powerbase-powerregistersuspendresumenotification). I'm just wondering where this function is called. I've already checked Qt, relevant Boost libraries and LLVM but none of these code bases contain |
|
That is my suspicion. As long as this can be reproduced and you check where the pointer points to it should be done. |
|
on it. |
|
Here is a blog from a similar failure. https://devblogs.microsoft.com/oldnewthing/20240913-00/?p=110257 |
|
....there's no symbol. |
just kidding! what's interesting is that stepping through every instruction seems to indicate that that's the only non-ntdll attempted call? I get the feeling it's not anything to do with pcre2_get_error_message and that's just the garbage it landed on. |
|
I do have to wonder if the reason that clangarm64 won't let me use the fcf-protection flag for a native build is because this is known to not work. |
|
Looks probable. Here is a debian discussion for it. https://lists.debian.org/debian-devel/2022/10/msg00314.html. Also GCC docs document fcf-protection as using Intel CET. |
|
At least the flag when compiling mingw-w64 shouldn't hurt as MSYS2 enabled it as well. They also have tests using It also looks like llvm-mingw enables it for all archs: https://github.com/mstorsjo/llvm-mingw/blob/master/Dockerfile |
|
probably worth noting that mguard=cf, enable-cfguard, and fcf-protection are all different flags |
|
only one that clangarm is giving me trouble for using is fcf |
|
I know but I'm also not using fcf. Sorry for the confusion, I initially thought as well that fcf might be the culprit but it is not used at all in my builds as it is only used when the condition |
|
Bash syntax, love it. gonna see what happens if its a personal build w/o fcf then |
|
misaligned ldr/str offset? |
|
|
This looks curious: |
|
guess the crt as compiled by the msys guys isn't above board here, huh edit: can't find any issue reports, granted, so that seems unlikely |
|
The issue with Note that I rebuilt all my packages using llvm tooling. (The only package where I used binutils was boot but only for windmc. Other tooling was taken from llvm. I think it would have been impossible to use binutils anyways because the misalignment issue goes in the way very soon.) |
|
perhaps even more strange is that this wasn't happening until i started adding the hardening flags. edit: and compiler-rt/ld.lld so maybe i should try removing those first edit edit: same deal |
|
culprit appears to either be fno-plt or fexceptions, per testing edit: looks like its fno-plt, gonna double check by readding the other options edit edit: yup, testing now edit edit edit: cfguard fast fail crash |
that's....new. https://github.com/golang/go/blob/master/src/runtime/zcallback_windows_arm64.s |
|
Ok, so what flags exactly did you have to add to the compilation of what to reproduce a crash? And did you reproduce the power callback crash or only the one pointing to the Go runtime? Unfortunately I'm really not familiar with what these callback from the Go runtime to be able to shed some light on what's perhaps going on. Maybe you're onto something, though :-) |
Same flags minus -fno-plt. That's the function pointer present when the same powercallback trips on resume from sleep. |
|
And I assume you're only recompiling Syncthing Tray but none of the dependencies like Qt, libc++ or mingw-w64? If the compilation of Syncthing Tray alone makes a difference that's maybe a good sign (and I won't have to recompile everything to create a fixed build). To exclude Go you can of course build with |
|
Correct. Not sure how to get the output of a no_libsyncthing build to attach to a running copy of syncthing, though. edit: its in settings, ignore me. will update. |
|
seems its surviving sleep-wake with external syncthing. next testing your current release build. |
|
Juding by https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html#index-fno-plt and other information I found about I think I only added this flag because it is also used in regular x86 mingw-w64 builds (https://aur.archlinux.org/cgit/aur.git/tree/mingw-env.sh?h=mingw-w64-environment). It might just ended up being used in regular x86 mingw-w64 builds because we generally try to stay close to how things are done in native Arch Linux packaging (to have the same level of hardening and optimization). |
|
still dies. seems its a function of whether libsyncthing is compiled in rather than a function of whether its actually using it. (also interesting that this ones back to the libpcre call) |
To be fair this didn't affect the crashing (although it did change the called-back function to something that at least made some sense). It only affected the alignment error on native build. |
That doesn't surprise me. A few years ago I had crashes when trying to run an x86 Windows build with Wine. It always crashed when it was compiled with libsyncthing support, even though none of the Go functions were ever executed. (It worked natively under Windows, though.) |
|
I asked about this on the LLVM discord and got answers:
About debugging the linker errors:
|
|
I've just released 1.7.0 and for this release I removed support for the built-in Syncthing library in ARM builds. Maybe this makes a difference. |
|
Appears it survives sleep-wake. Dunno if that counts as solving the issue, but it certainly solves my usecase. |
|
I suppose then the culprit is really the Go runtime which doesn't support Windows on ARM in the way I'm building my binaries (with recommend build/hardening flags and where the Go runtime is linked into the final executable as static library). Then I'll keep the built-in library disabled for now. I wouldn't close the issue right now. I'll at least update the notes on the website and README and maybe there's an issue on the Go side I can watch and block on. |
Bug description
Every time I put my Surface Pro 11 to sleep and wake it up, the Syncthing Tray icon has disappeared and syncthing isn't running anymore. Doesn't happen with any other app.
Addition: it also just crashed after I right-clicked on the icon, so I guess it's not only after sleep.
Version: syncthingtray-1.6.4-aarch64-w64-mingw32
Qt 6.8.1
The text was updated successfully, but these errors were encountered: