Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ClassNotFoundException - JLOptions #2

Closed
cnuernber opened this issue Sep 15, 2021 · 35 comments
Closed

ClassNotFoundException - JLOptions #2

cnuernber opened this issue Sep 15, 2021 · 35 comments

Comments

@cnuernber
Copy link
Owner

cnuernber commented Sep 15, 2021 •
edited
Loading

From the Julia discourse:

From leinegen repl I’m using -

(require '[libjulia-clj.julia :as julia])

What I’ve done -

Installed Julia 1.5.4, julia_home is set to point to the install directory
Used git to download LibJulia project
Used from LibJulia directory
lein repl

At user prompt used:

user> (require '[libjulia-clj.julia :as julia])

I’m getting:

Syntax error (ClassNotFoundException) compiling at (libjulia_clj\impl\jna.clj:1:1).
julia_clj.JLOptions

Note - I found that Julia 1.5.4 does have the Options.jl library. My path for
julia_home is pointing to Julia 1.5.4 install directory. I have not checked for
later Julia install on my system. My julia_home is set to point to Julia 1.5.4
install directory.
@cnuernber
Copy link
Owner Author

JLOptions is a class that is included with the jar and based on C struct definition exposed via julia.h. It describes the command line options that libjulia was initialized with.

There is a new ffi system that is much cleaner and more portable than the pure jna-based system - it would be nice to upgrade libjulia-clj to that version.

I think for this case there is a dependency conflict. Could you post your project.clj and the output lein deps :tree?

@jdwright007
Copy link

My project.clj is shown below. I only removed the

:java-source-paths ["java"]

line since with that I had a build issue. I'm using a binary
Julia 1.5.4 version that I did not build from source. Also, I'm
on a Window cpu. Both clojure and julia are working well
and environmental variables/constants are set correctly for
these environments.

I'm new to both clojure and julia so I don't really know my
way around these too much as yet. I'd like to get a Java to Julia
interface working. I tried another api, a direct java to julia api,
and got that working successfully. I'd also like to get your api
working too. Then I can weigh the benefits of each. I can use
the clojure to julia interface when needed by my requirements.

(defproject cnuernber/libjulia-clj "0.08-SNAPSHOT"
:description "Experimental Julia bindings for Clojure."
:url "https://github.com/cnuernber/libjulia-clj"
:license {:name "EPL-2.0"
:url "https://www.eclipse.org/legal/epl-2.0/"}
:dependencies [[org.clojure/clojure "1.10.1"]
[cnuernber/dtype-next "6.00-beta-16"]
[techascent/tech.jna "4.05"]]
;;sane logging please
:profiles {:dev {:dependencies [[ch.qos.logback/logback-classic "1.2.3"]]}
:openj9 {:java-cmd "scripts/openj9-java"
:jvm-opts ["-Xsigchain" "-Xrs"]}
:codox {:dependencies [[codox-theme-rdash "0.1.2"]]
:plugins [[lein-codox "0.10.7"]]
:codox {:project {:name "libjulia-clj"}
:metadata {:doc/format :markdown}
:themes [:rdash]
:source-paths ["src"]
:output-path "docs"
:doc-paths ["topics"]
:source-uri "https://github.com/cnuernber/libjulia-clj/blob/master/{filepath}#L{line}"
:namespaces [libjulia-clj.julia
libjulia-clj.modules.Base
libjulia-clj.modules.Core
libjulia-clj.modules.LinearAlgebra
libjulia-clj.modules.DifferentialEquations]}}}
:aliases {"codox" ["with-profile" "codox,dev" "codox"]})

  • John W

@jdwright007
Copy link

Here's my lein deps :tree -

PS C:\java_tools\julia\workspace\libjulia-clj> lein deps :tree
Possibly confusing dependencies found:
[org.clojure/clojure "1.10.1"]
overrides
[techascent/tech.jna "4.05"] -> [techascent/tech.resource "5.02"] -> [org.clojure/clojure "1.10.2-alpha1"]
and
[cnuernber/dtype-next "6.00-beta-16"] -> [techascent/tech.resource "5.02"] -> [org.clojure/clojure "1.10.2-alpha1"]
and
[cnuernber/dtype-next "6.00-beta-16"] -> [org.clojure/clojure "1.10.2-alpha1"]

Consider using these exclusions:
[techascent/tech.jna "4.05" :exclusions [org.clojure/clojure]]
[cnuernber/dtype-next "6.00-beta-16" :exclusions [org.clojure/clojure]]

[org.clojure/clojure "1.10.1"] -> [org.clojure/spec.alpha "0.2.176"]
overrides
[techascent/tech.jna "4.05"] -> [techascent/tech.resource "5.02"] -> [org.clojure/clojure "1.10.2-alpha1"] -> [org.clojure/spec.alpha "0.2.187"]
and
[cnuernber/dtype-next "6.00-beta-16"] -> [techascent/tech.resource "5.02"] -> [org.clojure/clojure "1.10.2-alpha1"] -> [org.clojure/spec.alpha "0.2.187"]
and
[cnuernber/dtype-next "6.00-beta-16"] -> [org.clojure/clojure "1.10.2-alpha1"] -> [org.clojure/spec.alpha "0.2.187"]

Consider using these exclusions:
[techascent/tech.jna "4.05" :exclusions [org.clojure/spec.alpha]]
[cnuernber/dtype-next "6.00-beta-16" :exclusions [org.clojure/spec.alpha]]

[ch.qos.logback/logback-classic "1.2.3" :scope "test"]
[ch.qos.logback/logback-core "1.2.3" :scope "test"]
[org.slf4j/slf4j-api "1.7.25" :scope "test"]
[clojure-complete "0.2.5" :exclusions [[org.clojure/clojure]]]
[cnuernber/dtype-next "6.00-beta-16"]
[camel-snake-kebab "0.4.2"]
[insn "0.4.0"]
[it.unimi.dsi/fastutil "8.2.1"]
[org.apache.commons/commons-math3 "3.6.1"]
[org.ow2.asm/asm "7.1"]
[org.roaringbitmap/RoaringBitmap "0.9.0"]
[org.roaringbitmap/shims "0.9.0" :scope "runtime"]
[org.xerial.larray/larray-mmap "0.4.1"]
[org.xerial.larray/larray-buffer "0.4.1"]
[primitive-math "0.1.6"]
[techascent/tech.resource "5.02"]
[org.clojure/tools.logging "1.1.0"]
[nrepl "0.8.3" :exclusions [[org.clojure/clojure]]]
[org.clojure/clojure "1.10.1"]
[org.clojure/core.specs.alpha "0.2.44"]
[org.clojure/spec.alpha "0.2.176"]
[techascent/tech.jna "4.05"]
[net.java.dev.jna/jna "5.6.0"]
PS C:\java_tools\julia\workspace\libjulia-clj>

@mkitti
Copy link

mkitti commented Sep 16, 2021

Following. I'll see if I can take a closer look this weekend.

@cnuernber
Copy link
Owner Author

cnuernber commented Sep 16, 2021 •
edited
Loading

So, the issue is that the libjulia project is a mixed java/clj project. Be removing the java-source-paths from the project file you are no longer compiling the java classes - one of which is JLOptions.java.

What happens when you try the clojars release of libjulia-clj?

I added an activate-julia script and updated the example project to use the clojars julia jar file and it produced the fractal correctly so I think the issue is a configuration issue.

@jdwright007
Copy link

I downloaded source for both java (11-15u) and julia (1.6.2) and neither one had the
JLOptions.java. I tried a lein build with the clojars release of libjulia-clj and this did not
work. I don't know what download to use that has the JLOptions.java.

I'm looking forward to getting this worked out so that I can work on combined
java/julia projects (that have a java/clojure/julia interface).

  • John W

@mkitti
Copy link

mkitti commented Sep 18, 2021

JLOptions.java is at https://github.com/cnuernber/libjulia-clj/blob/master/java/julia_clj/JLOptions.java

@jdwright007
Copy link

jdwright007 commented Oct 23, 2021 via email

@cnuernber
Copy link
Owner Author

cnuernber commented Oct 23, 2021 •
edited
Loading

I think what we should do is try out the test application and see what happens.

There is a test app that uses julia to efficiently compute a fractal.

Once you have JULIA_HOME exported and the clojure command line system installed, go into the examples/fractal directory.

(base) chrisn@chrisn-lt3:~/dev/cnuernber/libjulia-clj/examples/fractal$ clj
Clojure 1.10.3
(require '[julia-fractals :as jf])
nil
(require '[tech.v3.libs.buffered-image :as bufimg])
nil
(def img (jf/fractal))
#'user/img
img
#object[java.awt.image.BufferedImage 0x4f74edbf "BufferedImage@4f74edbf: type = 10 ColorModel: #pixelBits = 8 numComponents = 1 color space = java.awt.color.ICC_ColorSpace@3c1f2651 transparency = 1 has alpha = false isAlphaPre = false ByteInterleavedRaster: width = 1920 height = 1080 #numDataElements 1 dataOff[0] = 0"]
(bufimg/save! img "test.png")
true
user=>
(base) chrisn@chrisn-lt3:~/dev/cnuernber/libjulia-clj/examples/fractal$

This is a very simple example that should work as is mainly because we don't allocate memory in julia threads.

When you want to do something more involved we have to preload the jsig library which unfortunately I do not know how to do on windows but we can figure that out once we get this far.

What happens if you try this pathway?

@cnuernber
Copy link
Owner Author

cnuernber commented Oct 23, 2021 •
edited
Loading

That all being said, I do not see the jl_options member exported from the shared library any more.

@mkitti - nm -D doesn't show jl_options as being exported any more which is odd because the header shows it as exported. I think perhaps my usage of it, in that I set things directly is just not the right way to do it. I used it to disable threads and such here.

Should I be using a different function? One thought I had was to use the normal command line pathway to do this -

JL_DLLEXPORT void jl_parse_opts(int *argcp, char ***argvp);

I basically enable/disable signals, set num threads and set the optimization level.

@mkitti
Copy link

mkitti commented Oct 23, 2021

doesn't show jl_options as being exported any more which is odd because the header shows it as exported. I think perhaps my usage of it, in that I set things directly is just not the right way to do it. I used it to disable threads and such here.

See JuliaLang/julia#42416

@cnuernber
Copy link
Owner Author

OK, so then @jdwright007 was partially correct, this version of libjulia won't work with latest julia. I tested all this with 1.5.4.

@mkitti
Copy link

mkitti commented Oct 23, 2021

It looks like it was backported to Julia 1.7-rc2 which is available for binary download, so that might work?
https://julialang.org/downloads/

@cnuernber
Copy link
Owner Author

Perhaps, thanks for the pointer I will check it out.

@jdwright007
Copy link

jdwright007 commented Oct 23, 2021 via email

@cnuernber
Copy link
Owner Author

I just checked in a change while I was writing those instructions. If you get latest that should allow the above section to work as it is pure clojure.

@jdwright007
Copy link

jdwright007 commented Oct 24, 2021 via email

@jdwright007
Copy link

jdwright007 commented Oct 24, 2021 via email

@cnuernber
Copy link
Owner Author

cnuernber commented Oct 24, 2021 •
edited
Loading

The command line pathway like I showed you from the example directory won't work for the main project. That is a leiningen based project so the equivalent command for leiningen based projects is lein repl. I am trying 17-rc presently and we will see.

Leiningen based projects have project.clj in the root path, clojure command line based projects have deps.edn in the base path. We are all moving to deps.edn as it avoids a ton of issues that sprung up over time with leiningen based projects.

The example is how I recommend new projects being setup. Or, if you are using a primarily java project, just including the jar from the deps.edn dependencies in the example deps.edn.

I need to spend some time with this project - I have a new FFI layer that is much cleaner and supports multiple FFI backends including jdk-17 and java's native executable pathway, graal native. Upgrading to that backend will allow me to get rid of the java in the project completely and move this project to be a deps.edn-based project similar to the example fractal.

@cnuernber
Copy link
Owner Author

I just tested with 17-rc2 and that failed also. I made a branch and will attempt to upgrade libjulia-clj to latest of all the toys - will be in touch.

@jdwright007
Copy link

jdwright007 commented Oct 24, 2021 via email

@jdwright007
Copy link

jdwright007 commented Oct 25, 2021 via email

@cnuernber
Copy link
Owner Author

cnuernber commented Oct 25, 2021 •
edited
Loading

Thanks for your kind words!! I feel a bit like a prophet in the wilderness a lot of times with this stuff. I would also use Julia to do strict numerical computation such as probabilistic programming. It is hard to explain but Julia code is more composable in a lot of ways and nothing on the JVM seriously competes with Julia in terms of raw performance of straight line numeric code. Julia's linear algebra library and GPU bindings along with it's sophisticated runtime type-based dispatch and compilation system are really unique and I think complement any JVM language. The fractal example, for instance, is much faster done in Julia out of the box and serious algorithms like umap are faster than their carefully written numba versions.

There is a new version of the library and examples/fractal is updated and has a more thorough readme.

For some resources in learning Clojure specifically another project I am involved with and that is based on the same ffi system is libpython-clj. That project has a new to clojure section.

There are some gotchas and this all is pretty untested. Here are some things.

  • Julia is column major while tech.v3.tensor is row-major. So a jvm tensor will be transposed in Julia, something I take advantage of in my kmeans-based mnist demo.
  • You cannot call Clojure from a Julia task. This is because in order to implement green threads in Julia they do stack manipulations and the java JNI pathway expects the stack to look a certain way thus calling Java from Julia task usually hangs. @mkitti - dtype-next now supports jdk-17's new ffi system, the foreign incubator module. This is likely to have different ramifications and is something we can test out at a later date but I know it has faster base inter-language function calls.
  • For some platforms you have to enable libjsig. This is documented in the signals topic.

When you have time could you try out the fractals demo again and see what happens? Specifically instead of calling jf/fractal which is pure JVM stuff call jf/jl-fractal and you should see the julia system initialize and generate the image.

@mkitti
Copy link

mkitti commented Oct 25, 2021

By the way, I'm anticipating some regression on macOS between Julia 1.7 and the JVM:
JuliaInterop/JavaCall.jl#151

There are some changes in Julia 1.6.3 and later that changes how Julia deals with signals:
JuliaLang/julia#40056

@cnuernber
Copy link
Owner Author

Interesting! If anything julia-1.7.0-rc2 is more stable:

(base) chrisn@chrisn-lt3:~/dev/cnuernber/libjulia-clj$ clj
Clojure 1.10.2
(require '[libjulia-clj.julia :as jl])
nil
(jl/initialize! {:enable-signals? false :n-threads -1})
Oct 25, 2021 2:58:44 PM clojure.tools.logging$eval3223$fn__3226 invoke
INFO: Reference thread starting
Oct 25, 2021 2:58:44 PM clojure.tools.logging$eval3223$fn__3226 invoke
INFO: julia library arguments: ["--handle-signals=no" "--threads" "12"]
:ok
(jl/jl "Threads.@threads for i in 1:1000; zeros(1024, 1024) .+ zeros(1024, 1024); end")
nothing
user=>
(System/gc)
nil
user=>

That same multithreaded code used to crash the system either during running the threaded code or when System/gc was called.

@cnuernber
Copy link
Owner Author

You still need jsig, however. Even with --handle-signals=no the java GC can cause a sigsegv and Julia will bail with message:

(base) chrisn@chrisn-lt3:~/dev/cnuernber/libjulia-clj$ clj
Clojure 1.10.2
(require '[libjulia-clj.julia :as jl])
nil
(jl/initialize! {:enable-signals? false :n-threads -1})
Oct 25, 2021 3:13:56 PM clojure.tools.logging$eval3223$fn__3226 invoke
INFO: Reference thread starting
Oct 25, 2021 3:13:56 PM clojure.tools.logging$eval3223$fn__3226 invoke
INFO: julia library arguments: ["--handle-signals=no" "--threads" "12"]
:ok
(jl/cycle-gc!)
nil
(jl/jl "Threads.@threads for i in 1:1000; zeros(1024, 1024) .+ zeros(1024, 1024); end")
nothing
(jl/cycle-gc!)
nil
(System/gc)
nil
user=>
rlwrap: warning: clojure crashed, killed by SIGSEGV (core dumped).
rlwrap itself has not crashed, but for transparency,
it will now kill itself with the same signal


warnings can be silenced by the --no-warnings (-n) option
Segmentation fault (core dumped)

@cnuernber
Copy link
Owner Author

@jdwright007 - Another option, since you are bridging languages anyway, is tensorflow probability.

@jdwright007
Copy link

jdwright007 commented Oct 26, 2021 via email

@cnuernber
Copy link
Owner Author

hmm - on windows it will be just julia.dll, not libjulia.dll. That I think is a bug in my code. What files ending in .dll are installed under julia?

@mkitti
Copy link

mkitti commented Oct 26, 2021

On Windows, it should be libjulia.dll and be in the same folder as julia.exe, in the "bin" directory.

image

Note that by Julia 1.8, what was once libjulia.dll will be broken up into

  • libjulia.dll
  • libjulia-internal.dll
  • libjulia-codegen.dll

@cnuernber
Copy link
Owner Author

Very interesting -- A public API published for each of them would be great :-).

@cnuernber
Copy link
Owner Author

@jdwright007 - In latest I updated the system to search both lib and bin dirs for the julia shared library. When you get a chance could you give it another try?

@cnuernber
Copy link
Owner Author

I should also note that for probabilistic programming in Clojure you have:

  • Anglican - grandaddy of lots of probprog systems.
  • Daphne - new hotness.

@jdwright007
Copy link

jdwright007 commented Oct 27, 2021 via email

@cnuernber
Copy link
Owner Author

OK, wow, that is great! Thanks for sticking with it these last few days and for the motivation to bring this project forward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@cnuernber @mkitti @jdwright007