Julia-lang: Extending a method from an installed module, "using" order - module

I'm trying to extend a method from the JSON.jl module, specifically I'm doing this (extending 'JSON.lower', as documented, to allow serialisation),
generally - I'm asking what are the current Julia "best practices" for extending a method from another module - especially - if this is a method in-change of some of some policy of the module (as in this case in JSON.jl).
Is there actually a way to guarantee the updated method will be used by the imported module, except for patching it?
Let's demonstrate both desired and sinister behaviour:
module TestProgram
using JSON
import JSON.lower # not sure whether this is needed
JSON.print(Set([1,2,1]))
## output:
## {"dict":{"2":null,"1":null}}
## not the output I want or expect
## in this case might be a good idea to patch JSON.jl and pull request
## and "JSON.print" - is now compiled with this default behaviour
JSON.lower{T}(v::Set{T}) = collect(v) # overloading 'JSON.lower'
println(json(Set([1,2,1])))
## output:
## [2,1]
JSON.print(Set([1,2,1]))
## output still:
## {"dict":{"2":null,"1":null}}
end # module
If anywhere during runtime before - some piece of code already ran this method, json(v::Set{T}) - then this method is already compiled, and will not use my extention of 'JSON.lower'
This actually happen to me and was hard to debug, as the json() call was behaving as expected, while JSON.print() was behaving differently (just by change in some run-flow)
Since any module (JSON.jl here) can be used by another package I'm using - I have no way of knowing my code actually runs first and actually extends a method correctly
So - what are the best practices to ensure that?
[Currently tested this behaviour with Julia 0.4.6 and 0.5.0]

Yes, your analysis is correct on Julia 0.5 and prior. Many developers know this issue number by heart (#265). A large amount of work went into fixing this for the upcoming version 0.6; Julia now tracks the callers of every function and re-compiles them as needed.
In general, though, the best advice here is to patch the library directly and push your change. This is true in both 0.5 and 0.6, even with the run-time correctness fix. It's advised that a you shouldn't extend imported functions with types that you didn't define yourself since it changes that behavior for all other packages that depend upon it. This has become to be colloquially referred to as "type piracy" since you're commandeering the method for your own purposes — purposes that may not suit other callers.
As a temporary workaround, you could try adding the definition to your ~/.juliarc.jl file, which will get executed at startup and has a better chance at being defined before any other methods get compiled. Even this isn't bullet-proof, though, since the package could use precompilation to speed up its usage.

Related

Implementing Custom Cache Store

Just implemented a custom cache store based on the official existing RocksDb one for a different backend store.
That leads me to a number of concerns/questions:
Found out the hard way that PersistenceContextInitializerImpl is auto-generated and had added an import from Eclipse to resolve the issue. Now I have to leave it non-imported and showing as an error in Eclipse, is there a best practice way to handle this?
Why is RocksDBDbStoreTest#testSegmentsRemovedAndAdded call when segmented is false, since this calls removeSegments that contractually should not be called if not segmented?
Same class, why is buildConfig numSegments set or larger than 1 for non-segmented test cases?
Any example of store implementing the NonBlockingStore transactional methods? Mostly wondering to make sure that all calls are from the same thread?
Wanted to disable the compatibility test, since not supported in prior versions. Changed group to unstable or manual and would always still get called, which doesn't seem to match documentation. What is the right way to disable it from build time run?
Are there any kind of performance/stress tests for persistence store that can be executed or adapted?
Found out the hard way that PersistenceContextInitializerImpl is auto-generated and had added an import from Eclipse to resolve the issue. Now I have to leave it non-imported and showing as an error in Eclipse, is there a best practice way to handle this?
There should be a way to have it run the annotation processor. We use IntelliJ and it works fine OOTB.
Why is RocksDBDbStoreTest#testSegmentsRemovedAndAdded call when segmented is false, since this calls removeSegments that contractually should not be called if not segmented?
This is just a side effect of having a parameterized test like that. In actual runtime it won't be invoked. You can just have the test ignore it if it is segmented.
if (segmented) return;
Same class, why is buildConfig numSegments set or larger than 1 for non-segmented test cases?
Infinispan data container is always segmented when running any of the cluster modes. The store however is not required to be segmented in those cases. If the store is not segmented you can ignore any segment parameters as documented.
Any example of store implementing the NonBlockingStore transactional methods? Mostly wondering to make sure that all calls are from the same thread?
You can see some at https://github.com/infinispan/infinispan/blob/main/persistence/jdbc/src/main/java/org/infinispan/persistence/jdbc/stringbased/JdbcStringBasedStore.java#L724. The methods can and will be invoked from different metehods, thus why it stores a Map keyed by the Transaction.
Wanted to disable the compatibility test, since not supported in prior versions. Changed group to unstable or manual and would always still get called, which doesn't seem to match documentation. What is the right way to disable it from build time run?
Not sure I understand the question. For your store you shouldn't need any compatibility test, so just don't copy that test file.
Are there any kind of performance/stress tests for persistence store that can be executed or adapted?
We have https://github.com/infinispan/infinispan-benchmarks/blob/main/cachestores/src/main/java/org/infinispan/jmhbenchmarks/InfinispanHolder.java that should work.

Elixir's module attributes in Phoenix production

As far as I know module attributes evaluated during compilation time. I was trying to follow this post about mocking API in Elixir:
defmodule Example.User do
#github_api Application.get_env(:example, :github_api)
def get(username) when is_binary(username) do
#github_api.make_request(:get, "/users/#{username}")
end
end
And I'm wondering if that's gonna work in production at all. As far as I understand when this module is compiled there's no access to the Application. So my question is: can I use module attributes to store some config values that come from Application.get_env?
You absolutely can. As long as the application was compiled using MIX_ENV specified to environment you want the application running under, and as long as that call evaluates to what you expect for that environment, it'll all work fine.
For a deeper look at how module attributes are affected by compilation for an almost identical case as what you've described, take a look at this blog post here.

Change REPL module/namespace in Julia

I'm looking for a way to "enter" a module in the REPL, so that I can access all symbols without qualification (not just the exported ones), and any function (re)defined at the REPL gets in the specified module. (Basically this is the functionality of Common Lisp's in-package macro.)
This would be useful in a REPL-oriented workflow, as I would be able to write the same code in the REPL as in the module I am developing.
The manual recommends a workflow where I qualify everything, but that seems annoying.
I started a package called REPLMods.jl for this a while back. It should probably be polished up, but I haven't had the time.
I spoke to core Julia members and there was interest in getting it merged into base once things were clean, but again, no time!
I know this isn't quite what you're asking, but just in case the 'obvious' had not occured to you (or future visitors to the question), assuming you loaded a module with an annoyingly cumbersome name, e.g.
import LaTeXStrings
and you don't want to have to type LaTeXStrings all the time just to explore its accessibles, i.e.
LaTeXStrings.[TAB]
you can just assign the imported module as a whole to another variable, i.e.
const l = LaTeXStrings
I'm sure in the absence of a more appropriate built-in solution, at least typing l.[TAB] as opposed to LaTeXStrings.[TAB]is a lot more tolerable :)
(I find it odd, in fact, that julia doesn't seem to support the import LaTeXStrings as l syntax ...)
It's 2020, I'm using Julia 1.4, and was unable to get REPLMods.jl to work. I think the following seem good enough for the time being:
ExportAll.jl - see Exporting all symbols in Julia for a discussion (just that one shouldn't ExportAll to replace normal export)
and Revise.jl

How do I determine what to import using gnulib-tool

autoscan produced a series of function checks (AC_FUNC_* and AC_CHECK_FUNCS), and AC_CHECK_HEADERS. Now I need to pull in gnulib code. Do I just try to import every function and header that autoscan identified? Half of them don't show up in the gnulib-tool --list output. Or do I wait until ./configure fails on some platform?
My suggestion would be to ignore autoscan entirely and instead try to figure out which platform you're targeting.
In particular autoscan will try to add tests for functions that are available in every single platform since 1990, but might be missing on a non-SysV OS for instance.
Start with a base platform target (such as C99 and the latest POSIX), and then figure out which other OSes you want to target and what they are missing.
Don't try to cover bases for OSes you have no access to, it's likely you won't actually know what is needed for them to build, let alone run.
Full disclosure: I have changed my opinion on gnulib over the years and not suggest it to be used anymore, if at all possible.
There are two ways to determine which list of gnulib modules you might need:
You can scan your source code or your object files (with nm) by hand.
You can import all the POSIX header file modules via gnulib-tool, then add -DGNULIB_POSIXCHECK to the CFLAGS, compile your package, and analyze the resulting warnings.
Also, some gnulib overrides exist for the sake of old platforms only. You can read the list of what each override fixes, and apply your own judgement.

Why's a simple change to rt.jar causing the Java Runtime Environment to crash silently?

This is what I'm doing:
extract contents of my JRE's rt.jar
extract src.zip of my JDK (same version)
Now, if I copy Runtime.java from the extracted src folder and compile it using javac.exe without any modifications and then put it in the extracted rt folder to finally put everything back in a jar file using jar.exe, everything works as expected. The JRE runs fine.
However, if I make the slightest change to Runtime.java and compile it and put it in rt.jar, the JRE crashes whenever I attempt to start it. This is an example of a slight change that causes the silent crash:
/** Don't let anyone else instantiate this class */
private Runtime() {
System.out.println("This is a test.");
}
Instead of:
/** Don't let anyone else instantiate this class */
private Runtime() {}
Could anyone tell me why this is causing my JRE to crash?
Thanks in advance.
It's possible that System.out has not been initialised at the time that the Runtime() constructor runs. Usually console output is not considered a "slight" change, but at the wrong time it can invoke way too much stuff that may not be set up at all yet.
You're doing this all wrong. You can't distribute that modified JRE for a start, so it is only useful inside your organization . Install a SecurityManager and don't grant your codebase any of the RuntimePermissions you're trying to protect against.
#Tom - I advise you NOT to try to do this:
You cannot distribute the modified rt.jar file without violating the Sun binary license.
Even if you did, you would not be allowed to call it Java.
As you are finding, there are lots of complications that arise when you make changes, particularly when those changes might interfere with the JVM's behind the scenes initialization. And when things blow up during initialization, the JVM often cannot report the problem in an intelligible way.
If you do succeed in making the modified rt.jar work for one JRE, there is no guarantee that the same hacks will work for a different version.
Nobody in their right mind would knowingly use a modified JVM (especially one modified by a third-party) in a production app.
EDIT : judging from your other questions, I guess you are trying to reverse engineer or modify some third party Java application with a custom launcher. If you provided more information on what you were really trying to do, we might be able to suggest the right way to do it ... rather than using "desperate measures" such as modifying the JRE.
That's pretty strange, as I did the same trick with many classes in rt.jar in past.
Can you provide us with the crashed process output?