How to mimic Optaplanner 7's MoveIteratorFactoryToMoveSelectorBridge in Optaplanner 8 - kotlin

We have implemented a custom construction heuristic with Optaplanner 7. We didn't use a simple CustomPhaseCommand; Instead, we extend MoveSelectorConfig and override buildBaseMoveSelector to return our own MoveFactory wrapped in a MoveIteratorFactoryToMoveSelectorBridge. We decided to do so, because it gives us the following advantages:
global termination config is supported out of the box
type safe configuration from code (no raw Strings)
With Optaplanner 8 the method buildBaseMoveSelector is gone from the MoveSelectorConfig API and building a custom config class seems to be prevented in the new implementation of MoveSelectorFactory.
Is it still possible to inject a proper custom construction heuristic into the Optaplanner 8 configuration and if yes, how? Or should we be using a CustomPhaseCommand with a custom self-implemented termination?
EDIT:
For clarity, in Optaplanner 7 we had the following snippet in our Optaplanner-config (defined in kotlin code):
ConstructionHeuristicPhaseConfig().apply {
foragerConfig = ConstructionHeuristicForagerConfig().apply {
pickEarlyType = FIRST_FEASIBLE_SCORE
}
entityPlacerConfig = QueuedEntityPlacerConfig().apply {
moveSelectorConfigList = listOf(
CustomMoveSelectorConfig().apply {
someProperty = 1
otherProperty = 0
}
)
}
},
CustomMoveSelectorConfig extends MoveSelectorConfig and overrides buildBaseMoveSelector:
class CustomMoveSelectorConfig(
var someProperty: Int = 0,
var otherProperty: Int = 0,
) : MoveSelectorConfig<CustomMoveSelectorConfig>() {
override fun buildBaseMoveSelector(
configPolicy: HeuristicConfigPolicy?,
minimumCacheType: SelectionCacheType?,
randomSelection: Boolean,
): MoveSelector {
return MoveIteratorFactoryToMoveSelectorBridge(
CustomMoveFactory(someProperty, otherProperty),
randomSelection
)
}
To summarize: We really need to plug our own MoveSelector with the custom factory. I think this is not possible with Optaplanner 8 at the moment.

Interesting extension.
Motivation for the changes in 8:
the buildBaseMoveSelector was not public API (the config package was not in the api package, we only guaranteed XML backwards compatibility for package config in 7). Now, we also guarantee API backwards compatibility for package config, so including programmatic configuration, because we moved all build* methods out of it.
In 8.2 or later we want to internalize the configuration in the SolverFactory, so we can build thousands of Solver instances faster. For example, loading classes wouldn't to be done only at SolverFactory build, once, no longer at every Solver build.
Anyway, let's first see if you can use the default way to override the moves of the CH, by explicitly configuring the queuedEntityPlacer and it's MoveIteratorFactory? https://docs.optaplanner.org/latestFinal/optaplanner-docs/html_single/#allocateEntityFromQueueConfiguration
I guess not, because you'd need mimic support... for the selected entity you need to generate n moves, but always the same entity during 1 placement (hence the need for mimicing).
Clearly, the changes in 8 prevent users from plugging in their own MoveSelectors (= which is an internal API, but anyway). We might be able to add an internal API to allow that again.

Related

Hwo to convert Flux<Item> to List<Item> by blocking

Background
I have a legacy application where I need to return a List<Item>
There are many different Service classes each belonging to an ItemType.
Each service class calls a few different backend APIs and collects the responses to create a SubType of the Item.
So we can say, each service class implementation returns an Item
All backend API access code is using WebClient which returns Mono of some type, and I can zip all Mono within the service to create an Item
The user should be able to look up many different types of items in one call. This requires many backend calls
So for performance sake, I wanted to make this all asynchronous using reactor, so I introduced Spring Reactive code.
Problem
If my endpoint had to return Flux<Item> then this code work fine,
But this is some service code which is used by other legacy code caller.
So eventually I want to return the List<Item> but When I try to convert my Flux into the List I get an error
"message": "block()/blockFirst()/blockLast() are blocking,
which is not supported in thread reactor-http-nio-3",
Here is the service, which is calling a few other service classes.
Flux<Item> itemFlux = Flux.fromIterable(searchRequestByItemType.entrySet())
.flatMap(e ->
getService(e.getKey()).searchItems(e.getValue()))
.subscribeOn(Schedulers.boundedElastic());
Mono<List<Item>> listMono = itemFlux
.collectList()
.block(); //This line throws error
Here is what the above service is calling
default Flux<Item> searchItems(List<SingleItemSearchRequest> requests) {
return Flux.fromIterable(requests)
.flatMap(this::searchItem)
.subscribeOn(Schedulers.boundedElastic());
}
Here is what a single-item search is which is used by above
public Mono<Item> searchItem(SingleItemSearchRequest sisr) {
return Mono.zip(backendApi.getItemANameApi(sisr.getItemIdentifiers().getItemId()),
sisr.isAddXXXDetails()
?backendApi.getItemAXXXApi(sisr.getItemIdentifiers().getItemId())
:Mono.empty(),
sisr.isAddYYYDetails()
?backendApi.getItemAYYYApi(sisr.getItemIdentifiers().getItemId())
:Mono.empty())
.map(tuple3 -> Item.builder()
.name(tuple3.getT1())
.xxxDetails(tuple3.getT2())
.yyyDetails(tuple3.getT3())
.build()
);
}
Sample project to replicate the problem..
https://github.com/mps-learning/spring-reactive-example
I’m new to spring reactor, feel free to pinpoint ALL errors in the code.
UPDATE
As per Patrick Hooijer Bonus suggestion, updating the Mono.zip entries to always contain some default.
#Override
public Mono<Item> searchItem(SingleItemSearchRequest sisr) {
System.out.println("\t\tInside " + supportedItem() + " searchItem with thread " + Thread.currentThread().toString());
//TODO: how to make these XXX YYY calls conditionals In clear way?
return Mono.zip(getNameDetails(sisr).defaultIfEmpty("Default Name"),
getXXXDetails(sisr).defaultIfEmpty("Default XXX Details"),
getYYYDetails(sisr).defaultIfEmpty("Default YYY Details"))
.map(tuple3 -> Item.builder()
.name(tuple3.getT1())
.xxxDetails(tuple3.getT2())
.yyyDetails(tuple3.getT3())
.build()
);
}
private Mono<String> getNameDetails(SingleItemSearchRequest sisr) {
return mockBackendApi.getItemCNameApi(sisr.getItemIdentifiers().getItemId());
}
private Mono<String> getYYYDetails(SingleItemSearchRequest sisr) {
return sisr.isAddYYYDetails()
? mockBackendApi.getItemCYYYApi(sisr.getItemIdentifiers().getItemId())
: Mono.empty();
}
private Mono<String> getXXXDetails(SingleItemSearchRequest sisr) {
return sisr.isAddXXXDetails()
? mockBackendApi.getItemCXXXApi(sisr.getItemIdentifiers().getItemId())
: Mono.empty();
}
Edit: Below answer does not solve the issue, but it contains useful information about Thread switching. It does not work because .block() is no problem for non-blocking Schedulers if it's used to switch to synchronous code.
This is because the block operator inherited the reactor-http-nio-3 Thread from backendApi.getItemANameApi (or one of the other calls in Mono.zip), which is non-blocking.
Most operators continue working on the Thread on which the previous operator executed, this is because the Thread is linked to the emitted item. There are two groups of operators where the Thread of the output item differs from the input:
flatMap, concatMap, zip, etc: Operators that emit items from other Publishers will keep the Thread link they received from this inner Publisher, not from the input.
Time based operators like delayElements, interval, buffer(Duration), etc. will schedule their tasks on the provided Scheduler, or Schedulers.parallel() if none provided. The emitted items will then be linked to the Thread the task was scheduled on.
In your case, Mono.zip emits items from backendApi.getItemANameApi linked to reactor-http-nio-3, which gets propagated downstream, goes outside both the flatMap in searchItems and in itemFlux, until it reaches your block operator.
You can solve this by placing a .publishOn(Schedulers.boundedElastic()), either in searchItem, searchItems or itemFlux. This will cause the item to switch to a Thread in the provided Scheduler.
Bonus: Since you requested to pinpoint errors: Your Mono.zip will not work if sisr.isAddXXXDetails() is false, as Mono.zip discards any element it could not zip. Since you return a Mono.empty() in that case, no items can be zipped and it will return an empty Mono.
If we have only spring-boot-starter-webflux defined as application dependency, then springbok spin up a `Netty server.
One is not expected to block() in a reactive application using a non-blocking server.
However, once we add spring-boot-starter-web dependency then even with the presence of spring-boot-starter-webflux, springboot spinup a tomcat server. Which is a thread-per-request model and is expected to have blocking calls
So to solve my problem, all I had to do above is, to add spring-boot-starter-web dependency in pom.xml. After that applications is started in Tomcat
with timcat .collectList().block() works in Controller class to return the List<Item>.
Whereas with the Netty server I could return only Flux<Item> not List<Item>, which is expected.

Neo4j 3.5's embeded database does not seem to persist data

I am trying to build a small command line tool that will store data in a neo4j graph. To do this I have started experimenting with Neo4j3.5's embedded databases. After putting together the following example I have found that either the nodes I am creating are not being saved to the database or the method of database creation is overwriting my previous run.
The Example:
fun main() {
//Spin up data base
val graphDBFactory = GraphDatabaseFactory()
val graphDB = graphDBFactory.newEmbeddedDatabase(File("src/main/resources/neo4j"))
registerShutdownHook(graphDB)
val tx = graphDB.beginTx()
graphDB.createNode(Label.label("firstNode"))
graphDB.createNode(Label.label("secondNode"))
val result = graphDB.execute("MATCH (a) RETURN COUNT(a)")
println(result.resultAsString())
tx.success()
}
private fun registerShutdownHook(graphDb: GraphDatabaseService) {
// Registers a shutdown hook for the Neo4j instance so that it
// shuts down nicely when the VM exits (even if you "Ctrl-C" the
// running application).
Runtime.getRuntime().addShutdownHook(object : Thread() {
override fun run() {
graphDb.shutdown()
}
})
}
I would expect that every time I run main the resulting query count will increase by 2.
That is currently not the case and I can find nothing in the docs that references a different method of opening an already created embedded database. Am I trying to use the embedded database incorrectly or am I missing something? Any help or info would be appreciated.
build Info:
Kotlin jvm 1.4.21
Neo4j-comunity-3.5.35
Transactions in neo4j 3.x have a 3 stage model
create
success / failure
close
you missed the third, which would then commit or rollback.
You can use Kotlin's use as Transaction is an AutoCloseable

system_cpu_usage is Nan when compiled in native

In my quarkus application i'm using micrometer to retrieve metrics (like in this guide : https://quarkus.io/guides/micrometer).
In JVM mode everything works fine, but in native mode system_cpu_usage is "Nan".
I tried bumping micrometer to 1.8.4 and adding :
{
"name":"com.sun.management.OperatingSystemMXBean", "allPublicMethods": true
},
to my reflect-config.json but no luck. I also tried generating the reflect-config (and other native configuration files) with the graalvm tracing agent but still no luck.
This may be a bug.
Micrometer is looking for a few known implementations of the MXBean:
https://github.com/micrometer-metrics/micrometer/blob/b087856355667abf9bf2386265edef8642e0e077/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/system/ProcessorMetrics.java#L55
private static final List<String> OPERATING_SYSTEM_BEAN_CLASS_NAMES = Arrays.asList(
"com.ibm.lang.management.OperatingSystemMXBean", // J9
"com.sun.management.OperatingSystemMXBean" // HotSpot
);
so that it can find the methods that it should be invoking...
https://github.com/micrometer-metrics/micrometer/blob/b087856355667abf9bf2386265edef8642e0e077/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/system/ProcessorMetrics.java#L80
this.operatingSystemBean = ManagementFactory.getOperatingSystemMXBean();
this.operatingSystemBeanClass = getFirstClassFound(OPERATING_SYSTEM_BEAN_CLASS_NAMES);
Method getCpuLoad = detectMethod("getCpuLoad");
this.systemCpuUsage = getCpuLoad != null ? getCpuLoad : detectMethod("getSystemCpuLoad");
this.processCpuUsage = detectMethod("getProcessCpuLoad");
(Note specifically "getFirstClassFound", which is constrained against the first list).
Speculation on my part, but I suspect Graal is returning a different type, which is possible from here:
https://github.com/oracle/graal/blob/6ba65dad76a4f54fa59e1ed2a62dedd3afe39928/substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/management/ManagementSupport.java#L166
would take some digging to know which, but I would open an issue with Micrometer so we can sort it out.

Howto tell PowerBuilder to pass options to a JVM when starting?

What I want to do?
I want to create and consume java objects in PowerBuilder and call methods on it. This should happen with less overhead possible.
I do not want to consume java webservices!
So I've a working sample in which I can create a java object, call a method on this object and output the result from the called method.
Everything is working as expected. I'm using Java 1.8.0_31.
But now I want to attach my java IDE (IntelliJ) to the running JVM (started by PowerBuilder) to debug the java code which gets called by PowerBuilder.
And now my question.
How do I tell PowerBuilder to add special options when starting the JVM?
In special I want to add the following option(s) in some way:
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
The JVM is created like following:
LONG ll_result
inv_java = CREATE JavaVM
ll_result = inv_java.CreateJavaVM("C:\Development\tms java\pbJavaTest", FALSE)
CHOOSE CASE ll_result
CASE 1
CASE 0
CASE -1
MessageBox ( "", "jvm.dll was not found in the classpath.")
CASE -2
MessageBox ( "", "pbejbclient90.jar file was not found." )
CASE ELSE
MessageBox ( "", "Unknown result (" + String (ll_result ) +")" )
END CHOOSE
In the PowerBuilder help I found something about overriding the static registry classpath. There is something written about custom properties which sounds like what I'm looking for.
But there's no example on how to add JVM options to override default behavior.
Does anyone have a clue on how to tell PowerBuilder to use my options?
Or does anyone have any advice which could guide me in the right direction?
Update 1
I found an old post which solved my initial issue.
If someone else want to know how it works take a look at this post:
http://nntp-archive.sybase.com/nntp-archive/action/article/%3C46262213.6742.1681692777#sybase.com%3E
Hi, you need to set some windows registry entries.
Under HKEY_LOCAL_MACHINE\SOFTWARE\Sybase\Powerbuilder\9.0\Java, there
are two folders: PBIDEConfig and PBRTConfig. The first one is used when
you run your application from within the IDE, and the latter is used
when you run your compiled application. Those two folders can have
PBJVMconfig and PBJVMprops folders within them.
PBJVMconfig is for JVM configuration options such as -Xms. You have to
specify incremental key values starting from "0" by one, and one special
key "Count" to tell Powerbuilder how many options exists to enumerate.
PBJVMprops is for all -D options. You do not need to specify -D for
PBJVMProps, just the name of the property and its value, and as many
properties as you wish.
Let me give some examples:
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Sybase\PowerBuilder\9.0\Java\PBIDEConfig\PBJVMprops]
"java.security.auth.login.config"="auth.conf"
"user.language"="en"
[HKEY_LOCAL_MACHINE\SOFTWARE\Sybase\PowerBuilder\9.0\Java\PBRTConfig\PBJVMconfig]
"0"="-client"
"1"="-Xms128m"
"2"="-Xmx512m"
"Count"="3"
[HKEY_LOCAL_MACHINE\SOFTWARE\Sybase\PowerBuilder\9.0\Java\PBRTConfig\PBJVMprops]
"java.security.auth.login.config"="auth.conf"
"user.language"="en"
Regards,
Gokhan Demir
But now there's another issue...
PB isn't able to create EJB Proxies for my sample class which is really simple with java 1.8.0_31. They were created with the default version, which is 1.6.0_24.
public class Simple
{
public Simple()
{
}
public static String getValue()
{
return "blubber";
}
public int getInt32Value()
{
return 123456;
}
public double getDoubleVaue()
{
return 123.123;
}
public static void main(String[] args)
{
System.out.println(Simple.getValue());
}
}
The error is the following. :D
---------- Deploy: Deploy of project p_genapp_ejbclientproxy (15:35:18)
Retrieving PowerBuilder Proxies from EJB...
Generation Errors: Error: class not found: (
Deployment Error: No files returned for package/component 'Simple'. Error code: Unknown. Proxy was not created.
Done.
---------- Finished Deploy of project p_genapp_ejbclientproxy (15:35:19)
So the whole way isn't a option because we do not want to change the JAVA settings in PB back and forth just to generate new EJB Proxies for changed JAVA objects in the future...
So one option to test will be creating COM wrappers for JAVA classes to use them in PB...

Executing untrusted code

I'm building a C# application which uses plug-ins. The application must guarantee to the user that plug-ins will not do whatever they want on the user machine, and will have less privileges that the application itself (for example, the application can access its own log files, whereas plug-ins cannot).
I considered three alternatives.
Using System.AddIn. I tried this alternative first, because it seamed much powerful, but I'm really disappointed by the need of modifying the same code seven times in seven different projects each time I want to modify something. Besides, there is a huge number of problems to solve even for a simple Hello World application.
Using System.Activator.CreateInstance(assemblyName, typeName). This is what I used in the preceding version of the application. I can't use it nevermore, because it does not provide a way to restrict permissions.
Using System.Activator.CreateInstance(AppDomain domain, [...]). That's what I'm trying to implement now, but it seems that the only way to do that is to pass through ObjectHandle, which requires serialization for every used class. Although plug-ins contain WPF UserControls, which are not serializable.
So is there a way to create plug-ins containing UserControls or other non serializable objects and to execute those plug-ins with a custom PermissionSet ?
One thing you could do is set the current AppDomain's policy level to a restricted permission set and add evidence markers to restrict based on strong name or location. The easiest would probably be to require plugins are in a specific directory and give them a restrictive policy.
e.g.
public static void SetRestrictedLevel(Uri path)
{
PolicyLevel appDomainLevel = PolicyLevel.CreateAppDomainLevel();
// Create simple root policy normally with FullTrust
PolicyStatement fullPolicy = new PolicyStatement(appDomainLevel.GetNamedPermissionSet("FullTrust"));
UnionCodeGroup policyRoot = new UnionCodeGroup(new AllMembershipCondition(), fullPolicy);
// Build restrictred permission set
PermissionSet permSet = new PermissionSet(PermissionState.None);
permSet.AddPermission(new SecurityPermission(SecurityPermissionFlag.Execution));
PolicyStatement permissions = new PolicyStatement(permSet, PolicyStatementAttribute.Exclusive);
policyRoot.AddChild(new UnionCodeGroup(new UrlMembershipCondition(path.ToString()), permissions));
appDomainLevel.RootCodeGroup = policyRoot;
AppDomain.CurrentDomain.SetAppDomainPolicy(appDomainLevel);
}
static void RunPlugin()
{
try
{
SetRestrictedLevel(new Uri("file:///c:/plugins/*"));
Assembly a = Assembly.LoadFrom("file:///c:/plugins/ClassLibrary.dll");
Type t = a.GetType("ClassLibrary.TestClass");
/* Will throw an exception */
t.InvokeMember("DoSomething", BindingFlags.InvokeMethod | BindingFlags.Public | BindingFlags.Static,
null, null, null);
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
Of course this isn't rigorously tested and CAS policy is notoriously complex so there is always a risk that this code might allow some things to bypass the policy, YMMV :)