I got null element when reading from BigQuery table using dataflow as follows
private static class ParseBQInput extends DoFn<TableRow, KV<String, MyClass> > {
#Override
public void processElement(ProcessContext c) throws Exception {
TableRow row = c.element();
List<TableCell> cells = row.getF();
if (cells == null) {
throw new RuntimeException("Get null cells");
}
// Other operation
}
}
The cell I extracted with row.getF() is null regardless which table i read.
I noticed that there is similar question already asked here but the answer is sort of unclear to me... I have tried updated the library version to 1.3.0 but it seems that the problem is still not solved. (Due to some dependency issue, it is a little painful to go to version 1.4.0 at current...)
(As a newbie to stack overflow, i don't have enough points to post comments though, so have to ask again here...)
As noted in the updated https://stackoverflow.com/a/35326342/4392693, getF() will not be supported in newer versions. Please use get("fieldname"). Thank you for your patience.
Related
I am building a Capacited Vehicle Routing Problem with Time Windows, but with one small difference when compared to the one provided in examples from the documentation: I don't have a depot. Instead, each order has a pickup step, and a delivery step, in two different locations.
(like in the Vehicle Routing example from the documentation, the previousStep planning variable has the CHAINED graph type, and its valueRangeProviderRefs includes both Drivers, and Steps)
This difference adds a couple of constraints:
the pickup and delivery steps of a given order must be handled by the same driver
the pickup must be before the delivery
After experimenting with constraints, I have found that it would be more efficient to implement two types of custom moves:
assign both steps of an order to a driver
rearrange the steps of a driver
I am currently implementing that first custom move. My solver's configuration looks like this:
SolverFactory<RoutingProblem> solverFactory = SolverFactory.create(
new SolverConfig()
.withSolutionClass(RoutingProblem.class)
.withEntityClasses(Step.class, StepList.class)
.withScoreDirectorFactory(new ScoreDirectorFactoryConfig()
.withConstraintProviderClass(Constraints.class)
)
.withTerminationConfig(new TerminationConfig()
.withSecondsSpentLimit(60L)
)
.withPhaseList(List.of(
new LocalSearchPhaseConfig()
.withMoveSelectorConfig(CustomMoveListFactory.getConfig())
))
);
My CustomMoveListFactory looks like this (I plan on migrating it to an MoveIteratorFactory later, but for the moment, this is easier to read and write):
public class CustomMoveListFactory implements MoveListFactory<RoutingProblem> {
public static MoveListFactoryConfig getConfig() {
MoveListFactoryConfig result = new MoveListFactoryConfig();
result.setMoveListFactoryClass(CustomMoveListFactory.class);
return result;
}
#Override
public List<? extends Move<RoutingProblem>> createMoveList(RoutingProblem routingProblem) {
List<Move<RoutingProblem>> moves = new ArrayList<>();
// 1. Assign moves
for (Order order : routingProblem.getOrders()) {
Driver currentDriver = order.getDriver();
for (Driver driver : routingProblem.getDrivers()) {
if (!driver.equals(currentDriver)) {
moves.add(new AssignMove(order, driver));
}
}
}
// 2. Rearrange moves
// TODO
return moves;
}
}
And finally, the move itself looks like this (nevermind the undo or the isDoable for the moment):
#Override
protected void doMoveOnGenuineVariables(ScoreDirector<RoutingProblem> scoreDirector) {
assignStep(scoreDirector, order.getPickupStep());
assignStep(scoreDirector, order.getDeliveryStep());
}
private void assignStep(ScoreDirector<RoutingProblem> scoreDirector, Step step) {
StepList beforeStep = step.getPreviousStep();
Step afterStep = step.getNextStep();
// 1. Insert step at the end of the driver's step list
StepList lastStep = driver.getLastStep();
scoreDirector.beforeVariableChanged(step, "previousStep"); // NullPointerException here
step.setPreviousStep(lastStep);
scoreDirector.afterVariableChanged(step, "previousStep");
// 2. Remove step from current chained list
if (afterStep != null) {
scoreDirector.beforeVariableChanged(afterStep, "previousStep");
afterStep.setPreviousStep(beforeStep);
scoreDirector.afterVariableChanged(afterStep, "previousStep");
}
}
The idea being that at no point I'm doing an invalid chained list manipulation:
However, as the title and the code comment indicate, I am getting a NullPointerException when I call scoreDirector.beforeVariableChanged. None of my variables are null (I've printed them to make sure). The NullPointerException doesn't occur in my code, but deep inside Optaplanner's inner workings, making it difficult for me to fix it:
Exception in thread "main" java.lang.NullPointerException
at org.drools.core.common.NamedEntryPoint.update(NamedEntryPoint.java:353)
at org.drools.core.common.NamedEntryPoint.update(NamedEntryPoint.java:338)
at org.drools.core.impl.StatefulKnowledgeSessionImpl.update(StatefulKnowledgeSessionImpl.java:1579)
at org.drools.core.impl.StatefulKnowledgeSessionImpl.update(StatefulKnowledgeSessionImpl.java:1551)
at org.optaplanner.core.impl.score.stream.drools.DroolsConstraintSession.update(DroolsConstraintSession.java:49)
at org.optaplanner.core.impl.score.director.stream.ConstraintStreamScoreDirector.afterVariableChanged(ConstraintStreamScoreDirector.java:137)
at org.optaplanner.core.impl.domain.variable.inverserelation.SingletonInverseVariableListener.retract(SingletonInverseVariableListener.java:96)
at org.optaplanner.core.impl.domain.variable.inverserelation.SingletonInverseVariableListener.beforeVariableChanged(SingletonInverseVariableListener.java:46)
at org.optaplanner.core.impl.domain.variable.listener.support.VariableListenerSupport.beforeVariableChanged(VariableListenerSupport.java:170)
at org.optaplanner.core.impl.score.director.AbstractScoreDirector.beforeVariableChanged(AbstractScoreDirector.java:430)
at org.optaplanner.core.impl.score.director.AbstractScoreDirector.beforeVariableChanged(AbstractScoreDirector.java:390)
at test.optaplanner.solver.AssignMove.assignStep(AssignMove.java:98)
at test.optaplanner.solver.AssignMove.doMoveOnGenuineVariables(AssignMove.java:85)
at org.optaplanner.core.impl.heuristic.move.AbstractMove.doMove(AbstractMove.java:35)
at org.optaplanner.core.impl.heuristic.move.AbstractMove.doMove(AbstractMove.java:30)
at org.optaplanner.core.impl.score.director.AbstractScoreDirector.doAndProcessMove(AbstractScoreDirector.java:187)
at org.optaplanner.core.impl.localsearch.decider.LocalSearchDecider.doMove(LocalSearchDecider.java:132)
at org.optaplanner.core.impl.localsearch.decider.LocalSearchDecider.decideNextStep(LocalSearchDecider.java:116)
at org.optaplanner.core.impl.localsearch.DefaultLocalSearchPhase.solve(DefaultLocalSearchPhase.java:70)
at org.optaplanner.core.impl.solver.AbstractSolver.runPhases(AbstractSolver.java:98)
at org.optaplanner.core.impl.solver.DefaultSolver.solve(DefaultSolver.java:189)
at test.optaplanner.OptaPlannerService.testOptaplanner(OptaPlannerService.java:68)
at test.optaplanner.App.main(App.java:13)
Is there something I did wrong? It seems I am following the documentation for custom moves fairly closely, outside of the fact that I am using exclusively java code instead of drools.
The initial solution I feed to the solver has all of the steps assigned to a single driver. There are 15 drivers and 40 orders.
In order to bypass this error, I have tried a number of different things:
remove the shadow variable annotation, turn Driver into a problem fact, and handle the nextStep field myself => this makes no difference
use Simulated Annealing + First Fit Decreasing construction heuristics, and start with steps not assigned to any driver (this was inspired by looking up the example here, which is more complete than the one from the documentation) => the NullPointerException appears on afterVariableChanged instead, but it still appears.
a number of other things which were probably not very smart
But without a more helpful error message, I can't think of anything else to try.
Thank you for your help
I am trying to obtains list of broken constrains from a problem instance in OptaPlanner. I am using OptaPlanner version 7.0.0.Final and drools for rules engine (also 7.0.0.Final). The problem is solved correctly and without any error, but when I try to obtain broken constrains I get a NullPointer exception.
As far as I have researched, I found out, that this only happens, when I use drools accumulator without reverse operation (like max or min). Further I have made a custom accumulator, which is the exact copy from org.drools.core.base.accumulators.LongSumAccumulateFunction and everything works as expected, but as soon as I change the supportsReverse() function to return false, the NullPointer exception rises.
I have managed to reconstruct this problem in one of the provided examples - CloudBalancing. This is the change to CloudBalancingHelloWorld, it's only purpose is to obtain list of broken constraints as mentioned in this post.
public class CloudBalancingHelloWorld {
public static void main(String[] args) {
// Build the Solver
SolverFactory<CloudBalance> solverFactory = SolverFactory.createFromXmlResource(
"org/optaplanner/examples/cloudbalancing/solver/cloudBalancingSolverConfig.xml");
Solver<CloudBalance> solver = solverFactory.buildSolver();
// Load a problem with 400 computers and 1200 processes
CloudBalance unsolvedCloudBalance = new CloudBalancingGenerator().createCloudBalance(400, 1200);
// Solve the problem
CloudBalance solvedCloudBalance = solver.solve(unsolvedCloudBalance);
// Display the result
System.out.println("\nSolved cloudBalance with 400 computers and 1200 processes:\n"
+ toDisplayString(solvedCloudBalance));
//
//A Piece of code added - start
//
ScoreDirector<CloudBalance> scoreDirector = solver.getScoreDirectorFactory().buildScoreDirector();
scoreDirector.setWorkingSolution(solvedCloudBalance);
Collection<ConstraintMatchTotal> constrains = scoreDirector.getConstraintMatchTotals();
System.out.println(constrains.size());
//
//A Piece of code added - end
//
}
public static String toDisplayString(CloudBalance cloudBalance) {
StringBuilder displayString = new StringBuilder();
for (CloudProcess process : cloudBalance.getProcessList()) {
CloudComputer computer = process.getComputer();
displayString.append(" ").append(process.getLabel()).append(" -> ")
.append(computer == null ? null : computer.getLabel()).append("\n");
}
return displayString.toString();
}
}
And this is the change to requiredCpoPowerTotal rule. Please note that I have done this only to demonstrate the problem. Basicaly I have changed sum to max.
rule "requiredCpuPowerTotal"
when
$computer : CloudComputer($cpuPower : cpuPower)
accumulate(
CloudProcess(
computer == $computer,
$requiredCpuPower : requiredCpuPower);
$requiredCpuPowerTotal : max($requiredCpuPower);
(Integer) $requiredCpuPowerTotal > $cpuPower
)
then
scoreHolder.addHardConstraintMatch(kcontext, $cpuPower - (Integer) $requiredCpuPowerTotal);
end
I am really confused, because the error does not happen during planing phase, but when the scoreDirector recomputes the score to obtain broken constrains it does. I mean the same calculations must have happened during the planning phase right?
Anyway here is the stack trace
Exception in thread "main" Exception executing consequence for rule "requiredCpuPowerTotal" in org.optaplanner.examples.cloudbalancing.solver: java.lang.NullPointerException
at org.drools.core.runtime.rule.impl.DefaultConsequenceExceptionHandler.handleException(DefaultConsequenceExceptionHandler.java:39)
at org.drools.core.common.DefaultAgenda.handleException(DefaultAgenda.java:1256)
at org.drools.core.phreak.RuleExecutor.innerFireActivation(RuleExecutor.java:438)
at org.drools.core.phreak.RuleExecutor.fireActivation(RuleExecutor.java:379)
at org.drools.core.phreak.RuleExecutor.fire(RuleExecutor.java:135)
at org.drools.core.phreak.RuleExecutor.evaluateNetworkAndFire(RuleExecutor.java:88)
at org.drools.core.concurrent.AbstractRuleEvaluator.internalEvaluateAndFire(AbstractRuleEvaluator.java:34)
at org.drools.core.concurrent.SequentialRuleEvaluator.evaluateAndFire(SequentialRuleEvaluator.java:43)
at org.drools.core.common.DefaultAgenda.fireLoop(DefaultAgenda.java:1072)
at org.drools.core.common.DefaultAgenda.internalFireAllRules(DefaultAgenda.java:1019)
at org.drools.core.common.DefaultAgenda.fireAllRules(DefaultAgenda.java:1011)
at org.drools.core.impl.StatefulKnowledgeSessionImpl.internalFireAllRules(StatefulKnowledgeSessionImpl.java:1321)
at org.drools.core.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:1312)
at org.drools.core.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:1296)
at org.optaplanner.core.impl.score.director.drools.DroolsScoreDirector.getConstraintMatchTotals(DroolsScoreDirector.java:134)
at org.optaplanner.examples.cloudbalancing.app.CloudBalancingHelloWorld.main(CloudBalancingHelloWorld.java:52)
Caused by: java.lang.NullPointerException
at org.drools.core.base.accumulators.JavaAccumulatorFunctionExecutor$JavaAccumulatorFunctionContext.getAccumulatedObjects(JavaAccumulatorFunctionExecutor.java:208)
at org.drools.core.reteoo.FromNodeLeftTuple.getAccumulatedObjects(FromNodeLeftTuple.java:94)
at org.drools.core.common.AgendaItem.getObjectsDeep(AgendaItem.java:78)
at org.drools.core.reteoo.RuleTerminalNodeLeftTuple.getObjectsDeep(RuleTerminalNodeLeftTuple.java:359)
at org.optaplanner.core.api.score.holder.AbstractScoreHolder.extractJustificationList(AbstractScoreHolder.java:118)
at org.optaplanner.core.api.score.holder.AbstractScoreHolder.registerConstraintMatch(AbstractScoreHolder.java:88)
at org.optaplanner.core.api.score.buildin.hardsoft.HardSoftScoreHolder.addHardConstraintMatch(HardSoftScoreHolder.java:53)
at org.optaplanner.examples.cloudbalancing.solver.Rule_requiredCpuPowerTotal1284553313.defaultConsequence(Rule_requiredCpuPowerTotal1284553313.java:14)
at org.optaplanner.examples.cloudbalancing.solver.Rule_requiredCpuPowerTotal1284553313DefaultConsequenceInvokerGenerated.evaluate(Unknown Source)
at org.optaplanner.examples.cloudbalancing.solver.Rule_requiredCpuPowerTotal1284553313DefaultConsequenceInvoker.evaluate(Unknown Source)
at org.drools.core.phreak.RuleExecutor.innerFireActivation(RuleExecutor.java:431)
... 13 more
Thank you for any help in advance.
That NPE sounds like a bug in Drools. The ConstraintMatch API should always just work. Very that you get it against the latest master version. If so, please create a jira for this with a minimal reproducer and we'll look into it.
In Junit5 5.0.0 M4 I could do this:
#ParameterizedTest
#MethodSource("generateCollections")
void testCollections(Collection<Object> collection) {
assertOnCollection(collection);
}
private static Iterator<Collection<Object>> generateCollections() {
Random generator = new Random();
// We'll run as many tests as possible in 500 milliseconds.
final Instant endTime = Instant.now().plusNanos(500000000);
return new Iterator<Collection<Object>>() {
#Override public boolean hasNext() {
return Instant.now().isBefore(endTime);
}
#Override public Collection<Object> next() {
// Dummy code
return Arrays.asList("this", "that", Instant.now());
}
};
}
Or any number of other things that ended up with collections of one type or another being passed into my #ParameterizedTest. This no longer works: I now get the error
org.junit.jupiter.api.extension.ParameterResolutionException:
Error resolving parameter at index 0
I've been looking through the recent commits to SNAPSHOT and I there's a few changes in the area, but I can't see anything that definitely changes this.
Is this a deliberate change? I'd ask this on a JUnit5 developer channel but I can't find one. And it's not a bug per se: passing a collection is not a documented feature.
If this is a deliberate change, then this is a definite use-case for #TestFactory...
See https://github.com/junit-team/junit5/issues/872
The next snapshot build should fix the regression.
I have this sonar error Major:
Found 'UR'-anomaly for variable 'language' (lines '83'-'85')
in this function:
public void saveAll(List<Language> languages){
//Found 'UR'-anomaly for variable 'country' (lines '83'-'85').
//Code Smell Major Open Not assigned 20min effort Comment
for (Language language: languages) {
save(language);
}
}
how to fix this major error please, thanks for advance
Edit:
Found even more information it this other SO post. While that is more PMD centric, the background information can be of interest to you.
Java for each loop being flagged as UR anomaly by PMD.
This is a rule from PMD it seems. Definition:
The dataflow analysis tracks local definitions, undefinitions and
references to variables on different paths on the data flow. From
those informations there can be found various problems. 1. UR -
Anomaly: There is a reference to a variable that was not defined
before. This is a bug and leads to an error. 2. DU - Anomaly: A
recently defined variable is undefined. These anomalies may appear in
normal source text. 3. DD - Anomaly: A recently defined variable is
redefined. This is ominous but don't have to be a bug.
There is an open bug report for this:
https://sourceforge.net/p/pmd/bugs/1190/
In the example they report it for Arrays, but somebody commented that it happens for them also for collections.
Example:
public static void main(final String[] args) {
for (final String string : args) {
string.getBytes(); //UR Anomaly
}
for (int i = 0; i < args.length; i++) {
args[i].getBytes();
}
}
In our sonar setup we don't use this rule. Based on the available information you may wish not to use it in yours.
Recently, we started to work with play 2.2. Previously we were working with play 2.1.3.
In play 2.2 it says Akka.future and async methods are seen as deprecated. Also when we tried to run below piece of code fetchSample() through a loop, it took more time to complete in play 2.2.
So how can we replace the below deprecated code with the latest one?
private static Promise<SampleDBResponseBean> fetchSample(
final Document sampleDoc) throws Exception {
Promise<SampleBean> promiseOfSampleJson = Akka.future(
new Callable<SampleBean>() {
public SampleBean call() throws Exception
{
return doSomeCalc(sampleDoc);
}
});
}
private Result getAsyncResult(final SampleResponseBean sampleDbResponseBean) {
List<F.Promise<? extends SampleDBResponseBean>> promiseList = sampleDbResponseBean
.getSampleHelperList();
Promise<List<SampleDBResponseBean>> promiseJsonObjLists = Promise
.sequence(promiseList);
return async(
promiseJsonObjLists.map(
new Function<List<SampleDBResponseBean>, Result>() {
public Result apply(List<SampleDBResponseBean> sampleList) {
SampleResponseBean sampleResponseBean = new SampleResponseBean();
sampleResponseBean.setStatus("success");
sampleResponseBean.setSampleList(sampleList);
JsonNode jsNodeResponse = Json.toJson(sampleResponseBean);
return ok(jsNodeResponse);
}
}));
}
I had searched a lot of places not seeing any solution. The problem effects our code performance when comparing to 2.1.3.
Any ideas how can we implement the deprecated methods for the above 2 methods in play 2.2?
As pointed out in the migration docs:
http://www.playframework.com/documentation/2.2.x/Migration22
You want to use Promise.promise. This is also described in the documentation:
http://www.playframework.com/documentation/2.2.x/JavaAsync
And of course in the API docs:
http://www.playframework.com/documentation/2.2.x/api/java/play/libs/F.Promise.html#promise(play.libs.F.Function0)
One of the really nice things about Play 2.2 Java promises is now you can control exactly which execution context the code runs in, so you can create your own execution context, or get one from Akka, and so control exactly how many, in your case, concurrent DB operations are run across the whole app at the same time.