What's the use/significance of evaluate() and aggregate() methods in the GenericUDAFEvaluator class? How is it difference from what merge() is doing?
Any example on usage of the above would be helpful.
Usage of merge(): http://beekeeperdata.com/posts/hadoop/2015/08/17/hive-udaf-tutorial.html
Javadocs: https://hive.apache.org/javadocs/r0.10.0/api/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.html
Checking the source code helped clarify things. Here are the relevant code-snippets of the same from GenericUDAFEvaluator.class file
evaluate(): This method is ensuring to call terminatePartial() implementation in case the execution mode is PARTIAL1 or PARTIAL2, else terminate() would be invoked to output the final result.
public Object evaluate(AggregationBuffer agg) throws HiveException {
if ((this.mode == Mode.PARTIAL1) || (this.mode == Mode.PARTIAL2)) {
return terminatePartial(agg);
}
return terminate(agg);
}
aggregate(): This method is checking on execution mode to invoke iterate() or merge() alternatively.
public void aggregate(AggregationBuffer agg, Object[] parameters) throws HiveException {
if ((this.mode == Mode.PARTIAL1) || (this.mode == Mode.COMPLETE)) {
iterate(agg, parameters);
} else {
assert (parameters.length == 1);
merge(agg, parameters[0]);
}
}
The difference modes of execution are as follows in the lifecycle of an UDAF
• PARTIAL1
○ from original data to partial aggregation data: iterate() and terminatePartial() will be called. Invoked from mapper.
• PARTIAL2
○ from partial aggregation data to partial aggregation data: merge() and terminatePartial() will be called. Invoked from combiner.
• COMPLETE
○ from original data directly to full aggregation: iterate() and terminate() will be called. Invoked in cases where UDAF is not spawned across map-reduce stages. i.e the call is limited only to reducer stage.
• FINAL
from partial aggregation to full aggregation: merge() and terminate() will be called. Invoked for reduce stage of a map-reduce spanning UDAF call.
Related
I have the following function in my OperationWalker:
public override void VisitDynamicInvocation(IDynamicInvocationOperation operation)
{
var memberReferenceOp = (IDynamicMemberReferenceOperation)operation.Operation;
switch (memberReferenceOp.Instance.Type)
{
case INamedTypeSymbol type:
{
var memberName = memberReferenceOp.MemberName;
var members = type.GetMembers(memberName);
if (members.Length > 1)
{
// WHAT DO I DO HERE ???
}
else
{
Result.Add((IMethodSymbol)members[0]);
}
break;
}
case IDynamicTypeSymbol dynamicType:
Unresolved.Add((operation.Syntax, memberReferenceOp.MemberName));
break;
}
}
I am clueless when a method on a normal type (non dynamic) is called with a dynamic parameter and there is a choice of target methods with the same name. E.g.:
class A
{
public void Get(int i){}
public void Get(string s){}
public void Get(object o){}
public void Get(double s, int precision){}
}
...
dynamic x = ...;
A a;
a.Get(x)
In this case any of the first 3 A.Get methods may be called, depending on the actual type of x. But not the fourth method.
Is there a way in Roslyn to get this information? Specifically in this example, I would like to get the symbols for first 3 Get methods.
The logic is non trivial, because one needs to take into account:
Default parameters, so just counting the arguments may not be enough
Type conversions
Visibility Scope
Number of arguments
Parameters may be passed using the named syntax in arbitrary order
Combining it all together we get non trivial logic. Is there anything in the SemanticModel or anywhere else to help get the answer?
I figured it out and it is straightforward - SemanticModel.GetSymbolInfo. When there is exact match its Symbol property returns it. When there are multiple candidates, as may be the case when one of the passed arguments is dynamic, then the property CandidateSymbols holds all the options.
I have not tested it with extension methods, so it is possible there is a gap there.
I am writing a native plugin that, in some cases, has to call functions in the Flutter portion of the app, written in Dart.
How it's achieved, is explained here:
https://flutter.io/platform-channels/
Furthermore, an example of invoking a method from the native/platform part towards the Dart/non-native is here:
https://github.com/flutter/plugins/tree/master/packages/quick_actions
Now, this example is really nice in case the platform only needs to invoke a method, i.e. that call returns nothing/void, but in case it needs to invoke a function, i.e. needs a return value from the non-native/Dart part, I could not have found an example or documentation on the internet. I believe it can be implemented though, because in the native Java part, there is a method:
public void invokeMethod(String method, Object arguments, MethodChannel.Result callback)
So, there is a callback object that could have a return value from the non-native part - or, I am mistaken here, and there is currently no way of returning a value from the non-native Dart portion of the app?
The signature is void setMethodCallHandler(Future<dynamic> handler(MethodCall call)), so we need to provide a function at the Dart end that returns Future<dynamic>, for example _channel.setMethodCallHandler(myUtilsHandler);
Then implement the handler. This one handles two methods foo and bar returning respectively String and double.
Future<dynamic> myUtilsHandler(MethodCall methodCall) async {
switch (methodCall.method) {
case 'foo':
return 'some string';
case 'bar':
return 123.0;
default:
throw MissingPluginException('notImplemented');
}
}
At the Java end the return value is passed to the success method of the Result callback.
channel.invokeMethod("foo", arguments, new Result() {
#Override
public void success(Object o) {
// this will be called with o = "some string"
}
#Override
public void error(String s, String s1, Object o) {}
#Override
public void notImplemented() {}
});
In Swift, the return value is an Any? passed to the result closure. (Not implemented is signaled by the any parameter being the const NSObject value FlutterMethodNotImplemented.)
channel.invokeMethod("foo", arguments: args, result: {(r:Any?) -> () in
// this will be called with r = "some string" (or FlutterMethodNotImplemented)
})
I'm using some aspect around advice methods. I have methods for controller, service and repository.
#Around("execution(* com.abc..controller..*(..)) && #annotation(audit)")
public Object controllerAround(ProceedingJoinPoint proceedingJoinPoint, Audit audit) throws Throwable {
//some code here
return output;
}
#Around("execution(* com.abc..service.*Impl.*(..))")
public Object serviceAround(ProceedingJoinPoint proceedingJoinPoint) throws Throwable {
//some code here
return output;
}
#Around("execution(* com.abc..persistence.*Impl.*(..))")
public Object persistenceAround(ProceedingJoinPoint proceedingJoinPoint) throws Throwable {
//some code here
return output;
}
I have a query that, I need to check in serviceAround method pointcut expression that, whether it's comes from the controllerAround method. I tried using some flags, but spring doesn't support aspectj's if() pointcut primitive.
Any workaround for this will be appreciated. :)
What you actually need is cflow() or cflowbelow(), not if(). But those are also not supported by Spring AOP. So the remaining solution is to use the full power of AspectJ from within Spring via LTW (load-time weaving). How this is done is nicely documented.
The pointcut would look like this:
execution(* com.abc..service.*Impl.*(..)) &&
cflow(
execution(* com.abc..controller..*(..)) &&
#annotation(customAnnotation)
)
Or simpler, assuming you do not need the annotation in the advice method:
execution(* com.abc..service.*Impl.*(..)) &&
cflow(execution(* com.abc..controller..*(..)))
Attention: cflow() only works for control flows within one thread, not if e.g. *Impl*.(..) is executed in another thread than controller..*(..).
P.S.: Your sample code would probably not work because of a parameter name mismatch between customAnnotation and audit.
I have a situation as follows: I have a LoggingAspect with several pointcuts matching specific method executions in my main application. The corresponding advice bodies basically all look similar, causing a lot of code duplication:
void around() : download() {
String message = "Downloading, verifying (MD5) and unpacking";
SimpleLogger.verbose(message, IndentMode.INDENT_AFTER);
proceed();
SimpleLogger.verbose(message + " - done", IndentMode.DEDENT_BEFORE);
}
There is some variation, though. Sometimes the pointcut & advice have an arg or this parameter which is also printed to the log. Sometimes the "done" message is not printed if it s just a minor call not wrapping a lot of other calls, like this:
void around(BasicFilter filter) : fixFaultyLinkTargets() && this(filter) {
String message = "TOC file: checking for faulty link targets";
SimpleLogger.verbose(message, IndentMode.INDENT_AFTER);
proceed(filter);
SimpleLogger.dedent();
}
The constant thing is that I manually tell the logger
to increase the indent level after the first message is printed, i.e. directly before proceed() is called, and
to decrease the indent level before the final message is printed (if any is printed), i.e. directly after proceed() has returned.
My idea is that I would like to write a meta aspect (or call it a helper aspect) with a pointcut which intercepts the proceed() calls in LoggingAspect so as to automatically adjust the indentation level accordingly. But there seems to be no pointcut matching proceed(). I have tried call(SomeMethodInMyMainApp), even a pointcut matching everything in the logging aspect, but the pointcut matches anything I do not need, but never ever the proceed.
If anybody knows how I can do this, I would appreciate a hint or a code snippet.
An indirect way of doing this might be to intercept not the advice themselves, but the method calls (or executions) advised by those advice by creating an extra pointcut like this:
// ATTENTION: each new pointcut must also be added here
pointcut catchAll() : download() || fixFaultyLinkTargets() || ...;
void around() : catchAll() {
SimpleLogger.indent();
proceed();
SimpleLogger.dedent();
}
I would prefer another way though, without me having to remember to update the extra catchAll() pointcut everytime I change something in the logging aspect.
Suggestion wrap the proceed() in an anonymous class. And the write an aspect which adress this execution (but don't forget potential exceptions of proceed()).
My suggestion:
// AspectProceedCaller.java
public abstract class AspectProceedCaller {
public abstract Object doProceed();
};
// aspect ProceedCallerAspect.aj
aspect ProceedCallerAspect {
pointcut execProceedCaller() : execution( * AspectProceedCaller+.doProceed() );
Object around() : execProceedCaller() {
try {
SimpleLogger.indent();
return proceed();
}
finally {
SimpleLogger.dedent();
}
}
};
// Your application aspect
aspect AnyAspect {
pointcut anyPointcut() : ...;
Object around() : anyPointcut() {
AspectProceedCaller apc=new AspectProceedCaller() {
public Object doProceed() {
return proceed();
}
};
// DO Stuff before ....
Object retval = apc.doProceed();
// ... and after calling proceed.
return retval;
}
};
Best regards Marko
Please note: I am going to answer my own question here, adding more information and the additional feature of parametrisation to the solution suggested by loddar2012. Because his answer led me to the right direction, I am going to accept it even though this answer here really addresses all my needs from the original question, such as (quoting myself):
There is some variation, though. Sometimes the pointcut & advice have an arg or this parameter which is also printed to the log. Sometimes the "done" message is not printed if it s just a minor call not wrapping a lot of other calls
The basic thing we are dealing with here is what Ramnivas Laddad calls the worker object pattern in his book AspectJ in Action. His (and loddar2012's) idea is, in plain prose
to wrap a call into an instance of an anonymous class (the worker object) where
the base class or implemented interface provides a method intended to do the work,
the worker object provides a concrete implementation of the worker method and specifically calls proceed() therein,
the worker method can be called right after object creation (as we will do here) or later, maybe even in its own thread,
the worker object may be passed around or added to a scheduling queue (none of which we will need here).
An elegant solution if you need to execute your proceed() calls asynchronously would be to create instances of anonymous Runnable classes. We will use our own abstract base class LogHelper, though, because we want some more sugar in our tea, specifically the option to pass a log message and some other parameters influencing log output to each worker. So this is what I did (package names and imports not shown in the sample code):
Abstract worker base class:
abstract class LogHelper {
// Object state needed for logging
String message;
boolean logDone;
boolean indent;
LogType type;
// Main constructor
LogHelper(String message, boolean logDone, boolean indent, LogType type) {
this.message = message;
this.logDone = logDone;
this.indent = indent;
this.type = type;
}
// Convenience constructors for frequent use cases
LogHelper(String message, boolean logDone) {
this(message, logDone, true, LogType.VERBOSE);
}
LogHelper(String message) {
this(message, true);
}
// Worker method to be overridden by each anonymous subclass
abstract void log();
}
Logging advice capturing execution of worker objects:
aspect LoggingAspect
{
void around(LogHelper logHelper) :
execution(* LogHelper.log()) && this(logHelper)
{
try {
SimpleLogger.log(logHelper.type, logHelper.message);
if (logHelper.indent)
SimpleLogger.indent();
proceed(logHelper);
} finally {
if (logHelper.indent)
SimpleLogger.dedent();
if (logHelper.logDone)
SimpleLogger.log(logHelper.type, logHelper.message + " - done");
}
}
// (...)
}
As you can see, the logging advice does some things before calling proceed(logHelper) (i.e. executing the worker object's log() method) and some things afterwards, using the state information stored inside the worker object, such as
message to be logged,
log level (here called "type"),
flag specifying if indentation level should be raised before proceeding,
flag specifying if "done" message should be printed after worker execution.
Because in my use case all logged methods return void, there is no need to implement return value passing, but this would be easily possible, if necessary. The advice's return value would then just be Object and we would pass the result of proceed() back to our caller, no big deal.
Some advice capturing joinpoints to be logged and utilising parametrised worker objects to get the work done:
aspect LoggingAspect
{
// (...)
pointcut processBook() : execution(* OpenbookCleaner.downloadAndCleanBook(Book));
pointcut download() : execution(* Downloader.download());
pointcut cleanBook() : execution(* OpenbookCleaner.cleanBook(Book));
pointcut cleanChapter() : execution(* OpenbookCleaner.cleanChapter(Book, File));
pointcut initialiseTitle() : execution(* *Filter.initialiseTitle(boolean));
void around(final Book book) : processBook() && args(book) {
new LogHelper("Book: " + book.unpackDirectory) {
void log() { proceed(book); } }.log();
}
void around() : download() {
new LogHelper("Downloading, verifying (MD5) and unpacking") {
void log() { proceed(); } }.log();
}
void around() : cleanBook() {
new LogHelper("Filtering") {
void log() { proceed(); } }.log();
}
void around(final File origFile) : cleanChapter() && args(*, origFile) {
new LogHelper("Chapter: " + origFile.getName()) {
void log() { proceed(origFile); } }.log();
}
void around() : initialiseTitle() {
new LogHelper("Initialising page title", false) {
void log() { proceed(); } }.log();
}
}
The examples show how you can
instantiate an anonymous LogHelper as a worker object with one or more constructor parameters, setting its state
implement the log() method, optionally using joinpoint state bound via this() or args(),
call/run the worker object (the call will be intercepted by the logging advice's pointcut and the real logging business be done there).
In languages like Scala, one can have multiple definitions for one method name by changing the number of parameters and/or the type of the parameters of the method. This is called method overloading.
How is that different from multiple dispatch?
Thank you
Method overloading is resolved at compile time.
Multiple dispatch is resolved at runtime.
When using double dispatch the called method depends on the actual type of receiver and arguments. Method overloading however, only allows the called method to depend on the declared type of the parameters. Why? Java binds method calls at compile time with their full signature (early binding). The full signature includes all parameter types, hence when the actual type of an argument differs at runtime (polymoprhism), overloading does not work as you might expect!
void add(Foo o) { ... }
void add(Bar o) { ... }
void client() {
Foo o = new Bar();
add(o); // calls add(Foo) not add(Bar)!
}
using multiple dispatch however
void add(Foo o) { o.dispatch(this); }
void add(Bar o) { o.dispatch(this); }
void client() {
Foo o = new Bar();
add(o); // calls #dispatch as defined in Bar!
}
Things might slightly differ in Scala, though the general distinction should be the same as presented here in all programming languages.