Sometimes I want to filter out certain errors in a stream. I'd like to write something like this:
stream
.filterError (error) ->
error.type is 'foo'
But there is no filterError method.
As an alternative I thought I could use errors().mapError to map the errors into values, filter them, and then map them back into errors. However, I don't see a way to convert a value in a stream into an error.
# Filter only the errors we are interested in
errors = stream.errors()
.mapError (error) ->
error
.filter (error) ->
...
.mapValuesBackIntoErrors() # ?
The idea is that the stream in question either carries a value or an error. Both represent domain knowledge; the value means the system is in normal operation and the error means we have a domain error. Some of the domain errors are not such that we want to carry them, though, so I wish to filter them out.
The alternative works, of course, and you may use a combination of map and mapError to wrap normal values and errors in the style of the Either type in Haskell. For instance
stream.map((value) -> { value }).mapError((error) -> { error })
Now your stream would output something like this:
{ value: 1 }
{ value: 2 }
{ error: "cannot connect to host" }
On the other hand, actually implementing filterError wouldn't be too hard. Consider implementing this yourself and making a PR.
Related
I am attempting to put together a Cro service that has a react/whenever block consuming data "in the background" So unlike many examples of websocket usage with Cro, this has nothing to do with routes that may be accessed via the browser.
My use case is to consume message received via an MQTT topic and do some processing with them. At a later stage in development I might create a supply out of this data, but for now, when data is received it will be stored in a variable and dependant on certain conditions, be sent to another service via a http post.
My thought was to include a provider() in the Cro::HTTP::Server setup like so:
use Cro::HTTP::Log::File;
use Cro::HTTP::Server;
use Routes;
use DataProvider; # Here
my Cro::Service $http = Cro::HTTP::Server.new(
http => <1.1>,
host => ...,
port => ...,
application => [routes(), provider()], # Made this into an array of subs?
after => [
Cro::HTTP::Log::File.new(logs => $*OUT, errors => $*ERR)
]
);
And in the DataProvider.pm6:
use MQTT::Client;
sub provider() is export {
my $mqtt = MQTT::Client.new: server => 'localhost';
react {
whenever $mqtt.subscribe('some/mqtt/topic') {
say "+ topic: { .<topic> } => { .<message>.decode("utf8-c8") }";
}
}
}
This throws a bunch of errors:
A react block:
in sub provider at DataProvider.pm6 (DataProvider) line 5
in block <unit> at service.p6 line 26
Died because of the exception:
Invocant of method 'write' must be an object instance of type
'IO::Socket::Async', not a type object of type 'IO::Socket::Async'. Did
you forget a '.new'?
in method subscribe at /home/cam/raku/share/perl6/site/sources/42C762836A951A1C11586214B78AD34262EC465F (MQTT::Client) line 133
in sub provider at DataProvider.pm6 (DataProvider) line 6
in block <unit> at service.p6 line 26
To be perfectly honest, I am totally guessing that this is how I would approach the need to subscribe to data in the background of a Cro service, but I was not able to find any information on what might be considered the recommended approach.
Initially I had my react/whenever block in the main service.pm6 file, but that did not seem right. And needed to be wrapped in a start{} block because as I have just learned, react is blocking :) and cro was not able to actually start.
But following the pattern of how Routes are implemented seemed logical, but I am missing something. The error speaks about setting up a new method, but I'm not convinced that is the root cause. Routes.pm6 does not have a constructor.
Can anyone point me in the right direction please?
Thanks to all who have provided information, this has been a very valuable learning exercise.
The approach of passing additional sub routines, along side router() in the application parameter to Cro::HTTP::Server.new gave further trouble. (an array is not allowed, and broke routing)
Instead, I have moved the background work into a class of it's own, and given it a start and stop method more akin to Cro::HTTP::Server.
My new approach:
service.pm6
use Cro::HTTP::Log::File;
use Cro::HTTP::Server;
use Routes;
use KlineDataSubscriber; # Moved mqtt functionality here
use Database;
my $dsn = "host=localhost port=5432 dbname=act user=.. password=..";
my $dbh = Database.new :$dsn;
my $mqtt-host = 'localhost';
my $subscriber = KlineDataSubscriber.new :$mqtt-host;
$subscriber.start; # Inspired by $http.start below
my Cro::Service $http = Cro::HTTP::Server.new(
http => <1.1>,
host => ...,
port => ...,
application => routes($dbh), # Basically back the way it was originally
after => [
Cro::HTTP::Log::File.new(logs => $*OUT, errors => $*ERR)
]
);
$http.start;
say "Listening at...";
react {
whenever signal(SIGINT) {
say "Shutting down...";
$subscriber.stop;
$http.stop;
done;
}
}
And in KlineDataSubscriber.pm6
use MQTT::Client;
class KlineDataSubscriber {
has Str $.mqtt-host is required;
has MQTT::Client $.mqtt = Nil;
submethod TWEAK() {
$!mqtt = MQTT::Client.new: server => $!mqtt-host;
await $!mqtt.connect;
}
method start(Str $topic = 'act/feed/exchange/binance/kline-closed/+/json') {
start {
react {
whenever $!mqtt.subscribe($topic) {
say "+ topic: { .<topic> } => { .<message>.decode("utf8-c8") }";
}
}
}
}
method stop() {
# TODO Figure how to unsubscribe and cleanup nicely
}
}
This feels much more "Cro idiomatic" to me, but I would be happy to be corrected.
More importantly, it works as expected and I feel is somewhat future proof. I should be able to create a supply to make real-time data available to the router, and push data to any connected web clients.
I also intend to have a http GET endpoint /status with various checks to ensure everything healthy
The root cause
The error speaks about setting up a new method, but I'm not convinced that is the root cause.
It's not about setting up a new method. It's about a value that should be defined instead being undefined. That typically means a failure to attempt to initialize it, which typically means a failure to call .new.
Can anyone point me in the right direction please?
Hopefully this question helps.
Finding information on a recommended approach
I am totally guessing that this is how I would approach the need to subscribe to data in the background of a Cro service, but I was not able to find any information on what might be considered the recommended approach.
It might be helpful for you to list which of the get-up-to-speed steps you've followed from Getting started with Cro, including the basics but also the "Learn about" steps at the end.
The error message
A react block:
in sub provider ...
Died because of the exception:
...
in method subscribe ...
The error message begins with the built in react construct reporting that it caught an exception (and handled it by throwing its own exception in response). A "backtrace" corresponding to where the react appeared in your code is provided indented from the initial "A react block:".
The error message continues with the react construct summarizing its own exception (Died because ...) and explains itself by reporting the original exception, further indented, in subsequent lines. This includes another backtrace, this time one corresponding to the original exception, which will likely have occurred on a different thread with a different callstack.
(All of Raku's structured multithreading constructs[1] use this two part error reporting approach for exceptions they catch and handle by throwing another exception.)
The first backtrace indicates the react line:
in sub provider at DataProvider.pm6 (DataProvider) line 5
use MQTT::Client;
sub provider() is export {
my $mqtt = MQTT::Client.new: server => 'localhost';
react {
The second backtrace is about the original exception:
Invocant of method 'write' must be an object instance of type
'IO::Socket::Async', not a type object of type 'IO::Socket::Async'. ...
in method subscribe at ... (MQTT::Client) line 133
This reports that the write method called on line 133 of MQTT::Client requires its invocant is an instance of type 'IO::Socket::Async'. The value it got was of that type but was not an instance, but instead a "type object". (All values of non-native types are either type objects or instances of their type.).
The error message concludes with:
Did you forget a '.new'?
This is a succinct hint based on the reality that 99 times out of a hundred the reason a type object is encountered when an instance is required is that code has failed to initialize a variable. (One of the things type objects are used for is to serve the role of "undefined" in languages like Perl.)
So, can you see why something that should have been an initialized instance of 'IO::Socket::Async' is instead an uninitialized one?
Footnotes
[1] Raku's constructs for parallelism, concurrency, and asynchrony follow the structured programming paradigm. See Parallelism, Concurrency, and Asynchrony in Raku for Jonathan Worthington's video presentation of this overall approach. Structured constructs like react can cleanly observe, contain, and manage events that occur anywhere within their execution scope, including errors such as error exceptions, even if they happen on other threads.
You seem to be fine now but when I first saw this I made this https://github.com/jonathanstowe/Cro-MQTT which turns the MQTT client into a first class Cro service.
I haven't released it yet but it may be instructive.
Can I use antlr v4 for syntax check before I actually run the code?
Example :
I defined syntax: select * from table, I want to know the statement is correct or not before actually executing it.
Following is my code :
val listener = new SQLListener()
val loadLexer = new SQLLexer(new ANTLRInputStream(input))
val tokens = new CommonTokenStream(loadLexer)
val parser = new SQLParser(tokens)
val stat = parser.statement()
I tried but DefaultErrorStrategy won't throw an Exception
I tried this:
parser.addErrorListener(new BaseErrorListener {
override def syntaxError(recognizer: Recognizer[_, _ <: ATNSimulator],
offendingSymbol: scala.Any,
line: Int,
charPositionInLine: Int,
msg: String, e: RecognitionException ): Unit = {
println("==========2============"+msg)
throw new AssertionError("line: " + line + ", offset: " + charPositionInLine +
", symbol:" + offendingSymbol + " " + msg)
}
})
but get this:
Error: Note: the super classes of contain the following, non final members named syntaxError:
If the input contains any syntax errors, this will call the visitErrorNode method on the listener. So if you define that method in your listener, you'll see any errors that occur.
If your listener is directly executing the code (rather than first building an AST or other form of IR), you probably won't want your listener to even start executing when there's a syntax error. One way to achieve that would be to set the BailErrorStrategy instead of the DefaultErrorStrategy as the error handling strategy of your parser (using setErrorHandler on the parser). This will throw an exception as soon as a syntax error occurs.
If you don't want to abort on the first error and/or you want some additional checks beyond just syntax errors (like checking for certain types of semantic errors), an alternative is to have a listener just to perform those checks. Then you'd run your code-executing listener only if the error-checking listener does not find any errors.
You are on the right track here. Use your error listener to store the errors in a list while parsing. Afterwards you can then check that list.
That requires however not to do any action during the parsing process (e.g. in a parse listener) other than stuff related to the parsing process itself. Any follow up action (e.g. error markup in an editor) should be done after the parse run.
If you like to see an example of an application using this approach take a look at the parser module implementation of MySQL Workbench. It also demonstrates the 2-stage parsing strategy for quicker parsing.
I've read the various bits of literature, and I'm seeing the same problem that the questioner in
https://stackoverflow.com/a/25636911
was seeing.
My code looks like this:
coll = db.collection('foobar');
bulk = coll.initializeUnorderedBulkOp();
for entry in messages {
bulk.insert(entry);
}
bulk.execute(function (err, result) {
if (err) throw err
inserted += result.nInserted
});
bulk is an object
bulk.insert works just fine
bulk.execute is undefined
The answer in the stackoverflow question said, "only the callback flavor of db.collection() works, so I tried:
db.collection('foobar', function (err, coll) {
logger.debug "got here"
if (err) throw err
bulk = coll.initializeUnorderedBulkOp()
... same code as before
We never get to "got here" implying that the "callback flavor" of db.collection() was dropped for 3.0?
Unfortunately, my python is way better than my JS prototyping skills, so looking at the skin source code doesn't make any sense to me.
What is the right way, with mongoskin 2.1.0 and the 2.2.0 mongodb JS driver, to do a bulk operation, or is this not implemented at all anymore?
There are at least two answers:
(1) Use insert, but the array form, so you insert multiple documents with one call. Works like a charm.
(2) If you really need bulk operations, you'll need to switch from mongoskin to the native mongo interface, but just for that one call.
This kinda sucks because it's using a private interface in mongoskin, but it's also the most efficient way to stick with mongoskin:
(example in coffeescript)
// bulk write all the messages in "messages" to a collection
// and insert the server's current time in the recorded field of
// each message
// use the _native interface and wait for callback to get collection
db._native.collection collectionName, (err, collection) ->
bulk = collection.initializeUnorderedBulkOp()
for message in messages
bulk.find
_id: message._id
.upsert().updateOne
$set: message
$currentDate:
recorded: true
bulk.execute (err, result) ->
// ... error and result checking code
or (3) if you want to implement that $currentDate and not any generic bulk operation, refer to solution (1) but use the not-very-well-documented BSON object Timestamp() with no arguments:
for msg in messages:
msg.recorded = Timestamp()
db.mycollection.insert(msg)
which will do a bulk insert and set timestamp to the DB server's time at the time the record is written to the db.
Fairly new to node-webkit, so I'm still figuring out how everything works...
I have some logging in my app:
console.log("Name: %s", this.name);
It outputs to the browser console as expected:
Name: Foo
But in the invoking terminal, instead I get some fairly ugly output:
[7781:1115/085317:INFO:CONSOLE(43)] ""Name: %s" "Foo"", source: /file/path/to/test.js (43)
The numerical output within the brackets might be useful, but I don't know how to interpret it. The source info is fine. But I'd really like the printed string to be printf-style formatted, rather than shown as individual arguments.
So, is there a way to get stdout to be formatted either differently, or to call a custom output function of some sort so I can output the information I want?
I eventually gave up, and wrapped console.log() with:
log = function() {
console.log(util.format.apply(this, arguments));
}
The actual terminal console output is done via RenderFrameHostImpl::OnAddMessageToConsole in chromium, with the prefix info being generated via LogMessage::Init() in the format:
[pid:MMDD/HHMMSS:severity:filename(line)]
The javascript console.log is implemented in console.cc, via the Log() function. The printf style formatting is being done at a higher level, so that by the time the Log() function (or similar) are called, they are only passed a single string.
It's not a satisfying answer, but a tolerable workaround.
I was looking to create a command line interface to go along side my UI and had a similar problem. Along with not logging the values as I wanted I also wanted to get rid of the [pid:MMDD/HHMMSS:severity:filename(line)] output prefix so I added the following:
console.log = function (d) {
process.stdout.write(d + '\n');
};
so that console logging was set back to stdout without extra details. Unfortunately also a work around.
I'm learning go and trying to understand how to get more detailed error information out of the generic error type. The example I'll use is from the net package, specifically the DialTimeout function.
The signature is
func DialTimeout(network, address string, timeout time.Duration) (Conn, error)
The error type only defines an Error() string function. If I want to find out exactly why DialTimeout failed, how can I get that information? I found out that I can use type assertion to get the net.Error specific error:
con, err := net.DialTimeout("tcp", net.JoinHostPort(address, "22"),
time.Duration(5) * time.Second)
if err != nil {
netErr, ok := err.(net.Error)
if ok && netErr.Timeout() {
// ...
}
}
but that only tells me whether or not I had a timeout. For example, say I wanted to differentiate between a refused connection and no route to host. How can I do that?
Maybe DialTimeout is too high-level to give me that kind of detail, but even looking at syscall.Connect, I don't see how to get the specific error. It just says it returns the generic error type. Compare that to the Posix connect, which will let me know why it failed with the various return codes.
My general question is: how am I supposed to pull out error details from the generic error type if the golang docs don't tell me what type of errors may be returned?
Most networking operations return a *OpError which holds detailed information about the error
and implements the net.Error interface. So for most use cases it is sufficient to use net.Error
as you already did.
But for your case you'd want to assert the returned error to be *net.OpError and
use the internal error:
if err != nil {
if oerr, ok := err.(*OpError); ok {
// Do something with oerr.Err
}
}
As soon as you're doing this you are in the land of platform dependency as syscalls under Linux
can fail differently to those under Windows. For Linux you'd do something like this:
if oerr.Err == syscall.ECONNREFUSED {
// Connection was refused :(
}
The syscall package contains the important error constants for your platform. Unfortunately
the golang website only shows the syscall package for Linux amd64. See here for ECONNREFUSED.
Finding out types
The next time you're wondering what is actually returned by some function and you can't make
heads and tails of it, try using the %#v format specified in fmt.Printf (and friends):
fmt.Printf("%#v\n", err)
// &net.OpError{Op:"dial", Net:"tcp", Addr:(*net.TCPAddr)(0xc20006d390), Err:0x6f}
It will print detailed type information and is generally quite helpful.