I defined the following module in the Modules database under '/test/test.mjs':
// test.mjs
export function test(message){
xdmp.log("Received: "+message);
return "Received: "+message;
}
I have the following code in the query console and I run it from a test database:
'use strict';
declareUpdate(); // I do not need here in this test case but I need it for my real code
import { test } from '/test/test.mjs';
test("Hello");
It is working fine the first time.
If I run it again, I am getting:
[javascript] XDMP-MODNOTFOUND: result: evalScriptOrModule(q) -- Module /test/test.mjs not found"
If I am waiting between 5 or 6 minutes and I run it again, it is working.
=> So it seems that a timeout is set somewhere when I am using a module inside the Query console (between 2 executions). Could you please give me an hint about it?
We solved the problem. It seems that it was coming from the configuration of the app server that we defined. Since we played also with it. The root was not equal to "/" anymore. We set it to "/" and now we do not have timeout problem between 2 executions.
I am attempting to put together a Cro service that has a react/whenever block consuming data "in the background" So unlike many examples of websocket usage with Cro, this has nothing to do with routes that may be accessed via the browser.
My use case is to consume message received via an MQTT topic and do some processing with them. At a later stage in development I might create a supply out of this data, but for now, when data is received it will be stored in a variable and dependant on certain conditions, be sent to another service via a http post.
My thought was to include a provider() in the Cro::HTTP::Server setup like so:
use Cro::HTTP::Log::File;
use Cro::HTTP::Server;
use Routes;
use DataProvider; # Here
my Cro::Service $http = Cro::HTTP::Server.new(
http => <1.1>,
host => ...,
port => ...,
application => [routes(), provider()], # Made this into an array of subs?
after => [
Cro::HTTP::Log::File.new(logs => $*OUT, errors => $*ERR)
]
);
And in the DataProvider.pm6:
use MQTT::Client;
sub provider() is export {
my $mqtt = MQTT::Client.new: server => 'localhost';
react {
whenever $mqtt.subscribe('some/mqtt/topic') {
say "+ topic: { .<topic> } => { .<message>.decode("utf8-c8") }";
}
}
}
This throws a bunch of errors:
A react block:
in sub provider at DataProvider.pm6 (DataProvider) line 5
in block <unit> at service.p6 line 26
Died because of the exception:
Invocant of method 'write' must be an object instance of type
'IO::Socket::Async', not a type object of type 'IO::Socket::Async'. Did
you forget a '.new'?
in method subscribe at /home/cam/raku/share/perl6/site/sources/42C762836A951A1C11586214B78AD34262EC465F (MQTT::Client) line 133
in sub provider at DataProvider.pm6 (DataProvider) line 6
in block <unit> at service.p6 line 26
To be perfectly honest, I am totally guessing that this is how I would approach the need to subscribe to data in the background of a Cro service, but I was not able to find any information on what might be considered the recommended approach.
Initially I had my react/whenever block in the main service.pm6 file, but that did not seem right. And needed to be wrapped in a start{} block because as I have just learned, react is blocking :) and cro was not able to actually start.
But following the pattern of how Routes are implemented seemed logical, but I am missing something. The error speaks about setting up a new method, but I'm not convinced that is the root cause. Routes.pm6 does not have a constructor.
Can anyone point me in the right direction please?
Thanks to all who have provided information, this has been a very valuable learning exercise.
The approach of passing additional sub routines, along side router() in the application parameter to Cro::HTTP::Server.new gave further trouble. (an array is not allowed, and broke routing)
Instead, I have moved the background work into a class of it's own, and given it a start and stop method more akin to Cro::HTTP::Server.
My new approach:
service.pm6
use Cro::HTTP::Log::File;
use Cro::HTTP::Server;
use Routes;
use KlineDataSubscriber; # Moved mqtt functionality here
use Database;
my $dsn = "host=localhost port=5432 dbname=act user=.. password=..";
my $dbh = Database.new :$dsn;
my $mqtt-host = 'localhost';
my $subscriber = KlineDataSubscriber.new :$mqtt-host;
$subscriber.start; # Inspired by $http.start below
my Cro::Service $http = Cro::HTTP::Server.new(
http => <1.1>,
host => ...,
port => ...,
application => routes($dbh), # Basically back the way it was originally
after => [
Cro::HTTP::Log::File.new(logs => $*OUT, errors => $*ERR)
]
);
$http.start;
say "Listening at...";
react {
whenever signal(SIGINT) {
say "Shutting down...";
$subscriber.stop;
$http.stop;
done;
}
}
And in KlineDataSubscriber.pm6
use MQTT::Client;
class KlineDataSubscriber {
has Str $.mqtt-host is required;
has MQTT::Client $.mqtt = Nil;
submethod TWEAK() {
$!mqtt = MQTT::Client.new: server => $!mqtt-host;
await $!mqtt.connect;
}
method start(Str $topic = 'act/feed/exchange/binance/kline-closed/+/json') {
start {
react {
whenever $!mqtt.subscribe($topic) {
say "+ topic: { .<topic> } => { .<message>.decode("utf8-c8") }";
}
}
}
}
method stop() {
# TODO Figure how to unsubscribe and cleanup nicely
}
}
This feels much more "Cro idiomatic" to me, but I would be happy to be corrected.
More importantly, it works as expected and I feel is somewhat future proof. I should be able to create a supply to make real-time data available to the router, and push data to any connected web clients.
I also intend to have a http GET endpoint /status with various checks to ensure everything healthy
The root cause
The error speaks about setting up a new method, but I'm not convinced that is the root cause.
It's not about setting up a new method. It's about a value that should be defined instead being undefined. That typically means a failure to attempt to initialize it, which typically means a failure to call .new.
Can anyone point me in the right direction please?
Hopefully this question helps.
Finding information on a recommended approach
I am totally guessing that this is how I would approach the need to subscribe to data in the background of a Cro service, but I was not able to find any information on what might be considered the recommended approach.
It might be helpful for you to list which of the get-up-to-speed steps you've followed from Getting started with Cro, including the basics but also the "Learn about" steps at the end.
The error message
A react block:
in sub provider ...
Died because of the exception:
...
in method subscribe ...
The error message begins with the built in react construct reporting that it caught an exception (and handled it by throwing its own exception in response). A "backtrace" corresponding to where the react appeared in your code is provided indented from the initial "A react block:".
The error message continues with the react construct summarizing its own exception (Died because ...) and explains itself by reporting the original exception, further indented, in subsequent lines. This includes another backtrace, this time one corresponding to the original exception, which will likely have occurred on a different thread with a different callstack.
(All of Raku's structured multithreading constructs[1] use this two part error reporting approach for exceptions they catch and handle by throwing another exception.)
The first backtrace indicates the react line:
in sub provider at DataProvider.pm6 (DataProvider) line 5
use MQTT::Client;
sub provider() is export {
my $mqtt = MQTT::Client.new: server => 'localhost';
react {
The second backtrace is about the original exception:
Invocant of method 'write' must be an object instance of type
'IO::Socket::Async', not a type object of type 'IO::Socket::Async'. ...
in method subscribe at ... (MQTT::Client) line 133
This reports that the write method called on line 133 of MQTT::Client requires its invocant is an instance of type 'IO::Socket::Async'. The value it got was of that type but was not an instance, but instead a "type object". (All values of non-native types are either type objects or instances of their type.).
The error message concludes with:
Did you forget a '.new'?
This is a succinct hint based on the reality that 99 times out of a hundred the reason a type object is encountered when an instance is required is that code has failed to initialize a variable. (One of the things type objects are used for is to serve the role of "undefined" in languages like Perl.)
So, can you see why something that should have been an initialized instance of 'IO::Socket::Async' is instead an uninitialized one?
Footnotes
[1] Raku's constructs for parallelism, concurrency, and asynchrony follow the structured programming paradigm. See Parallelism, Concurrency, and Asynchrony in Raku for Jonathan Worthington's video presentation of this overall approach. Structured constructs like react can cleanly observe, contain, and manage events that occur anywhere within their execution scope, including errors such as error exceptions, even if they happen on other threads.
You seem to be fine now but when I first saw this I made this https://github.com/jonathanstowe/Cro-MQTT which turns the MQTT client into a first class Cro service.
I haven't released it yet but it may be instructive.
I am writing code to consume WinRT API on Windows 10 using C++ WRL (i.e. without using either C++/CX or C++/CLI). I am using GetAppListEntriesAsync() from Package class to get names of UWP applications installed on a system. GetAppListEntriesAsync() calls a callback method asynchronously. I want to wait till all such asynchronous methods have finished. Unlike C++/CX WRL doesn't have any await keyword to wait on asynchronous tasks.
Details
The following code is used to get the list.
ComPtr<ABI::Windows::Foundation::__FIAsyncOperation_1___FIVectorView_1_Windows__CApplicationModel__CCore__CAppListEntry_t> operation;
package3->GetAppListEntriesAsync(&operation);
package3 is pointer to type ABI::Windows::ApplicationModel::IPackage3:-
ComPtr<ABI::Windows::ApplicationModel::IPackage3> package3;
I invoke the asynchronous operation as follows.
operation->put_Completed(cb.Get())
Where cb holds reference to callback function that GetAppListEntriesAsync() calls asynchronously. (see update2)
There are several such asynchronous operations which are invoked. I want to run a piece of code which should run only after all such asynchronous callbacks have completed. C++/CX has await keyword for doing a wait on asynchronous tasks.
How should I do the same using WRL C++?
I tried using WaitForSingleObject() but it doesn't seem to work.
A very brute approach would be to keep checking status of every asynchronous operation in a while loop.
ComPtr<ABI::Windows::Foundation::IAsyncInfo> ai;
operation.As(&ai);
while (true)
{
ai->get_Status(&status);
if (status == Started)
{
continue;
}
else
{
break;
}
}
I am not sure if this is guaranteed to work, or whether if it will incur performance penalty. The COM apartment type for my code is APTTYPE_STA (note sure if this piece of information is relevant to the question; I wanted to set is to RO_INIT_MULTITHREADED but RoInitialize() fails, however WRL code seems to be working fine).
UPDATE 2:
Here is how cb (callback) is implemented.
auto cb = wrl::Callback<ABI::Windows::Foundation::IAsyncOperationCompletedHandler<ABI::Windows::Foundation::Collections::__FIVectorView_1_Windows__CApplicationModel__CCore__CAppListEntry_t *>>(
[](ABI::Windows::Foundation::__FIAsyncOperation_1___FIVectorView_1_Windows__CApplicationModel__CCore__CAppListEntry_t * ppp, AsyncStatus status) -> HRESULT
{
<do something in callback>;
SetEvent(threadCompleted.Get());
}
Trying to implement IAgileObject.
To make my callback implement IAgileObject I will have to use Implements<> WRL template class. so callback would be something like auto cb = Callback<Implements<IAgileObject,...>> but the problem is that IAsyncOperationCompletedHandler<ABI::Windows::Foundation::Collections::__FIVectorView_1_Windows__CApplicationModel__CCore__CAppListEntry_t *> doesn't derive from IInspectable, hence using Implements on that :-
Callback<Implements<ABI::Windows::Foundation::IAsyncOperationCompletedHandler<ABI::Windows::Foundation::Collections::__FIVectorView_1_Windows__CApplicationModel__CCore__CAppListEntry_t *>,IAgileObject>> gives the following error (Visual Studio 2017).
Error C2338 'I' has to derive from 'IWeakReference', 'IWeakReferenceSource' or 'IInspectable' AsyncTask c:\sw\tools\sdk\winsdk\win10\include\winrt\wrl\implements.h 413
My use case varies for this, but in general i'm trying to collect a bunch of elements and then apply _.map() to each. The problem is that this series of .getAttribute() calls can cause a test that works locally to fail against a remote server like sauce/android.
One example: collecting all <div class='article'><a href='articles/{id}'> on a page and then getting the hrefs. it might look something like this, and this approach will work until i test on a mobile (android) sauce environment. then I get a timeout.
Is it possible this is an issue related to my android environment capabilities? To piling up so many requests? I've tried scaling my test down from using 75 articles to only 45 and i've upped the timeout to 60s and still the mobile test fails. locally with chromedriver is fine, chrome desktop + sauce is fine.
Not my actual test but an approximation of the code i'm talking about:
/// ... return this.remote
.findAllByTagName('div.article a')
.then(function (articles) {
var promises = articles.map(function(article) {
return article.getAttribute('href');
});
Promise.all(promises)
.then(function (hrefs) {
uniques = _.uniq(hrefs);
assert(hrefs.length === uniques.length);
});
});
Since you're seeing a timeout error, I'd suggest continuing to increase the test timeout until the test passes. The mobile testing environments on Sauce are both slower to initialize and slower to operate than the desktop environments, so it's quite possible that a test with many requests is simply very slow.
One way to speed things up would be to use an execute block to gather the references, like:
.then(function (articles) {
return this.parent.execute(function (articles) {
return articles.map(function (node) {
return node.getAttribute('href');
});
}, [ articles ]);
})
In the above snippet, the articles element array is passed as an argument to the execute block. The remote WebDriver will deserialize the element references into actual DOM elements that can be operated on in the execute code. This is significantly more efficient than using individual getAttribute requests for each element since only a single request will be made to the remote browser.
Few month back I was working on some Odata WCF project and I had some problems with parsing custom headers for token auth (apiKey).
At that time, being quite a noob (still am!), I posted this SO question: JayData oData request with custom headers
Today I am working on a new project with Jaydata Odata server and client library and this:
application.context.prepareRequest = function (r) {
r[0].headers['apikey'] = '123456';
};
was working fine till I had to do a MERGE request. I found out that somehow MERGE request was overriding my headers so I investigated further.
It appears at first that in the oDataProvider.js (~line 617) in the _saveRest method the headers are not inherited:
request = {
requestUri: this.providerConfiguration.oDataServiceHost + '/',
headers: {
MaxDataServiceVersion: this.providerConfiguration.maxDataServiceVersion
}
};
but a few lines later we get:
this.context.prepareRequest.call(this, requestData);
which "should" call my own prepareRequest, but doesnt... Instead it still points to:
//Line 11302 jaydata.js
prepareRequest: function () { },
which of course does... nothing! Funnilly enough, when you execute a simple GET the same code supposedly on the same context instance works and points to my prepareRequest override.
I can assert with enough confidence that somehow the context between GET/MERGE is not the same instance. I cant see, however, any place where the context instance is reassigned.
Has anyone got a clue?
PS: this is NOT a CORS issue. My OPTIONS is passing fine and manually feeding the headers in oDataProvider works.
More
I followed the lead on different context instances and found something interesting. calling EntitySet.save() ends up calling the EntityContext constructor. see trace:
$data.Class.define.constructor (jaydata.js:10015)
EntityContext (VM110762:7)
Service (VM110840:8)
storeToken.factory (jaydata.js:14166)
$data.Class.define._getContextPromise (jaydata.js:13725)
$data.Class.define._getStoreContext (jaydata.js:13700)
$data.Class.define._getStoreEntitySet (jaydata.js:13756)
$data.Class.define.EntityInstanceSave (jaydata.js:13837)
$data.Entity.$data.Class.define.save (jaydata.js:9774)
(anonymous function) (app.js:162) // My save()
That explains why I get two different instances...
Hack
Replacing the prepareRequest function direcly in the class definition works, but its ugly!
for now I can cope with this:
$data.EntityContext.prototype.prepareRequest = function (r) {
r[0].headers['apikey'] = '12345';
};
This works fine as long as you only need to talk to a single endpoint.
Final word based on my experience
As much as I like JayData, it is obvious that they created a monster and its getting out of their hands (poor forum, no community, half-documented,...).
I chose JD because I was lazy and wanted to keep working with my old WCF DataService. Switching to Web API seemed wrong or too much work for me.
Also as a .net dev I liked strong typing of my entities and the ability to work with a concrete model generated from the JD tools. However, in the end, I was adding confusion. Every time my server side model changed I had to fetch the new metadata and scaffold a new entityModel.
I ended up by switching to Web Api and migrated my data service layer to Breeze. And seriously! its a breeze to work with it!
The documentation is absolutely brilliant and here on S.O you can always count on Ward or Jay Tarband to reply with a very high amount of professionalism.
In the end I realize this should probably be more a wiki than a Question.....