Scheduling localhost serverless cron: The model could not be resolved in registry - express

I'm hosting an ExpressJS/NodeJS API on AWS Lambda with Serverless framework. The API uses Knex.js and Bookshelf.js ORM.
I want to test scheduling a cron job locally. I'm using serverless-offline-scheduler to do this.
Question: My API runs fine if I call it from my client, but if I call a function via serverless scheduler, it complains that no models are in the registry. Why is this? I've already definitely included all necessary Models at the top of the OrderService.js file.
{
"errorMessage": "The model User could not be resolved from the registry plugin.",
"errorType": "Error",
"stackTrace": [
"Error: The model User could not be resolved from the registry plugin.",
" at new ModelNotResolved (/Users/danielturcotte/Sites/d2c/api_v4/node_modules/bookshelf/lib/plugins/registry.js:70:133)",
Serverless.yml:
functions:
app:
handler: handler.handler
events: ...
dequeue:
handler: ./services/OrderService.dequeue // Call dequeue function
events:
- schedule: rate(1 minute)
The handler calls root/services/OrderService.dequeue function, which contains
...
const dequeue = async function() {
await checkQueuedOrders();
};
module.exports = {
dequeue,
};
In my knexService.js file, I register Bookshelf models to the registry to remove circular dependencies:
const knexfile = require('./knexfile');
const config = require('./environment');
const environment = config.env.NODE_ENV || 'development';
const knex = require('knex')(knexfile[environment]);
knex.client.pool.numPendingCreates();
const bookshelf = require('bookshelf')(knex);
bookshelf.plugin('registry'); // Resolve circular dependencies with relations
bookshelf.plugin('visibility');
bookshelf.plugin('pagination');
module.exports.knex = knex;
module.exports.bookshelf = bookshelf;

Related

What are cloudflare KV preview_ids and how to get one?

I have a following wrangler.toml. When I would like to use dev or preview (e.g. npx wrangler dev or npx wrangler preview) wrangler asks to add a preview_id to the KV namespaces. Is this an identifier to an existing KV namespace?
I see there is a ticket in Cloudflare Workers GH at https://github.com/cloudflare/wrangler/issues/1458 that tells this ought to be clarified but the ticket is closed with adding an error message
In order to preview a worker with KV namespaces, you must designate a preview_id in your configuration file for each KV namespace you'd like to preview."
which is the reason I'm here. :)
As for larger context I would be really glad if someone could clarify: I see that if I give a value of an existing namespace, I can preview and I see a KV namespace of type __some-worker-dev-1234-workers_sites_assets_preview is generated in Cloudflare. This has a different identifier than the KV namespace pointed by the identifier used in the preview_id and the KV namespace pointed by the identifier I used in preview_id is empty. Why does giving an identifier of an existing KV namespace remove the error message, deploys the assets and allow for previwing but the actual KV namespace is empty and a new one is created?
How do does kv-asset-handler know to look into this generated namespace to retrieve the assets?
I'm currently testing with the default generated Cloudare Worker to my site and I wonder if I have misunderstood something or if there is some mechanics that bundles during preview/publish the site namespace to the scipt.
If there is some random mechanics with automatic mapping, can this be then so that every developer can have their own private preview KV namespace?
type = "javascript"
name = "some-worker-dev-1234"
account_id = "<id>"
workers_dev = true
kv_namespaces = [
{ binding = "site_data", id = "<test-site-id>" }
]
[site]
# The location for the site.
bucket = "./dist"
# The entry directory for the package.json that contains
# main field for the file name of the compiled worker file in "main" field.
entry-point = ""
[env.production]
name = "some-worker-1234"
zone_id = "<zone-id>"
routes = [
"http://<site>/*",
"https://www.<site>/*"
]
# kv_namespaces = [
# { binding = "site_data", id = "<production-site-id>" }
# ]
import { getAssetFromKV, mapRequestToAsset } from '#cloudflare/kv-asset-handler'
/**
* The DEBUG flag will do two things that help during development:
* 1. we will skip caching on the edge, which makes it easier to
* debug.
* 2. we will return an error message on exception in your Response rather
* than the default 404.html page.
*/
const DEBUG = false
addEventListener('fetch', event => {
try {
event.respondWith(handleEvent(event))
} catch (e) {
if (DEBUG) {
return event.respondWith(
new Response(e.message || e.toString(), {
status: 500,
}),
)
}
event.respondWith(new Response('Internal Error', { status: 500 }))
}
})
async function handleEvent(event) {
const url = new URL(event.request.url)
let options = {}
/**
* You can add custom logic to how we fetch your assets
* by configuring the function `mapRequestToAsset`
*/
// options.mapRequestToAsset = handlePrefix(/^\/docs/)
try {
if (DEBUG) {
// customize caching
options.cacheControl = {
bypassCache: true,
}
}
const page = await getAssetFromKV(event, options)
// allow headers to be altered
const response = new Response(page.body, page)
response.headers.set('X-XSS-Protection', '1; mode=block')
response.headers.set('X-Content-Type-Options', 'nosniff')
response.headers.set('X-Frame-Options', 'DENY')
response.headers.set('Referrer-Policy', 'unsafe-url')
response.headers.set('Feature-Policy', 'none')
return response
} catch (e) {
// if an error is thrown try to serve the asset at 404.html
if (!DEBUG) {
try {
let notFoundResponse = await getAssetFromKV(event, {
mapRequestToAsset: req => new Request(`${new URL(req.url).origin}/404.html`, req),
})
return new Response(notFoundResponse.body, { ...notFoundResponse, status: 404 })
} catch (e) {}
}
return new Response(e.message || e.toString(), { status: 500 })
}
}
/**
* Here's one example of how to modify a request to
* remove a specific prefix, in this case `/docs` from
* the url. This can be useful if you are deploying to a
* route on a zone, or if you only want your static content
* to exist at a specific path.
*/
function handlePrefix(prefix) {
return request => {
// compute the default (e.g. / -> index.html)
let defaultAssetKey = mapRequestToAsset(request)
let url = new URL(defaultAssetKey.url)
// strip the prefix from the path for lookup
url.pathname = url.pathname.replace(prefix, '/')
// inherit all other props from the default request
return new Request(url.toString(), defaultAssetKey)
}
}
In case the format is not obvious (it wasn't to me) here is a sample config block from the docs with the preview_id specified for a couple of KV Namespaces:
kv_namespaces = [
{ binding = "FOO", id = "0f2ac74b498b48028cb68387c421e279", preview_id = "6a1ddb03f3ec250963f0a1e46820076f" },
{ binding = "BAR", id = "068c101e168d03c65bddf4ba75150fb0", preview_id = "fb69528dbc7336525313f2e8c3b17db0" }
]
You can generate a new namespace ID in the Workers KV section of the dashboard or with the Wrangler CLI:
wrangler kv:namespace create "SOME_NAMESPACE" --preview
This answer applies to versions of Wrangler >= 1.10.0
wrangler asks to add a preview_id to the KV namespaces. Is this an identifier to an existing KV namespace?
Yes! The reason there is a different identifier for preview namespaces is so that when developing with wrangler dev or wrangler preview you don't accidentally write changes to your existing production data with possibly buggy or incompatible code. You can add a --preview flag to most wrangler kv commands to interact with your preview namespaces.
For your situation here there are actually a few things going on.
You are using Workers Sites
You have a KV namespace defined in wrangler.toml
Workers Sites will automatically configure a production namespace for each environment you run wrangler publish on, and a preview namespace for each environment you run wrangler dev or wrangler preview on. If all you need is Workers Sites, then there is no need at all to specify a kv-namepsaces table in your manifest. That table is for additional KV namespaces that you may want to read data from or write data to. If that is what you need, you'll need to configure your own namespace and add id to wrangler.toml if you want to use wrangler publish, and preview_id (which should be different) if you want to use wrangler dev or wrangler preview.

"Execution failed" when setting up API Gateway and Fargate with AWS CDK

I am trying to setup AWS API Gateway to access a fargate container in a private VPC as described here. For this I am using AWS CDK as described below. But when I curl the endpoint after successful cdk deploy I get "Internal Server Error" as a response. I can't find any additional information. For some reason API GW can't reach the container.
So when I curl the endpoint like this:
curl - i https://xxx.execute-api.eu-central-1.amazonaws.com/prod/MyResource
... I get the following log output in cloud watch:
Extended Request Id: NpuEPFWHliAFm_w=
Verifying Usage Plan for request: 757c6b9e-c4af-4dab-a5b1-542b15a1ba21. API Key: API Stage: ...
PI Key authorized because method 'ANY /MyResource/{proxy+}' does not require API Key. Request will not contribute to throttle or quota limits
Usage Plan check succeeded for API Key and API Stage ...
Starting execution for request: 757c6b9e-c4af-4dab-a5b1-542b15a1ba21
HTTP Method: GET, Resource Path: /MyResource/test
Execution failed due to configuration error: There was an internal error while executing your request
CDK Code
First I create a network load balanced fargate service:
private setupService(): NetworkLoadBalancedFargateService {
const vpc = new Vpc(this, 'MyVpc');
const cluster = new Cluster(this, 'MyCluster', {
vpc: vpc,
});
cluster.connections.allowFromAnyIpv4(Port.tcp(5050));
const taskDefinition = new FargateTaskDefinition(this, 'MyTaskDefinition');
const container = taskDefinition.addContainer('MyContainer', {
image: ContainerImage.fromRegistry('vad1mo/hello-world-rest'),
});
container.addPortMappings({
containerPort: 5050,
hostPort: 5050,
});
const service = new NetworkLoadBalancedFargateService(this, 'MyFargateServie', {
cluster,
taskDefinition,
assignPublicIp: true,
});
service.service.connections.allowFromAnyIpv4(Port.tcp(5050));
return service;
}
Next I create the VpcLink and the API Gateway:
private setupApiGw(service: NetworkLoadBalancedFargateService) {
const api = new RestApi(this, `MyApi`, {
restApiName: `MyApi`,
deployOptions: {
loggingLevel: MethodLoggingLevel.INFO,
},
});
// setup api resource which forwards to container
const resource = api.root.addResource('MyResource');
resource.addProxy({
anyMethod: true,
defaultIntegration: new HttpIntegration('http://localhost.com:5050', {
httpMethod: 'ANY',
options: {
connectionType: ConnectionType.VPC_LINK,
vpcLink: new VpcLink(this, 'MyVpcLink', {
targets: [service.loadBalancer],
vpcLinkName: 'MyVpcLink',
}),
},
proxy: true,
}),
defaultMethodOptions: {
authorizationType: AuthorizationType.NONE,
},
});
resource.addMethod('ANY');
this.addCorsOptions(resource);
}
Anyone has a clue what is wrong with this config?
After hours of trying I finally figured out that the security groups do not seem to be updated correctly when setting up the VpcLink with CDK. Broadening the allowed connection with
service.service.connections.allowFromAnyIpv4(Port.allTraffic())
solved it. Still need to figure out which minimum set needs to be set instead of allTrafic()
Additionally I replaced localhost in the HttpIntegration by the endpoint of the load balancer like this:
resource.addMethod("ANY", new HttpIntegration(
'http://' + service.loadBalancer.loadBalancerDnsName,
{
httpMethod: 'ANY',
options: {
connectionType: ConnectionType.VPC_LINK,
vpcLink: new VpcLink(this, 'MyVpcLink', {
targets: [service.loadBalancer],
vpcLinkName: 'MyVpcLink',
})
},
}
))

Frequent timeout with app using Serverless Framework (AWS Lambda/Gateway), Express, Mongoose/MongoDB Atlas

Trigger warning : Beginner question.
I built an api using Express and Mongoose with a MongoDB Atlas DB.
Most of the time, it works normally, but often I get timeout errors. This seems to happen very randomly and concerns all routes, etc... Precisely, I get :
`502 Internal server error via POSTMAN`
and in the Serverless Dashboard, I get :
invocation
time invoked 1 day ago, mar 08 at 1:38pm
fatal error Function execution duration going to exceeded configured timeout limit.
cold start
duration 48.9 s
memory used na
request
endpoint /{proxy+}
method POST
status 502
message Internal server error
latency 27 ms
and span & log :
I used this tutorial to wrap my express app to deploy it with serverless framework : https://dev.to/adnanrahic/a-crash-course-on-serverless-apis-with-express-and-mongodb-193k
Serverless.yml file :
service: serviceName
app: appName
org: orgName
provider:
name: aws
runtime: nodejs12.x
stage: ${env:NODE_ENV}
region: eu-central-1
environment:
NODE_ENV: ${env:NODE_ENV}
DB: ${env:DB}
functions:
app:
handler: server.run
events:
- http:
path: /
method: ANY
cors: true
- http:
path: /{proxy+}
method: ANY
cors: true
plugins:
- serverless-offline # Utiliser pour tester localement
- serverless-dotenv-plugin
server.js file :
const sls = require('serverless-http')
const app = require('./app')
module.exports.run = sls(app)
app.js file :
const express = require('express')
const cors = require('cors')
const bodyParser = require('body-parser')
const newRoutes = require('./routes/file')
const app = express()
app.use(bodyParser.json())
const helmet = require('helmet')
app.use(helmet())
app.options('*', cors())
app.use(cors({ allowedHeaders: 'Content-Type, Authorization' }))
app.use('/new-route', newRoutes)
app.use((error, req, res, next) => {
console.log(error)
const status = error.status || 500
const message = error.message
res.status(status).json({
status: status,
message: message
})
})
// Gère la connexion à la base de donnée :
require('./db')
module.exports = app
and finally db.js file :
const mongoose = require('mongoose')
mongoose
.connect(
process.env.DB, {
useNewUrlParser: true,
useUnifiedTopology: true
})
.then(() => {
console.log('connected')
})
.catch(err => console.log(err))
From what I have read, it is related to cold start in Lambda and the way API Gateway handles timeouts (!?). I have read this on mongoose documentation (https://mongoosejs.com/docs/lambda.html), and read also other tutorials, but I don't how exaclty I should adapt it to my situation.
Thank you for your help
Under your provider add timeout, maximum value of timeout in lambda is
900 seconds, place it according to your execution time like 30 seconds
and see what happens
provider:
timeout: 30
The error is clearly saying that it's execution exceeded timeout, since you have not configured timeout so it was using default timeout of 3 seconds, hopefully it will solve the issue
The issue is likely due to your open database connection. While this connection is established any calls to callback won't be returned to the client and your function will timeout.
You need to set context.callbackWaitsForEmptyEventLoop to false.
Here is the explanation from the docs:
callbackWaitsForEmptyEventLoop – Set to false to send the response right away when the callback executes, instead of waiting for the Node.js event loop to be empty. If this is false, any outstanding events continue to run during the next invocation.
With serverless-http you can set this option quite easily within your server.js file:
const sls = require('serverless-http')
const app = require('./app')
module.exports.run = sls(app, { callbackWaitsForEmptyEventLoop: false })

Botkit slackbot error "Could not load team while processing webhook"

I have created a simple express server and added a /slack/receive route to handle webhook requests from the Slack events API:
// routes.js (which is used by my app defined in server.js)
...
let slack = require('./controllers/slack');
router.post('/slack/receive', slack.receive);
...
I then use Botkit to create a simple Slack application:
// controllers/slack.js
'use strict';
const logger = require('../config/winston');
// initialise firebase storage for botkit
const admin = require('firebase-admin');
var serviceAccount = require('../config/firebase.json');
admin.initializeApp({
credential: admin.credential.cert(serviceAccount)
});
var db = admin.firestore();
db.settings({
timestampsInSnapshots: true
})
// initialise botkit for slack
const botkit = require('botkit');
const controller = botkit.slackbot({
storage: require('botkit-storage-firestore')({ database: db }),
clientId: process.env.SLACK_CLIENT_ID,
clientSecret: process.env.SLACK_CLIENT_SECRET,
clientSigningSecret: process.env.SLACK_SIGNING_SECRET,
redirectUri: process.env.SLACK_REDIRECT,
disable_startup_messages: true,
send_via_rtm: false,
debug: true,
scopes: ['bot', 'chat:write:bot'],
})
controller.hears('Hello', 'direct_mention,direct_message', (bot, message) => {
logger.info(message);
bot.reply(message, 'I heard a message!');
})
exports.receive = (req, res, next) => {
res.sendStatus(200);
logger.debug(req.body);
controller.handleWebhookPayload(req, res);
};
The server initialises correctly, but as soon as the slack webhook receives a request the following error happens:
Could not load team while processing webhook: Error: could not find team T5VDRMWKX
at E:\Documents\upper-revolutions\node_modules\botkit\lib\SlackBot.js:169:24
at firebaseRef.doc.get.then.catch.err (E:\Documents\upper-revolutions\node_modules\botkit-storage-firestore\src\index.js:86:13)
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:118:7)
So far I have found that:
Having/not having storage in the botkit slackbot makes no difference
The error happens within the handleWebhookPayload method as code within controller.hears() does not get executed
This error occurs because botkit needs some form of storage where it can store all the teams (channels and users too) and retrive it later on.
So, When your method handleWebhookPayload gets executed it calls another method called
findAppropriateTeam that will query for the specified team record in the storage provided by you (It might be mongoDB or a JSON file or other). The error is saying that you do not have any record in the storage with the id provided.
So this might implicate two things:
You did not provide a storage for botkit to work
You did not save the team id in the storage
The solution to the first problem is quite simple. You just need to install mongodb in your machine and then pass to botkit the MONGO_URL.
NOTE: I see that you are using the botkit simple storage and this might be the problem since I also have experieced some troubles with this kind of storage not saving records.
const controller = botkit.slackbot({
storage: 'mongodb//localhost:27017:/yourdb',
})
//OR
const controller = botkit.slackbot({
storage: process.env.MONGO_URL,
})
The possible solution to the second problem:
I will assume you are using botkit locally, so you must be using some tunneling like ngrok or localtunnel. In that case make sure:
You provided the redirect URL to Slack (Ex, https://your_url/oauth)
You accessed the https://your_url/login page
Botkit saves your team id on the provided storage when you access the /login route and authorizes the app. So if you skipped that part then botkit won't save your team id and therfore will throw an error when you receive events later on.
Check this like [https://github.com/howdyai/botkit/issues/938] for discutions on the topic
I hope this helps!

How is everyone going about implementing scheduled jobs / cloud jobs on parse-server?

According to the parse-server migration guide we could use something like Kue and Kue-UI to emulate the parse.com scheduled jobs functionality.
I haven't implemented Kue or Kue-ui, but looking at the guides, it doesn't look like it provides anywhere close to the same level of functionality as the existing parse.com scheduled jobs. Is this observation correct? Has someone implemented this? Is it true that jobs have to be scheduled through Kue in javascript and Kue-ui only provides a summary of the current status of the jobs and new schedules can't be added through Kue-ui?
Has anyone tried to achieve the same outcome with something like Jenkins? So this is what I had in mind:
each job would still be defined in the cloud code Parse.Cloud.job("job01", function(request, response) {));
modify parse-server slightly to expose job at a similar url to existing cloud functions e.g. /parse/jobs/job01 (this might exist in parse-server soon: github.com/ParsePlatform/parse-server/pull/2560)
create a new jenkins job do a curl at that url
define a cron like schedule for that jenkins job from within the jenkins web ui
I can see the benefits being:
little to no coding
setting up jenkins sounds like much less work then setting up kue, redis and kue-ui
existing cloud job / definitions stay exactly the same
schedule and manually trigger jobs through the jenkins web ui
The only thing that the current parse.com schedule jobs / cloud jobs can do that a jenkins based solution can't is being able to select a job name to create a new schedule for from a drop down list.
Am I missing something? How is everyone else going about this? Thanks.
I ended up doing this with ´node-schedule´. Not sure if it is the best option but it is working fine for me.
index.js
var schedule = require('node-schedule');
var request = require('request');
schedule.scheduleJob('*/15 * * * *', function() {
var options = {
url: serverUrl + '/functions/{function name}}',
headers: {
'X-Parse-Application-Id': appID,
'X-Parse-Master-Key': masterKey,
'Content-Type': 'application/json'
}
};
request.post(options, function (error, response, body) {
if (!error && response.statusCode == 200) {
console.log(body);
}
});
});
main.js
Parse.Cloud.define('{function name}', function(req, res) {
//Check for master key to prevent users to call this
if (req.master === true) {
//Do operations here
} else {
res.error('Need Master Key');
}
});
I use kue for this purpose. I've described the appoach in my article. In short, this function:
Parse.Cloud.job("sendReport", function(request, response) {
Parse.Cloud.httpRequest({
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
url: "https://example.com/url/", // Webhook url
body: "body goes here",
success: function(httpResponse) {
console.log("Successfully POSTed to the webhook");
},
error: function(httpResponse) {
console.error("Couldn't POST to webhook: " + httpResponse);
}
});
});
becomes this:
// Create a kue instance and a queue.
var kue = require('kue-scheduler');
var Queue = kue.createQueue();
var jobName = "sendReport";
// Create a job instance in the queue.
var job = Queue
.createJob(jobName)
// Priority can be 'low', 'normal', 'medium', 'high' and 'critical'
.priority('normal')
// We don't want to keep the job in memory after it's completed.
.removeOnComplete(true);
// Schedule it to run every 60 minutes. Function every(interval, job) accepts interval in either a human-interval String format or a cron String format.
Queue.every('60 minutes', job);
// Processing a scheduled job.
Queue.process(jobName, sendReport);
// The body of job goes here.
function sendReport(job, done) {
Parse.Cloud.httpRequest({
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
url: "https://example.com/url/", // Webhook url
body: "body goes here"}).then(function(httpResponse) {
console.log("Successfully POSTed to the webhook");
// Don't forget to run done() when job is done.
done();
}, function(httpResponse) {
var errorMessage = "Couldn't POST to webhook: " + httpResponse;
console.error(errorMessage);
// Pass Error object to done() to mark this job as failed.
done(new Error(errorMessage));
});
}
It works fine, though I noticed that sometimes kue-scheduler fires the event more often than needed. See this issue for more info: https://github.com/lykmapipo/kue-scheduler/issues/45
If you are on AWS this may be an option:
Create a AWS CloudWatch Event Rule that is triggered at specific intervals and calls a Lambda function. The Event Rule can pass parameters to the Lambda function.
Create a simple Lambda function that calls a Cloud Code function / job. If you receive the cloud code function name and other parameters from the Event Rule, you only need one generic Lambda function for any Cloud Code call.
This has several advantages as Event Rules are part of AWS infrastructure and can be easily integrated with other AWS services. For example, you can set up intelligent queueing of Event Rule calls so that if the previous call did not complete yet, you discard the next call in the queue, overflow to another queue, notify an operator, etc.
You can use the parse-server-scheduler npm-module.
It does not require any external server and simply allows you to setup scheduling in parse-dashboard.