Gulp task to SSH and then mysqldump - ssh

So I've got this scenario where I have separate Web server and MySQL server, and I can only connect to the MySQL server from the web server.
So basically everytime I have to go like:
step 1: 'ssh -i ~/somecert.pem ubuntu#1.2.3.4'
step 2: 'mysqldump -u root -p'password' -h 6.7.8.9 database_name > output.sql'
I'm new to gulp and my aim was to create a task that could automate all this, so running one gulp task would automatically deliver me the SQL file.
This would make the developer life a lot easier since it would just take a command to download the latest db dump.
This is where I got so far (gulpfile.js):
////////////////////////////////////////////////////////////////////
// Run: 'gulp download-db' to get latest SQL dump from production //
// File will be put under the 'dumps' folder //
////////////////////////////////////////////////////////////////////
// Load stuff
'use strict'
var gulp = require('gulp')
var GulpSSH = require('gulp-ssh')
var fs = require('fs');
// Function to get home path
function getUserHome() {
return process.env.HOME || process.env.USERPROFILE;
}
var homepath = getUserHome();
///////////////////////////////////////
// SETTINGS (change if needed) //
///////////////////////////////////////
var config = {
// SSH connection
host: '1.2.3.4',
port: 22,
username: 'ubuntu',
//password: '1337p4ssw0rd', // Uncomment if needed
privateKey: fs.readFileSync( homepath + '/certs/somecert.pem'), // Uncomment if needed
// MySQL connection
db_host: 'localhost',
db_name: 'clients_db',
db_username: 'root',
db_password: 'dbp4ssw0rd',
}
////////////////////////////////////////////////
// Core script, don't need to touch from here //
////////////////////////////////////////////////
// Set up SSH connector
var gulpSSH = new GulpSSH({
ignoreErrors: true,
sshConfig: config
})
// Run the mysqldump
gulp.task('download-db', function(){
return gulpSSH
// runs the mysql dump
.exec(['mysqldump -u '+config.db_username+' -p\''+config.db_password+'\' -h '+config.db_host+' '+config.db_name+''], {filePath: 'dump.sql'})
// pipes output into local folder
.pipe(gulp.dest('dumps'))
})
// Run search/replace "optional"
SSH into the web server runs fine, but I have an issue when trying to get the mysqldump, I'm getting this message:
events.js:85
throw er; // Unhandled 'error' event
^
Error: Warning:
If I try the same mysqldump command manually from the server SSH, I get:
Warning: mysqldump: unknown variable 'loose-local-infile=1'
Followed by the correct mylsql dump info.
So I think this warning message is messing up my script, I would like to ignore warnings in cases like this, but don't know how to do it or if it's possible.
Also I read that using the password directly in the command line is not really good practice.
Ideally, I would like to have all the config vars loaded from another file, but this is my first gulp task and not really familiar with how I would do that.
Can someone with experience in Gulp orient me towards a good way of getting this thing done? Or do you think I shouldn't be using Gulp for this at all?
Thanks!

As I suspected, that warning message was preventing the gulp task from finalizing, I got rid of it by commenting the: loose-local-infile=1 From /etc/mysql/my.cnf

Related

Using Deno to interact through SSH hangs on p.output()

I'm looking to create an SSH sub-process and then interact with the server. I'm hung up on a basic step which is to simply wait until the SSH process has connected. I know that this ssh command connects fine because when I run it with inherit instead of piped, the ssh shell shows up as expected.
If I understand correctly, p.output() listens for stdout until it reaches EOF. I'm assuming that when SSH has connected, it streams the stdout, but does not EOF, and so p.output() never gets called.
const encoder = new TextEncoder();
const decoder = new TextDecoder();
const p=Deno.run({
cmd: ["ssh", "root#mywebsite"],
stdout: "piped",
stderr: "piped",
stdin: "piped"
});
const command = (cmd : string) => p.stdin.write(encoder.encode(cmd))
const getOutput = async () => decoder.decode(await p.output())
await p.output() // <----- Hangs here
await command("cd /home/dev/www")
await command("ls -la")
console.log(await getOutput())
await p.status()
console.log("done")
It hangs because .output will resolve once the entire process output has been read, meaning that until ssh command has finished, it will not resolve.
Also have in mind that you need to add \n at the end of each command, otherwise it'll never be triggered.
await command("cd /home\n");
await command("ls -la\n");
// if you don't finish the ssh session, .output will never resolve
await command("exit\n");
// now it will work correctly
console.log(await getOutput());
In any case if you don't want to close the session to read the output of a given command, what you need to do is use p.stdout.readable or p.stdout.read(buf) instead.
for await const(const chunk of p.stdout.readable) {
// parse chunk and do something
}

Groovy URL getText() returns a PasswordAuthentication instance

I am trying to download the content of a password-protected Gerrit URL in a Jenkins pipeline Groovy script. HTTPBuilder is not accessible so I am using the URL class with Authenticator:
// To avoid pipline bailing out since data PasswordAuthentication is non-serializable
#NonCPS
def getToString(data) {
data.toString()
}
def fetchCommit(host, project, version) {
withCredentials([usernamePassword(credentialsId: 'my-credentials',
usernameVariable: 'user',
passwordVariable: 'PASSWORD')]) {
proj = java.net.URLEncoder.encode(project, 'UTF-8')
echo "Setting default authentication"
Authenticator.default = {
new PasswordAuthentication(env.user, env.PASSWORD as char[])
} as Authenticator
echo "https://${host}/a/projects/${proj}/commits/${version}"
url = "https://${host}/a/projects/${proj}/commits/${version}".toURL()
result = getToString(url.getText())
echo "${result}"
}
}
The result is a PasswordAuthentication instance, and not the expected data:
[Pipeline] echo
java.net.PasswordAuthentication#3938b0f1
I have been wrestling with this for a while. I have tried different ways to setup the authentication and reading the data, but those mostly end up with an exception. Using eachLine() on the url does not enter the closure at all. The job also exits far to quickly, giving the impression it not even tries to make a connection.
Refs:
https://kousenit.org/2012/06/07/password-authentication-using-groovy/

Soap client with Node and strong-soap returning error with cerficate

I am using strong-soap (but with node-soap is the same result) node module to connect with soap services.
In the first step I am creating the client and trying to connect one method in this case "doLogin" method.
My code is:
soap.createClient(url, clientOptions, (err, client) => {
var loginApi = { UserName: "xxxx", Password: "xxxxxx" };
var loginUser = {
userName: "comercial#xxxxx.com"
};
client.addSoapHeader(header);
//client.setSecurity(new soap.BasicAuthSecurity(loginApi));
// we now have a soapClient - we also need to make sure there's no `err` here.
client.doLogin(loginUser, (err, result) => {
//'result' is the response body
console.error(err);
console.log("Result: \n" + JSON.stringify(result));
});
But the variable err is returning this error in the console:
{ Error: unable to verify the first certificate
at TLSSocket.<anonymous> (_tls_wrap.js:1105:38)
at emitNone (events.js:106:13)
at TLSSocket.emit (events.js:208:7)
at TLSSocket._finishInit (_tls_wrap.js:639:8)
at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:469:38) code:
'UNABLE_TO_VERIFY_LEAF_SIGNATURE' }
and result is undefined.
why is happening this error?
result is undefined by the error?
I have faced same error, unable to verify the first certificate.
This is because of SSL cerficate isnt verified.
Your nodejs script calls your server, it is going to carry out the full TLS check process (as you would hope). This will check the certificates for validity etc.
To work around this issue, you can run the following Steps:
npm config set strict-ssl false
As a best practice, it is wise to set it back to true afterwords so you do not accidentally install an untrusted module that you actually do not trust.
After this,
npm cache clean --force
Add the following environment variable:
NODE_TLS_REJECT_UNAUTHORIZED=0
For Linux:
export NODE_TLS_REJECT_UNAUTHORIZED=0
For Nginx
NODE_TLS_REJECT_UNAUTHORIZED=0
For Window:
this will set for only current command prompt screen,
set NODE_TLS_REJECT_UNAUTHORIZED=0
This has solved issue for me. Please try
Note: Make sure you do not leave this option on in production. Please don't disable TLS checks at all.
FIXED:
I have added correct certificates and rejectUnauthorized: false to create client and added "envelope" directive to the headers and now it is working.
I donĀ“t like the instruction rejectUnauthorized: false by security topics and I would like to know how to remove this in production environment.
Thank you!!

deepstream error listen EADDRINUSE 127.0.0.1:6020

i try to run my first deepstream.io server from this link but i get this error :
error:
CONNECTION_ERROR | Error: listen EADDRINUSE 127.0.0.1:3003
PLUGIN_ERROR | connectionEndpoint wasn't initialised in time
f:\try\deep\node_modules\deepstream.io\src\utils\dependency-
initialiser.js:96
throw error
^
Error: connectionEndpoint wasn't initialised in time
at DependencyInitialiser._onTimeout
(f:\try\deep\node_modules\deepstream.io\src\utils\dependency-
initialiser.js:94:17)
at ontimeout (timers.js:386:14)
at tryOnTimeout (timers.js:250:5)
at Timer.listOnTimeout (timers.js:214:5)
and this is my code:
const DeepStreamServer = require("deepstream.io")
const C = DeepStreamServer.constants;
const server = new DeepStreamServer({
host:'localhost',
port:3003
})
server.start();
In deepstream 3.0 we released our HTTP endpoint, by default this runs alongside our websocket endpoint.
Because of this, passing the port option at the root level of the config no longer works (it overrides both the HTTP and websocket port options, as you can see in the screen capture provided, both endpoints are trying to start on the same port).
You can override each of these ports as follows:
const deepstream = require('deepstream.io')
const server = new deepstream({
connectionEndpoints: {
http: {
options: {
port: ...
}
},
websocket: {
options: {
port: ...
}
}
}
})
server.start()
Or you can define your config in a file and point to that while initialising deepstream[1].
[1] deepstream server configuration
One solution that i find is passing empty config object so inseted of :
const server = new DeepStreamServer({
host:'localhost',
port:3003
})
i'm just using this :
const server = new DeepStreamServer({})
and now everything work's well.
All the bellow is for Version 4.2.2 (last version by now)
I was having the same Port in use or config file not found errors. And i was using typescript and i didn't pay attention too to the output dir and build (which can be a problem when one use typescript and build). I was able to run the server in the end. And i had a lot of analysis.
I checked up the code source and i have seen how the config is loaded
const SUPPORTED_EXTENSIONS = ['.yml', '.yaml', '.json', '.js']
const DEFAULT_CONFIG_DIRS = [
path.join('.', 'conf', 'config'), path.join('..', 'conf', 'config'),
'/etc/deepstream/config', '/usr/local/etc/deepstream/config',
'/usr/local/etc/deepstream/conf/config',
]
DEFAULT_CONFIG_DIRS.push(path.join(process.argv[1], '..', 'conf', 'config'))
DEFAULT_CONFIG_DIRS.push(path.join(process.argv[1], '..', '..', 'conf', 'config'))
Also i tested different things and all. Here what i came up with:
First of all if we don't precise any parameter in the constructor. A config from the default directories will get to load. If there isn't then the server fail to run.
And one of the places where we can put a config is ./conf in the same folder as the server node script.
Secondly we can precise a config as a string path (parameter in the constructor). config.yml or one of the supported extensions. That will allow the server to load the server config + the permission.yml and users.yml configs. Which too are supposed to be in the same folder. If not in the same folder there load will fail, and therefor the permission plugin will not load. And so does the users config. And no fall back to default will happen.
Thirdly the supported extensions for the config files are: yml, yaml, json, js.
In nodejs context. If nothing precised. There is no fallback to some default config. The config need to be provided in one of the default folders, or by precising a path to it. Or by passing a config object. And all the optional options will default to some values if not provided ( a bit bellow there is an example that can show that ). Know however that precising an end point is very important and required.
To precise the path, we need to precise the path to the config.yml file (the server config) [example: path.join(__dirname, './conf/config.yml')]. Then from the same dir permission.yml and users.yml will be retrieved (the extension can be any of the supported one). We can not precise a path to a directory, it will fail.
We can precise the path to permission config or user config separatly within config.yaml as shown bellow:
# Permissioning example with default values for config-based permissioning
permission:
type: config
options:
path: ./permissions.yml
maxRuleIterations: 3
cacheEvacuationInterval: 60000
Finally we can pass an object to configure the server, or by passing null as a parameter and use .set methods (i didn't test the second method). For configuring the server we need to follow the same structure as the yml file. With sometimes a bit different naming. The typescript declaration files or types show us the way. With an editor like vscode. Even if we are not using typescript we can keep get the auto completion and type definitions.
And the simplest for equivalent to the previous version is :
const webSocketServer = new Deepstream({
connectionEndpoints: [
{
type: 'ws-websocket',
options: {
port: 6020,
host: '127.0.0.1',
urlPath: '/deepstream'
}
}
]
});
webSocketServer.start();
the above is the new syntax and way.
const server = new DeepStreamServer({
host:'localhost',
port:3003
})
^^^^^^^ is completely deprecated and not supported in version 4 (the doc is not updated).

Socket Hang Up when using https.request in node.js

When using https.request with node.js v04.7, I get the following error:
Error: socket hang up
at CleartextStream.<anonymous> (http.js:1272:45)
at CleartextStream.emit (events.js:61:17)
at Array.<anonymous> (tls.js:617:22)
at EventEmitter._tickCallback (node.js:126:26)
Simplified code that will generate the error:
var https = require('https')
, fs = require('fs')
var options = {
host: 'localhost'
, port: 8000
, key: fs.readFileSync('../../test-key.pem')
, cert: fs.readFileSync('../../test-cert.pem')
}
// Set up server and start listening
https.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'})
res.end('success')
}).listen(options.port, options.host)
// Wait a second to let the server start up
setTimeout(function() {
var clientRequest = https.request(options, function(res) {
res.on('data', function (chunk) {
console.log('Called')
})
})
clientRequest.write('')
clientRequest.end()
}, 1000)
I get the error even with the server and client running on different node instances and have tested with port 8000, 3000, and 443 and with and without the SSL certificates. I do have libssl and libssl-dev on my Ubuntu machine.
Any ideas on what could be the cause?
In
https.createServer(function (req, res) {
you are missing options when you create the server, should be:
https.createServer(options, function (req, res) {
with your key and cert inside
I had a very similar problem where the response's end event never fired.
Adding this line fixed the problem:
// Hack to emit end on close because of a core bug that never fires end
response.on('close', function () {response.emit('end')});
I found an example of this in the request library mentioned in the previous answer.
Short answer: Use the the latest source code instead of the one you have. Store it where you will and then require it, you are good to go.
In the request 1.2.0 source code, main.js line 76, I see
http.createClient(options.uri.port, options.uri.hostname, options.uri.protocol === 'https:');
Looking at the http.js source code, I see
exports.createClient = function(port, host) {
var c = new Client();
c.port = port;
c.host = host;
return c;
};
It is requesting with 3 params but the actual function only has 2. The functionality is replaced with a separate module for https.
Looking at the latest main.js source code, I see dramatic changes. The most important is the addition of require('https').
It appears that request has been fixed but never re-released. Fortunately, the fix seems to work if you just copy manually from the raw view of the latest main.js source code and use it instead.
I had a similar problem and i think i got a fix. but then I have another socket problem.
See my solution here: http://groups.google.com/group/nodejs/browse_thread/thread/9189df2597aa199e/b83b16c08a051706?lnk=gst&q=hang+up#b83b16c08a051706
key point: use 0.4.8, http.request instead of http.createClient.
However, the new problem is, if I let the program running for long time, (I actually left the program running but no activity during weekend), then I will get socket hang up error when I send a request to http Server. (not even reach the http.request). I don't know if it is because of my code, or it is different problem with http Server