youtube-dl not able to authenticate from amazon ec2 machine - authentication

I installed youtube-dl on my local machine using curl as mentioned in the official README here.
sudo curl -L https://yt-dl.org/downloads/latest/youtube-dl -o /usr/local/bin/youtube-dl
sudo chmod a+rx /usr/local/bin/youtube-dl
Now when I run below command on my local machine
youtube-dl --cookies cookie.txt https://www.youtube.com/watch?v=x-5V_RS3Q48 -u my_account#gmail.com -p my_pass_word
I am able to download the video without any hassle as shown below.
But when I try to download the same video on one of my ec2 instances, I get the following exception.
The installation procedure on both machine is exactly same, youtube-dl version is exactly same (2017.08.18), python version is same (2.7.6)
The only difference I could figure out is the kernel versions on both machines:
On My Local: Linux-3.19.0-25-generic-x86_64-with-Ubuntu-14.04-trusty
On EC2 Instance: Linux-3.13.0-74-generic-x86_64-with-Ubuntu-14.04-trusty
Also the video I am trying to download is private and uploaded by same user I am providing credentials of.
One important point to note is that EC2 machine is able to download the video without any trouble if I am not using the username & password (which is only possible for videos that are not private)
Thanks

Posting the answer in case someone else is stuck with the similar issue.
The Issue with the cookies not being generated on server edition Linux OS on ec2 instance provided by AWS.
According to what I learnt recently, we don't have support for firefox browser on these machines (at least by default) and that's why it was failing to create the cookie file.
Solution
I created the file locally and set expiry time to 20 years in future and moved that cookie on server ec2 instance and used that cookie to sign in rather than creating one.
Thanks

Related

I want to turn my localhost server into a real website

I created an application that runs on a localhost server using expressjs. And I also bought a domain.
I'm wondering if there is a way to take that localhost server and turn it into a real shared server
I tried once to use a hosting service like hostgator but I still don't know how I can turn the express app into a real website.
I have no experience with any web development services so please don't tell me to use ....... whatever because I will have no idea what that is.
For one thing it is not clear how your website actually works: if it is only express does it generate HTML or is it purely JSON passed to browser clients via get requests (to each their own).
There are so many options as to how you might do this: one of the best options is to first make sure your server runs on Docker. Find a tutorial on YouTube/google/Stack Overflow/Blogs on how to run your Express server with docker. If you do that you can deploy it to a Container manager like Google/Amazon/Digital Ocean. If this seems hard to you there are other options.
Presumably you run your server with something like npm start. This guide can show you how to do essentially that but on a cloud computer.
Before you begin make sure that you're locally working server is checked in to a cloud Git provider like Github, GitLab, Bitbucket, etc.
Since Amazon AWS, and Google Cloud have free tier or options for hosting for free for a certain amount of time (AWS 1 year) or for a certain amount of money (Google Cloud). These two seem like viable place to start.
If you find the option that you'd like you'll need to:
create an account
Create a server (choose a cheap one especially initially like mice/small/cheap etc).
Find a tutorial on how to "SSH" into that server (which basically means remotely control the terminal on that server). Google actually makes this fairly easy there's a big button that says SSH into this server.
Once you've logged into that Computer you'll be able to run the same commands you probably normally do on your home computer:
The computer you'll be getting is likely to be a virtual Linux Computer probably something like Linux Ubuntu. Find a tutorial on how to get git and node installed there (but it's something like sudo apt-get update && sudo apt-get install git node).
Once you have git and node try mkdir www and cd into that: mkdir www && cd project (This isn't critical but conventional.)
Copy the link that allows you to "Clone your repo using HTTPS" (there's a link at the top right of your GitHub (or others) repo that allows you to do that. You'll need to enter your password
Now all the files that you had on your computer are on this new computer.
Next you'll have to probably npm i to install your dependent NPM packages. (This assumes you properly used .gitignore to prevent GitHub from being filled with extra copies of your npm packages.)
Now you should be able to run your code as usual: npm run start
If all those steps work you'll want to get something that will run these "forever" like https://www.npmjs.com/package/forever npm i -g forever or even better: https://www.npmjs.com/package/pm2 will allow you to continuously run your express server.
Finally, you'll need to configure this server on AWS/Google/whatever service you're using to push traffic coming in on port 80 and 443 to port 3000 and open traffic to all. And depending on the service you chose that's different so find a tutorial for doing just that part.
This will only allow people across the internet to see your service on an AWS URL or a google URL. But it's a good chance to make sure everything works perfectly. Once you're happy with everything associate your purchased domain with that special AWS/Google domain. You can do that on the AWS side, or the GoDaddy/NameCheap/where-ever you bought your domain side.
For the docker option you can download aws-cli tools and upload your built docker container to AWS and have it available. Find a tutorial to do that.
Essentially your question is very broad so I sometimes brushed over some details, but this is essentially what you have to do.

Waiting for SSH to be available with docker-machine on windows 10

I installed docker machine, and then created a new docker-machine on Windows 10.
Now I run ls to see the list of docker machines.
Now I run the following command
docker-machine start hypervdockermachine
Now I am stuck at this
Waiting for SSH to be available...
Too many retries waiting for SSH to be available. Last error: Maximum number of retries (60) exceeded
I have seen the git hub issue here, but not clear what to do.
Is there a way to solve this problem? I am not good at ssh
UPDATE
I just found a workaround.
You can run the above commands with git bash.
Most important, you must run git bash as admin. Else you will end up scratching your head.
Even the basic
docker-machine ls
will not show up anything without being an admin.
Finally if you are seeing the following error
Unable to query docker version: Get https://192.168.0.105:2376/v1.15/version: x509: certificate signed by unknown authority
Then you have to look at this issue.
docker-machine regenerate-certs yourdockermachinename
If needed user --force option
I got into the same problem after I moved .docker to partition D: and created a symlink to C:\Users\username\.docker, following this SO answer. I removed the old machines and configured new ones, and tried to regenerate the certs as suggested in the OP workaround but the problem was not solved.
After googling, I found this OpenSSH wiki page
and suspected that the cause of the problem was related to permissions.
So I could solve the problem by trying two different things:
Delete .ssh (source)
fix permissions to D:\path\to\.docker, allowing only SYSTEM, Administrators and my user to have full control access (source). These permissions were the same defined for .docker when it was under C:\Users\username\, but moving the folder to another partition made it inherit different permissions. To avoid dealing to much with it, I keep inheritance enabled changed the permissions directly in D: rather than in .docker folder.

Docker for Win acme.json permissions

Traefik v1.3.1
Docker CE for Windows: 17.06.0-ce-win18 (12627)
I have the /acme folder routed to a host volume which contains the file acme.json. With the Traefik 1.3.1 update, I noticed that Traefik gets stuck in an infinite loop complaining that the "permissions 755 for /etc/traefik/acme/acme.json are too open, please use 600". The only solution I've found is to remove acme.json and let Traefik re-negotiate the certs. Unfortunately, if I need to restart the container, I have to remove acme.json again or I'm stuck with the same issue again!
My guess is that the issue lies with the Windows volume mapped to Docker but I was wondering what the recommended workaround would even be for this?
Can I change permissions on shared volumes for container-specific deployment requirements?
No, at this point, Docker for Windows does not enable you to control (chmod) the Unix-style permissions on shared volumes for deployed containers, but rather sets permissions to a default value of 0755 (read, write, execute permissions for user, read and execute for group) which is not configurable.
Traefik is not compatible with regular Windows due to the POSIX permissions check. It may work in the Windows Subsystem for Linux since that has a Unix-style permission system.
Stumbled across this issue when trying to get traefik running on Docker for Windows... ended up getting it working by adding a few lines to a dockerfile to create the acme.json and set permissions. I then built the image and despite throwing the "Docker image from Windows against a non-Windows Docker host security warning" when I checked permissions on the acme.json file it worked!
[
I setup a repo and have it auto building to the dockerhub here for further testing.
https://hub.docker.com/r/guerillamos/traefik/
https://github.com/guerillamos/traefikwin/blob/master/Dockerfile
Once I got that built I switched the image out in my docker-compose file and my DNS challenge to Cloudflare worked like a charm according to the logs.
I hope this helps someone!

Cloud9IDE ssh to own workspace - WITHOUT nodejs being installed

Quick summary:
Can I use cloud9 as an online shell terminal to connect to my own workspace (ec2 instance) WITHOUT having nodejs installed on that instance?
More details
I love the cloud9 online ide and am keen to use it for everything as I just have a chromebook. I just read about the new Ubuntu Snappy version of Ubuntu and wanted to launch an instance of it on amazon's ec2, ssh in, and play with it.
I can ssh in from my chromebook ok, but I'd like to know if there's a way to do this from cloud9? i.e. to use it is an online shell terminal, without first installing nodejs on the ec2 instance (which cloud9 as I understand it needs for the fancier ide features I could make do without for this use case.)
Thanks for the help in advance - first post on stackoverflow :)
Note: I'm a newish linux user. I've successfuly got cloud9's ide to work with a fresh ec2 regular ubuntu instace by connecting via ssh using my chromebook's crosh terminal and installing nodejs first, then switching to connect from cloud9 using the 'own ssh workspace' option. However I'm keen to see if I could have done this totally using cloud9 - ie used cloud9 like an online terminal to connect to the fresh ec2, then installed nodejs to turn on cloud9's fancy ide features. (or perhaps not install nodejs, and just use it as an online terminal e.g. to play with an image of ubuntu snappy quickly)
Unfortunately Cloud9 needs NodeJS on your server to work correctly. When you connect it to your workspace it should pop up with a prompt which after clicking next will automatically install all the dependencies Cloud9 needs to work.
(this is in response to your comment of 12/24)
You don't need Node installed on your Amazon server to make an ordinary ssh connnection. Perhaps you're copying the wrong key over: it's the one ending in .pub in ~/.ssh (e.g. id_rsa.pub).
Amazon has a help page for this process - basically you're adding the content of the public key on C9 to the file ~/.ssh/authorized_keys on your server:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/managing-users.html
Then, you'd ssh from C9 to your server like so:
ssh -i .ssh/<my public key> <myusername on amazon>#<amazon server IP>

Ldap server for developer

I am developing a project and that requires ldap validation. But, I don't have a developer/qa ldap server.
Does a small ldap server exist for windows for testing/development?.
I just want to test to validate a active account and detect if it is blocked or not, so i don't want to install a whole domain to do that.
---never mind---
I tried an compiled openldap but I was unable to understand it. Simply, I don't get how to connect to it, how to create a account and how to validate, the client ldap returned me some obfuscate error message, i tried several ways to do it and finally i give up.
Finally, i installed a domain, it was absurdly easy to install (2008 r2), restart the server and that's it.
Anyways, thanks for the advice of opendlap and aldps
If you're on Windows and use Active Directory, have a look at Active Directory Lightweight Directory Services (AD LDS) - a LDAP server you can install and use on your dev machine.
The open source LDAP server from OpenLDAP should give you what you need:
http://www.openldap.org/
Apache provide a directory server called "ApacheDS"(Apache Directory Server), and it provides a GUI management client called "Apache Directory Studio" which is based on Eclipse.
If you want to have a test only, this studio provides a built-in server for your test, easy to link.
You can also install the studio directly in Eclipse using this update site: http://directory.apache.org/studio/update/2.x/
Active Directory works fine as an LDAP server and its included in the Windows Server 2008 trial. See the answer to my question Testing LDAP Connections to Active Directory Server. I have it running in a KVM virtual machine on Linux and query it from an OpenLDAP based client.
Necromancing.
I've had the same problem.
OpenDS is very easy to get up and running, and doesn't require administrator rights.
You just need to download the ZIP file and run the installer.
The installer can populate the directory with test entries, too - if you want to see some example data.
That's exactly what you're looking for when wanting a simple dev test server.
Note:
OpenDS development has seized, and was forked into OpenDJ, a commercial project by forgerock.
While OpenDS still works on Java7, only OpenDJ will work with Java8.
However, OpenDJ is still FREE and OpenSource.
You can find the sourcecode here on Bitbucket
and you can grab it with git:
git clone https://stash.forgerock.org/scm/opendj/opendj.git
Forget OpenLDAP and AD-LDS; these are way too complicated for simple testing.
In addition, their user interface is horrible, and you need something that you can get up and running FAST, without admin rights, and have it populated with test data in a few minutes, not in a few weeks.
And ApacheDS will require administrator privileges, unfortunately (because it only works as windows service, and you can't start/stop these without being administrator).
So OpenDJ is the definite way to go.
Apache Directory Studio is a good client to browse, edit and import/export data via LDAP (LDIF).
However, despite Apache Directory Studio being written in Java, it adds a dependency to gtk, and only has binaries for x86/x64, which means it won't work on a Chromebook with ARM processor, or on a RaspberrryPI.
But with the test entries added automagically in OpenDJ/OpenDS (if you choose the option), you don't even need that.
When in doubt, use a web based interface that "talks LDAP".
Try OpenDS it is very simple and requires only Java.
You could roll your own LDAP server for testing pretty easily using godap: https://github.com/bradleypeabody/godap
It's written in Go. It's very small and simple.
You would basically need to copy the server example out of godap_test.go and wire it up however you need.
Try simple-ldap-server
I know its pretty late to answer this question. But for the reference of someone who runs into the same question.
I wrote a simple ldap server(using ldapjs on nodejs) for authentication testing purposes. Please feel free to use it. It's easy to configure. Can support both LDAP/LDAPS protocols, just require a json file including the user ids you want to add(or it comes with a pre-included users json file, which you can use if you want).
The project is on github. (I'll add a docker image too)
Feel free to visit and use
Docker image
Simple Ldap Server Git
OpenLDAP. Ships with most Unixes and Linuxes. For Windows it is available from several sources:
Cygwin
http://www.userbooster.de
as the Silver (free) edition of the CDS product http://www.symas.com/cds.shtml. This is crippled compared to the Userbooster version, which is complete.
You can use a Docker container with Samba as Domain controller, here I show how to setup one in just a few minutes
Basically you need to
Create an image with this (read the post if you want to know why)
$ git clone https://github.com/padiazg/alpine-samba-ad-container.git
$ cd alpine-samba-ad-container
# replace your-user with your username
$ docker build -t your-user/alpine-samba-ad-container .
Create some folders and files to persist the container data
mkdir /tmp/krb-conf
&& mkdir /tmp/krb-data
&& mkdir /tmp/smb-conf
&& modir /tmp/smb-data
&& touch /tmp/krb-conf/krb5.conf
Run the container
docker run -d \
-e SAMBA_ADMIN_PASSWORD=a-secure-password \
-e SAMBA_DOMAIN=local \
-e SAMBA_REALM=local.your-domain.io \
-e LDAP_ALLOW_INSECURE=true \
--mount type=bind,source=/tmp/krb-conf/krb5.conf,target=/etc/krb5.conf \
--mount type=bind,source=/tmp/krb-data,target=/var/lib/krb5kdc \
--mount type=bind,source=/tmp/smb-conf,target=/etc/samba \
--mount type=bind,source=/tmp/smb-data,target=/var/lib/samba \
-p 389:389 \
--name smb4ad \
your-user/alpine-samba-ad-container
And now you are good to go