I have spent a few hours hunting for the problem here.
I have been working with Macs for years now and never had this problem, and have ssh into eC2 instances thousands of times.
I recently received at work a new MacBook Pro.
SSH runs as a service, meaning here it does not return any error that it is not found.
But no matter what server or EC2 instance I try to ssh into, as I have done a million times before I get a timeout.
Before you ask, I have looked all over for this problem. I have also looked for the normal ~/.ssh directory, which seems to be missing and therefore cannot find any config file.
The following is the Mac info:
Catalina 10.15.2
Model Name: MacBook Pro
Model Identifier: MacBookPro16,1
Processor Name: 8-Core Intel Core i9
Processor Speed: 2.3 GHz
Number of Processors: 1
Total Number of Cores: 8
L2 Cache (per Core): 256 KB
L3 Cache: 16 MB
Hyper-Threading Technology: Enabled
Memory: 16 GB
Boot ROM Version: 1037.60.58.0.0 (iBridge: 17.16.12551.0.0,0)
Serial Number (system): C02ZNMV5MD6N
Hardware UUID: 27B1EDF5-B1D2-5F86-BD12-D646F36D9D2D
Activation Lock Status: Enabled
ETA: Yes, from a Windows machine I can access the EC2 network. Yes, I have the correct PEM file. And yes, I have made sure security groups in AWS are correct. For some reason the normal ssh -i etc. picked up directly from AWS connect for the EC2 instance always times out.
Crazy question: does the ssh in Catalina demand another command, addition or some other parameter besides -i?
(I do not seem to be able to ping, telnet etc. either. So something seems to be preventing the OS from going out on ssh port 22.)
Does anyone know of or has had this problem and a fix for it? I am fairly sure it is some type of configuration in ssh or in the Network configurations.
It is driving me crazy. Any and all help would be greatly appreciated!
New Macbook Pro owner/user here. Same issue, despite all configs being identical to my Windows 10 pc and Ubuntu 20 laptop.
For some reason, this doesn't work for me on my Macbook, but does work on Windows and Ubuntu.
ssh -i path-to-keyfile.pem user#ipaddress
But creating an SSH config file and adding my AWS keyfile to my keychain works:
open ~/.ssh/config if the config file exists, or touch ~/.ssh/config if not.
Edit this config file as follows:
Host *
AddKeysToAgent yes
UseKeychain yes
IdentityFile ~/.ssh/id_rsa
Note: I don't know for sure, but I imagine only the 'AddKeysToAgent' and 'UseKeychain' parts are what's important. I'm using the 'IdentityFile' part for connecting to my git repos.
Save the config file and exit. Next, make sure your keyfile isn't too open, otherwise you won't be able to add it to your keychain:
chmod 600 path-to-keyfile.pem
Finally, add the keyfile to your keychain:
ssh-add -K path-to-keyfile.pem
Now on Mac, I'm able to ssh into my AWS instance without the -i flag:
ssh aws-username#aws-ipaddress
Hope this helps. I found the solution here: https://www.cloudsavvyit.com/1795/how-to-add-your-ec2-pem-file-to-your-ssh-keychain/
PS - I'm also unable to sFTP into AWS using Filezilla on Mac, so I'm looking into this as well.
Update on Filezilla: A bit bizarre and I haven't figured out how to save my settings, but for now this answer works: https://superuser.com/questions/280808/filezilla-on-mac-sftp-with-passwordless-authentication
I've an f1-micro instance which I've been testing docker on created as such:
$ gcloud compute instances create dockerbox \
--image container-vm-v20140731 \
--image-project google-containers \
--zone europe-west1-b \
--machine-type f1-micro
This all works fine.
I'm now in the process of upgrading to a larger google compute engine VM. I've taken a snapshot of the fi-micro dockerbox, then used this as the Boot Source for the larger n1-standard-8 VM... this seems to create without problems until I try to ssh onto it.
via the command line:
$ gcloud compute --project "secure-electron-631" ssh --zone "europe-west1-b" "me#biggerbox"
ssh: connect to host xx.xx.xx.xx port 22: Connection timed out
ERROR: (gcloud.compute.ssh) Your SSH key has not propagated to your instance yet. Try running this command again.
via the browser, ssh connection I get:
Connection Failed
We are unable to connect to the VM on port 22. Please check that the VM is healthy and the SSH server is running.
I've tried multiple times but same result
I've confirmed it biggerbox is RUNNING. not sure about sshd
OK, problem seemed to stem from not detaching the micro instance from a mounted persistant disk when I took the snapshot. Detached and unmounted the PD volume and snapshotted the micro-instance again and based a new n1-standard-8 on it. Works ok now.
FYI, also handy for those troubleshooting GCE instance ssh:
https://github.com/GoogleCloudPlatform/compute-ssh-diagnostic-sh
Currently, I have two servers running on an EC2 instance (MongoDB and bottlepy). Everything works when I SSHed to the instance and started those two servers. However, when I closed the SSH session (the instance is still running), I lost those two servers. Is there a way to keep the server running after logging out? I am using Bitvise Tunnelier on Windows 7.
The instance I am using is Ubuntu Server 12.04.3 LTS.
For those landing here from a google search, I would like to add tmux as another option. tmux is widely used for this purpose, and is preinstalled on new Ubuntu EC2 instances.
Managing a single session
Here is a great answer by Hamish Downer given to a similar question over at askubuntu.com:
I would use a terminal multiplexer - screen being the best known, and tmux being a more recent implementation of the idea. I use tmux, and would recommend you do to.
Basically tmux will run a terminal (or set of terminals) on a computer. If you run it on a remote server, you can disconnect from it without the terminal dying. Then when you login in again later you can reconnect, and see all the output you missed.
To start it the first time, just type
tmux
Then, when you want to disconnect, you do Ctrl+B, D (ie press Ctrl+B, then release both keys, and then press d)
When you login again, you can run
tmux attach
and you will reconnect to tmux and see all the output that happened. Note that if you accidentally lose the ssh connection (say your network goes down), tmux will still be running, though it may think it is still attached to a connection. You can tell tmux to detach from the last connection and attach to your new connection by running
tmux attach -d
In fact, you can use the -d option all the time. On servers, I have this in my >.bashrc
alias tt='tmux attach -d'
So when I login I can just type tt and reattach. You can go one step further >if you want and integrate the command into an alias for ssh. I run a mail client >inside tmux on a server, and I have a local alias:
alias maileo='ssh -t mail.example.org tmux attach -d'
This does ssh to the server and runs the command at the end - tmux attach -d The -t option ensures that a terminal is started - if a command is supplied then it is not run in a terminal by default. So now I can run maileo on a local command line and connect to the server, and the tmux session. When I disconnect from tmux, the ssh connection is also killed.
This shows how to use tmux for your specific use case, but tmux can do much more than this. This tmux tutorial will teach you a bit more, and there is plenty more out there.
Managing multiple sessions
This can be useful if you need to run several processes in the background simultaneously. To do this effectively, each session will be given a name.
Start (and connect to) a new named session:
tmux new-session -s session_name
Detach from any session as described above: Ctrl+B, D.
List all active sessions:
tmux list-sessions
Connect to a named session:
tmux attach-session -t session_name
To kill/stop a session, you have two options. One option is to enter the exit command while connected to the session you want to kill. Another option is by using the command:
tmux kill-session -t session_name
If you don't want to run some process as a service (or via an apache module) you can (like I do for using IRC) use gnome-screen Install screen http://hostmar.co/software-small.
screen keeps running on your server even if you close the connection - and thus every process you started within will keep running too.
It would be nice if you provided more info about your environment but assuming it's Ubuntu Linux you can start the services in the background or as daemons.
sudo service mongodb start
nohup python yourbottlepyapp.py &
(Use nohup if you want are in a ssh session and want to prevent it from closing file descriptors)
You can also run your bottle.py app using Apache mod_wsgi. (Running under the apache service) More info here: http://bottlepy.org/docs/dev/deployment.html
Hope this helps.
Addition: (your process still runs after you exit the ssh session)
Take this example time.py
import time
time.sleep(3600)
Then run:
$ python3 time.py &
[1] 3027
$ ps -Af | grep -v grep | grep time.py
ubuntu 3027 2986 0 18:50 pts/3 00:00:00 python3 time.py
$ exit
Then ssh back to the server
$ ps -Af | grep -v grep | grep time.py
ubuntu 3027 1 0 18:50 ? 00:00:00 python3 time.py
Process still running (notice with no tty)
You will want the started services to disconnect from the controlling terminal. I would suggest you use nohup to do that, e.g.
ssh my.server "/bin/sh -c nohup /path/to/service"
you may need to put an & in there (in the quotes) to run it in the background.
As others have commented, if you run proper init scripts to start/stop services (or ubuntu's service command), you should not see this issue.
If on Linux based instances putting the job in the background followed by disown seems to do the job.
$ ./script &
$ disown
I am running an Ubuntu 12.04-based box inside of Vagrant using VirtualBox. So far, everything is fine - except for one thing:
Let's assume that the VM is running. Then, the host goes to standby-mode. After waking it up again, the VM is still running, but its internal clock continues where it stopped when the host went down. So this basically means: Put the host to sleep for 15 minutes, wake it up again, then the VM's internal clock is 15 minutes late.
How can I fix this (setting the time manually is not an option for obvious reasons ;-))? Is there a way to run a script inside of a Vagrant VM whenever the host system changes its state?
I've read in the documentation that by default the VirtualBox Guest Additions sync the time with the host every 10 seconds. Apparently this is not happening, but I can not find any place where it is disabled. So any ideas?
PS: The Guest Additions are installed and match the version of VirtualBox being used.
The documentation lacks some details here.
What VirtualBox does every 10 seconds is just slight adjustement (something like 0.005 seconds). Only when the time difference reaches a threshold (20 minutes by default) a "real" resync is done.
You can reduce the thresold (i.e. to 10 seconds) with the following command:
VBoxManage guestproperty set <vm-name> "/VirtualBox/GuestAdd/VBoxService/--timesync-set-threshold" 10000
Summarizing answers of #zilupe and #Slobodan Kovacevic, solution is to add following to Vagrantfile:
config.vm.provider 'virtualbox' do |vb|
vb.customize [ "guestproperty", "set", :id, "/VirtualBox/GuestAdd/VBoxService/--timesync-set-threshold", 1000 ]
end
This will synchronize clocks each time when desync becomes > 1s (1000ms)
I give an other solution to sync time between guest & host without installing Virtualbox guest addition:
install ntp on your guest, and de-comment these lines in /etc/ntp.conf:
disable auth
broadcastclient
Then, restart ntp with service ntp restart
Active broadcast on your host:
For Linux users, edit your /etc/ntp.conf file and configure broadcast (you must adapt IP):
broadcast 192.168.123.255
For Windows users, activate the "Windows Time" service. You can then read this page to configure it to broadcast time
Then, restart time service on host.
For me to get timesync working I had to do this:
vboxmanage setextradata «machine-name» "VBoxInternal/Devices/VMMDev/0/Config/GetHostTimeDisabled" 0
It turns the timesync on. It was, for some reason, off.
I found a solution:
install ntpdate
add "s" permission for ntpdate, this allows non-root users to run ntpdate as root: sudo chmod u+s /usr/sbin/ntpdate
add one line in ~/.bashrc: ntpdate -u ntp.ubuntu.com
After that, each time you login to the linux system, the time will be sync once.
you can install the VirtualBox Guest Additions in the VM to sync the time automatically by VB.
I have error The difference between the request time and the current time is too large when call method amazons3.ListObjects
ListObjectsRequest request = new ListObjectsRequest() {
BucketName = BucketName, Prefix = fullKey
};
using (ListObjectsResponse response = s3Client.ListObjects(request))
{
bool result = response.S3Objects.Count > 0;
return result;
}
What it could be?
The time on your local box is out of sync with the current time. Sync up your system clock and the problem will go away.
For those using Vagrant, a vagrant halt followed by vagrant up worked for me.
The clock is out of sync.
I followed the steps in this post to get it working again, but also had to run the following command.
sudo ntpdate ntp.ubuntu.com
sudo apt-get install ntp
If at any time you get a message saying the NTP socket is still in use, stop it with sudo /etc/init.d/ntp stop and re-run your command.
I had the same error and I'm using Docker for Mac. Simply restarting Docker worked for me.
On WSL 2 or any Deb-based Linux (Ubuntu, Mint ...):
Check date:
date
Now run:
sudo apt install ntpdate
sudo ntpdate time.nist.gov
Output example:
18 Feb 14:27:36 ntpdate[24008]: step time server 132.163.97.4 offset 1009.140848 sec
Check date again:
date
Alternatively look for correctClockSkew option in AWS CLI/SDK config, and set it to true
For those using Docker in Windows try restarting the Docker Engine in Setting->Reset->Restart Docker.
In case anyone finds this using Laravel and Homestead, simply running
homestead halt
followed by
homestead up
And you're good to go again.
2021 answer:
AWS.config.update({
accessKeyId: 'xxx',
secretAccessKey: 'xxxx',
correctClockSkew: true
});
As other's have said, your local clock is out of sync with AWS. You can keep it synced to Amazon's servers directly using NTP so you won't have to worry about clock drift now or in the future.
Note: The below instructions are for *nix users. I've added a comment with how you might do it in Windows, but as a non-Windows user I can't verify their accuracy.
To install NTP, simply choose one of the following, depending on your distribution:
apt-get install ntp
or
yum install ntp
etc.
Configure NTP to use Amazon servers, like so:
vim /etc/ntp.conf
And in it, comment out the default servers and add these:
server 0.amazon.pool.ntp.org iburst
server 1.amazon.pool.ntp.org iburst
server 2.amazon.pool.ntp.org iburst
server 3.amazon.pool.ntp.org iburst
Restart ntp service:
sudo service ntp restart
Source: https://allcloud.io/blog/how-to-fix-amazon-s3-requesttimetooskewed/
And a more general article on keeping your time synchronized with NTP:
https://www.digitalocean.com/community/tutorials/how-to-set-up-time-synchronization-on-ubuntu-12-04
This can also be caused by using async/await with the construction of the request object outside the task and the actual call to AWS inside the task. If there are lots of tasks running and the task isn't scheduled in time, or there is some other operation delaying the actual call to AWS, this exception may be thrown. This is more common than you might guess because the default task scheduler does not process tasks in FIFO order, resulting in starvation for some tasks, especially under heavy load.
This reset my system clock correctly on OSX. S3 uploads using the JS SDK works for me now in local dev
ntpdate us.pool.ntp.org
Read more about this here
if this problem in you localhost for windows 10
set time automatically ON and set time zone automatically ON
this solve my problem.
If you get this error in windows follow these steps to solve your problem.. Change your local time setting:
step 1: click on change date and time settings
step 2: from the popup Date and Time window click on Internet Time Tab
step 3: next Click on Change Settings
step 4: from the Server drop down select time.nist.gov or check this website
step 5: click on OK
Restart your console and check. It works...
For those facing same problem on Microsoft WLS2 Ubuntu, the only workarounds right now are:
sudo hwclock -s
Or
wsl --shutdown
Clock offset is occurring after waking up Windows from sleep. Keep an eye on https://github.com/microsoft/WSL/issues/5324 for fix from microsoft.
If you're working with a VM, restarting the VM just works on mine
If you are using a virtualbox, the time into virtual machine is sync with the time of the real machine. Just fix the time into the virtual machine will not fix the problem.
I had this error because my local machine's time and timezone were set incorrectly. Changing them to the correct time and timezone worked for me.
I had same problem in Windows 10 with Docker. You should run this commands step for step
docker run --rm --privileged alpine hwclock -s
again
docker run --rm --privileged alpine hwclock -s
and last command , don't forget to set your username and password and your timezone, to run minIO while Docker is
docker run -p 9000:9000 -e "MINIO_ACCESS_KEY=yourUserName" -e "MINIO_SECRET_KEY=YourPassword" -e "TZ=Europe/Berlin" -v /etc/localtime:/etc/localtime:ro minio/minio server /data
It is a little crude but this worked for me
Did a curl to s3 server
curl s3.amazonaws.com -v
Then got this
* Trying 52.216.141.158...
* TCP_NODELAY set
* Connected to s3.amazonaws.com (52.216.141.158) port 80 (#0)
> GET / HTTP/1.1
> Host: s3.amazonaws.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 307 Temporary Redirect
< x-amz-id-2: q2wUOf5ZC7iu2ymbRWUpZaM6GpPLLf/irrntuw/JNB7QYxDzQvcLHQbsbF2dp5zT8rBrGwqnOz0=
< x-amz-request-id: T4H1W4WKBE3F39RM
< Date: Sat, 09 Oct 2021 19:21:24 GMT
< Location: https://aws.amazon.com/s3/
< Server: AmazonS3
< Content-Length: 0
<
* Connection #0 to host s3.amazonaws.com left intact
* Closing connection 0
Got this date
Sat, 09 Oct 2021 19:21:24 GMT
Set the date in ubuntu
sudo date --set "Sat, 09 Oct 2021 19:21:24 GMT"
My code stopped throwing exceptions
Now I have a script that does this periodically every month
To get rid of this problem, you have to adjust the client's timing so that there is a maximum time stamp difference of up to 15 minutes. Also set the standard time and zone for your system.
Check the full details here.
I have the exact same error message but it's not the same cause as any of the others above.
In my case I have a React browser app doing something like this:
import { Storage } from '#aws-amplify/storage'
...
await Promise.all(files.map(file => Storage.put(...)))
I am uploading a lot of files over a slow network connection.
With this code, the promises are all started at once, so the request time for all the requests is the same, but because the browser (or amplify?) is throttling the number of concurrent connections, the later requests don't actually hit the server until more than 15 minutes after they were created.
The solution is to limit the concurrency of the promise creation - e.g. use something like bluebird Promise.map with the concurrency option
Using ntp may not work on all version of your Linux based server (e.g. an out of date Ubuntu server version that is no longer supported which will block you from downloading ntp if it is not already installed).
If this is your situation, you can set independent time zones for your Linux VM:
https://community.rackspace.com/products/f/25/t/650
After you do this you may need to reset the time/date. Instructions for doing this are in this article:
http://codeghar.wordpress.com/2007/12/06/manage-time-in-ubuntu-through-command-line
If u are in 2016 and in Istanbul here is a weird situation that Turkey decided not to switch to winter time standards anyway set your local timezone to Moscow then restart your machine.
I ran into this issue running Jet (Codeship) and Terraform on MacOS using Docker for Mac Beta channel 1.13.1-beta42.
Failed to read state: Error reloading remote state: RequestTimeTooSkewed: The difference between the request time and the current time is too large.
status code: 403, request id: 9D32BA2A5360FC18
This was resolved by restarting Docker.
I've just started getting this error, and syncing my clock doesn't help. (I've spent 2 hours syncing it to every timeserver I can find, including the AWS servers, but nothing makes a difference.)
Exactly the same thing started happening a year ago on Dec 31 2017. In that case, rebooting my system, and rebuilding my server (that uses the aws java sdk) fixed it. I don't know why. I assumed that AWS had some end-of-year timezone peculiarity. It's also possible that while I was doing these things, AWS timeservers fixed themselves. I have no way to test that hypothesis.
Now, the same thing has suddenly started to happen on Dec 30, 2018. It's not right at year-end, but close enough to seem suspicious. (Never got this error except on these dates.) Rebooting and rebuilding isn't helping this time.
My dev environment on this box is Windows 10 under Parallels. Nothing else on my system has changed - as I've double-checked by rolling back to prior Parallels snapshots. The clocks on both my host MacOS and the virtual Windows 10 are correct.
I'm suspecting an AWS bug.
Rebooting my windows server fixed it for me
The time was identical to ~1 second to the site time.in, so it wasn't off.
I was running into the same issue on my Mac. When I moved to a different timezone(PST to IST), somehow OSX was not picking timezone and time change automatically. So I had to set the two manually and that caused a lag of some 15-20 seconds on my laptop. After setting the automatic sync, the time got synched and the S3 copy command started working: For reference
You can use this tool for organizing your time with AWS and local system.
To synchronize time:
sudo yum -y install chrony
sudo systemctl enable chronyd
sudo systemctl start chronyd
This issue generally occurs when s3cmd client machine time is not synced with server.
Check time of both machine.
either sync time between them using date command
Client# sudo date --set="string"
Client# sudo date --set="15 MAY 2011 1:40 PM"
or
install chrony and restart its service on both machine
Client# sudo apt-get install chrony
Client# vi /etc/chrony/chrony.conf
pool ntp-server iburst
Client# sudo systemctl restart chronyd