Debugging Partial file aka net::ERR_INCOMPLETE_CHUNKED_ENCODING - api

I have a Dockerized service so theoretically they should be exactly the same across my two servers. The only difference is production is running on Digital Ocean with CoreOS stable (835.9.0) and the dev is running from my home server under Archlinux.
Problem I noticed that when my API returns a lot of results, on production the request seems to be cut short resulting in the infamous net::ERR_INCOMPLETE_CHUNKED_ENCODING in the browser. I can reproduce this issue like so:
curl -i 'http://greptweet.com/u/kaihendry/grep.php?q=http' >/dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 41274 0 41274 0 0 17846 0 --:--:-- 0:00:02 --:--:-- 17852
curl: (18) transfer closed with outstanding read data remaining
However is works fine on my home server:
curl -i 'http://gt.dabase.com/u/kaihendry/grep.php?q=http' >/dev/null
I am waiting to hear back from Digital Ocean. Is there anything else I might have missed? Content length? Compression?

The answer was actually in my error log if I cared to look closely:
[crit] 14#0: *3888 open() "/var/lib/nginx/tmp/fastcgi/2/03/0000000032" failed (13: Permission denied) while reading upstream, client:...
The fix was chmod -R 755 /var/lib/nginx.
This serverfault question is also related.

Related

"client_loop: send disconnect: Broken pipe" while running long experiments with bash script

I am connected through ssh to a linux virtual machine to run long experiments (3 hours per program) for academic research. When my computer is not used I get the error message: client_loop: send disconnect: Broken pipe. I have looked at this forum and tried many of the solutions such as:
in my ~/.ssh creating a file config (while creating using sudo chmod 644 ~/.ssh/config) and adding the following lines:
ServerAliveInterval 60
ServerAliveCountMax 100000
In /etc/ssh/ssh_config I have added the following:
Host*
ServerAliveInterval 60
ServerAliveCountMax 100000
And finally /etc/ssh/sshd_config I have the added the following:
TCPKeepAlive yes
ClientAliveInterval 60
ClientAliveCountMax 100000
I have all my macbook settings such that it won't go to sleep by using the following command sudo pmset -a disablesleep 1 and by changing all power saving methods.
However, while going away from the computer for ~1 hour of not using it actively (so screensaver is on the screen) I get this message.
I really don't know where to look at this point. The only things I can consider are MaxStartups 10:30:100 in /etc/ssh/sshd_config or ConnectTimeout 0 in /etc/ssh/ssh_config, but I wasn't entirely sure what the impact of changing these were.
Any suggestions to solve this problem would be appreciated!
Thanks!
edit/update: I notice that when I leave my computer on overnight but I am not running a bash script, that I do not get the broken pipe error.
edit/update 2: I find that I can leave my computer unattended for at least 30 minutes without a broken pipe error
I solve by adding following in my macbook /etc/ssh/ssh_config
Host *
ServerAliveInterval 60 #add this line

cURL: SSL connect error while trying to install kubectl on RHEL 7

I am following Kubernetes documentation to Install kubectl on Linux on my RHEL 7 server but I see an
curl: (35) SSL connect error
error while running the following command:
curl -kLO https://storage.googleapis.com/kubernetes-release/release/v1.7.0/bin/windows/amd64/kubectl
Any pointers to fix this issue will be very helpful for me to move forward.
I have just checked it and it appeared that the https://storage.googleapis.com/kubernetes-release/release/v1.7.0/bin/windows/amd64/kubectl doesn't exist . It looks like it is needed to add ".exe" at the end of the URL.
<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
<Details>
No such object: kubernetes-release/release/v1.7.0/bin/windows/amd64/kubectl
</Details>
</Error>
The official documentation on how to install kubectl on Linux asks to download the latest release _for_linux_ with the following command:
curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 44.5M 100 44.5M 0 0 15.8M 0 0:00:02 0:00:02 --:--:-- 15.8M
Additionally with the URL you have provided you are trying to download kubectl for windows on RHEL... (/bin/windows/amd64/kubectl in your url)
So, it is merely needed to add .exe in the end of kubectl if you need downloading it for Windows or replace windows with linux in URL :)

Redis stops working and becomes unresponsive

I have an application using Redis as a cache.
I have sporadic episodes where Redis suddenly becomes unresponsive. Even a simple curl test fails:
~ curl localhost:6379
curl: (7) Failed to connect to localhost port 6379: Connection refused
Redis console logs show no errors,
I am using this metrics exporter: https://github.com/oliver006/redis_exporter
During these episodes, redis_up is 0 and redis_exporter_last_scrape_error is 1 with no error detail.
FYI, I have 46 million short keys in Redis with empty string values.
How do I go about diagnosing and hopefully resolving this?

Cant SSH over IPSEC

I am in AWS and i have two VPCS between virgina and oregon and I am trying to SSH from either region.
My rules allow everything needed and I can ping
Virgina
IPSEC-Machine 10.10.1.47
Host-Machine-V 10.10.4.125
Oregon
IPSEC-Machine 10.20.0.97
Host-Machine-O 10.20.1.190
I can ping between regions
[ec2-user#ip-10-20-0-97 ~]$ ping 10.10.1.47
PING 10.10.1.47 (10.10.1.47) 56(84) bytes of data.
64 bytes from 10.10.1.47: icmp_seq=1 ttl=64 time=60.5 ms
--- 10.10.1.47 ping statistics ---
2 packets transmitted, 1 received, 50% packet loss, time 1002ms
rtt min/avg/max/mdev = 60.560/60.560/60.560/0.000 ms
SSH seems to work
[ec2-user#ip-10-10-1-47 ~]$ nc -v -w 1 10.20.1.190 -z 22
Connection to 10.20.1.190 22 port [tcp/ssh] succeeded!
[ec2-user#ip-10-10-1-47 ~]$
But when on 10.10.1.47 and I type ssh 10.20.1.190 it just hangs and I get nothing. The keys are all correct but even if I get a permission denied at this point I would be happy.
I'm not really sure what could be causing this, but here are a few things to try:
Use the EC2 hostname to connect - it's possible that something in the addressing is causing problems between the regions. My only guess at the moment is that the IP address is actually someone else's server in the Virginia region, and not your server in Oregon.
Run the nmap portscanner to ensure that the port is open. I saw that you used netcat, but a proper portscan may help.
Run ssh -vvv to get verbose output, this may give you some information to figure out what the problem is.

S3 Error: The difference between the request time and the current time is too large

I have error The difference between the request time and the current time is too large when call method amazons3.ListObjects
ListObjectsRequest request = new ListObjectsRequest() {
BucketName = BucketName, Prefix = fullKey
};
using (ListObjectsResponse response = s3Client.ListObjects(request))
{
bool result = response.S3Objects.Count > 0;
return result;
}
What it could be?
The time on your local box is out of sync with the current time. Sync up your system clock and the problem will go away.
For those using Vagrant, a vagrant halt followed by vagrant up worked for me.
The clock is out of sync.
I followed the steps in this post to get it working again, but also had to run the following command.
sudo ntpdate ntp.ubuntu.com
sudo apt-get install ntp
If at any time you get a message saying the NTP socket is still in use, stop it with sudo /etc/init.d/ntp stop and re-run your command.
I had the same error and I'm using Docker for Mac. Simply restarting Docker worked for me.
On WSL 2 or any Deb-based Linux (Ubuntu, Mint ...):
Check date:
date
Now run:
sudo apt install ntpdate
sudo ntpdate time.nist.gov
Output example:
18 Feb 14:27:36 ntpdate[24008]: step time server 132.163.97.4 offset 1009.140848 sec
Check date again:
date
Alternatively look for correctClockSkew option in AWS CLI/SDK config, and set it to true
For those using Docker in Windows try restarting the Docker Engine in Setting->Reset->Restart Docker.
In case anyone finds this using Laravel and Homestead, simply running
homestead halt
followed by
homestead up
And you're good to go again.
2021 answer:
AWS.config.update({
accessKeyId: 'xxx',
secretAccessKey: 'xxxx',
correctClockSkew: true
});
As other's have said, your local clock is out of sync with AWS. You can keep it synced to Amazon's servers directly using NTP so you won't have to worry about clock drift now or in the future.
Note: The below instructions are for *nix users. I've added a comment with how you might do it in Windows, but as a non-Windows user I can't verify their accuracy.
To install NTP, simply choose one of the following, depending on your distribution:
apt-get install ntp
or
yum install ntp
etc.
Configure NTP to use Amazon servers, like so:
vim /etc/ntp.conf
And in it, comment out the default servers and add these:
server 0.amazon.pool.ntp.org iburst
server 1.amazon.pool.ntp.org iburst
server 2.amazon.pool.ntp.org iburst
server 3.amazon.pool.ntp.org iburst
Restart ntp service:
sudo service ntp restart
Source: https://allcloud.io/blog/how-to-fix-amazon-s3-requesttimetooskewed/
And a more general article on keeping your time synchronized with NTP:
https://www.digitalocean.com/community/tutorials/how-to-set-up-time-synchronization-on-ubuntu-12-04
This can also be caused by using async/await with the construction of the request object outside the task and the actual call to AWS inside the task. If there are lots of tasks running and the task isn't scheduled in time, or there is some other operation delaying the actual call to AWS, this exception may be thrown. This is more common than you might guess because the default task scheduler does not process tasks in FIFO order, resulting in starvation for some tasks, especially under heavy load.
This reset my system clock correctly on OSX. S3 uploads using the JS SDK works for me now in local dev
ntpdate us.pool.ntp.org
Read more about this here
if this problem in you localhost for windows 10
set time automatically ON and set time zone automatically ON
this solve my problem.
If you get this error in windows follow these steps to solve your problem.. Change your local time setting:
step 1: click on change date and time settings
step 2: from the popup Date and Time window click on Internet Time Tab
step 3: next Click on Change Settings
step 4: from the Server drop down select time.nist.gov or check this website
step 5: click on OK
Restart your console and check. It works...
For those facing same problem on Microsoft WLS2 Ubuntu, the only workarounds right now are:
sudo hwclock -s
Or
wsl --shutdown
Clock offset is occurring after waking up Windows from sleep. Keep an eye on https://github.com/microsoft/WSL/issues/5324 for fix from microsoft.
If you're working with a VM, restarting the VM just works on mine
If you are using a virtualbox, the time into virtual machine is sync with the time of the real machine. Just fix the time into the virtual machine will not fix the problem.
I had this error because my local machine's time and timezone were set incorrectly. Changing them to the correct time and timezone worked for me.
I had same problem in Windows 10 with Docker. You should run this commands step for step
docker run --rm --privileged alpine hwclock -s
again
docker run --rm --privileged alpine hwclock -s
and last command , don't forget to set your username and password and your timezone, to run minIO while Docker is
docker run -p 9000:9000 -e "MINIO_ACCESS_KEY=yourUserName" -e "MINIO_SECRET_KEY=YourPassword" -e "TZ=Europe/Berlin" -v /etc/localtime:/etc/localtime:ro minio/minio server /data
It is a little crude but this worked for me
Did a curl to s3 server
curl s3.amazonaws.com -v
Then got this
* Trying 52.216.141.158...
* TCP_NODELAY set
* Connected to s3.amazonaws.com (52.216.141.158) port 80 (#0)
> GET / HTTP/1.1
> Host: s3.amazonaws.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 307 Temporary Redirect
< x-amz-id-2: q2wUOf5ZC7iu2ymbRWUpZaM6GpPLLf/irrntuw/JNB7QYxDzQvcLHQbsbF2dp5zT8rBrGwqnOz0=
< x-amz-request-id: T4H1W4WKBE3F39RM
< Date: Sat, 09 Oct 2021 19:21:24 GMT
< Location: https://aws.amazon.com/s3/
< Server: AmazonS3
< Content-Length: 0
<
* Connection #0 to host s3.amazonaws.com left intact
* Closing connection 0
Got this date
Sat, 09 Oct 2021 19:21:24 GMT
Set the date in ubuntu
sudo date --set "Sat, 09 Oct 2021 19:21:24 GMT"
My code stopped throwing exceptions
Now I have a script that does this periodically every month
To get rid of this problem, you have to adjust the client's timing so that there is a maximum time stamp difference of up to 15 minutes. Also set the standard time and zone for your system.
Check the full details here.
I have the exact same error message but it's not the same cause as any of the others above.
In my case I have a React browser app doing something like this:
import { Storage } from '#aws-amplify/storage'
...
await Promise.all(files.map(file => Storage.put(...)))
I am uploading a lot of files over a slow network connection.
With this code, the promises are all started at once, so the request time for all the requests is the same, but because the browser (or amplify?) is throttling the number of concurrent connections, the later requests don't actually hit the server until more than 15 minutes after they were created.
The solution is to limit the concurrency of the promise creation - e.g. use something like bluebird Promise.map with the concurrency option
Using ntp may not work on all version of your Linux based server (e.g. an out of date Ubuntu server version that is no longer supported which will block you from downloading ntp if it is not already installed).
If this is your situation, you can set independent time zones for your Linux VM:
https://community.rackspace.com/products/f/25/t/650
After you do this you may need to reset the time/date. Instructions for doing this are in this article:
http://codeghar.wordpress.com/2007/12/06/manage-time-in-ubuntu-through-command-line
If u are in 2016 and in Istanbul here is a weird situation that Turkey decided not to switch to winter time standards anyway set your local timezone to Moscow then restart your machine.
I ran into this issue running Jet (Codeship) and Terraform on MacOS using Docker for Mac Beta channel 1.13.1-beta42.
Failed to read state: Error reloading remote state: RequestTimeTooSkewed: The difference between the request time and the current time is too large.
status code: 403, request id: 9D32BA2A5360FC18
This was resolved by restarting Docker.
I've just started getting this error, and syncing my clock doesn't help. (I've spent 2 hours syncing it to every timeserver I can find, including the AWS servers, but nothing makes a difference.)
Exactly the same thing started happening a year ago on Dec 31 2017. In that case, rebooting my system, and rebuilding my server (that uses the aws java sdk) fixed it. I don't know why. I assumed that AWS had some end-of-year timezone peculiarity. It's also possible that while I was doing these things, AWS timeservers fixed themselves. I have no way to test that hypothesis.
Now, the same thing has suddenly started to happen on Dec 30, 2018. It's not right at year-end, but close enough to seem suspicious. (Never got this error except on these dates.) Rebooting and rebuilding isn't helping this time.
My dev environment on this box is Windows 10 under Parallels. Nothing else on my system has changed - as I've double-checked by rolling back to prior Parallels snapshots. The clocks on both my host MacOS and the virtual Windows 10 are correct.
I'm suspecting an AWS bug.
Rebooting my windows server fixed it for me
The time was identical to ~1 second to the site time.in, so it wasn't off.
I was running into the same issue on my Mac. When I moved to a different timezone(PST to IST), somehow OSX was not picking timezone and time change automatically. So I had to set the two manually and that caused a lag of some 15-20 seconds on my laptop. After setting the automatic sync, the time got synched and the S3 copy command started working: For reference
You can use this tool for organizing your time with AWS and local system.
To synchronize time:
sudo yum -y install chrony
sudo systemctl enable chronyd
sudo systemctl start chronyd
This issue generally occurs when s3cmd client machine time is not synced with server.
Check time of both machine.
either sync time between them using date command
Client# sudo date --set="string"
Client# sudo date --set="15 MAY 2011 1:40 PM"
or
install chrony and restart its service on both machine
Client# sudo apt-get install chrony
Client# vi /etc/chrony/chrony.conf
pool ntp-server iburst
Client# sudo systemctl restart chronyd