I have 6 machines: 3 running RHEL 3 (Taroon Update 9) and 3 other running RHEL 5.6 (Tikanga).
They all share a NFS volume containing
an Apache 2.2.4 installation
that Apache's configuration files
the PHP scripts and other content served by that Apache
On each machine, that shared Apache is started with the "-D $hostname" option to distinguish from the Apache instances running on the other machines. (Where $hostname is the hostname of the machine).
There is a load balancer (another Apache) in front of these 6 machines, balancing the requests between them.
The problem:
I've placed a .htaccess file in a directory containing these rules:
order deny,allow
deny from all
allow from my.very.own.ip
(I've tried the same rules in the configuration file between , with the same result)
Any request from any IP other than my.very.own.ip is denied by all 6 Apaches.
Requests from my.very.own.ip that hit the first 3 Apaches (running on RHEL 3) are accepted, which is normal.
Requests from my.very.own.ip that hit the last 3 Apaches (running on RHEL 5.6) are denied (HTTP 403, client denied by server configuration), which is not normal.
I've ran a strace on all the Apache servers (and their forked children) and this is how they look like:
- on a "good" Apache with request coming from my.very.own.ip
19553 read(7, "GET /my/website"..., 8000) = 1761
19553 gettimeofday({1343736878, 168708}, NULL) = 0
19553 gettimeofday({1343736878, 168787}, NULL) = 0
19553 gettimeofday({1343736878, 168835}, NULL) = 0
19553 stat64("/shark/apps/apache1/conf/www/maps/stationlist.txt", {st_mode=S_IFREG|0644, st_size=5557, ...}) = 0
19553 stat64("/shark/apps/apache1/conf/somedir/stationkidsurls.txt", {st_mode=S_IFREG|0644, st_size=6200, ...}) = 0
19553 stat64("/shark/www_docs/my/website/php/script.php", {st_mode=S_IFREG|0644, st_size=2449, ...}) = 0
19553 open("/shark/www_docs/.htaccess", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
19553 open("/shark/www_docs/my/.htaccess", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
19553 open("/shark/www_docs/my/website/.htaccess", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
19553 open("/shark/www_docs/my/website/php/.htaccess", O_RDONLY|O_LARGEFILE) = 10
19553 fstat64(10, {st_mode=S_IFREG|0644, st_size=51, ...}) = 0
19553 read(10, " order deny,allow\n deny f"..., 4096) = 51
19553 read(10, "", 4096) = 0
19553 close(10) = 0
19553 open("/shark/www_docs/my/website/php/script.php/.htaccess", O_RDONLY|O_LARGEFILE) = -1 ENOTDIR (Not a directory)
19553 getpid() = 19553
19553 setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={60, 0}}, NULL) = 0
19553 rt_sigaction(SIGPROF, {0x1280014, [PROF], SA_RESTORER|SA_RESTART, 0x3c10d8}, {0x1280014, [PROF], SA_RESTORER|SA_RESTART, 0x3c10d8}, 8) = 0
19553 rt_sigprocmask(SIG_UNBLOCK, [PROF], NULL, 8) = 0
19553 getcwd("/", 4095) = 2
19553 chdir("/shark/www_docs/my/website/php") = 0
- on a "bad" Apache with request coming from my.very.own.ip
1723 read(9, "GET /my/website"..., 8000) = 1761
1723 gettimeofday({1343736621, 548677}, NULL) = 0
1723 gettimeofday({1343736621, 548735}, NULL) = 0
1723 gettimeofday({1343736621, 548771}, NULL) = 0
1723 stat64("/shark/apps/apache1/conf/www/maps/stationlist.txt", {st_mode=S_IFREG|0644, st_size=5557, ...}) = 0
1723 stat64("/shark/apps/apache1/conf/somedir/stationkidsurls.txt", {st_mode=S_IFREG|0644, st_size=6200, ...}) = 0
1723 stat64("/shark/www_docs/my/website/php/script.php", {st_mode=S_IFREG|0644, st_size=2449, ...}) = 0
1723 open("/shark/www_docs/.htaccess", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
1723 open("/shark/www_docs/my/.htaccess", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
1723 open("/shark/www_docs/my/website/.htaccess", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
1723 open("/shark/www_docs/my/website/php/.htaccess", O_RDONLY|O_LARGEFILE) = 12
1723 fstat64(12, {st_mode=S_IFREG|0644, st_size=51, ...}) = 0
1723 read(12, " order deny,allow\n deny f"..., 4096) = 51
1723 read(12, "", 4096) = 0
1723 close(12) = 0
1723 open("/shark/www_docs/my/website/php/script.php/.htaccess", O_RDONLY|O_LARGEFILE) = -1 ENOTDIR (Not a directory)
1723 gettimeofday({1343736621, 550606}, NULL) = 0
1723 write(10, "[Tue Jul 31 08:10:21 2012] [erro"..., 159) = 159
729 <... read resumed> "[Tue Jul 31 08:10:21 2012] [erro"..., 65536) = 159
729 gettimeofday({1343736621, 550743}, NULL) = 0
729 gettimeofday({1343736621, 550786}, NULL) = 0
What do you think?
The only difference between these machines in the RHEL version. That may also mean that the NFS share ( /shark ) is mounted differently...
I've modified the .htaccess an changed the allow,deny rule with a Rewrite (requests not coming from my.very.own.ip are redirected to somewhere else). But... I'm still wondering why the same rules behave differently on different machines (same Apache, same configuration files).
Thank you!
Related
Ive launched a EMR cluster with jupyterhub included and set up LDAP following the guide below:
https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-jupyterhub-ldap-users.html
I can log into jupyterhub using LDAP but it doesnt launch the notebook server and throws a error:
500 : Internal Server Error
Spawner failed to start [status=1]. The logs for joe.blogg may contain
details.
You can try restarting your server from the home page.
Checking the jupyterhub logs i find the following error:
[I 2019-04-30 12:24:11.590 JupyterHub log:122] 302 GET / → /hub (#::ffff:172.31.150.206) 0.75ms
[I 2019-04-30 12:24:11.629 JupyterHub log:122] 302 GET /hub → /hub/home (joe.blogg#::ffff:172.31.150.206) 2.22ms
[I 2019-04-30 12:24:11.685 JupyterHub log:122] 200 GET /hub/home (joe.blogg#::ffff:172.31.150.206) 26.51ms
[I 2019-04-30 12:24:13.741 JupyterHub log:122] 302 GET /hub/spawn → /user/joe.blogg/ (joe.blogg#::ffff:172.31.150.206) 3.70ms
[I 2019-04-30 12:24:13.769 JupyterHub log:122] 302 GET /user/joe.blogg/ → /hub/user/joe.blogg/ (#::ffff:172.31.150.206) 0.51ms
[I 2019-04-30 12:24:13.847 JupyterHub spawner:978] Spawning jupyterhub-singleuser --port=46821 --debug
[D 2019-04-30 12:24:14.255 SingleUserNotebookApp application:177] Searching ['/home', '/home/users/joe.blogg/.jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2019-04-30 12:24:14.256 SingleUserNotebookApp application:555] Looking for jupyter_config in /etc/jupyter
[D 2019-04-30 12:24:14.256 SingleUserNotebookApp application:555] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2019-04-30 12:24:14.256 SingleUserNotebookApp application:555] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2019-04-30 12:24:14.256 SingleUserNotebookApp application:555] Looking for jupyter_config in /home/users/joe.blogg/.jupyter
[D 2019-04-30 12:24:14.256 SingleUserNotebookApp application:555] Looking for jupyter_config in /home
[D 2019-04-30 12:24:14.257 SingleUserNotebookApp application:555] Looking for jupyter_notebook_config in /etc/jupyter
[D 2019-04-30 12:24:14.257 SingleUserNotebookApp application:577] Loaded config file: /etc/jupyter/jupyter_notebook_config.py
[D 2019-04-30 12:24:14.258 SingleUserNotebookApp application:555] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 2019-04-30 12:24:14.258 SingleUserNotebookApp application:555] Looking for jupyter_notebook_config in /opt/conda/etc/jupyter
[D 2019-04-30 12:24:14.258 SingleUserNotebookApp application:577] Loaded config file: /opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2019-04-30 12:24:14.258 SingleUserNotebookApp application:555] Looking for jupyter_notebook_config in /home/users/joe.blogg/.jupyter
[D 2019-04-30 12:24:14.258 SingleUserNotebookApp application:555] Looking for jupyter_notebook_config in /home
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 528, in get
value = obj._trait_values[self.name]
KeyError: 'runtime_dir'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/bin/jupyterhub-singleuser", line 6, in <module>
main()
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 455, in main
return SingleUserNotebookApp.launch_instance(argv)
File "/opt/conda/lib/python3.6/site-packages/jupyter_core/application.py", line 266, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "<decorator-gen-7>", line 2, in initialize
File "/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1366, in initialize
self.init_configurables()
File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1100, in init_configurables
connection_dir=self.runtime_dir,
File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 556, in __get__
return self.get(obj, cls)
File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 535, in get
value = self._validate(obj, dynamic_default())
File "/opt/conda/lib/python3.6/site-packages/jupyter_core/application.py", line 99, in _runtime_dir_default
ensure_dir_exists(rd, mode=0o700)
File "/opt/conda/lib/python3.6/site-packages/jupyter_core/utils/__init__.py", line 13, in ensure_dir_exists
os.makedirs(path, mode=mode)
File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
[Previous line repeated 1 more times]
File "/opt/conda/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/users'
[W 2019-04-30 12:24:23.874 JupyterHub web:1588] 500 GET /hub/user/joe.blogg/ (::ffff:172.31.150.206): Spawner failed to start [status=1]. The logs for joe.blogg may contain details.
[E 2019-04-30 12:24:23.886 JupyterHub log:114] {
"X-Forwarded-Host": "ip-172-31-150-206.eu-west-1.compute.internal:9443",
"X-Forwarded-Proto": "https",
"X-Forwarded-Port": "9443",
"X-Forwarded-For": "::ffff:172.31.150.206",
"Upgrade-Insecure-Requests": "1",
"Cookie": "jupyter-hub-token=\"2|1:0|10:1556624768|17:jupyter-hub-token|44:ZThlOTZkYWM0NzRiNDRkMDlmYzdkNDUwOTUzMTNjYjA=|12a53077b8d92723bba01fc9273eb64050911e22317385f96c1c4f52ff5253a8\"; _xsrf=2|5370034d|137cc417d37f89a6aed65c0ec72ad572|1556623914",
"Connection": "close",
"Referer": "https://ip-172-31-150-206.eu-west-1.compute.internal:9443/hub/home",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.5",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0",
"Host": "ip-172-31-150-206.eu-west-1.compute.internal:9443"
}
[E 2019-04-30 12:24:23.886 JupyterHub log:122] 500 GET /hub/user/joe.blogg/ (joe.blogg#::ffff:172.31.150.206) 10088.04ms
[W 2019-04-30 12:24:42.370 JupyterHub user:458] joe.blogg's server never showed up at http://127.0.0.1:46821/user/joe.blogg/ after 30 seconds. Giving up
[E 2019-04-30 12:24:42.380 JupyterHub gen:914] Exception in Future <tornado.concurrent.Future object at 0x7f53b1b9ce10> after timeout
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 910, in error_callback
future.result()
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 445, in finish_user_spawn
yield spawn_future
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/user.py", line 476, in spawn
raise e
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/user.py", line 450, in spawn
resp = yield server.wait_up(http=True, timeout=spawner.http_timeout)
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/utils.py", line 180, in wait_for_http_server
timeout=timeout
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/utils.py", line 135, in exponential_backoff
raise TimeoutError(fail_message)
TimeoutError: Server at http://127.0.0.1:46821/user/joe.blogg/ didn't respond in 30 seconds
The /etc/jupyter/conf/jupyterhub_conf file looks like the following after following aws guide:
# Configuration file for jupyterhub.
import os
notebook_dir = os.environ.get('DOCKER_NOTEBOOK_DIR')
network_name='jupyterhub-network'
c.Spawner.debug = True
c.Spawner.environment = {'SPARKMAGIC_CONF_DIR':'/etc/jupyter/conf'}
c.JupyterHub.hub_ip = '0.0.0.0'
c.JupyterHub.admin_access = True
c.JupyterHub.ssl_key = '/etc/jupyter/conf/server.key'
c.JupyterHub.ssl_cert = '/etc/jupyter/conf/server.crt'
c.JupyterHub.port = 9443
c.Authenticator.admin_users = {'jovyan'}
c.JupyterHub.authenticator_class = 'ldapauthenticator.LDAPAuthenticator'
c.LDAPAuthenticator.use_ssl = False
c.LDAPAuthenticator.server_address = 'openldap.companyx.com'
c.LDAPAuthenticator.bind_dn_template = 'cn={username},ou=Users,dc=openldap,dc=companyx,dc=com'
Has anyone else managed to setup multi user access on AWS EMR jupyterHub please.
I had faced similar issues, Can you try commenting out
#c.JupyterHub.ssl_key = '/etc/jupyter/conf/server.key'
#c.JupyterHub.ssl_cert = '/etc/jupyter/conf/server.crt'
After this try accessing with http (instead of https), this should fix your problem (Hope without https its ok for you)
Apache2 crashes and there's no logging in the error.log, not even after setting the Loglevel to debug in the apache2.conf.
Owner of the error.log is root
No updates or changes were made on the Server
Tried to downgrade from Apache (apt-get install --reinstall apache2=2.4.18-2ubuntu3 apache2-bin=2.4.18-2ubuntu3 apache2-data=2.4.18-2ubuntu3) but that doesn't help.
service apache2 start:
Job for apache2.service failed because the control process exited with error code. See "systemctl status apache2.service" and "journalctl -xe" for details.
systemctl status apache2.service:
apache2.service - LSB: Apache2 web server
Loaded: loaded (/etc/init.d/apache2; bad; vendor preset: enabled)
Drop-In: /lib/systemd/system/apache2.service.d
└─apache2-systemd.conf
Active: failed (Result: exit-code) since Tue 2017-04-04 16:06:03 UTC; 1min 26s ago
Docs: man:systemd-sysv-generator(8)
Process: 11071 ExecStart=/etc/init.d/apache2 start (code=exited, status=1/FAILURE)
Apr 04 16:05:43 bpi-iot-ros systemd[1]: Starting LSB: Apache2 web server...
Apr 04 16:05:43 bpi-iot-ros apache2[11071]: * Starting Apache httpd web server apache2
Apr 04 16:06:03 bpi-iot-ros apache2[11071]: *
Apr 04 16:06:03 bpi-iot-ros apache2[11071]: * The apache2 instance did not start within 20 seconds. Please read the log files to discover problems
Apr 04 16:06:03 bpi-iot-ros systemd[1]: apache2.service: Control process exited, code=exited status=1
Apr 04 16:06:03 bpi-iot-ros systemd[1]: Failed to start LSB: Apache2 web server.
Apr 04 16:06:03 bpi-iot-ros systemd[1]: apache2.service: Unit entered failed state.
Apr 04 16:06:03 bpi-iot-ros systemd[1]: apache2.service: Failed with result 'exit-code'.
● service.service
Loaded: not-found (Reason: No such file or directory)
Active: inactive (dead)
journalctl -xe:
Apr 04 16:09:22 bpi-iot-ros apache2[11191]: *
Apr 04 16:09:22 bpi-iot-ros apache2[11191]: * The apache2 instance did not start within 20 seconds. Please read the log files to discover problems
Apr 04 16:09:22 bpi-iot-ros systemd[1]: apache2.service: Control process exited, code=exited status=1
Apr 04 16:09:22 bpi-iot-ros systemd[1]: Failed to start LSB: Apache2 web server.
-- Subject: Unit apache2.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit apache2.service has failed.
--
-- The result is failed.
Apr 04 16:09:22 bpi-iot-ros systemd[1]: apache2.service: Unit entered failed state.
Apr 04 16:09:22 bpi-iot-ros systemd[1]: apache2.service: Failed with result 'exit-code'.
Error log, last lines from Monday:
[Mon Apr 03 08:56:07.530514 2017] [:error] [pid 30594] [client 192.168.2.185:52276] PHP Fatal error: Uncaught Doctrine\DBAL\DBALException: Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] No such file or directory in /var/www/nextcloud/lib/private/DB/Connection.php:60\nStack trace:\n#0 /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect()\n#1 /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion()\n#2 /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform()\n#3 /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(621): Doctrine\DBAL\Connection->getDatabasePlatform()\n#4 /var/www/nextcloud/lib/private/DB/Connection.php(147): Doctrine\DBAL\Connection->setTransactionIsolation(2)\n#5 /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DriverManager.php(172): OC\DB\Connection->__construct(Array, Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Obj in /var/www/nextcloud/lib/private/DB/Connection.php on line 60
[Mon Apr 03 08:56:12.034222 2017] [mpm_prefork:notice] [pid 625] AH00169: caught SIGTERM, shutting down
Edit:
apache2ctl configtest no Syntax errors.
apache2ctl -X
Child Process seems to hang up, can reach the SSL Warning (selfsigned) when attempting to reach the sitestarting
did a strace of Apache starting, some Childprocess exited with Errorcode 1 (huge File so i only copied the last Lines)
11028 open("/lib/arm-linux-gnueabihf/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
11028 read(3, "\177ELF\1\1\1\3\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\331i\1\0004\0\0\0"..., 512) = 512
11028 lseek(3, 894128, SEEK_SET) = 894128
11028 read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2960) = 2960
11028 lseek(3, 888324, SEEK_SET) = 888324
11028 read(3, "A2\0\0\0aeabi\0\1(\0\0\0\0057-A\0\6\n\7A\10\1\t\2\n\3\f"..., 51) = 51
11028 fstat64(3, {st_mode=S_IFREG|0755, st_size=897088, ...}) = 0
11028 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6f72000
11028 mmap2(NULL, 963928, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6e3e000
11028 mprotect(0xb6f14000, 65536, PROT_NONE) = 0
11028 mmap2(0xb6f24000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xd6000) = 0xb6f24000
11028 mmap2(0xb6f27000, 9560, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb6f27000
11028 close(3) = 0
11028 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6f71000
11028 set_tls(0xb6f714c0, 0xb6f71b98, 0xb6f7c050, 0xb6f714c0, 0xb6f7c050) = 0
11028 mprotect(0xb6f24000, 8192, PROT_READ) = 0
11028 mprotect(0xb6f51000, 8192, PROT_READ) = 0
11028 mprotect(0x21000, 4096, PROT_READ) = 0
11028 mprotect(0xb6f7b000, 4096, PROT_READ) = 0
11028 munmap(0xb6f73000, 21672) = 0
11028 ioctl(1, TCGETS, 0xbedb2554) = -1 EINVAL (Invalid argument)
11028 brk(NULL) = 0x1d23000
11028 brk(0x1d44000) = 0x1d44000
11028 stat64("/root/.terminfo", 0x1d23160) = -1 ENOENT (No such file or directory)
11028 stat64("/etc/terminfo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
11028 stat64("/lib/terminfo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
11028 stat64("/usr/share/terminfo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
11028 gettimeofday({1491321517, 321307}, NULL) = 0
11028 access("/etc/terminfo/x/xterm", R_OK) = -1 ENOENT (No such file or directory)
11028 access("/lib/terminfo/x/xterm", R_OK) = 0
11028 open("/lib/terminfo/x/xterm", O_RDONLY|O_LARGEFILE) = 3
11028 fstat64(3, {st_mode=S_IFREG|0644, st_size=3361, ...}) = 0
11028 read(3, "\32\1)\0&\0\17\0\235\1u\5xterm|xterm-debian|X"..., 4096) = 3361
11028 read(3, "", 4096) = 0
11028 close(3) = 0
11028 gettimeofday({1491321517, 324975}, NULL) = 0
11028 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0
11028 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0
11028 ioctl(2, TIOCGWINSZ, {ws_row=54, ws_col=189, ws_xpixel=0, ws_ypixel=0}) = 0
11028 fstat64(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
11028 write(1, "\33[39;49m", 8) = 8
10941 <... read resumed> "\33[39;49m", 128) = 8
11028 exit_group(0) = ?
10941 read(3, <unfinished ...>
11028 +++ exited with 0 +++
10941 <... read resumed> "", 128) = 0
10941 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=11028, si_uid=0, si_status=0, si_utime=0, si_stime=1} ---
10941 sigreturn({mask=[]}) = 0
10941 close(3) = 0
10941 wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 11028
10941 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb6fae418) = 11029
11029 close(12 <unfinished ...>
10941 wait4(-1, <unfinished ...>
11029 <... close resumed> ) = 0
11029 close(11) = 0
11029 close(10) = 0
11029 execve("/bin/echo", ["/bin/echo", "-e", " \33[31mfailed!\33[39;49m"], [/* 19 vars */]) = 0
11029 brk(NULL) = 0x318000
11029 uname({sysname="Linux", nodename="bpi-iot-ros", ...}) = 0
11029 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
11029 mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6f2e000
11029 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
11029 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
11029 fstat64(3, {st_mode=S_IFREG|0644, st_size=21672, ...}) = 0
11029 mmap2(NULL, 21672, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb6f28000
11029 close(3) = 0
11029 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
11029 open("/lib/arm-linux-gnueabihf/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
11029 read(3, "\177ELF\1\1\1\3\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\331i\1\0004\0\0\0"..., 512) = 512
11029 lseek(3, 894128, SEEK_SET) = 894128
11029 read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2960) = 2960
11029 lseek(3, 888324, SEEK_SET) = 888324
11029 read(3, "A2\0\0\0aeabi\0\1(\0\0\0\0057-A\0\6\n\7A\10\1\t\2\n\3\f"..., 51) = 51
11029 fstat64(3, {st_mode=S_IFREG|0755, st_size=897088, ...}) = 0
11029 mmap2(NULL, 963928, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6e1d000
11029 mprotect(0xb6ef3000, 65536, PROT_NONE) = 0
11029 mmap2(0xb6f03000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xd6000) = 0xb6f03000
11029 mmap2(0xb6f06000, 9560, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb6f06000
11029 close(3) = 0
11029 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6f27000
11029 set_tls(0xb6f27870, 0xb6f27f48, 0xb6f31050, 0xb6f27870, 0xb6f31050) = 0
11029 mprotect(0xb6f03000, 8192, PROT_READ) = 0
11029 mprotect(0x24000, 4096, PROT_READ) = 0
11029 mprotect(0xb6f30000, 4096, PROT_READ) = 0
11029 munmap(0xb6f28000, 21672) = 0
11029 brk(NULL) = 0x318000
11029 brk(0x339000) = 0x339000
11029 fstat64(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(136, 0), ...}) = 0
11029 write(1, " \33[31mfailed!\33[39;49m\n", 22) = 22
11029 close(1) = 0
11029 close(2) = 0
11029 exit_group(0) = ?
11029 +++ exited with 0 +++
10941 <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 11029
10941 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=11029, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
10941 sigreturn({mask=[]}) = 11029
10941 close(12) = 0
10941 close(11) = 0
10941 exit_group(1) = ?
10941 +++ exited with 1 +++
(Posted on behalf of the OP).
I dont know exactly how, but it's solved... maybe reinstalling libapache2-mod-php7.0 was the failure.
I'm trying to run autossh (on a VM running CentOS6), but it's exiting immediately with the help message. I think this is a system issue because when I run it with the exact same parameters on another computer (running Ubuntu 14.04) it works fine. It's also fine when I run the same command but with ssh instead of autossh. So I tried strace to see if anything's wrong there. But if there is I'm not sure what it is. Any ideas?
Here's the autossh command: autossh -oStrictHostKeyChecking=no -oServerAliveInterval=15 -oServerAliveCountMax=4 -L 3130:localhost:3130 -N -i /path/to/some.pem user#remotehost
Here's the strace output (note myserver is an entry in .ssh/config that contains the same parameters as the previous command):
execve("/usr/local/bin/autossh", ["autossh", "myserver"], [/* 55 vars */]) = 0
brk(0) = 0xefc000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f26193cc000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=36751, ...}) = 0
mmap(NULL, 36751, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f26193c3000
close(3) = 0
open("/lib64/libnsl.so.1", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p#\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=113432, ...}) = 0
mmap(NULL, 2198192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f2618f95000
mprotect(0x7f2618fab000, 2093056, PROT_NONE) = 0
mmap(0x7f26191aa000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x7f26191aa000
mmap(0x7f26191ac000, 6832, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f26191ac000
close(3) = 0
open("/lib64/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\356\1\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1920936, ...}) = 0
mmap(NULL, 3750152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f2618c01000
mprotect(0x7f2618d8b000, 2097152, PROT_NONE) = 0
mmap(0x7f2618f8b000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18a000) = 0x7f2618f8b000
mmap(0x7f2618f90000, 18696, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f2618f90000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f26193c2000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f26193c1000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f26193c0000
arch_prctl(ARCH_SET_FS, 0x7f26193c1700) = 0
mprotect(0x7f2618f8b000, 16384, PROT_READ) = 0
mprotect(0x7f26191aa000, 4096, PROT_READ) = 0
mprotect(0x7f26193cd000, 4096, PROT_READ) = 0
munmap(0x7f26193c3000, 36751) = 0
write(2, "usage: autossh [-V] [-M monitor_"..., 69usage: autossh [-V] [-M monitor_port[:echo_port]] [-f] [SSH_OPTIONS]
) = 69
write(2, "\n", 1
) = 1
write(2, " -M specifies monitor port. M"..., 238 -M specifies monitor port. May be overridden by environment
variable AUTOSSH_PORT. 0 turns monitoring loop off.
Alternatively, a port for an echo service on the remote
machine may be specified. (Normally port 7.)
) = 238
write(2, " -f run in background (autoss"..., 85 -f run in background (autossh handles this, and does not
pass it to ssh.)
) = 85
write(2, " -V print autossh version and"..., 39 -V print autossh version and exit.
) = 39
write(2, "\n", 1
) = 1
write(2, "Environment variables are:\n", 27Environment variables are:
) = 27
write(2, " AUTOSSH_GATETIME - how lo"..., 259 AUTOSSH_GATETIME - how long must an ssh session be established
before we decide it really was established
(in seconds). Default is 30 seconds; use of -f
flag sets this to 0.
) = 259
write(2, " AUTOSSH_LOGFILE - file t"..., 107 AUTOSSH_LOGFILE - file to log to (default is to use the syslog
facility)
) = 107
write(2, " AUTOSSH_LOGLEVEL - level "..., 49 AUTOSSH_LOGLEVEL - level of log verbosity
) = 49
write(2, " AUTOSSH_MAXLIFETIME - set th"..., 65 AUTOSSH_MAXLIFETIME - set the maximum time to live (seconds)
) = 65
write(2, " AUTOSSH_MAXSTART - max ti"..., 69 AUTOSSH_MAXSTART - max times to restart (default is no limit)
) = 69
write(2, " AUTOSSH_MESSAGE - messag"..., 74 AUTOSSH_MESSAGE - message to append to echo string (max 64 bytes)
) = 74
write(2, " AUTOSSH_PATH - path t"..., 53 AUTOSSH_PATH - path to ssh if not default
) = 53
write(2, " AUTOSSH_PIDFILE - write "..., 49 AUTOSSH_PIDFILE - write pid to this file
) = 49
write(2, " AUTOSSH_POLL - how of"..., 70 AUTOSSH_POLL - how often to check the connection (seconds)
) = 70
write(2, " AUTOSSH_FIRST_POLL - time b"..., 71 AUTOSSH_FIRST_POLL - time before first connection check (seconds)
) = 71
write(2, " AUTOSSH_PORT - port t"..., 61 AUTOSSH_PORT - port to use for monitor connection
) = 61
write(2, " AUTOSSH_DEBUG - turn l"..., 104 AUTOSSH_DEBUG - turn logging to maximum verbosity and log to
stderr
) = 104
write(2, "\n", 1
) = 1
exit_group(1) = ?
+++ exited with 1 +++
I had exactly the same problem with autossh 1.4e on CentOS Linux 7. autossh stopped immediately and printed the help, without even trying to connect to SSH.
The solution was to specify -M 0 on the command line:
autossh -M 0 \
-oStrictHostKeyChecking=no \
-oServerAliveInterval=15 \
-oServerAliveCountMax=4 \
-L 3130:localhost:3130 \
-N -i /path/to/some.pem user#remotehost
Come on ... the autossh exits with
write(2, "usage: autossh [-V] [-M monitor_"..., 69usage: autossh [-V] [-M monitor_port[:echo_port]] [-f] [SSH_OPTIONS]
which means that you specified wrong arguments and it is trying to explain you how the syntax looks like. To see what is wrong, there are several things you can try:
Use -vvv switches to ssh to see more verbose log from the ssh.
Use AUTOSSH_DEBUG environment variable to get some debug logs from autossh.
Thanks for Big Query.
When trying to load the NYS Voter File into Big Query we see this:
The file is
-rw-rw-r-- 1 fedex1 fedex1 5.3G Jun 3 06:43 AllNYSVoters.txt
Is 5.3 Gigs too large to load from the command line?
It is only 15,486,253 rows (15 Million rows)
It appears stackoverflow does not like code only questions, so I am giving more details.
> You have encountered a bug in the BigQuery CLI. Google engineers
> monitor and answer questions on Stack Overflow, with the tag
> google-bigquery:
> http://stackoverflow.com/questions/ask?tags=google-bigquery Please
> include a brief description of the steps that led to this issue, as
> well as the following information:
>
> ========================================
> == Platform == CPython:2.7.3:Linux-3.8.11-x86_64-with-Ubuntu-12.04-precise
> == bq version ==
> 2.0.21
> == Command line == ['/var/host/media/removable/USB_Drive/google-cloud-sdk/platform/bq/bq.py',
> '--credential_file',
> '/home/x1/.config/gcloud/legacy_credentials/x7/singlestore.json',
> '--project', 'personal-real-estate', 'load', 'nys.all_voter_file',
> 'AllNYSVoters.txt',
> 'LASTNAME:string,FIRSTNAME:string,MIDDLENAME:string,NAMESUFFIX:string,RADDNUMBER:string,RHALFCODE:string,RAPARTMENT:string,RPREDIRECTION:string,RSTREETNAME:string,RPOSTDIRECTION:string,RCITY:string,RZIP5:string,RZIP4:string,MAILADD1:string,MAILADD2:string,MAILADD3:string,MAILADD4:string,DOB:string,GENDER:string,ENROLLMENT:string,OTHERPARTY:string,COUNTYCODE:integer,ED:integer,LD:integer,TOWNCITY:string,WARD:string,CD:integer,SD:integer,AD:integer,LASTVOTEDDATE:timestamp,PREVYEARVOTED:string,PREVCOUNTY:string,PREVADDRESS:string,PREVNAME:string,COUNTYVRNUMBER:string,REGDATE:timestamp,VRSOURCE:string,IDREQUIRED:string,IDMET:string,STATUS:string,REASONCODE:string,INACT_DATE:timestamp,PURGE_DATE:timestamp,SBOEID:string,VoterHistory:string']
> == UTC timestamp == 2014-06-11 11:42:34
> == Error trace == File "/var/host/media/removable/USB_Drive/google-cloud-sdk/platform/bq/bq.py",
> line 806, in RunSafely
> return_value = self.RunWithArgs(*args, **kwds) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/platform/bq/bq.py",
> line 1047, in RunWithArgs
> job = client.Load(table_reference, source, schema=schema, **opts) File
> "/var/host/media/removable/USB_Drive/google-cloud-sdk/platform/bq/bigquery_client.py",
> line 2045, in Load
> upload_file=upload_file, **kwds) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/platform/bq/bigquery_client.py",
> line 1642, in ExecuteJob
> job_id=job_id) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/platform/bq/bigquery_client.py",
> line 1627, in RunJobSynchronously
> upload_file=upload_file, job_id=job_id) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/platform/bq/bigquery_client.py",
> line 1535, in StartJob
> projectId=project_id).execute() File "/var/host/media/removable/USB_Drive/google-cloud-sdk/platform/bq/bigquery_client.py",
> line 308, in execute
> return super(BigqueryHttp, self).execute(**kwds) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/util.py",
> line 132, in positional_wrapper
> return wrapped(*args, **kwargs) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/bin/bootstrapping/../../lib/apiclient/http.py",
> line 688, in execute
> _, body = self.next_chunk(http=http, num_retries=num_retries) File
> "/var/host/media/removable/USB_Drive/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/util.py",
> line 132, in positional_wrapper
> return wrapped(*args, **kwargs) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/bin/bootstrapping/../../lib/apiclient/http.py",
> line 867, in next_chunk
> headers=headers) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/util.py",
> line 132, in positional_wrapper
> return wrapped(*args, **kwargs) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/client.py",
> line 490, in new_request
> redirections, connection_type) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/bin/bootstrapping/../../lib/httplib2/__init__.py",
> line 1586, in request
> (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File
> "/var/host/media/removable/USB_Drive/google-cloud-sdk/bin/bootstrapping/../../lib/httplib2/__init__.py",
> line 1333, in _request
> (response, content) = self._conn_request(conn, request_uri, method, body, headers) File
> "/var/host/media/removable/USB_Drive/google-cloud-sdk/bin/bootstrapping/../../lib/httplib2/__init__.py",
> line 1289, in _conn_request
> response = conn.getresponse() File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
> response.begin() File "/usr/lib/python2.7/httplib.py", line 407, in begin
> version, status, reason = self._read_status() File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
> line = self.fp.readline() File "/usr/lib/python2.7/socket.py", line 430, in readline
> data = recv(1) File "/usr/lib/python2.7/ssl.py", line 241, in recv
> return self.read(buflen) File "/usr/lib/python2.7/ssl.py", line 160, in read
> return self._sslobj.read(len)
> ========================================
>
> Unexpected exception in load operation: [Errno 104] Connection reset
> by peer
The last line of the log seems to show what the problem was:
[Errno 104] Connection reset by peer
Was there any interruption in the network connectivity?
I successfully installed apache on my machine,but when I try to run it throws core dump.
I ran it again using strace,it throws core dump at write system call.
Below is the extract from output of stace. What can I infer from this error ?
write(2, "/tools/abc/local/bin/apachectl: line 79: 14372 Segmentation fault (core dump"..., 120/tools/abc/local/bin/apachectl: line 79: 14372 Segmentation fault (core dumped) $HTTPD -k $ARGV
) = 120
EDIT
Output of strace -s 100 httpd -k start
execve("/tools/abc/local/bin/httpd", ["httpd", "-k", "start"], [/* 38 vars */]) = 0
brk(0) = 0xe7e000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ffa0aece000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ffa0aecd000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ffa0aecc000
arch_prctl(ARCH_SET_FS, 0x7ffa0aecd680) = 0
--- SIGSEGV (Segmentation fault) # 0 (0) ---
+++ killed by SIGSEGV (core dumped) +++
Memory fault(coredump)