Why won't chrony sync to GPS when serial data flows through socat? - gps

I'm using gpsd to sync time to a GPS. When I connect my GPS to /dev/ttyUSB0, and tell gpsd to listen on that port, chrony is happy to use it as a time source.
gpsd -D 5 -N -n /dev/ttyUSB0
However, as soon as I try and pipe that data through socat (which is how it needs to work in our production system), chrony won't use it as a source. This is the case even though gpsd, cgps, and gpsmon all seem perfectly happy with the GPS data they are getting.
Here's my socat:
socat -d -d pty,rawer,echo=0,link=/tmp/ttyVSP0 /dev/ttyUSB0,b4800
(my gpsd command is the same as above but with /tmp/ttyVSP0 as the port to listen to in this case).
I'm using chronyc sources to confirm when GPS is a chrony source.
My refclock line in my /etc/chrony/chrony.conf looks like this:
refclock SHM 0 refid GPS

pty ports are prevented from talking to ntp (and thus, chrony) by an early return meant to prevent code from being executed during testing.
void ntpshm_link_activate(struct gps_device_t *session)
/* set up ntpshm storage for a session */
{
/* don't talk to NTP when we're running inside the test harness */
if (session->sourcetype == source_pty)
return;
if (session->sourcetype != source_pps ) {
/* allocate a shared-memory segment for "NMEA" time data */
session->shm_clock = ntpshm_alloc(session->context);
if (session->shm_clock == NULL) {
gpsd_log(&session->context->errout, LOG_WARN,
"NTP: ntpshm_alloc() failed\n");
return;
}
}
Discovered thanks to this bug report

Related

Syslog-ng logs not processing certain logs possibly due to journal cursor issue

I'm using syslog-ng 3.37.1 on a VMware Photon 3.0 virtual appliance (preconfigured VM). The appliance is configured to write logs into certain files under /var/log folder as well as to remote syslog servers (optional).
Logs from facility 'auth' and 'authpriv' are configured to write to /var/log/auth.log, as well as send it over to remote syslog server when enabled.
In addition, there are other messages as well from kernel, systemd services as well as other processes, configured to be processed via syslog-ng.
Issue is that, logs from a few facilities (such as auth, authpriv, cron etc) are not processed (received?) by syslog-ng initially. So, any SSH events, TTY login events are not logged into the file and remote. However, many other events from kernel, systemd and other processes are logged fine.
Below is the configuration for auth.log, that does not log in the first boot.
filter f_auth { facility(auth) or facility(authpriv)); };
destination authlog { file("/var/log/auth.log" perm(0600)); };
log { source(s_local); filter(f_auth); destination(authlog); };
I updated the filter as below without any success
filter f_auth {
facility(auth) or facility(authpriv) or
match('sshd' value('PROGRAM')) or match('systemd-logind' value('PROGRAM'));
};
In journal logs I can observe the relevant logs, for example, below command to view SSH logs.
journalctl -f -u sshd
Additional syslog-ng service restart or config reload during appliance startup do not fix this.
The log file /var/log/auth.log (and also cron log etc) show zero size during this time. Syslog-ng log looks fine too.
However, if I generate some auth facility event (say, SSH/TTY login) and manually restart syslog-ng, all the log entries (including old events) are immediately written into filesystem log (/var/log/auth.log) and also sent to remote syslog server.
In the syslog-ng.log I find below entry when it starts working that way.
syslog-ng[481]: [date] Failed to seek journal to the saved cursor position; cursor='', error='Invalid argument (22)'
It makes me wonder if it is due to some bad cursor position. However, I can still see other systemd and kernel logs being logged fine. So, not sure.
What could be causing such behaviour? How can I ensure that syslog-ng is able to receive and process these logs without manual intervention?
Below is more detailed configuration for reference:
#version: 3.37
#include "scl.conf"
source s_local {
system();
internal();
udp();
};
destination d_local {
file("/var/log/messages");
file("/var/log/messages-kv.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3));
};
log {
source(s_local);
# uncomment this line to open port 514 to receive messages
#source(s_network);
destination(d_local);
};
filter f_auth {
facility(auth) or facility(authpriv)); # Also tried facility (auth, authpriv)
};
destination authlog { file("/var/log/auth.log" perm(0600)); };
log { source(s_local); filter(f_auth); destination(authlog); };
destination d_kern { file("/dev/console" perm(0600)); };
filter f_kern { facility(kern); };
log { source(s_local); filter(f_kern); destination(d_kern); };
destination d_cron { file("/var/log/cron" perm(0600)); };
filter f_cron { facility(cron); };
log { source(s_local); filter(f_cron); destination(d_cron); };
destination d_syslogng { file("/var/log/syslog-ng.log" perm(0600)); };
filter f_syslogng { program(syslog-ng); };
log { source(s_local); filter(f_syslogng); destination(d_syslogng); };
# A few more of above kind of configuration follows here.
# Add configuration files that have remote destination, filter and log configuration for remote servers
#include "remote/*.conf"
As can be seen, /var/log/auth.log should hold logs from auth facility, but the log remains blank until subsequent restart of syslog-ng after a syslog config change (via API) or manual login into the system. However, triggering automated restart of syslog-ng using cron (without additional syslog config change) does not help.
Any thoughts, suggestions?
This is probably caused by your real time clock going backwards. The notification mechanism in libsystemd does not work in this case.
There's a proof-of-concept patch in this syslog-ng issue:
https://github.com/syslog-ng/syslog-ng/issues/2836
But I've increased the priority to tackle that problem and fix this, as it is causing issue more often than I anticipated.
As a workaround you should synchronize the time for your VM, preferably so that during boot it waits until a sync and then keep the time synchronized by ntp.

Switching USB DWC3 controller from host to device mode

I need to use an embedded Linux platform as a USB device in order to stream audio and video from a smartphone. The platform has a USB A receptacle and doesn't support OTG (USB_ID pin is not connected on the host controller).
Now I try to switch from host to device mode using DWC3 controller and the debugfs interface. Therefore I activated DWC3 controller in the kernel configuration and set it to "Dual Role Mode". After mounting the file system I checked the current mode in /sys/kernel/debug/xxxxxxxx.usb3/mode with cat mode and got host as expected. But unfortunately I can't write device to the mode file. After entering the command echo device > mode it remains host and does not change. Does anyone know what could be causing it?
I know that this used to work out of the box, but with the kernel version I'm using in an embedded platform (5.4.0) it doesn't. In order to make it work I had to patch around a bug in the kernel:
diff --git a/drivers/usb/dwc3/core.c b/drivers/usb/dwc3/core.c
index bf1a7a9da..a78990664 100644
--- a/drivers/usb/dwc3/core.c
+++ b/drivers/usb/dwc3/core.c
## -110,9 +110,6 ## static void __dwc3_set_mode(struct work_struct *work)
unsigned long flags;
int ret;
- if (dwc->dr_mode != USB_DR_MODE_OTG)
- return;
-
if (dwc->current_dr_role == DWC3_GCTL_PRTCAP_OTG)
dwc3_otg_update(dwc, 0);
## -192,6 +189,12 ## void dwc3_set_mode(struct dwc3 *dwc, u32 mode)
dwc->desired_dr_role = mode;
spin_unlock_irqrestore(&dwc->lock, flags);
+ if (dwc->dr_mode != USB_DR_MODE_OTG)
+ {
+ __dwc3_set_mode(&dwc->drd_work);
+ return;
+ }
+
queue_work(system_freezable_wq, &dwc->drd_work);
}
After applying the above patch, rebooting to the new kernel, and running:
echo "host" | sudo tee /sys/kernel/debug/*.usb3/mode
I was able to switch modes like I expect (note: I had *.dwc3 instead of *.usb3 in the above command).

Nagios host notifications not sending via email or logging

I am re-doing our nagios infrastructure with puppet but I am currently stopped at a seemingly simple problem (most likely a config issue).
Using puppet, I spit out some basic nagios config files on disk. Nagios reloads fine and everything looks okay in the UI but, when I mark a host down, it does not send a notification.
nagios.log shows:
[1470699491] EXTERNAL COMMAND:
PROCESS_HOST_CHECK_RESULT;divcont01;1;test notification
[1470699491] PASSIVE HOST CHECK: divcont01;1;test notification
[1470699491] HOST ALERT: divcont01;DOWN;HARD;1;test notification
In production (where I have changed nothing), I see in nagios.log (after marking a host down in ui):
[1470678186] EXTERNAL COMMAND:
PROCESS_HOST_CHECK_RESULT;PALTL12;1;test ey
[1470678187] PASSIVE HOST CHECK: PALTL12;1;test ey
[1470678187] HOST ALERT: PALTL12;DOWN;HARD;1;test ey
[1470678187] HOST NOTIFICATION:
pal_infra;PALTL12;DOWN;host-notify-by-pom;test ey
[1470678187] HOST NOTIFICATION:
pal_infra;PALTL12;DOWN;host-notify-by-email;test ey
[1470678192] HOST ALERT: PALTL12;UP;HARD;1;PING OK - Packet loss = 0%,
RTA = 0.81 ms
[1470678192] HOST NOTIFICATION:
pal_infra;PALTL12;UP;host-notify-by-pom;PING OK - Packet loss = 0%,
RTA = 0.81 ms
[1470678192] HOST NOTIFICATION:
pal_infra;PALTL12;UP;host-notify-by-email;PING OK - Packet loss = 0%,
RTA = 0.81 ms
As seen in the logs, there is a HOST NOTIFICATION logged and sent directly after the HOST ALERT in prod. I have been exhaustively comparing config files today and I cannot find a reason why the new config stops short of the notification.
I have verified that notifications are enabled at the top level. I have verified that email can be sent from this box (though, I am using the logs to verify functionality, not email). I have also tried multiple other google suggestions (and will continue my search too).
Relevant config details below. Please pardon the verbosity of my configuration and lackluster stack-overflow formatting. Thank you in advance.
hosts/divcont01.cfg:
define host {
address snip
host_name divcont01
use generic-host-puppetized
}
host-templates/generic-host-puppetized.cfg:
define host {
check_command check-host-alive
check_interval 1
contact_groups generic-contactgroup
checks_enabled 1
event_handler_enabled 0
flap_detection_enabled 0
name generic-host-puppetized
hostgroups +generic-host-puppetized
max_check_attempts 4
notification_interval 4
notification_options d,u,r
notification_period 24x7
notifications_enabled 1
process_perf_data 0
register 0
retain_nonstatus_information 1
retain_status_information 1
}
hostgroups/generic-host-puppetized.cfg:
define hostgroup {
hostgroup_name generic-host-puppetized
}
contactgroups/generic-contactgroup.cfg
define contactgroup {
contactgroup_name generic-contactgroup
members generic-puppetized-contact
}
contacts/generic-puppetized-contact.cfg
define contact {
use generic-contact
contact_name generic-puppetized-contact
email <my email>
}
objects/templates.cfg (generic-contact config only)
define contact{
use my email
name generic-contact ; The name of this contact template
service_notification_period 24x7 ; service notifications can be sent anytime
host_notification_period 24x7 ; host notifications can be sent anytime
host_notification_commands generic-puppetized-contact-host-notify-by-email-low
service_notification_commands notify-by-email,service-notify-by-pom
service_notification_options u,c,r,f ; send notifications for all service states, flapping events, and scheduled downtime events
host_notification_options d,r,f ; send notifications for all host states, flapping events, and scheduled downtime events
register 0 ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL CONTACT, JUST A TEMPLATE!
}
commands/generic-puppetized-contact-host-notify-by-email-low.cfg:
define command {
command_line /etc/nagios/global/scripts/nagios-mailx.sh -t my email -s "** notification Host Alert: hostname is hoststate **" -m "***** Nagios ***** Notification Type: notification type Host: host State: hoststate Address: address Info: output Date/Time: date"
command_name generic-puppetized-contact-host-notify-by-email-low
}
Figured it out...I was building my system within another pre-existing system (dangerous, I know) and my contacts were actually pointing to a generic-contact that had its notifications disabled.
Whoops :)

logstash and centralised redis problems

I'm trying to get logstash working in a centralised setup using the docs as an example:
http://logstash.net/docs/1.2.2/tutorials/getting-started-centralized
I've got logstash (as indexer), redis, elasticsearch and standalone kibana3 running on my web server. I then need to run logstash as an agent on another server to collect apache logs and send them to the web server via redis. The number of agents will increase and the logs will vary, but for now I just want to get this working!
I need everything to run as a service so that all is well after reboots etc. All servers are running Ubuntu.
For all logstash instances (indexer and agent), I'm using the following init script (Ubuntu version, second gist):
https://gist.github.com/shadabahmed/5486949#file-logstash-ubuntu
For running redis as a service, I followed the instructions here:
http://redis.io/topics/quickstart (Installing redis more properly)
Elasticsearch is also running as a service.
On the web server, running redis-cli returns PONG correctly. Navigating to the correct Elasticsearch URL returns the correct JSON response. Navigating to the Kibana3 url gives me the dashboard, but no data. UFW is set to allow the redis port (at the moment from everywhere).
On the web server, my logstash.conf is:
input {
file {
path => "/var/log/apache2/access.log"
type => "apache-access"
sincedb_path => "/etc/logstash/.sincedb"
}
redis {
host => "127.0.0.1"
data_type => "list"
key => "logstash"
codec => json
}
}
filter {
grok {
type => "apache-access"
pattern => "%{COMBINEDAPACHELOG}"
}
}
output {
elasticsearch {
embedded => true
}
statsd {
# Count one hit every event by response
increment => "apache.response.%{response}"
}
}
From the agent server, I can telnet successfully to the web server IP and redis port. logstash is running. The logstash.conf file is:
input {
file {
path => "/var/log/apache2/shift.access.log"
type => "apache"
sincedb_path => "/etc/logstash/since_db"
}
stdin {
type => "example"
}
}
filter {
if [type] == "apache" {
grok {
pattern => "%{COMBINEDAPACHELOG}"
}
}
}
output {
stdout { codec => rubydebug }
redis { host => ["xx.xx.xx.xx"] data_type => "list" key => "logstash" }
}
If I comment out the stdin and stdout lines, I still don't get a result. The logstash logs do not give me any connection errors - only warnings about the deprecated grok settings format.
I have also tried running logstash from the command line (making sure to stop the demonised service first). The apache log file is correctly outputted in the terminal, so I know that logstash is accessing the log correctly. And I can write random strings and they are output in the correct logstash format.
The redis logs on the web server show no sign of trouble......
The frustrating thing is that this has worked once. One message from stdin made it all the way through to elastic search. That was this morning just after getting everything setup. Since then, I have had no luck and I have no idea why!
Any tips/pointers gratefully received... Solving my problem will stop me tearing out more of my hair which will also make my wife happy......
UPDATE
Rather than filling the comments....
Thanks to #Vor and #rutter, I've confirmed that the user running logstash can read/write to the logstash.log file.
I've run the agent with -vv and the logs are populated with e.g.:
{:timestamp=>"2013-12-12T06:27:59.754000+0100", :message=>"config LogStash::Outputs::Redis/#host = [\"XX.XX.XX.XX\"]", :level=>:debug, :file=>"/opt/logstash/logstash.jar!/logstash/config/mixin.rb", :line=>"104"}
I then input random text into the terminal and get stdout results. However, I do not see anything in the logs until AFTER terminating the logstash agent. After the agent is terminated, I get lines like these in the logstash.log:
{:timestamp=>"2013-12-12T06:27:59.835000+0100", :message=>"Pipeline started", :level=>:info, :file=>"/opt/logstash/logstash.jar!/logstash/pipeline.rb", :line=>"69"}
{:timestamp=>"2013-12-12T06:29:22.429000+0100", :message=>"output received", :event=>#<LogStash::Event:0x77962b4d #cancelled=false, #data={"message"=>"test", "#timestamp"=>"2013-12-12T05:29:22.420Z", "#version"=>"1", "type"=>"example", "host"=>"Ubuntu-1204-precise-64-minimal"}>, :level=>:info, :file=>"(eval)", :line=>"16"}
{:timestamp=>"2013-12-12T06:29:22.461000+0100", :level=>:debug, :host=>"XX.XX.XX.XX", :port=>6379, :timeout=>5, :db=>0, :file=>"/opt/logstash/logstash.jar!/logstash/outputs/redis.rb", :line=>"230"}
But while I do get messages in stdout, I get nothing in redis on the other server. I can however telnet to the correct port on the other server, and I get "ping/PONG" in telnet, so redis on the other server is working..... And there are no errors etc in the redis logs.
It looks to me very much like the redis plugin on the logstash shipper agent is not working as expected, but for the life of me, I can't see where the breakdown is coming from.....

how to syslog-ng to remote facility

i have a host running syslog-ng. it does all it's stuff locally fine (creating log files etc). however, i would like to forward ALL of it's logs to a remote machine - specifically to one facility on the remote machine (local4). i tried playing around with rewrite (set-facility) and templates within the destination (syntax errors) - but to no avail.
destination remote_server {
udp(\"172.18.192.8\" port (514));
udp(\"172.18.192.9\" port (514));
};
rewrite r_local4 {
set-facility(local4);
};
filter f_alllogs {
level (debug...emerg);
};
log {
source(local);
filter(f_alllogs);
rewrite(r_local4)
destination(remote_server);
};
AFAIK, currently it is not possible to modify the facility of a message in syslog-ng.
Is there a special reason you want to do it?