How to get rid of N/A from logs forwarded by nxlog? - nxlog

I have enabled debug mode in windows DNS server and the logfile is located at c:\logs\dns.log
<181>Jan 2 11:41:40 DC-SE-01 MSWinEventLog 1 N/A 1011398786 Tue Jan 2 11:41:40 2018 N/A N/A N/A N/A N/A N/A N/A N/A
<181>Jan 2 11:41:40 DC-IN-01 MSWinEventLog 1 N/A 1011398810 Tue Jan 2 11:41:40 2018 N/A N/A N/A N/A N/A N/A N/A N/A
<181>Jan 2 11:41:40 DC-IN-01 MSWinEventLog 1 N/A 1011398825 Tue Jan 2 11:41:40 2018 N/A N/A N/A N/A N/A N/A N/A 1/2/2018 11:41:38 AM 0A48 PACKET 00BACCA157DAE01 UDP Snd 11.11.201.81 3b20 R Q [8281 DR SERVFAIL] A (8)tnmaster(0) N/A
I think these messages are of little importance so how to get rid of these messages from nxlog and why are the "N/A" 's in there ?
Below are relevant parts of nxlog.conf file
<Input>
Module im_file
File "C:\logs\dns.log"
InputType LineBased
Exec $Message=$raw_event; $SyslogFacilityValue=22;
</Input>
<Output logger>
Module om_udp
Host 11.11.11.10
Port 514
Exec to_syslog_snare();
</Output>
<Route 3>
Path dnsdebug => logger
</Route>

The Snare syslog format is basically a tab delimited string that assumes certain fields such as the EventID since it was primarily designed to transfer the Windows Eventlog over syslog.
In order to generate the output these fields need to be populated. When you read the DNS log from a file obviously these fields are not automatically parsed, thus the output has N/A at those places.
For more information see the Snare topic in the NXLog User Guide.
Since you are trying to collect dns.log the Collecting DNS logs topic might be also relevant.

Related

Tensorflow 2.0 utilize all CPU cores 100%

My Tensorflow model makes heavy use of data preprocessing that should be done on the CPU to leave the GPU open for training.
top - 09:57:54 up 16:23, 1 user, load average: 3,67, 1,57, 0,67
Tasks: 400 total, 1 running, 399 sleeping, 0 stopped, 0 zombie
%Cpu(s): 19,1 us, 2,8 sy, 0,0 ni, 78,1 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
MiB Mem : 32049,7 total, 314,6 free, 5162,9 used, 26572,2 buff/cache
MiB Swap: 6779,0 total, 6556,0 free, 223,0 used. 25716,1 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17604 joro 20 0 22,1g 2,3g 704896 S 331,2 7,2 4:39.33 python
This is what top shows me. I would like to make this python process use at least 90% of available CPU across all cores. How can this be achieved?
GPU utilization is better, around 90%. Even though I don't know why it is not at 100%
Mon Aug 10 10:00:13 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.100 Driver Version: 440.100 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce RTX 208... Off | 00000000:01:00.0 On | N/A |
| 35% 41C P2 90W / 260W | 10515MiB / 11016MiB | 11% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1128 G /usr/lib/xorg/Xorg 102MiB |
| 0 1648 G /usr/lib/xorg/Xorg 380MiB |
| 0 1848 G /usr/bin/gnome-shell 279MiB |
| 0 10633 G ...uest-channel-token=1206236727 266MiB |
| 0 13794 G /usr/lib/firefox/firefox 6MiB |
| 0 17604 C python 9457MiB |
+-----------------------------------------------------------------------------+
All i found was a solution for tensorflow 1.0:
sess = tf.Session(config=tf.ConfigProto(
intra_op_parallelism_threads=NUM_THREADS))
I have an Intel 9900k and a RTX 2080 Ti and use Ubuntu 20.04
E: When I add the following code on top, it uses 1 core 100%
tf.config.threading.set_intra_op_parallelism_threads(1)
tf.config.threading.set_inter_op_parallelism_threads(1)
But increasing this number to 16 again only utilizes all cores ~30%
Just setting the set_intra_op_parallelism_threads and set_inter_op_parallelism_threads wasn't working for me. Incase someone else is in the same place, after a lot of struggle with the same issue, below piece of code worked for me in limiting the CPU usage of tensorflow below 500%:
import os
import tensorflow as tf
num_threads = 5
os.environ["OMP_NUM_THREADS"] = "5"
os.environ["TF_NUM_INTRAOP_THREADS"] = "5"
os.environ["TF_NUM_INTEROP_THREADS"] = "5"
tf.config.threading.set_inter_op_parallelism_threads(
num_threads
)
tf.config.threading.set_intra_op_parallelism_threads(
num_threads
)
tf.config.set_soft_device_placement(True)
There can be many issues for this, I solved it for me the following way:
Set
tf.config.threading.set_intra_op_parallelism_threads(<Your_Physical_Core_Count>) tf.config.threading.set_inter_op_parallelism_threads(<Your_Physical_Core_Count>)
both to your physical core count. You do not want Hyperthreading for highly vectorized operations as you cannot benefit from parallized operations when there aren't any gaps.
"With a high level of vectorization, the number of execution gaps is
very small and there is possibly insufficient opportunity to make up
any penalty due to increased contention in HT."
From: Saini et al, published by NASAA dvanced Supercomputing Division, 2011: The Impact of Hyper-Threading on Processor
Resource Utilization in Production Applications
EDIT: I am not sure anymore, if one of the two has to be 1. But one 100% needs to be set to Physical.

Why do I see ttyUSB2 on nmcli, not see wwan0?

I want to use NetworkManager to control wwan0. But I only see ttyUSB2 in NetworkManager device list.
I use "cat" command to make ttyUSB2 & ttyUSB3 in use, but it doesn't work.
The below is modemmanager info
root#imx6qsabresd:~# mmcli -m 0
/org/freedesktop/ModemManager1/Modem/0 (device id '74d82342f71de5945dddcc381409e231f5ed5df1')
-------------------------
Hardware | manufacturer: 'SIMCOM INCORPORATED'
| model: 'SIMCOM_SIM7600JC-H'
| revision: 'LE11B01SIM7600JC-H'
| supported: 'gsm-umts, lte'
| current: 'gsm-umts, lte'
| equipment id: '861478030131862'
-------------------------
System | device: '/sys/devices/soc0/soc/2100000.aips-bus/2184200.usb/ci_hdrc.1/usb1/1-1/1-1.1'
| drivers: 'option1, simcom_wwan'
| plugin: 'SimTech'
| primary port: 'ttyUSB2'
| ports: 'ttyUSB0 (qcdm), ttyUSB2 (at), ttyUSB3 (at), wwan0 (net)'
-------------------------
Numbers | own : 'unknown'
-------------------------
Status | lock: 'none'
| unlock retries: 'unknown'
| state: 'registered'
| power state: 'on'
| access tech: 'unknown'
| signal quality: '80' (recent)
-------------------------
Modes | supported: 'allowed: 2g; preferred: none
| allowed: 3g; preferred: none
| allowed: 2g, 3g; preferred: none
| allowed: 2g, 3g; preferred: 2g
| allowed: 2g, 3g; preferred: 3g
| allowed: 2g, 3g, 4g; preferred: none'
| current: 'allowed: any; preferred: none'
-------------------------
Bands | supported: 'unknown'
| current: 'unknown'
-------------------------
IP | supported: 'ipv4, ipv6, ipv4v6'
-------------------------
3GPP | imei: '861478030131862'
| enabled locks: 'none'
| operator id: '46692'
| operator name: 'Chunghwa Telecom'
| subscription: 'unknown'
| registration: 'home'
-------------------------
SIM | path: '/org/freedesktop/ModemManager1/SIM/0'
-------------------------
Bearers | paths: 'none'
Does anyone know how to make wwan0 appear in NetworkManager device list?
You see ttyUSB2 in NetworkManager because that is the "primary port" reported by ModemManager in the modem details.
In this specific case, ModemManager doesn't know how to use the WWAN port of the device (exposed by the non-standard simcom_wwan kernel driver), and therefore it will fallback to use PPP over a TTY port.
If you want to use the WWAN port with NM/MM you must not install the simcom_wwan kernel driver, and instead, you should just use the standard qmi_wwan kernel driver. Once you do that, you'll get a cdc-wdm port (that speaks QMI) and an associated WWAN port (in raw-ip mode), and those are nicely used by ModemManager/NetworkManager.
If you ask me, there should be no reason to use the simcom_wwan kernel driver by normal users. If your qmi_wwan driver doesn't expose the QMI ports of the modem, you may just need a newer kernel.

SQL Query Performance DISTINCT IN

I have one requirement in my project as follows :
There is a table like this :
UID VER STATUS
--------------------------
DOC001 VER.1 N/A
DOC001 VER.2 N/A
DOC001 VER.3 N/A
DOC001 VER.4 N/A
DOC002 VER.1 N/A
DOC002 VER.2 N/A
DOC002 VER.3 N/A
DOC003 VER.1 N/A
DOC003 VER.2 N/A
DOC003 VER.3 N/A
DOC003 VER.4 N/A
DOC003 VER.5 N/A
DOC003 VER.6 N/A
I need to change the status of each version after doing some validation. In this process, if VER.2 of Doc, say DOC001 has failed, I need to update the status of remaining versions i.e. VER.3 and VER.4 to FAIL irrespective of validation.
For this, I wrote SQL queries using DISTINCT, IN, ORDER BY clauses which is very slow.
The real time time DB is quite big and has millions of rows which increases the pressure on me to improve the performance.
Your suggestions and help is highly appreciated.
Thanks in advance.

Counting values from table results

I have a table of data that looks like the following:
ArtistName TrackName TrackID
1 Pendulum Slam 6
2 N/A N/A 26
3 N/A N/A 26
4 N/A N/A 26
5 Snow Patrol Chasing Cars 17
6 Snow Patrol Chasing Cars 17
7 Rihanna Love The Way You Lie 4
8 N/A N/A 26
9 N/A N/A 26
10 Kanye West Stronger 10
11 Rihanna Love The Way You Lie 4
12 N/A N/A 26
13 N/A N/A 26
14 Tinie Tempah Written In The Stars 8
15 N/A N/A 26
16 N/A N/A 26
17 Nero Crush On You 18
etc...
Basically what I'd like to do is count the number of occurrences of each TrackID, and display that in a column. The previous table is created from this query which combines a few other tables:
SELECT Artist_Details.ArtistName, Track_Details.TrackName, Sales_Records.TrackID
FROM Track_Details
INNER JOIN Sales_Records ON Track_Details.TrackID = Sales_Records.TrackID
JOIN Artist_Details ON Track_Details.ArtistID = Artist_Details.ArtistID;
The output format I'd like is:
ArtistName TrackName Track ID TotalSales
1 Pendulum Slam 6 8
2 Tinie Tempah Written In The Stars 8 5
3 Rihanna Love The Way You Lie 4 2
And finally, I'd like the value 26 to not be counted and to be ignored and not displayed in the results, with it sorted ascending by TotalSales. And if possible to limit this chart to 10 rows.
Thanks in advance, Mark
That looks like a slam dunk for group by:
SELECT top 10 Artist_Details.ArtistName, Track_Details.TrackName,
Sales_Records.TrackID, count(Sales_Records.TrackID) as TotalSales
FROM Track_Details
INNER JOIN Sales_Records ON Track_Details.TrackID = Sales_Records.TrackID
JOIN Artist_Details ON Track_Details.ArtistID = Artist_Details.ArtistID
WHERE Sales_Records.TrackID <> 26
GROUP BY Artist_Details.ArtistName, Track_Details.TrackName, Sales_Records.TrackID
ORDER BY count(Sales_Records.TrackID) desc

MySQL Performance inquiry

I have moved my drupal site from one mysql server to another one.
Old Machine has 1 cpu, 1GB Ram
New Machine has 4 cpu, 4GB Ram.
I have a huge negative difference in perfomance on this query ( 2 mins vs 2 secs )
select distinct c.client
from client_table c
LEFT JOIN reps r on ( c.client = r.client )
where r.user_id is NULL
AND c.client not in ( select distinct client from billing where first_purchase = 1 )
NEW OLD
| connect_timeout 10 |connect_timeout 5
| have_federated_engine DISABLED | have_federated_engine YES
| max_connections 100 | max_connections 400
| max_seeks_for_key 18446744073709551615 | max_seeks_for_key 4294967295
| max_write_lock_count 18446744073709551615 | max_write_lock_count 4294967295
| myisam_max_sort_file_size 9223372036853727232 | myisam_max_sort_file_size 2147483647
| max_binlog_cache_size 18446744073709547520 | max_binlog_cache_size 4294967295
| myisam_recover_options BACKUP | myisam_recover_options OFF
| range_alloc_block_size 4096 | range_alloc_block_size 2048
| table_cache 457 | table_cache 307
| version 5.0.67-0ubuntu6-log | version 5.0.51a-3ubuntu5.4-log
| version_compile_machine x86_64 | version_compile_machine i486
ONLY on NEW | relay_log |
ONLY on NEW | relay_log_index |
ONLY on NEW | relay_log_info_file | relay-log.info
ONLY on NEW innodb_adaptive_hash_index | ON
Any ideas on how to identify what is causing the problem or how to fix it?
You might need to rebuild your indexes on the new instance.
Make triple-sure you've rebuilt your indicies, they don't really carry over.
Try using the MySQL Query Profiler.
I would profile in both environments.
So how do you go about analyzing
database performance? There are three
forms of performance analysis that are
used to troubleshoot and tune database
systems:
Bottleneck analysis - focuses on
answering the questions: What is my
database server waiting on; what is a
user connection waiting on; what is a
piece of SQL code waiting on?
Workload analysis - examines the server and who
is logged on to determine the resource
usage and activity of each.
Ratio-based analysis - utilizes a
number of rule-of-thumb ratios to
gauge performance of a database, user
connection, or piece of code.