Pig Processing log file using - apache-pig

I have following logs : Can any one tell me how can I processes it using PigLatin ?
**
SYSTEM IP:192.168.68.78
Distro info:Red Hat Enterprise Linux Server release 6.6 (Santiago)
Kernel:Linux bugzilla-blr-in 2.6.32-504.16.2.el6.x86_64 #1 SMP Tue Mar 10 17:01:00 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
Uptime:12:27:42 up 8 days, 17:57, 0 users, load average: 0.00, 0.00, 0.00
Memory:Total:1869Mb Memory:Used:1567Mb Memory:Free:302Mb
Swap:Total:1999Mb Swap:Used:0Mb Swap:Free: 1999Mb
Architecture:x86_64
Processor:0:Intel(R) Xeon(R) CPU E5-2640 0 # 2.50GHz
Date:Wed Jun 29 12:27:42 IST 2016
SCRIPT USER
User:aimsadm (uid:503)
Groups:aimsadm
Working dir:/home/aimsadm
Home dir:/home/aimsadm
NETWORK DETAILS
Hostname:bugzilla-blr-in
IP ( ):127.0.0.1/8
IP (eth0):192.168.68.78/24
Gateway:192.168.68.1
Name Server:8.8.8.8
Name Server:192.168.68.80
LIST OF USERS:sdudam,sudutha,djegathesa,aimsadm,krishnang,
CLAMD STATUS: CLAM AV service is stopped or not installed
NAGIOS STATUS: Nagios service is running
OSSEC STATUS: Ossec service is stopped or not installed
NTPD STATUS: NTP service is running
HARDENING STATUS:Hardening Done
AD INTEGRATION STATUS:AD Integration Not Done
HARDWARE/PLATFORM DETAILS
Hardware Platform:64Bit
Hardware Info :DMI 2.3 present.
DMI: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090006 05/23/2012
OS DETAILS
Red Hat Enterprise Linux Server release 6.6 (Santiago)
Linux bugzilla-blr-in 2.6.32-504.16.2.el6.x86_64 #1 SMP Tue Mar 10 17:01:00 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
CPU INFO
model name : Intel(R) Xeon(R) CPU E5-2640 0 # 2.50GHz
MEMORY INFO
MemTotal: 1914776 kB
RAM:1 GB
HARD DISK DETAILS
MOUNT DETAILS
Filesystem:/dev/mapper/vg_bugzillablrin-LogVol00,Type:ext4,Total Size:22G,Used:2.4G,Avail:19G,Use%:12%,Mounted on:/
Filesystem:tmpfs,Type:tmpfs,Total Size:981M,Used:0,Avail:981M,Use%:0%,Mounted on:/dev/shm
Filesystem:/dev/sda1,Type:ext4,Total Size:297M,Used:95M,Avail:186M,Use%:34%,Mounted on:/boot
Filesystem:/dev/mapper/vg_bugzillablrin-LogVol01,Type:ext4,Total Size:21G,Used:5.8G,Avail:14G,Use%:30%,Mounted on:/var
LSBLK OUTPUT
NAME:sr0,
MAJ:MIN:11:0,RM:1,SIZE:1024M,RO:0,TYPE:rom,MOUNTPOINT::
NAME:sda,
MAJ:MIN:8:0,RM:0,SIZE:60G,RO:0,TYPE:disk,MOUNTPOINT::
NAME:sda1,
MAJ:MIN:8:1,RM:0,SIZE:300M,RO:0,TYPE:part,MOUNTPOINT::/boot
NAME:sda2,
MAJ:MIN:8:2,RM:0,SIZE:59.7G,RO:0,TYPE:part,MOUNTPOINT::
RUNNING SERVICES
auditd running...
crond running...
messagebus running...
nrpe running...
ntpd running...
rhnsd running...
rhsmcertd running...
rpcbind running...
openssh-daemon running...
**
SYSTEM IP:192.168.68.35
Distro info:CentOS release 6.6 (Final)
Kernel:Linux altifin-ci-app 2.6.32-504.16.2.el6.x86_64 #1 SMP Wed Apr 22 06:48:29 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Uptime:12:28:06 up 48 days, 20:31, 0 users, load average: 0.00, 0.00, 0.00
Memory:Total:11903Mb Memory:Used:1277Mb Memory:Free:10625Mb
Swap:Total:8191Mb Swap:Used:0Mb Swap:Free: 8191Mb
Architecture:x86_64
Processor:0:Intel(R) Xeon(R) CPU E5-2650 v2 # 2.60GHz
Processor:1:Intel(R) Xeon(R) CPU E5-2650 v2 # 2.60GHz
Date:Wed Jun 29 12:28:06 IST 2016
SCRIPT USER
User:aimsadm (uid:509)
Groups:aimsadm
Working dir:/home/aimsadm
Home dir:/home/aimsadm
NETWORK DETAILS
Hostname:altifin-ci-app
IP (lo):127.0.0.1/8
IP (eth0):192.168.68.35/24
Gateway:192.168.68.1
Name Server:192.168.68.10
Name Server:192.168.68.4
LIST OF USERS:altipay,aramesh,sdudam,nagios,kpankaj,sudutha,miyappan,skosanam,djegathesa,aimsadm,
CLAMD STATUS: CLAM AV service is stopped or not installed
NAGIOS STATUS: Nagios service is running
OSSEC STATUS: Ossec service is stopped or not installed
NTPD STATUS: NTP service is running
HARDENING STATUS:Hardening Done
AD INTEGRATION STATUS:AD Integration Not Done
HARDWARE/PLATFORM DETAILS
Hardware Platform:64Bit
Hardware Info :DMI 2.3 present.
DMI: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090006 05/23/2012
OS DETAILS
CentOS release 6.6 (Final)
Linux altifin-ci-app 2.6.32-504.16.2.el6.x86_64 #1 SMP Wed Apr 22 06:48:29 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
CPU INFO
model name : Intel(R) Xeon(R) CPU E5-2650 v2 # 2.60GHz
model name : Intel(R) Xeon(R) CPU E5-2650 v2 # 2.60GHz
MEMORY INFO
MemTotal: 12189032 kB
RAM:11 GB
HARD DISK DETAILS
MOUNT DETAILS
Filesystem:/dev/mapper/vg_altifinci-LogVol01,Type:ext4,Total Size:203G,Used:80G,Avail:113G,Use%:42%,Mounted on:/
Filesystem:tmpfs,Type:tmpfs,Total Size:6.3G,Used:0,Avail:6.3G,Use%:0%,Mounted on:/dev/shm
Filesystem:/dev/sda1,Type:ext4,Total Size:500M,Used:64M,Avail:410M,Use%:14%,Mounted on:/boot
LSBLK OUTPUT
NAME:sr0,
MAJ:MIN:11:0,RM:1,SIZE:1024M,RO:0,TYPE:rom,MOUNTPOINT::
NAME:sda,
MAJ:MIN:8:0,RM:0,SIZE:200G,RO:0,TYPE:disk,MOUNTPOINT::
NAME:sda1,
MAJ:MIN:8:1,RM:0,SIZE:500M,RO:0,TYPE:part,MOUNTPOINT::/boot
NAME:sda2,
MAJ:MIN:8:2,RM:0,SIZE:199.5G,RO:0,TYPE:part,MOUNTPOINT::
RUNNING SERVICES
abrtd running...
abrt-dump-oops running...
acpid running...
atd running...
auditd running...
automount running...
crond running...
cupsd running...
hald running...
mcelog running...
messagebus running...
MySQL but
rpc.statd running...
nrpe running...
ntpd running...
rpcbind running...
openssh-daemon running...


Yes.
There is way. Let me explain this. Though the given sample data falls into the category of 'unstructured', we always look for 'some thing' in it.
Having said that we look for a pattern, say line or lines having the required data you are looking into!
To achieve this we need to identify the 'pattern' from the sample data and use appropriate 'RegEx' (regular expression) to pull it.
Also, Pig comes with built-in jar 'piggybank' to support various pre-defined file formats including unstructured one like you said.Try with 'RegExLoader' class that is part of the below package from PIG's piggybank!!!
(Package org.apache.pig.piggybank.storage)
https://pig.apache.org/docs/r0.15.0/api/
Also, let all know the exact output you are looking into.

Related

Do Azure Container instance running on Linux or Windows host?

My question may sounds weird but i have a scenario which may be explained if you could confirm. Basically i experienced issue with Linux ACI that i do not have when i run my container either locally (on Windows 10 with wsl2) or on Linux VM.
So i try to detect the host OS using command uname -a
Step 1: On my Linux container running on my Windows10 with wsl2, i get:
Linux 1a67c6f4eb2d 5.10.16.3-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Step 2: On my Linux container running on my ubuntu VM i get:
Linux b06ba4a9f68b 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Step 3: on my Linux container running on Azure Container instance i get:
Linux SandboxHost-638071330812554054 5.10.102.2-microsoft-standard #1 SMP Mon Mar 7 17:36:34 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
When it reads microsoft-standard, does it mean that ACI is running on Windows Operating system ? That may explain my issue that i do not get on step 2 and step 1.
Do you know if it is possible to instantiate an ACI on Linux host instead ?
regards

Created both Windows and Linux based containers and got the host OS as windows only.
**Windows based container: **
Linux based container:
Based on above repro we can say that the host OS should be windows for all the container instances.
So, we can conclude that
When it reads Microsoft-standard, does it mean that ACI is running on Windows Operating system?
– Yes, the host OS should be Windows.
Do you know if it is possible to instantiate an ACI on Linux host instead?
– As per my knowledge, not supported as of today.

hashcat benchmark not starting at all

I'm trying to start hashcat on Win10 using my GPU AMD RX580.
What I did so far is:
Uninstalled all AMD drivers and rebooted
Started Driver Fusion cleaned all AMD Display drivers and rebooted
Manually removed OpenCL.dll from C:\windows\system32\ and c:\windows\syswow64 and rebooted
Installed AMD Crimsson driver edition - tried 17.8.2, 17.11.1, 17.11.4
My problem:
When I try to start a benchmark for WPA2 hash with PS D:\crack\hashcat-5.1.0> .\hashcat64.exe -m 2500 -b
It starts and just quits without any output:
PS D:\crack\hashcat-5.1.0> .\hashcat64.exe -m 2500 -b
hashcat (v5.1.0) starting in benchmark mode...
Benchmarking uses hand-optimized kernel code by default.
You can use it in your cracking session by setting the -O option.
Note: Using optimized kernel code limits the maximum supported password length.
To disable the optimized kernel code in benchmark mode, use the -w option.
* Device #2: Not a native Intel OpenCL runtime. Expect massive speed loss.
You can use --force to override, but do not report related errors.
OpenCL Platform #1: Advanced Micro Devices, Inc.
================================================
* Device #1: Ellesmere, 3264/4096 MB allocatable, 36MCU
* Device #2: Intel(R) Core(TM) i7-4790 CPU # 3.60GHz, skipped.
Benchmark relevant options:
===========================
* --optimized-kernel-enable
Hashmode: 2500 - WPA-EAPOL-PBKDF2 (Iterations: 4096)
PS D:\crack\hashcat-5.1.0>
Output from hashcat -I:
PS D:\crack\hashcat-5.1.0> .\hashcat64.exe -I
hashcat (v5.1.0) starting...
OpenCL Info:
Platform ID #1
Vendor : Advanced Micro Devices, Inc.
Name : AMD Accelerated Parallel Processing
Version : OpenCL 2.0 AMD-APP (2442.8)
Device ID #1
Type : GPU
Vendor ID : 1
Vendor : Advanced Micro Devices, Inc.
Name : Ellesmere
Version : OpenCL 2.0 AMD-APP (2442.8)
Processor(s) : 36
Clock : 1366
Memory : 3264/4096 MB allocatable
OpenCL Version : OpenCL C 2.0
Driver Version : 2442.8
Device ID #2
Type : CPU
Vendor ID : 128
Vendor : GenuineIntel
Name : Intel(R) Core(TM) i7-4790 CPU # 3.60GHz
Version : OpenCL 1.2 AMD-APP (2442.8)
Processor(s) : 8
Clock : 3599
Memory : 6131/24526 MB allocatable
OpenCL Version : OpenCL C 1.2
Driver Version : 2442.8 (sse2,avx)
PS D:\crack\hashcat-5.1.0>
The same happens if I try to start mask attack.
Other benchmarks seems to work but when it reaches the WPA2 it just returns me to the command prompt:
PS D:\crack\hashcat-5.1.0> .\hashcat64.exe -b
hashcat (v5.1.0) starting in benchmark mode...
Benchmarking uses hand-optimized kernel code by default.
You can use it in your cracking session by setting the -O option.
Note: Using optimized kernel code limits the maximum supported password length.
To disable the optimized kernel code in benchmark mode, use the -w option.
* Device #2: Not a native Intel OpenCL runtime. Expect massive speed loss.
You can use --force to override, but do not report related errors.
OpenCL Platform #1: Advanced Micro Devices, Inc.
================================================
* Device #1: Ellesmere, 3264/4096 MB allocatable, 36MCU
* Device #2: Intel(R) Core(TM) i7-4790 CPU # 3.60GHz, skipped.
Benchmark relevant options:
===========================
* --optimized-kernel-enable
Hashmode: 0 - MD5
Speed.#1.........: 12381.1 MH/s (96.87ms) # Accel:256 Loops:512 Thr:256 Vec:1
Hashmode: 100 - SHA1
Speed.#1.........: 4268.9 MH/s (70.02ms) # Accel:256 Loops:128 Thr:256 Vec:1
Hashmode: 1400 - SHA2-256
Speed.#1.........: 1870.1 MH/s (80.00ms) # Accel:256 Loops:64 Thr:256 Vec:1
Hashmode: 1700 - SHA2-512
Speed.#1.........: 461.1 MH/s (81.22ms) # Accel:128 Loops:32 Thr:256 Vec:1
Hashmode: 2500 - WPA-EAPOL-PBKDF2 (Iterations: 4096)
PS D:\crack\hashcat-5.1.0>
Any idea what could cause this behavior?

It looks like it was a crash in the OpenCL runtime. I have installed the newest Adrenalin Drivers - 19.3.2 , but went into another issue :
PS D:\crack\hashcat-5.1.0> .\hashcat64.exe -b -m 2500
hashcat (v5.1.0) starting in benchmark mode...
Benchmarking uses hand-optimized kernel code by default.
You can use it in your cracking session by setting the -O option.
Note: Using optimized kernel code limits the maximum supported password length.
To disable the optimized kernel code in benchmark mode, use the -w option.
OpenCL Platform #1: Advanced Micro Devices, Inc.
================================================
* Device #1: Ellesmere, 3264/4096 MB allocatable, 36MCU
Benchmark relevant options:
===========================
* --optimized-kernel-enable
Hashmode: 2500 - WPA-EAPOL-PBKDF2 (Iterations: 4096)
* Device #1: ATTENTION! OpenCL kernel self-test failed.
Your device driver installation is probably broken.
See also: https://hashcat.net/faq/wrongdriver
Speed.#1.........: 179.3 kH/s (87.14ms) # Accel:128 Loops:64 Thr:256 Vec:1
Started: Mon Mar 18 19:05:55 2019
Stopped: Mon Mar 18 19:06:26 2019
PS D:\crack\hashcat-5.1.0>
Then I used the --self-test-disable option and it started . I have also created a post in the hashcat forum : https://hashcat.net/forum/thread-8226-post-44141.html#pid44141 in case anyone is interested .

running gem5 with SPEC2006

when running GEM5 X86 in SE mode, I am trying to run bzip2 from SPEC2006, at first it was failing because it says it can't run dynamic execution so I compiled it with -static flag.
now I get this error:
gem5 Simulator System. http://gem5.org
gem5 is copyrighted software; use the --copyright option for details.
gem5 compiled Oct 27 2018 00:36:02
gem5 started Dec 22 2018 18:16:40
gem5 executing on Dan
command line: ./build/X86/gem5.opt configs/example/se.py -c /home/dan/SPEC2006/benchspec/CPU2006/401.bzip2/exe/bzip2_base.ia64-gcc42 -i /home/dan/SPEC2006/benchspec/CPU2006/401.bzip2/data/test/input/dryer.jpg
Could not import 03_BASE_FLAT
Could not import 03_BASE_NARROW
Global frequency set at 1000000000000 ticks per second
warn: DRAM device capacity (8192 Mbytes) does not match the address range assigned (4096 Mbytes)
0: system.remote_gdb.listener: listening for remote gdb #0 on port 7000
**** REAL SIMULATION ****
info: Entering event queue # 0. Starting simulation...
panic: Tried to write unmapped address 0xffffedd8. Inst is at 0x400da4
# tick 5500
[invoke:build/X86/arch/x86/faults.cc, line 160]
Memory Usage: 4316736 KBytes
Program aborted at tick 5500
Aborted (core dumped)
I am running gem5 on ubuntu 17.10.
I tried to find solutions in google but I didn't see any one referring to this problem, does anyone know how to fix the problem?

Please check your host machine configuration. Bzip2 does not work in a 32-bit machine. My desktop is dual core have 32-bit X86 architecture, I tried to run bzip2 it had shown same error.

Apache MINA - stuck on SSL connection

I am having troubles with Apache MINA core library.
When I deploy my application to a remote server some of the requests are not processed (around 2%). It looks like there might be a problem with SSL.
Log tail: http://pastebin.com/48bwWsjs
When request is not being processed, it is always stuck on the:
org.apache.mina.filter.ssl.SslFilter - Session Server[40](ssl...): Processing the SSL Data
Did something similar happened to any of you?
I tried Apache-mina 2.0.7 and 2.0.16
Env:
bash-4.2$ java -version
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
bash-4.2$ uname -a
Linux 8d9ad913fa03 4.4.39-34.54.amzn1.x86_64 #1 SMP Fri Dec 30 19:11:28 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Thanks any help!

Can't compile 64bit redis-server

I'm trying to compile the latest stable (2.8.19) version of Redis. Build is successfull as well as all tests, but unexpectedly server runs on 32bit arch.
Log entries:
# Warning: 32 bit instance detected but no memory limit set. Setting 3 GB maxmemory limit with 'noeviction' policy now.
Redis 2.8.19 (00000000/0) 32 bit
Running in stand alone mode
Port: 6582
PID: 2381
Redis-cli INFO display arch_bits:32. Previous instance (version 2.4.6) works well on arch_bits 64, but I don't know which way it was installed.
OS version info:
root:~# uname -a
Linux localhost 2.6.32-5-amd64 #1 SMP Tue Mar 8 22:49:26 UTC 2011 x86_64 GNU/Linux
root:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 6.0.1 (squeeze)
Release: 6.0.1
Codename: squeeze
root:~# arch
x86_64
What are the ways to fix this issue and run latest redis as 64bit?
UPD
Despite above commands output, dpkg --print-architecture returns i386 and all packages in system are all or i386. Only redis-server 2.4.*, installed as a package, is strangely ia64.
What can I do in this situation? The server was setup long time ago by another person, and I is still too newbie in Unix.

It seems, my server needs a full migration from 32 to 64-bit architeture.
Current task solved by downloading compiled 64-bit DEB-package and installing it manually.