Comma is always killed by oom when I try to edit a file in which I use 10 modules (mainly Cro::HTTP ) and after the start of analysis. I can see that many raku process are running (in order to analyze?) :
journalctl :
janv. 09 19:47:42 samuel-Virtual-Machine kernel: [ 4722] 1000 4722 654 29 40960 0 0 comma.sh
janv. 09 19:47:42 samuel-Virtual-Machine kernel: [ 4771] 1000 4771 1085318 155576 1949696 0 0 java
janv. 09 19:47:42 samuel-Virtual-Machine kernel: [ 4825] 1000 4825 783 35 40960 0 0 fsnotifier64
janv. 09 19:47:42 samuel-Virtual-Machine kernel: [ 5036] 1000 5036 52039 24008 364544 0 0 rakudo
janv. 09 19:47:42 samuel-Virtual-Machine kernel: [ 5038] 1000 5038 51119 25114 372736 0 0 rakudo
janv. 09 19:47:42 samuel-Virtual-Machine kernel: [ 5039] 1000 5039 52391 23805 368640 0 0 rakudo
janv. 09 19:47:42 samuel-Virtual-Machine kernel: [ 5047] 1000 5047 51473 22787 352256 0 0 rakudo
janv. 09 19:47:42 samuel-Virtual-Machine kernel: [ 5049] 1000 5049 51129 22929 356352 0 0 rakudo
janv. 09 19:47:42 samuel-Virtual-Machine kernel: [ 5050] 1000 5050 49796 21981 348160 0 0 rakudo
janv. 09 19:47:42 samuel-Virtual-Machine kernel: [ 5052] 1000 5052 50929 25154 368640 0 0 rakudo
janv. 09 19:47:42 samuel-Virtual-Machine kernel: [ 5057] 1000 5057 52078 23535 364544 0 0 rakudo
janv. 09 19:47:42 samuel-Virtual-Machine kernel: [ 5066] 1000 5066 51071 22735 348160 0 0 rakudo
janv. 09 19:47:42 samuel-Virtual-Machine kernel: [ 5075] 1000 5075 51254 22555 356352 0 0 rakudo
janv. 09 19:47:42 samuel-Virtual-Machine kernel: [ 5081] 1000 5081 49423 21271 335872 0 0 rakudo
janv. 09 19:47:42 samuel-Virtual-Machine kernel: [ 5093] 1000 5093 49375 21590 344064 0 0 rakudo
janv. 09 19:47:42 samuel-Virtual-Machine kernel: [ 5100] 1000 5100 50784 22763 352256 0 0 rakudo
janv. 09 19:47:42 samuel-Virtual-Machine kernel: [ 5104] 1000 5104 49360 21141 335872 0 0 rakudo
janv. 09 19:47:42 samuel-Virtual-Machine kernel: [ 5115] 1000 5115 46338 14169 282624 0 0 rakudo
janv. 09 19:47:42 samuel-Virtual-Machine kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/user#1000.service,task=java,pid=4771,uid=1000
janv. 09 19:47:42 samuel-Virtual-Machine kernel: Out of memory: Killed process 4771 (java) total-vm:4341272kB, anon-rss:622236kB, file-rss:0kB, shmem-rss:68kB, UID:1000 pgtables:1904kB oom_score_adj:0
janv. 09 19:47:42 samuel-Virtual-Machine kernel: oom_reaper: reaped process 4771 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:68kB
Is this a normal behavior? Is there an option to limit the number of parallel raku process ?
(I work on a small Vm with 4GB of memory).
It's expected that Comma will invoke the selected Raku compiler in order to obtain symbols from modules. That should take place once at the start of editing a file using a particular module and then be cached (and the caching is across the project as a whole).
Aside from the number of rakudo instances spawned, the memory usage of the java process itself looks a bit on the high side. Probably it's worth asking the Comma developers to take a look at it, and providing some more detailed information. Of note, the Help menu has a "Collect Logs and Diagnostic Data" option (which will provide a zip file that can be sent to the developers, although note it may also include some data about the project you are working on). Any other information to aid reproduction (such as the list of modules being used) would also be useful.
Related
I'm using ubuntu20.04 in vmware workstation. I allocate 8 cpu cores, 6G RAM, 4G swap and enough free disk space(at least 25G) to ubuntu20.04. But after running make modules -j4 for about 30 minutes the whole system freezes. The only thing I can do is moving the cursor in virtual machine, the click doesn't get response. When I click "hang up the vm" I get error "cannot hang up vm". After trying several times I can hang it up successsfully. And after I press "continue vm" vm works again but after running about 30 min the situation just occurs again.
gcc version, kernel version are listed below:(they are both compiling linux-4.8.5 kernel source code)
user#ubuntu:~$ gcc --version
gcc (GCC) 4.8.5
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
user#ubuntu:~$ uname -r
5.15.57
user#centos7:~$ gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
user#centos7:~$ uname -r
5.15.86
syslog when the freeze situation occurs:
Message from syslogd#ubuntu at Jan 26 18:55:37 ...
kernel:[ 7665.562563] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [cc1:100337]
Jan 26 18:55:37 ubuntu kernel: [ 7665.562563] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [cc1:100337]
Jan 26 18:55:37 ubuntu kernel: [ 7665.562667] Modules linked in: btrfs ... hid_generic
Jan 26 18:55:47 ubuntu kernel: [ 7665.564688] crc32_pclmul mptspi usbhid mptscsih ahci psmouse mptbase hid e1000 pata_acpi libahci scsi_transport_spi i2c_piix4
Jan 26 18:55:47 ubuntu kernel: [ 7665.564982] CPU: 6 PID: 100337 Comm: cc1 Tainted: G
L 5.15.0-52-generic #58~20.04.1-Ubuntu
Jan 26 18:55:47 ubuntu kernel: [ 7665.565049] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
Jan 26 18:55:47 ubuntu kernel: [ 7665.565083] RIP: 0010:exit_to_user_mode_prepare+0xea/0x1c0
Jan 26 18:55:47 ubuntu kernel: [ 7665.565393] Code: e3 02 75 71 fa 66 0f 1f 44 00 00 65 48 8b 04 25 c0 fb 01 00 48 8b 18 f7 c3 0e 30 02 00 0f 84 48 ff ff ff fb 66 0f 1f 44 00 00 <f6> c3 08 74 be e8 3c 23 c0 00 f6 c7 10 74 b9 4c 89 e7 e8 ff 49 13
Jan 26 18:55:47 ubuntu kernel: [ 7665.565437] RSP: 0000:ffffc251922e7ed8 EFLAGS: 00000202
Jan 26 18:55:47 ubuntu kernel: [ 7665.565441] RAX: 0000000000000000 RBX: 0000000000000008 RCX: 00000000000006e0
Jan 26 18:55:47 ubuntu kernel: [ 7665.565443] RDX: ffffa04a4033cb00 RSI: 0000000000000000 RDI: ffffc251922e7f58
Jan 26 18:55:47 ubuntu kernel: [ 7665.565446] RBP: ffffc251922e7f08 R08: 0000000000000000 R09: 000006f3506a78c1
Jan 26 18:55:47 ubuntu kernel: [ 7665.565448] R10: 000006f3502d6fc1 R11: 0000000000512638 R12: ffffc251922e7f58
Jan 26 18:55:47 ubuntu kernel: [ 7665.565450] R13: ffffa04a460d3200 R14: ffffa04a460d3200 R15: ffffa04a460d3200
Jan 26 18:55:47 ubuntu kernel: [ 7665.565452] FS: 00007fc500f3aac0(0000) GS:ffffa04af7f80000(0000) knlGS:0000000000000000
Jan 26 18:55:47 ubuntu kernel: [ 7665.565455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 26 18:55:47 ubuntu kernel: [ 7665.565458] CR2: 00007fc50060f000 CR3: 000000010b0ae004 CR4: 00000000003706e0
Jan 26 18:55:47 ubuntu kernel: [ 7665.565486] Call Trace:
Jan 26 18:55:47 ubuntu kernel: [ 7665.565515] <TASK>
Jan 26 18:55:47 ubuntu kernel: [ 7665.565669] irqentry_exit_to_user_mode+0x9/0x20
Jan 26 18:55:47 ubuntu kernel: [ 7665.565844] irqentry_exit+0x1d/0x30
Jan 26 18:55:47 ubuntu kernel: [ 7665.565848] sysvec_apic_timer_interrupt+0x4e/0x90
Jan 26 18:55:47 ubuntu kernel: [ 7665.565852] asm_sysvec_apic_timer_interrupt+0x1a/0x20
Jan 26 18:55:47 ubuntu kernel: [ 7665.565886] RIP: 0033:0x12cfc00
Jan 26 18:55:47 ubuntu kernel: [ 7665.565972] Code: 33 ff ff ff 83 e6 01 48 89 df e8 4b 0c ff ff 85 c0 0f 84 29 ff ff ff 80 bb 9c 00 00 00 53 0f 85 19 ff ff ff e9 6a ff ff ff 90 <4c> 89 e0 5b 5d 41 5c c3 48 8d 15 c2 3a 79 00 be 1d 0a 00 00 48 8d
Jan 26 18:55:47 ubuntu kernel: [ 7665.565974] RSP: 002b:00007ffd36e8a930 EFLAGS: 00000246
Jan 26 18:55:47 ubuntu kernel: [ 7665.565977] RAX: 0000000002569580 RBX: 00000000023c0a00 RCX: 0000000000000005
Jan 26 18:55:47 ubuntu kernel: [ 7665.565979] RDX: 0000000000000000 RSI: 000000000241dcf0 RDI: 00000000023c0a00
Jan 26 18:55:47 ubuntu kernel: [ 7665.565980] RBP: 00000000023c0a98 R08: 000000000000000f R09: 00007fc50177c000
Jan 26 18:55:47 ubuntu kernel: [ 7665.565981] R10: 0000000002569580 R11: 0256958002569580 R12: 000000000241dcf0
Jan 26 18:55:47 ubuntu kernel: [ 7665.565983] R13: 0000000000000000 R14: 00007ffd36e8a978 R15: 00007fc5007b8f00
Jan 26 18:55:47 ubuntu kernel: [ 7665.565986] </TASK>
Message from syslogd#ubuntu at Jan 26 18:56:06 ...
kernel:[ 7693.578970] watchdog: BUG: soft lockup - CPU#6 stuck for 48s! [cc1:100337]
Jan 26 18:56:06 ubuntu kernel: [ 7693.578970] watchdog: BUG: soft lockup - CPU#6 stuck for 48s! [cc1:100337]
Jan 26 18:56:09 ubuntu kernel: [ 7693.578983] Modules linked in: btrfs ... hid_generic
Jan 26 18:56:09 ubuntu kernel: [ 7693.579073] crc32_pclmul mptspi usbhid mptscsih ahci psmouse mptbase hid e1000 pata_acpi libahci scsi_transport_spi i2c_piix4
Jan 26 18:56:09 ubuntu kernel: [ 7693.579088] CPU: 6 PID: 100337 Comm: cc1 Tainted: G
L 5.15.0-52-generic #58~20.04.1-Ubuntu
Jan 26 18:56:09 ubuntu kernel: [ 7693.579093] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
Jan 26 18:56:09 ubuntu kernel: [ 7693.579135] RIP: 0010:exit_to_user_mode_prepare+0xea/0x1c0
Jan 26 18:56:09 ubuntu kernel: [ 7693.579146] Code: e3 02 75 71 fa 66 0f 1f 44 00 00 65 48 8b 04 25 c0 fb 01 00 48 8b 18 f7 c3 0e 30 02 00 0f 84 48 ff ff ff fb 66 0f 1f 44 00 00 <f6> c3 08 74 be e8 3c 23 c0 00 f6 c7 10 74 b9 4c 89 e7 e8 ff 49 13
Jan 26 18:56:09 ubuntu kernel: [ 7693.579149] RSP: 0000:ffffc251922e7ed8 EFLAGS: 00000202
Jan 26 18:56:09 ubuntu kernel: [ 7693.579152] RAX: 0000000000000000 RBX: 0000000000000008 RCX: 00000000000006e0
Jan 26 18:56:09 ubuntu kernel: [ 7693.579154] RDX: ffffa04a4033cb00 RSI: 0000000000000000 RDI: ffffc251922e7f58
Jan 26 18:56:09 ubuntu kernel: [ 7693.579155] RBP: ffffc251922e7f08 R08: 0000000000000000 R09: 000006f3506a78c1
Jan 26 18:56:09 ubuntu kernel: [ 7693.579157] R10: 000006f3502d6fc1 R11: 0000000000512638 R12: ffffc251922e7f58
Jan 26 18:56:09 ubuntu kernel: [ 7693.579159] R13: ffffa04a460d3200 R14: ffffa04a460d3200 R15: ffffa04a460d3200
Jan 26 18:56:09 ubuntu kernel: [ 7693.579161] FS: 00007fc500f3aac0(0000) GS:ffffa04af7f80000(0000) knlGS:0000000000000000
Jan 26 18:56:09 ubuntu kernel: [ 7693.579163] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 26 18:56:09 ubuntu kernel: [ 7693.579165] CR2: 00007fc50060f000 CR3: 000000010b0ae004 CR4: 00000000003706e0
Jan 26 18:56:09 ubuntu kernel: [ 7693.579169] Call Trace:
Jan 26 18:56:09 ubuntu kernel: [ 7693.579172] <TASK>
(same as above)
</TASK>
Message from syslogd#ubuntu at Jan 26 18:56:40 ...
kernel:[ 7729.590239] watchdog: BUG: soft lockup - CPU#6 stuck for 81s! [cc1:100337]
Message from syslogd#ubuntu at Jan 26 18:57:09 ...
kernel:[ 7757.599000] watchdog: BUG: soft lockup - CPU#6 stuck for 107s! [cc1:100337]
Message from syslogd#ubuntu at Jan 26 18:57:36 ...
kernel:[ 7785.607641] watchdog: BUG: soft lockup - CPU#6 stuck for 133s! [cc1:100337]
Message from syslogd#ubuntu at Jan 26 18:58:27 ...
kernel:[ 7813.614515] watchdog: BUG: soft lockup - CPU#6 stuck for 159s! [cc1:100337]
Message from syslogd#ubuntu at Jan 26 18:58:32 ...
kernel:[ 7841.620739] watchdog: BUG: soft lockup - CPU#6 stuck for 186s! [cc1:100337]
Jan 26 18:56:15 ubuntu kernel: [ 7702.812787] rcu: INFO: rcu_sched self-detected stall on CPU
Jan 26 18:56:15 ubuntu kernel: [ 7702.812875] rcu: 6-....: (1 GPs behind) idle=2df/1/0x4000000000000000 softirq=645654/645656 fqs=31
Jan 26 18:56:15 ubuntu kernel: [ 7702.812919] (t=15001 jiffies g=1424273 q=363)
Jan 26 18:56:15 ubuntu kernel: [ 7702.812947] rcu: rcu_sched kthread starved for 509 jiffies! g1424273 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=5
Jan 26 18:56:15 ubuntu kernel: [ 7702.812954] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.
Jan 26 18:56:15 ubuntu kernel: [ 7702.812956] rcu: RCU grace-period kthread stack dump:
Jan 26 18:56:15 ubuntu kernel: [ 7702.812957] task:rcu_sched state:R running task stack: 0 pid: 14 ppid: 2 flags:0x00004000
Jan 26 18:56:15 ubuntu kernel: [ 7702.813009] Call Trace:
Jan 26 18:56:15 ubuntu kernel: [ 7702.813013] <TASK>
(same as above)
</TASK>
Jan 26 18:56:15 ubuntu kernel: [ 7702.813330] rcu: Stack dump where RCU GP kthread last ran:
Jan 26 18:56:15 ubuntu kernel: [ 7702.813359] Sending NMI from CPU 6 to CPUs 5:
Jan 26 18:56:15 ubuntu kernel: [ 7702.813681] NMI backtrace for cpu 5
Jan 26 18:56:15 ubuntu kernel: [ 7702.813684] CPU: 5 PID: 843 Comm: gmain Tainted: G
L 5.15.0-52-generic #58~20.04.1-Ubuntu
Jan 26 18:56:15 ubuntu kernel: [ 7702.813688] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
Jan 26 18:56:15 ubuntu kernel: [ 7702.813690] RIP: 0010:native_write_msr+0x6/0x30
Jan 26 18:56:15 ubuntu kernel: [ 7702.813761] Code: fc ff b8 40 00 00 00 0f 00 d8 44 89 e7 e8 62 15 fc ff eb cb e8 fb d6 cc 00 66 66 2e 0f 1f 84 00 00 00 00 00 89 f9 89 f0 0f 30 <66> 90 c3 cc cc cc cc 55 48 c1 e2 20 89 f6 48 09 d6 31 d2 48 89 e5
Jan 26 18:56:15 ubuntu kernel: [ 7702.813764] RSP: 0000:ffffc251805c0f00 EFLAGS: 00000007
Jan 26 18:56:15 ubuntu kernel: [ 7702.813767] RAX: 000000009437f6ed RBX: 0000000000000000 RCX: 00000000000006e0
Jan 26 18:56:15 ubuntu kernel: [ 7702.813768] RDX: 0000000000000cb4 RSI: 000000009437f6ed RDI: 00000000000006e0
Jan 26 18:56:15 ubuntu kernel: [ 7702.813770] RBP: ffffc251805c0f08 R08: 0000000000000001 R09: 000007014874c7c1
Jan 26 18:56:15 ubuntu kernel: [ 7702.813771] R10: 7fffffffffffffff R11: ffffc251805c0ff8 R12: 000027a10367387b
Jan 26 18:56:15 ubuntu kernel: [ 7702.813772] R13: ffffa04af7f5fe00 R14: ffffa04af7f62db8 R15: ffffa04af7f62df8
Jan 26 18:56:15 ubuntu kernel: [ 7702.813796] FS: 00007f02c513e700(0000) GS:ffffa04af7f40000(0000) knlGS:0000000000000000
Jan 26 18:56:15 ubuntu kernel: [ 7702.813798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 26 18:56:15 ubuntu kernel: [ 7702.813800] CR2: 00007f00d5bf2000 CR3: 0000000117928001 CR4: 00000000003706e0
Jan 26 18:56:15 ubuntu kernel: [ 7702.813803] Call Trace:
Jan 26 18:56:15 ubuntu kernel: [ 7702.813805] <IRQ>
Jan 26 18:56:15 ubuntu kernel: [ 7702.813848] ? lapic_next_deadline+0x2c/0x40
Jan 26 18:56:15 ubuntu kernel: [ 7702.813936] clockevents_program_event+0xa8/0x130
Jan 26 18:56:15 ubuntu kernel: [ 7702.813968] tick_program_event+0x41/0x90
Jan 26 18:56:15 ubuntu kernel: [ 7702.813971] ? hrtimer_update_next_event+0x4f/0x70
Jan 26 18:56:15 ubuntu kernel: [ 7702.813973] hrtimer_interrupt+0x136/0x230
Jan 26 18:56:15 ubuntu kernel: [ 7702.813976] __sysvec_apic_timer_interrupt+0x5e/0xe0
Jan 26 18:56:15 ubuntu kernel: [ 7702.813978] sysvec_apic_timer_interrupt+0x7b/0x90
Jan 26 18:56:15 ubuntu kernel: [ 7702.813982] </IRQ>
Jan 26 18:56:15 ubuntu kernel: [ 7702.813983] <TASK>
Jan 26 18:56:15 ubuntu kernel: [ 7702.813983] asm_sysvec_apic_timer_interrupt+0x1a/0x20
Jan 26 18:56:15 ubuntu kernel: [ 7702.813986] RIP: 0010:exit_to_user_mode_prepare+0xea/0x1c0
Jan 26 18:56:15 ubuntu kernel: [ 7702.813989] Code: e3 02 75 71 fa 66 0f 1f 44 00 00 65 48 8b 04 25 c0 fb 01 00 48 8b 18 f7 c3 0e 30 02 00 0f 84 48 ff ff ff fb 66 0f 1f 44 00 00 <f6> c3 08 74 be e8 3c 23 c0 00 f6 c7 10 74 b9 4c 89 e7 e8 ff 49 13
Jan 26 18:56:15 ubuntu kernel: [ 7702.813991] RSP: 0000:ffffc2518434fed8 EFLAGS: 00000202
Jan 26 18:56:15 ubuntu kernel: [ 7702.813993] RAX: 0000000000000000 RBX: 0000000000000008 RCX: 00000000000006e0
Jan 26 18:56:15 ubuntu kernel: [ 7702.813994] RDX: ffffa04a40339900 RSI: 0000000000000000 RDI: ffffc2518434ff58
Jan 26 18:56:15 ubuntu kernel: [ 7702.813995] RBP: ffffc2518434ff08 R08: 0000000000000000 R09: 0000070146c988c1
Jan 26 18:56:15 ubuntu kernel: [ 7702.813996] R10: 00000701468c7fc1 R11: 00000000000024d8 R12: ffffc2518434ff58
Jan 26 18:56:15 ubuntu kernel: [ 7702.813998] R13: ffffa04a557ce400 R14: ffffa04a557ce400 R15: ffffa04a557ce400
Jan 26 18:56:15 ubuntu kernel: [ 7702.814000] irqentry_exit_to_user_mode+0x9/0x20
Jan 26 18:56:15 ubuntu kernel: [ 7702.814004] irqentry_exit+0x1d/0x30
Jan 26 18:56:15 ubuntu kernel: [ 7702.814007] sysvec_apic_timer_interrupt+0x4e/0x90
Jan 26 18:56:15 ubuntu kernel: [ 7702.814011] asm_sysvec_apic_timer_interrupt+0x1a/0x20
Jan 26 18:56:15 ubuntu kernel: [ 7702.814013] RIP: 0033:0x7f02c5e07508
Jan 26 18:56:15 ubuntu kernel: [ 7702.814017] Code: c0 74 0f e9 4a f5 ff ff 66 2e 0f 1f 84 00 00 00 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 31 f6 87 37 <83> fe 01 74 0b e9 ce f4 ff ff 66 0f 1f 44 00 00 c3 0f 1f 80 00 00
Jan 26 18:56:15 ubuntu kernel: [ 7702.814019] RSP: 002b:00007f02c513dc98 EFLAGS: 00000246
Jan 26 18:56:15 ubuntu kernel: [ 7702.814021] RAX: 0000000000000f6a RBX: 000000007fffffff RCX: 0000000000000004
Jan 26 18:56:15 ubuntu kernel: [ 7702.814023] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 000055920b115870
Jan 26 18:56:15 ubuntu kernel: [ 7702.814024] RBP: 0000000000000002 R08: 0000000000000002 R09: 0000000000000001
Jan 26 18:56:15 ubuntu kernel: [ 7702.814025] R10: 000055920b132e00 R11: 000055920b112580 R12: 0000000000000002
Jan 26 18:56:15 ubuntu kernel: [ 7702.814026] R13: 000055920b115870 R14: 000055920b112dd0 R15: 00007f02c513dcf4
Jan 26 18:56:15 ubuntu kernel: [ 7702.814028] </TASK>
Jan 26 18:56:15 ubuntu kernel: [ 7702.814382] NMI backtrace for cpu 6
Jan 26 18:56:15 ubuntu kernel: [ 7702.814389] CPU: 6 PID: 100337 Comm: cc1 Tainted: G
L 5.15.0-52-generic #58~20.04.1-Ubuntu
Jan 26 18:56:15 ubuntu kernel: [ 7702.814393] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
Jan 26 18:56:15 ubuntu kernel: [ 7702.814394] Call Trace:
Jan 26 18:56:15 ubuntu kernel: [ 7702.814395] <IRQ>
Jan 26 18:56:15 ubuntu kernel: [ 7702.814397] dump_stack_lvl+0x4a/0x63
Jan 26 18:56:15 ubuntu kernel: [ 7702.814484] dump_stack+0x10/0x16
Jan 26 18:56:15 ubuntu kernel: [ 7702.814487] nmi_cpu_backtrace.cold+0x4d/0x96
Jan 26 18:56:15 ubuntu kernel: [ 7702.814491] ? lapic_can_unplug_cpu+0x90/0x90
Jan 26 18:56:15 ubuntu kernel: [ 7702.814495] nmi_trigger_cpumask_backtrace+0xf3/0x100
Jan 26 18:56:15 ubuntu kernel: [ 7702.814594] arch_trigger_cpumask_backtrace+0x19/0x20
Jan 26 18:56:15 ubuntu kernel: [ 7702.814598] rcu_dump_cpu_stacks+0x132/0x16d
Jan 26 18:56:15 ubuntu kernel: [ 7702.814762] rcu_sched_clock_irq.cold+0x100/0x36a
Jan 26 18:56:15 ubuntu kernel: [ 7702.814787] ? account_system_index_time+0xa0/0xc0
Jan 26 18:56:15 ubuntu kernel: [ 7702.814831] update_process_times+0x94/0xd0
Jan 26 18:56:15 ubuntu kernel: [ 7702.814861] tick_sched_handle.isra.0+0x25/0x60
Jan 26 18:56:15 ubuntu kernel: [ 7702.814865] tick_sched_timer+0x71/0x90
Jan 26 18:56:15 ubuntu kernel: [ 7702.814868] ? tick_sched_do_timer+0x50/0x50
Jan 26 18:56:15 ubuntu kernel: [ 7702.814871] __hrtimer_run_queues+0xc8/0x1d0
Jan 26 18:56:15 ubuntu kernel: [ 7702.814873] hrtimer_interrupt+0x109/0x230
Jan 26 18:56:15 ubuntu kernel: [ 7702.814876] __sysvec_apic_timer_interrupt+0x5e/0xe0
Jan 26 18:56:15 ubuntu kernel: [ 7702.814878] sysvec_apic_timer_interrupt+0x7b/0x90
Jan 26 18:56:15 ubuntu kernel: [ 7702.814883] </IRQ>
Jan 26 18:56:15 ubuntu kernel: [ 7702.814883] <TASK>
Jan 26 18:56:15 ubuntu kernel: [ 7702.814884] asm_sysvec_apic_timer_interrupt+0x1a/0x20
Jan 26 18:56:15 ubuntu kernel: [ 7702.814887] RIP: 0010:exit_to_user_mode_prepare+0xea/0x1c0
Jan 26 18:56:15 ubuntu kernel: [ 7702.814890] Code: e3 02 75 71 fa 66 0f 1f 44 00 00 65 48 8b 04 25 c0 fb 01 00 48 8b 18 f7 c3 0e 30 02 00 0f 84 48 ff ff ff fb 66 0f 1f 44 00 00 <f6> c3 08 74 be e8 3c 23 c0 00 f6 c7 10 74 b9 4c 89 e7 e8 ff 49 13
Jan 26 18:56:15 ubuntu kernel: [ 7702.814893] RSP: 0000:ffffc251922e7ed8 EFLAGS: 00000202
Jan 26 18:56:15 ubuntu kernel: [ 7702.814896] RAX: 0000000000000000 RBX: 0000000000000008 RCX: 00000000000006e0
Jan 26 18:56:15 ubuntu kernel: [ 7702.814897] RDX: ffffa04a4033cb00 RSI: 0000000000000000 RDI: ffffc251922e7f58
Jan 26 18:56:15 ubuntu kernel: [ 7702.814899] RBP: ffffc251922e7f08 R08: 0000000000000000 R09: 000006f3506a78c1
Jan 26 18:56:15 ubuntu kernel: [ 7702.814927] R10: 000006f3502d6fc1 R11: 0000000000512638 R12: ffffc251922e7f58
Jan 26 18:56:15 ubuntu kernel: [ 7702.814929] R13: ffffa04a460d3200 R14: ffffa04a460d3200 R15: ffffa04a460d3200
Jan 26 18:56:15 ubuntu kernel: [ 7702.814932] irqentry_exit_to_user_mode+0x9/0x20
Jan 26 18:56:15 ubuntu kernel: [ 7702.814937] irqentry_exit+0x1d/0x30
Jan 26 18:56:15 ubuntu kernel: [ 7702.814940] sysvec_apic_timer_interrupt+0x4e/0x90
Jan 26 18:56:15 ubuntu kernel: [ 7702.814943] asm_sysvec_apic_timer_interrupt+0x1a/0x20
Jan 26 18:56:15 ubuntu kernel: [ 7702.814945] RIP: 0033:0x12cfc00
Jan 26 18:56:15 ubuntu kernel: [ 7702.814949] Code: 33 ff ff ff 83 e6 01 48 89 df e8 4b 0c ff ff 85 c0 0f 84 29 ff ff ff 80 bb 9c 00 00 00 53 0f 85 19 ff ff ff e9 6a ff ff ff 90 <4c> 89 e0 5b 5d 41 5c c3 48 8d 15 c2 3a 79 00 be 1d 0a 00 00 48 8d
Jan 26 18:56:15 ubuntu kernel: [ 7702.814951] RSP: 002b:00007ffd36e8a930 EFLAGS: 00000246
Jan 26 18:56:15 ubuntu kernel: [ 7702.814953] RAX: 0000000002569580 RBX: 00000000023c0a00 RCX: 0000000000000005
Jan 26 18:56:15 ubuntu kernel: [ 7702.814978] RDX: 0000000000000000 RSI: 000000000241dcf0 RDI: 00000000023c0a00
Jan 26 18:56:15 ubuntu kernel: [ 7702.814980] RBP: 00000000023c0a98 R08: 000000000000000f R09: 00007fc50177c000
Jan 26 18:56:15 ubuntu kernel: [ 7702.814981] R10: 0000000002569580 R11: 0256958002569580 R12: 000000000241dcf0
Jan 26 18:56:15 ubuntu kernel: [ 7702.814983] R13: 0000000000000000 R14: 00007ffd36e8a978 R15: 00007fc5007b8f00
Jan 26 18:56:15 ubuntu kernel: [ 7702.814985] </TASK>
Jan 26 18:56:40 ubuntu kernel: [ 7729.590239] watchdog: BUG: soft lockup - CPU#6 stuck for 81s! [cc1:100337]
Jan 26 18:56:40 ubuntu kernel: [ 7729.590249] Modules linked in: btrfs ... hid_generic
Jan 26 18:56:40 ubuntu kernel: [ 7729.590369] crc32_pclmul mptspi usbhid mptscsih ahci psmouse mptbase hid e1000 pata_acpi libahci scsi_transport_spi i2c_piix4
Jan 26 18:56:40 ubuntu kernel: [ 7729.590381] CPU: 6 PID: 100337 Comm: cc1 Tainted: G
L 5.15.0-52-generic #58~20.04.1-Ubuntu
Jan 26 18:56:40 ubuntu kernel: [ 7729.590385] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
Jan 26 18:56:40 ubuntu kernel: [ 7729.590387] RIP: 0010:exit_to_user_mode_prepare+0xea/0x1c0
Jan 26 18:56:40 ubuntu kernel: [ 7729.590394] Code: e3 02 75 71 fa 66 0f 1f 44 00 00 65 48 8b 04 25 c0 fb 01 00 48 8b 18 f7 c3 0e 30 02 00 0f 84 48 ff ff ff fb 66 0f 1f 44 00 00 <f6> c3 08 74 be e8 3c 23 c0 00 f6 c7 10 74 b9 4c 89 e7 e8 ff 49 13
Jan 26 18:56:40 ubuntu kernel: [ 7729.590397] RSP: 0000:ffffc251922e7ed8 EFLAGS: 00000202
Jan 26 18:56:40 ubuntu kernel: [ 7729.590400] RAX: 0000000000000000 RBX: 0000000000000008 RCX: 00000000000006e0
Jan 26 18:56:40 ubuntu kernel: [ 7729.590402] RDX: ffffa04a4033cb00 RSI: 0000000000000000 RDI: ffffc251922e7f58
Jan 26 18:56:40 ubuntu kernel: [ 7729.590403] RBP: ffffc251922e7f08 R08: 0000000000000000 R09: 000006f3506a78c1
Jan 26 18:56:40 ubuntu kernel: [ 7729.590405] R10: 000006f3502d6fc1 R11: 0000000000512638 R12: ffffc251922e7f58
Jan 26 18:56:40 ubuntu kernel: [ 7729.590407] R13: ffffa04a460d3200 R14: ffffa04a460d3200 R15: ffffa04a460d3200
Jan 26 18:56:40 ubuntu kernel: [ 7729.590409] FS: 00007fc500f3aac0(0000) GS:ffffa04af7f80000(0000) knlGS:0000000000000000
Jan 26 18:56:40 ubuntu kernel: [ 7729.590411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 26 18:56:40 ubuntu kernel: [ 7729.590412] CR2: 00007fc50060f000 CR3: 000000010b0ae004 CR4: 00000000003706e0
Jan 26 18:56:40 ubuntu kernel: [ 7729.590451] Call Trace:
Jan 26 18:56:40 ubuntu kernel: [ 7729.590453] <TASK>
(same as above)
</TASK>
Jan 26 18:57:09 ubuntu kernel: [ 7757.599000] watchdog: BUG: soft lockup - CPU#6 stuck for 107s! [cc1:100337]
Jan 26 18:57:09 ubuntu kernel: [ 7757.599009] Modules linked in: btrfs ... hid_generic
Jan 26 18:57:09 ubuntu kernel: [ 7757.599076] crc32_pclmul mptspi usbhid mptscsih ahci psmouse mptbase hid e1000 pata_acpi libahci scsi_transport_spi i2c_piix4
Jan 26 18:57:09 ubuntu kernel: [ 7757.599086] CPU: 6 PID: 100337 Comm: cc1 Tainted: G
L 5.15.0-52-generic #58~20.04.1-Ubuntu
Jan 26 18:57:09 ubuntu kernel: [ 7757.599089] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
Jan 26 18:57:09 ubuntu kernel: [ 7757.599091] RIP: 0010:exit_to_user_mode_prepare+0xea/0x1c0
Jan 26 18:57:09 ubuntu kernel: [ 7757.599097] Code: e3 02 75 71 fa 66 0f 1f 44 00 00 65 48 8b 04 25 c0 fb 01 00 48 8b 18 f7 c3 0e 30 02 00 0f 84 48 ff ff ff fb 66 0f 1f 44 00 00 <f6> c3 08 74 be e8 3c 23 c0 00 f6 c7 10 74 b9 4c 89 e7 e8 ff 49 13
Jan 26 18:57:09 ubuntu kernel: [ 7757.599099] RSP: 0000:ffffc251922e7ed8 EFLAGS: 00000202
Jan 26 18:57:09 ubuntu kernel: [ 7757.599101] RAX: 0000000000000000 RBX: 0000000000000008 RCX: 00000000000006e0
Jan 26 18:57:09 ubuntu kernel: [ 7757.599103] RDX: ffffa04a4033cb00 RSI: 0000000000000000 RDI: ffffc251922e7f58
Jan 26 18:57:09 ubuntu kernel: [ 7757.599104] RBP: ffffc251922e7f08 R08: 0000000000000000 R09: 000006f3506a78c1
Jan 26 18:57:09 ubuntu kernel: [ 7757.599105] R10: 000006f3502d6fc1 R11: 0000000000512638 R12: ffffc251922e7f58
Jan 26 18:57:09 ubuntu kernel: [ 7757.599106] R13: ffffa04a460d3200 R14: ffffa04a460d3200 R15: ffffa04a460d3200
Jan 26 18:57:09 ubuntu kernel: [ 7757.599108] FS: 00007fc500f3aac0(0000) GS:ffffa04af7f80000(0000) knlGS:0000000000000000
Jan 26 18:57:09 ubuntu kernel: [ 7757.599110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 26 18:57:09 ubuntu kernel: [ 7757.599111] CR2: 00007fc50060f000 CR3: 000000010b0ae004 CR4: 00000000003706e0
Jan 26 18:57:09 ubuntu kernel: [ 7757.599114] Call Trace:
Jan 26 18:57:16 ubuntu kernel: [ 7757.599116] <TASK>
(same as above)
</TASK>
Jan 26 18:57:36 ubuntu kernel: [ 7785.607641] watchdog: BUG: soft lockup - CPU#6 stuck for 133s! [cc1:100337]
Jan 26 18:57:36 ubuntu kernel: [ 7785.607651] Modules linked in: btrfs ... hid_generic
Jan 26 18:57:36 ubuntu kernel: [ 7785.607750] crc32_pclmul mptspi usbhid mptscsih ahci psmouse mptbase hid e1000 pata_acpi libahci scsi_transport_spi i2c_piix4
Jan 26 18:57:36 ubuntu kernel: [ 7785.607760] CPU: 6 PID: 100337 Comm: cc1 Tainted: G
L 5.15.0-52-generic #58~20.04.1-Ubuntu
Jan 26 18:57:36 ubuntu kernel: [ 7785.607764] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
Jan 26 18:57:36 ubuntu kernel: [ 7785.607765] RIP: 0010:exit_to_user_mode_prepare+0xea/0x1c0
Jan 26 18:57:36 ubuntu kernel: [ 7785.607772] Code: e3 02 75 71 fa 66 0f 1f 44 00 00 65 48 8b 04 25 c0 fb 01 00 48 8b 18 f7 c3 0e 30 02 00 0f 84 48 ff ff ff fb 66 0f 1f 44 00 00 <f6> c3 08 74 be e8 3c 23 c0 00 f6 c7 10 74 b9 4c 89 e7 e8 ff 49 13
Jan 26 18:57:36 ubuntu kernel: [ 7785.607774] RSP: 0000:ffffc251922e7ed8 EFLAGS: 00000202
Jan 26 18:57:36 ubuntu kernel: [ 7785.607776] RAX: 0000000000000000 RBX: 0000000000000008 RCX: 00000000000006e0
Jan 26 18:57:36 ubuntu kernel: [ 7785.607777] RDX: ffffa04a4033cb00 RSI: 0000000000000000 RDI: ffffc251922e7f58
Jan 26 18:57:36 ubuntu kernel: [ 7785.607779] RBP: ffffc251922e7f08 R08: 0000000000000000 R09: 000006f3506a78c1
Jan 26 18:57:36 ubuntu kernel: [ 7785.607780] R10: 000006f3502d6fc1 R11: 0000000000512638 R12: ffffc251922e7f58
Jan 26 18:57:36 ubuntu kernel: [ 7785.607781] R13: ffffa04a460d3200 R14: ffffa04a460d3200 R15: ffffa04a460d3200
Jan 26 18:57:36 ubuntu kernel: [ 7785.607782] FS: 00007fc500f3aac0(0000) GS:ffffa04af7f80000(0000) knlGS:0000000000000000
Jan 26 18:57:36 ubuntu kernel: [ 7785.607784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 26 18:57:36 ubuntu kernel: [ 7785.607785] CR2: 00007fc50060f000 CR3: 000000010b0ae004 CR4: 00000000003706e0
Jan 26 18:57:59 ubuntu kernel: [ 7785.607788] Call Trace:
Jan 26 18:58:27 ubuntu kernel: [ 7785.607790] <TASK>
(same as above)
</TASK>
Jan 26 18:58:27 ubuntu kernel: [ 7813.614515] watchdog: BUG: soft lockup - CPU#6 stuck for 159s! [cc1:100337]
Jan 26 18:58:27 ubuntu kernel: [ 7813.614524] Modules linked in: btrfs ... hid_generic
Jan 26 18:58:27 ubuntu kernel: [ 7813.614588] crc32_pclmul mptspi usbhid mptscsih ahci psmouse mptbase hid e1000 pata_acpi libahci scsi_transport_spi i2c_piix4
Jan 26 18:58:27 ubuntu kernel: [ 7813.614635] CPU: 6 PID: 100337 Comm: cc1 Tainted: G
L 5.15.0-52-generic #58~20.04.1-Ubuntu
Jan 26 18:58:27 ubuntu kernel: [ 7813.614638] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
Jan 26 18:58:27 ubuntu kernel: [ 7813.614640] RIP: 0010:exit_to_user_mode_prepare+0xea/0x1c0
Jan 26 18:58:27 ubuntu kernel: [ 7813.614646] Code: e3 02 75 71 fa 66 0f 1f 44 00 00 65 48 8b 04 25 c0 fb 01 00 48 8b 18 f7 c3 0e 30 02 00 0f 84 48 ff ff ff fb 66 0f 1f 44 00 00 <f6> c3 08 74 be e8 3c 23 c0 00 f6 c7 10 74 b9 4c 89 e7 e8 ff 49 13
Jan 26 18:58:27 ubuntu kernel: [ 7813.614648] RSP: 0000:ffffc251922e7ed8 EFLAGS: 00000202
Jan 26 18:58:27 ubuntu kernel: [ 7813.614650] RAX: 0000000000000000 RBX: 0000000000000008 RCX: 00000000000006e0
Jan 26 18:58:27 ubuntu kernel: [ 7813.614652] RDX: ffffa04a4033cb00 RSI: 0000000000000000 RDI: ffffc251922e7f58
Jan 26 18:58:27 ubuntu kernel: [ 7813.614653] RBP: ffffc251922e7f08 R08: 0000000000000000 R09: 000006f3506a78c1
Jan 26 18:58:27 ubuntu kernel: [ 7813.614654] R10: 000006f3502d6fc1 R11: 0000000000512638 R12: ffffc251922e7f58
Jan 26 18:58:27 ubuntu kernel: [ 7813.614656] R13: ffffa04a460d3200 R14: ffffa04a460d3200 R15: ffffa04a460d3200
Jan 26 18:58:27 ubuntu kernel: [ 7813.614657] FS: 00007fc500f3aac0(0000) GS:ffffa04af7f80000(0000) knlGS:0000000000000000
Jan 26 18:58:27 ubuntu kernel: [ 7813.614659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 26 18:58:27 ubuntu kernel: [ 7813.614660] CR2: 00007fc50060f000 CR3: 000000010b0ae004 CR4: 00000000003706e0
Jan 26 18:58:27 ubuntu kernel: [ 7813.614686] Call Trace:
(same as above)
</TASK>
Jan 26 18:58:32 ubuntu kernel: [ 7841.620739] watchdog: BUG: soft lockup - CPU#6 stuck for 186s! [cc1:100337]
Jan 26 18:58:32 ubuntu kernel: [ 7841.620748] Modules linked in: btrfs ... hid_generic
Jan 26 18:58:32 ubuntu kernel: [ 7841.620818] crc32_pclmul mptspi usbhid mptscsih ahci psmouse mptbase hid e1000 pata_acpi libahci scsi_transport_spi i2c_piix4
Jan 26 18:58:32 ubuntu kernel: [ 7841.620830] CPU: 6 PID: 100337 Comm: cc1 Tainted: G
L 5.15.0-52-generic #58~20.04.1-Ubuntu
Jan 26 18:58:32 ubuntu kernel: [ 7841.620833] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
Jan 26 18:58:32 ubuntu kernel: [ 7841.620835] RIP: 0010:exit_to_user_mode_prepare+0xea/0x1c0
Jan 26 18:58:32 ubuntu kernel: [ 7841.620842] Code: e3 02 75 71 fa 66 0f 1f 44 00 00 65 48 8b 04 25 c0 fb 01 00 48 8b 18 f7 c3 0e 30 02 00 0f 84 48 ff ff ff fb 66 0f 1f 44 00 00 <f6> c3 08 74 be e8 3c 23 c0 00 f6 c7 10 74 b9 4c 89 e7 e8 ff 49 13
Jan 26 18:58:32 ubuntu kernel: [ 7841.620844] RSP: 0000:ffffc251922e7ed8 EFLAGS: 00000202
Jan 26 18:58:32 ubuntu kernel: [ 7841.620847] RAX: 0000000000000000 RBX: 0000000000000008 RCX: 00000000000006e0
Jan 26 18:58:32 ubuntu kernel: [ 7841.620848] RDX: ffffa04a4033cb00 RSI: 0000000000000000 RDI: ffffc251922e7f58
Jan 26 18:58:32 ubuntu kernel: [ 7841.620886] RBP: ffffc251922e7f08 R08: 0000000000000000 R09: 000006f3506a78c1
Jan 26 18:58:32 ubuntu kernel: [ 7841.620888] R10: 000006f3502d6fc1 R11: 0000000000512638 R12: ffffc251922e7f58
Jan 26 18:58:32 ubuntu kernel: [ 7841.620890] R13: ffffa04a460d3200 R14: ffffa04a460d3200 R15: ffffa04a460d3200
Jan 26 18:58:32 ubuntu kernel: [ 7841.620891] FS: 00007fc500f3aac0(0000) GS:ffffa04af7f80000(0000) knlGS:0000000000000000
Jan 26 18:58:32 ubuntu kernel: [ 7841.620893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 26 18:58:32 ubuntu kernel: [ 7841.620895] CR2: 00007fc50060f000 CR3: 000000010b0ae004 CR4: 00000000003706e0
Jan 26 18:58:32 ubuntu kernel: [ 7841.620898] Call Trace:
Jan 26 18:58:32 ubuntu kernel: [ 7841.620900] <TASK>
(same as above)
</TASK>
Jan 26 19:05:42 ubuntu NetworkManager[768]: <info> [1674788384.3036] manager: NetworkManager state is now CONNECTED_SITE
I encountered freezing once, by adding RAM I solved the problem. But this time I add RAM to 8G, freezing still occurs. What also confuses me is that my centos vm with 3G RAM and 8 cpu cores run make modules -j8 successfully. I have no idea about the difference.
To check the cpu and memory usage I execute top at the same time. the picture is top's output when freezing occurs. top command output
I'm new to Redis and would like to play with the RedisTimeSeries module. Following the instructions I load the module using:
redis-server --loadmodule /path/to/module/redistimeseries.so
Obviously adjusting the path to match mine. The server starts and shows the following output:
88537:M 02 Apr 2022 12:19:24.785 # Server initialized
88537:M 02 Apr 2022 12:19:24.789 * <timeseries> RedisTimeSeries version 10609, git_sha=f36e5a703dc9a2487880087a34f6cb0e56d9a459
88537:M 02 Apr 2022 12:19:24.789 * <timeseries> Redis version found by RedisTimeSeries : 6.2.6 - oss
88537:M 02 Apr 2022 12:19:24.789 * <timeseries> loaded default CHUNK_SIZE_BYTES policy: 4096
88537:M 02 Apr 2022 12:19:24.789 * <timeseries> loaded server DUPLICATE_POLICY: block
88537:M 02 Apr 2022 12:19:24.789 * <timeseries> Setting default series ENCODING to: compressed
88537:M 02 Apr 2022 12:19:24.789 * <timeseries> Detected redis oss
88537:M 02 Apr 2022 12:19:24.789 * <timeseries> Enabled diskless replication
88537:M 02 Apr 2022 12:19:24.789 * Module 'timeseries' loaded from lib/redistimeseries.so
88537:M 02 Apr 2022 12:19:24.790 * Loading RDB produced by version 6.2.6
88537:M 02 Apr 2022 12:19:24.790 * RDB age 4 seconds
88537:M 02 Apr 2022 12:19:24.790 * RDB memory usage when created 1.02 Mb
88537:M 02 Apr 2022 12:19:24.790 # Done loading RDB, keys loaded: 0, keys expired: 0.
88537:M 02 Apr 2022 12:19:24.790 * DB loaded from disk: 0.000 seconds
88537:M 02 Apr 2022 12:19:24.790 * Ready to accept connections
When I connect using the redis-cli I run the sample command like so:
TS.CREATE temperature RETENTION 60 LABELS sensor_id 2 area_id 32
However, I receive the following error in the CLI:
(error) ERR unknown command `TS.CREATE`, with args beginning with: `temperature`, `RETENTION`, `60`, `LABELS`, `sensor_id`, `2`, `area_id`, `32`,
No time series commands are available. Can anyone help? I'm stumped.
As i know global variables(in userspace programs) are stored in coressponding DATA (and BSS) segment. But how and where are they stored in case of kernel code?
For an example let's look at one of Linux's global variable. Basically, Linux is an ELF image compressed with gzip or other compression tools. As stated on Wikipedia (https://en.wikipedia.org/wiki/Vmlinux):
Traditionally, when creating a bootable kernel image, the kernel is also compressed using gzip, or, since Linux 2.6.30,[3] using LZMA or bzip2, which requires a very small decompression stub to be included in the resulting image. The stub decompresses the kernel code, on some systems printing dots to the console to indicate progress, and then continues the boot process. Support for LZO,[4] xz[5] and LZ4[6] compression was added later.
On Ubuntu, there is a special script called extract-vmlinux which comes with your version of the Linux kernel (https://blog.packagecloud.io/eng/2016/03/08/how-to-extract-and-disassmble-a-linux-kernel-image-vmlinuz/). The Linux ELF file is /boot/vmlinuz. To decompress it, you must use the extract-vmlinux script which can be installed with
sudo apt-get install linux-headers-$(uname -r)
The script is safe to use because it is a standard script which is found in the official repository of packages for your distribution (most packages there are open source and safe). Now use
sudo /usr/src/kernels/$(uname -r)/scripts/extract-vmlinux /boot/vmlinuz-$(uname -r) > vmlinux
to extract the Linux kernel's ELF file from the compressed vmlinux image to your home repository. Now we can look at the ELF file of the Linux kernel. Use readelf -a vmlinux to have a summary of the content. The output for me is this:
user#user-System-Product-Name:~$ readelf -a vmlinux
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: EXEC (Executable file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0x1000000
Start of program headers: 64 (bytes into file)
Start of section headers: 46137728 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 5
Size of section headers: 64 (bytes)
Number of section headers: 35
Section header string table index: 34
Section Headers:
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
[ 0] NULL 0000000000000000 00000000
0000000000000000 0000000000000000 0 0 0
[ 1] .text PROGBITS ffffffff81000000 00200000
0000000000e025f7 0000000000000000 AX 0 0 4096
[ 2] .rodata PROGBITS ffffffff82000000 01200000
00000000004e46f2 0000000000000000 WA 0 0 4096
[ 3] .pci_fixup PROGBITS ffffffff824e4700 016e4700
00000000000032b0 0000000000000000 A 0 0 16
[ 4] .tracedata PROGBITS ffffffff824e79b0 016e79b0
0000000000000078 0000000000000000 A 0 0 1
[ 5] __ksymtab PROGBITS ffffffff824e7a28 016e7a28
0000000000010a34 0000000000000000 A 0 0 4
[ 6] __ksymtab_gpl PROGBITS ffffffff824f845c 016f845c
00000000000121ec 0000000000000000 A 0 0 4
[ 7] __kcrctab PROGBITS ffffffff8250a648 0170a648
00000000000058bc 0000000000000000 A 0 0 4
[ 8] __kcrctab_gpl PROGBITS ffffffff8250ff04 0170ff04
00000000000060a4 0000000000000000 A 0 0 4
[ 9] __ksymtab_strings PROGBITS ffffffff82515fa8 01715fa8
0000000000037bea 0000000000000001 AMS 0 0 1
[10] __init_rodata PROGBITS ffffffff8254dba0 0174dba0
00000000000002a8 0000000000000000 A 0 0 32
[11] __param PROGBITS ffffffff8254de48 0174de48
00000000000038e0 0000000000000000 A 0 0 8
[12] __modver PROGBITS ffffffff82551728 01751728
00000000000005a0 0000000000000000 WA 0 0 8
[13] __ex_table PROGBITS ffffffff82551cd0 01751cd0
0000000000001af4 0000000000000000 A 0 0 4
[14] .notes NOTE ffffffff825537c4 017537c4
00000000000001ec 0000000000000000 A 0 0 4
[15] .data PROGBITS ffffffff82600000 01800000
00000000003657c0 0000000000000000 WA 0 0 8192
[16] __bug_table PROGBITS ffffffff829657c0 01b657c0
00000000000178d4 0000000000000000 WA 0 0 1
[17] .vvar PROGBITS ffffffff8297e000 01b7e000
0000000000001000 0000000000000000 WA 0 0 16
[18] .data..percpu PROGBITS 0000000000000000 01c00000
000000000002f000 0000000000000000 WA 0 0 4096
[19] .init.text PROGBITS ffffffff829ae000 01dae000
00000000000772e5 0000000000000000 AX 0 0 16
[20] .altinstr_aux PROGBITS ffffffff82a252e5 01e252e5
0000000000002d9b 0000000000000000 AX 0 0 1
[21] .init.data PROGBITS ffffffff82a2a000 01e2a000
00000000001c74f0 0000000000000000 WA 0 0 8192
[22] .x86_cpu_dev.init PROGBITS ffffffff82bf14f0 01ff14f0
0000000000000028 0000000000000000 A 0 0 8
[23] .parainstructions PROGBITS ffffffff82bf1518 01ff1518
0000000000022b7c 0000000000000000 A 0 0 8
[24] .altinstructions PROGBITS ffffffff82c14098 02014098
0000000000006660 0000000000000000 A 0 0 1
[25] .altinstr_replace PROGBITS ffffffff82c1a6f8 0201a6f8
0000000000001789 0000000000000000 AX 0 0 1
[26] .iommu_table PROGBITS ffffffff82c1be88 0201be88
00000000000000f0 0000000000000000 A 0 0 8
[27] .apicdrivers PROGBITS ffffffff82c1bf78 0201bf78
0000000000000040 0000000000000000 WA 0 0 8
[28] .exit.text PROGBITS ffffffff82c1bfb8 0201bfb8
0000000000001f4b 0000000000000000 AX 0 0 1
[29] .smp_locks PROGBITS ffffffff82c1e000 0201e000
000000000000a000 0000000000000000 A 0 0 4
[30] .data_nosave PROGBITS ffffffff82c28000 02028000
0000000000001000 0000000000000000 WA 0 0 4
[31] .bss NOBITS ffffffff82c29000 02029000
00000000005d7000 0000000000000000 WA 0 0 4096
[32] .brk NOBITS ffffffff83200000 02029000
000000000002c000 0000000000000000 WA 0 0 1
[33] .init.scratch PROGBITS ffffffff83400000 02800000
0000000000400000 0000000000000000 WA 0 0 32
[34] .shstrtab STRTAB 0000000000000000 02c00000
0000000000000180 0000000000000000 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
L (link order), O (extra OS processing required), G (group), T (TLS),
C (compressed), x (unknown), o (OS specific), E (exclude),
l (large), p (processor specific)
There are no section groups in this file.
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
LOAD 0x0000000000200000 0xffffffff81000000 0x0000000001000000
0x00000000015539b0 0x00000000015539b0 R E 0x200000
LOAD 0x0000000001800000 0xffffffff82600000 0x0000000002600000
0x000000000037f000 0x000000000037f000 RW 0x200000
LOAD 0x0000000001c00000 0x0000000000000000 0x000000000297f000
0x000000000002f000 0x000000000002f000 RW 0x200000
LOAD 0x0000000001dae000 0xffffffff829ae000 0x00000000029ae000
0x0000000000e52000 0x0000000000e52000 RWE 0x200000
NOTE 0x00000000017537c4 0xffffffff825537c4 0x00000000025537c4
0x00000000000001ec 0x00000000000001ec 0x4
Section to Segment mapping:
Segment Sections...
00 .text .rodata .pci_fixup .tracedata __ksymtab __ksymtab_gpl __kcrctab __kcrctab_gpl __ksymtab_strings __init_rodata __param __modver __ex_table .notes
01 .data __bug_table .vvar
02 .data..percpu
03 .init.text .altinstr_aux .init.data .x86_cpu_dev.init .parainstructions .altinstructions .altinstr_replacement .iommu_table .apicdrivers .exit.text .smp_locks .data_nosave .bss .brk .init.scratch
04 .notes
There is no dynamic section in this file.
There are no relocations in this file.
The decoding of unwind sections for machine type Advanced Micro Devices X86-64 is not currently supported.
No version information found in this file.
Displaying notes found in: .notes
Owner Data size Description
Xen 0x00000006 Unknown note type: (0x00000006)
description data: 6c 69 6e 75 78 00
Xen 0x00000004 Unknown note type: (0x00000007)
description data: 32 2e 36 00
Xen 0x00000008 Unknown note type: (0x00000005)
description data: 78 65 6e 2d 33 2e 30 00
Xen 0x00000008 Unknown note type: (0x00000003)
description data: 00 00 00 80 ff ff ff ff
Xen 0x00000008 Unknown note type: (0x0000000f)
description data: 00 00 00 00 80 00 00 00
Xen 0x00000008 NT_VERSION (version)
description data: c0 e1 9a 82 ff ff ff ff
Xen 0x00000008 NT_ARCH (architecture)
description data: 00 20 00 81 ff ff ff ff
Xen 0x00000029 Unknown note type: (0x0000000a)
description data: 21 77 72 69 74 61 62 6c 65 5f 70 61 67 65 5f 74 61 62 6c 65 73 7c 70 61 65 5f 70 67 64 69 72 5f 61 62 6f 76 65 5f 34 67 62
Xen 0x00000004 Unknown note type: (0x00000011)
description data: 01 88 00 00
Xen 0x00000004 Unknown note type: (0x00000009)
description data: 79 65 73 00
Xen 0x00000008 Unknown note type: (0x00000008)
description data: 67 65 6e 65 72 69 63 00
Xen 0x00000010 Unknown note type: (0x0000000d)
description data: 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00
Xen 0x00000004 Unknown note type: (0x0000000e)
description data: 01 00 00 00
Xen 0x00000004 Unknown note type: (0x00000010)
description data: 01 00 00 00
Xen 0x00000008 Unknown note type: (0x0000000c)
description data: 00 00 00 00 00 80 ff ff
Xen 0x00000008 Unknown note type: (0x00000004)
description data: 00 00 00 00 00 00 00 00
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: c0263a3075bc0a9388365ddf35ab5422da3356a9
Linux 0x00000001 OPEN
description data: 00
Xen 0x00000008 Unknown note type: (0x00000012)
description data: 40 07 00 01 00 00 00 00
The important section for the question is the Program headers section:
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
LOAD 0x0000000000200000 0xffffffff81000000 0x0000000001000000
0x00000000015539b0 0x00000000015539b0 R E 0x200000
LOAD 0x0000000001800000 0xffffffff82600000 0x0000000002600000
0x000000000037f000 0x000000000037f000 RW 0x200000
LOAD 0x0000000001c00000 0x0000000000000000 0x000000000297f000
0x000000000002f000 0x000000000002f000 RW 0x200000
LOAD 0x0000000001dae000 0xffffffff829ae000 0x00000000029ae000
0x0000000000e52000 0x0000000000e52000 RWE 0x200000
NOTE 0x00000000017537c4 0xffffffff825537c4 0x00000000025537c4
0x00000000000001ec 0x00000000000001ec 0x4
I would guess that the RW sections are the data/bss segments of the static and freestanding kernel image. There isn't much bss data (uninitialized data) in this kernel image. You can see that by looking at the filesize versus the memsize of the different segments. In bss data, only the size is written in the executable. The loader will then assign zeroes to these segments. Since we have only memsizes which are equal to the filesizes, then we don't have much bss here.
Also, we can look at the different virtual addresses at which the kernel will load itself after decompression. This will be useful to find the segment in which a certain global variable resides.
Now, to determine where a global variable is loaded one can look at the System.map file in the /boot directory. For example, we can take a random global variable in kernel/sched/core.c called sysctl_sched_rt_period and find its address using
user#user-System-Product-Name:~$ sudo grep "sysctl_sched_rt_period" /boot/System.map-$(uname -r)
ffffffff8266eb24 D sysctl_sched_rt_period
The address is thus ffffffff8266eb24. This address resides in the RW segment loaded at ffffffff82600000 because the size of this segment is 37f000 and because ffffffff82600000 + 37f000 > ffffffff8266eb24. You can clearly see that this global variable is stored in the data segment of the kernel's executable.
Related post: How to select lines between two patterns?
#fedorqui, thanks for providing all these different options for awk. I have been using this to parse through var log messages when troubleshooting ooms and it works great. I want to extend this further but I have not been able to figure out how to proceed. What I am trying to do:
Print the lines between rss and out of memory. I have done that with the example
Order the sections between each match by rss field. I have not been able to figure this out
Add an extra column with its own header and perform some mathematical operation. I have been able to do this somewhat but I am running into some formatting issues. I am not sure how to skip the first and last line when adding the column so I lose those lines. I am also not able to keep the spacing from the original if I do any operations other than print.
Here's the command I am using right now:
less /var/log/messages'|awk '/swapents/{x=1; print "=================="};/Out of memory/{x=0} x'|sed 's/[]\[]//g'
Here's the source Data:
Sep 8 11:35:15 ip-10-23-15-70 kernel: 11810061.617265 pid uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Sep 8 11:35:15 ip-10-23-15-70 kernel: 11810061.622250 1828 0 1828 4331 116 14 3 0 -1000 udevd
Sep 8 11:35:15 ip-10-23-15-70 kernel: 11810061.627310 2664 0 2664 28002 53 23 3 0 -1000 auditd
Sep 8 11:35:15 ip-10-23-15-70 kernel: 11810061.633181 2680 0 2680 62032 1181 24 4 0 0 rsyslogd
Sep 8 11:35:15 ip-10-23-15-70 kernel: 11810061.638888 2694 0 2694 3444 61 11 3 0 0 irqbalance
Sep 8 11:35:15 ip-10-23-15-70 kernel: 11810061.644912 2710 81 2710 5430 56 14 3 0 0 dbus-daemon
Sep 8 11:35:15 ip-10-23-15-70 kernel: 11810061.651108 2779 0 2779 19958 203 42 3 0 -1000 sshd
Sep 8 11:35:15 ip-10-23-15-70 kernel: 11810061.656670 2789 0 2789 5622 56 17 3 0 0 xinetd
Sep 8 11:35:15 ip-10-23-15-70 kernel: 11810061.653452 Out of memory: Kill process 43390 (mysql) score 1000 or sacrifice child
blah
blah
blah
Sep 8 11:35:15 ip-10-23-15-70 kernel: 11810061.617265 pid uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Sep 8 11:35:15 ip-10-23-15-70 kernel: 11810061.622250 1828 0 1828 4331 116 14 3 0 -1000 udevd
Sep 8 11:35:15 ip-10-23-15-70 kernel: 11810061.627310 2664 0 2664 28002 53 23 3 0 -1000 auditd
Sep 8 11:35:15 ip-10-23-15-70 kernel: 11810061.633181 2680 0 2680 62032 1181 24 4 0 0 rsyslogd
Sep 8 11:35:15 ip-10-23-15-70 kernel: 11810061.638888 2694 0 2694 3444 61 11 3 0 0 irqbalance
Sep 8 11:35:15 ip-10-23-15-70 kernel: 11810061.644912 2710 81 2710 5430 56 14 3 0 0 dbus-daemon
Sep 8 11:35:15 ip-10-23-15-70 kernel: 11810061.651108 2779 0 2779 19958 203 42 3 0 -1000 sshd
Sep 8 11:35:15 ip-10-23-15-70 kernel: 11810061.656670 2789 0 2789 5622 56 17 3 0 0 xinetd
Sep 8 11:35:15 ip-10-23-15-70 kernel: 11810061.653452 Out of memory: Kill process 43390 (mysql) score 1000 or sacrifice child
Here's what my output looks like:
================== 0MB
Sep 8 11:35:15 pid 0MB name <---- should be header (Pid virt rss etc)
Sep 8 11:35:15 1828 0MB udevd
Sep 8 11:35:15 2664 0MB auditd
Sep 8 11:35:15 2680 4MB rsyslogd
Sep 8 11:35:15 2694 0MB irqbalance
Sep 8 11:35:15 2710 0MB dbus-daemon
Sep 8 11:35:15 2779 0MB sshd
Sep 8 11:35:15 2789 0MB xinetd
Sep 8 11:35:15 2822 0MB crond
Sep 8 11:35:15 Out 0MB or <---- should be footer (out of memory etc)
================== 0MB
Sep 8 11:35:15 pid 0MB name <---- should be header (Pid virt rss etc)
Sep 8 11:35:15 1828 0MB udevd
Sep 8 11:35:15 2664 0MB auditd
Sep 8 11:35:15 2680 4MB rsyslogd
Sep 8 11:35:15 2694 0MB irqbalance
Sep 8 11:35:15 2710 0MB dbus-daemon
Sep 8 11:35:15 2779 0MB sshd
Sep 8 11:35:15 2789 0MB xinetd
Sep 8 11:35:15 2822 0MB crond
Sep 8 11:35:15 Out 0MB or <---- should be footer (out of memory etc)
================== 0MB
You can see the from the output that the separator I added for each oom field, awk tries to calculate values for it, I would love to avoid this if possible. Also the header and footer are getting chopped off and it would be nice to avoid that too.
Here's what I would like:
========================
Sep 8 11:35:15 pid rss memused_MB oom_score_adj name
Sep 8 11:35:15 2664 53 {rss*4/1024} -1000 auditd
Sep 8 11:35:15 2789 56 {rss*4/1024} 0 xinetd
Sep 8 11:35:15 2710 56 {rss*4/1024} 0 dbus-dae
Sep 8 11:35:15 2694 61 {rss*4/1024} 0 irqbalan
Sep 8 11:35:15 1828 116 {rss*4/1024} -1000 udevd
Sep 8 11:35:15 2680 181 {rss*4/1024} 0 rsyslogd
Sep 8 11:35:15 2779 203 {rss*4/1024} -1000 sshd
Sep 8 11:35:15 Out of memory: Kill process 43390 (mysql) score 1000 or sacrifice child
========================
Sep 8 11:35:15 pid rss memused_MB oom_score_adj name
Sep 8 11:35:15 2664 53 {rss*4/1024} -1000 auditd
Sep 8 11:35:15 2789 56 {rss*4/1024} 0 xinetd
Sep 8 11:35:15 2710 56 {rss*4/1024} 0 dbus-dae
Sep 8 11:35:15 2694 61 {rss*4/1024} 0 irqbalan
Sep 8 11:35:15 1828 116 {rss*4/1024} -1000 udevd
Sep 8 11:35:15 2680 181 {rss*4/1024} 0 rsyslogd
Sep 8 11:35:15 2779 203 {rss*4/1024} -1000 sshd
Sep 8 11:35:15 Out of memory: Kill process 43390 (mysql) score 1000 or sacrifice child
========================
awk solution:
$ cat tst.awk
/swapents/ {
x=1;
print "=================="
printf( "%s %s %s pid\t%4s\tmemused_MB\toom_score_adj\tname\n", $1, $2, $3, "rss");
next
}
/Out of memory/ {
printf( "%s %s %s %s\n", $1, $2, $3, substr($0,index($0,$7)));
x=0
}
x {
printf( "%s %s %s %s\t%4d\t%10.5f\t%13d\t%s\n", $1, $2, $3, $7, $11, ($11*4)/1024, $15, $16 )
}
You can play around with the formatting, like the precision in column 6, using the specifiers in the printf function. Call this with:
$ awk -f tst.awk /var/log/messages
EDIT: with sorting
OP asked to sort the output by rss columns. Using a standard sort wouldn't work here because you want to sort in between starting and ending matches. You can solve this by saving the intermediate result in an array and sort that with a self-defined function. Like this:
$ cat tst2.awk
/swapents/ {
x=1;
print "=================="
printf( "%s %s %s pid\t%4s\tmemused_MB\toom_score_adj\tname\n", $1, $2, $3, "rss");
next
}
/Out of memory/ {
n=asort(a, sorted, "cmp_rss")
for (i=1; i<=n; i++) {
print sorted[i]
}
delete a;
printf( "%s %s %s %s\n", $1, $2, $3, substr($0,index($0,$7)));
x=0
}
x {
a[i++] = sprintf( "%s %s %s %s\t%4d\t%10.5f\t%13d\t%s", $1, $2, $3, $7, $11, ($11*4)/1024, $15, $16 );
}
function cmp_rss(i1, v1, i2, v2)
{
split(v1, a1, " ")
split(v2, a2, " ")
rss1=a1[5];
rss2=a2[5];
return (rss1 - rss2)
}
which leads to:
$ awk -f tst2.awk input.txt
==================
Sep 8 11:35:15 pid rss memused_MB oom_score_adj name
Sep 8 11:35:15 2664 53 0.20703 -1000 auditd
Sep 8 11:35:15 2710 56 0.21875 0 dbus-daemon
Sep 8 11:35:15 2789 56 0.21875 0 xinetd
Sep 8 11:35:15 2694 61 0.23828 0 irqbalance
Sep 8 11:35:15 1828 116 0.45312 -1000 udevd
Sep 8 11:35:15 2779 203 0.79297 -1000 sshd
Sep 8 11:35:15 2680 1181 4.61328 0 rsyslogd
Sep 8 11:35:15 Out of memory: Kill process 43390 (mysql) score 1000 or sacrifice child
==================
Sep 8 11:35:15 pid rss memused_MB oom_score_adj name
Sep 8 11:35:15 2664 53 0.20703 -1000 auditd
Sep 8 11:35:15 2710 56 0.21875 0 dbus-daemon
Sep 8 11:35:15 2789 56 0.21875 0 xinetd
Sep 8 11:35:15 2694 61 0.23828 0 irqbalance
Sep 8 11:35:15 1828 116 0.45312 -1000 udevd
Sep 8 11:35:15 2779 203 0.79297 -1000 sshd
Sep 8 11:35:15 2680 1181 4.61328 0 rsyslogd
Sep 8 11:35:15 Out of memory: Kill process 43390 (mysql) score 1000 or sacrifice child
Based on Marc Lambrichs response I was able to create this one liner that does the job. Thanks so much. The only thing missing now is sorting by the rss column, I haven't been able to get the rss field sorted though
less /var/log/messages|awk '/swapents/ {x=1; print "==================";gsub(/\[|\]/, "") ;printf "%s %s %s %s %10s %10s %10s %15s memory_used %-s\n", $1,$2,$3,$4,$7,$10,$11,$15,$16 ;next } {gsub(/\[|\]/, "")} /Out of memory/ {print $0 ;x=0 } x {printf "%s %s %s %s %10s %10s %10s %15s %9.2fMB %-s\n", $1,$2,$3,$4,$7,$10,$11,$15,$11*4/1024,$16}'
For readability, the awk code formatted:
/swapents/ {
x=1;
print "==================";
gsub(/\[|\]/, "");
printf "%s %s %s %s %10s %10s %10s %15s memory_used %-s\n", $1,$2,$3,$4,$7,$10,$11,$15,$16 ;
next
}
{
gsub(/\[|\]/, "")
}
/Out of memory/ {
print $0;
x=0
}
x {
printf "%s %s %s %s %10s %10s %10s %15s %9.2fMB %-s\n", $1,$2,$3,$4,$7,$10,$11,$15,$11*4/1024,$16
}
I'm new to supporting WAS (Websphere Application Server), currently I'm having issue with my WAS, my WAS was installed under AIX under 2 servers/nodes.
While investigating it, I found in our application log that there are some activity which is "Performing Cache Maintenance":
===================================
2017-01-14 01:31:52,619: [Cache Maintenance] com.ibm.srm.util.db.ServerCache refreshed 2017-01-14 01:31:53,314: [Cache Maintenance] Memory: available=[6884mb] used=[9500mb] %used avail=[58%] max=[16384mb] %used max=[58%] total=[16384mb] free=[6884mb] used by doMaintenance=[-251,201,3 92bytes] Time=[22,818ms] 2017-01-14 01:51:53,325: -------- Performing Cache Maintenance -------- 2017-01-14 01:51:53,325: null : QN=319 Select * from perform.cache_timestamps where row_class_name not like '%Cache' and row_class_name not like '%(SRM 6.0)' 2017-01-14 01:51:53,333: Returning 19 data records, QN=319,2 columns, Time: 8ms conn/query time: 5ms
2017-01-14 01:51:53,333: [Cache Maintenance] Memory: available=[5492mb] used=[10892mb] %used avail=[66%] max=[16384mb] %used max=[66%] total=[16384mb] free=[5492mb] used by doMaintenance=[532kb] Time=[8ms]
===================================
After this activity triggered, I found that mpmstats value for 'bsy' are keep increasing until reach MaxClient maximum value which is '4000':
===================================
[Sat Jan 14 01:38:58 2017] [notice] mpmstats: rdy 166 bsy 234 rd 0 wr 234 ka 0 log 0 dns 0 cls 0 [Sat Jan 14 01:38:58 2017] [notice] mpmstats: bsy: 234 in mod_was_ap22_http.c [Sat Jan 14 01:48:58 2017] [notice] mpmstats: rdy 195 bsy 505 rd 0 wr 505 ka 0 log 0 dns 0 cls 0 [Sat Jan 14 01:48:58 2017] [notice] mpmstats: bsy: 505 in mod_was_ap22_http.c [Sat Jan 14 01:58:58 2017] [notice] mpmstats: rdy 180 bsy 720 rd 0 wr 720 ka 0 log 0 dns 0 cls 0 [Sat Jan 14 01:58:58 2017] [notice] mpmstats: bsy: 720 in mod_was_ap22_http.c [Sat Jan 14 02:08:59 2017] [notice] mpmstats: rdy 105 bsy 895 rd 1 wr 894 ka 0 log 0 dns 0 cls 0 [Sat Jan 14 02:08:59 2017] [notice] mpmstats: bsy: 894 in mod_was_ap22_http.c [Sat Jan 14 02:18:59 2017] [notice] mpmstats: rdy 112 bsy 1088 rd 1 wr 1087 ka 0 log 0 dns 0 cls 0 [Sat Jan 14 02:18:59 2017] [notice] mpmstats: bsy: 1087 in mod_was_ap22_http.c
[Sat Jan 14 02:28:59 2017] [notice] mpmstats: rdy 158 bsy 1242 rd 1 wr 1241 ka 0 log 0 dns 0 cls 0
----
[Sat Jan 14 04:55:34 2017] [notice] mpmstats: rdy 0 bsy 4000 rd 0 wr 4000 ka 0 log 0 dns 0 cls 0 [Sat Jan 14 04:55:34 2017] [notice] mpmstats: bsy: 4000 in mod_was_ap22_http.c [Sat Jan 14 04:57:04 2017] [notice] mpmstats: reached MaxClients (4000/4000) [Sat Jan 14 04:57:04 2017] [notice] mpmstats: rdy 0 bsy 4000 rd 0 wr 4000 ka 0 log 0 dns 0 cls 0 [Sat Jan 14 04:57:04 2017] [notice] mpmstats: bsy: 4000 in mod_was_ap22_http.c [Sat Jan 14 04:58:34 2017] [notice] mpmstats: reached MaxClients (4000/4000) [Sat Jan 14 04:58:34 2017] [notice] mpmstats: rdy 0 bsy 4000 rd 0 wr 4000 ka 0 log 0 dns 0 cls 0
[Sat Jan 14 04:58:34 2017] [notice] mpmstats: bsy: 4000 in mod_was_ap22_http.c
===================================
It seem WAS are not processing the client request until it reached the Maximum value.
The questions are:
Is there any log that I can check about why WAS are not processing the Client request until it reached to the max value?
Does the "Cache Maintenance" activity hold WAS from processing the Client request? Because as mentioned from our developer this activity should not lead to this issue.
What is the procedure that I can take to identify/resolve this issue?
Appreciate if can help me for this thing as this issue already occurred for a long time but still not resolve.