Check if you can help me.
We have an old BeeGFS install running version 7.1.5 on EL7 and one of the TargetIDs gone offline (without replacing). After it came back buddy mirror entered in a failed state that we can’t recover.
If we try to change the Target back to online it fails:
[root#headnode beegfs]# beegfs-ctl --nodetype=storage --setstate --state=good --force --targetid=13
Node did not accept state change. Error: Unknown storage target
The state shows as this:
root#headnode ~]# beegfs-ctl --listtargets --nodetype=storage --state
TargetID Reachability Consistency NodeID
======== ============ =========== ======
1 Online Good 1
2 Online Good 2
3 Online Good 3
4 Online Good 4
5 Online Good 5
6 Online Good 6
7 Online Good 7
8 Online Good 8
9 Online Good 9
10 Online Good 10
11 Online Good 11
12 Online Good 12
13 Offline Good 13
14 Online Good 14
16 Online Good 13
Please note that a new TargetID numbered as 16 appeared where it should be 13.I tried to swap it back to 13 but I was unable to.
[root#headnode.mintrop.usp.br ~]# beegfs-ctl --removetarget 13
Given target is part of a buddy mirror group. Aborting.
[root#n13 ~]# beegfs-ctl --removemirrorgroup --mirrorgroupid=7 --nodetype=storage --dry-run
Could not remove buddy group: Communication error
I think we are doing something wrong, because of the buddy mirror setup that sometimes is difficult.
Any help is greatly appreciated.
Thank you.
PS: For completude, the checks seems to be fine:
[root#headnode.mintrop.usp.br ~]# beegfs-df
METADATA SERVERS:
TargetID Cap. Pool Total Free % ITotal IFree %
======== ========= ===== ==== = ====== ===== =
1 normal 218.2GiB 66.9GiB 31% 109.2M 107.8M 99%
STORAGE TARGETS:
TargetID Cap. Pool Total Free % ITotal IFree %
======== ========= ===== ==== = ====== ===== =
[ERROR from beegfs-storage n13.mintrop.usp.br [ID: 13]: Unknown storage target]
13 emergency 0.0GiB 0.0GiB 0% 0.0M 0.0M 0%
Solution found: Problem was in the node that was using different inputs than the headnode was seeing. The headnode sees the file below, which corresponds to each node in ascending order (n01, n02...n14):
[root#headnode ~]# cat /data1/beegfs/mgmtd/targetNumIDs
0-5E3B6573-1=1
0-5E3B6592-2=2
0-5E3B65B2-3=3
0-5E3B65D1-4=4
0-5E3B65F1-5=5
0-5E3B6610-6=6
0-5E3B6630-7=7
0-5E3B664F-8=8
0-5E3B666E-9=9
0-5E3B6690-A=A
0-5E3B66B1-B=B
0-5E3B66D2-C=C
0-5E3B66F3-D=D
0-5E3B6714-E=E
0-626C29BD-D=F
0-62853797-D=10
In the n13 file /data1/beegfs/storage/targetID was the corresponding number in tenth 0-62853797-D=10. If you do the calculation this corresponds to 16 in decimal:
[root#headnode~]# echo "obase=16; 16" | bc
10
So the solution was to change the targetID to the hexadecimal corresponding to the number 13:
[root#headnode~]# echo "obase=16; 13" | bc
D
This inside hn's /data1/beegfs/mgmtd/targetNumIDs file corresponds to 0-5E3B66F3-D=D. So two changes were made to n13. Inside the targetNumID and targetID files that had 16 and 0-62853797-D=10 respectively were replaced by:
[root#n13 ~]# cat /data1/beegfs/storage/targetNumID
13
[root#n13 ~]# cat /data1/beegfs/storage/targetID
0-5E3B66F3-D
Once this is done, restart the beegfs-storage services beegfs-meta.
root#headnode~]# beegfs-ctl --listtargets --nodetype=storage --state
TargetID Reachability Consistency NodeID
======== ============ =========== ======
1 Online Good 1
2 Online Good 2
3 Online Good 3
4 Online Good 4
5 Online Good 5
6 Online Good 6
7 Online Good 7
8 Online Good 8
9 Online Good 9
10 Online Good 10
11 Online Good 11
12 Online Good 12
13 Online Good 13
14 Online Good 14
Best regards
Jaqueline
Related
I was wondering if anyone can tell me what these mean. From most people posting about them, there is no more than double digits. However, I have 1051556645921812989870080 Media and Data Integrity Errors on my SK hynix PC711 on my new HP dev one. Thanks!
Here's my entire smartctl output
`smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.0.7-arch1-1] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Number: SK hynix PC711 HFS001TDE9X073N
Serial Number: KDB3N511010503A37
Firmware Version: HPS0
PCI Vendor/Subsystem ID: 0x1c5c
IEEE OUI Identifier: 0xace42e
Total NVM Capacity: 1,024,209,543,168 [1.02 TB]
Unallocated NVM Capacity: 0
Controller ID: 1
NVMe Version: 1.3
Number of Namespaces: 1
Namespace 1 Size/Capacity: 1,024,209,543,168 [1.02 TB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: ace42e 00254f98f1
Local Time is: Wed Nov 9 13:58:37 2022 EST
Firmware Updates (0x16): 3 Slots, no Reset required
Optional Admin Commands (0x001f): Security Format Frmw_DL NS_Mngmt Self_Test
Optional NVM Commands (0x005f): Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x1e): Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg Pers_Ev_Lg
Maximum Data Transfer Size: 64 Pages
Warning Comp. Temp. Threshold: 84 Celsius
Critical Comp. Temp. Threshold: 85 Celsius
Namespace 1 Features (0x02): NA_Fields
Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 6.3000W - - 0 0 0 0 5 5
1 + 2.4000W - - 1 1 1 1 30 30
2 + 1.9000W - - 2 2 2 2 100 100
3 - 0.0500W - - 3 3 3 3 1000 1000
4 - 0.0040W - - 3 3 3 3 1000 9000
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 0
1 - 4096 0 0
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 34 Celsius
Available Spare: 100%
Available Spare Threshold: 5%
Percentage Used: 0%
Data Units Read: 13,162,025 [6.73 TB]
Data Units Written: 3,846,954 [1.96 TB]
Host Read Commands: 156,458,059
Host Write Commands: 128,658,566
Controller Busy Time: 116
Power Cycles: 273
Power On Hours: 126
Unsafe Shutdowns: 15
Media and Data Integrity Errors: 1051556645921812989870080
Error Information Log Entries: 0
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 34 Celsius
Temperature Sensor 2: 36 Celsius
Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged`
Encountered a similar SMART reading from the same model.
I'm seeing a reported Media and Data Integrity Errors rate of a value that's over 2 ^ 84.
It could just be an error with its SMART implementation or the utility reading from it.
Converting your reported value of 1051556645921812989870080 to hex, we get 0xdead0000000000000000 big endian and 0x0000000000000000adde little endian.
Similarly, when I convert my value to hex, I get 0xffff0000000000000000 big endian and 0x0000000000000000ffff little endian, where f is just denotes a value other than 0.
I'm going to assume that the Media and Data Integrity Errors value has no actual meaning with regard to real errors. I doubt that both of us would have values that are padded with 16 0's when converted to hex. Something is sending/receiving/parsing bad data.
If you poke around the other reported SMART values in your post, and on my end, some of them don't seem to make much sense, either.
I am analysing the amount of hashes cracked over a set period of time.
I am looking to save the current status of the crack every 10 seconds.
'''
Recovered........: 132659/296112 (44.80%) Digests, 0/1 (0.00%) Salts
Recovered/Time...: CUR:3636,N/A,N/A AVG:141703,8502198,204052756 (Min,Hour,Day)
Progress.........: 15287255040/768199139595 (1.99%)
'''
I want these 3 lines of the status saved every 10 seconds or so.
Is it possible to do this within hashcat or will I need to make a separate script in python?
Getting the status every 10 seconds
You can enable printing the status with --status and you can set the status to prints every X seconds with --status-timer X. You can see these command line arguments on the hashcat options wiki page, or hashcat --help.
Example: hashcat -a 0 -m 0 example.hash example.dict --status --status-timer 10
Saving all the statuses
I'm assuming that you just want to save everything that gets printed by hashcat while it's running. An easy way to do this is just copy everything from stdout into a file. This is a popular s/o question, so we'll just use this answer.
To be safe, let's use -a which appends to the file, so we don't accidentally overwrite previous runs. All we need to do is put | tee -a file.txt after our hashcat call.
Solution
Give this a shot, it should save all the statuses (and everything else from stdout) to output.txt:
hashcat -a A -m M hashes.txt dictionary.txt --status --status-timer 10 | tee -a output.txt
Just swap out A, M, hashes.txt, and dictionary.txt with the arguments you're using.
If you need help getting just the "Recovered" lines from this output file, or if this doesn't work on your computer (I'm on OSX), let me know in a comment.
In addition to Andrew Zick's answer, note that for machine-readable status, hashcat has native support for machine-readable output - see the --machine-readable option. This produces tab-separated output like so:
STATUS 5 SPEED 111792 1000 EXEC_RUNTIME 0.007486 CURKU 1 PROGRESS 62 62 RECHASH 0 1 RECSALT 0 1 REJECTED 0 UTIL -1
STATUS 5 SPEED 14247323 1000 EXEC_RUNTIME 0.038953 CURKU 36 PROGRESS 2232 2232 RECHASH 0 1 RECSALT 0 1 REJECTED 0 UTIL -1
STATUS 5 SPEED 36929864 1000 EXEC_RUNTIME 1.661804 CURKU 1296 PROGRESS 80352 80352 RECHASH 0 1 RECSALT 0 1 REJECTED 0 UTIL -1
STATUS 5 SPEED 66538858 1000 EXEC_RUNTIME 3.237319 CURKU 46656 PROGRESS 28926722892672 RECHASH 0 1 RECSALT 0 1 REJECTED 0 UTIL -1
STATUS 5 SPEED 63562975 1000 EXEC_RUNTIME 3.480536 CURKU 1679616 PROGRESS 104136192 104136192 RECHASH 0 1 RECSALT 0 1 REJECTED 0 UTIL -1
... which is exactly what tools like Hashtopolis use to provide a front-end to hashcat output.
For machine-readable output, the options --outfile, and --outfile-format are available. See the Format section of the output of hashcat --help for the options to --outfile-format:
- [ Outfile Formats ] -
# | Format
===+========
1 | hash[:salt]
2 | plain
3 | hex_plain
4 | crack_pos
5 | timestamp absolute
6 | timestamp relative
This is what I have tested with the dis module in python -
>>> def f():
... a = 1
... b = 2
... c = 3
... a = b + c * a
... return a + c
...
>>> dis.dis(f)
2 0 LOAD_CONST 1 (1)
2 STORE_FAST 0 (a)
3 4 LOAD_CONST 2 (2)
6 STORE_FAST 1 (b)
4 8 LOAD_CONST 3 (3)
10 STORE_FAST 2 (c)
5 12 LOAD_FAST 1 (b)
14 LOAD_FAST 2 (c)
16 LOAD_FAST 0 (a)
18 BINARY_MULTIPLY
20 BINARY_ADD
22 STORE_FAST 0 (a)
6 24 LOAD_FAST 0 (a)
26 LOAD_FAST 2 (c)
28 BINARY_ADD
30 RETURN_VALUE
Those are instructions for a stack-based virtual machine. Is there any way to convert the above stack-based instructions into register-based instructions provided I have access to unlimited number of registers.
I only know about one tool which does that, we know that JVM is stack based but Dalvik VM is register-based. When we write code in Java, the class files contain stack based instructions and the dx tool converts the stack based instructions to register based instructions so that it can run in the Dalvik VM. So most probably there could be an algorithm somewhere which I have missed.
Also can there be an edge can where the stack could dynamically grow and shrink(which would be decided in runtime) , in that case it would be impossible to convert stack based instructions to register based. However one tool does it.
Can someone point me to the correct direction. Or know any algorithm which can help in this.
I have a dataset with the following details:
1.4 million nodes
2.9 million relationships
15 million properties (including gender, name, subscriber_id etc)
1 relationship type (Contacted)
I've batch imported the data into the database on my machine (64 bit, 16 core, 16 GB RAM) using https://github.com/jexp/batch-import/tree/20
I'm trying to index these nodes on Subscriber_ID, but I'm not really sure what I'm doing.
I ran
start n = node(*) set n:Subscribers
My understanding is this creates a label for each of the nodes (is this correct)
Next I ran
create index on :Subscribers(SUBSCRIBER_ID)
Which I think should create an index for all nodes with the 'Subscribers' label on the property 'SUBSCRIBER_ID'. (correct?)
Now when I go to Neo4j-sh and run
neo4j-sh (?)$ schema
==> Indexes
==> ON :Subscribers(SU_SUBSCRIBER_ID) ONLINE
==>
==> No constraints
But when I run the following it says there are no indices set for the nodes.
neo4j-sh (?)$ index --indexes
==> Node indexes:
==>
==> Relationship indexes:
I have a few questions
Do I have to tell it to index the existing data? If so how do I do
that?
How can I then use the index? I've read through the
documentation but I had a bit of trouble following it.
It looks
like I can have the indexes set up when I run the batch import
script, but I can't really understand how... could someone explain
please?
Here's an example of my data:
Nodes.txt
id SU_SUBSCRIBER_ID CU_FIRST_NAME gender SU_AGE
0 123456 Ann F 56
1 832746 ? UNKNOWN -1
2 546765 Tom UNKNOWN -1
3 768345 Anges F 72
4 267854 Aoibhlinn F 38
rels.csv
start end rel counter
0 3 CONTACTED 2
1 2 CONTACTED 1
1 4 CONTACTED 1
3 2 CONTACTED 2
4 1 CONTACTED 1
schema is the right command to look at.
Cypher uses the label indexes automatically for MERGE and MATCH.
With the Java Core-API you'd use db.findNodesByLabelAndProperty(label,property,value)
You did the right thing, except for one. You could have created the labels on the nodes while doing the batch-import.
Just add a l:label field to your CSV-file containing a comma separated list of labels per node. Like shown in the readme on that branch.
If you look at the original Wordnet search and select "Display options: Show Lexical File Info", you'll see an extremely useful classification of words called lexical file. Eg for "filling" we have:
<noun.substance>S: (n) filling, fill (any material that fills a space or container)
<noun.process>S: (n) filling (flow into something (as a container))
<noun.food>S: (n) filling (a food mixture used to fill pastry or sandwiches etc.)
<noun.artifact>S: (n) woof, weft, filling, pick (the yarn woven across the warp yarn in weaving)
<noun.artifact>S: (n) filling ((dentistry) a dental appliance consisting of ...)
<noun.act>S: (n) filling (the act of filling something)
The first thing in brackets is the "lexical file". Unfortunately I have not been able to find a SPARQL endpoint that provides this info
The latest RDF translation of Wordnet 3.0 points to two things:
Talis SPARQL endpoint. Use eg this query to check there's no such info:
DESCRIBE <http://purl.org/vocabularies/princeton/wn30/synset-chair-noun-1>
W3C's mapping description. Appendix D "Conversion details" describes something useful: wn:classifiedByTopic.
But it's not the same as lexical file, and is quite incomplete. Eg "chair" has nothing, while one of the senses of "completion" is in the topic "American Football"
DESCRIBE <http://purl.org/vocabularies/princeton/wn30/synset-completion-noun-1> ->
<j.1:classifiedByTopic rdf:resource="http://purl.org/vocabularies/princeton/wn30/synset-American_football-noun-1"/>
The question: is there a public Wordnet query API, or a database, that provides the lexical file information?
Using the Python NLTK interface:
from nltk.corpus import wordnet as wn
for synset in wn.synsets('can'):
print synset.lexname
I don't think you can find it in the RDF/OWL Representation of WordNet. It's in the WordNet distribution though: dict/lexnames. Here is the content of the file as of WordNet 3.0:
00 adj.all 3
01 adj.pert 3
02 adv.all 4
03 noun.Tops 1
04 noun.act 1
05 noun.animal 1
06 noun.artifact 1
07 noun.attribute 1
08 noun.body 1
09 noun.cognition 1
10 noun.communication 1
11 noun.event 1
12 noun.feeling 1
13 noun.food 1
14 noun.group 1
15 noun.location 1
16 noun.motive 1
17 noun.object 1
18 noun.person 1
19 noun.phenomenon 1
20 noun.plant 1
21 noun.possession 1
22 noun.process 1
23 noun.quantity 1
24 noun.relation 1
25 noun.shape 1
26 noun.state 1
27 noun.substance 1
28 noun.time 1
29 verb.body 2
30 verb.change 2
31 verb.cognition 2
32 verb.communication 2
33 verb.competition 2
34 verb.consumption 2
35 verb.contact 2
36 verb.creation 2
37 verb.emotion 2
38 verb.motion 2
39 verb.perception 2
40 verb.possession 2
41 verb.social 2
42 verb.stative 2
43 verb.weather 2
44 adj.ppl 3
For each entry of dict/data.*, the second number is the lexical file info. For example, this filling entry contains the number 13, which is noun.food.
07883031 13 n 01 filling 0 002 # 07882497 n 0000 ~ 07883156 n 0000 | a food mixture used to fill pastry or sandwiches etc.
It can be done through MIT JWI (MIT Java Wordnet Interface) a Java API to query Wordnet. There's a topic in this link showing how to implement a java class to access lexicographic
This is what worked for me,
Synset[] synsets = database.getSynsets(wordStr);
ReferenceSynset referenceSynset = (ReferenceSynset) synsets[i];
int lexicalCode =referenceSynset.getLexicalFileNumber();
Then use above table to deduce "lexnames" e.g. noun.time
If you're on Windows, chances are it is in your appdata, in the local directory. To get there, you will want to open your file browser, go to the top, and type in %appdata%
Next click on roaming, and then find the nltk_data directory. In there, you will have your corpora file. The full path is something like:
C:\Users\yourname\AppData\Roaming\nltk_data\corpora
and lexnames will present under
C:\Users\yourname\AppData\Roaming\nltk_data\corpora\wordnet.