Seaweedfs volume management - seaweedfs

I have 2 questions concerning a Seaweedfs cluster we have running. The leader is started with the following command:
/usr/local/bin/weed server -ip=192.168.13.154 -ip.bind=192.168.13.154 -dir=/opt/seaweedfs/volume-1,/opt/seaweedfs/volume-2,/opt/seaweedfs/volume-3 -master.dir=/opt/seaweedfs/master -master.peers=192.168.13.154:9333,192.168.13.155:9333,192.168.13.156:9333 -volume.max=30,30,30 -filer=true -s3=true -metrics.address=192.168.13.84:9091
Question 1
I created a master.toml file using weed scaffold -config=master:
[master.maintenance]
# periodically run these scripts are the same as running them from 'weed shell'
scripts = """
ec.encode -fullPercent=95 -quietFor=1h
ec.rebuild -force
ec.balance -force
volume.balance -force
"""
sleep_minutes = 17 # sleep minutes between each script execution
However the maintenance scripts seem to fail because
shell failed to keep connected to localhost:9333: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp [::1]:19333: connect: connection refused"
This makes sense since the master is bound to ip 192.168.13.154 and the maintenance script tries to connect to localhost. How can I specificy the master ip in the master.toml file?
Question 2
The amount of volumes seems to grow faster than the amount of disk space used. For example on the .154 server there are only 11 free volumes. But looking at the disk space there should be much more.
Status:
{
"Topology": {
"DataCenters": [
{
"Free": 16,
"Id": "DefaultDataCenter",
"Max": 270,
"Racks": [
{
"DataNodes": [
{
"EcShards": 0,
"Free": 0,
"Max": 90,
"PublicUrl": "192.168.13.155:8080",
"Url": "192.168.13.155:8080",
"Volumes": 90
},
{
"EcShards": 0,
"Free": 11,
"Max": 90,
"PublicUrl": "192.168.13.154:8080",
"Url": "192.168.13.154:8080",
"Volumes": 79
},
{
"EcShards": 0,
"Free": 5,
"Max": 90,
"PublicUrl": "192.168.13.156:8080",
"Url": "192.168.13.156:8080",
"Volumes": 85
}
],
"Free": 16,
"Id": "DefaultRack",
"Max": 270
}
]
}
],
"Free": 16,
"Max": 270,
"layouts": [
...
]
},
"Version": "30GB 1.44"
}
Disk (192.168.13.154):
/dev/sdb1 1007G 560G 397G 59% /opt/seaweedfs/volume-1
/dev/sdc1 1007G 542G 414G 57% /opt/seaweedfs/volume-2
/dev/sdd1 1007G 398G 559G 42% /opt/seaweedfs/volume-3
Is this related to the maintenance scripts not running properly, or is there something else I'm not understanding correctly?

Question 1: Added a fix https://github.com/chrislusf/seaweedfs/commit/56244fb9a13c75616aa8a9232c62d1b896906e98
Question 2: Likely related to master leadership changes.

Related

Bitcoin / DefiChain RPC rawTransaction

Hello i try to figure out how to encode normal rpc calls in an raw transaction.
Till now my problem is that i dont know what the hex must contain as string.
f.e.
rpc command: "method: 'compositeswap' {'from':'MyAddress','tokenFrom':'MyToken1','amountFrom':'0.001','to':'Address','tokenTo':'Token2','maxPrice':'0.01'}"
There seems to be OP-Codes to exists like OP_DEFI_TX_COMPOSITE_SWAP
how does the chain knows to execute an operation with params?
i tried to figure it out by trying to decode an actual transaction from the test wallet:
from the part
"scriptPubKey": {
"asm": "OP_RETURN 446654786917a914721d5b1c58d38af7b6797b385b6ac291b002f88c870000e1f5050000000017a914721d5b1c58d38af7b6797b385b6ac291b002f88c870b0000000000000000c74e71050000000000",
"hex": "6a4c50446654786917a914721d5b1c58d38af7b6797b385b6ac291b002f88c870000e1f5050000000017a914721d5b1c58d38af7b6797b385b6ac291b002f88c870b0000000000000000c74e71050000000000",
"type": "nulldata"
},
of
{
"txid": "9a98d693d4c5107647131ee1bb7a5b0cce0fcdbe390c9609a71f4b71157e39dc",
"hash": "a18a6fd4abf0ba1885febcf37a333b1d1f34b4de954f13538a6619b1d7b20042",
"version": 4,
"size": 309,
"vsize": 228,
"weight": 909,
"locktime": 0,
"vin": [
{
"txid": "9e1140197138ba5e247ab3b3f1f4881bf7be624a939073d9795242caf3634409",
"vout": 1,
"scriptSig": {
"asm": "0014451be7ab94ccd7eff0a33ab8fe997a75c62eb7dd",
"hex": "160014451be7ab94ccd7eff0a33ab8fe997a75c62eb7dd"
},
"txinwitness": [
"30440220552d8aa4e129f566bfe083b780e1dcf67a3ca0176e07407912451371f597bc620220698c6ac483e021b78c7d7bf42e14f1c619618d1941cf12fec7cf8302ece6d3ae01",
"03c7d2dbe5ee429de5d88e8594cda6ceb84268ebbf9d0b16b33664e999307f33e8"
],
"sequence": 4294967295
}
],
"vout": [
{
"value": 0,
"n": 0,
"scriptPubKey": {
"asm": "OP_RETURN 446654786917a914721d5b1c58d38af7b6797b385b6ac291b002f88c870000e1f5050000000017a914721d5b1c58d38af7b6797b385b6ac291b002f88c870b0000000000000000c74e71050000000000",
"hex": "6a4c50446654786917a914721d5b1c58d38af7b6797b385b6ac291b002f88c870000e1f5050000000017a914721d5b1c58d38af7b6797b385b6ac291b002f88c870b0000000000000000c74e71050000000000",
"type": "nulldata"
},
"tokenId": 0
},
{
"value": 183.03901748,
"n": 1,
"scriptPubKey": {
"asm": "OP_HASH160 721d5b1c58d38af7b6797b385b6ac291b002f88c OP_EQUAL",
"hex": "a914721d5b1c58d38af7b6797b385b6ac291b002f88c87",
"reqSigs": 1,
"type": "scripthash",
"addresses": [
"tgfbETCK2kYyvsnHbS41v9aicQzAXLsz9B"
]
},
"tokenId": 0
}
]
}
the
OP_RETURN 446654786917a914721d5b1c58d38af7b6797b385b6ac291b002f88c870000e1f5050000000017a914721d5b1c58d38af7b6797b385b6ac291b002f88c870b0000000000000000c74e71050000000000
cant be decoded sucessfully back into a string.
does s.o know what kind of encoding it is?
tried
bytess=bytes.fromhex("446654786917a914721d5b1c58d38af7b6797b385b6ac291b002f88c870000e1f5050000000017a914721d5b1c58d38af7b6797b385b6ac291b002f88c870b0000000000000000c74e71050000000000")
print(bytess.decode("latin-1"))
but only get
INFO (MainThread) 14.05.2022 22:02:39 DfTxi©r
INFO (MainThread) 14.05.2022 22:02:39 [
INFO (MainThread) 14.05.2022 22:02:39 XÓŠ÷¶y{8[j‘°øŒ‡ áõ ©r
You can't just decode the string after the OP Code. It is a concatenation of several input parameters depending on the function.
In your example
6a4c50446654786917a914721d5b1c58d38af7b6797b385b6ac291b002f88c870000e1f5050000000017a914721d5b1c58d38af7b6797b385b6ac291b002f88c870b0000000000000000c74e71050000000000
6a is OP_RETURN
4c50 is Number of bytes following (in HEX)
44665478 is "DfTx" in HEX
6917a914721d5b1c58d38af7b6797b385b6ac291b002f88c870000e1f5050000000017a914721d5b1c58d38af7b6797b385b6ac291b002f88c870b0000000000000000c74e71050000000000 = the parameters from your json. For your address you need Base58 (in this case) or Bech32 encoding to get the hex (some backward engineering necessary).
The part with lots of zeros is the amount/maxPrice.
The tokens are inserted by their IDs not the name.
I am thinking about writing a guide. So far, I am afraid you have to do some backward engineering. Hope this helps a little bit

Details are visible in my request, how do I filter this in my API request?

This is my first time working with an API and the various sources have been very helpful.However I am now struggling at the end.
API: Vincere API
API Documentation: https://api.vincere.io/documentation.html#tag/Introduction
What I am trying to do: The client has a list of jobs shown on their dashboard when logged in. On the dashboard there is a column listed as "Posted", this data below is then either "Posted" or "Not Posted". I would like to retrieve all jobs that are only listed as "Posted".
What I am able to do:
I am able to connect, authenticate and get the required data using the above documentation. I can then see whether a job should be private or public.
Problem I am facing:
I would like to only show the jobs that have
"private_job": 0,
At the moment my request is returning all jobs.
The request I am sending::
https://companyname.vincere.io/api/v2/job/search/fl=job_title,private_job;sort=created_date dec?limit=100
Result
{
"result": {
"start": 0,
"total": 355,
"items": [
{
"private_job": 0,
"job_title": "HR Manager"
},
{
"private_job": 0,
"job_title": "Accountant / Financial Manager - Cryptocurrency"
},
{
"private_job": 1,
"job_title": "Conveyancer"
},
What I have tried:
From reading some other questions and articles I tried the following but the result remained unchanged
https://companyname.vincere.io/api/v2/job/search/fl=job_title,private_job;sort=created_date dec?limit=100&private_job=0
Result
{
"result": {
"start": 0,
"total": 355,
"items": [
{
"private_job": 0,
"job_title": "HR Manager"
},
{
"private_job": 0,
"job_title": "Accountant / Financial Manager - Cryptocurrency"
},
{
"private_job": 1,
"job_title": "Conveyancer"
},
I also tried:
https://companyname.vincere.io/api/v2/job/search/fl=job_title,private_job;sort=created_date dec?limit=100&private_job=false
Result
Same as above
I would really appreciate some assistance with the above. Thank you in advance.
You need to add private_job to the q (query list). So I thinkwhat you're after is something like:
https://companyname.vincere.io/api/v2/job/search/fl=job_title,private_job;sort=created_date desc?q=private_job:0%23&limit=100

Web Activity endless running In Azure Data Factory

Currently it seems that web activity is broken.
When using simple pipeline
{
"name": "pipeline1",
"properties": {
"activities": [
{
"name": "Webactivity",
"type": "WebActivity",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"url": "https://www.microsoft.com/",
"connectVia": {
"referenceName": "AutoResolveIntegrationRuntime",
"type": "IntegrationRuntimeReference"
},
"method": "GET",
"body": ""
}
}
],
"annotations": []
}
}
When debugging it never finishes. There is "in progress" for several minutes.
I tried Web hook and it works.
Is there something else I could try?
A quick note on the "never finishes" issue: one of my pet peeves with Data Factory is that the default timeout for all activities is 7 DAYS. While I've had a few activities that needed to run for 7 hours, a WEEK is a ridiculous default timeout value. One of the first things I do in any production scenario is address the timeout values of all the activities.
As to the Web activity question: I set up a quick example in my test bed and it returned just fine:
Looking at the generated code, the only real difference I see is the absence of the "connectVia" property that was in your example:
Ok I've found it.
The default AutoResolveIntegrationRuntime only had managed private network which I couldn't change. So I created a new Integration Runtime with public network setting.
This is a litte bit strange as I started today with a brand new Azure Data Factory.
I wonder why I cannot change the default Integration Runtime to disable virtual network:

Media Source Extensions appendBuffer of WebM stream in random order

I am trying to achieve video downloading in parallel from multiple sources. However MSE appendBuffer method always fails when not following sequence order of video file.
I would like to append parts in random order and play video "as soon as possible".
I was exploring SourceBuffer mode property as well as timestampOffset. None of those were helpful.
I am wondering if source webm file i have could be in "not supported format" for such a task (sequential approach works fine).
source video file
Thank you for any advices.
UPDATE:
I tried to analyse well known example video file and i figured out that it is possible to append parts of it out of order. Seems like it is necessary to follow Cluster byte ranges:
<Cluster type="list" offset="4357">
<Timecode type="uint" value="0"/>
<SimpleBlock type="binary" size="7723" trackNum="1" timecode="0" presentationTimecode="0" flags="80"/>
<SimpleBlock type="binary" size="5" trackNum="2" timecode="0" presentationTimecode="0" flags="80"/>
...
</Cluster>
<Cluster type="list" offset="16187">
<Timecode type="uint" value="385"/>
<SimpleBlock type="binary" size="5" trackNum="2" timecode="0" presentationTimecode="385" flags="80"/>
<SimpleBlock type="binary" size="4968" trackNum="1" timecode="13" presentationTimecode="398" flags="80"/>
...
</Cluster>
After digging into webm format specification, compiling libwebm tools and studying DASH i finally figured out how to make MSE appendBuffer working in any order!
ffmpeg -i result.webm -g 10 -c:v libvpx resultClusters.webm (you can also use libvpx-vp9)
mkvmuxer_sample -i resultClusters.webm -o resultRepaired.webm
mse_json_manifest resultRepaired.webm >> manifest.json
You will get on stdout something like:
{
"type": "video/webm; codecs=\"vp8\"",
"duration": 27771.000000,
"init": { "offset": 0, "size": 258},
"media": [
{ "offset": 258, "size": 54761, "timecode": 0.000000 },
{ "offset": 55019, "size": 166431, "timecode": 2.048000 },
{ "offset": 221450, "size": 49258, "timecode": 4.130000 },
{ "offset": 270708, "size": 29677, "timecode": 6.148000 },
{ "offset": 300385, "size": 219929, "timecode": 8.232000 },
{ "offset": 520314, "size": 25132, "timecode": 10.335000 },
{ "offset": 545446, "size": 180777, "timecode": 12.440000 },
{ "offset": 726223, "size": 76107, "timecode": 14.471000 },
{ "offset": 802330, "size": 376557, "timecode": 14.794000 },
{ "offset": 1178887, "size": 247138, "timecode": 16.877000 },
{ "offset": 1426025, "size": 78468, "timecode": 18.915000 },
{ "offset": 1504493, "size": 25614, "timecode": 20.991000 },
{ "offset": 1530107, "size": 368277, "timecode": 23.093000 },
{ "offset": 1898384, "size": 382847, "timecode": 25.097000 },
{ "offset": 2281231, "size": 10808, "timecode": 27.135000 }
]
}
Now all you have to do is firstly load metadata xhr.setRequestHeader("Range", "bytes=0-257"); and then in ANY ORDER all other segments. E.g. second segment range is 55019-221449 bytes.
Explanation:
The most important thing is ffmpeg reencoding with group of frames set to the size of cluster you would like to have. In this example i choose pretty low threshold (each 10 frames) but you can choose higher causing fewer clusters are generated (less items in "media" array).
After that you have to fix cues in the classic way (using sample_muxer from libwebm) and you are ready to go.
Tested on: Chrome 51, Firefox 47.

successful snapshot fails to load some shards, RepositoryMissingException in elasticsearch

I had a backup successfully complete to my s3 bucket in elasticsearch:
{
"state": "SUCCESS",
"start_time": "2014-12- 06T00:12:39.362Z",
"start_time_in_millis": 1417824759362,
"end_time": "2014-12-06T00:33:34.352Z",
"end_time_in_millis": 1417826014352,
"duration_in_millis": 1254990,
"failures": [],
"shards": {
"total": 345,
"failed": 0,
"successful": 345
}
}
But when I restore from the snapshot, I have a few failed shards, with the following message:
[2014-12-08 00:00:05,580][WARN ][cluster.action.shard] [Sunder] [kibana-int][4] received shard failed for [kibana-int][4],
node[_QG8dkDaRD-H1uPL_p57lw], [P], restoring[elasticsearch:snapshot_1], s[INITIALIZING], indexUUID [SAuv_EU3TBGZ71NhkC7WOA],
reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[kibana-int][4] failed recovery];
nested: IndexShardRestoreFailedException[[kibana-int][4] restore failed];
nested: RepositoryMissingException[[elasticsearch] missing]; ]]
how I do reconcile the data, or if necessary remove the shards from my cluster to complete the recovery?