At first it's seem to be easy task since the config for creating this index is in /usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json
{
"date_index_name": {
"field": "timestamp",
"date_rounding": "d",
"index_name_prefix": "{{fields.index_prefix}}",
"index_name_format": "yyyy.mm.dd",
"ignore_failure": false
}
},
But changing the index_name_format to 'xxxx.ww' and restart filebeat, it still write to old index format. Anyone know if I need to do anything else to let filebeat recognizes new pipeline config?
OK, so problem resolved after digging into Wazuh documentation.
First, the format for weekly index should be 'YYYY.ww'
Second, turned out that pipeline.json is cached into elasticsearch, and need to be purged by:
DELETE _ingest/pipeline/filebeat-7.10.2-wazuh-alerts-pipeline
Related
I need to mark documets as expired after some time and therefore I am trying to use #refresh feature to re-run subscription and to compute my 'expired' flag. I know there is 'Document expiration' feature but this one removes data which I don't want.
I have turned Refresh feature in settings and added #refresh UTC datetime in metadata for required documents. For example I added manually this document:
{
"Name": "My data",
"#metadata": {
"#collection": "Testing",
"#refresh": "2021-04-30T07:41:35.4845961Z"
}
}
It looks like I am facing non deterministic behavior - sometimes refresh is processed sometimes not. I tried with different combinations of times and set through code or via Raven Studio.
Refresh interval is set to refresh but still says "in less than a minute"
I am using
Community license (Document refresh not mentioned here, but I don't see it mentioned for any other licenses as well)
community license extensions
tried more vresions of RavenDB with same result (5.1.7. was looking more promising as it worked for some time but after a while stopped):
4.2.111 server/studio version in Docker on Windows 10
5.1.7 server/studio version
C# RavenDB.Client 5.1.6
Did not find related issue in bug tracker
https://issues.hibernatingrhinos.com/issues/RavenDB?q=document%20refresh
Any ideas what to check or what might be the case?
EDIT: After turned logging into console I found some error log. It looks like
RavendbProject, Raven.Server.Documents.Expiration.ExpiredDocumentsCleaner, Failed to refresh documents on RavendbProject which are older than 05/17/2021 09:48:47, EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object.
RavendbProject | at Sparrow.Server.ByteStringContext`1.From(String value, ByteStringType type, ByteString& str) in C:\Builds\RavenDB-Stable-5.1\51024\src\Sparrow.Server\ByteString.cs:line 1297
RavendbProject | at Raven.Server.Documents.DocumentPutAction.PutDocument(DocumentsOperationContext context, String id, String expectedChangeVector, BlittableJsonReaderObject document, Nullable`1 lastModifiedTicks, String changeVector, DocumentFlags flags, NonPersistentDocumentFlags nonPersistentFlags) in C:\Builds\RavenDB-Stable-5.1\51024\src\Raven.Server\Documents\DocumentPutAction.cs:line 190
Also worth mentioning is that my document was stored in ClusterWide transaction and thus I can see in one of my documents corresponding flag:
"#flags": "FromClusterTransaction",
My current suspicion is that it may happen that one of these documents prevented other documents from being refreshed. After deleting cluster-transaction document, other documents in collection were refreshed
The bug related to document that was added via cluster transaction, the workaround for now would be to not use cluster transaction.
I have opened an issue on bug tracker,
https://issues.hibernatingrhinos.com/issue/RavenDB-16710
I'm setting up Superset (0.36.0) in production Mode (with Gunicorn), and I would like to set up impersonate while running Impala queries on my Kerberized Cluster, to each user of Superset have privilegies on tables/databases like he has on Hive/Hue/HDFS. I've tried to set "Impersonate the logged on user" to true in my database config, but it's not changing the user that is running the query, it's always using the celery-worker user.
My database config is:
Extras:
{
"metadata_params": {},
"engine_params": {
"connect_args": {
"port": 21050,
"use_ssl": "True",
"ca_cert": "/path/to/my/cert.pem",
"auth_mechanism": "GSSAPI"
}
},
"metadata_cache_timeout": {},
"schemas_allowed_for_csv_upload": []
}
My query resume in Cloudera Manager (5.13):
How can I enable Impersonate correctly in my Superset? Maybe there is something related to the config impala.doas.user in HiveServer2 connection, but I don't know how to config this properly.
I faced the same issue and I was to get it working for hive. The issue seems to be in the file hive.py located under the path ${YOUR_INSTALLATION_PATH}/superset/db_engine_specs
If you just comment out line 435, it should work. Unfortunately, I don't understand python well enough to tell you the exact reason.
I found this by brute force by running the source code and putting log statements
if (
backend_name == "hive"
# comment this line
# and "auth" in url.query.keys()
and impersonate_user is True
and username is not None
):
configuration["hive.server2.proxy.user"] = username
return configuration
Alternatively, if you do not want to modify the source code, you can modify the URL while creating the data source in superset as :
hive://<url>/default?auth=NONE ( when there is no security )
hive://<url>/default?auth=KERBEROS
hive://<url>/default?auth=LDAP
I'm using RavenDb Server and Client 3.5.0 and I have tried to get UniqueConstraint work without success.
The simple case:
using Raven.Client.UniqueConstraints;
public class User {
public string Id { get; set; }
[UniqueConstraint]
public string Email { get; set; }
}
The documentation says:
Drop the Raven.Bundles.UniqueContraints assembly in the Plugins
directory.
I did it by NuGet: Install-Package RavenDB.Bundles.UniqueConstraints -Version 3.5.0
and then paste the binary Raven.Bundles.UniqueConstraints.dll to folder Plugins that I created myself in Raven's root directory.
After save an User document I get this in Metadata:
"Ensure-Unique-Constraints": [
{
"Name": "Email",
"CaseInsensitive": false
}
]
All seems to work, but I still saving documents with the same email.
UniqueConstraintCheckResult<User> checkResult = session.CheckForUniqueConstraints(user);
// returns whether its constraints are available
if (checkResult.ConstraintsAreFree())
{
session.Store(user);
session.SaveChanges();
}
I check this link RavenDB UniqueConstraint doesn't seem to work and this one https://groups.google.com/forum/#!searchin/ravendb/unique|sort:relevance/ravendb/KzO-eIf9vV0/NJyJ4DNniFUJ and many other that people have the same problem without solution. In some cases they said that are checking if the property already exist in database manualy as solution.
The documentation also says:
To activate unique constraints server-wide, simply add Unique
Constraints to Raven/ActiveBundles configuration in the global
configuration file, or setup a new database with the unique
constraints bundle turned on using API or the Studio
but with no clue how to do that. I did some search and find a possible how:
In Studio, select database, go to Settings -> Database settings, and I found this config:
{
"Id": "TestRaven",
"Settings": {
"Raven/DataDir": "~\\TestRaven"
},
"SecuredSettings": {},
"Disabled": false
}
and I tried add this config:
"Settings": {
"Raven/DataDir": "~\\TestRaven",
"Raven/ActiveBundles": "UniqueConstraints"
}
Then I get an error when trying save it. The message erros says something like "the database is already created and cant modify or add bundles" and make a sugestion to add this line "Raven-Temp-Allow-Bundles-Change": true and I was able to save de settings with UniqueConstraint bundle configuration.
So far I think I did all requirement that documentation describe. The last one is:
Any bundle which is not added to ActiveBundles list, will not be
active, even if the relevant assembly is in the Plugins directory.
The only place that I found a bundle list is creating a new database in Studio, but the list is not editable, just an information about what already has enabled.
The documentation says a lot of requirements but just dont tell us how to do it, super smart, and we have to try gess how. I could get to here so far, but gess what? It still not working!
My question is, UniqueConstraints realy work in RavenDb? Have someone get this working?
If yes, cloud please tell me how?
Thank you in advance!
[Edited]
I forgot to mention that I added the follow line:
store.Listeners.RegisterListener(new UniqueConstraintsStoreListener());
And also tried with version 3.5.1.
The issue is that the specified name of the bundle is incorrect so it won't be active on the server side. Please use "Unique Constraints" instead of "UniqueConstraints" in "Raven/ActiveBundles" settings option.
I'm trying to test S3 backup/restore functionality.
What I did:
installed DSE +OpsCenter on Amazon
Scheduled hourly backup of all keyspaces (60 MB total size). got 10 backups in the morning.
Terminate instances and create new one
try to get my data back. No luck. OpsCenter can't connect to my S3 bucket
it takes >10 min now...
What do I do wrong?
UPD:
finally, got response:
I believe that this may be OPSC-5915 (sorry no public bug tracker) which is fixed in the upcoming 5.2.0 release.
The summary is that the API calls will still work as expected but the UI is not pushing the destination information to the API endpoint correctly.
You can confirm that this is the error you are experiencing thusly:
1) goto /etc/opscenter/clusters/<cluster_name>.conf (or similar location depending on if you've done a tarball install/etc)
2) Find the destination ID that matches your bucket, it'll look something like b699738d9bd8409c82e664b543f24030
3) Confirm the clustername in your opsc URLs, it'll look something like localhost:8888/my_cluster
4) Manually hit the API to retrieve your backup list
curl localhost:8888/<clustername>/backups?amount=6\&last_seen=\&list_all=1\&destination=<destination ID>
It'll look like this
curl localhost:8888/dse/backups?amount=6\&last_seen=\&list_all=1\&destination=b699738d9bd8409c82e664b543f24030
5) You should get back a json, confirm that your backup is listed
{"opscenter_adhoc_2014-12-17-20-22-57-UTC": {"keyspaces": {"OpsCenter":...
If you see your backup in the JSON, then opsc sees your backup and this is indeed OPSC-5915, so that's at least confirmed.
If this is your case, we can work around it by manually hitting the restore API (this is admittedly a bit more involved).
http://docs.datastax.com/en/opscenter/5.1/api/docs/backups.html#backups
It'll look a bit like this:
BACKUP='opscenter_4a269167-96c1-40c7-84b7-b070c6bcd0cd_2012-06-07-18-00-00-UTC'
curl -X POST
http://192.168.1.1:8888/Test_Cluster/backups/restore/$BACKUP
-d '{
"destination": "fe85800f3f4043a88fbe76fc45b22b19",
"keyspaces": {
"Keyspace1": {
"column-families: ["users", "dates"],
"truncate": true
},
"OpsCenter": {
"truncate": false
}
},
}'
I like to get response from Lucene.NET after N seconds, even no results yet. How?
Currently I am facing a problem. All Lucene.NET index is located in a central place, and each instance, after reboot, have to copy the index to local before search can happen.
The copy will be initiated after first Lucene.NET request and take few minutes to complete. Currently all Lucene.NET just hung and wait, so I like to FORCE them to response no matter what.
Please help.
[EIDT]
So the path is using TimeLimitingCollector, this gives me another question, how to use multiple connector together?
My original code is:
TopFieldCollector collector = TopFieldCollector.create(Sort.RELEVANCE, resultAmount,
false,
true /* trackDocScores */,
true /* trackMaxScore */,
false /* docsInOrder */);
searcher.Search(query, new PositiveScoresOnlyCollector(collector));
Where should I put TimeLimitingCollector?
You can use a TimeLimitingCollector.
[EDIT]
I am not familiar with Lucene.NET, but with Lucene Java you just need to wrap your collector inside a TimeLimitingCollector, and it will throw a time-out exception whenever trying to collect a document too late.