While running the sparql queries I get the warning
# hit complexity limit 20 times, increasing soft limit may give more results
and this is not dependent to any particular query but for all.I wanted to know how can I increase the soft limit if I am not using the http interface rather I am using the 4store client master
By default 4store gives a soft limit of about 1000. You have a couple of options to tweak/disable search breadth:
If using 4s-query you can override the setting with --soft-limit e.g. 4s-query --soft-limit 5000 demo 'SELECT ... etc.'. Setting it to -1 completely disables it.
If using a 3rd party client you can tweak soft limit with a cgi parameter e.g. sparql-query 'http://example.org/sparql/?soft-limit=-1'
You can set the soft limit globally when starting the sparql server with the -s parameter. See 4s-httpd --help.
Related
I'm currently studying Redis, and have the following case:
So what I have is a sorted set by google place id, for which all posts are sorted from recent to older.
The first page that is requested fetches posts < current timestamp.
When a cursor is sent to the backend, this cursor is a simple timestamp that indicates from where to fetch the next posts from the ZSET.
The query to retrieve posts by place id would be:
ZREVRANGEBYSCORE <gplaceId> <cur_timestamp> -INF WITHSCORES LIMIT <offset:timestamp as from where to fetch> <count:number of posts>
My question is what is the recommended way to clean up members of the ZSET.
Since I want to use Redis as a cache, I would prefer to limit the number of posts per place for example up until 50. When places get new posts when already 50 posts have been added to the set, I want to drop the last post from the set.
Of course I realise that I could make a manual check on every insert and perform operations as such, but I wonder if Redis has a recommended way of performing this cleanup. Alternatively I could create a scheduler for this, but I would prefer not having to do this.
Unfortunately Redis sorted set do not come with an out of the box feature for this. If sorted sets allowed a max size attribute with a configurable eviction strategy - you could have avoided some extra work.
See this related question:
How to specify Redis Sorted Set a fixed size?
In absence of such a feature, the two approaches you mentioned are right.
You can replace inserts with a transaction : insert, check size, delete if > 50
A thread that checks size of the set and trims it periodically
We have Oracle Unified Directory(OUD) with around 10 million users. We have read-only account that is used to for search operations on base dn. But sometimes this user account is being used to perform wildcard search like (uid=*) on base dn. This is causing timeout issues on OUD and impacting performance also. Is there a way to restrict this wildcard search on base dn?
You can set resource limited on the user that is performing the wildcard search.
This will of course cause all the queries they run to be limited to the amount of entries set and have unknown consequences in whatever app is using the returned data. It’s probably preferable to educate them on how to perform paged searches so you can still define limits and they can still consume the data.
Create an ldif file and run ldapmodify with limits suited for your environment.
dn: uid=baduser,ou=people,dc=example,dc=com
changetype: modify
add: ds-rlim-lookthrough-limit
ds-rlim-lookthrough-limit: 10000
-
add: ds-rlim-size-limit
ds-rlim-size-limit: 5000
-
add: ds-rlim-time-limit
ds-rlim-time-limit: 300
I am trying to understand how these limits work.
But I am not able to figure it out, why soft limit is needed in the first place when hard limit is present.
Most of the sites I have gone through talk about what they are, but none mentioned about how do they work.
In context of processes, Can anyone explain it how do these work?
Thank you.
Let us consider a user abc whose number of processes are set as
abc soft nproc 1024
abc hard nproc 20000
When this user logs in, the user has the effective soft limit of processes applied i.e., abc can run a maximum of 1024 processes.
When this limit is used up, the user will not be able to run any more processes unless the soft limit is increased further.
At this point, the user can optionally increase (but should not be greater than the hard limit) the process limit.
If the user tries to increase it to 21000 (which is above the hard limit of 20000)
ulimit -S -u 21000
-bash: ulimit: max user processes: cannot modify limit: Operation not permitted
But setting it to any value lesser or equal to the hard limit is accepted. Increasing hard limit can be done only by root.
Can't we just use the hard limits?
Yes, we can.
abc soft nproc 20000
abc hard nproc 20000
Setting soft limit equal to hard limit would make the hard limit be the default limit for abc user.
But it is not recommended as it could lead to a single user exhausting most of the processes (if the limit set is relatively high) that can be totally spawned (pid_max).
I'm trying to use HAProxy for rate-limiting.
I need to keep track of several endpoints and limit them individually.
So far I was using general purpose counters. However, there is only 3 of them, sc0 to sc2.
In the documentation, it mentions that all the operations on these counters take an optional table parameter, but it's not clear, then, if I can track different things on different tables, but using the same counters.
In other words: is the limit of 3 general purpose counters global, or per sticky table?
If, after proper table definition and track instructions, I do
sc1_inc_gpc0(table1)
(and, under different conditions)
sc1_inc_gpc0(table2)
And then have 2 acl rules like
acl X sc1_get_gpc0(table1) gt 1
acl Y sc1_get_gpc0(table2) gt 1
Will the two acls work indepentently, or they would to all effects track the same counter?
Thanks for all help!
(In case you are wondering: for a number of reasons, at the moment I could not use a different solution than HAProxy for rate-limiting)
Self answered after looking at the source code and testing.
Yes it is possible to use the same counter on different tables
Moreover, you can also increment the number of available counters at build time. The default is 3, but it can be set up to 10 for sure. Then generic versions of the functions like sc_gpc0_rate(<ctr>[,<table>]) can be used, passing the index of the new counter as first argument.
There is a limit of displaying 1000 rows for the tables in Datastax Devcenter. Any reason for having this option?
Because when queried as SELECT count(*) FROM tablename;
the performance from Cassandra is going to be same whether displaying 1000 records or complete records set.
DevCenter version 1.6.0 introduces result set paging which allows you to browse all the rows in your result set.
In DevCenter 1.6.0 the "with limit" value sets the paging size, e.g. the number of records to view per page and is still limited to 1000 maximum. However, now you can page forward (and back) through all of the query results.
A related new feature allows you to export all results to a file, either as CSV or INSERT statements. Right-click in the results view area and select "Export all results to File as [CSV|Insert]".
This is by design; consider it as a safeguard that prevents you from potentially fetching thousands or millions of rows by accident, which, among other problems, could have a serious impact on your network's bandwidth usage.
When you run the query in Datastax DevCenter 1.6 it displays the 1000 record in result as selected limit but if you export the same result to CSV it will give you all the record which you are looking for.
I run Datastax Devcenter 1.4.
I run the query with a limit and it provides me the actual count.
But LIMIT is limited to maximum value of signed integer (2147483647)
select count(*) from users LIMIT 2147483647;-- ALLOW FILTERING;