I've been trying to set the TTL for Redis Key from within Drupal, but it does not seem be working. Our settings.php is as follows:
// Set Redis as the default backend for any cache bin not otherwise specified.
$settings['cache']['default'] = 'cache.backend.redis';
$settings['redis.connection']['interface'] = 'PhpRedis';
$settings['redis.connection']['host'] = '127.0.0.1';
$settings['redis.connection']['port'] = '6379';
$settings['redis.connection']['base'] = 8; // This is the DB ID.
#$conf['cache_lifetime_redis'] = 21600;
#$settings['redis.settings']['perm_ttl_cache_field'] = "3 months";
$conf['redis_perm_ttl'] = "3 months";
I've tried with $conf['cache_lifetime_redis'] = 21600;, with $settings['redis.settings']['perm_ttl_cache_field'] = "3 months"; and $conf['redis_perm_ttl'] = "3 months"; to no avail. When connecting to REDIS, the TTL is still set to 1 year. What else needs to be done to make this work ? settings this at the server level is a no-go, due to being a shared environment.
Related
I'm trying to integrate a Datadog monitor check on sshd process in my terraform codebase, but I'm getting datadog_monitor.host_is_up2: error updating monitor: API error 400 Bad Request: {"errors":["The value provided for parameter 'query' is invalid"]}
What I did was to copy the monitor's query I created on the Datadog panel and pasted it into the tf file:
resource "datadog_monitor" "host_is_up2" {
name = "host is up"
type = "metric alert"
message = "Monitor triggered"
escalation_message = "Escalation message"
query = "process.up.over('process:ssh').last(4).count_by_status()"
thresholds {
ok = 0
warning = 1
critical = 2
}
notify_no_data = false
renotify_interval = 60
notify_audit = false
timeout_h = 60
include_tags = true
silenced {
"*" = 0
}
}
ofc the query example "avg(last_1h):avg:aws.ec2.cpu{environment:foo,host:foo} by {host} > 2" works
What's the right way to check via Datadog API or terraform if a specific service, like sshd, is up or not?
There are two error in your code:
The type used is wrong. It should be service check instead of metric alert.
You need to enclose process.up in a pair of ''.
Once done, your code will run flawlessly.
I have checked these questions but they did not help me to fix my issue. I am using Redis as a key value store for Rate Limiting in my Spring REST application using spring-data-redis library. I test with huge load. In that I use the following code to store a key and I am setting the expire time as well. Most of the time the key expires as expected. But some times the key is not expiring!
code snippet
RedisAtomicInteger counter = counter = new RedisAtomicInteger("mykey");
counter.expire(1, TimeUnit.MINUTES);
I checked the availability of the keys using redis-cli tool
keys *
and
ttl keyname
redis.conf having default values.
Any suggestions ?
Edit 1:
Full code:
The function is in an Aspect
public synchronized Object checkLimit(ProceedingJoinPoint joinPoint) throws Exception, Throwable {
boolean isKeyAvailable = false;
List<String> keysList = new ArrayList<>();
Object[] obj = joinPoint.getArgs();
String randomKey = (String) obj[1];
int randomLimit = (Integer) obj[2];
// for RedisTemplate it is already loaded as
// #Autowired
// private RedisTemplate template;
// in this class
Set<String> redisKeys = template.keys(randomKey+"_"randomLimit+"*");
Iterator<String> it = redisKeys.iterator();
while (it.hasNext()) {
String data = it.next();
keysList.add(data);
}
if (keysList.size() > 0) {
isKeyAvailable = keysList.get(0).contains(randomKey + "_" + randomLimit);
}
RedisAtomicInteger counter = null;
// if the key is not there
if (!isKeyAvailable) {
long expiryTimeStamp = 0;
int timePeriodInMintes = 1;
expiryTimeStamp = new Date(System.currentTimeMillis() + timePeriodInMintes * 60 * 1000).getTime();
counter = new RedisAtomicInteger(randomKey+ "_"+ randomLimit + "_" + expiryTimeStamp,template.getConnectionFactory());
counter.incrementAndGet();
counter.expire(timePeriodInMintes, TimeUnit.MINUTES);
break;
} else {
String[] keys = keysList.get(0).split("_");
String rLimit = keys[1];
counter = new RedisAtomicInteger(keysList.get(0), template.getConnectionFactory());
int count = counter.get();
// If count exceeds throw error
if (count != 0 && count >= Integer.parseInt(rLimit)) {
throw new Exception("Error");
}
else {
counter.incrementAndGet();
}
}
return joinPoint.proceed();
}
when these lines run
RedisAtomicInteger counter = counter = new RedisAtomicInteger("mykey");
counter.expire(1, TimeUnit.MINUTES);
I can see
75672562.380127 [0 10.0.3.133:65462] "KEYS" "mykey_1000*"
75672562.384267 [0 10.0.3.133:65462] "GET" "mykey_1000_1475672621787"
75672562.388856 [0 10.0.3.133:65462] "SET" "mykey_1000_1475672621787" "0"
75672562.391867 [0 10.0.3.133:65462] "INCRBY" "mykey_1000_1475672621787" "1"
75672562.395922 [0 10.0.3.133:65462] "PEXPIRE" "mykey_1000_1475672621787" "60000"
...
75672562.691723 [0 10.0.3.133:65462] "KEYS" "mykey_1000*"
75672562.695562 [0 10.0.3.133:65462] "GET" "mykey_1000_1475672621787"
75672562.695855 [0 10.0.3.133:65462] "GET" "mykey_1000_1475672621787"
75672562.696139 [0 10.0.3.133:65462] "INCRBY" "mykey_1000_1475672621787" "1"
in Redis log, when I "MONITOR" it in
Edit:
Now with the updated code I believe your methodology is fundamentally flawed aside from what you're reporting.
The way you've implemented it you require running KEYS in production - this is bad. As you scale out you will be causing a growing, and unnecessary, system blocking load on the server. As every bit of documentation on it says, do not use keys in production. Note that encoding the expiration time in the key name gives you no benefit. If you made that part of the key name a the creation timestamp, or even a random number nothing would change. Indeed, if you removed that bit, nothing would change.
A more sane route would instead be to use a keyname which is not time-dependent. The use of expiration handles that function for you. Let us call your rate-limited thing a "session". Your key name sans the timestamp is the "session ID". By setting an expiration of 60s on it, it will no longer be available at the 61s mark. So you can safely increment and compare the result to your limit without needing to know the current time or expiry time. All you need is a static key name and an appropriate expiration set on it.
If you INCR a non-existing key, Redis will return "1" meaning it created the key and incremented it in a single step/call. so basically the logic goes like this:
create "session" ID
increment counter using ID
compare result to limit
if count == 1, set expiration to 60s
id count > limit, reject
Step 3.1 is important. A count of 1 means this is a new key in Redis, and you want to set your expiration on it. Anything else means the expiration should already have been set. If you set it in 3.2 you will break the process because it will preserve the counter for more than 60s.
With this you don't need to have dynamic key names based on expiration time, and thus don't need to use keys to find out if there is an existing "session" for the rate-limited object. It also makes your code much simpler and predictable, as well as reduce round trips to Redis - meaning it will be lower load on Redis and perform better. As to how to do that w/the client library you're using I can't say because I'm not that familiar with it. But the basic sequence should be translatable to it as it is fairly basic and simple.
What you haven't shown, however, is anything to support the assertion that the expiration isn't happening. All you've done is show that Redis is indeed being told to and setting an expiration. In order to support your claim you need to show that the key does not expire. Which means you need to show retrieval of the key after the expiration time, and that the counter was not "reset" by being recreated after the expiration. One way you can see the expiration is happening is to use keyspace notifications. With that you will be able to see Redis saying a key was expired.
Where this process will fail a bit is if you do multiple windows for rate-limiting, or if you have a much larger window (ie. 10 minutes) in which case sorted sets might be a more sane option to prevent front-loading of requests - if desired. But as your example is written, the above will work just fine.
I'm able to configure a DataSource Resource in TomEE by modifying the "conf/tomee.xml" file. However, it's sort of awkward to automate this modification, as I have to insert the DataSource definition before the "" line. I heard from a comment in a related SO posting from me that it's easier to append to the "system.properties" file.
So, I tried translating this:
<Resource id="sus2" type="DataSource">
JdbcDriver = oracle.jdbc.driver.OracleDriver
MaxActive = 10
MinIdle = 2
MaxIdle = 2
MaxWait = 10000
JdbcUrl = jdbc:oracle:thin:#${DB_HOST}:${DB_PORT}:${DB_SID}
UserName = ${DB_USER}
Password = ${DB_PASSWORD}
</Resource>
Which works, to the following:
db = new://Resource?type=DataSource
db.id = Resource/sus2
db.JdbcDriver = oracle.jdbc.driver.OracleDriver
db.MaxActive = 10
db.MinIdle = 2
db.MaxIdle = 2
db.MaxWait = 10000
db.JdbcUrl = jdbc:oracle:thin:#${DB_HOST}:${DB_PORT}:${DB_SID}
db.UserName = ${DB_USER}
db.Password = ${DB_PASSWORD}
which does not work. It fails, saying it couldn't find the "Resource/sus2" resource.
The configuration reference can be found at http://tomee.apache.org/ng/admin/configuration/resources.html
You have to understand that XML attributes becomes URI query parameters then I think it will work.
In other words:
db = new://Resource?type=DataSource
becomes
sus2 = new://Resource?type=DataSource
and your db.id doesn't do anything - I think it is logged.
In short: replace all your "db" by "sus2" and it will work
I have a setup where celery_result_backend has been configured to 'amqp'. I can see my tasks getting executed by the worker in logs. But
It is creating the queue with task id but its status is expired.I am not getting the result (result = AsyncResult(taskid); result.get() hangs). I tried all the backed supported:
1)Mysql: It is not putting data to the celery created tables
2) Redis: It is not putting data to the db
I two centos system.
1) I am calling the delay method to send the task to proper rabbitmq. And the worker is listening to the queue, from there it will pick the task and process(I can see task in the queue and getting executed by the worker in machine 2 But the result is not being put into the backend.
).Here I am doing the result.get() It hangs.
2) The worker is running on it to execute the task.It executes the task but I think not able to put the rersult
Settings:
RABBITMQ_BROKER_HOST = '10.213.166.133'
RABBITMQ_BROKER_PORT = dqms_settings.RABBITMQ_BROKER_PORT
RABBITMQ_BROKER_VHOST = dqms_settings.RABBITMQ_BROKER_VHOST
RABBITMQ_BROKER_USERNAME = dqms_settings.RABBITMQ_BROKER_USERNAME
RABBITMQ_BROKER_PASSWORD = dqms_settings.RABBITMQ_BROKER_PASSWORD
BROKER_URL = 'amqp://%s:%s#%s:%s/%s' % (RABBITMQ_BROKER_USERNAME,
RABBITMQ_BROKER_PASSWORD,
RABBITMQ_BROKER_HOST,
RABBITMQ_BROKER_PORT,
RABBITMQ_BROKER_VHOST)
#CELERY_TASK_RESULT_EXPIRES = 18000
#CELERY_IGNORE_RESULT = True
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
#CELERY_RESULT_BACKEND = 'db+mysql://svcacct-dqms:s3cretP#ssw0rd#10.213.166.202:3306/dqms'
#CELERY_RESULT_BACKEND = 'amqp'
#CELERY_AMQP_TASK_RESULT_EXPIRES = 1000
#CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = TIME_ZONE
CELERYD_PREFETCH_MULTIPLIER = dqms_settings.CELERYD_PREFETCH_MULTIPLIER
CELERY_DEFAULT_QUEUE = dqms_settings.CELERY_DEFAULT_QUEUE
CELERY_DEFAULT_EXCHANGE_TYPE = dqms_settings.CELERY_DEFAULT_EXCHANGE_TYPE
CELERY_DEFAULT_ROUTING_KEY = dqms_settings.CELERY_DEFAULT_ROUTING_KEY
CELERY_QUEUES = dqms_settings.CELERY_QUEUES
CELERY_ROUTES = dqms_settings.CELERY_ROUTES
CELERYD_HIJACK_ROOT_LOGGER = dqms_settings.CELERYD_HIJACK_ROOT_LOGGER
CELERY_ACKS_LATE = dqms_settings.CELERY_ACKS_LATE
CELERY_RESULT_BACKEND = 'redis://:s3cretP#ssw0rd#10.213.166.204:6379/5' #'djcelery.backends.database.DatabaseBackend'
#CELERY_REDIS_MAX_CONNECTIONS = 6
#CELERY_ALWAYS_EAGER = False
Can some one help why it is not putting the result in the queue?
This is a issue which is happening quite common now.
setting CELERY_ALWAYS_EAGER to TRUE will do the work
However this is not the best solution in production scenario.
I am attempting to replicate a SQL CE 3.5 SP1 database but upon syncrhonization, I am thrown the following error:
"Failure to connect to SQL Server with provided connection information. SQL Server does not exist, access is denied because the IIS user is not a valid user on the computer running SQL Server, or the password is incorrect."
I am using the Windows Mobile 6 Professional emulator and the machine I am attempting to connect to is a Windows Virtual Machine running Windows XP Professional SP3. I have configured the network adapter settings for the emulator (I can access web pages), verified user permissions, double checked IIS settings, and triple checked my connection string:
SqlCeReplication rpl = null;
try
{
// Creates the replication object.
rpl = new SqlCeReplication();
// Establishes the connection string.
rpl.SubscriberConnectionString = #"Data Source = \Program Files\ParkSurvey\ParkSurvey.sdf; Password = *; Temp File Max Size = 512;
Max Database Size = 512; Max Buffer Size = 512; Flush Interval = 20; Autoshrink Threshold = 10; Default Lock Escalation = 100";
// Sets the Publisher properties.
rpl.PublisherSecurityMode = SecurityType.NTAuthentication;
rpl.Publisher = "PUBLISHER";
rpl.PublisherLogin = "INDICOPUBLIC\\subuser";
rpl.PublisherPassword = "*";
rpl.PublisherDatabase = "PUBLISHER";
rpl.Publication = "ParkSurveyPublication";
// Sets the internet replication properties.
rpl.InternetUrl = "http://replication/sqlce/sqlcesa35.dll";
rpl.InternetLogin = "INDICOPUBLIC\\subuser";
rpl.InternetPassword = "*";
rpl.ConnectionManager = true;
// Sets the Distributor properties.
rpl.Distributor = "PUBLISHER";
rpl.DistributorLogin = "INDICOPUBLIC\\subuser";
rpl.DistributorPassword = "psrAdmin";
rpl.DistributorSecurityMode = SecurityType.NTAuthentication;
// Sets the timeout properties.
rpl.ConnectionRetryTimeout = 120;
rpl.ConnectTimeout = 6000;
rpl.ReceiveTimeout = 6000;
rpl.SendTimeout = 6000;
// Sets the Subscriber properties.
rpl.Subscriber = "ParkSurveySubscriber";
rpl.HostName = "Mobile1";
rpl.CompressionLevel = 6;
rpl.ExchangeType = ExchangeType.BiDirectional;
// Call the replication methods.
rpl.Synchronize();
}
catch (SqlCeException sqlEx)
{
MessageBox.Show(sqlEx.Message);
}
finally
{
// Disposing the replication object
if (rpl != null)
{
rpl.Dispose();
}
}
I have also attempted to open the host machine itself in File Explorer on the mobile emulator and am prompted that "The network path was not found.". This leads me to believe it is ActiveSync issue within the emulator itself. Does anyone have any advice?
Try with IP adresse instead of hostname, and test the agent URL from IE on the device. Make sure to use the latest build of 3.5 SP2 on all components if your DB server is SQL 2012