Kubernetes configmap with Redis - redis

I was following this tutorial to setup a configmap for a redis.conf. After I create the Redis deployment, I check to ensure that the redis.conf file is in each of the pods, and they are there. The problem is that when go in the redis-cli and check the configuration there, the redis.conf values aren't used. The default values are being used as if the Redis did not start up with the redis.conf file.
redis.conf
maxclients 2000
requirepass "test"
redis-config configmap
{
"apiVersion": "v1",
"data": {
"redis-config": "maxclients 2000\nrequirepass \"test\"\n\n"
},
"kind": "ConfigMap",
"metadata": {
"creationTimestamp": "2018-03-07T15:28:19Z",
"name": "redis-config",
"namespace": "default",
"resourceVersion": "2569562",
"selfLink": "/api/v1/namespaces/default/configmaps/redis-config",
"uid": "29d250ea-221c-11e8-969f-06c0c8d545d2"
}
}
k8 redis manifest.json
{
"kind" : "Deployment",
"apiVersion" : "extensions/v1beta1",
"metadata" : {
"name" : "redis-master",
"creationTimestamp" : null
},
"spec" : {
"replicas" : 2,
"template" : {
"metadata" : {
"creationTimestamp" : null,
"labels" : {
"app" : "redis",
"role" : "master",
"tier" : "backend"
}
},
"spec" : {
"hostNetwork" : true,
"nodeSelector" :{ "role": "cache"},
"containers" : [{
"name" : "master",
"image" : "redis",
"ports" : [{
"containerPort" : 6379,
"hostPort" : 6379,
"protocol" : "TCP"
}
],
"volumeMounts" : [{
"mountPath" : "/redis-master",
"name": "config"
}
],
"resources" : {},
"terminationMessagePath" : "/dev/termination-log",
"imagePullPolicy" : "IfNotPresent"
}],
"volumes" : [{
"name" : "config",
"configMap" : {
"name" : "redis-config",
"items": [{
"key": "redis-config",
"path": "redis.conf"
}]
}
}
],
"restartPolicy" : "Always",
"terminationGracePeriodSeconds" : 30,
"dnsPolicy" : "ClusterFirst",
"securityContext" : {}
}
}
},
"status" : {}
}
Now I know the tutorial uses a Pod kind, and I am using a Deployment kind, but I don't think that is the issue here.

It looks like you are pulling the default redis container. If you check the redis Dokerfiles, for example https://github.com/docker-library/redis/blob/d53b982b387634092c6f11069401679034054ecb/4.0/alpine/Dockerfile, at the bottom, they have:
CMD ["redis-server"]
which will start redis with the default configuration.
Per redis documentation:
https://redis.io/topics/quickstart
under "Starting Redis" section, if you want to provide a different configuration, you would need to start redis with:
redis-server <config file>
Additionally the example in Kubernetes documentation uses a different redis containter:
image: kubernetes/redis
And from the Dokerfile: https://github.com/kubernetes/kubernetes/blob/master/examples/storage/redis/image/Dockerfile, it seems like that one starts Redis with the provided configuration.

Related

Setting API Key required to true using serverless.template in AWS API Gateway

I am deploying an ASP.Net Core project on AWS Lambda and I am struggling with making the API Key required.
Here is my Json template:
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Transform" : "AWS::Serverless-2016-10-31",
"Description" : "An AWS Serverless Application that uses the ASP.NET Core framework running in Amazon Lambda.",
"Parameters" : {
"ShouldCreateBucket" : {
"Type" : "String",
"AllowedValues" : ["true", "false"],
"Description" : "If true then the S3 bucket that will be proxied will be created with the CloudFormation stack."
},
"BucketName" : {
"Type" : "String",
"Description" : "Name of S3 bucket that will be proxied. If left blank a name will be generated.",
"MinLength" : "0"
}
},
"Conditions" : {
"CreateS3Bucket" : {"Fn::Equals" : [{"Ref" : "ShouldCreateBucket"}, "true"]},
"BucketNameGenerated" : {"Fn::Equals" : [{"Ref" : "BucketName"}, ""]}
},
"Resources" : {
"AspNetCoreFunction" : {
"Type" : "AWS::Serverless::Function",
"Properties": {
"Handler": "SmartClockAPI::SmartClockAPI.LambdaEntryPoint::FunctionHandlerAsync",
"Runtime": "dotnetcore2.1",
"CodeUri": "",
"MemorySize": 256,
"Timeout": 30,
"Role": null,
"Policies": [ "AWSLambdaFullAccess","AmazonCognitoPowerUser","AmazonAPIGatewayAdministrator"],
"Environment" : {
"Variables" : {
"AppS3Bucket" : { "Fn::If" : ["CreateS3Bucket", {"Ref":"Bucket"}, { "Ref" : "BucketName" } ] }
}
},
"Events": {
"PutResource": {
"Type": "Api",
"Properties": {
"Path": "/{proxy+}",
"Method": "ANY"
}
}
}
}
},
"BasicUsagePlan" : {
"Type" : "AWS::ApiGateway::UsagePlan",
"Properties" : {
"UsagePlanName" : "Basic plan",
"Quota" : {
"Limit" : 100,
"Period" : "MONTH"
},
"Throttle" : {
"RateLimit" : 10,
"BurstLimit" : 10
}
}
},
"Bucket" : {
"Type" : "AWS::S3::Bucket",
"Condition" : "CreateS3Bucket",
"Properties" : {
"BucketName" : { "Fn::If" : ["BucketNameGenerated", {"Ref" : "AWS::NoValue" }, { "Ref" : "BucketName" } ] }
}
}
},
"Outputs" : {
"ApiURL" : {
"Description" : "API endpoint URL for Prod environment",
"Value" : { "Fn::Sub" : "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/" }
},
"S3ProxyBucket" : {
"Value" : { "Fn::If" : ["CreateS3Bucket", {"Ref":"Bucket"}, { "Ref" : "BucketName" } ] }
}
}
}
What I am trying to achieve is setting this value to true from the Json template.
I was expecting some extra property for Proxy where I can specify this value.
Any ideas?
Use AWS::ApiGateway::UsagePlanKey
{
"Type" : "AWS::ApiGateway::UsagePlanKey",
"Properties" : {
"KeyId" : String,
"KeyType" : String,
"UsagePlanId" : String
}
}
Add a DependsOn to ApiKey, ApiUsagePlan, and ApiUsagePlanKey to ensure they were created in the correct order. This is a nice working example

ELasticsearch Post bulk on elastic xpack role

I have an Elastic cluster with xpack enable.
I'd like to make a backup of all xpack roles created :
GET _xpack/security/role
=> I get a big JSON, ex :
{
"kibana_dashboard_only_user": {
"cluster": [],
"indices": [
{
"names": [
".kibana*"
],
"privileges": [
"read",
"view_index_metadata"
]
}
],
"run_as": [],
"metadata": {
"_reserved": true
},
"transient_metadata": {
"enabled": true
}
},
"watcher_admin": {
"cluster": [
"manage_watcher"
],
"indices": [
{
"names": [
".watches",
".triggered_watches",
".watcher-history-*"
],
"privileges": [
"read"
]
}
],
"run_as": [],
"metadata": {
"_reserved": true
},
"transient_metadata": {
"enabled": true
}
},
....
}
And now I'd like to put it back in the cluster (or another). I cannot just PUT it to _xpack/security/role. If i understand correctly I have to use bulk :
$ curl --user elastic:password https://elastic:9200/_xpack/security/_bulk?pretty -XPOST -H 'Content-Type: application/json' -d '
{"index":{"_index": "_xpack/security/role"}}
{"ROOOOLE" : {"cluster" : [ ],"indices" : [{"names" : [".kibana*"],"privileges" : ["read","view_index_metadata"]}],"run_as" : [ ],"metadata" : {"_reserved" : true},"transient_metadata" : {"enabled" : true}}}
'
But I get an error:
{
"took" : 3,
"errors" : true,
"items" : [
{
"index" : {
"_index" : "_xpack/security/role",
"_type" : "security",
"_id" : null,
"status" : 400,
"error" : {
"type" : "invalid_index_name_exception",
"reason" : "Invalid index name [_xpack/security/role], must not contain the following characters [ , \", *, \\, <, |, ,, >, /, ?]",
"index_uuid" : "_na_",
"index" : "_xpack/security/role"
}
}
}
]
}
Is there a way to do this easily? Or do I have to parse the JSON, and put each role one by one to:
_xpack/security/role/rolexxx
_xpack/security/role/roleyyy
...
More globally, is there a way to get all data of an index (config index), then upload it back or put it into another cluster?

AWS cloudformation : how to properly create a redis cache cluster

I want to create an elasticache instance using redis.
I think that I should use it "cluster mode disabled" because everything will fit into one server.
In order to not have a SPOF, I want to create a read replica that will be promoted by AWS in case of a failure of the master.
If possible, it would be great to balance the read only operations between master and slave, but it is not mandatory.
I created a functioning master/read-replica using the aws console then used cloudformer to create the cloudformation json conf. Cloudformer has created me two unlinked AWS::ElastiCache::CacheCluster, but by reading the doc. I don't understand how to link them... For now I have this configuration :
{
"cachehubcache001": {
"Type": "AWS::ElastiCache::CacheCluster",
"Properties": {
"AutoMinorVersionUpgrade": "true",
"AZMode": "single-az",
"CacheNodeType": "cache.t2.small",
"Engine": "redis",
"EngineVersion": "3.2.4",
"NumCacheNodes": "1",
"PreferredAvailabilityZone": { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "Az1B"]},
"PreferredMaintenanceWindow": "sun:04:00-sun:05:00",
"CacheSubnetGroupName": {
"Ref": "cachesubnethubprivatecachesubnetgroup"
},
"VpcSecurityGroupIds": [
{
"Fn::GetAtt": [
"sgiHubCacheSG",
"GroupId"
]
}
]
}
},
"cachehubcache002": {
"Type": "AWS::ElastiCache::CacheCluster",
"Properties": {
"AutoMinorVersionUpgrade": "true",
"AZMode": "single-az",
"CacheNodeType": "cache.t2.small",
"Engine": "redis",
"EngineVersion": "3.2.4",
"NumCacheNodes": "1",
"PreferredAvailabilityZone": { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "Az1A"]},
"PreferredMaintenanceWindow": "sun:02:00-sun:03:00",
"CacheSubnetGroupName": {
"Ref": "cachesubnethubprivatecachesubnetgroup"
},
"VpcSecurityGroupIds": [
{
"Fn::GetAtt": [
"sgiHubCacheSG",
"GroupId"
]
}
]
}
},
}
I know that it is wrong, but I can't figure out how to create a correct replica. I can't understand the AWS doc, for a start I can't figure out wich Type I should use between :
http://docs.aws.amazon.com/fr_fr/AWSCloudFormation/latest/UserGuide/aws-resource-elasticache-replicationgroup.html
http://docs.aws.amazon.com/fr_fr/AWSCloudFormation/latest/UserGuide/aws-properties-elasticache-cache-cluster.html
Since cloudformer created AWS::ElastiCache::CacheCluster I'll go with it, but I've got the feeling that it should have created only one resource, and used the NumCacheNodes parameter in order to create two resources.
redis can't use :
NumCacheNodes
AZMode and PreferredAvailabilityZones
so I don't know how to make this solution multi-AZ...
I managed to do this using AWS::ElastiCache::ReplicationGroup, the NumCacheClusters parameter provide the possibility to have numerous servers. Beware : it seem that you have to handle the connection to master/slave yourself (but in case of a master's failure, aws should normally detect it and change the dns of a slave to permit you point do not change your configuration). here is a sample :
"hubElastiCacheReplicationGroup" : {
"Type" : "AWS::ElastiCache::ReplicationGroup",
"Properties" : {
"ReplicationGroupDescription" : "Hub WebServer redis cache cluster",
"AutomaticFailoverEnabled" : "false",
"AutoMinorVersionUpgrade" : "true",
"CacheNodeType" : "cache.t2.small",
"CacheParameterGroupName" : "default.redis3.2",
"CacheSubnetGroupName" : { "Ref": "cachesubnethubprivatecachesubnetgroup" },
"Engine" : "redis",
"EngineVersion" : "3.2.4",
"NumCacheClusters" : { "Ref" : "ElasticacheRedisNumCacheClusters" },
"PreferredMaintenanceWindow" : "sun:04:00-sun:05:00",
"SecurityGroupIds" : [ { "Fn::GetAtt": ["sgHubCacheSG", "GroupId"] } ]
}
},

restart jobtracker through cloudera manager API

I am trying to restart Mapreduce Jobtracker through Cloudera Manager API. Stats for Jobtracker is as follows :
local-iMac-399:$ curl -u 'admin:admin' 'http://hadoop-namenode.dev.com:7180/api/v6/clusters/Cluster%201/services/mapreduce/roles/mapreduce-JOBTRACKER-0675ebab2b87e3869e0d90167cf4bf86'
{
"name" : "mapreduce-JOBTRACKER-0675ebab2b87e3869e0d90167cf4bf86",
"type" : "JOBTRACKER",
"serviceRef" : {
"clusterName" : "cluster",
"serviceName" : "mapreduce"
},
"hostRef" : {
"hostId" : "24259373-7e71-4089-8251-faf055e42ad7"
},
"roleUrl" : "http://hadoop-namenode.dev.com:7180/cmf/roleRedirect/mapreduce-JOBTRACKER-0675ebab2b87e3869e0d90167cf4bf86",
"roleState" : "STARTED",
"healthSummary" : "GOOD",
"healthChecks" : [ {
"name" : "JOB_TRACKER_FILE_DESCRIPTOR",
"summary" : "GOOD"
}, {
"name" : "JOB_TRACKER_GC_DURATION",
"summary" : "GOOD"
}, {
"name" : "JOB_TRACKER_HOST_HEALTH",
"summary" : "GOOD"
}, {
"name" : "JOB_TRACKER_LOG_DIRECTORY_FREE_SPACE",
"summary" : "GOOD"
}, {
"name" : "JOB_TRACKER_SCM_HEALTH",
"summary" : "GOOD"
}, {
"name" : "JOB_TRACKER_UNEXPECTED_EXITS",
"summary" : "GOOD"
}, {
"name" : "JOB_TRACKER_WEB_METRIC_COLLECTION",
"summary" : "GOOD"
} ],
"configStalenessStatus" : "STALE",
"haStatus" : "ACTIVE",
"maintenanceMode" : false,
"maintenanceOwners" : [ ],
"commissionState" : "COMMISSIONED",
"roleConfigGroupRef" : {
"roleConfigGroupName" : "mapreduce-JOBTRACKER-BASE"
}
}
local-iMac-399:$
Dont know How do I use API to restart just Jobtracker ?
I tried to restart Hive service using following command but got some error
local-iMac-399:$curl -X POST -u 'admin:admin' 'http://hadoop-namenode.dev.com:7180/api/v6/clusters/Cluster%201/services/hive/roleCommands/restart'
{
"message" : "No content to map due to end-of-input\n at [Source: org.apache.cxf.transport.http.AbstractHTTPDestination$1#4169c499; line: 1, column: 1]"
}
I would appreciate if someone help in understanding how to use Cloudera Manager API
Based on the information provided, this is how you'd invoke the CM API JobTracker restart
curl -u 'admin:admin' -X POST -H "Content-Type:application/json" -d '{"items":["mapreduce-JOBTRACKER-0675ebab2b87e3869e0d90167cf4bf86"]}' 'http://hadoop-namenode.dev.com:7180/api/v6/clusters/Cluster%201/services/mapreduce/roleCommands/restart'

How can I access protected S3 files in a CFN script?

I am trying to retrieve a file in my cloudformation script. If I make the file publicly available, then it works fine. If the file is private, then the cfn script fails, but with a 404 error in /var/log/. Trying to retrieve the file via wget results in the appropriate 403 error.
How can I retrieve private files from S3?
My file clause looks like:
"files" : {
"/etc/httpd/conf/httpd.conf" : {
"source" : "https://s3.amazonaws.com/myConfigBucket/httpd.conf"
}
},
I added an authentication clause and appropriate parameter:
"Parameters" : {
"BucketRole" : {
"Description" : "S3 role for access to bucket",
"Type" : "String",
"Default" : "S3Access",
"ConstraintDescription" : "Must be a valid IAM Role"
}
}
"AWS::CloudFormation::Authentication": {
"default" : {
"type": "s3",
"buckets": [ "myConfigBucket" ],
"roleName": { "Ref" : "BucketRole" }
}
},
My IAM Role looks like:
{
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": "*"
}
]
}
The solution is to add an IamInstanceProfile property to the instance creation:
"Parameters" : {
...
"RoleName" : {
"Description" : "IAM Role for access to S3",
"Type" : "String",
"Default" : "DefaultRoleName",
"ConstraintDescription" : "Must be a valid IAM Role"
}
},
"Resources" : {
"InstanceName" : {
"Type" : "AWS::EC2::Instance",
"Properties" : {
"ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "64"] },
"InstanceType" : { "Ref" : "InstanceType" },
"SecurityGroups" : [ {"Ref" : "SecurityGroup"} ],
"IamInstanceProfile" : { "Ref" : "RoleName" },
"KeyName" : { "Ref" : "KeyName" }
}
},
...