When using OpenShift Aggregated Logging I get logs nicely fed into elasticsearch. However, the line as logged by apache ends up in a message field.
I'd like to create queries in Kibana where I can access the url, the status code and other fields individually. For that the special apache access log parsing needs to be done.
How can I do that?
This is an example entry as seen in kibana:
{
"_index": "42-steinbruchsteiner-staging.3af0bedd-eebc-11e6-af4b-005056a62fa6.2017.03.29",
"_type": "fluentd",
"_id": "AVsY3aSK190OXhxv4GIF",
"_score": null,
"_source": {
"time": "2017-03-29T07:00:25.595959397Z",
"docker_container_id": "9f4fa85a626d2f5197f0028c05e8e42271db7a4c674cc145204b67b6578f3378",
"kubernetes_namespace_name": "42-steinbruchsteiner-staging",
"kubernetes_pod_id": "56c61b65-0b0e-11e7-82e9-005056a62fa6",
"kubernetes_pod_name": "php-app-3-weice",
"kubernetes_container_name": "php-app",
"kubernetes_labels_deployment": "php-app-3",
"kubernetes_labels_deploymentconfig": "php-app",
"kubernetes_labels_name": "php-app",
"kubernetes_host": "itsrv1564.esrv.local",
"kubernetes_namespace_id": "3af0bedd-eebc-11e6-af4b-005056a62fa6",
"hostname": "itsrv1564.esrv.local",
"message": "10.1.3.1 - - [29/Mar/2017:01:59:21 +0200] "GET /kwf/status/health HTTP/1.1" 200 2 "-" "Go-http-client/1.1"\n",
"version": "1.3.0"
},
"fields": {
"time": [
1490770825595
]
},
"sort": [
1490770825595
]
}
Disclaimer: I did not test this out in openshift. I don't know which tech stack you are using for your microservice.
This is how I do this in a spring boot application (with logback) deployed in Kubernetes.
1. Use logstash encoder for logback (This will write logs in Json format which is more ELK stack friendly)
I have a gradle dependency to enable this
compile "net.logstash.logback:logstash-logback-encoder:3.5"
Then configure LogstashEncoder as encoder in the appender, in logback-spring.groovy/logback-spring.xml (or logabck.xml)
2. Have some filters or libraries to write the access log
For 2. Either use
A. Use "net.rakugakibox.springbootext:spring-boot-ext-logback-access:1.6" library
(This is what I am using)
It gives in a nice json format, as follows
{
"#timestamp":"2017-03-29T09:43:09.536-05:00",
"#version":1,
"#message":"0:0:0:0:0:0:0:1 - - [2017-03-29T09:43:09.536-05:00] \"GET /orders/v1/items/42 HTTP/1.1\" 200 991",
"#fields.method":"GET",
"#fields.protocol":"HTTP/1.1",
"#fields.status_code":200,
"#fields.requested_url":"GET /orders/v1/items/42 HTTP/1.1",
"#fields.requested_uri":"/orders/v1/items/42",
"#fields.remote_host":"0:0:0:0:0:0:0:1",
"#fields.HOSTNAME":"0:0:0:0:0:0:0:1",
"#fields.content_length":991,
"#fields.elapsed_time":48,
"HOSTNAME":"ABCD"
}
OR
B. Use Logback's Tee Filter
OR
C. Spring's CommonsRequestLoggingFilter (Did not really test this out)
Add a bean definition
#Bean
public CommonsRequestLoggingFilter requestLoggingFilter() {
CommonsRequestLoggingFilter crlf = new CommonsRequestLoggingFilter();
crlf.setIncludeClientInfo(true);
crlf.setIncludeQueryString(true);
crlf.setIncludePayload(true);
return crlf;
}
Then set org.springframework.web.filter.CommonsRequestLoggingFilter to DEBUG, this can be done using the application.properties by adding:
logging.level.org.springframework.web.filter.CommonsRequestLoggingFilter=DEBUG
Related
Im trying to learn more about the Revit API as it works with BIM 360. I know I can determine isCloudInModel (which is available in the 2019 API, not the 2018 - and hopefully its available intentionally this time) and get the path using these strategies https://forums.autodesk.com/t5/revit-api-forum/determine-central-file-type-file-server-our-cloud/td-p/6506907 .
BIM 360 models don't seem to subscribe to Sync events, Im struggling to understand if I can subscribe to any events that have to do with synchronization, saving, opening or anything else.
Is there a reference for API and Limitations of models that are hosted on BIM 360?
Through the Application/ControlledApplication classes you have methods for this. You just need to have the logic for seeing if the application that is triggering the DocumentSynchronizing, DocumentSaving type Events is a BIM360 model. Which you can do with how you find the path.
With the Forge API, through the Data Management API, you can use the PostModelJob endpoint which has this functionality:
Every time you synchronize the current model with the central model, it sets the data attribute to null. When you publish the latest version to BIM 360 Docs (using the PublishModel command), it sets the status to processing or complete.
This will give you the information on if the model has been synced. You also have to be using BIM 360 Docs and not BIM 360 Team.
Successful Retrieval of C4R Publish Status - Model Needs Publishing (200)
Note that if you have updated the central model, the data attribute is set to null until you publish it.
Request
curl -X POST -v "https://developer.api.autodesk.com/data/v1/projects/b.project.id.xyz/commands/" -H "Authorization: Bearer kEnG562yz5bhE9igXf2YTcZ2bu0z" -H "Content-Type: application/vnd.api+json" -d '
{
"jsonapi": {
"version": "1.0"
},
"data": {
"type": "commands",
"attributes": {
"extension": {
"type": "commands:autodesk.bim360:C4RModelGetPublishJob",
"version": "1.0.0"
}
},
"relationships": {
"resources": {
"data": [ { "type": "items", "id": "urn:adsk.wip:dm.file:hC6k4hndRWaeIVhIjvHu8w" } ]
}
}
}
}'
Show Less
Response
{
"data": null
"jsonapi": {
"version": "1.0"
}
}
I am trying to get started with a monitoring server solution. I got the Sensu Clients, RabbitMQ and Uchiwa configured but then I tried using Graphite but there were so many parts to configure I tried InfluxDB instead. I am stuck configuring Sensu to InfluxDB.
Is there a part missing in the below configuration?
Client [Sensu] > RabbitMQ <> Sensu Server <> InfluxDB <> Grafana
Any suggestions?
cat influx.json
{
"influxdb": {
"hosts" : ["192.168.1.1"],
"host" : "192.168.1.1",
"port" : "8086",
"database" : "sensumetrics",
"time_precision": "s",
"use_ssl" : false,
"verify_ssl" : false,
"initial_delay" : 0.01,
"max_delay" : 30,
"open_timeout" : 5,
"read_timeout" : 300,
"retry" : null,
"prefix" : "",
"denormalize" : true,
"status" : true
}
}
cat handler.json
{
"handlers": {
"influxdb": {
"type": "pipe",
"command": "/opt/sensu/embedded/bin/metrics-influxdb.rb"
}}}
checks1,
{
"checks": {
"check_memory_linux": {
"handlers": ["influxdb","default"],
"command": "/opt/sensu/embedded/bin/check-memory-percent.rb -w 90 -c 95",
"interval": 60,
"occurrences": 5,
"subscribers": [ "TEST" ]
}}}
checks2,
{
"checks": {
"check_cpu_linux-elkctrl-pipe": {
"type": "metric",
"command": "/opt/sensu/embedded/bin/check-cpu.rb -w 80 -c 90",
"subscribers": ["TEST"],
"interval": 10,
"handlers": ["debug","influxdb"]
}}}
To use InfluxDB to persist your data, you must have:
InfluxDB plugin installed (also, installation and usage instructions here)
Definitions for the plugin (an influxdb.json containin at least the host, port, user, password and database to be used by Sensu)
The definition, as other config files, must be in /etc/sensu/conf.d/
Handler configuration set properly (also in conf.d)
Mutator for InfluxDB (extensions)
Your checks must send results to the handler, so their definition must contain:
"handlers": [
"influxdb"
]
Or whatever name you gave your handler.
Case, if the influxdb config you provided above is the full extent of your configuration, it would seem to be missing the username/password attributes required by the influxdb configuration. If they're present, but not provided in the post, no big deal. However, I'd recommend doing the following for your Sensu logs:
grep -i influxdb /var/logs/sensu/sensu-server.log
And seeing if the check result is getting sent to your influxdb instance. If they are, you should be receiving an error that might be pointing a bit more to what's going on.
You can also check your influxdb logs to see if they're getting a post from your Sensu server:
journalctl -u influxdb.service -f
But yeah, if the username/password is missing from the configuration, that'd be the first place that I start.
I recently went through the tutorial for load balancing apps in DCOS using marathon-lb (in the example they balance some nginx containers: https://dcos.io/docs/1.9/networking/marathon-lb/marathon-lb-advanced-tutorial/). I am trying to use this approach to internally load balance my own custom application. The custom app I am using is a play scala app. I have the internal marathon-lb set up and can successfully use it for the nginx container but when I try to use my own docker image I cannot get this to work. I start up my service with my custom image and I can access the service fine by using the IP and port that gets assigned to it (i.e. if the service gets deployed on 10.0.0.0 and is available on port 1234 then curl http://10.0.0.0:1234/ works as expected and I can also make my api calls as defined in my application routes). However, when I try to access the app through the load balancer (curl -i http://marathon-lb-internal.marathon.mesos:10002, where 10002 is the service port) then I get this message:
HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
For reference, here is my json file I'm using to start my custom service:
{
"id": "my-app",
"container": {
"type": "DOCKER",
"docker": {
"image": "my_repo/my_image:1.0.0",
"network": "BRIDGE",
"portMappings": [
{ "hostPort": 0, "containerPort": 9000, "servicePort": 10002, "protocol": "tcp" }
],
"parameters": [
{ "key": "env", "value": "USER_NAME=user" },
{ "key": "env", "value": "USER_PASSWORD=password" }
],
"forcePullImage": true
}
},
"instances": 1,
"cpus": 1,
"mem": 1000,
"healthChecks": [{
"protocol": "HTTP",
"path": "/v1/health",
"portIndex": 0,
"timeoutSeconds": 10,
"gracePeriodSeconds": 10,
"intervalSeconds": 2,
"maxConsecutiveFailures": 10
}],
"labels":{
"HAPROXY_GROUP":"internal"
},
"uris": [ "https://s3.amazonaws.com/my_bucket/my_docker_credentials" ]
}
I had the same problem and found the solution here
marathon-lb health check failing on all spray.io containers
Need to add
"HAPROXY_0_BACKEND_HTTP_HEALTHCHECK_OPTIONS": " http-send-name-header Host\n timeout check {healthCheckTimeoutSeconds}s\n"
To your config so that the REST layer doesn't bark on the health check from marathon
Question: how can I split swagger definition across files? What are the possibilities in that area? The question details are described below:
example of what I want - in RAML
I do have experience in RAML and what I do is, for example:
/settings:
description: |
This resource defines application & components configuration
get:
is: [ includingCustomHeaders ]
description: |
Fetch entire configuration
responses:
200:
body:
example: !include samples/settings.json
schema: !include schemas/settings.json
The last two lines are important here - theones with !include <filepath> - in RAML I can split my entire contract into many files that just get included dynamically by the RAML parser (and RAML parser is used by all tools that base on RAML).
My benefit from this is that:
I get my contract more clear and easier to maintain, because schemas are not inline
but that's really important: I can reuse the schema files within other tools to do validation, mock generation, stubs, generate tests, etc. In other words, this way I can reuse schema files within both contract (RAML, this case) and other tools (non-RAML, non-swagger, just JSONschema-based ones).
back to Swagger
As far as I read, swagger supports $ref keyword which allows to load external files. But is that files fetched through HTTP/AJAX or can they just be local files?
And is that supported by the whole specification or is it just some tools that support it and some that don't?
What I found here is that the input for swagger has to be one file. And this is extremely inconvenient for big projects:
because of size
and because I can't reuse the schema if I want to use something non-swagger
Or, in other words, can I achieve the same with swagger, that I can with RAML - in terms of splitting files?
The specification allows for references in multiple locations but not everywhere. These references are resolved depending on where the specification is being hosted--and what you're trying to do.
For something like rendering a dynamic user interface, then yes you do need to eventually load the entire definition into "a single object" which may be composed from many files. If performing a code generation, the definitions may be loaded directly from the file system. But ultimately there are swagger parsers doing the resolution, which is much more fine grained and controllable in Swagger than other definition formats.
In your case, you would use a JSON pointer to the schema reference:
responses:
200:
description: the response
schema:
via local reference
$ref: '#/definitions/myModel'
via absolute reference:
$ref: 'http://path/to/your/resource'
via relative reference, which would be 'relative to where this doc is loaded'
$ref: 'resource.json#/myModel
via inline definition
type: object
properties:
id:
type: string
When I split OpenAPI V3 files using references, I try to avoid the sock drawer anti-pattern and instead use functional groupings for the YAML files.
I also make it so that each YAML file itself is a valid OpenAPI V3 spec.
I start out with the openapi.yaml file.
openapi: 3.0.3
info:
title: MyAPI
description: |
This is the public API for my stuff.
version: "3"
tags:
# NOTE: the name is needed as the info block uses `title` rather than name
- name: Authentication
$ref: 'authn.yaml#/info'
paths:
# NOTE: here are the references to the other OpenAPI files
# from the path. Note because OpenAPI requires paths to
# start with `/` and that is already used as a separator
# replace the `/` with `%2F` for the path reference.
'/authn/start':
$ref: 'authn.yaml#/paths/%2Fstart'
Then in the functional group:
openapi: 3.0.3
info:
title: Authentication
description: |
This is the authentication module.
version: "3"
paths:
# NOTE: don't include the `/authn` prefix here that top level grouping is
# in the `openapi.yaml` file.
'/start':
get:
responses:
"200":
description: OK
By doing this separation you can independently test each file or the whole API as a group.
There may be points where you repeat yourself, but by doing this you limit the chance of breaking changes to other API endpoints when using a "common" library.
However, you should still have a common definition library for some things such as:
errors
security
There is a limitation on this approach and that's the "Discriminators" (it may be a ReDoc issue though, but if you had types that have discriminators outside of the openapi.yaml ReDoc fails to render correctly.
See this answer for details on how to split your Swagger documentation across many files. This is done using JSON, but the same concept can apply to RAML.
EDIT: Adding content of link here
The basic structure of your Swagger JSON should look something like this:
{
"swagger": "2.0",
"info": {
"title": "",
"version": "version number here"
},
"basePath": "/",
"host": "host goes here",
"schemes": [
"http"
],
"produces": [
"application/json"
],
"paths": {},
"definitions": {}
}
The paths and definitions are where you need to insert the paths that your API supports and the model definitions describing your response objects. You can populate these objects dynamically. One way of doing this could be to have a separate file for each entity's paths and models.
Let's say one of the objects in your API is a "car".
Path:
{
"paths": {
"/cars": {
"get": {
"tags": [
"Car"
],
"summary": "Get all cars",
"description": "Returns all of the cars.",
"responses": {
"200": {
"description": "An array of cars",
"schema": {
"type": "array",
"items": {
"$ref": "#/definitions/car"
}
}
},
"404": {
"description": "error fetching cars",
"schema": {
"$ref": "#/definitions/error"
}
}
}
}
}
}
Model:
{
"car": {
"properties": {
"_id": {
"type": "string",
"description": "car unique identifier"
},
"make": {
"type": "string",
"description": "Make of the car"
},
"model":{
"type": "string",
"description": "Model of the car."
}
}
}
}
You could then put each of these in their own files. When you start your server, you could grab these two JSON objects, and append them to the appropriate object in your base swagger object (either paths or definitions) and serve that base object as your Swagger JSON object.
You could also further optimize this by only doing the appending once when the server is started (since the API documentation will not change while the server is running). Then, when when the "serve Swagger docs" endpoint is hit, you can just return the cached Swagger JSON object that you created when the server was started.
The "serve Swagger docs" endpoint can be intercepted by catching a request to /api-docs like below:
app.get('/api-docs', function(req, res) {
// return the created Swagger JSON object here
});
You can use $ref but not have good flexibility, I suggest you process YAML with an external tool like 'Yamlinc' that mix multiple files into one using '$include' tag.
read more: https://github.com/javanile/yamlinc
I'm building several base images for our infrastructure and would like to mimic the Docker Hub nomenclature for the image tags. For example, Java image on Docker Hub includes several aliases for the same image, e.g. 8 and latest is the same image.
If I were to replicate this system in ImageStreams, I would need to create a BuildConfig with an output specification like this:
"output": {
"to": {
"kind": "ImageStreamTag"
"name": "jdk:8"
}
}
Obviously, this only includes one tag, so even if I were to write
"output": {
"to": {
"kind": "ImageStreamTag"
"name": "jdk:8"
},
"to": {
"kind": "ImageStreamTag"
"name": "jdk:latest"
}
}
only the latest definition would actually be executed.
Is there any proper way to push the same image into different tags apart from creating a different BuildConfig (which would probably "build" from Docker image to Docker image)?
There is a card on the trello board to do this: https://trello.com/c/nOX8FTRq/686-5-support-multiple-tags-for-a-build-output .
You should also be able to do this using oc tag to avoid having to run the same build twice.