How to get the response from Azure Data Lake Rest API using Azure API Management? - azure-data-lake

I want to make a call to the Azure Data Lake Gen 1 using Data Lake rest API. I tried making the call through Javascript but because I ran into CORS, I decided to have a layer between JS and ADLS using Azure API Management. So basically a user would call the APIM and the APIM internally would call the ADLS Rest API and send me the response. Using the CURL command as reference,
curl -X POST https://login.microsoftonline.com/<TENANT-ID>/oauth2/token \
-F grant_type=client_credentials \
-F resource=https://management.core.windows.net/ \
-F client_id=<CLIENT-ID> \
-F client_secret=<AUTH-KEY>
I am trying to fetch the OAuth Token from ADLS using APIM. This is what I have so far,
The APIM Policies :
<policies>
<inbound>
<base />
<set-method>POST</set-method>
<set-header name="Content-Type" exists-action="override">
<value>"application/x-www-form-urlencoded"</value>
</set-header>
</inbound>
<backend>
<set-body>{
'grant_type':'client_credentials',
'resource':'https://management.core.windows.net/',
'client_id':'#################################',
'client_secret':'#######################'
}</set-body>
</backend>
<outbound>
</outbound>
<on-error>
<base />
</on-error>
</policies>
I can successfully make a call to the ADLS but how do I consume the response from APIM that is the OAuth token and route it to my caller in APIM?

Depending on how you configured your API you may also need to add rewrite-uri and/or set-backend-service into inbound to direct call to https://login.microsoftonline.com.
Otherwise try:
<policies>
<inbound>
<base />
<set-method>POST</set-method>
<set-header name="Content-Type" exists-action="override">
<value>"application/x-www-form-urlencoded"</value>
</set-header>
<set-body>{
'grant_type':'client_credentials',
'resource':'https://management.core.windows.net/',
'client_id':'#################################',
'client_secret':'#######################'
}</set-body>
</inbound>
<backend>
<forward-request/>
</backend>
<outbound>
<base />
</outbound>
<on-error>
<base />
</on-error>
</policies>

Related

Access denied Error while uploading file to google cloud storage using Apigee API proxy

I am getting access denied error while trying to upload a file to google cloud storage using apigee api proxy. here is the error
<HTML>
<HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<H1>Access Denied</H1>
You don't have permission to access
"http://dev-apis.ihg.com/ics/libraries/v1/files" on this server.<P>
Reference #18.df2a0660.1669823422.237e2655
</BODY>
</HTML>
Here is the curl I have -
curl --location --request POST 'https://dev-apis.ihg.com/ics/libraries/v1/files' \
--header 'X-IHG-API-KEY: n5AwmgY0sDfjVQA9SIizlgOvYeZunbFp' \
--header 'X-IHG-AGENT-TOKEN: XXXXXXXX \
--header 'SMUSERNAME: XXXXXXX' \
--header 'Cookie: XXXXX' \
--form '=#"TIDW.DAT"' \
--form 'uploadDto="{
\"fileType\":\"TA_Uploads\",
\"fileSubType\":\"TRUE\",
\"supportCaseNbr\":\"9999\",
\"comments\":\"File upload\",
\"requesterName\":\"XXXXX\"
}"'
I have added streaming property for request and response in proxy endpoint as well as target endpoint.
<Properties>
<Property name="response.streaming.enabled">true</Property>
<Property name="request.streaming.enabled">true</Property>
</Properties>
When I tried to run my upload API from local it worked and I was able to upload file in google cloud storage.
When I tried to trace this request in Apigee edge I couldn't see the request landing to get traced. It appears it is failing at Apigee end when it is trying to process the message.
Please help me get over this.

Is there a way to setup apigee to use external authentication server

Is there a way to setup apigee to use external authentication server, to better explain the situation:
we generate a token from an app (test.app.com/services/oauth2/token)
we send this token with the request to the apigee endpoint
the apigee should call (test.app.com/services/oauth2/introspect) with authentication header (basic) and with body containing the token to verify it.
Im fairly new in apigee and have not seen similar implementations in my search
I am not sure if you figured another approach to this but I would you a service callout policy to call the authentication server as below:
<ServiceCallout async="false" continueOnError="false" enabled="true" name="SC-GetAuthPass">
<DisplayName>SC-GetAuthPass</DisplayName>
<Properties/>
<Request clearPayload="true" variable="ABCDLogin">
<Set>
<Headers>
<Header name="Content-Type">application/x-www-form-urlencoded</Header>
</Headers>
<Verb>POST</Verb>
<FormParams>
<FormParam name="username">{server_username}</FormParam>
<FormParam name="password">{server_password}</FormParam>
</FormParams>
</Set>
<IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
</Request>
<Response>calloutResponseGetAuthPass</Response>
<HTTPTargetConnection>
<Properties/>
<URL>{authentication_server_url}</URL>
</HTTPTargetConnection>
Then I could just extract the token and pass it to Apigee
<JSONPayload>
<Variable name="authpass">
<JSONPath>$.</JSONPath>
</Variable>
</JSONPayload>
<Source clearPayload="false">calloutResponseGetAuthPass</Source>
Then you could just call the verification endpoint with the extracted token (I assume this would be another service callout)

How to pass Authorization Token to my backend service through WSO2 (API Manager) version 4.1.0?

I need to pass Authorization Token from API Manager (WSO2) to my Backend using policies like Header policy but it works for me only using cURL but not with UI.
curl -k -X 'GET'
'https://localhost:8243/test/1.0.0/support/get/55'
-H 'accept: /'
-H 'Authorization: Bearer gatewayToken
-H 'Authorization: Bearer BackendToken'
Screenshot of the result after adding Header policy
By default the authorization header is dropped from the API gateway after validating the request. So you can't send the same header here with multiple values. If you want to send a custom header in the UI, you can do this as follows.
In the API publisher click on the API and go to the API configuration section.
Click on the resource tab and select a HTTP method.
Under parameters, you can add a header. Then from the Swagger UI, you can set this header when trying out the request.
Using API policies in API Manager 4.1.0, you can add AddHeader policy and send any static headers to the backend services.
You should be able to achieve this with a custom Operation Policy and adding that to the inflow of the API.
<sequence xmlns="http://ws.apache.org/ns/synapse" name="TokenExchange">
<property name="Custom" expression="get-property('transport', 'Custom')"/>
<property name="Authorization" expression="get-property('Custom')" scope="transport"/>
<property name="Custom" scope="transport" action="remove"/>
</sequence>
Here, you have to send the backend token with a different header value (Custom) and gateway will automatically forward it to the backend under the Authorization header.
You can refer [1] to get an idea about this flow. Since the mediation sequences are replaced with Operation policies in APIM 4.1.0, you will have to refer the doc [1] and create and apply the Operation policy accordingly.
[1] - https://apim.docs.wso2.com/en/4.0.0/deploy-and-publish/deploy-on-gateway/api-gateway/message-mediation/passing-a-custom-authorization-token-to-the-backend/#passing-a-custom-authorization-token-to-the-backend

Send logs directly to Loki without use of agents

Is there a way to send logs to Loki directly without having to use one of it's agents?
For example, if I have an API, is it possible to send request/response logs directly to Loki from an API, without the interference of, for example, Promtail?
Thanks in advance!
Loki HTTP API
Loki HTTP API allows pushing messages directly to Grafana Loki server:
POST /loki/api/v1/push
/loki/api/v1/push is the endpoint used to send log entries to Loki.
The default behavior is for the POST body to be a snappy-compressed
protobuf message:
Protobuf definition
Go client library
Alternatively, if the Content-Type header is set to application/json,
a JSON post body can be sent in the following format:
{
"streams": [
{
"stream": {
"label": "value"
},
"values": [
[ "<unix epoch in nanoseconds>", "<log line>" ],
[ "<unix epoch in nanoseconds>", "<log line>" ]
]
}
]
}
You can set Content-Encoding: gzip request header and post gzipped
JSON.
Example:
curl -v -H "Content-Type: application/json" -XPOST -s "http://localhost:3100/loki/api/v1/push" --data-raw \
'{"streams": [{ "stream": { "foo": "bar2" }, "values": [ [ "1570818238000000000", "fizzbuzz" ] ] }]}'
So it is easy to create JSON-formatted string with logs and send it to the Grafana Loki.
Libraries
There are some libraries implementing several Grafana Loki protocols.
There is also (my) zero-dependency library in pure Java 1.8, which implements pushing logs in JSON format to Grafana Loki. Works on Java SE and Android platform:
https://github.com/mjfryc/mjaron-tinyloki-java
Security
Above API doesn't support any access restrictions as written here - when using over public network, consider e.g. configuring Nginx proxy with HTTPS from Certbot and Basic Authentication.
Yes. You can send logs directly from a Java application to loki.
It can be done using the loki4j configuration in your java springboot project. Add these below dependencies to pom.xml
<dependency>
<groupId>com.github.loki4j</groupId>
<artifactId>loki-logback-appender</artifactId>
<version>1.2.0</version>
</dependency>
Run loki either directly or from docker depending on how you have installed loki on your system. I use docker instances of loki and grafana.
Create a logback.xml in your springboot project with the following contents
<property name="HOME_LOG" value="app.log" />
<appender name="FILE-ROLLING"
class="com.github.loki4j.logback.Loki4jAppender">
<http>
<url>http://localhost:3100/loki/api/v1/push</url>
</http>
<format>
<label>
<pattern>app=my-app,host=${HOSTNAME},level=%level</pattern>
</label>
<message>
<pattern>l=%level h=${HOSTNAME} c=%logger{20} t=%thread | %msg %ex
</pattern>
</message>
<sortByTime>true</sortByTime>
</format>
</appender>
<logger name="com.vasanth.loki" level="debug" additivity="false">
<appender-ref ref="FILE-ROLLING" />
</logger>
<root level="error">
<appender-ref ref="FILE-ROLLING" />
</root>
</configuration>
Configure your logger names in the above example and make sure you have given the proper loki URL - You are basically telling the application to write logs into an output stream going directly to the loki URL instead of the traditional way of writing logs to a file through log4j configuration and then using promtail to fetch these logs and load into loki.

How to bridge from a webhook URL to a HTTP POST request?

I want to make a HTTP POST request to Twilio but the calling service only allows me to enter a webhook URL.
I was trying to bridge this with apigee's API proxy but I could not figure out how to make it work.
The flow is like this:
A chat bot on motion.ai calls a web hook URL at a certain point.
The call should make an outbound call via twilio.com which requires a HTTP POST request, see here.
The POST request looks like this:
$ curl -XPOST https://api.twilio.com/2010-04-01/Accounts/<...>/Calls.json \
--data-urlencode "Url=http://demo.twilio.com/docs/voice.xml" \
--data-urlencode "To=<...>" \
--data-urlencode "From=<...>" \
-u '<...>:<...>'
What is the easiest way to bridge this?
I managed to setup an API proxy with Apigee to convert the HTTP GET request to a HTTP POST request.
Create a API proxy in Apigee and add a Basic Authentication policy:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<BasicAuthentication async="false" continueOnError="false" enabled="true" name="Basic-Authentication-1">
<DisplayName>Basic Authentication-1</DisplayName>
<Operation>Encode</Operation>
<IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
<User ref="request.queryparam.username"/>
<Password ref="request.queryparam.password"/>
<AssignTo createNew="false">request.header.Authorization</AssignTo>
<Source>request.header.Authorization</Source>
</BasicAuthentication>
Next add a Assign Message policy:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<AssignMessage async="false" continueOnError="false" enabled="true" name="Assign-Message-1">
<DisplayName>ConvertQueryToFormParameters</DisplayName>
<Properties/>
<Copy source="request">
<Headers/>
<QueryParams/>
<FormParams/>
<Payload/>
<Verb/>
<StatusCode/>
<ReasonPhrase/>
<Path/>
</Copy>
<Add/>
<Set>
<FormParams>
<FormParam name="To">{request.queryparam.To}</FormParam>
<FormParam name="From">{request.queryparam.From}</FormParam>
<FormParam name="Url">{request.queryparam.Url}</FormParam>
</FormParams>
<Verb>POST</Verb>
</Set>
<IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
</AssignMessage>
Then you can make a POST request to Twilio by simply calling
https://<yourApigeeApiUrl>.apigee.net/<yourApiName>?username=<yourTwilioApiUsername>&password=<yourTwilioApiPassword>&...