I wanted to use NiFi's posthttp/invokeHttp processor to post a PDF to an API.
But considering the following cURL request to replicate in NiFi:
curl -X POST "http://ipaddress:port/api/" -H "accept: application/json" -H
"Content-Type: multipart/form-data" -F "pdf_file=#sample.pdf;
type=application/pdf"
Which property takes the -F information in nifi attributes?
Configuration for invokehttp right now:
error:
"400 Bad Request: The browser (or proxy) sent a request that this server could not understand."
Configration for posthttp right now:
error:
server logs: readv() failed (104: Connection reset by peer) while reading upstream
In older version of nifi you will have to use your own script to build a multipart request and then use invoke to create post request. You can refer to this post for a ExecuteGroovyScript example.
https://stackoverflow.com/a/57204862
Since Nifi 1.12 you can directly use invokeHTTP by setting content-type
https://stackoverflow.com/a/69284300
When you use PostHttp/InvokeHttp you wouldn't be referencing an external file, you would be sending the content of the flow file. So you would first need to bring sample.pdf into NiFi by using GetFile or ListFile/FetchFile and then flow file coming out of those processors represents the PDF, and you would route that flow file to InvokeHttp which would POST the content of the flow file (the pdf).
Related
The app that I am trying to test has an SFTP server that can be queried via API. Swagger shows the following sample cURL request to get domain files information, and I am having a hard time making this call with Karate:
curl -X GET -H 'Accept: application/json' 'https://{host}:{port}/api/{clientId}/'
I do the following where baseUrl is defined as https://sftp.mydomain.com:22 where 22 is the port number that I can successfully use to connect to the SFTP server via Cyberduck:
Feature:
Background:
* url baseUrl
* def moduleBase = '/api/12345/'
Scenario:
* path moduleBase
When method get
Then status 200
The error that I get is this:
ERROR com.intuit.karate - src/test/java/mytest.feature:9
When method get
http call failed after 815 milliseconds for url: https://sftp.mydomain.com:22/api/12345/
What am I doing wrong?
Clearly it may not be HTTP so I don't think Swagger and all is legit.
Maybe you can just delegate to the OS. Refer: https://stackoverflow.com/a/64352676/143475
So I am having difficulties sending a curl request to Hive. I want to take the json flow-file that I have created and send it as a command to Hive but I keep getting errors when I try to configure InvokeHTTP processor. For reference here is my workflow as it currently stands.
Replace Text-> Update Attributes - > InvokeHTTP->Put processor
I have tried mostly to get InvokeHTTP processor to work. The configurations that I have are:
1.HTTP Method: POST
2.Remote URL: ${https://hive-prod-1.sample_text/alert}
3.SSL Context Service: StandardSSLContext Service
4.Proxy Type: https
-Content-type: application/json
I then added a property
5.curl: curl-XPOST-H"Authorization: Bearer xWJbexxxxxxxx -H "Content-Type: application/json'
I am not sure if my configuration is incorrect or if there is another issue going on.
When I tried to use/configure ExecuteStreamCommand:
1.Command Arguments: curl-XPOST-H"Authorization xxxxx -H "Content-type: application/json
2.Command Path: application/json
Argument Delimiter: ;
Again, I am not sure if the configuration if correct for either of these processors or if it has something to do with a cert. When I run it I also get the error message 'java.lang.illegalstateexception: trustmanagerfactory =is not initialized.
It sounds as if you have not successfully/completely configured the SSLContextService which is required for InvokeHTTP when connecting to a service which uses TLS. Your Hive instance is protected with TLS, so you need to obtain the public certificate of the Hive instance (you can do this via a browser, using openssl s_client, etc.), load the public certificate into a Java Keystore (JKS) formatted truststore file as a trustedCertEntry, and then point the SSLContextService to that truststore file. For more information, look at the first section of Tomas Zezula's article on NiFi SSL configuration.
How would I go about translating the following curl command to a Jitterbit operation?
curl -i -u username:password -X POST -F file=#/path/to/file.csv
https://website.com/api/filepost
Currently I have my Operation structured as follows:
Script:
$jitterbit.target.http.form_data = true;
$jitterbit.target.http.form_data.filename = "file.csv";
$jitterbit.target.http.form_data.name = "file";
Source
A CSV file without headers, which matches the API's specifications (sent the same file successfully via curl)
Transformation:
Text to Text - both source and target use the same file format as the Source file
API Endpoint
Currently I authenticate successfully, but I get a 400/Bad Request error message saying "No file attached".
Full error message:
The operation "2. POST Preapplicants - CSV to API" failed.
Fatal Error
Failed to post to the url 'https://website.com/api/filepost’.
The last (and probably most relevant) error was: The server
returned HTTP Status Code : 400 Bad Request Error is: The
request could not be processed by the server due to invalid
syntax. Headers sent by the server: HTTP/1.1 400 Bad Request
Server: nginx/1.10.3 (Ubuntu) Content-Type: application/json
Transfer-Encoding: chunked Connection: keep-alive Cache-Control:
no-cache Date: Tue, 12 Sep 2017 18:55:38 GMT The response was:
{"message":"No file attached."}
I solved this problem by doing the following:
1. Changing from a transformation operation to an archive operation (using the same source, target, and script)
2. Changing the content-type of my HTTP connection to multipart/form-data (the default content-type passed by curl)
you can do it that way. what I did was use a file list operation stored that into an array then uses a Base64EncodeFile() function to upload the file
So I got the data that is being sent to a specific server. Now I want to do the same using curl from my local machine to play around with specific repsonses from the server and learn more about curl as well.
Here is part of my data
POST /auth HTTP/1.1
platform: android
X-Auth-Token: <censored>
Content-Type: application/json; charset=utf-8
Host: api.blabla.com
Accept-Encoding: gzip
And the data that is being sent:
{"blabla_token": "sdsadsad", "blahblah_id": "23213", "locale": "us"}
Now when I try cURL in my dos shell, I try
curl --insecure -X POST https://api.blabla.com/auth --data '{"blabla_token": "sdsadsad", "blahblah_id": "23213", "locale": "us"}'
The response I get from cURL is this:
{"code":401,"error":"blablaTokenRequired"}
Even though I specified the token. So there are two possible scenarios because the token is correct:
It has something to do with the SSL thing? (I use --insecure because I get an SSL error otherwise)
Something about my command is not correct but I can't figure out what.
Can someone kindly help me out? I am trying everything I can without success
I am not sure if I understand your application specific right, but probably one thing you need to take into account:
man curl says:
-d, --data <data>
(HTTP) Sends the specified data in a POST request to the HTTP server, in the same way that a browser does when
a user has filled in an HTML form and presses the submit button. This will cause curl to pass the data to the
server using the content-type application/x-www-form-urlencoded. Compare to -F, --form.
-d, --data is the same as --data-ascii. --data-raw is almost the same but does not have a special interpreta‐
tion of the # character. To post data purely binary, you should instead use the --data-binary option. To URL-
encode the value of a form field you may use --data-urlencode.
As I can't see in your example the necessity of sending data as HTML form input, probably your application expects just a "raw" POST body and then you have to try this:
curl --insecure -X POST https://api.blabla.com/auth --data--binary '{"blabla_token": "sdsadsad", "blahblah_id": "23213", "locale": "us"}'
PS and for sure this is error is not about using --insecure which just asks curl to neglect ssl verification
you forgot the headers and enabling compressed encoding (gzip), however, i believe you can't force curl to only support gzip encoding using the curl command line alone, you will have to use libcurl, this will make the request say "Accept-Encoding: gzip,deflate" on most systems, using --compressed .. if that's not acceptable to you, rewrite it using libcurl (where you can force it to say only "gzip", if you wish, via CURLOPT_ENCODING )
curl -X POST https://api.blabla.com/auth --data '{"blabla_token": "sdsadsad", "blahblah_id": "23213", "locale": "us"}' --header 'platform: android' --header 'X-Auth-Token: <censored>' --header 'Content-Type: application/json; charset=utf-8' --header 'Host: api.blabla.com' --compressed
another gotcha: on some systems, there will be a default useragent header (like debian 6), while on some systems, curl comes without a default useragent (like debian 8).. you might want to use --user-agent '' too
I'm trying to add a user to a Github repository via their API, but I always get a 502 Bad Gateway error.
With curl I send a request like this (<...> replaced by a real owner, repo, etc.):
curl -i -H 'Authorization: token xxxxxxxxxx' -XPUT https://api.github.com/repos/<owner>/<repo>/collaborators/<username>
I also tried it with this url:
curl -i -H 'Authorization: token xxxxxxxxxx' -XPUT https://api.github.com/teams/<id>/members/<username>
As token I used a newly created Personal Access Tokens
But both times I get this back
HTTP/1.0 502 Bad Gateway
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>
A GET on each URL works fine but a DELETE doesn't work either. So maybe it has to do with curl.
Quoting the reply from GitHub's support with changes in italic:
You're just getting trolled by HTTP and curl.
When you make a PUT request with no body, curl doesn't explicitly set a Content-Length header for that request. However, PUT requests with no Content-Length confuse servers and they respond in weird ways.
Can you please try explicitly setting the Content-Lenght header to 0, or supplying an empty body when making that request (so that curl can set the header for you)? You can accomplish that adding -d "" in your command.