today I am trying to download an webpage url using curl. And here I see a peculiar problem.
When I run my code from local machine, its collect everything well. But when I run my code from my production server, its get 503 error.
I see that site use cloudflare service. But my question is
If they protect scrap from production, why can't protect when I trying from locally?
Can anyone help me please?
my code is :
<!DOCTYPE html>
<html>
<head>
<meta http-equiv=Content-Type content="text/html; charset=UTF-8">
</head>
<body>
<?php
include('simple_html_dom.php');
$new_url = 'https://www.bagdoom.com/electronics/computers/webcams.html';
$header = array();
$header[0] = "Accept: text/xml,application/xml,application/xhtml+xml,";
$header[0] .= "text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
$header[] = "Cache-Control: max-age=0";
$header[] = "Connection: keep-alive";
$header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
$header[] = "Accept-Language: en-us,en;q=0.5";
$header[] = "Pragma: ";
$curl = curl_init();
curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7');
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($curl, CURLOPT_HTTPHEADER, $header);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_URL, $new_url);
curl_setopt($curl, CURLOPT_REFERER, $new_url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
$str = curl_exec($curl);
curl_close($curl);
$html_list = new simple_html_dom();
$new_html_list = $html_list->load($str);
foreach($new_html_list->find("#awac_catalog_container li.item h2.product-name a") as $e){
echo $e->href . '<br>';
}
?>
</body>
</html>
One of the services Cloudflare offers is IP Reputation checks; in short if a site has been seen to be abusive across the Cloudflare network - it can be challenged with a captcha or JavaScript challenge page.
It is likely that the IP Address reputation of your server on production is quite low (especially if the IP Address has been recycled by a cloud server provider). The IP reputation of your local server, may however, be considerably more trustworthy.
In order to bypass this, you will need to get the site to whitelist your servers IP Address in the Firewall app in the Cloudflare dashboard.
Related
Trying to authorize an 'app' in 3dcart
The api is fairly well documented using Swagger and they have this page w/ instructions https://apirest.3dcart.com/v1/getting-started/index.html#getting-started
You are supposed to make this first call, which is supposed to give you token you use in the subsequent step
https://apirest.3dcart.com/oauth/authorize?
Client_Id=22613fdfc5a6200bece02a29524XXXXX
&state=12345
&response_type=code
&store_url=www.***********.com
&redirect_uri=www.!!!!!!!!!!.com <============== WHAT SHOULD THIS BE
Using postman, I can play with the various parameters.
I beleive clientId should be the guid-ish string you get associated w/ each store that has subscribed to your app.
Store Url - i know this. Should I include the https:// protocol in the string?
It sounds like STATE is just something you pass in and the store parrots it back to you.
redirect_uri - not sure what this is/ should be
I can get status 200 OK, but i am getting an HTML page .
Pasting same URL in a browser, I am getting a login page. I login as an admin user and then get 404 because it is redirecting me to the redirect_uri
??? I really just want to login. I tried passing in a blank uri and omitting the parameter. Both yield
Status 400 Error 101 redirect_uri is required
Am I doing this correctly? Is there an example of authorizing so you can use the API?
I have downloaded the GIT repository. That helps a bit, but still missing something
tyia
Using the URL you're talking about will allow the customer to grant access to your application. The Redirect URI in the URL you have the customer click on MUST match the Redirect URI you have set up on your app in 3dcart Developer.
After that, there are 5 steps to getting the Authorization Token and then use it to work with the 3dcart API.
The redirect URI is your endpoint where 3dcart will send the code.
Step one:
Set up your Redirect URIs: under the Oauth tab on your application on 3dcart Developers
Step 2:
3dcart will post to this URL a "Code". You can get this from the URL with GET
$code = $_GET['code'];
Step 3:
Use Curl (or whatever) to make a postback to 3dcart with the code
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://apirest.3dcart.com/oauth/token");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
$data = array('Code' => $code, 'client_id' => '{your public app key}', 'client_secret' => '{your private app key}', 'grant_type' => 'authorization_code');
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'application/x-www-form-urlencoded'
));
$response = curl_exec($ch);
Step 4:
parse the response to get the access_token:
$info = json_decode($response, true);
$access_token = $info['access_token'];
Step 5:
Use the Access token to make calls to 3dcart using the following parameters in curl
$auth = array(
"Content-Type: application/xml",
"Accept: application/json",
"Authorization: Bearer $access_token",
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://apirest.3dcart.com/3dCartWebAPI/v2/{API END POINT}");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_HTTPHEADER, $auth);
$response = curl_exec($ch);
curl_close($ch);
Using Postman, I have been able to successfully create an API call to retrieve information from the webserver.
However, I have been unsuccessful in doing the same with PHP. Following are a few screenshots from Postman which I am trying to replicate in PHP.
This is my PHP code -
$filters = array("1234", "28");
$selectors = array("Brand", "Name");
$body = array('Filter' => array('SKU'=>$filters, 'OutputSelector' => $selectors));
$url = "https://xyz.neto.com.au/do/WS/NetoAPI";
$method = "POST";
$headers = array(
"NETOAPI_ACTION: GetItem",
"NETOAPI_KEY: xxxxxxx",
"Accept: application/json",
"NETOAPI_USERNAME: puneeth"
);
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, $method);
curl_setopt($ch, CURLOPT_POSTFIELDS, $body);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
$response = curl_exec($ch);
$err = curl_error($ch);
curl_close($ch);
if ($err) {
echo "cURL Error #:" . $err;
} else {
echo $response;
}
P.S I have deliberately fudged the url and key.
I am trying to follow instructions as outlined here - https://developers.neto.com.au/documentation/engineers/api-documentation/products/getitem
Few things that might be preventing your code from working:
1. Array to string conversion
You have declared the body of the request as an array in $body. You need to JSON-encode it before passing it to cURL:
$body = array('Filter' => array('SKU'=>$filters, 'OutputSelector' => $selectors));
$bodyJSON = json_encode($body);
[...]
curl_setopt($ch, CURLOPT_POSTFIELDS, $bodyJSON);
2. HTTPS request
If you're getting an empty response, you'll also need to configure cURL for SSL communication. A quick fix for this is to ignore the remote server certificate validity:
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
Although this works, see this answer for a better solution.
3. Missing Content-Type
When sending raw postfields, some servers need to know what type of data you're sending. This is done by setting a Content-Type header:
$headers = array(
"Content-Type: application/json",
[...]
);
Recently, Twitch bring a new API version using new endpoints etc..
I was working on the V5, but I didn't use Curl command line, I work with URL.
So I decide to look at the references of the new version, trying for example to getting the followers and found this :
https://api.twitch.tv/helix/users/follows?to_id='user ID'
So I replace the user_ID by an id (mine and/or another) and get :
{
"error":"Unauthorized",
"status":401,"message":"Must provide a valid Client-ID or OAuth token"
}
When I was working on the V5, I was putting the client_id and the oauth at the end of the URL like this :
https://api.twitch.tv/kraken/channels/CHANNELNAME?client_id=xXxXxXxXxX&oauth_token=aaaabbbbccc111
And it was working, but in the new API, I already have a parameter so I added the client_id and token after with a & connector... But still have the same error.
I also try to put them before the to_id parameter, but same...
So my question is really stupid but anyone know the URL format on the new API?
You should send your Client-ID in request's header now, not as a param in URL. But there's other problem with SSL/HTTPS in this case if you use curl.
Here is a solution to your problem
DEFINE (TWITCH_API_KEY,'YOUR_KEY_HERE');
$url = 'https://api.twitch.tv/helix/streams/metadata';
$ch = curl_init();
$headers=['Client-ID: '.TWITCH_API_KEY];
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
$result = curl_exec ($ch);
curl_close ($ch);
echo $result;
I need to call Shopify's API's to create a new product or get details of existing products. How can i do this. Basically how to make a connection of salesforce with shopify?
Shopify provide REST API, so you need to use it for integration with your Salesforce solution. On Salesforce side you need to do few steps for implementation of connection.
Usual you need to make a post/get requests to some URL you would do the following.
Add your domain under Remote Site Settings
To send an outbound calls (POST/GET requests) from Apex in Salesforce, you need to add the domain to Remote Site Settings in setup.
Create and send the request in your APEX controller
For instance, for POST request you need to implement something like this
HttpRequest req = new HttpRequest();
HttpResponse res = new HttpResponse();
Http http = new Http();
req.setEndpoint('https://someurl.com/api');
req.setMethod('POST');
//these parts of the POST you may want to customize
req.setCompressed(false);
req.setBody('key1=value1&key2=value2');
req.setHeader('Content-Type', 'application/x-www-form-urlencoded');
try {
res = http.send(req);
} catch(System.CalloutException e) {
System.debug('Callout error: '+ e);
}
System.debug(res.getBody());
For more information take a look on Salesforce documentation:
Apex Web Services and Callouts
HttpRequest Class
HttpResponse Class
I just recently actually managed to develop what (I think) you are looking for in PHP and since i had plenty trouble finding all the info I actually went ahead and created an account just to answer this for you :)
my PHP code is as follows:
//Modify these
$API_KEY = 'yourAPIkey';
$SECRET = 'yourAPIsercer(password)';
$TOKEN = 'zzz';
$STORE_URL = 'yourstore.myshopify.com';
$url = 'https://' . $API_KEY . ':' . $SECRET . '#' . $STORE_URL . '/admin/products.xml';
$xmlsrc = <<<XML
<?xml version='1.0' encoding='UTF-8'?>
<product>
<title>TITLEH!!!</title>
<body-html>asdasdfda</body-html>
<product-type>Photoshop</product-type>
<variants type="array">
<variant>
<price>3.00</price>
<inventory-quantity>7</inventory-quantity>
</variant>
</variants>
<vendor>JLH</vendor>
</product>
XML;
$session = curl_init();
curl_setopt($session, CURLOPT_URL, $url);
curl_setopt($session, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($session, CURLOPT_POSTFIELDS, $xmlsrc);
curl_setopt($session, CURLOPT_HEADER, false);
curl_setopt($session, CURLOPT_HTTPHEADER, array('Accept: application/xml', 'Content-Type: application/xml'));
curl_setopt($session, CURLOPT_RETURNTRANSFER, true);
if(ereg("^(https)",$url)) curl_setopt($session,CURLOPT_SSL_VERIFYPEER,false);
$result = curl_exec($session);
curl_close($session);
$doc = new DOMDocument();
$doc->preserveWhiteSpace = true;
$doc->formatOutput = true;
$doc->loadXML($result);
echo $doc->saveXML();
I believe it's straightforward enough.
It's also quite easy to modify it for different actions once you get the hang of it...
I got alot of useful info from this thread: https://ecommerce.shopify.com/c/shopify-discussion/t/php-api-example-26017
and the rest i deducted from the shopify API reference: https://help.shopify.com/api/reference/product
(note that API reference uses json examples and i am working with XML)
I hope this helps you.
I am using rest api to fetch basic profile data from linkedin with php code. I am successfully able to generate access code and access token but I am getting ssl required error whenever I tried to get basic profile using following url
https://api.linkedin.com/v1/people/~?format=json
I followed all steps to make authenticated requests as listed there https://developer.linkedin.com/docs/oauth2
I am using a non ssl url as call back parameter , Is it necessary to use ssl url ? If not then why I am getting this error .
looking for a solution
below is code to get profile fields
$host = "api.linkedin.com";
$path = "/v1/people/~)";
//$arr = array('oauth2_access_token' => $access['access_token']);
$token = $access['access_token'];
$header[] = "Authorization: Bearer ".$token;
$header[] = "Connection: keep-alive";
$header[] = "Keep-Alive: 300";
$ch = curl_init();
// endpoint url
curl_setopt($ch, CURLOPT_URL, $host . $path);
// set request as regular post
//curl_setopt($ch, CURLOPT_HTTPGET, true);
// set http version
curl_setopt($ch, CURLOPT_HTTP_VERSION, 'HTTP/1.1');
// set data to be send
//curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($arr));
// set header
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
// set header
curl_setopt($ch, CURLOPT_CERTINFO, true);
// return transfer as string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
curl_close($ch);
var_export($response);
My solution was to use https:// instead of just http://