While using Amadeus Java SDK Point of Interest API on test environment, I provided the longitude and latitude received from the city search API, but I got "Invalid Data Received" with detailed message as "Sandbox coordinates out of the allowed box". I am getting similar response while trying directly from Amadeus Self Service API's.
Below is the request URL and response received:-
Request
https://test.api.amadeus.com/v1/reference-data/locations/pois?latitude=77.10309&longitude=28.5665&radius=1&page%5Blimit%5D=10&page%5Boffset%5D=0
Response
{
"errors": [
{
"status": 400,
"code": 4926,
"title": "INVALID DATA RECEIVED",
"detail": "Sandbox coordinates out of the allowed box",
"source": {
"parameters": [
"latitude",
"longitude",
"radius"
]
}
}
]
}
This is because you use the test environment which is free of charge but limited (limited number of calls per API per month and limited set of data as compared to production).
You can find here the data available in test for all our APIs and here the list of supported cities for Points Of Interest in test.
Note that in addition to the limited number of cities you will get only a maximum of 10 POIs.
in all Amadeus Self-Service APIs only some Airport codes work. For example "FRA" works, but "TXL" doesn't. Is this because the API is in Beta and I only use the Sandbox version?
Example:
https://test.api.amadeus.com/v1/shopping/flight-destinations?origin=FRA&oneWay=false&nonStop=false
WORKS
https://test.api.amadeus.com/v1/shopping/flight-destinations?origin=TXL&oneWay=false&nonStop=false
{
"errors": [
{
"status": 500,
"code": 141,
"title": "SYSTEM ERROR HAS OCCURRED",
"detail": "DATA DOMAIN NOT FOUND FOR REQUEST"
}
]
}
The APIs available in the test environment have a limit set of data (cache or fake data).
In the test environment, this API doesn't have data for TXL as origin, for Germany you have FRA and MUC.
So far our data set covers more the US and some big cities in the world. We will publish soon the list of available data on our portal.
You can find the list of available data in the test environment on our GitHub page.
I would like to use Azure to create environments under a single ResourceGroup for clients comprising:
X web servers
Y app servers
1 database
Ideally, the database would be hosted on a server that is part of a different resource group, so I can leverage elastic pools across multiple clients.
When I attempt to use New-AzureRmResourceGroupDeployment to do this, I get an error stating that the parent SqlServer cannot be found.
I have created a SqlServer via PowerShell:
New-AzureRmSqlServer -servername "MyServer" -ResourceGroupName "Common" -Location "South Central US"
I then attempt a deployment for ClientA with:
New-AzureRmResourceGroup -Name 'ClientA' -Location "South Central US"
New-AzureRmResourceGroupDeployment -ResourceGroupName 'ClientA' -TemplateFile azure.ClientDeployment.json -Client 'ClientA'
My deployment configuration is:
{
"parameters": { ... }
"variables": { ... }
"resources": [
{
"name": "MySqlServer/ClientA",
"type": "Microsoft.Sql/servers/databases",
"location": "[resourceGroup().location]",
"apiVersion": "2014-04-01",
"tags": {},
"location": "[resourceGroup().location]",
"properties": {
"edition": "Basic"
},
}
],
"outputs": { }
}
Results in the error message:
New-AzureRmResourceGroupDeployment : 5:04:33 PM - Resource Microsoft.Sql/servers/databases 'MySqlServer/ClientA' failed with message '{
"code": "NotFound",
"message": "Server 'MySqlServer' does not exist in resource group 'ClientA' in subscription '{...}'.",
"target": null,
"details": [],
"innererror": []
}
Note that both resource groups (Common and ClientA) are in the same subscription and location.
Is it possible to have a SqlServer part of one resource group, and the SqlDatabase part of a different resource group?
Separating SQLServer with one Subscriptions(location)and it's Database to another Subscriptions is not possible as of now.
See the similar question here in MSDN forum
An Azure resource group is also defined to be a strong container for
its resources, which defines their identity (look at the URI of all
Azure resources), and propagation of role-based access rights, and
controls resource lifetimes - thus granting access to a resource group
grants rights to its resources and deleting a resource group deletes
the resources it contains.
Combining the above means that the resource group that contains a
server must also contain the databases hosted/nested on that server.
Even if you try to create all in a single subscription and try to move the particular SQLDB resource group after. It won't allow doing that (In Portal)
For your case, I think to create all the services in a common subscription where the location common or easily accessible by all the client.
You can use AzureSpeed to get the latency for all the location from your current location and try to create the subscription in a common location which is having minimal latency for all your clients.
I do an up scale with a code like this on an Azure SQL Database:
ALTER DATABASE [MYDATABASE] Modify (SERVICE_OBJECTIVE = 'S1');
How is it possible to know from a c# code when Azure has completed the job and the table is already available?
Checking for SERVICE_OBJECTIVE value is not enough, the process still continue further.
Instead of performing this task in T-SQL I would perform the task from C# using an API call over to the REST API, you can find all of the details on MSDN.
Specifically, you should look at the Get Create or Update Database Status API method which allows you to call the following URL:
GET https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/{resource-group-name}/providers/Microsoft.Sql/servers/{server-name}/databases/{database-name}}/operationResults/{operation-id}?api-version={api-version}
The JSON body allows you to pass the following parameters:
{
"id": "{uri-of-database}",
"name": "{database-name}",
"type": "{database-type}",
"location": "{server-location}",
"tags": {
"{tag-key}": "{tag-value}",
...
},
"properties": {
"databaseId": "{database-id}",
"edition": "{database-edition}",
"status": "{database-status}",
"serviceLevelObjective": "{performance-level}",
"collation": "{collation-name}",
"maxSizeBytes": {max-database-size},
"creationDate": "{date-create}",
"currentServiceLevelObjectiveId":"{current-service-id}",
"requestedServiceObjectiveId":"{requested-service-id}",
"defaultSecondaryLocation": "{secondary-server-location}"
}
}
In the properties section, the serviceLevelObjective property is the one you can use to resize the database. To finish off you can then perform a GET on the Get Database API method where you can compare both the currentServiceLevelObjectiveId and requestedServiceObjectiveId properties to ensure your command has been successful.
Note: Don't forget to pass all of the common parameters required to make API calls in Azure.
I am wondering how the various app statistics sites get app-details from Google Play. As GP does not have a public API. An example is Appaware.com - they have full details for Google Play apps.
A possible solution is scraping, however it doesn't work because Google will block you when you start sending hundreds of requests to them.
Any ideas?
p.s. "Google Play Developers API" is not a choice as it lets you access app-details only for your apps.
They use either the mobile API used by Android devices (i.e. with this library) or scrape the Google Play website. Both methods are subject to rate limiting, so they put pauses in between requests.
The mobile device API is completely undocumented and very difficult to program against. I would recommend scraping.
There is no official API or feed that you can use.
Android Marketing API is used to get the All app details from google store, You can check it out at here: https://code.google.com/p/android-market-api/
Unfortunately Google Play (previously known as Android Market) does not expose an official API.
To get the data you need, you could develop your own HTML crawler, parse the page and extract the app meta-data you need. This topic has been covered in other questions, for instance here.
If you don't want to implement all that by yourself (as you mentioned it's a complex project to do), you could use a third-party service to access Android apps meta-data through a JSON-based API.
For instance, 42matters.com (the company I work for) offers an API for both Android and iOS, you can see more details here.
The endpoints range from "lookup" (to get one app's meta-data, probably what you need) to "search", but we also expose "rank history" and other stats from the leading app stores. We have extensive documentation for all supported features, you find them in the left panel: 42matters docs
I hope this helps, otherwise feel free to get in touch with me. I know this industry quite well and can point you in the right direction.
Regards,
Andrea
The request might be blocked if using requests as default user-agent in requests library is a python-requests.
An additional step could be to rotate user-agent, for example, to switch between PC, mobile, and tablet, as well as between browsers e.g. Chrome, Firefox, Safari, Edge and so on. User-agent rotation can be used in combo with proxy rotation (ideally residential) + CAPTCHA solver.
At the moment, the Google Play Store has been heavily redesigned, now it is almost completely dynamic. However, all the data can be extracted from the inline JSON.
For scraping dynamic sites, selenium or playwright webdriver is great. However, in our case, using BeautifulSoup and regular expression is faster to extract data from the page source.
We must extract certain <script> element from all <script> elements in the HTML, by using regular expression, and transform in to a dict with json.loads():
# https://regex101.com/r/zOMOfo/1
basic_app_info = json.loads(re.findall(r"<script nonce=\"\w+\" type=\"application/ld\+json\">({.*?)</script>", str(soup.select("script")[11]), re.DOTALL)[0])
Check code in online IDE.
from bs4 import BeautifulSoup
import requests, re, json, lxml
# https://requests.readthedocs.io/en/latest/user/quickstart/#custom-headers
headers = {
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
}
# https://requests.readthedocs.io/en/latest/user/quickstart/#passing-parameters-in-urls
params = {
"id": "com.nintendo.zara", # app name
"gl": "US", # country of the search
"hl": "en_GB" # language of the search
}
html = requests.get("https://play.google.com/store/apps/details", params=params, headers=headers, timeout=30)
soup = BeautifulSoup(html.text, "lxml")
# where all app data will be stored
app_data = {
"basic_info":{
"developer":{},
"downloads_info": {}
}
}
# [11] index is a basic app information
# https://regex101.com/r/zOMOfo/1
basic_app_info = json.loads(re.findall(r"<script nonce=\"\w+\" type=\"application/ld\+json\">({.*?)</script>", str(soup.select("script")[11]), re.DOTALL)[0])
# https://regex101.com/r/6Reb0M/1
additional_basic_info = re.search(fr"<script nonce=\"\w+\">AF_initDataCallback\(.*?(\"{basic_app_info.get('name')}\".*?)\);<\/script>",
str(soup.select("script")), re.M|re.DOTALL).group(1)
app_data["basic_info"]["name"] = basic_app_info.get("name")
app_data["basic_info"]["type"] = basic_app_info.get("#type")
app_data["basic_info"]["url"] = basic_app_info.get("url")
app_data["basic_info"]["description"] = basic_app_info.get("description").replace("\n", "") # replace new line character to nothing
app_data["basic_info"]["application_category"] = basic_app_info.get("applicationCategory")
app_data["basic_info"]["operating_system"] = basic_app_info.get("operatingSystem")
app_data["basic_info"]["thumbnail"] = basic_app_info.get("image")
app_data["basic_info"]["content_rating"] = basic_app_info.get("contentRating")
app_data["basic_info"]["rating"] = round(float(basic_app_info.get("aggregateRating").get("ratingValue")), 1) # 4.287856 -> 4.3
app_data["basic_info"]["reviews"] = basic_app_info.get("aggregateRating").get("ratingCount")
app_data["basic_info"]["reviews"] = basic_app_info.get("aggregateRating").get("ratingCount")
app_data["basic_info"]["price"] = basic_app_info["offers"][0]["price"]
app_data["basic_info"]["developer"]["name"] = basic_app_info.get("author").get("name")
app_data["basic_info"]["developer"]["url"] = basic_app_info.get("author").get("url")
# https://regex101.com/r/C1WnuO/1
app_data["basic_info"]["developer"]["email"] = re.search(r"[a-zA-Z0-9_.+-]+#[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+", additional_basic_info).group(0)
# https://regex101.com/r/Y2mWEX/1 (a few matches but re.search always matches the first occurence)
app_data["basic_info"]["release_date"] = re.search(r"\d{1,2}\s[A-Z-a-z]{3}\s\d{4}", additional_basic_info).group(0)
# https://regex101.com/r/7yxDJM/1
app_data["basic_info"]["downloads_info"]["long_form_not_formatted"] = re.search(r"\"(\d+,?\d+,?\d+\+)\"\,(\d+),(\d+),\"(\d+M\+)\"", additional_basic_info).group(1)
app_data["basic_info"]["downloads_info"]["long_form_formatted"] = re.search(r"\"(\d+,?\d+,?\d+\+)\"\,(\d+),(\d+),\"(\d+M\+)\"", additional_basic_info).group(2)
app_data["basic_info"]["downloads_info"]["as_displayed_short_form"] = re.search(r"\"(\d+,?\d+,?\d+\+)\"\,(\d+),(\d+),\"(\d+M\+)\"", additional_basic_info).group(4)
app_data["basic_info"]["downloads_info"]["actual_downloads"] = re.search(r"\"(\d+,?\d+,?\d+\+)\"\,(\d+),(\d+),\"(\d+M\+)\"", additional_basic_info).group(3)
# https://regex101.com/r/jjsdUP/1
# [2:] skips 2 PEGI logo thumbnails and extracts only app images
app_data["basic_info"]["images"] = re.findall(r",\[\d{3,4},\d{3,4}\],.*?(https.*?)\"", additional_basic_info)[2:]
try:
# https://regex101.com/r/C1WnuO/1
app_data["basic_info"]["video_trailer"] = "".join(re.findall(r"\"(https:\/\/play-games\.\w+\.com\/vp\/mp4\/\d+x\d+\/\S+\.mp4)\"", additional_basic_info)[0])
except:
app_data["basic_info"]["video_trailer"] = None
print(json.dumps(app_data, indent=2, ensure_ascii=False))
Example output:
[
{
"basic_info": {
"developer": {
"name": "Nintendo Co., Ltd.",
"url": "https://supermariorun.com/",
"email": "supermariorun-support#nintendo.co.jp"
},
"downloads_info": {
"long_form_not_formatted": "100,000,000+",
"long_form_formatted": "100000000",
"as_displayed_short_form": "100M+",
"actual_downloads": "213064462"
},
"name": "Super Mario Run",
"type": "SoftwareApplication",
"url": "https://play.google.com/store/apps/details/Super_Mario_Run?id=com.nintendo.zara&hl=en_GB&gl=US",
"description": "Control Mario with just a tap!",
"application_category": "GAME_ACTION",
"operating_system": "ANDROID",
"thumbnail": "https://play-lh.googleusercontent.com/3ZKfMRp_QrdN-LzsZTbXdXBH-LS1iykSg9ikNq_8T2ppc92ltNbFxS-tORxw2-6kGA",
"content_rating": "Everyone",
"rating": 4.0,
"reviews": "1645926",
"price": "0",
"release_date": "22 Mar 2017",
"images": [
"https://play-lh.googleusercontent.com/yT8ZCQHNB_MGT9Oc6mC5_mQS5vZ-5A4fvKQHHOl9NBy8yWGbM5-EFG_uISOXmypBYQ6G",
"https://play-lh.googleusercontent.com/AvRrlEpV8TCryInAnA__FcXqDu5d3i-XrUp8acW2LNmzkU-rFXkAKgmJPA_4AHbNjyY",
"https://play-lh.googleusercontent.com/AESbAa4QFa9-lVJY0vmAWyq2GXysv5VYtpPuDizOQn40jS9Z_ji8HXHA5hnOIzaf_w",
"https://play-lh.googleusercontent.com/KOCWy63UI2p7Fc65_X5gnIHsErEt7gpuKoD-KcvpGfRSHp-4k8YBGyPPopnrNQpdiQ",
"https://play-lh.googleusercontent.com/iDJagD2rKMJ92hNUi5WS2S_mQ6IrKkz6-G8c_zHNU9Ck8XMrZZP-1S_KkDsA6KDJ9No",
# ...
]
A possible good solution with shorter and simpler code could be Google Play Store API from SerpApi. It's a paid API with a free plan.
The difference is that it will bypass blocks (including CAPTCHA) from Google, no need to create the parser and maintain it.
SerpApi simple code example:
from serpapi import GoogleSearch
import os, json
params = {
"api_key": os.getenv("API_KEY"), # your serpapi api key
"engine": "google_play_product", # parsing engine
"store": "apps", # app page
"gl": "us", # country of the search
"product_id": "com.nintendo.zara", # low review count example to show it exits the while loop
"all_reviews": "true" # shows all reviews
}
search = GoogleSearch(params) # where data extraction happens
results = search.get_dict()
print(json.dumps(results["product_info"], indent=2, ensure_ascii=False))
print(json.dumps(results["media"], indent=2, ensure_ascii=False))
# other data
Output exactly the same as in the previous solution.
There's a Scrape Google Play Store App in Python blog post if you need a little bit more code explanation.
Disclaimer, I work for SerpApi.