I'm building a website and I'm using the Spotify API as a music library. I would like to add more filters and order options to search traks than the api allows me to so I was wondering what track/song data can I save to my DB from the API, like artist name or popularity.
I would like to save: Name, Artists, Album and some other stuff. Is that possible or is it against the terms and conditions?
Thanks in advance!
Yes, it is possible.
Data is stored in Spotify API in endpoints.
Spotify API endpoint reference here.
Each endpoint deals with the specific kind of data being requested by the client (you).
I'll give you one example. The same logic applies for all other endpoints.
import requests
"""
Import library in order to make api calls.
Alternatively, ou can also use a wrapper like "Spotipy"
instead of requesting directely.
"""
# hit desired endpoint
SEARCH_ENDPOINT = 'https://api.spotify.com/v1/search'
# define your call
def search_by_track_and_artist(artist, track):
path = 'token.json' # you need to get a token for this call
# endpoint reference page will provide you with one
# you can store it in a file
with open(path) as t:
token = json.load(t)
# call API with authentication
myparams = {'type': 'track'}
myparams['q'] = "artist:{} track:{}".format(artist,track)
resp = requests.get(SEARCH_ENDPOINT, params=myparams, headers={"Authorization": "Bearer {}".format(token)})
return resp.json()
try it:
search_by_track_and_artist('Radiohead', 'Karma Police')
Store the data and process it as you wish. But you must comply with Spotify terms in order to make it public.
sidenote: Spotipy docs.
Related
I am running the following code to retrieve account info by hashtag from the unofficial TikTok Api.
API Repository - https://github.com/davidteather/TikTok-Api
Class Definition - https://dteather.com/TikTok-Api/docs/TikTokApi/tiktok.html#TikTokApi.hashtag
However it seems the maximum number of responses I can get on any hashtag is 500 or so.
Is there a way I can request more? Say...10K lines of account info?
from TikTokApi import TikTokApi
import pandas as pd
hashtag = "ugc"
count = 50000
with TikTokApi() as api:
tag = api.hashtag(name=hashtag)
print(tag.info())
lst = []
for video in tag.videos(count=count):
lst.append(video.author.as_dict)
df = pd.DataFrame(lst)
print(df)
The above code for hashtag "ugc" produces only 482 results. Whereas I know there is significantly more results available from TikTok.
It’s ratelimit, TikTok will block you after a certain amount of requests.
Add proxy support like this:
proxies = open('proxies.txt', 'r').read().splitlines()
proxy = random.choice(proxies)
proxies = {'http': f'http://{proxy}', 'https': f'https://{proxy}'}
Could be more precise but it's a way to rotate them
I work with analytics for organizational YouTube channels for my employer. I've been successfully retrieving data using YouTube DATA API, however I am unable to obtain metrics from YouTube Analytics API (a'la reports.query).
Here are the particulars...
Authentication: SAML (organizational GSuite account)
Authorization: OAuth2
My organizational account is an "owner" (though not primary owner) of the YouTube channel.
Google Cloud Platform project: internal (organizational)
I've tried various SCOPE combinations.
When I specify ids="channel==<channel_id>", I receive a 403 (forbidden) response.
When I specify ids="channel==MINE", I get a 200 status with headers but depending on dimension and metric selections, either no records, or a single record with zeros for each metric. My suspicion is that "channel==MINE" is looking for "my" channel rather than the brand account's channel as it wouldn't know which brand channel otherwise.
My question is, how can I assure that my organizational account can obtain data using YouTube Analytics API as well as YouTube Reporting API? Is there an administrative site that assigns these access rights? If so, where is it?
The following code is essentially sample code from the API Explorer...
import os
import google_auth_oauthlib.flow
import googleapiclient.discovery
import googleapiclient.errors
scopes = ["https://www.googleapis.com/auth/youtube.readonly"]
def main():
# Disable OAuthlib's HTTPS verification when running locally.
# *DO NOT* leave this option enabled in production.
os.environ["OAUTHLIB_INSECURE_TRANSPORT"] = "1"
api_service_name = "youtubeAnalytics"
api_version = "v2"
client_secrets_file = "client_secret.json"
# Get credentials and create an API client
flow = google_auth_oauthlib.flow.InstalledAppFlow.from_client_secrets_file(
client_secrets_file, scopes)
# credentials = flow.run_console()
credentials = flow.run_local_server()
youtube_analytics = googleapiclient.discovery.build(
api_service_name, api_version, credentials=credentials)
request = youtube_analytics.reports().query(
ids="channel==MINE",
startDate="2020-09-01",
endDate="2020-09-30",
dimensions="day",
metrics="views",
)
response = request.execute()
print(response)
if __name__ == "__main__":
main()
The code as shown, yields...
{'kind': 'youtubeAnalytics#resultTable', 'columnHeaders': [{'name': 'day', 'columnType': 'DIMENSION', 'dataType': 'STRING'}, {'name': 'views', 'columnType': 'METRIC', 'dataType': 'INTEGER'}], 'rows': []}
With ids set to "channel==UC6L0DBYWqAkmwfawTUMaR3g", the followiing is returned...
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://youtubeanalytics.googleapis.com/v2/reports?ids=channel%3D%3DUC6L0DBYWqAkmwfawTUMaR3g&startDate=2020-09-01&endDate=2020-09-30&dimensions=day&metrics=views&alt=json returned "Forbidden">
I have an app where the user details are passed as a JWT containing information about the current user and it's roles.
Everytime the user is logged in (via a KeyCloak instance), the information from the JWT is parsed on my end in a function that updates the user object via SQLAlchemy. However, since there is no user object being passed back and forth in the backend, I have to parse the JWT for roles for every action that requires it. I also have a need for auditing, and due to the structure of the app, this module does not necessarily have access to the request objects at the time of logging.
I'm looking for a neat way to make something like flask_users current_user() functionality work by mapping JWT -> ORM user object, to be able to transparently get the current user. Is there any way to go about this? The user registration and so on is completely separate from the app, and Flask only knows which user it is based on tokens in the requests that are being sent.
TLDR; Is there a way to load a user from the DB based on an already issued JWT (which contains information to map to a user), and is there perhaps already a lib or extension to flask that supports this?
I use a decorator to parse the JWT token using pyjwt.
Then from the parsed token you can get the user and do the proper authorization.
If you don't want to add the decorator to all your functions that require authorization you can use Flasks before_request.
from functools import wraps
from flask import Response, current_app, request
from jwt import decode
from jwt.exceptions import (DecodeError, ExpiredSignatureError,
InvalidSignatureError)
def authorize(func):
#wraps(func)
def check_authorization(*args, **kwargs):
try:
jwt_token = request.cookies.get('auth_token') # get token from request
if jwt_token is None:
return Response(status=401, headers={'WWW-Authenticate': 'Bearer'})
token = decode(
jwt_token,
key='pub_key', # public key to validate key
algorithms=['RS256'], # list of algs the key could be signed
verify=True
)
# you can call another function to do check user roles here
# e.g authorize(token['sub'])
return func(*args, **kwargs)
except (InvalidSignatureError, DecodeError, ExpiredSignatureError):
return Response(
response='{ "error": "token_invalid"}',
status=401,
headers={'WWW-Authenticate': 'Bearer'})
return check_authorization
This is supported with flask-jwt-extended: https://flask-jwt-extended.readthedocs.io/en/stable/complex_objects_from_token/
With reference to https://docs.pro.coinbase.com/#get-account-history
HTTP REQUEST
GET /accounts//holds
I am struggling to produce python code to get the account holds via API pagination request and I could not find any example of implementing it.
Could you please advise me on how to proceed with this one?
According to Coinbase Pro documentation, pagination works like so (example with the holds endpoint):
import requests
account_id = ...
url = f'https://api.pro.coinbase.com/accounts/{account_id}/holds'
response = requests.get(url)
assert response.ok
first_page = response.json() # here is the first page
cursor = response.headers['CB-AFTER']
response = requests.get(url, params={'after': cursor})
assert response.ok
second_page = response.json() # then the next one
cursor = response.headers['CB-AFTER']
# and so on, repeat until response.json() is an empty list
You should properly wrap this into helper functions or class, or, even better, use an existing library and save dramatic time.
I am creating the application which need to track all tweets from user who registered to my application, i tried to track those with streaming API , there are public API, user API , and site API,
in those API it just have an option to follow the user ID by add the comma separated user ID
https://dev.twitter.com/streaming/overview/request-parameters#follow
but i think it is not flexible, if there are a new user registered , i need to rebuild the HTTP request , and also if there are so many users try to listen this stream and query will be so long,
it will be
https://stream.twitter.com/1.1/statuses/filter.json?follow=[user1],[user2],[user3]........[userN],
i afraid the query wont fit, i just need a parameter to filter all user who registered in my application such as, for example.
https://stream.twitter.com/1.1/statuses/filter.json?application=[applicationID]
but i think twitter dev does not provide it
so, is there any way to filter stream by application ID?
I didn't see anything like tracking by application id. If your query become too complex (too many follows/keywords), public streaming api will reject it,
and you can't open more than 2 connections with user stream. So, last solution is using Site Stream, -> you can open as many user connections as you have users registered to your app.
BUT the docs says :
"Site Streams is currently in a closed beta. Applications are no
longer being accepted."
Contact twitter to be sure
Arstechnica has a very interesting article about it. Take a look at this code and the link in the end of this post
If you are using python pycurl will do the job. Its provides a way to execute a function for every little piece of data received.
import pycurl, json
STREAM_URL = "http://chirpstream.twitter.com/2b/user.json"
USER = "YOUR_USERNAME"
PASS = "XXXXXXXXX"
userlist = ['user1',...,'userN']
def on_receive(self, data):
self.buffer += data
if data.endswith("rn") and self.buffer.strip():
content = json.loads(self.buffer)
self.buffer = ""
if "text" in content and content['user'] in userlist:
#do stuff
conn = pycurl.Curl()
conn.setopt(pycurl.USERPWD, "%s:%s" % (USER, PASS))
conn.setopt(pycurl.URL, STREAM_URL)
conn.setopt(pycurl.WRITEFUNCTION, on_receive)
conn.perform()
You can find more information here Real time twitter stream api