I'm trying to create a react native app that uses data from a website that is not mine. I need to enter a login and password to get into this website.
I was thinking about using a web scraper on the backend to accomplish this, but I wasn't sure this was the best approach or even possible. How should I go about doing this?
If you could find the API from the network tabs, then use that API instead. Web Scrapers are more prone to be unstable, you wouldn't want to change the scraper again and again if someone from the website changes something in their UI. Reach out for API first, scraper should be your last option.
Related
I'm enjoying developing cross-browser web extensions, the main target being Chrome, so much that I started to think to develop one for my company. I find a chrome extension quite a cheap and efficient way to deploy internal apps. The main purpose is to host a couple of dynamic dashboards that fetch data from various APIs by using cross-domain ajax in background scripts. I finalized the app and I was also able to implement the authentication via chrome.identity and Azure AD.
However, I am struggling to find a safe way to customise the content.
I mean, when the extension is installed it requires to login to azure via the chrome.identity flow. Then I get a token that I use to query ms graph and get the user ID, name, email and basic info.
Until I get this information I want the browser action (popup) to be unavailable to the user as well as any other extension pages. After a successful login I would like to show the content on the pop up and to let the user access the pages, but here I want to customize the experience.
I know how to use the user id retrieved from the api call to customize the extension, but I think it is not safe because all the code is in the client.
If I code something like
if (user === logged) show something
it will be damn easy for a malicious user to look at the code and bypass it, or even to impersonate another user. And chrome extension cannot be obfuscated.
Any help?
Thanks
I'm attempting to create a small website using Directus as a back end, since it has a nice interface for managing a database. The front end will be coded in VueJS, and I'd like them both on the same site. I'm using Apache. For example:
www.example.com = VueJS frontend that pulls information from the Directus backend
www.example.com/admin = Directus admin link
www.example.com/api/[whatever] = RESTful api access
I've messed around with virtualhosts a bit but I can only move the entire directus app to a different port, which I don't want to do since I don't want admins to have to type the port number in manually if they want to access the example.com/admin interface.
I think you can put the API on sub-domain like api.example.com that way both will be available on the same server.
I am just starting with the Directus and met with the same issue of putting both components and thought to put as written above.
If you followed some other approach, pls post here so that it can help people like me.
I have created a small mixin library for Vue.JS 3 and Directus API that makes it easier for developers to make API calls quicker and with almost no code needed.
It supports fetchCollection, fetchBtId, Sort, Filter and Search API endpoints.
Check it out. Hope it helps anybody in the future.
https://github.com/Slaveworx/api-rabbit
I'm using the dropboxd service under Linux, which requires you to log into their website e.g. https://www.dropbox.com/cli_link?host_id=2173bf325f94beee3b1879d2c7b49e69 to link the machine to your account.
Is there any programatic way to do this (ideally using Java)? To access the website above it seems you need to login using forms (which seems tricky to do programatically), and their basic REST API (https://www.dropbox.com/developers/core/docs) doesnt seem to cover the cli_link command.
I could write an app to do the sync using their full API, but it seems like overkill since aside from the cli_link requirement the basic dropboxd does all that I need.
The official Dropbox desktop client is unrelated to the API, though both the API and the Linux CLI require user interaction on the Dropbox web site (once per link) to authorize the linking. Also, note that automating/scraping the site itself is not allowed by the terms:
https://www.dropbox.com/terms#acceptable_use
Not really a solution for DropBox users, but in the end we just moved over to use MediaFire instead. That has a full REST API and doesnt require any manual intervention.
I'm using Java with Google Plus API. I'm using OAuth 2.0. When a user is authenticated, an access code is returned in a browser. Now, given that the code must accompany a call to the Google Plus API, I currently have to manually copy the code and use it in making calls to the Google Plus API. What I wish to do, however, is to programmatically retrieve this code; eliminate the manual copying.
Any assistance will be highly appreciated. Thanks in advance.
It sounds like you're writing a command line or some other non-web application that uses the Google+ API. This throws a little bit of a wrench into the token delivery via HTTP redirect. Without the redirect there's no way for the OAuth web pages to communicate with your code and hence you must copy and paste it.
There is one work around that seems to work pretty well. You can set up a local web server, such as an embedded Jetty, and complete the OAuth flow by redirecting the user back to their locally running web server.
You can see an example of this implemented in oacurl which is hosted here: http://code.google.com/p/oacurl/
I was asked to build a control-system for a Ebay-like Finnish auction-site huuto.net.
The system would reopen closed auctions by a specific rules. It would be completely external from the main site, running at an external website.
The site is however unwilling to release its API and Schema. I know no way to build such a system without knowing its API.
How do you build an internet site without its API and Schema?
You could try some form of automatic browsing: mechanize
Edit:
Examples here.
I think you're asking about building a site that interacts with another site without using a well-defined API. Is that right?
You can interact with an external site without using an official API - in order to do so, you need to imitate a normal site visitor and send your requests to the site frontend (in much the same way as a web crawler does). Tools like hpricot, mechanize and curl can help you parse the content of pages and send requests, but in doing so your system may be quite brittle. Any change to the target site might mean you have to rewrite portions of your system.
It might be possible to get the data you need by screen scraping the site. You could perform the operations you want to do by POSTing data into their forms or using a WebClient type API to make your program act like a web browser but that's likely to be an extremely brittle solution.
Honestly though, without an API, there really is no good solution.
you either need access to the database or an API, otherwise no point in even trying.