I'm new to next.js. I'm trying to cache the whole HTML page in next.js. So, that I can reduce the response time in the next call for that page.
I tried by creating a custom server and then save the response that came from rendertoHTML()/render(). But it didn't return any response.
Redis is not for caching whole HTML pages or any other complex object types for that matter. It's key–value database. In Redis you should keep your object simple.
For your case it's best to use stale-while-revalidate (SWR) cache-control headers in combination with getServerSideProps for server-rendering in NextJS.
See here for example.
Related
I am trying to use sw-precache to enhance the off-line experience in a web app.
As I understand it, the library (and correct me if I am wrong) is to pre-cache all static assets. What about new requests?
Say I want to cache the response of a new get fetch request.
Is there away to override/extend the already provided
self.addEventListener('fetch', ... in the generated sw file
I may be very much missing something here. enlgihten me please...
You need to use the option runtimeCaching. This option enables caching for those new fetch requests.
You pass it an array of objects specifying what urlPattern you want to match, the handler you wish to use for the matched request, and some other options you can check in the link.
When the javascript is loaded, it makes a another ajax request where cookies should be set in the response. However, Splash does not keep any cookies across multiple requests, is there a way to keep the cookies across all requests? Or even assign them manually between each requests.
Yes, there is an example in scrapy-splash README - see Session Handling section. In short, first, make sure that all settings are correct. Then use SplashRequest(url, endpoint='execute', args={'lua_source': script}) to send scrapy requests. Rendering script should be like this:
function main(splash)
splash:init_cookies(splash.args.cookies)
-- ... your script
return {
cookies = splash:get_cookies(),
-- ... other results, e.g. html
}
end
There is also a complete example with cookie handling, header handling, etc. in scrapy-splash README - see a last example here.
I am crawling a website that returns pages with a captcha and a status code 200 suggesting everything is ok. This causes the page to be put into scrapy's cache.
I want to recrawl these pages later. But if they are in the cache, they won't get recrawled.
Is it feasible to overload the process_response function from the httpcache middleware or to look for a specific string in the reponse html and override the 200 code with an error code?
What would be the easiest way to keep scrapy from putting certain responses into the cache.
Scrapy uses scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware to cache http responses. To ignore this caching you can just set request meta keyword dont_cache to True like:
yield Request(url, meta={'dont_cache': True})
The docs above also mention how to disable it project-wide with a setting if you are interested in that too.
I am using node.js to write a web service, it calls an API for some data but I am limited by the API to a number of calls per month, so I wish to cache the data I retrieve from the API so I can serve it up with the cached data, and re-fetch the data from the API at a timed interval.
Is this a good approach for this problem? And what caching framework should I use? I looked at node-redis but I don't think a key value store is appropriate for the data.
Thanks!
I would disagree with you regarding Redis. Redis is a very powerful key-value store that can easily be used for what you want. It is designed to have stuff dumped in it and taken out again. In your situation, you can easy cache the API response by saving it into Redis with the query as the key (if this is a REST API you're calling, you could just use the URL or serialized data as the key) and simply cache the response as a stringified JSON object (or XML string if you happen to be getting that).
You can also set an expiry on the cached data, and it will be cleared when the time is expired.
You could then wrap your API call in a helper function which checks the cache, and returns the value if it's present. If it's not it makes the API request, adds it to the cache, then returns it.
This is probably the most straightforward solution and seems to cover your use case pretty well.
I need to download a few web pages for later usee in my application, and I can't find an easy way to accomplish this task. I would prefer a solution where I don't need to parse the HTML to get the URLs of the images and other resources, but rather download these somehow automatically.
OK guys, here is my solution:
created my own cache object, derived from NSURLCache
added to it a "state" enum variable, with the possible states of: SAVING, LOADING, NOTHING
overwritten cachedResponseForRequest to do things according to the state
SAVING: created a NSMutableDictionary to store every download request
Downloaded the file in the request to a flat file, added the path to the file to the dictionary as an object, with the URL as the key
LOADING: used this dictionary as they did in this example to load back the stored content: http://cocoawithlove.com/2010/09/substituting-local-data-for-remote.html
set my cache object as the shared cache object using [NSURLCache setSharedURLCache:myCacheObject];
After this, when I want to save something I set the cache's state to SAVING, and load a request to an UIWebView. After this I set the state back to LOADING, load a request to an UIWebView, and if I stored my request previously, my cache will load it from the disk.
I think ASIHTTPRequest framework can be useful for you - try ASIWebPageRequest and see if it supports all features you need.