DuckDuckGo API - How to get more results? - api

The default search using the DuckDuckGo API returns only the results on the first page (around 25 I guess). Is there any way to get more results or navigate to the 2nd, 3rd pages of the search results?
Websites like Faroo have a parameter called s (which stands for start) which can be set to 1 if we want the first 10 results, to 11 if we want the next 10 results and so on. Is there something like that for DuckDuckGo, too?

According to DuckDuckGo Search API documentation, all the available parameters are:
q: query
format: output format (json or xml)
If format=='json', you can also pass:
callback: function to callback (JSONP format) pretty: 1 to make JSON
look pretty (like JSONView for Chrome/Firefox)
no_redirect: 1 to skip HTTP redirects (for !bang commands).
no_html: 1 to remove HTML from text, e.g. bold and italics.
skip_disambig: 1 to skip disambiguation (D) Type.
In particular, note that:
This API does not include all of our links, however. That is, it is
not a full search results API or a way to get DuckDuckGo results into
your applications beyond our instant answers.

TL/DR; - Install TamperMonkey, add the short script below (full instructions follow) and the browser will lazy-load the next page(s) automatically as you scroll.
After coming to this answer via Google and not finding the information I was seeking, I wrote this small TamperMonkey script to do the job. I post it here for future googlers.
The below userscript will work with Chrome, Firefox and Opera. Instructions for installation follow below the script, and a brief explanation of what TamperMonkey is follows below that.
This script is inspired by, and named similarly to (in honor of), Endless Google by Tumpio.
// ==UserScript==
// #name Endless DuckDuckGo
// #namespace http://tampermonkey.net/
// #match https://duckduckgo.com/?q=*
// #require http://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js
// #grant none
// ==/UserScript==
(function() {
'use strict';
$(window).scroll(function(){
var els = document.querySelectorAll('.result.result--more');
if (els.length){
var elmore = document.querySelectorAll('.result--more__btn.btn.btn--full');
if (elmore.length){
elmore[0].click();
}
}
});
})(); //
How To Install the Above Script:
Install the TamperMonkey extension for Chrome (or the "add-in" for Firefox).
You will see the TamperMonkey icon appear at the top of the browser
Do a search on DuckDuckGo
Click on the TamperMonkey icon and from the drop-down menu, choose Dashboard
Along the tabs at the top of the Dashboard page, click on the [+] icon at left of the tab strip
The TamperMonkey editor will open up with a blank UserScript template. Delete that entire sample script and replace it with the script from this post.
Save [Ctrl] + [s]
Run another DuckDuckGo search and scroll down the page... True happiness is yours.
What Is TamperMonkey:
A good overview is here.
TamperMonkey is a browser extension, and there is a version of TamperMonkey for each major browser. You probably already use the AdBlock or uBlock browser extensions (if not, WHY NOT?), this is just another extension like those. Anyway, to install for Chrome or Brave, go to the Chrome Web Store and search for TamperMonkey by Jan Binoc. Install it. (Yes, it's safe - there are hundreds of thousands of users, mostly coders). Please consider donating - Jan deserves your support (and no, I don't know him, and yes I donated.)
Before TamperMonkey, there was another extension called GreaseMonkey that did the same thing but only worked on Firefox. However, the GreaseMonkey authors stopped maintaining it or something, and Jan Binoc stepped up to the plate with TamperMonkey.
TamperMonkey allows us to inject our own code into ANY webpage, to programmatically manipulate the web page on our local computers. How does that work? Simplistic Explanation: When you view a web page, you never actually view it "directly from the web server" - your browser first downloads a local copy of the web page code to your browser's cache folder and displays it to you from there. Therefore, TamperMonkey can intercept the page as it loads from cache (on your local hard drive) into the browser and modify it before it is displayed. That explanation is super-simplistic and not fully technically accurate, but in essence that is exactly how it works, and why TamperMonkey works. Most Importantly: The above few lines explain why the page does not change for anyone else - just for you, on your own computer.
TamperMonkey is an excellent reason to learn a bit of javascript/css/html. Using it, you can do stuff like hiding or re-arranging images on a webpage, removing clutter from a page, totally reformatting a page, etc. For example, one of my fav News sites has lots of clutter. So, I go to their RSS feed page, which acts like a great index of articles, but that also has too much stuff I don't want to see (mainly unnecessary thumbnail images and too-narrow columns). I wrote a short TM script to hide all the images and widen the columns and now, instead of seeing 5 or 6 article summaries per screen, I see ~ 20.
The absolute best, most concise, primer for html/css/js that I've ever seen is on Lynda.com. (You might already have access via your local library card - I was greatly surprised to find out that I do.) There is a series by Emma Saunders called D3.js Essential Training for Data Scientists. The course begins with two short tutorials (Recalling HTML Basics (4m) and Understanding HTML5 (3m) ) in html/css/js that are worth a university course tuition by themselves. Why can't everyone teach like this? Anyway, that's all you need - those first two (3 and 4 min) videos. Now, go tweak a webpage.
(Final disclaimer: No, I don't know Emma Saunders either, nor do I have anything to do with either Binoc's or Saunders' products in any way. I'm just a run-of-the-mill user and fan.)

Related

Can vscode's markdown preview scripts trigger actions directly in an extension?

I'm writing a vscode extension where I'm hoping to squeeze more dynamic functionality out of markdown preview. Effectively the problem I'm trying to solve is:
In markdown preview, there's a checkbox
When user clicks the checkbox in markdown preview, send a message/event to the vscode extension runtime
Vscode extension can listen for this message/event and store the action in local storage
Checkbox state is saved - and subsequent renders of the markdown preview can use this action
Ideally, I'd like to do this while keeping the default markdown preview security (https://code.visualstudio.com/Docs/languages/markdown#_strict). After all, I don't need the extension to or markdown preview script to talk to a remote server - I just want them to be able to talk to one another.
Problem as code
To write the problem as sudo code, I want my markdown preview script to contain something like:
const button = ... // get button element
button.addEventListener('click', () => {
... /*
* Send a message to the vscode extension. Something like:
* `vscode.postMessage('vscode.my-extension.preview-action' + value)`
* (which I can't get to work, I'll discuss why)
*/
});
where then my extension can listen for messages like 'vscode.my-extension.preview-action'.
What I've Tried Already
I have tried acquireVsCodeApi() but because the markdown extension already does that, I can't do it again in the subsequent loaded script. I've also tried registering a uri handler but as far as I can try out the preview script still needs to fetch to that uri, which is still blocked by the default markdown security settings.
Perhaps markdown preview scripts are not the place to do this kind of thing, but I just wanted to leverage as much as possible that's already there with the vscode markdown extension. I want to supplement markdown but not replace it, the functionality I want to add is just icing on markdown documentation.
I've read https://code.visualstudio.com/api/extension-guides/markdown-extension#adding-advanced-functionality-with-scripts and it doesn't tell me much about markdown extension scripts capabilities and limitations.
Thanks to #LexLi I looked at some of the source code in the markdown extension and was able to come up with an ugly hack to make this work in preview scripts. Markdown allows normal clicks. And vscode extensions can handle normal clicks. I've paraphrased the code so there could be small syntax errors.
In the extension I did this:
vscode.window.registerUriHandler({
handleUri(uri: vscode.Uri): vscode.ProviderResult<void> {
console.log(`EXTENSION GOT URL: ${uri.toString()}`);
},
});
Then I made sure my extension/preview script put this in the document
<!-- in the preview script I place a button like this -->
<!-- it even works with hidden :) so I can do more app customization -->
<a
hidden
id="my-extension-messager"
href="vscode://publisher-id.my-extension"
>
cant see me but I'm there
</a>
Then my preview script I can even set href before faking a click:
const aMessager = document.querySelector("#my-extension-messager");
console.log('client is setting attribute and clicking...')
aMessager.setAttribute('href', 'vscode://publisher-id.my-extension?action=do-something');
aMessager.click();
console.log('client clicked');
Logs I saw (trimmed/tweaked from my particular extension to match the contrived example):
client is setting attribute and clicking...
client clicked
[Extension Host] EXTENSION GOT URL: vscode://publisher-id.my-extension?action%3Ddo-something
It's a hack but I can do a lot with this. Within the URL I can encode data back to the extension and kind of pass whatever I want (as long as data is relatively small).

Automate a button click on chrome://extensions page using selenium webdriver

I'm trying to write an automated test that will automate the process of updating a google chrome extension. I'm not aware of another method of doing this automatically so here is what I'm currently trying to do:
Open the chrome extensions page (as far as I'm aware this is just an html page unless I'm missing something).
Click on the "Update extensions" button
Here is what I have tried having opened the chrome extensions page:
IwebElement UpdateButton = driver.findelement(By.Id("update-extensions-now"));
UpdateButton.Click();
For some reason the button click is not registering. I have tried some other locators such as CSS path and Xpath but they don't work either. Also, when I debug this test, it passes fine so I know it's not an issue with any of my locators. I have (as a test) tried to automate clicks on the other elements on this page and it's the same issue. I can't get a handle on any elements on the chrome://extensions page at all.
Has anyone encountered this or have any ideas as to what's going on?
You can use the Chrome extensions API to auto-update required extension.
Find the file "manifest.json" in the default Google Chrome
C:\Users\*UserName*\AppData\Local\Google\Chrome\User Data\Default\Extensions
There find the update URL of your extension:
{
"name": "My extension",
...
"update_url": "http://myhost.com/mytestextension/updates.xml",
...
}
The returned XML by the Google server looks like:
<?xml version='1.0' encoding='UTF-8'?>
<gupdate xmlns='http://www.google.com/update2/response' protocol='2.0'>
<app appid='yourAppID'>
<updatecheck codebase='http://myhost.com/mytestextension/mte_v2.crx' version='2.0' />
</app>
</gupdate>
appid
The extension or app ID, generated based on a hash of the public key, as described in Packaging. You can find the ID of an extension or Chrome App by going to the Extensions page (chrome://extensions).
codebase
A URL to the .crx file.
version
Used by the client to determine whether it should download the .crx file specified by codebase. It should match the value of "version" in the .crx file's manifest.json file.
The update manifest XML file may contain information about multiple extensions by including multiple elements.
Another option is to use the --extensions-update-frequency command-line flag to set a more frequent interval in seconds. For example, to make checks run every 45 seconds, run Google Chrome like this:
chrome.exe --extensions-update-frequency=45
Note that this affects checks for all installed extensions and apps, so consider the bandwidth and server load implications of this. You may want to temporarily uninstall all but the one you are testing with, and should not run with this option turned on during normal browser usage.
The request to update each individual extension would be:
http://test.com/extension_updates.php?x=id%3DyourAppID%26v%3D1.1
You can find even more detailed information on exntesions developers site: https://developer.chrome.com/extensions
If you look at the HTML of the "chrome://extensions" page you will notice that the "Update extensions now" button is contained within an iframe. You need to switch to the iframe before trying to register a button click. i.e:
(This is in c#. Note that this code is written from memory so it may not be 100% accurate. Also, you will want to write more robust method. This code just quickly demonstrates that by switching to the iframe, it will work ok)
String ChromeExtensionsPage = "chrome://extensions";
driver.Navigate().GoToUrl(ChromeExtensionsPage);
driver.Switchto().Frame("DesiredFrame");
IwebElement UpdateButton = driver.findelement(By.Id("DesiredButtonID"));
UpdateButton.Click();

Safari extension options pages with access to background page

I'm developing a cross-platform browser extension, and have based all my code on the Chrome-way of doing this. I have counted on that the background page will be accessible from the options page, which in Safari extensions turns out to be not possible (since there is no such thing as an options-page). You can only access safari.extension.globalPage.contentWindow from within the extension popup, and the background page itself.
Now, I have an options page, which is an html-page within the extension bundle, and so far I haven't found a way for Safari to give it extension "rights". The closest I have come is adding a content script that's only added on the options page. This seems a bit silly, since the html page itself is in the extension bundle?!
Others have suggested using asynchronous ping-pong style message event handlers, and even the canLoad-mechanism (which is "only" able to run in a beforeload-event). I have been able to hack the canLoad-mechanism for synchronous messaging by forging the BeforeLoadEvent:
// Content script (run from anywhere)
var result = safari.self.tab.canLoad(new BeforeLoadEvent, "data")
-> "return value"
// Background page
safari.application.addEventListener('message', function(e) {
if ( e.name === "canLoad" )
e.message = "return value";
}, true);
It's a hack, but it works. However, I am crippled by the message transport serialization, since I need to be able access methods and data on my objects from the background page. Is there anyway around this?
Possible ways that might work but I don't know if possible:
Access options-page window-object from backgrounds page. Is that possible?
Message passing, need to bypass message serialization
Any shared/global object that I can attach objects to and fetch from the options page?
Make Safari run the options.html page from outside the content-script sandbox? It works in Chrome since they are both within the extension-bundle. It's quite annoying that safari doesn't do this too.
Run the options-page from within the popup. This is promising, but it crashes safari (which is very promising!). However, from the looks of it it's just something to do with a CSS animation in my options.html page. The biggest issue is that it has to be able to open an OAuth2 popup, but thanks to being able to programmatically open the popover, it might be a non-issue. However, this option is the most realistic, but I would rather have it open in a new tab.
Any suggestions and hackish workarounds would really help.

Automatic text translation at MSDN pages - How to turn off?

Is there a way to turn off the automatic text translation at the MSDN library pages ?
I do prefer English text but due to having a German IP address Microsoft activates the automatic translation on every new page load which gives me a yellow box with a German translation of the text I am currently hovering over with the mouse.
This happens regardless what language is initially set in the right upper corner and regardless of whether I am logged in or not.
I can't tell how annoying this is !!
Any ideas, anyone ?
When you hit the "Original" radio button at the top, you see English, with German in the yellow hover box.
If you visit the original English site, you don't see a translation, not even on hover.
You switch to English by replacing /de-de/ in the URL with /en-us/. As in
German (translation or original with translation on hover):
http://msdn.microsoft.com/de-de/library/system.diagnostics.contracts.contractargumentvalidatorattribute(v=vs.110).aspx
English only (no translation):
http://msdn.microsoft.com/en-us/library/system.diagnostics.contracts.contractargumentvalidatorattribute(v=vs.110).aspx
If you are a firefox user, you can use Redirector addon. Create a new redirect and set it up like this:
It will automatically redirect all msdn requests to english non-translated versions.
Found it! I mean, it's 2016, 3 years late, and maybe they just added it recently, but when you scroll all the way down there's a small button in the left bottom corner where you can choose language you want to use (more specifically a country "you're from").
MSDN uses the prefered language from your web browser settings.
http://social.msdn.microsoft.com/Forums/en-US/6543407d-f743-48fb-965b-b8af9f9a0cb1/howto-disable-automatic-translation-into-german?forum=msdnfeedback
This is due to the Accept-Language header:
http://www.w3.org/International/questions/qa-accept-lang-locales
So setting your browser to prefer English language websites should fix this problem. W3C has an overview how to do that on different browsers here:
http://www.w3.org/International/questions/qa-lang-priorities.en.php
There is a chrome addon aswell
Switcheroo-Redirector
I got tired of replacing manually the url of the MSDN docs to target en-us in the url, so I came up with this little user script for the very handy Tampermonkey extension (available on Chrome, Microsoft Edge, Opera, and Firefox)
// ==UserScript==
// #name MSDN docs [en-us] redirect
// #version 0.1
// #description Redirects to the en-us version of the current MSDN doc page
// #grant none
// #match https://learn.microsoft.com/*
// ==/UserScript==
(function () {
let pathname = window.location.pathname.split('/');
if (pathname[1].toLowerCase() !== 'en-us') {
pathname[1] = 'en-us';
pathname = pathname.join('/');
window.location.href = window.location.origin + pathname + window.location.search;
}
})();
Once you have the extension installed,
Click on its icon
Click on Create a new script...
Paste the previous code
Save it (Ctrl + S or File > Save).
Test the redirection: https://learn.microsoft.com/fr-fr/dotnet/csharp/programming-guide/classes-and-structs/classes
The #match property will ensure that this script is only run against MSDN doc pages.
Recently I came across the same problem. And I solved it with Chrome extension ModHeader.
I configured and it works:
I know it's an old question, by maybe this insight will be useful to someone.
I almost always open msdn through a search in google. It most of the time offered me site translated to my local language (through a part of the address with locale), sometimes accompanied by original (English) version next to it. If I click on the original language link, it does not translate anything, so it is not automatic translation based on my localization.
What solved my problem was to change google search settings to prefer English, rather than my native language. Go to google search settings, set Which language should Google products use? to English, then in Currently showing search results in: click Edit and check other languages you are likely to search in.
It will also change the UI language for google. I know it might be a high price to pay, but I believe it is worth it. If you search for a query typed in given language, results will most likely result in this language pages anyway.
Instead of extensions, which will consume memory and are a bit overkill for that kind of thing, you can use a custom search query.
Chrome
Settings => Manage Search Engines, add this entry:
Engine: MSDN US
Keyword: ms
URL: https://social.msdn.microsoft.com/Search/en-US?query=%s (or whatever the proper url is at the time of your reading, just use %s wherever it needs the actual query string)
Now, in the address bar, just type ms [SPACE]. As soon as you press the space, it will prompt you with Search on MSDN US:. Just type your query now. For instance ms string will redirect you to the MSDN-US version of the search results for string. Of course you can change the title and keyword.
I'm sure the other common browsers expose that kind of functionality too. On Firefox, I used to plug custom search engines on the search bar.
This is a neat trick that I use for all kinds of searches (SO, Amazon, Wikipedia in different languages, etc.). It's very efficient.
Usually there is a language link at the bottom of the page where you can change language (even though a permanent site specific setting would be much nicer).
In IE in Internet Options Panel you have Apperrance part in General Tab. Add preffered language as a first and from now on all pages from MSDN will be presented in choosen language
I'm using NoScript addon with Firefox (actually Waterfox), just forbib "m-msft.com", the translator will be turned off. I think you can use other plugins in other browser to forbid the domain too. NoScript is a must have addon for any serious web user, and UserStyles, of course.

How to stop firefox from downloading and applying CSS via a firefox extension?

Thanks to everyone in advance -
So I have been banging on this issue for quite a while now and have burned through all my options. My current approach to canceling css requests is with nsIRequest.cancel inside of nsIWebProgressListener.onStateChange. This works most of the time, except when things are a little laggy a few will slip through and jump out of the loadgroup before I can get to them. This is obviously a dirty solution.
I have read through the following links to try and get a better idea of how to disable css before a nsIRequest is created...no dice.
https://developer.mozilla.org/en/Document_Loading_-_From_Load_Start_to_Finding_a_Handler
https://developer.mozilla.org/en/The_life_of_an_HTML_HTTP_request
https://developer.mozilla.org/en/Bird's_Eye_View_of_the_Mozilla_Framework
How do I disable css via presentation objects/interfaces? Is this possible? Inside of nsIDocShell there are a few attributes that kind of imply you can disable css via the browsers docshell - allowPlugins, allowJavascript, allowMetaRedirects, allowSubframes, allowImages.
Any suggestions?
Thanks,
Sam
The menu option that disables style sheets uses a function
setStyleDisabled(true)
so you probably can just call this function whenever new browser tab is created. Style sheets are still requested from server, but not applied. This function is not very sophisticated and doesn't mess with nsIRequest, source:
function setStyleDisabled(disabled) {
getMarkupDocumentViewer().authorStyleDisabled = disabled;
}
Digging in Web Developer Toolbar source code I have noticed that their "disable stylesheets" function loops trough all document.styleSheets and sets the disabled property to true, like:
/* if DOM content is loaded */
var sheets = document.styleSheets;
for(var i in sheets){ sheets[i].disabled = true; }
So if the key is to not apply CSS to pages, one of the above solutions should work. But if you really need to stop style sheets from being downloaded from servers, I'm affraid nsIRequest interception is your only option.
Set permissions.default.stylesheet to 2 and voilĂ !
You can actually use the permissions manager to block or allow stylesheets on a host-by-host basis.
Unfortunately there doesn't seem to be a simple flag like allowImages. The bugzilla adding for that is https://bugzilla.mozilla.org/show_bug.cgi?id=340746. You can now vote for it using the new bugzilla voting functionality. You can also add yourself to the CC list to be notified if anyone ever works on it.
A related request is to just give us basic HTML parsing support, which may be what you are trying to do. Unfortunately that isn't supported yet either, but you can vote/track the bugzilla for that at https://bugzilla.mozilla.org/show_bug.cgi?id=102699.
So the only workable solution seems to be some sort of interception as #pawal suggests. Here is a link that talks about the basics of interception to at least get you/us started https://developer.mozilla.org/en/XUL_School/Intercepting_Page_Loads. It lists several options that I list below.
These first few seem to just be at the page/document level so I don't think they help:
Load Events (addEventListener load)
Web Progress Listeners (nsIWebProgressListener) - I tried this approach, it only seems to be called for the page itself, not for content within the page.
Document Loader Service - A global version of nsIWebProgressListener so I think it has the same problem (page level only)
That leaves two others I have not tried yet. They work globally so you would need to filter them to just the browser/pages you care about.
HTTP Observers - Seems like it might work, need to verify it calls back for CSS
Content Policy - Seems like the best option to me since it explicitly is called for CSS, someday I hope to try it :)