Here is the Microsoft Python example of using spellchecker API:
import http.client, urllib.parse, json
text = 'Hollo, wrld!'
data = {'text': text}
# NOTE: Replace this example key with a valid subscription key.
key = 'MY_API_KEY'
host = 'api.cognitive.microsoft.com'
path = '/bing/v7.0/spellcheck?'
params = 'mkt=en-us&mode=proof'
headers = {'Ocp-Apim-Subscription-Key': key,
'Content-Type': 'application/x-www-form-urlencoded'}
# The headers in the following example
# are optional but should be considered as required:
#
# X-MSEdge-ClientIP: 999.999.999.999
# X-Search-Location: lat: +90.0000000000000;long: 00.0000000000000;re:100.000000000000
# X-MSEdge-ClientID: <Client ID from Previous Response Goes Here>
conn = http.client.HTTPSConnection(host)
body = urllib.parse.urlencode(data)
conn.request ("POST", path + params, body, headers)
response = conn.getresponse()
output = json.dumps(json.loads(response.read()), indent=4)
print (output)
And it works well for mkt=en-us. But if I try to change it, for example to 'fr-FR'. It always answers me with a blank response to any input text.
{
"_type": "SpellCheck",
"flaggedTokens": []
}
Has anybody encountered the similar problem? May it be connected with my trial api key (though they do not mention that trial supports only English)?
Well, I've found out what the problem was. 'mode=proof' — advanced spellchecker currently available only if 'mkt=en-us' (for some Microsoft reasons it does not available even if 'mkt=en-uk'). For all other languages, you should use 'mode=spell'.
The main difference between 'proof' and 'spell' is described like this:
The Spell mode finds most spelling mistakes but doesn't find some of the grammar errors that Proof catches (for example, capitalization and repeated words).
Related
I have an API which gets the success or error message on console.I am new to python and trying to read the response. Google throws so many examples to use subprocess but I dont want to run,call any command or sub process. I just want to read the output after below API call.
This is the response in console when success
17:50:52 | Logged in!!
This is the github link for the sdk and documentation
https://github.com/5paisa/py5paisa
This is the code
from py5paisa import FivePaisaClient
email = "myemailid#gmail.com"
pw = "mypassword"
dob = "mydateofbirth"
cred={
"APP_NAME":"app-name",
"APP_SOURCE":"app-src",
"USER_ID":"user-id",
"PASSWORD":"pw",
"USER_KEY":"user-key",
"ENCRYPTION_KEY":"enc-key"
}
client = FivePaisaClient(email=email, passwd=pw, dob=dob,cred=cred)
client.login()
In general it is bad practice to get a value from STDOUT. There are some ways but it's pretty tricky (it's not made for it). And the problem doesn't come from you but from the API which is wrongly designed, it should return a value e.g. True or False (at least) to tell you if you logged in, and they don't do it.
So, according to their documentation it is not possible to know if you're logged in, but you may be able to see if you're logged in by checking the attribute client_code in the client object.
If client.client_code is equal to something then it should be logged in and if it is equal to something else then not. You can try comparing it's value when you successfully login or when it fails (wrong credential for instance). Then you can put a condition : if it is None or False or 0 (you will have to see this by yourself) then it is failed.
Can you try doing the following with a successful and failed login:
client.login()
print(client.client_code)
Source of the API:
# Login function :
# (...)
message = res["body"]["Message"]
if message == "":
log_response("Logged in!!")
else:
log_response(message)
self._set_client_code(res["body"]["ClientCode"])
# (...)
# _set_client_code function :
def _set_client_code(self, client_code):
try:
self.client_code = client_code # <<<< That's what we want
except Exception as e:
log_response(e)
Since this questions asks how to capture "stdout" one way you can accomplish this is to intercept the log message before it hits stdout.
The minimum code to capture a log message within a Python script looks this:
#!/usr/bin/env python3
import logging
logger = logging.getLogger(__name__)
class RequestHandler(logging.Handler):
def emit(self, record):
if record.getMessage().startswith("Hello"):
print("hello detected")
handler = RequestHandler()
logger.addHandler(handler)
logger.warning("Hello world")
Putting it all together you may be able to do something like this:
import logging
from py5paisa import FivePaisaClient
email = "myemailid#gmail.com"
pw = "mypassword"
dob = "mydateofbirth"
cred={
"APP_NAME":"app-name",
"APP_SOURCE":"app-src",
"USER_ID":"user-id",
"PASSWORD":"pw",
"USER_KEY":"user-key",
"ENCRYPTION_KEY":"enc-key"
}
client = FivePaisaClient(email=email, passwd=pw, dob=dob,cred=cred)
class PaisaClient(logging.Handler):
def __init__():
self.loggedin = False # this is the variable we can use to see if we are "logged in"
def emit(self, record):
if record.getMessage().startswith("Logged in!!")
self.loggedin = True
def login():
client.login()
logging.getLogger(py5paisa) # get the logger for the py5paisa library
# tutorial here: https://betterstack.com/community/questions/how-to-disable-logging-from-python-request-library/
logging.basicConfig(handlers=[PaisaClient()], level=0, force=True)
c = PaisaClient()
c.login()
So I'm studying Scraping website using Selenium:Chrome on Perl, I just wondering how can I download all pdf files from year 2017 to 2021 and store it into a folder from this website https://www.fda.gov/drugs/warning-letters-and-notice-violation-letters-pharmaceutical-companies/untitled-letters-2021 . So far this is what I've done
use strict;
use warnings;
use Time::Piece;
use POSIX qw(strftime);
use Selenium::Chrome;
use File::Slurp;
use File::Copy qw(copy);
use File::Path;
use File::Path qw(make_path remove_tree);
use LWP::Simple;
my $collection_name = "mre_zen_test3";
make_path("$collection_name");
#DECLARE SELENIUM DRIVER
my $driver = Selenium::Chrome->new;
#NAVIGATE TO SITE
print "trying to get toc_url\n";
$driver->navigate('https://www.fda.gov/drugs/warning-letters-and-notice-violation-letters-pharmaceutical-companies/untitled-letters-2021');
sleep(8);
#GET PAGE SOURCE
my $toc_content = $driver->get_page_source();
$toc_content =~ s/[^\x00-\x7f]//g;
write_file("toc.html", $toc_content);
print "writing toc.html\n";
sleep(5);
$toc_content = read_file("toc.html");
This script only download the entire content of the website. Hope someone here can help me and teach me. Thank you very much.
Here is some working code, to help you get going hopefully
use warnings;
use strict;
use feature 'say';
use Path::Tiny; # only convenience
use Selenium::Chrome;
my $base_url = q(https://www.fda.gov/drugs/)
. q(warning-letters-and-notice-violation-letters-pharmaceutical-companies/);
my $show = 1; # to see navigation. set to false for headless operation
# A little demo of how to set some browser options
my %chrome_capab = do {
my #cfg = ($show)
? ('window-position=960,10', 'window-size=950,1180')
: 'headless';
'extra_capabilities' => { 'goog:chromeOptions' => { args => [ #cfg ] } }
};
my $drv = Selenium::Chrome->new( %chrome_capab );
my #years = 2017..2021;
foreach my $year (#years) {
my $url = $base_url . "untitled-letters-$year";
$drv->get($url);
say "\nPage title: ", $drv->get_title;
sleep 1 if $show;
my $elem = $drv->find_element(
q{//li[contains(text(), 'PDF')]/a[contains(text(), 'Untitled Letter')]}
);
sleep 1 if $show;
# Downloading the file is surprisingly not simple with Selenium (see text)
# But as we found the link we can get its url and then use Selenium-provided
# user-agent (it's LWP::UserAgent)
my $href = $elem->get_attribute('href');
say "pdf's url: $href";
my $response = $drv->ua->get($href);
die $response->status_line if not $response->is_success;
say "Downloading 'Content-Type': ", $response->header('Content-Type');
my $filename = "download_$year.pdf";
say "Save as $filename";
path($filename)->spew( $response->decoded_content );
}
This takes shortcuts, switches approaches, and sidesteps some issues (which one need resolve for a fuller utility of this useful tool). It downloads one pdf from each page; to download all we need to change the XPath expression used to locate them
my #hrefs =
map { $_->get_attribute('href') }
$drv->find_elements(
# There's no ends-with(...) in XPath 1.0 (nor matches() with regex)
q{//li[contains(text(), '(PDF)')]}
. q{/a[starts-with(#href, '/media/') and contains(#href, '/download')]}
);
Now loop over the links, forming filenames more carefully, and download each like in the program above. I can fill the gaps further if there's need for that.
The code puts the pdf files on disk, in its working directory. Please review that before running this so to make sure that nothing gets overwritten!
See Selenium::Remove::Driver for starters.
Note: there is no need for Selenium for this particular task; it's all straight-up HTTP requests, no JavaScript. So LWP::UserAgent or Mojo would do it just fine. But I take it that you want to learn how to use Selenium, since it often is needed and is useful.
I'm running Terraform using VScode editor which uses PowerShell as the default shell and getting the same error when I try to validate it or to run terraform init/plan/apply through VScode, external PowerShell or CMD.
The code was running without any issues until I added Virtual Machine creation code. I have clubbed the variables.tf, terraform.tfvars and the main Terraform code below.
terraform.tfvars
web_server_location = "West US 2"
resource_prefix = "web-server"
web_server_address_space = "1.0.0.0/22"
web_server_address_prefix = "1.0.1.0/24"
Environment = "Test"
variables.tf
variable "web_server_location" {
type = string
}
variable "resource_prefix" {
type = string
}
variable "web_server_address_space" {
type = string
}
#variable for network range
variable "web_server_address_prefix" {
type = string
}
#variable for Environment
variable "Environment" {
type = string
}
terraform_example.tf
# Configure the Azure Provider
provider "azurerm" {
# whilst the `version` attribute is optional, we recommend pinning to a given version of the Provider
version = "=2.0.0"
features {}
}
# Create a resource group
resource "azurerm_resource_group" "example_rg" {
name = "${var.resource_prefix}-RG"
location = var.web_server_location
}
# Create a virtual network within the resource group
resource "azurerm_virtual_network" "example_vnet" {
name = "${var.resource_prefix}-vnet"
resource_group_name = azurerm_resource_group.example_rg.name
location = var.web_server_location
address_space = [var.web_server_address_space]
}
# Create a subnet within the virtual network
resource "azurerm_subnet" "example_subnet" {
name = "${var.resource_prefix}-subnet"
resource_group_name = azurerm_resource_group.example_rg.name
virtual_network_name = azurerm_virtual_network.example_vnet.name
address_prefix = var.web_server_address_prefix
}
# Create a Network Interface
resource "azurerm_network_interface" "example_nic" {
name = "${var.resource_prefix}-NIC"
location = azurerm_resource_group.example_rg.location
resource_group_name = azurerm_resource_group.example_rg.name
ip_configuration {
name = "internal"
subnet_id = azurerm_subnet.example_subnet.id
private_ip_address_allocation = "Dynamic"
public_ip_address_id = azurerm_public_ip.example_public_ip.id
}
}
# Create a Public IP
resource "azurerm_public_ip" "example_public_ip" {
name = "${var.resource_prefix}-PublicIP"
location = azurerm_resource_group.example_rg.location
resource_group_name = azurerm_resource_group.example_rg.name
allocation_method = var.Environment == "Test" ? "Static" : "Dynamic"
tags = {
environment = "Test"
}
}
# Creating resource NSG
resource "azurerm_network_security_group" "example_nsg" {
name = "${var.resource_prefix}-NSG"
location = azurerm_resource_group.example_rg.location
resource_group_name = azurerm_resource_group.example_rg.name
# Security rule can also be defined with resource azurerm_network_security_rule, here just defining it inline.
security_rule {
name = "RDPInbound"
priority = 100
direction = "Inbound"
access = "Allow"
protocol = "Tcp"
source_port_range = "*"
destination_port_range = "3389"
source_address_prefix = "*"
destination_address_prefix = "*"
}
tags = {
environment = "Test"
}
}
# NIC and NSG association
resource "azurerm_network_interface_security_group_association" "example_nsg_association" {
network_interface_id = azurerm_network_interface.example_nic.id
network_security_group_id = azurerm_network_security_group.example_nsg.id
}
# Creating Windows Virtual Machine
resource "azurerm_virtual_machine" "example_windows_vm" {
name = "${var.resource_prefix}-VM"
location = azurerm_resource_group.example_rg.location
resource_group_name = azurerm_resource_group.example_rg.name
network_interface_ids = [azurerm_network_interface.example_nic.id]
vm_size = "Standard_B1s"
delete_os_disk_on_termination = true
storage_image_reference {
publisher = "MicrosoftWindowsServer"
offer = "WindowsServerSemiAnnual"
sku = "Datacenter-Core-1709-smalldisk"
version = "latest"
}
storage_os_disk {
name = "myosdisk1"
caching = "ReadWrite"
create_option = "FromImage"
storage_account_type = "Standard_LRS"
}
os_profile {
computer_name = "hostname"
admin_username = "adminuser"
admin_password = "Password1234!"
}
os_profile_windows_config {
disable_password_authentication = false
}
tags = {
environment = "Test"
}
}
Error:
PS C:\Users\e5605266\Documents\MyFiles\Devops\Terraform> terraform init
There are some problems with the configuration, described below.
The Terraform configuration must be valid before initialization so that
Terraform can determine which modules and providers need to be installed.
Error: Invalid character
on terraform_example.tf line 89, in resource "azurerm_virtual_machine" "example_windows_vm":
89: location = azurerm_resource_group.example_rg.location
This character is not used within the language.
Error: Invalid expression
on terraform_example.tf line 89, in resource "azurerm_virtual_machine" "example_windows_vm":
89: location = azurerm_resource_group.example_rg.location
Expected the start of an expression, but found an invalid expression token.
Error: Argument or block definition required
on terraform_example.tf line 90, in resource "azurerm_virtual_machine" "example_windows_vm":
90: resource_group_name = azurerm_resource_group.example_rg.name
An argument or block definition is required here. To set an argument, use the
equals sign "=" to introduce the argument value.
Error: Invalid character
on terraform_example.tf line 90, in resource "azurerm_virtual_machine" "example_windows_vm":
90: resource_group_name = azurerm_resource_group.example_rg.name
This character is not used within the language.
*
I've encountered this problem myself in several different contexts, and it does have a common solution which is no fun at all: manually typing the code back in...
This resource block seems to be where it runs into problems:
resource "azurerm_virtual_machine" "example_windows_vm" {
name = "${var.resource_prefix}-VM"
location = azurerm_resource_group.example_rg.location
resource_group_name = azurerm_resource_group.example_rg.name
network_interface_ids = [azurerm_network_interface.example_nic.id]
vm_size = "Standard_B1s"
delete_os_disk_on_termination = true
storage_image_reference {
publisher = "MicrosoftWindowsServer"
offer = "WindowsServerSemiAnnual"
sku = "Datacenter-Core-1709-smalldisk"
version = "latest"
}
storage_os_disk {
name = "myosdisk1"
caching = "ReadWrite"
create_option = "FromImage"
storage_account_type = "Standard_LRS"
}
os_profile {
computer_name = "hostname"
admin_username = "adminuser"
admin_password = "Password1234!"
}
os_profile_windows_config {
disable_password_authentication = false
}
tags = {
environment = "Test"
}
}
Try copying that back into your editor as is. I cannot see any problematic characters in it, and ironically StackOverflow may have done you a solid and filtered them out. Literally copy/pasting it over the existing block may remedy the situation.
I have seen Terraform examples online with stylish double quotes (which aren't ASCII double quotes and won't work) many times. That may be what you are seeing.
Beyond that, you'd need to push your code to GitHub or similar so I can see the raw bytes for myself.
In the off-chance this helps someone who runs into this error and comes across it on Google, I just thought I would post my situation and how I fixed it.
I have an old demo Terraform infrastructure that I revisited after months and, long story short, I issued this command two days ago and forgot about it:
terraform plan -out=plan.tf
This creates a zip archive of the plan. Upon coming back two days later and running a terraform init, my terminal scrolled garbage and "This character is not used within the language." for about 7 seconds. Due to the .tf extension, terraform was looking at the zip data and promptly pooping its pants.
Through moving individual tf files to a temp directory and checking their validity with terraform init, I found the culprit, deleted it, and functionality was restored.
Be careful when exporting your plan files, folks!
I ran into the same problem and found this page.
I solved the issue and decided to post here.
I opened my plan file in Notepad++ and selected View-Show all symbols.
I removed all the TAB characters and replaced them with spaces.
In my case, the problem was fully resolved by this.
In my case, when I ran into the same problem ("This character is not used within the language"), I found the encoding of the files was UTF-16 (it was a generated file from PS). Changing the file encoding to UTF-8 (as mentioned in this question) solved the issue.
I found I got this most often when I go from Windows to linux. The *.tf file does not like the windows TABs and Line Breaks.
I tried to some of the same tools I use when I have this problem with *.sh, but so far I've resorted to manually cleaning up the lines I've seen in there error.
In my case, the .tf file was generated by the following command terraform show -no-color > my_problematic.tf, and this file's encoding is in "UTF-16 LE BOM", converting it to UTF-8 fixed my issue.
I followed softlayer-object-storage-python in order to return a list of my objects matching a specific criteria.
This code seems to just return everything in my container no matter what I put into the search
sl_storage = object_storage.get_client(
username = environment['slos_username'],
password = environment['api_key'],
auth_url = environment['auth_url']
)
# get container
sl_container = sl_storage[environment['object_container']]
# get list, the search function doesn't actually work...
containers = sl_container.search("icm10restapi-qa.zip.*")
I expect only to get back things that start with icm10restapi-qa.zip.
I also tried using ^=icm10restapi-qa.zip but no luck either.
Reviewing the method, it seems that there is not possible to filter the objects as you would like:
https://github.com/softlayer/softlayer-object-storage-python/blob/master/object_storage/client.py#L147
API Operations for Search Services
My apologizes for the inconveniences, I recommended to try filter these in your code.
Updated
This script will help to filter your objects with the name which starts as specific string
import object_storage
import pprint
# Declare username, apikey and datacenter
USERNAME = 'set me'
API_KEY = 'set me'
DATACENTER = 'https://dal05.objectstorage.softlayer.net/auth/v1.0/'
# Creating object storage connection
sl_storage = object_storage.get_httplib2_client(USERNAME, API_KEY, auth_url=DATACENTER)
# Declare name to filter
name = 'icm10restapi-qa.zip'
# Filtering
containers = sl_storage.search(name)
for container in containers['results']:
if container.__dict__['name'].startswith(name):
print(container)
I have made a web2py web application. The api endpoints exposed are as follows.
"/comments[comments]"
"/comments/id/{comments.id}"
"/comments/id/{comments.id}/:field"
"/comments/user-id/{comments.user_id}"
"/comments/user-id/{comments.user_id}/:field"
"/comments/date-commented/{comments.date_commented.year}"
"/comments/date-commented/{comments.date_commented.year}/:field"
"/comments/date-commented/{comments.date_commented.year}/{comments.date_commented.month}"
"/comments/date-commented/{comments.date_commented.year}/{comments.date_commented.month}/:field"
"/comments/date-commented/{comments.date_commented.year}/{comments.date_commented.month}/{comments.date_commented.day}"
"/comments/date-commented/{comments.date_commented.year}/{comments.date_commented.month}/{comments.date_commented.day}/:field"
"/comments/date-commented/{comments.date_commented.year}/{comments.date_commented.month}/{comments.date_commented.day}/{comments.date_commented.hour}"
"/comments/date-commented/{comments.date_commented.year}/{comments.date_commented.month}/{comments.date_commented.day}/{comments.date_commented.hour}/:field"
"/comments/date-commented/{comments.date_commented.year}/{comments.date_commented.month}/{comments.date_commented.day}/{comments.date_commented.hour}/{comments.date_commented.minute}"
"/comments/date-commented/{comments.date_commented.year}/{comments.date_commented.month}/{comments.date_commented.day}/{comments.date_commented.hour}/{comments.date_commented.minute}/:field"
"/comments/date-commented/{comments.date_commented.year}/{comments.date_commented.month}/{comments.date_commented.day}/{comments.date_commented.hour}/{comments.date_commented.minute}/{comments.date_commented.second}"
"/comments/date-commented/{comments.date_commented.year}/{comments.date_commented.month}/{comments.date_commented.day}/{comments.date_commented.hour}/{comments.date_commented.minute}/{comments.date_commented.second}/:field"
"/comments/complaint-id/{comments.complaint_id}"
"/comments/complaint-id/{comments.complaint_id}/:field"
The comments model is as follows
models/db.py
db.define_table(
'comments',
Field('user_id', db.auth_user),
Field('comment_made', 'string', length=2048),
Field('date_commented', 'datetime', default=datetime.now),
Field('complaint_id', db.complaints),
Field('detailed_status', 'string', length=2048),
)
I have been successful in retriving a single comment via the following request
localhost:8000/api/comments/id/1.json
Now I wish to retrieve all the comments. I am not able to figure out how to use /comments[comments] to retrieve all comments.?
I have tried
localhost:8000/api/comments.json
But it gives an output with "invalid path"
I have realized requests such as http://localhost:8000/api/comments/complaint-id/1.json
also give "invalid path" as output.
Please help.
EDIT:
Controllers/default.py
#request.restful()
def api():
response.view='generic.' + request.extension
def GET(*args,**kargs):
patterns='auto'
parser = db.parse_as_rest(patterns,args,kargs)
if parser.status == 200:
return dict(content=parser.response)
else:
raise HTTP(parser.status,parser.error)
def POST(*args,**kargs):
return dict()
return locals()
routes.py in the main web2py folder to change the default application:
routers = dict(
BASE = dict(
default_application='GRS',
)
)
Another observation:
I added another endpoint as below:
def comm():
"""" Comments api substitute"""
rows=db().select(db.comments.ALL) ## this line shows error
# rows = db(db.comments.id > 0).select()
#rows=[[5,6],[3,4],[1,2]]
#for row in rows:
# print row.id
return dict(message=rows)
Even now I am not able to retrieve all comments with "/comm.json". This gives a web2py error ticket which says "need more than 1 value to unpack" on the line "rows=db.select(db.comments.ALL)". Are the above invalid path and this error related in someway?