I installed a python library called cdsapiin google colab.
To use it I need to locate its config file (which in a general Linux system is $HOME/.cdsapirc) and add my account key to it.
More details can be found here (https://cds.climate.copernicus.eu/api-how-to).
I am having a problem with this step
Copy the code displayed beside, in the file $HOME/.cdsapirc (in your
Unix/Linux environment): url: {api-url} key: {uid}:{api-key}
I tried using !cd /home/ in colab notebook but it doesn't contain this file.
I have also tried !cat /home/.cdsapirc, it gave error:
cat: /home/.cdsapirc: No such file or directory
I achieved this successfully. My code in Colab is as follows:
First, create '.cdsapirc' and write your key in root dir:
url = 'url: https://cds.climate.copernicus.eu/api/v2'
key = 'key: your uid and key'
with open('/root/.cdsapirc', 'w') as f:
f.write('\n'.join([url, key]))
with open('/root/.cdsapirc') as f:
print(f.read())
Then, install cdsapi:
!pip install cdsapi
Run example:
import cdsapi
c = cdsapi.Client()
c.retrieve("reanalysis-era5-pressure-levels",
{
"variable": "temperature",
"pressure_level": "1000",
"product_type": "reanalysis",
"year": "2008",
"month": "01",
"day": "01",
"time": "12:00",
"format": "grib"
}, "/target/dir/download.grib")
The target dir could be your google drive folder.
You can specify your UID, API key and CDS API endpoint directly as arguments into the constructor:
uid = <YOUR UID HERE>
apikey = <YOUR APIKEY>
c = cdsapi.Client(key=f"{uid}:{apikey}", url="https://cds.climate.copernicus.eu/api/v2")
Related
I am trying to poll csv files from S3 buckets using Filepulse source connector. When the task starts I get the following error. What additional libraries do I need to add to make this work from S3 bucket ? Config file below.
Where did I go wrong ?
Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:208)
java.nio.file.FileSystemNotFoundException: Provider "s3" not installed
at java.base/java.nio.file.Path.of(Path.java:212)
at java.base/java.nio.file.Paths.get(Paths.java:98)
at io.streamthoughts.kafka.connect.filepulse.fs.reader.LocalFileStorage.exists(LocalFileStorage.java:62)
Config file :
{
"name": "FilePulseConnector_3",
"config": {
"connector.class": "io.streamthoughts.kafka.connect.filepulse.source.FilePulseSourceConnector",
"filters": "ParseCSVLine, Drop",
"filters.Drop.if": "{{ equals($value.artist, 'U2') }}",
"filters.Drop.invert": "true",
"filters.Drop.type": "io.streamthoughts.kafka.connect.filepulse.filter.DropFilter",
"filters.ParseCSVLine.extract.column.name": "headers",
"filters.ParseCSVLine.trim.column": "true",
"filters.ParseCSVLine.seperator": ";",
"filters.ParseCSVLine.type": "io.streamthoughts.kafka.connect.filepulse.filter.DelimitedRowFilter",
"fs.cleanup.policy.class": "io.streamthoughts.kafka.connect.filepulse.fs.clean.LogCleanupPolicy",
"fs.cleanup.policy.triggered.on":"COMMITTED",
"fs.listing.class": "io.streamthoughts.kafka.connect.filepulse.fs.AmazonS3FileSystemListing",
"fs.listing.filters":"io.streamthoughts.kafka.connect.filepulse.fs.filter.RegexFileListFilter",
"fs.listing.interval.ms": "10000",
"file.filter.regex.pattern":".*\\.csv$",
"offset.policy.class":"io.streamthoughts.kafka.connect.filepulse.offset.DefaultSourceOffsetPolicy",
"offset.attributes.string": "name",
"skip.headers": "1",
"topic": "connect-file-pulse-quickstart-csv",
"tasks.reader.class": "io.streamthoughts.kafka.connect.filepulse.fs.reader.LocalRowFileInputReader",
"tasks.file.status.storage.class": "io.streamthoughts.kafka.connect.filepulse.state.KafkaFileObjectStateBackingStore",
"tasks.file.status.storage.bootstrap.servers": "172.27.157.66:9092",
"tasks.file.status.storage.topic": "connect-file-pulse-status",
"tasks.file.status.storage.topic.partitions": 10,
"tasks.file.status.storage.topic.replication.factor": 1,
"tasks.max": 1,
"aws.access.key.id":"<<>>",
"aws.secret.access.key":"<<>>",
"aws.s3.bucket.name":"mytestbucketamtrak",
"aws.s3.region":"us-east-1"
}
}
What should I put in the libraries to make this work ? Note : The lenses connector sources from S3 bucket without issues. So its not a credentials issue.
As mentioned in comments by #OneCricketeer
Suggest you follow - github.com/streamthoughts/kafka-connect-file-pulse/issues/382 pointed to root cause.
Modifying the config file to use this property sourced the file:
"tasks.reader.class": "io.streamthoughts.kafka.connect.filepulse.fs.reader.AmazonS3RowFileInputReader"
I have this specific use case where from my extension, I want users to be able to type !myamltemplate in a yaml file and be able to load in a designated yaml template for users to start editing. This is similar to the ! Emmet that VSCode currently supports.
So is there a way to create my custom Emmets and if not what's the best way to implement this use case.
Edit1:
I see we have this doc on how to use custom Emmets but the example shows using a html one.
Can this be used for yaml files as well ?
Can the yaml template be loaded from a file
I would like to also have diagnostics and validations on the user edited values in yaml file
Edit2:
I was able to achieve my use case with Snippets
// package.json
"contributes": {
"snippets": [
{
"language": "yaml",
"path": "./snippets.json"
}
]
}
// snippets.json
{
"yaml": {
"prefix": "!mytemplate",
"body": [
"name: install and configure DB",
"hosts: testServer",
"vars:",
" oracle_db_port_value : 1521",
"tasks:",
" -name: Install the Oracle DB",
" yum: <code to install the DB>",
" ",
" -name: Ensure the installed service is enabled and running",
"service:",
" name: <your service name>"
],
"description": "Creates a yaml definition from template"
}
}
Questions
Is this the best approach for this use case ?
Where can I find samples for .tmSnippets as described herein Using TextMate snippets
Thank you!
Currently I have a list of dictionaries in a .bzl file:
test_data = [
{ "name": "test", "data": "test_data"}
]
That I load in a BUILD file and perform some magic with list comprehension...
[
foo(name=data["name"], data=data["data"])
for data in test_data
]
I need to be able to pull this file in from S3 and provide the contents of the BUILD file the same way I do with the static .bzl file.
I am trying to set up config.json for Bitbucket Cloud to automatically update dependencies in npm repos of Bitbucket Cloud. I found one example, but cannot figure out two things:
endpoint - what should go there (ABC)? - our company's bitbucket namespace link looks like: https://bitbucket.org/uvxyz/
Can I use renovate-bot to issue PRs without bitbucket pipelines? If so, can I make renovate to update only particular repo or repos via config.json mods or I shall put renovate.json file in each repo where automatic dependency update is required?
appreciate any examples on the latter.
config.json:
module.exports = {
"platform": "bitbucket",
"username": "<my.username>",
"password": "<bitbucket token on my account>",
"endpoint": "ABC",
"hostRules": [
{
"hostType": "bitbucket",
"domainName": "ABC",
"timeout": 10000,
"username": "<my.username>",
"password": "<bitbucket token on my account>"
}
]
};
according to the code:
const BITBUCKET_PROD_ENDPOINT = 'https://api.bitbucket.org/';
const defaults = { endpoint: BITBUCKET_PROD_ENDPOINT };
there's a default, it works for me without setting it
what you see in the documentation is all you need
I was able to get renovate working with BB after putting the following configuration into its config.js file as
{
hostType: 'bitbucket',
matchHost: 'https://api.bitbucket.org/2.0/',
username: "bb-username",
password: "<special app password generated for bb-username>",
}
for BitBucket app passwords please look at
https://support.atlassian.com/bitbucket-cloud/docs/create-an-app-password/
and
https://support.atlassian.com/bitbucket-cloud/docs/app-passwords/
Is it possible to copy app.js (from dist folder) to a shared drive (inside my organization)? a scripts setting inside Package.json
npm install copy-files-from-to is a good add-on but doesn't copy to a shared drive. I tried fetching a file from a shared drive as it works well but the other way doesn't seem to work.
Here is a copy of the json file:
{ "copyFiles": [ { "from": "dist/js/app.js", "to": "\\\\<Shared Path>\\app.js" } ] }