So I'm trying to setup this script that pipes this data via an API into BigQuery.
It's all being done on the command line, and I've already successfully setup the framework behind it. Specifically, setting up the schema.json file.
When I run the following, it successfully uploads:
bq load --source_format=NEWLINE_DELIMITED_JSON --max_bad_records=10 program_users gs://internal/program_user.json program_users_schema.json
As I said, this successfully pipes into BQ, which is great. But the problem is, this API only allows a max of 50 records at a time when there are over 1000.
EDIT: The initial call to retrieve the records looks like this:
$ curl -s https://api.programs.io/users/57f263fikgi33d8ea7ff4 -u 'dG9rOmU5NGFjYTkwXzliNDFfNGIyJ24iYzA0XzU0NDg3MjE5ZWJkZD=': -H 'Accept:application/json'
Would anyone have a solution to this, particularly, one that is done in the command line?
Related
I am looking to write a script that will send an ssh command over. However, there will be sub commands after a command is entered. I was hoping to do this in a single line.
Example: ssh command
>showstuff
response:>what stuff would you like to show?(1,2,3,4):
>1
response:>would you like to show more info on that stuff?(y,n):
>y
response:>here you go
Is there something to do this in a single line? ie showstuff->1->y
Or is there another easy enough way to handle this?
I found a great library for connecting with SSH. Here's an example of what I used:
Chilkat Example
Link for the library is at the top of the example. I used the ChannelSendString then the ChannelReceiveUntilMatch to wait for the response. Then I sent another command via the ChannelSendString command. Worked like a charm!
I've used this library for a bunch of solutions now.
I'm running a process in a screen (on Ubuntu 13.10, if it matters). I can execute a command within that screen with:
screen -p 0 -X eval 'stuff \"$command\"\015'
I'm not 100% sure what this command is doing to begin with, though it's functioning correctly. The reason behind it is I'm running a Minecraft server (still) and this screens in to the correct screen, and throws the command on the running command line. So that's good, so far.
But what I'd like is to be able to run this command with a return value. So for example, if I were to run a "list" command, it'd tell me how many people and who is online, but I need to capture that output and put it somewhere.
Anyone know of a way to accomplish this? I can't tell the minecraft server command line to redirect the output somewhere else since it doesn't have direct command line access, so the only way I could do this would be to grab all output of the screen while I'm connected ... but I'm not sure if that's possible.
I think you may be able to view the logs? can you not view a running log of the server ?
I'm following the tutorial of loading a dvdrental database as shown in the link below:
http://www.postgresqltutorial.com/load-postgresql-sample-database/
I have done the first step successfully but when I came to the step 2 and 3 which navigating the bin folder and loading the data I was stuck. It keeps prompt me with the invalid command message as shown in the picture below.
Anyone know how to solve this issue? Thanks in advance for all the help!
You should be running pg_restore from the shell, not from inside psql. pg_restore is a separate command line program, not a psql command.
Is there an easy way to directly download all the data contained in a certain dataset on Google BigQuery? I'm actually downloading "as csv", making one query after another, but it doesn't allow me to get more than 15k rows, and rows i need to download are over 5M.
Thank you
You can run BigQuery extraction jobs using the Web UI, the command line tool, or the BigQuery API. The data can be extracted
For example, using the command line tool:
First install and auth using these instructions:
https://developers.google.com/bigquery/bq-command-line-tool-quickstart
Then make sure you have an available Google Cloud Storage bucket (see Google Cloud Console for this purpose).
Then, run the following command:
bq extract my_dataset.my_table gs://mybucket/myfilename.csv
More on extracting data via API here:
https://developers.google.com/bigquery/exporting-data-from-bigquery
Detailed step-by-step to download large query output
enable billing
You have to give your credit card number to Google to export the output, and you might have to pay.
But the free quota (1TB of processed data) should suffice for many hobby projects.
create a project
associate billing to a project
do your query
create a new dataset
click "Show options" and enable "Allow Large Results" if the output is very large
export the query result to a table in the dataset
create a bucket on Cloud Storage.
export the table to the created bucked on Cloud Storage.
make sure to click GZIP compression
use a name like <bucket>/prefix.gz.
If the output is very large, the file name must have an asterisk * and the output will be split into multiple files.
download the table from cloud storage to your computer.
Does not seem possible to download multiple files from the web interface if the large file got split up, but you could install gsutil and run:
gsutil -m cp -r 'gs://<bucket>/prefix_*' .
See also: Download files and folders from Google Storage bucket to a local folder
There is a gsutil in Ubuntu 16.04 but it is an unrelated package.
You must install and setup as documented at: https://cloud.google.com/storage/docs/gsutil
unzip locally:
for f in *.gz; do gunzip "$f"; done
Here is a sample project I needed this for which motivated this answer.
For python you can use following code,it will download data as a dataframe.
from google.cloud import bigquery
def read_from_bqtable(bq_projectname, bq_query):
client = bigquery.Client(bq_projectname)
bq_data = client.query(bq_query).to_dataframe()
return bq_data #return dataframe
bigQueryTableData_df = read_from_bqtable('gcp-project-id', 'SELECT * FROM `gcp-project-id.dataset-name.table-name` ')
yes steps suggested by Michael Manoochehri are correct and easy way to export data from Google Bigquery.
I have written a bash script so that you do not required to do these steps every time , just use my bash script .
below are the github url :
https://github.com/rajnish4dba/GoogleBigQuery_Scripts
scope :
1. export data based on your Big Query SQL.
2. export data based on your table name.
3. transfer your export file to SFtp server.
try it and let me know your feedback.
to help use ExportDataFromBigQuery.sh -h
I have a .bat file shown below in which I want to redirect the whole contents present in my IDE to some text file.
D:\WindRiver\wrenv.exe -p vxworks653-2.2.3 run
D:\WindRiver\wrenv.exe -p vxworks653-2.2.3>C:\ThreePartition\output.txt
PAUSE
I am able to just get some partial output i.e I am unable to get the errors which are thrown during compilation or building process.
Is this correct or Can anyone suggest any other way??
Thanks a lot
Maddy
You can try this:
D:\WindRiver\wrenv.exe -p vxworks653-2.2.3 > C:\ThreePartition\output.txt 2>&1
You can find a good explanation here. Basically you need to redirect both stdout AND stderr to your file.
Best regards.
Your batch is redirecting all messages from wrenv.exe that are sent to the standard output.
I never used WinRiver but usually IDEs manage the console internally and don't log any messages on the standard output/error stream.
It is maybe possible to set the output of the console of the IDE though. If it is, try to set it to the standard output.
I think you want to combine both those lines into one:
D:\WindRiver\wrenv.exe -p vxworks653-2.2.3 run >C:\ThreePartition\output.txt
OK, looking at your posts here, here and here, it seems you want to log the compilation process. The command for that will be something like (all on one line):
make ThreePartition.mak >C:\ThreePartition\output.txt
Assuming there's a file called ThreePartition.mak.
The command you've been using so far is designed to simply open an interface where you can type commands, which is why you get no output. If you want to log simulation, or a kernel build, there is a file called vxworks_cli_tools_users_guide_6.6.pdf which describes the command line interface, including vxprj in full detail.
Also, are you really using a nant script to call a .vbs to call a .bat to call wrenv.exe? I'm sure there's a simpler way to do that.