I query the Big Query table from PHP script and get the result. I want to save the result in new resultant table for future need...
The option #Pentium10 mentioned works, but there is another way to do it, when you already have query results and you want to save them.
All queries in BigQuery generate output tables. If you don't specify your own destination table, the output table will be an automatically generated table that only sticks around for 24 hours. You can, however, copy that temporary table to a new destination table and it will stick around for as long as you like.
To get the destination table, you need to look up the job (if you use the jobs.query() api, the job id is in the jobReference field of the response (see here). To look up the job, you can use jobs.get() with that job id, and you'll get back the destination table information (datasetId and tableId) from the configuration.query.destinationTable (the job object is described here).
You can copy that destination table to your own table by using the jobs.insert() call with a copy configuration section filled out. Info on copying a table is here.
Passing parameters to BQ call is tricky.
This should work in more recent cloud library versions:
public function runBigQueryJobIntoTable($query, $project, $dataset, $table)
{
$bigQuery = new BigQueryClient(['projectId' => $project]);
$destinationTable = $bigQuery->dataset($dataset)->table($table);
$queryJobConfig = $bigQuery->query($query)->destinationTable($destinationTable);
$job = $bigQuery->startQuery($queryJobConfig);
$queryResults = $job->queryResults();
while (!$queryResults->isComplete()) {
sleep(1);
$queryResults->reload();
}
return true;
}
For old versions:
public function runBigQueryJobIntoTable($query, $project, $dataset, $table)
{
$bigQuery = new BigQueryClient(['projectId' => $project]);
$jobConfig = [
'destinationTable' => [
'projectId' => $project,
'datasetId' => $dataset,
'tableId' => $table
]
];
$job = $bigQuery->runQueryAsJob($query, ['jobConfig' => $jobConfig]);
$queryResults = $job->queryResults();
while (!$queryResults->isComplete()) {
sleep(1);
$queryResults->reload();
}
return true;
}
You need to set the destinationTable with the call, the results will be written to the table you set.
https://developers.google.com/bigquery/querying-data#asyncqueries
Related
I use pg_dump to make a copy of the database, copy1.sql.
I run an up migration to create a new instance
up: asyn (queryInterface) => {
return await queryInterface.bulkInsert('keys', [{ clientKey: 'key123' }]);
}
I run a down migration to delete the instance
down: async (queryInterface) => {
return await queryInterface.bulkDelete('keys', { clientKey: ['key123'] });
}
I do another pg_dump of the database, copy2.sql. I compare the first copy of the database with the second copy of the database to show that the down migration worked properly by running a bash script
diff "copy1.sql" "copy2.sql"
The difference is
-SELECT pg_catalog.setval('public.keys_id_seq', 6, true);
+SELECT pg_catalog.setval('public.keys_id_seq', 7, true);
This makes my test fail because the copies of both databases are not identical due to this difference. Even though I deleted that key, it's saying the next id sequence is going to start at 8 instead of 7 according to this document. The table rows that currently exists are 1 through 6. Is there a way to delete the instance so that the sequence will start at 7 instead of 8? Meaning both copies of the database should have
SELECT pg_catalog.setval('public.keys_id_seq', 6, true);
Are there options I can include? Maybe something like
down: async (queryInterface) => {
return await queryInterface.bulkDelete('keys', { clientKey: ['key123'] }, { resetIdSequence: true });
}
You can reset sequence using the truncate table command. Truncate table command erases all table data. For example:
truncate table table_name restart identity;
Second way manual resetting using setval. Example:
select setval('your_table_id_seq', 1, false);
If you don't delete all table data, then recommended set sequence value to the maximum id of records. Example:
select setval('your_table_id_seq', COALESCE((select max(id)+1 from your_table), 1), false);
I know it might be too late for you but I had the same problem and resolved it by adding {restartIdentity: true} on my migration file, like this (example of one of my tables):
async down(queryInterface, Sequelize) {
await queryInterface.dropTable('card', {restartIdentity: true});
}
To be sure it works I tried several "turns" with those commands on the terminal :
npx sequelize db:migrate, npx sequelize db:seed:all : everything is on place :)
npx sequelize db:migrate:undo:all : no tables, good!
npx sequelize db:migrate, npx sequelize db:seed:all : everything is good, foreign keys are still the good ones, great !
So for your code you could try this :
down: async (queryInterface) => {
return await queryInterface.bulkDelete('keys', { clientKey: ['key123'] }, {restartIdentity: true});
}
Hope this helps ;)
Short question: I would like to split a BQ table into multiple small tables, based on the distinct values of a column. So, if column country has 10 distinct values, it should split the table into 10 individual tables, with each having respective country data. Best, if done from within a BQ query (using INSERT, MERGE, etc.).
What I am doing right now is importing data to gstorage -> local storage -> doing splits locally and then pushing into tables (which is kind of a very time consuming process).
Thanks.
If the data has the same schema, just leave it in one table and use the clustering feature: https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#creating_a_clustered_table
#standardSQL
CREATE TABLE mydataset.myclusteredtable
PARTITION BY dateCol
CLUSTER BY country
OPTIONS (
description="a table clustered by country"
) AS (
SELECT ....
)
https://cloud.google.com/bigquery/docs/clustered-tables
The feature is in beta though.
You can use Dataflow for this. This answer gives an example of a pipeline that queries a BigQuery table, splits the rows based on a column and then outputs them to different PubSub topics (which could be different BigQuery tables instead):
Pipeline p = Pipeline.create(PipelineOptionsFactory.fromArgs(args).withValidation().create());
PCollection<TableRow> weatherData = p.apply(
BigQueryIO.Read.named("ReadWeatherStations").from("clouddataflow-readonly:samples.weather_stations"));
final TupleTag<String> readings2010 = new TupleTag<String>() {
};
final TupleTag<String> readings2000plus = new TupleTag<String>() {
};
final TupleTag<String> readingsOld = new TupleTag<String>() {
};
PCollectionTuple collectionTuple = weatherData.apply(ParDo.named("tablerow2string")
.withOutputTags(readings2010, TupleTagList.of(readings2000plus).and(readingsOld))
.of(new DoFn<TableRow, String>() {
#Override
public void processElement(DoFn<TableRow, String>.ProcessContext c) throws Exception {
if (c.element().getF().get(2).getV().equals("2010")) {
c.output(c.element().toString());
} else if (Integer.parseInt(c.element().getF().get(2).getV().toString()) > 2000) {
c.sideOutput(readings2000plus, c.element().toString());
} else {
c.sideOutput(readingsOld, c.element().toString());
}
}
}));
collectionTuple.get(readings2010)
.apply(PubsubIO.Write.named("WriteToPubsub1").topic("projects/fh-dataflow/topics/bq2pubsub-topic1"));
collectionTuple.get(readings2000plus)
.apply(PubsubIO.Write.named("WriteToPubsub2").topic("projects/fh-dataflow/topics/bq2pubsub-topic2"));
collectionTuple.get(readingsOld)
.apply(PubsubIO.Write.named("WriteToPubsub3").topic("projects/fh-dataflow/topics/bq2pubsub-topic3"));
p.run();
Ok so basically I need to convert this regular sql statement to the syntax joomla uses via
https://api.joomla.org/11.4/Joomla-Platform/Database/JDatabaseQuery.html
here is my statement
SET #myunsubid = (
SELECT subid
FROM aqbi8_acymailing_subscriber s
WHERE s.email = 'email#email.co.nz'
);
SELECT #myunsubid;
UPDATE aqbi8_acymailing_listsub a
SET a.`status` = 1
WHERE a.subid = #myunsubid AND a.listid = 232
So id like it to be like
$db->set(#myunsubid = ( $db->select($db->quoteName('subid') )
$db->from($db->quoteName('aqbi8_acymailing_subscriber s') )
$db->where($db->quoteName('s.email') = 'email#email.co.nz')
)
$db->update($db->quoteName('aqbi8_acymailing_listsub a'))
$db->set($db->quoteName('a.status') = 1)
$db->where ($db->quoteName('a.subid') = #myunsubid AND $db->quoteName('a.listid') = 232 )
But this isnt quite right. please help!
I actually figured it out got it to work like this.
$db = &JDatabase::getInstance($option);
$query = $db->getQuery(true);
// make a variable for subID
$query->select($db->quoteName(array('subid')));
$query->from($db->quoteName('aqbi8_acymailing_subscriber'));
$query->where($db->quoteName('email') . " = '" . $email ."'");
$db->setQuery($query);
$db->execute();
$test = $db->loadObjectList();
print_r( $test );
$myid = $test[0]->subid;
$query->clear();
// // Create Database query
$fields = $db->quoteName('status') . ' = 1';
$conditions = array(
$db->quoteName('subid') . ' = ' . $myid,
$db->quoteName('listid') . ' = ' . $listid
);
// // update query
$query->update($db->quoteName('aqbi8_acymailing_listsub'))->set($fields)->where($conditions);
$db->setQuery($query);
$db->execute();
You don't need to make two trips to the database, you can write a subquery into your UPDATE's WHERE condition (no mysql variables or table aliases are necessary).
Raw Query:
UPDATE aqbi8_acymailing_listsub
SET status = 1
WHERE listid = 232
AND subid = (
SELECT subid
FROM aqbi8_acymailing_subscriber
WHERE `email` = 'email#email.co.nz'
)
Tested Code:
$db = JFactory::getDBO();
try {
$subquery = $db->getQuery(true)
->select('subid')
->from('#__acymailing_subscriber')
->where("email = 'email#email.co.nz'");
$query = $db->getQuery(true)
->update("#__acymailing_listsub")
->set("status = 1")
->where(["listid = 232", "personid = ($subquery)"]); // or make 2 where() calls
echo $query->dump(); // if you want to see; *during development ONLY
$db->setQuery($query);
$db->execute();
if ($affrows = $db->getAffectedRows()) {
JFactory::getApplication()->enqueueMessage("Updated. Rows affected: $affrows", 'success');
} else {
JFactory::getApplication()->enqueueMessage("Logic Error", 'error');
}
} catch (Exception $e) {
JFactory::getApplication()->enqueueMessage("Query Syntax Error: " . $e->getMessage(), 'error'); // never show getMessage() to public
}
It is not clear if any of your values are coming from users/untrusted sources, so be sure to follow good practices when writing variables into your queries -- like casting integers with (int) and calling $db->quote() on string values.
If you want to see a complex/convoluted UPDATE query with several other tables and techniques blended in, here is a comprehensive post: https://joomla.stackexchange.com/a/22916/12352
Please DON'T USE JDatabase Object to update Joomla tables, when there's an API available for the extension.
Whilst I appreciate the OP's question is pertaining to how to update the joomla database using the joomla database object (JDatabase), I propose a safer and more robust method, the "ACYMailing API".
"BUT WHY?", I hear you ask...
Good question!!!
There are 2 pitfalls in updating the joomla database directly - be it on the command-line, in a GUI such as MySQL Workbench or PHPMyAdmin, or even with the Joomla Database Object. Simply put, they both concern compatibility - 1. regarding third party integrations, and 2. concerning the future compatibility of your code. In a nutshell, whenever there's a an API for interacting with a component, I'd use it, over JDatabase every time to future proof your code, and ensure that all pre and post save, update, delete... ...move, and publish plugin events take care of your integrations, just as if you'd performed the action authentically.
To elaborate on these points a bit...
Most Joomla extensions (core and 3rd-party) make use of Joomla's powerful plugin architecture. By doing so, extensions can perform actions at key points in the application's life cycle. For example, after deleting a record from a table belonging to component1, delete related records from a table relating to compnent2. Therefore, one run's the risk of breaking the behaviour/functionality of the component in question - i.e. ACY Mailing, as in your case. Potentially, other core/3rd-party extensions that rely on ACY's data, that would otherwise, get updated through onAfterSave() or onAfterDelete() plugin events, as they will not get called.
There's a big risk that your code to break with future Joomla/ACY Mailing updates, if/when the table structure changes.
OK, so how do we use the API?
The following example code displays everything that you should need to update a subscription record. Each step explains the code, which for reference, is summarised in doc and inline comments in the code itself. To begin, navigate to the file where you are entering your code, then...
STEP BY STEP
STEP 1: Check the existence of ACY Mailing by attempting to include it's helper class, as follows. N.B. If the include_once() fails, you should see the echo statement, indicating that ACY Mailing IS NOT installed.
// load the ACY Mailing helper - bail out if not
if(!include_once(rtrim(JPATH_ADMINISTRATOR, DIRECTORY_SEPARATOR) . DIRECTORY_SEPARATOR . 'components' . DIRECTORY_SEPARATOR . 'com_acymailing' . DIRECTORY_SEPARATOR . 'helpers' . DIRECTORY_SEPARATOR . 'helper.php')){
echo 'This code can not work without the AcyMailing Component';
return false;
}
STEP 2: Set-up your parameters by inputting values into the following 3 variables. See examples in code comments.
// array $lists An array of integer IDs (primary keys) of the lists you want the user to be subscribed to (can be empty).
// e.g. array(2,4,6)
$lists = array();
// array $unsubs An array of integer IDs (primary keys) of the lists you want the user to be un-subscribed from (can be empty).
// e.g. array(2,4,6)
$unsubs = array();
// string $userID Numeric Joomla User or user e-mail. For example: '42' or 'name#domain.com'
$userID = '';
STEP 3: Add the following code to find the ACY Mailing user, from the Joomla User ID/Email address passed in to the ->subid() method, and bail out if not found.
// instantiate the ACY Mailing Subscriber (user) Class
$user = acymailing_get('class.subscriber');
// find the ACY Mailing user id (subid) from the joomla ID or email address set in $userID
$subID = $user->subid($userID);
// No ACY Mailing user/subscriber?
if(empty($subID))
return; // bail out
STEP 4: Add the following code to check, and setup the data for any of the subscriptions/unsubscriptions you've configured to update ($lists and $unsubs arrays). If any found, they will be updated. If not found, return.
// create an array to store data in
$data = array();
// Set up new newsletter subscriptions from the $lists array()
if(!empty($lists)) foreach($lists as $listId)
$data[$listId] = array("status" => 1);
// Set up un-subscriptions from the $unsubs array()
if(!empty($unsubs)) foreach($unsubs as $listId)
$data[$listId] = array('status' => 0);
// no data, bail out...
if(empty($data))
return; //there is nothing to do...
// update the user's subscription records, creating/removing subscriptions/unsubsriptions accordingly
$user->saveSubscription($subID, $data);
I need to save same model($modelcrite) multiple times after changing a id here is the code
protected function saveed($data1,$studentid,$modelcrite,$model)
{
$index = 0;
foreach ($data1 as $key => $value) {
$studentid[$index]=(string)$key;
$modelcrite->setAttribute('st_id',$key);
if ($modelcrite->validate()){
$modelcrite->setScenario('insert');
$modelcrite->save();
}
else {
$this->Delete($model->ass_id);
return FALSE;
}
$index=$index+1;
}
return TRUE;
}
but the problem is when second time save the $modelcrite value it is updating the database.I need to save it as new one
please can any one tell how can i do that.thank you.
I believe that you confusion comes from the fact that an ActiveRecord model does not represent a table, but instead represents a row in the table. So changing the terminology a bit, when you pass in the new model(row) $modelcrite ActiveRecord recognizes that this is a new model(row), and the save performs an insert. At this point now Active record realizes that the model(row) exists in the table, and any additional saves against that model(row) will generate an update.
With that background in mind what you need to do is in your foreach loop create a new model for the insert, and set the attributes of the new model to $modelcrite, then set the student_id, then validate and save.
Something like this:
foreach ($data1 as $key => $value) {
$newModel = new CriteModel();
$newModel->attributes = $modelcrite->attributes;
$studentid[$index]=(string)$key;
$newModel->setAttribute('st_id',$key);
if ($newModel->validate()){
$newModel->save();
I am trying to get the actual response (the data) from my database using prepared statements:
$stmt=$dbconn->prepare("SELECT user_videos FROM public.account_recover_users WHERE user_mail= :email");
$stmt->execute(array(':videos'=>$json_videos,':email'=>$email));
I know that $stmt->execute(array(':videos'=>$json_videos,':email'=>$email)); will return a boolean, not the actual data. But how to get the data from my database into an array? I will need to later return that data, the script is accessed via a GET request, and I will need to do exit("{'data':$data_from_db}"); so I don't want to fetch each row using foreach($stmt as $row). Just pass it all as it is.
$results = array();
while($row = $stmt->fetch(PDO::FETCH_ASSOC)){
$results[] = $row;
}