KnexJS raw query in migration - sql

I have a problem with the following migration in KnexJS, working with PostgreSQL:
exports.up = (knex) => {
knex.raw('CREATE EXTENSION IF NOT EXISTS "uuid-ossp"');
return knex.schema.createTable('car_brands', (table) => {
table.uuid('brandId').unique().notNullable().primary().defaultTo(knex.raw('uuid_generate_v4()'));
table.string('name').notNullable().unique();
table.timestamp('created_at').notNullable().defaultTo(knex.raw('now()'));
table.timestamp('updated_at').notNullable().defaultTo(knex.raw('now()'));
});
};
exports.down = (knex) => {
knex.raw('drop extension if exists "uuid-ossp"');
return knex.schema.dropTable('car_brands');
};
I am using the UUID type for my default values, by using the
defaultTo(knex.raw('uuid_generate_v4()')).
However, when running the above migration, by:
knex migrate:latest --env development --knexfile knexfile.js --debug true
I get an error that:
function uuid_generate_v4() does not exist
Do you know why the knex.raw() query method is not working?

The problem is you are running
knex.raw('CREATE EXTENSION IF NOT EXISTS "uuid-ossp"');
and
knex.schema.createTable('car_brands');
asynchronously, so the first query is not executed before the second one.
Rewrite it using async/await:
exports.up = async (knex) => {
await knex.raw('CREATE EXTENSION IF NOT EXISTS "uuid-ossp"');
return knex.schema.createTable('car_brands', (table) => {
table.uuid('brandId').unique().notNullable().primary().defaultTo(knex.raw('uuid_generate_v4()'));
table.string('name').notNullable().unique();
table.timestamp('created_at').notNullable().defaultTo(knex.raw('now()'));
table.timestamp('updated_at').notNullable().defaultTo(knex.raw('now()'));
});
};
or using Promises:
exports.up = (knex) => {
knex.raw('CREATE EXTENSION IF NOT EXISTS "uuid-ossp"')
.then(() => {
return knex.schema.createTable('car_brands', (table) => {
table.uuid('brandId').unique().notNullable().primary().defaultTo(knex.raw('uuid_generate_v4()'));
table.string('name').notNullable().unique();
table.timestamp('created_at').notNullable().defaultTo(knex.raw('now()'));
table.timestamp('updated_at').notNullable().defaultTo(knex.raw('now()'));
});
})
};

These are the steps I took to resolve this issue in my app with PostgreSQL, Objection and KNEX.
Go to your database to verify your extension is available.
postgres=# SELECT * FROM pg_extension;
Verify "uuid-ossp" is installed in the database_name you need.
database_name=# CREATE EXTENSION "uuid-ossp"
Back to your app, go to the KNEX migration file where you are altering your table.
t.uuid('user_id').defaultTo(knex.raw('uuid_generate_v4()'));
Use the KNEX command to get the Batch running:
knex migrate:latest
Insert a new raw in your table and verify your UUID has been auto-generated.
I hope these steps can be helpful.

Related

Laravel Row duplication inserted, with updateOrCreate method with Race-Condition

i have function in my controller that create a forecast :
public function updateOrCreate(Request $request, $subdomain, $uuid)
{
$fixture = Fixture::where('uuid',$uuid)->firstOrFail();
request()->validate([
'local_team_score' => 'integer|min:0',
'visitor_team_score' => 'integer|min:0',
'winner_team_id' => 'integer|nullable'
]);
if ($fixture->status !== "PENDING"){
return response()->json([
'message' => "You can not add or modify a forecast if the fixture is not pending"
], 403);
}
$winner_team = null;
// local team win
if ($request->local_team_score > $request->visitor_team_score) {
$winner_team = $fixture->localTeam;
}elseif ($request->local_team_score < $request->visitor_team_score){ //visitor win
$winner_team = $fixture->visitorTeam;
}else{ // draw
$winner_team = FixtureTeam::where('team_id',$request->winner_team_id)->first();
}
$user = auth('api')->user();
$platform = Platform::first();
$forecast = Forecast::updateOrCreate([
'user_id' => $user->id,
'fixture_id' => $fixture->id,
'platform_id' => $platform->id
],[
'local_team_score' => $request->local_team_score,
'visitor_team_score' => $request->visitor_team_score,
'winner_team_id' => is_null($winner_team) ? null : $winner_team->team_id
]);
$forecast->load('winnerTeam');
return new ForecastResource($forecast);
}
As you can see i use updateOrCreate methods to add or update a forecast.
The problem is when 2 requests from the same user run at the same time (and no forecast is already created) 2 row are inserted.
Do you have a solution ?
I See that the problem is not new but i could not find a solution https://github.com/laravel/framework/issues/19372
updateOrCreate does 2 steps:
tries to fetch the record
depending on the outcome does an update or a create.
This operation is not atomic, meaning that between step 1 and 2 another process could create the record and you would end up with duplicates (your situation).
To solve your problem you need following:
determine what columns would give the uniqueness of the record and add an unique index (probably compound between user_id, fixture_id, platform_id)
you need to let database handle the upsert (ON DUPLICATE KEY UPDATE in MySQL, ON CONFLICT (...) DO UPDATE SET in Postgres, etc). This can be achieved in Laravel by using the upsert(array $values, $uniqueBy, $update = null) instead of updateOrCreate.

Use another column as default value in migration

I am trying to add a new column to existing tableName table which has a column anotherColumn.
exports.up = function (knex, Promise) {
return knex.schema.table('tableName', table => {
table.string('newColumn').defaultTo(table.anotherColumn);
});
};
How should I get the value of the existing column into this new column?
The short answer is, you can't: the defaultTo value can't come from another column. However, if you're just trying to have the default take place at the time of migration you could do this:
exports.up = knex =>
knex.schema.table('tableName', t => {
t.string('newColumn')
})
.then(() => knex('tableName').update('newColumn', knex.ref('anotherColumn'));
It should hopefully be obvious that this will not update new rows being inserted following the migration: for that you'd need a trigger, or to ensure that you covered it in your insert code.

Data transformation in logstash

I have a data source, a read-only SQL server, and by combining tables from this server, I am making a log, that I need to upload it Elastic Search.
To do this, I have an API call to the data source, and then have the data transformation happen in Logstash, and then upload it to ES.
I have done the data transformation several times before, in SQL. In SQL I would JOIN several tables and INSERT the query results into a log table, but I don't have SQL option in this setup, I need to do the transformation in logstash.
What I am asking for is best-practice suggestions for logstash.
input {
jdbc {
jdbc_driver_library => "mysql-connector-java-5.1.38-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost/student?user=root&password="
jdbc_user => "Croos"
parameters => {
}
schedule => "* * * * *"
statement => "SELECT * from subject WHERE id > :sql_last_value"
use_column_value => true
tracking_column => id
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
}
This link may help full to you.

cakephp see the compiled SQL Query before execution

My query gets the timeout error on each run. Its a pagination with joins.
I want to debug the SQL, but since I get a timeout, I can't see it.
How can I see the compiled SQL Query before execution?
Some cake code:
$this -> paginate = array(
'limit' => '16',
'joins' => array( array(
'table' => 'products',
'alias' => 'Product',
'type' => 'LEFT',
'conditions' => array('ProductModel.id = Product.product_model_id')
)),
'fields' => array(
'COUNT(Product.product_model_id) as Counter',
'ProductModel.name'
),
'conditions' => array(
'ProductModel.category_id' => $category_id,
),
'group' => array('ProductModel.id')
);
First off, set the debug variable to 2 in app/config/config.php.
Then add:
<?php echo $this->element('sql_dump');?>
at the end of your layout. This should actually be commented out in your default cake layout.
You will now be able see all SQL queries that go to the database.
Now copy the query and use the SQL EXPLAIN command (link is for MySQL) over the database to see what the query does in the DBMS. For more on CakePHP debugging check here.
Since your script doesn't even render you can try to get the latest log directly from the datasource with:
function getLastQuery()
{
$dbo = $this->getDatasource();
$logs = $dbo->getLog();
$lastLog = end($logs['log']);
return $lastLog['query'];
}
This needs to be in a model since the getDatasource() function is defined in a model.
Inspect the whole $logs variable and see what's in there.
One more thing you can do is ....
Go to Cake/Model/DataSource/DboSource.php and locate function execute() and print $sql variable.
That should print the sql.
This certainly is not be the cleanest way (as you are changing Cake directory) .. but certainly would be quickest just to debug if something is not working with sql.
Try...
function getLastQuery($model) {
$dbo = $model->getDatasource();
$logData = $dbo->getLog();
$getLog = end($logData['log']);
echo $getLog['query'];
}
Simple way to show all executed query of your given model:
$sqllog = $this->ModelName->getDataSource()->getLog(false, false);
debug($sqllog);
class YourController extends AppController {
function testfunc(){
$this->Model->find('all', $options);
echo 'SQL: '.$this->getLastQuery();
}
function getLastQuery()
{
$dbo = ConnectionManager::getDataSource('default');
$logs = $dbo->getLog();
$lastLog = end($logs['log']);
return $lastLog['query'];
}
}
or you can get all the query by adding following line in to the function execute() in lib/Cake/Model/DataSource.php
Debugger::dump($sql);
set the debug variable to 2 in app/config/config.php.
echo $this->Payment->save();
Out put like =>SQL Query: INSERT INTO photoora_photoorange.payments VALUES (*******)
[insert query][2]
set the debug variable to 2 in app/config/config.php.
And

Symfony file upload - "Array" stored in database instead of the actual filename

I'm using Symfony 1.4.4 and Doctrine and I need to upload an image on the server.
I've done that hundreds of times without any problem but this time something weird happens : instead of the filename being stored in the database, I find the string "Array".
Here's what I'm doing:
In my Form:
$this->useFields(array('filename'));
$this->embedI18n(sfConfig::get('app_cultures'));
$this->widgetSchema['filename'] = new sfWidgetFormInputFileEditable(array(
'file_src' => '/uploads/flash/'.$this->getObject()->getFilename(),
'is_image' => true,
'edit_mode' => !$this->isNew(),
'template' => '<div id="">%file%</div><div id=""><h3 class="">change picture</h3>%input%</div>',
));
$this->setValidator['filename'] = new sfValidatorFile(array(
'mime_types' => 'web_images',
'path' => sfConfig::get('sf_upload_dir').'/flash',
));
In my action:
public function executeIndex( sfWebRequest $request )
{
$this->flashContents = $this->page->getFlashContents();
$flash = new FlashContent();
$this->flashForm = new FlashContentForm($flash);
$this->processFlashContentForm($request, $this->flashForm);
}
protected function processFlashContentForm($request, $form)
{
if ( $form->isSubmitted( $request ) ) {
$form->bind( $request->getParameter( $form->getName() ), $request->getFiles( $form->getName() ) );
if ( $form->isValid() ) {
$form->save();
$this->getUser()->setFlash( 'notice', $form->isNew() ? 'Added.' : 'Updated.' );
$this->redirect( '#home' );
}
}
}
Before binding my parameters, everything's fine, $request->getFiles($form->getName()) returns my files.
But afterwards, $form->getValue('filename') returns the string "Array".
Did it happen to any of you guys or do you see anything wrong with my code?
Edit: I added the fact that I'm embedding another form, which may be the problem (see Form code above).
Alright, I got it. I wasn't properly declaring my validator.
What i should've done is:
$this->setValidator('filename', new sfValidatorFile(array(
'mime_types' => 'web_images',
'path' => sfConfig::get('sf_upload_dir').'/flash',
)));
Silly mistake, I hope that will help those who have the same problem.
Alternatively you can use;
$this->validatorSchema['filename']
in place of;
$this->setValidator['filename']