Laravel seed table from multiple csv files - sql

I'm very new to Laravel and Database and I'm trying to understand how to insert data into my database. Please be patient the question can sounds dummy for you.
STEP
I created a table in migrations. Example of a table:
public function up(){
Schema::create('job-urls', function (Blueprint $table) {
$table->increments('id');
$table->foreign('job_id')->references('id')->on('jobs');
$table->string('url')->index();
$table->string('hash');
$table->timestamp('created_at')->nullable();
$table->timestamp('updated_at')->nullable();
STEP
I have two csv file that correspond to the field url and hash and I want to insert them. I created a new file in migration called populate_jobs_url
class PopulateJoburls extends Migration
{
/**
* Run the migrations.
*
* #return void
*/
public function up(){
$fileurls = fopen('../data/urls.csv', 'r');
$filehash = fopen('../data/urls_hash.csv', 'r');
while (($row = fgetcsv($fileurls, 0, ',')) !=FALSE){
DB::table('joburls')->insert(
array(
'url' => $row,
)
);
}
while (($row = fgetcsv($filehash, 0, ',')) !=FALSE){
DB::table('joburls')->insert(
array(
'hash' => $row,
)
);
}
}
Can you help me to understand how I check if the table is correctly filled? Is this approach correct? How could I insert data otherwise in my Database? Unfortunately all examples on the web deal with inserting manually data with a form.
Thanks

Seeding the table inside of a migration file is not the best practise. You can take advantage of Seeders, which is right way to fill your table with test or actual data.
First, create a seeder file with php artisan make:seeder PopulateJobUrls command. Then you can arrange your seeder like this:
<?php
use Illuminate\Database\Seeder;
use Illuminate\Support\Facades\DB;
class PopulateJobUrls extends Seeder
{
/**
* Run the database seeds.
*
* #return void
*/
public function run()
{
$fileurls = fopen('../data/urls.csv', 'r');
$filehash = fopen('../data/urls_hash.csv', 'r');
// Rest of your seeding logic...
}
}
You should reference your seeder from database/seeds/DatabaseSeeder.php in the run method:
$this->call(PopulateJobUrls::class);
Run php artisan db:seed or if you want to be more specific, php artisan db:seed --class=PopulateJobUrls and you are good to go with your correctly filled data!

Related

Column does not exist Laravel Factory

I have a table
public function up()
{
Schema::create('parties', function (Blueprint $table) {
$table->id();
$table->integer('place_id');
$table->tinyInteger('status')->default(0);
$table->dateTime('utc_date');
$table->dateTime('local_date');
$table->timestamps();
});
Schema::table('parties', function (Blueprint $table) {
$table->index('place_id');
$table->foreign('place_id')
->references('id')
->on('places')
->onDelete('restrict');
});
}
and model factory
<?php
/** #var \Illuminate\Database\Eloquent\Factory $factory */
use App\Models\Party;
use Faker\Generator as Faker;
$factory->define(Party::class, function (Faker $faker) {
$date = $faker->dateTimeBetween(now()->subDays(3), now());
return [
'place_id' => $faker->numberBetween(1, 9),
'status' => $faker->numberBetween(0, 3),
'utc_date' => $date,
'local_date' => \Carbon\Carbon::make($date)->addHours(3),
];
});
If i call the method create in HomeController it works correctly
factory(Party::class, 10)->create();
but when i call it in my tests I got an error
SQLSTATE[42703]: Undefined column: 7 ERROR: column "utc_date" of
relation "parties" does not exist LINE 1: insert into "parties"
("place_id", "status", "utc_date", "lo...
^ (SQL: insert into "parties" ("place_id", "status", "utc_date", "local_date",
"updated_at", "created_at") values (5, 3, 2020-08-04 00:34:52,
2020-08-04 03:34:52, 2020-08-04 11:42:18, 2020-08-04 11:42:18)
returning "id")
<?php
namespace Tests\Feature\API;
use App\User;
use Tests\TestCase;
use Laravel\Sanctum\Sanctum;
use App\Models\{Party, Place};
use Illuminate\Foundation\Testing\RefreshDatabase;
class PartiesControllerTest extends TestCase
{
use RefreshDatabase;
private User $user;
protected function setUp(): void
{
parent::setUp();
$this->user = factory(User::class)->create();
}
public function test_get_parties()
{
factory(Party::class)->create([
'place_id' => factory(Place::class)->create()->id,
]);
dd(Party::all());
Sanctum::actingAs($this->user);
}
}
I am using second database for testing. I connected to this db and found that this table has not been refreshed. I don't know why because I used RefreshDatabase trait. I added this field manually for solving this problem.
Try to run php artisan migrate:fresh. If you use another database for testing and using another .env file then try to run php artisan migrate:fresh --env=testing.
For example: you can use file env.testing and APP_ENV=testing inside this file.
Maybe you added this field later and did not refresh DB. Also, check if the field exists in visual DB.
If it didn't help try to run composer dump-autoload and retry.
please make sure the fillable array in model has this key,
and try using carbon instead of faker with date
(Carbon::now())->subDays(3);

Create tables and fill them with my own data after migration - Laravel

I have a few tables in a database that need specific data. I know I can always save the SQL command and execute them but I wonder if Laravel has some sort of specific command.
You can use seeders for test data, like #Alexey says. If you need the data to persist in all environments (e.g. local and production), you can insert the data after creating the table, with the Query Builder.
e.g.
<?php
class CreatePostsTable extends Migration
{
/**
* Run the migrations.
*
* #return void
*/
public function up()
{
Schema::create('posts', function (Blueprint $table) {
$table->increments('id');
$table->string('title');
$table->text('body');
$table->timestamps();
});
DB::table('posts')->insert([
'title' => 'Hello, world!',
'body' => 'This post will be created after migrating.',
]);
}
Laravel uses seeder classes to fill tables with data after migrations:
You need to create and register seeders and then run this command to run migration and seed the data:
php artisan migrate --seed

Laravel - seeding large SQL file

A memory exhaustion happens when I run my DB seed script in production.
Below is my seed script.
class MembershipTableSeeder extends Seeder
{
public function run()
{
DB::table('members')->delete();
foreach (range(1, 99) as $days){
Members::create(array('membership_code' => 'test'.$days));
}
DB::unprepared(file_get_contents(app_path()."/database/seeds/members.sql"));
}
}
So what I did was add a no-limit on my seed script.
ini_set('memory_limit', '-1');
The problem now is that when I run the script it logs the output into the terminal the content of the SQL script (which is very, very big).
Is there a good way of running a SQL dump inside my DB seeds that doesn't consume much memory? What I did now was run it manually:
mysql -uuser -p db < script.sql
For others who prefer a more Laravel-ish solution, this is how I handled it:
/**
* This class is responsible for running the data dump sql.
* It is recommended to update this class instead of creating new ones for new database content dumps.
*/
class DatabaseDumpSeeder extends Seeder
{
/**
* Run the database seeds.
* #throws \Exception
*/
public function run()
{
// Note: these dump files must be generated with DELETE (or TRUNCATE) + INSERT statements
$sql = file_get_contents(__DIR__ . '/dumps/dump-20150709.sql');
if (! str_contains($sql, ['DELETE', 'TRUNCATE'])) {
throw new Exception('Invalid sql file. This will not empty the tables first.');
}
// split the statements, so DB::statement can execute them.
$statements = array_filter(array_map('trim', explode(';', $sql)));
foreach ($statements as $stmt) {
DB::statement($stmt);
}
}
}
The problem happens because when using Db::unprepared it also logs the query to the laravel.log file, making in background much more actions then you think, from this side you have memory exhaust. If you are not running the safe mode I would stick to executing the console command like this:
exec("mysql -u ".\Config::get('database.mysql.user')." -p".\Config::get('database.mysql.password')." ".\Config::get('database.mysql.database')." < script.sql")
Create Seeder File "PostalCodeTableSeeder.php" in
Project_directory/database/seeds
use Illuminate\Database\Seeder;
class PostalCodeTableSeeder extends Seeder {
/**
* Run the database seeds.
*
* #return void
*/
public function run()
{
// =============================================================
// file Path -> Project/app/configs/database.php
// get the database name, database username, database password
// =============================================================
$db = \Config::get('database.connections.mysql.database');
$user = \Config::get('database.connections.mysql.username');
$pass = \Config::get('database.connections.mysql.password');
// $this->command->info($db);
// $this->command->info($user);
// $this->command->info($pass);
// running command line import in php code
exec("mysql -u " . $user . " -p" . $pass . " " . $db . " &lt postal_codes.sql");
// postal_codes.sql is inside root folder
}
}
Also add the class name into
Project_directory/database/seed/DatabaseSeeder.php like code below
use Illuminate\Database\Seeder;
class DatabaseSeeder extends Seeder
{
/**
* Run the database seeds.
*
* #return void
*/
public function run()
{
$this->call(PostalCodeTableSeeder::class);
// $this->call(UsersTableSeeder::class);
}
}
I had a strange issue where importing a large SQL file as a migration caused the line to not be added to the migrations table.
This is how I fixed it.
$path = 'database/data.sql';
$command = "mysql -h".env('DB_HOST')." -u".env('DB_USERNAME')." ".(env('DB_PASSWORD')?"-p'".env('DB_PASSWORD')."'":'')." ".env('DB_DATABASE')." < ".$path;
exec($command);
I notice that there are similar answers to this, but my method checks .env and not the app config, support for remote MySQL hosts, also works with no password on local and passwords with special characters that would break the command line.

Tx_Extbase_Domain_Repository_FrontendUserRepository->findAll() not working in typo3 4.5.30?

I am trying to run a simple query off of the Tx_Extbase_Domain_Repository_FrontendUserRepository. I cannot get anything to work except findByUid(), not even findAll().
In my controller I have this code which seems to work:
/**
* #var Tx_Extbase_Domain_Repository_FrontendUserRepository
*/
protected $userRepository;
/**
* Inject the user repository
* #param Tx_Extbase_Domain_Repository_FrontendUserRepository $userRepository
* #return void */
public function injectFrontendUserRepository(Tx_Extbase_Domain_Repository_FrontendUserRepository $userRepository) {
$this->userRepository = $userRepository;
}
/**
* action create
*
* #param Tx_BpsCoupons_Domain_Model_Coupon $newCoupon
* #return void
*/
public function createAction(Tx_BpsCoupons_Domain_Model_Coupon $newCoupon) {
...... some code .....
$user = $this->userRepository->findByUid(($GLOBALS['TSFE']->fe_user->user[uid]));
$newCoupon->setCreator($user);
...... some code .....
}
but in another function I want to look up a user not by uid but by a fe_users column called vipnumber (an int column) so I tried
/**
* check to see if there is already a user with this vip number in the database
* #param string $vip
* #return bool
*/
public function isVipValid($vip) {
echo "<br/>" . __FUNCTION__ . __LINE__ . "<br/>";
echo "<br/>".$vip."<br/>";
//$ret = $this->userRepository->findByUid(15); //this works!! but
$query = $this->userRepository->createQuery();
$query->matching($query->equals('vip',$vip) );
$ret = $query->execute(); //no luck
.................
and neither does this
$ret = $this->userRepository->findAll();
How can one work but not the others? In my setup I already put
config.tx_extbase.persistence.classes.Tx_Extbase_Domain_Model_FrontendUser.mapping.recordType >
which seems to be necessary for the fiondByUid to work, is i t preventing the other from working?
I am using typo3 v 4.5.30 with extbase 1.3
Thanks
If $this->userRepository->findByUid(15); works, there is no reason why $this->userRepository->findAll(); should not. However $this->userRepository->findAll(); returns not a single Object but a collection of all objects, so you have to iterate over them.
If you add a column to the fe_users, you have to add it to TCA and to your extbase model (you need a getter and a setter), too! After that you can call findByProperty($property) in your repository. In your case that would be
$user = $this->userRepository->findByVipnumber($vip);
This will return all UserObjects that have $vip set as their Vipnumber. If you just want to check if that $vip is already in use, you can call
$user = $this->userRepository->countByVipnumber($vip);
instead. Which obviously returns the number of Users that have this $vip;
You never use $query = $this->createQuery(); outside your Repository.
To add the property to the fronenduser Model you create your own model Classes/Domain/Model/FronendUser.php:
class Tx_MyExt_Domain_Model_FrontendUser extends Tx_Extbase_Domain_Model_FrontendUser {
/**
* #var string/integer
*/
protected $vipnumber;
}
Add a getter and a setter. Now you create your own FrontendUserRepository and extend the extbase one like you did with the model. You use this repository in your Controller. Now you're almost there: Tell Extbase via typoscript, that your model is using the fe_users table and everything should work:
config.tx_extbase {
persistence{
Tx_MyExt_Domain_Model_FrontendUser{
mapping {
tableName = fe_users
}
}
}
}
To disable storagePids in your repository in general, you can use this code inside your repository:
/**
* sets query settings repository-wide
*
* #return void
*/
public function initializeObject() {
$querySettings = $this->objectManager->create('Tx_Extbase_Persistence_Typo3QuerySettings');
$querySettings->setRespectStoragePage(FALSE);
$this->setDefaultQuerySettings($querySettings);
}
After this, your Querys will work for all PIDs.
I didn't have the opportunity to work with frontend users yet, so I don't know if the following applies in this case:
In a custom table I stumbled uppon the fact, that extbase repositories automatically have a look at the pids stored in each entry and check it against a set storage pid (possibly also the current pid if not set). Searching for a uid usually means you have a specific dataset in mind so automatic checks for other values could logically be ignored which would support your experiences. I'd try to set the storage pid for your extension to the place the frontend users are stored in ts-setup:
plugin.[replace_with_extkey].persistence.storagePid = [replace_with_pid]

how to seed in Yii?

I'm wondering how one can seed in Yii a table once it is created with migration?
I've got a migration with an up-method:
public function up()
{
$this->createTable('users',array('id'=>"pk",
'login'=>'string NOT NULL'));
echo "table 'users' is created.\n";
return true;
}
I've got as well corresponding Users model and its CRUD actions. When I try to execute another migration with an up-method
public function up()
{
$user = new Users;
$user->login = "Bob";
return $user->save();
}
I get the following error:
PHP Error[2]: include(users.php): failed to open stream: No such file or directory
in file MyYiiRoot\yii\framework\YiiBase.php at line 421
I've managed to achieve the desired result by using query builder (by means of insert command), but I hope there is a nicer way out.
Use
public function safeUp()
{
$this->insert('users',array(
'login'=>'Bob'));
}
You can also do update, delete and a host of other actions. Look at http://www.yiiframework.com/doc/api/1.1/CDbMigration for more information