I have tried
val schemas = addresses.schema
val setup = schemas.create
val db = Database.forConfig("h2disk")
Await.result(db.run(setup), Duration.Inf)
but, apparently, it is not working. Here are some logs
[error] Caused by: org.h2.jdbc.JdbcSQLException: Schema "apps" not found; SQL statement:
[error] create table "apps"."t_address" ("name" VARCHAR,"domain" VARCHAR,"t_address_id" VARCHAR NOT NULL PRIMARY KEY) [90079-196]
[error] at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
[error] at org.h2.message.DbException.get(DbException.java:179)
[error] at org.h2.message.DbException.get(DbException.java:155)
[error] at org.h2.command.Parser.getSchema(Parser.java:688)
[error] at org.h2.command.Parser.getSchema(Parser.java:694)
We can try
val schemas = addresses.schema
val setup = DBIO.seq(sqlu"""create schema apps;""", schemas.create)
val db = Database.forConfig("h2disk")
Await.result(db.run(setup), Duration.Inf)
Notes: the schema name for some dbms is case sensitive, e.g. H2 will automatically convert schema to APPS
I had to use bind variables to make it work as follow : (prefixing the variable with #)
src: https://scala-slick.org/doc/3.3.3/sql.html#splicing-literal-values
val schemaName = "something"
val schemas = Cases(schemaName).schema
val setup = DBIO.seq(
sqlu"""create schema #${schemaName} AUTHORIZATION postgres""",
// create table schemas
schema.createIfNotExists
//add default data
...
// add rights
...
)
all the tables are defined like
class Cases(_tableTag: Tag, schemaName: String) extends profile.api.Table[CasesRow](_tableTag, Some(schemaName), "cases") {
....
}
def Cases(schema: String) = new TableQuery(tag => new Cases(tag,schemaName = schema))
Related
First of all, I'm n.b to spring and jpa. so, Sorry for the rudimentary question.
These days I tried to make server system to location points storing using springboot + jpa + docker + postgresql /kotlin
my idea is server get client call and store locations periodically
so, I using #ElementCollection for store location item with #Embeddable
but, I got exception from springTest code
Hibernate:
insert
into
pos_info_pos_list
(pos_info_id, pos_list_order, accuracy, event_time, geo_lati, geo_long)
values
(?, ?, ?, ?, ?, ?)
2022-11-12 22:07:34.963 WARN 25880 --- [ main] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 0, SQLState: 23505
2022-11-12 22:07:34.963 ERROR 25880 --- [ main] o.h.engine.jdbc.spi.SqlExceptionHelper : ERROR: duplicate key value violates unique constraint "pos_info_pos_list_pkey"
Detail: Key (pos_info_id, pos_list_order)=(1, 0) already exists.
I'll explain the table structure below
PosInfo(one), PosData(many)
oneToMany relation
I want to use ordercolumn for performance and want posList size limitation(MAX_POS_DATA_SIZE = 200)
#Entity
data class PosInfo(
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
var id: Long? = null
) {
#ElementCollection(fetch = FetchType.EAGER, targetClass = PosData::class)
#OrderColumn
val posList: MutableList<PosData> = mutableListOf()
fun addPosData(posData: PosDataDto) {
while (posList.size >= MAX_POS_DATA_SIZE) {
posList.removeFirst()
}
val newData = PosData(posData.geoLati, posData.geoLong, posData.eventTime, posData.accuracy)
posList.add(newData)
}
}
PosData table
#Embeddable
data class PosData(
#Column
val geoLati: String,
#Column
val geoLong: String,
#Column
val eventTime: Long,
#Column
val accuracy: Int,
)
SpringTestCode is
first of all, insert maxSize posData then add more one data again
#Test
fun addPathMax() {
val dummyPosData = PosDataDto("", "", System.currentTimeMillis(), 0)
val dummyPosData2 = PosDataDto("yyyy", "eeeee", System.currentTimeMillis(), 0)
val id = "KSH"
service.tryAddUser(id, "")
val userInfo = service.getUserInfo(id)
assertThat(userInfo).isNotNull
val posIndex = userInfo!!.posIndex
val posInfo = service.getPosInfo(posIndex)
assertThat(posInfo).isNotNull
for (i in 0 until MAX_POS_DATA_SIZE) {
posInfo!!.addPosData(dummyPosData)
}
service.updatePosInfo(posInfo!!)
println("Next Input Check KSH_TEST")
val posInfo2 = service.getPosInfo(posIndex)
posInfo2!!.addPosData(dummyPosData2)
service.updatePosInfo(posInfo2!!)
}
#Transactional
service.updatePosInfo <= it just call to crudRepository save method
but I got duplicate key again and again
Q1. Shouldn't the 'pos_list_order' be 'existing last +1' since the first data of the previous data was erased and the new data was inserted? why '0'?
// Key (pos_info_id, pos_list_order)=(1, 0) already exists.
Q2. Is this structure not good for updating and storing location data periodically?(using ElementCollection, should I use OneToMany?)
=To be honest, I've tried "one To Many" before. By the way, I gave up because I was tired of fixing strange build errors. I came back with "Element Collection," which I thought was easy
Thank you in advance for all the helpful comments
===========================
= I already tried before below
OneToMany with mapped, but it made many error and when I tried insert more value, it was made all delete row and re-install all and + newer again
ElementCollection looks simple, but it was made duplicated exception again and again
I already checked using below
#CollectionTable(
name = "pos_data",
joinColumns = [JoinColumn(name = "pos_info_id")]
)
JpaRepository.save then flush doesn't work
but same result, I don't know why.. really sad
I got a solution
Now this problem was caused by my poor understanding of 'Transactional'
it's fixed with below annotation
#Transactional(propagation = Propagation.REQUIRES_NEW)
#Transactional(propagation = Propagation.REQUIRES_NEW)
#Rollback(false)
#Test
fun addPathMax() {
val dummyPosData = PosDataDto("", "", System.currentTimeMillis(), 0)
val dummyPosData2 = PosDataDto("yyyy", "eeeee", System.currentTimeMillis(), 0)
val id = "KSH"
service.tryAddUser(id, "")
val userInfo = service.getUserInfo(id)
assertThat(userInfo).isNotNull
val posIndex = userInfo!!.posIndex
val posInfo = service.getPosInfo(posIndex)
assertThat(posInfo).isNotNull
for (i in 0 until Constants.MAX_POS_DATA_SIZE) {
posInfo!!.addPosData(dummyPosData)
}
service.updatePosInfo(posInfo!!)
println("Next Input Check KSH_TEST")
val posInfo2 = service.getPosInfo(posIndex)
posInfo2!!.addPosData(dummyPosData2)
service.updatePosInfo(posInfo2!!)
}
I thought service already including 'Transactional' annotation
so it can be made query persist context to database
but it was not
I am trying to bind my data class for a sql query but I am getting a error when I am not using all the parameters from my data class. Is there a way to check in the sql query which parameters needs binding and which ones do not or allow to bind parameters which are not used. The error looks as following:
Request error
java.lang.IllegalArgumentException: Identifier 'deleted_at' is not a valid identifier. Should be of the pattern '\$([\d]+)'.
at io.r2dbc.postgresql.ExtendedQueryPostgresqlStatement.getIndex(ExtendedQueryPostgresqlStatement.java:196)
Caused by: java.lang.IllegalArgumentException: Identifier 'deleted_at' is not a valid identifier. Should be of the pattern '\$([\d]+)'.
at io.r2dbc.postgresql.ExtendedQueryPostgresqlStatement.getIndex(ExtendedQueryPostgresqlStatement.java:196)
And this is the code I use:
Repository:
client
.sql(
"""
INSERT INTO app_user_settings (uuid, allows_to_process_transactions, app_user_id) VALUES (:uuid, :allows_to_process_transactions, :app_user_id)
RETURNING *
""".trimIndent()
)
.bind(AppUserSettingsWriteConverter(), appUserSettings)
.map(AppUserSettingsReadConverter()::convert)
.awaitOne()
Custom bind method:
fun <T> DatabaseClient.GenericExecuteSpec.bind(
convertor: Converter<T, OutboundRow>,
value: T
): DatabaseClient.GenericExecuteSpec {
val outboundRow = convertor.convert(value!!)!!
val t = outboundRow.toMap()
var execution = this
t.forEach { (t, u) ->
execution = execution.bind(t.toString(), u)
}
return execution
}
WriteConverter:
class AppUserSettingsWriteConverter : Converter<AppUserSettings, OutboundRow> {
override fun convert(source: AppUserSettings): OutboundRow {
val outboundRow = OutboundRow()
if (source.isSaved()) {
outboundRow[SqlIdentifier.unquoted("id")] = Parameter.from(source.id)
}
outboundRow[SqlIdentifier.unquoted("uuid")] = Parameter.from(source.uuid)
outboundRow[SqlIdentifier.unquoted("allows_to_process_transactions")] = Parameter.from(source.allowsToProcessTransactions)
outboundRow[SqlIdentifier.unquoted("app_user_id")] = Parameter.from(source.appUserId)
outboundRow[SqlIdentifier.unquoted("deleted_at")] = Parameter.fromOrEmpty(source.deletedAt, ZonedDateTime::class.java)
return outboundRow
}
}
I am using now a check if deleted_at is empty and then not bind it but would prefer if there is another way to do it.
after setting up a schema in an H2 database for unit testing, the unit tests relying on the schema could not find it.
import java.sql.DriverManager
Class.forName("org.h2.Driver")
val setupConn = DriverManager.getConnection("jdbc:h2:mem:test_data_metrics;MODE=PostgreSQL", "sa", "")
val setupStmt = setupConn.createStatement
// setup schema at the beginning of our test
setupStmt.execute("CREATE SCHEMA IF NOT EXISTS my_test_schema AUTHORIZATION sa;")
setupStmt.execute("GRANT ALL ON SCHEMA my_test_schema TO sa;")
setupStmt.execute("CREATE TABLE IF NOT EXISTS my_test_schema.my_test_table (test_id VARCHAR(255), test_column VARCHAR(255));")
setupStmt.executeQuery("select * from my_test_schema.my_test_table")
// res4: java.sql.ResultSet = rs3: org.h2.result.LocalResultImpl#3eb10d62 columns: 2 rows: 0 pos: -1
// this seems to work correctly ^^^
setupStmt.close
setupConn.close
// now run our test using the schema we just set up
val conn = DriverManager.getConnection("jdbc:h2:mem:test_data_metrics;SCHEMA=my_test_schema;MODE=PostgreSQL", "sa", "")
val stmt = conn.createStatement
stmt.executeQuery("select * from my_test_table where test_id = '1'")
// org.h2.jdbc.JdbcSQLSyntaxErrorException: Schema "MY_TEST_SCHEMA" not found; SQL statement:
// SET SCHEMA my_test_schema [90079-200]
// ^^^^ something has gone horribly wrong
You can simply add ;DB_CLOSE_DELAY=-1 to the JDBC URL; no need to have an active connection.
https://h2database.com/html/commands.html#set_db_close_delay
If you use a some recent version of H2, you may also want to add ;DATABASE_TO_LOWER=TRUE for better compatibility with PostgreSQL; the PostgreSQL compatibility mode by itself doesn't imply this setting.
This is embarrassing, but I didn't realize that when I closed the connection to my in memory database, the database would dry up and blow away. This seems obvious in retrospect. The solution is to keep the first connection to the database open throughout testing.
import java.sql.DriverManager
Class.forName("org.h2.Driver")
val setupConn = DriverManager.getConnection("jdbc:h2:mem:test_data_metrics;MODE=PostgreSQL", "sa", "")
val setupStmt = setupConn.createStatement
// setup schema at the beginning of our test
setupStmt.execute("CREATE SCHEMA IF NOT EXISTS my_test_schema AUTHORIZATION sa;")
setupStmt.execute("GRANT ALL ON SCHEMA my_test_schema TO sa;")
setupStmt.execute("CREATE TABLE IF NOT EXISTS my_test_schema.my_test_table (test_id VARCHAR(255), test_column VARCHAR(255));")
setupStmt.executeQuery("select * from my_test_schema.my_test_table")
// res4: java.sql.ResultSet = rs3: org.h2.result.LocalResultImpl#3eb10d62 columns: 2 rows: 0 pos: -1
// DON'T CLOSE THE CONNECTION YET!
//setupStmt.close
//setupConn.close
val conn = DriverManager.getConnection("jdbc:h2:mem:test_data_metrics;SCHEMA=my_test_schema;MODE=PostgreSQL", "sa", "")
val stmt = conn.createStatement
stmt.executeQuery("select * from my_test_table where test_id = '1'")
// res5: java.sql.ResultSet = rs4: org.h2.result.LocalResultImpl#293e66e4 columns: 2 rows: 0 pos: -1
// ^^^^ huzzah!
val df = spark.read.parquet(path)
val IP ="190.176.35.145"
val port = "9000"
val table = "table1"
val user = "defalut"
val password = "default"
I don't know how to write df directly into clickhouse,
and I am not finding any similar answers.
Writing to the clickhouse database is similar to writing any other database through JDBC. Just make sure to import the ClickHouseDriver class to your code.
The username and password are passed into the ckProperties object.
The write command is as follows, you can replace the database name in the string:
import ru.yandex.clickhouse._
val jdbcUrl = "jdbc:clickhouse://190.176.35.145:9000/your_database_name"
val ckProperties = new Properties()
df.write.mode("append").option("driver", "ru.yandex.clickhouse.ClickHouseDriver").jdbc(jdbcUrl, table = "table1", ckProperties)
In my code I call stored procedure like this (and it works perfectly):
{ ? = call schema.package.function(?) }
I need to call it like this because jdbc connection is set to another schema.
But for now I can't test it because H2 database doesn't support packages. So if I change my jdbc url database name to the one I require and delete "schema" from the call everything is ok while testing.
#Test
fun test() {
val session = em.entityManager.unwrap(Session::class.java)
session.doWork {
val st = it.createStatement()
st.execute("create schema if not exists mySchema")
st.execute("create alias mySchema.myPackage.myFunction for " // the error happens here +
"\"${this.javaClass.name}.myFunction\"")
}
val response = dao.myFunction("1")
//test stuff
}
How can I change my test because now it's giving me the syntax error?