Get back newly inserted row in Postgres with sqlx - sql

I use https://github.com/jmoiron/sqlx to make queries to Postgres.
Is it possible to get back the whole row data when inserting a new row?
Here is the query I run:
result, err := Db.Exec("INSERT INTO users (name) VALUES ($1)", user.Name)
Or should I just use my existing user struct as the source of truth about the new entry in the database?

Here is docs about transaction of sqlx:
The result has two possible pieces of data: LastInsertId() or RowsAffected(), the availability of which is driver dependent. In MySQL, for instance, LastInsertId() will be available on inserts with an auto-increment key, but in PostgreSQL, this information can only be retrieved from a normal row cursor by using the RETURNING clause.
So I made a complete demo for how to execute transaction using sqlx, the demo will create an address row in addresses table and then create a user in users table using the new address_id PK as user_address_id FK of the user.
package transaction
import (
"database/sql"
"github.com/jmoiron/sqlx"
"log"
"github.com/pkg/errors"
)
import (
"github.com/icrowley/fake"
)
type User struct {
UserID int `db:"user_id"`
UserNme string `db:"user_nme"`
UserEmail string `db:"user_email"`
UserAddressId sql.NullInt64 `db:"user_address_id"`
}
type ITransactionSamples interface {
CreateUserTransaction() (*User, error)
}
type TransactionSamples struct {
Db *sqlx.DB
}
func NewTransactionSamples(Db *sqlx.DB) ITransactionSamples {
return &TransactionSamples{Db}
}
func (ts *TransactionSamples) CreateUserTransaction() (*User, error) {
tx := ts.Db.MustBegin()
var lastInsertId int
err := tx.QueryRowx(`INSERT INTO addresses (address_id, address_city, address_country, address_state) VALUES ($1, $2, $3, $4) RETURNING address_id`, 3, fake.City(), fake.Country(), fake.State()).Scan(&lastInsertId)
if err != nil {
tx.Rollback()
return nil, errors.Wrap(err, "insert address error")
}
log.Println("lastInsertId: ", lastInsertId)
var user User
err = tx.QueryRowx(`INSERT INTO users (user_id, user_nme, user_email, user_address_id) VALUES ($1, $2, $3, $4) RETURNING *;`, 6, fake.UserName(), fake.EmailAddress(), lastInsertId).StructScan(&user)
if err != nil {
tx.Rollback()
return nil, errors.Wrap(err, "insert user error")
}
err = tx.Commit()
if err != nil {
return nil, errors.Wrap(err, "tx.Commit()")
}
return &user, nil
}
Here is test result:
☁ transaction [master] ⚡ go test -v -count 1 ./...
=== RUN TestCreateUserTransaction
2019/06/27 16:38:50 lastInsertId: 3
--- PASS: TestCreateUserTransaction (0.01s)
transaction_test.go:28: &transaction.User{UserID:6, UserNme:"corrupti", UserEmail:"reiciendis_quam#Thoughtstorm.mil", UserAddressId:sql.NullInt64{Int64:3, Valid:true}}
PASS
ok sqlx-samples/transaction 3.254s

This is a sample code that works with named queries and strong type structures for inserted data and ID.
Query and struct included to cover used syntax.
const query = `INSERT INTO checks (
start, status) VALUES (
:start, :status)
returning id;`
type Row struct {
Status string `db:"status"`
Start time.Time `db:"start"`
}
func InsertCheck(ctx context.Context, row Row, tx *sqlx.Tx) (int64, error) {
return insert(ctx, row, insertCheck, "checks", tx)
}
// insert inserts row into table using query SQL command
// table used only for loging, actual table name defined in query
// should not be used from services directly - implement strong type wrappers
// function expects query with named parameters
func insert(ctx context.Context, row interface{}, query string, table string, tx *sqlx.Tx) (int64, error) {
// convert named query to native parameters format
query, args, err := tx.BindNamed(query, row)
if err != nil {
return 0, fmt.Errorf("cannot bind parameters for insert into %q: %w", table, err)
}
var id struct {
Val int64 `db:"id"`
}
err = sqlx.GetContext(ctx, tx, &id, query, args...)
if err != nil {
return 0, fmt.Errorf("cannot insert into %q: %w", table, err)
}
return id.Val, nil
}

PostgreSQL supports RETURNING syntax for INSERT statements.
Example:
INSERT INTO users(...) VALUES(...) RETURNING id, name, foo, bar
Documentaion: https://www.postgresql.org/docs/9.6/static/sql-insert.html
The optional RETURNING clause causes INSERT to compute and return value(s) based on each row actually inserted (or updated, if an ON CONFLICT DO UPDATE clause was used). This is primarily useful for obtaining values that were supplied by defaults, such as a serial sequence number. However, any expression using the table's columns is allowed. The syntax of the RETURNING list is identical to that of the output list of SELECT. Only rows that were successfully inserted or updated will be returned.

Related

Inserting empty string or null into postgres as null using jackc/pgx

I'm using an external json API that's inconsistent in the way it handles missing values. Sometimes json values show up as empty strings and other times as null. For example...
Case1: datedec and curr are both empty strings.
{
"symbol": "XYZ",
"dateex": "2020-09-01",
"datedec": "",
"amount": "1.25",
"curr": "",
"freq": "annual"
}
Case2: datedec is null. curr is populated.
{
"symbol": "XYZ",
"dateex": "2020-09-01",
"datedec": null,
"amount": "1.25",
"curr": "USD",
"freq": "annual"
}
Here is the struct I'm using to represent a dividend:
type Dividend struct {
symbol string `json:"symbol"`
dateex string `json:"dateex"`
datedec string `json:"datedec"`
amount string `json:"amount"`
curr string `json:"curr"`
freq string `json:"freq"`
}
The problem I'm having is how to insert either an empty string or null, into the database as NULL. I know I could use an omitempty json tag, but then how would I write a function to handle values I don't know will be missing? For example, Here is my current function to insert a dividend into postgresql using the jackc/pgx package:
func InsertDividend(d Dividend) error {
sql := `INSERT INTO dividends
(symbol, dateex, datedec, amount, curr, freq)
VALUES ($1, $2, $3, $4, $5, $6)`
conn, err := pgx.Connect(ctx, "DATABASE_URL")
// handle error
defer conn.Close(ctx)
tx, err := conn.Begin()
// handle error
defer tx.Rollback(ctx)
_, err = tx.Exec(ctx, sql, d.symbol, d.dateex, d.datedec, d.amount, d.curr, d.freq)
// handle error
}
err = tx.Commit(ctx)
// handle error
return nil
}
If a value (e.g. datedec or curr) is missing, then this function will error. From this post Golang Insert NULL into sql instead of empty string I saw how to solve Case1. But is there a more general way to handle both cases (null or empty string)?
I've been looking through the database/sql & jackc/pgx documentation but I have yet to find anything. I think the sql.NullString has potential but I'm not sure how I should be doing it.
Any suggestions will be appreciated. Thanks!
There are a number of ways you can represent NULL when writing to the database. sql.NullString is an option as is using a pointer (nil = null); the choice really comes down to what you find easer to understand. Rus Cox commented:
There's no effective difference. We thought people might want to use NullString because it is so common and perhaps expresses the intent more clearly than *string. But either will work.
I suspect that using pointers will be the simplest approach in your situation. For example the following will probably meet your needs:
type Dividend struct {
Symbol string `json:"symbol"`
Dateex string `json:"dateex"`
Datedec *string `json:"datedec"`
Amount string `json:"amount"`
Curr *string `json:"curr"`
Freq string `json:"freq"`
}
func unmarshal(in[]byte, div *Dividend) {
err := json.Unmarshal(in, div)
if err != nil {
panic(err)
}
// The below is not necessary unless if you want to ensure that blanks
// and missing values are both written to the database as NULL...
if div.Datedec != nil && len(*div.Datedec) == 0 {
div.Datedec = nil
}
if div.Curr != nil && len(*div.Curr) == 0 {
div.Curr = nil
}
}
Try it in the playground.
You can use the Dividend struct in the same way as you are now when writing to the database; the SQL driver will write the nil as a NULL.
you can also use pgtypes and get the SQL Driver value from any pgtype using the Value() func:
https://github.com/jackc/pgtype
https://github.com/jackc/pgtype/blob/master/text.go
type Dividend struct {
symbol pgtype.Text `json:"symbol"`
dateex pgtype.Text `json:"dateex"`
datedec pgtype.Text `json:"datedec"`
amount pgtype.Text `json:"amount"`
curr pgtype.Text `json:"curr"`
freq pgtype.Text `json:"freq"`
}
func InsertDividend(d Dividend) error {
// --> get SQL values from d
var err error
symbol, err := d.symbol.Value() // see https://github.com/jackc/pgtype/blob/4db2a33562c6d2d38da9dbe9b8e29f2d4487cc5b/text.go#L174
if err != nil {
return err
}
dateex, err := d.dateex.Value()
if err != nil {
return err
}
// ...
sql := `INSERT INTO dividends
(symbol, dateex, datedec, amount, curr, freq)
VALUES ($1, $2, $3, $4, $5, $6)`
conn, err := pgx.Connect(ctx, "DATABASE_URL")
defer conn.Close(ctx)
tx, err := conn.Begin()
defer tx.Rollback(ctx)
// --> exec your query using the SQL values your get earlier
_, err = tx.Exec(ctx, sql, symbol, dateex, datedec, amount, curr, freq)
// handle error
}
err = tx.Commit(ctx)
// handle error
return nil
}

golang null.String decoding not working correctly

Trying to fix this problem i'm having with my api im building.
db:
DROP TABLE IF EXISTS contacts CASCADE;
CREATE TABLE IF NOT EXISTS contacts (
uuid UUID UNIQUE PRIMARY KEY,
first_name varchar(150),
);
DROP TABLE IF EXISTS workorders CASCADE;
CREATE TABLE IF NOT EXISTS workorders (
uuid UUID UNIQUE PRIMARY KEY,
work_date timestamp WITH time zone,
requested_by UUID REFERENCES contacts (uuid) ON UPDATE CASCADE ON DELETE CASCADE,
);
struct:
https://gopkg.in/guregu/null.v3
type WorkorderNew struct {
UUID string `json:"uuid"`
WorkDate null.Time `json:"work_date"`
RequestedBy null.String `json:"requested_by"`
}
api code:
workorder := &models.WorkorderNew{}
if err := json.NewDecoder(r.Body).Decode(workorder); err != nil {
log.Println("decoding fail", err)
}
// fmt.Println(NewUUID())
u2, err := uuid.NewV4()
if err != nil {
log.Fatalf("failed to generate UUID: %v", err)
}
q := `
INSERT
INTO workorders
(uuid,
work_date,
requested_by
)
VALUES
($1,$2,$3)
RETURNING uuid;`
statement, err := global.DB.Prepare(q)
global.CheckDbErr(err)
fmt.Println("requested by", workorder.RequestedBy)
lastInsertID := ""
err = statement.QueryRow(
u2,
workorder.WorkDate,
workorder.RequestedBy,
).Scan(&lastInsertID)
global.CheckDbErr(err)
json.NewEncoder(w).Encode(lastInsertID)
When I send an API request with null as the value it works as expected
but when i try to send a "" as the value for the null.String or the null.Time it fails
works:
{
"work_date":"2016-12-16T19:00:00Z",
"requested_by":null
}
not working:
{
"work_date":"2016-12-16T19:00:00Z",
"requested_by":""
}
Basically when i call the QueryRow and save to database the workorder.RequestedBy value should be a null and not the "" value im getting
thanks
If you want to treat empty strings as nulls you have at least two options.
"Extend" null.String:
type MyNullString struct {
null.String
}
func (ns *MyNullString) UnmarshalJSON(data []byte) error {
if string(data) == `""` {
ns.Valid = false
return nil
}
ns.String.UnmarshalJSON(data)
}
Or use NULLIF in the query:
INSERT INTO workorders (
uuid
, work_date
, requested_by
) VALUES (
$1
, $2
, NULLIF($3, '')
)
RETURNING uuid
Update:
To extend the null.Time you have to understand that the type of null.Time.Time is a struct. The builtin len function works on slices, arrays, pointers to arrays, maps, channels, and strings. Not structs. So in this case you can check the data argument, which is a byte slice, by converting it to a string and comparing it to a string that contains an empty string, i.e. it has two double quotes and nothing else.
type MyNullTime struct {
null.Time
}
func (ns *MyNullTime) UnmarshalJSON(data []byte) error {
if string(data) == `""` {
ns.Valid = false
return nil
}
return ns.Time.UnmarshalJSON(data)
}

Go SQL query inconsistency

I am experiencing some really weird inconsistencies when executing queries, and was wondering if anyone knew why.
Imagine I have a struct defined as follows:
type Result struct {
Afield string `db:"A"`
Bfield interface{} `db:"B"`
Cfield string `db:"C"`
Dfield string `db:"D"`
}
And a MySQL Table with the following cols:
A : VARCHAR(50)
B : INT
C : VARCHAR(50)
D : VARCHAR(50)
The query I would like to execute:
SELECT A, B, C, D FROM table WHERE A="a"
first way it can be executed:
db.Get(&result, `SELECT A, B, C, D FROM table WHERE A="a"`)
second way it can be executed:
db.Get(&result, `SELECT A, B, C, D FROM table WHERE A=?`, "a")
The inconsistencies I am experiencing are as follows: When executing the query the first way, the type of Bfield is int. However, when executing the query the second time, it is []uint8.
This outcome is occurring for example when B is 1.
Why is the type of Bfield different depending on how the query is executed?
connection declaration:
// Connection is an interface for making queries.
type Connection interface {
Exec(query string, args ...interface{}) (sql.Result, error)
Get(dest interface{}, query string, args ...interface{}) error
Select(dest interface{}, query string, args ...interface{}) error
}
EDIT
This is also happening using the Go database/sql package + driver. The queries below are assigning Bfield to []uint8 and int64 respectively.
db is of type *sql.DB
query 1:
db.QueryRow(SELECT A, B, C, D FROM table WHERE A="a").Scan(&result.Afield, &result.Bfield, &result.Cfield, &result.Dfield)
-- > type of Bfield is []uint8
query 2:
db.QueryRow(SELECT A, B, C, D FROM table WHERE A=?, "a").Scan(&result.Afield, &result.Bfield, &result.Cfield, &result.Dfield)
--> type of Bfield is int64
EDIT
Something else to note, when chaining multiple WHERE clauses, as long as at least 1 is populated using ?, the query will return int. Otherwise if they are all populated in the string, it will return []uint8
Short answer: because the MySQL driver uses a different protocol for queries with and without parameters. Use a prepared statement to get consistent results.
The following explanation refers to the standard MySQL driver github.com/go-sql-driver/mysql, version 1.4
In the first case, the driver sends the query directly to MySQL, and interprets the result as a *textRows struct. This struct (almost) always decodes results into a byte slice, and leaves the conversion to a better type to the Go sql package. This works fine if the destination is an int, string, sql.Scanner etc, but not for interface{}.
In the second case, the driver detects that there are arguments and returns driver.ErrSkip. This causes the Go SQL package to use a PreparedStatement. And in that case, the MySQL driver uses a *binaryRows struct to interpret the results. This struct uses the declared column type (INT in this case) to decode the value, in this case to decode the value into an int64.
Fun fact: if you provide the interpolateParams=true parameter to the database DSN (e.g. "root:testing#/mysql?interpolateParams=true"), the MySQL driver will prepare the query on the client side, and not use a PreparedStatement. At this point both types of query behave the same.
A small proof of concept:
package main
import (
"database/sql"
"log"
_ "github.com/go-sql-driver/mysql"
)
type Result struct {
Afield string
Bfield interface{}
}
func main() {
db, err := sql.Open("mysql", "root:testing#/mysql")
if err != nil {
log.Fatal(err)
}
defer db.Close()
if _, err = db.Exec(`CREATE TABLE IF NOT EXISTS mytable(A VARCHAR(50), B INT);`); err != nil {
log.Fatal(err)
}
if _, err = db.Exec(`DELETE FROM mytable`); err != nil {
log.Fatal(err)
}
if _, err = db.Exec(`INSERT INTO mytable(A, B) VALUES ('a', 3)`); err != nil {
log.Fatal(err)
}
var (
usingLiteral Result
usingParam Result
usingLiteralPrepared Result
)
row := db.QueryRow(`SELECT B FROM mytable WHERE A='a'`)
if err := row.Scan(&usingLiteral.Bfield); err != nil {
log.Fatal(err)
}
row = db.QueryRow(`SELECT B FROM mytable WHERE A=?`, "a")
if err := row.Scan(&usingParam.Bfield); err != nil {
log.Fatal(err)
}
stmt, err := db.Prepare(`SELECT B FROM mytable WHERE A='a'`)
if err != nil {
log.Fatal(err)
}
defer stmt.Close()
row = stmt.QueryRow()
if err := row.Scan(&usingLiteralPrepared.Bfield); err != nil {
log.Fatal(err)
}
log.Printf("Type when using literal: %T", usingLiteral.Bfield) // []uint8
log.Printf("Type when using param: %T", usingParam.Bfield) // int64
log.Printf("Type when using prepared: %T", usingLiteralPrepared.Bfield) // int64
}
Your first SQL string, in MySql is ambigous and can have too meaning as explained on StackOverflow in following address
When to use single quotes, double quotes, and back ticks in MySQL
Depending on SQL-MODE, your SQL command can be interpreted as
SELECT A, B, C, D FROM table WHERE A='a'
that is what I think you are expecting.
or as
SELECT A, B, C, D FROM table WHERE A=`a`
To avoid this ambiguity, can you make a new FIRST test in replacing double quotes by single quote ?
If the same behavior continue to be there, my answer is not a good response.
If BOTH SQL select return same value, your question has been solved.
Using ` character, you pass a variable name and not a string value !

golang sqlite can't define Query variable

It seems golang's sqlite package doesn't like my db.Query statement, though it's exactly like the one found in the example on github.
db, err := sql.Open("sqlite3", "./database.db")
if err != nil {
log.Fatal(err)
}
defer db.Close()
rows, err = db.Query("select id, name from job")
if err != nil {
log.Fatal(err)
}
defer rows.Close()
fmt.Println("Jobs:")
for rows.Next() {
var name string
var id int
fmt.Printf("%v %v\n", id, name)
}
This is the error I'm getting:
./test.go:7: undefined: rows
./test.go:7: cannot assign to rows
./test.go:11: undefined: rows
./test.go:14: undefined: rows
Edit: I've tried using grave accent and single quote strings for db.Query() as well, to no avail.
You cannot assign values to to undeclared variables.
rows, err = db.Query("select id, name from job")
Should be :
rows, err := db.Query("select id, name from job")
Theoretically this should solve the problem, but I haven't tried.
You should also add :
rows.Scan(&id, &name)
Before the printf function so as to actually assign the row's value to the id & name variables otherwise will print an empty string & 0.

Error with scan of nil float value using Go and database/sql

I'm writing a program that needs to determine opening values for a table prior to doing some Inserts and Updates for that table. The table in question (PostgreSql in this case) could have zero rows initially. When I select the opening values, if there are zero rows, the total for the value of balances is being returned as nil. This causes the scan to fail with message :
Error on scan of test01 opening Row Count. Error = sql: Scan error on column
index 1: converting string "<nil>" to a float64: strconv.ParseFloat:
parsing "<nil>": invalid syntax
While I can "solve" the problem by doing two selects, one to select the COUNT(*) and the other to SUM() the balances if the row-count exceeds zero, it does not seem an elegant solution, and may not always solve the problem, and the two values selected (number of rows and total of balances) are not at the same point in time.
Is there a way to solve this problem doing one select of the table?
A small test program illustrating the problem is below. When there are rows in the table being selected, the program works fine. However if there are zero rows, the above error results.
Example Test Program:
package main
import (
"database/sql"
"fmt"
_ "github.com/lib/pq"
)
var db *sql.DB
func main() {
var err error
db, err = sql.Open("postgres",
"user=test dbname=testdb password=test sslmode=disable")
if err != nil {
fmt.Sprintf("Failed to open Db Connection. Error = %s\n", err)
return
}
defer fCloseDb()
var row *sql.Row
var sSql string = "SELECT COUNT(*), SUM(dbalance) FROM test01"
if row = db.QueryRow(sSql); row == nil {
println("No row returned selecting opening count(*) from test01")
return
}
var iRowCount int64 = 0
var fBalTot float64 = 0.00
if err = row.Scan(&iRowCount, &fBalTot); err != nil {
fmt.Printf("Error on scan of test01 opening Row Count. Error = %s\n", err)
return
}
fmt.Printf("Row Count = %d, Balance total value = %.2f\n", iRowCount, fBalTot)
}
func fCloseDb() {
if db != nil {
db.Close()
println("Db Closed")
}
}
The structure of the table is as follows :
sSql = "CREATE TABLE IF NOT EXISTS test01 " +
"(ikey SERIAL Primary Key, " +
"sname varchar(22) not null, " +
"dbalance decimal(12,2) not null)"
Many thanks, that worked :
"SELECT COUNT(*), coalesce(SUM(dbalance), 0.00) FROM test01"
I believe that coalesce returns the first non-null value.