How to execute an IN lookup in SQL using Golang? - sql

What does Go want for the second param in this SQL query.
I am trying to use the IN lookup in postgres.
stmt, err := db.Prepare("SELECT * FROM awesome_table WHERE id= $1 AND other_field IN $2")
rows, err := stmt.Query(10, ???)
What I really want:
SELECT * FROM awesome_table WHERE id=10 AND other_field IN (this, that);

It looks like you may be using the pq driver. pq recently added Postgres-specific Array support via pq.Array (see pull request 466). You can get what you want via:
stmt, err := db.Prepare("SELECT * FROM awesome_table WHERE id= $1 AND other_field = ANY($2)")
rows, err := stmt.Query(10, pq.Array([]string{'this','that'})
I think this generates the SQL:
SELECT * FROM awesome_table WHERE id=10 AND other_field = ANY('{"this", "that"}');
Note this utilizes prepared statements, so the inputs should be sanitized.

Query just takes varargs to replace the params in your sql
so, in your example, you would just do
rows, err := stmt.Query(10)
say, this and that of your second example were dynamic, then you'd do
stmt, err := db.Prepare("SELECT * FROM awesome_table WHERE id=$1 AND other_field IN ($2, $3)")
rows, err := stmt.Query(10,"this","that")
If you have variable args for the "IN" part, you can do (play)
package main
import "fmt"
import "strings"
func main() {
stuff := []interface{}{"this", "that", "otherthing"}
sql := "select * from foo where id=? and name in (?" + strings.Repeat(",?", len(stuff)-1) + ")"
fmt.Println("SQL:", sql)
args := []interface{}{10}
args = append(args, stuff...)
fakeExec(args...)
// This also works, but I think it's harder for folks to read
//fakeExec(append([]interface{}{10},stuff...)...)
}
func fakeExec(args ...interface{}) {
fmt.Println("Got:", args)
}

Incase anyone like me was trying to use an array with a query, here is an easy solution.
get https://github.com/jmoiron/sqlx
ids := []int{1, 2, 3}
q,args,err := sqlx.In("SELECT id,username FROM users WHERE id IN(?);", ids) //creates the query string and arguments
//you should check for errors of course
q = sqlx.Rebind(sqlx.DOLLAR,q) //only if postgres
rows, err := db.Query(q,args...) //use normal POSTGRES/ANY SQL driver important to include the '...' after the Slice(array)

With PostgreSQL, at least, you have the option of passing the entire array as a string, using a single placeholder:
db.Query("select 1 = any($1::integer[])", "{1,2,3}")
That way, you can use a single query string, and all the string concatenation is confined to the parameter. And if the parameter is malformed, you don't get an SQL injection; you just get something like: ERROR: invalid input syntax for integer: "xyz"
https://groups.google.com/d/msg/golang-nuts/vHbg09g7s2I/RKU7XsO25SIJ

if you use sqlx, you can follow this way:
https://github.com/jmoiron/sqlx/issues/346
arr := []string{"this", "that"}
query, args, err := sqlx.In("SELECT * FROM awesome_table WHERE id=10 AND other_field IN (?)", arr)
query = db.Rebind(query) // sqlx.In returns queries with the `?` bindvar, rebind it here for matching the database in used (e.g. postgre, oracle etc, can skip it if you use mysql)
rows, err := db.Query(query, args...)

var awesome AwesomeStruct
var awesomes []*AwesomeStruct
ids := []int{1,2,3,4}
q, args, err := sqlx.In(`
SELECT * FROM awesome_table WHERE id=(?) AND other_field IN (?)`, 10, ids)
// use .Select for multiple return
err = db.Select(&awesomes, db.SQL.Rebind(q), args...)
// use .Get for single return
err = db.Get(&awesome, db.SQL.Rebind(q), args...)

//I tried a different way. A simpler and easier way, maybe not too efficient.
stringedIDs := fmt.Sprintf("%v", ids)
stringedIDs = stringedIDs[1 : len(stringedIDs)-1]
stringedIDs = strings.ReplaceAll(stringedIDs, " ", ",")
query := "SELECT * FROM users WHERE id IN (" + stringedIDs + ")"
//Then follow your standard database/sql Query
rows, err := db.Query(query)
//error checking
if err != nil {
// Handle errors
} else {
// Process rows
}

Rather pedestrian and only to be used if server generated. Where UserIDs is a slice (list) of strings:
sqlc := `select count(*) from test.Logins where UserID
in ("` + strings.Join(UserIDs,`","`) + `")`
errc := db.QueryRow(sqlc).Scan(&Logins)

You can also use this direct conversion.
awesome_id_list := []int{3,5,8}
var str string
for _, value := range awesome_id_list {
str += strconv.Itoa(value) + ","
}
query := "SELECT * FROM awesome_table WHERE id IN (" + str[:len(str)-1] + ")"
WARNING
This is method is vulnerable to SQL Injection. Use this method only if awesome_id_list is server generated.

Related

Make a query to both "id" or "slug" with a single variable

I have a table "articles" where there're "id" and "slug" among other things. On an html page I have a list of links to articles. A link can contain either "id" or "slug" in it.
But if in a URL there's only a number, it doesn't still mean that it's an id -- therefore, casting to int to determine whether or not it's slug or id, won't work.
/articles/my_article
/articles/35
/articles/666 --> still may be slug
I have this sql query:
import (
"github.com/jackc/pgx/v4"
//.........
)
// [..........]
vars := mux.Vars(req)
q1 := `
SELECT
ar.id,
[.........]
FROM
articles AS ar
WHERE ar.slug = $1 OR ar.id = $1`
ar := Article{}
row := db.QueryRow(context.Background(), q1, vars["id_or_slug"])
switch err := row.Scan(&ar.Id, /*[.......]*/); err {
case pgx.ErrNoRows:
wrt.WriteHeader(http.StatusNotFound)
wrt.Write([]byte("article not found"))
case nil:
// good, article found
I get:
ERROR: operator does not exist: bigint = text (SQLSTATE 42883)
You can "attempt" to convert the value to an integer and if the conversion fails just ignore the error and provide an id value known to not be present in the db.
Doing the conversion with Go:
slug := mux.Vars(req)["id_or_slug"]
// option 1:
id, err := strconv.ParseInt(slug, 10, 64)
if err != nil {
id = -1 // provide a value that you're certain will not be present in the db
}
// option 2:
// if id 0 is good enough, you can skip error checking
// and use the following instead of the above.
id, _ := strconv.ParseInt(slug, 10, 64)
query := `SELECT ... FROM articles AS a
WHERE a.slug = $1
OR a.id = $2`
row := db.QueryRow(query, slug, id)
Doing the conversion with postgres: (the following postgres snippet was taken from here.
)
-- first create a postgres function that will do the conversion / cast
create or replace function cast_to_int(text, integer) returns integer as $$
begin
return cast($1 as integer);
exception
when invalid_text_representation then
return $2;
end;
$$ language plpgsql immutable;
... and then utilizing that in go:
slug := mux.Vars(req)["id_or_slug"]
query := `SELECT ... FROM articles AS a
WHERE a.slug = $1
OR a.id = cast_to_int($1::text, -1)` // use the postgres function in the go query string
row := db.QueryRow(query, slug)

sql: expected 3 destination arguments in Scan, not 1 in Golang

I am writing a generic code to query data from any RDS table. I have gone through many StackOverflow answers but none of it worked for me. I have gone through below links:-
panic: sql: expected 1 destination arguments in Scan, not <number> golang, pq, sql
How to query any table of RDS using Golang SDK
My first code is
package main
import (
"fmt"
)
type BA_Client struct {
ClientId int `json:ClientId;"`
CompanyName string `json:CompanyName;"`
CreateDate string `json:CreateDate;"`
}
func main() {
conn, _ := getConnection() // Det database connection
query := `select * from IMBookingApp.dbo.BA_Client
ORDER BY
ClientId ASC
OFFSET 56 ROWS
FETCH NEXT 10 ROWS ONLY ;`
var p []byte
err := conn.QueryRow(query).Scan(&p)
if err != nil {
fmt.Println("Error1", err)
}
fmt.Println("Data:", p)
var m BA_Client
err = json.Unmarshal(p, &m)
if err != nil {
fmt.Println("Error2", err)
}
}
My second code is
conn, _ := getConnection()
query := `select * from IMBookingApp.dbo.BA_Client__c
ORDER BY
ClientId__c ASC
OFFSET 56 ROWS
FETCH NEXT 10 ROWS ONLY ;`
rows, err := conn.Query(query)
if err != nil {
fmt.Println("Error:")
log.Fatal(err)
}
println("rows", rows)
defer rows.Close()
columns, err := rows.Columns()
fmt.Println("columns", columns)
if err != nil {
panic(err)
}
for rows.Next() {
receiver := make([]*string, len(columns))
err := rows.Scan(&receiver )
if err != nil {
fmt.Println("Error reading rows: " + err.Error())
}
fmt.Println("Data:", p)
fmt.Println("receiver", receiver)
}
With both the codes, I am getting the same error as below
sql: expected 3 destination arguments in Scan, not 1
Can it be because I am using SQL Server and not MySQL? Appreciate if anyone can help me to find the issue.
When you want to use a slice of inputs for your row scan, use the variadic 3-dots notation ... to convert the slice into individual parameters, like so:
err := rows.Scan(receiver...)
Your (three in this case) columns, using the variadic arguments will effectively expand like so:
// len(receiver) == 3
err := rows.Scan(receiver[0], receiver[1], receiver[2])
EDIT:
SQL Scan method parameter values must be of type interface{}. So we need an intermediate slice. For example:
is := make([]interface{}, len(receiver))
for i := range is {
is[i] = receiver[i]
// each is[i] will be of type interface{} - compatible with Scan()
// using the underlying concrete `*string` values from `receiver`
}
// ...
err := rows.Scan(is...)
// `receiver` will contain the actual `*string` typed items
try this (no ampersand in front of receiver) for second example:
err := rows.Scan(receiver)
receiver and *string are already a reference types. No need to add another reference to it.
specific field table
query := `select * from IMBookingApp.dbo.BA_Client__c
ORDER BY
ClientId__c ASC
OFFSET 56 ROWS
FETCH NEXT 10 ROWS ONLY ;`
change to
query := `select ClientId, CompanyName, CreateDate from IMBookingApp.dbo.BA_Client__c
ORDER BY
ClientId__c ASC
OFFSET 56 ROWS
FETCH NEXT 10 ROWS ONLY ;`

"WHERE ... IN (...)" using pgx [duplicate]

I am trying to execute the following query against the PostgreSQL database in Go using pq driver:
SELECT COUNT(id)
FROM tags
WHERE id IN (1, 2, 3)
where 1, 2, 3 is passed at a slice tags := []string{"1", "2", "3"}.
I have tried many different things like:
s := "(" + strings.Join(tags, ",") + ")"
if err := Db.QueryRow(`
SELECT COUNT(id)
FROM tags
WHERE id IN $1`, s,
).Scan(&num); err != nil {
log.Println(err)
}
which results in pq: syntax error at or near "$1". I also tried
if err := Db.QueryRow(`
SELECT COUNT(id)
FROM tags
WHERE id IN ($1)`, strings.Join(stringTagIds, ","),
).Scan(&num); err != nil {
log.Println(err)
}
which also fails with pq: invalid input syntax for integer: "1,2,3"
I also tried passing a slice of integers/strings directly and got sql: converting Exec argument #0's type: unsupported type []string, a slice.
So how can I execute this query in Go?
Pre-building the SQL query (preventing SQL injection)
If you're generating an SQL string with a param placeholder for each of the values, it's easier to just generate the final SQL right away.
Note that since values are strings, there's place for SQL injection attack, so we first test if all the string values are indeed numbers, and we only proceed if so:
tags := []string{"1", "2", "3"}
buf := bytes.NewBufferString("SELECT COUNT(id) FROM tags WHERE id IN(")
for i, v := range tags {
if i > 0 {
buf.WriteString(",")
}
if _, err := strconv.Atoi(v); err != nil {
panic("Not number!")
}
buf.WriteString(v)
}
buf.WriteString(")")
Executing it:
num := 0
if err := Db.QueryRow(buf.String()).Scan(&num); err != nil {
log.Println(err)
}
Using ANY
You can also use Postgresql's ANY, whose syntax is as follows:
expression operator ANY (array expression)
Using that, our query may look like this:
SELECT COUNT(id) FROM tags WHERE id = ANY('{1,2,3}'::int[])
In this case you can declare the text form of the array as a parameter:
SELECT COUNT(id) FROM tags WHERE id = ANY($1::int[])
Which can simply be built like this:
tags := []string{"1", "2", "3"}
param := "{" + strings.Join(tags, ",") + "}"
Note that no check is required in this case as the array expression will not allow SQL injection (but rather will result in a query execution error).
So the full code:
tags := []string{"1", "2", "3"}
q := "SELECT COUNT(id) FROM tags WHERE id = ANY($1::int[])"
param := "{" + strings.Join(tags, ",") + "}"
num := 0
if err := Db.QueryRow(q, param).Scan(&num); err != nil {
log.Println(err)
}
This is not really a Golang issue, you are using a string to compare to integer (id) in your SQL request. That means, SQL receive:
SELECT COUNT(id)
FROM tags
WHERE id IN ("1, 2, 3")
instead of what you want to give it. You just need to convert your tags into integer and passe it to the query.
EDIT:
Since you are trying to pass multiple value to the query, then you should tell it:
params := make([]string, 0, len(tags))
for i := range tags {
params = append(params, fmt.Sprintf("$%d", i+1))
}
query := fmt.Sprintf("SELECT COUNT(id) FROM tags WHERE id IN (%s)", strings.Join(params, ", "))
This will end the query with a "($1, $2, $3...", then convert your tags as int:
values := make([]int, 0, len(tags))
for _, s := range tags {
val, err := strconv.Atoi(s)
if err != nil {
// Do whatever is required with the error
fmt.Println("Err : ", err)
} else {
values = append(values, val)
}
}
And finally, you can use it in the query:
Db.QueryRow(query, values...)
This should do it.
Extending #icza solution, you can use pq.Array instead of building the params yourself.
So using his example, the code can look like this:
tags := []string{"1", "2", "3"}
q := "SELECT COUNT(id) FROM tags WHERE id = ANY($1::int[])"
num := 0
if err := Db.QueryRow(q, pq.Array(tags)).Scan(&num); err != nil {
log.Println(err)
}

Cross-database prepared statement binding (like and where in) in Golang

After reading many tutorials, I found that there are many ways to bind arguments on prepared statement in Go, some of them
SELECT * FROM bla WHERE x = ?col1 AND y = ?col2
SELECT * FROM bla WHERE x = ? AND y = ?
SELECT * FROM bla WHERE x = :col1 AND y = :col2
SELECT * FROM bla WHERE x = $1 AND y = $2
First question, what is the cross-database way to bind arguments? (that works on any database)
Second question, none of the tutorial I've read mention about LIKE statement, how to bind arguments for LIKE-statement correctly?
SELECT * FROM bla WHERE x LIKE /*WHAT?*/
Third question, also none of them give an example for IN statement, how to bind arguments for IN statement correctly?
`SELECT * FROM bla WHERE x IN ( /*WHAT?*/ )
What is the cross-database way to bind arguments?
With database/sql, there is none. Each database has its own way to represent parameter placeholders. The Go database/sql package does not provide any normalization facility for the prepared statements. Prepared statement texts are just passed to the underlying driver, and the driver typically just sends them unmodified to the database server (or library for embedded databases).
How to bind arguments for LIKE-statement correctly?
You can use parameter placeholders after a like statement and bind it as a string. For instance, you could write a prepared statement as:
SELECT a from bla WHERE b LIKE ?
Here is an example (error management handling omitted).
package main
import (
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
)
// > select * from bla ;
// +------+------+
// | a | b |
// +------+------+
// | toto | titi |
// | bobo | bibi |
// +------+------+
func main() {
// Open connection
db, err := sql.Open("mysql", "root:XXXXXXX#/test")
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
defer db.Close()
// Prepare statement for reading data
stmtOut, err := db.Prepare("SELECT a FROM bla WHERE b LIKE ?")
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
defer stmtOut.Close()
var a string
b := "bi%" // LIKE 'bi%'
err = stmtOut.QueryRow(b).Scan(&a)
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
fmt.Printf("a = %s\n", a)
}
Note that the % character is part of the bound string, not of the query text.
How to bind arguments for IN statement correctly?
None of the databases I know allows binding a list of parameters directly with a IN clause. This is not a limitation of database/sql or the drivers, but this is simply not supported by most database servers.
You have several ways to work the problem around:
you can build a query with a fixed number of placeholders in the IN clause. Only bind the parameters you are provided with, and complete the other placeholders by the NULL value. If you have more values than the fixed number you have chosen, just execute the query several times. This is not extremely elegant, but it can be effective.
you can build multiple queries with various number of placeholders. One query for IN ( ? ), a second query for IN (?, ?), a third for IN (?,?,?), etc ... Keep those prepared queries in a statement cache, and choose the right one at runtime depending on the number of input parameters. Note that it takes memory, and generally the maximum number of prepared statements is limited, so it cannot be used when the number of parameters is high.
if the number of input parameters is high, insert them in a temporary table, and replace the query with the IN clause by a join with the temporary table. It is effective if you manage to perform the insertion in the temporary table in one roundtrip. With Go and database/sql, it is not convenient because there is no way to batch queries.
Each of these solutions has drawbacks. None of them is perfect.
I'm a newbie to Go but just to answer the first part:
First question, what is the cross-database way to bind arguments? (that works on any database)
If you use sqlx, which is a superset of the built-in sql package, then you should be able to use sqlx.DB.Rebind to achieve that.
I had this same question, and after reading the answers started to look for other solution on how to bind arguments for the IN statement.
Here is an example of what I did, not the most elegant solution, but works for me.
What I did was to create a select query with the parameters statically set on the query, and not using the bind feature at all.
It could be a good idea to sanitize the string that comes from the Marshal command, to be sure and safe, but I don't need it now.
package main
import (
"database/sql"
"encoding/json"
"fmt"
"log"
_ "github.com/go-sql-driver/mysql"
)
type Result struct {
Identifier string
Enabled bool
}
func main() {
// Open connection
db, err := sql.Open("mysql", "username:password#tcp(server-host)/my-database")
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
defer db.Close()
// this is an example of a variable list of IDs
idList := []string{"ID1", "ID2", "ID3", "ID4", "ID5", "IDx"}
// convert the list to a JSON string
formatted, _ := json.Marshal(idList)
// a JSON array starts and ends with '[]' respectivelly, so we replace them with '()'
formatted[0] = '('
formatted[len(formatted)-1] = ')'
// create a static select query
query := fmt.Sprintf("SELECT identifier, is_enabled FROM some_table WHERE identifier in %s", string(formatted))
// prepare que query
rows, err := db.Query(query)
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
defer rows.Close()
var result []Result
// fetch rows
for rows.Next() {
var r0 Result
if err := rows.Scan(&r0.Identifier, &r0.Enabled); err != nil {
log.Fatal(err)
}
// append the row to the result
result = append(result, r0)
}
if err := rows.Err(); err != nil {
log.Fatal(err)
}
fmt.Printf("result = %v\n", result)
}

golang sqlite can't define Query variable

It seems golang's sqlite package doesn't like my db.Query statement, though it's exactly like the one found in the example on github.
db, err := sql.Open("sqlite3", "./database.db")
if err != nil {
log.Fatal(err)
}
defer db.Close()
rows, err = db.Query("select id, name from job")
if err != nil {
log.Fatal(err)
}
defer rows.Close()
fmt.Println("Jobs:")
for rows.Next() {
var name string
var id int
fmt.Printf("%v %v\n", id, name)
}
This is the error I'm getting:
./test.go:7: undefined: rows
./test.go:7: cannot assign to rows
./test.go:11: undefined: rows
./test.go:14: undefined: rows
Edit: I've tried using grave accent and single quote strings for db.Query() as well, to no avail.
You cannot assign values to to undeclared variables.
rows, err = db.Query("select id, name from job")
Should be :
rows, err := db.Query("select id, name from job")
Theoretically this should solve the problem, but I haven't tried.
You should also add :
rows.Scan(&id, &name)
Before the printf function so as to actually assign the row's value to the id & name variables otherwise will print an empty string & 0.