I have a dataset without column names defined. How can I assign these programmatically using gota? For comparison, Pandas has df.rename.
Say my data looks like this:
1,2,3
4,5,6
7,8,9
I basically want this in my dataframe:
A,B,C
1,2,3
4,5,6
7,8,9
You can call ReadCSV with load options. In go, this method is called functional options.
This is the example in gota:
func main() {
f, err := os.Open("sample2.csv")
if err != nil {
panic(err.Error())
}
defer f.Close()
names := dataframe.Names("A", "B", "C")
noHeader := dataframe.HasHeader(false)
df := dataframe.ReadCSV(f, names, noHeader)
r := df.Records()
fmt.Println(r)
}
Functions Load* and Read* in gota all have LoadOption:
https://pkg.go.dev/github.com/go-gota/gota/dataframe
Related
I'm getting a weird behavior in Rego and I wonder why does it happen.
Link to Rego Playground
When I create an empty array, and than assign to it new array, the count of the first array is still zero:
package play
x[{"msg": msg}]{
c := []
a := [1,2]
b := [3,4]
c = array.concat(a,b)
count(c) > 0
msg := "Length of c is greater than zero"
}
And the output is:
{
"x": []
}
So, I have 2 questions:
Why do I get false in the line count(c)> 0?
How can I assign array to existing one? ( I need it because I have function that returns array and I'm trying to return the concatenation of 2 arrays. e.g.:
func[{"msg": msg}] = c{
a := [1,2]
b := [3,4]
c = array.concat(a,b)
}
Thanks!
Rego values and variables are immutable, so there's no way to assign a new value to an already existing variable. Your example compiles due to using the unification operator (=) rather than the assignment operator (:=).
In the example you provided, simply remove the first assignment:
package play
x[{"msg": msg}]{
a := [1,2]
b := [3,4]
c := array.concat(a,b)
count(c) > 0
msg := "Length of c is greater than zero"
}
I am writing a generic code to query data from any RDS table. I have gone through many StackOverflow answers but none of it worked for me. I have gone through below links:-
panic: sql: expected 1 destination arguments in Scan, not <number> golang, pq, sql
How to query any table of RDS using Golang SDK
My first code is
package main
import (
"fmt"
)
type BA_Client struct {
ClientId int `json:ClientId;"`
CompanyName string `json:CompanyName;"`
CreateDate string `json:CreateDate;"`
}
func main() {
conn, _ := getConnection() // Det database connection
query := `select * from IMBookingApp.dbo.BA_Client
ORDER BY
ClientId ASC
OFFSET 56 ROWS
FETCH NEXT 10 ROWS ONLY ;`
var p []byte
err := conn.QueryRow(query).Scan(&p)
if err != nil {
fmt.Println("Error1", err)
}
fmt.Println("Data:", p)
var m BA_Client
err = json.Unmarshal(p, &m)
if err != nil {
fmt.Println("Error2", err)
}
}
My second code is
conn, _ := getConnection()
query := `select * from IMBookingApp.dbo.BA_Client__c
ORDER BY
ClientId__c ASC
OFFSET 56 ROWS
FETCH NEXT 10 ROWS ONLY ;`
rows, err := conn.Query(query)
if err != nil {
fmt.Println("Error:")
log.Fatal(err)
}
println("rows", rows)
defer rows.Close()
columns, err := rows.Columns()
fmt.Println("columns", columns)
if err != nil {
panic(err)
}
for rows.Next() {
receiver := make([]*string, len(columns))
err := rows.Scan(&receiver )
if err != nil {
fmt.Println("Error reading rows: " + err.Error())
}
fmt.Println("Data:", p)
fmt.Println("receiver", receiver)
}
With both the codes, I am getting the same error as below
sql: expected 3 destination arguments in Scan, not 1
Can it be because I am using SQL Server and not MySQL? Appreciate if anyone can help me to find the issue.
When you want to use a slice of inputs for your row scan, use the variadic 3-dots notation ... to convert the slice into individual parameters, like so:
err := rows.Scan(receiver...)
Your (three in this case) columns, using the variadic arguments will effectively expand like so:
// len(receiver) == 3
err := rows.Scan(receiver[0], receiver[1], receiver[2])
EDIT:
SQL Scan method parameter values must be of type interface{}. So we need an intermediate slice. For example:
is := make([]interface{}, len(receiver))
for i := range is {
is[i] = receiver[i]
// each is[i] will be of type interface{} - compatible with Scan()
// using the underlying concrete `*string` values from `receiver`
}
// ...
err := rows.Scan(is...)
// `receiver` will contain the actual `*string` typed items
try this (no ampersand in front of receiver) for second example:
err := rows.Scan(receiver)
receiver and *string are already a reference types. No need to add another reference to it.
specific field table
query := `select * from IMBookingApp.dbo.BA_Client__c
ORDER BY
ClientId__c ASC
OFFSET 56 ROWS
FETCH NEXT 10 ROWS ONLY ;`
change to
query := `select ClientId, CompanyName, CreateDate from IMBookingApp.dbo.BA_Client__c
ORDER BY
ClientId__c ASC
OFFSET 56 ROWS
FETCH NEXT 10 ROWS ONLY ;`
I am experiencing some really weird inconsistencies when executing queries, and was wondering if anyone knew why.
Imagine I have a struct defined as follows:
type Result struct {
Afield string `db:"A"`
Bfield interface{} `db:"B"`
Cfield string `db:"C"`
Dfield string `db:"D"`
}
And a MySQL Table with the following cols:
A : VARCHAR(50)
B : INT
C : VARCHAR(50)
D : VARCHAR(50)
The query I would like to execute:
SELECT A, B, C, D FROM table WHERE A="a"
first way it can be executed:
db.Get(&result, `SELECT A, B, C, D FROM table WHERE A="a"`)
second way it can be executed:
db.Get(&result, `SELECT A, B, C, D FROM table WHERE A=?`, "a")
The inconsistencies I am experiencing are as follows: When executing the query the first way, the type of Bfield is int. However, when executing the query the second time, it is []uint8.
This outcome is occurring for example when B is 1.
Why is the type of Bfield different depending on how the query is executed?
connection declaration:
// Connection is an interface for making queries.
type Connection interface {
Exec(query string, args ...interface{}) (sql.Result, error)
Get(dest interface{}, query string, args ...interface{}) error
Select(dest interface{}, query string, args ...interface{}) error
}
EDIT
This is also happening using the Go database/sql package + driver. The queries below are assigning Bfield to []uint8 and int64 respectively.
db is of type *sql.DB
query 1:
db.QueryRow(SELECT A, B, C, D FROM table WHERE A="a").Scan(&result.Afield, &result.Bfield, &result.Cfield, &result.Dfield)
-- > type of Bfield is []uint8
query 2:
db.QueryRow(SELECT A, B, C, D FROM table WHERE A=?, "a").Scan(&result.Afield, &result.Bfield, &result.Cfield, &result.Dfield)
--> type of Bfield is int64
EDIT
Something else to note, when chaining multiple WHERE clauses, as long as at least 1 is populated using ?, the query will return int. Otherwise if they are all populated in the string, it will return []uint8
Short answer: because the MySQL driver uses a different protocol for queries with and without parameters. Use a prepared statement to get consistent results.
The following explanation refers to the standard MySQL driver github.com/go-sql-driver/mysql, version 1.4
In the first case, the driver sends the query directly to MySQL, and interprets the result as a *textRows struct. This struct (almost) always decodes results into a byte slice, and leaves the conversion to a better type to the Go sql package. This works fine if the destination is an int, string, sql.Scanner etc, but not for interface{}.
In the second case, the driver detects that there are arguments and returns driver.ErrSkip. This causes the Go SQL package to use a PreparedStatement. And in that case, the MySQL driver uses a *binaryRows struct to interpret the results. This struct uses the declared column type (INT in this case) to decode the value, in this case to decode the value into an int64.
Fun fact: if you provide the interpolateParams=true parameter to the database DSN (e.g. "root:testing#/mysql?interpolateParams=true"), the MySQL driver will prepare the query on the client side, and not use a PreparedStatement. At this point both types of query behave the same.
A small proof of concept:
package main
import (
"database/sql"
"log"
_ "github.com/go-sql-driver/mysql"
)
type Result struct {
Afield string
Bfield interface{}
}
func main() {
db, err := sql.Open("mysql", "root:testing#/mysql")
if err != nil {
log.Fatal(err)
}
defer db.Close()
if _, err = db.Exec(`CREATE TABLE IF NOT EXISTS mytable(A VARCHAR(50), B INT);`); err != nil {
log.Fatal(err)
}
if _, err = db.Exec(`DELETE FROM mytable`); err != nil {
log.Fatal(err)
}
if _, err = db.Exec(`INSERT INTO mytable(A, B) VALUES ('a', 3)`); err != nil {
log.Fatal(err)
}
var (
usingLiteral Result
usingParam Result
usingLiteralPrepared Result
)
row := db.QueryRow(`SELECT B FROM mytable WHERE A='a'`)
if err := row.Scan(&usingLiteral.Bfield); err != nil {
log.Fatal(err)
}
row = db.QueryRow(`SELECT B FROM mytable WHERE A=?`, "a")
if err := row.Scan(&usingParam.Bfield); err != nil {
log.Fatal(err)
}
stmt, err := db.Prepare(`SELECT B FROM mytable WHERE A='a'`)
if err != nil {
log.Fatal(err)
}
defer stmt.Close()
row = stmt.QueryRow()
if err := row.Scan(&usingLiteralPrepared.Bfield); err != nil {
log.Fatal(err)
}
log.Printf("Type when using literal: %T", usingLiteral.Bfield) // []uint8
log.Printf("Type when using param: %T", usingParam.Bfield) // int64
log.Printf("Type when using prepared: %T", usingLiteralPrepared.Bfield) // int64
}
Your first SQL string, in MySql is ambigous and can have too meaning as explained on StackOverflow in following address
When to use single quotes, double quotes, and back ticks in MySQL
Depending on SQL-MODE, your SQL command can be interpreted as
SELECT A, B, C, D FROM table WHERE A='a'
that is what I think you are expecting.
or as
SELECT A, B, C, D FROM table WHERE A=`a`
To avoid this ambiguity, can you make a new FIRST test in replacing double quotes by single quote ?
If the same behavior continue to be there, my answer is not a good response.
If BOTH SQL select return same value, your question has been solved.
Using ` character, you pass a variable name and not a string value !
After reading many tutorials, I found that there are many ways to bind arguments on prepared statement in Go, some of them
SELECT * FROM bla WHERE x = ?col1 AND y = ?col2
SELECT * FROM bla WHERE x = ? AND y = ?
SELECT * FROM bla WHERE x = :col1 AND y = :col2
SELECT * FROM bla WHERE x = $1 AND y = $2
First question, what is the cross-database way to bind arguments? (that works on any database)
Second question, none of the tutorial I've read mention about LIKE statement, how to bind arguments for LIKE-statement correctly?
SELECT * FROM bla WHERE x LIKE /*WHAT?*/
Third question, also none of them give an example for IN statement, how to bind arguments for IN statement correctly?
`SELECT * FROM bla WHERE x IN ( /*WHAT?*/ )
What is the cross-database way to bind arguments?
With database/sql, there is none. Each database has its own way to represent parameter placeholders. The Go database/sql package does not provide any normalization facility for the prepared statements. Prepared statement texts are just passed to the underlying driver, and the driver typically just sends them unmodified to the database server (or library for embedded databases).
How to bind arguments for LIKE-statement correctly?
You can use parameter placeholders after a like statement and bind it as a string. For instance, you could write a prepared statement as:
SELECT a from bla WHERE b LIKE ?
Here is an example (error management handling omitted).
package main
import (
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
)
// > select * from bla ;
// +------+------+
// | a | b |
// +------+------+
// | toto | titi |
// | bobo | bibi |
// +------+------+
func main() {
// Open connection
db, err := sql.Open("mysql", "root:XXXXXXX#/test")
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
defer db.Close()
// Prepare statement for reading data
stmtOut, err := db.Prepare("SELECT a FROM bla WHERE b LIKE ?")
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
defer stmtOut.Close()
var a string
b := "bi%" // LIKE 'bi%'
err = stmtOut.QueryRow(b).Scan(&a)
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
fmt.Printf("a = %s\n", a)
}
Note that the % character is part of the bound string, not of the query text.
How to bind arguments for IN statement correctly?
None of the databases I know allows binding a list of parameters directly with a IN clause. This is not a limitation of database/sql or the drivers, but this is simply not supported by most database servers.
You have several ways to work the problem around:
you can build a query with a fixed number of placeholders in the IN clause. Only bind the parameters you are provided with, and complete the other placeholders by the NULL value. If you have more values than the fixed number you have chosen, just execute the query several times. This is not extremely elegant, but it can be effective.
you can build multiple queries with various number of placeholders. One query for IN ( ? ), a second query for IN (?, ?), a third for IN (?,?,?), etc ... Keep those prepared queries in a statement cache, and choose the right one at runtime depending on the number of input parameters. Note that it takes memory, and generally the maximum number of prepared statements is limited, so it cannot be used when the number of parameters is high.
if the number of input parameters is high, insert them in a temporary table, and replace the query with the IN clause by a join with the temporary table. It is effective if you manage to perform the insertion in the temporary table in one roundtrip. With Go and database/sql, it is not convenient because there is no way to batch queries.
Each of these solutions has drawbacks. None of them is perfect.
I'm a newbie to Go but just to answer the first part:
First question, what is the cross-database way to bind arguments? (that works on any database)
If you use sqlx, which is a superset of the built-in sql package, then you should be able to use sqlx.DB.Rebind to achieve that.
I had this same question, and after reading the answers started to look for other solution on how to bind arguments for the IN statement.
Here is an example of what I did, not the most elegant solution, but works for me.
What I did was to create a select query with the parameters statically set on the query, and not using the bind feature at all.
It could be a good idea to sanitize the string that comes from the Marshal command, to be sure and safe, but I don't need it now.
package main
import (
"database/sql"
"encoding/json"
"fmt"
"log"
_ "github.com/go-sql-driver/mysql"
)
type Result struct {
Identifier string
Enabled bool
}
func main() {
// Open connection
db, err := sql.Open("mysql", "username:password#tcp(server-host)/my-database")
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
defer db.Close()
// this is an example of a variable list of IDs
idList := []string{"ID1", "ID2", "ID3", "ID4", "ID5", "IDx"}
// convert the list to a JSON string
formatted, _ := json.Marshal(idList)
// a JSON array starts and ends with '[]' respectivelly, so we replace them with '()'
formatted[0] = '('
formatted[len(formatted)-1] = ')'
// create a static select query
query := fmt.Sprintf("SELECT identifier, is_enabled FROM some_table WHERE identifier in %s", string(formatted))
// prepare que query
rows, err := db.Query(query)
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
defer rows.Close()
var result []Result
// fetch rows
for rows.Next() {
var r0 Result
if err := rows.Scan(&r0.Identifier, &r0.Enabled); err != nil {
log.Fatal(err)
}
// append the row to the result
result = append(result, r0)
}
if err := rows.Err(); err != nil {
log.Fatal(err)
}
fmt.Printf("result = %v\n", result)
}
It seems golang's sqlite package doesn't like my db.Query statement, though it's exactly like the one found in the example on github.
db, err := sql.Open("sqlite3", "./database.db")
if err != nil {
log.Fatal(err)
}
defer db.Close()
rows, err = db.Query("select id, name from job")
if err != nil {
log.Fatal(err)
}
defer rows.Close()
fmt.Println("Jobs:")
for rows.Next() {
var name string
var id int
fmt.Printf("%v %v\n", id, name)
}
This is the error I'm getting:
./test.go:7: undefined: rows
./test.go:7: cannot assign to rows
./test.go:11: undefined: rows
./test.go:14: undefined: rows
Edit: I've tried using grave accent and single quote strings for db.Query() as well, to no avail.
You cannot assign values to to undeclared variables.
rows, err = db.Query("select id, name from job")
Should be :
rows, err := db.Query("select id, name from job")
Theoretically this should solve the problem, but I haven't tried.
You should also add :
rows.Scan(&id, &name)
Before the printf function so as to actually assign the row's value to the id & name variables otherwise will print an empty string & 0.