I'm trying to put a (key,value) in aerospike
that it's value is: ("mykey",3).
is this the correct technique?
because i cannot fetch it...
key, err := as.NewKey("namespace", "set","mykey")
if err != nil {
log.Fatal(err)
}
exists, err := client.Exists(Policy, key)
if exists {
// Read a record
record, err := client.Get(Policy, key)
if err != nil {
log.Fatal(err)
}
var newval = 3
}
bin1 := as.NewBin("bin1", newval )
// Write a record
err = client.PutBins(WritePolicy, key, bin1)
Related
I am dealing with a bit over 15 billion rows of data in various text files. I am trying to insert them into MariaDB using golang. Golang is a fast language and is often used for big data but I cannot get more than 10k-15k inserts a second, at this rate its gonna take over 15 days, I need this data imported sooner than that. I have tried various batch sizes but they all give about the same results.
function I'm using to handle file data:
func handlePath(path string) {
file, err := os.Open(path)
if err != nil {
fmt.Printf("error opening %v: %v", path, err)
return
}
defer file.Close()
scanner := bufio.NewScanner(file)
var temp_lines []string
for scanner.Scan() {
if len(temp_lines) == line_batch {
insertRows(temp_lines)
temp_lines = []string{}
}
temp_lines = append(temp_lines, scanner.Text())
}
insertRows(temp_lines)
fmt.Printf("\nFormatted %v\n", path)
if err := scanner.Err(); err != nil {
fmt.Printf("\nScanner error %v\n", err)
return
}
}
function I'm using for inserting:
func insertRows(rows []string) {
var Args []string
for _, row := range rows {
line_split := strings.Split(row, "|")
if len(line_split) != 6 {return}
database_id := line_split[0]
email := line_split[1]
password := line_split[2]
username := line_split[3]
ip := line_split[4]
phone := line_split[5]
arg := fmt.Sprintf("('%v','%v','%v','%v','%v','%v')",database_id,email,password,username,ip,phone)
Args = append(Args, arg)
}
sqlQuery := fmt.Sprintf("INSERT INTO new_table (database_id, email, password, username, ip, phone_number) VALUES %s", strings.Join(Args, ","))
_, err := db.Exec(sqlQuery)
if err != nil {
//fmt.Printf("%v\n", err)
return
}
total+=line_batch
writes++
}
Server specs:
server
I'm trying to use pq.CopyIn to do bulk imports as described here:
https://godoc.org/github.com/lib/pq
The import is much faster than other methods I've tried but I am finding that a unique constraint violation in just one record will cause the entire import to fail.
Is there any way to set ON CONFLICT DO NOTHING using pq.CopyIn.
Here is a copy of my table structure
CREATE TABLE test (
id serial PRIMARY KEY,
unique_token VARCHAR ( 10 ) UNIQUE NOT NULL,
frequency INT DEFAULT 0
);
I tried using #mkopriva answer below but I'm getting Error: pq: null
value in column "id" violates not-null constraint
Code sample below
tx, _ := db.Begin()
_, err = tx.Exec(`CREATE TEMP TABLE token_temp ON COMMIT DROP AS
SELECT id, unique_token FROM test WITH NO DATA`)
if err != nil {
return err
}
stmt, err := tx.Prepare(pq.CopyIn("token_temp", "unique_token"))
if err != nil {
fmt.Println("error here")
return err
}
for _, token := range tokenList {
_, err = stmt.Exec(token)
if err != nil {
return err
}
}
_, err = stmt.Exec()
if err != nil {
log.Fatal(err)
}
err = stmt.Close()
if err != nil {
log.Fatal(err)
}
_, err = tx.Exec(`INSERT INTO test SELECT id, unique_token FROM
token_temp ON CONFLICT(unique_token) DO UPDATE SET frequency=
test.frequency + 1 `)
if err != nil {
fmt.Println("Error")
return err
}
err = tx.Commit()
if err != nil {
log.Fatal(err)
}
pq.CopyIn internally uses COPY FROM which has no support for the ON CONFLICT clause.
What you can do, however, is to create a temporary table that has no constraints, copy the data into that temporary table, and then do an INSERT into the target table, with your ON CONFLICT clause, using the temporary table as the source of the data to be inserted.
An example should make this more clear, say you have a users table that looks like this:
CREATE TABLE users (
id serial PRIMARY KEY
, name text
, email text UNIQUE
);
And say you have a slice of users like this:
var users = []User{
{Name: "John Doe", Email: "jdoe#example.com"},
{Name: "Joe Blow", Email: "jblow#example.com"},
{Name: "Jane Doe", Email: "jdoe#example.com"}, // duplicate email!
{Name: "Foo Bar", Email: "fbar#example.com"},
}
With that you can do the following:
_, err = txn.Exec(`
CREATE TEMP TABLE users_temp
ON COMMIT DROP
AS SELECT * FROM users
WITH NO DATA`)
if err != nil {
panic(err)
}
stmt, err := txn.Prepare(pq.CopyIn("users_temp", "name", "email"))
if err != nil {
panic(err)
}
for _, u := range users {
if _, err := stmt.Exec(u.Name, u.Email); err != nil {
panic(err)
}
}
if _, err := stmt.Exec(); err != nil {
panic(err)
}
if err := stmt.Close(); err != nil {
panic(err)
}
_, err = txn.Exec(`
INSERT INTO users (name, email)
SELECT name, email FROM users_temp
ON CONFLICT DO NOTHING`)
if err != nil {
panic(err)
}
if err := txn.Commit(); err != nil {
panic(err)
}
After you run the above you can do SELECT * FROM users; and you'll get this:
id | name | email
----+----------+-------------------
1 | John Doe | jdoe#example.com
2 | Joe Blow | jblow#example.com
4 | Foo Bar | fbar#example.com
(3 rows)
For you specific example and requirement you can do something like this in the INSERT ... SELECT ... query:
_, err = txn.Exec(`
INSERT INTO test (unique_token, frequency)
SELECT unique_token, COUNT(*) FROM token_temp
GROUP BY unique_token`)
if err != nil {
panic(err)
}
I'm working on a simple REST API and I'm having troubles with the PATCH method. I don't know how can i tell the method and the query which fields i want to update(for example which fields are passed as JSON) in the database. Here is what i have so far.
func PatchServer(c echo.Context) error {
patchedServer := new(structs.Server)
requestID := c.Param("id")
if err := c.Bind(patchedServer); err != nil {
return err
}
sql := "UPDATE servers SET server_name = CASE WHEN ? IS NOT NULL THEN ? END WHERE id = ?"
stmt, err := db.Get().Prepare(sql)
if err != nil {
panic(err)
}
_, err2 := stmt.Exec(patchedServer.Name, patchedServer.Name, requestID)
if err2 != nil {
panic(err2)
}
fmt.Println(patchedServer.ID, patchedServer.Name, patchedServer.Components)
fmt.Println("Requested id: ", requestID)
return c.JSON(http.StatusOK, "Patched!")
}
For the context, I'm new to go and I'm creating a program that can copy tables from Oracle to MySQL.
I use database/sql go package, so I assume it can be used for migrating any kind of database.
To simplify my question I'm coping on the same MySQL database table name world.city to world.city_copy2.
with my following code, I ended up with the same last values in all the rows in the table :-(
do I somehow need to read through all the values inside the loop? what is the efficient way to do that?
package main
import (
"database/sql"
"fmt"
"strings"
_ "github.com/go-sql-driver/mysql"
)
const (
user = "user"
pass = "testPass"
server = "localhost"
)
func main() {
fmt.Print("test")
conStr := fmt.Sprintf("%s:%s#tcp(%s)/world", user, pass, server)
db, err := sql.Open("mysql", conStr)
if err != nil {
panic(err.Error())
}
defer db.Close()
err = db.Ping()
if err != nil {
panic(err.Error())
}
rows, err := db.Query("SELECT * FROM city")
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
columns, err := rows.Columns()
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
// Make a slice for the values
values := make([]sql.RawBytes, len(columns))
// rows.Scan wants '[]interface{}' as an argument, so we must copy the
// references into such a slice
scanArgs := make([]interface{}, len(values))
for i := range values {
scanArgs[i] = &values[i]
}
// that string will be generated according to len of columns
placeHolders := "( ?, ?, ?, ?, ? )"
// slice will contain all the values at the end
bulkValues := []interface{}{}
valueStrings := make([]string, 0)
for rows.Next() {
// get RawBytes from data
err = rows.Scan(scanArgs...)
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
valueStrings = append(valueStrings, placeHolders)
bulkValues = append(bulkValues, scanArgs...)
//
}
stmStr := fmt.Sprintf("INSERT INTO city_copy2 VALUES %s", strings.Join(valueStrings, ","))
_, err = db.Exec(stmStr, bulkValues...)
if err != nil {
panic(err.Error())
}
}
I have checked out the docs of the library, and it seems that the problem here is that bulkValues keeps the address of the pointer so when scanArgs changes, bulkValues also changes to latest value of that scanArgs.
You need to use the values variable to get the values like below:
func main() {
fmt.Print("test")
conStr := fmt.Sprintf("%s:%s#tcp(%s)/soverflow", user, pass, server)
db, err := sql.Open("mysql", conStr)
if err != nil {
panic(err.Error())
}
defer db.Close()
err = db.Ping()
if err != nil {
panic(err.Error())
}
rows, err := db.Query("SELECT * FROM city")
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
columns, err := rows.Columns()
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
// Make a slice for the values
values := make([]sql.RawBytes, len(columns))
// rows.Scan wants '[]interface{}' as an argument, so we must copy the
// references into such a slice
scanArgs := make([]interface{}, len(values))
for i := range values {
scanArgs[i] = &values[i]
}
// that string will be generated according to len of columns
placeHolders := "( ?, ?, ?, ?, ? )"
// slice will contain all the values at the end
bulkValues := []interface{}{}
valueStrings := make([]string, 0)
// make an interface to keep the record's value
record := make([]interface{}, len(columns))
for rows.Next() {
// get RawBytes from data
err = rows.Scan(scanArgs...)
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
valueStrings = append(valueStrings, placeHolders)
for i, col := range values {
// you need to be carefull with the datatypes here
// check out the docs for details on here
record[i] = string(value)
}
bulkValues = append(bulkValues, record...)
}
stmStr := fmt.Sprintf("INSERT INTO city_copy2 VALUES %s", strings.Join(valueStrings, ","))
_, err = db.Exec(stmStr, bulkValues...)
if err != nil {
panic(err.Error())
}
}
You can also find the example of the documentation here.
Note: There might be more efficient ways to copy database from psql to mysql but this answer only gives a quick solution for this particular issue that you are having.
In my html page, user clicks on Download button after entering the start and end date to download data in csv format.
I have used jquery to get the user inputs from the screen and pass the values to main.go
Now in main.go, I am trying to run a select query to read all the data in database between the two dates that is coming from jquery.
Something like below:
func adminPage(w http.ResponseWriter, r *http.Request) {
dsn := "server=eccdb1677.md3q.ford.com;user id=prx_mdesk_appl;password=******"
db, err := sql.Open("mssql", dsn)
if err != nil {
log.Fatal(err)
}
err = db.Ping()
if err != nil {
log.Fatal(err)
}
defer db.Close()
r.ParseForm()
StartDate := r.FormValue("startdate")
EndDate := r.FormValue("enddate")
rows, _ := db.Query("SELECT * FROM mdesk.dbo.tbl_fdbk WHERE CreateDate BETWEEN StartDate and EndDate")
for rows.Next() {
var (
CDSID string
CreateDate int
Rating int
Comments string
)
err = rows.Scan(&CDSID, &CreateDate, &Rating, &Comments)
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
file, err := os.Create("reports.csv")
if err != nil {
log.Fatal(err)
}
defer file.Close()
w := csv.NewWriter(file)
err = w.WriteAll(rows)
if err != nil {
log.Fatal("Error writing record to csv:", err)
}
}
}
This code has error in so many levels, but right now I'm focusing more on how to write the select query to read the data between start date and end date.
Any help would be appreciated.