Load and Store in Go Language [duplicate] - serialization

This question already has answers here:
Efficient Go serialization of struct to disk
(3 answers)
Closed 6 years ago.
I am new to Go and was wondering if there is a way to load and store precomputed variables in Go like pickle in Python.
My code is creating a map and an array from some data and I don't want to spend time in computation of those, every time the code runs.
I want to load that map and array directly next time I run the code.
Can someone help me with this?
TIA :)

I don't know about how pickle work, if you want to dump a struct into file, may be you can use gob package, see more detail for this How do I dump the struct into the byte array without reflection?。
Also, I found a package that can read and write Python's pickle https://github.com/hydrogen18/stalecucumber.

Calculate your variables once and save them all once in a file,
then open that file and load them all.
When there is no file to open, it is the first time, so calculate and save it once.
You may use your own file format, if you like, or use standard library, like "encoding/json", "encoding/gob", "encoding/csv", "encoding/xml", ....
This:
data := calcOnce()
reads file:
rd, err := ioutil.ReadFile(once)
and if there is no error loads all the variables, otherwise calculates and saves them once.
Here's the working code:
1- Using "encoding/json", try it on The Go Playground:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
)
type Data struct {
A [2]int
B map[int]string
}
func main() {
data := calcOnce()
fmt.Println(data) // {[101 102] map[1:Hello 2:World.]}
}
func calcOnce() Data {
const once = "date.json"
rd, err := ioutil.ReadFile(once)
if err != nil {
//calc and save once:
data := Data{[2]int{101, 102}, map[int]string{1: "Hello ", 2: "World."}}
buf, err := json.Marshal(data)
if err != nil {
panic(err)
}
//fmt.Println(string(buf))
err = ioutil.WriteFile(once, buf, 0666)
if err != nil {
panic(err)
}
return data
}
var d *Data
err = json.Unmarshal(rd, &d)
if err != nil {
panic(err)
}
return *d
}
2- Using "encoding/gob", try it on The Go Playground:
package main
import (
"bytes"
"encoding/gob"
"fmt"
"io/ioutil"
)
type Data struct {
A [2]int
B map[int]string
}
func main() {
data := calcOnce()
fmt.Println(data) // {[1010 102] map[2:World. 1:Hello ]}
}
func calcOnce() Data {
const once = "date.bin"
rd, err := ioutil.ReadFile(once)
if err != nil {
//calc and save once:
data := Data{[2]int{101, 102}, map[int]string{1: "Hello ", 2: "World."}}
buf := &bytes.Buffer{}
err = gob.NewEncoder(buf).Encode(data)
if err != nil {
panic(err)
}
err = ioutil.WriteFile(once, buf.Bytes(), 0666)
if err != nil {
panic(err)
}
return data
}
var d Data
err = gob.NewDecoder(bytes.NewReader(rd)).Decode(&d)
if err != nil {
panic(err)
}
return d
}
3- for protobuf see: Efficient Go serialization of struct to disk

Perhaps the gob package is closest:
https://golang.org/pkg/encoding/gob/

Related

How to optimize database connections

In my Go application I use crontab package to run Tracker function every minute. As you can notice from the code I call PostgreSQL function. To interact with the PostgreSQL database, I use the gorm package. Application worked several days without any problem but now I notice an error in logs: pq: sorry, too many clients already. I know that same questions was asked several times in StackOverflow before. For example in this post people advice to use Exec or Scan methods. In my case as you can see I use Exec method but anyway I have error. As far as I understand, each database request makes a separate connection and does not close it. I can't figure out what I'm doing wrong.
main.go:
package main
import (
"github.com/mileusna/crontab"
)
func main() {
database.ConnectPostgreSQL()
defer database.DisconnectPostgreSQL()
err = crontab.New().AddJob("* * * * *", controllers.Tracker); if err != nil {
utils.Logger().Fatal(err)
return
}
}
tracker.go:
package controllers
import (
"questionnaire/database"
"time"
)
var Tracker = func() {
err := database.DBGORM.Exec("CALL tracker($1)", time.Now().Format("2006-01-02 15:04:05")).Error; if err != nil {
utils.Logger().Println(err) // ERROR: pq: sorry, too many clients already
return
}
}
PostgreSQL.go:
package database
import (
"fmt"
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/postgres"
"github.com/joho/godotenv"
"questionnaire/utils"
)
var DBGORM *gorm.DB
func ConnectPostgreSQL() {
err := godotenv.Load(".env")
if err != nil {
utils.Logger().Println(err)
panic(err)
}
databaseUser := utils.CheckEnvironmentVariable("PostgreSQL_USER")
databasePassword := utils.CheckEnvironmentVariable("PostgreSQL_PASSWORD")
databaseHost := utils.CheckEnvironmentVariable("PostgreSQL_HOST")
databaseName := utils.CheckEnvironmentVariable("PostgreSQL_DATABASE_NAME")
databaseURL:= fmt.Sprintf("host=%s user=%s dbname=%s password=%s sslmode=disable", databaseHost, databaseUser, databaseName, databasePassword)
DBGORM, err = gorm.Open("postgres", databaseURL)
if err != nil {
utils.Logger().Println(err)
panic(err)
}
err = DBGORM.DB().Ping()
if err != nil {
utils.Logger().Println(err)
panic(err)
}
DBGORM.LogMode(true)
}
func DisconnectPostgreSQL() error {
return DBGORM.Close()
}

Golang - Read Os.stdin input but don't echo it

In a golang program I'm reading the Os.Stdin input from a bufio.Reader.
After enter is pressed, the program reads the input and it is then printed onto the console. Is it possible to not print the input onto the console? After reading it, I process the input and reprint it (and no longer need the original input).
I read the data like this:
inputReader := bufio.NewReader(os.Stdin)
for {
outgoing, _ := inputReader.ReadString('\n')
outs <- outgoing
}
I cannot think to other methods than to use ANSI escape codes to clear the terminal and move the cursor to a specific location (in your case to column 1:row 1).
var screen *bytes.Buffer = new(bytes.Buffer)
var output *bufio.Writer = bufio.NewWriter(os.Stdout)
And here are some basic helper methods to ease your job working with terminal.
// Move cursor to given position
func moveCursor(x int, y int) {
fmt.Fprintf(screen, "\033[%d;%dH", x, y)
}
// Clear the terminal
func clearTerminal() {
output.WriteString("\033[2J")
}
Then inside your function you need to clear the terminal and move the cursor to the first column and first row of the terminal window. At the end you have to output the computed result.
for {
outgoing, err := input.ReadString('\n')
if err != nil {
break
}
if _, err := fmt.Sscanf(outgoing, "%f", input); err != nil {
fmt.Println("Input error!")
continue
}
// Clear console
clearTerminal()
moveCursor(1,1)
fmt.Println(outs) // prints the computed result
}
It seems you are looking for a terminal-specific function to disable echo. This is usually used when writing passwords on the terminal (you can type but you don't see the characters).
I suggest you give a try to terminal.ReadPassword it should work nicely and probably in the most cross-platform compatible way.
prompt := ""
t := terminal.NewTerminal(os.Stdin, prompt)
for {
outgoing, err := t.ReadPassword(prompt)
if err != nil {
log.Fatalln(err)
}
outs <- outgoing
}
other than crypto/ssh/terminal;
package main
import (
"bufio"
"fmt"
"os"
"os/exec"
)
func raw(start bool) error {
r := "raw"
if !start {
r = "-raw"
}
rawMode := exec.Command("stty", r)
rawMode.Stdin = os.Stdin
err := rawMode.Run()
if err != nil {
return err
}
return rawMode.Wait()
}
// http://tldp.org/HOWTO/Bash-Prompt-HOWTO/x361.html
func main() {
var rs []rune
raw(true)
for {
inp := bufio.NewReader(os.Stdin)
r, _, err := inp.ReadRune()
if err != nil {
raw(false)
panic(err)
}
if r == '\x03' { // ctrl+c
break
} else if r == '\r' { // enter
fmt.Print(string(rs), "\n\r")
rs = []rune{}
continue
} else if r == '\u007f' { // backspace
fmt.Printf("\033[1D\033[K")
continue
}
rs = append(rs, r)
}
raw(false)
}

Go ioutil using too many file descriptors/leak?

I am going through a list of files and Unmarshalling the xml data in them into an array of structs rArray. I intend to process about 18000 files. When I get to about 1300 files processed, the program panics and says that too many files are open. If I limit the amount of files processed to a safe amount of 1000, the program does not crash.
As seen below, I am using ioutil.ReadFile to read the file data.
for _, f := range files {
func() {
data, err := ioutil.ReadFile("./" + recordDir + "/" + f.Name())
if err != nil {
fmt.Println("error reading %v", err)
return
} else {
if (strings.Contains(filepath.Ext(f.Name()), "xml")) {
//unmarshal data and put into struct array
err = xml.Unmarshal([]byte(data), &rArray[a])
if err != nil {
fmt.Println("error decoding %v: %v",f.Name(), err)
return
}
}
}
}()
}
I am not sure if Go is using too many file descriptors or not closing the files fast enough.
After reading https://groups.google.com/forum/#!topic/golang-nuts/7yXXjgcOikM and viewing the ioutil source in http://golang.org/src/pkg/io/ioutil/ioutil.go, the code for ioutil.ReadFile shows that it uses defer to close the file. defer runs when calling function is returned and ReadFile() is the calling function. Am I correct in this understanding?
I also tried wrapping the ioutil.ReadFile part of my code in a function, but it makes no difference.
My ulimit is set to unlimited.
UPDATE:
I believe that the error of too many files is actually occurring during my Unzip function.
func Unzip(src, dest string) error {
r, err := zip.OpenReader(src)
if err != nil {
return err
}
for _, f := range r.File {
rc, err := f.Open()
if err != nil {
panic(err)
}
path := filepath.Join(dest, f.Name)
if f.FileInfo().IsDir() {
os.MkdirAll(path, f.Mode())
} else {
f, err := os.OpenFile(
path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, f.Mode())
if err != nil {
panic(err)
}
_, err = io.Copy(f, rc)
if err != nil {
panic(err)
}
f.Close()
}
rc.Close()
}
r.Close()
return nil
}
I initially got the Unzip function from https://gist.github.com/hnaohiro/4572580, but upon further inspection, the use of defer in the gist author's function seemed wrong as the file would only be closed after the Unzip() function returned which is too late becuase then 18000 file descriptors will be open. ;)
I replaced the deferred Closes with explicit Close() as shown above, but am still getting the same "too many open files" error. Is there a problem with my modified Unzip function?
UPDATE # 2
Oops, I was running this on Heroku and was pushing to the wrong app with my changes this entire time. Lesson learned: verify target app in heroku toolbelt.
Unzip code from https://gist.github.com/hnaohiro/4572580 does not work as it does not close files until all files processed.
My unzip code with explicit close above works and so does the defer version in #peterSO's answer.
I would modify the Unzip function from https://gist.github.com/hnaohiro/4572580 to the following:
package main
import (
"archive/zip"
"io"
"log"
"os"
"path/filepath"
)
func unzipFile(f *zip.File, dest string) error {
rc, err := f.Open()
if err != nil {
return err
}
defer rc.Close()
path := filepath.Join(dest, f.Name)
if f.FileInfo().IsDir() {
err := os.MkdirAll(path, f.Mode())
if err != nil {
return err
}
} else {
f, err := os.OpenFile(
path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, f.Mode())
if err != nil {
return err
}
defer f.Close()
_, err = io.Copy(f, rc)
if err != nil {
return err
}
}
return nil
}
func Unzip(src, dest string) error {
r, err := zip.OpenReader(src)
if err != nil {
return err
}
defer r.Close()
for _, f := range r.File {
err := unzipFile(f, dest)
if err != nil {
return err
}
}
return nil
}
func main() {
err := Unzip("./sample.zip", "./out")
if err != nil {
log.Fatal(err)
}
}

json.RawMessage from database/sql json column getting overwritten

Getting strange behaviour with a struct with embedded json.
package main
import (
"database/sql"
"encoding/json"
"fmt"
_ "github.com/lib/pq"
)
type Article struct {
Id int
Doc *json.RawMessage
}
func main() {
db, err := sql.Open("postgres", "postgres://localhost/json_test?sslmode=disable")
if err != nil {
panic(err)
}
_, err = db.Query(`create table if not exists articles (id serial primary key, doc json)`)
if err != nil {
panic(err)
}
_, err = db.Query(`truncate articles`)
if err != nil {
panic(err)
}
docs := []string{
`{"type":"event1"}`,
`{"type":"event2"}`,
}
for _, doc := range docs {
_, err = db.Query(`insert into articles ("doc") values ($1)`, doc)
if err != nil {
panic(err)
}
}
rows, err := db.Query(`select id, doc from articles`)
if err != nil {
panic(err)
}
articles := make([]Article, 0)
for rows.Next() {
var a Article
err := rows.Scan(
&a.Id,
&a.Doc,
)
if err != nil {
panic(err)
}
articles = append(articles, a)
fmt.Println("scan", string(*a.Doc), len(*a.Doc))
}
fmt.Println()
for _, a := range articles {
fmt.Println("loop", string(*a.Doc), len(*a.Doc))
}
}
Output:
scan {"type":"event1"} 17
scan {"type":"event2"} 17
loop {"type":"event2"} 17
loop {"type":"event2"} 17
So the articles end up pointing to the same json.
Am I doing something wrong?
UPDATE
Edited to a runnable example. I'm using Postgres and lib/pq.
I ran into this same issue and after looking at if for a long time I read the doc on Scan and it says
If an argument has type *[]byte, Scan saves in that argument a copy of the corresponding data. The copy is owned by the caller and can be modified and held indefinitely. The copy can be avoided by using an argument of type *RawBytes instead; see the documentation for RawBytes for restrictions on its use.
What I think is happening if you use *json.RawMessage then Scan does not see it as a *[]byte and does not copy into it. So you get in internal slice on the next loop Scan overwrites.
Change your Scan to cast the *json.RawMessage to a *[]byte so Scan will copy the values to it.
err := rows.Scan(
&a.Id,
(*[]byte)(a.Doc),
)
In case that helps anyone :
I used masebase anwser to INSERT a json.RawMessage property of my struct in a postgresql db column having jsonb column type.
All you need to do is cast : ([]byte)(a.Doc) in the insert binding method (without the * in my case).

getpasswd functionality in Go?

Situation:
I want to get a password entry from the stdin console - without echoing what the user types. Is there something comparable to getpasswd functionality in Go?
What I tried:
I tried using syscall.Read, but it echoes what is typed.
The following is one of best ways to get it done.
First get term package by go get golang.org/x/term
package main
import (
"bufio"
"fmt"
"os"
"strings"
"syscall"
"golang.org/x/term"
)
func main() {
username, password, _ := credentials()
fmt.Printf("Username: %s, Password: %s\n", username, password)
}
func credentials() (string, string, error) {
reader := bufio.NewReader(os.Stdin)
fmt.Print("Enter Username: ")
username, err := reader.ReadString('\n')
if err != nil {
return "", "", err
}
fmt.Print("Enter Password: ")
bytePassword, err := term.ReadPassword(int(syscall.Stdin))
if err != nil {
return "", "", err
}
password := string(bytePassword)
return strings.TrimSpace(username), strings.TrimSpace(password), nil
}
http://play.golang.org/p/l-9IP1mrhA
Just saw a mail in #go-nuts maillist. There is someone who wrote quite a simple go package to be used. You can find it here: https://github.com/howeyc/gopass
It something like that:
package main
import "fmt"
import "github.com/howeyc/gopass"
func main() {
fmt.Printf("Password: ")
pass := gopass.GetPasswd()
// Do something with pass
}
Since Go ~v1.11 there is an official package golang.org/x/term which replaces the deprecated crypto/ssh/terminal. It has, among other things, the function term.ReadPassword.
Example usage:
package main
import (
"fmt"
"os"
"syscall"
"golang.org/x/term"
)
func main() {
fmt.Print("Password: ")
bytepw, err := term.ReadPassword(int(syscall.Stdin))
if err != nil {
os.Exit(1)
}
pass := string(bytepw)
fmt.Printf("\nYou've entered: %q\n", pass)
}
I had a similar usecase and the following code snippet works well for me. Feel free to try this if you are still stuck here.
import (
"fmt"
"golang.org/x/crypto/ssh/terminal"
)
func main() {
fmt.Printf("Now, please type in the password (mandatory): ")
password, _ := terminal.ReadPassword(0)
fmt.Printf("Password is : %s", password)
}
Of course, you need to install terminal package using go get beforehand.
you can do this by execing stty -echo to turn off echo and then stty echo after reading in the password to turn it back on
Here is a solution that I developed using Go1.6.2 that you might find useful.
It only uses the following standard packages: bufio, fmt, os, strings and syscall. More specifically, it uses syscall.ForkExec() and syscall.Wait4() to invoke stty to disable/enable terminal echo.
I have tested it on Linux and BSD (Mac). It will not work on windows.
// getPassword - Prompt for password. Use stty to disable echoing.
import ( "bufio"; "fmt"; "os"; "strings"; "syscall" )
func getPassword(prompt string) string {
fmt.Print(prompt)
// Common settings and variables for both stty calls.
attrs := syscall.ProcAttr{
Dir: "",
Env: []string{},
Files: []uintptr{os.Stdin.Fd(), os.Stdout.Fd(), os.Stderr.Fd()},
Sys: nil}
var ws syscall.WaitStatus
// Disable echoing.
pid, err := syscall.ForkExec(
"/bin/stty",
[]string{"stty", "-echo"},
&attrs)
if err != nil {
panic(err)
}
// Wait for the stty process to complete.
_, err = syscall.Wait4(pid, &ws, 0, nil)
if err != nil {
panic(err)
}
// Echo is disabled, now grab the data.
reader := bufio.NewReader(os.Stdin)
text, err := reader.ReadString('\n')
if err != nil {
panic(err)
}
// Re-enable echo.
pid, err = syscall.ForkExec(
"/bin/stty",
[]string{"stty", "echo"},
&attrs)
if err != nil {
panic(err)
}
// Wait for the stty process to complete.
_, err = syscall.Wait4(pid, &ws, 0, nil)
if err != nil {
panic(err)
}
return strings.TrimSpace(text)
}
Required launching stty via Go ForkExec() function:
package main
import (
os "os"
bufio "bufio"
fmt "fmt"
str "strings"
)
func main() {
fmt.Println();
if passwd, err := Getpasswd("Enter password: "); err == nil {
fmt.Printf("\n\nPassword: '%s'\n",passwd)
}
}
func Getpasswd(prompt string) (passwd string, err os.Error) {
fmt.Print(prompt);
const stty_arg0 = "/bin/stty";
stty_argv_e_off := []string{"stty","-echo"};
stty_argv_e_on := []string{"stty","echo"};
const exec_cwdir = "";
fd := []*os.File{os.Stdin,os.Stdout,os.Stderr};
pid, err := os.ForkExec(stty_arg0,stty_argv_e_off,nil,exec_cwdir,fd);
if err != nil {
return passwd, os.NewError(fmt.Sprintf("Failed turning off console echo for password entry:\n\t%s",err))
}
rd := bufio.NewReader(os.Stdin);
os.Wait(pid,0);
line, err := rd.ReadString('\n');
if err == nil {
passwd = str.TrimSpace(line)
} else {
err = os.NewError(fmt.Sprintf("Failed during password entry: %s",err))
}
pid, e := os.ForkExec(stty_arg0,stty_argv_e_on,nil,exec_cwdir,fd);
if e == nil {
os.Wait(pid,0)
} else if err == nil {
err = os.NewError(fmt.Sprintf("Failed turning on console echo post password entry:\n\t%s",e))
}
return passwd, err
}
Here is a version specific to Linux:
func terminalEcho(show bool) {
// Enable or disable echoing terminal input. This is useful specifically for
// when users enter passwords.
// calling terminalEcho(true) turns on echoing (normal mode)
// calling terminalEcho(false) hides terminal input.
var termios = &syscall.Termios{}
var fd = os.Stdout.Fd()
if _, _, err := syscall.Syscall(syscall.SYS_IOCTL, fd,
syscall.TCGETS, uintptr(unsafe.Pointer(termios))); err != 0 {
return
}
if show {
termios.Lflag |= syscall.ECHO
} else {
termios.Lflag &^= syscall.ECHO
}
if _, _, err := syscall.Syscall(syscall.SYS_IOCTL, fd,
uintptr(syscall.TCSETS),
uintptr(unsafe.Pointer(termios))); err != 0 {
return
}
}
So to use it:
fmt.Print("password: ")
terminalEcho(false)
var pw string
fmt.Scanln(&pw)
terminalEcho(true)
fmt.Println("")
It's the TCGETS syscall that is linux specific. There are different syscall values for OSX and Windows.
You could also use PasswordPrompt function of https://github.com/peterh/liner package.
Turning off echo before typing and turning on to turn it back on after typing.
Without third library, you can find ways to do with it on unix shown above. But it's difficult on Windows.
You can achieve it by method SetConsoleMode with windows kernel32.dll referring to the accepted answer from C: How to disable echo in windows console?
func GetPassword(prompt string) (err error, text string) {
var modeOn, modeOff uint32
stdin := syscall.Handle(os.Stdin.Fd())
err = syscall.GetConsoleMode(stdin, &modeOn)
if err != nil {
return
}
modeOff = modeOn &^ 0x0004
proc := syscall.MustLoadDLL("kernel32").MustFindProc("SetConsoleMode")
fmt.Print(prompt)
_, _, _ = proc.Call(uintptr(stdin), uintptr(modeOff))
_, err = fmt.Scanln(&text)
if err != nil {
return
}
_, _, _ = proc.Call(uintptr(stdin), uintptr(modeOn))
fmt.Println()
return nil, strings.TrimSpace(text)
}
You can get the behavior you want with the Read method from the os.File object (or the os.Stdin variable). The following sample program will read a line of text (terminated with by pressing the return key) but won't echo it until the fmt.Printf call.
package main
import "fmt"
import "os"
func main() {
var input []byte = make( []byte, 100 );
os.Stdin.Read( input );
fmt.Printf( "%s", input );
}
If you want more advanced behavior, you're probably going to have to use the Go C-wrapper utilities and create some wrappers for low-level api calls.