I am trying to implement a simple api in Golang. My experience in the backend is more with python and node, so I am having some difficulty printing out data held within the interface since it won't allow me to index it. I have searched around and several people have asked similar questions when the interface is one value, but not when the interface is a slice, I believe ([]interface{}). I have tried vaping the interface to no avail.
When I point the browser to /quandl/ddd/10 I would like to fmt.Println the specific numerical data, i.e. ("2017-01-13",
15.67,
16.41,
15.67,
16.11,
3595248,
0,
1,
15.67,
16.41,
15.67,
16.11,
3595248
])
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"net/url"
"github.com/fatih/color"
"github.com/gorilla/mux"
)
type QuandlResponse struct {
SourceCode string `json:"source_code"`
SourceName string `json:"source_name"`
Code string `json:"code"`
Frequency string `json:"frequency"`
FromDate string `json:"from_date"`
ToDate string `json:"to_date"`
Columns []string `json:"column_names"`
Data interface{} `json:"data"`
}
func getContent(w http.ResponseWriter, r *http.Request) {
stock := mux.Vars(r)["stock"]
limit := mux.Vars(r)["limit"]
url := "https://www.quandl.com/api/v1/datasets/WIKI/" + url.QueryEscape(stock) + ".json?&limit=" + url.QueryEscape(limit) + "&auth_token=XXXXX"
response, err := http.Get(url)
if err != nil {
fmt.Println(err)
}
contents, err := ioutil.ReadAll(response.Body)
var result QuandlResponse
json.Unmarshal(contents, &result)
json.NewEncoder(w).Encode(result)
fmt.Println(result.Data[0])
}
func callAll() {
rabbit := mux.NewRouter()
rabbit.HandleFunc("/quandl/{stock}/{limit}", getContent)
http.ListenAndServe(":8000", rabbit)
}
func main() {
color.Blue("Running Server #localhost:8000")
callAll()
}
If you know that the type of Data is []interface{}, you can do a type assertion:
slice := result.Data.([]interface{})
fmt.Println(slice[0])
If there are several possibilities for the type of Data, you can use a type switch:
switch data := result.Data.(type) {
case []interface{}:
fmt.Println(data[0])
case string:
fmt.Println(data)
default:
// unexpected type
}
You may also want to look at the reflect package if your requirements are more complicated.
Related
I am having problems getting the golang validator to work with SQL null types. Here's an example of what I tried:
package main
import (
"database/sql"
"database/sql/driver"
"log"
"gopkg.in/go-playground/validator.v9"
)
// NullInt64
type NullInt64 struct {
sql.NullInt64
Set bool
}
func MakeNullInt64(valid bool, val int64) NullInt64 {
n := NullInt64{}
n.Set = true
n.Valid = valid
if valid {
n.Int64 = val
}
return n
}
func (n *NullInt64) Value() (driver.Value, error) {
if !n.NullInt64.Valid {
return nil, nil
}
return n.NullInt64.Int64, nil
}
type Thing struct {
N2 NullInt64 `validate:"min=10"`
N3 int64 `validate:"min=10"`
N4 *int64 `validate:"min=10"`
}
func main() {
validate := validator.New()
n := int64(6)
number := MakeNullInt64(true, n)
thing := Thing{number, n, &n}
e := validate.Struct(thing)
log.Println(e)
}
When I run this code, I only get this output:
Key: 'Thing.N3' Error:Field validation for 'N3' failed on the 'min'
tag
Key: 'Thing.N4' Error:Field validation for 'N4' failed on the
'min' tag
The problem is that I want it to also show that Thing.N2 failed for the same reasons as Thing.N3 and Thing.N4.
I tried introducing the func (n *NullInt64) Value() method because it was mentioned in the documentation. But I think I misunderstood something. Can anyone tell me what I did wrong?
UPDATE
There is an Example specifically for that. You may check it out. My other proposed solution should still work though.
Since the value you are trying to validate is Int64 inside sql.NullInt64, the easiest way would be to remove the validate tag and just register a Struct Level validation using:
validate.RegisterStructValidation(NullInt64StructLevelValidation, NullInt64{})
while NullInt64StructLevelValidation is a StructLevelFunc that looks like this:
func NullInt64StructLevelValidation(sl validator.StructLevel) {
ni := sl.Current().Interface().(NullInt64)
if ni.NullInt64.Int64 < 10 {
sl.ReportError(ni.NullInt64.Int64, "Int64", "", "min", "")
}
}
Note #1: this line thing := Thing{number,&number,n,&n} has one argument too many. I assume you meant thing := Thing{number, n, &n}
Note #2: Go tooling including gofmt is considered to be one of the most powerful features of the language. Please consider using it/them.
EDIT #1:
I don't think implementing Valuer interface is of any value in this context.
Would it be possible to use Gob encoding for appending structs in series to the same file using append? It works for writing, but when reading with the decoder more than once I run into:
extra data in buffer
So I wonder if that's possible in the first place or whether I should use something like JSON to append JSON documents on a per line basis instead. Because the alternative would be to serialize a slice, but then again reading it as a whole would defeat the purpose of append.
The gob package wasn't designed to be used this way. A gob stream has to be written by a single gob.Encoder, and it also has to be read by a single gob.Decoder.
The reason for this is because the gob package not only serializes the values you pass to it, it also transmits data to describe their types:
A stream of gobs is self-describing. Each data item in the stream is preceded by a specification of its type, expressed in terms of a small set of predefined types.
This is a state of the encoder / decoder–about what types and how they have been transmitted–, a subsequent new encoder / decoder will not (cannot) analyze the "preceeding" stream to reconstruct the same state and continue where a previous encoder / decoder left off.
Of course if you create a single gob.Encoder, you may use it to serialize as many values as you'd like to.
Also you can create a gob.Encoder and write to a file, and then later create a new gob.Encoder, and append to the same file, but you must use 2 gob.Decoders to read those values, exactly matching the encoding process.
As a demonstration, let's follow an example. This example will write to an in-memory buffer (bytes.Buffer). 2 subsequent encoders will write to it, then we will use 2 subsequent decoders to read the values. We'll write values of this struct:
type Point struct {
X, Y int
}
For short, compact code, I use this "error handler" function:
func he(err error) {
if err != nil {
panic(err)
}
}
And now the code:
const n, m = 3, 2
buf := &bytes.Buffer{}
e := gob.NewEncoder(buf)
for i := 0; i < n; i++ {
he(e.Encode(&Point{X: i, Y: i * 2}))
}
e = gob.NewEncoder(buf)
for i := 0; i < m; i++ {
he(e.Encode(&Point{X: i, Y: 10 + i}))
}
d := gob.NewDecoder(buf)
for i := 0; i < n; i++ {
var p *Point
he(d.Decode(&p))
fmt.Println(p)
}
d = gob.NewDecoder(buf)
for i := 0; i < m; i++ {
var p *Point
he(d.Decode(&p))
fmt.Println(p)
}
Output (try it on the Go Playground):
&{0 0}
&{1 2}
&{2 4}
&{0 10}
&{1 11}
Note that if we'd use only 1 decoder to read all the values (looping until i < n + m, we'd get the same error message you posted in your question when the iteration reaches n + 1, because the subsequent data is not a serialized Point, but the start of a new gob stream.
So if you want to stick with the gob package for doing what you want to do, you have to slightly modify, enhance your encoding / decoding process. You have to somehow mark the boundaries when a new encoder is used (so when decoding, you'll know you have to create a new decoder to read subsequent values).
You may use different techniques to achieve this:
You may write out a number, a count before you proceed to write values, and this number would tell how many values were written using the current encoder.
If you don't want to or can't tell how many values will be written with the current encoder, you may opt to write out a special end-of-encoder value when you don't write more values with the current encoder. When decoding, if you encounter this special end-of-encoder value, you'll know you have to create a new decoder to be able to read more values.
Some things to note here:
The gob package is most efficient, most compact if only a single encoder is used, because each time you create and use a new encoder, the type specifications will have to be re-transmitted, causing more overhead, and making the encoding / decoding process slower.
You can't seek in the data stream, you can only decode any value if you read the whole file from the beginning up until the value you want. Note that this somewhat applies even if you use other formats (such as JSON or XML).
If you want seeking functionality, you'd need to manage an index file separately, which would tell at which positions new encoders / decoders start, so you could seek to that position, create a new decoder, and start reading values from there.
Check a related question: Efficient Go serialization of struct to disk
In addition to the above, I suggest using an intermediate structure to exclude the gob header:
package main
import (
"bytes"
"encoding/gob"
"fmt"
"io"
"log"
)
type Point struct {
X, Y int
}
func main() {
buf := new(bytes.Buffer)
enc, _, err := NewEncoderWithoutHeader(buf, new(Point))
if err != nil {
log.Fatal(err)
}
enc.Encode(&Point{10, 10})
fmt.Println(buf.Bytes())
}
type HeaderSkiper struct {
src io.Reader
dst io.Writer
}
func (hs *HeaderSkiper) Read(p []byte) (int, error) {
return hs.src.Read(p)
}
func (hs *HeaderSkiper) Write(p []byte) (int, error) {
return hs.dst.Write(p)
}
func NewEncoderWithoutHeader(w io.Writer, sample interface{}) (*gob.Encoder, *bytes.Buffer, error) {
hs := new(HeaderSkiper)
hdr := new(bytes.Buffer)
hs.dst = hdr
enc := gob.NewEncoder(hs)
// Write sample with header info
if err := enc.Encode(sample); err != nil {
return nil, nil, err
}
// Change writer
hs.dst = w
return enc, hdr, nil
}
func NewDecoderWithoutHeader(r io.Reader, hdr *bytes.Buffer, dummy interface{}) (*gob.Decoder, error) {
hs := new(HeaderSkiper)
hs.src = hdr
dec := gob.NewDecoder(hs)
if err := dec.Decode(dummy); err != nil {
return nil, err
}
hs.src = r
return dec, nil
}
Additionally to great icza answer, you could use the following trick to append to a gob file with already written data: when append the first time write and discard the first encode:
Create the file Encode gob as usual (first encode write headers)
Close file
Open file for append
Using and intermediate writer encode dummy struct (which write headers)
Reset the writer
Encode gob as usual (writes no headers)
Example:
package main
import (
"bytes"
"encoding/gob"
"fmt"
"io"
"io/ioutil"
"log"
"os"
)
type Record struct {
ID int
Body string
}
func main() {
r1 := Record{ID: 1, Body: "abc"}
r2 := Record{ID: 2, Body: "def"}
// encode r1
var buf1 bytes.Buffer
enc := gob.NewEncoder(&buf1)
err := enc.Encode(r1)
if err != nil {
log.Fatal(err)
}
// write to file
err = ioutil.WriteFile("/tmp/log.gob", buf1.Bytes(), 0600)
if err != nil {
log.Fatal()
}
// encode dummy (which write headers)
var buf2 bytes.Buffer
enc = gob.NewEncoder(&buf2)
err = enc.Encode(Record{})
if err != nil {
log.Fatal(err)
}
// remove dummy
buf2.Reset()
// encode r2
err = enc.Encode(r2)
if err != nil {
log.Fatal(err)
}
// open file
f, err := os.OpenFile("/tmp/log.gob", os.O_WRONLY|os.O_APPEND, 0600)
if err != nil {
log.Fatal(err)
}
// write r2
_, err = f.Write(buf2.Bytes())
if err != nil {
log.Fatal(err)
}
// decode file
data, err := ioutil.ReadFile("/tmp/log.gob")
if err != nil {
log.Fatal(err)
}
var r Record
dec := gob.NewDecoder(bytes.NewReader(data))
for {
err = dec.Decode(&r)
if err == io.EOF {
break
}
if err != nil {
log.Fatal(err)
}
fmt.Println(r)
}
}
I don't understand type conversion. I know this isn't right, all I get is a bunch of hieroglyphs.
f, _ := os.Open("test.pdf")
defer f.Close()
io.Copy(os.Stdout, f)
I want to work with the strings....
I tried some go pdf libs, and found sajari/docconv works like I expect.
easy to use, here is a example:
package main
import (
"fmt"
"log"
"code.sajari.com/docconv"
)
func main() {
res, err := docconv.ConvertPath("your-file.pdf")
if err != nil {
log.Fatal(err)
}
fmt.Println(res)
}
It's because the PDF doesn't only contain the text, but it also contains the formats (fonts, padding, margin, position, shapes, image) information.
In case you need to read the plain text without format. I have forked a repository and implement the function to do that. You can check it at https://github.com/ledongthuc/pdf
I also have put an example, help it useful for you.
package main
import (
"bytes"
"fmt"
"github.com/ledongthuc/pdf"
)
func main() {
content, err := readPdf("test.pdf") // Read local pdf file
if err != nil {
panic(err)
}
fmt.Println(content)
return
}
func readPdf(path string) (string, error) {
r, err := pdf.Open(path)
if err != nil {
return "", err
}
totalPage := r.NumPage()
var textBuilder bytes.Buffer
for pageIndex := 1; pageIndex <= totalPage; pageIndex++ {
p := r.Page(pageIndex)
if p.V.IsNull() {
continue
}
textBuilder.WriteString(p.GetPlainText("\n"))
}
return textBuilder.String(), nil
}
all I get is a bunch of hieroglyphs.
What you get is the content of a pdf file, which is not clear text.
If you want to read a pdf file in Go, use one of the golang pdf libraries like rsc.io/pdf, or one of those libraries like yob/pdfreader.
As mentioned here:
I doubt there is any 'solid framework' for this kind of stuff. PDF format isn't meant to be machine-friendly by design, and AFAIK there is no guaranteed way to parse arbitrary PDFs.
Disclaimer: I wish you a merry XMas and I hope my question does not disturb you!
sample.go:
package main
import(
"fmt"
"os"
)
type sample struct {
value int64
}
func (s sample) useful() {
if s.value == 0 {
fmt.Println("Error: something is wrong!")
os.Exit(1)
} else {
fmt.Println("May the force be with you!")
}
}
func main() {
s := sample{42}
s.useful()
s.value = 0
s.useful()
}
// output:
// May the force be with you!
// Error: something is wrong!
// exit status 1
I did a lot of research on how to use interfaces in golang testing. But so far I was not able to wrap my head around this completely. At least I can not see how interfaces help me when I need to "mock" (apologies for using this word) golang std. library packages like "fmt".
I came up with two scenarios:
use os/exec to test the command line interface
wrap fmt package so I have control and am able to check the output strings
I do not like both scenarios:
I experience going through the actual command line a convoluted and not-performant (see below). Might have portability issues, too.
I believe this is the way to go but I fear that wrapping the fmt package might be a lot of work (at least wrapping the time package for testing turned out a non-trivial task (https://github.com/finklabs/ttime)).
Actual Question here: Is there another (better/simpler/idiomatic) way?
Note: I want to do this in pure golang, I am not interested in the next testing framework.
cli_test.go:
package main
import(
"os/exec"
"testing"
)
func TestCli(t *testing.T) {
out, err := exec.Command("go run sample.go").Output()
if err != nil {
t.Fatal(err)
}
if string(out) != "May the force be with you!\nError: this is broken and not useful!\nexit status 1" {
t.Fatal("There is something wrong with the CLI")
}
}
Chapter 11 of Kerningham's Book gives a good solution to this question.
The trick is to change the calls to fmt.Printline() to calls to
fmt.Fprint(out, ...) where out is initialised to os.Stdout
This can be overwritten in the test harness to new(bytes.Buffer) allowing the
test to capture the output.
See https://github.com/adonovan/gopl.io/blob/master/ch11/echo/echo.go and
https://github.com/adonovan/gopl.io/blob/master/ch11/echo/echo_test.go
edited by OP...
sample.go:
package main
import(
"fmt"
"os"
"io"
)
var out io.Writer = os.Stdout // modified during testing
var exit func(code int) = os.Exit
type sample struct {
value int64
}
func (s sample) useful() {
if s.value == 0 {
fmt.Fprint(out, "Error: something is wrong!\n")
exit(1)
} else {
fmt.Fprint(out, "May the force be with you!\n")
}
}
func main() {
s := sample{42}
s.useful()
s.value = 0
s.useful()
}
// output:
// May the force be with you!
// Error: this is broken and not useful!
// exit status 1
cli_test.go:
package main
import(
"bytes"
"testing"
)
func TestUsefulPositive(t *testing.T) {
bak := out
out = new(bytes.Buffer)
defer func() { out = bak }()
s := sample{42}
s.useful()
if out.(*bytes.Buffer).String() != "May the force be with you!\n" {
t.Fatal("There is something wrong with the CLI")
}
}
func TestUsefulNegative(t *testing.T) {
bak := out
out = new(bytes.Buffer)
defer func() { out = bak }()
code := 0
osexit := exit
exit = func(c int) { code = c }
defer func() { exit = osexit }()
s := sample{0}
s.useful()
if out.(*bytes.Buffer).String() != "Error: something is wrong!\n" {
t.Fatal("There is something wrong with the CLI")
}
if code != 1 {
t.Fatal("Wrong exit code!")
}
}
Am I missing something here or are you talking of testable examples?
Basically, it works like this: In a *_test.go file, you need to adhere to the convention Example[[T][_M]] where T is a placeholder for the type and M a placeholder for the method you want to display the testable example as example code in the Godoc. If the function is just called Example(), the code will be shown as a package example.
Below the last line of the code of your example, you can put a comment like this
// Output:
// Foo
Now go test will make sure that the testable example function either exactly puts out everything below // Output: (including whitespace) or it will make the test fail.
Here is an actual example for an testable example
func ExampleMongoStore_Get() {
sessionId := "ExampleGetSession"
data, err := ms.Get(sessionId)
if err == sessionmw.ErrSessionNotFound {
fmt.Printf("Session '%s' not found\n", sessionId)
data = make(map[string]interface{})
data["foo"] = "bar"
ms.Save(sessionId, data)
}
loaded, _ := ms.Get(sessionId)
fmt.Printf("Loaded value '%s' for key '%s' in session '%s'",
loaded["foo"],
"foo", sessionId)
// Output:
// Session 'ExampleGetSession' not found
// Loaded value 'bar' for key 'foo' in session 'ExampleGetSession'
}
Edit: Have a look at the output of above example at godoc.org
I'm completely new to Go and I'm trying to read a binary file, either byte by byte or several bytes at a time. The documentation doesn't help much and I cannot find any tutorial or simple example (by the way, how could Google give their language such an un-googlable name?). Basically, how can I open a file, then read some bytes into a buffer? Any suggestion?
For manipulating files, the os package is your friend:
f, err := os.Open("myfile")
if err != nil {
panic(err)
}
defer f.Close()
For more control over how the file is open, see os.OpenFile() instead (doc).
For reading files, there are many ways. The os.File type returned by os.Open (the f in the above example) implements the io.Reader interface (it has a Read() method with the right signature), it can be used directly to read some data in a buffer (a []byte) or it can also be wrapped in a buffered reader (type bufio.Reader).
Specifically for binary data, the encoding/binary package can be useful, to read a sequence of bytes into some typed structure of data. You can see an example in the Go doc here. The binary.Read() function can be used with the file read using the os.Open() function, since as I mentioned, it is a io.Reader.
And there's also the simple to use io/ioutil package, that allows you to read the whole file at once in a byte slice (ioutil.ReadFile(), which takes a file name, so you don't even have to open/close the file yourself), or ioutil.ReadAll() which takes a io.Reader and returns a slice of bytes containing the whole file. Here's the doc on ioutil.
Finally, as others mentioned, you can google about the Go language using "golang" and you should find all you need. The golang-nuts mailing list is also a great place to look for answers (make sure to search first before posting, a lot of stuff has already been answered). To look for third-party packages, check the godoc.org website.
HTH
This is what I use to read an entire binary file into memory
func RetrieveROM(filename string) ([]byte, error) {
file, err := os.Open(filename)
if err != nil {
return nil, err
}
defer file.Close()
stats, statsErr := file.Stat()
if statsErr != nil {
return nil, statsErr
}
var size int64 = stats.Size()
bytes := make([]byte, size)
bufr := bufio.NewReader(file)
_,err = bufr.Read(bytes)
return bytes, err
}
For example, to count the number of zero bytes in a file:
package main
import (
"fmt"
"io"
"os"
)
func main() {
f, err := os.Open("filename")
if err != nil {
fmt.Println(err)
return
}
defer f.Close()
data := make([]byte, 4096)
zeroes := 0
for {
data = data[:cap(data)]
n, err := f.Read(data)
if err != nil {
if err == io.EOF {
break
}
fmt.Println(err)
return
}
data = data[:n]
for _, b := range data {
if b == 0 {
zeroes++
}
}
}
fmt.Println("zeroes:", zeroes)
}
You can't whimsically cast primitive types to (char*) like in C, so for any sort of (de)serializing of binary data use the encoding/binary package.
http://golang.org/pkg/encoding/binary .
I can't improve on the examples there.
Here is an example using Read method:
package main
import (
"io"
"os"
)
func main() {
f, e := os.Open("a.go")
if e != nil {
panic(e)
}
defer f.Close()
for {
b := make([]byte, 10)
_, e = f.Read(b)
if e == io.EOF {
break
} else if e != nil {
panic(e)
}
// do something here
}
}
https://golang.org/pkg/os#File.Read