I'm rather new to both golang and encoding.com and I'm trying to use the encoding.com API wrapper to transcode a simple video file, but I'm rather confused by the format to use.
When looking at the tests I can see how to call the AddMedia function (https://github.com/nytimes/encoding-wrapper/blob/master/encodingcom/media_test.go#L9-L39) but unfortunately it doesn't work for me.
package main
import ("github.com/NYTimes/encoding-wrapper/encodingcom")
func main() {
client, err := encodingcom.NewClient("https://manage.encoding.com", "123", "key")
format := encodingcom.Format{
Output: []string{"https://key:secret#bucket.s3.amazonaws.com/aladin.ogg"},
VideoCodec: "libtheora",
AudioCodec: "libvorbis",
Bitrate: "400k",
AudioBitrate: "64k",
}
addMediaResponse, err := client.AddMedia([]string{"https://samples.mplayerhq.hu/h264/Aladin.mpg"},
[]encodingcom.Format{format}, "us-east-1")
}
}
The error "raised" is
APIError.Errors.Errors0: Output format 'https://key:secret#bucket.s3.amazonaws.com/aladin.aac' is not allowed! (format #0)
APIError.Message:
and I really don't get it, the Output element in the Format looks missplaced, am I reading the test wrong? Using the API builder the format parameter should receive only the format, for example "ogg", and there's a "destination" parameter for S3. It also doesn't specify if the url must be urlencoded, but honestly I don't think so. Still keys and secrets can contain for example the char '/'
Any more experienced gopher?
Related
The following code tries to select some data from a file stored on S3:
let client = S3Client::new(Region::default());
let source = ... object providing bucket and key ...;
let r = SelectObjectContentRequest {
bucket: source.bucket,
key: source.key,
expression: "select id from S3Object[*].id".to_string(),
expression_type: "SQL".to_string(),
input_serialization: InputSerialization {
json: Some(JSONInput { type_: Some("LINES".to_string()) }),
..Default::default()
},
output_serialization: OutputSerialization {
json: Some(JSONOutput { record_delimiter: Some("\n".to_string()) }),
..Default::default()
},
..Default::default()
};
It causes the following error:
The specified method is not allowed against this
resource.POST
The example is a 1:1 port of a working Python/boto3 example, so I'm quite sure it should work. I found this issue, which is a few month old and the status is not clear to me. How do I get this working with Rust?
Unfortunately s3 select still doesn't work on the latest rusoto_s3-0.40.0. The issue you linked has all the answer. The problems are twofold.
First, right now the s3 select request rusoto sends out has a bogus query string. It should be /ObjectName?select&select-type=2, but rusoto encodes it to be /bjectName?select%26select-type=2. That's the error you saw.
To verify, run your project like so:
$ RUST_LOG=rusoto,hyper=debug cargo run
You will see logs from rusoto and hyper. Sure enough it emits an incorrect URI. One can even dig into the code responsible:
let mut params = Params::new();
params.put("select&select-type", "2");
request.set_params(params);
It is supposed to be:
let mut params = Params::new();
params.put("select-type", "2");
params.put("select", "");
request.set_params(params);
Although the fix seems trivial, remember these are glue code generated from AWS botocore service manifests, not manually coded. To incorporate the fix is not that straightforward.
Second, the bigger problem. The AWS s3 select response uses a customized binary format. rusoto simply doesn't have a deserializer for that yet.
I'm newer to the Go language and it's resources, but have been looking around for quite some time without any luck of finding what I'm looking for. So if there is a resource out there for it, I apologize for the duplicate question and would appreciate being directed that way.
My goal is simply to build a web scraper. I'm using chromedp, which has features to focus on elements, fill in text, etc. I want to create a test environment/server to test it with during development. The main reason being that I do not want to constantly create GET requests from a website (out of common courtesy), but also be able to work offline and in addition it should also make testing a little faster. I stumbled across the go-vcr library and have been trying to get that to work, but to no avail. I can get it to record and create a .yaml, but I can't figure out how to test beyond the raw html that bounces back and gets stored in the .yaml file. My understanding is that it's possible to replicate the website and functionality of it using the library, but I'm unable to piece together how to do that.
Is what I'm trying to do possible, or is the go-vcr library (or any test/fake server for that matter) only capable of returning static data, therefore rendering anything I want to test with the web scraper not possible?
I haven't posted any code simply because I haven't pieced together much more than the examples given from the repository for the go-vcr.
I hope I was able to explain that in a way that made sense. If not I'd be happy to answer questions to clarify.
Update: Adding the example code for sake of ease. I understand how this part of it works (I think) and I can use it for testing whether or not I grabbed the proper elements of a static page, but ideally (as an example) I want to be able to fill in a text box with my program and test whether or not I successfully found the text box and filled it in without hitting the live webpage.
package vcr_test
import (
"io/ioutil"
"net/http"
"strings"
"testing"
"github.com/dnaeon/go-vcr/recorder"
)
func TestSimple(t *testing.T) {
// Start our recorder
r, err := recorder.New("fixtures/golang-org")
if err != nil {
t.Fatal(err)
}
defer r.Stop() // Make sure recorder is stopped once done with it
// Create an HTTP client and inject our transport
client := &http.Client{
Transport: r, // Inject as transport!
}
url := "http://golang.org/"
resp, err := client.Get(url)
if err != nil {
t.Fatalf("Failed to get url %s: %s", url, err)
}
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
t.Fatalf("Failed to read response body: %s", err)
}
wantTitle := "<title>The Go Programming Language</title>"
bodyContent := string(body)
if !strings.Contains(bodyContent, wantTitle) {
t.Errorf("Title %s not found in response", wantTitle)
}
}
I've read the various bits of literature, and I'm seeing the same problem that the questioner in
https://stackoverflow.com/a/25636911
was seeing.
My code looks like this:
coll = db.collection('foobar');
bulk = coll.initializeUnorderedBulkOp();
for entry in messages {
bulk.insert(entry);
}
bulk.execute(function (err, result) {
if (err) throw err
inserted += result.nInserted
});
bulk is an object
bulk.insert works just fine
bulk.execute is undefined
The answer in the stackoverflow question said, "only the callback flavor of db.collection() works, so I tried:
db.collection('foobar', function (err, coll) {
logger.debug "got here"
if (err) throw err
bulk = coll.initializeUnorderedBulkOp()
... same code as before
We never get to "got here" implying that the "callback flavor" of db.collection() was dropped for 3.0?
Unfortunately, my python is way better than my JS prototyping skills, so looking at the skin source code doesn't make any sense to me.
What is the right way, with mongoskin 2.1.0 and the 2.2.0 mongodb JS driver, to do a bulk operation, or is this not implemented at all anymore?
There are at least two answers:
(1) Use insert, but the array form, so you insert multiple documents with one call. Works like a charm.
(2) If you really need bulk operations, you'll need to switch from mongoskin to the native mongo interface, but just for that one call.
This kinda sucks because it's using a private interface in mongoskin, but it's also the most efficient way to stick with mongoskin:
(example in coffeescript)
// bulk write all the messages in "messages" to a collection
// and insert the server's current time in the recorded field of
// each message
// use the _native interface and wait for callback to get collection
db._native.collection collectionName, (err, collection) ->
bulk = collection.initializeUnorderedBulkOp()
for message in messages
bulk.find
_id: message._id
.upsert().updateOne
$set: message
$currentDate:
recorded: true
bulk.execute (err, result) ->
// ... error and result checking code
or (3) if you want to implement that $currentDate and not any generic bulk operation, refer to solution (1) but use the not-very-well-documented BSON object Timestamp() with no arguments:
for msg in messages:
msg.recorded = Timestamp()
db.mycollection.insert(msg)
which will do a bulk insert and set timestamp to the DB server's time at the time the record is written to the db.
Fairly new to node-webkit, so I'm still figuring out how everything works...
I have some logging in my app:
console.log("Name: %s", this.name);
It outputs to the browser console as expected:
Name: Foo
But in the invoking terminal, instead I get some fairly ugly output:
[7781:1115/085317:INFO:CONSOLE(43)] ""Name: %s" "Foo"", source: /file/path/to/test.js (43)
The numerical output within the brackets might be useful, but I don't know how to interpret it. The source info is fine. But I'd really like the printed string to be printf-style formatted, rather than shown as individual arguments.
So, is there a way to get stdout to be formatted either differently, or to call a custom output function of some sort so I can output the information I want?
I eventually gave up, and wrapped console.log() with:
log = function() {
console.log(util.format.apply(this, arguments));
}
The actual terminal console output is done via RenderFrameHostImpl::OnAddMessageToConsole in chromium, with the prefix info being generated via LogMessage::Init() in the format:
[pid:MMDD/HHMMSS:severity:filename(line)]
The javascript console.log is implemented in console.cc, via the Log() function. The printf style formatting is being done at a higher level, so that by the time the Log() function (or similar) are called, they are only passed a single string.
It's not a satisfying answer, but a tolerable workaround.
I was looking to create a command line interface to go along side my UI and had a similar problem. Along with not logging the values as I wanted I also wanted to get rid of the [pid:MMDD/HHMMSS:severity:filename(line)] output prefix so I added the following:
console.log = function (d) {
process.stdout.write(d + '\n');
};
so that console logging was set back to stdout without extra details. Unfortunately also a work around.
I'm learning go and trying to understand how to get more detailed error information out of the generic error type. The example I'll use is from the net package, specifically the DialTimeout function.
The signature is
func DialTimeout(network, address string, timeout time.Duration) (Conn, error)
The error type only defines an Error() string function. If I want to find out exactly why DialTimeout failed, how can I get that information? I found out that I can use type assertion to get the net.Error specific error:
con, err := net.DialTimeout("tcp", net.JoinHostPort(address, "22"),
time.Duration(5) * time.Second)
if err != nil {
netErr, ok := err.(net.Error)
if ok && netErr.Timeout() {
// ...
}
}
but that only tells me whether or not I had a timeout. For example, say I wanted to differentiate between a refused connection and no route to host. How can I do that?
Maybe DialTimeout is too high-level to give me that kind of detail, but even looking at syscall.Connect, I don't see how to get the specific error. It just says it returns the generic error type. Compare that to the Posix connect, which will let me know why it failed with the various return codes.
My general question is: how am I supposed to pull out error details from the generic error type if the golang docs don't tell me what type of errors may be returned?
Most networking operations return a *OpError which holds detailed information about the error
and implements the net.Error interface. So for most use cases it is sufficient to use net.Error
as you already did.
But for your case you'd want to assert the returned error to be *net.OpError and
use the internal error:
if err != nil {
if oerr, ok := err.(*OpError); ok {
// Do something with oerr.Err
}
}
As soon as you're doing this you are in the land of platform dependency as syscalls under Linux
can fail differently to those under Windows. For Linux you'd do something like this:
if oerr.Err == syscall.ECONNREFUSED {
// Connection was refused :(
}
The syscall package contains the important error constants for your platform. Unfortunately
the golang website only shows the syscall package for Linux amd64. See here for ECONNREFUSED.
Finding out types
The next time you're wondering what is actually returned by some function and you can't make
heads and tails of it, try using the %#v format specified in fmt.Printf (and friends):
fmt.Printf("%#v\n", err)
// &net.OpError{Op:"dial", Net:"tcp", Addr:(*net.TCPAddr)(0xc20006d390), Err:0x6f}
It will print detailed type information and is generally quite helpful.