I am try to upload AsyncRead trait to s3, but I can not figure out how to convert the AsyncRead object to ByteStream which the library (aws-sdk-s3) know how to work with.
The signature of my function is like this:
async fn upload(&self, key: String, data: &mut impl AsyncRead) -> Result<(),Error>;
let byte_stream = codec::FramedRead::new(read, codec::BytesCodec::new()).map(|r| r.freeze());
let res = s3
.put_object()
.key(id)
.body(ByteStream::new(byte_stream))
.send()
.await;
The errors that I got are:
the trait bound &impl 'async_trait + AsyncRead: AsyncRead is not satisfied
FramedRead<&impl 'async_trait + AsyncRead, BytesCodec> is not an iterator
I found this and tried to implement it like the "good way" section but I got compile errors.
I also looked at the documentation and found nothing that can help with the conversion (found only the other way around).
How can I upload an AsyncRead object to s3?
Related
I am developing an API using the tower_web framework, and for some API calls, it would be best to just return bytes with a content type like "application/octet-stream". This data is not directly read from a file, but generated dynamically, so I can't refer to the example code for serving a static file.
For implementing the desired functionality, I tried using a http::Response<Vec<u8>> with a body using a u8 array. Here are the relevant snippets of my code, for testing purposes I used a u8 array filled with zeroes, that would later be the data I want to send to the client:
extern crate tower_web;
use tower_web::ServiceBuilder;
use http::Response as HttpResponse;
const SERVER_ADDRESS : &str = "127.0.0.1:8080";
[...]
#[derive(Clone, Debug)]
pub struct HttpApi;
impl_web! {
impl HttpApi {
[...]
#[get("/test")]
#[content_type("application/octet-stream")]
fn test(&self) -> http::Response<Vec<u8>> {
let response = HttpResponse::builder().status(200).body([0u8;16]);
println!("{:?}", response);
response
}
}
}
pub fn main() {
let addr = SERVER_ADDRESS.parse().expect("Invalid address");
println!("Listening on http://{}", addr);
ServiceBuilder::new()
.resource(HttpApi)
.run(&addr)
.unwrap();
}
However, when I try to build it using cargo, I get this error:
the trait `tower_web::codegen::futures::Future` is not implemented for `http::Response<Vec<u8>>`
I also tried setting the type returned by the function to a Result wrapping the http::Response<Vec<u8>> and (), as it would work that way when I try returning strings or json. This gave me the error
the trait `BufStream` is not implemented for `Vec<u8>`
Is there any simple way to solve this, or any other approach for returning bytes as the body of a HTTP Response in Tower Web?
apologies if this is a dumb question, I am new enough to using rust and i am having a hard time figuring out how to read an object from s3 using the rusoto library.
https://rusoto.github.io/rusoto/rusoto_s3/
so far i've worked out this much:
let mut get_object_request = rusoto_s3::GetObjectRequest::default();
get_object_request.bucket = bucket.to_owned();
get_object_request.key = object_id.to_owned();
let resp = client.get_object(get_object_request)
.await
.map_err(|_| {
FailedToParseFile(format!(
"Failed to retrieve file from s3, bucket: {}, object id: {}",
bucket, object_id
))
})?;
let mut resp_body = match resp.body {
Some(body) => {body}
None => { return Err(ClientError::FailedToObtainConfiguredFile(
format!("Failed to retrieve the file: {} in {}", object_id, bucket)
))}
};
However i've no idea how to turn the streaming body returned from this into a usable string. I've tried a few things to get this working but none seem to work for me.
let mut byte_buffer = String::new();
resp_body.into_blocking_read()
.read_to_string(&mut byte_buffer)
.map_err(|_| {
FailedToParseFile(format!(
"Failed to parse the received file, bucket: {}, object id: {}",
bucket, object_id
))
})?;
let resp_body_bytes = byte_buffer.into_bytes();
As i am using tokio (rocket), this blocking queue doesn't seem to be an option for me causing it to panic on attempting to block on async thread. I started looking at the method 'into_async_read()' rather than blocking but i'm unfamiliar with this too and seem to be struggling using it as intended, this is starting to seem pretty convoluted maybe i'm missing something, any help with this would be appreciated.
let mut byte_buffer = String::new();
Pin::new(&mut resp_body.into_async_read())
.read_to_string(&mut byte_buffer)
.await;
This actually seems to do what I need it to, messing about with this I didn't realise the result of .read_to_string(&mut byte_buffer) could be awaited in order to fill the supplied buffer.
I'm new to Kotlin/Native! I'd like to create an array of CPointers to pass it around, but having hard time creating one.
In C/C++ void* a[] = {test} is enough. But I'm not able to do the same in K/N.
I've tried val a: CValuesRef<out COpaquePointerVar> = cValuesOf(test)
But it results in the following error:
Tried looking at docs and find it on the web, but none of them answered correctly.
Any help is appreciated!!
So I basically did what I wanted using a StableRef
on_exit(staticCFunction { _, argsPtr ->
val argsStableRef = argsPtr!!.asStableRef<List<COpaquePointer>>()
val args = argsStableRef.get()
// Cleanup code
argsStableRef.dispose()
}, StableRef.create(listOf(/* All the pointers */)).asCPointer())
Basically transforming the List<COpaquePointer> to a StableRef and extracting pointer out of it, then when required deref it via the asStableRef and after that dispose it to ensure memory have been freed.
I've been toying around with Rust and have come across the following code:
fn request(&url) -> Result<(), Box<dyn std::error::Error>> {
let mut res = reqwest::get(&url)?;
let mut body = String::new();
res.read_to_string(&mut body)?;
println!("Status: {}", res.status());
println!("Headers:\n{:#?}", res.headers());
println!("Body:\n{}", body);
Ok(())
}
It is my understanding that:
fn request(&url) -> Result<(), Box<dyn std::error::Error>> {
Defines a function that has a single (borrowed) parameter and uses Result to handle errors.
let mut res = reqwest::get(&url)?;
Defines a mutable variable to store the response object from the reqwest crate's get method.
let mut body = String::new();
Defines a mutable variable to store the responseText string.
res.read_to_string(&mut body)?;
This method stores the responseText in the body variable.
println!("Status: {}", res.status());
println!("Headers:\n{:#?}", res.headers());
println!("Body:\n{}", body);
Prints three formatted strings (with trailing new lines) containing the response status, headers and body.
Ok(())
Handles errors via Result..?
Questions:
What do the empty parenthesis in Result<() and OK(()) mean/do?
What is Box<dyn std::error::Error>?
You're absolutely correct in your understanding.
A Result is an Enum which can either be "Ok" or "Err" - if Ok, then there can be some value of okayness (a result, response, data, output, whatever); similarly, if Err, then there's some concrete error you may want to communicate. With that let's break down the result.
The should be read like this: Result<TypeOfValueIfOkay, TypeOfErrorWhenNotOkay>. These two sub-types can be anything, but they have to be something - can't just ignore it.
So if TypeOfValueIfOkay has to be something, but if you don't want to return something, you can return an empty Tuple. That's the () in Result. It's just efficiently saying "I return nothing at all when everything goes well".
So then the second part TypeOfErrorWhenNotOkay can also just be any type - a string, an int, whatever. It helps for the type to implement the std::error::Error trait helping callers standardize a bit.
Returning "some dynamic object but that implements trait std::error::Error" requires Rust to know the exact size of this value if it is to return it on the caller's stack (the caller's stack needs to be sized to accept it.)
This is where the Box type comes in - it pushes the actual value onto the heap and holds a pointer to it (which can be of predictable fixed size no matter the actual value on the heap.) The <dyn std::error::Error> is an assurance that whatever the boxed value is, it implements the Error trait.
So now the final Ok(()) makes sense. If you read Ok(value): it says the Result enum is variant Ok with the value of "empty tuple" (), i.e. nothing.
I have written the following function that reads the contents of a text file and panic!s if an error is encountered.
fn get_file_contents(name: String) -> Result<String, io::Error> {
let mut f = try!(File::open(name));
let mut contents = String::new();
try!(f.read_to_string(&mut contents));
Ok(contents)
}
And the contents are extracted from the Result using:
let file_contents = match get_file_contents(file_name) {
Ok(contents) => contents,
Err(err) => panic!("{}", err)
};
I am now trying to reimplement this in an object oriented manner using structures and implementations. I created the following structure:
struct FileReader {
file_name: String,
file_contents: String,
}
and implemented the following methods:
impl FileReader {
fn new(fname: &str) -> FileReader {
FileReader {
file_name: fname.to_string(),
file_contents: String::new(),
}
}
fn get_file_contents(&mut self) {
let mut f = match File::open(&self.file_name) {
Ok(file) => file,
Err(err) => panic!("{}", err)
};
match f.read_to_string(&mut self.file_contents) {
Ok(size) => size,
Err(err) => panic!("{}", err)
};
}
}
In the OO approach, I haven't used the try! macro as I don't want the method to return any value. Is my OO implementation of get_file_contents a typical way of achieving this functionality? If not, can you please suggest an alternative way?
In the OO approach, I haven't used the try! macro as I don't want the method to return any value.
It's unclear why you think that "object oriented" means "doesn't return a value". If an error can occur, the code should indicate that.
Many languages have the equivalent of exceptions — out of band values that are thrown (also known as "returned") from a function or method. Note that this means that these languages allow for two disjoint types to be returned from a given function: the "normal" type and the "exceptional" type. That is a close equivalent for Rust's Result: Result<NormalType, ExceptionalType>.
Exceptional isn't a great term for this, as you should expect that opening a file should fail. There's an infinite number of ways that it could not work, but only a narrow subset of ways that it can succeed.
Panicking is closer to "kill the entire program / thread right now". Unlike C, you are forced to either deal with a problem, pass it back to the caller, or kill the program (panic).
If you would have thrown an exception in a language that supports them, use a Result. If you would have killed the program, or don't want to handle an error, use a panic.
If you want to panic in your particular case, use unwrap, or even better, expect:
fn get_file_contents(&mut self) {
let mut f = File::open(&self.file_name).expect("Couldn't open file");
f.read_to_string(&mut self.file_contents).expect("Couldn't read file");
}
seems kind of clunky to have to deal with the Result for each method.
Which is why the Error Handling section of The Rust Programming Language spends a good amount of time discussing the try! macro:
A cornerstone of error handling in Rust is the try! macro. The try! macro abstracts case analysis like combinators, but unlike combinators, it also abstracts control flow. Namely, it can abstract the early return pattern seen above.
(this makes more sense in context of the page)
I don't want my code to try and recover from the error (most likely caused by the file not being found) - I want it to print a useful error message and then die
Then by all means, panic. There's more succinct AND more detailed ways to do it (as shown above).