I want to send some queries to SQL in Rust. To achieve this I'm using the crate sqlx. I send queries to my database file from separate coroutines. And the code works okay if the amount of coroutines is about 1000. If there's more, there's an error
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Database(SqliteError { code: 5, message: "database is locked" })', src\main.rs:56:43
I understand that 10 thousand coroutines is too much for one file, but I expect a corouting to wait until the database is unlocked rather than return an error.
Here's my code
use std::error::Error;
use tokio::task::JoinHandle;
use sqlx::sqlite::SqlitePool;
async fn routine(pool: &SqlitePool, age: i32, id: i32) -> Result<String, Box<dyn Error>> {
if age % 2 == 0 {
sqlx::query!("UPDATE users SET salary = salary + 1 WHERE id = ?1", id).execute(pool).await?;
let row = sqlx::query!(r#"SELECT salary FROM users WHERE id = ?1"#, id)
.fetch_one(pool)
.await?;
Ok(format!("{}: {}", id, row.salary))
} else {
let res = sqlx::query!(r#"SELECT salary FROM users WHERE age = ?1 ORDER BY id"#, age)
.fetch_all(pool)
.await?
.iter()
.map(|it| it.salary.to_string())
.collect::<Vec<String>>()
.join(", ");
Ok(res)
}
}
#[tokio::main]
async fn main() {
let n = std::env::args().nth(1).unwrap().parse().unwrap(); // drops if n is 10_000
let pool = SqlitePool::connect("sqlite://database.db").await.unwrap();
let handles: Vec<JoinHandle<_>> = (1..n).map(|_| {
let pool = pool.clone();
tokio::spawn(async move {
routine(&pool, 0, 0).await.unwrap()
})
}).collect();
for handle in handles {
handle.await.unwrap();
}
}
Could you please explain how to wait and not to get errors?
Related
I have some tests which have some variables that hold some important data and I'd like to print their data when an assertion fails. Getting the data I need consumes the variables, so the printing code must own the variables. In this example, I'd want to call dump_foo_data once an assertion fails:
struct Foo();
fn dump_foo_data(f: Foo) {
eprintln!("Behold, Foo data: ");
}
#[test]
fn my_test() {
let f = Foo();
eprintln!("begin");
// do a test
&f;
let success = true;
assert!(success);
// do another test
&f;
let success = false;
assert!(success);
}
I can make a very bad solution by making dump_foo_data non-returning and panic:
fn dump_foo_data(f: Foo) -> ! {
eprintln!("Behold, Foo data: ");
panic!();
}
Then instead of using assert!, I check the failure with an if and maybe call dump_foo_data:
let success = true;
if !success {
dump_foo_data(f);
}
This is too many lines of code, and I need to specify f. In reality, I have more than one variable like f that I need to dump data from, so it's not very nice to list out single relevant local variable in every check.
I couldn't figure out how to write a macro to make this better because I'd still need to pass every relevant local variable to the macro.
I couldn't think of a way to use std::panic either. update_hook would need to take ownership of f, then I couldn't use it in tests.
Is there any good way to do this in Rust?
Edit: I've thought of another approach: put each relevant local in an Rc then pass each of those to std::panic::update_hook. I've not confirmed whether this'll work yet.
Edit 2: Maybe I could abuse break to do what I explained with goto in a comment.
One way that doesn't use any macro or shared-interior-mutability-reference magic might be to repossess f:
fn check_or_dump(success: bool, f: Foo) -> Foo {
match success {
true => f,
false => panic!("Behold foo data: {:?}", dump_foo_data(f)),
}
}
You use it like this:
let f = Foo();
let success = true;
let f = check_or_dump(success, f);
let success = false;
let f = check_or_dump(success, f);
// and so on.
Here's a solution without macro or interior mutability and that doesn't require you to list all the variables on each check. It is inspired by this answer:
struct Foo();
fn dump_foo_data(_f: Foo) {
eprintln!("Behold, Foo data: ");
}
#[test]
fn my_test() {
let f = Foo();
let doit = || -> Option<()> {
eprintln!("begin");
// do a test
&f;
let success = true;
success.then_some(())?;
// do another test
&f;
let success = false;
success.then_some(())?;
Some(())
};
if let None = doit() {
dump_foo_data (f);
panic!("Test failure");
}
}
Playground
I've worked out a solution using the panic handler:
use std::rc::Rc;
use std::cell::{Cell, RefCell};
use std::panic::PanicInfo;
thread_local! {
static TL_PANIC_TARGETS: RefCell<Vec<Rc<dyn PanicTrigger>>> = RefCell::new(vec![]);
}
pub trait PanicTrigger {
fn panic_trigger(self: Rc<Self>);
}
pub fn register_panic_trigger<P: PanicTrigger + 'static>(p: Rc<P>) {
TL_PANIC_TARGETS.with(|v: _| {
v.borrow_mut().push(p.clone());
});
}
#[ctor::ctor]
fn set_panic_hook() {
let old_hook = std::panic::take_hook();
std::panic::set_hook(Box::new(move |pi: &PanicInfo| {
run_panic_triggers(pi);
old_hook(pi);
}));
}
fn run_panic_triggers(_: &PanicInfo) {
TL_PANIC_TARGETS.with(|v: _| {
for pt in v.take() {
pt.panic_trigger();
}
});
}
struct Foo();
fn dump_foo_data(_f: Foo) {
eprintln!("Behold, Foo data: ");
}
impl PanicTrigger for Cell<Option<Foo>> {
fn panic_trigger(self: Rc<Self>) {
if let Some(f) = self.take() {
dump_foo_data(f);
}
}
}
#[test]
fn my_test() {
let f = Rc::new(Cell::new(Some(Foo())));
register_panic_trigger(f.clone());
let success = true;
assert!(success);
let success = false;
assert!(success);
}
fn main() { }
Basically, you put the relevant data in an Rc and keep a local reference and put one in TLS for the panic handler. You need to put it in an Option in a Cell so that you can move out of it.
Types that don't need to be owned to print relevant data can be registered too, and you don't need to implement PanicTrigger on a Cell<Option<T>>, just T.
This is thread-safe.
Because the data is so wrapped up, it's harder to manipulate in the test body. But now you can use normal assert!. It's a trade-off.
I've been spending way too much time trying to solve this problem. So the code that I posted below does work in terms of downloading a file, but the problem is, the flow has a very unexpected behaviour. The response.content.readAvailable() method call seems to block until it's completely done downloading the whole file at which point the emit progress happens, so you end up waiting a long time for the file to download, and then in a split second you get all of the progress updates. So I'm wondering if there is a way to do this where I read in a certain number of bytes at a time and then emit a progress and then repeat until the file is done downloading? Or maybe a way to hook into the readAvailable() method and update the progress that way? Any help with this would be greatly appreciated.
Here's the code I found and modified, but still does not work right:
suspend fun HttpClient.downloadFile(
output: File,
downloadUrl: String,
md5Hash: String,
) = flow {
try {
val response = get<HttpResponse> { url(downloadUrl) }
val data = ByteArray(response.contentLength()?.toInt() ?: 0)
val contentLn = response.contentLength()?.toInt() ?: 0
var offset = 0
var bytesRemaining = contentLn
do {
val chunkSize = min(maxChunkSize, bytesRemaining)
logger?.d { "Read Available:" }
val result = response.content.readAvailable(data, offset, length = chunkSize)
val progress = ((offset / contentLn.toDouble()) * 100).toInt()
emit(DownloadResult.Progress(progress))
logger?.d { "logged progress: $progress" }
// delay(6000L) this was to test my assumption that the readAvalible was blocking.
offset += chunkSize
bytesRemaining -= chunkSize
} while (result != -1)
if (response.status.isSuccess()) {
if (data.md5().hex == md5Hash) {
output.write(data)
emit(DownloadResult.Success)
} else {
emit(DownloadResult.ErrorCorruptFile)
}
} else {
emit(DownloadResult.ErrorBadResponseCode(response.status.value))
}
} catch (e: TimeoutCancellationException) {
emit(DownloadResult.ErrorRequestTimeout("Connection timed out", e))
}
}
Finally after a stupid amount of time I solved this. What you need to use is this. That gives you access to the byte channel as it is downloading.
and a very crude implementation (that I'm not yet done with) is this:
get<HttpStatement>(url = downloadUrl).execute {
var offset = 0
val byteBufferSize = 1024 * 100
val channel = it.receive<ByteReadChannel>()
val contentLen = it.contentLength()?.toInt() ?: 0
val data = ByteArray(contentLen)
do {
val currentRead = channel.readAvailable(data, offset, byteBufferSize)
val progress = if(contentLen == 0) 0 else ( offset / contentLen.toDouble() ) * 100
logger?.d { "progress: $progress" }
offset += currentRead
} while (currentRead >= 0)
}
two things to not with this solution. 1.) I'm in the context of HttpClient, so that's how I have access to get(). 2.) I'm creating a byte buffer size of 1024 * 100 in order to not let the readAvailable method block for too long, though this might not be necessary... the one nice thing about it is that it determines how frequently you will be publishing your progress updates.
I've been spending way too much time trying to solve this problem. So the code that I posted below does work in terms of downloading a file, but the problem is, the flow has a very unexpected behaviour. The response.content.readAvailable() method call seems to block until it's completely done downloading the whole file at which point the emit progress happens, so you end up waiting a long time for the file to download, and then in a split second you get all of the progress updates. So I'm wondering if there is a way to do this where I read in a certain number of bytes at a time and then emit a progress and then repeat until the file is done downloading? Or maybe a way to hook into the readAvailable() method and update the progress that way? Any help with this would be greatly appreciated.
Here's the code I found and modified, but still does not work right:
suspend fun HttpClient.downloadFile(
output: File,
downloadUrl: String,
md5Hash: String,
) = flow {
try {
val response = get<HttpResponse> { url(downloadUrl) }
val data = ByteArray(response.contentLength()?.toInt() ?: 0)
val contentLn = response.contentLength()?.toInt() ?: 0
var offset = 0
var bytesRemaining = contentLn
do {
val chunkSize = min(maxChunkSize, bytesRemaining)
logger?.d { "Read Available:" }
val result = response.content.readAvailable(data, offset, length = chunkSize)
val progress = ((offset / contentLn.toDouble()) * 100).toInt()
emit(DownloadResult.Progress(progress))
logger?.d { "logged progress: $progress" }
// delay(6000L) this was to test my assumption that the readAvalible was blocking.
offset += chunkSize
bytesRemaining -= chunkSize
} while (result != -1)
if (response.status.isSuccess()) {
if (data.md5().hex == md5Hash) {
output.write(data)
emit(DownloadResult.Success)
} else {
emit(DownloadResult.ErrorCorruptFile)
}
} else {
emit(DownloadResult.ErrorBadResponseCode(response.status.value))
}
} catch (e: TimeoutCancellationException) {
emit(DownloadResult.ErrorRequestTimeout("Connection timed out", e))
}
}
Finally after a stupid amount of time I solved this. What you need to use is this. That gives you access to the byte channel as it is downloading.
and a very crude implementation (that I'm not yet done with) is this:
get<HttpStatement>(url = downloadUrl).execute {
var offset = 0
val byteBufferSize = 1024 * 100
val channel = it.receive<ByteReadChannel>()
val contentLen = it.contentLength()?.toInt() ?: 0
val data = ByteArray(contentLen)
do {
val currentRead = channel.readAvailable(data, offset, byteBufferSize)
val progress = if(contentLen == 0) 0 else ( offset / contentLen.toDouble() ) * 100
logger?.d { "progress: $progress" }
offset += currentRead
} while (currentRead >= 0)
}
two things to not with this solution. 1.) I'm in the context of HttpClient, so that's how I have access to get(). 2.) I'm creating a byte buffer size of 1024 * 100 in order to not let the readAvailable method block for too long, though this might not be necessary... the one nice thing about it is that it determines how frequently you will be publishing your progress updates.
I want to return the Result as shown in below from the for loop. Please help which would be the best way solve this error. I tried the pattern matching with returning None which works. But I need to return Error.
pub fn get_account(&self) -> Result<Keys, Error> {
//PATH is default home directory
let values = match load_json_file(PATH + "/keys.json") {
Ok(account) => Ok(account),
Err(e) => {
return Err(Error::Invalid_Tx(
"The sender address cannot be nil".to_owned(),
))
}
};
let accounts: Vec<Keys> = values.unwrap();
let sender_address = self.sender.unwrap();
for acc in accounts {
if acc.address == sender_address {
return Ok(acc);
};
};
Ok(())
}
expected struct commands::key::Keys, found ()rustc(E0308)
You are trying to return two different types from the same function:
line 15: Ok(acc) is of type Result<Keys, Error>
line 18: Ok(()) has type Result<(), Error>
If "no result" is a valid return value, then you can change the function signature to:
pub fn get_account(&self) -> Result<Option<Keys>, Error>;
And then modify those return values to be Ok(Some(acc)) and Ok(None) respectively.
If "no result" is an error then you need to modify the Error type to include this variant. For example:
enum Error {
Invalid_Tx(String),
NotFound,
}
And return Err(Error::NotFound) at the end.
You can also tidy this function up a lot, by using thiserror, which is a popular crate for defining error types:
use thiserror::Error; // thiserror = "1.0.21"
#[derive(Debug, Error)]
enum Error {
#[error("The sender address cannot be nil")]
InvalidTx,
#[error("The key was not found")]
NotFound,
}
pub fn get_account(&self) -> Result<Keys, Error> {
let accounts: Vec<Keys> = load_json_file(PATH + "/keys.json")
.map_err(|_| Error::InvalidTx)?;
let sender_address = self.sender.unwrap();
accounts
.into_iter()
.find(|acc| acc.address == sender_address)
.ok_or(Error::NotFound)
}
This is better because the Strings in the errors do not need to be allocated, but they are still available as static string slices if they are needed for display. I also got rid of the for loop altogether, which makes the function shorter and cleaner.
I need to realize TCP server, that stores a variable M. At the beginning, M = 1.
But, when the conntection set between server and client, and client sends another value of variable N, server must do next: M = M * N. And return that value to client.
And server must save that new value of variable M!
And, when next new client set connection, it will work with new value of variable.
Example:
Server: M = 1; Client: N = 5; Server: M = 5;
Server: A = 5; Client: N = 8; Server: M = 40;
This is my code of my server. And it works in that way:(not saving new value)
Example:
Server: M = 1; Client: N = 5; Server: M = 5;
Server: A = 1; Client: N = 8; Server: M = 8;
Maybe should I do a global variable? Or something about it... Give me some advice. Thank you.
fn handle_client(mut stream: TcpStream, a:&mut i32) {
let mut data = [0 as u8; 30]; // using 30 byte buffer
while match stream.read(&mut data) {
Ok(size) => {
if size>0{
let temp = str::from_utf8(&data[0..size]).unwrap().to_string();
let temp: i32 = temp.trim().parse().unwrap();
*a = *a * temp;
let st = a.to_string();
// String в u8
let data = st.as_bytes();
stream.write(&data).unwrap();
}
true
},
Err(_) => {
println!("An error occurred, terminating connection with {}", stream.peer_addr().unwrap());
stream.shutdown(Shutdown::Both).unwrap();
false
}
} {}
}
fn main() {
let listener = TcpListener::bind("0.0.0.0:7956").unwrap();
let mut a: i32 = 1;
// accept connections and process them, spawning a new thread for each one
println!("Server listening on port 7956");
println!("A = 1");
for stream in listener.incoming() {
match stream {
Ok(stream) => {
println!("New connection: {}", stream.peer_addr().unwrap());
thread::spawn(move|| {
// connection succeeded
handle_client(stream, &mut a)
});
}
Err(e) => {
println!("Error: {}", e);
/* connection failed */
}
}
}
// close the socket server
drop(listener);
}
Why doesn't your code work?
Because you move a into the closure, thus copying it. In order to se that, you just have to ask: What do you need to move into the closure, such that &mut a is of type &mut i32?
Exactly, a "raw" i32. How does the compiler do that? By copying it. That means changes to the variable "inside" the closure will not be reflected "outside".
Well, then how to fix it?
Let's start from the beginning: If you simply don't borrow in the call, you will move a mutable reference into the closure - that doesn't work though, due to two reasons: Firstly, it is obviously not thread safe, and secondly, the lifetime of the borrow cant even be guaranteed to be long enough ('static).
How do we work around that: For the first part, the thread-safety, we could use a Mutex. That's overkill for a simple integer though. We use an atomic. For the second part, we could use an Arc - I again went with the simpler option, make it a static ("global") variable.
Edit: In the comments it was raised to my attention, that maybe my views of a "simple" solution to this problem are a little skewed, this is a link to the chapter on thread-safety in the book, which uses an Arc<Mutex<_>> to count to ten in parallel. Every Rust programmer should have read that section (and the entire book!).
So here is your code, with only the relevant parts rewritten:
use std::sync::atomic;
fn handle_client(mut stream: TcpStream) {
let mut data = [0 as u8; 30]; // using 30 byte buffer
while match stream.read(&mut data) {
Ok(size) => {
if size > 0 {
let temp = std::str::from_utf8(&data[0..size]).unwrap().to_string();
let temp: i32 = temp.trim().parse().unwrap();
let prev_a_val = loop {
let a_acquired = a.load(atomic::Ordering::Acquire);
match a.compare_exchange_weak(
a_acquired,
a_acquired * temp,
atomic::Ordering::AcqRel,
atomic::Ordering::Acquire,
) {
Ok(value) => break value,
Err(_) => {}
}
};
let st = (prev_a_val * temp).to_string();
// String в u8
let data = st.as_bytes();
stream.write(&data).unwrap();
}
true
}
Err(_) => {
println!(
"An error occurred, terminating connection with {}",
stream.peer_addr().unwrap()
);
stream.shutdown(Shutdown::Both).unwrap();
false
}
} {}
}
static a: atomic::AtomicI32 = atomic::AtomicI32::new(1);
fn main() {
let listener = TcpListener::bind("0.0.0.0:7956").unwrap();
// accept connections and process them, spawning a new thread for each one
println!("Server listening on port 7956");
println!("A = 1");
for stream in listener.incoming() {
match stream {
Ok(stream) => {
println!("New connection: {}", stream.peer_addr().unwrap());
thread::spawn(|| {
// connection succeeded
handle_client(stream)
});
}
Err(e) => {
println!("Error: {}", e);
/* connection failed */
}
}
}
}