I am trying to implement a client-server application using TLS (openssl). I followed the example given in rust doc for my code's structure: example
Server Code
fn handle_client(mut stream: SslStream<TcpStream>){
println!("Passed in handling method");
let mut data = vec![];
let length = stream.read(&mut data).unwrap();
println!("read successfully; size read:{}", length);
stream.write(b"From server").unwrap();
stream.flush().unwrap();
println!("{}", String::from_utf8_lossy(&data));
}
fn main() {
//remember: certificate should always be signed
let mut acceptor = SslAcceptor::mozilla_intermediate(SslMethod::tls()).unwrap();
acceptor.set_private_key_file("src/keyfile/key.pem", SslFiletype::PEM).unwrap();
acceptor.set_certificate_file("src/keyfile/certs.pem",SslFiletype::PEM).unwrap();
acceptor.check_private_key().unwrap();
let acceptor = Arc::new(acceptor.build());
let listener = TcpListener::bind("127.0.0.1:9000").unwrap();
for stream in listener.incoming(){
match stream{
Ok(stream)=>{
println!("a receiver is connected");
let acceptor = acceptor.clone();
//thread::spawn(move || {
let stream = acceptor.accept(stream).unwrap();
handle_client(stream);
//});
}
Err(_e)=>{println!{"connection failed"}}
}
}
println!("Server");
}
Client Code
fn main() {
let mut connector = SslConnector::builder(SslMethod::tls()).unwrap();
connector.set_verify(SslVerifyMode::NONE); //Deactivated verification due to authentication error
connector.set_ca_file("src/keyfile/certs.pem");
let connector = connector.build();
let stream = TcpStream::connect("127.0.0.1:9000").unwrap();
let mut stream = connector.connect("127.0.0.1",stream).unwrap();
stream.write(b"From Client").unwrap();
stream.flush().unwrap();
println!("client sent its message");
let mut res = vec![];
stream.read_to_end(&mut res).unwrap();
println!("{}", String::from_utf8_lossy(&res));
// stream.write_all(b"client").unwrap();
println!("Client");
}
The Server code and the client code both compile without issues, albeit with some warnings. The client is able to connect to the server. But when the client writes its message From Client to the stream, the stream.read called in handle_client() returns nothing. Furthermore, when the server writes its message From Server, the client is able to receive that.
Hence, is there an issue with the way I use SslStream or on the way I configured my server?
I presume when you say stream.read returns nothing, that it returns a zero value indicating that nothing was read.
The Read trait API says this:
This function does not provide any guarantees about whether it blocks
waiting for data, but if an object needs to block for a read and
cannot, it will typically signal this via an Err return value.
If n is 0, then it can indicate one of two scenarios:
This reader has reached its "end of file" and will likely no longer be
able to produce bytes. Note that this does not mean that the reader
will always no longer be able to produce bytes.
The buffer specified was 0 bytes in length.
It is not an error if the returned value n is smaller than the buffer
size, even when the reader is not at the end of the stream yet. This
may happen for example because fewer bytes are actually available
right now (e. g. being close to end-of-file) or because read() was
interrupted by a signal.
So, you need to repeatedly call read until you receive all the bytes you expect, or you get an error.
If you know exactly how much you want to read (as you do in this case) you can call read_exact which will read exactly the number of bytes needed to fill the supplied buffer.
If you want to read up until a delimeter (such as a newline or other character) you can use a BufReader, which provides methods such as read_until or read_line.
Related
I'm making a small ncurses application in Rust that needs to communicate with a child process. I already have a prototype written in Common Lisp. I'm trying to rewrite it because CL uses a huge amount of memory for such a small tool.
I'm having some trouble figuring out how to interact with the sub-process.
What I'm currently doing is roughly this:
Create the process:
let mut program = match Command::new(command)
.args(arguments)
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
{
Ok(child) => child,
Err(_) => {
println!("Cannot run program '{}'.", command);
return;
}
};
Pass it to an infinite (until user exits) loop, which reads and handles input and listens for output like this (and writes it to the screen):
fn listen_for_output(program: &mut Child, output_viewer: &TextViewer) {
match program.stdout {
Some(ref mut out) => {
let mut buf_string = String::new();
match out.read_to_string(&mut buf_string) {
Ok(_) => output_viewer.append_string(buf_string),
Err(_) => return,
};
}
None => return,
};
}
The call to read_to_string however blocks the program until the process exits. From what I can see read_to_end and read also seem to block. If I try running something like ls which exits right away, it works, but with something that doesn't exit like python or sbcl it only continues once I kill the subprocess manually.
Based on this answer, I changed the code to use BufReader:
fn listen_for_output(program: &mut Child, output_viewer: &TextViewer) {
match program.stdout.as_mut() {
Some(out) => {
let buf_reader = BufReader::new(out);
for line in buf_reader.lines() {
match line {
Ok(l) => {
output_viewer.append_string(l);
}
Err(_) => return,
};
}
}
None => return,
}
}
However, the problem still remains the same. It will read all lines that are available, and then block. Since the tool is supposed to work with any program, there is no way to guess out when the output will end, before trying to read. There doesn't appear to be a way to set a timeout for BufReader either.
Streams are blocking by default. TCP/IP streams, filesystem streams, pipe streams, they are all blocking. When you tell a stream to give you a chunk of bytes it will stop and wait till it has the given amout of bytes or till something else happens (an interrupt, an end of stream, an error).
The operating systems are eager to return the data to the reading process, so if all you want is to wait for the next line and handle it as soon as it comes in then the method suggested by Shepmaster in Unable to pipe to or from spawned child process more than once (and also in his answer here) works.
Though in theory it doesn't have to work, because an operating system is allowed to make the BufReader wait for more data in read, but in practice the operating systems prefer the early "short reads" to waiting.
This simple BufReader-based approach becomes even more dangerous when you need to handle multiple streams (like the stdout and stderr of a child process) or multiple processes. For example, BufReader-based approach might deadlock when a child process waits for you to drain its stderr pipe while your process is blocked waiting on it's empty stdout.
Similarly, you can't use BufReader when you don't want your program to wait on the child process indefinitely. Maybe you want to display a progress bar or a timer while the child is still working and gives you no output.
You can't use BufReader-based approach if your operating system happens not to be eager in returning the data to the process (prefers "full reads" to "short reads") because in that case a few last lines printed by the child process might end up in a gray zone: the operating system got them, but they're not large enough to fill the BufReader's buffer.
BufReader is limited to what the Read interface allows it to do with the stream, it's no less blocking than the underlying stream is. In order to be efficient it will read the input in chunks, telling the operating system to fill as much of its buffer as it has available.
You might be wondering why reading data in chunks is so important here, why can't the BufReader just read the data byte by byte. The problem is that to read the data from a stream we need the operating system's help. On the other hand, we are not the operating system, we work isolated from it, so as not to mess with it if something goes wrong with our process. So in order to call to the operating system there needs to be a transition to "kernel mode" which might also incur a "context switch". That is why calling the operating system to read every single byte is expensive. We want as few OS calls as possible and so we get the stream data in batches.
To wait on a stream without blocking you'd need a non-blocking stream. MIO promises to have the required non-blocking stream support for pipes, most probably with PipeReader, but I haven't checked it out so far.
The non-blocking nature of a stream should make it possible to read data in chunks regardless of whether the operating system prefers the "short reads" or not. Because non-blocking stream never blocks. If there is no data in the stream it simply tells you so.
In the absense of a non-blocking stream you'll have to resort to spawning threads so that the blocking reads would be performed in a separate thread and thus won't block your primary thread. You might also want to read the stream byte by byte in order to react to the line separator immediately in case the operating system does not prefer the "short reads". Here's a working example: https://gist.github.com/ArtemGr/db40ae04b431a95f2b78.
P.S. Here's an example of a function that allows to monitor the standard output of a program via a shared vector of bytes:
use std::io::Read;
use std::process::{Command, Stdio};
use std::sync::{Arc, Mutex};
use std::thread;
/// Pipe streams are blocking, we need separate threads to monitor them without blocking the primary thread.
fn child_stream_to_vec<R>(mut stream: R) -> Arc<Mutex<Vec<u8>>>
where
R: Read + Send + 'static,
{
let out = Arc::new(Mutex::new(Vec::new()));
let vec = out.clone();
thread::Builder::new()
.name("child_stream_to_vec".into())
.spawn(move || loop {
let mut buf = [0];
match stream.read(&mut buf) {
Err(err) => {
println!("{}] Error reading from stream: {}", line!(), err);
break;
}
Ok(got) => {
if got == 0 {
break;
} else if got == 1 {
vec.lock().expect("!lock").push(buf[0])
} else {
println!("{}] Unexpected number of bytes: {}", line!(), got);
break;
}
}
}
})
.expect("!thread");
out
}
fn main() {
let mut cat = Command::new("cat")
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.expect("!cat");
let out = child_stream_to_vec(cat.stdout.take().expect("!stdout"));
let err = child_stream_to_vec(cat.stderr.take().expect("!stderr"));
let mut stdin = match cat.stdin.take() {
Some(stdin) => stdin,
None => panic!("!stdin"),
};
}
With a couple of helpers I'm using it to control an SSH session:
try_s! (stdin.write_all (b"echo hello world\n"));
try_s! (wait_forˢ (&out, 0.1, 9., |s| s == "hello world\n"));
P.S. Note that await on a read call in async-std is blocking as well. It's just instead of blocking a system thread it only blocks a chain of futures (a stack-less green thread essentially). The poll_read is the non-blocking interface. In async-std#499 I've asked the developers whether there's a short read guarantee from these APIs.
P.S. There might be a similar concern in Nom: "we would want to tell the IO side to refill according to the parser's result (Incomplete or not)"
P.S. Might be interesting to see how stream reading is implemented in crossterm. For Windows, in poll.rs, they are using the native WaitForMultipleObjects. In unix.rs they are using mio poll.
Tokio's Command
Here is an example of using tokio 0.2:
use std::process::Stdio;
use futures::StreamExt; // 0.3.1
use tokio::{io::BufReader, prelude::*, process::Command}; // 0.2.4, features = ["full"]
#[tokio::main]
async fn main() {
let mut cmd = Command::new("/tmp/slow.bash")
.stdout(Stdio::piped()) // Can do the same for stderr
.spawn()
.expect("cannot spawn");
let stdout = cmd.stdout().take().expect("no stdout");
// Can do the same for stderr
// To print out each line
// BufReader::new(stdout)
// .lines()
// .for_each(|s| async move { println!("> {:?}", s) })
// .await;
// To print out each line *and* collect it all into a Vec
let result: Vec<_> = BufReader::new(stdout)
.lines()
.inspect(|s| println!("> {:?}", s))
.collect()
.await;
println!("All the lines: {:?}", result);
}
Tokio-Threadpool
Here is an example of using tokio 0.1 and tokio-threadpool. We start the process in a thread using the blocking function. We convert that to a stream with stream::poll_fn
use std::process::{Command, Stdio};
use tokio::{prelude::*, runtime::Runtime}; // 0.1.18
use tokio_threadpool; // 0.1.13
fn stream_command_output(
mut command: Command,
) -> impl Stream<Item = Vec<u8>, Error = tokio_threadpool::BlockingError> {
// Ensure that the output is available to read from and start the process
let mut child = command
.stdout(Stdio::piped())
.spawn()
.expect("cannot spawn");
let mut stdout = child.stdout.take().expect("no stdout");
// Create a stream of data
stream::poll_fn(move || {
// Perform blocking IO
tokio_threadpool::blocking(|| {
// Allocate some space to store anything read
let mut data = vec![0; 128];
// Read 1-128 bytes of data
let n_bytes_read = stdout.read(&mut data).expect("cannot read");
if n_bytes_read == 0 {
// Stdout is done
None
} else {
// Only return as many bytes as we read
data.truncate(n_bytes_read);
Some(data)
}
})
})
}
fn main() {
let output_stream = stream_command_output(Command::new("/tmp/slow.bash"));
let mut runtime = Runtime::new().expect("Unable to start the runtime");
let result = runtime.block_on({
output_stream
.map(|d| String::from_utf8(d).expect("Not UTF-8"))
.fold(Vec::new(), |mut v, s| {
print!("> {}", s);
v.push(s);
Ok(v)
})
});
println!("All the lines: {:?}", result);
}
There's numerous possible tradeoffs that can be made here. For example, always allocating 128 bytes isn't ideal, but it's simple to implement.
Support
For reference, here's slow.bash:
#!/usr/bin/env bash
set -eu
val=0
while [[ $val -lt 10 ]]; do
echo $val
val=$(($val + 1))
sleep 1
done
See also:
How do I synchronously return a value calculated in an asynchronous Future in stable Rust?
If Unix support is sufficient, you can also make the two output streams as non-blocking and poll over them as you would do it on TcpStream with the set_nonblocking function.
The ChildStdout and ChildStderr returned by the Command spawn are Stdio (and contain a file descriptor), you can modify directly the read behavior of these handle to make it non-blocking.
Based on the work of jcreekmore/timeout-readwrite-rs and anowell/nonblock-rs, I use this wrapper to modify the stream handles:
extern crate libc;
use std::io::Read;
use std::os::unix::io::AsRawFd;
use libc::{F_GETFL, F_SETFL, fcntl, O_NONBLOCK};
fn set_nonblocking<H>(handle: &H, nonblocking: bool) -> std::io::Result<()>
where
H: Read + AsRawFd,
{
let fd = handle.as_raw_fd();
let flags = unsafe { fcntl(fd, F_GETFL, 0) };
if flags < 0 {
return Err(std::io::Error::last_os_error());
}
let flags = if nonblocking{
flags | O_NONBLOCK
} else {
flags & !O_NONBLOCK
};
let res = unsafe { fcntl(fd, F_SETFL, flags) };
if res != 0 {
return Err(std::io::Error::last_os_error());
}
Ok(())
}
You can manage the two streams as any other non-blocking stream. The following example is based on the polling crate which makes really easy to handle read event and BufReader for line reading:
use std::process::{Command, Stdio};
use std::path::PathBuf;
use std::io::{BufReader, BufRead};
use std::thread;
extern crate polling;
use polling::{Event, Poller};
fn main() -> Result<(), std::io::Error> {
let path = PathBuf::from("./worker.sh").canonicalize()?;
let mut child = Command::new(path)
.stdin(Stdio::null())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.expect("Failed to start worker");
let handle = thread::spawn({
let stdout = child.stdout.take().unwrap();
set_nonblocking(&stdout, true)?;
let mut reader_out = BufReader::new(stdout);
let stderr = child.stderr.take().unwrap();
set_nonblocking(&stderr, true)?;
let mut reader_err = BufReader::new(stderr);
move || {
let key_out = 1;
let key_err = 2;
let mut out_closed = false;
let mut err_closed = false;
let poller = Poller::new().unwrap();
poller.add(reader_out.get_ref(), Event::readable(key_out)).unwrap();
poller.add(reader_err.get_ref(), Event::readable(key_err)).unwrap();
let mut line = String::new();
let mut events = Vec::new();
loop {
// Wait for at least one I/O event.
events.clear();
poller.wait(&mut events, None).unwrap();
for ev in &events {
// stdout is ready for reading
if ev.key == key_out {
let len = match reader_out.read_line(&mut line) {
Ok(len) => len,
Err(e) => {
println!("stdout read returned error: {}", e);
0
}
};
if len == 0 {
println!("stdout closed (len is null)");
out_closed = true;
poller.delete(reader_out.get_ref()).unwrap();
} else {
print!("[STDOUT] {}", line);
line.clear();
// reload the poller
poller.modify(reader_out.get_ref(), Event::readable(key_out)).unwrap();
}
}
// stderr is ready for reading
if ev.key == key_err {
let len = match reader_err.read_line(&mut line) {
Ok(len) => len,
Err(e) => {
println!("stderr read returned error: {}", e);
0
}
};
if len == 0 {
println!("stderr closed (len is null)");
err_closed = true;
poller.delete(reader_err.get_ref()).unwrap();
} else {
print!("[STDERR] {}", line);
line.clear();
// reload the poller
poller.modify(reader_err.get_ref(), Event::readable(key_err)).unwrap();
}
}
}
if out_closed && err_closed {
println!("Stream closed, exiting process thread");
break;
}
}
}
});
handle.join().unwrap();
Ok(())
}
Additionally, used with a wrapper over an EventFd, it becomes possible to easily stop the process from another thread without blocking nor active polling and uses and only a single thread.
EDIT: It seems the polling crate sets automatically the polled handles in non-blocking mode following my tests. The set_nonblocking function is still useful in case you want to directly use the nix::poll object.
I have encountered enough use-cases where it was useful to interact with a subprocess over line-delimited text that I wrote a crate for it, interactive_process.
I expect the original problem has long since been solved, but I thought it might be helpful to others.
I am receiving messages on UDP in multiple threads. After each reception I raise MessageReceived.OnNext(message).
Because I am using multiple threads the messages raised unordered which is a problem.
How can I order the raise of the messages by the message counter?
(lets say there is a message.counter property)
Must take in mind a message can get lost in the communication (lets say if we have a counter hole after X messages that the hole is not filled I raise the next message)
Messages must be raised ASAP (if the next counter received)
In stating the requirement for detecting lost messages, you haven't considered the possibility of the last message not arriving; I've added a timeoutDuration which flushes the buffered messages if nothing arrives in the given time - you may want to consider this an error instead, see the comments for how to do this.
I will solve this by defining an extension method with the following signature:
public static IObservable<TSource> Sort<TSource>(
this IObservable<TSource> source,
Func<TSource, int> keySelector,
TimeSpan timeoutDuration = new TimeSpan(),
int gapTolerance = 0)
source is the stream of unsorted messages
keySelector is a function that extracts an int key from a message. I assume the first key sought is 0; amend if necessary.
timeoutDuration is discussed above, if omitted, there is no timeout
tolerance is the maximum number of messages held back while waiting for an out of order message. Pass 0 to hold any number of messages
scheduler is the scheduler to use for the timeout and is supplied for test purposes, a default is used if not given.
Walkthrough
I'll present a line-by-line walkthrough here. The full implementation is repeated below.
Assign Default Scheduler
First of all we must assign a default scheduler if none was supplied:
scheduler = scheduler ?? Scheduler.Default;
Arrange Timeout
Now if a time out was requested, we will replace the source with a copy that will simply terminate and send OnCompleted if a message doesn't arrive in timeoutDuration.
if(timeoutDuration != TimeSpan.Zero)
source = source.Timeout(
timeoutDuration,
Observable.Empty<TSource>(),
scheduler);
If you wish to send a TimeoutException instead, just delete the second parameter to Timeout - the empty stream, to select an overload that does this. Note we can safely share this with all subscribers, so it is positioned outside the call to Observable.Create.
Create Subscribe handler
We use Observable.Create to build our stream. The lambda function that is the argument to Create is invoked whenever a subscription occurs and we are passed the calling observer (o). Create returns our IObservable<T> so we return it here.
return Observable.Create<TSource>(o => { ...
Initialize some variables
We will track the next expected key value in nextKey, and create a SortedDictionary to hold the out of order messages until they can be sent.
int nextKey = 0;
var buffer = new SortedDictionary<int, TSource>();
Subscribe to the source, and handle messages
Now we can subscribe to the message stream (possibly with the timeout applied). First we introduce the OnNext handler. The next message is assigned to x:
return source.Subscribe(x => { ...
We invoke the keySelector function to extract the key from the message:
var key = keySelector(x);
If the message has an old key (because it exceeded our tolerance for out of order messages) we are just going to drop it and be done with this message (you may want to act differently):
// drop stale keys
if(key < nextKey) return;
Otherwise, we might have the expected key, in which case we can increment nextKey send the message:
if(key == nextKey)
{
nextKey++;
o.OnNext(x);
}
Or, we might have an out of order future message, in which case we must add it to our buffer. If we do this, we must also ensure our buffer hasn't exceeded our tolerance for storing out of order messages - in this case, we will also bump nextKey to the first key in the buffer which because it is a SortedDictionary is conveniently the next lowest key:
else if(key > nextKey)
{
buffer.Add(key, x);
if(gapTolerance != 0 && buffer.Count > gapTolerance)
nextKey = buffer.First().Key;
}
Now regardless of the outcome above, we need to empty the buffer of any keys that are now ready to go. We use a helper method for this. Note that it adjusts nextKey so we must be careful to pass it by reference. We simply loop over the buffer reading, removing and sending messages as long as the keys follow on from each other, incrementing nextKey each time:
private static void SendNextConsecutiveKeys<TSource>(
ref int nextKey,
IObserver<TSource> observer,
SortedDictionary<int, TSource> buffer)
{
TSource x;
while(buffer.TryGetValue(nextKey, out x))
{
buffer.Remove(nextKey);
nextKey++;
observer.OnNext(x);
}
}
Dealing with errors
Next we supply an OnError handler - this will just pass through any error, including the Timeout exception if you chose to go that way.
Flushing the buffer
Finally, we must handle OnCompleted. Here I have opted to empty the buffer - this would be necessary if an out of order message held up messages and never arrived. This is why we need a timeout:
() => {
// empty buffer on completion
foreach(var item in buffer)
o.OnNext(item.Value);
o.OnCompleted();
});
Full Implementation
Here is the full implementation.
public static IObservable<TSource> Sort<TSource>(
this IObservable<TSource> source,
Func<TSource, int> keySelector,
int gapTolerance = 0,
TimeSpan timeoutDuration = new TimeSpan(),
IScheduler scheduler = null)
{
scheduler = scheduler ?? Scheduler.Default;
if(timeoutDuration != TimeSpan.Zero)
source = source.Timeout(
timeoutDuration,
Observable.Empty<TSource>(),
scheduler);
return Observable.Create<TSource>(o => {
int nextKey = 0;
var buffer = new SortedDictionary<int, TSource>();
return source.Subscribe(x => {
var key = keySelector(x);
// drop stale keys
if(key < nextKey) return;
if(key == nextKey)
{
nextKey++;
o.OnNext(x);
}
else if(key > nextKey)
{
buffer.Add(key, x);
if(gapTolerance != 0 && buffer.Count > gapTolerance)
nextKey = buffer.First().Key;
}
SendNextConsecutiveKeys(ref nextKey, o, buffer);
},
o.OnError,
() => {
// empty buffer on completion
foreach(var item in buffer)
o.OnNext(item.Value);
o.OnCompleted();
});
});
}
private static void SendNextConsecutiveKeys<TSource>(
ref int nextKey,
IObserver<TSource> observer,
SortedDictionary<int, TSource> buffer)
{
TSource x;
while(buffer.TryGetValue(nextKey, out x))
{
buffer.Remove(nextKey);
nextKey++;
observer.OnNext(x);
}
}
Test Harness
If you include nuget rx-testing in a console app, the following will run given you a test harness to play with:
public static void Main()
{
var tests = new Tests();
tests.Test();
}
public class Tests : ReactiveTest
{
public void Test()
{
var scheduler = new TestScheduler();
var xs = scheduler.CreateColdObservable(
OnNext(100, 0),
OnNext(200, 2),
OnNext(300, 1),
OnNext(400, 4),
OnNext(500, 5),
OnNext(600, 3),
OnNext(700, 7),
OnNext(800, 8),
OnNext(900, 9),
OnNext(1000, 6),
OnNext(1100, 12),
OnCompleted(1200, 0));
//var results = scheduler.CreateObserver<int>();
xs.Sort(
keySelector: x => x,
gapTolerance: 2,
timeoutDuration: TimeSpan.FromTicks(200),
scheduler: scheduler).Subscribe(Console.WriteLine);
scheduler.Start();
}
}
Closing comments
There's all sorts of interesting alternative approaches here. I went for this largely imperative approach because I think it's easiest to follow - but there's probably some fancy grouping shenanigans you can employ to do this to. One thing I know to be consistently true about Rx - there's always many ways to skin a cat!
I'm also not entirely comfortable with the timeout idea here - in a production system, I would want to implement some means of checking connectivity, such as a heartbeat or similar. I didn't get into this because obviously it will be application specific. Also, heartbeats have been discussed on these boards and elsewhere before (such as on my blog for example).
Strongly consider using TCP instead if you want reliable ordering - that's what it's for; otherwise, you'll be forced to play a guessing game with UDP and sometimes you'll be wrong.
For example, imagine that you receive the following datagrams in this order: [A, B, D]
When you receive D, how long should you wait for C to arrive before pushing D?
Whatever duration you choose you may be wrong:
What if C was lost during transmission and so it will never arrive?
What if the duration you chose is too short and you end up pushing D but then receive C?
Perhaps you could choose a duration that heuristically works best, but why not just use TCP instead?
Side Note:
MessageReceived.OnNext implies that you're using a Subject<T>, which is probably unnecessary. Consider converting the async UdpClient methods into observables directly instead, or convert them by writing an async iterator via Observable.Create<T>(async (observer, cancel) => { ... }).
I have a type that contains a byte of data, and takes a channel to post new data there. Other code can read the last written byte of data using a Read function.
Edit: for actual, runnable code, see https://github.com/ariejan/i6502/pull/3 especially files acia6551.go and acia6551_test.go. Tests results can be viewed here: https://travis-ci.org/ariejan/i6502/jobs/32862705
I have the following:
// Emulates a serial interface chip of some kind.
type Unit struct {
// Channel used for others to use, bytes written here will be placed in rxChar
Rx chan byte
// Internal store of the last byte written.
rxChar byte // Internal storage
}
// Used internally to read data store in rxChar
func (u *Unit) Read() byte {
return u.rxChar
}
// Create new Unit and go-routing to listen for Rx bytes
func NewUnit(rx chan byte) *Unit {
unit := &Unit{Rx: rx}
go func() {
for {
select {
case data := <-unit.Rx:
unit.rxData = data
fmt.Printf("Posted 0x%02X\n", data)
}
}
}()
return unit
}
My test looks like this:
func TestUnitRx(t *testing.T) {
rx := make(chan byte)
u := NewUnit(rx)
// Post a byte to the Rx channel
// This prints "Posted 0x42", as you'd expect
rx <- 0x42
// Using testing
// Should read last byte, 0x42 but fails.
fmt.Println("Reading value...")
assert.Equal(t, 0x42, u.Read())
}
At first I figured the "Reading value" happened before the go-routing got around to writing the data. But the "Posted" message is always printed before "Reading".
So, two questions remain:
Is this the best way to handle an incoming stream of bytes (at 9600 baud ;-))
If this is the right way, how can I properly test it or what is wrong with my code?
Guessing by the pieces posted here, it doesn't look like you have anything guaranteeing the order of operations when accessing the stored data. You can use a mutex around any data shared between goroutines.
A better option here is to use buffered channels of length 1 to write, store, and read the bytes.
It's always a good idea to test your program with -race to use the race detector.
Since this looks very "stream" like, you very well may want some buffering, and to look at some examples of how the io.Reader and io.Writer interfaces are often used.
I am reading data from a NSFileHandle (from a NSPipe) using a readabilityHandler block:
fileHandle.readabilityHandler = ^( NSFileHandle *handle ) {
[self processData: [handle availableData]];
}
This works fine, I get all the data I expect fed to my processData method. The problem is that I need to know when the last chunk of data was read. availableData should return an empty NSData instance if it reached end-of-file, but the problem is that the reachability handler is not called again on EOF.
I can’t find anything about how to get some kind of notification or callback on EOF. So what am I missing? Is Apple really providing an asynchronous reading API without an EOF callback?
By the way, I cannot use the runloop based readInBackgroundAndNotify method since I don’t have a runloop available. If I cannot get this to work with the NSFileHandle API I probably will directly use a dispatch source to do the IO.
I personally compare current file offset with current file position and stop reading.
extension FileHandle {
func stopReadingIfPassedEOF() {
let pos = offsetInFile
let len = seekToEndOfFile()
if pos < len {
// Resume reading.
seek(toFileOffset: pos)
}
else {
// Stop.
// File offset pointer stays at the EOF.
readabilityHandler = nil
}
}
}
I couldn't understand why it's been designed in this way for a long time, but now I think this could be intentional.
In my opinion, Apple basically defines FileHandle as an infinite stream, therefore, EOF is not well defined unless you close the file. FileHandle seem to be more like a "channel".
It's also unclear what happens if another process appends/delete some data to/from the file while you're reading from it. What would be the EOF in this case? As far as I find, there's no mention about this case in Apple documentation. As far as I know, there's no true exclusive file I/O lock in macOS like other Unix-like systems.
In my opinion, availableData can return empty data at any time if I/O is not fast enough, and readabilityHandler just don't care about EOF.
I believe the accepted answer is actually incorrect. The readabilityHandler is indeed called when EOF is reached. That is signaled by having availableData be of 0 size.
Here’s a simple playground that attests to this.
import Foundation
import PlaygroundSupport
let pipe = Pipe()
pipe.fileHandleForReading.readabilityHandler = { fh in
let d = fh.availableData
print("Data length: \(d.count)")
if (d.count == 0) {
fh.readabilityHandler = nil
}
}
pipe.fileHandleForWriting.write("Hello".data(using: .utf8)!)
pipe.fileHandleForWriting.closeFile()
PlaygroundPage.current.needsIndefiniteExecution = true
I'm afraid you're out of luck doing this with NSFileHandle if you can't use readInBackgroundAndNotify.
Two solutions I see:
Create a runloop and then use readInBackgroundAndNotify.
Roll your own implementation using dispatch_io_*
I am finding the doc for CFStreamCreatePairWithSocketToCFHost confusing:
Specifically, its not clear to me how the function can set the readStream pointer to null on error.
as far as I understand, the pointer is passed by value - so the function can only change the objected pointed to by the pointer.
Right now I can't figure out how to detect connection errors.
Relevant doc snippet:
Creates readable and writable streams connected to a given CFHost object.
void CFStreamCreatePairWithSocketToCFHost (
CFAllocatorRef alloc,
CFHostRef host,
SInt32 port,
CFReadStreamRef *readStream,
CFWriteStreamRef *writeStream
);
readStream
Upon return, contains a CFReadStream object connected to the host host on port port, or NULL if there is a failure during creation. If you pass NULL, the function will not create a readable stream. Ownership follows the Create Rule.
This is my connecting code, it goes all the way to NSLog(#"Connected") even when the server is down.
NSLog(#"Attempting to (re)connect to %#:%d", m_host, m_port);
while(TRUE)
{
CFHostRef host = CFHostCreateWithName(kCFAllocatorDefault, (CFStringRef)m_host);
if (!host)
{
NSLog(#"Error resolving host %#", m_host);
[NSThread sleepForTimeInterval:5.0];
continue;
}
CFStreamCreatePairWithSocketToCFHost(kCFAllocatorDefault, host , m_port, &m_in, &m_out);
CFRelease(host);
if (!m_in)
{
NSLog(#"Error");
}
CFStreamClientContext context = {0, self,nil,nil,nil};
if (CFReadStreamSetClient(m_in, kCFStreamEventHasBytesAvailable | kCFStreamEventErrorOccurred | kCFStreamEventEndEncountered, networkReadEvent, &context))
{
CFReadStreamScheduleWithRunLoop(m_in, CFRunLoopGetCurrent(),kCFRunLoopCommonModes);
}
if (CFWriteStreamSetClient(m_out, kCFStreamEventErrorOccurred | kCFStreamEventEndEncountered, networkWriteEvent, &context))
{
CFWriteStreamScheduleWithRunLoop(m_out, CFRunLoopGetCurrent(),kCFRunLoopCommonModes);
}
BOOL success = CFReadStreamOpen(m_in);
CFErrorRef error = CFReadStreamCopyError(m_in);
if (!success || (error && CFErrorGetCode(error) != 0))
{
NSLog(#"Connect error %s : %d", CFErrorGetDomain(error), CFErrorGetCode(error));
[NSThread sleepForTimeInterval:5.0];
}
else
{
NSLog(#"Connected");
break;
}
}
From the "CFNetwork Programming Guide":
Opening a stream can be a lengthy process, so the CFReadStreamOpen and CFWriteStreamOpen functions avoid blocking by returning TRUE to
indicate that the process of opening the stream has begun. To check
the status of the open, call the functions CFReadStreamGetStatus and
CFWriteStreamGetStatus, which returnkCFStreamStatusOpening if the open
is still in progress, kCFStreamStatusOpen if the open is complete,
orkCFStreamStatusErrorOccurred if the open has completed but failed.
In most cases, it doesn’t matter whether the open is complete because
the CFStream functions that read and write will block until the stream
is open.
Also check out the kCFStreamEventOpenCompleted,
(http://developer.apple.com/library/ios/#documentation/CoreFoundation/Reference/CFStreamConstants/Reference/reference.html)
: a stream event that reports the successful completion of the opening
process. So to conclude, after calling CFReadStreamOpen (or Write),
which will probably succeed, register to listen to the "OpenCompleted"
event to identify a "real" success.
Surely after you call CFStreamCreatePairWithSocketToCFHost() just test readstream to see if it's NULL?
As you're passing in the memory location of the readstream pointer, the function can easily set that to whatever value it chooses (either a reference to a created object, or alternatively NULL).
Edit
I've tried your code, and I agree, it's very confusing. It appears that the CFReadStreamRef is readily created and opened, even for a nonsense host (I literally used "nonsense"). I don't believe this function will return NULL pointers for an unreachable host.
I suppose this makes sense, in as far as until one tries to open the stream, whether it will work or not is unknown.
So, the readStream param is a pointer to the CFReadStreamRef and, as such, can definitely be set to NULL by the function. &foo means "address of foo" and if you have the address you can set the value.
My reading of the documentation for CFStreamCreatePairWithSocketToCFHost is that they will be set to NULL on failure, but that failure is not about connection failure, but other kinds of failure (memory, etc). So not likely you'll get an error there.
Looks to me like the issue is that CFReadStreamOpen can return immediately with true when it can open the stream in the background and so this code is not really opening the stream or testing that it's been opened, merely queuing it for opening). From the documentation for CFReadStreamOpen:
" If the stream can open in the background without blocking, this function always returns true."
So I think you will need to follow the rest of the instructions for CFReadStreamOpen and schedule the stream on a run loop, or perhaps poll (though obviously polling in a tight loop isn't likely to work).
In the documentation for CFReadStreamOpen we see:
Opening a stream causes it to reserve all the system resources it requires. If the stream can open in the background without blocking, this function always returns true.
I suspect that the stream is opening in the background, and thus you are saying "Connected" before it actually opens. You've already scheduled the stream with a runloop, so if you let the run loop run, you'll probably get a callback with the event type set to kCFStreamEventErrorOccurred, and from there you can process the error appropriately.