How to perform Depth First Search (DFS) inside a module starting from its ports? - yosys

I am trying to implement a new pass to calculate the sequential depth and complexity of a given module in Yosys. To do so, I am getting inspired by scc pass.
To implement it, I need to specifically perform DFS starting from the module's input ports. To do so, I am trying to find all of the cells which are immediately connected to the input ports. I am starting from the module's port and find the associate wires:
SigPool inputPorts;
for (auto &it : module->ports)
if (module->wires_[it]->port_input)
inputPorts.add((sigmap(RTLIL::SigSpec(module->wires_[it]))));
but the problem that I have is that I am not able to find the cells which are immediately connected to the input ports from there (there is not APR for that purpose in the wires/sigspec/sigpool types).
Any help/hint would be greatly appreciated.

I think the following code example (although not DFS) should contain all the relevant idiomatic Yosys code snippets that you'd need:
// create canonical versions of all sigbits
SigMap sigmap(module);
// first layer of bits
pool<SigBit> input_bits;
for (auto wire : module->wires())
if (wire->port_input)
for (auto bit : sigmap(wire))
input_bits.insert(bit);
// index: from sigbit to driven cells
dict<SigBit, pool<Cell*>> bit2cells;
// index: from cell to driven sigbits
dict<Cell*, pool<SigBit>> cell2bits;
for (auto cell : module->cells())
for (auto &conn : cell->connections()) {
if (cell->input(conn.first)) {
for (auto bit : sigmap(conn.second))
bit2cells[bit].insert(cell);
}
if (cell->output(conn.first)) {
for (auto bit : sigmap(conn.second))
cell2bits[cell].insert(bit);
}
}
pool<SigBit> queue = input_bits;
pool<Cell*> visited_cells;
while (!queue.empty())
{
log("------\n");
pool<Cell*> this_iter_cells;
for (auto bit : queue)
for (auto cell : bit2cells[bit])
if (!visited_cells.count(cell)) {
log(" %s\n", log_id(cell));
visited_cells.insert(cell);
this_iter_cells.insert(cell);
}
queue.clear();
for (auto cell : this_iter_cells)
for (auto bit : cell2bits[cell])
queue.insert(bit);
}
The most important things to take away from that:
Use SigMap to create canonical representations of signal bits (in case two wires are connected to each other and are just "alias names" for the same actual bit).
Break up your algorithm in two stages: (1) create the index structures that you need and (2) perform the actual algorithm.
You might also find the answers to this and this question useful.
(I'm writing this answer in a hurry.. Please ask further questions in comments below if anything is unclear.)

I was able to implement that part as follows, that works but I am not sure if it is the best way. any advice/comment would be greatly appreciated.
// for all input the cells of the module find its incoming signals (inputSignals in this code), and for each bit of them do the following:
for (auto &bit : inputSignals)
if (bit.wire != NULL) {
if (inputPortsSet.count(bit.wire->port_id)) {
log ("found an input port\n");
log ("%s :",cell->name.c_str());
log(" %s ", bit.wire->name.c_str());
log(" %d \n", bit.wire->port_id);
break;
}
}

Related

Optaplanner: NullPointerException when calling scoreDirector.beforeVariableChanged in a simple custom move

I am building a Capacited Vehicle Routing Problem with Time Windows, but with one small difference when compared to the one provided in examples from the documentation: I don't have a depot. Instead, each order has a pickup step, and a delivery step, in two different locations.
(like in the Vehicle Routing example from the documentation, the previousStep planning variable has the CHAINED graph type, and its valueRangeProviderRefs includes both Drivers, and Steps)
This difference adds a couple of constraints:
the pickup and delivery steps of a given order must be handled by the same driver
the pickup must be before the delivery
After experimenting with constraints, I have found that it would be more efficient to implement two types of custom moves:
assign both steps of an order to a driver
rearrange the steps of a driver
I am currently implementing that first custom move. My solver's configuration looks like this:
SolverFactory<RoutingProblem> solverFactory = SolverFactory.create(
new SolverConfig()
.withSolutionClass(RoutingProblem.class)
.withEntityClasses(Step.class, StepList.class)
.withScoreDirectorFactory(new ScoreDirectorFactoryConfig()
.withConstraintProviderClass(Constraints.class)
)
.withTerminationConfig(new TerminationConfig()
.withSecondsSpentLimit(60L)
)
.withPhaseList(List.of(
new LocalSearchPhaseConfig()
.withMoveSelectorConfig(CustomMoveListFactory.getConfig())
))
);
My CustomMoveListFactory looks like this (I plan on migrating it to an MoveIteratorFactory later, but for the moment, this is easier to read and write):
public class CustomMoveListFactory implements MoveListFactory<RoutingProblem> {
public static MoveListFactoryConfig getConfig() {
MoveListFactoryConfig result = new MoveListFactoryConfig();
result.setMoveListFactoryClass(CustomMoveListFactory.class);
return result;
}
#Override
public List<? extends Move<RoutingProblem>> createMoveList(RoutingProblem routingProblem) {
List<Move<RoutingProblem>> moves = new ArrayList<>();
// 1. Assign moves
for (Order order : routingProblem.getOrders()) {
Driver currentDriver = order.getDriver();
for (Driver driver : routingProblem.getDrivers()) {
if (!driver.equals(currentDriver)) {
moves.add(new AssignMove(order, driver));
}
}
}
// 2. Rearrange moves
// TODO
return moves;
}
}
And finally, the move itself looks like this (nevermind the undo or the isDoable for the moment):
#Override
protected void doMoveOnGenuineVariables(ScoreDirector<RoutingProblem> scoreDirector) {
assignStep(scoreDirector, order.getPickupStep());
assignStep(scoreDirector, order.getDeliveryStep());
}
private void assignStep(ScoreDirector<RoutingProblem> scoreDirector, Step step) {
StepList beforeStep = step.getPreviousStep();
Step afterStep = step.getNextStep();
// 1. Insert step at the end of the driver's step list
StepList lastStep = driver.getLastStep();
scoreDirector.beforeVariableChanged(step, "previousStep"); // NullPointerException here
step.setPreviousStep(lastStep);
scoreDirector.afterVariableChanged(step, "previousStep");
// 2. Remove step from current chained list
if (afterStep != null) {
scoreDirector.beforeVariableChanged(afterStep, "previousStep");
afterStep.setPreviousStep(beforeStep);
scoreDirector.afterVariableChanged(afterStep, "previousStep");
}
}
The idea being that at no point I'm doing an invalid chained list manipulation:
However, as the title and the code comment indicate, I am getting a NullPointerException when I call scoreDirector.beforeVariableChanged. None of my variables are null (I've printed them to make sure). The NullPointerException doesn't occur in my code, but deep inside Optaplanner's inner workings, making it difficult for me to fix it:
Exception in thread "main" java.lang.NullPointerException
at org.drools.core.common.NamedEntryPoint.update(NamedEntryPoint.java:353)
at org.drools.core.common.NamedEntryPoint.update(NamedEntryPoint.java:338)
at org.drools.core.impl.StatefulKnowledgeSessionImpl.update(StatefulKnowledgeSessionImpl.java:1579)
at org.drools.core.impl.StatefulKnowledgeSessionImpl.update(StatefulKnowledgeSessionImpl.java:1551)
at org.optaplanner.core.impl.score.stream.drools.DroolsConstraintSession.update(DroolsConstraintSession.java:49)
at org.optaplanner.core.impl.score.director.stream.ConstraintStreamScoreDirector.afterVariableChanged(ConstraintStreamScoreDirector.java:137)
at org.optaplanner.core.impl.domain.variable.inverserelation.SingletonInverseVariableListener.retract(SingletonInverseVariableListener.java:96)
at org.optaplanner.core.impl.domain.variable.inverserelation.SingletonInverseVariableListener.beforeVariableChanged(SingletonInverseVariableListener.java:46)
at org.optaplanner.core.impl.domain.variable.listener.support.VariableListenerSupport.beforeVariableChanged(VariableListenerSupport.java:170)
at org.optaplanner.core.impl.score.director.AbstractScoreDirector.beforeVariableChanged(AbstractScoreDirector.java:430)
at org.optaplanner.core.impl.score.director.AbstractScoreDirector.beforeVariableChanged(AbstractScoreDirector.java:390)
at test.optaplanner.solver.AssignMove.assignStep(AssignMove.java:98)
at test.optaplanner.solver.AssignMove.doMoveOnGenuineVariables(AssignMove.java:85)
at org.optaplanner.core.impl.heuristic.move.AbstractMove.doMove(AbstractMove.java:35)
at org.optaplanner.core.impl.heuristic.move.AbstractMove.doMove(AbstractMove.java:30)
at org.optaplanner.core.impl.score.director.AbstractScoreDirector.doAndProcessMove(AbstractScoreDirector.java:187)
at org.optaplanner.core.impl.localsearch.decider.LocalSearchDecider.doMove(LocalSearchDecider.java:132)
at org.optaplanner.core.impl.localsearch.decider.LocalSearchDecider.decideNextStep(LocalSearchDecider.java:116)
at org.optaplanner.core.impl.localsearch.DefaultLocalSearchPhase.solve(DefaultLocalSearchPhase.java:70)
at org.optaplanner.core.impl.solver.AbstractSolver.runPhases(AbstractSolver.java:98)
at org.optaplanner.core.impl.solver.DefaultSolver.solve(DefaultSolver.java:189)
at test.optaplanner.OptaPlannerService.testOptaplanner(OptaPlannerService.java:68)
at test.optaplanner.App.main(App.java:13)
Is there something I did wrong? It seems I am following the documentation for custom moves fairly closely, outside of the fact that I am using exclusively java code instead of drools.
The initial solution I feed to the solver has all of the steps assigned to a single driver. There are 15 drivers and 40 orders.
In order to bypass this error, I have tried a number of different things:
remove the shadow variable annotation, turn Driver into a problem fact, and handle the nextStep field myself => this makes no difference
use Simulated Annealing + First Fit Decreasing construction heuristics, and start with steps not assigned to any driver (this was inspired by looking up the example here, which is more complete than the one from the documentation) => the NullPointerException appears on afterVariableChanged instead, but it still appears.
a number of other things which were probably not very smart
But without a more helpful error message, I can't think of anything else to try.
Thank you for your help

Handling bad messages using Kafka's Streams API

I have a basic stream processing flow which looks like
master topic -> my processing in a mapper/filter -> output topics
and I am wondering about the best way to handle "bad messages". This could potentially be things like messages that I can't deserialize properly, or perhaps the processing/filtering logic fails in some unexpected way (I have no external dependencies so there should be no transient errors of that sort).
I was considering wrapping all my processing/filtering code in a try catch and if an exception was raised then routing to an "error topic". Then I can study the message and modify it or fix my code as appropriate and then replay it on to master. If I let any exceptions propagate, the stream seems to get jammed and no more messages are picked up.
Is this approach considered best practice?
Is there a convenient Kafka streams way to handle this? I don't think there is a concept of a DLQ...
What are the alternative ways to stop Kafka jamming on a "bad message"?
What alternative error handling approaches are there?
For completeness here is my code (pseudo-ish):
class Document {
// Fields
}
class AnalysedDocument {
Document document;
String rawValue;
Exception exception;
Analysis analysis;
// All being well
AnalysedDocument(Document document, Analysis analysis) {...}
// Analysis failed
AnalysedDocument(Document document, Exception exception) {...}
// Deserialisation failed
AnalysedDocument(String rawValue, Exception exception) {...}
}
KStreamBuilder builder = new KStreamBuilder();
KStream<String, AnalysedPolecatDocument> analysedDocumentStream = builder
.stream(Serdes.String(), Serdes.String(), "master")
.mapValues(new ValueMapper<String, AnalysedDocument>() {
#Override
public AnalysedDocument apply(String rawValue) {
Document document;
try {
// Deserialise
document = ...
} catch (Exception e) {
return new AnalysedDocument(rawValue, exception);
}
try {
// Perform analysis
Analysis analysis = ...
return new AnalysedDocument(document, analysis);
} catch (Exception e) {
return new AnalysedDocument(document, exception);
}
}
});
// Branch based on whether analysis mapping failed to produce errorStream and successStream
errorStream.to(Serdes.String(), customPojoSerde(), "error");
successStream.to(Serdes.String(), customPojoSerde(), "analysed");
KafkaStreams streams = new KafkaStreams(builder, config);
streams.start();
Any help greatly appreciated.
Right now, Kafka Streams offers only limited error handling capabilities. There is work in progress to simplify this. For now, your overall approach seems to be a good way to go.
One comment about handling de/serialization errors: handling those error manually, requires you to do de/serialization "manually". This means, you need to configure ByteArraySerdes for key and value for you input/output topic of your Streams app and add a map() that does the de/serialization (ie, KStream<byte[],byte[]> -> map() -> KStream<keyType,valueType> -- or the other way round if you also want to catch serialization exceptions). Otherwise, you cannot try-catch deserialization exceptions.
With your current approach, you "only" validate that the given string represents a valid document -- but it could be the case, that the message itself is corrupted and cannot be converted into a String in the source operator in the first place. Thus, you don't actually cover deserialization exception with you code. However, if you are sure a deserialization exception can never happen, you approach would be sufficient, too.
Update
This issues is tackled via KIP-161 and will be included in the next release 1.0.0. It allows you to register an callback via parameter default.deserialization.exception.handler. The handler will be invoked every time a exception occurs during deserialization and allows you to return an DeserializationResponse (CONTINUE -> drop the record an move on, or FAIL that is the default).
Update 2
With KIP-210 (will be part of in Kafka 1.1) it's also possible to handle errors on the producer side, similar to the consumer part, by registering a ProductionExceptionHandler via config default.production.exception.handler that can return CONTINUE.
Update Mar 23, 2018: Kafka 1.0 provides much better and easier handling for bad error messages ("poison pills") via KIP-161 than what I described below. See default.deserialization.exception.handler in the Kafka 1.0 docs.
This could potentially be things like messages that I can't deserialize properly [...]
Ok, my answer here focuses on the (de)serialization issues as this might be the most tricky scenario to handle for most users.
[...] or perhaps the processing/filtering logic fails in some unexpected way (I have no external dependencies so there should be no transient errors of that sort).
The same thinking (for deserialization) can also be applied to failures in the processing logic. Here, most people tend to gravitate towards option 2 below (minus the deserialization part), but YMMV.
I was considering wrapping all my processing/filtering code in a try catch and if an exception was raised then routing to an "error topic". Then I can study the message and modify it or fix my code as appropriate and then replay it on to master. If I let any exceptions propagate, the stream seems to get jammed and no more messages are picked up.
Is this approach considered best practice?
Yes, at the moment this is the way to go. Essentially, the two most common patterns are (1) skipping corrupted messages or (2) sending corrupted records to a quarantine topic aka a dead letter queue.
Is there a convenient Kafka streams way to handle this? I don't think there is a concept of a DLQ...
Yes, there is a way to handle this, including the use of a dead letter queue. However, it's (at least IMHO) not that convenient yet. If you have any feedback on how the API should allow you to handle this -- e.g. via a new or updated method, a configuration setting ("if serialization/deserialization fails send the problematic record to THIS quarantine topic") -- please let us know. :-)
What are the alternative ways to stop Kafka jamming on a "bad message"?
What alternative error handling approaches are there?
See my examples below.
FWIW, the Kafka community is also discussing the addition of a new CLI tool that allows you to skip over corrupted messages. However, as a user of the Kafka Streams API, I think ideally you want to handle such scenarios directly in your code, and fallback to CLI utilities only as a last resort.
Here are some patterns for the Kafka Streams DSL to handle corrupted records/messages aka "poison pills". This is taken from http://docs.confluent.io/current/streams/faq.html#handling-corrupted-records-and-deserialization-errors-poison-pill-messages
Option 1: Skip corrupted records with flatMap
This is arguably what most users would like to do.
We use flatMap because it allows you to output zero, one, or more output records per input record. In the case of a corrupted record we output nothing (zero records), thereby ignoring/skipping the corrupted record.
Benefit of this approach compared to the others ones listed here: We need to manually deserialize a record only once!
Drawback of this approach: flatMap "marks" the input stream for potential data re-partitioning, i.e. if you perform a key-based operation such as groupings (groupBy/groupByKey) or joins afterwards, your data will be re-partitioned behind the scenes. Since this might be a costly step we don't want that to happen unnecessarily. If you KNOW that the record keys are always valid OR that you don't need to operate on the keys (thus keeping them as "raw" keys in byte[] format), you can change from flatMap to flatMapValues, which will not result in data re-partitioning even if you join/group/aggregate the stream later.
Code example:
Serde<byte[]> bytesSerde = Serdes.ByteArray();
Serde<String> stringSerde = Serdes.String();
Serde<Long> longSerde = Serdes.Long();
// Input topic, which might contain corrupted messages
KStream<byte[], byte[]> input = builder.stream(bytesSerde, bytesSerde, inputTopic);
// Note how the returned stream is of type KStream<String, Long>,
// rather than KStream<byte[], byte[]>.
KStream<String, Long> doubled = input.flatMap(
(k, v) -> {
try {
// Attempt deserialization
String key = stringSerde.deserializer().deserialize(inputTopic, k);
long value = longSerde.deserializer().deserialize(inputTopic, v);
// Ok, the record is valid (not corrupted). Let's take the
// opportunity to also process the record in some way so that
// we haven't paid the deserialization cost just for "poison pill"
// checking.
return Collections.singletonList(KeyValue.pair(key, 2 * value));
}
catch (SerializationException e) {
// log + ignore/skip the corrupted message
System.err.println("Could not deserialize record: " + e.getMessage());
}
return Collections.emptyList();
}
);
Option 2: dead letter queue with branch
Compared to option 1 (which ignores corrupted records) option 2 retains corrupted messages by filtering them out of the "main" input stream and writing them to a quarantine topic (think: dead letter queue). The drawback is that, for valid records, we must pay the manual deserialization cost twice.
KStream<byte[], byte[]> input = ...;
KStream<byte[], byte[]>[] partitioned = input.branch(
(k, v) -> {
boolean isValidRecord = false;
try {
stringSerde.deserializer().deserialize(inputTopic, k);
longSerde.deserializer().deserialize(inputTopic, v);
isValidRecord = true;
}
catch (SerializationException ignored) {}
return isValidRecord;
},
(k, v) -> true
);
// partitioned[0] is the KStream<byte[], byte[]> that contains
// only valid records. partitioned[1] contains only corrupted
// records and thus acts as a "dead letter queue".
KStream<String, Long> doubled = partitioned[0].map(
(key, value) -> KeyValue.pair(
// Must deserialize a second time unfortunately.
stringSerde.deserializer().deserialize(inputTopic, key),
2 * longSerde.deserializer().deserialize(inputTopic, value)));
// Don't forget to actually write the dead letter queue back to Kafka!
partitioned[1].to(Serdes.ByteArray(), Serdes.ByteArray(), "quarantine-topic");
Option 3: Skip corrupted records with filter
I only mention this for completeness. This option looks like a mix of options 1 and 2, but is worse than either of them. Compared to option 1, you must pay the manual deserialization cost for valid records twice (bad!). Compared to option 2, you lose the ability to retain corrupted records in a dead letter queue.
KStream<byte[], byte[]> validRecordsOnly = input.filter(
(k, v) -> {
boolean isValidRecord = false;
try {
bytesSerde.deserializer().deserialize(inputTopic, k);
longSerde.deserializer().deserialize(inputTopic, v);
isValidRecord = true;
}
catch (SerializationException e) {
// log + ignore/skip the corrupted message
System.err.println("Could not deserialize record: " + e.getMessage());
}
return isValidRecord;
}
);
KStream<String, Long> doubled = validRecordsOnly.map(
(key, value) -> KeyValue.pair(
// Must deserialize a second time unfortunately.
stringSerde.deserializer().deserialize(inputTopic, key),
2 * longSerde.deserializer().deserialize(inputTopic, value)));
Any help greatly appreciated.
I hope I could help. If yes, I'd appreciate your feedback on how we could improve the Kafka Streams API to handle failures/exceptions in a better/more convenient way than today. :-)
For the processing logic you could take this approach:
someKStream
.mapValues(inputValue -> {
// for each execution the below "return" could provide a different class than the previous run!
// e.g. "return isFailedProcessing ? failValue : successValue;"
// where failValue and successValue have no related classes
return someObject; // someObject class vary at runtime depending on your business
}) // here you'll have KStream<whateverKeyClass, Object> -> yes, Object for the value!
// you could have a different logic for choosing
// the target topic, below is just an example
.to((k, v, recordContext) -> v instanceof failValueClass ?
"dead-letter-topic" : "success-topic",
// you could completelly ignore the "Produced" part
// and rely on spring-boot properties only, e.g.
// spring.kafka.streams.properties.default.key.serde=yourKeySerde
// spring.kafka.streams.properties.default.value.serde=org.springframework.kafka.support.serializer.JsonSerde
Produced.with(yourKeySerde,
// JsonSerde could be an instance configured as you need
// (with type mappings or headers setting disabled, etc)
new JsonSerde<>()));
Your classes, though different and landing into different topics, will serialize as expected.
When not using to(), but instead one wants to continue with other processing, he could use branch() with splitting the logic based on the kafka-value class; the trick for branch() is to return KStream<keyClass, ?>[] in order to further allow one to cast to the appropriate class the individual array items.
If you want to send an exception (custom exception) to another topic (ERROR_TOPIC_NAME):
#Bean
public KStream<String, ?> kafkaStreamInput(StreamsBuilder kStreamBuilder) {
KStream<String, InputModel> input = kStreamBuilder.stream(INPUT_TOPIC_NAME);
return service.messageHandler(input);
}
public KStream<String, ?> messageHandler(KStream<String, InputModel> inputTopic) {
KStream<String, Object> output;
output = inputTopic.mapValues(v -> {
try {
//return InputModel
return normalMethod(v);
} catch (Exception e) {
//return ErrorModel
return errorHandler(e);
}
});
output.filter((k, v) -> (v instanceof ErrorModel)).to(KafkaStreamsConfig.ERROR_TOPIC_NAME);
output.filter((k, v) -> (v instanceof InputModel)).to(KafkaStreamsConfig.OUTPUT_TOPIC_NAME);
return output;
}
If you want to handle Kafka exceptions and skip it:
#Autowired
public ConsumerErrorHandler(
KafkaProducer<String, ErrorModel> dlqProducer) {
this.dlqProducer = dlqProducer;
}
#Bean
ConcurrentKafkaListenerContainerFactory<?, ?> kafkaListenerContainerFactory(
ConcurrentKafkaListenerContainerFactoryConfigurer configurer,
ObjectProvider<ConsumerFactory<Object, Object>> kafkaConsumerFactory) {
ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
configurer.configure(factory, kafkaConsumerFactory.getIfAvailable());
factory.setErrorHandler(((exception, data) -> {
ErrorModel errorModel = ErrorModel.builder().message()
.status("500").build();
assert data != null;
dlqProducer.send(new ProducerRecord<>(DLQ_TOPIC, data.key().toString(), errorModel));
}));
return factory;
}
All above answers although valid and useful, they are assuming that your streams topology is stateless. For example going back to the original example,
master topic -> my processing in a mapper/filter -> output topics
"my processing in a mapper/filter" should be stateless. I.e. Not re-partitioning (aka writing to a persistent re-partition topic) or doing a toTable() (aka writing to a changelog topic). If the processing fails further down the topology and you commit the transaction (by following any of the 3 option mention above - flatmap, branch or filter - then you have to cater for manually or programmatically eventually deleting that inconsistent state. That would mean writing extra custom code for automatic this.
I would personally expect Streams to also give you a LogAndSkip option for any unhandled runtime exception, not only for deserialization and production ones.
Has anyone any ideas on this?
I don't believe these examples work at all when working with Avro.
When the schema can't be resolved (i.e there is bad/non-avro message corrupting the topic, for example) there is no key or value to deserialize in the first place because by the time the DSL .branch() code is called, the exception has already been thrown (or handled).
Can anyone confirm if this i indeed the case? The very fluent approach you refer to here isn't possible when working with Avro?
KIP-161 does explain how to use a handler, however, it's much more fluent to see it as part of the topology.

Some basic HTTP protocol questions (programming and theory)

Before you decide its a tl:dr (too long, didnt read) post try to read at least some, since Its a question broken down in a lot of small pieces. Some of which you can probably answer and help me.
Please try to help me as much as you can. These types of problems are very common on the internet and I think you will help me and much more people after me.
I am currently researching HTTP services and the protocol itself so that I can discover if it is useful to me.
I have some basic questions as well as some code that needs to be discussed.
First I would like to know how does the communication start? I have discovered that the client sends a message in which it requests a resource (is this correct?). Then what happens? I (as a server) have to reply with what?
Do I need to append a carriage return and a line feed after every response? Somewhere it says there even need to be two (\r\n\r\n).
How can an asynchronous writing be established? (I hope this question is understandable) My primary goal is to achieve a connection between a client and a server and then a continuous data stream from server to the client. Does the client need to reply for every message it gets?
I hope I made my questions clear, since I'm not an expert in these things (yet, I am very interested in it).
And for the programming part of my problem.
I have managed to put together a simple program in Qt in C++ (server side) and a simple client in Objective C (iOS). The client connects and I can read the request header. It is like this:
Data available, incoming: "GET / HTTP/1.1
Host: localhost:9990
Connection: close
User-Agent: CFStream%20test/1.0 CFNetwork/609 Darwin/12.2.0
Should I reply to this header manually? And if so, what?
The client side code looks like this (i know its not pseudo but i think its pretty self-explanatory):
- (void)setupStream
{
NSURL *url = [NSURL URLWithString:#"http://localhost:9990"];
CFHTTPMessageRef message = CFHTTPMessageCreateRequest(NULL, (CFStringRef)#"GET", (CFURLRef)url, kCFHTTPVersion1_1);
stream = CFReadStreamCreateForHTTPRequest(NULL, message);
CFRelease(message);
if (!CFReadStreamSetProperty(stream, kCFStreamPropertyHTTPShouldAutoredirect, kCFBooleanTrue))
{
NSLog(#"Some error.");
}
CFDictionaryRef proxySettings = CFNetworkCopySystemProxySettings();
CFReadStreamSetProperty(stream, kCFStreamPropertyHTTPProxy, proxySettings);
CFRelease(proxySettings);
if (!CFReadStreamOpen(stream))
{
CFRelease(stream);
NSLog(#"Error opening stream.");
}
CFStreamClientContext context = {0, self, NULL, NULL, NULL};
CFReadStreamSetClient(stream, kCFStreamEventHasBytesAvailable | kCFStreamEventErrorOccurred, readStreamCallback, &context);
CFReadStreamScheduleWithRunLoop(stream, CFRunLoopGetCurrent(), kCFRunLoopCommonModes);
NSLog(#"Done");
}
This is the setup stream method. The stream variable is a class variable of type CFReadStreamRef.
The callback looks like this:
static void readStreamCallback(CFReadStreamRef aStream, CFStreamEventType event, void *client)
{
ViewController *controller = (ViewController*)client;
[controller handleEvent:event forStream:aStream];
}
And the handle event like this:
- (void)handleEvent:(CFStreamEventType)event forStream:(CFReadStreamRef)aStream
{
if (aStream != stream)
{
return;
}
NSLog(#"Handle event callback");
switch (event)
{
case kCFStreamEventHasBytesAvailable:
NSLog(#"Work log");
UInt8 bytes[11];
CFIndex length;
length = CFReadStreamRead(stream, bytes, 11); //I know 11 bytes is hard coded, its in testing stage now. Feel free to suggest me how to do it better.
if (length == -1)
{
NSLog(#"Error, data length = -1");
return;
}
NSLog(#"Len: %li, data: %s", length, bytes);
break;
default:
NSLog(#"Other event");
break;
}
}
And thats practically all the client code that is worth mentioning. The Qt Server part (I will only post the important parts) is done like this: (this is a subclassed QTcpServer class). First the startServer(); is called:
bool Server::startServer()
{
if (!this->listen(QHostAddress::Any, 9990))
return false;
return true;
}
When there is a connection incoming the incomingConnection is fired off with the socket descriptor as a parameter:
void Server::incomingConnection(int handle)
{
qDebug("New client connected");
ServerClient *client = new ServerClient(handle, this); //The constructor takes in the socket descriptor needed to set up the socket and the parent (this)
client->setVectorLocation(clients.count()); //This is a int from a Qvector in which i append the clients, its not important for understanding right now.
connect(client, SIGNAL(clientDisconnected(int)), this, SLOT(clientDisconnected(int)), Qt::QueuedConnection); //When the client socket emits a disconnected signal the ServerClient class emits a client disconnected signal which the server uses to delete that client from the vector (thats why I use "setVectorLocation(int)") - not important right now
clients.push_back(client); //And then I append the client to the QVector - not important right now
}
The ClientServer class constructor just creates a new socket and connects the required methods:
ServerClient::ServerClient(int handle, QObject *parent) :
QObject(parent)
{
socket = new QTcpSocket(this); //Socket is a class variable
connect(socket, SIGNAL(disconnected()), this, SLOT(disconnected()));
connect(socket, SIGNAL(readyRead()), this, SLOT(readyRead()));
socket->setSocketDescriptor(handle);
}
Ready read just writes me the data incoming (it wont be much user later i think):
void ServerClient::readyRead()
{
qDebug() << "Data available, incoming: " << socket->readAll();
}
And finally the write data:
void ServerClient::writeData(QByteArray *data)
{
data->append("\r\n\r\n"); //I have read this must be appended to all outgoing data from a HTTP server
socket->write(*data);
socket->flush();
qDebug() << "Written data to client: " << *data;
}
This code however does not always work. Sometimes when I write message like "Message" the client recieves all the data and some things that shouldnt be there (the new line and a wierd symbol - can NSLog cause this?). Sometimes when I send "Hellow" the client only gets "Hel" and some other funky stuff.
What are the problems? What should I pay more attention about? Anything that will help me will be MUCH appreciated. And please dont paste in some links that contain a book with a few hundred pages, Im sure this can be solved just by explaining things to me.
THANKS A LOT!
Jan.
You asked many questions ... and that's a perfectly legitimate thing to do :)
I confess - it was too long, I didn't read :(
BUT ...
1) Yes, the HTTP protocol does expect na "CRLF" ("\r\n"). Many servers and many clients are "forgiving", but strictly speaking - yes, you need them.
REFERENCE: RFC 2616
2) Wanting to understand HTTP "internals" is also perfectly legitimate - I applaud you.
One good way is to read the RFC(s).
Another is to use a "telnet" client: http://blog.tonycode.com/tech-stuff/http-notes/making-http-requests-via-telnet
Yet another is to study requests and responses in FF Firebug
3) Socket programming is another issue - which explains why sometimes you might read "hello world", and other times you might just get "hel".
Strong recommendation: Beej's Guide to Network Programming
4) Finally, no way would I actually write a server in Qt with C++ (except maybe as a toy "science experiment", or for some really off-the-wall requirement)
I would definitely write server code in C# (for Windows servers), Java (for everything else) or a scripting language I felt comfortable with (Perl, Ruby/RoR, Python and Lua all come to mind).
IMHO .. and hope that helps!
Your questions pretty much amount to "how does HTTP work", and the full answer lies in the specification.

MidiReadProc - using srcConnRefCon to listen to only one source

I am trying to write a basic app that uses CoreMidi to receive midi events from a specific source. I understand that all midi events that come into a port call the proc that I connected via MidiInputPortCreate(). I also understand that when using MidiPortConnectSource() that you can send an identifier (connRefCon) to help know what the source is. But I'm not sure how to use it.
I figure that within my MidiReadProc that I can use the scrConnRefCon and an if statement to listen to a specific source, but I still dont know what *void I should pass to separate each source. Ideally my ReadProc will look something like this:
void SourceReadProc (const MIDIPacketList *pktlist,
void *readProcRefCon,
void *srcConnRefCon)
{
if (srcConnRefCon == mySourceChoice) {
// pass the pktlist to do something
}
};
Any help will be greatly appreciated.
GW
After a break I've come back to this project with a fresh perspective. When I call MIDIPortConnectSource and pass a unique connRefCon it's not apparently passing for each endpoint. Here's my code:
ItemCount count = MIDIGetNumberOfSources();
for (Itemcount i=0; i<count; i++) {
MIDIEndpointRef endpoint = MIDIGetSource(i);
MIDIObjectGetStringProperty(endpoint,kMIDIPropertyName, &midiEndpointSourceName);
NSLog(#"Source %lu: %#", i, midiEndpointSourceName);
MIDIPortConnectSource(midiSourcePort, endpoint, (void*)&i);
}
Then my read proc:
void SourceReadProc (const MIDIPacketList *pktlist,
void *readProcRefCon,
void *srcConnRefCon)
{
ItemCount *source = (ItemCount*) srcConnRefCon;
NSLog(#"source: %lu", *source);
}
I've hooked up two different midi sources and I can find them both just fine. My first code reports that there are two sources and tells me their names. But my read proc says that the sources is always the first source. I've tried three different data types when passing the connRefCon with no luck. I feel that my issue must be with the MIDIPortConnectSource.
Any help or even troubleshooting ideas would be great. I wish that CoreMIDI had functions to query what's connected to ports so I could check that, but alas, there's not.
The srcConnRefCon is useful if you've made multiple MIDIPortConnectSource() calls. Most commonly, it's a pointer to an object representing the source, but it could be anything. If you just want to disambiguate multiple sources, you could, say, use a string.
MIDIPortConnectSource(port, endpoint1, (void *)"endpoint1");
MIDIPortConnectSource(port, endpoint2, (void *)"endpoint2");
Then, in your SourceReadProc, you'd do something like this:
char *source = (char *)srcConnRefCon;
if (!strcmp(source, "endpoint1")) {
// Process packets from source 1
}
Make sure the allocation lifetime of whatever you pass in extends as long as the port is connected - otherwise you'll get a dangling pointer, which can be hell to debug.

sprintf() and WriteFile() affecting string Buffer

I have a very weird problem which I cannot seem to figure out. Unfortunately, I'm not even sure how to describe it without describing my entire application. What I am trying to do is:
1) read a byte from the serial port
2) store each char into tagBuffer as they are read
3) run a query using tagBuffer to see what type of tag it is (book or shelf tag)
4) depending on the type of tag, output a series of bytes corresponding to the type of tag
Most of my code is implemented and I can get the right tag code sent back out the serial port. But there are two lines that I've added as debug statements which when I tried to remove them, they cause my program to stop working.
The lines are the two lines at the very bottom:
sprintf(buf,"%s!\n", tagBuffer);
WriteFile(hSerial,buf,strlen(buf), &dwBytesWritten,&ovWrite);
If I try to remove them, "tagBuffer" will only store the last character as oppose being a buffer. Same thing with the next line, WriteFile().
I thought sprintf and WriteFile are I/O functions and would have no effect on variables.
I'm stuck and I need help to fix this.
//keep polling as long as stop character '-' is not read
while(szRxChar != '-')
{
// Check if a read is outstanding
if (HasOverlappedIoCompleted(&ovRead))
{
// Issue a serial port read
if (!ReadFile(hSerial,&szRxChar,1,
&dwBytesRead,&ovRead))
{
DWORD dwErr = GetLastError();
if (dwErr!=ERROR_IO_PENDING)
return dwErr;
}
}
// resets tagBuffer in case tagBuffer is out of sync
time_t t_time = time(0);
char buf[50];
if (HasOverlappedIoCompleted(&ovWrite))
{
i=0;
}
// Wait 5 seconds for serial input
if (!(HasOverlappedIoCompleted(&ovRead)))
{
WaitForSingleObject(hReadEvent,RESET_TIME);
}
// Check if serial input has arrived
if (GetOverlappedResult(hSerial,&ovRead,
&dwBytesRead,FALSE))
{
// Wait for the write
GetOverlappedResult(hSerial,&ovWrite,
&dwBytesWritten,TRUE);
if( strlen(tagBuffer) >= PACKET_LENGTH )
{
i = 0;
}
//load tagBuffer with byte stream
tagBuffer[i] = szRxChar;
i++;
tagBuffer[i] = 0; //char arrays are \0 terminated
//run query with tagBuffer
sprintf(query,"select type from rfid where rfidnum=\"");
strcat(query, tagBuffer);
strcat(query, "\"");
mysql_real_query(&mysql,query,(unsigned int)strlen(query));
//process result and send back to handheld
res = mysql_use_result(&mysql);
while(row = mysql_fetch_row(res))
{
printf("result of query is %s\n",row[0]);
string str = "";
str = string(row[0]);
if( str == "book" )
{
WriteFile(hSerial,BOOK_INDICATOR,strlen(BOOK_INDICATOR),
&dwBytesWritten,&ovWrite);
}
else if ( str == "shelf" )
{
WriteFile(hSerial,SHELF_INDICATOR,strlen(SHELF_INDICATOR),
&dwBytesWritten,&ovWrite);
}
else //this else doesn't work
{
WriteFile(hSerial,NOK,strlen(NOK),
&dwBytesWritten,&ovWrite);
}
}
mysql_free_result(res);
// Display a response to input
//printf("query is %s!\n", query);
//printf("strlen(tagBuffer) is %d!\n", strlen(tagBuffer));
//without these, tagBuffer only holds the last character
sprintf(buf,"%s!\n", tagBuffer);
WriteFile(hSerial,buf,strlen(buf), &dwBytesWritten,&ovWrite);
}
}
With those two lines, my output looks like this:
s sh she shel shelf shelf0 shelf00 BOOKCODE shelf0001
Without them, I figured out that tagBuffer and buf only stores the most recent character at any one time.
Any help at all will be greatly appreciated. Thanks.
Where are you allocating tagbuffer, how large is it?
It's possible that you are overwriting 'buf' because you are writing past the end of tagbuffer.
It seems unlikely that those two lines would have that effect on a correct program - maybe you haven't allocated sufficient space in buf for the whole length of the string in tagBuffer? This might cause a buffer overrun that is disguising the real problem?
The first thing I'd say is a piece of general advice: bugs aren't always where you think they are. If you've got something going on that doesn't seem to make sense, it often means that your assumptions somewhere else are wrong.
Here, it does seem very unlikely that an sprintf() and a WriteFile() will change the state of the "buf" array variable. However, those two lines of test code do write to "hSerial", while your main loop also reads from "hSerial". That sounds like a recipie for changing the behaviour of your program.
Suggestion: Change your lines of debugging output to store the output somewhere else: to a dialog box, or to a log file, or similar. Debugging output should generally not go to files used in the core logic, as that's too likely to change how the core logic behaves.
In my opinion, the real problem here is that you're trying to read and write the serial port from a single thread, and this is making the code more complex than it needs to be. I suggest that you read the following articles and reconsider your design:
Serial Port I/O from Joseph Newcomer's website.
Serial Communications in Win32 from MSDN.
In a multithreaded implementation, whenever the reader thread reads a message from the serial port you would then post it to your application's main thread. The main thread would then parse the message and query the database, and then queue an appropriate response to the writer thread.
This may sound more complex than your current design, but really it isn't, as Newcomer explains.
I hope this helps!