Who can recommend analog for dateset
worldcitiespop
on https://public.opendatasoft.com/?
I use this data set early, but now, this data set is not available.
Or may be, you know another open data source with public api.
Related
I'm developing the API for the application using protobuf and grpc.
I need to send the data with the arbitrary size. Sometimes it is small, sometimes huge. For example Nympy array. If the size is small I want to send it through protobuf, if the size if huge I want to dump data into file and send the filepath to this file through protobuf.
To do so I've created a following .proto messages:
message NumpyTroughProtobuf {
repeated int32 shape = 1;
repeated float array = 2;
}
message NumpyTroughfile {
string filepath = 1;
}
message NumpyTrough {
google.protobuf.Any data = 1;
}
The logic is simple: If the size is big I use data as NumpyTroughfile or if small data as NumpyTroughProtobuf.
Problem (what I want to avoid):
The mechanism of data transformation is the part of my app.
In the current approach I have to check and covert the data before I create the message NumpyTrough. So I have to add some logic into my application which will care of data check and cast. The same I have to do for any language which I use (for example if I send massages from Python to C++).
What I want to do:
The mechanism of data transformation is the part of customized protobuf.
I want to hide the data transformation. I want that my app to send a pure Numpy array into NumpyTrough.data field. All data transformation should be hided.
So I want that the logic of data transformation be the part of custom Protobuf field, not the part of my application.
Which meant that I would like to create a custom type of the field. I just implement the behavior of this filed (marshal/unmarshal) for any languages which I use. Then I can just directly send Numpy data into this custom field and this field will decide how to proceed: turn the data in into file or via other method, send trough Protobuf and restore on the receiver side.
Somethig like this https://github.com/gogo/protobuf/blob/master/custom_types.md but it seems this is not a part of protobuf ecosystem.
Protobuf only defines schema.
You can't add logic to a protobuf definition.
Protobug Any represents arbitrary binary data and so -- somewhere -- you'll need to explain to your users what it represents in order that they can ship data in the correct format to your service.
You get to decide how to distribute the processing of the data:
Either partly client-side functionality that performs preprocessing of the data and ships the output (either as structured data using non-Any types or, if still necessary as Any).
Or partly server-side that receives entirely unprocessed client-side data shipped through Any
Or some combination of the two
NOTE You may want to consider shipping the data regardless of size as file references to simplify your implementation. You're correct to bias protobuf to smaller message sizes but, depending on the file size distribution, does it make sense to complicate your implementation with 2 paths?
I need some help in creating a VI that generates virtual or calculated channels based on several channels I measure.
e.g.
I measure voltage on several AI, lets say, ch A,B,C,D,E were B,C and E represent current on a shunt and would like to calculate a the power of the system
Q[A] = B+C
R[W] = A*Q
S[W] = D*E
T[W] = R+S
I would like to load the equations externally from a configuration file that may vary from one project to another equations would come in a format of a string Q=A+B , R= A*Q .....
*(during a run equation and channel count don't change - only when loading config).
The main issues that I am facing is that the inputs to each equation may have dependencies on virtual channels that do not have data yet
Was trying to use:
formula nodes/ Math scripts: https://zone.ni.com/reference/en-XX/help/371361R-01/lvconcepts/formula_nodes/
https://knowledge.ni.com/KnowledgeArticleDetails?id=kA03q000000x30HCAQ&l=en-IL
All data that should be chunked into a data stream (continues sampling) that can be presented on a Chart/Graph and saved to CSV/TDMS
do I need some additional packages?
I have tried the following based on the the example given - getting strange result
Answer
The elements you are looking for are not the Formula/Math Nodes but rather the:
Formula Parsing VIs
Using these VIs you are able to pass a calculation in the form of a string and an array of variable names and then evaluate the formula. This allows for run-time variable scripting, where most other nodes require compile time formula evaluation (With the exception of the python node).
Example
Example of using a very simple program to evaluate two different calculations using the same values and variables.
I'd like to use the Event Processing functionality to create derived measurements.
Our devices report a measurement that has a target value we store, per-device, in a managed object fragment. I want to create a device measurement that is the difference between the measurement and the target, it seems an ideal candidate for RTP/CEL implementation.
I've looked at the real time processing guide but can't quite work out how to read a fragment from a managed object.
Edit. I updated my code and I'm nearly there, I just can't work out how to use the incoming deviceId from the measurement as the place to get my fragment data from.
create variable ManagedObject device = findManagedObjectById(m.measurement.source.value);
create variable BigDecimal setpoint = getNumber(device, "uty_Setpoint.value");
insert into CreateMeasurement
select
m.measurement.time as time,
m.measurement.source.value as source,
"uty_Pressure_delta" as type,
{"uty_Pressure_delta.T.value", getNumber(m, "uty_Pressure.pressure.value") - setpoint,
"uty_Pressure_delta.T.unit", "percent" } as fragments
from MeasurementCreated m
where getNumber(m, "uty_Pressure.pressure.value") is not null
First step is to query the object based on the source of the measurement
findManagedObjectById(measurement.source.value)
You can also take a look at the geofence example which also reads some parameter from the device object.
http://cumulocity.com/guides/event-language/geofence/
Afterwards you can access the fragment with these functions:
getNumber(deviceObject, "myFragmentInDevice.value")
getString(deviceObject, "myFragmentInDevice.value")
I would ensure that the target value is stored as a number and not a string because then you can use the getNumber function which will return a BigDecimal.
http://cumulocity.com/guides/event-language/functions/#utility-functions
struct stud
{
char name[10];
int rno;
}s[10];
I want to send the data of structure array s from a kernel module to a file in userspace. One way is to combine the data to form a string and send through copy_to_user() but it'll further require tokenization to separate out the data in userspace.
Plz suggest some effective method.
#Gaurav, go through the below link and decide upon which mechanism to use to transfer data from kernel space to user space or vice-versa.
http://people.ee.ethz.ch/~arkeller/linux/kernel_user_space_howto.html
Hope link helps to solve your problem!.
I am trying to build a resource file for a website basically jamming all the images into a compressed file that is then unpacked on the output buffers to the client.
my question is in vb2005 can a filestream be multi threaded if you know the size of the converted file, ala like a bit torrent and work on pieces of the filestream ( the individual files in this case) and add them to the resource filestream when they are done instead of one at a time?
If you need something similar to the torrents way of writing to a file, this is how I would implement it:
Open a FileStream on Thread T1, and create a queue "monitor" for step 2
Create a queue that will be read from T1, but written by multiple network reader threads. (the queue data structure would look like this: (position where to write, size of data buffer, data buffer).
Fire up the threads
:)
Anyway, from your comments, your problem seems to be another one..
I have found something in, but I'm not sure if it works:
If you want to write data to a file,
two parallel methods are available,
WriteByte() and Write(). WriteByte()
writes a single byte to the stream:
byte NextByte = 100;
fs.WriteByte(NextByte);
Write(), on the other hand, writes out
an array of bytes. For instance, if
you initialized the ByteArray
mentioned before with some values, you
could use the following code to write
out the first nBytes of the array:
fs.Write(ByteArray, 0, nBytes);
Citation from:
Nagel, Christian, Bill Evjen, Jay
Glynn, Morgan Skinner, and Karli
Watson. "Chapter 24 - Manipulating
Files and the Registry". Professional
C# 2005 with .NET 3.0. Wrox Press. ©
2007. Books24x7. http://common.books24x7.com/book/id_20568/book.asp
(accessed July 22, 2009)
I'm not sure if you're asking if a System.IO.FileStream object can be read from or written to in a multi-threaded fashion. But the answer in both cases is no. This is not a supported scenario. You will need to add some form of locking to ensure serialized access to the resource.
The documentation calls out multi-threaded access to the object as an unsupported scenario
http://msdn.microsoft.com/en-us/library/system.io.filestream.aspx