Filtering posts by timestamp and userId - api

I have the following object in Couchbase:
{
"postReplyId": "Reply_9AE1F47E585522FD1D2EFFEA7671C0855BBFDA991698B23886E37D1C65DAC8AF_1375468399745",
"userId": "User_9AE1F47E585522FD1D2EFFEA7671C0855BBFDA991698B23886E37D1C65DAC8AF",
"postId": "Message_9AE1F47E585522FD1D2EFFEA7671C0855BBFDA991698B23886E37D1C65DAC8AF_1375457606125",
"post_reply_message": "Testing",
"attachments": {
"images": [
],
"audio": [
],
"videos": [
]
},
"upVoters": [
],
"downVoters": [
],
"upVotes": 0,
"report": 0,
"reporters": [
],
"timestamp": 1375468399745,
"mtype": "reply"
}
I would like to have a view and return all the posts created during the last 30 minutes by the user x
I did:
function (doc, meta) {
if(doc.mtype == "reply") {
var dt = new Date();
if((dt.getTime() - doc.timestamp) < 1800000 )
emit(doc.userId, doc);
}
}
and i pass the userIds as multiple keys in the URL, but I get old results
Can someone suggest a solution?

A view runs as documents are added/modified, and only upon request or when automatically updated. It doesn't rerun constantly, and more importantly, it does not re-run for already added documents. So, as written your view would only contain old results.
You need to emit all documents and include the timestamp as part of the emit, so that you can use that as part of the query to the view (a time range).
So, in your emit function, you might instead (untested code):
function (doc, meta) {
if (doc.mtype === "reply") {
// dt will be when the document is processed by the view, which may
// not when the document was added.
var dt = new Date();
var year = dt.getFullYear();
var month = dt.getMonth() + 1; // make month 1-12
var day = dt.getDate();
var hours = dt.getHours();
var minutes = dt.getMinutes();
var seconds = dt.getSeconds();
// emit the full key, including the user id and the date of the document.
emit([doc.userId, year, month, day, hours, minutes, seconds], doc._id);
}
}
Then your query might be like this range (broken into several lines for readability):
/salamis_db/_design/docs/_view/by_user_date?
startkey=["the-userId", 2013, 8, 7, 10, 30, 0]
&endkey=["the-userId", 2013, 8, 7, 11, 00, 00]
Although endkey wouldn't strictly be necessary, I've left it in for clarity.
Because of the way Couchbase views work, a view may not always contain all data though (from here):
Irrespective of the stale parameter, documents can only be indexed by
the system once the document has been persisted to disk. If the
document has not been persisted to disk, use of the stale will not
force this process. You can use the observe operation to monitor when
documents are persisted to disk and/or updated in the index.
Also, note that documents are not immediately added to the view by default. Read this for more information.

Related

PostMan - Generate random number range and random future date

there are certain discussions on generating random stuff within a range on stack but none of them answer the actual question.
So task is the following
Need to generate a random number within a range, let's say from 1 to 13010, and put this number into the body. example
{
"userid": {{randomNumberFromRange}},
"date": "{{RandomDateInFuture}}"
}
the main requirement - this should work in POST request body. Any ideas?
In Pre-request:
pm.variables.set("randomNumberFromRange", _.random(1, 13010));
function randomDate(start, end) {
return new Date(start.getTime() + Math.random() * (end.getTime() - start.getTime()))
}
pm.variables.set("RandomDateInFuture", randomDate(new Date(), new Date(2030, 0, 1)));
Result:
{
"userid": 3538,
"date": "2023-07-19T22:17:13.379Z"
}
Reference: https://gist.github.com/miguelmota/5b67e03845d840c949c4

SSIS Import data that is NOT columnar into SQL

I am fairly new to SSIS and need a little help getting started. I have several reports that come out of our mainframe. The reports are not in a columnar format. The date record is at the top then there might be some initial data then there might be a little more. So I need to read in each line look to see what the text reads and figure out if I need the data or move to the next row.
This is a VERY rough example of what the report I want to import into a SQL table.
DATE: 01/08/2020 FACILITY NAME PAGE1
REVENUE USAGE FOR ACCOUNTING PERIOD 02
----TOTAL---- ----TOTAL---- ----OTHER---- ----INSURANCE---- ----INSURANCE2----
SERVICE CODE - 123456789 DESCRIPTION: WIDGETS
CURR 2,077
IP 0.0000 3 2,345 0.00
143
OP 0.0000 2 1,231 0.00
YTD 5
IP 0.0000
76
OP 0.0000
etc . . . .. .
SERVICE CODE
After the SERVICE CODE the data will start to repeat like it is above. This is the basic idea of a report.
I want to get the Date then the Service Code, Description, Current IP Volume, Current IP Dollar, Current OP Volume, Current OP Dollar, YTD IP Volume, YTD IP Dollar, YTD OP Volume, YTD OP Dollar . . then repeat.
Just to clarify, I am not asking anyone to do this for me. I want to learn how to do this. I have looked on how to do this but every example I have looked at talks about doing this with a CSV, tab, or Excel file. i do not have that type of file so I was asking what I need to look at. I currently use Monarch to format the file, but again I want to learn more about SSIS and this is a perfect way to learn. Asking the vendor to redo the report is not an option plus I want to learn how to do this. Thank you I just wanted to get that out there.
Any help would be greatly appreciated.
Rodger
As stated in comments, you could do this using a script task. The basics steps are:
Define a DataTable to store your data.
Use a StreamReader to read your report.
Process this using a combination of conditionals, String Methods, and parsing to extract the relevant fields from the relevant line:
Write the DataTable to the database using SqlBulkCopy
The following would go inside your Main method in your script task:
//Define a table to store your data
var table = new DataTable
{
Columns =
{
{ "ServiceCode", typeof(string) },
{ "Description", typeof(string) },
{ "CurrentIPVolume", typeof(int) },
{ "CurrentIPDollar,", typeof(decimal) },
{ "CurrentOPVolume", typeof(int) },
{ "CurrentOPDollar", typeof(decimal) },
{ "YTDIPVolume", typeof(int) },
{ "YTDIPDollar,", typeof(decimal) },
{ "YTDOPVolume", typeof(int) },
{ "YTDOPDollar", typeof(decimal) }
}
};
var filePath = #"Your File Path";
using (var reader = new StreamReader(filePath))
{
string line = null;
DataRow row = null;
// As YTD and Curr are identical, we will need a flag later to mark our position within the record
bool ytdFlag= false;
//Loop through every line in the file
while ((line = reader.ReadLine()) != null)
{
//if the line is blank, move on to the next
if (string.IsNullOrWhiteSpace(line)
continue;
// If the line starts with service code, then it marks the start of a new record
if (line.StartsWith("SERVICE CODE"))
{
//If the current value for row is not null then this is
//not the first record, so we need to add the previous
//record to the tale before continuing
if (row != null)
{
table.Rows.Add(row);
ytdFlag= false; // New record, reset YTD flag
}
row = table.NewRow();
//Split the line now based on known values:
var tokens = line.Split(new string[] { "SERVICE CODE - ", "DESCRIPTION: "}, StringSplitOptions.None);
row[0] = tokens[0];
row[1] = tokens[1];
}
if (line.StartsWith("CURR"))
{
//Process the row --> "CURR 2,077"
//Not sure what 2,077 is, but this will parse it
int i = 0;
if (int.TryParse(line.Substring(4).Trim().Replace(",", ""), out i))
{
//Do something with your int
Console.WriteLine(i);
}
}
if (line.StartsWith(" IP"))
{
//Start at after IP then split the line into the 4 numbers
var tokens = line.Substring(3).Split(new [] { " "}, StringSplitOptions.RemoveEmptyEntries);
//If we have gone past the CURR record, then at to YTD Columns
if (ytdFlag)
{
row[6] = int.Parse(tokens[1]);
row[7] = decimal.Parse(tokens[1]);
}
//Otherwise we are still in the CURR section:
else
{
row[2] = int.Parse(tokens[1]);
row[3] = decimal.Parse(tokens[1]);
}
}
if (line.StartsWith(" OP"))
{
//Start at after OP then split the line into the 4 numbers
var tokens = line.Substring(3).Split(new [] { " "}, StringSplitOptions.RemoveEmptyEntries);
//If we have gone past the CURR record, then at to YTD Columns
if (ytdFlag)
{
row[8] = int.Parse(tokens[1]);
row[9] = decimal.Parse(tokens[1]);
}
//Otherwise we are still in the CURR section:
else
{
row[4] = int.Parse(tokens[1]);
row[5] = decimal.Parse(tokens[1]);
}
//After we have processed an OP record, we must set the YTD Flag to true.
//Doesn't matter if it is the YTD OP record, since the flag will be reset
//By the next line that starts with SERVICE CODE anyway
ytdFlag= true;
}
}
}
//Now that we have processed the file, we can write the data to a database
using (var sqlBulkCopy = new SqlBulkCopy("Your Connection String"))
{
sqlBulkCopy.DestinationTableName = "dbo.YourTable";
//If necessary add column mappings, but if your DataTable matches your database table
//then this is not required
sqlBulkCopy.WriteToServer(table);
}
This is a very quick example, far from the finished article, and I have done little or no testing, but it should give you the gist of how it could be done, and get you started on one possible solution.
It can definitely be cleaned up and refactored, but I have tried to make it as clear as possible what is going on, rather than trying to write the most efficient code ever. It should also (hopefully) demonstrate what a monumental pain this is to do, and very minor report changes things like an extra space be "OP" will break the whole thing.
So again, I would re-iterate, if you can get the data in a standard flat file format, with one line per record, you should. I do however appreciate that sometimes these things are out of your control, and I have had to write incredibly ugly import routines like this in the past, so I feel your pain if you can't get the data in a consumable format.

Adding x axis labels when using dojox.charting.DataSeries

I'm creating a Dojo line chart from a dojo.data.ItemFileReadStore using a dojox.charting.DataSeries. I'm using the third parameter (value) of the constructor of DataSeries to specify a method which will generate the points on the chart. e.g.
function formatLineGraphItem(store,item)
{
var o = {
x: graphIndex++,
y: store.getValue(item, "fileSize"),
};
return o;
}
The graphIndex is an integer which is incremented for every fileSize value. This gives me a line chart with the fileSize shown against a numeric count. This works fine.
What I'd like is to be able to specify the x axis label to use instead of the value of graphIndex i.e. the under lying data will still be 1,2,3,4 but the label will show text (in this case the time at which the file size was captured).
I can do this by passing in an array of labels into the x asis when I call chart.addAxis() but this requires me to know the the values before I iterate through the data. e.g.
var dataSeriesConfig = {query: {id: "*"}};
var xAxisLabels = [{text:"2011-11-20",value:1},{text:"2011-11-21",value:2},{text:"2011-11-22",value:3}];
var chart1 = new dojox.charting.Chart("chart1");
chart1.addPlot("default", {type: "Lines", tension: "4"});
chart1.addAxis("x", {labels: xAxisLabels});
chart1.addAxis("y", {vertical: true});
chart1.addSeries("Values", new dojox.charting.DataSeries(dataStore, dataSeriesConfig, formatLineGraphItem));
chart1.render();
The xAxisLabels array can be created by preparsing the dataSeries but it's not a very nice work around.
Does anyone have any ideas how the formatLineGraphItem method could be extended to provide the x axis labels. Or does anyone have any documentation on what values the object o can contain?
Thanks in advance!
This will take a unix timestamp, multiply the value by 1000 (so that it has microseconds for JavaScript, and then pass the value to dojo date to format it).
You shouldn't have any problems editing this to the format you need.
You provided examples that your dates are like "1", "2", "3", which is clearly wrong. Those aren't dates.. so this is the best you can do unless you edit your question.
chart1.addAxis("x",{
labelFunc: function(n){
if(isNaN(dojo.number.parse(n)) || dojo.number.parse(n) % 1 != 0){
return " ";
}
else {
// I am assuming that your timestamp needs to be multiplied by 1000.
var date = new Date(dojo.number.parse(n) * 1000);
return dojo.date.locale.format(date, {
selector: "date",
datePattern: "dd MMMM",
locale: "en"
});
}
},
maxLabelSize: 100
}

Couchdb views and many (thousands) document types

I'm studing CouchDB and I'm picturing a worst case scenario:
for each document type I need 3 view and this application can generate 10 thousands of document types.
With "document type" I mean the structure of the document.
After insertion of a new document, couchdb make 3*10K calls to view functions searching for right document type.
Is this true?
Is there a smart solution than make a database for each doc type?
Document example (assume that none documents have the same structure, in this example data is under different keys):
[
{
"_id":"1251888780.0",
"_rev":"1-582726400f3c9437259adef7888cbac0"
"type":'sensorX',
"value":{"ValueA":"123"}
},
{
"_id":"1251888780.0",
"_rev":"1-37259adef7888cbac06400f3c9458272"
"type":'sensorY',
"value":{"valueB":"456"}
},
{
"_id":"1251888780.0",
"_rev":"1-6400f3c945827237259adef7888cbac0"
"type":'sensorZ',
"value":{"valueC":"789"}
},
]
Views example (in this example only one per doc type)
"views":
{
"sensorX": {
"map": "function(doc) { if (doc.type == 'sensorX') emit(null, doc.valueA) }"
},
"sensorY": {
"map": "function(doc) { if (doc.type == 'sensorY') emit(null, doc.valueB) }"
},
"sensorZ": {
"map": "function(doc) { if (doc.type == 'sensorZ') emit(null, doc.valueC) }"
},
}
The results of the map() function in CouchDB is cached the first time you request the view for each new document. Let me explain with a quick illustration.
You insert 100 documents to CouchDB
You request the view. Now the 100 documents have the map() function run against them and the results cached.
You request the view again. The data is read from the indexed view data, no documents have to be re-mapped.
You insert 50 more documents
You request the view. The 50 new documents are mapped and merged into the index with the old 100 documents.
You request the view again. The data is read from the indexed view data, no documents have to be re-mapped.
I hope that makes sense. If you're concerned about a big load being generated when a user requests a view and lots of new documents have been added you could look at having your import process call the view (to re-map the new documents) and have the user request for the view include stale=ok.
The CouchDB book is a really good resource for information on CouchDB.
James has a great answer.
It looks like you are asking the question "what are the values of documents of type X?"
I think you can do that with one view:
function(doc) {
// _view/sensor_value
var val_names = { "sensorX": "valueA"
, "sensorY": "valueB"
, "sensorZ": "valueC"
};
var value_name = val_names[doc.type];
if(value_name) {
// e.g. "sensorX" -> "123"
// or "sensorZ" -> "789"
emit(doc.type, doc.value[value_name]);
}
}
Now, to get all values for sensorY, you query /db/_design/app/_view/sensor_value with a parameter ?key="sensorX". CouchDB will show all values for sensorX, which come from the document's value.valueA field. (For sensorY, it comes from value.valueB, etc.)
Future-proofing
If you might have new document types in the future, something more general might be better:
function(doc) {
if(doc.type && doc.value) {
emit(doc.type, doc.value);
}
}
That is very simple, and any document will work if it has a type and value field. Next, to get the valueA, valueB, etc. from the view, just do that on the client side.
If using the client is impossible, use a _list function.
function(head, req) {
// _list/sensor_val
//
start({'headers':{'Content-Type':'application/json'}});
// Updating this will *not* cause the map/reduce view to re-build.
var val_names = { "sensorX": "valueA"
, "sensorY": "valueB"
, "sensorZ": "valueC"
};
var row;
var doc_type, val_name, doc_val;
while(row = getRow()) {
doc_type = row.key;
val_name = val_names[doc_type];
doc_val = row.value[val_name];
send("Doc " + row.id + " is type " + doc_type + " and value " + doc_val);
}
}
Obviously use send() to send whichever format you prefer for the client (such as JSON).

Changing constraints on the fly

I have a dijit.form.NumberTextBox input field that starts out with these parms:
new dijit.form.NumberTextBox({
id: din1,
style: "width:60px",
constraints: {
places: 0,
pattern: '######'
}
},
din1);
Everything works great..My question is I would like to change 'places' and 'pattern' parms on the fly. So I wrote this to change 'places' and 'patterns' parms:
var myFldObj = dijit.byId(din1);
if (myFldObj) {
var myConstObj = myFldObj.attr('constraints');
if (myConstObj) {
myConstObj.places = 2;
myConstObj.pattern = '#####.0';
}
}
So, after I show the form again, I'd expect the entry field to allow 2 decimal places but the form still acts like places=0 and pattern='######'. When I check the values of 'places' and 'pattern' I get what I'd expect (2 and #####.0). My question:
Can you change these values on the fly??
OR
Do you have to destroy the original dijit object and recreate with new parms??
Thx!!
So, here is what worked for me:
First, I think this is a bug because an input field that starts out like
new dijit.form.NumberTextBox({
id: "fieldID",
style: "width:60px",
constraints: {
places: 0
}
},
"fieldID");
that is then changed on the fly with code like:
NOTE: ntbArry - Array of dijit.form.NumberTextBox objs tied to a html
input tag id.
for (var x=0;x < ntbArry.length;x++) {
var handle = ntbArry[x];
if (handle) {
handle.attr('constraints').places = 2;
handle.attr('constraints').pattern = '#####.0#';
}
}
Does not exhibit the same behavior as a field created this way (no constraints mods on the fly):
new dijit.form.NumberTextBox({
id: "fieldID",
style: "width: 60px",
constraints: {
places: 2,
pattern: '#####.0#'
}
},
"fieldID");
It's close in behavior but every time you type a decimal point, the error message pops up stating invalid entry. This message doesn't pop up when typing the decimal point on a field that was originally created with the constraints places=2 and pattern '#####.0#'.
So, to get original behavior I wanted:
fieldIDEvents is an array of dojo events tied to NumberTextBox fields.
Before continuing disconnect dojo events
for (var x=0;x < fieldIDEvents.length;x++) {
var handle = fieldIDEvents[x];
if (handle) {
dojo.disconnect(handle);
}
}
then destroy the NumberTextBox dojo objects
for (var x=0;x < ntbArry.length;x++) {
var handle = ntbArry[x];
if (handle) {
handle.destroy();
ntbArry[x] = null;
}
}
Next, place the input tag back into the html because it gets destroyed:
NOTE: tdtag and an id on a html td tag which should contain the input tag.
var fld1 = this.document.getElementById("tdtag");
if (fld1) {
//alert("\""+fld1.innerHTML+"\"");
fld1.innerHTML = "<input id=\"fieldID\">";
}
Now, create the NumberTextBox object again:
ntbArry[0] = new dijit.form.NumberTextBox({
id: "fieldID",
style: "width: 60px",
constraints: {
places: 2,
pattern: '#####.0#'
}
},
"fieldID");
It's a few extra steps but, at least I know this is what works for me..If I'm missing something basic, let me know, it's easy to miss the small details with this stuff.
I use Dojo 1.3 and I can see that dijit.form.NumberTextBox has no pattern and places properties, but has editOptions property. So I would try to change the constraints like this:
myConstObj.editOption.places = 2;