I need to paginate through all of the records from Hubspot API and I am getting stuck at offset pagination loop.
According to the Hubspot's API documentation, there is no "total records" path available in the response but instead "has-more" filed tells us if there are any more records we can pull from this portal.
two parameters that can be used for pagination are
vidOffset & has-more
here's what qliksense script looks like for custom pagination via rest connector.
LIB CONNECT TO 'HubSpot ';
// Action required: Implement the logic to retrieve the total records from the REST source and assign to the 'total' local variable.
Let total = 0;
Let totalfetched = 0;
Let startAt = 0;
Let pageSize = 100;
for startAt = 0 to total step pageSize
RestConnectorMasterTable:
SQL SELECT
"has-more",
"vid-offset",
"__KEY_root",
(SELECT
"addedAt",
"vid" AS "vid_u0",
"canonical-vid",
"portal-id",
"is-contact",
"profile-token",
"profile-url",
"__KEY_contacts",
"__FK_contacts",
(SELECT
"#Value",
"__FK_merged-vids"
FROM "merged-vids" FK "__FK_merged-vids" ArrayValueAlias "#Value"),
(SELECT
"__KEY_properties",
"__FK_properties",
(SELECT
"value",
"__FK_firstname"
FROM "firstname" FK "__FK_firstname"),
(SELECT
"value" AS "value_u0",
"__FK_lastmodifieddate"
FROM "lastmodifieddate" FK "__FK_lastmodifieddate"),
(SELECT
"value" AS "value_u1",
"__FK_company"
FROM "company" FK "__FK_company"),
(SELECT
"value" AS "value_u2",
"__FK_lastname"
FROM "lastname" FK "__FK_lastname")
FROM "properties" PK "__KEY_properties" FK "__FK_properties"),
(SELECT
"#Value" AS "#Value_u0",
"__FK_form-submissions"
FROM "form-submissions" FK "__FK_form-submissions" ArrayValueAlias "#Value_u0"),
(SELECT
"vid",
"saved-at-timestamp",
"deleted-changed-timestamp",
"__KEY_identity-profiles",
"__FK_identity-profiles",
(SELECT
"type",
"value" AS "value_u3",
"timestamp",
"is-primary",
"__FK_identities"
FROM "identities" FK "__FK_identities")
FROM "identity-profiles" PK "__KEY_identity-profiles" FK "__FK_identity-profiles"),
(SELECT
"#Value" AS "#Value_u1",
"__FK_merge-audits"
FROM "merge-audits" FK "__FK_merge-audits" ArrayValueAlias "#Value_u1")
FROM "contacts" PK "__KEY_contacts" FK "__FK_contacts")
FROM JSON (wrap on) "root" PK "__KEY_root"
WITH CONNECTION(Url "https://api.hubapi.com/contacts/v1/lists/all/contacts/all");
// Action required: change URL included in 'WITH CONNECTION' as needed to support pagination for the REST source.
// Please see the documentation for "Loading paged data."
NEXT startAt;
Need to understand how to set this up as per my API paramteres i.e. offset & hasmore property. How do i loop through all of the vidoffset values so that i can get all of the records until has-more becomes false?
Here's my json response
Please try recursive call to do you need to put your call in subroutine than check for has_more and if it is equal to True call subroutine again. Also Url parameter have to be updated every time with new vid-offset value. Here is example (tested it is working):
SUB getOnePage(vOffset)
LIB CONNECT TO [hubspot_api];
RestConnectorMasterTable:
SQL SELECT
(...)
FROM JSON (wrap on) "root" PK "__KEY_root"
WITH CONNECTION (Url "https://api.hubapi.com/contacts/v1/lists/all/contacts/all/?hapikey=YOURKEY=$(vOffset)");
LET vHasMore = Peek('has-more',-1);
LET vVidOffset = Peek('vid-offset',-1);
DROP Table root;
IF vHasMore = 'True' then
CALL getOnePage($(vVidOffset));
EndIf
EndSub
Because of repeated keys in every CALL we need to change settings in REST connector as follow:
Related
Using Microsoft Visual FoxPro 9, I have a custom function, "newid()", inside of the stored procedures for Main:
function newId
parameter thisdbf
regional keynm, newkey, cOldSelect, lDone
keynm=padr(upper(thisdbf),50)
cOldSelect=alias()
lDone=.f.
do while not lDone
select keyvalue from main!idkeys where keyname=keynm into array akey
if _tally=0
insert into main!idkeys (keyname) value (keynm)
loop
endif
newkey=akey+1
update main!idkeys set keyvalue=newkey where keyname=keynm and keyvalue=akey
if _tally=1
lDone=.t.
endif
enddo
if not empty(cOldSelect)
select &cOldSelect
else
select 0
endif
return newkey
This function is used to generate a new ID for records added to the database.
It is called as the default value:
I would like to call this newid() function and retrieve its returned value. When executing SELECT newid("TABLENAME"), the error is is thrown:
Invalid subscript reference
How can I call the newid() function and return the newkey in Visual FoxPro 9?
As an addition to what Stefan Wuebbe said,
You actually had your answer in your previous question here that you forgot to update.
From your previous question, as I understand you are coming from a T-SQL background. While in T-SQL (and in SQL generally) there is:
Select < anyVariableOrFunction >
that returns a single column, single row result, in VFP 'select' like that has another meaning:
Select < aliasName >
aliasName is an alias of a working area (or it could be number of a work area) and is used to change the 'current workarea'. When it was used in xBase languages like FoxPro (and dBase), those languages didn't yet meet ANSI-SQL if I am not wrong. Anyway, in VFP there are two Select, this one and SELECT—SQL which definitely requires a FROM clause.
VFP has direct access to variables and function calls though, through the use of = operator.
SELECT newid("TABLENAME")
in T-SQL, would be (you are just displaying the result):
? newid("TABLENAME")
To store it in a variable, you would do something like:
local lnId
lnId = newid("TABLENAME")
* do something with m.lnId
* Note the m. prefix, it is a built-in alias for memory variables
After having said all these, as per your code.
It looks like it has been written by a very old FoxPro programmer and I must admit I am seeing it the first time in my life that someone used "REGIONAL" keyword in VFP. It is from FoxPro 2.x days I know but I didn't see anyone use it up until now :) Anyway, that code doesn't seem to be robust enough in a multiuser environment, you might want to change it. VFP ships with a NewId sample code and below is the slightly modified version that I have been using in many locations and proved to be reliable:
Function NewID
Lparameters tcAlias,tnCount
Local lcAlias, lnOldArea, lcOldReprocess, lcTable, lnTagNo, lnNewValue, lnLastValue, lcOldSetDeleted
lnOldArea = Select()
lnOldReprocess = Set('REPROCESS')
* Uppercase Alias name
lcAlias = Upper(Iif(Parameters() = 0, Alias(), tcAlias))
* Lock reprocess - try once
Set Reprocess To 1
If !Used("IDS")
Use ids In 0
Endif
* If no entry yet create
If !Seek(lcAlias, "Ids", "tablename")
Insert Into ids (tablename, NextID) Values (lcAlias,0)
Endif
* Lock, increment id, unlock, return nextid value
Do While !Rlock('ids')
* Delay before next lock trial
lnStart = Seconds()
Do While Seconds()-lnStart < 0.01
Enddo
Enddo
lnLastValue = ids.NextID
lnNewValue = m.lnLastValue + Evl(m.tnCount,1)
*Try to query primary key tag for lcAlias
lcTable = Iif( Used(lcAlias),Dbf(lcAlias), Iif(File(lcAlias+'.dbf'),lcAlias,''))
lcTable = Evl(m.lcTable,m.lcAlias)
If !Empty(lcTable)
Use (lcTable) In 0 Again Alias '_GetPKKey_'
For m.lnTagNo=1 To Tagcount('','_GetPKKey_')
If Primary(m.lnTagNo,'_GetPKKey_')
m.lcOldSetDeleted = Set("Deleted")
Set Deleted Off
Select '_GetPKKey_'
Set Order To Tag (Tag(m.lnTagNo,'_GetPKKey_')) ;
In '_GetPKKey_' Descending
Locate
lnLastValue = Max(m.lnLastValue, Evaluate(Key(m.lnTagNo,'_GetPKKey_')))
lnNewValue = m.lnLastValue + Evl(m.tnCount,1)
If Upper(m.lcOldSetDeleted) == 'ON'
Set Deleted On
Endif
Exit
Endif
Endfor
Use In '_GetPKKey_'
Select ids
Endif
* Increment
Replace ids.NextID With m.lnNewValue In 'ids'
Unlock In 'ids'
Select (lnOldArea)
Set Reprocess To lnOldReprocess
Return ids.NextID
Endfunc
Note: If you use this, as I see from your code, you would need to change the "id table" name to idkeys, field names to keyname, keyvalue:
ids => idKeys
tablename => keyName
nextId => keyValue
Or in your database just create a new table with this code:
CREATE TABLE ids (TableName c(50), NextId i)
INDEX on TableName TAG TableName
When executing SELECT newid("TABLENAME")
The error: Invalid subscript reference is thrown
The SQL Select command in Vfp requires a From clause.
Running a procedure or a function can, or better usually needs to be done differently:
For example, in the IDE's Command Window you can do a
? newid("xy") && the function must be "in scope",
&& i.e in your case the database that contains the "Stored
&& Procedure" must have been opened in advance
&& or you store the function result in a variable
Local lnNextID
lnNextID = newid("xy")
Or you can use it in an SQL SELECT when you have a From alias
CREATE CURSOR placebo (col1 Int)
INSERT INTO placebo VALUES (8)
Select newid("xy") FROM placebo
I have a field in my postgres db called authorizedUserNumber. This field is set by default to 0, and does not auto-increment, because it is only asigned when a user has been fully onboarded.
Okay, so let's say a new user has been fully onboarded and I want to assign a unique number to the field authorizedUserNumber. In the event I have multiple servers running, I want to detect collisions of unique numbers in this field, so as to protect against race conditions.
I thought of defining authorizedUserNumber as a Sequelize unique field, and trying something like this:
// get current max authorizedUserNumber
let userWithMaxOnboardedauthorizedUserNumber = await connectors.usersClinical.findAll({
attributes: [
sequelize.fn('MAX', sequelize.col('authorizedUserNumber'))
],
});
let newLatestauthorizedUserNumber = userWithMaxOnboardedauthorizedUserNumber[0].authorizedUserNumber;
newLatestauthorizedUserNumber += 1;
let numAttempts = 0
let done = false;
while ((!done) && (numAttempts <= 50)){
try{
user = await updateUser(user.userId, {authorizedUserNumber: newLatestauthorizedUserNumber});
done = true;
}catch(e){
// a unique field will throw an error if you try to store a duplicate value to it
console.log(`Collision in assigning unique authorizedUserNumber. UserId: ${user.userId}`);
newLatestauthorizedUserNumber += 1;
numAttempts+= 1;
}
}
if (!done){
console.error(`Could not assign unique authorizedUserNumber. UserId: ${user.userId}`);
}
The problem with this code, is that if the field authorizedUserNumber is defined as unique, then I can't put a default value in it. So there's no way to have it be empty prior to having the correct value placed in it.
What's the best practice for dealing with this sort of situation?
UPDATE:
Thanks to #Belayer for providing the solution.
Here are some notes on how I implemented it in Sequelize/Postgres.
Sequelize, AFAICT, does not yet support sequences. So I used a raw query in Sequelize to create the sequence:
let sql = `
CREATE SEQUENCE authorizedUserNumber_seq
START WITH 1
INCREMENT BY 1;`
let result;
try{
result = await db.query(sql);
console.log(`sql code to create authorizedUserNumber_seq has been run successfully.`)
}
catch(e){
result = null;
console.error(`Error in creating authorizedUserNumber_seq.`)
}
Then when it's time to authorize the new user and assign a unique user number, I again use a raw query, with the following sql:
let sql = `UPDATE usersClinical
SET "authorizedUserNumber" = nextval('authorizedUserNumber_seq')
WHERE "userId" = '${user.userId}';`
Rather than defaulting to 0 just let the column be null when not set. Since the default is null there can be any number of them without violating a unique constraint. Then create a sequence for that column (do not set as column default). There is no requirement for a sequence to auto-increment, the nextval can be assigned when needed. Make the assignment from the sequence when the new user becomes fully on-boarded.
create table users ( id integer generated always as identity
, name text
, assignedid integer
, constraint assigned_uk unique (assignedid)
) ;
create sequence user_assigned_seq;
You can even make the assignment when user is created if desired. (see demo )
Instead of creating a unique constraint, you can create a unique index like this:
CREATE UNIQUE INDEX ON tab (nullif(authorizedUserNumber, 0));
this is my first question on stackOverflow.
I'm working with Corona and I'm having an issue accessing a SQLdb (I'm a bit of a SQL noob.)
I'm trying to access and return a value I've stored in the database.
Here's some code samples:
print("---------------- How I Create New Player Save Data")
local entry = [[CREATE TABLE IF NOT EXISTS playerData (key STRING PRIMARY KEY, content INTEGER);]]
db:exec(entry)
entry = [[INSERT INTO playerData VALUES ("LastLoginTime", 0);]]
db:exec( entry )
entry = [[INSERT INTO playerData VALUES ("Credits", 1000);]]
db:exec( entry )
entry = [[INSERT INTO playerData VALUES ("Level", 1);]]
db:exec( entry )
Now this function works, it will print everything in the db (i pass in 'dbName'):
--print all the table contents
for row in db:nrows("SELECT * FROM "..dbName) do
local text = row.key..": "..row.content
end
This doesn't work, it returns '0':
local grabCredits = "SELECT content FROM playerData WHERE key='Credits'"
local credits = db:exec(grabCredits)
print("-- value: "..credits)
Neither does this, also returns '0':
local grabCredits = "SELECT key FROM playerData WHERE content>=10"
local credits = db:exec(grabCredits)
print("-- value: "..credits)
I don't understand what I'm doing wrong. Maybe I need to use another function call on the db other than exec(). I realize I could iterate through the db every time I want to access a single entry, but that just seems inefficient.
Any help is very much appreciated.
If you want results, you must use some form of iterator. SQLite always returns rows for a query result.
This is similar to what I'm using to retrieve one result from my database and works well for me.
local query = "SELECT content FROM playerData WHERE key = 'Credits' LIMIT 1"
local queryResultTable = {}
local queryFunction = function(userData, numberOfColumns, columnValues, columnTitles)
for i = 1, numberOfColumns do
queryResultTable[columnTitles[i]] = columnValues[i]
end
end
db:exec(query, queryFunction)
for k,v in pairs(queryResultTable) do
print(k,v)
end
if ($node->taxonomy) {
$query = 'SELECT DISTINCT(t.nid), n.nid, n.title FROM {node} n INNER JOIN {term_node} t ON n.nid = t.nid WHERE n.nid != %d AND (';
$args = array($node->nid);
$tids = array();
foreach ($node->taxonomy as $term) {
$tids[] = 't.tid = %d';
$args[] = $term->tid;
}
$query .= implode(' OR ', $tids) . ')';
$result = db_query_range($query, $args, 0, 10);
while ($o = db_fetch_object($result)) {
echo l($o->title, 'node/' . $o->nid);
}
}
the code is from a drupal guru. . used to get the article's title under the same term in node.tpl.php, i have researched it two days, although know some part of it. the principle of the code i still don't know. expect someone can explain more details about it for me .many thanks.
Short version:
It gets the array of tags of the node, retrieves the first 10 nodes that use at least one of these tags and outputs a link for each of these 10 results.
Detailed version:
First of all, the variable "$node" is an object that contains the data about a specific node (e.g. a Page or Story node).
For example, "$node->title" would be the title of that node.
"$node->taxonomy" tests is that node is tagged (because if it has no tags, it cannot retrieve the other nodes using the same tag(s).
When there is one or several tags associated with that node/page/story, $node->taxonomy is an array .
Now about the SQL query:
"node" is the database table that stores the base fields (non-CCK) of every node.
"term_node" is the database table that contains the combination of tag (which is called a "taxonomy term") and node.
In both tables, "nid" is the "unique Node ID" (which is an internal autoincremented number). Because this column is in both tables, this is how the tables are joined together.
In "term_node", "tid" is the "unique Term ID" (which is also an internal autoincremented number).
The "node" table is aliased "n", therefore "n.nid" means "the Node ID stored in table node".
The "term_node" table is aliased "t", therefore "t.tid" means "the Term ID stored in table term_node".
The "foreach" loop goes thru the array of tags to extract the TermID of each tag used by the node in order to add it in the SQL query, and implode converts to a string.
The loop stores a piece of SQL query for each tag in variable $tids and stores the actual value in variable $args because Drupal database calls are safer when the arguments are passed separately from the SQL query: "%d" means "integer number".
"db_query_range" is a function that selects multiple rows in the database: here, "0 10" means "retrieve the first 10 results".
"db_fetch_object" in the "while" loop retrieves each result and stores it in the variable "$o", which is an object.
Therefore "$o->title" contains the value of the column "title" retrieved by the SQL query.
The function "l" is the drupal functin that creates the code for an HTML link: the first argument is the name of the link, the second argument is the drupal path: in Drupal, any node can be accessed by default using "www.yoursite.com/node/NodeID",
which is why it gives the path "node/123" (where 123 is the "Node ID").
This function is useful because it transparently handles custom paths, so if your node has a custom path to access it using "www.yoursite.com/my-great-page" instead, it will create a link to that page instead of "www.yoursite.com/node/123" automatically.
I wouldn't exactly call the guy who wrote this a guru, you could do this a lot prettier. Anyways what he does it create a query that looks like this:
SELECT DISTINCT(t.nid), n.nid, n.title FROM {node} n
INNER JOIN {term_node} t ON n.nid = t.nid
WHERE n.nid != %d
AND (t.tid = %d OR t.tid = %d OR ... t.tid = %d);
The end result is that he selects all the node ids and titles (only once) that share at least one term with the selected node, but isn't the node itself.
So the main program is in C#. Inserting new records into a VFP database table. It was taking too long to generate the next ID for the record via
select max(id)+1 from table
, so I put that code into a compile dll in VFP and am calling that COM object through C#.
The COM object returns the new ID in about 250ms. I then just do an update through OLEDB. The problem I am having is that after the COM object returns the newly inserted ID, I cannot immediately find it from C# via the OLEDB
select id form table where id = *newlyReturnedID*
returns 0 rows back. If I wait an unknown time period the query will return 1 row. I can only assume it returns 0 rows immediately because it has yet to add the newly minted ID into the index and therefore the select cannot find it.
Has anyone else ever run into something similar? If so, how did you handle it?
DD
Warning: your code is flawed in a multi-user environment. Two people could run the query at the same time and get the same ID. One of them will fail on the INSERT if the column has a primary or candidate key, which is a best practice for key fields.
My recommendation is to either have the ID be a auto-incrementing integer field (I'm not a fan of them), or even better, create a table of keys. Each record in the table is for a table that gets keys assigned. I use the a structure similar to this:
Structure for: countergenerator.dbf
Database Name: conferencereg.dbc
Long table name: countergenerator
Number of records: 0
Last updated: 11/08/2008
Memo file block size: 64
Code Page: 1252
Table Type: Visual FoxPro Table
Field Name Type Size Nulls Next Step Default
----------------------------------------------------------------------------------------------------------------
1 ccountergenerator_pk Character 36 N guid(36)
2 ckey Character (Binary) 50 Y
3 ivalue Integer 4 Y
4 mnote Memo 4 Y "Automatically created"
5 cuserid Character 30 Y
6 tupdated DateTime 8 Y DATETIME()
Index Tags:
1. Tag Name: PRIMARY
- Type: primary
- Key Expression: ccountergenerator_pk
- Filter: (nothing)
- Order: ascending
- Collate Sequence: machine
2. Tag Name: CKEY
- Type: regular
- Key Expression: lower(ckey)
- Filter: (nothing)
- Order: ascending
- Collate Sequence: machine
Now the code for the stored procedure in the DBC (or in another program) is this:
FUNCTION NextCounter(tcAlias)
LOCAL lcAlias, ;
lnNextValue, ;
lnOldReprocess, ;
lnOldArea
lnOldArea = SELECT()
IF PARAMETERS() < 1
lcAlias = ALIAS()
IF CURSORGETPROP("SOURCETYPE") = DB_SRCLOCALVIEW
*-- Attempt to get base table
lcAlias = LOWER(CURSORGETPROP("TABLES"))
lcAlias = SUBSTR(lcAlias, AT("!", lcAlias) + 1)
ENDIF
ELSE
lcAlias = LOWER(tcAlias)
ENDIF
lnOrderNumber = 0
lnOldReprocess = SET('REPROCESS')
*-- Lock until user presses Esc
SET REPROCESS TO AUTOMATIC
IF !USED("countergenerator")
USE EventManagement!countergenerator IN 0 SHARED ALIAS countergenerator
ENDIF
SELECT countergenerator
IF SEEK(LOWER(lcAlias), "countergenerator", "ckey")
IF RLOCK()
lnNextValue = countergenerator.iValue
REPLACE countergenerator.iValue WITH countergenerator.iValue + 1
UNLOCK
ENDIF
ELSE
* Create the new record with the starting value.
APPEND BLANK IN countergenerator
SCATTER MEMVAR MEMO
m.cKey = LOWER(lcAlias)
m.iValue = 1
m.mNote = "Automatically created by stored procedure."
m.tUpdated = DATETIME()
GATHER MEMVAR MEMO
IF RLOCK()
lnNextValue = countergenerator.iValue
REPLACE countergenerator.iValue WITH countergenerator.iValue + 1
UNLOCK
ENDIF
ENDIF
SELECT (lnOldArea)
SET REPROCESS TO lnOldReprocess
RETURN lnNextValue
ENDFUNC
The RLOCK() ensures there is no contention for the records and is fast enough to not have bottleneck the process. This is way safer than the approach you are currently taking.
Rick Schummer
VFP MVP
VFP needs to FLUSH its workareas.