Is there a way through Selenium to see the time that the first google analytics event was fired? I was able to retrieve the network calls, but I don't see a payload.
wd = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
wd.get("https://target.com")
test = wd.execute_script("var performance = window.performance || window.mozPerformance || window.msPerformance || window.webkitPerformance || {}; var network = performance.getEntries() || {}; return (network);")
for item in test:
print(item)
The output just shows a list of network calls, but no payload information.
Related
I have a flask app that is functioning to expectations, and I am now trying to add a message notification section to my page. The difficulty I am having is that the database changes I am trying to rely upon do not seem to be updating in a timely fashion.
The html code is elementary:
<ul id="out" cols="85" rows="14">
</ul><br><br>
<script type="text/javascript">
var ul = document.getElementById("out");
var eventSource = new EventSource("/stream_game_channel");
eventSource.onmessage = function(e) {
ul.innerHTML += e.data + '<br>';
}
</script>
Here is the msg write code that the second user is executing. I know the code block is run because the redis trigger is properly invoked:
msg_join = Messages(game_id=game_id[0],
type="gameStart",
msg_from=current_user.username,
msg_to="Everyone",
message=f'{current_user.username} has requested to join.')
db.session.add(msg_join)
db.session.commit()
channel = str(game_id[0]).zfill(5) + 'startGame'
session['channel'] = channel
date_time = datetime.utcnow().strftime("%Y/%m/%d %H:%M:%S")
redisChannel.set(channel, date_time)
Here is the flask stream code, which is correctly triggered by a new redis time, but when I pull the list of messages, the new message the the second user has added is not yet accessible:
#games.route('/stream_game_channel')
def stream_game_channel():
#stream_with_context
def eventStream():
channel = session.get('channel')
game_id = int(left(channel, 5))
cnt = 0
while cnt < 1000:
print(f'cnt = 0 process running from: {current_user.username}')
time.sleep(1)
ntime = redisChannel.get(channel)
if cnt == 0:
msgs = db.session.query(Messages).filter(Messages.game_id == game_id)
msg_list = [i.message for i in msgs]
cnt += 1
ltime = ntime
lmsg_list = msg_list
for i in msg_list:
yield "data: {}\n\n".format(i)
elif ntime != ltime:
print(f'cnt > 0 process running from: {current_user.username}')
time.sleep(3)
msgs = db.session.query(Messages).filter(Messages.game_id == game_id)
msg_list = [i.message for i in msgs]
new_messages = # need to write this code still
ltime = ntime
cnt += 1
yield "data: {}\n\n".format(msg_list[len(msg_list)-len(lmsg_list)])
return Response(eventStream(), mimetype="text/event-stream")
The syntactic error that I am running into is that the msg_list is exactly the same length (i.e the pushed new message does not get written when i expect it to). Strangely, the second user's session appears to be accessing this information because its stream correctly reflects the addition.
I am using an Amazon RDS MySQL database.
The solution was to utilize a db.session.commit() before my db.session.query(Messages).filter(...) even where no writes were pending. This enabled an immediate read from a different user session, and my code commenced to react to the change in message list length properly.
I have a strange issue where I have passed in parameters from a URL, into my Express server,
When I get the req.params.code & req.params.mode variables, they are different than what is passed in through the URL.
Allow me to show you...
Here is the Express code:
router.get('/verify/:user/:mode/:code', function(req,res){
console.log("STARTING VERIFICATION");
var code = req.params.code;
console.log('code: ' + code);
var user = req.params.user;
console.log('user: ' + user);
var mode = req.params.mode;
console.log('mode: ' + mode);
console.log('req.params: ' + JSON.stringify(req.params));
var regex = new RegExp(["^", req.params.user, "$"].join(""), "i");
console.log('REGEX: ' + regex);
var verified = false;
console.log('req.params: ' + req.params);
console.log('req.body: ' + req.body);
console.log("rx: "+ regex);
console.log('req.params.code: ' + req.params.code);
console.log('req.params.user: ' + req.params.user);
etc... etc... etc...
Here is the output in the console:
STARTING VERIFICATION
code: background-cycler.js
user: admin
mode: js
req.params: {"user":"admin","mode":"js","code":"background-cycler.js"}
REGEX: /^admin$/i
req.params: [object Object]
req.body: [object Object]
rx: /^admin$/i
req.params.code: background-cycler.js
req.params.user: admin
Here is the URL that is passed into the browser:
https://examplesite.com/verify/admin/sms/9484
I want to say that this code worked prior to dusting it off and moving an instance to google's cloud compute...
As you can see, the parameters passed in to the verify, code should be 9484 and mode should be sms. Instead i'm getting an unintended js filename, and a js mode instead.
UPDATE: As requested I added this within the Express route function:
console.log(req.originalUrl);
and I get this result:
/verify/admin/js/background-cycler.js
I can verify the URL that sent this was:
https://examplesite.com/verify/admin/sms/9484
I'm trying to pull data from the Crunchbase Open Data Map to a Google Spreadsheet. I'm following Ben Collins's script but it no longer works since the upgrade from v3 to v3.1. Anyone had any luck modifying the script for success?
var USER_KEY = 'insert your API key in here';
// function to retrive organizations data
function getCrunchbaseOrgs() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheetByName('Organizations');
var query = sheet.getRange(3,2).getValue();
// URL and params for the Crunchbase API
var url = 'https://api.crunchbase.com/v/3/odm-organizations?query=' + encodeURI(query) + '&user_key=' + USER_KEY;
var json = getCrunchbaseData(url,query);
if (json[0] === "Error:") {
// deal with error with fetch operation
sheet.getRange(5,1,sheet.getLastRow(),2).clearContent();
sheet.getRange(6,1,1,2).setValues([json]);
}
else {
if (json[0] !== 200) {
// deal with error from api
sheet.getRange(5,1,sheet.getLastRow(),2).clearContent();
sheet.getRange(6,1,1,2).setValues([["Error, server returned code:",json[0]]]);
}
else {
// correct data comes back, filter down to match the name of the entity
var data = json[1].data.items.filter(function(item) {
return item.properties.name == query;
})[0].properties;
// parse into array for Google Sheet
var outputData = [
["Name",data.name],
["Homepage",data.homepage_url],
["Type",data.primary_role],
["Short description",data.short_description],
["Country",data.country_code],
["Region",data.region_name],
["City name",data.city_name],
["Blog url",data.blog_url],
["Facebook",data.facebook_url],
["Linkedin",data.linkedin_url],
["Twitter",data.twitter_url],
["Crunchbase URL","https://www.crunchbase.com/" + data.web_path]
];
// clear any old data
sheet.getRange(5,1,sheet.getLastRow(),2).clearContent();
// insert new data
sheet.getRange(6,1,12,2).setValues(outputData);
// add image with formula and format that row
sheet.getRange(5,2).setFormula('=image("' + data.profile_image_url + '",4,50,50)').setHorizontalAlignment("center");
sheet.setRowHeight(5,60);
}
}
}
This code no longer pulls data as expected.
I couldn't confirm about the error messages when you ran the script. So I would like to show about the clear difference point. It seems that the endpoint was changed from https://api.crunchbase.com/v/3/ to https://api.crunchbase.com/v3.1/. So how about this modification?
From :
var url = 'https://api.crunchbase.com/v/3/odm-organizations?query=' + encodeURI(query) + '&user_key=' + USER_KEY;
To :
var url = 'https://api.crunchbase.com/v3.1/odm-organizations?query=' + encodeURI(query) + '&user_key=' + USER_KEY;
Note :
From your script, I couldn't also find query. So if the script doesn't work even when you modified the endpoint, please confirm about it. You can see the detail of API v3 Compared to API v3.1 is here.
References :
API v3 Compared to API v3.1
Using the API
If this was not useful for you, I'm sorry.
UPDATE 1: I've created a GIST with actual running code in a test jig to show exactly what I'm running up against. I've included working bot tokens (to a throw-away bot) and access to a telegram chat that the bot is already in, in case anyone wants to take a quick peek. It's
https://gist.github.com/pleasantone/59efe5f9d7f0bf1259afa0c1ae5a05fe
UPDATE 2: I've looked at the following articles for answers already (and a ton more):
https://github.com/francois2metz/html5-formdata/blob/master/formdata.js
PhantomJS - Upload a file without submitting a form
https://groups.google.com/forum/#!topic/casperjs/CHq3ZndjV0k
How to instantiate a File object in JavaScript?
How to create a File object from binary data in JavaScript
I've got a program written in casperjs (phantomjs) that successfully sends messages to Telegram via the BOT API, but I'm pulling my hair out trying to figure out how to send up a photo.
I can access my photo either as a file, off the local filesystem, or I've already got it as a base64 encoded string (it's a casper screen capture).
I know my photo is good, because I can post it via CURL using:
curl -X POST "https://api.telegram.org/bot<token>/sendPhoto" -F chat_id=<id> -F photo=#/tmp/photo.png
I know my code for connecting to the bot api from within capserjs is working, as I can do a sendMessage, just not a sendPhoto.
function sendMultipartResponse(url, params) {
var boundary = '-------------------' + Math.floor(Math.random() * Math.pow(10, 8));
var content = [];
for (var index in params) {
content.push('--' + boundary + '\r\n');
var mimeHeader = 'Content-Disposition: form-data; name="' + index + '";';
if (params[index].filename)
mimeHeader += ' filename="' + params[index].filename + '";';
content.push(mimeHeader + '\r\n');
if (params[index].type)
content.push('Content-Type: ' + params[index].type + '\r\n');
var data = params[index].content || params[index];
// if (data.length !== undefined)
// content.push('Content-Length: ' + data.length + '\r\n');
content.push('' + '\r\n');
content.push(data + '\r\n');
};
content.push('--' + boundary + '--' + '\r\n');
utils.dump(content);
var xhr = new XMLHttpRequest();
xhr.open("POST", url, false);
if (true) {
/*
* Heck, try making the whole thing a Blob to avoid string conversions
*/
body = new Blob(content, {type: "multipart/form-data; boundary=" + boundary});
utils.dump(body);
} else {
/*
* this didn't work either, but both work perfectly for sendMessage
*/
body = content.join('');
xhr.setRequestHeader("Content-Type", "multipart/form-data; boundary=" + boundary);
// xhr.setRequestHeader("Content-Length", body.length);
}
xhr.send(body);
casper.log(xhr.responseText, 'error');
};
Again, this is in a CASPERJS environment, not a nodejs environment, so I don't have things like fs.createReadableStream or the File() constructor.
I have a Google Apps Script function used for setting up accounts for new employees in our Google Apps domain.
The first thing it does is makes calls to the Google Admin Settings API and retrieves the currentNumberOfUsers and maximumNumberOfUsers, so it can see if there are available seats (otherwise a subsequent step where the user is created using the Admin SDK Directory API would fail).
It's been working fine until recently when our domain had to migrate from Postini to Google Vault for email archiving.
Before the migration, when creating a Google Apps user using the Admin SDK Directory API, it would increment the currentNumberOfUsers by 1 and the new user account user would automatically have access to all Google Apps services.
Now after the migration, when creating a Google Apps user, they aren't automatically assigned a "license," so I modified my script to use the Enterprise License Manager API and now it assigns a "Google-Apps-For-Business" license. That works fine.
However, the currentNumberOfUsers is now different from the number of assigned licenses, and "Google-Apps-For-Business" is only one of several different types of licenses available.
I can get the current number of assigned "Google-Apps-For-Business" licenses by running this:
var currentXml = AdminLicenseManager.LicenseAssignments.listForProductAndSku('Google-Apps', 'Google-Apps-For-Business', 'domain.com', {maxResults: 1000});
var current = currentXml.items.toString().match(/\/sku\/Google-Apps-For-Business\/user\//g).length;
But the number that produces is different from currentNumberOfUsers.
All I really need to do now is get the maximum number of owned "Google-Apps-For-Business" licenses so the new employee setup script can determine whether there are any available.
I checked the API Reference documentation for the following APIs but...
Enterprise License Manager API → Doesn't have a method for getting the maximum or available number of licenses.
Google Admin Settings API → Doesn't deal with licenses, only "users."
Admin SDK Directory API User resource → Doesn't deal with licenses.
Google Apps Reseller API → This API seems to have what I need, but it's only for Reseller accounts.
I know I can program my new employee setup script to just have a try/catch seeing if it would be able to create the user and assign the license, and end the script execution gracefully if it can't, but that doesn't seem efficient.
Also, part of the old script was that if there were less than X seats available, it would email me a heads-up to order more. I can program a loop that attempts to repeatedly create dummy users and assign them licenses and count the number of times it can do that before it fails, then delete all the dummy users, but, once again, that's not efficient at all.
Any ideas?
Update 3/11/2020: Since the Admin Settings API had shut down a few years ago I've been using the Enterprise License Manager API to get the current number of used licenses, like this:
function getCurrentNumberOfUsedGoogleLicenses(skuId) {
var success = false, error = null, count = 0;
var adminEmail = 'admin#domain.com';
var gSuiteDomain = adminEmail.split('#')[1];
// for more information on the domain-wide delegation:
// https://developers.google.com/identity/protocols/OAuth2ServiceAccount#delegatingauthority
// the getDomainWideDelegationService() function uses this:
// https://github.com/gsuitedevs/apps-script-oauth2
var service = getDomainWideDelegationService('EnterpriseLicenseManager: ', 'https://www.googleapis.com/auth/apps.licensing', adminEmail);
if (skuId == 'Google-Apps-Unlimited') var productId = 'Google-Apps';
else return { success: success, error: "Unsupported skuId", count: count };
var requestBody = {};
requestBody.headers = {'Authorization': 'Bearer ' + service.getAccessToken()};
requestBody.method = "GET";
requestBody.muteHttpExceptions = false;
var data, pageToken, pageTokenString;
var maxAttempts = 5;
var currentAttempts = 0;
var pauseBetweenAttemptsSeconds = 3;
loopThroughPages:
do {
if (typeof pageToken === 'undefined') pageTokenString = "";
else pageTokenString = "&pageToken=" + encodeURIComponent(pageToken);
var url = 'https://www.googleapis.com/apps/licensing/v1/product/' + productId + '/sku/' + skuId + '/users?maxResults=1000&customerId=' + gSuiteDomain + pageTokenString;
try {
currentAttempts++;
var response = UrlFetchApp.fetch(url, requestBody);
var result = JSON.parse(response.getContentText());
if (result.items) {
var licenseAssignments = result.items;
var licenseAssignmentsString = '';
for (var i = 0; i < licenseAssignments.length; i++) {
licenseAssignmentsString += JSON.stringify(licenseAssignments[i]);
}
if (skuId == 'Google-Apps-Unlimited') count += licenseAssignmentsString.match(/\/sku\/Google-Apps-Unlimited\/user\//g).length;
currentAttempts = 0; // reset currentAttempts before the next page
}
} catch(e) {
error = "Error: " + e.message;
if (currentAttempts >= maxAttempts) {
error = 'Exceeded ' + maxAttempts + ' attempts to get license count: ' + error;
break loopThroughPages;
}
} // end of try catch
if (result) pageToken = result.nextPageToken;
} while (pageToken);
if (!error) success = true;
return { success: success, error: error, count: count };
}
However, there still does not appear to be a way to get the maximum number available to the domain using this API.
Use CustomerUsageReports.
jay0lee is kind enough to provide the GAM source code in Python. I crudely modified the doGetCustomerInfo() function into Apps Script thusly:
function getNumberOfLicenses() {
var tryDate = new Date();
var dateString = tryDate.getFullYear().toString() + "-" + (tryDate.getMonth() + 1).toString() + "-" + tryDate.getDate().toString();
while (true) {
try {
var response = AdminReports.CustomerUsageReports.get(dateString,{parameters : "accounts:gsuite_basic_total_licenses,accounts:gsuite_basic_used_licenses"});
break;
} catch(e) {
//Logger.log(e.warnings.toString());
tryDate.setDate(tryDate.getDate()-1);
dateString = tryDate.getFullYear().toString() + "-" + (tryDate.getMonth() + 1).toString() + "-" + tryDate.getDate().toString();
continue;
}
};
var availLicenseCount = response.usageReports[0].parameters[0].intValue;
var usedLicenseCount = response.usageReports[0].parameters[1].intValue;
Logger.log("Available licenses:" + availLicenseCount.toString());
Logger.log("Used licenses:" + usedLicenseCount.toString());
return availLicenseCount;
}
I would recommend exploring GAM which is a tool that gives command line access to the administration functions of your domain.