My application needs to manage 3 types of events: creation, edit and deletion.
Can I manage all these events with just one ReadSideProcessor?
Is there any particular order I should prepare the statements in the prepare method?
Yes, ReadSideProcessor.defineEventHandlers accept a builder which can hold multiple events.
Since all events are unique, the order in which to define them is no important.
Think about is as a hashmap(event, view-store logic)
see below,
#Override
public EventHandlers defineEventHandlers(EventHandlersBuilder builder) {
// when Account created, insert account table;
builder.setEventHandler(TransactionEvent.AccountCreatedEvent.class, (ev, offset) -> {
System.out.println("offset ->" + offset);
BoundStatement st = writeAccount.bind()
.setString("account_id", ev.id)
.setString("name", ev.name);
BoundStatement stOffset = writeOffset.bind(offset);
return completedStatements(Arrays.asList(st, stOffset));
});
// when Deposit, insert history and update balance
builder.setEventHandler(TransactionEvent.MoneyDepositedEvent.class, (ev, offset) -> {
System.out.println("offset ->" + offset);
BoundStatement historyInsert = writeHistory.bind()
.setString("account_id", ev.id)
.setLong("amount", ev.amount)
.setString("type", "DEPOSIT")
.setTimestamp("at", toTimestamp(offset));
BoundStatement accountUpdate = updateAccount.bind()
.setString("account_id", ev.id)
.setLong("balance", ev.balance + ev.amount);
return completedStatements(Arrays.asList(historyInsert, accountUpdate, writeOffset.bind(offset)));
});
// when Withdrawal, insert history and update balance
builder.setEventHandler(TransactionEvent.MoneyWithdrawnEvent.class, (ev, offset) -> {
System.out.println("offset ->" + offset);
BoundStatement historyInsert = writeHistory.bind()
.setString("account_id", ev.id)
.setLong("amount", ev.amount)
.setString("type", "WITHDRAWAL")
.setTimestamp("at", toTimestamp(offset));
BoundStatement accountUpdate = updateAccount.bind()
.setString("account_id", ev.id)
.setLong("balance", ev.balance - ev.amount);
return completedStatements(Arrays.asList(historyInsert, accountUpdate, writeOffset.bind(offset)));
});
return builder.build();
}
https://github.com/khazzz/lagomk
Look at blog-impl project
BlogEventProcessor class.
Related
I have a problem with retrieving a data from bigquery table inside DoFn. I can't find example to extract values from TypedRead.
This is a simplified pipeline. I would like to check does record with target SSN exists or not in bigquery table. The target SSN will be received via pubsub in real pipeline, I have replaced it with array of strings.
final BigQueryIoTestOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().as(BigQueryIoTestOptions.class);
final List<String> SSNs = Arrays.asList("775-89-3939");
Pipeline p = Pipeline.create(options);
PCollection<String> ssnCollection = p.apply("GetSSNParams", Create.of(SSNs)).setCoder(StringUtf8Coder.of());
ssnCollection.apply("SelectFromBQ", ParDo.of(new DoFn<String, TypedRead<TableRow>>() {
#ProcessElement
public void processElement(ProcessContext c) throws Exception {
TypedRead<TableRow> tr =
BigQueryIO.readTableRows()
.fromQuery("SELECT pid19PatientSSN FROM dataset.table where pid19PatientSSN = '" + c.element() + "' LIMIT 1");
c.output(tr);
}
}))
.apply("ParseResponseFromBigQuery", ParDo.of(new DoFn<TypedRead<TableRow>, Void>() {
#ProcessElement
public void processElement(ProcessContext c) throws Exception {
System.out.println(c.element().toString());
}
}));
p.run();
Big query returns PCollection only, we can get the result as entry set like the below example or we can serialize to objects as well like mentioned here
If you want to query from BigQuery middle of your pipeline use BigQuery instead of BigQueryIO like mentioned here
BigQueryIO Example:
PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
Pipeline pipeline = Pipeline.create(options);
PCollection<TableRow> result = pipeline.apply(BigQueryIO.readTableRows()
.fromQuery("SELECT id, name FROM [project-test:test_data.test] LIMIT 1"));
result.apply(MapElements.via(new SimpleFunction<TableRow, Void>() {
#Override
public Void apply(TableRow obj) {
System.out.println("***" + obj);
obj.entrySet().forEach(
(k)-> {
System.out.println(k.getKey() + " :" + k.getValue());
}
);
return null;
}
}));
pipeline.run().waitUntilFinish();
BigQuery Example:
// [START bigquery_simple_app_client]
BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();
// [END bigquery_simple_app_client]
// [START bigquery_simple_app_query]
QueryJobConfiguration queryConfig =
QueryJobConfiguration.newBuilder(
"SELECT "
+ "CONCAT('https://stackoverflow.com/questions/', CAST(id as STRING)) as url, "
+ "view_count "
+ "FROM `bigquery-public-data.stackoverflow.posts_questions` "
+ "WHERE tags like '%google-bigquery%' "
+ "ORDER BY favorite_count DESC LIMIT 10")
// Use standard SQL syntax for queries.
// See: https://cloud.google.com/bigquery/sql-reference/
.setUseLegacySql(false)
.build();
// Create a job ID so that we can safely retry.
JobId jobId = JobId.of(UUID.randomUUID().toString());
Job queryJob = bigquery.create(JobInfo.newBuilder(queryConfig).setJobId(jobId).build());
// Wait for the query to complete.
queryJob = queryJob.waitFor();
// Check for errors
if (queryJob == null) {
throw new RuntimeException("Job no longer exists");
} else if (queryJob.getStatus().getError() != null) {
// You can also look at queryJob.getStatus().getExecutionErrors() for all
// errors, not just the latest one.
throw new RuntimeException(queryJob.getStatus().getError().toString());
}
// [END bigquery_simple_app_query]
// [START bigquery_simple_app_print]
// Get the results.
QueryResponse response = bigquery.getQueryResults(jobId);
TableResult result = queryJob.getQueryResults();
// Print all pages of the results.
for (FieldValueList row : result.iterateAll()) {
String url = row.get("url").getStringValue();
long viewCount = row.get("view_count").getLongValue();
System.out.printf("url: %s views: %d%n", url, viewCount);
}
// [END bigquery_simple_app_print]
I'm very new at coding and I'm having some issues. I'd like to display the followers of followers of ..... of followers of some specific users in Twitter. I have coded this and I can set a limit for the depth. But, while running the code with a small sample, I saw that I run into the same users again and my code re-display the followers of these users. How can I avoid this and skip to the next user? You can find my code below:
By the way, while running my code, I encounter with a 401 error. In the list I'm working on, there's a private user, and when my code catches that user, it stops. Additionally, how can I deal with this issue? I'd like to skip such users and prevent my code to stop.
Thank you for your help in advance!
PS: I know that I'll encounter with a 429 error working with a large sample. After fixing these issues, I'm planning to review relevant discussions to deal with.
public class mainJava {
public static Twitter twitter = buildConfiguration.getTwitter();
public static void main(String[] args) throws Exception {
ArrayList<String> rootUserIDs = new ArrayList<String>();
Scanner s = new Scanner(new File("C:\\Users\\ecemb\\Desktop\\rootusers1.txt"));
while (s.hasNextLine()) {
rootUserIDs.add(s.nextLine());
}
s.close();
for (String rootUserID : rootUserIDs) {
User rootUser = twitter.showUser(rootUserID);
List<User> userList = getFollowers(rootUser, 0);
}
}
public static List<User> getFollowers(User parent, int depth) throws Exception {
List<User> userList = new ArrayList<User>();
if (depth == 2) {
return userList;
}
IDs followerIDs = twitter.getFollowersIDs(parent.getScreenName(), -1);
long[] ids = followerIDs.getIDs();
for (long id : ids) {
twitter4j.User child = twitter.showUser(id);
userList.add(child);
getFollowers(child, depth + 1);
System.out.println(depth + "th user: " + parent.getScreenName() + " Follower: " + child.getScreenName());
}
return userList;
}
}
I guess graph search algorithms can be implemented for this particular issue. I chose Breadth First Search algorithm because visiting root user's followers at first would be better. You can check this link to additional information about algorithm.
Here is my implementation for your problem:
public List<User> getFollowers(User parent, int startDepth, int finalDepth) {
List<User> userList = new ArrayList<User>();
Queue<Long> queue = new LinkedList<Long>();
HashMap<Long, Integer> discoveredUserId = new HashMap<Long, Integer>();
try {
queue.add(parent.getId());
discoveredUserId.put(parent.getId(), 0);
while (!queue.isEmpty()) {
long userId = queue.remove();
int discoveredDepth = discoveredUserId.get(userId);
if (discoveredDepth == finalDepth) {
continue;
}
User user = twitter.showUser(userId);
handleRateLimit(user.getRateLimitStatus());
if (user.isProtected()) {
System.out.println(user.getScreenName() + "'s account is protected. Can't access followers.");
continue;
}
IDs followerIDs = null;
followerIDs = twitter.getFollowersIDs(user.getScreenName(), -1);
handleRateLimit(followerIDs.getRateLimitStatus());
long[] ids = followerIDs.getIDs();
for (int i = 0; i < ids.length; i++) {
if (!discoveredUserId.containsKey(ids[i])) {
discoveredUserId.put(ids[i], discoveredDepth + 1);
User child = twitter.showUser(ids[i]);
handleRateLimit(child.getRateLimitStatus());
userList.add(child);
if (discoveredDepth >= startDepth && discoveredDepth < finalDepth) {
System.out.println(discoveredDepth + ". user: " + user.getScreenName() + " has " + user.getFollowersCount() + " follower(s) " + (i + 1) + ". Follower: " + child.getScreenName());
}
queue.add(ids[i]);
} else {//prints to console but does not check followers. Just for data consistency
User child = twitter.showUser(ids[i]);
handleRateLimit(child.getRateLimitStatus());
if (discoveredDepth >= startDepth && discoveredDepth < finalDepth) {
System.out.println(discoveredDepth + ". user: " + user.getScreenName() + " has " + user.getFollowersCount() + " follower(s) " + (i + 1) + ". Follower: " + child.getScreenName());
}
}
}
}
} catch (TwitterException e) {
e.printStackTrace();
}
return userList;
}
//There definitely are more methods for handling rate limits but this worked for me well
private void handleRateLimit(RateLimitStatus rateLimitStatus) {
//throws NPE here sometimes so I guess it is because rateLimitStatus can be null and add this conditional expression
if (rateLimitStatus != null) {
int remaining = rateLimitStatus.getRemaining();
int resetTime = rateLimitStatus.getSecondsUntilReset();
int sleep = 0;
if (remaining == 0) {
sleep = resetTime + 1; //adding 1 more second
} else {
sleep = (resetTime / remaining) + 1; //adding 1 more second
}
try {
Thread.sleep(sleep * 1000 > 0 ? sleep * 1000 : 0);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
in this code HashMap<Long, Integer> discoveredUserId is used to prevent program checking same users repeatedly and storing in which depth we faced with this user.
and for private users, there is isProtected() method in twitter4j library.
Hope this implementation helps.
I am performing a POC to check Streaming API stability, POC is as follows
Program 1 : subscribe to pushtopic created against Account object
Program 2 : create, update & delete single record after every 10 min interval
Both this programs were kept running for more than 12 hours (left overnight), after that I verified if all notification are received or not and found that after sometime (in this case it was nearly ~ 2 hours 45 min ) no notification were received, I repeated this twice and both case it stops getting notification after sometime.
Test code used
Streaming API client (using EMP connector)
public class SFPoc {
static Long count = 0L;
static Long Leadcount = 0L;
public static void main(String[] argv) throws Exception {
String userName = "<user_name>";
String password = "<pwd>";
String pushTopicName = "/topic/AccountPT";
String pushTopicNameLead = "/topic/Leadwhere";
long replayFrom = EmpConnector.REPLAY_FROM_EARLIEST;
String securityToken = "<token>";
BayeuxParameters custom = getBayeuxParamWithSpecifiedAPIVersion("37.0");
BayeuxParameters params = null;
try {
params = login(userName, password + securityToken, custom);
} catch (Exception e) {
e.printStackTrace();
}
Consumer<Map<String, Object>> consumer = event -> System.out.println(String.format("Received:\n%s ** Recieved at %s, event count total %s", event, LocalDateTime.now() , ++count));
Consumer<Map<String, Object>> consumerLead = event -> System.out.println(String.format("****** LEADS ***** Received:\n%s ** Recieved at %s, event count total %s", event, LocalDateTime.now() , ++Leadcount));
EmpConnector connector = new EmpConnector(params);
connector.start().get(10, TimeUnit.SECONDS);
TopicSubscription subscription = connector.subscribe(pushTopicName, replayFrom, consumer).get(10, TimeUnit.SECONDS);
TopicSubscription subscriptionLead = connector.subscribe(pushTopicNameLead, replayFrom, consumerLead).get(10, TimeUnit.SECONDS);
System.out.println(String.format("Subscribed: %s", subscription));
System.out.println(String.format("Subscribed: %s", subscriptionLead));
}
private static BayeuxParameters getBayeuxParamWithSpecifiedAPIVersion(String apiVersion) {
BayeuxParameters params = new BayeuxParameters() {
#Override
public String version() {
return apiVersion;
}
#Override
public String bearerToken() {
return null;
}
};
return params;
}
}
Code which is doing record create/update/delete periodically to generate events
import com.sforce.soap.enterprise.*;
import com.sforce.soap.enterprise.Error;
import com.sforce.soap.enterprise.sobject.Account;
import com.sforce.soap.enterprise.sobject.Contact;
import com.sforce.ws.ConnectionException;
import com.sforce.ws.ConnectorConfig;
import java.time.LocalDateTime;
public class SFDCDataAdjustment {
static final String USERNAME = "<username>";
static final String PASSWORD = "<pwd&securitytoken>";
static EnterpriseConnection connection;
static Long count = 0L;
public static void main(String[] args) {
ConnectorConfig config = new ConnectorConfig();
config.setUsername(USERNAME);
config.setPassword(PASSWORD);
//config.setTraceMessage(true);
try {
connection = Connector.newConnection(config);
// display some current settings
System.out.println("Auth EndPoint: "+config.getAuthEndpoint());
System.out.println("Service EndPoint: "+config.getServiceEndpoint());
System.out.println("Username: "+config.getUsername());
System.out.println("SessionId: "+config.getSessionId());
// run the different examples
while (true) {
createAccounts();
updateAccounts();
deleteAccounts();
Thread.sleep(1 * 10 * 60 * 1000);
}
} catch (ConnectionException e1) {
e1.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
// queries and displays the 5 newest contacts
private static void queryContacts() {
System.out.println("Querying for the 5 newest Contacts...");
try {
// query for the 5 newest contacts
QueryResult queryResults = connection.query("SELECT Id, FirstName, LastName, Account.Name " +
"FROM Contact WHERE AccountId != NULL ORDER BY CreatedDate DESC LIMIT 5");
if (queryResults.getSize() > 0) {
for (int i=0;i<queryResults.getRecords().length;i++) {
// cast the SObject to a strongly-typed Contact
Contact c = (Contact)queryResults.getRecords()[i];
System.out.println("Id: " + c.getId() + " - Name: "+c.getFirstName()+" "+
c.getLastName()+" - Account: "+c.getAccount().getName());
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
// create 5 test Accounts
private static void createAccounts() {
System.out.println("Creating a new test Account...");
Account[] records = new Account[1];
try {
// create 5 test accounts
for (int i=0;i<1;i++) {
Account a = new Account();
a.setName("OptyAccount "+i);
records[i] = a;
}
// create the records in Salesforce.com
SaveResult[] saveResults = connection.create(records);
// check the returned results for any errors
for (int i=0; i< saveResults.length; i++) {
if (saveResults[i].isSuccess()) {
System.out.println(i+". Successfully created record - Id: " + saveResults[i].getId() + "At " + LocalDateTime.now());
System.out.println("************Event Count************" + ++count);
} else {
Error[] errors = saveResults[i].getErrors();
for (int j=0; j< errors.length; j++) {
System.out.println("ERROR creating record: " + errors[j].getMessage());
}
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
// updates the 5 newly created Accounts
private static void updateAccounts() {
System.out.println("Update a new test Accounts...");
Account[] records = new Account[1];
try {
QueryResult queryResults = connection.query("SELECT Id, Name FROM Account ORDER BY " +
"CreatedDate DESC LIMIT 1");
if (queryResults.getSize() > 0) {
for (int i=0;i<queryResults.getRecords().length;i++) {
// cast the SObject to a strongly-typed Account
Account a = (Account)queryResults.getRecords()[i];
System.out.println("Updating Id: " + a.getId() + " - Name: "+a.getName());
// modify the name of the Account
a.setName(a.getName()+" -- UPDATED");
records[i] = a;
}
}
// update the records in Salesforce.com
SaveResult[] saveResults = connection.update(records);
// check the returned results for any errors
for (int i=0; i< saveResults.length; i++) {
if (saveResults[i].isSuccess()) {
System.out.println(i+". Successfully updated record - Id: " + saveResults[i].getId() + "At " + LocalDateTime.now());
System.out.println("************Event Count************" + ++count);
} else {
Error[] errors = saveResults[i].getErrors();
for (int j=0; j< errors.length; j++) {
System.out.println("ERROR updating record: " + errors[j].getMessage());
}
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
// delete the 5 newly created Account
private static void deleteAccounts() {
System.out.println("Deleting new test Accounts...");
String[] ids = new String[1];
try {
QueryResult queryResults = connection.query("SELECT Id, Name FROM Account ORDER BY " +
"CreatedDate DESC LIMIT 1");
if (queryResults.getSize() > 0) {
for (int i=0;i<queryResults.getRecords().length;i++) {
// cast the SObject to a strongly-typed Account
Account a = (Account)queryResults.getRecords()[i];
// add the Account Id to the array to be deleted
ids[i] = a.getId();
System.out.println("Deleting Id: " + a.getId() + " - Name: "+a.getName());
}
}
// delete the records in Salesforce.com by passing an array of Ids
DeleteResult[] deleteResults = connection.delete(ids);
// check the results for any errors
for (int i=0; i< deleteResults.length; i++) {
if (deleteResults[i].isSuccess()) {
System.out.println(i+". Successfully deleted record - Id: " + deleteResults[i].getId() + "At " + LocalDateTime.now());
System.out.println("************Event Count************" + ++count);
} else {
Error[] errors = deleteResults[i].getErrors();
for (int j=0; j< errors.length; j++) {
System.out.println("ERROR deleting record: " + errors[j].getMessage());
}
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
Further updates got below mentioned error after which notification were
2017-03-09T19:30:28.346 ERROR [com.salesforce.emp.connector.EmpConnector] - connection failure, reconnecting
org.cometd.common.TransportException: {httpCode=503}
at org.cometd.client.transport.LongPollingTransport$2.onComplete(LongPollingTransport.java:278)
at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:193)
After this reconnect also happened and handshake also happened but error seems to be in resubscribe() EMP connector seems to be not able to resubscribe for some reason
Note I am using "resubscribe-on-disconnect" branch of EMP connetor
We have determined there was a bug on the server side in a 403 case. The Streaming API uses a session routing cookie and this cookie periodically expires. When it expires, the session is routed to another server, and this responds with a 403. In the current version, this 403 response does not include connect advice, and the client does not attempt to reconnect. This has been fixed and the fix is currently live. My understanding is that this should fix the reconnect problem exhibited by the clients.
I am trying to send a message to the player with his saved friends. These friends are saved in a .yml file, but only the UUIDS of each individual player.
I am then trying to replace the UUID or convert it to the players name when the message is displayed (if that made sense)
CODE:
p.sendMessage("§7▄▄▄▄▄▄▄▄▄▄▄▄§aFriend System - page 1 of 1§7▄▄▄▄▄▄▄▄▄▄▄▄");
int i = 1;
int length = cfg.getList(p.getUniqueId() + ".Friends").size();
if (length != 0)
{
while (i <= length)
{
String uuid = (String)cfg.getList(p.getUniqueId() + ".Friends").get(i - 1);
ProxiedPlayer p2 = ProxyServer.getInstance().getPlayer(UUID.fromString(uuid));
if (p2 != null)
{
TextComponent prefix = new TextComponent(Main.prefix);
TextComponent join = new TextComponent("§a§lONLINE");
prefix.addExtra("§9" + p2.getName());
prefix.addExtra(" ");
prefix.addExtra(join);
p.sendMessage(prefix);
}
else
{
String name = getNamebyUUID(uuid);
if (name != null)
{
p.sendMessage(Main.prefix + "§9" + name + " §c§lOFFLINE");
Main.names.put(uuid, name);
}
else if (Main.names.containsKey(uuid))
{
p.sendMessage(Main.prefix + "§9" + (String)Main.names.get(uuid) + " §8[§c§lOFFLINE§8]");
}
else
{
p.sendMessage(Main.prefix + "§cThis is not a valid player!");
}
}
i++;
}
}
else
{
p.sendMessage(Main.prefix + "§cYou don't have any friends.");
}
}
catch (IOException e)
{
e.printStackTrace();
}
}
Related Question: Get Offline Player by UUID
If the Player is online:
String playerUUID;
Bukkit.getPlayer(playerUUID).getDisplayName();
If the Player is offline:
Not trully possible. The Player can change names at any time, and Bukkit can't keep that information and keep updating all the player's name whenever they change.
You can either use the online-player-only method above, or store the player's name together with the uuid.
#UPDATE
As stated by the user Pokechu22, Bukkit does save the last name the user used before logging out. It can be retrieved this way:
OfflinePlayer off = Bukkit.getOfflinePlayer(String uuid);
String lastKnownName = off.getName();
But be very careful! It might not be the up-to-date name of the player.
I want to display a value of a table column(which shows overall amount) in a labelfield.
Here is my code that i have used in developing a label
Border myBorder = BorderFactory.createBitmapBorder(
new XYEdges(20, 16, 27, 23),
Bitmap.getBitmapResource("border.png"));
LabelField myField = new LabelField("Total Amount Owed: Rs ",LabelField.USE_ALL_WIDTH | LabelField.FIELD_HCENTER)
{
protected void paint(Graphics g) {
g.setColor(Color.BLUE);
super.paint(g);
}
};
myField.setBorder(myBorder);
add(myField);
This is the sql statement i want to use to fetch the value from table:
Statement statementG56 = db.createStatement("SELECT owe FROM GTemp5");
statementG56.prepare();
statementG56.execute();
How to include that select statement into my labelfield so that the amount fetched stands beside the label title.
This is how i implemented. Hope it helps future readers.
try
{
//Open or create the database
Database db = DatabaseFactory.openOrCreate("database1.db");
Statement statementGF55 = db.createStatement("CREATE TABLE IF NOT EXISTS GTemp5(owe INTEGER)");
statementGF55.prepare();
statementGF55.execute();
statementGF55.close();
Statement statementGF56 = db.createStatement("SELECT owe FROM GTemp5");
statementGF56.prepare();
statementGF56.execute();
Cursor c = statementGF56.getCursor();
while(c.next())
{
System.out.println("Inside while for fetching total owed value");
Row r;
r = c.getRow();
Border myBorder = BorderFactory.createBitmapBorder(
new XYEdges(20, 16, 27, 23),
Bitmap.getBitmapResource("border.png"));
String pp = "" + r.getObject(0);
LabelField myField = new LabelField("Total Amount: Rs " +pp,LabelField.USE_ALL_WIDTH | LabelField.FIELD_HCENTER)
{
protected void paint(Graphics g)
{
g.setColor(Color.RED);
super.paint(g);
}
};
myField.setBorder(myBorder);
vfm.add(myField);
}
}
catch(Exception e)
{
System.out.println( e.getMessage() );
e.printStackTrace();
}