I want to simulate a loop in Hive. The problem is that of link traversal.
Raw Data:
phone,name
1234,Sam
2345,John
3456,Max
7899,Sam
7899,Tim
8899,Tim
8898,Dan
8899,John
I am only interested in getting phone numbers.
If I take 1234,Sam and go to 7899,Sam that would be 1-hop because Sam used 7899 number too. Then if I go from 7899,Sam to 8899,Tim that would be 1-hop because 7899 was used by Tim and Tim also used 8899. Now where we started from, 1234, to where we are now, 8899, we are 2-hops away. I can do this with a shell script with loops but I am interested to find if this is possible to do purely in Hive. I want the loop to stop if any one criteria is met:
10 hops reached OR no new numbers detected
(If the conditions above are hard to implement, I can just have one condition - 10hops reached)
Visual representation:
1 hop:
1234 -> Sam
Sam linked to 7899
2 hops:
7899 -> Tim
Tim linked to 8899
so 1234 and 8899 are 2 hops away.
Any tips are welcome!
Related
I am stuck trying to create an attribute rule/field calculation using Arcade where upstream areas are added to a downstream area called "Cumulative Area"
NodeFrom
NodeTo
Area
CumulativeArea
1
2
10
10
2
3
5
15
4
3
6
6
4
3
6
6
3
5
8
20
Essentially I am thinking of the SQL equivalent of a SUM WHERE NodeTo = NodeFrom. Or an excel equivalent of [Area + SUMIF(CumulativeArea, NodeFrom, NodeTo)]
This is for QA'ing sewer calculations so please let me know if there are any tools out there that may already have this kind of functionality.
Thanks!
I think you're just tired and need to re-evalute the tools you're already using. Other tools won't be node specific. In faster settings the area is more difficult to process at the speed you require. Try disassembly of your current tool for full copyright. You're on the right track.
I'm creating a game matchmaking system using Redis based on MMR, which is a number that pretty much sums up the skill of a player. Therefore the system can match him/her with others who are pretty much with the same skill.
For example if a player with MMR of 1000 joins the queue, system will try to find other ppl with MMR of range 950 to 1050 to match with this player. But if after one minute it cannot find any player with given stats it will scale up the range to 900 to 1100 (a constant threshold).
What I want to do is really easy with relational database design but I can't figure out how to do it with Redis.
The queue table implementation would be like this:
+----+---------+------+-------+
| ID | USER_ID | MMR | TRIES |
+----+---------+------+-------+
| 1 | 50 | 1000 | 1 |
| 2 | 70 | 1500 | 1 |
| 3 | 350 | 1200 | 1 |
+----+---------+------+-------+
So when a new player queues up, it will check it's MMR against other players in the queue if it finds one between 5% Threshold it will match the two players if not it will add the new player to the table and wait for new players to queue up to compare or to pass 1 minute and the cronjob increment the tries and retry to match players.
The only way I can imagine is to use two separate keys for the low and high of each player in the queue like this
MatchMakingQueue:User:1:Low => 900
MatchMakingQueue:User:1:High => 1100
but the keys will be different and I can't get for example all users in between range of low of 900 to high of 1100!
I hope I've been clear enough any help would be much appreciated.
As #Guy Korland had suggested, a Sorted Set can be used to track and match players based on their MMR, and I do not agree with the OP's "won't scale" comment.
Basically, when a new player joins, the ID is added to a zset with the MMR as its score.
ZADD players:mmr 1000 id:50
The matchmaking is made for each user, e.g. id:50 with the following query:
ZREVRANGEBYSCORE players:mmrs 1050 950 LIMIT 0 2
A match is found if two IDs are returned and at least one of them is different than that of the new player. To make the match, both IDs (the new player's and the matched with one) need to be removed from the set - I'd use a Lua script to implement this piece of logic (matching and removing) for atomicity and communication reduction, but it can be done in the client as well.
There are different ways to keep track of the retries, but perhaps the simplest one is to use another Sorted Set, where the score is that metric.
The following pseudo Redis Lua code is a minimal example of the approach:
local kmmrs, kretries = KEYS[1], KEYS[2]
local id = ARGV[1]
local mmr = redis.call('ZSCORE', kmmrs, id)
local retries = redis.call('ZSCORE', kretries, id)
local min, max = mmr*(1-0.05*retries), mmr*(1+0.05*retries)
local candidates = redis.call('ZREVRANGEBYSCORE', kmmrs, max, min, 'LIMIT', 0, 2)
if #candidates < 2 then
redis.call('ZINCRBY', kretries, 1, id)
return nil
end
local reply
if candidates[1] ~= id then
reply = candidates[1]
else
reply = candidates[2]
end
redis.call('ZREM', kmmrs, id, reply)
redis.call('ZREM', kretries, id, reply)
return reply
Let me get the problem right! Your problem is that you want to find all the users in a given range of MMR value. What if You make other users say that "I'm falling in this range".
Read about Redis Pub/Sub.
When a user joins in, publish its MMR to rest of the players.
Write the code on the user side to check if his/her MMR is falling in the range.
If it is, user will publish back to a channel that it is falling in that range. Else, user will silently discard the message.
repeat these steps if you get no response back in 1 minute.
You can make one channel (let's say MatchMMR) for all users to publish MMR for match request which should be suscribed by all the users. And make user specific channel in case somebody has a MMR in the calculated range.
Form you published messages such that you can send all the information like "retry count", "match range percentage", "MMR value" etc. so that your code at user side can calculate if it is the right fit for the MMR.
Read mode about redis Pub/Sub at: https://redis.io/topics/pubsub
My team have settled on MSpec for our BDD testing framework, which from their usage so far looks really good - but I'm struggling with the documentation/google for finding any implementation similar to SpecFlow's 'Scenario Outline'. I've shown an example of this below, but basically it allows you to write one 'test' and run it multiple times from a table (example) of inputs/expected outputs. I'll be embarrassed if answer turns out to be a LMGTFY but I've not been able to find anything myself. I don't want to say to the team it's not possible if I've just not found how to do it in MSpec (or understood MSpec properly). I wonder if this is why in some of the pro's/con's for MSpec I see references to the number of classes you can end up with listed as a negative.
Example of SpecFlow Scenario Outline
Scenario Outline: Successfully Convert Seconds to Minutes Table
When I navigate to Seconds to Minutes Page
And type seconds for <seconds>
Then assert that <minutes> minutes are displayed as answer
Examples:
| seconds | minutes |
| 1 day, 1 hour, 1 second | 1500 |
| 5 days, 3 minutes | 7203 |
| 4 hours | 240 |
| 180 seconds | 3 |
From: https://gist.github.com/angelovstanton/615da65a8f821d7a43c92ef9e2fd0b01#file-energyandpowerconvertcalculator-feature
Short answer, this is current not supported by by mspec. We planned this several years back, but the contribution never made it back into master.
If you want scenario outlines either use a different framework or create parameterized static methods in a helper class and call these from your context classes. Which will leave you with 1 class per scenario.
I am trying to figure out how to breakdown a vehicle vin number.
There is an explanation of how a VIN is build (http://en.wikipedia.org/wiki/Vehicle_Identification_Number#Components_of_the_VIN) but it fails to explain what to do with manufacturers that only have 2 digits assigned instead of 3 digits.
If I understand correct what is written there then every VIN number must be 17 characters long, and the first 3 characters are the WMI (World Manufaturer Identification).
Then there is a list of WMI on the same page, but some manufacturers only have 2 characters in that list, not 3.
How to read such a VIN number ? Will it be only 16 characters long or how do I regognize that a WMI is 2 or 3 characters ?
for example nissan has WMI = JN which is only 2 characters.
2 VIN numbers for Nissan that I know that are valid are :
JN1UC4E26F9001391 and JNKCP0106TT541680
How can I know that for these 2 VIn numbers only the first 2 digits are to be read and used for the WMI ?
In you examples it is in fact 3 characters that Set the WMI information
JN1UC4E26F9001391
NISSAN MOTOR COMPANY, LTD. JN1,1N4 3995 RESEARCH PARK DRIVE ANN ARBOR MI 48104 PASSENGER CAR SEE MEMO 7/29/1986
JNKCP0106TT541680
NISSAN RESEARCH & DEVELOPMENT JN1,JNK 750 17TH STREET, N.W. WASHINGTON DC 20006 PASSENGER CAR ALL 1/13/1992
1st character- Identifies the country in which the vehicle was manufactured.
For example: U.S.A.(1or 4), Canada(2), Mexico(3), Japan(J), Korea(K), England(S), Germany(W), Italy(Z)
2nd character- Identifies the manufacturer. For example; Audi(A),
BMW(B), Buick(4), Cadillac(6), Chevrolet(1), Chrysler(C), Dodge(B),
Ford(F), GM Canada(7), General Motors(G), Honda(H), Jaquar(A), Lincoln(L), Mercedes Benz(D), Mercury(M), Nissan(N), Oldsmobile(3), Pontiac(2or5), Plymouth(P), Saturn(8), Toyota(T), VW(V), Volvo(V).
3rd character- Identifies vehicle type or manufacturing division.
Per VIN descriptor
This link contains the most current WMI information for all, this is were I got my data
WMI for all manufactures
In any Case it is not 3 or 2 for WMI it is
1 for country
2 for manufacturer
3 for type
this gets a little tricky because IF "2" = "N" that is not necessarily Nissan, for example 1N9 = NOMAD CUSTOM CYCLES INC.
Let me know if that helps or if you find a place to get VDS information
Your wiki page seems pretty clear that it's either a 3-character WMI, or a 2-character WMI followed by a 9.
There are an abundance of libraries out there, on GitHub and elsewhere, which are designed to decode VINs. "Do not do a thing already done."
Is it possible to handle relations between jobs (in this case services) in jsprit?
For instance one job has to start at the same time with one or more other jobs.
Or one job must not start before the end of another job (normal sequence).
If not, do you know other java libraries that can handle such kind of restrictions?
Thank you!
Yes you can handle such relations with jsprit.
It is a bit of coding, but - I think - easy to implement. To illustrate it, I prepared you five examples that are based on the Vehicle Routing Problems (VRP) developed by Christofides-Mingozzi-Toth's* (first benchmarking instance). Follow the links behind the individual headlines and you get to the code. Note that the first activity in a route is marked with a triangle. The labels denote jobIds.
noConstraints
capacityConstraints
job 13 and 21 in same route
job 13 and 21 in same route AND 21 before 13
job 13 and 21 in same route AND 13 right after 21
Visit http://jsprit.github.io/ and you get to a number of code examples and docs.
*N. Christofides, A. Mingozzi, and P. Toth. The vehicle routing problem. In N. Christofides, A. Mingozzi, P. Toth, and C. Sandi, editors, Combinatorial Optimization. John Wiley, Chichester, 1979.