I am working on a Hive code to check if an IP belongs to a subnet. But the IP that I have is in string format. The usual way of doing it in SQL is :
ip::inet << '1.2.3.4'::inet
How do I do the same in Hive?
As pointed out by #[Z4-tier], there is no in-built method for Hive for doing this.
I was able to write a udf in python and then call the function over the ip in hive.
def subnet_to_mask(subnet):
c = int(subnet)
mask = (0xffffffff >> (32 - c)) << (32 - c)
return str((0xff000000 & mask) >> 24) + '.' + str((0x00ff0000 & mask) >> 16) + '.' + str((0x0000ff00 & mask) >> 8) + '.' + str((0x000000ff & mask))
def ip_to_number(ip):
ip_no = 0
for i, octet in enumerate(ip.split('.')):
ip_no += int(octet) << (24 - (8 * i))
return ip_no
def ip_in_subnet(ip, subnet):
if len(subnet.split('/')) < 2:
return ip == subnet.split('/')[0]
else:
network_ip, subnet = subnet.split('/')
subnet = subnet_to_mask(subnet)
return (ip_to_number(ip) & ip_to_number(subnet)) == (ip_to_number(network_ip) & ip_to_number(subnet))
You can then use it in hiveql like:
select ip, ip_in_subnet(ip, subnet) from tableABC;
Related
I trying to use matlab to solve a polynomial with variable parameters. I am using fsolve. The code is given below:
function my_fsolve3()
% Constant Parameters
A = 6.24;
B = 5.68e-5;
C = 1.7e-3;
D = 6.55e-8;
E = 5.3e-8;
F = 9.46e-1;
t = [0;600;1200;1800;2400;3000;10200;17400;24600;31800;...
39000;46200;53400;60600;67800;75000;82200;89400;96600;103800;...
111000;118200;125400;132600;139800;147000;154200;161400;168600;175800;183000];
G = [0.00;4.88E-06;4.88E-06;5.11E-06;5.11E-06;5.35E-06;5.35E-06;5.36E-06;5.36E-06;5.07E-06;....
5.07E-06;4.93E-06;5.30E-06;5.30E-06;5.30E-06;9.46E-06;9.46E-06;1.13E-05;1.15E-05;1.10E-05;...
1.10E-05;1.04E-05;1.04E-05;1.04E-05;1.03E-05;1.04E-05;1.06E-05;1.13E-05;1.13E-05;1.13E-05;1.03E-05];
H = [0.00;3.34E-01;6.79E-01;1.04E+00;1.41E+00;6.00E+00;1.07E+01;1.56E+01;2.07E+01;2.59E+01;...
3.14E+01;3.67E+01;4.18E+01;4.66E+01;5.09E+01;5.51E+01;5.90E+01;6.23E+01;6.56E+01;6.87E+01;...
7.12E+01;7.36E+01;7.59E+01;7.78E+01;7.95E+01;8.11E+01;8.24E+01;8.24E+01;8.23E+01;8.21E+01;8.20E+01];
I = [0.00;4.88E-06;4.88E-06;5.11E-06;5.11E-06;5.35E-06;5.35E-06;5.36E-06;5.36E-06;5.07E-06;...
5.07E-06;4.93E-06;5.30E-06;5.30E-06;5.30E-06;9.46E-06;9.46E-06;1.13E-05;1.15E-05;1.10E-05;...
1.10E-05;1.04E-05;1.04E-05;1.04E-05;1.03E-05;1.04E-05;1.06E-05;1.13E-05;1.13E-05;1.13E-05;1.03E-05];
J = [1.78E-07;7.41E-06;9.33E-06;1.20E-05;1.05E-05;1.74E-05;3.72E-05;3.55E-05;1.00E-04;4.07E-02;...
2.45E-01;6.17E-01;1.32E+00;2.29E+00;2.34E+00;2.40E+00;1.82E+00;1.38E+00;2.09E+00;1.82E+00;...
1.58E+00;2.29E+00;1.62E+00;1.12E+00;8.91E-01;8.51E-01;7.59E-01;8.71E-01;1.12E+00;1.00E+00;8.51E-01];
K = [9.75;8.13;8.03;7.92;7.98;7.76;7.43;7.45;7.00;4.39;...
3.61;3.21;2.88;2.64;2.63;2.62;2.74;2.86;2.68;2.74;...
2.8;2.64;2.79;2.95;3.05;3.07;3.12;3.06;2.95;3.00;3.07];
for i = 1:length(t)
s(i) = fsolve(#(x) x(i) + 2.* G(i) - ((H(i).* A.*x(i))/(x(i).^2 + A.*x(i) + A.*B))- 2.*((H(i).*A.*B)/(x(i).^2 ...
+ A.*x(i) + A.*B))-((I(i).*C.*x(i))/(x(i).^2 + C.*x(i) + C.*D))- 2.*((I(i).*C.*D)/(x(i).^2 ...
+ C.*x(i) + C.*D))- E/x(i), F);
end
L = 6 - log10(s);
plot(t,L)
hold on
plot (t,K,'v')
xlabel('Time (sec)')
ylabel('pH')
hold off
legend('pH-Mod','pH-Exp')
end
I do not know how can I ensure that the index value does not exceed the number of the array elements. In line 34, I indexed to the end of the independent variable, thinking that it will prevent the error. The error message is
Index exceeds the number of array elements (1).
Error in
my_fsolve3>#(x)x(i)+2.*G(i)-((H(i).*A.*x(i))/(x(i).^2+A.*x(i)+A.*B))-2.*((H(i).*A.*B)/(x(i).^2+A.*x(i)+A.*B))-((I(i).*C.*x(i))/(x(i).^2+C.*x(i)+C.*D))-2.*((I(i).*C.*D)/(x(i).^2+C.*x(i)+C.*D))-E/x(i)
(line 36)
s(i) = fsolve(#(x) x(i) + 2.* G(i) - ((H(i).* A.*x(i))/(x(i).^2 + A.*x(i) + A.*B))- 2.*((H(i).*A.*B)/(x(i).^2 ...
Error in fsolve (line 242)
fuser = feval(funfcn{3},x,varargin{:});
Error in my_fsolve3 (line 36)
s(i) = fsolve(#(x) x(i) + 2.* G(i) - ((H(i).* A.*x(i))/(x(i).^2 + A.*x(i) + A.*B))- 2.*((H(i).*A.*B)/(x(i).^2 ...
Caused by:
Failure in initial objective function evaluation. FSOLVE cannot continue.
Your anonymous function has x as an argument which is initialized to the scalar F.
You try to access x(i) which fails for i>1 because x is scalar. Solution: replace x(i) by x.
s(i) = fsolve(#(x) x + 2.* G(i) - ((H(i).* A.*x)/(x.^2 + A.*x + A.*B))- 2.*((H(i).*A.*B)/(x.^2 ...
+ A.*x + A.*B))-((I(i).*C.*x)/(x.^2 + C.*x + C.*D))- 2.*((I(i).*C.*D)/(x.^2 ...
+ C.*x + C.*D))- E/x, F);
By the way, you do not need the dot operator in multiplications of scalars, a simple * will do in your code.
I am passing a url using a string from a cell in an excel workbook.
The string is : Ben & Jerry's 2017 Base PIC - 1.
The following is my code to pass the string in a url:
Dim targetUrl As String: targetUrl =
"https://catalina.my.salesforce.com/_ui/search/ui/UnifiedSearchResults?searchType=2&sen=aAh&str=" & i & "#!/initialViewMode=summary"
where 'i' is the string that is passed.
This is the url obtained in chrome :
https://catalina.my.salesforce.com/_ui/search/ui/UnifiedSearchResults?str=Ben+&+Jerry%27s+2017+Base+PIC+-+1+=&searchType=2&sen=aAh#!/fen=aAh&initialViewMode=detail&str=Ben
As you can see, the '&' isn't encoded and the search doesn't include any value after 'Ben'.
Could someone guide me on how to solve this issue? Thank you.
To encode the parameter:
Sub Test()
Dim url As String
url = "https://catalina.my.salesforce.com/_ui/search/ui/UnifiedSearchResults" _
& "?searchType=2" _
& "&sen=aAh" _
& "&str=" & EncodeURL("Ben & Jerry's 2017 Base PIC - 1") _
& "#!/initialViewMode=summary"
Debug.Print url
End Sub
Public Function EncodeURL(url As String) As String
Dim buffer As String, i As Long, c As Long, n As Long
buffer = String$(Len(url) * 12, "%")
For i = 1 To Len(url)
c = AscW(Mid$(url, i, 1)) And 65535
Select Case c
Case 48 To 57, 65 To 90, 97 To 122, 45, 46, 95 ' Unescaped 0-9A-Za-z-._ '
n = n + 1
Mid$(buffer, n) = ChrW(c)
Case Is <= 127 ' Escaped UTF-8 1 bytes U+0000 to U+007F '
n = n + 3
Mid$(buffer, n - 1) = Right$(Hex$(256 + c), 2)
Case Is <= 2047 ' Escaped UTF-8 2 bytes U+0080 to U+07FF '
n = n + 6
Mid$(buffer, n - 4) = Hex$(192 + (c \ 64))
Mid$(buffer, n - 1) = Hex$(128 + (c Mod 64))
Case 55296 To 57343 ' Escaped UTF-8 4 bytes U+010000 to U+10FFFF '
i = i + 1
c = 65536 + (c Mod 1024) * 1024 + (AscW(Mid$(url, i, 1)) And 1023)
n = n + 12
Mid$(buffer, n - 10) = Hex$(240 + (c \ 262144))
Mid$(buffer, n - 7) = Hex$(128 + ((c \ 4096) Mod 64))
Mid$(buffer, n - 4) = Hex$(128 + ((c \ 64) Mod 64))
Mid$(buffer, n - 1) = Hex$(128 + (c Mod 64))
Case Else ' Escaped UTF-8 3 bytes U+0800 to U+FFFF '
n = n + 9
Mid$(buffer, n - 7) = Hex$(224 + (c \ 4096))
Mid$(buffer, n - 4) = Hex$(128 + ((c \ 64) Mod 64))
Mid$(buffer, n - 1) = Hex$(128 + (c Mod 64))
End Select
Next
EncodeURL = Left$(buffer, n)
End Function
Can anyone on here help me on decrypting the SSL encryption that protects this LUA script linked at the end of this topic? Basically they are encoded with Base64 then SSL, but I have no idea how to do the SSL portion. They are used with a program called Bot of Legends, and someone told me that it is possible to break the encryption by dumping the decryption function of said program and using that to get the SSL key, but I have no clue where to even start on that. Basically these scripts work by connecting to an authentication server that is coded into the script, and I have gotten a few on my own by sniffing the traffic to their auth server from network packets to get their server link and essentially created my own auth server with Apache, then redirected the network traffic that goes to their server to my own from the script to get the script validated response. For some scripts that have stronger encryption, its not that easy and I would have to get to the source code to remove the coding that runs the auth server checks. Up until a few days ago I had no knowledge on how lua coding worked and how to even compute how auth server checks could be even possible for coding in a simple text file due to lua obfuscation. So bear with me, I would like if someone can chime in and give me an idea on what I can do.
Regards,
Chris
*** PasteBin link to the script in question in raw format: http://pastebin.com/raw.php?i=bG0VqQGW
The Base64 section is first with the SSL section at the bottom.
print("SSL Decoder version 2.0")
print("Copyright (C) 2015")
print("Decoding Started...")
local infilename = select(1,...)
local outfilename = select(2,...)
local infile = io.open(infilename, "r")
if not infile then
error("Failed to open input file.")
end
local intext = infile:read("*a")
infile:close()
local ssltabletext = intext:match("SSL%s*%(%s*%{([%s,0-9]*)%}%s*%)")
if not ssltabletext then
error("Could not find ssl table in source file.")
end
local ssltable = load("return {"..ssltabletext.."}")()
if #ssltable < 255 then
error("SSL table is too short -- can't find table encryption key.")
end
-- find decryption key for the ssl table
local decrypt = {}
decrypt[0] = 0
for i = 1,255 do
local dec = i
local enc = ssltable[i]
assert(decrypt[enc] == nil)
decrypt[enc] = dec
end
-- decrypt ssl table
for i = 256, #ssltable - 256 do -- not sure what last 256 bytes are
ssltable[i] = decrypt[ssltable[i] ]
end
-- If this does a stack overflow, easy to change to something dumb but more robust
local sslcode = string.char(table.unpack(ssltable, 256, #ssltable - 256))
-- This is interesting --
--print(sslcode)
local keyindex = sslcode:match("local Key%s*=%s*'()")
if not keyindex then
error("Could not find key in decoded ssl table.")
end
local key = sslcode:sub(keyindex)
local length = 0
while true do
local c = key:sub(length+1, length+1)
if c == "" then
error("Key string was not terminated.")
elseif c == "'" then
break
elseif c == "\\" then
local c2 = key:sub(length+2, length+2)
if c2:match("%d") then
local c3 = key:sub(length+3, length+3)
if c3:match("%d") then
local c4 = key:sub(length+4, length+4)
if c4:match("%d") then
length = length + 4
else
length = length + 3
end
else
length = length + 2
end
elseif c2 == "x" then
length = length + 4
else
length = length + 2
end
else
length = length + 1
end
end
key = key:sub(1, length)
if #key == 0 then
error("Key is empty")
end
print("Key Found! > " .. key)
print("Decoding finished, outfile is at > " .. outfilename)
-- find base64
local b64 = intext:match("_G.ScriptCode%s*=%s*Base64Decode%s*%(%s*\"([a-zA-Z0-9/+]*=*)\"%s*%)")
if not b64 then
error("Could not find Base-64 encrypted code in source file.")
end
-- base64 decode
local b64val = {}
for i = 0, 25 do
do
local letter = string.byte("A")
b64val[string.char(letter+i)] = i
end
do
local letter = string.byte("a")
b64val[string.char(letter+i)] = i + 26
end
end
for i = 0, 9 do
local numeral = string.byte("0")
b64val[string.char(numeral+i)] = i + 52
end
b64val["+"] = 62
b64val["/"] = 63
b64val["="] = 0
local encoded = b64:gsub("(.)(.)(.)(.)",function(a,b,c,d)
local n = b64val[a] * (64 * 64 * 64) + b64val[b] * (64 * 64) + b64val[c] * 64 + b64val[d]
local b1 = n % 256; n = (n - b1) / 256
local b2 = n % 256; n = (n - b2) / 256
local b3 = n
if d == "=" then
if c == "=" then
assert(b1 == 0 and b2 == 0)
return string.char(b3)
else
assert(b1 == 0)
return string.char(b3, b2)
end
else
return string.char(b3, b2, b1)
end
end)
-- decode
local decoded = encoded:gsub("()(.)", function(i, c)
local b = c:byte()
local ki = ((i - 1) % #key) + 1
local k = key:byte(ki,ki)
b = b - k
if b < 0 then b = b + 256 end
return string.char(b)
end)
-- verify
local result, err = load(decoded)
if not result then
error("Decoded file could not be loaded -- it may be corrupt... ("..tostring(err)..")")
end
-- output
local outfile = io.open(outfilename, "wb")
if not outfile then
error("Failed to open output file.")
end
outfile:write(decoded)
outfile:close()
This code is by Extreme Coders (https://reverseengineering.stackexchange.com/users/1413/extreme-coders)
how to use it , u need to get lua52.exe
save the code into a text file and name it ssl.lua (for example)
now run cmd and type lua52 ssl yourscript.lua decryptedscript.lua
it will run and decrypt it.
I'm new to VBA, and I have a question, i.e I have a mathematical function 1 + 2x¹ + 3x² + 4x³ + ... +10x⁹ and I need to resolve it into two ways:
I can use raising operations(analog pow in Pascal) and IF statement;
without rising operations and IF statement...
I have tried this one:
Public Function test(x)
test = 1 + 2*x^1 + 3*x^2 + 4*x^3 + 5*x^4 + 6*x^5 + 7*x^6 + 8*x^7 + 9*x^8 + 10*x^9
End Function
but I think it returns the wrong answer - 2441406 with calling =test(5)
So can anyone give any advice, or help with my problem?
If you can't use VBA for this, there is a formula solution. Assuming your variable x is in cell A1, you would use this formula in another cell (I used B1):
=SUMPRODUCT(ROW($1:$10)*A1^(ROW($1:$10)-1))
When A1 = 5, it returned 23803711 as expected.
You will need * as the multiper:
Public Function test(x)
test = _
1 _
+ 2 * (x ^ 1) _
+ 3 * (x ^ 2) _
+ 4 * (x ^ 3) _
+ 5 * (x ^ 4) _
+ 6 * (x ^ 5) _
+ 7 * (x ^ 6) _
+ 8 * (x ^ 7) _
+ 9 * (x ^ 8) _
+ 10 * (x ^ 9)
End Function
I have access to some VC++ source code for which I am trying to convert to VB.NET. I previously asked a question regarding bit shifting, and although the answers given made sense and seemed rather simple to convert over to VB.NET, I am having difficulty getting things to work out. Here is some VC++ code that I am needing to convert to VB.NET:
#define bitShift(_val) \
((u64)(((((u64)_val) & 0xff00000000000000ull) >> 56) | \
((((u64)_val) & 0x00ff000000000000ull) >> 40) | \
((((u64)_val) & 0x0000ff0000000000ull) >> 24) | \
((((u64)_val) & 0x000000ff00000000ull) >> 8 ) | \
((((u64)_val) & 0x00000000ff000000ull) << 8 ) | \
((((u64)_val) & 0x0000000000ff0000ull) << 24) | \
((((u64)_val) & 0x000000000000ff00ull) << 40) | \
((((u64)_val) & 0x00000000000000ffull) << 56)))
Now, the returned value will be used as the counter for AES decryption in CTR Mode. The following VC++ code is used to calculate the counter:
u8 counter[16];
*(u64 *)(counter + 0) = bitShift(i);
*(u64 *)(counter + 8) = 0;
This is where I am currently at with the VB.NET code:
Public Function SwapBits(ByVal value As Int64) As Int64
Dim uvalue As UInt64 = CULng(value)
Dim swapped As UInt64 = ((&HFF00000000000000UL) And (uvalue >> 56) Or _
(&HFF000000000000L) And (uvalue >> 40) Or _
(&HFF0000000000L) And (uvalue >> 24) Or _
(&HFF00000000L) And (uvalue >> 8) Or _
(&HFF000000UI) And (uvalue << 8) Or _
(&HFF0000) And (uvalue << 24) Or _
(&HFF00) And (uvalue << 40) Or _
(&HFF) And (uvalue << 56))
Return CLng(swapped)
End Function
Here is the code used to create the counter:
Dim blocks As Integer = file_size / 16
For i As Integer = 0 To blocks - 1
Dim buffer As Byte() = New Byte(15) {}
Array.Copy(BitConverter.GetBytes(SwapBits(CULng(i))), 0, buffer, 0, 8)
'AES decryption takes place after this...
The counter is 16 bytes, but only the first 8 bytes are encrypted using AES 128 bit EBC and then XOR'd with the current encrypted block of data which is also 16 bytes (AES CTR Mode). I can get the code to run without any errors, but the output of decrypted data is incorrect which leads me to believe I am not calculating the counter which is being used for encryption correctly.
Once again, any help is obviously appreciated, and thanks in advance!
EDIT: Current SwapBits function... still not right though
Public Function SwapBits(ByVal uvalue As UInt64) As UInt64
Dim swapped As UInt64 = ((((uvalue) And &HFF00000000000000) >> 56) Or _
(((uvalue) And &HFF000000000000) >> 40) Or _
(((uvalue) And &HFF0000000000) >> 24) Or _
(((uvalue) And &HFF00000000) >> 8) Or _
(((uvalue) And &HFF000000) << 8) Or _
(((uvalue) And &HFF0000) << 24) Or _
(((uvalue) And &HFF00) << 40) Or _
(((uvalue) And &HFF) << 56))
Return swapped
End Function
This actually causes an "Arithmetic operation resulted in an overflow." error when uvalue reaches a value of 128. When the value of 1 is passed to SwapBits, my return value = 72057594037927936. My interpretation of the VC++ code is that my counter should simply be a 16 byte array incrementing by 1 each time. For example, if
uvalue = 1
then my counter needs to be
0000000100000000
if
uvalue = 25
then my counter needs to be
0000002500000000
etc, etc... Or I am misinterpreting something somewhere?
Not sure what you're expecting from the C++ code. But when I use this:
#include <iostream>
using namespace std;
#define bitShift(_val) \
((unsigned __int64)(((((unsigned __int64)_val) & 0xff00000000000000ull) >> 56) | \
((((unsigned __int64)_val) & 0x00ff000000000000ull) >> 40) | \
((((unsigned __int64)_val) & 0x0000ff0000000000ull) >> 24) | \
((((unsigned __int64)_val) & 0x000000ff00000000ull) >> 8 ) | \
((((unsigned __int64)_val) & 0x00000000ff000000ull) << 8 ) | \
((((unsigned __int64)_val) & 0x0000000000ff0000ull) << 24) | \
((((unsigned __int64)_val) & 0x000000000000ff00ull) << 40) | \
((((unsigned __int64)_val) & 0x00000000000000ffull) << 56)))
int main()
{
unsigned __int64 test = bitShift(25);
return 0;
}
I get the exact same return value(1801439850948198400 || &H1900000000000000) as this:
Dim result As ULong = SwapBits(25)
Public Function SwapBits(ByVal uvalue As UInt64) As UInt64
Dim swapped As UInt64 = ((((uvalue) And &HFF00000000000000UL) >> 56) Or _
(((uvalue) And &HFF000000000000UL) >> 40) Or _
(((uvalue) And &HFF0000000000UL) >> 24) Or _
(((uvalue) And &HFF00000000UL) >> 8) Or _
(((uvalue) And &HFF000000UL) << 8) Or _
(((uvalue) And &HFF0000UL) << 24) Or _
(((uvalue) And &HFF00UL) << 40) Or _
(((uvalue) And &HFFUL) << 56))
Return swapped
End Function
I don't have much experience in C++, care to share what this is doing:
u8 counter[16];
*(u64 *)(counter + 0) = bitShift(i);
*(u64 *)(counter + 8) = 0;
basically that section of code increments the first 8 bytes of counter by 1 each iteration 0f i, starting with the right most byte and expanding left for each carryover. For instance, if the counter reaches 999 counter[7] will hold 231(&HE7) and counter[6] 3(&H3) which when you look at the whole array gives, &H000000000003E7 which equals 999 decimal.
Something tells me conversion is better done using the GetBytes and ToUInt64() methods, a for loop and a temporary variable. It would be much easier to read and probably fast enough for most purposes.