I have an in-memory table with 3 million records.
Part of the stock data is as follows:
How can I quickly get the last n rows of this table, say 35,000 rows?
Here are two solutions to your problem.
I’ll use the following simulated data:
def createData(num){
date = take(2022.01.01, num)
time = 00:00:00.000 + take(0..num, num)
a1 = rand(1000, num)
a2 = rand(1000, num)
a3 = rand(1000, num)
a4= rand(1000, num)
a5 = rand(100.0, num)
a6 = rand(100.0, num)
a7 = rand(100.0, num)
a8 = rand(100.0, num)
return table(date, time, a1, a2, a3, a4, a5, a6, a7, a8)
}
num = 3000000
t = createData(num)
Solution 1:
timer{
result = (select * from t order by time desc limit 35000).sortBy!(`time)
}
//Time elapsed: 11.732 ms
Solution 2:
timer{
rowOffset = size(t) - 35000
rowCount = 35000
result = select * from t order by time limit rowOffset, rowCount
}
//Time elapsed: 14.013 ms
Related
I´m currently working on a small tool for powerpoint to make certain processes easier and one of them is to create a gradient with 2 stops and then extract the colors inbetween so that I can simply create a gradient in powerpoint on a shape, select it, choose how many colors should be generated out of the gradient and then create a group of shapes with the individual colors.
However creating the shapes with the colors is not an issue so I would simply like to know whether or not there is any way to extract the colors as described and perhaps how I would achieve it.
Thanks in advance
(Edit : I was able to resolve the issue myself. The code is attached in my answer)
Alright, I was able to find a solution myself by breaking down both colors into the respective values and then increase/decrease each value by a percentage of the difference between both colors. I attached the code for anyone interested.
Sub extractGradient()
Dim sld As Slide
Dim nShape As shape
Dim c1, c2, r1, g1, b1, r2, g2, b2, rDiff, gDiff, bDiff, cR, cG, cB As Long
Dim range As Integer
Dim colors As Collection
Set sld = Application.ActiveWindow.View.Slide
Set colors = New Collection
range = 1000
With ActiveWindow.Selection.ShapeRange(1).Fill.ForeColor
c1 = .RGB
r1 = .RGB Mod 256
g1 = .RGB \ 256 Mod 256
b1 = .RGB \ 65536 Mod 256
End With
With ActiveWindow.Selection.ShapeRange(2).Fill.ForeColor
c2 = .RGB
r2 = .RGB Mod 256
g2 = .RGB \ 256 Mod 256
b2 = .RGB \ 65536 Mod 256
End With
rDiff = Abs(r2 - r1)
gDiff = Abs(g2 - g1)
bDiff = Abs(b2 - b1)
colors.Add c1
For i = 1 To range - 1
cR = IIf(r1 > r2, r1 - (rDiff / range * i), r1 + (rDiff / range * i))
cG = IIf(g1 > g2, g1 - (gDiff / range * i), g1 + (gDiff / range * i))
cB = IIf(b1 > b2, b1 - (bDiff / range * i), b1 + (bDiff / range * i))
colors.Add (RGB(cR, cG, cB))
Next i
colors.Add c2
count = 0
For Each c In colors
Set nShape = sld.Shapes.AddShape(Type:=msoShapeRectangle, left:=(1 * count), top:=50, width:=1, height:=50)
nShape.Fill.ForeColor.RGB = c
nShape.Line.Visible = msoFalse
count = count + 1
Next c
End Sub
For example, if I want to find
1085912312763120759250776993188102125849391224162 = a^9+b^9+c^9+d
the code needs to brings
a=3456
b=78525
c=217423
d=215478
I do not need specific values, only that they comply with the fact that a, b and c have 6 digits at most and d is as small as possible.
Is there a quick way to find it?
I appreciate any help you can give me.
I have tried with nested loops but it is extremely slow and the code gets stuck.
Any help in VB or other code would be appreciated. I think the structure is more important than the language in this case
Imports System.Numerics
Public Class Form1
Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
Dim Value As BigInteger = BigInteger.Parse("1085912312763120759250776993188102125849391224162")
Dim powResult As BigInteger
Dim dResult As BigInteger
Dim a As Integer
Dim b As Integer
Dim c As Integer
Dim d As Integer
For i = 1 To 999999
For j = 1 To 999999
For k = 1 To 999999
powResult = BigInteger.Add(BigInteger.Add(BigInteger.Pow(i, 9), BigInteger.Pow(j, 9)), BigInteger.Pow(k, 9))
dResult = BigInteger.Subtract(Value, powResult)
If Len(dResult.ToString) <= 6 Then
a = i
b = j
c = k
d = dResult
RichTextBox1.Text = a & " , " & b & " , " & c & " , " & d
Exit For
Exit For
Exit For
End If
Next
Next
Next
End Sub
End Class
UPDATE
I wrote the code in vb. But with this code, a is correct, b is correct but c is incorrect, and the result is incorrect.
a^9 + b^9 + c^9 + d is a number bigger than the initial value.
The code should brings
a= 217423
b= 78525
c= 3456
d= 215478
Total Value is ok= 1085912312763120759250776993188102125849391224162
but code brings
a= 217423
b= 78525
c= 65957
d= 70333722607339201875244531009974
Total Value is bigger and not equal=1085935936469985777155428248430866412402362281319
Whats i need to change in the code to make c= 3456 and d= 215478?
the code is
Imports System.Numerics
Public Class Form1
Private Function pow9(x As BigInteger) As BigInteger
Dim y As BigInteger
y = x * x ' x^2
y *= y ' x^4
y *= y ' x^8
y *= x ' x^9
Return y
End Function
Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
Dim a, b, c, d, D2, n As BigInteger
Dim aa, bb, cc, dd, ae As BigInteger
D2 = BigInteger.Parse("1085912312763120759250776993188102125849391224162")
'first solution so a is maximal
d = D2
'a = BigIntegerSqrt(D2)
'RichTextBox1.Text = a.ToString
For a = 1 << ((Convert.ToInt32(Math.Ceiling(BigInteger.Log(d, 2))) + 8) / 9) To a > 0 Step -1
If (pow9(a) <= d) Then
d -= pow9(a)
Exit For
End If
Next
For b = 1 << ((Convert.ToInt32(Math.Ceiling(BigInteger.Log(d, 2))) + 8) / 9) To b > 0 Step -1
If (pow9(b) <= d) Then
d -= pow9(b)
Exit For
End If
Next
For c = 1 << ((Convert.ToInt32(Math.Ceiling(BigInteger.Log(d, 2))) + 8) / 9) To c > 0 Step -1
If (pow9(c) <= d) Then
d -= pow9(c)
Exit For
End If
Next
' minimize d
aa = a
bb = b
cc = c
dd = d
If (aa < 10) Then
ae = 0
Else
ae = aa - 10
End If
For a = aa - 1 To a > ae Step -1 'a goes down few iterations
d = D2 - pow9(a)
For n = 1 << ((Convert.ToInt32(Math.Ceiling(BigInteger.Log(d, 2))) + 8) / 9) To b < n 'b goes up
If (pow9(b) >= d) Then
b = b - 1
d -= pow9(b)
Exit For
End If
Next
For c = 1 << ((Convert.ToInt32(Math.Ceiling(BigInteger.Log(d, 2))) + 8) / 9) To c > 0 Step -1 'c must be search fully
If pow9(c) <= d Then
d -= pow9(c)
Exit For
End If
Next
If d < dd Then 'remember better solution
aa = a
bb = b
cc = c
dd = d
End If
If a < ae Then
Exit For
End If
Next
a = aa
b = bb
c = cc
d = dd
' a,b,c,d is the result
RichTextBox1.Text = D2.ToString
Dim Sum As BigInteger
Dim a9 As BigInteger
Dim b9 As BigInteger
Dim c9 As BigInteger
a9 = BigInteger.Pow(a, 9)
b9 = BigInteger.Pow(b, 9)
c9 = BigInteger.Pow(c, 9)
Sum = BigInteger.Add(BigInteger.Add(BigInteger.Add(a9, b9), c9), d)
RichTextBox2.Text = Sum.ToString
Dim Subst As BigInteger
Subst = BigInteger.Subtract(Sum, D2)
RichTextBox3.Text = Subst.ToString
End Sub
End Class
[Update]
The below code is an attempt to solve a problem like OP's, yet I erred in reading it.
The below is for 1085912312763120759250776993188102125849391224162 = a^9+b^9+c^9+d^9+e and to minimize e.
Just became too excite about OP's interesting conundrum and read too quick.
I review this more later.
OP's approach is O(N*N*N*N) - slow
Below is a O(N*N*log(N)) one.
Algorithm
Let N = 1,000,000. (Looks like 250,000 is good enough for OP's sum of 1.0859e48.)
Define 160+ wide integer math routines.
Define type: pow9
int x,y,
int160least_t z
Form array pow9 a[N*N] populated with x, y, x^9 + y^9, for every x,y in the [1...N] range.
Sort array on z.
Cost so far O(N*N*log(N).
For array elements indexed [0... N*N/2] do a binary search for another array element such that the sum is 1085912312763120759250776993188102125849391224162
Sum closest is the answer.
Time: O(N*N*log(N))
Space: O(N*N)
Easy to start with FP math and then later get a better answer with crafter extended integer math.
Try with smaller N and total sum targets to iron out implementation issues.
In case a,b,c,d might be zero I got an Idea for fast and simple solution:
First something better than brute force search of a^9 + d = x so that a is maximal (that ensures minimal d)...
let d = 1085912312763120759250776993188102125849391224162
find max value a such that a^9 <= d
this is simple as we know 9th power will multiply the bitwidth of operand 9 times so the max value can be at most a <= 2^(log2(d)/9) Now just search all numbers from this number down to zero (decrementing) until its 9th power is less or equal to x. This value will be our a.
Its still brute force search however from much better starting point so much less iterations are required.
We also need to update d so let
d = d - a^9
Now just find b,c in the same way (using smaller and smaller remainder d)... these searches are not nested so they are fast ...
b^9 <= d; d-=b^9;
c^9 <= d; c-=b^9;
To improve speed even more you can hardcode the 9th power using power by squaring ...
This will be our initial solution (on mine setup it took ~200ms with 32*8 bits uints) with these results:
x = 1085912312763120759250776993188102125849391224162
1085912312763120759250776993188102125849391224162 (reference)
a = 217425
b = 65957
c = 22886
d = 39113777348346762582909125401671564
Now we want to minimize d so simply decrement a and search b upwards until still a^9 + b^9 <= d is lower. Then search c as before and remember better solution. The a should be search downwards to meet b in the middle but as both a and bhave the same powers only few iterations might suffice (I used 50) from the first solution (but I have no proof of this its just my feeling). But still even if full range is used this has less complexity than yours as I have just 2 nested fors instead of yours 3 and they all are with lower ranges...
Here small working C++ example (sorry do not code in BASIC for decades):
//---------------------------------------------------------------------------
typedef uint<8> bigint;
//---------------------------------------------------------------------------
bigint pow9(bigint &x)
{
bigint y;
y=x*x; // x^2
y*=y; // x^4
y*=y; // x^8
y*=x; // x^9
return y;
}
//---------------------------------------------------------------------------
void compute()
{
bigint a,b,c,d,D,n;
bigint aa,bb,cc,dd,ae;
D="1085912312763120759250776993188102125849391224162";
// first solution so a is maximal
d=D;
for (a=1<<((d.bits()+8)/9);a>0;a--) if (pow9(a)<=d) break; d-=pow9(a);
for (b=1<<((d.bits()+8)/9);b>0;b--) if (pow9(b)<=d) break; d-=pow9(b);
for (c=1<<((d.bits()+8)/9);c>0;c--) if (pow9(c)<=d) break; d-=pow9(c);
// minimize d
aa=a; bb=b; cc=c; dd=d;
if (aa<50) ae=0; else ae=aa-50;
for (a=aa-1;a>ae;a--) // a goes down few iterations
{
d=D-pow9(a);
for (n=1<<((d.bits()+8)/9),b++;b<n;b++) if (pow9(b)>=d) break; b--; d-=pow9(b); // b goes up
for (c=1<<((d.bits()+8)/9);c>0;c--) if (pow9(c)<=d) break; d-=pow9(c); // c must be search fully
if (d<dd) // remember better solution
{
aa=a; bb=b; cc=c; dd=d;
}
}
a=aa; b=bb; c=cc; d=dd; // a,b,c,d is the result
}
//-------------------------------------------------------------------------
The function bits() just returns number of occupied bits (similar to log2 but much faster). Here final results:
x = 1085912312763120759250776993188102125849391224162
1085912312763120759250776993188102125849391224162 (reference)
a = 217423
b = 78525
c = 3456
d = 215478
It took 1689.651 ms ... As you can see this is much faster than yours however I am not sure with the number of search iterations while fine tuning ais OK or it should be scaled by a/b or even full range down to (a+b)/2 which will be much slower than this...
One last thing I did not bound a,b,c to 999999 so if you want it you just add if (a>999999) a=999999; statement after any a=1<<((d.bits()+8)/9)...
[Edit1] adding binary search
Ok now all the full searches for 9th root (except of the fine tunnig of a) can be done with binary search which will improve speed a lot more while ignoring bigint multiplication complexity leads to O(n.log(n)) against your O(n^3)... Here updated code (will full iteration of a while fitting so its safe):
//---------------------------------------------------------------------------
typedef uint<8> bigint;
//---------------------------------------------------------------------------
bigint pow9(bigint &x)
{
bigint y;
y=x*x; // x^2
y*=y; // x^4
y*=y; // x^8
y*=x; // x^9
return y;
}
//---------------------------------------------------------------------------
bigint binsearch_max_pow9(bigint &d) // return biggest x, where x^9 <= d, and lower d by x^9
{ // x = floor(d^(1/9)) , d = remainder
bigint m,x;
for (m=bigint(1)<<((d.bits()+8)/9),x=0;m.isnonzero();m>>=1)
{ x|=m; if (pow9(x)>d) x^=m; }
d-=pow9(x);
return x;
}
//---------------------------------------------------------------------------
void compute()
{
bigint a,b,c,d,D,n;
bigint aa,bb,cc,dd;
D="1085912312763120759250776993188102125849391224162";
// first solution so a is maximal
d=D;
a=binsearch_max_pow9(d);
b=binsearch_max_pow9(d);
c=binsearch_max_pow9(d);
// minimize d
aa=a; bb=b; cc=c; dd=d;
for (a=aa-1;a>=b;a--) // a goes down few iterations
{
d=D-pow9(a);
for (n=1<<((d.bits()+8)/9),b++;b<n;b++) if (pow9(b)>=d) break; b--; d-=pow9(b); // b goes up
c=binsearch_max_pow9(d);
if (d<dd) // remember better solution
{
aa=a; bb=b; cc=c; dd=d;
}
}
a=aa; b=bb; c=cc; d=dd; // a,b,c,d is the result
}
//-------------------------------------------------------------------------
function m.isnonzero() is the same as m!=0 just faster... The results are the same as above code but the time duration is only 821 ms for full iteration of a which would be several thousands seconds with previous code.
I think except using some polynomial discrete math trick I do not know of there is only one more thing to improve and that is to compute consequent pow9 without multiplication which will boost the speed a lot (as bigint multiplication is slowest operation by far) like I did in here:
How to get a square root for 32 bit input in one clock cycle only?
but I am too lazy to derive it...
Private Function lang_array(ByVal temp As Double) As Double
'This program call the subroutine langmuir to calculate langmuir constants
'of each molecule for the small cage and for the large cage
Dim molenum As Integer = 1 ' Number of molecular species
Dim sigma(10) As Double ' value of sigma for each species
sigma(0) = 0.00000000030197
Dim radius(10) As Double ' value of core radius for each species
radius(0) = 0.000000000072
Dim edep_div_k(10) As Double ' value of energy depth/k for each species
edep_div_k(0) = 171.3
Dim z_small As Double = 20 ' coordination number for small cage
Dim z_large As Double = 24 'coordination number for large cage
Dim r_small As Double = 0.000000000391 ' radius of small cage
Dim r_large As Double = 0.000000000433 ' radius of large cage
Dim n_small As Double = 1.0 / 23.0 'number of small cage per one water molecule
Dim n_large As Double = 3.0 / 23.0 'number of large cage per one water molecule
'In langmuir constant array c[k][i], suffix k changes with molecular species
' and suffix i changes with cavity type
'In suffix i, 0 indicates small cavity and 1 indicates large cavity.
Dim c1, c2, c3, c4, c5 As Double
Dim c(2, 2) As Double
Dim k As Integer
For k = 0 To molenum
c1 = z_small
c2 = r_small
c3 = sigma(k)
c4 = radius(k)
c5 = edep_div_k(k)
c(k, 0) = langmuir(temp, c1, c2, c3, c4, c5)
Next
' Calculate langmuir constant for large cavity
For k = 0 To molenum
c1 = z_large
c2 = r_large
c3 = sigma(k)
c4 = radius(k)
c5 = edep_div_k(k)
c(k, 1) = langmuir(temp, c1, c2, c3, c4, c5)
Next
**Return (c)**
End Function
' All I am trying to return the value of array (c(k,0) & c(k,1)) but I unable to do that. Can you help please
change the signature to:
Private Function lang_array(ByVal temp As Double) As Double(,)
should work then
I had a Quickbird image and after running the FLAASH, I would like to normalize the image as following: if the pixel>0 or =10000, multiply it by 1; if pixel < or = 0 multiple it by 0; if pixel >0 and <10000 multiple it by its float value and devide the result by 10 000.
I wrote the IDL code as following, but my conditional statement is error. Could you please help me to fix the conditional statement.
Thanks so much for your help.
Lien
My IDL code:
pro correct_reflec
fname='D:\Quick\BA.dat'
envi_open_file, fname, r_fid=fid,NO_REALIZE=1
ENVI_FILE_QUERY,fid,DIMS=dims,NS=ns,NL=nl,NB=nb, pos=pos
map_info=envi_get_map_info(fid=fid)
b1 = ENVI_GET_DATA(FID=fid, dims=dims, pos=1)
ns=n_elements(b1[*,0])
nl=n_elements(b1[0,*])
br=fltarr(ns,3,nl)
CASE 1 of
b1 le 0: br(*,0,*) = (b1)*0
b1 ge 10000: br(*,0,*)= (b1)* 1
else: br(*,0,*)= b1*foat(b1)/(10000)
ENDCASE
b2 = ENVI_GET_DATA(FID=fid, dims=dims, pos=1)
CASE 1 of
b2 le 0: br(*,1,*) = b2*0
b2 ge 10000:br(*,1,*)=b2*1
else: br(*,1,*)=b2*foat(b2)/10000
ENDCASE
b3 = ENVI_GET_DATA(FID=fid, dims=dims, pos=2)
CASE 1 of
b3 le 0:br(*,2,*) = b3*0
b3 ge 10000: br(*,2,*)=b3*1
else br(*,2,*)=b3*foat(b3)/10000
ENDCASE
envi_write_envi_file, br, map_info=map_info, out_name='D:\Quick\test', r_fid=fid
END
You are missing a ":" in the ELSE clause of the third CASE statement. Is that where the syntax error is showing?
I've been doing past paper questions and keep coming up against these questions that deal with 3 valued logic. My notes mention it but don't give examples that relate to those asked in exams.
I understand the basis that True = 1, False = 0 & Unknown = 1/2 as well as And = Min, Or = Max and Not(x) = 1-x. However I do not know how to apply it to questions such as those below:
In SQL, discuss the possible truth values of the following expression:
R.a > R.b OR R.a <= 0 OR R.b >= 0
Justify your answer.
And:
The phone and age fields of the Owner table might have null values in
them. Considering all possible combinations, show which of the three
truth values might be returned by the expression:
phone = ’141-3304913’ OR age <50 OR age >= 50
Any help in clarifying these for me would be really appreciated :)
I will focus on the concrete example, which is more proper for clarifying things.
Put simply, your logical expression is made of a conjunction of three clauses
C1: phone = '141-3304913'
C2: age < 50
C3: age >= 50
for which tri-boolean logic states that the result is
True, if any clause is true
False, if all clauses are false
Unknown, in all the other cases
Consequently, if the value associated with True is the largest, with False is the smallest, and with Unknown is any intermediate value, then taking the MAX for a conjunction proves correct. Similarly, a disjunction works with the MIN function. Negation works as long as we interpret any value between 0 and 1 (excluded) as Unknown; clearly, if we take 1/2 then the negation function is "stable", but that does not really matter in mathematical terms.
More operatively, the clauses clearly react to the following values (instances) of your phone variable P and your age variable A:
P1 such that P1 = '141-3304913'
P2 such that P2 <> '141-3304913'
P3 such that P3 = NULL
A1 such that A1 < 50
A2 such that A2 >= 50
A3 such that A3 = NULL
In terms of satisfaction of the clauses, we have
P1 -> C1 = 1
P2 -> C1 = 0
P3 -> C1 = 1/2
A1 -> C2 = 1, C3 = 0
A2 -> C2 = 0, C3 = 1
A3 -> C2 = C3 = 1/2
In general there exist 3*3 possible combinations, since each of your two variables takes three possible values:
P1 A1: C1 = 1, C2 = 1, C3 = 0 -> MAX(1,1,0) = 1 -> true
P1 A2: C1 = 1, C2 = 0, C3 = 1 -> MAX(1,0,1) = 1 -> true
P1 A3: C1 = 1, C2 = 1/2, C3 = 1/2 -> MAX(1,1/2,1/2) = 1 -> true
P2 A1: C1 = 0, C2 = 1, C3 = 0 -> MAX(0,1,0) = 1 -> true
P2 A2: C1 = 0, C2 = 0, C3 = 1 -> MAX(0,0,1) = 1 -> true
P2 A3: C1 = 0, C2 = 1/2, C3 = 1/2 -> MAX(0,1/2,1/2) = 1/2 -> unknown
P3 A1: C1 = 1/2, C2 = 1, C3 = 0 -> MAX(1/2,1,0) = 1 -> true
P3 A2: C1 = 1/2, C2 = 0, C3 = 1 -> MAX(1/2,0,1) = 1 -> true
P3 A3: C1 = 1/2, C2 = 1/2, C3 = 1/2 -> MAX(1/2,1/2,1/2) = 1/2 -> unknown
In particular, since C2 and C3 are mutually exclusive, you never get False as a result of the conjunction.
The expression R.a > R.b OR R.a <= 0 OR R.b >= 0 instead presents these cases:
R.a <= 0, R.a > 0, R.a = unknown
R.b >= 0, R.b < 0, R.b = unknown
R.a - R.b > 0, R.a - R.b <= 0, R.a - R.b = unknown
Apparently we have three variables and 27 possible cases, but several related to R.a - R.b can be trivially ruled out.