haproxy: passing variables to lua action - variables

In haproxy, I implemented a simple way to delay clients sending too many requests too fast:
In haproxy.cfg:
acl conn_rate_high sc1_conn_rate(ipsticktable) gt 20
http-request lua.delay_request if conn_rate_high
In the lua file:
function delay_request(txn)
core.msleep(1000)
end
core.register_action("delay_request", { "http-req" }, delay_request);
This works okay, but what if I want to delay more if there are even more requests? I can have multiple delay functions with different delays, and multiple acls and conditions, but this gets clunky really fast.
The sensible thing would be to calculate the delay inside the lua function from the connection rate. My problem is, that I cannot pass the connection rate value to the lua function.
Here is what I tried and does not work:
In haproxy.cfg:
http-request set-var(txn.sc1_conn_rate) sc1_conn_rate(ipsticktable)
In the lua script:
function delay_request(txn)
local cr = tostring(txn:get_var("sc1_conn_rate"))
txn:Alert("conn_rate: ")
txn:Alert(cr)
txn:Alert("\n")
core.msleep(tonumber(cr) * 500)
end
The output in the log is:
Nov 18 14:41:06 haproxy[10988]: conn_rate:
Nov 18 14:41:06 haproxy[10988]: conn_rate:
Nov 18 14:41:06 haproxy[10988]: nil
Nov 18 14:41:06 haproxy[10988]: nil
Nov 18 14:41:06 haproxy[10988]: conn_rate:
Nov 18 14:41:06 haproxy[10988]: conn_rate:
Nov 18 14:41:06 haproxy[10988]: nil
Nov 18 14:41:06 haproxy[10988]: nil
Nov 18 14:41:06 haproxy[10988]: .
Nov 18 14:41:06 haproxy[10988]: .
Nov 18 14:41:06 haproxy[10988]: Lua function 'delay_request': runtime error: /etc/haproxy/delay.lua:6: attempt to perform arithmetic on a nil value from /etc/haproxy/delay.lua:6 C function line 1.
Nov 18 14:41:06 haproxy[10988]: Lua function 'delay_request': runtime error: /etc/haproxy/delay.lua:6: attempt to perform arithmetic on a nil value from /etc/haproxy/delay.lua:6 C function line 1.
So, as far as I understand, get_var got nil for some reason. :(
My main question is: How do I pass the value of sc1_conn_rate(ipsticktable) to the lua function?
Bonus question: Why are all alerts printed twice?

Related

Circuits v3.2 Manager.py **events.effects** Issue

Has anyone noticed errors or event.effect errors when using Version 3.2 of Circuits? I've received the message:
Apr 15 03:48:05 **** resilient-circuits[11462]: event.effects -= 1
Apr 15 03:48:05 **** resilient-circuits[11462]: *AttributeError: 'ThreatServiceLookupEvent' object has no attribute 'effects'*
which seems to be coming from circuits. Specifically line 738 of the the Manager.py script.
[Manager.py]: <https://github.com/circuits/circuits/blob/59b2a7be553fa788d06e7575b39a5eb2ec96f884/circuits/core/manager.py#L738>

AWK complains about number of fields when extracting variables

I have a script to parse a TeamCity directory map file. The script works, but I want to know why refactoring it into using variables breaks it with a seemingly unrelated error message and how I can still have it work using variables.
MAP=/opt/TeamCity/buildAgent/work/directory.map
sed -n -e '1,3d;1,/#/{/#/!p}' $MAP | \
awk ' {
n=split($0, array, "->");
printf(substr(array[1], 6) substr(array[2],2,16) "\n");
}
'
This prints
nicecorp::Master 652293808ace4eb5
nicecorp::Reset Database 652293808ace4eb5
nicecorp::test-single-steps 652293808ace4eb5
nicecorp::Develop 652293808ace4eb5
nicecorp::Pull Requests 652293808ace4eb5
Which is pretty much what I want.
The refactoring that breaks
But then I was trying to extract the sub strings into variables, and the script broke. I changed the last printf statement into this
proj=substr(array[1], 6);
tcdir=substr(array[2],2,16);
printf($proj" " $tcdir);
That just prints this error, although I thought it was more or less the same?
awk: program limit exceeded: maximum number of fields size=32767
FILENAME="-" FNR=1 NR=1
This error seems a bit weird, given that my total input is about 500 bytes, 60 times less than the limit they complain about with regards to fields.
AWK version: mawk (1994)
Data format ($ head -10 directory.map):
#Don't edit this file!
#Nov 5, 2019 1:49:26 PM UTC
--version=2
bt30=nicecorp::Master -> 652293808ace4eb5 |?| Oct 29, 2019 4:14:27 PM UTC |:| default
bt32=nicecorp::Reset Database -> 652293808ace4eb5 |?| Oct 30, 2019 1:01:48 PM UTC |:| default
bt33=nicecorp::test-single-steps -> b96874cc9acaf874 |?| Nov 4, 2019 4:20:13 PM UTC |:| default
bt33=nicecorp::test-single-steps -> 652293808ace4eb5 |?| Nov 5, 2019 9:00:37 AM UTC |:| default
bt28=nicecorp::Develop -> 652293808ace4eb5 |?| Nov 5, 2019 1:07:53 PM UTC |:| default
bt29=nicecorp::Pull Requests -> 652293808ace4eb5 |?| Nov 5, 2019 1:18:08 PM UTC |:| default
#
The source of the problem is that the print statement in the refactor is using shell notation for variable ($proj instead of proj, $tcdir instead of tcdir).
When those values are numeric (e.g., tcdir=652293808ace4eb5 for the first line), awk (mawk in this case) will try to print 652293808-th column. Current version of gawk will not fail here - they will realize there are only few columns, and will show empty string for those field (or the full line for $0, if the value is non numeric)
Older version may attempt to extend the field list array to match the requested number, resulting in limit exceeded message.
Also note two minor issues - refactored code uses proj as format - it will get confused if '%' is included. Also, missing newlines. Did you really mean printf and not print ?
Fix:
proj=substr(array[1], 6);
tcdir=substr(array[2],2,16);
# Should consider print, instead of printf
printf(proj " " tcdir "\n");
# print proj, tcdir
The problem was syntax. I was using the shell style $tcdir to insert the value of the variable instead of simply tcdir. By (some unknown to me) means the tcdir portion of $tcdir is resolved to some numeric field value, meaning I am trying to print the value of a field, not the variable tcdir.

showing results of tcp-variants-comparison.cc under ns3 3.28

I am looking for a way to show the results of the file "tcp-variants-comparison.cc" under ns3 (3.28) used with Ubuntu 18.04.
I found here an old topic from 2013, but it seems not to work correctly in my current environment.
P.S: I am a newbie in ns3, so i will appreciate any help.
regards
cedkhader
Running ./waf --run "tcp-variants-comparison --tracing=1" yields the following files:
-rw-rw-r-- 1 112271415 Aug 5 15:52 TcpVariantsComparison-ascii
-rw-rw-r-- 1 401623 Aug 5 15:52 TcpVariantsComparison-cwnd.data
-rw-rw-r-- 1 1216177 Aug 5 15:52 TcpVariantsComparison-inflight.data
-rw-rw-r-- 1 947619 Aug 5 15:52 TcpVariantsComparison-next-rx.data
-rw-rw-r-- 1 955550 Aug 5 15:52 TcpVariantsComparison-next-tx.data
-rw-rw-r-- 1 38 Aug 5 15:51 TcpVariantsComparison-rto.data
-rw-rw-r-- 1 482134 Aug 5 15:52 TcpVariantsComparison-rtt.data
-rw-rw-r-- 1 346427 Aug 5 15:52 TcpVariantsComparison-ssth.data
You can use other command line arguments to generate the desired output, see list below.
Program Arguments:
--transport_prot: Transport protocol to use: TcpNewReno, TcpHybla, TcpHighSpeed, TcpHtcp, TcpVegas, TcpScalable, TcpVeno, TcpBic, TcpYeah, TcpIllinois, TcpWestwood, TcpWestwoodPlus, TcpLedbat [TcpWestwood]
--error_p: Packet error rate [0]
--bandwidth: Bottleneck bandwidth [2Mbps]
--delay: Bottleneck delay [0.01ms]
--access_bandwidth: Access link bandwidth [10Mbps]
--access_delay: Access link delay [45ms]
--tracing: Flag to enable/disable tracing [true]
--prefix_name: Prefix of output trace file [TcpVariantsComparison]
--data: Number of Megabytes of data to transmit [0]
--mtu: Size of IP packets to send in bytes [400]
--num_flows: Number of flows [1]
--duration: Time to allow flows to run in seconds [100]
--run: Run index (for setting repeatable seeds) [0]
--flow_monitor: Enable flow monitor [false]
--pcap_tracing: Enable or disable PCAP tracing [false]
--queue_disc_type: Queue disc type for gateway (e.g. ns3::CoDelQueueDisc) [ns3::PfifoFastQueueDisc]
--sack: Enable or disable SACK option [true]
in ns3.36.1 I used this command
./ns3 run examples/tcp/tcp-variants-comparison.cc -- --tracing=1
and output look like this
TcpVariantsComparison-ascii
TcpVariantsComparison-cwnd.data
TcpVariantsComparison-inflight.data
TcpVariantsComparison-next-rx.data
TcpVariantsComparison-next-tx.data
TcpVariantsComparison-rto.data
TcpVariantsComparison-rtt.data
TcpVariantsComparison-ssth.data

Cratedb: One node went down abruptly from 2 node cluster

We are using 2 node cratedb cluster (v2.3.4). It was running fine for more than a month without any issues. Recently we came to know that one node went away without any external interference. We are unable to find the Root cause for this incident.
Below are the logs. Please help.
Apr 12 23:47:04 STATS-DB-M crate[162556]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
Apr 12 23:47:04 STATS-DB-M crate[162556]: at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Apr 12 23:47:04 STATS-DB-M crate[162556]: [2018-04-12T23:47:04,027][WARN ][o.e.c.a.s.ShardStateAction] [crate3] [online_dlr_report_cache_20180412][7] received shard failed for shard id [[online_dlr_report_cache_20180412][7]], allocation id [NahsM0yfRPaHA5waOpu5OA], primary term [2], message [mark copy as stale]
Apr 12 23:47:04 STATS-DB-M crate[162556]: [2018-04-12T23:47:04,027][WARN ][o.e.c.a.s.ShardStateAction] [crate3] [online_dlr_report_cache_20180412][1] received shard failed for shard id [[online_dlr_report_cache_20180412][1]], allocation id [haMsWkQGTe-yTIfGSkLbHw], primary term [2], message [mark copy as stale]
Apr 12 23:47:04 STATS-DB-M crate[162556]: [2018-04-12T23:47:04,026][WARN ][o.e.c.a.s.ShardStateAction] [crate3] [online_dlr_report_cache_20180412][1] received shard failed for shard id [[online_dlr_report_cache_20180412][1]], allocation id [ZfHGc1DiTZmJ2JQ3YoA_Yg], primary term [1], message [failed to perform indices:crate/data/write/upsert on replica [online_dlr_report_cache_20180412][1], node[1RRQy42EQ8meT7S40loaEw], [R], s[STARTED], a[id=ZfHGc1DiTZmJ2JQ3YoA_Yg]], failure [RemoteTransportException[[crate3][192.168.1.50:4300][indices:crate/data/write/upsert[r]]]; nested: IllegalStateException[active primary shard cannot be a replication target before relocation hand off [online_dlr_report_cache_20180412][1], node[1RRQy42EQ8meT7S40loaEw], [P], s[STARTED], a[id=ZfHGc1DiTZmJ2JQ3YoA_Yg], state is [STARTED]]; ]
Apr 12 23:47:04 STATS-DB-M crate[162556]: org.elasticsearch.transport.RemoteTransportException: [crate3][192.168.1.50:4300][indices:crate/data/write/upsert[r]]
Apr 12 23:47:04 STATS-DB-M crate[162556]: Caused by: java.lang.IllegalStateException: active primary shard cannot be a replication target before relocation hand off [online_dlr_report_cache_20180412][1], node[1RRQy42EQ8meT7S40loaEw], [P], s[STARTED], a[id=ZfHGc1DiTZmJ2JQ3YoA_Yg], state is [STARTED]
Apr 12 23:47:10 STATS-DB-M systemd[1]: crate.service: main process exited, code=exited, status=126/n/a
Apr 12 23:47:10 STATS-DB-M systemd[1]: Unit crate.service entered failed state.
Apr 12 23:47:10 STATS-DB-M systemd[1]: crate.service failed.
the logs don't give a hint on why the node went down. do you have additional information?
generally we'd suggest using a 3 node cluster minimum for being able to have a quorum when a node goes down.
if you have more information let us know.
thanks, joe

How can i get OptaPlanner Construction Heuristics to use different values?

When i use the Construction Heuristics i always get the same values assigned to my PlanningVariables which leads to a bad initial solution.
I am quite sure i make an error here but cannot see where i fail.
My Problem Description:
I try to assign resources to WorkOrders. One WorkOrder can have several ResourceAssignments. This ResourceAssignment is my main PlanningEntity.
The resource is the PlanningVariable.
WorkOrders are also PlanningEntities but that information is only necessary here to show i have more than one PlanningEntity and therefore cannot go with the more simple ConstructionHeuristic configurations.
I use value range provider from entity so i can give back different sets of Resources for different types of ResourceAssignments. (e.g. Human, Forklift, Handscanner ...)
My problem is that the Construction Heuristic seems to assign always the same Resource to all ResourceAssignment.
Here is the output from the CH - as you can see the same Human Resource (id 100) is assigned to both ResourceAssignments. This is the same for all following PlanningEntities.
2015-07-23 10:57:12,291 [main] INFO Solving started: time spent (185), best score (uninitialized/-9900hard/0soft), environment mode (FAST_ASSERT), random (JDK with seed 0).
2015-07-23 10:57:12,429 [main] DEBUG CH step (0), time spent (325), score (-9900hard/0soft), selected move count (200), picked move (ResourceAssignment{id=0, workorder=WorkOrder{id=0, name=WO0, start=Thu Jul 23 12:57:11 CEST 2015, stop=Thu Jul 23 14:57:11 CEST 2015, offset=0}, resource=null, type=human} => Resource{type=human, id=100}).
2015-07-23 10:57:12,523 [main] DEBUG CH step (1), time spent (419), score (-9900hard/0soft), selected move count (200), picked move (ResourceAssignment{id=1, workorder=WorkOrder{id=0, name=WO0, start=Thu Jul 23 12:57:11 CEST 2015, stop=Thu Jul 23 14:57:11 CEST 2015, offset=0}, resource=null, type=human} => Resource{type=human, id=100}).
2015-07-23 10:57:12,576 [main] DEBUG CH step (2), time spent (472), score (-9900hard/0soft), selected move count (99), picked move (ResourceAssignment{id=2, workorder=WorkOrder{id=0, name=WO0, start=Thu Jul 23 12:57:11 CEST 2015, stop=Thu Jul 23 14:57:11 CEST 2015, offset=0}, resource=null, type=forklift} => Resource{type=forklift, id=0}).
2015-07-23 10:57:12,667 [main] DEBUG CH step (3), time spent (563), score (-9900hard/0soft), selected move count (200), picked move (ResourceAssignment{id=3, workorder=WorkOrder{id=1, name=WO1, start=Thu Jul 23 12:57:11 CEST 2015, stop=Thu Jul 23 14:57:11 CEST 2015, offset=0}, resource=null, type=human} => Resource{type=human, id=100}).
2015-07-23 10:57:12,757 [main] DEBUG CH step (4), time spent (653), score (-9900hard/0soft), selected move count (200), picked move (ResourceAssignment{id=4, workorder=WorkOrder{id=1, name=WO1, start=Thu Jul 23 12:57:11 CEST 2015, stop=Thu Jul 23 14:57:11 CEST 2015, offset=0}, resource=null, type=human} => Resource{type=human, id=100}).
2015-07-23 10:57:12,795 [main] DEBUG CH step (5), time spent (691), score (-9900hard/0soft), selected move count (99), picked move (ResourceAssignment{id=5, workorder=WorkOrder{id=1, name=WO1, start=Thu Jul 23 12:57:11 CEST 2015, stop=Thu Jul 23 14:57:11 CEST 2015, offset=0}, resource=null, type=forklift} => Resource{type=forklift, id=0}).
2015-07-23 10:57:12,867 [main] DEBUG CH step (6), time spent (763), score (-9900hard/0soft), selected move count (200), picked move (ResourceAssignment{id=6, workorder=WorkOrder{id=2, name=WO2, start=Thu Jul 23 12:57:11 CEST 2015, stop=Thu Jul 23 14:57:11 CEST 2015, offset=0}, resource=null, type=human} => Resource{type=human, id=100}).
2015-07-23 10:57:12,917 [main] DEBUG CH step (7), time spent (813), score (-9900hard/0soft), selected move count (200), picked move (ResourceAssignment{id=7, workorder=WorkOrder{id=2, name=WO2, start=Thu Jul 23 12:57:11 CEST 2015, stop=Thu Jul 23 14:57:11 CEST 2015, offset=0}, resource=null, type=human} => Resource{type=human, id=100}).
It seems like it ignores all other moves - and of course i have a rule that punishes using the same resource at the same time with a hard score.
Here is my solution configuration.
<?xml version="1.0" encoding="UTF-8"?>
<solver>
<environmentMode>FAST_ASSERT</environmentMode>
<termination>
<unimprovedSecondsSpentLimit>10000</unimprovedSecondsSpentLimit>
<secondsSpentLimit>220000</secondsSpentLimit>
</termination>
<!-- Domain model configuration -->
<solutionClass>com.opal.solver.resource.ResourceAssignmentSolution</solutionClass>
<entityClass>com.opal.solver.resources.entity.ResourceAssignment</entityClass>
<entityClass>com.opal.solver.resources.entity.WorkOrder</entityClass>
<!-- Score configuration -->
<scoreDirectorFactory>
<scoreDefinitionType>HARD_SOFT</scoreDefinitionType>
<scoreDrl>resourceAsssignmentRules.drl</scoreDrl>
</scoreDirectorFactory>
<constructionHeuristic>
<queuedEntityPlacer>
<entitySelector id="placerEntitySelector">
<cacheType>PHASE</cacheType>
<entityClass>com.opal.solver.resources.entity.ResourceAssignment</entityClass>
</entitySelector>
<changeMoveSelector>
<entitySelector mimicSelectorRef="placerEntitySelector"/>
<valueSelector>
<selectionOrder>ORIGINAL</selectionOrder>
<variableName>resource</variableName>
</valueSelector>
</changeMoveSelector>
</queuedEntityPlacer>
</constructionHeuristic>
</solver>
I am sure I just miss some core concept of optaplanner here. I would be grateful for any hint into the right direction.
Looks like you have a verbose configuration of FIRST_FIT right now. Use FIRST_FIT_DECREASING instead, by declaring difficulty comparison and using that construction heuristic type.
<constructionHeuristic>
<constructionHeuristicType>FIRST_FIT_DECREASING</>
</>
See docs chapter on First Fit Decreasing. Normally, FFD is clearly better than FF (for example 4% on CloudBalancing on average), but use Local Search to improve upon that solution.
If you're really looking for random different CH results (for example for GeneticAlgorithm population) use selectionOrder RANDOM instead of ORGINAL.