I'm using the mongodb3 cookbook (took from chef supermarket) in my environment. When ever we pass input for mongos template like [10.10.0.10,127.0.0.1] , the output file is created with improper syntax:
PFB out put -
net:
port: 27017
bindIp: ! '[10.10.0.10,127.0.0.1]'
There's a ! and '' around the square brackets instead of only [10.10.0.10,127.0.0.1]
I am using below function to convert.
module Mongodb3Helper
def mongodb_config(config)
config.to_hash.compact.to_yaml
end
end
class Hash
def compact
inject({}) do |new_hash, (k, v)|
if v.is_a?(Hash)
v = v.compact
new_hash[k] = v unless v.empty?
else
new_hash[k] = v unless v.nil?
end
new_hash
end
end
end
Your code, while horribly unmaintainable due to making changes to global objects, is fine. As pointed out several times in the comments, the problem is with the data you are feeding in to that code.
Related
I recently got into camelot and but for some reason need to use my own backend to remove text during the imaging process.
Here is what I tried based on the documentation:
class ConversionBackend(object):
def convert(pdf_path, png_path):
arg2 = '-sOutputFile=' + png_path
p = subprocess.Popen(['/usr/bin/gs', '-sDEVICE=png16m', '-dNOPAUSE', '-dBATCH', '-dQUIET', "-dFILTERIMAGE", "-dFILTERTEXT", "-r300",str(arg2), pdf_path], stdout = subprocess.PIPE)
pass`
Aside from the fact that my ghostscript use is not optimal, all of this runs does work when I run it step by step in the python console. However when calling camelot using
tables = camelot.read_pdf(path, line_scale=100, split_text=True, flag_size=True, layout_kwargs={'detect_vertical': False, 'char_margin': 2.0}, pages='all', backend=ConversionBackend())
Camelot still executes my command but with complete disregard for the backend=ConversionBackend()
Any ideas on how to fix this?
Sh4yce
I have been puzzled by how to define and use custom module in Julia.
For example, I defined a module named myMoldule to wrap a mutable struct Param and a function add in D:\\run\\defineModule.jl:
module myMoldule
export Param, add
mutable struct Param
x ::Int64
y ::Int64
end
function add(x::Int64, y::Int64)
sum ::Int64
sum = x + y
return sum
end
end
and used this module in D:\\run\\useModule.jl like:
include("D:\\run\\defineModule.jl")
using .myMoldule
function testModule()
param = Param(1, 2)
sum = add(param.x, param.y)
println(sum)
end
An error occurred when running testModule() as follows:
julia> testModule()
ERROR: UndefVarError: Param not defined
Stacktrace:
[1] testModule() at D:\run\useModule.jl:8
[2] top-level scope at none:1
Note that I used the absolute path in the include(...) to avoid using LOAD_PATH stuff, and added . before the module name (i.e., using .myMoldule).
What seems to be the problem?
P.S.: Julia version information:
julia> versioninfo()
Julia Version 1.5.2
Commit 539f3ce943 (2020-09-23 23:17 UTC)
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: Intel(R) Core(TM) i7-8700K CPU # 3.70GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
Environment:
JULIA_DEPOT_PATH = C:\Users\f\.julia;C:\opt\JuliaPro-1.5.2-1\Julia-1.5.2\local\share\julia;C:\opt\JuliaPro-1.5.2-1\Julia-1.5.2\share\julia
JULIA_LOAD_PATH = #;#v#.#;#stdlib
JULIA_NUM_THREADS = 6
JULIA_PKG_SERVER = pkg.juliahub.com
Corrections to be made:
sum is a function in Base you should use a different name
no need to declare sum variable (and it should be named something like mysum)
Remove space before ::
Module names should start with a CapitalLetter
You have a typo in module name perhaps you are loading a different module than you think?
Once corrected your code works.
The issue happens because you are using .MyModule1, not in the MyModule2, and thus you import Param to the Main module but not to the MyModule2, thus Module2 does not see Param.
If you will put using ..MyModule1 (two dots instead of one as you there is one more level) into MyModule2 this issue will go.
However, your code still will not work, since julia's include function just runs all the content of the included file, thus even you included the same file, you will create different instances of the modules. This warning WARNING: replacing module... indicates that somewhere in your code you might use the different version of the module (in you case, Main.Module1 and Main.Module2.Module1).
The common practice in julia is to include all the files in one place (they should be included only once). For instance, you can put all the includes in the file useModule.jl:
include("./defineModule1.jl")
include("./defineModule2.jl")
using .MyModule1
using .MyModule2
function testModule()
param = Param(1, 2)
# call myAdd to get the sum of param.x and param.y
sumValue = myAdd(param)
println(sumValue)
# call mySubtract to get the difference of param.x and param.y
difValue = mySubtract(param)
println(difValue)
end
Do not include files in other places. e.g.
defineModule2.jl content:
module MyModule2
using ..MyModule1
export myAdd, mySubtract
function myAdd(param::Param)
return param.x + param.y
end
function mySubtract(param::Param)
return param.x - param.y
end
end # end of module
Questioner's note: The following new question voted downwards (perhaps) was extended form the original one and was well answered by Vitaliy Yakovchuk.
I fixed all the issues pointed out by Przemyslaw Szufel. In my case above, it's not the improper ways of naming that cause the issue.
Now, I have a better exmaple to clarify my issue.
Suppose that, to meet the needs, I have to seperate my julia source code into two modules, e.g., define of a mutable struct Param in defineModule1.jl and define of functions in defineModule2.jl. The code scripts are as follows:
"D:\\run\\defineModule1.jl":
module MyModule1
export Param
mutable struct Param
x::Int64
y::Int64
end
end # end of module
"D:\\run\\defineModule2.jl":
include("D:\\run\\defineModule1.jl"); using .MyModule1
module MyModule2
export myAdd, mySubtract
function myAdd(param::Param)
return param.x + param.y
end
function mySubtract(param::Param)
return param.x - param.y
end
end # end of module
Note that Param is not defined here, and to make Param available, a line include("D:\\run\\defineModule1.jl"); using .MyModule1 is added as the first line of this file.
"D:\\run\\useModule.jl":
include("D:\\run\\defineModule1.jl"); using .MyModule1
include("D:\\run\\defineModule2.jl"); using .MyModule2
function testModule()
param = Param(1, 2)
# call myAdd to get the sum of param.x and param.y
sumValue = myAdd(param)
println(sumValue)
# call mySubtract to get the difference of param.x and param.y
difValue = mySubtract(param)
println(difValue)
end
Note that both function myAdd(param) and mySubtract(param) in the script defineModule2.jl need the predefined mutable struct Param in defineModule1.jl.
This is what I got when I run D:\\run\\useModule.jl:
julia> include("D:\\run\\useModule.jl")
WARNING: replacing module MyModule1.
WARNING: replacing module MyModule1.
WARNING: replacing module MyModule2.
ERROR: LoadError: LoadError: UndefVarError: Param not defined
Stacktrace:
[1] top-level scope at D:\run\defineModule2.jl:7
[2] include(::String) at .\client.jl:457
[3] top-level scope at D:\run\useModule.jl:2
[4] include(::String) at .\client.jl:457
[5] top-level scope at none:1
in expression starting at D:\run\defineModule2.jl:7
in expression starting at D:\run\useModule.jl:2
I believed that, by using the following lines in the beginning of "D:\run\useModule.jl", the mutable struct Param should have be found:
include("D:\\run\\defineModule1.jl"); using .MyModule1
include("D:\\run\\defineModule2.jl"); using .MyModule2
...
Still, error LoadError: UndefVarError: Param not defined is reported.
So, Why can't Param be found by D:\\run\\useModule.jl?
Before ask my question, I have to explain you my project and how I'm working. I make a server for a game. This server need a lot of development, 800k lines of code.
After more than one month. It appears there is a cheat some players are using to destroy player's fun.
I have to admit it's a bit hard for me to figure out how to defend myself about this. The server owners community are just under attack too much and everyone is trying to defend himself as they can.
I'm working in Lua, JSON, HTML and CSS for this project, 99% Lua.
What things the game client has control over :
The client can do commands but commands are only allowed to admin. Except if the injection of his own file make a new command. And can inject code inside the client game(explain after).
What the server does :
I guess, here is where we may can do something. All servers have tons of mods installed, so all client are downloading all my mod/client scripts to trigger my servers events.
What is the process of sending a command :
the client can use chat /command [attribut][attribut2]
the client can run the client's console and : command [attribut][attribut2]
How cheaters are doing it?
1) Injector hidden in the memory with windows executable.
2) They are loading a client file to make clients commands to call server triggers*.
(because, yes, it's poor everyone is using same scripts so it's known triggers names)
*) To do that they open a client console and writing inside : "exec c:/filename.lua" )
50% of cheat scripts are obfuscated with AHCI & variable replacement like the following.
What does a cheat look like?
I cannot post the full code here >Body is limited to 30000 characters; you entered 208906.< >obfuscated< >not indented(1 ligne)<
-A global function :
function IllIlllIllIlllIlllIlllIll(IllIlllIllIllIll) if (IllIlllIllIllIll==(((((919 + 636)-636)*3147)/3147)+919))end end)
-A global variable :
IllIIllIIllIII(IllIIIllIIIIllI(IlIlIlIlIlIlIlIlII,IIIIIIIIllllllllIIIIIIII))()
-A global variable :
Xxxxx5 = {}
-A global variable :
Xxxxx5.debug = true
-A local variable :
local logged = true
Where I need you help
I would like to detect not the injection. But, I have no control at all, on this. I would like to detect one of these thing : (Remember I can inject my client lua / Json scripts to do things)
when someone is typing : exec in the console ?
when someone is using a global variable/function ? (since I got them "Xxxxx5.debug = true")
when someone is trying to detect or scan my triggers ?
something I didn't think yet ?
Detect if function exists ?
I have global and local function in the client script :
local function d(e)
local f = {}
local h = GetGameTimer() / 200
f.r = math.floor(math.sin(h * e + 0) * 127 + 128)
f.g = math.floor(math.sin(h * e + 2) * 127 + 128)
f.b = math.floor(math.sin(h * e + 4) * 127 + 128)
return f
end
function Dt(text, x, y)
..stuff
end
A good information is :
I succeed to do a good protection but it's very basic....
Most of cheats are using the same commands to be oppened. I just patched the shordcuts to acces them. Client get kick if they use them.
Maybe it's a solution if better coded ? But, if player is in the chat or console it's no more working so I don't figure out how it's working when player is in this state. (Key disabled)
Why I'm asking for help ?
I can assure I dev all my scripts, reworked all my downloaded scripts, find all other solutions, without any help. I prefer learn to fish than ask for a fish. But this time tried many things that are not working.
What I've tried : no one of these conditions worked :
-- if (_G["Xxxxx.IsMenuOpened"] ~= nil) then
-- while true do
-- print = 'Xxxxx'
-- end
-- end
-- if Xxxxx ~= nil then
-- setfenv(Xxxxx , nil )
-- end
-- if rawget( _G, Xxxxx) ~= nil then -- "var" n'est pas déclaré
-- print 'Xxxxx DETECTER !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!'
-- end
-- if rawget( _G, Xxxx) ~= nil then -- "var" n'est pas déclaré
-- print 'Xxxxx DETECTER !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!'
-- end
-- if (_G["Xxxx.IsAnyMenuOpened"] ~= nil) then
-- while true do
-- print = 'Xxxxx'
-- end
-- print = 'xxxxx'
-- end
-- if source == '[Xxxx]' or source == 'Xxxxx' then
-- while true do
-- print = 'Xxxxx'
-- end
-- print = 'xxxxx'
-- end
-- if (_G["Xxxxx"] ~= nil) then
-- while true do
-- print = 'Xxxxx'
-- end
-- print = 'xxxxx'
-- end
-- if Xxxxx ~= nil or Lynx8.debug = true or Lynx8.debug = false then
-- reason2 = 'Xxxxx DETECTE'
-- TriggerServerEvent("Drop_player" , reason2 )
-- end
-- if Xxxxx.Display() ~= nil then
-- reason2 = 'Xxxxx DETECTE'
-- TriggerServerEvent("Drop_player" , reason2 )
-- end
After more research... Here I come maybe with a good solution.
I have most of used cheats in my possession. So all of them have global variable and function.
So I searched may there is a way to lock Global variable by using and locking them before cheaters.
http://lua-users.org/lists/lua-l/2003-08/msg00081.html
So, injection will fail because used variable will be lockeD.
I read this : And It look's it's a good solution for me yes.
Now I have more precise questions, may you can help me with this :
I don't know multiples things :
$ cat z.lua
This come from the pasted url. What's this ?
Cound you confirm ?
In lua Or Json, Is there a way to lock variable before use ?
Currently, I am working with a EV3 lego robot that is controlled by several neurons. Now I want to modify the code (running on
python3) in such a way that one can change certain parameter values on the run via the shell (Ubuntu) in order to manipulate the robot's dynamics at any time (and for multiple times). Here is a schema of what I have achieved so far based on a short example code:
from multiprocessing import Process
from multiprocessing import SimpleQueue
import ev3dev.ev3 as ev3
class Neuron:
(definitions of class variables and update functions)
def check_input(queue):
while (True):
try:
new_para = str(input("Type 'parameter=value': "))
float(new_para[2:0]) # checking for float in input
var = new_para[0:2]
if (var == "k="): # change parameter k
queue.put(new_para)
elif (var == "g="): # change parameter g
queue.put(new_para)
else:
print("Error". Type 'k=...' or 'g=...')
queue.put(0) # put anything in queue
except (ValueError, EOFError):
print("New value is not a number. Try again!")
(some neuron-specific initializations)
queue = SimpleQueue()
check = Process(target=check_input, args=(queue,))
check.start()
while (True):
if (not queue.empty()):
cmd = queue.get()
var = cmd[0]
val = float(cmd[2:])
if (var == "k"):
Neuron.K = val
elif (var == "g"):
Neuron.g = val
(updating procedure for neurons, writing data to file)
Since I am new to multiprocessing there are certainly some mistakes concerning taking care of locking, efficiency and so on but the robot moves and input fields occur in the shell. However, the current problem is that it's actually impossible to make an input:
> python3 controller_multiprocess.py
> Type 'parameter=value': New value is not a number. Try again!
> Type 'parameter=value': New value is not a number. Try again!
> Type 'parameter=value': New value is not a number. Try again!
> ... (and so on)
I know that this behaviour is caused by putting the exception of EOFError due to the fact that this error occurs when the exception is removed (and the process crashes). Hence, the program just rushes through the try-loop here and assumes that no input (-> empty string) was made over and over again. Why does this happen? - when not called as a threaded procedure the program patiently waits for an input as expected. And how can one fix or bypass this issue so that changing parameters gets possible as wanted?
Thanks in advance!
My application is crashing really, really hard, and it appears to be related to the database. The application deals with lots and lots of data, and hundreds of simultaneous users. In an effort to speed up data loads, I am loading some records like this:
def load(filename)
rc = Publication.connection.raw_connection
rc.exec("COPY invoice_line_items FROM STDIN WITH CSV HEADER")
# open up your CSV file looping through line by line and getting the line into a format suitable for pg's COPY...
error = false
begin
CSV.foreach(filename) do |line|
until rc.put_copy_data( line.to_csv )
ErrorPrinter.print " waiting for connection to be writable..."
sleep 0.1
end
end
rescue Errno => err
User.inform_admin(false, User.me, "Line Item import failed with #{err.class.name} the following error: #{err.message}", err.backtrace)
error = true
else
rc.put_copy_end
while res = rc.get_result
if (res.result_status != 1)
User.inform_admin(false, User.me, "Line Item import result of COPY was: %s" % [ res.res_status(res.result_status) ], "")
error = true
end
end
end
end
I also have Sidekiq running with about 90 threads. Does this method of loading put an exclusive lock on that table? Is it possible that these jobs are running into each other? If they are, am I better off just doing inserts?
COPY takes the same level of lock as INSERT. (It's missing from the explicit locking chapter, but visible in the source code). So whatever's giving you trouble, it's probably not that.
You should be looking at pg_locks and pg_stat_activity to see if anything's stuck on a lock. More info on other questions on SO or DBA.SE, the manual, and the PostgreSQL wiki.