Rabbitmq from version 3.7 uses 2 formats:
erlang-style legacy format, used by default in rabbitmq.config file,
properties-style new format, used by default in rabbitmq.conf file.
https://blog.rabbitmq.com/posts/2018/02/new-configuration-format-in-rabbitmq-3-7/
New options that appear in rabbitmq are often documented only in new format and it is not obvious how to achieve the same setting in the legacy format.
Where can I find such mapping?
That article mentioned in the question says that new format is translated on the fly using Cuttlefish and it uses rabbit.schema mapping file to control this translation. You can manually find there how new properties are translated to legacy format.
Example:
Because of this line in rabbit.schema
{mapping, "heartbeat", "rabbit.heartbeat", [{datatype, integer}]}.
we can infer that new format entry
heartbeat = 60
will translate to legacy format
[{
rabbit, [
{heartbeat, 60}
]
}].
See also:
new config template
old config template
Related
I have multiple ecospold (version 1) files with LCI data that I want to convert to a custom excel format. I need all data given in the ecospold file. For my own convinience I want to use python to complete this task.
My research until now has lead me to the following conclusions:
There exist at least two converters (by GLAD and openLCA) to convert ecospold formats (1 and 2) to e.g. the ILCD. But those formats are not helping me to go anywhere, since I need to have all the data accessible in python and in order to then write it into my custom excel format.
To get the data in python, the package pyecospold (https://github.com/sami-m-g/pyecospold) seems to be a suitable choice.
According to the README that can be found at the pyecospold github repository,
ecoSpold = parse_file("data/v1/v1_1.xml") # Replace with your own XML file
should do the job. So I implemented the following lines:
import os
from pyecospold import parse_file, save_file, Defaults
from lxml import etree
cd = os.getcwd()
path_input = cd + r'\inputs\ecospold_test.xml'
# Parse the required XML file to EcoSpold class.
es = parse_file('inputs/ecospold_test.xml')
Now I run into the error:
TypeError: parse_file() missing 2 required positional arguments: 'schema_path' and 'ecospold_lookup'
I understood that a schema in xsd format is needed, therefore I got the schema files from the github and amended my last line of code:
es = parse_file('inputs/ecospold_test.xml', 'inputs/schemas/v1/EcoSpold01Dataset.xsd')
Now there is still one argument missing:
TypeError: parse_file() missing 1 required positional argument: 'ecospold_lookup'
Since I have no experience in parsing xml files in python, I have no idea what to do with this. Additionally, I am confused why the README does not say anything about those additionally needed arguments.
My second idea was to use brightway to get the data into python. But since brightway itself is quite an extensive package, I could not find a simple (or any) way to do this. (Sadly, the notebooks linked in the answer of this question Import Ecoinvent 2.2 Ecospold files into Brightway do not exist anymore)
Another option would of course be to write my own parser. But because I am lacking experience and pyecospold does exactly this (at least in my understanding), I would like to avoid this option.
Additionally, there in openLCA it is possible to read in ecospold files and then export them to an excel format. From this excel format I could of course make my custom excel format. The problem here is that I have no idea how to automize this, because I do not want to read in and export each file individually and manually in openLCA.
If anyone has an idea on how to solve one of my subproblems or a good alternative on how to solve my general problem, I would be very thankful. :)
I have a json coming in like this:
{
"app" : "hw",
"content" : "hello world",
"time" : "2018-05-06 12:53:04"
}
I wish to push to S3 in the following file format:
/upper-directory/$jsonfield1/$jsonfield2/$date/$HH
I know I can achieve:
/upper-directory/$date/$HH
with TimeBasedPartitioner and Topic.dir, but how do I put in the 2 json fields as well?
You need to write your own Partitioner to achieve a combination of TimeBased and Field Partitioners
That means make a new Java project, look at the source code for a reference point, build a JAR out of the project, and then copy the jar into kafka-connect-storage-common on all servers running Kafka Connect, which is picked up by the S3 connector. After you've copy the JAR, you will need to reboot the Connect process.
Note: there's already a PR that is trying to add this - https://github.com/confluentinc/kafka-connect-storage-common/pull/73/files
I am having few ISO8583 logs in a text file. I want to parse these logs from this text file and write them to any database with some descriptive information such as class of the message, message function, message origin, processing code, response code etc.
I am new to the BASE24/ISO8583 and was trying to find any ready-made parser for this. Is there any such parser available ? Does jPOS provides such functionality ?
EDIT
I have the logs in ISO8583 format in ".log" file as given below:
MTI : 0200
Field-3 : 201234
Field-4 : 000000010000
Field-7 : 0110722180
Field-11 : 123456
Field-44 : A5DFGR
Field-105 : ABCDEFGHIJ 1234567890
This is same as the format given in the link shared by you.
It also consists of hex dump but I dont want to parse that.
The code given in the link is doing packing and unpacking of the message where as what I am trying is to read these logs (in unpacked form) and write them into a database table.
I think i need to write my own code for this and use the jPOS packagers in it.
It really depends on the format of the log file - are the ISO8583 messages - HexStrings, and HexDump an XML representation of ISO8583, some other application trace file ?
Once you know the format and it might require some massaging - you will want to research the ISOMsg.unpack() methods using the appropriate jPOS packager. the packager defines the field structure - of the various ISO8583 fields and field construction (lengths, character set, etc.)
a good example was found at the following blog post: looking at the "Parse (unpack) ISO Message" seciton http://jimmod.com/blog/2011/07/26/jimmys-blog-iso-8583-tutorial-build-and-parse-iso-message-using-jpos-library/
You mention - Base24 - jPOS does have a few packagers that might be close starting point.:
https://github.com/jpos/jPOS/blob/master/jpos/src/dist/cfg/packager/base24.xml
Those human-readable log formats are usually difficult to parse without loosing information. Moreover, the logs are probably PCI compliant so there's a lot of masked information there. You want to ask for ah hex dump of the messages.
what is displayed in log file is parsed ISO.Hence you need not use jpos.jpos is only for packing and unpacking when you transmit the message.
Assign the field to variable and write in DB
for example,Field 39 is response code.
Using jpos is good idea. You should go for your custom packager design class.
When running a Neo4J database server standalone (on Ubuntu 14.04), configuration options are set for the global installation in etc/neo4j/neo4j.conf or possibly $NEO4J_HOME/conf/neo4j.conf.
However, when instantiating a Neo4j database from Java or Scala using Apache's Neo4jGraph class (org.apache.tinkerpop.gremlin.neo4j.structure.Neo4jGraph), there is no global installation, and the constructor does not (as far as I can tell) look for any configuration files.
In particular, when running the test suite for my application, I end up with many simultaneous instances of Neo4jGraph, which ends up throwing a java.net.BindException: Address already in use because all of these instances are trying to communicate over a small range of ports for online backup, which I don't actually need. These channels are set with config options dbms.backup.address (default value: 127.0.0.1:6362-6372) and dbms.backup.enabled (default value: true).
My problem would be solved by setting dbms.backup.enabled to false, or expanding the port range.
Things that have not worked:
Creating /etc/neo4j/neo4j.conf containing the line dbms.backup.enabled=false.
Creating the same file in my project's src/main/resources directory.
Creating the same file in src/main/resources/neo4j.
Manually setting the configuration property inside the Scala code:
val db = new Neo4jGraph(dataDirectory)
db.configuration.addProperty("dbms.backup.enabled",false)
or
db.configuration.addProperty("neo4j.conf.dbms.backup.enabled",false)
or
db.configuration.addProperty("gremlin.neo4j.conf.dbms.backup.enabled",false)
How should I go about setting this property?
Neo4jGraph configuration through TinkerPop is accomplished by a pass-through of configuration keys. In TinkerPop 3.x, that would mean that all Neo4j keys prefixed with gremlin.neo4j.conf that are provided via Configuration object to Neo4jGraph.open() or GraphFactory.open() will be passed down directly to the Neo4j instance. You can see examples of this here in the TinkerPop documentation on high availability configuration.
In TinkerPop 2.x, the same approach was taken however the key prefix was instead blueprints.neo4j.conf.* as discussed here.
Manipulating db.configuration after the database connection had already been opened was definitely futile.
stephen mallette's answer was on the right track, but this particular configuration doesn't appear to pass through in the way his linked example does. There is a naming mismatch between the configuration keys expected in neo4j.conf and those expected in org.neo4j.backup.OnlineBackupKernelExtension. Instead of dbms.backup.address and dbms.backup.enabled, that class looks for config keys online_backup_server and online_backup_enabled.
I was not able to get these keys passed down to the underlying Neo4jGraphAPI instance correctly. What I had to do, instead, was the following:
import org.neo4j.tinkerpop.api.impl.Neo4jFactoryImpl
import scala.collection.JavaConverters._
val factory = new Neo4jFactoryImpl()
val config = Map(
"online_backup_enabled" -> "true",
"online_backup_server" -> "0.0.0.0:6350-6359"
).asJava
val db = Neo4jGraph.open(factory.newGraphDatabase(dataDirectory,config))
With this initialization, the instance correctly listened for backups on port 6350; changing "true" to "false" disabled backup listening.
Using Neo4j 3.0.0 the following disables port listening for me (Java code)
import org.apache.commons.configuration.BaseConfiguration;
import org.apache.tinkerpop.gremlin.neo4j.structure.Neo4jGraph;
BaseConfiguration conf = new BaseConfiguration();
conf.setProperty(Neo4jGraph.CONFIG_DIRECTORY, "/path/to/db");
conf.setProperty(Neo4jGraph.CONFIG_CONF + "." + "dbms.backup.enabled", "false");
graph = Neo4jGraph.open(config);
For example, the .gitattributes file of lg2s has the line *.cs diff=csharp. the output of the codes
using (var repo = new Repository(#"path\to\lg2s"))
{
var tree1 = repo.Lookup<Commit>("a845db9").Tree;
var tree2 = repo.Lookup<Commit>("d677741").Tree;
var patches = repo.Diff.Compare<Patch>(tree1, tree2);
foreach (var patch in patches)
{
Console.WriteLine(patch.Patch);
}
}
is (shortly)
diff --git a/LibGit2Sharp/RepositoryStatus.cs b/LibGit2Sharp/RepositoryStatus.cs
## -59,8 +59,8 ## namespace LibGit2Sharp
but the output of git bash is (shortly)
diff --git a/LibGit2Sharp/RepositoryStatus.cs b/LibGit2Sharp/RepositoryStatus.cs
## -59,8 +59,8 ## internal RepositoryStatus(Repository repo, StatusOptions optio
the second line not same.
Looks like there are an in-depth discussion Add .gitattributes support libgit2/#508, but the PR Add APIs for git attributes libgit2/#516 is merged or not?
How many languages were supported now?
Can we specify the rules without gitattributes file?
This is actually an issue of diff drivers and custom function context rules for diffs, not one of attributes.
Those files do have the diff=csharp attribute; the problem is that the attribute doesn't mean anything to libgit2.
When git and libgit2 generate diffs, they use regular expressions to find the name of the function in which a diff hunk occurs. They also have a fallback rule if no regular expression is available to find the first line that did not start with whitespace.
git contains a table of built in diff drivers that includes a definition of the csharp driver.
libgit2 does not contain that table (as discussed in the PR that merged diff driver support) and hence has no built in notion of what pattern to use for diff=csharp.
Your alternatives for solving this are either:
Make an entry in your .git/config file (or your $HOME/.gitconfig) that implements diff.csharp.xfuncname with a pattern to search, or
Start a Pull Request to libgit2 that imports the driver definitions. The issue with importing (and the reason I didn't take it on) is that it will involve getting permission from the original authors of the code to relicense it for use in libgit2).
If you decide to start a Pull Request and you get the author of the CSharp xfuncname patterns to agree to relicense the code for libgit2, let me know and I'll be happy to collaborate on the actual implementation!