Puppet. Change any config/text file in most efficient way - config

I am learning how to use Puppet. An now I am trying to change config file for nscd. I need to change such lines:
server-user nscd
paranoia yes
And let's suppose that full config looks as next:
$ cat /etc/nscd/nscd.conf
logfile /var/log/nscd.log
threads 4
max-threads 32
server-user nobody
stat-user somebody
debug-level 0
reload-count 5
paranoia no
restart-interval 3600
Previously I have wrote such module for replacing needed lines and it looks as follow:
include nscd
class nscd {
define line_replace ($match) {
file_line { $name:
path => '/etc/nscd/nscd.conf',
line => $name,
match => $match,
notify => Service["nscd"]
}
}
anchor{'nscd::begin':}
->
package { 'nscd':
ensure => installed,
}
->
line_replace {
"1" : name => "server-user nscd", match => "^\s*server-user.*$";
"2" : name => "paranoia yes", match => "^\s*paranoia.*$";
}
->
service { 'nscd':
ensure => running,
enable => "true",
}
->
anchor{'nscd::end':}
}
Is it possible to make the same in more efficient way? With arrays or like that?

I recommend you to use the inifile puppet module to easy manage INI-style files like this, but also you can take advantage of the create_resources function:
include nscd
class nscd {
$server_user_line = 'server-user nscd'
$paranoia_line = 'paranoia yes'
$defaults = {
'path' => '/etc/nscd/nscd.conf',
'notify' => Service["nscd"],
}
$lines = {
$server_user_line => {
line => $server_user_line,
match => "^\s*server-user.*$",
},
$paranoia_line => {
line => $paranoia_line,
match => "^\s*paranoia.*$",
}
}
anchor{'nscd::begin':}
->
package { 'nscd':
ensure => installed,
}
->
create_resources(file_line, $lines, $defaults)
->
service { 'nscd':
ensure => running,
enable => "true",
}
->
anchor{'nscd::end':}
}

So I wrote such code:
class nscd($parameters) {
define change_parameters() {
file_line { $name:
path => '/etc/nscd.conf',
line => $name,
# #name.split[0..-2] will remove last element,
# does not matter how many elements in line
match => inline_template('<%="^\\s*"+(#name.split[0..-2]).join("\\s*")+".*$" %>'),
}
}
anchor{'nscd::begin':}
->
package { 'nscd':
ensure => installed,
}
->
change_parameters { $parameters: }
->
service { 'nscd':
ensure => 'running',
enable => true,
hasrestart => true
}
->
anchor{'nscd::end':}
}
And class can be launched by passing list/array to class:
class { 'nscd':
parameters =>
[' server-user nscd',
' paranoia yes',
' enable-cache hosts yes smth',
' shared hosts yes']
}
Then each element from array goes to change_parameters function as $name argument after that inline_template module will generate regexp with ruby one line code.
And the same for each element from list/array.
But anyway I think better to use erb template for such changing.

Related

Logstash mutate gsub for s3 special characters

I'm trying to upload a file from Logstash to s3. Therefore, I want to replace all special characters in the field that will be the s3 key.
The filter that I'm using in my conf:
filter {
mutate {
gsub => [ "log.file.path", "[=#&<>{}:,~#`%\;\$\+\?\\\^\[\]\|\s+]", "_" ]
}
}
I also added an output to file to test the gsub :
output {
file {
codec => rubydebug
path => "/tmp/test_gsub"
}
s3 {
....
}
}
An example of output in /tmp/test_gsub that shows that the gsub didn't work:
"#timestamp" => 2020 - 06 - 04T08: 40: 17.564Z,
"log" => {
"offset" => 1784971,
"file" => {
"path" => "/var/log/AVI1:VM_B30/app.log"
}
},
"message" => "just random message",
The log.file.path still has the : in the path. I would expect the path to change to /var/log/AVI1_VM_B30/app.log
Update
Tried also to use the following regex but still got same result :
filter {
mutate {
gsub => [ "log.file.path", "[:]", "_" ]
}
}
What worked for me in the end :
filter {
mutate {
gsub => [ "[log][file][path]", "[=#&<>{}:,~#`%\;\$\+\?\\\^\[\]\|\s+]", "_" ]
}

How can I better handle config files?

In a package I'm writing, I have a config module that looks like this:
use v6.d;
use JSON::Fast;
use PSBot::Tools;
sub EXPORT(--> Hash) {
my Str $path = do if %*ENV<TESTING> {
$*REPO.Str.IO.child('META6.json').e
?? $*REPO.Str.IO.child('t/config.json').Str # For when testing using zef
!! $*REPO.Str.IO.parent.child('t/config.json').Str; # For when testing using prove
} elsif $*DISTRO.is-win {
"%*ENV<LOCALAPPDATA>\\PSBot\\config.json"
} else {
"$*HOME/.config/PSBot/config.json"
};
unless $path.IO.e {
note "PSBot config at $path does not exist!";
note "Copy psbot.json.example there and read the README for instructions on how to set up the config file.";
exit 1;
}
with from-json slurp $path -> %config {
%(
USERNAME => %config<username>,
PASSWORD => %config<password>,
AVATAR => %config<avatar>,
HOST => %config<host>,
PORT => %config<port>,
SERVERID => %config<serverid>,
COMMAND => %config<command>,
ROOMS => set(%config<rooms>.map: &to-roomid),
ADMINS => set(%config<admins>.map: &to-id),
MAX_RECONNECT_ATTEMPTS => %config<max_reconnect_attempts>,
GIT => %config<git>,
DICTIONARY_API_ID => %config<dictionary_api_id>,
DICTIONARY_API_KEY => %config<dictionary_api_key>,
YOUTUBE_API_KEY => %config<youtube_api_key>,
TRANSLATE_API_KEY => %config<translate_api_key>
)
}
}
Every time I make changes to the config file, I have to delete the precomp files for the changes to appear. Is there a way I can write this so the exports aren't defined at compile time so users don't have to do this?
Assuming I understand your intentions correctly, one way to do it would be this:
get rid of the EXPORT sub
place the computation of $path and %config into the module's mainline
declare your 'constants' as terms such as
sub term:<USERNAME> is export { %config<username> }
After reading you guys' comments and the answer by #Christoph, I settled on this. This does what I want to do:
use v6.d;
use JSON::Fast;
use PSBot::Tools;
unit module PSBot::Config;
my Str $path = do if %*ENV<TESTING> {
%?RESOURCES<test/config.json>.Str
} elsif $*DISTRO.is-win {
Qh[%*ENV<LOCALAPPDATA>\PSBot\config.json]
} else {
"$*HOME/.config/PSBot/config.json"
};
unless $path.IO.e {
note "PSBot config at $path does not exist!";
note "Copy config.json.example there and read the README for instructions on how to set up the config file.";
exit 1;
}
my %config = from-json slurp $path;
sub term:<USERNAME> is export { %config<username> }
sub term:<PASSWORD> is export { %config<password> }
sub term:<AVATAR> is export { %config<avatar> }
sub term:<HOST> is export { %config<host> }
sub term:<PORT> is export { %config<port> }
sub term:<SERVERID> is export { %config<serverid> }
sub term:<COMMAND> is export { %config<command> }
sub term:<ROOMS> is export { set(%config<rooms>.map: &to-roomid) }
sub term:<ADMINS> is export { set(%config<admins>.map: &to-id) }
sub term:<MAX_RECONNECT_ATTEMPTS> is export { %config<max_reconnect_attempts> }
sub term:<GIT> is export { %config<git> }
sub term:<DICTIONARY_API_ID> is export { %config<dictionary_api_id> }
sub term:<DICTIONARY_API_KEY> is export { %config<dictionary_api_key> }
sub term:<YOUTUBE_API_KEY> is export { %config<youtube_api_key> }
sub term:<TRANSLATE_API_KEY> is export { %config<translate_api_key> }

Amazon S3 PHP SDK, Transfer folder, but skip specific files?

I am using the Amazon PHP SDK to upload a folder on my server to a bucket. This is working great:
$skip = ['index.html', '_metadata.txt', '_s3log.txt'];
$meta = [
'key' => $options->EWRbackup_s3_key,
'region' => $options->EWRbackup_s3_region,
'bucket' => $options->EWRbackup_s3_bucket,
'directory' => 's3://'.$options->EWRbackup_s3_bucket.'/'.$subdir,
];
$client = new S3Client([
'version' => 'latest',
'region' => $meta['region'],
'credentials' => [
'key' => $meta['key'],
'secret' => $options->EWRbackup_s3_secret,
]
]);
$s3log = fopen($subpath.'/_s3log.txt', 'w+');
fwrite($s3log, "-- Connecting to ".$meta['region'].":".$meta['bucket']."...\n");
$manager = new Transfer($client, $subpath, $meta['directory'], [
'before' => function ($command)
{
$filename = basename($command->get('SourceFile'));
fwrite($this->s3log, "-- Sending file $filename...\n");
},
]);
$manager->transfer();
fwrite($s3log, "-- Disconnecting from ".$meta['key'].":".$meta['bucket']."...");
fclose($s3log);
However, in the folder I am uploading using the Transfer method, there are 3 files I want to skip. They are defined in the $skip variable on line one. I was wondering if there was a way I could get the Transfer to skip these 3 files...
I modified the AWS client in a WordPress plugin I created. The AWS/S3/Transfer.php file is where the uploads are managed, in this case. I modified the private function upload($filename) to look for a boolean return value from the before function:
private function upload($filename)
{
$args = $this->s3Args;
$args['SourceFile'] = $filename;
$args['Key'] = $this->createS3Key($filename);
$command = $this->client->getCommand('PutObject', $args);
if ($this->before) {
if (false!==call_user_func($this->before, $command)) {
return $this->client->executeAsync($command);
}
} else {
return $this->client->executeAsync($command);
}
}
This replaces the original lines:
$this->before and call_user_func($this->before, $command);
return $this->client->executeAsync($command);
with
if ($this->before) {
if (false!==call_user_func($this->before, $command)) {
return $this->client->executeAsync($command);
}
} else {
return $this->client->executeAsync($command);
}
Then, in your declared before function, you can return false if you do not want this particular file uploaded.
I was able to do this because I can control when the AWS PHP SDK is updated and can therefore modify the code it contains. I have not found any hooks in the PHP SDK to do this in a better way.

not able to specify type for fields in elastic search index

I am using logstash, elastic search and Kibana.
input file is in .csv format
I first created the following mapping through the Dev Tools > console in Kibana:
PUT /defects
{
"mappings": {
"type_name":{
"properties" : {
"Detected on Date" :{
"type": "date"
},
"DEFECTID": {
"type": "integer"
}
}
}
}
}
It was successful. Then created a logstash conf file and ran it.
Here is my logstash.conf file:
input {
file {
path => ["E:/d drive/ELK/data/sample.csv"]
start_position => "beginning"
sincedb_path => "/dev/nul"
}
}
filter {
csv {
columns => ["DEFECTID","Detected on Date","SEVERITY","STATUS"]
separator => ","
}
}
output {
elasticsearch {
action => "index"
hosts => "localhost:9200"
manage_template => false
template_overwrite => true
index => "defects"
}
}
I created index pattern defects* in Kibana. when i look at the type of the fields, all are shown as string. Pls Let me know where i am missing

Customizing Xmultiselect in Yii

I am not able to make the left list of multiselect widget in empty mode on load. It shows error when I set null value to the left list. This is my code:
$this->widget('ext.widgets.multiselects.XMultiSelects', array(
'leftTitle' => '',
'leftName' => 'Certificate[selected][]',
'leftList' => SpecificCertification::model()->findCertificate(),// here I need to make the list empty
'rightTitle' => '',
'rightName' => 'Certificate[all][]',
'rightList' => SpecificCertification::model()->findCertificates(),
'size' => 10,
));
How can I make the left list empty ?
You need to open file widget XMultiSelects.php and modify it to fit your need
public function init()
{
/* Comment out the below validation
if(!isset($this->leftList))
{
throw new CHttpException(500,'"leftList" have to be set!');
}
if(!isset($this->rightList))
{
throw new CHttpException(500,'"rightList" have to be set!');
}
*/
}
Add validation for leftList and rightList such as below
if($this->leftList){
foreach($this->leftList as $value=>$label)
{
echo "<option value=\"{$value}\">{$label}</option>\n";
}
}
and
if($this->rightList){
foreach($this->rightList as $value=>$label)
{
echo "<option value=\"{$value}\">{$label}</option>\n";
}
}
After then, you can set null for them like what you did