I have an AWS ECS Fargate task which is running in an isolated subnet (no internet connectivity) in Amazon Linux. I have a ClamAv and Clamd installed using yum command. I have two questions around clamav.
As per the ClamAv website, latest version available is 0.105 (https://www.clamav.net/downloads). But yum is still installing 0.103. Why so?
bash-4.2# yum info clamd
Loaded plugins: ovl, priorities
228 packages excluded due to repository priority protections
Installed Packages
Name : clamd
Arch : x86_64
Version : 0.103.6
Release : 1.el7
Size : 248 k
Repo : installed
From repo : epel
Summary : The Clam AntiVirus Daemon
URL : https://www.clamav.net/
License : GPLv2
Description : The Clam AntiVirus Daemon
: See the README file how this can be done with a minimum of effort.
: This package contains a generic system wide clamd service which is
: e.g. used by the clamav-milter package.
Since clamav is running in the ECS task without any internet, it's not possible to update the virus definition and binary path update using yum update command.
Virus definition update (Solved): I'm downloading virus definitions to the s3. ECS task is downloading virus definition from the s3 using s3 endpoint to the clamav virus definition location "/var/lib/clamav/". So virus definition update is working this way without internet connectivity.
ClamAv binary/path update: Currently I'm struggling to update the clamav binary. Is there a way to download the ClamAv binary path and update Clamav version by running some command?
Related
Dears,
I am managing a pool of servers running under RHEL 7.6, I created a local repository of RHEL packages to be able to udpated the other servers by limiting the internet access to the server hosting the local repository.
I used the reposync command to populate my repository but I am downloading a huge number of rpms packages!
I would like to reduce the set of packages to download to the ones already deployed on all the severs, I can do the list using the rpm command, (~750 packages).
I read that there is an includepkgs directive to be used with the reposync command.
How is it working, what is the required format?
I know it is possible to use the yumdownloader command to update the local repository, how is it possible to populate the repository for the first time ?
Any help advice would be appreciated
Regards
Fdv
It seems that the best option is to limit to the last version of the packages by using the option :
-n, --newest-only Download only newest packages per-repo
When I running command below,
s3-dist-cp --src s3://test/9.19 --dest hdfs:///user/hadoop/test
I got a error about auxService.
20/02/03 07:52:13 INFO mapreduce.Job: Task Id : attempt_1580716305878_0001_m_000000_2, Status : FAILED
Container launch failed for container_1580716305878_0001_01_000004 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
In many QnA, I found a solution like this
link.
But there is no process for nodemanager.
[hadoop#ip-172-31-37-115 ~]$ initctl list | grep yarn
hadoop-yarn-timelineserver start/running, process 8149
hadoop-yarn-resourcemanager start/running, process 17331
hadoop-yarn-proxyserver start/running, process 8147
My EMR was created by quick menu with emr-5.28.0.
Is there anyone knows about this problem?
Thanks!
I'm sure there's some way to update the configs, but what I did was create a cluster using the 'advanced' setup and chose these software packages:
Ganglia
Hive
Hue
Mahout
Pig
Tez
Spark
Hadoop
(8 in total)
Most of those, except spark, are installed with the default settings (the first radio button for software packages in quick setup). One of these software packages or something related to it is what causes s3-dist-cp to be installed, and I was able to use it with no problems with that setup.
I am trying to deploy a production cluster for my Flink program. I am using a standard hadoop-core EMR cluster with Flink 1.3.2 installed, using YARN to run it.
I am trying to configure my RocksDB to write my checkpoints to an S3 bucket. I am trying to go through these docs: https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/aws.html#set-s3-filesystem. The problem seems to be getting the dependencies working correctly. I receive this error when trying run the program:
java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V
at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize(EmrFileSystem.java:93)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:328)
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:350)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:99)
at org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:282)
at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createStreamFactory(RocksDBStateBackend.java:273
I have tried both leaving and adjusting the core-site.xml and leaving it as is. I have tried setting the HADOOP_CLASSPATH to the /usr/lib/hadoop/share that contains(what I assume are) most of the JARs described in the above guide. I tried downloading the hadoop 2.7.2 binaries, and copying over them into the flink/libs directory. All resulting in the same error.
Has anyone successfully gotten Flink being able to write to S3 on EMR?
EDIT: My cluster setup
AWS Portal:
1) EMR -> Create Cluster
2) Advanced Options
3) Release = emr-5.8.0
4) Only select Hadoop 2.7.3
5) Next -> Next -> Next -> Create Cluster ( I do fill out names/keys/etc)
Once the cluster is up I ssh into the Master and do the following:
1 wget http://apache.claz.org/flink/flink-1.3.2/flink-1.3.2-bin-hadoop27-scala_2.11.tgz
2 tar -xzf flink-1.3.2-bin-hadoop27-scala_2.11.tgz
3 cd flink-1.3.2
4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 -d
5 Change conf/flink-conf.yaml
6 ./bin/flink run -m yarn-cluster -yn 1 ~/flink-consumer.jar
My conf/flink-conf.yaml I add the following fields:
state.backend: rocksdb
state.backend.fs.checkpointdir: s3:/bucket/location
state.checkpoints.dir: s3:/bucket/location
My program's checkpointing setup:
env.enableCheckpointing(getCheckpointRate,CheckpointingMode.EXACTLY_ONCE)
env.getCheckpointConfig.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION)
env.getCheckpointConfig.setMinPauseBetweenCheckpoints(getCheckpointMinPause)
env.getCheckpointConfig.setCheckpointTimeout(getCheckpointTimeout)
env.getCheckpointConfig.setMaxConcurrentCheckpoints(1)
env.setStateBackend(new RocksDBStateBackend("s3://bucket/location", true))
If there are any steps you think I am missing, please let me know
I assume that you installed Flink 1.3.2 on your own on the EMR Yarn cluster, because Amazon does not yet offer Flink 1.3.2, right?
Given that, it seems as if you have a dependency conflict. The method org.apache.hadoop.conf.Configuration.addResource(Lorg/apache/hadoop/conf/Configuration) was only introduced with Hadoop 2.4.0. Therefore I assume that you have deployed a Flink 1.3.2 version which was built with Hadoop 2.3.0. Please deploy a Flink version which was built with the Hadoop version running on EMR. This will most likely solve all dependency conflicts.
Putting the Hadoop dependencies into the lib folder seems to not reliably work because the flink-shaded-hadoop2-uber.jar appears to have precedence in the classpath.
I'm currently setting up a Kaltura streaming server on CentOS 7 with mariaDB. When I come to the point the installation manual requires me to install oflaDemo via browser, I only get an empty list. No connection errors occour. The debug output states:
Host: vstream-dev.my.domain
Trying to connect
Net status: NetConnection.Connect.Success
Got the application list
Got the application list
Got the application list
So, in theory there shouldn't be a problem.
Firewall is down for testing/devel
SELinux is off (permissive)
The only error that ocoured during the installation process was packet mysql-server is not installed. But the manual states that I should use mariaDB on CentOS 7.
I tried to clone https://github.com/Red5/red5-examples and link the ofla Demo folder to /usr/lib/red5/webapps/ with no success.
Ok, I solved it.
What I did:
I cloned the repo with the red5 examples: https://github.com/Red5/red5-examples and navigated into the subfolder oflaDemo (with pom.xml).
Then I had to install maven with
yum install -y maven
and do a maven build
mvn clean install
After that, I was able to grab the file target/red5-example-oflaDemo-2.0.war. I extracted this file into a folder oflaDemo in /var/lib/red5/webapps and restarted the server. Finally, I did mkdir /usr/lib/red5/webapps/oflaDemo/streams to create a folder for the streams.
After that, I was able to navigate to the demo via
http://my.domain:5080/oflaDemo/
I am trying to build tensorflow cpu version on my centos 6.5, but i am stuck :-
bazel build -c opt //tensorflow/cc:tutorials_example_trainer
........
WARNING: Sandboxed execution is not supported on your system and thus hermeticity of actions cannot be guaranteed. See http://bazel.io/docs/bazel-user-manual.html#sandboxing for more information. You can turn off this warning via --ignore_unsupported_sandboxing.
INFO: Downloading from http://www.ijg.org/files/jpegsrc.v9a.tar.gz: 0B
it tries to download jpeg/eigen/png tar files but is unable to do so due to lack of internet connectivity on my machine .
I can download & put all these dependencies somewhere within the tensorflow sourcecode directory so that build procedure automatically detect them.
Could you please suggest the path to that directory ( relative to tensorflow src root directory)? or is there a file which needs modification?
I tried placing it under $TENSOR_SRC_ROOT/tensorflow/contrib/cmake/external, but that did not help.
Eagerly awaiting your replies,
thanks to bazel development team, setting HTTP_PROXY & HTTPS_PROXY in my environment resolved the issue.