I am using python get data from an api trying to get a process, host name, start time of the process, user name, ip address, domain, and time stamp of each connection. Each process can have multiple results for the following columns that needs to move down each row. What is happening is that everything is being written into Row 2 as it gets over written. Here is my script.
wb = Workbook()
ws1 = wb.create_sheet("Sheet_A")
ws1.title = "Results"
ws1['A1'] = "Process"
ws1['B1'] = "Host Name"
ws1['C1'] = "Process Time"
ws1['D1'] = "User Name"
ws1['E1'] = "Domain"
ws1['F1'] = "Netconn Time"
c = CbResponseAPI()
rows = 1
base_query = c.select(Process).where('start:-500m (hostname:1 OR hostname:2 OR hostname:3 OR hostname:4) netconn_count:[1 TO *] -ipaddr:192.168.0.0/16 -ipaddr:10.0.0.0/8 -ipaddr:172.16.0.0/12 -ipaddr:127.0.0.1')
for proc in base_query:
# print proc
ws1['A2'] = proc.process_name
ws1['B2'] = proc.hostname
ws1['C2'] = proc.start
ws1['D2'] = proc.username
rows+=1
for netconn in proc.netconns:
ws1['E2'] = netconn.domain
ws1['F2'] = netconn.timestamp
ws1['G2'] = netconn.remote_ip
rows+=1
wb.save('Rav.xlsx')
Here is my edit
for i, proc in enumerate(base_query):
# print proc
ws1[f'A{i+2}'] = proc.process_name
ws1[f'B{i+2}'] = proc.hostname
ws1[f'C{i+2}'] = proc.start
ws1[f'D{i+2}'] = proc.username
rows+=1
for netconn in proc.netconns:
ws1[f'E{i+2}'] = netconn.domain
ws1[f'F{i+2}'] = netconn.timestamp
ws1[f'G{i+2}'] = netconn.remote_ip
rows+=1
Here is the output of print(proc) and print(netconn)
process_md5 : f8d0c92070e59a059a889d5e269c0da9
sensor_id : 656
filtering_known_dlls : False
modload_count : 45
parent_unique_id : 00000290-0000-023c-01d5-e25a6a09eddc-000000000001
cmdline : C:\Windows\ADWS\Microsoft.ActiveDirectory.WebServices.exe
last_update : 2020-02-17 14:48:26.878000
id : 00000290-0000-0760-01d5-e25a77d2e409
parent_name : services.exe
group : default group
parent_id : 00000290-0000-023c-01d5-e25a6a09eddc
hostname :
filemod_count : 0
start : 2020-02-13 10:43:46.254000
comms_ip : 0.0.0.0
regmod_count : 5
interface_ip : 10.140.51.10
process_pid : 1888
username : SYSTEM
process_sha256 : d40478a82bb2993f39a3ed6066cd0599be37ff9a0898636a680926fe145c64d6
terminated : False
emet_config :
process_name : microsoft.activedirectory.webservices.exe
emet_count : 0
last_server_update : 2020-02-17 14:49:19.877000
path : c:\windows\adws\microsoft.activedirectory.webservices.exe
netconn_count : 12008
parent_pid : 572
crossproc_count : 11
segment_id : 1581950959796
host_type : domain_controller
processblock_count : 0
os_type : windows
childproc_count : 0
unique_id : 00000290-0000-0760-01d5-e25a77d2e409-017053a000b4
Cb Network Connection event:
timestamp : 2020-02-17 14:44:26.844000
domain :
remote_ip : ::1
remote_port : 389
proto : IPPROTO_TCP
direction : Outbound
local_ip : ::1
local_port : 52995
proxy_ip : 0.0.0.0
proxy_port : 0
You just need to enumerate the for loop to give yourself a counter, to move to the next line. Then add 2 to i because you're starting on row 2, and i will start at 0.
for i, proc in enumerate(base_query):
# print proc
ws1[f'A{i+2}'] = proc.process_name
ws1[f'B{i+2}'] = proc.hostname
ws1[f'C{i+2}'] = proc.start
ws1[f'D{i+2}'] = proc.username
for netconn in proc.netconns:
ws1[f'E{i+2}'] = netconn.domain
ws1[f'F{i+2}'] = netconn.timestamp
ws1[f'G{i+2}'] = netconn.remote_ip
wb.save('Rav.xlsx')
Here's my working example:
import openpyxl
from openpyxl import Workbook
wb = Workbook()
ws1 = wb.create_sheet("Sheet_A")
ws1.title = "Results"
ws1['A1'] = "Process"
ws1['B1'] = "Host Name"
ws1['C1'] = "Process Time"
ws1['D1'] = "User Name"
ws1['E1'] = "Domain"
ws1['F1'] = "Netconn Time"
loop = [['process_name', 'hostname'],['process_name', 'hostname'],['process_name', 'hostname'],]
for i, proc in enumerate(loop):
# print proc
ws1[f'A{i+2}'] = proc[0]
ws1[f'B{i+2}'] = proc[1]
wb.save('Rav.xlsx')
Output
Related
I am creating Redis in AWS using Terraform. But When I execute terraform apply command for first time it creates without issues. But If I re-run Terraform apply below TF code destroys the Redis and starts re-creating it instead it should tell me that it already exists start focusing on other newly added resources .
Is it expected behaviour of Redis?
Adding terraform plan in the question:
-/+ resource "aws_elasticache_replication_group" "redis" {
apply_immediately = true
at_rest_encryption_enabled = true
auto_minor_version_upgrade = false
automatic_failover_enabled = true
+ configuration_endpoint_address = (known after apply)
engine = "redis"
engine_version = "5.0.4"
~ id = "dev-af-redis" -> (known after apply)
maintenance_window = "sun:06:00-sun:07:00"
~ member_clusters = [
- "ca-cng-dev-af-redis-001",
- "ca-cng-dev-af-redis-002",
] -> (known after apply)
node_type = "cache.t2.medium"
~ number_cache_clusters = 2 -> (known after apply)
parameter_group_name = "default.redis5.0"
port = 6379
~ primary_endpoint_address = "master.dev-af-redis.qxyj8a.euc1.cache.amazonaws.com" -> (known after apply)
replication_group_description = "Airflow Cluster"
replication_group_id = "dev-af-redis"
security_group_ids = [
"sg-094175ad3062da04d",
]
~ security_group_names = [] -> (known after apply)
- snapshot_retention_limit = 0 -> null
~ snapshot_window = "02:30-03:30" -> (known after apply)
subnet_group_name = "dev-subnet-group-airflow"
tags = {
"Application" = "project"
"BusinessUnit" = "subproject"
"Classification" = "private"
"Environment" = "development"
"Name" = "dev-airflow-redis"
"TechnicalOwner" = "ops"
"Tier" = "orchestration"
}
transit_encryption_enabled = true
+ cluster_mode {
+ num_node_groups = 1
+ replicas_per_node_group = 1 # forces replacement
}
}
Plan: 1 to add, 0 to change, 1 to destroy.
TF code which used to create Redis:-
resource "aws_elasticache_replication_group" "cng_redis" {
replication_group_description = "Cluster"
replication_group_id = "dev-af-redis"
engine = "redis"
engine_version = "5.0.4"
node_type = "cache.t2.medium "
port = 6379
subnet_group_name = "dev-subnet-group-airflow"
security_group_ids = ["${aws_security_group.airflow_sg.id}"]
parameter_group_name = "default.redis5.0"
at_rest_encryption_enabled = true
transit_encryption_enabled = true
maintenance_window = "sun:06:00-sun:07:00"
auto_minor_version_upgrade = false
apply_immediately = true
automatic_failover_enabled = true
cluster_mode {
num_node_groups = "1"
replicas_per_node_group = "1"
}
tags = merge(
var.common_tags,
map("Classification", "private"),
map("Name", "airflow-redis")
)
}
Here is a solution ("this is not a bug, it's a feature" case, I suppose ;) ): https://github.com/terraform-providers/terraform-provider-aws/issues/4817#issuecomment-463993424
I tested it and it works.
You have to add parameter group with cluster-enabled set to yes.
I'm using Redis 5.0.5, so to my aws_elasticache_replication_group I added:
resource "aws_elasticache_replication_group" "elc-rep-group" {
...
automatic_failover_enabled = true #this is required, when cluster-enabled parameter is on
parameter_group_name = "default.redis5.0.cluster.on"
...
}
I'm trying to fetch data which includes some German word with umlaut characters. following the bellow structure everything is fine in windows machine :
Sys.setlocale('LC_ALL','C')
library(RMySQL)
conn <- dbConnect(MySQL(), user = "user", dbname = "database",
host = "host", password = "pass")
sql.query <- paste0("some query")
df <- dbSendQuery(conn, sql.query)
names <- fetch(df, -1)
dbDisconnect(conn)
As an example I have :
names[1230]
[1] "StrĂ¼bbel"
What should I change in order to get the same result in Linux Ubuntu ?
the query will run without problem, but the result is :
names[1230]
[1] "Str\374bbel"
I have checked This solution, but when I put the 'set character set "utf8"' inside of query I'm getting the following error :
df <- dbSendQuery(conn, sql.query, 'set character set "utf8"')
names <- fetch(df, -1)
Error in .local(conn, statement, ...) :
unused argument ("set character set \"utf8\"")
I should mention the encoding for the result is unknown :
Encoding(names[1230])
[1] "unknown"
and doing the :
Encoding(names[1230]) <- "UTF-8"
names[1230]
[1] "Str<fc>bbel"
does not solve the problem !
Instead of :
Sys.setlocale('LC_ALL','C')
You have to use :
Sys.setlocale('LC_ALL','en_US.UTF-8')
and in the sql query :
library(RMySQL)
conn <- dbConnect(MySQL(), user = "user", dbname = "database",
host = "host", password = "pass")
sql.query <- paste0("some query")
dbSendQuery(conn,'set character set "utf8"')
df <- dbSendQuery(conn, sql.query)
names <- fetch(df, -1)
dbDisconnect(conn)
Not sure if this solution will help you but you could try such approach:
con <- dbConnect(MySQL(), user = "user", dbname = "database",
host = "host", password = "pass", encoding = "ISO-8859-1")
If this encoding doesn't work then try "brute force" with different variants
The following code (within my shiny app) is giving me this error:
"You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 1"
shinyServer(function(input, output, session) {
dataTable <- reactive ({
data <- input$dataset
con <-
dbConnect(
drv = dbDriver("MySQL"),
dbname = "Database",
host = 'remote',
port = 3306,
user = "user",
password = "password")
on.exit(dbDisconnect(con))
dbGetQuery(con, paste("select * from ", data, ";"))
})
output$myTable <- renderDataTable({
datatable(dataTable(),
rownames = FALSE,
filter = "top",
extensions = 'Buttons',
options = list(dom = 'Bfrtip', buttons = I('colvis')))
})
})
shinyUI(fluidPage(
titlePanel("Data Search"),
# SidePanel -------------------------------------------
# -The Input/Dropdown Menu that Control the Output
sidebarLayout(
sidebarPanel(
selectInput(
inputId = "dataset",
label = "Select Dataset",
choices = c("", "Schools", "GradRates"),
selected = "",
multiple = FALSE),
width = 3
),
# MainPanel -------------------------------------------
# -The Output/Table Displayed Based on Input
mainPanel(
dataTableOutput(outputId = "myTable"),
width = 9
)
))
You have most likely problem with this line
dbGetQuery(con, paste("select * from ", data, ";"))
It appears that variable data doesn't contain table name as expected. Check your code where you are inserting table name into data.
I'm very new to Python coding and have run into an issue while trying to upgrade some code. I'm working with an app that pulls data via an API from stored data from a scan.
here is the code as it sits working
def _collect_one_host_scan_info(self, host_id, sid, scan_info):
"""
The method to collect all the vulnerabilities of one host and generate the event data.
"""
count = 0
host_uri = self.endpoint + '/' + str(sid) + '/hosts/' + str(host_id)
result = self.client.request(host_uri).get("content")
# if there is exception in request, return None
if result is None:
_LOGGER.info("There is exception in request, return None")
return None
else:
host_info = result.get("info", {})
host_end_time = host_info.get("host_end", "")
if self.ckpt.is_new_host_scan(host_end_time,
self.config.get("start_date")):
self.source = self.url + self.endpoint + '/' + str(
sid) + '/hosts/' + str(host_id)
for vuln in result.get("vulnerabilities", []):
vuln["sid"] = sid
vuln["host_id"] = host_id
#get the port info
plugin_id = vuln.get("plugin_id", "")
port_info = []
if plugin_id:
plugin_uri = "{}/plugins/{}".format(host_uri,
plugin_id)
plugin_outputs = self.client.request(plugin_uri).get(
"content", {}).get("outputs")
ports = []
for output in plugin_outputs:
ports.extend(output.get("ports", {}).keys())
for port in ports:
port_elem = {}
port_items = re.split(r"\s*/\s*", port)
port_elem["port"] = int(port_items[0])
if port_items[1]:
port_elem["transport"] = port_items[1]
if port_items[2]:
port_elem["protocol"] = port_items[2]
port_info.append(port_elem)
vuln = dict(vuln, **scan_info)
vuln = dict(vuln, **host_info)
if port_info:
vuln["ports"] = port_info
entry = NessusObject(
vuln.get("timestamp"), self.sourcetype, self.source,
vuln)
self._print_stream(entry)
count += 1
return count
The data that is being pulled from looks like this
"outputs": [
{
"ports": {
"445 / tcp / cifs": [
{
"hostname": "computer.domain.com"
}
]
},
"has_attachment": 0,
"custom_description": null,
"plugin_output": "\nPath : c:\\program files (x86)\\folder\\bin\\fax.exe\nUsed by services : RFDB\nFile write allowed for groups : Domain Users\nFull control of directory allowed for groups : Domain Users\n\nPath : c:\\program files (x86)\\folder\\bin\\faxrpc.exe\nUsed by services : RFRPC\nFile write allowed for groups : Domain Users\nFull control of directory allowed for groups : Domain Users\n\nPath : c:\\program files (x86)\\folder\\bin\\faxserv.exe\nUsed by services : RFSERVER\nFile write allowed for groups : Domain Users\nFull control of directory allowed for groups : Domain Users\n`,
"hosts": null,
"severity": 3
}
with the working code the return is
ports{}.port 445
ports{}.protocol tcp
ports{}.transport cifs
What I really would like is to grab the "plugin_output" data with the "port" data
I'm currently just trying to replace the "port" data with "plugin_output" data
#get the output info
plugin_id = vuln.get("plugin_id", "")
output_info = []
if plugin_id:
plugin_uri = "{}/plugins/{}".format(host_uri,
plugin_id)
plugin_outputs = self.client.request(plugin_uri).get(
"content", {}).get("outputs")
outputs = []
for output in plugin_outputs:
outputs.extend(output.get("plugin_output", "").keys())
for plugin in plugin_outputs:
plugin_elem = {}
plugin_items = re.split(r"nPath\s*", plugin)
plugin_elem["location1"] = plugin_items[0]
if plugin_items[1]:
plugin_elem["location2"] = plugin_items[1]
if plugin_items[2]:
plugin_elem["location3"] = plugin_items[2]
output_info.append(plugin_elem)
vuln = dict(vuln, **scan_info)
vuln = dict(vuln, **host_info)
if output_info:
vuln["plugin_output"] = output_info
entry = NessusObject(
vuln.get("timestamp"), self.sourcetype, self.source,
vuln)
self._print_stream(entry)
count += 1
what I've done as you can see if just replace the "ports" data with "plugin_output" data and the error received is
AttributeError: 'unicode' object has no attribute key
Well after further efforts I was able to figure out what I needed to do with the code. It was much easier than I thought it would be but sometime when learning a new language its hard to envision what is needed. Code posted below.
def _collect_one_host_scan_info(self, host_id, sid, scan_info):
"""
The method to collect all the vulnerabilities of one host and generate
the event data.
"""
count = 0
host_uri = self.endpoint + '/' + str(sid) + '/hosts/' + str(host_id)
result = self.client.request(host_uri).get("content")
# if there is exception in request, return None
if result is None:
_LOGGER.info("There is exception in request, return None")
return None
else:
host_info = result.get("info", {})
host_end_time = host_info.get("host_end", "")
if self.ckpt.is_new_host_scan(host_end_time,
self.config.get("start_date")):
self.source = self.url + self.endpoint + '/' + str(
sid) + '/hosts/' + str(host_id)
for vuln in result.get("vulnerabilities", []):
vuln["sid"] = sid
vuln["host_id"] = host_id
plugin_id = vuln.get("plugin_id", "")
# get plugin_output data
plugin_output_info = []
if plugin_id:
plugin_uri = "{}/plugins/{}".format(host_uri,
plugin_id)
plugin_outputs = self.client.request(plugin_uri).get(
"content", {}).get("outputs", [])
data_output = []
for output in plugin_outputs:
items = output.get("plugin_output", 'no value')
item = str(items)
#clean = re.sub('[^a-zA-Z0-9-()_*.(:\\)]', ' ', item)
plugin_output_info.append(item)
# get the port info
port_info = []
if plugin_id:
plugin_uri = "{}/plugins/{}".format(host_uri,
plugin_id)
plugin_outputs = self.client.request(plugin_uri).get(
"content", {}).get("outputs", [])
ports = []
for output in plugin_outputs:
ports.extend(output.get("ports", {}).keys())
for port in ports:
port_elem = {}
port_items = re.split(r"\s*/\s*", port)
port_elem["port"] = int(port_items[0])
if port_items[1]:
port_elem["transport"] = port_items[1]
if port_items[2]:
port_elem["protocol"] = port_items[2]
port_info.append(port_elem)
vuln = dict(vuln, **scan_info)
vuln = dict(vuln, **host_info)
if port_info:
vuln["ports"] = port_info
if plugin_output_info:
vuln["plugin_output"] = plugin_output_info
entry = NessusObject(
vuln.get("timestamp"), self.sourcetype, self.source,
vuln)
self._print_stream(entry)
count += 1
return count
I am new to apache flume.
I am trying to see how I can get a json (as http source), parse it and store it to a dynamic path on hdfs according to the content.
For example:
if the json is:
[{
"field1" : "value1",
"field2" : "value2"
}]
then the hdfs path will be:
/some-default-root-path/value1/value2/some-value-name-file
Is there such configuration of flume that enables me to do that?
Here is my current configuration (accepts a json via http, and stores it in a path according to timestamp):
#flume.conf: http source, hdfs sink
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = org.apache.flume.source.http.HTTPSource
a1.sources.r1.port = 9000
#a1.sources.r1.handler = org.apache.flume.http.JSONHandler
# Describe the sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = /user/uri/events/%y-%m-%d/%H%M/%S
a1.sinks.k1.hdfs.filePrefix = events-
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = minute
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
Thanks!
The solution was in the flume documentation for the hdfs sink:
Here is the revised configuration:
#flume.conf: http source, hdfs sink
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = org.apache.flume.source.http.HTTPSource
a1.sources.r1.port = 9000
#a1.sources.r1.handler = org.apache.flume.http.JSONHandler
# Describe the sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = /user/uri/events/%{field1}
a1.sinks.k1.hdfs.filePrefix = events-
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = minute
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
and the curl:
curl -X POST -d '[{ "headers" : { "timestamp" : "434324343", "host" :"random_host.example.com", "field1" : "val1" }, "body" : "random_body" }]' localhost:9000