Deep Security API - Intrusion Prevention Rules - Error - api

I have a Python API query to gather all the Intrusion Prevention Rules and the ID of the computers associated with each but I get an error after around 14000 records which is :
An exception occurred when calling ComputerIntrusionPreventionRuleDetailsApi.lis
t_intrusion_prevention_rules_on_computer: (500)
Reason:
HTTP response headers: HTTPHeaderDict({'X-Frame-Options': 'SAMEORIGIN', 'X-XSS-P
rotection': '1;mode=block', 'Cache-Control': 'no-cache,no-store', 'Pragma': 'no-
cache', 'X-DSM-Version': 'Deep Security/12.0.296', 'Content-Type': 'application/
json', 'Content-Length': '35', 'Date': 'Fri, 16 Oct 2020 14:04:02 GMT', 'Connect
ion': 'close'})
HTTP response body: {"message":"Internal server error"}
My Script is the following :
# -*- coding: utf-8 -*-
from __future__ import print_function
import sys, warnings
import pymssql
import datetime
import deepsecurity
import json
import requests
import urllib3
from deepsecurity.rest import ApiException
from urllib3.exceptions import InsecureRequestWarning
from pprint import pprint
urllib3.disable_warnings(InsecureRequestWarning)
if not sys.warnoptions:
warnings.simplefilter("ignore")
configuration = deepsecurity.Configuration()
configuration.host = "Server/api/"
# Authentication
configuration.api_key['api-secret-key'] = 'Key'
# Initialization
# Set Any Required Values
conn = pymssql.connect("localhost","" ,"", "DeepSecurity")
cursor = conn.cursor()
cursor2 = conn.cursor()
api_instance = deepsecurity.ComputerIntrusionPreventionRuleDetailsApi(deepsecurity.ApiClient(configuration))
api_instance2 = deepsecurity.ComputersApi(deepsecurity.ApiClient(configuration))
api_version = 'v1'
overrides = False
try:
recorddt = datetime.datetime.now()
api_response2 = api_instance2.list_computers(api_version, overrides=overrides)
for y in api_response2.computers:
api_response = api_instance.list_intrusion_prevention_rules_on_computer(y.id,api_version,overrides=overrides)
for x in api_response.intrusion_prevention_rules:
strCVE=(x.cve)
clean_cve=str(strCVE).replace("['", "").replace("']", "").replace("'", "")
cursor.executemany("INSERT INTO ip_rules VALUES (%d, %s, %s ,%s,%s) ", [(x.id,x.name,clean_cve,recorddt,y.id)])
conn.commit()
except ApiException as e:
print("An exception occurred when calling ComputerIntrusionPreventionRuleDetailsApi.list_intrusion_prevention_rules_on_computer: %s\n" % e)

I guess it happened while looping (list_intrusion_prevention_rules_on_computer) with different computer id (as y.id).
Deep Security Manager seems to be able to identify the exception and return 500 Internal server error (and with header information). So, you might want to check if any exceptions in server0.log where you might get some cues.
You also want to identify which computer(s) failed to get prevention rules assigned and retry again.

Related

Custom general error handler for Flask API server [duplicate]

Is there a way to add a global catch-all error handler in which I can change the response to a generic JSON response?
I can't use the got_request_exception signal, as it is not allowed to modify the response (http://flask.pocoo.org/docs/0.10/signals/).
In contrast all signal handlers are executed in undefined order and do not modify any data.
I would prefer to not wrap the app.handle_exception function as that feels like internal API. I guess I'm after something like:
#app.errorhandler()
def handle_global_error(e):
return "Global error"
Note the errorhandler does not take any parameters, meaning it would catch all exceptions/status codes which does not have a specific error handler attached to them. I know I can use errorhandler(500) or errorhandler(Exception) to catch exceptions, but if I do abort(409) for example, it will still return a HTML response.
You can use #app.errorhandler(Exception):
Demo (the HTTPException check ensures that the status code is preserved):
from flask import Flask, abort, jsonify
from werkzeug.exceptions import HTTPException
app = Flask('test')
#app.errorhandler(Exception)
def handle_error(e):
code = 500
if isinstance(e, HTTPException):
code = e.code
return jsonify(error=str(e)), code
#app.route('/')
def index():
abort(409)
app.run(port=1234)
Output:
$ http get http://127.0.0.1:1234/
HTTP/1.0 409 CONFLICT
Content-Length: 31
Content-Type: application/json
Date: Sun, 29 Mar 2015 17:06:54 GMT
Server: Werkzeug/0.10.1 Python/3.4.3
{
"error": "409: Conflict"
}
$ http get http://127.0.0.1:1234/notfound
HTTP/1.0 404 NOT FOUND
Content-Length: 32
Content-Type: application/json
Date: Sun, 29 Mar 2015 17:06:58 GMT
Server: Werkzeug/0.10.1 Python/3.4.3
{
"error": "404: Not Found"
}
If you also want to override the default HTML exceptions from Flask (so that they also return JSON), add the following before app.run:
from werkzeug.exceptions import default_exceptions
for ex in default_exceptions:
app.register_error_handler(ex, handle_error)
For older Flask versions (<=0.10.1, i.e. any non-git/master version at the moment), add the following code to your application to register the HTTP errors explicitly:
from werkzeug import HTTP_STATUS_CODES
for code in HTTP_STATUS_CODES:
app.register_error_handler(code, handle_error)
This is Flask 0.12 compatible, and a very good solution to the problem (it allows one to render errors in JSON or any other format)
from functools import wraps
from flask import Flask, redirect, jsonify
app = Flask(__name__)
def get_http_exception_handler(app):
"""Overrides the default http exception handler to return JSON."""
handle_http_exception = app.handle_http_exception
#wraps(handle_http_exception)
def ret_val(exception):
exc = handle_http_exception(exception)
return jsonify({'code':exc.code, 'message':exc.description}), exc.code
return ret_val
# Override the HTTP exception handler.
app.handle_http_exception = get_http_exception_handler(app)
https://github.com/pallets/flask/issues/671#issuecomment-12746738
Far from elegant, but the following works for tying all subclasses of HTTPException to a single error handler:
from flask import jsonify
from werkzeug.exceptions import HTTPException
def handle_error(error):
code = 500
if isinstance(error, HTTPException):
code = error.code
return jsonify(error='error', code=code)
for cls in HTTPException.__subclasses__():
app.register_error_handler(cls, handle_error)
A cleaner way to implement this in Flask >=0.12 would be to explicitly register the handler for every Werkzeug exception:
from flask import jsonify
from werkzeug.exceptions import HTTPException, default_exceptions
app = Flask('test')
def handle_error(error):
code = 500
if isinstance(error, HTTPException):
code = error.code
return jsonify(error='error', code=code)
for exc in default_exceptions:
app.register_error_handler(exc, handle_error)
It is possible to register error handlers for very generic base classes such as HTTPException or even Exception. However, be aware that these will catch more than you might expect.
For example, an error handler for HTTPException might be useful for turning the default HTML errors pages into JSON. However, this handler will trigger for things you don’t cause directly, such as 404 and 405 errors during routing. Be sure to craft your handler carefully so you don’t lose information about the HTTP error.
from flask import Flask, abort, jsonify, json
from werkzeug.exceptions import HTTPException
app = Flask('test')
app.config['JSON_SORT_KEYS'] = False
#app.errorhandler(HTTPException)
def handle_exception(e):
"""Return JSON instead of HTML for HTTP errors."""
# start with the correct headers and status code from the error
response = e.get_response()
# replace the body with JSON
response.data = json.dumps({
"error": {
"code": e.code,
"name": e.name,
"description": e.description,
}
})
print(response.data)
response.content_type = "application/json"
return response
#app.route('/')
def index():
abort(409)
#app.route('/aloha')
def aloha():
abort(400, "I'm not in the mood to talk!")
app.run(port=1234)
output:
An error handler for Exception might seem useful for changing how all errors, even unhandled ones, are presented to the user. However, this is similar to doing except Exception: in Python, it will capture all otherwise unhandled errors, including all HTTP status codes.
In most cases it will be safer to register handlers for more specific exceptions. Since HTTPException instances are valid WSGI responses, you could also pass them through directly.
from werkzeug.exceptions import HTTPException
#app.errorhandler(Exception)
def handle_exception(e):
# pass through HTTP errors
if isinstance(e, HTTPException):
return e
# now you're handling non-HTTP exceptions only
return render_template("500_generic.html", e=e), 500
Error handlers still respect the exception class hierarchy. If you register handlers for both HTTPException and Exception, the Exception handler will not handle HTTPException subclasses because it the HTTPException handler is more specific.
Based on
Plain (non-HTML) error pages in REST api
I wanted to return json without changing any of my code at all, so I just added the following on the top of my code
#app.errorhandler(500)
def error_500(exception):
return jsonify({"error": str(exception)}), 500, {'Content-Type': 'application/json'}
#app.errorhandler(400)
def error_400(exception):
return jsonify({"error": str(exception)}), 400, {'Content-Type': 'application/json'}
If the Exceptions doesn't work, you may try app.register_error_handler (or use app.errorhandler in a non-decorator way)
Source: https://github.com/pallets/flask/issues/1837

Telegram bot Message Handler error [Local varialbe referenced before assignment]

from dotenv import load_dotenv
import os
from panda import *
from telegram.ext import *
from telegram.update import *
load_dotenv('.env')
Token =os.getenv('TOKEN')
print("The bot connected .....")
# commands handler
# start message
def start_command(update,context):
update.message.reply_text("Hello am mr panda am here to help you: ")
# help command
def help_command(update,context):
res = panda.help()
update.message.reply_text(res)
# message handler
**def message_handle(update,context):
message = str(update.message.text).lower()
respose = panda.hello(message)
update.message.reply_text(respose)**
# errror handler
def error(update,context):
print(f"Update the context error : {context.error}")
# main function
def main():
global message
updater =Updater(Token,use_context=True)
dp = updater.dispatcher
# command handlers
dp.add_handler(CommandHandler("start",start_command))
dp.add_handler(CommandHandler("help",help_command))
**# message handlers
dp.add_handler(MessageHandler(Filters.text,message_handle))**
# error handlers
dp.add_error_handler(error)
updater.start_polling()
updater.idle()
main()
This was the code Am getting error
Update the context error: local variable 'message' referenced before assignment
I think there is an error on the highlighted portions I do little searches and I referred to the documentation too I can't catch the error.
Does anyone have solution that would be great :)

Trying to execute https get request in haskell with oAuth2

This is my attempt at trying to connect to TDAmeritrade's api
to receive a json response using the Network.HTTP.Req library in haskell.
{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE OverloadedStrings #-}
module Main where
import Lib
import Wuss
import Control.Concurrent
import Control.Monad
import Data.Text
import Network.WebSockets
import Network.Connection
import Network.WebSockets.Stream
import Data.Text.Lazy
import Data.ByteString
import Lens.Micro.GHC.Internal
import Network.Curl
import Data.ByteString.Lazy.UTF8
import Network.HTTP.Req
import Control.Monad
import Control.Monad.IO.Class
import Data.Aeson
import Data.Maybe
import Data.Monoid
import GHC.Generics
import Network.HTTP.Req
import Data.ByteString.Char8
tok = Data.ByteString.Char8.pack "accesstoken"
main :: IO ()
main = runReq defaultHttpConfig $ do
v <- req GET (https "api.tdameritrade.com" /: "v1" /: "userprincipals?fields=streamerConnectionInfo") (NoReqBody) jsonResponse (oAuth2Bearer tok)
liftIO $ Data.ByteString.Char8.putStrLn (responseBody v)
if done correctly the response should only give an error that the auth token is incorrect because i purposely did not provide a valid token here. "accesstoken"
if i use bsResponse instead of jsonResponse it type checks but it does not type check with jsonResponse. The response from TDAmeritrade is going to be in JSON.
this is the error:
No instance for (FromJSON Data.ByteString.Char8.ByteString)
arising from a use of ‘req’
this is the result if I use bsResponse:
port = 443
secure = True
requestHeaders = [("Authorization","<REDACTED>")]
path = "/v1/userprincipals%3Ffields=streamerConnectionInfo"
queryString = ""
method = "GET"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
}
(StatusCodeException (Response {responseStatus = Status {statusCode = 400, statusMessage = "Bad Request"}, responseVersion = HTTP/1.1, responseHeaders = [("Date","Wed, 08 Sep 2021 15:27:41 GMT"),("Content-Type","application/json"),("Content-Length","78"),("Connection","keep-alive"),("Host","api.tdameritrade.com"),("X-Forwarded-Port","9002"),("X-Forwarded-Proto","http"),("Accept-Encoding","gzip"),("Authorization","Bearer accesstoken"),("NS-Proxy-Client-IP","82.46.185.98"),("Access-Control-Allow-Origin",""),("Access-Control-Allow-Headers","origin, x-requested-with, accept, authorization, content-type, correlationid, apikey, application-name"),("Access-Control-Max-Age","3628800"),("Access-Control-Allow-Methods","GET, PUT, POST, DELETE, OPTIONS, HEAD, PATCH"),("X-Xss-Protection","1; mode=block"),("X-Content-Type-Options","nosniff"),("X-Frame-Options","SAMEORIGIN"),("Content-Security-Policy","frame-ancestors 'self'"),("Cache-Control","no-cache,no-store,must-revalidate"),("Strict-Transport-Security","max-age=31536000")], responseBody = (), responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}) "\n {\n \"error\":\"The endpoint doesnt exist.\"\n \t\t}\n "))
for reference this curl command in the terminal yields the correct response:
curl -X GET --header "Authorization: Bearer authorizationtoken" "https://api.tdameritrade.com/v1/userprincipals?fields=streamerConnectionInfo"
I am completely lost. please help
To fix your immediate error you have to use the (=:) function construct the request parameter instead of manually writing the ?fields=streamerConnectionInfo part. And you also need to disable error status code checking. This is working code that replicates your curl command:
{-# LANGUAGE OverloadedStrings #-}
import qualified Data.ByteString.Char8
import Network.HTTP.Req
import Control.Monad.IO.Class
import Data.Text
tok = Data.ByteString.Char8.pack "accesstoken"
main :: IO ()
main = runReq defaultHttpConfig { httpConfigCheckResponse = \_ _ _ -> Nothing } $ do
v <- req GET (https "api.tdameritrade.com" /: "v1" /: "userprincipals") (NoReqBody) bsResponse ("fields" =: ("streamerConnectionInfo" :: Text) <> oAuth2Bearer tok)
liftIO $ Data.ByteString.Char8.putStrLn (responseBody v)
Another problem is accessing fields from the resulting JSON body. I would recommend starting by reading the documentation of Aeson which is the main Haskell JSON package. If you use the jsonResponse request type then it automatically converts the response to any type with a FromJSON instance.
Perhaps this stackoverflow answer about working with JSON in Haskell can also help you.

flask_cors is registered but still getting CORS error

I have a Vue frontend that uses Axios to POST to my Flask API. I've registered flask_cors to my Flask instance but I'm still get a CORS error.
flask_cors registered in app/__init__.py
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
from config import Config
from flask_marshmallow import Marshmallow
from flask_cors import CORS
app = Flask(__name__)
app.config.from_object(Config)
cors = CORS(app)
db = SQLAlchemy(app)
migrate = Migrate(app, db)
ma = Marshmallow(app)
from . import routes, models
app/routes.py
from flask import request, jsonify, current_app
from . import app, db
from .models import SetAnon
#app.route('/sets', methods=['POST'])
def sets():
data = request.get_json()
_set = SetAnon(
col1=data['somedata']
)
db.session.add(_set)
db.session.commit()
return "set saved", 201
Vue frontend is making POST request with axios:
import axios from 'axios'
const API_URL = 'http://127.0.0.1:5000'
export function saveSet(set) {
return axios.post(`${API_URL}/sets/`, set)
}
Getting this error in browser console
xhr.js?ec6c:172 OPTIONS http://127.0.0.1:5000/sets/ 404 (NOT FOUND)
Access to XMLHttpRequest at 'http://127.0.0.1:5000/sets/' from origin 'http://localhost:8080' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: It does not have HTTP ok status.
Looks like an issue with trailing backslash. Your routes.py defines the route as '/sets' however the frontend is calling '/sets/'.
Change the routes.py to this and it should work -
from flask import request, jsonify, current_app
from . import app, db
from .models import SetAnon
#app.route('/sets/', methods=['POST']) # Added trailing backslash
def sets():
data = request.get_json()
_set = SetAnon(
col1=data['somedata']
)
db.session.add(_set)
db.session.commit()
return "set saved", 201

REST call to sslCertificate.insert fails with "The SSL certificate could not be parsed", but gcloud works fine

Here's a example program that generates the error:
import urllib3
import certifi
from google.auth.transport.urllib3 import AuthorizedHttp
from google.oauth2 import service_account
import boto3
from pprint import pprint
import json
# parameters
JSON_SERVICE_ACCOUNT_FILE = "/home/mwilbert/gcp/matt-vm-service-account.json"
SCOPES = [
'https://www.googleapis.com/auth/devstorage.read_write',
'https://www.googleapis.com/auth/compute'
]
TEST_PROJECTID = 'mwilbert-workspace'
TEST_FULLCHAIN_FILE = '/etc/letsencrypt/live/fake.mapgeo.io/fullchain.pem'
TEST_PRIVKEY_FILE = '/etc/letsencrypt/live/fake.mapgeo.io/privkey.pem'
http = urllib3.PoolManager(cert_reqs='CERT_REQUIRED',ca_certs=certifi.where())
credentials = service_account.Credentials.from_service_account_file(JSON_SERVICE_ACCOUNT_FILE)
scoped_credentials = credentials.with_scopes(SCOPES)
authed_http = AuthorizedHttp(scoped_credentials,http)
requestBody = {
"name":"deleteme",
"description": "fake domain cert",
"certificate": TEST_FULLCHAIN_FILE,
"privateKey": TEST_PRIVKEY_FILE,
}
rb = json.dumps(requestBody)
url = 'https://www.googleapis.com/compute/v1/projects/%s/global/sslCertificates' % TEST_PROJECTID
r = authed_http.urlopen('POST',
url,
headers={'Content-Type': 'application/json'},
body=rb)
response = json.loads(r.data)
pprint(response)
However if I pass the same arguments to gcloud
gcloud compute ssl-certificates create deleteme
--certificate=/etc/letsencrypt/live/fake.mapgeo.io/fullchain.pem
--private-key=/etc/letsencrypt/live/fake.mapgeo.io/privkey.pem
--description="fake domain cert" --project=mwilbert-workspace
from the same terminal session the certificate is created. The certificate is in .PEM format.
Has anyone gotten this call to work? If so, any pointers?