Traverse directory at URL to root in Python - python-3.8

How can you traverse directory to get to root in Python? I wrote some code using BeautifulSoup, but it says 'module not found'. So I have this:
#
# There is a directory traversal vulnerability in the
# following page http://127.0.0.1:8082/humantechconfig?file=human.conf
# Write a script which will attempt various levels of directory
# traversal to find the right amount that will give access
# to the root directory. Inside will be a human.conf with the flag.
#
# Note: The script can timeout if this occurs try narrowing
# down your search
import urllib.request
import os
req = urllib.request.urlopen("http://127.0.0.1:8082/humantechconfig?file=human.conf")
dirName = "/tmp"
def getListOfFiles(dirName):
listOfFile = os.listdir(dirName)
allFiles = list()
for entry in listOfFile:
# Create full path
fullPath = os.path.join(dirName, entry)
if os.path.isdir(fullPath):
allFiles = allFiles + getListOfFiles(fullPath)
else:
allFiles.append(fullPath)
return allFiles
listOfFiles = getListOfFiles(dirName)
print(listOfFiles)
for file in listOfFiles:
if file.endswith(".conf"):
f = open(file, "r")
print(f.read())
This outputs:
/tmp/level-0/level-1/level-2/human.conf
User : Human 66
Flag: Not-Set (Must be Root Human)
However. If I change the URL to 'http://127.0.0.1:8082/humantechconfig?file=../../../human.conf' it gives me the output:
User : Human 66
Flag: Not-Set (Must be Root Human)
User : Root Human
Flag: Well done the flag is: {}
The level of directory traversal it is at fluctuates wildly, from /tmp/level-2 to /tmp/level-15; if it's at the one I wrote, then it says I'm 'Root Human'. But it won't give me the flag, despite the fact that I am suddenly 'Root Human'. Is there something wrong with the way I am traversing directory?
It doesn't seem to matter at all if I take away the req = urllib.request.urlopen("http://127.0.0.1:8082/humantechconfig?file=human.conf") line. How can I actually send the code to that URL?
Thanks!

cyber discovery moon base challenge?
For this one, you need to keep adding '../' in front of human.conf (for example 'http://127.0.0.1:8082/humantechconfig?file=../human.conf') which becomes your URL. This URL you need to request (using urllib.request.urlopen(URL)).
The main bit of the challenge is to attach the ../ multiple times which shall not be very hard using a simple loop. You don't need to use the OS.
Make sure to break the loop once you find the flag (or it will go into an infinite loop and give you errors).

Related

My workflow ignores the path decided via Inputfunction

I have two possible path for Trinity, genome free (GF) and genome guided (GG). For deciding which way to go I use the variable GUIDED from a config and depending on it i give the path to files created by either the GG or GF part.
The problem is that no matter what the Input function returns, snakemake always tries to run the GG part. (except for the exception ofc)
def GenomeDependentInput()->str:
guided = config["GUIDED"]
if guided == "GF":
print(rules.aggregate_GF.output.fasta) #this print is run by snakemake and gives the correct path ...Results/trinityGF/{species}_Trinity_GF.fasta
return rules.aggregate_GF.output.fasta
elif guided == "GG":
print(rules.aggregateTrinity.output.fasta) # this is not (good)
return rules.aggregateTrinity.output.fasta
else:
raise ValueError("Please fill in the GUIDED variable in the config")
rule Transdecoder:
input:
fasta = GenomeDependentInput()
output:
pep = path.join(TRANS_DIR, "{species}", path.basename(GenomeDependentInput()) + ".transdecoder.pep")
envmodules:
config["PERL"],
config["PYTHON3"]
script:
"scripts/TransDecoder.py"
So I found that down the line another rule uses only the GG part and I had to use the GenomeDependentInput function there too.
Furthermore the Transdecoder rule wasn't even active as the Transdecoder output wasn't yet used as input.
Maybe this will help somebody else so I'll just leave this here.

Snakemake: Generic input function for different file locations

I have two locations where my huge data can be stored: /data and /work.
/data is the folder where (intermediate) results are moved to after quality control. It is mounted read-only for the standard user.
/work is the folder where new results are written to. Obviously, it is writable.
I do not want to copy or link data from /data to /work.
So I run my snakemake from within the /work folder and want my input function first to check, if the required file exists in /data (and return the absolute /data path) and if not return the relative path in the /work directory.
def in_func(wildcards):
file_path = apply_wildcards('{id}/{visit}/{id}_{visit}-file_name_1.txt', wildcards)
full_storage_path = os.path.join('/data', file_path)
if os.path.isfile(full_storage_path):
file_path = full_storage_path
return {'myfile': file_path}
rule do_something:
input:
unpack(in_func),
params = '{id}/{visit}/{id}_{visit}_params.txt',
This works fine but I would have to define separate input functions for every rule because the file names differ. Is is possible to write a generic input function that takes as input the file name e.g {id}/{visit}/{id}_{visit}-file_name_1.txt and the wildcards?
I also tried something like
def in_func(file_path):
full_storage_path = os.path.join('/data', file_path)
if os.path.isfile(full_storage_path):
file_path = full_storage_path
file_path
rule do_something:
input:
myfile = in_func('{id}/{visit}/{id}_{visit}-file_name_1.txt')
params = '{id}/{visit}/{id}_{visit}_params.txt',
But then I do not have access to the wildcards in in_func(), do I?
Thanks,
Jan
You could use something like this:
def handle_storage(pattern):
def handle_wildcards(wildcards):
f = pattern.format(**wildcards)
f_data = os.path.join("/data", f)
if os.path.exists(f_data):
return f_data
return f
return handle_wildcards
rule do_something:
input:
myfile = handle_storage('{id}/{visit}/{id}_{visit}-file_name_1.txt')
params = '{id}/{visit}/{id}_{visit}_params.txt',
In other words, the function handle_storage returns a pointer to the handle_wildcards function that is tailored for the particular pattern. The latter is then automatically applied by Snakemake once the wildcard values are known. Inside that function, we first format the pattern and then check if it exists in /data.

Registering a new Command Line Option in RYU App

I need to be able to read in a path file from my simple_switch.py application.I have added the following code to my simple_switch.py in python.
LOG = logging.getLogger(__name__)
CONF = cfg.CONF
CONF.register_cli_opts([
cfg.StrOpt('path-file', default='test.txt',
help='path-file')
])
I attempt to start the application as follows.
bin/ryu-manager --observe-links --path-file test.txt ryu/app/simple_switch.py
However I get the following error.
usage: ryu-manager [-h] [--app-lists APP_LISTS] [--ca-certs CA_CERTS]
[--config-dir DIR] [--config-file PATH]
[--ctl-cert CTL_CERT] [--ctl-privkey CTL_PRIVKEY]
[--default-log-level DEFAULT_LOG_LEVEL] [--explicit-drop]
[--install-lldp-flow] [--log-config-file LOG_CONFIG_FILE]
[--log-dir LOG_DIR] [--log-file LOG_FILE]
[--log-file-mode LOG_FILE_MODE]
[--neutron-admin-auth-url NEUTRON_ADMIN_AUTH_URL]
[--neutron-admin-password NEUTRON_ADMIN_PASSWORD]
[--neutron-admin-tenant-name NEUTRON_ADMIN_TENANT_NAME]
[--neutron-admin-username NEUTRON_ADMIN_USERNAME]
[--neutron-auth-strategy NEUTRON_AUTH_STRATEGY]
[--neutron-controller-addr NEUTRON_CONTROLLER_ADDR]
[--neutron-url NEUTRON_URL]
[--neutron-url-timeout NEUTRON_URL_TIMEOUT]
[--noexplicit-drop] [--noinstall-lldp-flow]
[--noobserve-links] [--nouse-stderr] [--nouse-syslog]
[--noverbose] [--observe-links]
[--ofp-listen-host OFP_LISTEN_HOST]
[--ofp-ssl-listen-port OFP_SSL_LISTEN_PORT]
[--ofp-tcp-listen-port OFP_TCP_LISTEN_PORT] [--use-stderr]
[--use-syslog] [--verbose] [--version]
[--wsapi-host WSAPI_HOST] [--wsapi-port WSAPI_PORT]
[--test-switch-dir TEST-SWITCH_DIR]
[--test-switch-target TEST-SWITCH_TARGET]
[--test-switch-tester TEST-SWITCH_TESTER]
[app [app ...]]
ryu-manager: error: unrecognized arguments: --path-file
It does look like I need to register a new command line option somewhere before I can use it.Can some-one point out to me how to do that? Also can someone explain how to access the file(text.txt) inside the program?
You're on the right track, however the CONF entry that you are creating actually needs to be loaded before your app is loaded, otherwise ryu-manager has no way of knowing it exists!
The file you are looking for is flags.py, under the ryu directory of the source tree (or under the root installation directory).
This is how the ryu/tests/switch/tester.py Ryu app defines it's own arguments, so you might use that as your reference:
CONF.register_cli_opts([
# tests/switch/tester
cfg.StrOpt('target', default='0000000000000001', help='target sw dp-id'),
cfg.StrOpt('tester', default='0000000000000002', help='tester sw dp-id'),
cfg.StrOpt('dir', default='ryu/tests/switch/of13',
help='test files directory')
], group='test-switch')
Following this format, the CONF.register_cli_opts takes an array of config types exactly as you have done it (see ryu/cfg.py for the different types available).
You'll notice that when you run the ryu-manager help, i.e.
ryu-manager --help
the list that comes up is sorted by application (e.g. the group of arguments under 'test-switch options'). For that reason, you will want to specify a group name for your set of commands.
Now let us say that you used the group name 'my-app' and have an argument named 'path-file' in that group, the command line argument will be --my-app-path-file (this can get a little long), while you can access it in your application like this:
from ryu import cfg
CONF = cfg.CONF
path_file = CONF['my-app']['path_file']
Note the use of dash versus the use of underscores.
Cheers!

collective.xsendfile, ZODB blobs and UNIX file permissions

I am currently trying to configure collective.xsendfile, Apache mod_xsendfile and Plone 4.
Apparently the Apache process does not see blobstrage files on the file system because they contain permissions:
ls -lh var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x18/0xd5/0x19/0x038ea09d0eddc611.blob
-r-------- 1 plone plone 1006K May 28 15:30 var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x18/0xd5/0x19/0x038ea09d0eddc611.blob
How do I configure blobstorage to give additional permissions, so that Apache could access these files?
The modes with which the blobstorage writes it's directories and files is hardcoded in ZODB.blob. Specifically, the standard ZODB.blob.FileSystemHelper class creates secure directories (only readable and writable for the current user) by default.
You could provide your own implementation of FileSystemHelper that would either make this configurable, or just sets the directory modes to 0750, and then patch ZODB.blob.BlobStorageMixin to use your class instead of the default:
import os
from ZODB import utils
from ZODB.blob import FilesystemHelper, BlobStorageMixin
from ZODB.blob import log, LAYOUT_MARKER
class GroupReadableFilesystemHelper(FilesystemHelper):
def create(self):
if not os.path.exists(self.base_dir):
os.makedirs(self.base_dir, 0750)
log("Blob directory '%s' does not exist. "
"Created new directory." % self.base_dir)
if not os.path.exists(self.temp_dir):
os.makedirs(self.temp_dir, 0750)
log("Blob temporary directory '%s' does not exist. "
"Created new directory." % self.temp_dir)
if not os.path.exists(os.path.join(self.base_dir, LAYOUT_MARKER)):
layout_marker = open(
os.path.join(self.base_dir, LAYOUT_MARKER), 'wb')
layout_marker.write(self.layout_name)
else:
layout = open(os.path.join(self.base_dir, LAYOUT_MARKER), 'rb'
).read().strip()
if layout != self.layout_name:
raise ValueError(
"Directory layout `%s` selected for blob directory %s, but "
"marker found for layout `%s`" %
(self.layout_name, self.base_dir, layout))
def isSecure(self, path):
"""Ensure that (POSIX) path mode bits are 0750."""
return (os.stat(path).st_mode & 027) == 0
def getPathForOID(self, oid, create=False):
"""Given an OID, return the path on the filesystem where
the blob data relating to that OID is stored.
If the create flag is given, the path is also created if it didn't
exist already.
"""
# OIDs are numbers and sometimes passed around as integers. For our
# computations we rely on the 64-bit packed string representation.
if isinstance(oid, int):
oid = utils.p64(oid)
path = self.layout.oid_to_path(oid)
path = os.path.join(self.base_dir, path)
if create and not os.path.exists(path):
try:
os.makedirs(path, 0750)
except OSError:
# We might have lost a race. If so, the directory
# must exist now
assert os.path.exists(path)
return path
def _blob_init_groupread(self, blob_dir, layout='automatic'):
self.fshelper = GroupReadableFilesystemHelper(blob_dir, layout)
self.fshelper.create()
self.fshelper.checkSecure()
self.dirty_oids = []
BlobStorageMixin._blob_init = _blob_init_groupread
Quite a hand-full, you may want to make this a feature request for ZODB3 :-)
While setting up a backup routine for a ZOPE/ZEO setup, I ran into the same problem with blob permissions.
After trying to apply the monkey patch that Mikko wrote (which is not that easy) i came up with a "real" patch to solve the problem.
The patch suggested by Martijn is not complete, it still does not set the right mode on blob files.
So here's my solution:
1.) Create a patch containing:
Index: ZODB/blob.py
===================================================================
--- ZODB/blob.py (Revision 121959)
+++ ZODB/blob.py (Arbeitskopie)
## -337,11 +337,11 ##
def create(self):
if not os.path.exists(self.base_dir):
- os.makedirs(self.base_dir, 0700)
+ os.makedirs(self.base_dir, 0750)
log("Blob directory '%s' does not exist. "
"Created new directory." % self.base_dir)
if not os.path.exists(self.temp_dir):
- os.makedirs(self.temp_dir, 0700)
+ os.makedirs(self.temp_dir, 0750)
log("Blob temporary directory '%s' does not exist. "
"Created new directory." % self.temp_dir)
## -359,8 +359,8 ##
(self.layout_name, self.base_dir, layout))
def isSecure(self, path):
- """Ensure that (POSIX) path mode bits are 0700."""
- return (os.stat(path).st_mode & 077) == 0
+ """Ensure that (POSIX) path mode bits are 0750."""
+ return (os.stat(path).st_mode & 027) == 0
def checkSecure(self):
if not self.isSecure(self.base_dir):
## -385,7 +385,7 ##
if create and not os.path.exists(path):
try:
- os.makedirs(path, 0700)
+ os.makedirs(path, 0750)
except OSError:
# We might have lost a race. If so, the directory
# must exist now
## -891,7 +891,7 ##
file2.close()
remove_committed(f1)
if chmod:
- os.chmod(f2, stat.S_IREAD)
+ os.chmod(f2, stat.S_IRUSR | stat.S_IRGRP)
if sys.platform == 'win32':
# On Windows, you can't remove read-only files, so make the
You can also take a look at the patch here -> http://pastebin.com/wNLYyXvw
2.) Store the patch under name 'blob.patch' in your buildout root directory
3.) Extend your buildout configuration:
parts +=
patchblob
postinstall
[patchblob]
recipe = collective.recipe.patch
egg = ZODB3
patches = blob.patch
[postinstall]
recipe = plone.recipe.command
command =
chmod -R g+r ${buildout:directory}/var
find ${buildout:directory}/var -type d | xargs chmod g+x
update-command = ${:command}
The postinstall sections sets desired group read permissions on already existing blobs. Note, also execute permission must be given to the blob folders, that group can enter the directories.
I've tested this patch with ZODB 3.10.2 and 3.10.3.
As Martijn suggested, this should be configurable and part of the ZODB directly.

Python file location conventions and Import errors

I'm new to Python and I simply don't know how to handle this specific problem:
I'm trying to run an executable (named ros_M5e.py) that's located in the directory /opt/ros/diamondback/stacks/hrl/hrl_rfid/src/hrl_rfid/ros_M5e.py (annoyingly long filepath, I know, but necessary). However, within the ros_M5e.py file there is a call to another file that is further up the file path: from hrl.hrl_rfid.msg import RFIDread. The directory msg indeed is located at /opt/ros/diamondback/stacks/hrl/hrl_rfid/ and it does indeed contain the file RFIDread. However, whenever I try to execute ros_M5e.py I get this error:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/hrl/hrl_rfid/src/hrl_rfid/ros_M5e.py", line 37, in <module>
from hrl.hrl_rfid.msg import RFIDread
ImportError: No module named hrl.hrl_rfid.msg
Would someone with some expertise please shine some light on this problem for me? It seems like just a rudimentary file location problem, but I just don't know the appropriate Python conventions to fix it. I've tried putting the ros_M5e.py file in the same directory as the files it calls and changing the filepaths but to no avail.
Thanks a lot,
Khiya
Sure, I can help you get it up and running.
From the StackOverflow posting, it would seem that you're checking out the stack to /opt/ros/diamondback. This is no good, as it is a system path. You need to install into your local path. The reason for "readonly" on the repository is that you do not have permissions to make changes to the code -- it will still work just fine for you on your local machine. I spent a fair amount of time showing how to use this package (at least the python version) here:
http://www.ros.org/wiki/hrl_rfid
I'll try to do a quick run-through for installing it.... Run the following commands:
cd
mkdir sandbox
cd sandbox/
svn checkout http://gt-ros-pkg.googlecode.com/svn/trunk/hrl/hrl_rfid hrl_rfid (double-check that this checkout works OK!)
Add the following line to the bottom of your bashrc to tell ROS where to find the new package. (You may use "gedit ~/.bashrc")
export ROS_PACKAGE_PATH=$ROS_PACKAGE_PATH:$HOME/sandbox/hrl_rfid
Now execute the following:
roscd hrl_rfid (did you end up in the correct directory?)
rosmake hrl_rfid (did it make without errors?)
roscd hrl_rfid/src/hrl_rfid
At this point everything is actually installed correctly. By default, ros_M5e.py assumes that the reader is located at "/dev/robot/RFIDreader". Unless you've already altered udev rules, this will not be the case on your machine. I suggest running through the code:
http://www.ros.org/wiki/hrl_rfid
using iPython (a command-line python prompt that will let you execute python commands one at a time) to make sure everything is working (replace /dev/ttyUSB0 with whatever device your RFID reader is connected as):
import lib_M5e as M5e
r = M5e.M5e( '/dev/ttyUSB0', readPwr = 3000 )
r.ChangeAntennaPorts( 1, 1 )
r.QueryEnvironment()
r.TrackSingleTag( 'test_tag_id_' )
r.ChangeTagID( 'test_tag_id_' )
r.QueryEnvironment()
r.TrackSingleTag( 'test_tag_id_' )
r.ChangeAntennaPorts( 2, 2 )
r.QueryEnvironment()
This means that the underlying library is working just fine. Next, test ROS (make sure "roscore" is running!), by putting this in a python file and executing:
import lib_M5e as M5e
def P1(r):
r.ChangeAntennaPorts(1,1)
return 'AntPort1'
def P2(r):
r.ChangeAntennaPorts(2,2)
return 'AntPort2'
def PrintDatum(data):
ant, ids, rssi = data
print data
r = M5e.M5e( '/dev/ttyUSB0', readPwr = 3000 )
q = M5e.M5e_Poller(r, antfuncs=[P1, P2], callbacks=[PrintDatum])
q.query_mode()
t0 = time.time()
while time.time() - t0 < 3.0:
time.sleep( 0.1 )
q.track_mode( 'test_tag_id_' )
t0 = time.time()
while time.time() - t0 < 3.0:
time.sleep( 0.1 )
q.stop()
OK, everything works now. You can make your own node that is tuned to your setup:
#!/usr/bin/python
import ros_M5e as rm
def P1(r):
r.ChangeAntennaPorts(1,1)
return 'AntPort1'
def P2(r):
r.ChangeAntennaPorts(2,2)
return 'AntPort2'
ros_rfid = rm.ROS_M5e( name = 'my_rfid_server',
readPwr = 3000,
portStr = '/dev/ttyUSB0',
antFuncs = [P1, P2],
callbacks = [] )
rospy.spin()
ros_rfid.stop()
Or, ping me back and I can tweak ros_M5e.py to take an optional "portStr" -- though I recommend making your own so that you can name your antennas sensibly. Also, I highly recommend setting udev rules to ensure that the RFID reader always gets assigned to the same device: http://www.hsi.gatech.edu/hrl-wiki/index.php/Linux_Tools#udev_for_persistent_device_naming
BUS=="usb", KERNEL=="ttyUSB*", SYSFS{idVendor}=="0403", SYSFS{idProduct}=="6001", SYSFS{serial}=="ftDXR6FS", SYMLINK+="robot/RFIDreader"
If you do not do this... there is no guarantee that the reader will always be enumerated at /dev/ttyUSBx.
Let me know if you have any further problems.
~Travis Deyle (Hizook.com)
PS -- Did you modify ros_M5e.py to "from hrl.hrl_rfid.msg import RFIDread"? In the repo, it is "from hrl_rfid.msg import RFIDread". The latter is correct. As long as you have your ROS_PACKAGE_PATH correctly defined, and you've run rosmake on the package, then the import statement should work just fine. Also, I would not recommend posting ROS-related questions to StackOverflow. Very few people on here are going to be familiar with the ROS ecosystem (which is VERY complex). Please post questions here instead:
http://answers.ros.org/
http://code.google.com/p/gt-ros-pkg/issues/list
You need to make sure that following are true:
Directory /opt/ros/diamondback/stacks/ is in your python path.
/opt/ros/diamondback/stacks/hr1 contains __init__.py
/opt/ros/diamondback/stacks/hr1/hr1_rfid contians __init__.py
/opt/ros/diamondback/stacks/hr1/hr1_rfid/msg contians __init__.py
As the asker explained in comments that the RFIDRead does not have .py extension, so here is how that can be imported.
import imp
imp.load_source('RFIDRead', '/opt/ros/diamondback/stacks/hr1/hr1_rfid/msg/RFIDRead.msg')
Check out imp documentation for more information.