My flume config:
agent.sinks = s3hdfs
agent.sources = MySpooler
agent.channels = channel
agent.sinks.s3hdfs.type = hdfs
agent.sinks.s3hdfs.hdfs.path = s3a://mybucket/test
agent.sinks.s3hdfs.hdfs.filePrefix = FilePrefix
agent.sinks.s3hdfs.channel = channel
agent.sinks.s3hdfs.hdfs.useLocalTimeStamp = true
agent.sources.MySpooler.channels = channel
agent.sources.MySpooler.type = spooldir
agent.sources.MySpooler.spoolDir = /flume_to_aws
agent.sources.MySpooler.fileHeader = true
agent.channels.channel.type = memory
agent.channels.channel.capacity = 100
Now I will add a file in /flume_to_aws folder with the following content (text):
Oracle and SQL Server
After it is uploaded in S3, I downloaded the file and opened it, and it show the following text:
SEQ!org.apache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable
Œúg ÊC•ý¤ïM·T.C ! †"ûþ Oracle and SQL ServerÿÿÿÿŒúg ÊC•ý¤ïM·T.C
Why the file is not uploaded only with the text "Oracle and SQL Server"??
Problem solved. I have found this question in stackoverflow here
Flume is generating files in binary format instead of text format.
So, I have added the following lines:
agent.sinks.s3hdfs.hdfs.writeFormat = Text
agent.sinks.s3hdfs.hdfs.fileType = DataStream
Update Changes, problem remain****************
cheregeckodriver_autoinstaller.install()
profile = webdriver.FirefoxProfile("/Users/Toshiba/AppData/Roaming/Mozilla/Firefox/Profiles/zq6j04pa.default")
profile.set_preference("dom.webdriver.enabled",False)
profile.set_preference("useAutomationExtension", False)
profile.update_preferences()
desired = DesiredCapabilities.FIREFOX
path = '/Users/Toshiba/Desktop/backup/PythonLearning/Learning/Selenium/geckodriver'
binary = FirefoxBinary("/Users/Toshiba/AppData/Local/Mozilla Firefox/firefox.exe")
ua = UserAgent()
user_agent = ua.random
options = webdriver.FirefoxOptions()
options.add_argument(f'user-agent={user_agent}')
# options.add_argument('-incognito')
wd = webdriver.Firefox(executable_path=path, firefox_profile=profile, desired_capabilities=desired, firefox_binary=binary)
url = "https://docs.google.com/forms/u/0/"
wd.get(url)
# login to google
username_class = "whsOnd zHQkBf".replace(" ",".")
wd.find_element_by_class_name(username_class).send_keys(username)
time.sleep(random.uniform(1, 3))
btn_next_class = "VfPpkd-RLmnJb"
wd.find_elements_by_class_name(btn_next_class)[-1].click()
Problem
Automate testing software detected
Couldn't sign you in
This browser or app may not be secure. Learn more
Try using a different browser. If you’re already using a supported browser, you can refresh your screen and try again to sign in.
I am getting the following problem with this plugin. using trac 0.12.5 on centos 5. the following log is from a comment. user responding is sharif.uddin, user created is jason. and user i am trying to cc to is ramy.
Trac[paradox:env] INFO: Reloading environment due to configuration change
Trac[paradox:env] INFO: -------------------------------- environment startup [Trac 0.12.5] --------------------------------
Trac[paradox:api] INFO: Synchronized '(default)' repository in 0.01 seconds
Trac[paradox:api] INFO: Synchronized '(default)' repository in 0.00 seconds
Trac[paradox:env] INFO: Reloading environment due to configuration change
Trac[paradox:env] INFO: -------------------------------- environment startup [Trac 0.12.5] --------------------------------
Trac[paradox:api] INFO: Synchronized '(default)' repository in 0.06 seconds
Trac[paradox:SmtpLdapEmailSender] INFO: Binding to LDAP as cn=Administrator,cn=Users,dc=domain,dc=com
Trac[paradox:SmtpLdapEmailSender] INFO: Updating list of recipients
Trac[paradox:SmtpLdapEmailSender] WARNING: Searching LDAP server ldap://echo.uk.domain.com for user jasona#DOMAIN.COM
Trac[paradox:SmtpLdapEmailSender] WARNING: Found e-mail address: Jason.Aftalion#domain.com
Trac[paradox:SmtpLdapEmailSender] WARNING: Searching LDAP server ldap://echo.uk.domain.com for user sharifu#DOMAIN.COM
Trac[paradox:SmtpLdapEmailSender] WARNING: Found e-mail address: Sharif.Uddin#domain.com
Trac[paradox:SmtpLdapEmailSender] WARNING: Searching LDAP server ldap://echo.uk.domain.com for user Ramy.Mahmoud#domain.com
Trac[paradox:notification] INFO: Sending notification through SMTP at hero.uk.domain.com:25 to ['Jason.Aftalion#domain.com', 'Sharif.Uddin#domain.com', u'Ramy.Mahmoud#domain.com', 'support#domain.com']
Trac[paradox:api] INFO: Synchronized '(default)' repository in 0.01 seconds
Trac[paradox:api] INFO: Synchronized '(default)' repository in 0.00 seconds
I have the following code for the plugin in site-packages
class SmtpLdapEmailSender(SmtpEmailSender):
implements(IEmailSender)
email_ldap_serveruri = Option('notification', 'email_ldap_serveruri', '',
"""AD LDAP Server to use for looking up e-mail addresses""")
email_ldap_port = IntOption('notification', 'email_ldap_port', 389, """AD LDAP Server port""")
email_ldap_binddn = Option('notification', 'email_ldap_binddn', '',
"""Bind DN for LDAP lookup. If not given, Kerberos auth will be used for current user""")
email_ldap_bindpw = Option('notification', 'email_ldap_bindpw', '', """Password for non-kerberos auth""")
email_ldap_basedn = Option('notification', 'email_ldap_basedn', '', """Base DN to use for LDAP searches""")
email_attr = 'mail'
def __init__(self):
self.log.warn("Initialising LDAP object with URI: ", self.email_ldap_serveruri)
self.ldap_conn=ldap.initialize(self.email_ldap_serveruri)
def send(self, from_addr, recipients, message):
#self.log.warn(recipients)
if self.email_ldap_binddn != None:
self.log.info("Binding to LDAP as " + self.email_ldap_binddn)
self.ldap_conn.bind_s(self.email_ldap_binddn, self.email_ldap_bindpw, ldap.AUTH_SIMPLE)
else:
self.log.info("Binding to LDAP with Kerberos")
self.ldap_conn.bind_s()
#Iterate through recipients, checking for correct e-mail addresses in LDAP
#Output in ldapRecipients
self.log.info("Updating list of recipients")
new_recipients = []
def isset(variable):
return variable in locals() or variable in globals()
for i, addr in enumerate(recipients):
self.log.warn("Searching LDAP server %s for user %s", self.email_ldap_serveruri, addr)
search_string = 'userPrincipalName=' + addr
result = self.ldap_conn.search_s(self.email_ldap_basedn, ldap.SCOPE_SUBTREE, search_string, [self.email_attr])
#result is formatted as a string (result) in a list of [attr values], in a dictionary of {attr_name=>attr_values}
#in a tuple of (DN, Entry), within a list of results. So result for principle name jasona#domain.com would be
#[('CN=Jason Aftalion,OU=TechSupport,OU=Woking,OU=Sites,DC=domain,DC=com', {'mail': ['Jason.Aftalion#domain.com']})]
#self.log.error(addr)
if len(result) > 0:
if result[0][1][self.email_attr][0]:
self.log.warn("Found e-mail address: " + result[0][1][self.email_attr][0])
new_recipients.append(result[0][1][self.email_attr][0])
else:
self.log.warn("Could not find e-mail address")
new_recipients.append(addr)
else:
new_recipients.append(addr)
new_recipients.append("support#domain.com")
#self.log.error(new_recipients)
return super(SmtpLdapEmailSender,self).send(from_addr, new_recipients, message)
Also when the email gets sent out there is no one on the to address. I think i need to add the u before the quites open on the email if you see the log line Trac[paradox:notification] INFO: Sending notification through SMTP at hero.uk.domain.com:25 to ['Jason.Aftalion#domian.com', 'Sharif.Uddin#domian.com', u'Ramy.Mahmoud#domain.com'] . ramy is the only one that appears in the email as it is placed on the cc section of the ini file.
UPDATE
[root#hero plugins]# easy_install http://trac-hacks.org/svn/announcerplugin/trunk
Downloading http://trac-hacks.org/svn/announcerplugin/trunk
Doing subversion checkout from http://trac-hacks.org/svn/announcerplugin/trunk to /tmp/easy_install-hkATrd/trunk
Processing trunk
Running setup.py -q bdist_egg --dist-dir /tmp/easy_install-hkATrd/trunk/egg-dist-tmp-dGEGqu
File "build/bdist.linux-i686/egg/announcer/opt/bitten/announce.py", line 71
yield
^
SyntaxError: invalid syntax
zip_safe flag not set; analyzing archive contents...
TracAnnouncer 1.0dev-r12503 is already the active version in easy-install.pth
Installed /usr/lib/python2.4/site-packages/TracAnnouncer-1.0dev_r12503-py2.4.egg
Processing dependencies for TracAnnouncer==1.0dev-r12503
Finished processing dependencies for TracAnnouncer==1.0dev-r12503
UPDATE 2
It installed successfully
easy_install http://trac-hacks.org/svn/announcerplugin/trunk
Downloading http://trac-hacks.org/svn/announcerplugin/trunk
Doing subversion checkout from http://trac-hacks.org/svn/announcerplugin/trunk to /tmp/easy_install-AGCmXH/trunk
Processing trunk
Running setup.py -q bdist_egg --dist-dir /tmp/easy_install-AGCmXH/trunk/egg-dist-tmp-6tmNSt
zip_safe flag not set; analyzing archive contents...
Removing TracAnnouncer 1.0dev-r12503 from easy-install.pth file
Adding TracAnnouncer 1.0dev-r13963 to easy-install.pth file
Installed /usr/lib/python2.4/site-packages/TracAnnouncer-1.0dev_r13963-py2.4.egg
Processing dependencies for TracAnnouncer==1.0dev-r13963
Finished processing dependencies for TracAnnouncer==1.0dev-r13963
I get following in log file now. I cannot install python 2.7
Trac[paradox:env] INFO: -------------------------------- environment startup [Trac 0.12.5] --------------------------------
Trac[paradox:loader] ERROR: Skipping "announcer.email_decorators = announcer.email_decorators":
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/Trac-0.12.5-py2.4.egg/trac/loader.py", line 68, in _load_eggs
entry.load(require=True)
File "/usr/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg/pkg_resources.py", line 1954, in load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])
File "build/bdist.linux-i686/egg/announcer/email_decorators.py", line 7, in ?
ImportError: No module named utils
Trac[paradox:api] INFO: Synchronized '(default)' repository in 0.66 seconds
local ini file
cat /data/intranet/html/trac/paradox/conf/trac.ini
# -*- coding: utf-8 -*-
[changeset]
max_diff_files = 0
[components]
acct_mgr.admin.accountmanageradminpanel = disabled
acct_mgr.api.accountmanager = disabled
acct_mgr.db.sessionstore = disabled
acct_mgr.guard.accountguard = disabled
acct_mgr.macros.accountmanagerwikimacros = disabled
acct_mgr.notification.accountchangelistener = disabled
acct_mgr.notification.accountchangenotificationadminpanel = disabled
acct_mgr.register.emailcheck = disabled
acct_mgr.register.emailverificationmodule = disabled
advancedworkflow.controller.ticketworkflowopownercomponent = disabled
advancedworkflow.controller.ticketworkflowopownerfield = disabled
advancedworkflow.controller.ticketworkflowopownerprevious = disabled
advancedworkflow.controller.ticketworkflowopresetmilestone = disabled
advancedworkflow.controller.ticketworkflowoprunexternal = disabled
advancedworkflow.controller.ticketworkflowopstatusprevious = disabled
advancedworkflow.controller.ticketworkflowoptriage = disabled
advancedworkflow.controller.ticketworkflowopxref = disabled
announcer.api.announcementsystem = enabled
announcer.api.subscriptionresolver = enabled
announcer.distributors.mail.emaildistributor = enabled
announcer.distributors.mail.sendmailemailsender = enabled
announcer.distributors.mail.smtpemailsender = enabled
announcer.email_decorators.announceremaildecorator = enabled
announcer.email_decorators.staticemaildecorator = enabled
announcer.email_decorators.threadingemaildecorator = enabled
announcer.email_decorators.ticketaddlheaderemaildecorator = enabled
announcer.email_decorators.ticketsubjectemaildecorator = enabled
announcer.email_decorators.wikisubjectemaildecorator = enabled
announcer.filters.defaultpermissionfilter = enabled
announcer.formatters.ticketformatter = enabled
announcer.formatters.wikiformatter = enabled
announcer.opt.subscribers.allticketsubscriber = enabled
announcer.opt.subscribers.generalwikisubscriber = enabled
announcer.opt.subscribers.joinablegroupsubscriber = enabled
announcer.opt.subscribers.ticketcomponentownersubscriber = enabled
announcer.opt.subscribers.ticketcomponentsubscriber = enabled
announcer.opt.subscribers.ticketcustomfieldsubscriber = enabled
announcer.opt.subscribers.userchangesubscriber = enabled
announcer.opt.subscribers.watchsubscriber = enabled
announcer.pref.announcerpreferences = enabled
announcer.pref.subscriptionmanagementpanel = enabled
announcer.producers.attachmentchangeproducer = enabled
announcer.producers.ticketchangeproducer = enabled
announcer.producers.wikichangeproducer = enabled
announcer.resolvers.defaultdomainemailresolver = enabled
announcer.resolvers.sessionemailresolver = enabled
announcer.resolvers.specifiedemailresolver = enabled
announcer.resolvers.specifiedxmppresolver = enabled
announcer.subscribers.carboncopysubscriber = enabled
announcer.subscribers.ticketownersubscriber = enabled
announcer.subscribers.ticketreportersubscriber = enabled
announcer.subscribers.ticketupdatersubscriber = enabled
spectrum.smtpldapemailsender.smtpldapemailsender = enabled
tracopt.mimeview.php.phprenderer = enabled
[header_logo]
alt =
link = http://intranet/trac/paradox/
src = common/trac_banner.png
[inherit]
file = /usr/share/trac/conf/trac.ini
[logging]
log_level = INFO
log_type = file
[project]
descr = Paradox replacement
name = Paradox
url = http://intranet/sidb
[ticket]
default_component = other
default_milestone = create/update project
default_version = v12
[ticket-workflow]
accept = new -> assigned
accept.operations = set_owner_to_self
accept.permissions = TICKET_MODIFY
leave = * -> *
leave.default = 1
leave.operations = leave_status
reassign = new,assigned,reopened -> new
reassign.operations = set_owner
reassign.permissions = TICKET_MODIFY
reopen = closed -> reopened
reopen.operations = del_resolution
reopen.permissions = TICKET_CREATE
resolve = new,assigned,reopened -> closed
resolve.operations = set_resolution
resolve.permissions = TICKET_MODIFY
[trac]
base_url = http://intranet/trac/paradox/
check_auth_ip = true
metanav = login,logout,settings,help,about
repository_dir = /data/subversion/paradox
[notification]
smtp_always_cc = Ramy.Mahmoud#domain.com
[announcer]
use_public_cc = true
global ini file
cat ../conf/trac.ini
[announcer]
use_public_cc = true
#admit_domains =
#always_notify_component_owner = true
#always_notify_owner = true
#always_notify_reporter = true
#always_notify_updater = true
#default_email_format = text/html
#email_address_resolvers = SpecifiedEmailResolver, SessionEmailResolver
#ignore_domains =
#mime_encoding = base64
#smtp_always_bcc =
#smtp_always_cc =
#smtp_default_domain =
#smtp_enabled = true
#smtp_from = trac-no-reply#domain.com
#smtp_from_name = Trac
#smtp_password =
#smtp_port = 25
#smtp_replyto = no-reply#domain.com
#smtp_server = hero
#smtp_subject_prefix = __default__
#smtp_timeout = 30
#smtp_user =
#t#icket_email_header_fields = owner, reporter, milestone, component, priority, severity.
#ticket_email_subject = Ticket #${ticket.id}: ${ticket['summary']}.
#ticket_subject_template = $prefix $ticket.id: $summary
#use_public_cc = false
#use_short_addr = false
#use_tls = false
#email_enabled = true
[notification]
always_notify_owner = false
always_notify_reporter = true
always_notify_updater = true
#mime_encoding = base64
#smtp_always_cc = sharifu#domain.com
#smtp_default_domain = domain.com
smtp_enabled = true
smtp_from = trac#domain.com
smtp_password =
smtp_port = 25
smtp_replyto = no-reply#domain.com
smtp_server = hero.uk.domain.com
smtp_subject_prefix = __default__
smtp_user =
use_public_cc = false
use_short_addr = false
use_tls = false
#ignore_domains = domain.com
email_sender=SmtpLdapEmailSender
email_ldap_serveruri = ldap://echo.uk.domain.com
email_ldap_port = 389
email_ldap_basedn = ou=Sites,dc=domain,dc=com
email_ldap_binddn = cn=Administrator,cn=Users,dc=domain,dc=com
email_ldap_bindpw = ****
[ldap]
enable = true
global_perms = true
host = echo
basedn = dc=domain,dc=com
user_rdn = ou=sites
group_rdn = cn=users
store_bind = true
bind_user = cn=Administrator,cn=users,dc=domain,dc=com
bind_passwd = ****
[trac]
base_url = http://intranet/trac/
#permission_store = LdapPermissionStore
[logging]
log_format = Trac[$(basename)s:$(module)s] $(levelname)s: $(message)s
log_type = syslog
log_level = WARN
[components]
webadmin.* = enabled
#ldapauth.* = enabled
#ldapplugin.* = enabled
#ldapplugin.api.ldappermissiongroupprovider = enabled
#ldapplugin.api.ldappermissionstore = disabled
ticketdelete.* = enabled
tracopt.ticket.deleter = enabled
tracwysiwyg.* = enabled
advancedworkflow.* = enabled
#tickettemplate.* = enabled
tracopt.ticket.commit_updater.committicketreferencemacro = enabled
tracopt.ticket.commit_updater.committicketupdater = enabled
ticketchangesets.* = enabled
ticketlog.* = enabled
#announcer.* = enabled
#announcer.api.announcementsystem = enabled
#announcer.distributors.mail.emaildistributor = enabled
#announcer.formatters.ticket.ticketformatter = enabled
#announcer.formatters.wiki.wikiformatter = enabled
#announcer.pref.announcerpreferences = enabled
#announcer.producers.attachment.attachmentchangeproducer = enabled
#announcer.producers.ticket.ticketchangeproducer = enabled
#announcer.producers.wiki.wikichangeproducer = enabled
#announcer.resolvers.sessionemail.sessionemailresolver = enabled
#announcer.subscribers.ticket_compat.carboncopysubscriber = enabled
#announcer.subscribers.ticket_compat.legacyticketsubscriber = enabled
#announcer.subscribers.ticket_components.ticketcomponentsubscriber = enabled
#announcer.subscribers.ticket_custom.ticketcustomfieldsubscriber = enabled
#announcer.subscribers.watch_users.userchangesubscriber = enabled
#announcer.subscribers.watchers.watchsubscriber = enabled
#[tickettemplate]
#field_list = summary, description, reporter, owner, priority, cc, milestone, component, version, type
#enable_custom = true
[ticket-changesets]
check_perms = true
collapsed = false
commands.close = close closed closes fix fixed fixes
commands.refs = addresses re references refs see
compact = true
envelope =
hide_when_none = false
notify = true
resolution = fixed
ticket_comments = true
[ticket]
commit_ticket_update_envelope = []
commit_ticket_update_commands.close =
commit_ticket_update_commands.refs = <ALL>
commit_ticket_update_check_perms = true
commit_ticket_update_notify = true
[ticketlog]
; optional: custom your log message pattern
log_pattern = \s*#%s\s+.*
; optional: set log message's max length, default is no limit
log_message_maxlength = 100
When i leave a comment i see the following extra bit in log file
Trac[paradox:api] ERROR: AnnouncementSystem failed.
Traceback (most recent call last):
File "build/bdist.linux-i686/egg/announcer/api.py", line 560, in _real_send
File "build/bdist.linux-i686/egg/announcer/api.py", line 311, in subscriptions
TypeError: itemgetter expected 1 arguments, got 4
QUESTION 2
Do emails notification not get sent out when adding attachments? i found the following in the log...
Trac[paradox:api] INFO: Synchronized '(default)' repository in 0.01 seconds
Trac[paradox:attachment] INFO: New attachment: ticket:48: RE Conversation with Ringo Au.msg by sharifu#DOMAIN.COM
Trac[paradox:api] ERROR: AnnouncementSystem failed.
Traceback (most recent call last):
File "build/bdist.linux-i686/egg/announcer/api.py", line 560, in _real_send
File "build/bdist.linux-i686/egg/announcer/api.py", line 311, in subscriptions
TypeError: itemgetter expected 1 arguments, got 4
I do want to setup pelican in such way that it will allow me to compile static markdown files by keeping their location but changing the compiled file extensions on the destination.
Here is an example:
content/about.md --> output/about.html
content/foo/bar.md --> output/foo/bar.html
Inside config I tried these:
PAGE_URL = '{path}.html'
PAGE_SAVE_AS = '{path}.html'
ARTICLE_URL = '{path}.html'
ARTICLE_SAVE_AS = '{path}.html'
The problem is that the .md part is not removed and that {path} contains the entire path with the filename.
I know that I could use the {slug} and manually enter the slug inside bug that's not cool at all as it would require me to add a slug for each page.
You can try this in your pelican configuration:
FILENAME_METADATA = '(?P<slug>.*)' # use markdown file name as the slug meta
USE_FOLDER_AS_CATEGORY = True # use folder name as posts' category
ARTICLE_URL = '{category}/{slug}.html'
ARTICLE_SAVE_AS = ARTICLE_URL
PAGE_URL = '{slug}.html'
PAGE_SAVE_AS = PAGE_URL
CATEGORY_URL = '{slug}/index.html'
CATEGORY_SAVE_AS = CATEGORY_URL
For details about these variables, see http://docs.getpelican.com/en/3.1.1/settings.html
This used to work but after some modifications by the other programmers it just fails to work. I have this code on my Bootstrap:
protected function _initDatabase ()
{
$resource = $this->getPluginResource('multidb');
$resource->init();
Zend_Registry::set('gtap', $resource->getDb('gtap'));
Zend_Registry::set('phpbb', $resource->getDb('phpbb'));
}
Upon loading, this error shows up:
Fatal error: Call to a member function init() on a non-object in
/var/www/gamebowl3/application/Bootstrap.php on line 105
My php.ini has this entry on tis include_path:
.:/usr/share/php:/etc/apache2/libraries
and the i can see that multidb.php is located in:
/etc/apache2/librarties/Zend/Application/Resource
Can somebody tell me what causes the error? Thanks!
I just found out that the problem is in application.ini. Added a newly-introduced setting to the usual set of configs. Here it is:
;Gtap Database
resources.multidb.gtap.adapter = "PDO_MYSQL"
resources.multidb.gtap.host = "localhost"
resources.multidb.gtap.username = "root"
resources.multidb.gtap.password = "letmein1"
resources.multidb.gtap.dbname = "gtap"
resources.multidb.gtap.isDefaultTableAdapter = true
resources.multidb.gtap.default = true
;Forum Database
resources.multidb.phpbb.adapter = "PDO_MYSQL"
resources.multidb.phpbb.host = "localhost"
resources.multidb.phpbb.username = "root"
resources.multidb.phpbb.password = "letmein1"
resources.multidb.phpbb.dbname = "phpbb"
resources.multidb.phpbb.isDefaultTableAdapter = false
Also, make sure you have the latest Zend Framework Library and add it to PHP's include path. That should fix everything up.