I have a small problem with AXTLS. I'm trying to establish a SSL communication with a server, but it looks like it double-frees something in "ssl_client_new" function.
The code creates a new ctx (ssl_ctx_new) and loads the certificates (ssl_obj_load) during initialization (no problem here), the socket is connected and when I call "ssl_client_new", libc detects a double free:
"double free or corruption (fasttop): 0x0809f740"
I started the application under gdb, and this is the backtrace:
Program received signal SIGABRT, Aborted. 0xb7fdd424 in
__kernel_vsyscall () (gdb) bt
#0 0xb7fdd424 in __kernel_vsyscall ()
#1 0xb7d371ef in __GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0xb7d3a835 in __GI_abort () at abort.c:91
#3 0xb7d722fa in __libc_message (do_abort=2, fmt=0xb7e6a3bc "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:201
#4 0xb7d7ce42 in malloc_printerr (action=<optimized out>, str=<optimized out>, ptr=0x809f740) at malloc.c:5007
#5 0x080712fd in asn1_name ()
#6 0x0806849f in x509_new ()
#7 0x080642cc in process_certificate ()
#8 0x08067caa in do_clnt_handshake ()
#9 0x08066675 in basic_read ()
#10 0x08067a07 in ssl_client_new ()
The ctx pointer and the sockets are ok, and last two parameters are NULL and 0. (http://axtls.sourceforge.net/dox/group_c_api.html#ga4eef242a559b06d72b862c1e5ab3d0a2)
Here is the strace for my socket:
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 8
fcntl64(8, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(8, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(8, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("81.12.132.173")}, 16) = -1 EINPROGRESS (Operation now in progress)
select(9, NULL, [8], NULL, {10, 0}) = 1 (out [8], left {9, 996818})
getsockopt(8, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
fcntl64(8, F_SETFL, O_RDWR) = 0
fcntl64(8, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(8, F_SETFL, O_RDWR) = 0
write(8, "\26\3\1\0003\1\0\0/\3\1P#hf?\222Y\3046n\215\364\317\34-D8\311\270=\225"..., 56) = 56
read(8, "\26\3\1\0J", 5) = 5
read(8, "\2\0\0F\3\1Q\301_\210\267\200\352*}H\330\265\n;\33\253\31\24\320\377+\3\371\276g\362"..., 74) = 74
read(8, "\26\3\1\20)", 5) = 5
read(8, "\v\0\20%\0\20\"\0\5\0070\202\5\0030\202\3\353\240\3\2\1\2\2\23w\0\0~\263\3446"..., 4137) = 2632
read(8, "1\0270\25\6\3U\4\n\f\16Vodafone Group1)0'\6\3U"..., 1505) = 1505
So ssl handshake started.
And immediatly after this:
open("/dev/tty", O_RDWR|O_NOCTTY|O_NONBLOCK) = 9
writev(9, [{"*** glibc detected *** ", 23}, {"./HTTPS_TOOL", 11}, {": ", 2}, {"double free or corruption (fastt"..., 35}, {": 0x", 4}, {"082ba740", 8}, {" ***\n", 5}], 7*** glibc detected *** ./HTTPS_TOOL: double free or corruption (fasttop): 0x082ba740 ***
) = 88
Any ideas? Which can be the problem and what can I do?
Thank you.
Are you still getting the issue? I haven't had any reports of this and it may be a way the library is being used.
The library has been checked many times in Valgrind and there aren't any recent issues. Could you perhaps run it in Valgrind to verify?
Related
clang is:
└──( /opt/homebrew/Cellar/llvm/13.0.1_1/bin/clang++ --version
Homebrew clang version 13.0.1
Target: arm64-apple-darwin21.1.0
Thread model: posix
InstalledDir: /opt/homebrew/Cellar/llvm/13.0.1_1/bin
ASAN_OPTIONS is detect_stack_use_after_return=false detect_leaks=1
CMakeLists.txt has:
add_compile_options(-fsanitize=address -fsanitize=leak -g)
add_link_options(-fsanitize=address)
after the program exits, lots of leaks report, but in stacktrace, all objective-c function show as <unknown module>, and the address 0x3f368001a2e367b4 also seems wrong:
Direct leak of 40 byte(s) in 1 object(s) allocated from:
#0 0x10343b678 in wrap_malloc+0x8c (libclang_rt.asan_osx_dynamic.dylib:arm64+0x3f678)
#1 0x19ff4d638 in _Block_copy+0x3c (libsystem_blocks.dylib:arm64+0x1638)
#2 0x3f368001a2e367b4 (<unknown module>)
#3 0x5c5b8001a2e363c4 (<unknown module>)
#4 0xea608001a2dec090 (<unknown module>)
#5 0x8e440001a2debf14 (<unknown module>)
#6 0x875a8001a2debdd0 (<unknown module>)
#7 0x2498001a2debd28 (<unknown module>)
#8 0x2a408001a2e3eab8 (<unknown module>)
#9 0x731e8001a31bbce4 (<unknown module>)
#10 0x48600001029f37dc (<unknown module>)
#11 0x102942cac in bgfx::gl::RendererContextGL::setRenderContextSize(unsigned int, unsigned int, unsigned int) renderer_gl.cpp:3475
#12 0x1028f97d0 in bgfx::gl::RendererContextGL::init(bgfx::Init const&) renderer_gl.cpp:1817
#13 0x1028f9024 in bgfx::gl::rendererCreate(bgfx::Init const&) renderer_gl.cpp:4044
#14 0x102865bc0 in bgfx::rendererCreate(bgfx::Init const&) bgfx.cpp:2606
#15 0x102866714 in bgfx::Context::rendererExecCommands(bgfx::CommandBuffer&) bgfx.cpp:2657
but while debugging, these symbols are correct shown in debugger:
so how to make them right in LeakSanitizer's report? thanks very much.
I have an Arduino which is communicating properly with my Mac over serial using the pySerialTransfer library, running for hours. Then there is an interruption of the serial of some sort - though I've been unable to identify the cause when this happens overnight, I can reproduce the behavior pretty easily by just unplugging the Arduino USB cable from the laptop. The python code on my laptop continues to run, but gets into this infinite error loop:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/pySerialTransfer/pySerialTransfer.py", line 257, in send
self.connection.write(stack)
File "/usr/local/lib/python3.7/site-packages/serial/serialposix.py", line 571, in write
raise SerialException('write failed: {}'.format(e))
serial.serialutil.SerialException: write failed: [Errno 6] Device not configured
SENT (12 byte struct): (0, -55.836434114277004, 31.732435543849192)
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/serial/serialposix.py", line 537, in write
n = os.write(self.fd, d)
OSError: [Errno 6] Device not configured
During handling of the above exception, another exception occurred:
I seem unable to catch the exceptions using any try / except blocks in my python code, so that I can detect this and restart or try to reconnect. Below is my current python code - appreciate any pointers! Thank you.
#!/usr/bin/python3
import sys
import time
import arduino
import messageboard
VERBOSE = arduino.VERBOSE
SN = '75835343130351802272' # arduino uno serial
SIMULATE_ARDUINO = arduino.SIMULATE_ARDUINO
if '-s' in sys.argv:
SIMULATE_ARDUINO = True
COMMAND_DELAY_TIME = 1 # send a command to servos every n seconds
def main():
servos = arduino.OpenArduino(sn=SN)
format_string = '<lff'
format_byte_size = arduino.SizeOf(format_string)
azimuth=90
while True:
if not servos and not SIMULATE_ARDUINO:
servos = arduino.OpenArduino(sn=SN)
if VERBOSE:
print('Reopening connection')
if servos or SIMULATE_ARDUINO:
if azimuth == 90:
azimuth = 85
else:
azimuth = 90
values = (0, azimuth, 0)
arduino.StuffObject(servos, values, format_string, format_byte_size)
if not SIMULATE_ARDUINO:
servos.send(format_byte_size)
time.sleep(COMMAND_DELAY_TIME)
if __name__ == "__main__":
main()
I started down the tack of approaches to kill function calls that run too long, with potential solutions using StopIt and multiprocessing. But those failed because it seems that what was actually happening was that pySerialTransfer was spawning a separate thread to handle the serial communication and then returning control back to the calling function, but keeping that separate thread alive, and that separate thread was the one stuck in infinite error loops. So I couldn't detect the condition by the solutions to terminate long-running function calls. And as mentioned in the comment thread, no exceptions were passed up to the caller either, so try-except clauses also failed.
However, the one thing I could detect was that an error to stderr was logged, and so if I could redirect that and detect a new error there, then that would get me down the right path. Enter this guidance, which steered me towards this solution:
#!/usr/bin/python3
import sys
import time
import io
import arduino
import messageboard
import contextlib
VERBOSE = arduino.VERBOSE
SN = '75835343130351802272' # arduino uno serial
SIMULATE_ARDUINO = arduino.SIMULATE_ARDUINO
if '-s' in sys.argv:
SIMULATE_ARDUINO = True
COMMAND_DELAY_TIME = 1 # send a command to servos every n seconds
def main():
servos = arduino.OpenArduino(sn=SN)
format_string = '<lff'
format_byte_size = arduino.SizeOf(format_string)
azimuth=90
while True:
if not servos and not SIMULATE_ARDUINO:
servos = arduino.OpenArduino(sn=SN)
if VERBOSE:
print('Reopening connection')
if servos or SIMULATE_ARDUINO:
if azimuth == 90:
azimuth = 85
else:
azimuth = 90
values = (0, azimuth, 0)
# Detects any exceptions that print to stderr but are not raised to caller
with io.StringIO() as buf, contextlib.redirect_stderr(buf):
arduino.StuffObject(servos, values, format_string, format_byte_size)
if not SIMULATE_ARDUINO:
servos.send(format_byte_size)
# if there's an exception, probably the connection failed; clear the connection
# so that it can be reopened at top of loop
if buf.getvalue():
servos = None
time.sleep(COMMAND_DELAY_TIME)
if __name__ == "__main__":
main()
This leads to the following stdio output when I manually unplug the USB cable to arduino and then plug it back in, 2x during the testing.
$ python3 arduino_servo_test.py
SENT (12 byte struct): (0, 85, 0)
SENT (12 byte struct): (0, 90, 0)
SENT (12 byte struct): (0, 85, 0)
SENT (12 byte struct): (0, 90, 0)
SENT (12 byte struct): (0, 85, 0)
SENT (12 byte struct): (0, 90, 0)
SENT (12 byte struct): (0, 85, 0)
SENT (12 byte struct): (0, 90, 0)
SENT (12 byte struct): (0, 85, 0)
SENT (12 byte struct): (0, 90, 0)
SENT (12 byte struct): (0, 85, 0)
SENT (12 byte struct): (0, 90, 0)
SENT (12 byte struct): (0, 85, 0)
SENT (12 byte struct): (0, 90, 0)
Reopening connection
Reopening connection
Reopening connection
Reopening connection
SENT (12 byte struct): (0, 85, 0)
SENT (12 byte struct): (0, 90, 0)
SENT (12 byte struct): (0, 85, 0)
SENT (12 byte struct): (0, 90, 0)
SENT (12 byte struct): (0, 85, 0)
SENT (12 byte struct): (0, 90, 0)
SENT (12 byte struct): (0, 85, 0)
SENT (12 byte struct): (0, 90, 0)
SENT (12 byte struct): (0, 85, 0)
SENT (12 byte struct): (0, 90, 0)
SENT (12 byte struct): (0, 85, 0)
SENT (12 byte struct): (0, 90, 0)
Reopening connection
Reopening connection
Reopening connection
Reopening connection
Reopening connection
Reopening connection
Reopening connection
SENT (12 byte struct): (0, 85, 0)
SENT (12 byte struct): (0, 90, 0)
SENT (12 byte struct): (0, 85, 0)
SENT (12 byte struct): (0, 90, 0)
SENT (12 byte struct): (0, 85, 0)
SENT (12 byte struct): (0, 90, 0)
SENT (12 byte struct): (0, 85, 0)
SENT (12 byte struct): (0, 90, 0)
SENT (12 byte struct): (0, 85, 0)
SENT (12 byte struct): (0, 90, 0)
Problem solved - perhaps not the most elegant solution, but it seems to work.
I wanted to try out the XLA backend on tensorflow 1.1.0 which I built from source to support the XLA compiler. Also I am using Ubuntu 16.04. My model runs fine without the XLA backend. It takes about 0.8 seconds to compute a single training step on my GTX 1080 GPU. However when I enable the XLA compiler it makes it up to the point where I make the first call to session.run in my model and then just hangs there. My CPU and GPU are at about zero usage.
config = tf.ConfigProto()
config.graph_options.optimizer_options.global_jit_level = tf.OptimizerOptions.ON_1
sess = tf.Session(config = config)
m = model.CharacterTranslator(sess, MAX_LENGTH)
m.init_variables()
best_cost = None
m.restore('/media/chase/98d61322-9ea7-473e-b835-8739c77d1e1e/Translator/model.chk')
while True:
#session.run is called inside of here
m.train(random.sample(training_data, 40000), 64, False)
c = m.train(validation_data, 64, True)[0]
if best_cost is None or c < best_cost:
count = 0
best_cost = c
print('Saving...')
m.save('/media/chase/98d61322-9ea7-473e-b835-8739c77d1e1e/Translator/model.chk')
else:
count += 1
if count == 10:
break
...
def train(self, training_data, batch_size, validate = False, verbose = True):
total_cost = 0
total_acc = 0
total_time = 0
last_chars = 0
total_batches = len(training_data) // batch_size
for i, batch in enumerate(_batch(training_data, batch_size, False)):
x, y = zip(*batch)
x, xl = zip(*[self._vectorize_sent(s) for s in x])
y, yl = zip(*[self._vectorize_sent(s) for s in y])
start_time = time.time()
c, a, g, l, _ = self.session.run((self.cost, self.accuracy, self.global_step, self.learning_rate, self.null_train_step if validate else self.train_step), {
self.source_text: x,
self.target_text: y,
self.target_length: yl,
self.teacher_forcing: True,
})
end_time = time.time()
total_cost += c
total_acc += a
total_time += end_time - start_time
if verbose:
msg = '%s b(%d / %d) g(%d) c(%e) a(%0.4f) lr(%e) dt(%0.2f)' % ('Validating' if validate else 'Training', i, total_batches, g, total_cost / (i + 1), total_acc / (i + 1), l, total_time / (i + 1))
msg += ' ' * max(0, last_chars - len(msg))
last_chars = len(msg)
print(msg, end = '\r')
if verbose:
print()
return total_cost / (i + 1), total_acc / (i + 1)
It produces the following tensorflow output when I try to run it.
2017-04-26 05:15:58.421388: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-04-26 05:15:58.421698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties:
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.7335
pciBusID 0000:01:00.0
Total memory: 7.92GiB
Free memory: 7.33GiB
2017-04-26 05:15:58.421708: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0
2017-04-26 05:15:58.421711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0: Y
2017-04-26 05:15:58.421719: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0)
/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gradients_impl.py:93: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
2017-04-26 05:17:17.107616: I tensorflow/compiler/xla/service/platform_util.cc:58] platform CUDA present with 1 visible devices
2017-04-26 05:17:17.107635: I tensorflow/compiler/xla/service/platform_util.cc:58] platform Host present with 8 visible devices
2017-04-26 05:17:17.108265: I tensorflow/compiler/xla/service/service.cc:183] XLA service 0xa103840 executing computations on platform Host. Devices:
2017-04-26 05:17:17.108274: I tensorflow/compiler/xla/service/service.cc:191] StreamExecutor device (0): <undefined>, <undefined>
2017-04-26 05:17:17.108393: I tensorflow/compiler/xla/service/platform_util.cc:58] platform CUDA present with 1 visible devices
2017-04-26 05:17:17.108398: I tensorflow/compiler/xla/service/platform_util.cc:58] platform Host present with 8 visible devices
2017-04-26 05:17:17.108602: I tensorflow/compiler/xla/service/service.cc:183] XLA service 0xe383100 executing computations on platform CUDA. Devices:
2017-04-26 05:17:17.108607: I tensorflow/compiler/xla/service/service.cc:191] StreamExecutor device (0): GeForce GTX 1080, Compute Capability 6.1
I then attached gdb to the process to see what it was doing. It looks like it is just sitting on a pthread conditional wait.
#0 pthread_cond_wait##GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f715569291c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2 0x00007f716d85257b in tensorflow::DirectSession::WaitForNotification(tensorflow::Notification*, long long) ()
from /usr/local/lib/python3.5/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#3 0x00007f716d85262d in tensorflow::DirectSession::WaitForNotification(tensorflow::DirectSession::RunState*, tensorflow::CancellationManager*, long long) ()
from /usr/local/lib/python3.5/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#4 0x00007f716d85d287 in tensorflow::DirectSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*) ()
from /usr/local/lib/python3.5/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#5 0x00007f716c3259d1 in TF_Run_Helper(tensorflow::Session*, char const*, TF_Buffer const*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tensorflow::Tensor> > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, TF_Tensor**, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, TF_Buffer*, TF_Status*) [clone .constprop.554] ()
from /usr/local/lib/python3.5/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#6 0x00007f716c32639a in TF_Run ()
from /usr/local/lib/python3.5/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#7 0x00007f716c0ab351 in tensorflow::TF_Run_wrapper_helper(TF_DeprecatedSession*, char const*, TF_Buffer const*, _object*, tensorflow::gtl::InlinedVector<char const*, 8> const&, tensorflow::gtl::InlinedVector<char const*, 8> const&, TF_Status*, tensorflow::gtl::InlinedVector<_object*, 8>*, TF_Buffer*) ()
Does anyone have any idea why my tensorflow model with the XLA backend gets stuck?
I was wondering if there is a cil code list, I think 16 = false and 17 = true, but not 100% sure, also think -1 = 0 , but if anyone has a website to help with this that would be great.
16 = false
17 = true ?
You can find a complete listing of all opcodes in Partition III of ECMA-335, but as for the specific instructions you listed:
02 (0x02) = ldarg.0
16 (0x10) = starg.s
17 (0x11) = ldloc.s
18 (0x12) = ldloca.s
19 (0x13) = stloc.s
0x16 = ldc.i4.0 // could be a 0 or false or '\0' ... the exact type depends on how it's used.
0x17 = ldc.i4.1 // could be a 1 or true or '\u0001' ... again, the exact type depends on how it's used.
0x18 = ldc.i4.2
0x19 = ldc.i4.3
I get quite a few of the below errors in my Magento exception log. Does anyone know how I would go about fixing this?
2011-12-10T22:13:41+00:00 ERR (3):
exception 'PDOException' with message 'SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')' at line 1' in /home/mysite/public_html/lib/Zend/Db/Statement/Pdo.php:228
Stack trace:
#0 /home/mysite/public_html/lib/Zend/Db/Statement/Pdo.php(228): PDOStatement->execute(Array)
#1 /home/mysite/public_html/lib/Varien/Db/Statement/Pdo/Mysql.php(110): Zend_Db_Statement_Pdo->_execute(Array)
#2 /home/mysite/public_html/lib/Zend/Db/Statement.php(300): Varien_Db_Statement_Pdo_Mysql->_execute(Array)
#3 /home/mysite/public_html/lib/Zend/Db/Adapter/Abstract.php(479): Zend_Db_Statement->execute(Array)
#4 /home/mysite/public_html/lib/Zend/Db/Adapter/Pdo/Abstract.php(238): Zend_Db_Adapter_Abstract->query('SELECT `catalog...', Array)
#5 /home/mysite/public_html/lib/Varien/Db/Adapter/Pdo/Mysql.php(389): Zend_Db_Adapter_Pdo_Abstract->query('SELECT `catalog...', Array)
#6 /home/mysite/public_html/lib/Zend/Db/Adapter/Abstract.php(753): Varien_Db_Adapter_Pdo_Mysql->query(Object(Varien_Db_Select), Array)
#7 /home/mysite/public_html/app/code/core/Mage/Core/Model/Resource/Db/Abstract.php(375): Zend_Db_Adapter_Abstract->fetchRow(Object(Varien_Db_Select))
#8 /home/mysite/public_html/app/code/core/Mage/Core/Model/Abstract.php(225): Mage_Core_Model_Resource_Db_Abstract->load(Object(Mage_Catalog_Model_Category), Object(Mage_Catalog_Model_Category), NULL)
#9 /home/mysite/public_html/app/code/local/Mage/Catalog/Block/Product/Newcustom.php(80): Mage_Core_Model_Abstract->load(Object(Mage_Catalog_Model_Category))
#10 /home/mysite/public_html/app/code/local/Mage/Catalog/Block/Product/Newcustom.php(145): Mage_Catalog_Block_Product_Newcustom->_getProductCollection()
#11 /home/mysite/public_html/app/code/core/Mage/Core/Block/Abstract.php(862): Mage_Catalog_Block_Product_Newcustom->_beforeToHtml()
#12 /home/mysite/public_html/app/code/core/Mage/Core/Model/Email/Template/Filter.php(191): Mage_Core_Block_Abstract->toHtml()
#13 [internal function]: Mage_Core_Model_Email_Template_Filter->blockDirective(Array)
#14 /home/mysite/public_html/lib/Varien/Filter/Template.php(134): call_user_func(Array, Array)
#15 /home/mysite/public_html/app/code/core/Mage/Core/Model/Email/Template/Filter.php(502): Varien_Filter_Template->filter('<h3>New Watches...')
#16 /home/mysite/public_html/app/code/core/Mage/Cms/Block/Block.php(54): Mage_Core_Model_Email_Template_Filter->filter('<h3>New Watches...')
#17 /home/mysite/public_html/app/code/core/Mage/Core/Block/Abstract.php(863): Mage_Cms_Block_Block->_toHtml()
#18 /home/mysite/public_html/app/code/core/Mage/Core/Block/Text/List.php(43): Mage_Core_Block_Abstract->toHtml()
#19 /home/mysite/public_html/app/code/core/Mage/Core/Block/Abstract.php(863): Mage_Core_Block_Text_List->_toHtml()
#20 /home/mysite/public_html/app/code/core/Mage/Core/Block/Abstract.php(582): Mage_Core_Block_Abstract->toHtml()
#21 /home/mysite/public_html/app/code/core/Mage/Core/Block/Abstract.php(526): Mage_Core_Block_Abstract->_getChildHtml('right', true)
#22 /home/mysite/public_html/app/design/frontend/default/twistedtime/template/page/2columns-right.phtml(56): Mage_Core_Block_Abstract->getChildHtml('right')
#23 /home/mysite/public_html/app/code/local/Mage/Core/Block/Template.php(469): include('/home/mysite/...')
#24 /home/mysite/public_html/app/code/local/Mage/Core/Block/Template.php(531): Mage_Core_Block_Template->fetchView('frontend/defaul...')
#25 /home/mysite/public_html/app/code/local/Mage/Core/Block/Template.php(559): Mage_Core_Block_Template->renderView()
#26 /home/mysite/public_html/app/code/core/Mage/Core/Block/Abstract.php(863): Mage_Core_Block_Template->_toHtml()
#27 /home/mysite/public_html/app/code/core/Mage/Core/Model/Layout.php(529): Mage_Core_Block_Abstract->toHtml()
#28 /home/mysite/public_html/app/code/core/Mage/Core/Controller/Varien/Action.php(391): Mage_Core_Model_Layout->getOutput()
#29 /home/mysite/public_html/app/code/core/Mage/Catalog/Helper/Product/View.php(147): Mage_Core_Controller_Varien_Action->renderLayout()
#30 /home/mysite/public_html/app/code/core/Mage/Catalog/controllers/ProductController.php(132): Mage_Catalog_Helper_Product_View->prepareAndRender(27, Object(Mage_Catalog_ProductController), Object(Varien_Object))
#31 /home/mysite/public_html/app/code/core/Mage/Core/Controller/Varien/Action.php(420): Mage_Catalog_ProductController->viewAction()
#32 /home/mysite/public_html/app/code/core/Mage/Core/Controller/Varien/Router/Standard.php(250): Mage_Core_Controller_Varien_Action->dispatch('view')
#33 /home/mysite/public_html/app/code/community/ArtsOnIT/OfflineMaintenance/Controller/Router/Standard.php(46): Mage_Core_Controller_Varien_Router_Standard->match(Object(Mage_Core_Controller_Request_Http))
#34 /home/mysite/public_html/app/code/core/Mage/Core/Controller/Varien/Front.php(176): ArtsOnIT_OfflineMaintenance_Controller_Router_Standard->match(Object(Mage_Core_Controller_Request_Http))
#35 /home/mysite/public_html/app/code/core/Mage/Core/Model/App.php(349): Mage_Core_Controller_Varien_Front->dispatch()
#36 /home/mysite/public_html/app/Mage.php(640): Mage_Core_Model_App->run(Array)
#37 /home/mysite/public_html/index.php(80): Mage::run('', 'store')
#38 {main}
Check filters in your _getProductCollection function. If you don't have any then check init methods of all custom models you have.