I am new to KFServing and Kubeflow.
I was following https://github.com/kubeflow/kfserving/tree/master/docs/samples/v1alpha2/tensorflow to deploy a simple inference service.
However, when looking at the logs, I am unable to find the container storage-initializer. The only containers my predict service pod has are kfserving and queue-proxy.
I am currently on Kubeflow 1.2 and Kubernetes 1.17 on IBM Cloud.
Error Message Image
storage-initializer is an init container, so if you describe the pod you won't find it in the containers section of pod spec but in the initContainers section.
$ kubectl get pod flowers-sample-predictor-default-00002-deployment-58bb9557sf7g2 -o json | jq .status.initContainerStatuses
[
{
"containerID": "docker://e40e5f86401b3715118b873fec4ae6c3ef57765ffbb5c9ab48757234c4f53b6f",
"image": "gcr.io/kfserving/storage-initializer:v0.5.0",
"imageID": "docker-pullable://gcr.io/kfserving/storage-initializer#sha256:1d396c0c50892f5562a1c24d925691ec786e5d48e08200f3f9bb17bb48da40ae",
"lastState": {},
"name": "storage-initializer",
"ready": true,
"restartCount": 0,
"state": {
"terminated": {
"containerID": "docker://e40e5f86401b3715118b873fec4ae6c3ef57765ffbb5c9ab48757234c4f53b6f",
"exitCode": 0,
"finishedAt": "2021-02-27T20:13:25Z",
"reason": "Completed",
"startedAt": "2021-02-27T20:13:11Z"
}
}
}
]
I'm not familiar with the model label you are using, can you retry by using the app label or the pod name directly?
$ kubectl logs -l app=flowers-sample-predictor-default-00002 -c storage-initializer
[I 210227 20:13:12 initializer-entrypoint:13] Initializing, args: src_uri [gs://kfserving-samples/models/tensorflow/flowers] dest_path[ [/mnt/models]
[I 210227 20:13:12 storage:43] Copying contents of gs://kfserving-samples/models/tensorflow/flowers to local
[W 210227 20:13:15 _metadata:104] Compute Engine Metadata server unavailable onattempt 1 of 3. Reason: timed out
[W 210227 20:13:15 _metadata:104] Compute Engine Metadata server unavailable onattempt 2 of 3. Reason: [Errno 113] No route to host
[W 210227 20:13:18 _metadata:104] Compute Engine Metadata server unavailable onattempt 3 of 3. Reason: timed out
[W 210227 20:13:18 _default:250] Authentication failed using Compute Engine authentication due to unavailable metadata server.
[I 210227 20:13:19 storage:127] Downloading: /mnt/models/0001/saved_model.pb
[I 210227 20:13:19 storage:127] Downloading: /mnt/models/0001/variables/variables.data-00000-of-00001
[I 210227 20:13:25 storage:127] Downloading: /mnt/models/0001/variables/variables.index
[I 210227 20:13:25 storage:76] Successfully copied gs://kfserving-samples/models/tensorflow/flowers to /mnt/models
We are currently in the process of setting up a CI/CD pipeline for our SCP CF environment based on Project Piper. After a lot of trial & error, we have made significant process, however we are still not able to run our karma tests.
We understood that we need to use the karma-webdriver-launcher, however we are not able to start/connect to Chrome in the sidecar container.
Relevant logs:
[Pipeline] sh
+ cd ./nodejs
+ npm test
> nodejs#1.0.0 test /var/jenkins_home/workspace/CICD_Multibranch_master/nodejs
> karma start ./karma.conf.js
10 07 2019 14:42:53.897:DEBUG [config]: Loading config /var/jenkins_home/workspace/CICD_Multibranch_master/nodejs/karma.conf.js
10 07 2019 14:42:53.906:DEBUG [karma-server]: Final config [{"LOG_DISABLE":"1","LOG_ERROR":"2","LOG_WARN":"3","LOG_INFO":"4","LOG_DEBUG":"5","frameworks":"6","protocol":"7","port":9876,"listenAddress":"8","hostname":"9","httpsServerConfig":"10","basePath":"11","files":"12","browserConsoleLogOptions":"13","customContextFile":null,"customDebugFile":null,"customClientContextFile":null,"exclude":"14","logLevel":"5","colors":false,"autoWatch":false,"autoWatchBatchDelay":250,"restartOnFileChange":false,"usePolling":true,"reporters":"15","singleRun":true,"browsers":"16","captureTimeout":60000,"proxies":"17","proxyValidateSSL":true,"preprocessors":"18","urlRoot":"19","reportSlowerThan":0,"loggers":"20","transports":"21","forceJSONP":false,"plugins":"22","client":"23","defaultClient":"23","browserDisconnectTimeout":2000,"browserDisconnectTolerance":0,"browserNoActivityTimeout":30000,"processKillTimeout":2000,"concurrency":null,"failOnEmptyTestSuite":true,"retryLimit":2,"detached":false,"crossOriginAttribute":true,"browserSocketTimeout":20000,"cmd":"24","configFile":"25","customLaunchers":"26","junitReporter":"27"},"OFF","ERROR","WARN","INFO","DEBUG",["28"],"http:","0.0.0.0","localhost",{},"/var/jenkins_home/workspace/CICD_Multibranch_master/nodejs",["29","30"],{"level":"31","format":"32","terminal":true},["25"],["33"],["34"],{},{},"/",["35"],["36","37"],["38","39","40","41","42"],{"args":"43","useIframe":true,"runInParent":false,"captureConsole":true,"clearContext":true},"start","/var/jenkins_home/workspace/CICD_Multibranch_master/nodejs/karma.conf.js",{"chromeSel":"44"},{"outputFile":"45","suite":"46"},"qunit",{"pattern":"47","served":true,"included":true,"watched":true,"nocache":false,"weight":"48"},{"pattern":"49","served":true,"included":true,"watched":true,"nocache":false,"weight":"50"},"debug","%b %T: %m","junit","chromeSel",{"type":"51","layout":"52"},"polling","websocket","karma-qunit","karma-junit-reporter","karma-chrome-launcher","karma-webdriver-launcher",{"launcher:chromeSel":"53"},[],{"base":"54","config":"55","browserName":"56"},"test-results-karma.xml","","/var/jenkins_home/workspace/CICD_Multibranch_master/nodejs/server.js",[1,0,0,0,0,0],"/var/jenkins_home/workspace/CICD_Multibranch_master/nodejs/tests/sampleTest.spec.js",[1,0,0,0,0,0],"console",{"type":"57","pattern":"58"},["59",null],"WebDriver",{"hostname":"60","port":4444},"chrome","pattern","%d{DATE}:%p [%c]: %m","factory","selenium"]
10 07 2019 14:42:53.907:DEBUG [plugin]: Loading plugin karma-qunit.
10 07 2019 14:42:53.908:DEBUG [plugin]: Loading plugin karma-junit-reporter.
10 07 2019 14:42:53.923:DEBUG [plugin]: Loading plugin karma-chrome-launcher.
10 07 2019 14:42:53.931:DEBUG [plugin]: Loading plugin karma-webdriver-launcher.
10 07 2019 14:42:54.179:DEBUG [plugin]: Loading inlined plugin (defining launcher:chromeSel).
10 07 2019 14:42:54.193:DEBUG [web-server]: Instantiating middleware
10 07 2019 14:42:54.194:DEBUG [reporter]: Trying to load reporter: junit
10 07 2019 14:42:54.195:DEBUG [reporter]: Trying to load color-version of reporter: junit (junit_color)
10 07 2019 14:42:54.195:DEBUG [reporter]: Couldn't load color-version.
10 07 2019 14:42:54.224:INFO [karma-server]: Karma v4.1.0 server started at http://0.0.0.0:9876/
10 07 2019 14:42:54.225:INFO [launcher]: Launching browsers chromeSel with concurrency unlimited
10 07 2019 14:42:54.231:INFO [launcher]: Starting browser chrome via Remote WebDriver
10 07 2019 14:42:54.232:DEBUG [launcher]: null -> BEING_CAPTURED
10 07 2019 14:42:54.232:DEBUG [temp-dir]: Creating temp dir at /tmp/karma-89562642
10 07 2019 14:42:54.235:DEBUG [WebDriver]: WebDriver config: {"hostname":"selenium","port":4444}
10 07 2019 14:42:54.235:DEBUG [WebDriver]: Browser capabilities: {"platform":"ANY","testName":"Karma test","tags":[],"version":"","base":"WebDriver","browserName":"chrome"}
10 07 2019 14:43:54.281:WARN [launcher]: chrome via Remote WebDriver have not captured in 60000 ms, killing.
10 07 2019 14:43:54.286:DEBUG [launcher]: BEING_CAPTURED -> BEING_KILLED
10 07 2019 14:43:54.355:INFO [WebDriver]: Killed Karma test.
10 07 2019 14:43:54.355:DEBUG [launcher]: Process chrome via Remote WebDriver exited with code -1 and signal timeout
10 07 2019 14:43:54.356:DEBUG [temp-dir]: Cleaning temp dir /tmp/karma-89562642
10 07 2019 14:43:54.358:INFO [launcher]: Trying to start chrome via Remote WebDriver again (1/2).
karma.conf.js:
// Karma configuration
module.exports = function(config) {
var webdriverConfig = {
hostname: 'selenium',
port: 4444
}
config.set({
// base path that will be used to resolve all patterns (eg. files, exclude)
basePath: '',
// frameworks to use
// available frameworks: https://npmjs.org/browse/keyword/karma-adapter
frameworks: ['qunit'],
plugins: ['karma-qunit','karma-junit-reporter','karma-chrome-launcher', 'karma-webdriver-launcher'],
// list of files / patterns to load in the browser
files: [
'server.js',
'tests/sampleTest.spec.js'
],
// list of files to exclude
exclude: [
],
// preprocess matching files before serving them to the browser
// available preprocessors: https://npmjs.org/browse/keyword/karma-preprocessor
preprocessors: {
},
// test results reporter to use
// possible values: 'dots', 'progress'
// available reporters: https://npmjs.org/browse/keyword/karma-reporter
reporters: ['junit'],
// web server port
//port: 9876,
//hostname: localhost,
// enable / disable colors in the output (reporters and logs)
colors: false,
// level of logging
// possible values: config.LOG_DISABLE || config.LOG_ERROR || config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG
logLevel: config.LOG_DEBUG,
// enable / disable watching file and executing tests whenever any file changes
autoWatch: false,
customLaunchers: {
'chromeSel': {
base: 'WebDriver',
config: webdriverConfig,
browserName: 'chrome'
}
},
// start these browsers
// available browser launchers: https://npmjs.org/browse/keyword/karma-launcher
browsers: ['chromeSel'],
// Continuous Integration mode
// if true, Karma captures browsers, runs the tests and exits
singleRun: true,
junitReporter: {
outputFile: 'test-results-karma.xml',
suite: ''
}
});
};
Jenkinsfile:
#!groovy
#Library('piper-lib-os') _
node(){
stage('Prepare') {
deleteDir()
checkout scm
setupCommonPipelineEnvironment
}
stage('Build') {
karmaExecuteTests script: this, modules: ['./nodejs'],
installCommand: "npm install karma karma-qunit karma-junit-reporter karma-chrome-launcher qunit karma-webdriver-launcher",
runCommand: 'npm test'
mtaBuild script:this, buildTarget: 'CF', applicationName: 'appLibrary'
}
stage('Deploy to QA') {
testsPublishResults script: this, junit: [updateResults: true, archive: true]
cloudFoundryDeploy(
script: this,
cloudFoundry: [apiEndpoint: 'https://api.cf.eu10.hana.ondemand.com', manifest: 'manifest.yml', org:'xxx', space:'xxx', credentialsId: 'xxx'],
deployTool: 'mtaDeployPlugin'
)
}
}
Thanks a lot,
Nico
To run the karmaExecuteTests step on your Jenkins, you need a Docker deamon installed. I assume this is the case.
In the Docker sidecar pattern the containers can talk to each other using their container names (karma & selenium). So for the browser (config.customLaunchers.chromeSel.config.hostname) you set the hostname correctly to selenium but you also need to set config.hostname to karma.
I would like to use RabbitMQ to send messages from a webapp backend to a second module. On my laptop, it works, but when I deploy the application on a VPS, even in dev mode, it doesn't work anymore... Could you please help me solve this out?
Current status :
If I check the queues on the VPS where both modules are installed, then, it looks ok (messages are added in the queue)
$ rabbitmqctl list_queues
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
MyMessages 2
When I launch the second module, I get following log :
Waiting for a request on queue : MyMessages, hosted at localhost
Comming from the following java code :
public static void main(String[] args) throws IOException, TimeoutException {
RabbitMQConsumer rabbitMQConsumer = new RabbitMQConsumer();
rabbitMQConsumer.waitForRequests();
System.out.println("Waiting for a request on queue : " + AppConfig.QUEUE_NAME + ", hosted at " + AppConfig.QUEUE_HOST);
}
public RabbitMQConsumer() throws IOException, TimeoutException {
mapper = new ObjectMapper();
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost(AppConfig.QUEUE_HOST);
Connection connection = connectionFactory.newConnection();
channel = connection.createChannel();
}
public void waitForRequests() throws IOException {
DefaultConsumer consumer = new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
try {
System.out.println("Message received ! ");
channel.basicAck(envelope.getDeliveryTag(), false);
} catch (Exception e) {
e.printStackTrace();
}
}
};
channel.queueDeclare(AppConfig.QUEUE_NAME, true, false, false, null);
channel.basicConsume(AppConfig.QUEUE_NAME, consumer);
}
I think both modules are looking at the same queue, there are messages in the quue, so... to me, it looks like messages are not consummed... I've looked at the status of rabbitMQ, but I do not know how to use it :
$ invoke-rc.d rabbitmq-server status
● rabbitmq-server.service - RabbitMQ broker
Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2018-04-07 18:24:59 CEST; 1h 38min ago
Process: 17103 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl shutdown (code=exited, status=0/SUCCESS)
Main PID: 17232 (beam.smp)
Status: "Initialized"
Tasks: 84 (limit: 4915)
CGroup: /system.slice/rabbitmq-server.service
├─17232 /usr/lib/erlang/erts-9.3/bin/beam.smp -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 1280000 -K true -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/r
abbitmq/lib/rabbitmq_server-3.7.4/ebin -noshell -noinput -s rabbit boot -sname rabbit#vps5322 -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_err
or_logger false -rabbit lager_log_root "/var/log/rabbitmq" -rabbit lager_default_file "/var/log/rabbitmq/rabbit#vps5322.log" -rabbit lager_upgrade_file "/var/log/rabbitmq/rabbit#vps5322_upgrade.log" -r
abbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.7.4/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/
mnesia/rabbit#vps5322-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit#vps5322" -kernel inet_dist_listen_m
in 25672 -kernel inet_dist_listen_max 25672
├─17319 /usr/lib/erlang/erts-9.3/bin/epmd -daemon
├─17453 erl_child_setup 1024
├─17475 inet_gethost 4
└─17476 inet_gethost 4
Apr 07 18:24:57 vps5322 rabbitmq-server[17232]: ## ##
Apr 07 18:24:57 vps5322 rabbitmq-server[17232]: ## ## RabbitMQ 3.7.4. Copyright (C) 2007-2018 Pivotal Software, Inc.
Apr 07 18:24:57 vps5322 rabbitmq-server[17232]: ########## Licensed under the MPL. See http://www.rabbitmq.com/
Apr 07 18:24:57 vps5322 rabbitmq-server[17232]: ###### ##
Apr 07 18:24:57 vps5322 rabbitmq-server[17232]: ########## Logs: /var/log/rabbitmq/rabbit#vps5322.log
Apr 07 18:24:57 vps5322 rabbitmq-server[17232]: /var/log/rabbitmq/rabbit#vps5322_upgrade.log
Apr 07 18:24:57 vps5322 rabbitmq-server[17232]: Starting broker...
Apr 07 18:24:59 vps5322 rabbitmq-server[17232]: systemd unit for activation check: "rabbitmq-server.service"
Apr 07 18:24:59 vps5322 systemd[1]: Started RabbitMQ broker.
Apr 07 18:24:59 vps5322 rabbitmq-server[17232]: completed with 0 plugins.
Finally, note that the webapp application is a PlayFramework app, with these dependencies :
libraryDependencies ++= Seq(
guice,
"com.rabbitmq" % "amqp-client" % "5.2.0"
)
Whereas the second module is a pure java code, based on maven, with the following pom :
<dependency>
<groupId>com.rabbitmq</groupId>
<artifactId>amqp-client</artifactId>
<version>5.2.0</version>
</dependency>
Any idea of the problem?
Thank you very much !!
Finally I've found the problem. This configuration is actually working, but I could not see it because of a crash in my own app that was not logged because of an error in my log4J configuration.
Just in case, the error I had was that a local library included in my pom with a relative path (${project.basedir}) was found by my IDE but not anymore once deployed on a VPS. To solve this, I've just moved this (hopefully) very small library directly into my project. After solving this issue, I had to reset rabbitMQ and then it was all fine :
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
Thank you very much,
Regards,
I am using sails#beta.
I am trying to create several Room models (see definition) below, the problem is that I can successfully create Room models with the same attribute 'name', although attribute 'name' has a unique validation.
The validation isn't detected until the next restart of 'sails' server, then I get this output:
C:\eMali_dev\server>sails lift
info: Starting app...
Express midleware for passport
Waterline encountered a fatal error when trying to perform the `alter` auto-migration strategy.
In a couple of seconds, the data (cached in memory) will be logged to stdout.
(a failsafe put in place to preserve development data)
In the mean time, here's the error:
Error (E_UNKNOWN) :: Encountered an unexpected error:
MongoError: E11000 duplicate key error index: eMali_beta_dev.room.$name_1 dup key: { : "e1" }
Details:
{ error: 'E_UNKNOWN',
summary: 'Encountered an unexpected error',
status: 500,
raw: 'MongoError: E11000 duplicate key error index: eMali_beta_dev.room.$name_1 dup key: { : "e1" }' }
================================
Data backup:
================================
[ { name: 'e1',
center: '53471e2318b931dc1b69f3a8',
min_age: 1,
max_age: 6,
createdAt: Fri Apr 11 2014 00:41:39 GMT+0200 (W. Europe Summer Time),
updatedAt: Fri Apr 11 2014 00:41:39 GMT+0200 (W. Europe Summer Time),
_id: { _bsontype: 'ObjectID', id: 'SG\u001e#\u001811Ü\u001bióc' } },
{ name: 'e2',
center: '53471e2318b931dc1b69f3a8',
min_age: 1,
max_age: 7,
createdAt: Fri Apr 11 2014 00:41:39 GMT+0200 (W. Europe Summer Time),
updatedAt: Fri Apr 11 2014 00:41:39 GMT+0200 (W. Europe Summer Time),
_id: { _bsontype: 'ObjectID', id: 'SG\u001e#\u001811Ü\u001bióª' } },
{ name: 'e1',
center: '53471e2318b931dc1b69f3a8',
min_age: 1,
max_age: 6,
createdAt: Fri Apr 11 2014 00:41:39 GMT+0200 (W. Europe Summer Time),
updatedAt: Fri Apr 11 2014 00:41:39 GMT+0200 (W. Europe Summer Time),
_id: { _bsontype: 'ObjectID', id: 'SG\u001e#\u001811Ü\u001bió«' } } ]
error: A hook (`orm`) failed to load!
Room model:
module.exports = {
schema: true,
attributes: {
name: {
type: 'string',
required: true,
unique: true,
minLength: 3
},
center: {
model: 'center'
},
min_age: {
type: 'integer',
required: true
},
max_age: {
type: 'integer',
required: true
}
}
}
This was a bug in sails and it was fixed. Thanks everybody
You need to change the database from default Disk database to a database of your choice.
I was facing a similar problem and changing database from "Disk" to "MongoDB" fixed my issue.
It seems the default database is unable to enforce unique constraint.
I hope it solves your issue.
I met a problem while creating
"New Domain" iso/nfs storage,it prints "Error while executing action
New NFS Storage Domain: Storage domain remote path not mounted"
and the error code is 477.
I followed the "http://wiki.ovirt.org/wiki/Troubleshooting_NFS_Storage_Issues" and find that the vdsm user can't now use mount.
"mount: only root can do that"
the version I use:
oVirt Engine Version: 3.1.0-2.fc17
oVirt Node Hypervisor 2.5.4-0.1.fc17
the error log:
2012-11-08 09:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) Checking autorecoverable storage domains done
2012-11-08 09:17:28,920 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp--0.0.0.0-8009-2) calling GetConfigurationValueQuery (StorageDomainNameSizeLimit) with null version,
using default general for version
2012-11-08 09:17:29,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp--0.0.0.0-8009-3) [7720b88f] START, ValidateStorageServerConnectionVDSCommand(vdsId = 12bcf124-29a4-11e2-bcba-00505680002a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 6556c55d-42a4-4dcc-832c-4d8987ebe6bd, connection: 200.200.101.219:/usr/lwq/iso };]), log id: 52777a80
2012-11-08 09:17:29,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp--0.0.0.0-8009-3) [7720b88f] FINISH, ValidateStorageServerConnectionVDSCommand, return: {6556c55d-42a4-4dcc-832c-4d8987ebe6bd=0}, log id: 52777a80
2012-11-08 09:17:29,392 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--0.0.0.0-8009-3) [7720b88f] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2012-11-08 09:17:29,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--0.0.0.0-8009-3) [7720b88f] START, ConnectStorageServerVDSCommand(vdsId = 12bcf124-29a4-11e2-bcba-00505680002a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 6556c55d-42a4-4dcc-832c-4d8987ebe6bd, connection: 200.200.101.219:/usr/lwq/iso };]), log id: 36cb94f
2012-11-08 09:17:29,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--0.0.0.0-8009-3) [7720b88f] FINISH, ConnectStorageServerVDSCommand, return: {6556c55d-42a4-4dcc-832c-4d8987ebe6bd=477}, log id: 36cb94f
2012-11-08 09:17:29,658 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (ajp--0.0.0.0-8009-3) [7720b88f] The connection with details 200.200.101.219:/usr/lwq/iso failed because of
error code 477 and error message is: 477
2012-11-08 09:17:29,717 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp--0.0.0.0-8009-11) [1661aa36] Running command: AddNFSStorageDomainCommand internal: false. En
tities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2012-11-08 09:17:29,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp--0.0.0.0-8009-11) [1661aa36] START, CreateStorageDomainVDSCommand(vdsId = 12bcf12
4-29a4-11e2-bcba-00505680002a, storageDomain=org.ovirt.engine.core.common.businessentities.storage_domain_static#4a900545, args=200.200.101.219:/usr/lwq/iso), log id: 50b803a0
2012-11-08 09:17:35,233 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--0.0.0.0-8009-11) [1661aa36] Failed in CreateStorageDomainVDS method
2012-11-08 09:17:35,234 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--0.0.0.0-8009-11) [1661aa36] Error code StorageDomainFSNotMounted and error message VDSGeneri
cException: VDSErrorException: Failed to CreateStorageDomainVDS, error = Storage domain remote path not mounted: ('/rhev/data-center/mnt/200.200.101.219:_usr_lwq_iso',)
2012-11-08 09:17:35,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--0.0.0.0-8009-11) [1661aa36] Command org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageD
omainVDSCommand return value
As far as I know VDSM does sudo to run privileged commands, but still that mount error is strange. Can you send share the vdsm log?
Also, you may get more attention on the engine-users or vdsm mailing lists.