I wonder if it is possible to make the LLVM opt tool to report in detail which optimization passes are successfully used during each optimization level like -O1,-O2, etc.
For example, here is the naive bitcode of the file foo.bc:
define dso_local i32 #main() #0 {
%1 = alloca i32, align 4
%2 = alloca i32, align 4
%3 = alloca i32, align 4
store i32 0, i32* %1, align 4
store i32 1, i32* %2, align 4
store i32 1, i32* %3, align 4
%4 = load i32, i32* %2, align 4
ret i32 %4
}
And here is the bitcode obtained after running opt -O1 foo.bc -o foo.opt.bc
; Function Attrs: noinline norecurse nounwind readnone uwtable
define dso_local i32 #main() local_unnamed_addr #0 {
ret i32 1
}
How can I obtain information about passes applied by LLVM opt in the above optimization?
I tried to use several flags of opt of LLVM 10.0, such as --pass-remarks-filter, --debugify-each, --verify-each, --lto-pass-remarks-filter, etc but none of them work.
Does anyone know how to get such kind of information?
See the -debug-pass family of options:
opt --help-list-hidden | grep debug-pass -A 5
--debug-pass=<value> - Print PassManager debugging information
=Disabled - disable debug output
=Arguments - print pass arguments to pass to 'opt'
=Structure - print pass structure before run()
=Executions - print pass name before it is executed
=Details - print pass details when it is executed
Examples:
=Arguments
$ opt -O1 foo.bc -debug-pass=Arguments -o /dev/null
Pass Arguments: -tti -tbaa -scoped-noalias -assumption-cache-tracker -targetlibinfo -verify -ee-instrument -simplifycfg -domtree -sroa -early-cse -lower-expect
Pass Arguments: -targetlibinfo -tti -targetpassconfig -tbaa -scoped-noalias -assumption-cache-tracker -profile-summary-info -forceattrs -inferattrs -ipsccp -called-value-propagation -attributor -globalopt -domtree -mem2reg -deadargelim -domtree -basicaa -aa -loops -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -instcombine -simplifycfg -basiccg -globals-aa -prune-eh -always-inline -functionattrs -domtree -sroa -basicaa -aa -memoryssa -early-cse-memssa -simplifycfg -domtree -basicaa -aa -loops -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -instcombine -libcalls-shrinkwrap -loops -branch-prob -block-freq -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -pgo-memop-opt -simplifycfg -reassociate -domtree -loops -loop-simplify -lcssa-verification -lcssa -basicaa -aa -scalar-evolution -loop-rotate -memoryssa -licm -loop-unswitch -simplifycfg -domtree -basicaa -aa -loops -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -instcombine -loop-simplify -lcssa-verification -lcssa -scalar-evolution -indvars -loop-idiom -loop-deletion -loop-unroll -phi-values -memdep -memcpyopt -sccp -demanded-bits -bdce -basicaa -aa -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -instcombine -postdomtree -adce -simplifycfg -domtree -basicaa -aa -loops -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -instcombine -barrier -basiccg -rpo-functionattrs -globalopt -globaldce -basiccg -globals-aa -domtree -float2int -lower-constant-intrinsics -domtree -loops -loop-simplify -lcssa-verification -lcssa -basicaa -aa -scalar-evolution -loop-rotate -loop-accesses -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -loop-distribute -branch-prob -block-freq -scalar-evolution -basicaa -aa -loop-accesses -demanded-bits -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -loop-vectorize -loop-simplify -scalar-evolution -aa -loop-accesses -lazy-branch-prob -lazy-block-freq -loop-load-elim -basicaa -aa -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -instcombine -simplifycfg -domtree -basicaa -aa -loops -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -instcombine -loop-simplify -lcssa-verification -lcssa -scalar-evolution -loop-unroll -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -instcombine -memoryssa -loop-simplify -lcssa-verification -lcssa -scalar-evolution -licm -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -transform-warning -alignment-from-assumptions -strip-dead-prototypes -domtree -loops -branch-prob -block-freq -loop-simplify -lcssa-verification -lcssa -basicaa -aa -scalar-evolution -block-freq -loop-sink -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -instsimplify -div-rem-pairs -simplifycfg -verify -write-bitcode
Pass Arguments: -domtree
Pass Arguments: -targetlibinfo -domtree -loops -branch-prob -block-freq
Pass Arguments: -targetlibinfo -domtree -loops -branch-prob -block-freq
=Structure
$ opt -O1 foo.bc -debug-pass=Structure -o /dev/null
Pass Arguments: -tti -tbaa -scoped-noalias -assumption-cache-tracker -targetlibinfo -verify -ee-instrument -simplifycfg -domtree -sroa -early-cse -lower-expect
Target Transform Information
Type-Based Alias Analysis
Scoped NoAlias Alias Analysis
Assumption Cache Tracker
Target Library Information
FunctionPass Manager
Module Verifier
Instrument function entry/exit with calls to e.g. mcount() (pre inlining)
Simplify the CFG
Dominator Tree Construction
SROA
Early CSE
Lower 'expect' Intrinsics
Pass Arguments: -targetlibinfo -tti -targetpassconfig -tbaa -scoped-noalias -assumption-cache-tracker -profile-summary-info -forceattrs -inferattrs -ipsccp -called-value-propagation -attributor -globalopt -domtree -mem2reg -deadargelim -domtree -basicaa -aa -loops -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -instcombine -simplifycfg -basiccg -globals-aa -prune-eh -always-inline -functionattrs -domtree -sroa -basicaa -aa -memoryssa -early-cse-memssa -simplifycfg -domtree -basicaa -aa -loops -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -instcombine -libcalls-shrinkwrap -loops -branch-prob -block-freq -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -pgo-memop-opt -simplifycfg -reassociate -domtree -loops -loop-simplify -lcssa-verification -lcssa -basicaa -aa -scalar-evolution -loop-rotate -memoryssa -licm -loop-unswitch -simplifycfg -domtree -basicaa -aa -loops -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -instcombine -loop-simplify -lcssa-verification -lcssa -scalar-evolution -indvars -loop-idiom -loop-deletion -loop-unroll -phi-values -memdep -memcpyopt -sccp -demanded-bits -bdce -basicaa -aa -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -instcombine -postdomtree -adce -simplifycfg -domtree -basicaa -aa -loops -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -instcombine -barrier -basiccg -rpo-functionattrs -globalopt -globaldce -basiccg -globals-aa -domtree -float2int -lower-constant-intrinsics -domtree -loops -loop-simplify -lcssa-verification -lcssa -basicaa -aa -scalar-evolution -loop-rotate -loop-accesses -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -loop-distribute -branch-prob -block-freq -scalar-evolution -basicaa -aa -loop-accesses -demanded-bits -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -loop-vectorize -loop-simplify -scalar-evolution -aa -loop-accesses -lazy-branch-prob -lazy-block-freq -loop-load-elim -basicaa -aa -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -instcombine -simplifycfg -domtree -basicaa -aa -loops -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -instcombine -loop-simplify -lcssa-verification -lcssa -scalar-evolution -loop-unroll -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -instcombine -memoryssa -loop-simplify -lcssa-verification -lcssa -scalar-evolution -licm -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -transform-warning -alignment-from-assumptions -strip-dead-prototypes -domtree -loops -branch-prob -block-freq -loop-simplify -lcssa-verification -lcssa -basicaa -aa -scalar-evolution -block-freq -loop-sink -lazy-branch-prob -lazy-block-freq -opt-remark-emitter -instsimplify -div-rem-pairs -simplifycfg -verify -write-bitcode
Target Library Information
Target Transform Information
Target Pass Configuration
Type-Based Alias Analysis
Scoped NoAlias Alias Analysis
Assumption Cache Tracker
Profile summary info
ModulePass Manager
Force set function attributes
Infer set function attributes
Interprocedural Sparse Conditional Constant Propagation
FunctionPass Manager
Dominator Tree Construction
Called Value Propagation
Deduce and propagate attributes
Global Variable Optimizer
FunctionPass Manager
Dominator Tree Construction
Natural Loop Information
Branch Probability Analysis
Block Frequency Analysis
FunctionPass Manager
Dominator Tree Construction
Promote Memory to Register
Dead Argument Elimination
FunctionPass Manager
Dominator Tree Construction
Basic Alias Analysis (stateless AA impl)
Function Alias Analysis Results
Natural Loop Information
Lazy Branch Probability Analysis
Lazy Block Frequency Analysis
Optimization Remark Emitter
Combine redundant instructions
Simplify the CFG
CallGraph Construction
Globals Alias Analysis
Call Graph SCC Pass Manager
Remove unused exception handling info
Inliner for always_inline functions
Deduce function attributes
FunctionPass Manager
Dominator Tree Construction
SROA
Basic Alias Analysis (stateless AA impl)
Function Alias Analysis Results
Memory SSA
Early CSE w/ MemorySSA
Simplify the CFG
Dominator Tree Construction
Basic Alias Analysis (stateless AA impl)
Function Alias Analysis Results
Natural Loop Information
Lazy Branch Probability Analysis
Lazy Block Frequency Analysis
Optimization Remark Emitter
Combine redundant instructions
Conditionally eliminate dead library calls
Natural Loop Information
Branch Probability Analysis
Block Frequency Analysis
Lazy Branch Probability Analysis
Lazy Block Frequency Analysis
Optimization Remark Emitter
PGOMemOPSize
Simplify the CFG
Reassociate expressions
Dominator Tree Construction
Natural Loop Information
Canonicalize natural loops
LCSSA Verifier
Loop-Closed SSA Form Pass
Basic Alias Analysis (stateless AA impl)
Function Alias Analysis Results
Scalar Evolution Analysis
Loop Pass Manager
Rotate Loops
Memory SSA
Loop Pass Manager
Loop Invariant Code Motion
Unswitch loops
Simplify the CFG
Dominator Tree Construction
Basic Alias Analysis (stateless AA impl)
Function Alias Analysis Results
Natural Loop Information
Lazy Branch Probability Analysis
Lazy Block Frequency Analysis
Optimization Remark Emitter
Combine redundant instructions
Canonicalize natural loops
LCSSA Verifier
Loop-Closed SSA Form Pass
Scalar Evolution Analysis
Loop Pass Manager
Induction Variable Simplification
Recognize loop idioms
Delete dead loops
Unroll loops
Phi Values Analysis
Memory Dependence Analysis
MemCpy Optimization
Sparse Conditional Constant Propagation
Demanded bits analysis
Bit-Tracking Dead Code Elimination
Basic Alias Analysis (stateless AA impl)
Function Alias Analysis Results
Lazy Branch Probability Analysis
Lazy Block Frequency Analysis
Optimization Remark Emitter
Combine redundant instructions
Post-Dominator Tree Construction
Aggressive Dead Code Elimination
Simplify the CFG
Dominator Tree Construction
Basic Alias Analysis (stateless AA impl)
Function Alias Analysis Results
Natural Loop Information
Lazy Branch Probability Analysis
Lazy Block Frequency Analysis
Optimization Remark Emitter
Combine redundant instructions
A No-Op Barrier Pass
CallGraph Construction
Deduce function attributes in RPO
Global Variable Optimizer
FunctionPass Manager
Dominator Tree Construction
Natural Loop Information
Branch Probability Analysis
Block Frequency Analysis
Dead Global Elimination
CallGraph Construction
Globals Alias Analysis
FunctionPass Manager
Dominator Tree Construction
Float to int
Lower constant intrinsics
Dominator Tree Construction
Natural Loop Information
Canonicalize natural loops
LCSSA Verifier
Loop-Closed SSA Form Pass
Basic Alias Analysis (stateless AA impl)
Function Alias Analysis Results
Scalar Evolution Analysis
Loop Pass Manager
Rotate Loops
Loop Access Analysis
Lazy Branch Probability Analysis
Lazy Block Frequency Analysis
Optimization Remark Emitter
Loop Distribution
Branch Probability Analysis
Block Frequency Analysis
Scalar Evolution Analysis
Basic Alias Analysis (stateless AA impl)
Function Alias Analysis Results
Loop Access Analysis
Demanded bits analysis
Lazy Branch Probability Analysis
Lazy Block Frequency Analysis
Optimization Remark Emitter
Loop Vectorization
Canonicalize natural loops
Scalar Evolution Analysis
Function Alias Analysis Results
Loop Access Analysis
Lazy Branch Probability Analysis
Lazy Block Frequency Analysis
Loop Load Elimination
Basic Alias Analysis (stateless AA impl)
Function Alias Analysis Results
Lazy Branch Probability Analysis
Lazy Block Frequency Analysis
Optimization Remark Emitter
Combine redundant instructions
Simplify the CFG
Dominator Tree Construction
Basic Alias Analysis (stateless AA impl)
Function Alias Analysis Results
Natural Loop Information
Lazy Branch Probability Analysis
Lazy Block Frequency Analysis
Optimization Remark Emitter
Combine redundant instructions
Canonicalize natural loops
LCSSA Verifier
Loop-Closed SSA Form Pass
Scalar Evolution Analysis
Loop Pass Manager
Unroll loops
Lazy Branch Probability Analysis
Lazy Block Frequency Analysis
Optimization Remark Emitter
Combine redundant instructions
Memory SSA
Canonicalize natural loops
LCSSA Verifier
Loop-Closed SSA Form Pass
Scalar Evolution Analysis
Loop Pass Manager
Loop Invariant Code Motion
Lazy Branch Probability Analysis
Lazy Block Frequency Analysis
Optimization Remark Emitter
Warn about non-applied transformations
Alignment from assumptions
Strip Unused Function Prototypes
FunctionPass Manager
Dominator Tree Construction
Natural Loop Information
Branch Probability Analysis
Block Frequency Analysis
Canonicalize natural loops
LCSSA Verifier
Loop-Closed SSA Form Pass
Basic Alias Analysis (stateless AA impl)
Function Alias Analysis Results
Scalar Evolution Analysis
Block Frequency Analysis
Loop Pass Manager
Loop Sink
Lazy Branch Probability Analysis
Lazy Block Frequency Analysis
Optimization Remark Emitter
Remove redundant instructions
Hoist/decompose integer division and remainder
Simplify the CFG
Module Verifier
Bitcode Writer
Pass Arguments: -domtree
FunctionPass Manager
Dominator Tree Construction
Pass Arguments: -targetlibinfo -domtree -loops -branch-prob -block-freq
Target Library Information
FunctionPass Manager
Dominator Tree Construction
Natural Loop Information
Branch Probability Analysis
Block Frequency Analysis
Pass Arguments: -targetlibinfo -domtree -loops -branch-prob -block-freq
Target Library Information
FunctionPass Manager
Dominator Tree Construction
Natural Loop Information
Branch Probability Analysis
Block Frequency Analysis
=Executions
$ opt -O1 foo.bc -debug-pass=Executions -o /dev/null
… (includes previous levels)
[2020-07-15 12:09:58.850749079] 0x559a227eda10 Executing Pass 'Force set function attributes' on Module 'foo.bc'...
[2020-07-15 12:09:58.850794318] 0x559a227eda10 Freeing Pass 'Force set function attributes' on Module 'foo.bc'...
[2020-07-15 12:09:58.850834995] 0x559a227eda10 Executing Pass 'Infer set function attributes' on Module 'foo.bc'...
[2020-07-15 12:09:58.850877016] 0x559a227eda10 Freeing Pass 'Infer set function attributes' on Module 'foo.bc'...
[2020-07-15 12:09:58.850917851] 0x559a227eda10 Executing Pass 'Interprocedural Sparse Conditional Constant Propagation' on Module 'foo.bc'...
[2020-07-15 12:09:58.850967874] 0x559a227cb5c0 Executing Pass 'Dominator Tree Construction' on Function 'main'...
[2020-07-15 12:09:58.851024434] 0x559a227cb5c0 Executing Pass 'Dominator Tree Construction' on Function 'f'...
[2020-07-15 12:09:58.851078626] 0x559a227eda10 Freeing Pass 'Interprocedural Sparse Conditional Constant Propagation' on Module 'foo.bc'...
[2020-07-15 12:09:58.851118672] 0x559a227eda10 Executing Pass 'Called Value Propagation' on Module 'foo.bc'...
[2020-07-15 12:09:58.851176380] 0x559a227eda10 Freeing Pass 'Called Value Propagation' on Module 'foo.bc'...
[2020-07-15 12:09:58.851195948] 0x559a227eda10 Executing Pass 'Deduce and propagate attributes' on Module 'foo.bc'...
[2020-07-15 12:09:58.851222525] 0x559a227eda10 Freeing Pass 'Deduce and propagate attributes' on Module 'foo.bc'...
[2020-07-15 12:09:58.851252769] 0x559a227eda10 Executing Pass 'Global Variable Optimizer' on Module 'foo.bc'...
[2020-07-15 12:09:58.851285163] 0x559a227eda10 Made Modification 'Global Variable Optimizer' on Module 'foo.bc'...
[2020-07-15 12:09:58.851306058] 0x559a227eda10 Freeing Pass 'Global Variable Optimizer' on Module 'foo.bc'...
[2020-07-15 12:09:58.851332678] 0x559a227eda10 Executing Pass 'Function Pass Manager' on Module 'foo.bc'...
[2020-07-15 12:09:58.851361051] 0x559a2281c6c0 Executing Pass 'Dominator Tree Construction' on Function 'main'...
[2020-07-15 12:09:58.851390305] 0x559a2281c6c0 Executing Pass 'Promote Memory to Register' on Function 'main'...
[2020-07-15 12:09:58.851419137] 0x559a2281c6c0 Freeing Pass 'Dominator Tree Construction' on Function 'main'...
…
=Details
$ opt -O1 foo.bc -debug-pass=Details -o /dev/null
… (includes Structure level)
[2020-07-15 12:12:50.727901681] 0x55963d24ca10 Executing Pass 'Force set function attributes' on Module 'foo.bc'...
-*- 'Force set function attributes' is the last user of following pass instances. Free these instances
[2020-07-15 12:12:50.727919514] 0x55963d24ca10 Freeing Pass 'Force set function attributes' on Module 'foo.bc'...
[2020-07-15 12:12:50.727930575] 0x55963d24ca10 Executing Pass 'Infer set function attributes' on Module 'foo.bc'...
0x55963d229c20 Required Analyses: Target Library Information
-*- 'Infer set function attributes' is the last user of following pass instances. Free these instances
[2020-07-15 12:12:50.727951877] 0x55963d24ca10 Freeing Pass 'Infer set function attributes' on Module 'foo.bc'...
[2020-07-15 12:12:50.727962569] 0x55963d24ca10 Executing Pass 'Interprocedural Sparse Conditional Constant Propagation' on Module 'foo.bc'...
0x55963d229c80 Required Analyses: Assumption Cache Tracker, Dominator Tree Construction, Target Library Information
[2020-07-15 12:12:50.727991878] 0x55963d22aa00 Executing Pass 'Dominator Tree Construction' on Function 'main'...
[2020-07-15 12:12:50.728019459] 0x55963d22aa00 Executing Pass 'Dominator Tree Construction' on Function 'f'...
-*- 'Interprocedural Sparse Conditional Constant Propagation' is the last user of following pass instances. Free these instances
[2020-07-15 12:12:50.728048312] 0x55963d24ca10 Freeing Pass 'Interprocedural Sparse Conditional Constant Propagation' on Module 'foo.bc'...
[2020-07-15 12:12:50.728059628] 0x55963d24ca10 Executing Pass 'Called Value Propagation' on Module 'foo.bc'...
-*- 'Called Value Propagation' is the last user of following pass instances. Free these instances
[2020-07-15 12:12:50.728087385] 0x55963d24ca10 Freeing Pass 'Called Value Propagation' on Module 'foo.bc'...
[2020-07-15 12:12:50.728098125] 0x55963d24ca10 Executing Pass 'Deduce and propagate attributes' on Module 'foo.bc'...
0x55963d22c270 Required Analyses: Target Library Information
-*- 'Deduce and propagate attributes' is the last user of following pass instances. Free these instances
[2020-07-15 12:12:50.728123960] 0x55963d24ca10 Freeing Pass 'Deduce and propagate attributes' on Module 'foo.bc'...
[2020-07-15 12:12:50.728134651] 0x55963d24ca10 Executing Pass 'Global Variable Optimizer' on Module 'foo.bc'...
0x55963d22c2d0 Required Analyses: Target Library Information, Target Transform Information, Dominator Tree Construction, Block Frequency Analysis
[2020-07-15 12:12:50.728178301] 0x55963d24ca10 Made Modification 'Global Variable Optimizer' on Module 'foo.bc'...
-*- 'Global Variable Optimizer' is the last user of following pass instances. Free these instances
[2020-07-15 12:12:50.728193095] 0x55963d24ca10 Freeing Pass 'Global Variable Optimizer' on Module 'foo.bc'...
[2020-07-15 12:12:50.728203671] 0x55963d24ca10 Executing Pass 'Function Pass Manager' on Module 'foo.bc'...
[2020-07-15 12:12:50.728214720] 0x55963d27bb00 Executing Pass 'Dominator Tree Construction' on Function 'main'...
[2020-07-15 12:12:50.728227201] 0x55963d27bb00 Executing Pass 'Promote Memory to Register' on Function 'main'...
0x55963d27b620 Required Analyses: Assumption Cache Tracker, Dominator Tree Construction
…
The -time-passes option gives you that (and the time taken per pass).
This option is available in many LLVM tools, but is not listed in the manual, as it intended as a debugging tool for compiler/pass authors.
Since the release of LLVM-13, the new pass manager is turned on by default.
It seems one of suggested options, -debug-pass=<value> is not longer working.
There is another option -debug-pass-manager which could be used to list all running passes.
I followed the following steps：
1.CityScapes data set preparation
2.Generate TFRecords of CityScapes
3.Download the pre-training model
4.Run official instruction
python deeplab/train.py \
--logtostderr \
--training_number_of_steps=1000 \
--train_split="train" \
--model_variant="xception_65" \
--atrous_rates=6 \
--atrous_rates=12 \
--atrous_rates=18 \
--output_stride=16 \
--decoder_output_stride=4 \
--train_crop_size=513 \
--train_crop_size=513 \
--train_batch_size=1 \
--dataset="cityscapes" \
--tf_initial_checkpoint='/root/newP/official_tf/models-master/research/deeplab/backbone/deeplabv3_cityscapes_train/model.ckpt' \
--train_logdir='/root/newP/official_tf/models-master/research/deeplab/exp/train_on_train_set/train' \
--dataset_dir='/root/dataset/cityscapesScripts/tfrecord'
An error occurred while training deeplabv3++ using CityScapes dataset
“data split name train not recognized”.
I found the problem after debugging: "train" no longer exit in
"_CITYSCAPES_INFORMATION.splits_to_sizes".
Content in the code：
_CITYSCAPES_INFORMATION = DatasetDescriptor(
splits_to_sizes={'train_fine': 2975,
'train_coarse': 22973,
'trainval_fine': 3475,
'trainval_coarse': 23473,
'val_fine': 500,
'test_fine': 1525},
num_classes=19,
ignore_label=255,
)
I tried several others "train_fine","train_coarse".A new error occurred:
"Total size of new array must be unchanged for image_pooling/weights lh_shape: [(1, 1, 2048, 256)], rh_shape: [(1, 1, 320, 256)]".
May I ask what modifications I should do?
I found that the latest version of the pretraining model had a problem, and I could run it directly when I was not using the pretraining model.
[https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md][1]
Following this answer, I'm trying to add the sqlite (sqlite3) Qt5 plugin I forgot to enable during the last Yocto build. Here what I did:
Under my own custom layer (meta-custom-layer/recipes-core) I added a file qtbase_%.bbappend.
Inside I put:
PACKAGECONFIG_append = " sql-sqlite"
PACKAGECONFIG[sql-sqlite] = "-sql-sqlite,-no-sql-sqlite,sqlite3"
Then I deleted the tmp folder and issued bitbake qtbase. I didn't removed the sstate-cache because I added something rather removed or changed.
After parsing the recipes it successfully rebuilt the tmp folder but I cannot find anything related to the requested plugin (it should be libqsqlite.so).
I didn't understand the answer provided in the link above?
What is the right method to add this plugin?
UPDATE
To be usre there's nothing else to tune, here the contents of the image bb file:
SUMMARY = "blabla"
LICENSE = "Proprietary"
include recipes-st/images/st-image.inc
inherit core-image distro_features_check
CONFLICT_DISTRO_FEATURES = "x11 wayland"
IMAGE_LINGUAS = "en-us"
IMAGE_FEATURES += "splash package-management ssh-server-dropbear"
IMAGE_ROOTFS_MAXSIZE = ""
IMAGE_QT_MANDATORY_PART = " \
qtbase \
qtbase-plugins \
qtbase-tools \
"
IMAGE_QT_OPTIONAL_PART = " \
qtserialport \
"
CORE_IMAGE_EXTRA_INSTALL += " \
systemd-networkd-configuration \
\
packagegroup-framework-tools-core-base \
packagegroup-framework-tools-kernel-base \
packagegroup-framework-tools-network-base \
packagegroup-framework-tools-python2-base \
packagegroup-framework-tools-python3-base \
\
packagegroup-framework-tools-core \
packagegroup-framework-tools-kernel \
packagegroup-framework-tools-network \
packagegroup-framework-tools-python2 \
packagegroup-framework-tools-python3 \
\
packagegroup-core-eclipse-debug \
\
${IMAGE_QT_MANDATORY_PART} \
${IMAGE_QT_OPTIONAL_PART} \
"
and here the contents of the RDEPENDS_${PN} var in layers/meta-qt5/recipes-qt/packagegroups/packagegroup-qt5-toolchain-target.bb:
RDEPENDS_${PN} += " \
packagegroup-core-standalone-sdk-target \
libsqlite3-dev \
qtbase-dev \
qtbase-mkspecs \
qtbase-plugins \
qtbase-staticdev \
qtconnectivity-dev \
qtconnectivity-mkspecs \
qtmqtt-dev \
qtmqtt-mkspecs \
qtserialport-dev \
qtserialport-mkspecs \
qtserialbus-dev \
qtserialbus-mkspecs \
qtsystems-dev \
qtsystems-mkspecs \
qttools-dev \
qttools-mkspecs \
qttools-staticdev \
qtwebsockets-dev \
qtwebsockets-mkspecs \
qtwebchannel-dev \
qtwebchannel-mkspecs \
"
The PACKAGECONFIG is already there:
PACKAGECONFIG[sql-sqlite] = "-sql-sqlite -system-sqlite,-no-sql-sqlite,sqlite3"
Your problem is most likely due to you redefining in (wrongfully as you can see).
You do you have to define new PACKAGECONFIG. Just enable it with:
PACKAGECONFIG_append = " sql-sqlite"
I have two machine, each of which has 4 GPUs. I use
with tf.device('/job:worker/replica:%d/task:%d/gpu:%d' % (FLAGS.replica_id, FLAGS.task_id, FLAGS.gpu_device_id)):
to dictate device, but failed with these error log:
tensorflow.python.framework.errors.InvalidArgumentError: Cannot assign a device to node 'init_all_tables': Could not satisfy explicit device specification '/job:worker/replica:1/task:4/device:GPU:0' because no devices matching that specification are registered in this process; available devices: /job:ps/replica:0/task:0/cpu:0,
/job:worker/replica:0/task:0/cpu:0, /job:worker/replica:0/task:0/gpu:0, /job:worker/replica:0/task:0/gpu:1, /job:worker/replica:0/task:0/gpu:2, /job:worker/replica:0/task:0/gpu:3, /job:worker/replica:0/task:1/cpu:0, /job:worker/replica:0/task:1/gpu:0, /job:worker/replica:0/task:1/gpu:1, /job:worker/replica:0/task:1/gpu:2, /job:worker/replica:0/task:1/gpu:3, /job:worker/replica:0/task:2/cpu:0, /job:worker/replica:0/task:2/gpu:0, /job:worker/replica:0/task:2/gpu:1, /job:worker/replica:0/task:2/gpu:2, /job:worker/replica:0/task:2/gpu:3, /job:worker/replica:0/task:4/cpu:0, /job:worker/replica:0/task:4/gpu:0, /job:worker/replica:0/task:4/gpu:1, /job:worker/replica:0/task:4/gpu:2, /job:worker/replica:0/task:4/gpu:3, /job:worker/replica:0/task:5/cpu:0, /job:worker/replica:0/task:5/gpu:0, /job:worker/replica:0/task:5/gpu:1, /job:worker/replica:0/task:5/gpu:2, /job:worker/replica:0/task:5/gpu:3, /job:worker/replica:0/task:6/cpu:0, /job:worker/replica:0/task:6/gpu:0, /job:worker/replica:0/task:6/gpu:1, /job:worker/replica:0/task:6/gpu:2, /job:worker/replica:0/task:6/gpu:3, /job:worker/replica:0/task:7/cpu:0, /job:worker/replica:0/task:7/gpu:0, /job:worker/replica:0/task:7/gpu:1, /job:worker/replica:0/task:7/gpu:2, /job:worker/replica:0/task:7/gpu:3
it seems like tensorflow can't find machine B ? but I have totally same hardware and software configuration on both machine.
the start script:
# machine 10.10.12.28
~/models/inception/bazel-bin/inception/imagenet_distributed_train \
--batch_size=32 \
--data_dir=/data1/imagenet1k \
--job_name='worker' \
--replica_id=0 \
--task_id=0 \
--gpu_device_id=0 \
--ps_hosts='10.10.102.28:2220' \
--worker_hosts='10.10.102.28:2221,10.10.102.28:2222,10.10.102.28:2223,10.10.102.29:2224,10.10.102.29:2221,10.10.102.29:2222,10.10.102.29:2223,10.10.102.29:2224' &
~/models/inception/bazel-bin/inception/imagenet_distributed_train \
--batch_size=32 \
--data_dir=/data1/imagenet1k \
--job_name='worker' \
--replica_id=0 \
--task_id=1 \
--gpu_device_id=1 \
--ps_hosts='10.10.102.28:2220' \
--worker_hosts='10.10.102.28:2221,10.10.102.28:2222,10.10.102.28:2223,10.10.102.29:2224,10.10.102.29:2221,10.10.102.29:2222,10.10.102.29:2223,10.10.102.29:2224' &
~/models/inception/bazel-bin/inception/imagenet_distributed_train \
--batch_size=32 \
--data_dir=/data1/imagenet1k \
--job_name='worker' \
--replica_id=0 \
--task_id=2 \
--gpu_device_id=2 \
--ps_hosts='10.10.102.28:2220' \
--worker_hosts='10.10.102.28:2221,10.10.102.28:2222,10.10.102.28:2223,10.10.102.29:2224,10.10.102.29:2221,10.10.102.29:2222,10.10.102.29:2223,10.10.102.29:2224' &
~/models/inception/bazel-bin/inception/imagenet_distributed_train \
--batch_size=32 \
--data_dir=/data1/imagenet1k \
--job_name='worker' \
--replica_id=0 \
--task_id=3 \
--gpu_device_id=3 \
--ps_hosts='10.10.102.28:2220' \
--worker_hosts='10.10.102.28:2221,10.10.102.28:2222,10.10.102.28:2223,10.10.102.29:2224,10.10.102.29:2221,10.10.102.29:2222,10.10.102.29:2223,10.10.102.29:2224' &
CUDA_VISIBLE_DEVICES='' ~/models/inception/bazel-bin/inception/imagenet_distributed_train \
--job_name='ps' \
-task_id=0 \
--ps_hosts='10.10.102.28:2220' \
--worker_hosts='10.10.102.28:2221,10.10.102.28:2222,10.10.102.28:2223,10.10.102.29:2224,10.10.102.29:2221,10.10.102.29:2222,10.10.102.29:2223,10.10.102.29:2224' &
# machine 10.10.12.29
~/models/inception/bazel-bin/inception/imagenet_distributed_train \
--batch_size=32 \
--data_dir=/data1/imagenet1k \
--job_name='worker' \
--replica_id=1 \
--task_id=4 \
--gpu_device_id=0 \
--ps_hosts='10.10.102.28:2220' \
--worker_hosts='10.10.102.28:2221,10.10.102.28:2222,10.10.102.28:2223,10.10.102.29:2224,10.10.102.29:2221,10.10.102.29:2222,10.10.102.29:2223,10.10.102.29:2224' &
~/models/inception/bazel-bin/inception/imagenet_distributed_train \
--batch_size=32 \
--data_dir=/data1/imagenet1k \
--job_name='worker' \
--replica_id=1 \
--task_id=5 \
--gpu_device_id=1 \
--ps_hosts='10.10.102.28:2220' \
--worker_hosts='10.10.102.28:2221,10.10.102.28:2222,10.10.102.28:2223,10.10.102.29:2224,10.10.102.29:2221,10.10.102.29:2222,10.10.102.29:2223,10.10.102.29:2224' &
~/models/inception/bazel-bin/inception/imagenet_distributed_train \
--batch_size=32 \
--data_dir=/data1/imagenet1k \
--job_name='worker' \
--replica_id=1 \
--task_id=6 \
--gpu_device_id=2 \
--ps_hosts='10.10.102.28:2220' \
--worker_hosts='10.10.102.28:2221,10.10.102.28:2222,10.10.102.28:2223,10.10.102.29:2224,10.10.102.29:2221,10.10.102.29:2222,10.10.102.29:2223,10.10.102.29:2224' &
~/models/inception/bazel-bin/inception/imagenet_distributed_train \
--batch_size=32 \
--data_dir=/data1/imagenet1k \
--job_name='worker' \
--replica_id=1 \
--task_id=7 \
--gpu_device_id=3 \
--ps_hosts='10.10.102.28:2220' \
--worker_hosts='10.10.102.28:2221,10.10.102.28:2222,10.10.102.28:2223,10.10.102.29:2224,10.10.102.29:2221,10.10.102.29:2222,10.10.102.29:2223,10.10.102.29:2224' &
TL;DR: Don't ever use '/replica:%d' in your device specification.
The problem seems to be in your device string:
'/job:worker/replica:%d/task:%d/gpu:%d' % (FLAGS.replica_id, FLAGS.task_id, FLAGS.gpu_device_id)
The device specification '/replica:%d' is not supported in the open-source version of TensorFlow (but it is retained for some backwards compatibility reasons). The replica ID should be 0 for all tasks. You can solve this immediately by passing 0 as the --replica_id for each task, but you should really remove that flag from your version of the code.
Sometimes, due to the nicety of a compiler, language or luck of a programmer, a programmer may get away with some pretty sloppy code. In order to produce the cleanest build possible, i'd like to configure my Xcode settings to complain about the slightest suspected impropriety.
From your experience, what would you suggest?
The following comes to mind, but is there more?
i use something like this in my codebase (make an xcconfig). it's slightly dated (partly for toolset compatibility), but generally compatible with c, c++, objc, objc++ using Xcode's gcc4.2 and clang:
GCC_TREAT_IMPLICIT_FUNCTION_DECLARATIONS_AS_ERRORS = YES
GCC_TREAT_NONCONFORMANT_CODE_ERRORS_AS_WARNINGS = YES
GCC_TREAT_WARNINGS_AS_ERRORS = NO
GCC_WARN_64_TO_32_BIT_CONVERSION = YES
GCC_WARN_ABOUT_DEPRECATED_FUNCTIONS = YES
GCC_WARN_ABOUT_GLOBAL_CONSTRUCTORS = YES
GCC_WARN_ABOUT_INVALID_OFFSETOF_MACRO = YES
GCC_WARN_ABOUT_MISSING_FIELD_INITIALIZERS = YES
GCC_WARN_ABOUT_MISSING_NEWLINE = YES
GCC_WARN_ABOUT_MISSING_PROTOTYPES = YES
GCC_WARN_ABOUT_POINTER_SIGNEDNESS = YES
GCC_WARN_ABOUT_RETURN_TYPE = YES
GCC_WARN_ALLOW_INCOMPLETE_PROTOCOL = YES
GCC_WARN_CHECK_SWITCH_STATEMENTS = YES
GCC_WARN_EFFECTIVE_CPLUSPLUS_VIOLATIONS = YES
GCC_WARN_FOUR_CHARACTER_CONSTANTS = YES
GCC_WARN_HIDDEN_VIRTUAL_FUNCTIONS = YES
GCC_WARN_INHIBIT_ALL_WARNINGS = NO
GCC_WARN_INITIALIZER_NOT_FULLY_BRACKETED = YES
GCC_WARN_MISSING_PARENTHESES = YES
GCC_WARN_NON_VIRTUAL_DESTRUCTOR = YES
GCC_WARN_PEDANTIC = YES
GCC_WARN_SHADOW = YES
GCC_WARN_SIGN_COMPARE = YES
GCC_WARN_STRICT_SELECTOR_MATCH = YES
GCC_WARN_TYPECHECK_CALLS_TO_PRINTF = YES
GCC_WARN_UNDECLARED_SELECTOR = YES
GCC_WARN_UNKNOWN_PRAGMAS = YES
GCC_WARN_UNUSED_FUNCTION = YES
GCC_WARN_UNUSED_LABEL = YES
GCC_WARN_UNUSED_PARAMETER = YES
GCC_WARN_UNUSED_VALUE = YES
GCC_WARN_UNUSED_VARIABLE = YES
// this gets sloppy
WARNING_CFLAGS = -Waddress -Woverflow -Winline -pedantic -Wundef -Wstrict-overflow -Wstrict-overflow=4 -Wsequence-point -Wdeprecated-declarations -Wendif-labels -Winit-self -Wstrict-aliasing=2 -Wstrict-aliasing -Wextra -Wcast-align -Wcast-qual -Wchar-subscripts -Wcomment -Wconversion -Wfloat-equal -Wformat -Wformat=2 -Wformat-extra-args -Wformat-security -Wimplicit -Winit-self -Winvalid-pch -Wmissing-braces -Wmissing-field-initializers -Wmissing-format-attribute -Wmultichar -Wparentheses -Wpointer-arith -Wredundant-decls -Wreturn-type -Wsequence-point -Wshadow -Wstack-protector -Wsign-compare -Wswitch -Wswitch-default -Wswitch-enum -Wtrigraphs -Wundef -Wunknown-pragmas -Wunused -Wunused-function -Wunused-label -Wunused-parameter -Wunused-value -Wunused-variable -Wwrite-strings -Wvariadic-macros -Winline -Wformat-nonliteral $(inherited)
WARNING_CPLUSPLUSFLAGS = -Wno-long-long -Waddress -Woverflow -Winline -Wno-non-virtual-dtor -pedantic -Wundef -Wstrict-overflow -Wstrict-overflow=4 -Wsequence-point -Wdeprecated-declarations -Wendif-labels -Winit-self -Wstrict-aliasing=2 -Wstrict-aliasing -Wextra -Wcast-align -Wcast-qual -Wchar-subscripts -Wcomment -Wconversion -Wfloat-equal -Wformat -Wformat=2 -Wformat-extra-args -Wformat-security -Wimplicit -Winit-self -Winvalid-offsetof -Winvalid-pch -Wmissing-braces -Wmissing-field-initializers -Wmissing-format-attribute -Wmultichar -Wparentheses -Wpointer-arith -Wredundant-decls -Wreturn-type -Wsequence-point -Wshadow -Wstack-protector -Wsign-compare -Wswitch -Wswitch-default -Wswitch-enum -Wtrigraphs -Wundef -Wunknown-pragmas -Wunused -Wunused-function -Wunused-label -Wunused-parameter -Wunused-value -Wunused-variable -Wwrite-strings -Wvariadic-macros -Wreorder -Weffc++ -Wdeprecated -Woverloaded-virtual -Wsign-promo -Wold-style-cast -Winline -Wformat-nonliteral $(inherited)
// not categorized as a warning, but will enable warnings
GCC_STRICT_ALIASING = YES
// Apple headers are too much for this
GCC_WARN_MULTIPLE_DEFINITION_TYPES_FOR_SELECTOR = NO
// useful for some output (conversions), but useless for the other half (implicit conversions which do not apply to c++). clang covers many of the cases with its warnings.
GCC_WARN_PROTOTYPE_CONVERSION = NO
// disabled by default - *do* enable in optimized configurations
GCC_WARN_UNINITIALIZED_AUTOS = NO
// clang-ana:
CLANG_ANALYZER_DEADCODE_DEADSTORES = YES
CLANG_ANALYZER_DEADCODE_IDEMPOTENT_OPERATIONS = YES
CLANG_ANALYZER_OBJC_SELF_INIT = YES
if you've already dropped gcc, then you can remove many which clang does not support. clang has a few options of its own (varies greatly by the distributions you're using).