Ubuntu 22.04. I set VK_LAYER_PATH in /etc/enviroment like VK_LAYER_PATH=/etc/vulkan/implicit_layer.d where stored single json file with AMDVLK (need for test). But when I check vulkaninfo, it says that I still have 10 layers (without VK_LAYER_PATH vulkaninfo shows me 12 layers) , among them, as example, MangoHUD. I know where MangoHUD stored its layer json file. This is totally different folder (/usr/share/vulkan/**). From documentation I see
"Setting the VK_LAYER_PATH environment variable overrides the default loader layer search mechanism. When set, the loader will search only the directory(s) identified by the VK_LAYER_PATH environment variable for layer manifest files."
But it looks like Loader still goes through all default folders to look json files.
Found information that VK_LAYER_PATH only override explicit layers. Implicit will be still looked by default folders. Is it correct? If yes, do we have any option to override implicit search path?
You've probably already found the Loader Layer Interface doc and others like it in the same folder.
Since the loading of implicit layers is supposed to be automatic, I don't think that there is a way to set a search path just for implicit layers by design. You could post a question to the same Vulkan-Loader repo to get a better opinion.
If your goal is to prevent the loading of a set of implicit layers in order to test your own implicit layer by itself, then you can disable implicit layers one by one. Each layer manifest specifies an environment variable that can be set to disable that layer.
Related
I've started using Kubeflow Pipelines to run data processing, training and predicting for a machine learning project, and I'm using InputPath and OutputhPath to pass large files between components.
I'd like to know how, if it's possible, do I set the path that OutputPath would look for a file in in a component, and where InputPath would load a file in a component.
Currently, the code stores them in a pre-determined place (e.g. data/my_data.csv), and it would be ideal if I could 'tell' InputPath/OutputPath this is the file it should copy, instead of having to rename all the files to match what OutputPath expects, as per below minimal example.
#dsl.pipelines(name='test_pipeline')
def pipeline():
pp = create_component_from_func(func=_pre_process_data)()
# use pp['pre_processed']...
def pre_process_data(pre_processed_path: OutputPath('csv')):
import os
print('do some processing which saves file to data/pre_processed.csv')
# want to avoid this:
print('move files to OutputPath locations...')
os.rename(f'data/pre_processed.csv', pre_processed_path)
Naturally I would prefer not to update the code to adhere to Kubeflow pipeline naming convention, as that seems like very bad practice to me.
Thanks!
Update - See ark-kun's comment, the approach in my original answer is deprecated and should not be used. It is better to let Kubeflow Pipelines specify where you should store your pipeline's artifacts.
For lightweight components (such as the one in your example), Kubeflow Pipelines builds the container image for your component and specifies the paths for inputs and outputs (based upon the types you use to decorate your component function). I would recommend using those paths directly, instead of writing to one location and then renaming the file. The Kubeflow Pipelines samples follow this pattern.
For reusable components, you define the pipeline inputs and outputs as part of the YAML specification for the component. In that case you can specify your preferred location for the output files. That being said, reusable components take a bit more effort to create, since you need to build a Docker container image and component specification in YAML.
This is not supported by the system.
Components should use the system-provided paths.
This is important, because on some execution engines the data is mounted to those paths. And sometimes these paths have certain restrictions or might even be unchangeable. So the system must have the freedom to choose the paths.
Usually, good programs do not hard-code any absolute paths inside their code, but rather receive the paths from the command line.
In any case, it's pretty easy to copy the files from or to the system-provided paths (as you already do in the code).
Currently in our project we have layered architecture implemented in following way where Controller, Service, Repository are placed in the same package for each feature, for instance:
feature1:
Feature1Controller
Feature1Service
Feature1Repository
feature2:
Feature2Controller
Feature2Service
Feature2Repository
I've found following example of arch unit test where such classes are placed in dedicated packages https://github.com/TNG/ArchUnit-Examples/blob/master/example-junit5/src/test/java/com/tngtech/archunit/exampletest/junit5/LayeredArchitectureTest.java
Please suggest whether there is possibility to test layered architecture when all layers are in single package
If the file name conventions are followed properly across your project, how about you write custom test cases instead of using layeredArchitecture().
For Example:
classes().that().haveSimpleNameEndingWith("Service")
.should().onlyBeAccessed().byClassesThat().haveSimpleNameEndingWith("Controller")
noClasses().that().haveSimpleNameEndingWith("Service")
.should().accessClassesThat().haveSimpleNameEndingWith("Controller")
I know this question is rather old. But for the record, this has been possible for a while using predicates for the layers, e.g.
layeredArchitecture().consideringAllDependencies()
.layer("Controllers").definedBy(HasName.Predicates.nameEndingWith("Controller"))
.layer("Services").definedBy(HasName.Predicates.nameEndingWith("Service"))
.layer("Repository").definedBy(HasName.Predicates.nameEndingWith("Repository"))
.whereLayer("Controllers").mayNotBeAccessedByAnyLayer()
.whereLayer("Services").mayOnlyBeAccessedByLayers("Controllers")
.whereLayer("Repository").mayOnlyBeAccessedByLayers("Services")
However, I'm not sure how well this works in practice. Because usually you don't just have classes following this naming pattern and that's it. A service might also have some POJO as method parameter type (e.g. MyInput) and that should maybe for example not be used by repositories as well. Also, using forward dependency rules (mayOnlyAccessLayers(..)) this might then cause unwanted violations.
Considering all freedom that yocto gives to the developer, I have a question.
I would like to make this my_file.inc available only for recipes in one particular meta-layer. I know, that, for instance, using INHERIT keyword inside the local.conf will make my_class.bbclass file available globally for each recipe.
Is it a good practice to add this:
require my_file.inc
inside layer.conf? Or should I change my_file.inc to the my_file.bbclass, and, add INHERIT = "my_file.bbclass" to the layer.conf?
Any other possibilities?
Even if it seems to work, neither of your approaches is technically completely correct. The key point is that all .conf files are parsed first and everything they contain is globally visible throughout the whole build process. So if you add something through the layer.conf file, itis not being pulled in through an unexpected place, it also is not being limited that layer only and might therefore cause breakage at other places.
While I do not have a really good and clean solution, maybe the following can help you:
You can make your custom recipes react on certain keywords in DISTRO_FEATURES or MACHINE_FEATURES. Then you can create a two-staged approach:
Add the desired keyword in local.conf (or your MACHINE, or DISTRO, or whatever configuration)
Make the recipes react to it. If you need the mechanism in several places, then it might be useful to pour it into a .bbclass that your layer brings along and that you pull in for the respective recipes.
This way the effect is properly contained.
Maybe part 5.1.3.2 from the Yocto Project answers your question:
Avoid duplicating include files. Use append files (.bbappend) for each recipe that uses an include file. Or, if you are introducing a new recipe that requires the included file, use the path relative to the original layer directory to refer to the file. For example, use require recipes-core/package/file.inc instead of require file.inc. If you're finding you have to overlay the include file, it could indicate a deficiency in the include file in the layer to which it originally belongs. If this is the case, you should try to address that deficiency instead of overlaying the include file. For example, you could address this by getting the maintainer of the include file to add a variable or variables to make it easy to override the parts needing to be overridden.
So to avoid duplicate inclusion later, it would be better not to include your .inc file(s) this way.
If your project depends on TensorFlow it is recommended that you add...
load("//tensorflow:workspace.bzl", "tf_workspace")
tf_workspace()
...to your WORKSPACE file, which will load all of TF's dependencies.
However, if you look at TensorFlow's workspace.bzl file...
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/workspace.bzl
you can see that it depends on rules from #io_bazel_rules_closure. This means you also have to define this #io_bazel_rules_closure rule in your WORKSPACE file and keep it in sync with TensorFlow, even if you don't need it anywhere else in your project.
Is there a way to add the load() command somehow/somewhere into the tf_workspace() macro?
Thanks!
No, there is no way to add this rule in tf_workspace(), since the skylark rule tf_workspace() defined in https://github.com/tensorflow/tensorflow/blob/master/tensorflow/workspace.bzl needs to load #io_bazel_rules_closure.
There are basically two ways to make this work
either the tensorflow project redefines its rules so that it only uses internal rules or native rules.
or bazel is able to load the workspace of a dependency (and I assume load all the transitive dependencies too). This is a hard problem and is tracked in #1943.
I have a custom algorithm for processing a graph which accepts a txt file as input. Because it is a large scale graph I want to implement it in the apache giraph framework. I' ve done a lot of research but I am still not sure if I am in the right path.
I am reading a .csv file which contain the graph data and using a parser I am converting it to the txt file and uploading to the HDFS file system of hadoop.
I have read the SimpleShortestPathsVertex example from the apache quick start guide and I can see that processes the data from a file in HDFS using the jar-with-dependencies jar file.
My problem is that I haven't yet understood how can I add my algorithm in the apache giraph framework and start the process of the graph. Can I add my algorithm to apache framework using eclipse and modify it from there or there is any other way?
Thank you!
Have a look here:
https://cwiki.apache.org/confluence/display/GIRAPH/Shortest+Paths+Example
Where you able to run this example?
If yes.
Familiarize yourself with the different Writable formats of hadoop! Else it is hard to use these to your algorithm.
All computation concerning the graph is done in the compute() function.
(If you're more advanced have a look into the workerContext preSuperstep and Aggregators!)
You can change the example, but as soon as you use other data types you have to change your VertexReader and VertexWriter.
If you have a specific Algorithm in mind, make up your mind what you need for the computation and specify the layout of your input file. Then adapt your VertexReader and -Writer. And then finaly start the implement your compute() function!
Of course you can use eclipse! Simply Reference the Giraph jar (For me it is "giraph-0.1-jar-with-dependencies.jar") And start coding.
All you need is a instance of these files specific to your algorithm:
YourGiraphJob (the file starting the Hadoop/Giraph job)
YourVertex (Specifies your compute() function executed on each Vertex)
YourInputFormat (Specifying the Writable formats of YourReader)
YourOutputFormat (Specifying the Writable formats of YourWriter)
YourReader (Specifies how your inputFile is transformed e.g. that for each line a Vertex can be initialized using given information)
YourWriter (Specifies how your outputFile is generated from the vertices)
(optionaly a WorkerContext if you want to use Aggregators.)
Simply checkout: http://giraph.apache.org/source-repository.html
using eclipse and you should have the code including an example application you can toy around with!