My team has been working with serverless and we're trying to establish a new standard in the company for a file organization that eases the collaborative development.
We plan to isolate the lambdas/functions handler each in their folder, alongside the function .yml file and other necessary configs.
Example expected directory structure (lean):
-- /app
--- /functions
---- /func_a
----- func_a.py
----- func_a.yml
---- /func_b
----- func_b.py
----- func_b.yml
- serverles.yml
The problem so far is that we have to manually declare all external config function files in the serverless.yml file, which breaks the whole purpose of the idea.
The question is: is there a way to automate this import?
What we've searched so far:
Wildcard path - does not work for file variables. Eg.: ${file(./app/functions/*.yml)} or ${file(./app/functions/{any+})}
Extending configuration using custom plugins - does not seem to be able to modify the functions list. Only found information about: DefineTopLevelProperty, defineCustomProperties, defineFunctionEvent, defineFunctionEventProperties, defineFunctionProperties, defineProvider.
Info from here: Serverless Doc - base schema link broken, so no other information aside from the one in the page.
What we thought to be options:
Maybe is there a plugin that does that? Didn't find any.
Create an isolated custom function (python) that is called before running sls deploy and creates the final serverles.yml file from a template by traversing all folders.
What is the better and most natural approach to that?
I think that for your use case, you might be better of with considering using JS/TS configuration file format instead of YAML. That allows you to use regular JS to define your config which makes importing such parts of configurations much easier. See the TS template for example on how to use it: https://github.com/serverless/serverless/blob/master/lib/plugins/create/templates/aws-nodejs-typescript/serverless.ts
Related
Using ezplatform with "kaliop/ezmigrationbundle": "^3.0".
I read and re-read the manual but cannot see anything about auto-generating yml files from exsiting content types; does anyone have any experience with this and happen to know if/where the docs might live?
We have a list of content types that were created in the backend via the gui and now we need to create migration files for them for better development with the dev team.
Update: This is available on v4+ https://github.com/kaliop-uk/ezmigrationbundle/issues/34#issuecomment-317524072#issuecomment-317524072
This is available on v4+ and answers the question
https://github.com/kaliop-uk/ezmigrationbundle/issues/34#issuecomment-317524072#issuecomment-317524072
I guess that is not how it works!
the bundle generate just an empty yaml file for you and you should fill up the content types (or any other Backend Changes you want) yourself in that yaml file and then take it and apply to your stage or Live Environment.
so unlike Symfony DoctorineMigrationBunlde, this bundle does not read the difference and produce stuff itself.
Using NiFi v0.6.1 is there a way to import backups/archives?
And by backups I mean the files that are generated when you call
POST /controller/archive using the REST api or "Controller Settings" (tool bar button) and then "Back-up flow" (link).
I tried unzipping the backup and importing it as a template but that didn't work. But after comparing it to an exported template file, the formats are reasonably different. But perhaps there is a way to transform it into a template?
At the moment my current work around is to not select any components on the top level flow and then select "create template"; which will add a template with all my components. Then I just export that. My issue with this is it's a bit more tricky to automate via the REST API. I used Fiddler to determine what the UI is doing and it first generates a snippet that includes all the components (labels, processors, connections, etc.). Then it calls create template (POST /nifi-api/contorller/templates) using the snippet ID. So the template call is easy enough but generating the definition for the snippet is going to take some work.
Note: Once the following feature request is implemented I'm assuming I would just use that instead:
https://cwiki.apache.org/confluence/display/NIFI/Configuration+Management+of+Flows
The entire flow for a NiFi instance is stored in a file called flow.xml.gz in the conf directory (flow.xml.tar in a cluster). The back-up functionality is essentially taking a snapshot of that file at the given point in time and saving it to the conf/archive directory. At a later point in time you could stop NiFi and replace conf/flow.xml.gz with one of those back-ups to restore the flow to that state.
Templates are a different format from the flow.xml.gz. Templates are more public facing and shareable, and can be used to represent portions of a flow, or the entire flow if no components are selected. Some people have used templates as a model to deploy their flows, essentially organizing their flow into process groups and making template for each group. This project provides some automation to work with templates: https://github.com/aperepel/nifi-api-deploy
You just need to stop NiFi, replace the nifi flow configuration file (for example this could be flow.xml.gz in the conf directory) and start NiFi back up.
If you have trouble finding it check your nifi.properties file for the string nifi.flow.configuration.file= to find out what you've set this too.
If you are using clustered mode you need only do this on the NCM.
Somewhat related to this question - Parametrizing node name in Elixir Exrm - is there a way to dynamically set the content of the rel/vm.args file?
In the title, I suggest the use of config.exs, but I'm interested in any scheme that will allow me to add, remove and edit vm arguments at build time.
A bash script might be an idea, but a solution that would also work on Windows, and is preferably based on Elixir code would be the idea.
You can commit to your project repo a rel/vm.args file that you would like to be used instead of the Exrm generated one. Exrm will automatically use this file instead.
I have a situation where I would like to ignore specific folders inside of where Flyway is looking for the migration files.
Example
/db/Migration
2.0-newBase.sql
/oldScripts
1.1-base.sql
1.2-foo.sql
I want to ignore everything inside of the 'oldScripts' sub folder. Is there a flag that I can set in Flyway configs like ignoreFolder=SOME_FOLDER or scanRecursive=false?
An example for why I would do this is say, I have 1000 scripts in my migration folder. If we onboard a new member, instead of having them run the migration on 1000 files, they could just run the one script (The new base) and proceed from there. The alternative would be to never sync those files in the first place, but then people would need to remember to check source control to prior migrations instead of just looking on their local drive.
This is not currently supported directly. You could put both directories at the same level in the hierarchy (without nesting them) and selectively configure flyway.locations to achieve the same thing.
Since Flyway 6.4.0 wildcards are supported in flyway.locations. Examples:
db/**/test
db/release1.*
db/release1.?
More info at https://flywaydb.org/blog/organising-your-migrations
I am using the Intellij 11.1.5. We are a large team, and have a pretty complex project setup. so we've made a template and when someone needs a new project set up, we just clone it and she is pretty much ready to go. One other thing i would like to automate is the creation of run configurations. One such configuration starts a custom bat file that requires a parameter representing a path that is user specific. I wanted to know if can store that value as a path variable specific to each project. Maybe somewhere in the .idea folder in my project. I know that Intellij stores it in its .IntelliJIdea11\config\options\path.macros.xml file, but is there a way to tweak that?
Any other idea that would allow me to locally store a parameter passed to the run config script would be usefull.
Thanks
I'm afraid you can't do it in IDEA, but you can use some environment variable directly in the .bat file instead of using the parameter (or rewrite the batch script to detect this value automatically, if possible). Instruct your users to define this environment variable.
IDEA Path variables are global and cannot be made project specific.