Enviroment:Docker
After I invoke endpoint :importConfiguration,Unomi does not import profile csv.
Related
I have a python module (odoo.py) that creates a connection with an Odoo instance.
import odoorpc
from .env import ENV
ODOO = odoorpc.ODOO(ENV["ODOO_HOST"], port=ENV["ODOO_PORT"])
ODOO.login(db=ENV["ODOO_DB"], login=ENV["ODOO_USR"], password=ENV["ODOO_PWD"])
ODOO constant is imported by several modules containing Mongoengine classes to perform operations on the Odoo server. Here is an example:
from mongoengine import *
from .odoo import ODOO
class Category(DynamicDocument):
name = StringField(required=True)
odoo_id = StringField()
def publish(self):
_model = ODOO.env["product.category"]
self.odoo_id = _model.create({"name": self.name})
self.save()
Problem is that importing odoo.py takes about 10 seconds during application startup and it's REALLY annoying in development.
Let's consider this pseudo main.py:
import sys
from .category import Category
cat = Category(name=sys.argv[1])
if not 'test' in cat.name:
cat.publish()
In this example if sys.argv[1] contains the string 'test' then ODOO constant will never be used and waiting for its import would be totally useless.
Is there a way to "background" importing of odoo.py and wait for it only when ODOO constant is actually needed?
I am changing an API created in a flask to FastAPI, but I don't know how to change the following code.
Specifically I want to get guidance on how I can get db defined below in fastapi so that I don't have to change my downstream code which uses db.session.query. In short, I am looking for fastapi version for below code.
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()
Thanks !
You can follow the offcial fastapi guide regarding orm bests practices. Fastapi is not tiedup with any orm or db client so you can implement sqlalchemy same way than flask (with fastapi you can do async db or orm calls wich is nice)
Taken from the doc:
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
SQLALCHEMY_DATABASE_URL = "sqlite:///./sql_app.db"
# SQLALCHEMY_DATABASE_URL = "postgresql://user:password#postgresserver/db"
engine = create_engine(
SQLALCHEMY_DATABASE_URL, connect_args={"check_same_thread": False}
)
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
Base = declarative_base()
A more complexe project that implement async sqlalchemy calls and crud interface
I'm trying to execute the following DAG in Airflow Composer on google cloud and I keep getting the same error:
The conn_id hard_coded_project_name isn't defined
Maybe someone can point me to the right direction?
from airflow.models import DAG
import os
from airflow.operators.dummy import DummyOperator
from airflow.operators.bash_operator import BashOperator
from airflow.operators.python_operator import PythonOperator
import datetime
import pandas as pd
from airflow.contrib.operators.gcs_to_bq import GoogleCloudStorageToBigQueryOperator
from airflow.contrib.operators.bigquery_operator import BigQueryOperator
from airflow.providers.google.cloud.operators import bigquery
from airflow.contrib.hooks.bigquery_hook import BigQueryHook
default_args = {
'start_date': datetime.datetime(2020, 1, 1),
}
PROJECT_ID = os.environ.get("GCP_PROJECT_ID", "hard_coded_project_name")
def list_dates_in_df():
hook = BigQueryHook(bigquery_conn_id=PROJECT_ID,
use_legacy_sql=False)
bq_client = bigquery.Client(project = hook._get_field("project"),
credentials = hook._get_credentials())
query = "select count(*) from LP_RAW.DIM_ACCOUNT;"
df = bq_client.query(query).to_dataframe()
with DAG(
'df_test',
schedule_interval=None,
catchup = False,
default_args=default_args
) as dag:
list_dates = PythonOperator(
task_id ='list_dates',
python_callable = list_dates_in_df
)
list_dates
It means that PROJECT_ID as seen in line
PROJECT_ID = os.environ.get("GCP_PROJECT_ID", "hard_coded_project_name")
was assigned value hard_coded_project_name since GCP_PROJECT_ID has no value.
Then at line
hook = BigQueryHook(bigquery_conn_id=PROJECT_ID...
the string hard_coded_project_name is automatically associated with a connection id in Airflow and it does not have a value or it does not exist.
To avoid this error you can do either steps to fix this.
Create a connection id for both GCP_PROJECT_ID and hard_coded_project_name just so we are sure that both have values. But if we don't want to create a connection for GCP_PROJECT_ID, make sure that hard_coded_project_name has a value so there will be a fallback option. You can do this by
Opening your Airflow instance.
Click "Admin" > "Connections"
Click "Create"
Fill up "Conn Id", "Conn Type" as "hard_coded_project_name" and "Google Cloud Platform" respectively.
Fill up "Project Id" with your actual project id value
Do these steps another time to create GCP_PROJECT_ID
The connection should look like this (at minimum, providing the projectID will work. But feel free to add the keyfile or its content and scope so you won't be having problems on authentication moving forward):
You can use bigquery_default instead of hard_coded_project_name so by default it will point to the project that runs the Airflow instance.
Your updated PROJECT_ID assignment code will be
PROJECT_ID = os.environ.get("GCP_PROJECT_ID", "bigquery_default")
Also when testing your code you might encounter an error at line
bq_client = bigquery.Client(project = hook._get_field("project")...
since Client() does not exist on from airflow.providers.google.cloud.operators import bigquery you should use from google.cloud import bigquery instead.
Here is a snippet of the test where I only created hard_coded_project_name so PROJECT_ID will use this connection.I got the count of a table of mine and it worked:
Here is a snippet of the test I made when I used bigquery_default where I got the count of a table of mine and it worked:
When I select Code → Optimize Imports or Code → Reformat Code, the IntelliJ does optimize and sort imports, but even though I am using code style settings from Kotlin code style, the imports are not sorted in lexicographic order (not entirely at least). For example, this is the output produced:
import com.fasterxml.jackson.databind.ObjectMapper
import io.dropwizard.jackson.Jackson
import io.kotlintest.assertSoftly
import io.kotlintest.matchers.types.shouldBeNull
import io.kotlintest.shouldBe
import io.kotlintest.specs.ShouldSpec
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import org.intellij.lang.annotations.Language
import java.time.Instant // This should not be at the bottom!!!
while this is what expected by ktlint:
import com.fasterxml.jackson.databind.ObjectMapper
import io.dropwizard.jackson.Jackson
import io.kotlintest.assertSoftly
import io.kotlintest.matchers.types.shouldBeNull
import io.kotlintest.shouldBe
import io.kotlintest.specs.ShouldSpec
import java.time.Instant // should be here instead
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import org.intellij.lang.annotations.Language
In other words, it seems the formatter always puts imports from java.* package at the bottom no matter what. Is there a way to make it be in line with what ktlint expects (and what would actually constitute lexicographic order)? Any hidden option I am missing or something?
I am using IntelliJ IDEA 2019.3.1 (Ultimate Edition) with the Kotlin plugin version 1.3.61-release-IJ2019.3-1. Version of ktlint is 0.36.0
This is a bug/missing functionality in the Kotlin IDEA plugin: https://youtrack.jetbrains.com/issue/KT-10974. Please vote.
You can change IntelliJ's import order for Kotlin to satisfy ktlint:
In Settings > Editor > Code Style > Kotlin you can change Import Layout by removing everything except import all other imports from the list:
Before:
After:
What worked for me is manually removing all imports and then reimporting each import.
Ive been going at this for several hours but im afraid I still don't gronk flask app context and how my app should be implemented with Blueprints.
Ive taken a look at the this and this and have tried a few different recommendations but there must be something wrong with my basic approach.
I have one 'main' blueprint setup under the following PJ structure:
project/
app/
main/
__init__.py
routes.py
forms.py
helper.py
admin/
static/
templates/
__init__.py
models.py
app/init.py:
from flask import Flask
from config import config
from flask.ext.sqlalchemy import SQLAlchemy
from flask.ext.bootstrap import Bootstrap
db = SQLAlchemy()
bootstrap = Bootstrap()
def create_app(config_version='default'):
app = Flask(__name__)
app.config.from_object(config[config_version])
bootstrap.init_app(app)
from .main import main as main_blueprint
app.register_blueprint(main_blueprint)
db.init_app(app)
return app
app/main/init.py
from flask import Blueprint
main = Blueprint('main',__name__)
from . import routes, helper
app/main/helper.py
#!/usr/bin/env python
from . import main
from ..models import SKU, Category, db
from flask import current_app
def get_categories():
cat_list = []
for cat in db.session.query(Category).all():
cat_list.append((cat.id,cat.category))
return cat_list
Everything worked fine until I created the get_categoriesfunction in helpers.py to pull a dynamic list for a select form in app/main/forms.py. When I fireup WSGI, however, I get this error:
RuntimeError: application not registered on db instance and no application bound to current context
It would appear the db referenced in helper is not associated with an app context but when I try to create one within the function, it has not worked.
What am I doing wrong and is there a better way to organize helper functions when using Blueprints?
Documentation on database contexts here here.
My first thought was that you weren't calling db.init_app(app), but was corrected in the comments.
In app/main/helper.py, you're importing the database via from ..models import SKU, Category, db. This database object will not have been initialized with the app that you've created.
The way that I've gotten around this is by having another file, a shared.py in the root directory. In that file, create the database object,
from flask.ext.sqlalchemy import SQLAlchemy
db = SQLAlchemy()
In your app/init.py, don't create a new db object. Instead, do
from shared import db
db.init_app(app)
In any place that you want to use the db object, import it from shared.py. This way, the object in the shared file will have been initialized with the app context, and there's no chance of circular imports (which is one of the problems that you can run into with having the db object outside of the app-creating file).