We use Talend as ETL and PBI as Dashboard tool. We have scheduled when issues must be launched, but we want to launch it after ETL loading process. Is there any way to tell to PBI it refresh dataset when ETL processes has finished?
you can use Powe- Automate to ensure automatic refresh takes place for a powerBI
you can schedule it or you can make it triggered by file or token.
you need to check which version of PBI you are using & if it include Power automate feature.
Related
We have a data pipeline with ELT in BigQuery. We have several transformations. Some of those transformations depend on other transformations happening before.
With BigQuery scheduled queries we can only set a time, so either a lot of time the system is idle if we have large buffers, or when dependent scheduled queries are too near to each other, they overlap. How would one model a transformation pipeline in BigQuery with dependencies?
[Edit] I know about external tools like AirFlow but would like to use only Google services.
We can use workflow orchestrator solutions like Composer Airflow(Costly) or Cloud serverless workflows to manage the dependencies and the time of execution.
I have a job of running certain set of queries on daily basis to fetch data from my datbase. Queries are run on sas enterprise guide.
So basically I need to automate this process.Please suggest some code so that automatically at a particular time of the day, these queries are automatically run and I get my data.
Enterprise Guide help for "Scheduling projects and process flows" describes the steps needed.
Automating Projects
Scheduling projects and process flows
In SAS Enterprise Guide, you can use the Microsoft Windows Task Scheduler to schedule projects and process flows to run at a specified time or as the result of a system event. By default, when you open the Task Scheduler, a script is automatically created in SAS Enterprise Guide to run the project or process flow. When you schedule the project or process flow, the Task Scheduler creates a scheduled task that includes the script and the criteria that specify when the task should be run. The scheduled task is added to the project tree.
Note You must save the project to your local computer before you can create a scheduled task.
I am trying to run load tests on my existing selenium web tests and my api(unit) tests. The tests run in Visual studio using load test editor but does not collect all the metrics like response time and requests per seconds. Are there any additional parameters that I need to add to collect all the metrics ?
Load testing; how many selenium clients are you running? One or two will not generate much load. First issue to think about; you need load generators and selenium is a poor way to go about this (unless you are running grid headless but still).
So the target server is what, Windows Server 2012? Google Create a Data Collector Set to Monitor Performance Counters.
Data collection and analysis of same is your second issue to think about. People pays loads of money for tools like LoadRunner because they provide load generators and sophisticated data collection of servers, database, WANs and LANS and analysis reports to pinpoint bottlenecks. Doing this manually is hard and not easily repeatable. Most folks who start down your path eventually abandon it. Look into the various load/performance tools to see what works best for you and that you can afford.
I've successfully created a query with the Extractor tool found in Import.io. It does exactly what I want it to do, however I need to now run this once or twice a day. Is the purpose of Import.io as an API to allow me to build logic such as data storage and schedules tasks (running queries multiple times a day) with my own application or are there ways to scheduled queries and make use of long-term storage of my results completely within the Import.io service?
I'm happy to create a Laravel or Rails app to make requests to the API and store the information elsewhere but if I'm reinventing the wheel by doing so and they provides the means to address this then that is a true time saver.
Thanks for using the new forum! Yes, we have moved this over to Stack Overflow to maximise the community atmosphere.
At the moment, Import does not have the ability to schedule crawls. However, this is something we are going to roll out in the near future.
For the moment, there is the ability to set a Cron job to run when you specify.
Another solution if you are using the free version is to use a CI tool like travis or jenkins to schedule your API scripts.
You can query live the extractors so you don't need to make them run manually every time. This will consume one of your requests from your limit.
The endpoint you can use is:
https://extraction.import.io/query/extractor/extractor_id?_apikey=apikey&url=url
Unfortunately the script will not be a very simple one since most websites have very different respond structures towards import.io and as you may already know, the premium version of the tool provides now with scheduling capabilities.
We are developing an application which uses Spark & Hive to do static and ad-hoc reporting. For these static reports, they take a number of parameters and then run over a data set. We would like to make it easier to test performance of these reports on a cluster.
If we have a test cluster running with a sufficient sample data set which developers can share. To speed up development time, what is the best way to deploy a Spark application to a Spark cluster (in standalone) via an IDE?
I'm thinking we would create an SBT task which would run the spark submit script. Is there a better way?
Eventually this will feed into some automated performance testing which we plan to run as a twice daily Jenkins job. If its an SBT deploy task, it makes it easy to call in Jenkins. Is there a better way to do this?
I've found a project on GitHub, maybe you can get some inspiration.
Maybe just add a for loop for submitting jobs and increase the loop times to find the performance limit, not sure if I'm right or not.