I need to replicate data from a source HANA DB to other HANA DB on real time basis(on trigger event like update/delete/insert). I have searched for the solution and found one i.e. SAP LT Replication Server. But I need custom implementation like taking data from source HANA DB and putting in a queue. Is it possible? And need some suggetions regarding SAP LT Replication Server.
Your question is pretty broad, so I recommend to familiarize yourself more with the use case and the available technology.
Besides SAP LT replication, you could potentially use the Smart Streaming option to facilitate real-time data transfer.
However, database level real-time replication is a rather specific requirement and can easily be confused with more general integration requirements. Be very careful about the application level you integrate your applications on.
You can use SAP HANA EIM Smart Data Integration. The HANA adapter is trigger based, supports real time scenarios including DDL propagation.
Within the target HANA DB you would create a reptask or a realtime flow-graph.
You can find the details in the Administration Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data Quality.
Related
I am looking for the recommended way of streaming database change from cloud SQL (postgres) to bigQuery ? I am seeing that CDC streaming does not seems available for postgres, does anyone know the timeline of this feature ?
Thanks a lot for you help.
Jonathan.
With Datastream for BigQuery, you can now replicate data and schema updates from operational databases directly into BigQuery.
Datastream reads and delivers every change—insert, update, and delete—from your MySQL, PostgreSQL, AlloyDB, and Oracle databases into BigQuery with minimal latency. The source database can be hosted on-premises, on Google Cloud services such as Cloud SQL or Bare Metal Solution for Oracle, or anywhere else on any cloud.
https://cloud.google.com/datastream-for-bigquery
You have to create an ETL process. That will allow you to automatically transform data from Postgres into BigQuery. You can do that using many ways, but I will point you to the two main approaches that I've already implemented:
Way 1:
Set Up the ETL Process manually:
Create your ETL using open source tools...
This method involves the use of the COPY command to migrate data from PostgreSQL tables and standard file-system files. It can be used as a normal SQL statement with SQL functions or PL/pgSQL procedures which gives a lot of flexibility to extract data as a full dump or incrementally. You need to know that it is a time-consuming process and would need you to invest in engineering bandwidth!
Also, you could try different tech stacks to implement the above, and I recommended this one Java Spring Data Flow
Way 2:
Using DataFlow
You can automate the ETL process using GCP's DataFlow without coding your own solution. It is faster and it cost, of course.
DataFlow is Unified stream and batch data processing that's
serverless, fast, and cost-effective.
Check more details and learn in a minute here
Also check this
In our current system, we have a lot of ECC tables replicated to SAP HANA with SDI (Smart Data Integration). Replication tasks can be real-time or on demand, but sometimes a replication task comes too late and the data in the replicated table is very different from the source table.
What would be the best approach in SAP HANA to check these delta values?
ERP system uses DB2 database
DB2LogReaderAdapter is used to read DB2 database tables
Remote source is created in the Cloud (Virtual table)
There are about 260 replication tasks
Replication tasks contain only one object
Replication tasks are based on virtual tables
The biggest issue faced right now is latency in the remote source tables (delta values)
There is no easy/straightforward way to "check" delta values here.
The 260 replication tasks are processed independently from each other; regardless of transactional compounding in the source system.
That means, that if table A and B are updated in the same transaction, but replicated in separate tasks to HANA, the data will be written to HANA in separate transactions. The data in HANA will be lagging behind the source system.
Usually, this difference should only last a relatively short time (maybe a few secs.), but, of course, if you do aggregation queries and want to see current valid sums etc. this leads to wrong data.
One way to deal with this is to implement the queries in a way that takes this into account, by e.g. filtering on data that has been changed half an hour ago (or longer), and to exclude newer data.
Note that as the replication via LogReader is de-coupled from the source system's transaction processing, this problem of "lagging data" is built-in conceptionally and cannot be generally avoided.
All one can do is to reduce the extend of the lag and cope with the differences in the upstream processing.
This very issue is one of the reasons for why remote data access is usually preferred over replication for cases like operational reporting.
And if you do need data-loading (e.g. to avoid additional load on the source system) then a ETL/ELT approach into data stores (DWH/BW-like) makes the situation a lot better structures.
In fact, the current S/4 HANA & BW/4 HANA setups usually use a combination of scheduled data loads and ad-hoc fetching of new data via operational delta queues from the source system.
Lars,
If we need to replicate data from ECC on Oracle to a HANA instance, should we use SLT (because of cluster tables for example) or SDI already covers all functionality SLT provides?
Regards, Chris
Is it possible to virtualize SAP HANA tables in Azure SQL Data Warehouse? If so, please provide link to the documentation or details.
We are currently using Smart Data Access to virtualize tables between HANA tenants and it works well for our scenario. However, Data Warehouse has recently been introduced to our environment and the requirement is to have the data (virtual or replicated) in there as well.
Our current workaround is a script that replicates the data, but now we are having issues with keeping the data in sync. Moreover, we would prefer not to replicate data at all if possible.
Thanks,
Mike
No, it is not possible to virtualise Hana tables from Azure SQL Data Warehouse.
SAP customers typically export data to Azure storage, then ingest to Azure SQL Data Warehouse using the Polybase feature for fast parallel ingestion.
Is there a good data integration tool available for SAP HANA which could accomplish the following :
Consuming data periodically (a user defined interval) from REST based web service (A simple URL containing XML)
Parsing the XML and extracting the data
Populating the associated table
I am aware that SQL Server Integration Service is one such tool available for Microsoft SQL Server, which does the above. Would like to know the equivalent in HANA. I did explore SAP Cloud Integration service and Business Object Data Services tool, but would like to have a first hand opinion on the same.
SAP HANA Smart Data Integration (SDI) is a standard product option that you can use for this. As it is a feature of SAP HANA, no additional server is required for this solution. This blog gives a good overview SAP SDI BLOG.
Data Services can of course also used for that, but would probably be the 'Ferrari for driving to the bakery'.
Can HANA store all the data carried by SAP BI in HANA "in-memory"?
Are there certain BI tools that are available only in SAP BI and not possible in SAP HANA due to architectural differences in nature of Datawarehouse and Database tools now and in future? If yes, what are those?
How does a SAP BI developer use HANA? Does he write HANA Sql Scripts on SAP BI along with ABAP code for BI customization?
ad 1) Yes it can, but it doesn't have to (dynamic tiering and memory displacement can be used).
ad 2) Datawarehouses and DBMS are two different things, so there are lots of differences. If your question is more directed at whether or not SAP HANA can and will replace SAP BW anytime soon, I recommend the following blog posts:
https://blogs.saphana.com/2012/06/13/does-sap-hana-replace-bw-hint-no/
https://blogs.saphana.com/2012/06/20/does-sap-hana-replace-bw-hint-no-part-2/
https://blogs.saphana.com/2014/04/17/does-sap-hana-replace-bw-hint-still-no/
https://blogs.saphana.com/2015/07/06/sap-hana-replace-bw-hint-no/
ad 3) Yes, that's the case. For example expert routines can often be scripted in SQLScript, thereby avoiding data transfer between HANA server and SAP BW application server.
Can HANA store all the data carried by SAP BI in HANA "in-memory"?
Yes it can store all the data carried by SAP BI. Normally BI will have two databases, the first database is installed by default during the installation (that might be any database e.g DB2, MS SQL Server. If you want to keep the data as HANA you have to manually select it and HANA DB has to be physically installed). This will be used for auditing or running the BI System.
Whereas the other database or second database is where the actual data resides, where we are required to do reporting in BI by taking the database as a source.
Are there certain BI tools that are available only in SAP BI and not possible in SAP HANA due to architectural differences in nature of Datawarehouse and Database tools now and in future? If yes, what are those?
HANA is actually a database, whereas BI is a few reporting tools with various flavours such as webi, crystal.
A connector is required to connect between BI and HANA, if we want to connect only the HANA DB with other tools (you can connect ones such as MS Excel).
How does a SAP BI developer use HANA? Does he write HANA Sql Scripts on SAP BI along with ABAP code for BI customization?
A BI Developer will only use the BI tool and all the features which are there in the BI reporting tool, whereas writing SQL Scripts and HANA ABAP code can be written at a backend database level depending upon the requirements by a consultant called HANA Developer, whose responsibility it is to build models at the backend level and make them available to integrate with BI and allow the HANA model to pull in the BI Reporting tool for reporting purposes.