In Hitachi Pentaho 8.2 Data Integration (Spoon) we are trying to configure "MultisubnetFailover=True" for a SQL Server database connection.
For database connections we are using "MS-SQL Server(Native)/Native JDBC". The problem is that I can't find where to set this property in the Database Connection component, and it doesn't look like I can specify the complete connection string for the driver to use. Where can I set this property?
It doesn't look like the Options tab is the correct place for setting this property, and I don't see a way to just specify the entire connection string myself. Also, ODBC can't work for us because the ETL changes the server and database dynamically (we use Pentaho variables to vary those) and ODBC is only one hardcoded connection.
Any help will be appreciate it.
Related
I'm a web developer that has been tasked with creating some sort of mechanism for moving data from an IBM AS400 to a SQL server. Unfortunately, linked servers are out of the question in this case as the SQL Server is just Standard Edition (db2 providers not available in this version) and the AS400 server is on a separate server. I've researched adding some sort of trigger on the AS400 table that calls a web service that would insert data into the SQL server, but that doesn't seem like the best method. Does anyone have any suggestions on the process to get the data from the AS400 to the SQL Server when it is committed to the AS400?
This solution assumes you are familiar with SQL Server Integration Services (SSIS):
Connection to AS400
Create a new ADO.Net connection Manager
Set the Provider to .Net Provider --> ODBC Data Provider
Create a DSN (Control Panel -->Administrative Tools-->Data Sources ODBC -->System DSN)
In the connection manager for Data source specification select the DSN created. Provide the login information.
Test the connection.
Data flow source:
Use the DataReader source
In Advanced Editor select the Ado.Net connection manager just created.
In Component Properties tab --> Custom properties, in SQLCommand specify the required query string (select * from DatabaseName.TableName)
Check the column mappings for accuracy
Go to Input and Output properties -->Data reader output -->External columns (Select the columns which were of type varchar in the table, they will now be of the datatype UnicodeString (DT_WSTR). This is because by default DataReader reads strings as unicode strings. This implies that in the destination table in SQL these columns must be of type unicode i.e NVARCHAR instead of VARCHAR)
Answer sourced from www.sqlservercentral.com/Forums
I synchronize my web applications with an IBM i. But I have my own database design and wrote a sync program on the Windows side.
Having the same database design I wonder why I would need a copy on SQL server. I would access the IBM directly. Install the drivers as #Kamran Farzami suggested and use them. That way there would be no lag between writes on the mainframe and your queries.
If a lag is acceptable for you and you can't access the IBM i directly, I see three main options:
Pull the data from your Windows system with the OLE DB driver. Using the .NET driver you can use the relative record number (RRN) to remember where you stopped synchronizing.
Read the journal files and make them available by creating a webservice on the IBM i.
Read the journal files in a scheduled job and push the changes from the journal to a webservice which updates the SQL server.
Option 1 only works if the files you sync are not reorganized. The RGZPFM command changes the record numbers. If that's okay, you can get the RRN in your SELECT statement: select *, RRN(MYTABLE) as RRNMYTABLE from MYTABLE
The web service server is included in OS400 since V5R4. So you should be able to use option 2.
I've done something similar where the SQL server was in a remote (Honduras) location where the internet connection was unreliable. It was a short VB program, using the OLE DB driver, running on the server that connected to the AS400 when it was available (or "slept" when the connection was down). When available the program would update/synchronize a uniquely keyed mirror file. Another program uploaded individual transaction records to a separate table (file).
We'd also periodically update SQL Server master tables (i.e. item master) from the AS400. That also utilized a VB program (could be any language using the driver) initiated on the server. It isn't exactly elegant, but more practical than an AS/400 trigger to a web service, I believe.
My application (a testcase automation tool) does not support calling ODBC's SQLSetStatmentOption. I need to set the cursor option SQL_SOPT_SS_CURSOR_OPTIONS to SQL_CO_FFO_AF. Right now I connect to the server using a connection string (no DSN). I am using sqlsrv32.dll or Sqlncl10.dll.
Is there a way to set the cursor option in a DSN (or other layer) between my tool and the MSSQL server? The tool handles creating the cursor (or results set) and iterating through the result set in a way that seems to obscure any ability to manage it directly. It does allow for calling SQLSetConnectAttr() but as far as I can tell there's no way to set a cursor option in there.
All of this is an attempt to speed up retrieving data over a remote connection. In SQL Management Studio I get comparable response times to the local or remote DB, but in the automation tool the remote queries take hundreds of times longer, probably because it seems to be making a round trip to the sever for every row.
There is no way to set statement options in the connection string that I know of. In ODBC 1.0 and 2.0 you could make calls to SQLSetConnectOption for statement attributes and they set those attributes in each statement created in that connection but a) that is an old version of the API (although it probably still works) b) you still cannot do it from the DSN. In unixODBC you can set some environment and connection attributes in the DSN but your obviously Windows.
I'm not convinced anyway that you've identified the problem correctly and even if you have, how do you know your app does not use SQLGetData (which is disabled when you set SQL_CO_FFO_AF). Are you sure you've not enabled MARs. Did you try getting an ODBC trace to see what the app is doing?
In my current project, the database name keeps changing and I do not want to modify my OLE DB Sources each time. Thus I want to set it as a variable. That is, in each OLE DB Source, I could use it as a parameter. However, it seems SSIS doesn't allow database name to be transferred as a parameter. So what should I do to minimize my changes?
I wish to do something like this SQL statement in OLE DB Source:
Select OrderID
From ?.Order
And I just need to modify my variable when the database name is changed.
Thanks for any solutions!
you need to add package configurations to your package and set the database name of the connection manager as one of the configurations
Just right click any blank area on the control flow and select "package configuration" and follow the wizard.
I Suggest you start with a XML configuration because its easier to understand.
A couple of suggestions for you:
http://social.msdn.microsoft.com/Forums/en-us/sqlintegrationservices/thread/586bf0a7-16c1-4a24-a9b2-7b8fb0cffda9
http://social.msdn.microsoft.com/forums/en-US/sqlintegrationservices/thread/b72e1b45-9a78-4865-912a-4587fa72c7f7
I like to manage Server\Instance.Database at the connection manager and Schema.Table or View at the task or adapter level. I recommend externalizing connection manager connection strings. You can use Package Configurations to do this, or you can manage it yourself (I use a table with ConnectionName and ConnectionString columns, and encrypt the ConnectionString column). In my opinion, a hard-coded database name inside an SSIS package is not a best practice.
I recently recorded a couple sessions on SSIS Connections management - one for 2005, 2008, and 2008 R2; another for 2012. I have also recorded a session on managing configurations in SSIS (although the audio is poor).
I hope these help.
Andy
Check out this terrific post on how to parameterize the database name. This a very thorough blog post on the topic of parameters and environments in SSIS.
http://www.sqlchick.com/entries/2015/1/4/parameterizing-connections-and-values-at-runtime-using-ssis-environment-variables
I need to connect and send/receive information from an MS SQL server in my Lotus Notes app using #formula in realtime (I can connect using an agent, but I need to use inline code for this).
The commands themselves seem pretty straight forward, but setting up the configurations seems to be a topic with scarce documentation. Apparently I need to install an ODBC driver. Where would I find that, and do I install that onto the server or onto the workstations that will run this app?
If any Lotus gurus could step me through setting this up, it would be greatly appreciated.
Thanks
You'll need to install the ODBC driver on the workstations that run this app, if the users will be triggering the ODBC connections. If at all possible, I highly suggest setting this up on the server side, and having it run via an agent. That'll save you from a few headaches, including having to maintain the ODBC connections on each workstation and worrying if each workstation has access to the data and server.
You first just want to make sure your ODBC setup is correct. You'll need the appropriate driver, of course, and the connection information. This site has a walkthrough to give you an idea of how to setup an ODBC database connection
If you have MS Access you can use it to test querying from the ODBC data source. Once you've tested the connection works, you'll just refer to the data source name (DSN) in your #DbColumn, #DbLookup, or #DbCommand formulas.
Back to my suggestion on setting this up on the server side, that would mean you'd keep a copy of the data you're querying within the Notes database itself, and then users would be interacting with read-only data in Notes. You could schedule updates regularly on the server side of that read-only data and effectively create a cache of the data in your Notes environment. Then that data would replicate around to other replicas of the database, but remove the trouble of the ODBC connection being needed everywhere.
If you need realtime data, though, that solution is out the window and you'll have to go with a local solution. In that case, you might want to look at the LCConnection class or using an ADODB.Connection from script, as both will allow you to create DSN-less connections to data sources. You'd then save the trouble of requiring ODBC data sources on each workstation, and only have to worry about whether they can access the server from their workstation.
I would add another option to Ken's list. It involves having the server do the queries of the external database (therefore you are only setting up ODBC in on the server - you don't have to deal with it on the workstations). You create an agent that is launched on the server using the 'run on server' technique. When the workstation needs to query the external data, the code creates a throw-away document in the database, puts the query criteria into the temporary document, saves the document, then calls the 'run on server' agent passing a reference to the temporary document. The server launches the agent, reads the criteria from the temporary document, does the query, and writes the results back to the temporary document. Then the workstation can access the query results from the temporary document. A scheduled agent can delete the temp docs on a regular basis.
It sounds complicated, and it all has to be done in script, but I've done this in many applications and it is fast, flexible, easy to administer, and gives your applications a lot of power. Note that end users must have the ACL rights to create a document in the db (the temp doc) in order for this to work.
Good luck!
Is there a tool for windows that we can use to inspect any SQL commands that go through a particular ODBC data source?
You can make ODBC log out everything it's doing:
http://support.microsoft.com/kb/274551
http://msdn.microsoft.com/en-us/library/ms711020%28VS.85%29.aspx
You can also do it programmatically:
... One can do this by calling SQLSetConnectAttr and set the SQL_ATTR_TRACE attribute in the connection to SQL_OPT_TRACE_ON. So, by doing this you would be enabling/disabling it for the connection duration.
http://decipherinfosys.wordpress.com/2009/01/17/odbc-tracing/
If you're using SQL Server, look at the SQL Server Profiler. Profiler allows you to monitor/trace all communications between your application and the SQL Server, including which procedures are called, parameter values, etc, without having to modify your application.
If you're using a different server, you should be able to find a sql proxy that will do the same thing.