I am in a process to copy data from one SQL database (Source) and move to another SQL Database (destination) through data factory using c# code.
I am able to copy all the data from a source table to destination table, but i want to move filtered data only, like SELECT * FROM Source.tbl WHERE Category = 5. There would be around 10-15 table that i would move data. Can you provide me sample code which may help me?
My code for moving single table all data..
// Authenticate and create a data factory management client
var context = new AuthenticationContext("https://login.windows.net/" + tenantID);
ClientCredential cc = new ClientCredential(AppID, AuthKey);
AuthenticationResult result = context.AcquireTokenAsync("https://management.azure.com/", cc).Result;
ServiceClientCredentials cred = new TokenCredentials(result.AccessToken);
var client = new DataFactoryManagementClient(cred) { SubscriptionId = SubscriptionID };
// Create data factory
Factory dataFactory = new Factory { Location = Region, Identity = new FactoryIdentity() };
// This line throws error, we cannot proceed further. unless we get access of creating DF or update or access.
client.Factories.CreateOrUpdate(ResourceGroup, DataFactoryName, dataFactory);
var DF = client.Factories.Get(ResourceGroup, DataFactoryName);
while (DF.ProvisioningState == "PendingCreation")
{
System.Threading.Thread.Sleep(1000);
}
LinkedServiceResource storageLinkedService = new LinkedServiceResource(
new AzureSqlDatabaseLinkedService
{
ConnectionString = new SecureString(SourceSQLConnString)
}
);
client.LinkedServices.CreateOrUpdate(ResourceGroup, DataFactoryName, SourceSQLLinkedServiceName, storageLinkedService);
LinkedServiceResource sqlDbLinkedService = new LinkedServiceResource(
new AzureSqlDatabaseLinkedService
{
ConnectionString = new SecureString(DestSQLConnString)
}
);
client.LinkedServices.CreateOrUpdate(ResourceGroup, DataFactoryName, DestSQLLinkedServiceName, sqlDbLinkedService);
DatasetResource SourceSQLDataSet = new DatasetResource(
new AzureSqlTableDataset
{
LinkedServiceName = new LinkedServiceReference
{
ReferenceName = SourceSQLLinkedServiceName
},
TableName = Table,
}
);
client.Datasets.CreateOrUpdate(ResourceGroup, DataFactoryName, SourceSQLDataSetName, SourceSQLDataSet);
// Create a Azure SQL Database dataset
DatasetResource DestSQLDataSet = new DatasetResource(
new AzureSqlTableDataset
{
LinkedServiceName = new LinkedServiceReference
{
ReferenceName = DestSQLLinkedServiceName
},
TableName = Table
}
);
client.Datasets.CreateOrUpdate(ResourceGroup, DataFactoryName, DestSQLDataSetName, DestSQLDataSet);
PipelineResource pipeline = new PipelineResource
{
Activities = new List<Activity>
{
new CopyActivity
{
Name = "CopyFromSQLToSQL",
Inputs = new List<DatasetReference>
{
new DatasetReference()
{
ReferenceName = SourceSQLDataSetName
}
},
Outputs = new List<DatasetReference>
{
new DatasetReference
{
ReferenceName = DestSQLDataSetName
}
},
Source = new SqlSource(),
Sink = new SqlSink { }
}
}
};
client.Pipelines.CreateOrUpdate(ResourceGroup, DataFactoryName, PipelineName, pipeline);
// Create a pipeline run
CreateRunResponse runResponse = client.Pipelines.CreateRunWithHttpMessagesAsync(ResourceGroup, DataFactoryName, PipelineName).Result.Body;
// Monitor the pipeline run
PipelineRun pipelineRun;
while (true)
{
pipelineRun = client.PipelineRuns.Get(ResourceGroup, DataFactoryName, runResponse.RunId);
if (pipelineRun.Status == "InProgress")
System.Threading.Thread.Sleep(15000);
else
break;
}
You could put your query into the SqlReaderQuery property of your sql Source.
I talked to the Data Factory support, they said we have not implemented yet,
Create Data Factory,
Create linked services
In loop create datasets and Create copy activity
Related
I am making an API using ASP.netCore to fetch data from teradata. I have 2 store procedure on Teradata DB one for encryption one for description. What I want to achieve is when user call the get method to fetch data, my Store procedure should run first and then it should return all table to user(changed as SP).
I have tried a lot but didn't manage to find any way to call store procedure and Select * table together. If I run the only store procedure OR call only select * table it work. I want to call both (SP and select*) so first SP should run and change the data then return the changed data table.
My codes are following.
Codes for GetDataTable
public IEnumerable<myTable> GetData(IConfiguration _configuration)
{
try
{
myTable TableReturn = new myTable();
List<myTable> dataReturn = new List<myTable>();
var connectionString = _configuration.GetConnectionString("TeradataConnectionString");
using (var connection = new TdConnection(connectionString))
{
'Date_Of_Birth', ENCRYPTEDSTRING);";
TdCommand cmd = new TdCommand("SELECT * FROM Project.myTable;", connection) ;
TdCommand Db = (TdCommand)cmd;
Db.Connection = connection;
connection.Open();
TdDataReader reader = Db.ExecuteReader();
if (reader.HasRows)
{
while (reader.Read())
{
var newTable = new myTable();
newTable.Employee_Id = (string)reader.GetString(0);
newTable.First_name = (string)reader.GetString(1);
newTable.Last_Name = (string)reader.GetString(2);
newTable.Date_Of_Birth = (string)reader.GetString(3);
newTable.Phone = (string)reader.GetString(4);
newTable.Em_Position = (string)reader.GetString(5);
dataReturn.Add(newTable);
}
}
else
{
Console.WriteLine("no rows found");
}
}
return dataReturn;
}
catch (Exception e)
{
throw new Exception();
}
}
My codes for store procedure are following
public IEnumerable<DecryptedData> DecryptedData(IConfiguration _configuration)
{
// DecryptedData DecryptedReturn = new DecryptedData();
List<DecryptedData> dataReturn = new List<DecryptedData>();
var connectionString = _configuration.GetConnectionString("TeradataConnectionString");
using (var connection = new TdConnection(connectionString))
{
string command = "CALL Project.Decryption(STATUS,'Project' ,'myTable', 'Date_Of_Birth', ENCRYPTEDSTRING);";
TdCommand cmd = new TdCommand(command, connection);
TdCommand Db = (TdCommand)cmd;
Db.Connection = connection;
connection.Open();
TdDataReader reader = Db.ExecuteReader();
if (reader.HasRows)
{
while (reader.Read())
{
var DecryptedData = new DecryptedData();
DecryptedData.Date_Of_Birth = reader.GetString(1);
dataReturn.Add(DecryptedData);
Console.WriteLine(dataReturn);
}
}
else
{
Console.WriteLine("no rows found");
}
}
return dataReturn;
}
I cannot get OnBeforeStore to fire using a BulkInsert.
It fires fine during a regular Store operation.
I'm trying to use an Invoice number generator to add a formatted number and I'd like to do that in OnBeforeStore.
See code example:
static async Task GenerateInvoiceTest()
{
using var store = new DocumentStore
{
Urls = new string[] { "https://localhost:8080" },
Database = "APC",
};
//this never fires using BulkInsert
store.OnBeforeStore += (s, e) =>
{
if (!(e.Entity is Invoice invoice)) return;
if (invoice.InvoiceNumber != 0) return;
invoice.InvoiceNumber = new Random().Next(1, 1000);
};
store.Initialize();
//sample invoices
var invoices = new List<Invoice>
{
new Invoice { StartDate = DateTime.Now, EndDate = DateTime.Now.AddDays(3) },
new Invoice { StartDate = DateTime.Now, EndDate = DateTime.Now.AddDays(3) },
};
//bulk insert test
using var session = store.OpenAsyncSession();
using var bulkInsert = store.BulkInsert();
invoices.ForEach(i => bulkInsert.Store(i));
//this does NOT fire OnBeforeStore
await session.SaveChangesAsync();
foreach (var invoice in invoices)
{
//always prints 0
Console.WriteLine(invoice.InvoiceNumber);
}
//regular test
var otherInvoice = new Invoice { StartDate = DateTime.Now, EndDate = DateTime.Now.AddDays(3) };
await session.StoreAsync(otherInvoice);
//this DOES fire OnBeforeStore
await session.SaveChangesAsync();
}
OnBeforeStore is invoked as part of the Session SaveChanges method
See this documentation about OnBeforeStore
http://localhost:54391/docs/article-page/5.0/Csharp/client-api/session/how-to/subscribe-to-events
The event takes argument BeforeStoreEventArgs that consists of the Session entity's ID and the entity itself.
You define OnBeforeStore on the 'Store' but it is Not for use with bulkInsert.
It is for when saving from a Session.
BulkInsert operates on the Store itself, not on the Session
I try to create memory index with following code but it creates regular index.
Any idea?
var node = new Uri("http://localhost:9200");
var settings = new ConnectionSettings(node);
var client = new Elasticsearch.Net.ElasticsearchClient(settings);
var js = JsonConvert.SerializeObject("{settings: { index.store.type: memory}");
var index = client.IndicesCreate("sampleIndex", js);
Your second argument to the IndicesCreate method call is not correct. See the code below for a better way to achieve it. The first three lines are the same. In the fourth one, we properly create the settings for the index.
_body = new {
settings = new {
index = new {
store = new {
type = "memory"
}
}
}
};
var index = client.IndicesCreate("sampleIndex", _body);
If you have for example this code:
var db = new NotesDatabase("https://address.com/mobile.nsf", "Database");
var view = db.getView("PolicyData");
var vec = view.getAllEntries();
if(!vec) {
alert("nothing in view");
return;
}
var ve = vec.getFirstEntry();
it will fail because vec is null. Why?
The reason is that you need to synchronize the view with the Domino backend first, so that it's data is in the the local SQLite database of the device.
This would work:
var db = new NotesDatabase("https://address.com/mobile.nsf", "Database");
var view = db.getView("PolicyData");
var callback = function() {
var vec = view.getAllEntries();
if(!vec) {
alert("nothing in view");
return;
}
var ve = vec.getFirstEntry();
}
view.update(callback);
See http://www.youatnotes.com/web/youatnotes/wiki-dtg.nsf/dx/NotesView#Methods for more details about the update method.
I am trying to query on both Release and Iteration so I can fill out a drop down list with these various values. I'm not quite sure how to do this, however. What are the members of the object that come back via the query if we are able to do this? (Name, FormattedID, CreationDate, etc). Do we just create a new request of type "Release" and "Iteration" ?
Thanks!
Here is a code that queries on releases based on a project reference. If this project is not in a default workspace of the user that runs the code we either need to hardcode the workspace reference or get it from the project.
class Program
{
static void Main(string[] args)
{
RallyRestApi restApi;
restApi = new RallyRestApi("user#co.com", "TopSecret1984", "https://rally1.rallydev.com", "1.40");
var projectRef = "/project/22222222"; //use your project OID
DynamicJsonObject itemWorkspace = restApi.GetByReference(projectRef, "Workspace");
var workspaceRef = itemWorkspace["Workspace"]["_ref"];
Dictionary<string, string> result = new Dictionary<string, string>();
try
{
Request request = new Request("Release");
request.ProjectScopeDown = false;
request.ProjectScopeUp = false;
request.Workspace = workspaceRef;
request.Fetch = new List<string>()
{
"Name"
};
// request.Query = new Query("Project.ObjectID", Query.Operator.Equals, "22222222"); //also works
request.Query = new Query("Project", Query.Operator.Equals, projectRef);
QueryResult queryResult = restApi.Query(request);
foreach (var r in queryResult.Results)
{
Console.WriteLine("Name: " + r["Name"]);
}
}
catch
{
Console.WriteLine("problem!");
}
}
}
}