I have access to a system that if you enter in a url into a browser, it will automatically download a .csv file. The web address is formatted like below
https://secure.company.com/csv.cgi?user=username;password=password
What I would like to do is somehow get MS SQL Server 2012 to do all of the download and importing into a table automatically.
Is that even possible?
As a noob guess would it be done via a stored procedure?
If so how would one code such a thing?
How about using something like wget to download the file, then check out this blog post for how to import the CSV file into the database.
Even though there might be a way to send http request using SQL Server, using SQL queries, I’d strongly recommend you do this from C#. You can either develop a small application or use a CLR stored procedure for this.
Just update YourTable and download path and you should be fine.
protected const string FILE_NAME = #"C:\tablevalues.csv";
protected const string SQL_INSERT = "BULK INSERT YourTable FROM {0} WITH ( FIELDTERMINATOR = ',', ROWTERMINATOR = '\n')";
protected void DownloadFile()
{
WebClient webClient = new WebClient();
webClient.DownloadFile("https://secure.company.com/csv.cgi?user=username;password=password", FILE_NAME);
SqlConnection sqlConnection = new SqlConnection("connection string");
SqlCommand sqlCommand = new SqlCommand(string.Format(SQL_INSERT, FILE_NAME), sqlConnection);
sqlConnection.Open();
sqlCommand.ExecuteNonQuery();
sqlConnection.Close();
sqlConnection.Dispose();
}
Related
hello I create an intranet page with MVC, also I want to connect an ORA-SQL database.
my idea was to create a folder (Query) [in my Project] in which my SQL files are stored:
- DevStats.sql
inside the query.
Now I want to start a query:
OracleCommand command = connection.CreateCommand ();
command.CommandText = query;
What is the most efficient way to read the file (StreamReader?)? And where can I find the file then (relative path)?
using (StreamReader sr = new StreamReader(#"Query\DevStats.sql", Encoding.Default))
{
return sr.ReadToEnd();
}
is null
#"\Query\DevStats.sql"
is also null
thanks
You can use HttpServerUtility.MapPath() to get physical file path that corresponds to the specified virtual path.
Simple Code could be like
HttpServerUtility.MapPath("/Query/DevStats.sql");
HttpServerUtility.MapPath
As I know SQL Server since version 2012 has a new feature, FileTable. It allows us to store files in the file system and to use them from T-SQL.
I am trying to use this feature and I have no idea how to do it properly.
Generally, I don't know how to access files stored in the file table. Let's suppose I have asp.net MVC app and there are a lot of images which I show on web pages in img tags. I would like to store these images in Filetable and access them as files from the filesystem. But I don't know where these files are stored and how to use them as files. Now my images are stored in web application directory in folder images and I write something like this:
<img src='/images/mypicture.png' />
And if I move my images to file table what I should write in src?
<img src='path-toimage-in-filetable' />
I don't think you still need this, anyways I'll post my answer for anyone else interested.
First, a filetable still being a table, so, if you want to access to data from it you need to use a Select SQL statement. So you'd need something like:
select name, file_stream from filetable_name
where
name = 'file_name',
file_type = 'file_extension'
just execute an statement like this in your asp.net app, then fetch the results and use the file_stream column to get the binary data of the stored file. If you want to retrieve the file from HTML, first you need to create an action in your controller, which will return the retrieved file:
public ActionResult GetFile(){
..
return File(file.file_stream,file.file_type);
}
After this, put in you HTML tag something like:
<img src="/controller/GetFile" />
hope this could help!
If you want to know the schema of a filetable see
here
I assume by FileTable you actually mean FileStream. A couple notes about that:
This feature is best used if your files are actually files
The files should be, on average, greater than 1mb - although there can be exceptions to this rule, if they're smaller than 1mb on average, you may be better off using a VARBINARY(MAX) or XML data type as appropriate. If your images are very small on average (only a few KB), consider using a VARBINARY(MAX) column.
Accessing these files will require an open transaction and that the database is properly configured for FILESTREAM
You can get some significant advantages bypassing the normal SQL engine/database file method of data access by telling SQL Server that you want to access the file directly, however it's not meant for directly accessing the file on the file system and attempting to do so can break SQL's management of these files (transactional consistency, tracking, locking, etc.).
It's pretty likely that your use case here would be better served by using a CDN and storing image URLs in the table if you really need SQL for this. You can use FILESTREAM to do this (see code sample below for one implementation), but you'll be hammering your SQL server for every request unless you store the images somewhere else anyway that the browser can properly cache (my example doesn't do that) - and if you store them somewhere else for rendering int he browser you might as well store them there to begin with (you won't have transactional consistency for those images once they're copied to some other drive/disk/location anyway).
With all that said, here's an example of how you'd access the FILESTREAM data using ADO.NET:
public static string connectionString = ...; // get your connection string from encrypted config
// assumes your FILESTREAM data column is called Img in a table called ImageTable
const string sql = #"
SELECT
Img.PathName(),
GET_FILESTREAM_TRANSACTION_CONTEXT()
FROM ImageTagble
WHERE ImageId = #id";
public string RetreiveImage(int id)
{
string serverPath;
byte[] txnToken;
string base64ImageData = null;
using (var ts = new TransactionScope())
{
using (var conn = new SqlConnection(connectionString))
{
conn.Open();
using (SqlCommand cmd = new SqlCommand(sql, conn))
{
cmd.Parameters.Add("#id", SqlDbType.Int).Value = id;
using (SqlDataReader rdr = cmd.ExecuteReader())
{
rdr.Read();
serverPath = rdr.GetSqlString(0).Value;
txnToken = rdr.GetSqlBinary(1).Value;
}
}
using (var sfs = new SqlFileStream(serverPath, txnToken, FileAccess.Read))
{
// sfs will now work basically like a FileStream. You can either copy it locally or return it as a base64 encoded string
using (var ms = new MemoryStream())
{
sfs.CopyTo(ms);
base64ImageData = Convert.ToBase64String(ms.ToArray());
}
}
}
ts.Complete();
// assume this is PNG image data, replace PNG with JPG etc. as appropraite. Might store in table if it will vary...
return "data:img/png;base64," + base64ImageData;
}
}
Obviously, if you have lots of images to handle like this this is not an ideal method - don't try to make an instance of SQL server into what you should be using a CDN for.... However, if you have other really good reasons, you should try to grab as many images as possible in a single request/transaction (e.g. if you know you're displaying 50 images on a page, get all 50 with a single transaction scope, don't use 50 transaction scopes - this code won't handle that).
I'm just learning to use SQLite and I was curious if such is possible:
Encryption of the database file?
Password protect opening of the database?
PS. I know that there is this "SQLite Encryption Extension (SEE).", but according to the documentation, "The SEE is licensed software...." and "The cost of a perpetual source code license for SEE is US $2000."
SQLite has hooks built-in for encryption which are not used in the normal distribution, but here are a few implementations I know of:
SEE - The official implementation.
wxSQLite - A wxWidgets style C++ wrapper that also implements SQLite's encryption.
SQLCipher - Uses openSSL's libcrypto to implement.
SQLiteCrypt - Custom implementation, modified API.
botansqlite3 - botansqlite3 is an encryption codec for SQLite3 that can use any algorithms in Botan for encryption.
sqleet - another encryption implementation, using ChaCha20/Poly1305 primitives. Note that wxSQLite mentioned above can use this as a crypto provider.
The SEE and SQLiteCrypt require the purchase of a license.
Disclosure: I created botansqlite3.
You can password protect SQLite3 DB.
For the first time before doing any operations, set password as follows.
SQLiteConnection conn = new SQLiteConnection("Data Source=MyDatabase.sqlite;Version=3;");
conn.SetPassword("password");
conn.open();
then next time you can access it like
conn = new SQLiteConnection("Data Source=MyDatabase.sqlite;Version=3;Password=password;");
conn.Open();
This wont allow any GUI editor to view Your data.
Later if you wish to change the password, use conn.ChangePassword("new_password");
To reset or remove password, use conn.ChangePassword(String.Empty);
The .net library System.Data.SQLite also provides for encryption.
You can get sqlite3.dll file with encryption support from http://system.data.sqlite.org/.
1 - Go to http://system.data.sqlite.org/index.html/doc/trunk/www/downloads.wiki and download one of the packages. .NET version is irrelevant here.
2 - Extract SQLite.Interop.dll from package and rename it to sqlite3.dll. This DLL supports encryption via plaintext passwords or encryption keys.
The mentioned file is native and does NOT require .NET framework. It might need Visual C++ Runtime depending on the package you have downloaded.
UPDATE
This is the package that I've downloaded for 32-bit development: http://system.data.sqlite.org/blobs/1.0.94.0/sqlite-netFx40-static-binary-Win32-2010-1.0.94.0.zip
Keep in mind, the following is not intended to be a substitute for a proper security solution.
After playing around with this for four days, I've put together a solution using only the open source System.Data.SQLite package from NuGet. I don't know how much protection this provides. I'm only using it for my own course of study. This will create the DB, encrypt it, create a table, and add data.
using System.Data.SQLite;
namespace EncryptDB
{
class Program
{
static void Main(string[] args)
{
string connectionString = #"C:\Programming\sqlite3\db.db";
string passwordString = "password";
byte[] passwordBytes = GetBytes(passwordString);
SQLiteConnection.CreateFile(connectionString);
SQLiteConnection conn = new SQLiteConnection("Data Source=" + connectionString + ";Version=3;");
conn.SetPassword(passwordBytes);
conn.Open();
SQLiteCommand sqlCmd = new SQLiteCommand("CREATE TABLE data(filename TEXT, filepath TEXT, filelength INTEGER, directory TEXT)", conn);
sqlCmd.ExecuteNonQuery();
sqlCmd = new SQLiteCommand("INSERT INTO data VALUES('name', 'path', 200, 'dir')", conn);
sqlCmd.ExecuteNonQuery();
conn.Close();
}
static byte[] GetBytes(string str)
{
byte[] bytes = new byte[str.Length * sizeof(char)];
bytes = System.Text.Encoding.Default.GetBytes(str);
return bytes;
}
}
}
Optionally, you can remove conn.SetPassword(passwordBytes);, and replace it with conn.ChangePassword("password"); which needs to be placed after conn.Open(); instead of before. Then you won't need the GetBytes method.
To decrypt, it's just a matter of putting the password in your connection string before the call to open.
string filename = #"C:\Programming\sqlite3\db.db";
string passwordString = "password";
SQLiteConnection conn = new SQLiteConnection("Data Source=" + filename + ";Version=3;Password=" + passwordString + ";");
conn.Open();
You can always encrypt data on the client side. Please note that not all of the data have to be encrypted because it has a performance issue.
You can use SQLite's function creation routines (PHP manual):
$db_obj->sqliteCreateFunction('Encrypt', 'MyEncryptFunction', 2);
$db_obj->sqliteCreateFunction('Decrypt', 'MyDecryptFunction', 2);
When inserting data, you can use the encryption function directly and INSERT the encrypted data or you can use the custom function and pass unencrypted data:
$insert_obj = $db_obj->prepare('INSERT INTO table (Clear, Encrypted) ' .
'VALUES (:clear, Encrypt(:data, "' . $passwordhash_str . '"))');
When retrieving data, you can also use SQL search functionality:
$select_obj = $db_obj->prepare('SELECT Clear, ' .
'Decrypt(Encrypted, "' . $passwordhash_str . '") AS PlainText FROM table ' .
'WHERE PlainText LIKE :searchterm');
Well, SEE is expensive. However SQLite has interface built-in for encryption (Pager). This means, that on top of existing code one can easily develop some encryption mechanism, does not have to be AES. Anything really.
Please see my post here: https://stackoverflow.com/a/49161716/9418360
You need to define SQLITE_HAS_CODEC=1 to enable Pager encryption. Sample code below (original SQLite source):
#ifdef SQLITE_HAS_CODEC
/*
** This function is called by the wal module when writing page content
** into the log file.
**
** This function returns a pointer to a buffer containing the encrypted
** page content. If a malloc fails, this function may return NULL.
*/
SQLITE_PRIVATE void *sqlite3PagerCodec(PgHdr *pPg){
void *aData = 0;
CODEC2(pPg->pPager, pPg->pData, pPg->pgno, 6, return 0, aData);
return aData;
}
#endif
There is a commercial version in C language for SQLite encryption using AES256 - it can also work with PHP, but it needs to be compiled with PHP and SQLite extension. It de/encrypts SQLite database file on the fly, file contents are always encrypted. Very useful.
http://www.iqx7.com/products/sqlite-encryption
I had also similar problem. Needed to store sensitive data in simple database (SQLite was the perfect choice except security). Finally I have placed database file on TrueCrypt encrypted valume.
Additional console app mounts temporary drive using TrueCrypt CLI and then starts the database application. Waits until the database application exits and then dismounts the drive again.
Maybe not suitable solution in all scenarios but for me working well ...
i want to view the test.db file, i search for it's editor but didn't get any one
So please help to see the it in editor as like sql server.
i found some sqlite editor but it's not an sqlite file on most forum it say that it is an paradox .db file.
So how do i open it
Thanks
To access Paradox tables in .NET you can use ODBC. Here's a small example (in C#):
private static void RunMinimumParadoxTest()
{
const string ConnectionStringFormat =
"Driver={{Microsoft Paradox Driver (*.db )}};Uid={0};UserCommitSync=Yes;Threads=3;SafeTransactions=0;" +
"ParadoxUserName={0};ParadoxNetStyle=4.x;ParadoxNetPath={1};PageTimeout=5;MaxScanRows=8;" +
"MaxBufferSize=65535;DriverID=538;Fil=Paradox 7.X;DefaultDir={2};Dbq={2};CollatingSequence={3}";
DbProviderFactory factory = DbProviderFactories.GetFactory("System.Data.Odbc");
using (DbConnection connection = factory.CreateConnection())
{
string userName = "Tor";
string paradoxNetPath = #"C:\BdeNet";
string databasePath = #"C:\LangloMainSrv\LData\Ordre\LordWin\Database2011";
string collatingSequence = "Norwegian-Danish";
connection.ConnectionString =
String.Format(ConnectionStringFormat, userName, paradoxNetPath, databasePath, collatingSequence);
connection.Open();
using (DbCommand command = connection.CreateCommand())
{
command.CommandText = "select Count(*) from [OrdreDet] where [Ordrenr] = 81699002";
object itemCount = command.ExecuteScalar();
Console.WriteLine("Order items: {0}", itemCount);
Console.ReadKey();
}
}
}
Also see the following link for more details: http://msdn.microsoft.com/en-us/library/ms710922(VS.85).aspx.
A Paradox db file contains just one flat table. The actual structure of the DB file changed over time and different versions. But you can usually open the DB file with MS Excel - of course that changed over different versions too.
As noted above, other database applications, also including Paradox for Dos and Paradox for Windows, will open the file and other features as well. The key, for example is in the PX file with the same table name.
All of this assumes the table is not password protected, which an application database could be - or that you know the password. Beware if you get an error to that effect.
You can open and view Paradox database files using Database Desktop that is shipped with Borland C++Builder. A free alternative is BB's Database Desktop. The software may require administrator privileges to run correctly.
You can use gnumeric spreadsheet, paradox-db-reader or BB database desktop to read db paradox file.
BB database dekstop able to read XG0 file too.
BB's Database Desktop now called JEDI Database Desktop, but project is closed and it couldn't edit my table. I have had to use some hack: open *.db file in MS Excel 2007, edit it, export to *.csv, close file then Open *.db file in Paradox Data Editor 3.2.0, clear all table data and import previosly saved csv-file. And it works (don't know why but this app can't insert row in my file itself)!
The following piece of code works with regular SQL and SMO. I'm trying to get it to work with SQL Azure. According to this MSDN article, a limited subset of functionality that I need (database and login creation) should be supported. All the business checking whether an object exists will also fail: server.Logins[loginName] != null or server.Databases.Contains(dbName). I can create a database if I dont check whether it exists or not, but i cant create a login. Anyone else ran into the same problem?
string connectionString =
"Server=tcp:XXXXXX.database.windows.net;Database=MyDatabase;User ID=XXXXXXX;Password=XXXXXX;Trusted_Connection=False;Encrypt=True;TrustServerCertificate=true;"
using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.Open();
ServerConnection serverConnection = new ServerConnection(connection);
Server server = new Server(serverConnection);
Login login = new Login(server, "NewLogin");
login.LoginType = LoginType.SqlLogin;
login.Create("NewStrongPwd123***");
}
Create failed for Login 'NewLogin'.
at Microsoft.SqlServer.Management.Smo.SqlSmoObject.CreateImpl()
at Microsoft.SqlServer.Management.Smo.Login.Create(SecureString password)
at Microsoft.SqlServer.Management.Smo.Login.Create(String password)
Proposed answers to this question were identified on the MSDN Forum including a working approach. Please take a look at: http://social.msdn.microsoft.com/Forums/en-US/ssdsgetstarted/thread/26e42082-e649-4cde-916d-c1da2275e377