I am trying to carry a subset of the keys of a redis database to another remote one which is not directly accessible.
That is why I am searching for a method to export the subset of the keys (like the command keys prefix* gives all the keys that start with 'prefix', something like dump prefix*). And then carry the exported file and import into the remote server.
Existing solutions that I found are asking to dump keys individually. The method I am looking for is to export keys that match a pattern (and then import them in the remote server).
You can use the SCAN command to iterate those keys. For each key, use the DUMP command to get the serialized value, and use the RESTORE command to write it to another Redis database.
Related
I have implemented storing the results of a selection from a database (list) in Redis to speed up data loading on the site. When a cms user performs any operations (creates, deletes or edits article), the keys in redis are dropped and load fresh data from the database.
But sometimes it happens that one or two users do not drop their keys after performing operations with articles and old data remains in redis. The Internet was available, nothing was turned off. Why does this happen - are there any typical reasons that I need to know?
Do I need to block the database so that there are no multiple connections? But redis seems to be single-threaded. What am I doing wrong? The function for key drop is very simple:
function articlesRedisDrop($prefix)
{
$keys = $redis->keys($prefix."*");
if(count($keys) > 0)
{
foreach($keys as $key)
{
$redis->del($key);
}
}
}
guess that an atomic question. After $redis->keys($prefix."*"), before $redis->del($key), another connection added refresh data to redis.
You can try to combine get and del operations to single lua script.
local keys = redis.call("keys",string.format("%s.*",KEYS[1]))
for _, key in pairs(keys) do
redis.call("del", key)
end
then run the script with eval command and prefix param. If you meet performance problem with keys command, you can try scan or store all prefix keys to a set then get and delete all.
Is it possible to load CSV file directly into Redis (5.X) or CSV first needs to be converted into JSON and then load it programmatically?
Depending on how you want to store the data in your CSV file, you may or may not need to process it programmatically. For example, running this:
redis-cli SET foo "$(cat myfile.csv)"
will result with the contents of the files stored in the key 'foo' as a Redis String. If you want to store each line in its own data structure under a key (perhaps a Hash with all the columns), you'll need to process it with code and populate the database accordingly.
Note: there is no need, however, to convert it to JSON.
What is the way to migrate the data from DB 0 to DB 1 in the same instance of Redis?
I have found many resources about migrating from one instance to another (SAVE or replication), but not from one index to another; maybe I don't use the correct search keywords…
The commands COPY and MOVE let's you copy/move keys from one DB to another.
My problem statement is that I have a csv blob and I need to import that blob into a sql table. Is there an utility to do that?
I was thinking of one approach, that first to copy blob to on-premise sql server using AzCopy utility and then import that file in sql table using bcp utility. Is this the right approach? and I am looking for 1-step solution to copy blob to sql table.
Regarding your question about the availability of a utility which will import data from blob storage to a SQL Server, AFAIK there's none. You would need to write one.
Your approach seems OK to me. Though you may want to write a batch file or something like that to automate the whole process. In this batch file, you would first download the file on your computer and the run the BCP utility to import the CSV in SQL Server. Other alternatives to writing batch file are:
Do this thing completely in PowerShell.
Write some C# code which makes use of storage client library to download the blob and once the blob is downloaded, start the BCP process in your code.
To pull a blob file into an Azure SQL Server, you can use this example syntax (this actually works, I use it):
BULK INSERT MyTable
FROM 'container/folder/folder/file'
WITH ( DATA_SOURCE = 'ds_blob',BATCHSIZE=10000,FIRSTROW=2);
MyTable has to have identical columns (or it can be a view against a table that yields identical columns)
In this example, ds_blob is an external data source which needs to be created beforehand (https://learn.microsoft.com/en-us/sql/t-sql/statements/create-external-data-source-transact-sql)
The external data source needs to use a database contained credential, which uses an SAS key which you need to generate beforehand from blob storage https://learn.microsoft.com/en-us/sql/t-sql/statements/create-database-scoped-credential-transact-sql)
The only downside to this mehod is that you have to know the filename beforehand - there's no way to enumerate them from inside SQL Server.
I get around this by running powershell inside Azure Automation that enumerates blobds and writes them into a queue table beforehand
I am trying to export and import Vertica schema from one physical cluster to another physical cluster.
My Test instance has one single cluster and my production instance has 3 clusters.
I explored following options, but they are limited to moving data on one physical Vertica instance:
EXPORT TO VERTICA ..
COPY schema.table FROM VERTICA ...
Would like to know if there is an option to move the Vertica schema from one physical Vertica instance to another, with different cluster configuration.
This is tricky manipulation, which have many issues:
If you copy over DDLs, you will lose the current value of sequences, which might mean duplicate primary key when you insert data.
If columns are set up as AUTO_INCREMENT, you will not be able to insert data in it as it is on the source (you cannot force an auto_increment column, although I believe this might have been fixed in new releases).
If you copy DDLs between clusters with a different number of nodes, if node names are part of projection definition, you will end up with something you do not want.
As you noticed, different networks will prevent the use of CONNECT.
An attempt to help out with this has been done in python via the pyvertica utility, and specially the vertica_migrate script. You can find the doc at https://pyvertica.readthedocs.org .
This is a tricky job, and I know there are some issues in this script, although it already helped me a lot.
Hope this helped,
You can use either COPY FROM VERTICA or EXPORT TO VERTICA to import/export the data to another Vertica database (regardless of node configuration). Also, the target table must already exist. You can use EXPORT_OBJECTS to export the DDL. Both methods allow for data migration from a version that's an earlier release from the last major release (running 6.x, you can import from 5.x).
In the example below, I'll use EXPORT TO VERTICA to export data from the source database to the target database.
You must first create a connection to the other database:
CONNECT TO VERTICA VMart USER dbadmin PASSWORD '' ON 'VerticaTarget',5433;
Then use EXPORT TO VERTICA to export the data from the source to the target database:
EXPORT TO VERTICA VMart.schema.customer_dimension FROM schema.customer_dimension;
|______________________________| |_______________________|
| |
Target Source
DISCONNECT VMart;