I want to save the addresses of some 100,000 people in Azure SQL Database. The schema for the address table will look like this.
The size of each table field is 250 NVARCHAR.
First Name
Middle Name
Last Name
Email
Phone 1
Phone 2
Phone 3
Fax 1
Twitter
Facebook
LinkedIn
Address 1
Address 2
City
State
Country
How much GB of storage I need to store 100,000 addresses and the additional space required by the SQL Server?
Does Microsoft Azure charge for accessing the server to retrieve or save data per call?
The Azure Active Directory B2C is suitable for managing large amount of people and their data.
Related
My company uses the metabase for data analysis. The data I need to build the dashboard on the metabase is divided into 2, part of the data is retrieved from the SQL querying on the metabase and the other part is using google sheets as manual data. How can I join the data of the metabase and google sheet to create the dashboard on the metabase.
For example:
The data I need to build the dashboard on the metabase:
Name Age Adress Salary
Smith 25 Evans Mills $9000
The data is retrieved from the SQL querying on the metabase:
Name Age Adress
Smith 25 Evans Mills
Manual data on google sheets:
Salary
$9000
As far as my understanding of metabase, one of its limit is that it can not run queries across different databases.
However, I have helped our customer solve similar problems like yours. The software architecture is like this:
Metabase -> Presto SQL/Trino -> different database and different data source
In this design:
Metabase handles the dashboard part of work.
Trino handles the joining across different data sources.
Note: In our customer's case, the integration really requires certain programming work. It is not a quite trivial job.
I have a segmentation project I am working on for my company and we have to create a pipeline to gather data from our app users and when they fit a segment then the app will receive that information and do something with it (not in my scope). So currently, the client connects and authenticates to an endpoint that allows their client to send JSON data to an Elasticsearch cluster (app started, level completed, etc). I'm then using an Azure Function to grab the live data every 5 minutes and store it in an Azure Blob Storage which then creates a queue that Snowflake reads and ingests the JSON files. We'd then use Snowflake to run a task per segment (that will be decided by the analysts or executives) and the data will be outputted to a table like the one below:
AccountID
Game
SegmentID
CreatedAt
DeletedAt
123456789
Game 1
1
2021-04-20
2021-04-21
123456789
Game 1
2
2021-04-20
123456789
Game 1
3
2021-04-20
Where SegmentID can represent something like
SegmentID
SegmentType
SegmentDescription
1
5 Day Streak
User played for 5 consecutive days
2
10 Day Streak
User played for 10 consecutive days
3
15 Day Streak
User played for 15 consecutive days
In the next step of the pipeline, the same API the user authenticated with should post a request when the game boots up to grab all the segments that the user matches. The dev team will then decide where, when in the session and how to use the information to personalize content. Something like:
select
SegmentID
from
SegmentTable
where
AccountID='{AccountID the App authenticated with}' and
Game='{Game the App authenticated with}' and
DeletedAt is null
Response:
SegmentID
2
3
Serialised:
{"SegmentID": [2,3]}
We expect to have about 300K-500K users per day. My question would be, what would be the most efficient and cost-effective way to get this information from Snowflake back to the client so that this amount of users wouldn't have issues when querying the same endpoint and it won't be costly.
OK, so a bit of a workaround, but I created an external function on Snowflake (using Azure Functions) that upserts data in a local MongoDB cluster. So the API connects to the MongoDB instance which can handle the large volume of concurrent connections and since it is on a local server it is quite cheap. The only cost is the data transfer from Snowflake to MongoDB and the running of the App Service Plan on Azure Functions (could not use consumption-based as to send data to our internal server I needed to create a VNET, NAT Gateway and a Static Outbound IP Address in Azure) and the API Management Service I had to create in Azure.
So how it works? For each stored procedure in Snowflake, at the end I am collecting the segments which have changed (New row or DELETED_AT not null) and triggering the external function which upserts the data in MongoDB using the pymongo client.
Persons: theoretically, unlimited number of persons can be registered in the system. Contact information linked to each person should also be able to be added without limits.
UUID
Name
Surname
Company
Communication information:
Information Type: Telephone Number, E-mail Address, Location
Information Content
Report:
Location information
Most -> Least to be listed by their number of locations
The number of people registered in the directory at that location
Number of phone numbers registered in the phone book at that location
I have 2 tables, how should the relationship be between them and I am confused since the Communication information attribute has 2 attributes in itself. Should communication information be in a separate table?
I would suggest to divide this into three tables:
Persons(UUID,Name,Surname,Company)
CommunicationType(Telephone Number, E-mail Address,)
CommunicationInformation(Person_ID,CommunicationType_ID, ......)
I am currently working on a development project for which I need to make a database (I will be using Postgre).
Part of that database must store a phonebook for every company, and also for every user. Those two "types" of phonebook are stored in two tables, although they contain the same type of information :
PersonalPhoneBook(IdOwner, IdUser)
//Ids of the owner and a user he's in contact with.
CompanyPhoneBook(IdCompany, IdUser)
//Ids of the company and a user it's in contact with
The personal phonebook will be retrieved with a request like:
SELECT * F
ROM User
WHERE IdUser IN
(SELECT IdUser FROM PersonalPhoneBook WHERE IdOwner = x);
(this request isn't optimized, but you get the gist).
I should also mention that every user and company has as many details (phone numbers, addresses, ...) as they want, thanks to these three tables :
ContactDetail(IdContactDetail, LbContactDetail, ValueContactDetail)
Company_ContactDetail(IdCompany, IdContactDetail)
User_ContactDetail(IdUser, IdContactDetail)
Now there is something that I didn't take in count in that model : the users and companies will want to include in their phone books some people that aren't users in the database.
So I've been exploring several solutions, and I'd need advice to determine which is best:
Making two other tables to store a fixed number of details (2 phone numbers, 1 address) on those "outsiders".
Making all "outsiders" users (but I find that pretty ugly to store them together)
Store an independent phonebook (LDAP-type) for every company and every user (but then data is replicated)
Keep the contact detail system for companies and users and use a LDAP system for the others
Other...?
I am developing a Client-Server based app using WPF, SQL-Server and WCF. This is basically a kind of POS application (not using any POS library but features are almost same). Server side coding is almost done. Now I am moving to client side coding. I am new in WCF (first app on WCF) and Client-Server architecture, so I surrounded by some doubts before I begin my coding. According to my Client (Person), he should be able to print item sales bill (Memo) from Client computers as well as Server. Every memo should have a Memo No for that transaction which should increment automatically and salesman should be able to see which memo no he is going to use. So for this purpose I have added a Memo No field on Application Sales Windows which would show Memo No (Last Memo No + 1) from database.
Here comes the problem, suppose there are 3 sales counters and when salesman opened their respective Sales Windows for billing then all of them would see (last Memo No + 1). And if 3 of them bill simultaneously then same memo no will be stored for 3 different transactions.
I must tell you that I have 2 tables in database to store sales. Table 1 stores discount, grand total, salesman id and Memo no and Table 2 stores memo no, item code, quantity, bill date etc. So its Memo no which binds two tables and if this get manipulated then you know what mess it can create in Monthly Reports.
This situation in overflowing my brain and came here to get solution. What I can do to overcome this. Any suggestion, link, code will be very helpful.
You could consider amending your design so that a memo number is assigned when a client begins a sale.
One way to implement this would be to use another table to store the sequence of memo numbers:
CREATE TABLE MemoSequence
(MemoId int identity(1,1),
DateAssigned datetime
)
Each time the sales window is opened on the client, the current date/time should be inserted into MemoSequence.DateAssigned, and the corresponding MemoId value returned (using an OUTPUT clause or SCOPE_IDENTITY()) to the client as the MemoId.
That way, each client would get a unique memo number assigned from the start of the transaction.
NB - this may not be a suitable solution if it is a requirement that there are no gaps in the sequence of Memo numbers.