I have been a Delphi programmer for 25 years, but managed to avoid SQL until now. I was a dBase expert back in the day. I am using Firebird 3.0 SuperServer as a service on a Windows server 2012 box. I run a UDP listener service written in Delphi 2007 to receive status info from a software product we publish.
The FB database is fairly simple. I use the user's IP address as the primary key and record reports as they come in. I am currently getting about 150,000 reports a day and they are logged in a text file.
Rather than insert every report into a table, I would like to increment an integer value in a single record with a "running total" of reports received from each IP address. It would save a LOT of data.
The table has fields for IP address (Primary Key), LastSeen (timestamp), and Hits (integer). There are a few other fields but they aren't important.
I use UPDATE OR INSERT INTO when the report is received. If the IP address does not exist, a new row is inserted. If it does exist, then the record is updated.
I would like it to increment the "Hits" field by +1 every time I receive a report. In other words, if "Hits" already = 1, then I want to inc(Hits) on UPDATE to 2. And so on. Basically, the "Hits" field would be a running total of the number of times an IP address sends a report.
Adding 3 million rows a month just so I can get a COUNT for a specific IP address does not seem efficient at all!
Is there a way to do this?
The UPDATE OR INSERT statement is not suitable for this, as you need to specify the values to update or insert, so you will end up with the same value for both the insert and the update. You could address this by creating a before insert trigger that will always assign 1 to the field that holds the count (ignoring the value provided by the statement for the insert), but it is probably better to use MERGE, as it gives you more control about the resulting action.
For example:
merge into user_stats
using (
select '127.0.0.1' as ipaddress, timestamp '2021-05-30 17:38' as lastseen
from rdb$database
) as src
on user_stats.ipaddress = src.ipaddress
when matched then
update set
user_stats.hits = user_stats.hits + 1,
user_stats.lastseen = max_value(user_stats.lastseen , src.lastseen)
when not matched then
insert (ipaddress, hits, lastseen) values (src.ipaddress, 1, src.lastseen)
However, if you get a lot of updates for the same IP address, and those updates are processed concurrently, this can be rather error-prone due to update conflicts. You can address this by inserting individual hits, and then have a background process to summarize those records into a single record (e.g. daily).
Also keep in mind that having a single record removes the possibility to perform more analysis (e.g. distribution of hits, number of hits on day X or a time HH:mm, etc).
Related
I have a single database table on a relational database. Data will be loaded into it. I then want to have multiple servers processing that data concurrently (I don't want to have only one server running at a time). E.g. each server will:
Query for a fixed number of rows
Do some work for each row retrieved
Update each row to show it has been processed
How do I ensure that each row is only processed once? Note I don't want to pre-assign a row of data to a server; i'm designing for high availability so the solution should keep running if one or more servers goes down.
The solution I've gone for so far is as follows:
The table has three columns: LOCKED_BY (VARCHAR), LOCKED_AT (TIMESTAMP) and PROCESSED (CHAR)
Each server starts by attempting to "pseudo-lock" some rows by doing:
UPDATE THE_TABLE
SET LOCKED_BY= $servername,
LOCKED_AT = CURRENT_TIMESTAMP,
WHERE (LOCKED_BY = null OR (CURRENT_TIMESTAMP- LOCKED_AT > $timeout)
AND PROCSSED = 'N'
i.e. try to "pseudo-lock" rows that aren't locked already or where the pseudo-lock as expired. Only do this for unprocessed rows.
More than one server may have attempted this at the same time. The current server needs to query to find out if it was successful in the "pseudo-lock":
SELECT * FROM THE_TABLE
WHERE LOCKED_BY = $server_name
AND PROCESSED = 'N'
If any rows are returned the server can process them.
Once the processing has been done the row is updated
UPDATE THE_TABLE SET PROCESSED = 'Y' WHERE PRIMARYKEYCOL = $pk
Note: the update statement should ideally limit the number of rows updated.
If you are open to changing platform then I would suggest moving to a modern, cloud-based solution like Snowflake. This will do what you want but in the background and by default - so you don't need to know what it's doing or how it's doing it (unless you want to).
This may come across as patronising, which is not my intention, but what you are attempting (in the way you are attempting it) is very complex; so if you don't already know how to do it then someone telling you how to do it is not going to give you the skills/experience you need to be able to implement it successfully
Sql question.
I have a customer table with:
User id, name, email, phone
The customer can update their name, email and phone at anytime on an app.
How can I find out which user id had changes in name, email or phone number on a particular date?
Since your table doesn't store the date that they made the changes, you can't.
If you add a column with a datetime type (or whatever your specific database product provides) - you could call it LastModified or something like that - then the solution becomes trivial.
I'd give you a specific example, but because you didn't tell us what database engine you use, I can't guarantee to get the syntax right.
This is an issue with RDBMSes, you cannot as they generally store say a "photograph" of your data in time not a "film" of how it got there.
Based on the RDBMS you use, you can introduce an updated_at field which will hold when the last change happened to that row either from the "UPDATE" statement (say 'UPDATE phone=000, updated_at=now() WHERE user_id=999') or set it up to autoupdate see: create column for auto-date in postgresql
Check this simple piece of code that uses a generator to create unique primary keys in a Firebird table:
CREATE OR ALTER TRIGGER ON_BEFOREINSERT_PK_BOOKING_ITEM FOR BOOKING_ITEM BEFORE INSERT POSITION 0
AS
BEGIN
IF ((NEW.booking_item_id IS NULL) OR (NEW.booking_item_id = 0)) THEN BEGIN
SELECT GEN_ID(LastIdBookingItem, 1) FROM RDB$DATABASE INTO :NEW.booking_item_id;
END
END!
This trigger grabs and increments then assigns a generated value for the booking item id thus creating an auto-incremented key for the BOOKING_ITEM table. The trigger even checks that the booking id has not already been assigned a value.
The problem is the auto-incremented value will be lost (wasted) if, for some reason, the BOOKING_ITEM record cannot be posted.
I have a couple of ideas on how to avoid this wasting but have concerns about each one. Here they are:
Decrement the counter if a posting error occurs. Within the trigger I set up a try-except block (do try-except blocks even exist in Firebird PSQL?) and run a SELECT GEN_ID(LastIdBookingItem, -1) FROM RDB$DATABASEon post exceptions. Would this work? What if another transaction sneaks in and increments the generator before I decrement it? That would really mess things up.
Use a Temporary Id. Set the id to some unique temp value that I change to the generator value I want on trigger AFTER INSERT. This method feels somewhat contrived and requires a way that ensures that the temp id is unique. But what if the booking_item_id was supplied client side, how would I distinguish that from a temp id?. Plus I need another trigger
Use Transaction Control. This is like option 1. except instead of using the try-except block to reset the generator I start a transaction and then roll it back if the record fails to post. I don't know the syntax for using transaction control. I thought I read somewhere that SAVEPOINT/SET TRANSACTION is not allowed in PSQL. Plus the roll back would have to happen in the AFTER INSERT trigger so once again I need another trigger.
Surely this is an issue for any Firebird developer that wants to use Generators. Any other ideas? Is there something I'm missing?
Sequences are outside transaction control, and meddling with them to get 'gapless' numbers will only cause troubles because another transaction could increment the sequence as well concurrently, leading to gaps+duplicates instead of no gaps:
start: generator value = 1
T1: increment: value is 2
T2: increment: value is 3
T1: 'rollback', decrement: value is 2 (and not 1 as you expect)
T3: increment: value is 3 => duplicate value
Sequences should primarily be used for generating artificial primary keys, and you shouldn't care about the existence of gaps: it doesn't matter as long as the number uniquely identifies the record.
If you need an auditable sequence of numbers, and the requirement is that there are no gaps, then you shouldn't use a database sequence to generate it. You could use a sequence to assign numbers after creating and committing the invoice itself (so that it is sure it is persisted). An invoice without a number is simply not final yet. However even here there is a window of opportunity to get a gap, eg if an error or other failure occurs between assigning the invoice number and committing.
Another way might be to explicitly create a zero-invoice (marked as cancelled/number lost) with the gap numbers, so that the auditor knows what happened to that invoice.
Depending on local law and regulations, you shouldn't 're-use' or recycle lost numbers as that might be construed as fraud.
You might find other ideas in "An Auditable Series of Numbers". This also contains a Delphi project using IBObjects, but the document itself describes the problem and possible solutions pretty well.
What if, instead of using generators, you create a table with as many columns as the number of generators, giving each column the name of a generator. Something like:
create table generators
(
invoiceNumber integer default 0 not null,
customerId integer default 0 not null,
other generators...
)
Now, you have a table where you can increment invoice number using a SQL inside a transaction, something like:
begin transaction
update generator set invoiceNumber = invoiceNumber + 1 returning invoiceNumber;
insert into invoices set ..........
end transaction.
if anything goes wrong, the transaction would be rolled-back, together with the new
invoice number. I think there would be no more gaps in the sequence.
Enio
Im building a system for my company to keep track of internal orders, inbetween our warehouses, we have material that goes out warehouse 1 to warehouse 2 and we kind of lose track of how much of "x" is in warehouse 1 and how much in warehouse 2, so i want to implement this access db where a user fills a form and says: order 1: 500 of "x" order 2: 300 of "y". then another user fills an exit form where he says 1 of "x" going out, so i would need the program to keep track of total order and how much as gone out to fill order 1 and so on...
My idea here is to have both an order number and an id number for each of "x" everytime someoneone assembles 1 "x" they fill the form and print a label directly from the access (i have this part working already) while keeping a record of when it was assembled, who verified and what was verified (it will work as a quality control also).
What i dont know is how to program the db so when it reaches 500 of "x", the id number for "x" starts again from 1
This is the one major issue with my program right now, i'm not experienced in access db's or vba, but im getting there with a tip and a trick from here and there, so, no need to be careful with the technical language, i will google it if i have to :p
EDIT:
The table structure goes as follows:
1 table as the main table where I record the check that is made for every product, where I include the model of the product, the said ID that I want to reset after a number of products checked, and a concatenated field that includes most of this information to generate a qr code.
Then there is a table for the Order Number, which is connected to a form to record each new order with a date/time field, the order number itself and the number of products. This number of products must then be called from the code that will count how many products have been checked to date and keep the order number field updated so we can keep track of the order.
Then there is another minor table just to get values for the form, the product models
Thank you for your answers ;)
See this MSDN Documentation
Unfortunately in Access, you cannot 'reset' an ID field, unless you move the records to a newly created table and use that table for every 500 records.
As for the user control and login form, I'm afraid those are separate questions that must be asked in a different thread.
To get you started:
You can set the RecordSource of a form to a table, and when users make entries, the data will be saved to the table. You can also use a form with controls (text boxes, comboboxes, etc.) and create a button that runs a query to insert these records into a table.
The login piece - you can encrypt the database with a password. That may/may not be sufficient.
I would suggest you change your schema, if possible. Something like the following:
Orders
OrderID (Autonumber)
ProductID (link to your Products table)
QuantityRequested
Deliverables
DeliverableID (Autonumber)
OrderID (link to your Orders table)
SequenceNumber: in the BeforeInsert event set this value equal to:
DCount("*", "Deliverables", "OrderID=" & Me.OrderID) + 1
I'm assuming that your form has a control named OrderID that is bound to the OrderID field of the Deliverables table.
The code uses the DCount() function to get the count of all the other deliverables that have already been created for this order. If this is the first deliverable, DCount() will return 0. It then adds 1 to this count to get the sequence number of the next deliverable.
If the new SequenceNumber is greater than the quantity requested, you could display a message saying that the order has been filled and cancel the creation of the Deliverable record.
This is just one approach and it is not a complete solution. I'm assuming that once assigned a sequence number a deliverable cannot be deleted. You might need to make allowances for deliverables that get lost or damaged. You could incorporate a status field to the Deliverable table to deal with this, but you would still need to make a decision about what to do with the SequenceNumber.
Using a split database, everyone gets a front end with a local table I use as a 'cart' like in online shopping.
I'm copying records to a local table from stock. I don't want the record I copied across to be allowed to be transferred over again making duplicates. I also don't want to delete the original record, just modify it.
So I want them to edit the records copy locally then hit a button that will update the record on the database back end. If they don't hit the button and close the front end, no changes are made. Assume the temp table is wiped on start up.
To stop duplicate records I want to hide select records from the particular user of the front end database only. So if the Access app crashes the record isn't hidden for all users.
Idea: What If I add a Stock_ID (hidden) field to the local table? Then I can poll the column and if any Stock_ID matches the ID of the record I want to copy a message box says Error, record already exists and cancels the record copy?
I think you're saying you want to show the front end user only those stock records whose Stock_ID values are not present in the local table.
If that is correct, you can use an "unmatched query" to display those stock records.
SELECT s.*
FROM
stock AS s
LEFT JOIN [local] AS l
ON s.Stock_ID = l.Stock_ID
WHERE l.Stock_ID Is Null;
The Access query designer has a query wizard for this task. It should be worth a look.
When you say "hide select records", what combinations? Hide all of a certain type from ALL users; hide certain records from SOME users? In your split database, does EACH user have a copy of the front-end, or do all share the same front-end? There must be some criteria that determines who sees what records? Once that is identified, then a solution can follow.