File table contains - File_Id and File_data(BLOB)
how can I know the size of the binary file stored in File_data column. length function gives the length of file but how to know the size in KB.
This gives a number in bytes, devide it by 1024 to get size in KB.
Select sum(BIGINT(length(blob_column)))
from table;
BLOB Length is not quite the same as size.
You first need to answer NO to all of these questions for it to be the file size:
Is the BLOB column declared as COMPACT?
Is the table data compressed? (Off by default)
Also consider LOB locator overhead.
Basically the answer is, that you can't really 100% determine the BLOB / actual file size from the column via length method.
Related
I have a small Firebird 2.5 database with a blob field called "note" declared as this:
BLOB SUB_TYPE 1 SEGMENT SIZE 80 CHARACTER SET UTF8
The data base page size is:
16.384 (That I'm suspecting is too high)
I have ran this select in order to discover the average size of the blob fields available:
select avg(octet_length(items.note)) from items
and got this information:
2.671
As a beginner, I would like to know the better segment size for this blob field and the best database page size in your opinion (I know that this depends of other information, but I still don't know how to figure it out).
Blobs in Firebird are stored in separate pages of your database. The exact storage format depends on the size of your blob. As described in Blob Internal Storage:
Blobs are created as part of a data row, but because a blob could be
of unlimited length, what is actually stored with the data row is a
blobid, the data for the blob is stored separately on special blob
pages elsewhere in the database.
[..]
A blob page stores data for a blob. For large blobs, the blob page
could actually be a blob pointer page, i.e. be used to store pointers
to other blob pages. For each blob that is created a blob record is
defined, the blob record contains the location of the blob data, and
some information about the blobs contents that will be useful to the
engine when it is trying to retrieve the blob. The blob data could be
stored in three slightly different ways. The storage mechanism is
determined by the size of the blob, and is identified by its level
number (0, 1 or 2). All blobs are initially created as level 0, but
will be transformed to level 1 or 2 as their size increases.
A level 0 blob is a blob that can fit on the same page as the blob
header record, for a data page of 4096 bytes, this would be a blob of
approximately 4052 bytes (Page overhead - slot - blob record header).
In other words, if your average size of blobs is 2671 bytes (and most larger ones are still smaller than +/- 4000 bytes), then likely a page size of 4096 is optimal as it will reduce wasted space from on average 16340 - 2671 = 13669 bytes to 4052 - 2671 = 1381 bytes.
However for performance itself this likely hardly going to matter, and smaller page sizes have other effects that you will need to take into account. For example a smaller page size will also reduce the maximum size of a CHAR/VARCHAR index key, indexes might become deeper (more levels), and less records fit in a single page (or wider records become split over multiple pages).
Without measuring and testing it is hard to say if using 4096 for the page size is the right size for your database.
As to segment sizes: it is a historic artifact that is best ignored (and left off). Sometimes applications or drivers incorrectly assume that blobs need to be written or read in the specified segment size. In those rare cases specifying a larger segment size might improve performance. If you leave it off, Firebird will default to a value of 80.
From Binary Data Types:
Segment Size: Specifying the BLOB segment is throwback to times past,
when applications for working with BLOB data were written in C
(Embedded SQL) with the help of the gpre pre-compiler. Nowadays, it is
effectively irrelevant. The segment size for BLOB data is determined
by the client side and is usually larger than the data page size, in
any case.
I'm developing a Java web application that deals with large amounts of text (HTML code strings encoded using base64), which I need to save in my database. I'm using Firebird 2.0, and every time I try to insert a new record with strings longer than 32767 characters, I receive the following error:
GDS Exception. 335544726. Error reading data from the connection.
I have done some research about it, and apparently this is the character limit for Firebird, both for query strings and records in the database. I have tried a couple of things, like splitting the string in the query and then concatenating the parts, but it didn't work. Does anyone know any workarounds for this issue?
If you need to save large amount of text data in the database - just use BLOB fields. Varchar field size is limited to 32Kb.
For better performance you can use binary BLOBs and save there zipped data.
Firebird query strings are limited to 64 kilobytes in Firebird 2.5 and earlier. The maximum length of a varchar field is 32766 byte (which means it can only store 8191 characters when using UTF-8!). The maximum size of a row (with blobs counting for 8 bytes) is 64 kilobytes as well.
If you want to store values longer than 32 kilobytes, you need to use a BLOB SUB_TYPE TEXT, and you need to use a prepared statement to set the value.
In google Bigquery ....I have loaded a 1.5GB csv file from googlstorage after successfully loading,.... my table size is 250MB why this is so?
Likely because the binary encoding of numbers is more efficient than encoding them as strings. For example, the string "1234567890" takes 10 bytes (at least, or 20 bytes if it is UTF-16 encoded), but it can be represented by a 4 byte integer which only takes, ehm, 4 bytes.
Furthermore, the table in bigquery can also leave out the separators, because it knows how many bytes each field is wide. Thats another byte saved for every ,.
When trying to load data into a big query table, I get an error telling me a row is larger than the maximum allowed size. I could not find this limitation anywhere in the documentation. What is the limit? And is there a workaround?
The file is compressed json and is 360M.
2018 update: 100 MB maximum row size. https://cloud.google.com/bigquery/quotas
The maximum row size is 64k. See: https://developers.google.com/bigquery/docs/import#import
The limitation for json will likely increase soon.
2013 update: The maximum row size is 1MB, and 20MB for JSON.
See: https://developers.google.com/bigquery/preparing-data-for-bigquery#dataformats
2017 update: 10MB for CSV & JSON. https://cloud.google.com/bigquery/quotas#import
Except 1MB if streaming, https://cloud.google.com/bigquery/quotas#streaminginserts
Is there a limit to the amount of data that can be put in a single row in BigQuery? Is there a limit on the size of a single column entry (cell)? (in bytes)
Is there a limitation when importing from Cloud Storage?
The largest size of a single row allowed is 1MB for CSV and 2 MB for JSON. There are no limits on field sizes, but obviously they must be under the row size as well.
These limits are described here.