how do i store image in room database, From Json.
as i was trying to store with Byte it gives me error - java.lang.NumberFormatException: For input string: "https://upload.wikimedia.org/wikipedia/commons/4/41/Sunflower_from_Silesia2.jpg"
data Class
#Entity(tableName = "actor",indices = arrayOf(Index(value= arrayOf("id"),unique = true)))
data class ActorItem(
#PrimaryKey(autoGenerate = true)
val id_:Int,
#SerializedName("age")
#ColumnInfo(name = "age")
val age: String,
#SerializedName("id")
#ColumnInfo(name = "id")
val id: Int,
#SerializedName("image")
#ColumnInfo(name="image")
val image: Byte,
#SerializedName("name")
#ColumnInfo(name = "name")
val name: String
)
here is Json
[
{"id": 1,
"name": "Hero",
"image": "https://upload.wikimedia.org/wikipedia/commons/4/41/Sunflower_from_Silesia2.jpg",
"age": "23"}
]
Are you trying to store the path to an image in the DB, or the image itself?
If the path, then image would be of type String (not Byte). Storing a path (or name) in the DB, and not the image contents, is generally considered best practice. It requires downloading and saving the image in the filesystem outside of the DB.
If you are trying to store the image contents in the DB though, then the field type would be ByteArray (not Byte), the column type would be Blob, and you will need to download the image and write the bytes in to the DB yourself.
Related:
How insert image in room persistence library?
Simple solution is to store image file in internal directory(App-specific storage) and store the internal private path in your room database column.
Internal App storage Docs :-
https://developer.android.com/training/data-storage/app-specific
Or make base64 of file image file and store base64 string in your database column
From Json. as i was trying to store with Byte it gives me error - java.lang.NumberFormatException: For input string:
What number, between 0 and 255 does https://upload.wikimedia.org/wikipedia/commons/4/41/Sunflower_from_Silesia2.jpg resolve to 0? 1? 2? .... 255? (Rhetorical) Why? (Rhetorical)
how do i store image in room database
You VERY PROBABLY SHOULD NOT but instead store the image as a file in the Apps's storage (there would be no real difference in storage space) but probably a very noticeable improvement in response times.
The value https://upload.wikimedia.org/wikipedia/commons/4/41/Sunflower_from_Silesia2.jpg is, to many humans, obviously a link to a file that can be downloaded (an image). It does not equate to a byte or any number.
However, the file stored at that location is a series (aka stream) of bytes.
Thus in Room a ByteArray (which Room will assign a type affinity of BLOB the the column). So the column type should either be ByteArray or a type that would require a type converter where the type converter returns a ByteArray.
So instead of val image: Byte, you would probably have val image: ByteArray,.
To get the ByteArray you could (assuming permissions etc are all setup) use something like (but not restricted to):-
return URL(url).readBytes()
where url, in your case, would be the String https://upload.wikimedia.org/wikipedia/commons/4/41/Sunflower_from_Silesia2.jpg
IMPORTANT
However, at 2.7Mb
that is very likely to cause issues. Not due to SQLite limitations but due to limitations of the Android API which retrieves data from the SQLite database via a Cursor which is a buffer that is limited in size (4Mb). As such any image that is close to 4Mb may be stored but it couldn't be retrieved without complications AND highly inefficient/slow processing.
Demonstration of why NOT to store images, like the one mentioned in the question.** in the database**
As a demonstration consider the following which does store the equivalent of images (not actual images) in the image column of you table
(i.e. ByteArrays, the content unless actually displaying the image is irrelevant Room nor SQLite knows the difference between an image and any other BLOB value)
using a slightly modified version of your ActorItem class, as :-
#Entity(tableName = "actor",indices = arrayOf(Index(value= arrayOf("id"),unique = true)))
data class ActorItem(
#PrimaryKey(autoGenerate = true)
val id_:Int,
#ColumnInfo(name = "age")
val age: String,
#ColumnInfo(name = "id")
val id: Int,
#ColumnInfo(name="image")
val image: ByteArray,
#ColumnInfo(name = "name")
val name: String
) {
#androidx.room.Dao
interface Dao {
#Insert
fun insert(actorItem: ActorItem)
}
}
i.e. the important difference is a type of ByteArray as opposed to Byte for the image column
for brevity/convenience the DAO class has been included (it is sufficient just to insert some columns to demonstrate why saving the image is not a very good idea)
To accompany is an #Database class TheDatabase :-
#Database(entities = [ActorItem::class], version = 1, exportSchema = false)
abstract class TheDatabase: RoomDatabase() {
abstract fun getActorItemDao(): ActorItem.Dao
companion object {
private var instance: TheDatabase? = null
fun getInstance(context: Context): TheDatabase {
if ( instance == null) {
instance = Room.databaseBuilder(context,TheDatabase::class.java,"thedatabase.db")
.allowMainThreadQueries()
.build()
}
return instance as TheDatabase
}
}
allowMainThreadQueries included for brevity and convenience
Finally putting the above into action via an activity is MainActivity :-
class MainActivity : AppCompatActivity() {
lateinit var db: TheDatabase
lateinit var dao: ActorItem.Dao
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
db = TheDatabase.getInstance(this)
dao = db.getActorItemDao()
val url = "https://upload.wikimedia.org/wikipedia/commons/4/41/Sunflower_from_Silesia2.jpg"
try {
for (i in 1..100) {
dao.insert(
ActorItem(
0, "The age", i,
getBitmapFromURLAsString(
url,
/* The ByteArray (bitmap) */
/* BUT for demonstration of issues associated with size issues
start at 1.08Mb size incrementing at 8kb per row
last would be 1.8Mb (100 * 8kb)
*/
i * (4096 * 2) + (1024 * 1024)),
getNameFromURLAsString(url)
)
)
}
} catch (e: Exception) {
e.printStackTrace()
}
}
fun getBitmapFromURLAsString(url: String, size: Int): ByteArray {
/* Fake the byte array allowing the size of the bytearray to be specified */
val ba = ByteArray(size)
var byte: Byte = 8
for(i in 0 until (size)) {
ba[i] = byte++.mod(Byte.MAX_VALUE)
}
return ba
/* actual would be something like */
/* WARNING image Sunflower_from_Silesia2.jpg is 2.7Mb will likely cause issues */
//return URL(url).readBytes()
}
fun getNameFromURLAsString(url: String): String {
val split = url.split("/")
return split.get(split.size -1)
}
}
So the activity will try to insert 100 rows with a ByteArray in the image column (answer to how to store image in principle). For each row the size of the ByteArray is increased by 8k (the first row is 1.08Mb i.e. 1Mb and 8k in size). The name column
The above runs successfully without any trapped exceptions. And all 100 rows are inserted:-
using query to extract the length of each image column shows the size (of the last rows) :-
First warning sign that things are perhaps not that good
Running the query takes hardly any time at all. Refreshing, moving from start to end from the table view takes quite a bit of time (a minute (will be dependant upon PC/Laptop used)).
Second warning sign
Running the App takes a few seconds.
Third warning sign
Use i * (4096 * 2) + (1024 * 1024 * 2)), (i.e. start at 2Mb up to 2.8Mb), run the App and try to view via App Inspection and :-
As can be seen the Rows exist and have the expected data in them :-
Try to look at the Actor table, DatabaseInspector doesn't show the contents .
Run the query SELECT substr(image,1024 * 1024) FROM actor (i.e. 8k for the first row 1.8k for the 100th row) WAIT (for a minute or so), scroll to the last, WAIT (for a minutes or so) and :-
You should use ByteArray (on java it means byte[]). Sqlite supports saving byte arrays like this: How to store image in SQLite database
And on room database, you can just use ByteArray type for you image, and room will finish the rest of work.
Related
I'm a bit lost with this issue.
The Pre-packaged database has an invalid schema error has the following output:
Expected
TableInfo{name='account', columns={client_alt_phone_on_route_sheets=Column{name='client_alt_phone_on_route_sheets', type='INTEGER', affinity='3', notNull=true, primaryKeyPosition=0, defaultValue='0'}, client_titles_on_address_labels=Column{name='client_titles_on_address_labels', type='INTEGER', affinity='3', notNull=false, primaryKeyPosition=0, defaultValue='0'}, client_titles_on_invoices=Column{name='client_titles_on_invoices', type='INTEGER', affinity='3', notNull=false, primaryKeyPosition=0, defaultValue='0'}}, foreignKeys=[], indices=[]}
Found
TableInfo{name='account', columns={client_alt_phone_on_route_sheets=Column{name='client_alt_phone_on_route_sheets', type='BOOLEAN', affinity='1', notNull=true, primaryKeyPosition=0, defaultValue='FALSE'}, client_titles_on_address_labels=Column{name='client_titles_on_address_labels', type='BOOLEAN', affinity='1', notNull=false, primaryKeyPosition=0, defaultValue='FALSE'}, client_titles_on_invoices=Column{name='client_titles_on_invoices', type='BOOLEAN', affinity='1', notNull=false, primaryKeyPosition=0, defaultValue='FALSE'}}, foreignKeys=[], indices=[]}
I'm ommitting some other columns because they are not the problem and their output matches. The problem is with the columns that expected INTEGER but found BOOLEAN.
The database schema is as follows:
CREATE TABLE account
(
client_alt_phone_on_route_sheets BOOLEAN DEFAULT FALSE NOT NULL,
client_titles_on_address_labels BOOLEAN DEFAULT FALSE,
client_titles_on_invoices BOOLEAN DEFAULT FALSE,
// Omitted rows
);
Initially, I did created the account Room #Entity with the BOOLEAN columns of type Boolean:
#Entity(tableName = "account")
data class Account(
// Omitted data
// #FIXME
#ColumnInfo(name = "client_alt_phone_on_route_sheets", defaultValue = "FALSE") val routeSheetsClientAltPhone: Boolean,
// #FIXME
#ColumnInfo(name = "client_titles_on_address_labels", defaultValue = "FALSE") val clientTitlesOnAddressLabels: Boolean?,
// #FIXME
#ColumnInfo(name = "client_titles_on_invoices", defaultValue = "FALSE") val clientTitlesOnInvoices: Boolean?,
)
Then, the first time the error was thrown I did changed the Boolean type columns to Int type:
#Entity(tableName = "account")
data class Account(
// Omitted data
// #FIXME
#ColumnInfo(name = "client_alt_phone_on_route_sheets", defaultValue = "0") val routeSheetsClientAltPhone: Int,
// #FIXME
#ColumnInfo(name = "client_titles_on_address_labels", defaultValue = "0") val clientTitlesOnAddressLabels: Int?,
// #FIXME
#ColumnInfo(name = "client_titles_on_invoices", defaultValue = "0") val clientTitlesOnInvoices: Int?,
)
However, the error kept happening. I tried with this answer migrating a single column to see if it then matches in the expected/found output.
private val MIGRATION_1_2 = object : Migration(1,2) {
override fun migrate(database: SupportSQLiteDatabase) {
database.execSQL(
"ALTER TABLE account ADD COLUMN client_alt_phone_on_route_sheets INTEGER NOT NULL DEFAULT(0)"
)
}
}
private fun buildDatabase(context: Context) = Room.databaseBuilder(
context.applicationContext,
Database::class.java,
Constants.DATABASE_NAME
).addMigrations(MIGRATION_1_2).createFromAsset("database.db").build()
But apparently migration did not work, nor changed anything in the output because the error is still the same. Also, I'm not sure which one is the Room database and which one is the database from assets. My hunch is that the 'Found' output is the one to match from assets because of the BOOLEAN type, but that's not completely clear. Why does it seems like changing the Room #ColumnInfo value type from Boolean to Int doesn't seem to take effect? If migration above needs to be implemented for every column of type BOOLEAN on my database, what is the proper way to apply migration for multiple tables? (since I have more tables that has this BOOLEAN type, though they're not in the error)
Why does it seems like changing the Room #ColumnInfo value type from Boolean to Int doesn't seem to take effect?
Because to Room they are the same type that is any type Long/long -> Boolean that is an integer (as opposed to decimal) is given a column type of INTEGER in the create table SQL.
Room will only create column types of INTEGER, TEXT, REAL or BLOB.
SQLite however is flexible in that you can create columns with virtually any type (as long s the type doesn't break parser rules such as being a keyword). SQLite then uses a set of rules to assign a type affinity (one of the 4 listed above or the catch-all NUMERIC (which room does not support/allow)).
So from the above the issue is with the pre-packaged database. That is the pre-packaged database has a column defined with BOOLEAN room expects INTEGER and therefore you MUST change the column type from BOOLEAN to INTEGER.
You can make this change by using:-
ALTER TABLE account RENAME TO account_old;
CREATE TABLE IF NOT EXISTS account (client_alt_phone_on_route_sheets INTEGER NOT NULL DEFAULT VALUE false, .....);
INSERT INTO account SELECT * FROM account_old;
DROP TABLE IF EXISTS account_old;
Note you can obtain the CREATE TABLE SQL from room if you compile the project after creating the entities and the #Database annotated class with the entities defined in the list of entities.
After a successful compilation Room will have generated some java code in the class named the same as the #Database annotated class but suffixed with _Impl. In this class there will be a method createAllTables the SQL for the creation of the tables (indexes and views). This SQL is what room expects
You have various places where you can alter the table(s), the simplest are to:-
Make the changes (as above) in the pre-packaged database, then copy the changed file into the assets folder, or
Use the prePackagedDatabaseCallback call back see How do I use Room's prepackagedDatabaseCallback?
2. The callback is invoked after copying the file but before the validation (Expected v Found) is performed.
I am trying to create an embedded field. This is a simple example but I can't get this simple example to work. Eventually I need to have 3 levels of embedded items but trying to get this test case to work.
#Entity(tableName = "userItemsEntity")
#Parcelize
data class Item(
var objecttype: String?,
#PrimaryKey(autoGenerate = false)
var objectid: Int?,
var subtype: String?,
var collid: Int?,
#Embedded
var name: Name?
) : Parcelable
#Parcelize
data class Name(
var primary: Boolean? = true,
var sortindex: Int? = null,
var content: String? = null) : Parcelable
When I try and compile it it complains on the DAO that the updateItem()
SQL error or missing database (no such column: name)
DAO function
#Query("UPDATE userItemsEntity SET " +
"objecttype=:objecttype, objectid=:objectid, subtype=:subtype, collid=:collid, name=:name " +
"WHERE objectid=:objectid")
fun updateItem(
objecttype: String?,
objectid: Int,
subtype: String?,
collid: Int?,
name: Name?)
The reason is as it says there is no name column. Rather the table consists of the columns, as per the member variables of the EMBEDDED class (i.e. primary, sortindex and content).
i.e. the table create SQL is/will be :-
CREATE TABLE IF NOT EXISTS `userItemsEntity` (`objecttype` TEXT, `objectid` INTEGER, `subtype` TEXT, `collid` INTEGER, `primary` INTEGER, `sortindex` INTEGER, `content` TEXT, PRIMARY KEY(`objectid`))
Room knows to build the respective Name object from those columns when extracting rows.
So you could use :-
#Query("UPDATE userItemsEntity SET " +
"objecttype=:objecttype, objectid=:objectid, subtype=:subtype, collid=:collid, `primary`=:primary, sortindex=:sortindex, content=:content " +
"WHERE objectid=:objectid")
fun updateItem(
objecttype: String?,
objectid: Int,
subtype: String?,
collid: Int?,
primary: Boolean?,
sortindex: Int?,
content: String?
)
note that primary is an SQLite token and thus enclosed in grave accents to ensure that it is not treated as a token. Otherwise you would get :-
There is a problem with the query: [SQLITE_ERROR] SQL error or missing database (near "primary": syntax error)
However, as you are using a WHERE clause based upon the primary key (objectid) then the update will only apply to a single row and as such you can simply use:-
#Update
fun update(item: Item): Int
obviously the function's name need not be update it could be any valid name that suits.
this has the advantage of not only being simpler but of returning the number of rows updated (would be 1 if the row exists, otherwise 0)
Impementing a name column
If you want a name column and for that name column to hold a Name object. Then, as SQLite does not have storage/column types for objects then you would not EMBED the Name class.
You would have var name: Name? with an appropriate TypeConverter that would convert the Name object into a type that SQLite caters for :-
TEXT (String),
REAL (Float, Double...),
INTEGER (Long, Int ...) or
BLOB (ByteArray)).
Typically String is used and typically GSON is used to convert from an object to a JOSN String.
SQlite does have a NUMERIC type. However, Room doesn't support it's use. I believe because the other types cover all types of data and NUMERIC is a catch-all/default.
However, using a JSON representation of an object, introduces bloat and reduces the usefulness of the converted data from an SQL aspect.
For example say you had :-
#Entity(tableName = "userOtherItemsEntity")
#Parcelize
data class OtherItem (
var objecttype: String?,
#PrimaryKey(autoGenerate = false)
var objectid: Int?,
var subtype: String?,
var collid: Int?,
var name: OtherName?) : Parcelable
#Parcelize
data class OtherName(
var primary: Boolean? = true,
var sortindex: Int? = null,
var content: String? = null) : Parcelable
Then the underlying table does have the name column. The CREATE SQL, generated by Room, would be :-
CREATE TABLE IF NOT EXISTS `userOtherItemsEntity` (`objecttype` TEXT, `objectid` INTEGER, `subtype` TEXT, `collid` INTEGER, `name` TEXT, PRIMARY KEY(`objectid`))
However, you would need TypeConverters which could be :-
#TypeConverter
fun fromOtherName(othername: OtherName ): String {
return Gson().toJson(othername)
}
#TypeConverter
fun toOtherName(json: String): OtherName {
return Gson().fromJson(json,OtherName::class.java)
}
the first using Gson to convert the object to a JSON string, e.g. when inserting data
the second converts the JSON string to an OtherName object.
using Item with Name embedded then data would be stored along the lines of :-
Whilst with the OtherItem with OtherName being converted then the data (similar data) would be along the lines of :-
in the former the 3 Name columns would take up about (1 + 1 + 12) = 16 bytes.
in the latter, The OtherName columns (discounting the word Other whenever used) would take uo some 55 bytes.
the latter may require more complex and resource expensive searches if the components of the OtherName are to be included in searches.
e.g. #Query("SELECT * FROM userItemsEntity WHERE primary") as opposed to #Query("SELECT * FROM userOtherItemsEntity WHERE instr(name,'primary\":true') > 0")
I have a User entity that holds a Character entity in a #OneToOne relation. However I wantt he Character record to be removed as soon as it gets detached from the User entity.
Here is my User.kt entity class:
// User.kt
#Entity
class User(
#Id
var id: String,
var email: String,
#OneToOne(cascade = [CascadeType.ALL], orphanRemoval = true)
var character: Character?,
var isAdmin: Boolean
) { // ... }
This is the unit test I wrote to test this behaviour:
// UserRepositoryTest.kt
#Test
fun `should remove orphan character entity when being removed from user entity`() {
val user = UserTestTemplate.testUser()
val character = CharacterTestTemplate.testCharacter()
user.character = character
userRepository.save(user)
user.character = null
userRepository.save(user)
val actual = userRepository.findById(user.id).orElse(null)
assertThat(actual).isNotNull()
assertThat(actual.character).isNull()
val savedCharacter = characterRepository.findById(character.id)
assertThat(savedCharacter.get()).isNull() // fails
}
I added the CascadeType.ALL and orphanRemoval = true option since those are the only things I read about being related to my request.
What I do in the unit test is creating a user and character instance. Then adding the character instance to the user and saving the user via the UserRepository. Thanks to CascadeType.ALL the character instance will be saved automatically. Now I'd like to have the same thing in reverse when removing the character from the user. This however does not work as expected as you can see in the last line of the unit test
Two things to be aware of:
transactional write behind pattern
first level cache
#Test
fun `should remove orphan character entity entity`() {
val user = UserTestTemplate.testUser()
val character = CharacterTestTemplate.testCharacter()
user.character = character
userRepository.save(user)
user.character = null
//use saveAndFlush here to force immediate DB update
//otherwise may be deferred until transactional method returns
userRepository.saveAndFlush(user)
//clear the persistence context to ensure you will be reading from
//the database rather than first level cache
//entityManager is injected to test via #PersistenceContext annotation
entityManager.clear();
//now you are guaranteed a db read reflecting all flushed updates
val actual = userRepository.findById(user.id).orElse(null)
assertThat(actual).isNotNull()
assertThat(actual.character).isNull()
val savedCharacter = characterRepository.findById(character.id)
assertThat(savedCharacter.get()).isNull() // fails
}
I want to send data (number) from "edit text section" of Sub-Activity1 (users input a simple number)and receive in another Sub-activity2, and depending on the number I want to show different sets of text. I am a beginner and I am stuck where in Sub-Activity 2 as it returns error for val str where I want to receive and manipulate the number received from Sub-Activity 1 editText.
Sub-Activity 1 :
<Send & Open Sub-Activity2>
getResult.setOnClickListener {
val intent = Intent(this, subactivity::class.java)
val name: String = editTextID.getText().toString()
intent.putExtra(name:"editTextID",value:"7.0")
startActivity(intent)
This returns no error.
Sub-Activity 2: <Receive & Manipulate the text>
class subactivity2 : AppCompatActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_subactivity2)
val str =intent.getStringExtra("editTextID")
when str == 7.0 {
infoTextView.textview.text= "textIwannaShow"
}
}
Franz might be right about you wanting to pass a Double, but since you're passing it as a String and also naming your val str, I'll assume you do want to pass a string (that just happens to represent a number). In which case you need to compare to a string
if (str == "7.0") {
infoTextView.textview.text = "textIwannaShow"
}
or if you do want a when block
when(str) {
"7.0" -> infoTextView.textview.text = "textIwannaShow"
}
If you actually want to work with numbers you'll have to call toDouble() on your string at some point to convert it to one. toDoubleOrNull would be better if you're taking that number from a text input (in case the user doesn't enter a valid number), but you're not actually using the value taken from the EditText
In your Sub-Activity 2, you are receiving a String not an Integer.
So, you should change your code from this
val str =intent.getStringExtra("editTextID")
To this
val str =intent.getIntExtra("editTextID", 0)
Anyway, in the example you are passing 7.0 which is Double, so you probably need this instead of above code
val str =intent.getDoubleExtra("editTextID", 0.0)
We’re trying to decide between providing generic vs specific record formats for consumption by our clients
with an eye to providing an online schema registry clients can access when the schemas are updated.
We expect to send out serialized blobs prefixed with a few bytes denoting the version number so schema
retrieval from our registry can be automated.
Now, we’ve come across code examples illustrating the relative adaptability of the generic format for
schema changes but we’re reluctant to give up the type safety and ease-of-use provided by the specific
format.
Is there a way to obtain the best of both worlds? I.e. could we work with and manipulate the specific generated
classes internally and then have them converted them to generic records automatically just before serialization?
Clients would then deserialize the generic records (after looking up the schema).
Also, could clients convert these generic records they received to specific ones at a later time? Some small code examples would be helpful!
Or are we looking at this all the wrong way?
What you are looking for is Confluent Schema registry service and libs which helps to integrate with this.
Providing a sample to write Serialize De-serialize avro data with a evolving schema. Please note providing sample from Kafka.
import io.confluent.kafka.serializers.KafkaAvroDeserializer;
import io.confluent.kafka.serializers.KafkaAvroSerializer;
import org.apache.avro.generic.GenericRecord;
import org.apache.commons.codec.DecoderException;
import org.apache.commons.codec.binary.Hex;
import java.util.HashMap; import java.util.Map;
public class ConfluentSchemaService {
public static final String TOPIC = "DUMMYTOPIC";
private KafkaAvroSerializer avroSerializer;
private KafkaAvroDeserializer avroDeserializer;
public ConfluentSchemaService(String conFluentSchemaRigistryURL) {
//PropertiesMap
Map<String, String> propMap = new HashMap<>();
propMap.put("schema.registry.url", conFluentSchemaRigistryURL);
// Output afterDeserialize should be a specific Record and not Generic Record
propMap.put("specific.avro.reader", "true");
avroSerializer = new KafkaAvroSerializer();
avroSerializer.configure(propMap, true);
avroDeserializer = new KafkaAvroDeserializer();
avroDeserializer.configure(propMap, true);
}
public String hexBytesToString(byte[] inputBytes) {
return Hex.encodeHexString(inputBytes);
}
public byte[] hexStringToBytes(String hexEncodedString) throws DecoderException {
return Hex.decodeHex(hexEncodedString.toCharArray());
}
public byte[] serializeAvroPOJOToBytes(GenericRecord avroRecord) {
return avroSerializer.serialize(TOPIC, avroRecord);
}
public Object deserializeBytesToAvroPOJO(byte[] avroBytearray) {
return avroDeserializer.deserialize(TOPIC, avroBytearray);
} }
Following classes have all the code you are looking for.
io.confluent.kafka.serializers.KafkaAvroDeserializer;
io.confluent.kafka.serializers.KafkaAvroSerializer;
Please follow the link for more details :
http://bytepadding.com/big-data/spark/avro/avro-serialization-de-serialization-using-confluent-schema-registry/
Can I convert between them?
I wrote the following kotlin code to convert from a SpecificRecord to GenericRecord and back - via JSON.
PositionReport is an object generated off of avro with the avro plugin for gradle - it is:
#org.apache.avro.specific.AvroGenerated
public class PositionReport extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {
...
The functions used are below
/**
* Encodes a record in AVRO Compatible JSON, meaning union types
* are wrapped. For prettier JSON just use the Object Mapper
* #param pos PositionReport
* #return String
*/
private fun PositionReport.toAvroJson() : String {
val writer = SpecificDatumWriter(PositionReport::class.java)
val baos = ByteArrayOutputStream()
val jsonEncoder = EncoderFactory.get().jsonEncoder(this.schema, baos)
writer.write(this, jsonEncoder)
jsonEncoder.flush()
return baos.toString("UTF-8")
}
/**
* Converts from Genreic Record into JSON - Seems smarter, however,
* to unify this function and the one above but whatevs
* #param record GenericRecord
* #param schema Schema
*/
private fun GenericRecord.toAvroJson(): String {
val writer = GenericDatumWriter<Any>(this.schema)
val baos = ByteArrayOutputStream()
val jsonEncoder = EncoderFactory.get().jsonEncoder(this.schema, baos)
writer.write(this, jsonEncoder)
jsonEncoder.flush()
return baos.toString("UTF-8")
}
/**
* Takes a Generic Record of a position report and hopefully turns
* it into a position report... maybe it will work
* #param gen GenericRecord
* #return PositionReport
*/
private fun toPosition(gen: GenericRecord) : PositionReport {
if (gen.schema != PositionReport.getClassSchema()) {
throw Exception("Cannot convert GenericRecord to PositionReport as the Schemas do not match")
}
// We will convert into JSON - and use that to then convert back to the SpecificRecord
// Probalby there is a better way
val json = gen.toAvroJson()
val reader: DatumReader<PositionReport> = SpecificDatumReader(PositionReport::class.java)
val decoder: Decoder = DecoderFactory.get().jsonDecoder(PositionReport.getClassSchema(), json)
val pos = reader.read(null, decoder)
return pos
}
/**
* Converts a Specific Record to a Generic Record (I think)
* #param pos PositionReport
* #return GenericData.Record
*/
private fun toGenericRecord(pos: PositionReport): GenericData.Record {
val json = pos.toAvroJson()
val reader : DatumReader<GenericData.Record> = GenericDatumReader(pos.schema)
val decoder: Decoder = DecoderFactory.get().jsonDecoder(pos.schema, json)
val datum = reader.read(null, decoder)
return datum
}
There are a couple difference however between the two:
Fields in the SpecificRecord that are of Instant type will be encoded in the GenericRecord as long and Enums are slightly different
So for example in my unit test of this function time fields are tested like this:
val gen = toGenericRecord(basePosition)
assertEquals(basePosition.getIgtd().toEpochMilli(), gen.get("igtd"))
And enums are validated by string
val gen = toGenericRecord(basePosition)
assertEquals(basePosition.getSource().toString(), gen.get("source").toString())
So to convert between you can do:
val gen = toGenericRecord(basePosition)
val newPos = toPosition(gen)
assertEquals(newPos, basePosition)