Decrypt KMS encrypted S3 objects from SES in Java - kotlin

I am using an SES receipt rule to store KMS encrypted emails in S3 and looking at decrypting in a SpringBoot/Kotlin application. The only way I could achieve this using the AWS Java SDK v2 was decrypting myself using the BouncyCastle crypto lib. If I downgrade to AWS Java SDK v1 I can easily decrypt using AmazonS3EncryptionClient without worrying about the crypto operations but its deprecated.
What is the preferred way to read these KMS encrypted S3 objects from SES using the latest AWS Java SDK?
Below are my various attempts:
AWS SDKv1 - works perfectly but deprecated
val provider = KMSEncryptionMaterialsProvider(kmsArn)
val s3EncryptionClient = AmazonS3EncryptionClient.encryptionBuilder()
.withEncryptionMaterials(provider)
.withCryptoConfiguration(CryptoConfiguration()
.withCryptoMode(CryptoMode.AuthenticatedEncryption)
.withAwsKmsRegion(Region.getRegion(Regions.US_WEST_2)))
.build()
println(s3EncryptionClient.getObjectAsString(bucket, key))
AWS SDKv1 (V2) - fails on the EncryptionContext validation?
val provider = KMSEncryptionMaterialsProvider(kmsArn)
val s3EncryptionClient = AmazonS3EncryptionClientV2Builder.standard()
.withRegion(Regions.US_WEST_2)
.withCryptoConfiguration(CryptoConfigurationV2()
.withCryptoMode(CryptoMode.AuthenticatedEncryption)
.withAwsKmsRegion(com.amazonaws.regions.Region.getRegion(Regions.US_WEST_2)))
.withEncryptionMaterialsProvider(kmsProvider)
.build()
val result = s3EncryptionClient.getObject(bucket, key)
println(String(result.objectContent.readAllBytes()))
AWS SDKv2 + BouncyCastle - works perfectly but I have to do all the decryption myself
fun getAsDecryptedString(bucket: String, key: String): String {
Security.addProvider(BouncyCastleProvider())
val headObjectResponse = s3AsyncClient.headObject(
HeadObjectRequest.builder().bucket(bucket).key(key).build()
).get()
val metadata = headObjectResponse.metadata()
val kmsKeyBase64 = metadata["x-amz-key-v2"]
val iv = metadata["x-amz-iv"]
val algo = checkNotNull(metadata["x-amz-cek-alg"]) { "x-amz-cek-alg metadata is required" }
val encryptionContext = objectMapper.readValue(metadata["x-amz-matdesc"], typeRef)
val kmsKey = SdkBytes.fromByteArray(Base64.getDecoder().decode(kmsKeyBase64))
val kmsDecryptResult = kmsClient.decrypt(
DecryptRequest.builder()
.ciphertextBlob(kmsKey)
.encryptionContext(encryptionContext)
.build()
)
val request = GetObjectRequest.builder().bucket(bucket).key(key).build()
val getObjectResponse = s3AsyncClient.getObject(request, AsyncResponseTransformer.toBytes()).get()
val keyBytes = kmsDecryptResult.plaintext().asByteArray()
val bytesToDecrypt = getObjectResponse.asByteArray()
val ivBytes = Base64.getDecoder().decode(iv)
return decryptWithAES(keyBytes, bytesToDecrypt, ivBytes, algo)
}
private fun decryptWithAES(keyBytes: ByteArray, bytesToDecrypt: ByteArray, ivBytes: ByteArray, algo: String): String {
val secretKey = SecretKeySpec(keyBytes, "AES")
val iv = IvParameterSpec(ivBytes)
synchronized(Cipher::class.java) {
val cipher = Cipher.getInstance(algo, "BC")
cipher.init(Cipher.DECRYPT_MODE, secretKey, iv)
val plainText = ByteArray(cipher.getOutputSize(bytesToDecrypt.size))
var ptLength = cipher.update(bytesToDecrypt, 0, bytesToDecrypt.size, plainText, 0)
ptLength += cipher.doFinal(plainText, ptLength)
val decryptedString = String(plainText)
return decryptedString.trim { it <= ' ' }
}
}

Related

Kafka Exception: The group member needs to have a valid member id before actually entering a consumer group

I have this consumer that appears to be connected to the kafka topic but after seeing the logs I'm seeing that the consumer Join group failed with org.apache.kafka.common.errors.MemberIdRequiredException: The group member needs to have a valid member id before actually entering a consumer group. I've tried looking at a few other setups online and I don't see anyone explicitly setting a member ID. I don't see this exception thrown locally so I'm hoping that maybe I just need to do some fine-tuning on the consumer configurations but I'm not sure what needs to be changed.
Kafka Consumer
#KafkaListener(
topics = ["my.topic"],
groupId ="kafka-consumer-group"
)
fun consume(consumerRecord: ConsumerRecord<String, Record>, ack: Acknowledgment) {
val value = consumerRecord.value()
val eventKey = consumerRecord.key()
val productNotificationEvent = buildProductNotificationEvent(value)
val consumerIdsByEventKey = eventKey.let { favoritesRepository.getConsumerIdsByEntityId(it) }
ack.acknowledge()
}
Kafka Consumer Config
#EnableKafka
#Configuration
class KafkaConsumerConfig(
#Value("\${KAFKA_SERVER}")
private val kafkaServer: String,
#Value("\${KAFKA_SASL_USERNAME}")
private val userName: String,
#Value("\${KAFKA_SASL_PASSWORD}")
private val password: String,
#Value("\${KAFKA_SCHEMA_REGISTRY_URL}")
private val schemaRegistryUrl: String,
#Value("\${KAFKA_SCHEMA_REGISTRY_USER_INFO}")
private val schemaRegistryUserInfo: String,
#Value("\${KAFKA_CONSUMER_MAX_POLL_IN_MS}")
private val maxPollIntervalMsConfig: Int,
#Value("\${KAFKA_CONSUMER_MAX_POLL_RECORDS}")
private val maxPollRecords: Int
) {
#Bean
fun consumerFactory(): ConsumerFactory<String?, Any?> {
val props: MutableMap<String, Any> = mutableMapOf()
props[ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG] = kafkaServer
props[ConsumerConfig.GROUP_ID_CONFIG] = KAFKA_CONSUMER_GROUP_ID
props[ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG] = StringDeserializer::class.java
props[ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG] = KafkaAvroDeserializer::class.java
props[ConsumerConfig.AUTO_OFFSET_RESET_CONFIG] = "earliest"
props[ConsumerConfig.MAX_POLL_RECORDS_CONFIG] = maxPollRecords
props[ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG] = maxPollIntervalMsConfig
props[CommonClientConfigs.SECURITY_PROTOCOL_CONFIG] = "SASL_SSL"
props[SaslConfigs.SASL_MECHANISM] = "PLAIN"
val module = "org.apache.kafka.common.security.plain.PlainLoginModule"
val jaasConfig = String.format(
"%s required username=\"%s\" password=\"%s\";",
module,
userName,
password
)
props[SaslConfigs.SASL_JAAS_CONFIG] = jaasConfig
props[KafkaAvroDeserializerConfig.VALUE_SUBJECT_NAME_STRATEGY] = TopicRecordNameStrategy::class.java
props[KafkaAvroDeserializerConfig.SCHEMA_REGISTRY_URL_CONFIG] = schemaRegistryUrl
props[KafkaAvroDeserializerConfig.BASIC_AUTH_CREDENTIALS_SOURCE] = "USER_INFO"
props[KafkaAvroDeserializerConfig.USER_INFO_CONFIG] = schemaRegistryUserInfo
return DefaultKafkaConsumerFactory(props)
}
#Bean
fun kafkaListenerContainerFactory(): ConcurrentKafkaListenerContainerFactory<String, Any>? {
val factory = ConcurrentKafkaListenerContainerFactory<String, Any>()
factory.consumerFactory = consumerFactory()
factory.containerProperties.ackMode = ContainerProperties.AckMode.MANUAL_IMMEDIATE
factory.containerProperties.isSyncCommits = true
return factory
}
companion object {
private const val KAFKA_CONSUMER_GROUP_ID = "kafka-consumer-group"
}
}

Verifying signature from .ps7 file (PKCS#7/CMS) in Kotlin

I have a file fileA.json.p7s and I'm trying to validate the detached signature using Bouncy Castle.
However, the code fails on verification and returns the error:
org.bouncycastle.cms.CMSSignerDigestMismatchException: message-digest attribute value does not match calculated value
Does anyone know how to fix this error?
Here is my code:
fun verifySignature(): Boolean {
val signatureContent = File("src/test/resources/fileA.json.p7s").readBytes()
val cmsSignedData = CMSSignedData(
signatureContent
)
val certStore = cmsSignedData.certificates
val signers = cmsSignedData.signerInfos
val signerCollection = signers.signers
val iterator = signerCollection.iterator()
while (iterator.hasNext()){
val signer = iterator.next()
val certCollection = certStore.getMatches(signer.sid as Selector<X509CertificateHolder?>)
val certIt = certCollection.iterator()
val cert = certIt.next() as X509CertificateHolder
if (signer.verify(JcaSimpleSignerInfoVerifierBuilder().build(cert))) { // this is where the code fails
return true
}
}
return false
}

AES encryption of Image In Kotlin

I am trying to encrypt and decrypt the image by AES in Kotlin
Key Generation Function-
override suspend fun key_genration(callback: Enc) {
val keygenerator = KeyGenerator.getInstance("AES")
keygenerator.init(128)
val secretKey: Key = keygenerator.generateKey()
val secretKey1 = secretKey.encoded
val result = Firebase.firestore.collection("profiles").document("YHMauLouORRtrYBV2h4dHJ5A0s72").update("key","$secretKey1")
}
Encryption Function -
override suspend fun encryption(imageView: ImageView, Key: ByteArray, name: String, uid: String) {
val bitmap = (imageView.drawable as BitmapDrawable).bitmap
val baos = ByteArrayOutputStream()
bitmap.compress(Bitmap.CompressFormat.JPEG, 100, baos) // bm is the bitmap object
val ba: ByteArray = baos.toByteArray()
val cipher = Cipher.getInstance("AES")
val keySpec = SecretKeySpec(Key, "AES")
Log.d("key11119","$keySpec")
cipher.init(Cipher.ENCRYPT_MODE, keySpec)
val txt = cipher.doFinal(ba)
Log.d("poott0","$txt")
val p = txt.toString(Charsets.ISO_8859_1)
Log.d("poott","$p")
// val encryptText = String(txt, Charset.defaultCharset())
val file = File.createTempFile("$name", ".txt")
file.writeText(p)
//val storageReference = FirebaseStorage.getInstance().getReference("image/$uid/$name")
val storageReference = FirebaseStorage.getInstance()
.getReference("/image/0XhL4jD4XCemk38rcRkIEjJMgjh2/Aadhar/")
storageReference.putFile(Uri.fromFile(file))
}
Decryption Function -
val img = view.findViewById<ImageView>(R.id.imageView2)
Firebase.firestore.collection("profiles").whereEqualTo("UID", "YHMauLouORRtrYBV2h4dHJ5A0s72").get()
.addOnSuccessListener {
val key = it.first().toObject(docretreving::class.java).key
Log.d("key111","$key")
Log.d("pooptt","Success")
val storageRef =
FirebaseStorage.getInstance().reference?.child("/image/0XhL4jD4XCemk38rcRkIEjJMgjh2/Aadhar/")
val localfile = File.createTempFile("temp",".txt")
storageRef.getFile(localfile)
.addOnSuccessListener {
Log.d("pooptt","Success")
val inputStream: InputStream = localfile.inputStream()
val inputString = inputStream.bufferedReader().use { it.readText() }
println(inputString)
Log.d("pooptt", inputString)
val key1 = "mxkLZmSFE1aKWzr6JyybjQ==".toByteArray()
Log.d("key111","$key1")
val keySpec = SecretKeySpec(key1, "AES")
val isro= inputString.toByteArray(Charsets.ISO_8859_1)
val cipher = Cipher.getInstance("AES")
cipher.init(Cipher.DECRYPT_MODE,keySpec)
var decryptedText : ByteArray = cipher.doFinal(isro)
Log.d("key110","$decryptedText")
val baos = ObjectInputStream(ByteArrayInputStream(decryptedText))
val bitmap = baos.readObject() as Bitmap
img.setImageBitmap(bitmap)
}
}
It says invalid Block size in decryption function , I am not sure if I am doing this correct. The key generating function the key and gives to encryption function[For now I was just testing with 1 key converted to base64 ]
Thanks in Advance
You should not put the ciphertext through a buffered reader, which handles text and may change binary data. Files and ciphertext both consist of bytes, you should treat them as such and not convert them to text. If you need text, use base 64.

GCM encryption built in MAC check failure

I am trying to write a couple functions that can encrypt and decrypt text using AES/GCM encryption in conjunction with PBKDF2 key generation. I am transitioning my code from CTR (SIC) encryption, and the MAC check failure is killing me when all else is working.
fun encryptAESBasic(input: String, password: String): String {
val masterpw = getKey(password)
val random = SecureRandom()
val salt = ByteArray(16)
random.nextBytes(salt)
val factory: SecretKeyFactory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1")
val spec: KeySpec = PBEKeySpec(masterpw.toString().toCharArray(), salt, 100, 256)
val tmp: SecretKey = factory.generateSecret(spec)
val cipher = Cipher.getInstance("AES/CTR/NoPadding")
val iv = ByteArray(16)
SecureRandom().nextBytes(iv)
cipher.init(Cipher.ENCRYPT_MODE, tmp, IvParameterSpec(iv))
val cipherText: ByteArray = cipher.doFinal(input.toByteArray(Charset.forName("UTF-8")))
val ivstring: String = Base64.encodeToString(iv, Base64.NO_WRAP)
val saltystring: String = Base64.encodeToString(salt, Base64.NO_WRAP)
val cipherstring: String = Base64.encodeToString(cipherText, Base64.NO_WRAP)
val returnstring: String = ivstring + "-" + saltystring + "-" + cipherstring
return returnstring
}
fun decryptAESBasic(text: String, password: String): String {
val arr = text.split("-")
val iv = Base64.decode(arr[0].toByteArray(Charset.forName("UTF-8")), Base64.NO_WRAP)
val salt = Base64.decode(arr[1].toByteArray(Charset.forName("UTF-8")), Base64.NO_WRAP)
val data = arr[2].toByteArray(Charset.forName("UTF-8"))
val masterpw = getKey(password)
val factory: SecretKeyFactory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1")
val spec: KeySpec = PBEKeySpec(masterpw.toString().toCharArray(), salt, 100, 256)
val tmp: SecretKey = factory.generateSecret(spec)
val key: ByteArray = tmp.getEncoded()
val cipher = Cipher.getInstance("AES/CTR/NoPadding")
cipher.init(Cipher.DECRYPT_MODE, tmp, IvParameterSpec(iv))
val credential: ByteArray = cipher.doFinal(Base64.decode(data, Base64.NO_WRAP))
return credential.toString(Charset.forName("UTF-8"))
}
fun getKey(masterPass: String): ByteArray {
return masterPass.padEnd(32, '.').toByteArray(Charset.forName("UTF-8"))
}
Again, this code works, but I would like to change it from CTR to GCM but every time that I do I am met with a "mac check in GCM failed" error. Any help explaining how/why this is happening would be tremendously appreciated.
E/AndroidRuntime( 6461): Caused by: javax.crypto.AEADBadTagException: mac check in GCM failed
E/AndroidRuntime( 6461): at java.lang.reflect.Constructor.newInstance0(Native Method)
E/AndroidRuntime( 6461): at java.lang.reflect.Constructor.newInstance(Constructor.java:343)
E/AndroidRuntime( 6461): at com.android.org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher$AEADGenericBlockCipher.doFinal(BaseBlockCipher.java:1485)
E/AndroidRuntime( 6461): at com.android.org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineDoFinal(BaseBlockCipher.java:1217)
E/AndroidRuntime( 6461): at javax.crypto.Cipher.doFinal(Cipher.java:2055)
E/AndroidRuntime( 6461): at design.codeux.autofill_service.FlutterMyAutofillServiceKt.decryptAESBasic(FlutterMyAutofillService.kt:1003)
Instead of using IvParameterSpec in cipher.init(), use GCMParameterSpec

Reading a json file(approx 180 MB) without key names from S3 to a data class using object mapper (in kotlin)

My sample json in s3 looks like this :
{"380682":{"ids":[380682, 765894, 9875086]},"1995539":{"ids":[1995539, 234567, 987654]}}
I want to map it to List of Activities where:
data class Activities(
val key: String,
val values: List<String>
)
My application requires me to use Object Mapper only.
Currently I am doing it this way but it takes way too long:
val activitiesList = mutableListOf<Activities>()
val response: Map<String, Map<String, List<String>>>
val reader: Reader = InputStreamReader(S3ObjectInputStream, StandardCharsets.UTF_8)
val jsonNode: JsonNode
reader.use { jsonNode = objectMapper.readTree(it) }
response = objectMapper.convertValue(jsonNode, object : TypeReference<Map<String, Map<String, List<String>>>>() {})
for ((key, value) in response) {
value.get("ids")?.let {
Activities(
key = key,
values = it)
}?.let { ActivitiesList.add(it) }
Please suggest a faster and cleaner way to do this...