I am trying to write unit test for a dynamodb get call , the get call gets the records of type SdkIterable<Page> , below is the get method:
fun getByStatus(status: Status): List<MODEL> {
val attributeValue: AttributeValue = AttributeValue.builder().s(status.toString())
.build()
val queryConditional: QueryConditional = QueryConditional.keyEqualTo(
Key.builder()
.partitionValue(attributeValue).build()
)
val results: SdkIterable<Page<MODEL>> = secondaryIndex?.query(
QueryEnhancedRequest.builder()
.queryConditional(queryConditional)
.build()
) ?: throw IllegalArgumentException(
"No Global Secondary Index found with name : $GLOBAL_SECONDARY_INDEX for " +
" table : $DYNAMODB_TABLE_NAME "
)
var items = mutableListOf<MODEL>()
results.forEach { page ->
items.addAll(page.items())
}
return items
}
I am trying to write the unit test of this as below:
#Test
fun `get items by status should return items`() {
val entity = getTestEntity()
val attributeValue: AttributeValue = AttributeValue.builder().s(Status.NEW.toString)
.build()
val queryConditional: QueryConditional = QueryConditional.keyEqualTo(
Key.builder()
.partitionValue(attributeValue).build()
)
val request = QueryEnhancedRequest.builder()
.queryConditional(queryConditional)
.build()
every { unrecoverableEventEntityIndex.query(request) } returns iterable.iterator() as SdkIterable<Page<Model>>//want to return the SdkIterable with some records
verify { repository.getByStatus().size>0 }
}
I am not able to mock the secondaryIndex?.query call, i.e. can't find any documentation around initialising the
SdkIterable<Page>
or its implementation with some data so that I can return it while mocking the
secondaryIndex?.query
.
I am using Kotlin and enhanced dynamodb client for dynamodb operations.
Any help would be really appreciated.
Thanks in advance.
Related
I have a StateFlow containing a simple list of strings. I want to be able to filter that list of Strings. Whenever the filter gets updated, I want to push out a new update to the StateFlow.
class ResultsViewModel(
private val api: API
) : ViewModel() {
var filter: String = ""
val results: StateFlow<List<String>> = api.resultFlow()
.stateIn(viewModelScope, SharingStarted.Eagerly, emptyList())
}
It's easy enough to stick a map onto api.resultFlow():
val results: StateFlow<List<String>> = api.resultFlow()
.map { result ->
val filtered = mutableListOf<String>()
for (r in result) {
if (r.contains(filter)) {
filtered.add(r)
}
}
filtered
}
.stateIn(viewModelScope, SharingStarted.Eagerly, emptyList())
But how do I get the flow to actually emit an update when filter changes? Right now, this only works with whatever the initial value of filter is set to.
You could have the filter update another StateFlow that is combined with the other one. By the way, there's filter function that is easier to use than manually creating another list and iterating to get your results.
class ResultsViewModel(
private val api: API
) : ViewModel() {
private val filterFlow = MutableStateFlow("")
var filter: String
get() = filterFlow.value
set(value) {
filterFlow.value = value
}
val results: StateFlow<List<String>> =
api.resultFlow()
.combine(filterFlow) { list, filter ->
list.filter { it.contains(filter) }
}
.stateIn(viewModelScope, SharingStarted.Eagerly, emptyList())
}
We have some code to read data from DynamoDB:
suspend fun getKeys(owner: String): Set<String> {
...
val query = ...
query.subscribe { page -> foo(page) }.await()
return ...
}
The subscribe{} above is defined in AWS SDK:
default CompletableFuture<Void> subscribe(Consumer<T> consumer)
I'd like to unit test the logic in this function, for subscribe I don't care about it, hopefully just mock and do nothing.
I tried mock a callback in my unit test (irrelevant code removed):
class BucketRepoTest(
#Mock private val query: SdkPublisher<Page<DbBucket>>
) {
#Test
fun `get keys should be working`() {
val callback = mock<(Page<DbBucket>) -> Unit>()
val result = mock<CompletableFuture<Void>>()
whenever(query.subscribe(callback)).thenReturn(result)
runBlocking {
val keys = data.getKeys("Charlie")
assert(keys.isEmpty())
}
}
}
But when I run the test I got NPE:
query.subscribe { page -> foo(page) } must not be null
java.lang.NullPointerException: query.subscribe { page -> foo(page) } must not be null
at com.myApp.getKeys(myfile.kt:75)
at ...
Any idea how to fix it?
The result cannot be void.
val result = mock<CompletableFuture>()
try to create a mock data and pass as result
val result = the object or data type you want to more
Example
data Result( Charlie = "john", age = "23")
then
class BucketRepoTest(
#Mock private val query: SdkPublisher<Page<DbBucket>> ) {
#Test
fun `get keys should be working`() {
val callback = mock<(Page<DbBucket>) -> Unit>()
val result = Result()
whenever(query.subscribe(callback)).thenReturn(result)
runBlocking {
val keys = data.getKeys("Charlie")
assert(keys.isEmpty())
}
}
}
I am trying to return List<List<Map<String, String>>> from a function in kotlin. I'm new to kotlin.
Edit1
Here's how I am attempting to to this
val a = mutableListOf(mutableListOf(mutableMapOf<String, String>()))
The problem with the above variable is, I am unable to figure out how to insert data into this variable. I tried with this:
val a = mutableListOf(mutableListOf(mutableMapOf<String, String>()))
val b = mutableListOf(mutableMapOf<String, String>())
val c = mutableMapOf<String, String>()
c.put("c", "n")
b.add(c)
a.add(b)
This is giving me:
[[{}], [{}, {c=n}]]
What I want is [[{c=n}]]
Can someone tell me how I can insert data into it?
The end goal I am trying to achieve is to store data in the form of List<List<Map<String, String>>>
EDIT 2
The function for which I am trying to write this dat structure:
fun processReport(file: Scanner): MutableList<List<Map<String, String>>> {
val result = mutableListOf<List<Map<String, String>>>()
val columnNames = file.nextLine().split(",")
while (file.hasNext()) {
val record = mutableListOf<Map<String, String>>()
val rowValues = file.nextLine()
.replace(",(?=[^\"]*\"[^\"]*(?:\"[^\"]*\"[^\"]*)*$)".toRegex(), "")
.split(",")
for (i in rowValues.indices) {
record.add(mapOf(columnNames[i] to rowValues[i]))
print(columnNames[i] + " : " + rowValues[i] + " ")
}
result.add(record)
}
return result
}
You don't need to use mutable data structures. You can define it like this:
fun main() {
val a = listOf(listOf(mapOf("c" to "n")))
println(a)
}
Output:
[[{c=n}]]
If you wanted to use mutable data structures and add the data later, you could do it like this:
fun main() {
val map = mutableMapOf<String, String>()
val innerList = mutableListOf<Map<String, String>>()
val outerList = mutableListOf<List<Map<String, String>>>()
map["c"] = "n"
innerList.add(map)
outerList.add(innerList)
println(outerList)
}
The output is the same, although the lists and maps are mutable.
In response to the 2nd edit. Ah, you're parsing a CSV. You shouldn't try to do that yourself, but you should use a library. Here's an example using Apache Commons CSV
fun processReport(file: File): List<List<Map<String, String>>> {
val parser = CSVParser.parse(file, Charset.defaultCharset(), CSVFormat.DEFAULT.withHeader())
return parser.records.map {
it.toMap().entries.map { (k, v) -> mapOf(k to v) }
}
}
For the following CSV:
foo,bar,baz
a,b,c
1,2,3
It produces:
[[{foo=a}, {bar=b}, {baz=c}], [{foo=1}, {bar=2}, {baz=3}]]
Note that you can simplify it further if you're happy returning a list of maps:
fun processReport(file: File): List<Map<String, String>> {
val parser = CSVParser.parse(file, Charset.defaultCharset(), CSVFormat.DEFAULT.withHeader())
return parser.records.map { it.toMap() }
}
Output:
[{foo=a, bar=b, baz=c}, {foo=1, bar=2, baz=3}]
I'm using Charset.defaultCharset() here, but you should change it to whatever character set the CSV is in.
I am trying to build pipelines for events with Apache Beam. What I wanted to do is reading streaming data from GCP PubSub and read related metadata from MySQL using the ids in the events, then combine those two streams and write to my clickhouse database.
But JdbcIO.readall() doesn't seem to return its stream. As you can see on the ClickhousePipeline, after applying CoGroupByKey.create(), I am trying to combine two PCollection, but userMetaData comes in empty, and ParDo that chained right after UserMetadataEnricher() hasn't executed too.
In withRowMapper on UserMetadataEnricher, I added println() to chech if it's running, and It worked properly and print the results from my database, however, It doesn't return data to the next pipeline.
I guess the problem is related to Windowing, I checked it's working when I tested it without windowing. But, PubSubIO is Unbounded PCollection, so I have to apply window to use JDBCIO.readall() right? I have no clue to solve this problem. I hope to get an answer soon!
MainPipeline
object MainPipeline {
#JvmStatic
fun run(options: MainPipelineOptions) {
val p = Pipeline.create(options)
val events = p
.apply(
"Read DetailViewEvent PubSub",
PubsubIO.readStrings().fromSubscription(options.inputSubscription)
)
.apply(
"Extract messages",
ParseJsons.of(FoodDetailViewEvent::class.java)
.exceptionsInto(
TypeDescriptors.kvs(TypeDescriptors.strings(), TypeDescriptors.strings())
)
.exceptionsVia { KV.of(it.element(), it.exception().javaClass.canonicalName) }
)
val validEvents =
events.output().setCoder(SerializableCoder.of(FoodDetailViewEvent::class.java))
val invalidEvents = events.failures()
invalidEvents.apply(FailurePipeline(options))
validEvents.apply(ClickhousePipeline(options))
p.run().waitUntilFinish()
}
#JvmStatic
fun main(args: Array<String>) {
val options = PipelineOptionsFactory
.fromArgs(*args)
.withValidation()
.`as`(MainPipelineOptions::class.java)
run(options)
}
}
ClickhousePipeline
class ClickhousePipeline(private val options: MainPipelineOptions) :
PTransform<PCollection<DetailViewEvent>, PDone>() {
override fun expand(events: PCollection<DetailViewEvent>): PDone {
val windowedEvents = events
.apply(
"Window", Window
.into<DetailViewEvent>(GlobalWindows())
.triggering(
Repeatedly
.forever(
AfterProcessingTime
.pastFirstElementInPane()
.plusDelayOf(Duration.standardSeconds(5))
)
)
.withAllowedLateness(Duration.ZERO).discardingFiredPanes()
)
val userIdDetailViewEvents = windowedEvents
.apply(
MapElements.via(object :
SimpleFunction<DetailViewEvent, KV<String, DetailViewEvent>>() {
override fun apply(input: DetailViewEvent): KV<String, DetailViewEvent> {
return KV.of(input.userInfo.userId, input)
}
})
)
val userMetaData = userIdDetailViewEvents
.apply(
MapElements.via(object :
SimpleFunction<KV<String, DetailViewEvent>, String>() {
override fun apply(input: KV<String, DetailViewEvent>): String {
return input.key!!
}
})
)
.apply(
UserMetadataEnricher(options)
)
.apply(
ParDo.of(
object : DoFn<UserMetadata, KV<String, UserMetadata>>() {
#ProcessElement
fun processElement(
#Element data: UserMetadata,
out: OutputReceiver<KV<String, UserMetadata>>
) {
println("User:: ${data}") // Not printed!!
out.output(KV.of(data.userId, data))
}
})
)
val sourceTag = object : TupleTag<DetailViewEvent>() {}
val userMetadataTag = object : TupleTag<UserMetadata>() {}
val joinedPipeline: PCollection<KV<String, CoGbkResult>> =
KeyedPCollectionTuple.of(sourceTag, userIdDetailViewEvents)
.and(userMetadataTag, userMetaData)
.apply(CoGroupByKey.create())
val enrichedData = joinedPipeline.apply(
ParDo.of(object : DoFn<KV<String, CoGbkResult>, ClickHouseModel>() {
#ProcessElement
fun processElement(
#Element data: KV<String, CoGbkResult>,
out: OutputReceiver<ClickHouseModel>
) {
val name = data.key
val source = data.value.getAll(sourceTag)
val userMetadataSource = data.value.getAll(userMetadataTag)
println("==========================")
for (metadata in userMetadataSource.iterator()) {
println("Metadata:: $metadata") // This is always empty
}
for (event in source.iterator()) {
println("Event:: $event")
}
println("==========================")
val sourceEvent = source.iterator().next()
if (userMetadataSource.iterator().hasNext()) {
val userMetadataEvent = userMetadataSource.iterator().next()
out.output(
ClickHouseModel(
eventType = sourceEvent.eventType,
userMetadata = userMetadataEvent
)
)
}
}
})
)
val clickhouseData = enrichedData.apply(
ParDo.of(object : DoFn<ClickHouseModel, Row>() {
#ProcessElement
fun processElement(context: ProcessContext) {
val data = context.element()
context.output(
data.toSchema()
)
}
})
)
return clickhouseData
.setRowSchema(ClickHouseModel.schemaType())
.apply(
ClickHouseIO.write(
"jdbc:clickhouse://127.0.0.1:8123/test?password=example",
"clickhouse_test"
)
)
}
}
UserMetadataEnricher
class UserMetadataEnricher(private val options: MainPipelineOptions) :
PTransform<PCollection<String>, PCollection<UserMetadata>>() {
#Throws(Exception::class)
override fun expand(events: PCollection<String>): PCollection<UserMetadata> {
return events
.apply(
JdbcIO.readAll<String, UserMetadata>()
.withDataSourceConfiguration(
JdbcIO.DataSourceConfiguration.create(
"com.mysql.cj.jdbc.Driver", "jdbc:mysql://localhost:3306/beam-test"
)
.withUsername("root")
.withPassword("example")
)
.withQuery("select id,name,gender from user where id = ?")
.withParameterSetter { id: String, preparedStatement: PreparedStatement ->
preparedStatement.setString(1, id)
}
.withCoder(
SerializableCoder.of(
UserMetadata::class.java
)
)
.withRowMapper
{
println("RowMapper:: ${it.getString(1)}") // printed!!
UserMetadata(
it.getString(1),
it.getString(2),
it.getString(3)
)
}
)
}
}
output
RowMapper:: test-02
RowMapper:: test-01
==========================
Event:: DetailViewEvent(...)
==========================
==========================
Event:: DetailViewEvent(...)
==========================
Update 1 (GlobalWindow to FixedWindow)
using AfterProcessing
I've changed my window setting and added print to SimpleFunction that is assigned to userIdDetailViewEvents.
Window.into<FoodDetailViewEvent>(FixedWindows.of(Duration.standardSeconds(30)))
.triggering(
Repeatedly.forever(
AfterProcessingTime.pastFirstElementInPane().plusDelayOf(
Duration.standardSeconds(1)
)
)
)
.withAllowedLateness(Duration.ZERO)
.discardingFiredPanes()
)
And It prints:
userIdDetailViewEvents Called
userIdDetailViewEvents Called
RowMapper:: test-02
RowMapper:: test-01
==========================
Event:: DetailViewEvent(...)
==========================
==========================
Event:: DetailViewEvent(...)
==========================
Using AfterWatermark
Window.into<FoodDetailViewEvent>(FixedWindows.of(Duration.standardSeconds(30)))
.triggering(
Repeatedly.forever(
AfterWatermark.pastEndOfWindow()
)
)
.withAllowedLateness(Duration.ZERO)
.discardingFiredPanes()
outputs
userIdDetailViewEvents Called
userIdDetailViewEvents Called
RowMapper:: test-02
RowMapper:: test-01
I think using AfterWatermark is correct but It is hanging on somewhere... I guess It's JdbcIO
You are correct, the standard configuration doesn't return results with unbounded collections.
As long as you have a functional window / trigger combination the trick is to set
.withOutputParallelization(false); to the JdbcIO.<>readAll() call as follows:
p.apply("Read from JDBC", JdbcIO.<>readAll()
.withDataSourceConfiguration(getConfiguration())
.withQuery(getStreamingQuery())
.withParameterSetter((JdbcIO.PreparedStatementSetter<String>) (element, preparedStatement) -> {
// Prepare statement here.
})
.withRowMapper((JdbcIO.RowMapper<>) results -> {
// Map results here.
}).withOutputParallelization(false));
I struggled with this for hours but came across the code in this article and it finally worked.
I used
Window.into(new GlobalWindows()).triggering(
Repeatedly.forever(AfterPane.elementCountAtLeast(1))
)
.discardingFiredPanes())
for my trigger to process one element at a time.
GlobalWindow never closes so it's not a suitable option for unbounded data source like pubsub.
I would recommend using FixedWindow(<Some time range>) instead.
You can read more about windows here https://beam.apache.org/documentation/programming-guide/#windowing
I'm a newbie about Kotlin. My first project is to consume a rest api. I already made that using retrofit. But I have a problem when I'm logging the response, my data class is null. I don't know where is the error.
My Rerofit Client
object RetrofitClient {
var retrofit: Retrofit? = null
fun getClient(baseUrl: String): Retrofit? {
if (retrofit == null) {
//TODO While release in Google Play Change the Level to NONE
val interceptor = HttpLoggingInterceptor()
interceptor.level = HttpLoggingInterceptor.Level.BODY
val client = OkHttpClient.Builder()
.addInterceptor(interceptor)
.connectTimeout(100, TimeUnit.SECONDS)
.readTimeout(100, TimeUnit.SECONDS)
.build()
retrofit = Retrofit.Builder()
.client(client)
.baseUrl(baseUrl)
.addConverterFactory(GsonConverterFactory.create())
.build()
}
return retrofit
}
}
My Interface
public interface ApiLoginService {
#POST("UserManagementCoreAPI/api/v1/users")
fun loginService(#Body request: RequestBody): Call<DataLogin>
}
object ApiUtils {
val BASE_URL = "------"
val apiLoginService: ApiLoginService
get() = RetrofitClient.getClient(BASE_URL)!!.create(ApiLoginService::class.java)
}
My class data
data class DataLogin (
#SerializedName("employeeId") val employeeId : String,
#SerializedName("fullName") val fullName : String,
#SerializedName("loginConfins") val loginConfins : String,
#SerializedName("branchId") val branchId : String,
#SerializedName("isActive") val isActive : String
)
Main Activity
mApiLoginService!!.loginService(requestBody).enqueue(object : Callback<DataLogin>{
override fun onResponse(call: Call<DataLogin>, response: Response<DataLogin>) {
if(response.isSuccessful()){
if(response.body().toString() == null){
Log.d(tag,"Null")
}else{
Log.d(tag,"Logging In " + response.body()!!)
progressBar.visibility = View.GONE
btn_submit.visibility = View.VISIBLE
startActivity(Intent(this#MainActivity, HomeActivity::class.java))
}
}else{
Toast.makeText(applicationContext, "Invalid username or password", Toast.LENGTH_LONG).show()
Log.d(tag,"Error " + response.errorBody().toString())
progressBar.visibility = View.GONE
btn_submit.visibility = View.VISIBLE
}
}
override fun onFailure(call: Call<DataLogin>, t: Throwable) {
progressBar.visibility = View.GONE
btn_submit.visibility = View.VISIBLE
}
})
My Respone Log
message: Logging In DataLogin(employeeId=null, fullName=null, loginConfins=null, branchId=null, isActive=null)
I don't know where is the error and why my data is null. If the response succeeds is still gives me null.
This is a postman example
You have an issue with your schema , Your DataLogin class is different of your postman schema , Retrofit is waiting for : fullName, isActive ...., and the response is : header , data .. , you have to create class that contains header as variable of type Header(errors:List<AnotherClass>), data as variable of type Data(data(List<DataLogin>),totalRecord:Int), i would suggest if you use helper website like JSON to Class , parse your postman response there , and he will give you your correct response class but it's will be in java , you have to rewrite the code yourself of just copy paste in android studio and he will convert the code to Kotlin for you. (in the web site , check Source type: JSON)
You have to match the json structure with your data classes if you do not provide a custom adapter to Gson. So if you want to have a result, maybe something like this will work:
data class Response(
#SerializedName("headers") val headers: List<Any>,
#SerializedName("data") val data: Data
)
data class Data(
#SerializedName("data") val data: List<DataLogin>,
#SerializedName("totalRecord") val totalRecord: Int
)
data class DataLogin (
#SerializedName("employeeId") val employeeId : String,
#SerializedName("fullName") val fullName : String,
#SerializedName("loginConfins") val loginConfins : String,
#SerializedName("branchId") val branchId : String,
#SerializedName("isActive") val isActive : String
)
You need to return a Response object from your retrofit call.
Also a few tips about kotlin, Gson works well for Java, but it has some issues with kotlin (about null safety). I use Moshi when the project language is kotlin and it works well with it.
Try to avoid using !! in kotlin because it will cause RuntimeException. There are other ways of checking null and to protect your code from RuntimeExceptions.