I want to save chekpoint tests in a location on amazon S3, this is the part of my scala code on DStream,using below format but getting the error..
Exception in thread "main" java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties (respectively).
Code:
val creatingFunc = { ()=>
// Create a StreamingContext
val ssc = new StreamingContext(sc, Seconds(batchIntervalSeconds))
val ggsnLines = ssc.fileStream[LongWritable, Text, TextInputFormat]("C:\\Users\\Mbazarganigilani\\Documents\\RA\\GGSN\\Files1",filterF,false)
val ccnLines= ssc.fileStream[LongWritable, Text, TextInputFormat]("C:\\Users\\Mbazarganigilani\\Documents\\RA\\CCN\\Files1",filterF,false)
val probeLines= ssc.fileStream[LongWritable, Text, TextInputFormat]("C:\\Users\\Mbazarganigilani\\Documents\\RA\\Probe\\Files1",filterF,false)
val ggssnArrays=ggsnLines.map(x=>(x._1,x._2.toString())).filter(!_._2.contains("ggsnIPAddress")).map(x=>(x._1,x._2.split(",")))
ggssnArrays.foreachRDD(s=> {
s.collect().take(10).foreach(u=>println(u._2.mkString(",")))
})
ssc.remember(Minutes(1)) // To make sure data is not deleted by the time we query it interactively
ssc.checkpoint("s3n://probecheckpoints/checkpoints")
println("Creating function called to create new StreamingContext")
newContextCreated = true
ssc
}
def main(args:Array[String]): Unit =
{
//the minremeberduration is set to read the previous files from the directory
//the kyroclasses serialization needs to be enabled for the filestream
if (stopActiveContext) {
StreamingContext.getActive.foreach { _.stop(stopSparkContext = false) }
}
// Get or create a streaming context
val hadoopConfiguration:Configuration=new Configuration()
hadoopConfiguration.set("fs.s3n.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "AKIAIOPSJVBDTEUHUJCQ")
hadoopConfiguration.set("fs.s3n.awsSecretAccessKey", "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")
//val ssc = StreamingContext.getActiveOrCreate(creatingFunc)
val ssc=StreamingContext.getActiveOrCreate("s3n://probecheckpoints/SparkCheckPoints",creatingFunc,hadoopConfiguration,false)
if (newContextCreated) {
println("New context created from currently defined creating function")
} else {
println("Existing context running or recovered from checkpoint, may not be running currently defined creating function")
}
// Start the streaming context in the background.
ssc.start()
Related
I did check the similar question here, but it didn't work as expected for me, I still need to reconnect to the created database and I'm not sure how or even if I can avoid that.
Here is my code:
hikari.properties:
jdbcUrl=jdbc:mariadb://localhost:3306/
driverClassName=org.mariadb.jdbc.Driver
username=root
dataSource.databaseName=DBNAME //this doesn't seem to do much, I'm getting the same behavior with or without it
fun initDB() {
val config = HikariConfig("/hikari.properties")
val ds = HikariDataSource(config)
transaction(connect(ds)) {
SchemaUtils.createDatabase("DBNAME")
}
config.jdbcUrl = "jdbc:mariadb://localhost:3306/DBNAME"
//ds.jdbcUrl = "jdbc:mariadb://localhost:3306/DBNAME" //THIS WILL NOT WORK
val ds2 = HikariDataSource(config)
transaction(connect(ds2)) {
SchemaUtils.create( Tables... )
}
}
The reason I make a new datasource, is because otherwise I'll get this error:
java.lang.IllegalStateException: The configuration of the pool is sealed once started. Use HikariConfigMXBean for runtime changes. HikariConfigMXBean doesn't seem allow jdbcUrl changes.
There must be a more elegant way to do this, right?
It is not a good practice. Creating and filling db it is two separate processes. DB creates once by the DBA, then you just connect and use it. Your approach has huge security violation. User from the datasource must have create db privilege.
But if you want proceed with your current approach, first you should create a db without using datasource, then create datasource and connect to the db. Something like this
import com.zaxxer.hikari.HikariConfig
import com.zaxxer.hikari.HikariDataSource
import org.jetbrains.exposed.sql.Database
import org.jetbrains.exposed.sql.SchemaUtils
import org.jetbrains.exposed.sql.Table
import org.jetbrains.exposed.sql.transactions.transaction
fun main(args: Array<String>) {
//LOAD proerties
val config = HikariConfig("/hikari.properties")
//Hikari properties content
// jdbcUrl=jdbc:mysql://localhost:3306/hikari
// driverClassName=com.mysql.cj.jdbc.Driver
// username=root
// password=<pwd here>
//Here replace dbname from jdbc url to empty
transaction(Database.connect(url = config.jdbcUrl.replace("hikari", ""),
driver = config.driverClassName,
user = config.username,
password = config.password)) {
SchemaUtils.createDatabase("hikari")
}
val ds = HikariDataSource(config)
transaction(Database.connect(ds)) {
SchemaUtils.create(Users)
}
}
object Users : Table() {
val id = varchar("id", 10) // Column<String>
val name = varchar("name", length = 50) // Column<String>
override val primaryKey = PrimaryKey(id, name = "PK_User_ID")
}
Normally I use Code A to create a database Tasks.db with Room when the app run for the first time, I hope that Room doesn't create the database Tasks.db again when I run the app again, how can I do?
Code A
val result = Room.databaseBuilder(
context.applicationContext,
ToDoDatabase::class.java, "Tasks.db"
).build()
This is safe to use as it is. You'll only get a new ToDoDatabase instance that you can access your database file through, but the file on disk won't be erased and recreated if it already exists.
You can also use method onCreate() of RoomDatabase.Callback, which is invoke only the first time you create a data base:
val result = Room.databaseBuilder(context.applicationContext,
ToDoDatabase::class.java, "Tasks.db").addCallback(dbCallback).build()
...
var dbCallback: RoomDatabase.Callback = object : RoomDatabase.Callback() {
override fun onCreate(db: SupportSQLiteDatabase) {
Executors.newSingleThreadScheduledExecutor().execute {
Log.i(TAG, "create database")
result!!.getDao().insertAll(...) // add default data
...
}
}
}
Since the improved privacy changes on Android 10 Android 10 Privacy changes, I've noticed that my screenshot failure test watcher rule in Kotlin, that extends the Espresso BasicScreenCaptureProcessor no longer saves failure screenshots because I am using the deprecated getExternalStoragePublicDirectory on Android 10.
The concept currently implement is very similar to How to take screenshot at the point where test fail in Espresso?
class TestScreenCaptureProcessor : BasicScreenCaptureProcessor() {
init {
this.mDefaultScreenshotPath = File(
File(
Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_PICTURES),
"Failure_Screenshots"
).absolutePath
)
}
As seen in other posts, I could use the getInstrumentation().getTargetContext().getApplicationContext().getExternalFilesDir(DIRECTORY_PICTURES)
that would store the file in - /sdcard/Android/data/your.package.name/files/Pictures directory, but the connectedAndroidTest gradle task deletes the app at the end along with it the folders listed above.
I wondered if anyone else had come across something similar and has considered a way of storing failure screenshots on Android 10, in a location that will not be deleted when the tests have finished running & somewhere that Espresso Instrumentation tests can access.
My UI tests are run on a variety of devices, so need a generic way of storing the files is required to suite all models.
After a lot of research I found a way to save screenshots in kotlin based on the SDK version using MediaStore.
/**
* storeFailureScreenshot will store the bitmap based on the SDK level of the
* device. Due to security improvements and changes to how data can be accessed in
* SDK levels >=29 Failure screenshots will be stored in
* sdcard/DIRECTORY_PICTURES/Failure_Screenshots.
*/
fun storeFailureScreenshot(bitmap: Bitmap, screenshotName: String) {
val contentResolver = InstrumentationRegistry.getInstrumentation().targetContext.applicationContext.contentResolver
// Check SDK version of device to determine how to save the screenshot.
if (android.os.Build.VERSION.SDK_INT >= 29) {
useMediaStoreScreenshotStorage(
contentValues,
contentResolver,
screenshotName,
SCREENSHOT_FOLDER_LOCATION,
bitmap
)
} else {
usePublicExternalScreenshotStorage(
contentValues,
contentResolver,
screenshotName,
SCREENSHOT_FOLDER_LOCATION,
bitmap
)
}
}
/**
* This will be used by devices with SDK versions >=29. This is to overcome scoped
* storage considerations now in the SDK version listed to help limit file
* clutter. A Uniform resource identifier (Uri) is used to insert bitmap into
* the gallery using the contentValues previously specified. The contentResolver
* provides application access to content model to access and publish data in a
* secure manner, using MediaStore collections to do so. Files will
* be stored in sdcard/Pictures
*/
private fun useMediaStoreScreenshotStorage(
contentValues: ContentValues,
contentResolver: ContentResolver,
screenshotName: String,
screenshotLocation: String,
bitmap: Bitmap
) {
contentValues.put(MediaStore.MediaColumns.DISPLAY_NAME, "$screenshotName.jpeg")
contentValues.put(MediaStore.Images.Media.RELATIVE_PATH, Environment.DIRECTORY_PICTURES + screenshotLocation)
val uri: Uri? = contentResolver.insert(MediaStore.Images.Media.EXTERNAL_CONTENT_URI, contentValues)
if (uri != null) {
contentResolver.openOutputStream(uri)?.let { saveScreenshotToStream(bitmap, it) }
contentResolver.update(uri, contentValues, null, null)
}
}
/**
* Method to access internal storage on a handset with SDK version below 29.
* Directory will be in sdcard/Pictures. Relevant sub directories will be created
* & screenshot will be stored as a .jpeg file.
*/
private fun usePublicExternalScreenshotStorage(
contentValues: ContentValues,
contentResolver: ContentResolver,
screenshotName: String,
screenshotLocation: String,
bitmap: Bitmap
) {
val directory = File(
Environment.getExternalStoragePublicDirectory(
Environment.DIRECTORY_PICTURES + screenshotLocation).toString())
if (!directory.exists()) {
directory.mkdirs()
}
val file = File(directory, "$screenshotName.jpeg")
saveScreenshotToStream(bitmap, FileOutputStream(file))
val values = contentValues
contentResolver.insert(MediaStore.Images.Media.EXTERNAL_CONTENT_URI, values)
}
/**
* Assigns the assignments about the Image media including, image type & date
* taken. Content values are used so the contentResolver can interpret them. These
* are applied to the contentValues object.
*/
val contentValues = ContentValues().apply {
put(MediaStore.Images.Media.MIME_TYPE, "image/jpeg")
put(MediaStore.Images.Media.DATE_TAKEN, System.currentTimeMillis())
}
/**
* Compresses the bitmap object to a .jpeg image format using the specified
* OutputStream of bytes.
*/
private fun saveScreenshotToStream(bitmap: Bitmap, outputStream: OutputStream) {
outputStream.use {
try {
bitmap.compress(Bitmap.CompressFormat.JPEG, 50, it)
} catch (e: IOException) {
Timber.e("Screenshot was not stored at this time")
}
}
}
Used in conjunction with a TestWatcher that takes a screenshot on a UI test failure. This is then added to the test class as a rule.
private val deviceLanguage = Locale.getDefault().language
/**
* Finds current date and time & is put into format of Wed-Mar-06-15:52:17.
*/
fun getDate(): String = SimpleDateFormat("EEE-MMMM-dd-HH:mm:ss").format(Date())
/**
* ScreenshotFailureRule overrides TestWatcher failed rule and instead takes a
* screenshot using the UI Automation takeScreenshot method and the
* storeFailureScreenshot to decide where to store the bitmap when a failure
* occurs.
*/
class ScreenshotFailureRule : TestWatcher() {
override fun failed(e: Throwable?, description: Description) {
val screenShotName = "$deviceLanguage-${description.methodName}-${getDate()}"
val bitmap = getInstrumentation().uiAutomation.takeScreenshot()
storeFailureScreenshot(bitmap, screenShotName)
}
}
file is stored in sdcard/Pictures/Failure_Screenshots with a name of
en-testMethodName-Day-Month-Date-HH_MM_SS
Rule called using:
val screenshotFailureRule = ScreenshotFailureRule()
I want to upload avatars. My endpoints are set with OpenAPI3RouterFactory(api.yaml file).
api.yaml:
#other endpoints
/api/v1/upload-avatar:
post:
summary: Uploading avatars endpoint
operationId: upload-avatar
tags:
- sign up
- registration
#other ones
HandlerVerticle.kt:
//OpenApi3Router is set
//other endpoints
routerFactory.addHandlerByOperationId("upload-avatar", BodyHandler.create().setDeleteUploadedFilesOnEnd(true).setUploadsDirectory("mp-upload").setMergeFormAttributes(true))
routerFactory.addHandlerByOperationId("upload-avatar", { routingContext ->
val fileUploadSet = routingContext.fileUploads()
val fileUploadIterator = fileUploadSet.iterator()
while (fileUploadIterator.hasNext()) {
val fileUpload = fileUploadIterator.next()
val uploadedFile = vertx.fileSystem().readFileBlocking(fileUpload.uploadedFileName())
try {
val fileName = URLDecoder.decode(fileUpload.fileName(), "UTF-8")
vertx.fileSystem().writeFileBlocking(fileName , uploadedFile)
routingContext.response().end()
} catch (e: Exception) {
e.printStackTrace()
}
}
})
//other routes
when I upload the image it uploads successful, but in server side vertx creates wrong upload directory, and it doesn't delete temporary uploaded files. Can anyone help me?
Not sure if this should be considered a bug, or something intentional, but...
OpenAPI3RouterFactoryImpl.getRouter() always overrides BodyHandler
No matter what you set there before.
I opened a new issue for now: https://github.com/vert-x3/vertx-web/issues/860
The plugin play-reactivemongo offers an easy way to upload a file:
def upload = Action(gridFSBodyParser(gridFS)) { request =>
val futureFile: Future[ReadFile[BSONValue]] = request.body.files.head.ref
futureFile.map { file =>
// do something
Ok
}.recover { case e: Throwable => InternalServerError(e.getMessage) }
}
Unfortunately this solution doesn't suit me because:
I would like only my DAO layer to depend on reactive-mongo.
I need to save the file only if a user is authenticated (with SecureSocial) and use some user's properties as checks and metadata.
If no user is authenticated the request body shouldn't be parsed at all (see also this question).
It would be something along the lines
def upload = SecuredAction { request =>
val user = request.user
val enumerator = an enumrator from the body parsing ???
myDAO.saveFile(user, enumerator)
object myDAO {
def saveFile(user:User, enumerator:Enumerator[Array[Byte]]) = {
...
val fileToSave = DefaultFileToSave(...)
gridfs.save(enumerator, fileToSave)
...
}
}
Unfortunately it seems there is no way to get an enumerator from the parsing of the request body. The only way seems to provide the Action with a parser and an Iteratee that will be fed with the the body being parsed.
I couldn't figure out how to achieve it in a reactive way (without using a temporary file or storing the body in memory). Is it at all possible?
Actually, you might consider not using girdFS built-in parser at all:
val gfs = new GridFS(db)
// the controller method, Authenticated here is custom object extending ActionBuilder
def upload = Authenticated.async(parse.multipartFormData) { request =>
...
request.body.file("photo") match {
// handle error cases
...
case Some(photo) =>
val fileToSave = DefaultFileToSave(photo.filename, photo.contentType)
// here some more operations, basically you don't need the and need only photo.ref.file
val enumerator = Enumerator(Image(photo.ref.file).fitToWidth(120).write)
gfs.save(enumerator, fileToSave) map {
//handle responses and stuff
...
}
}
}
}