I am using Webflux in Spring Boot 2.0.3.RELEASE to create REST API. With that implementation, I customize and use the webSessionManager as below.
#EnableWebFluxSecurity
#Configuration
class SecurityConfiguration {
#Bean
fun webSessionManager(): WebSessionManager {
return DefaultWebSessionManager().apply {
sessionIdResolver = HeaderWebSessionIdResolver().apply {
headerName = "X-Sample"
}
sessionStore = InMemoryWebSessionStore()
}
}
// ...
}
And in order to test the REST API, I created a test code as follows. (addUser and signin are extension functions.)
#RunWith(SpringRunner::class)
#SpringBootTest
#AutoConfigureWebTestClient
#FixMethodOrder(MethodSorters.NAME_ASCENDING)
class UserTests {
#Autowired
private lateinit var client: WebTestClient
#Test
fun testGetUserInfo() {
client.addUser(defaultUser)
val sessionKey = client.signin(defaultUser)
client.get().uri(userPath)
.header("X-Sample", sessionKey)
.exchange()
.expectStatus().isOk
.expectBody()
.jsonInStrict("""
{
"user": {
"mail_address": "user#example.com"
}
}
""".trimIndent())
}
// ...
}
The test failed. It is refused by authorization. However, if I start the server and run it from curl it will succeed in the authorization.
After investigating the cause, it turned out that org.springframework.test.web.reactive.server.AbstractMockServerSpec set webSessionManager to DefaultWebSessionManager. Default is used, not the webSessionManager I customized. For this reason, it could not get the session ID.
AbstractMockServerSpec.java#L41
AbstractMockServerSpec.java#L72-L78
How can I change the webSessionManager of AbstractMockServerSpec?
Also, I think that it is better to have the following implementation, what do you think?
abstract class AbstractMockServerSpec<B extends WebTestClient.MockServerSpec<B>>
implements WebTestClient.MockServerSpec<B> {
// ...
private WebSessionManager sessionManager = null;
// ...
#Override
public WebTestClient.Builder configureClient() {
WebHttpHandlerBuilder builder = initHttpHandlerBuilder();
builder.filters(theFilters -> theFilters.addAll(0, this.filters));
if (this.sessionManager != null) {
builder.sessionManager(this.sessionManager);
}
this.configurers.forEach(configurer -> configurer.beforeServerCreated(builder));
return new DefaultWebTestClientBuilder(builder);
}
// ...
}
Spring Framework's AbstractMockServerSpec is providing a method to customize the WebSessionManager already.
Thanks for opening SPR-17094, this problem will be solved with that ticket - the AbstractMockServerSpec is already looking into the application context for infrastructure bits, this should check for a WebSessionManager as well.
Related
I'm using a spring security in my app. I need it to secure two api's endpoints. Here is security config:
#Configuration
#EnableWebSecurity
class SecurityConfig(
private val authProps: AuthenticationProperties
) : WebSecurityConfigurerAdapter() {
override fun configure(auth: AuthenticationManagerBuilder) {
val encoder = PasswordEncoderFactories.createDelegatingPasswordEncoder()
auth.inMemoryAuthentication()
.passwordEncoder(encoder)
.withUser(authProps.user)
.password(encoder.encode(authProps.password))
.roles("USER")
}
override fun configure(http: HttpSecurity) {
http.csrf().disable()
.authorizeRequests()
.antMatchers("/firstapi/**", "/secondapi/**").authenticated()
.anyRequest().permitAll()
.and()
.httpBasic()
}
}
Also, i'm integrating with an external service, that uses oauth2, here it's restTemplate config -
#Configuration
class ExternalServiceConfiguration {
#Bean("externalProperties")
#ConfigurationProperties("http.external.api")
fun externalHttpClientConfig() = HttpClientProperties()
#Bean("externalDetails")
#ConfigurationProperties("http.external.api.security.oauth2")
fun externalOAuth2Details() = ResourceOwnerPasswordResourceDetails()
#Bean("externalRestTemplate")
fun externalClientRestTemplate(
#Qualifier("externalProperties") externalHttpClientProperties: HttpClientProperties,
#Qualifier("externalDetails") externalOAuth2Details: ResourceOwnerPasswordResourceDetails,
customizerProviders: ObjectProvider<RestTemplateCustomizer>,
objectMapper: ObjectMapper,
): RestTemplate {
val template = OAuth2RestTemplate(externalOAuth2Details).apply {
messageConverters = listOf(MappingJackson2HttpMessageConverter(objectMapper))
requestFactory = requestFactory(externalHttpClientProperties)
errorHandler = IntegrationResponseErrorHandler()
}
customizerProviders.orderedStream().forEach { it.customize(template) }
return template
}
}
Somehow spring-security-oauth clashes with spring-security. When i try to obtain a token i'm failing at lib class method:
AccessTokenProviderChain.obtainAccessToken(...)
, because instead of null i have an AnonymousAuthenticationToken authentication at context, when it calls
SecurityContextHolder.getContext().getAuthentication()
So spring security merges anonymous context somehow, i am not sure when and how, so that it affects external calls.
Could anyone help me with advice, how can i find the collision???
I trying to create API with ktor.
Test for ktor api using JUnit5 and I need use the custom properties for database information in 'Application.conf' like below:
ktor {
deployment {
.....
}
application {
.....
}
database {
host = ${?DB_HOST}
user = ${?DB_USER}
pass = ${?DB_PASS}
}
}
I following official guidline for useing custom properties:
HoconApplicationConfig
private val testEnv = createTestEnvironment {
config = HoconApplicationConfig(ConfigFactory.load("application.conf"))
}
class MyTest {
#Test
fun Testing() {
withApplication(testEnv) {
handleRequest(HttpMethod.Get, "/foo").apply {
...
}
}
}
}
Succeed test when only one function in class, but defined second test function then test failed with Exception.
I want define multiple function to simplify the definition and checking in case of failure.
class MyTest {
#Test
fun Testing() {
withApplication(testEnv) {
handleRequest(HttpMethod.Get, "/foo").apply {
....
}
}
}
#Test
fun Testing2() {
withApplication(testEnv) {
handleRequest(HttpMethod.Get, "/foo/bar").apply {
...
}
}
}
}
io.ktor.server.engine.BaseApplicationResponse$ResponseAlreadySentException: Response has already been sent
I know all test will success using 'withTestApplication' but can't use the custom properties.
How can I use the custom properties, and define multiple test funcitons?
Environment:
macOS Big Sur 11.5
IntelliJ Ultimate 2021.1.3
kotlin 1.5.21
ktor 1.6.1
junit-jupiter 5.7.0
Sorry for bad English.
Best Regards.
I'm trying to implement transactions with spring-data-r2dbc repositories in combination with the TransactionalDatabaseClient as such:
class SongService(
private val songRepo: SongRepo,
private val databaseClient: DatabaseClient
){
private val tdbc = databaseClient as TransactionalDatabaseClient
...
...
fun save(song: Song){
return tdbc.inTransaction{
songRepo
.save(mapRow(song, albumId)) //Mapping to a row representation
.delayUntil { savedSong -> tdbc.execute.sql(...).fetch.rowsUpdated() } //saving a many to many relation
.map(::mapSong) //Mapping back to actual song and retrieve the relationship data.
}
}
}
I currently have a config class (annotated with #Configuration and #EnableR2dbcRepositories) that extends from AbstractR2dbcConfiguration. In here I override the databaseClient method to return a TransactionalDatabaseClient. This should be the same instance as in the SongService class.
When running the code in a test with just subscribing and printing, I get org.springframework.transaction.NoTransactionException: ReactiveTransactionSynchronization not active and the relationship data is not returned.
When using project Reactors stepverifier though, i get java.lang.IllegalStateException: Connection is closed. Also in this case, the relationship data is not returned.
Just for the record, I have seen https://github.com/spring-projects/spring-data-r2dbc/issues/44
Here is a working Java example:
#Autowired TransactionalDatabaseClient txClient;
#Autowired Mono<Connection> connection;
//You Can also use: #Autowired Mono<? extends Publisher> connectionPublisher;
public Flux<Void> example {
txClient.enableTransactionSynchronization(connection);
// Or, txClient.enableTransactionSynchronization(connectionPublisher);
Flux<AuditConfigByClub> audits = txClient.inTransaction(tx -> {
txClient.beginTransaction();
return tx.execute().sql("SELECT * FROM audit.items")
.as(Item.class)
.fetch()
.all();
}).doOnTerminate(() -> {
txClient.commitTransaction();
});
txClient.commitTransaction();
audits.subscribe(item -> System.out.println("anItem: " + item));
return Flux.empty()
}
I just started reactive so not too sure what I'm doing with my callbacks haha. But I decided to go with TransactionalDatabaseClient over DatabaseClient or Connection since I'll take all the utility I can get while R2dbc is in its current state.
In your code did you actually instantiate a Connection object? If so I think you would have done it in your configuration. It can be utilized throughout the app the same as DatabaseClient, but it is slightly more intricate.
If not:
#Bean
#Override // I also used abstract config
public ConnectionFactory connectionFactory() {
...
}
#Bean
TransactionalDatabaseClient txClient() {
...
}
//TransactionalDatabaseClient will take either of these as arg in
//#enableTransactionSynchronization method
#Bean
public Publisher<? extends Connection> connectionPublisher() {
return connectionFactory().create();
}
#Bean
public Mono<Connection> connection() {
return = Mono.from(connectionFactory().create());
}
If you are having problems translating to Kotlin, there is an alternative way to enable synchronization that could work:
// From what I understand, this is a useful way to move between
// transactions within a single subscription
TransactionResources resources = TransactionResources.create();
resources.registerResource(Resource.class, resource);
ConnectionFactoryUtils
.currentReactiveTransactionSynchronization()
.subscribe(currentTx -> sync.registerTransaction(Tx));
Hope this translates well for Kotlin.
I've tried to use WebClient with LoadBalancerExchangeFilterFunction:
WebClient config:
#Bean
public WebClient myWebClient(final LoadBalancerExchangeFilterFunction lbFunction) {
return WebClient.builder()
.filter(lbFunction)
.defaultHeader(ACCEPT, APPLICATION_JSON_VALUE)
.defaultHeader(CONTENT_ENCODING, APPLICATION_JSON_VALUE)
.build();
}
Then I've noticed that calls to underlying service are not properly load balanced - there is constant difference of RPS on each instance.
Then I've tried to move back to RestTemplate. And it's working fine.
Config for RestTemplate:
private static final int CONNECT_TIMEOUT_MILLIS = 18 * DateTimeConstants.MILLIS_PER_SECOND;
private static final int READ_TIMEOUT_MILLIS = 18 * DateTimeConstants.MILLIS_PER_SECOND;
#LoadBalanced
#Bean
public RestTemplate restTemplateSearch(final RestTemplateBuilder restTemplateBuilder) {
return restTemplateBuilder
.errorHandler(errorHandlerSearch())
.requestFactory(this::bufferedClientHttpRequestFactory)
.build();
}
private ClientHttpRequestFactory bufferedClientHttpRequestFactory() {
final SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setConnectTimeout(CONNECT_TIMEOUT_MILLIS);
requestFactory.setReadTimeout(READ_TIMEOUT_MILLIS);
return new BufferingClientHttpRequestFactory(requestFactory);
}
private ResponseErrorHandler errorHandlerSearch() {
return new DefaultResponseErrorHandler() {
#Override
public boolean hasError(ClientHttpResponse response) throws IOException {
return response.getStatusCode().is5xxServerError();
}
};
}
Load balancing using WebClient config up to 11:25, then switching back to RestTemplate:
Is there a reason why there is such difference and how I can use WebClient to have same amount of RPS on each instance? Clue might be that older instances are getting more requests than new ones.
I've tried bit of debugging and same (defaults like ZoneAwareLoadBalancer) logic is being called.
I did simple POC and everything works exactly the same with web client and rest template for default configuration.
Rest server code:
#SpringBootApplication
internal class RestServerApplication
fun main(args: Array<String>) {
runApplication<RestServerApplication>(*args)
}
class BeansInitializer : ApplicationContextInitializer<GenericApplicationContext> {
override fun initialize(context: GenericApplicationContext) {
serverBeans().initialize(context)
}
}
fun serverBeans() = beans {
bean("serverRoutes") {
PingRoutes(ref()).router()
}
bean<PingHandler>()
}
internal class PingRoutes(private val pingHandler: PingHandler) {
fun router() = router {
GET("/api/ping", pingHandler::ping)
}
}
class PingHandler(private val env: Environment) {
fun ping(serverRequest: ServerRequest): Mono<ServerResponse> {
return Mono
.fromCallable {
// sleap added to simulate some work
Thread.sleep(2000)
}
.subscribeOn(elastic())
.flatMap {
ServerResponse.ok()
.syncBody("pong-${env["HOSTNAME"]}-${env["server.port"]}")
}
}
}
In application.yaml add:
context.initializer.classes: com.lbpoc.server.BeansInitializer
Dependencies in gradle:
implementation('org.springframework.boot:spring-boot-starter-webflux')
Rest client code:
#SpringBootApplication
internal class RestClientApplication {
#Bean
#LoadBalanced
fun webClientBuilder(): WebClient.Builder {
return WebClient.builder()
}
#Bean
#LoadBalanced
fun restTemplate() = RestTemplateBuilder().build()
}
fun main(args: Array<String>) {
runApplication<RestClientApplication>(*args)
}
class BeansInitializer : ApplicationContextInitializer<GenericApplicationContext> {
override fun initialize(context: GenericApplicationContext) {
clientBeans().initialize(context)
}
}
fun clientBeans() = beans {
bean("clientRoutes") {
PingRoutes(ref()).router()
}
bean<PingHandlerWithWebClient>()
bean<PingHandlerWithRestTemplate>()
}
internal class PingRoutes(private val pingHandlerWithWebClient: PingHandlerWithWebClient) {
fun router() = org.springframework.web.reactive.function.server.router {
GET("/api/ping", pingHandlerWithWebClient::ping)
}
}
class PingHandlerWithWebClient(private val webClientBuilder: WebClient.Builder) {
fun ping(serverRequest: ServerRequest) = webClientBuilder.build()
.get()
.uri("http://rest-server-poc/api/ping")
.retrieve()
.bodyToMono(String::class.java)
.onErrorReturn(TimeoutException::class.java, "Read/write timeout")
.flatMap {
ServerResponse.ok().syncBody(it)
}
}
class PingHandlerWithRestTemplate(private val restTemplate: RestTemplate) {
fun ping(serverRequest: ServerRequest) = Mono.fromCallable {
restTemplate.getForEntity("http://rest-server-poc/api/ping", String::class.java)
}.flatMap {
ServerResponse.ok().syncBody(it.body!!)
}
}
In application.yaml add:
context.initializer.classes: com.lbpoc.client.BeansInitializer
spring:
application:
name: rest-client-poc-for-load-balancing
logging:
level.org.springframework.cloud: DEBUG
level.com.netflix.loadbalancer: DEBUG
rest-server-poc:
listOfServers: localhost:8081,localhost:8082
Dependencies in gradle:
implementation('org.springframework.boot:spring-boot-starter-webflux')
implementation('org.springframework.cloud:spring-cloud-starter-netflix-ribbon')
You can try it with two or more instances for server and it works exactly the same with web client and rest template.
Ribbon use by default zoneAwareLoadBalancer and if you have only one zone all instances for server will be registered in "unknown" zone.
You might have a problem with keeping connections by web client. Web client reuse the same connection in multiple requests, rest template do not do that. If you have some kind of proxy between your client and server then you might have a problem with reusing connections by web client. To verify it you can modify web client bean like this and run tests:
#Bean
#LoadBalanced
fun webClientBuilder(): WebClient.Builder {
return WebClient.builder()
.clientConnector(ReactorClientHttpConnector { options ->
options
.compression(true)
.afterNettyContextInit { ctx ->
ctx.markPersistent(false)
}
})
}
Of course it's not a good solution for production but doing that you can check if you have a problem with configuration inside your client application or maybe problem is outside, something between your client and server. E.g. if you are using kubernetes and register your services in service discovery using server node IP address then every call to such service will go though kube-proxy load balancer and will be (by default round robin will be used) routed to some pod for that service.
You have to configure Ribbon config to modify the load balancing behavior (please read below).
By default (which you have found yourself) the ZoneAwareLoadBalancer is being used. In the source code for ZoneAwareLoadBalancer we read:
(highlighted by me are some mechanics which could result in the RPS pattern you see):
The key metric used to measure the zone condition is Average Active Requests, which is aggregated per rest client per zone. It is the
total outstanding requests in a zone divided by number of available targeted instances (excluding circuit breaker tripped instances).
This metric is very effective when timeout occurs slowly on a bad zone.
The LoadBalancer will calculate and examine zone stats of all available zones. If the Average Active Requests for any zone has reached a configured threshold, this zone will be dropped from the active server list. In case more than one zone has reached the threshold, the zone with the most active requests per server will be dropped.
Once the the worst zone is dropped, a zone will be chosen among the rest with the probability proportional to its number of instances.
If your traffic is being served by one zone (perhaps the same box?) then you might get into some additionally confusing situations.
Please also note that without using LoadBallancedFilterFunction the average RPS is the same as when you use it (on the graph all lines converge to the median line) after the change, so globally looking both load balancing strategies consume the same amount of available bandwidth but in a different manner.
To modify your Ribbon client settings, try following:
public class RibbonConfig {
#Autowired
IClientConfig ribbonClientConfig;
#Bean
public IPing ribbonPing (IClientConfig config) {
return new PingUrl();//default is a NoOpPing
}
#Bean
public IRule ribbonRule(IClientConfig config) {
return new AvailabilityFilteringRule(); // here override the default ZoneAvoidanceRule
}
}
Then don't forget to globally define your Ribbon client config:
#SpringBootApplication
#RibbonClient(name = "app", configuration = RibbonConfig.class)
public class App {
//...
}
Hope this helps!
I got something like this:
private val client = HttpClient {
install(JsonFeature) {
serializer = GsonSerializer()
}
install(ExpectSuccess)
}
and make request like
private fun HttpRequestBuilder.apiUrl(path: String, userId: String? = null) {
header(HttpHeaders.CacheControl, "no-cache")
url {
takeFrom(endPoint)
encodedPath = path
}
}
but I need to check request and response body, is there any way to do it? in console/in file?
You can achieve this with the Logging feature.
First add the dependency:
implementation "io.ktor:ktor-client-logging-native:$ktor_version"
Then install the feature:
private val client = HttpClient {
install(Logging) {
logger = Logger.DEFAULT
level = LogLevel.ALL
}
}
Bonus:
If you need to have multiple HttpClient instances throughout your application and you want to reuse some of the configuration, then you can create an extension function and add the common logic in there. For example:
fun HttpClientConfig<*>.default() {
install(Logging) {
logger = Logger.DEFAULT
level = LogLevel.ALL
}
// Add all the common configuration here.
}
And then initialize your HttpClient like this:
private val client = HttpClient {
default()
}
I ran into this as well. I switched to using the Ktor OkHttp client as I'm familiar with the logging mechanism there.
Update your pom.xml or gradle.build to include that client (copy/paste from the Ktor site) and also add the OkHttp Logging Interceptor (again, copy/paste from that site). Current version is 3.12.0.
Now configure the client with
val client = HttpClient(OkHttp) {
engine {
val loggingInterceptor = HttpLoggingInterceptor()
loggingInterceptor.level = Level.BODY
addInterceptor(loggingInterceptor)
}
}
Regardless of which client you use or framework you are on, you can implement your own logger like so:
private val client = HttpClient {
// Other configurations...
install(Logging) {
logger = CustomHttpLogger()
level = LogLevel.BODY
}
}
Where CustomHttpLogger is any class that implements the ktor Logger interface, like so:
import io.ktor.client.features.logging.Logger
class CustomHttpLogger(): Logger {
override fun log(message: String) {
Log.d("loggerTag", message) // Or whatever logging system you want here
}
}
You can read more about the Logger interface in the documentation here or in the source code here
It looks like we should handle the response in HttpReceivePipeline. We could clone the origin response and use it for logging purpose:
scope.receivePipeline.intercept(HttpReceivePipeline.Before) { response ->
val (loggingContent, responseContent) = response.content.split(scope)
launch {
val callForLog = DelegatedCall(loggingContent, context, scope, shouldClose = false)
....
}
...
}
The example implementation could be found here: https://github.com/ktorio/ktor/blob/00369bf3e41e91d366279fce57b8f4c97f927fd4/ktor-client/ktor-client-core/src/io/ktor/client/features/observer/ResponseObserver.kt
and would be available in next minor release as a client feature.
btw: we could implement the same scheme for the request.
A custom structured log can be created with the HttpSend plugin
Ktor 2.x:
client.plugin(HttpSend).intercept { request ->
val call = execute(request)
val response = call.response
val durationMillis = response.responseTime.timestamp - response.requestTime.timestamp
Log.i("NETWORK", "[${response.status.value}] ${request.url.build()} ($durationMillis ms)")
call
}
Ktor 1.x:
client.config {
install(HttpSend) {
intercept { call, _ ->
val request = call.request
val response = call.response
val durationMillis = response.responseTime.timestamp - response.requestTime.timestamp
Log.i("NETWORK", "[${response.status.value}] ${request.url} ($durationMillis ms)")
call
}
}
}
Check out Kotlin Logging, https://github.com/MicroUtils/kotlin-logging it isused by a lot of open source frameworks and takes care of all the prety printing.
You can use it simply like this:
private val logger = KotlinLogging.logger { }
logger.info { "MYLOGGER INFO" }
logger.warn { "MYLOGGER WARNING" }
logger.error { "MYLOGGER ERROR" }
This will print the messages on the console.