MobileFirst 8 : Unexpected error encountered while storing data - ibm-mobilefirst

We are using UserAuthenticationSecurityCheck to authenticate user.
If verification is successful, the MFP server will store the user attributes.
public class AuthSecurityCheck extends UserAuthenticationSecurityCheck {
static Logger logger = Logger.getLogger(AuthSecurityCheck.class.getName());
private String userId, displayName;
private JSONObject attrObject;
private String errorMessage;
#Override
protected AuthenticatedUser createUser() {
Map<String, Object> userAttrsMap = new HashMap<String, Object>();
userAttrsMap.put("attributes",attrObject);
return new AuthenticatedUser(userId, displayName, this.getName(), userAttrsMap);
}
...
}
But if we store larger data(when userAttrsMap is large enough), we will get the 500 error.
errorMsg: Unexpected error encountered while storing data
The error is shown below:
Full source is on Github: https://github.com/DannyYang/PMR_CreateUserStoredLargeData
MFP version:
cordova-plugin-mfp 8.0.2017102115
MFP DevelopKit : 8.0.0.00-20171024-064640

The issue happens owing to the size of the data you are holding within the AuthenticatedUser object and thereby the Securitycheck's state.
MFP runtime saves the state of the securitycheck along with all the attributes to the attribute store . This involves serializing the security check state and persisting it to the DB. With a large object ( the custom map you have) this persistence operation fails and ends in a transaction rollback. This happens because the data you are trying to persist is too big and exceeds the allocated size.
SecurityCheck’s design consideration is to use it for a security check ( validation) and creating an identity object. Within your security check implementation, you have the following:
//Here the large data is assigned to the variable.
attrObject = JSONObject.parse(largeJSONString);
//This data is set into the AuthenticatedUser object.
Map<String, Object> userAttrsMap = new HashMap<String, Object>();
userAttrsMap.put("attributes",attrObject);
return new AuthenticatedUser(userId, displayName, this.getName(), userAttrsMap);
In this scenario this large data becomes part of the Securitycheck itself and will be serialized and attempted for persistence into the attribute store. When this data does not fit in the column, the transaction is rolled back and the error condition is propagated to the end user. Hence the error message you see - ” Unexpected error occured while storing data”. Enabling detailed trace will indicate the actual cause of the issue in the server trace logs.
Either way, this approach is not recommended at all in production systems because:
a) Every request from the client reaching the server goes through security introspection which will involve the server to load, check and update the securitycheck’s state. On systems taking heavy load ( production ) this can and will have performance costs. The process involves serializing the data and deserializing it later. In a distributed topology ( cluster or farms ) the request may end up in any of the nodes and these nodes will have to load and later save the security check's state to the store. All this will impact performance of your system.
b) At the end of successful authentication, the AuthenticatedUser object is propagated to the client application indicating completion of the login flow . Even if the SecurityCheck state were to be stored successfully in the attribute store ( with large data) transmitting large payloads over the network just to indicate successful login will be counter productive. For the enduser it may appear as if nothing has happened since they entered the credentials, while data indicating success is still getting downloaded.
c) Under heavy loads , the server will be strained from both a) and b) above.
You should consider cutting down the data that is propagated to the client within the authenticateduser object. Keep the data minimal within the AuthenticatedUser object. Instead, you should offload obtaining large data to resource adapters , that can be accessed post successful login .

Related

Can I allow multiple http clients to consume a Flowable stream of data with resteasy-rxjava2 / quarkus?

Currently I am able to see the streaming values exposed by the code below, but only one http client will receive the continuous stream of values, the others will not be able to.
The code, a modified version of the quarkus quickstart for kafka reactive streaming is:
#Path("/migrations")
public class StreamingResource {
private volatile Map<String, String> counterBySystemDate = new ConcurrentHashMap<>();
#Inject
#Channel("migrations")
Flowable<String> counters;
#GET
#Path("/stream")
#Produces(MediaType.SERVER_SENT_EVENTS) // denotes that server side events (SSE) will be produced
#SseElementType("text/plain") // denotes that the contained data, within this SSE, is just regular text/plain data
public Publisher<String> stream() {
Flowable<String> mainStream = counters.doOnNext(dateSystemToCount -> {
String key = dateSystemToCount.substring(0, dateSystemToCount.lastIndexOf("_"));
counterBySystemDate.put(key, dateSystemToCount);
});
return fromIterable(counterBySystemDate.values().stream().sorted().collect(Collectors.toList()))
.concatWith(mainStream)
.onBackpressureLatest();
}
}
Is it possible to make any modification that would allow multiple clients to consume the same data, in a broadcast fashion?
I guess this implies letting go of backpressure, because that would imply a state per consumer?
I saw that Observable is not accepted as a return type in the resteasy-rxjava2 for the Server Side Events media-tpe.
Please let me know any ideas,
Thank you
Please find the full code in Why in multiple connections to PricesResource Publisher, only one gets the stream?

Akka HTTP Source Streaming vs regular request handling

What is the advantage of using Source Streaming vs the regular way of handling requests? My understanding that in both cases
The TCP connection will be reused
Back-pressure will be applied between the client and the server
The only advantage of Source Streaming I can see is if there is a very large response and the client prefers to consume it in smaller chunks.
My use case is that I have a very long list of users (millions), and I need to call a service that performs some filtering on the users, and returns a subset.
Currently, on the server side I expose a batch API, and on the client, I just split the users into chunks of 1000, and make X batch calls in parallel using Akka HTTP Host API.
I am considering switching to HTTP streaming, but cannot quite figure out what would be the value
You are missing one other huge benefit: memory efficiency. By having a streamed pipeline, client/server/client, all parties safely process data without running the risk of blowing up the memory allocation. This is particularly useful on the server side, where you always have to assume the clients may do something malicious...
Client Request Creation
Suppose the ultimate source of your millions of users is a file. You can create a stream source from this file:
val userFilePath : java.nio.file.Path = ???
val userFileSource = akka.stream.scaladsl.FileIO(userFilePath)
This source can you be use to create your http request which will stream the users to the service:
import akka.http.scaladsl.model.HttpEntity.{Chunked, ChunkStreamPart}
import akka.http.scaladsl.model.{RequestEntity, ContentTypes, HttpRequest}
val httpRequest : HttpRequest =
HttpRequest(uri = "http://filterService.io",
entity = Chunked.fromData(ContentTypes.`text/plain(UTF-8)`, userFileSource))
This request will now stream the users to the service without consuming the entire file into memory. Only chunks of data will be buffered at a time, therefore, you can send a request with potentially an infinite number of users and your client will be fine.
Server Request Processing
Similarly, your server can be designed to accept a request with an entity that can potentially be of infinite length.
Your questions says the service will filter the users, assuming we have a filtering function:
val isValidUser : (String) => Boolean = ???
This can be used to filter the incoming request entity and create a response entity which will feed the response:
import akka.http.scaladsl.server.Directives._
import akka.http.scaladsl.model.HttpResponse
import akka.http.scaladsl.model.HttpEntity.Chunked
val route = extractDataBytes { userSource =>
val responseSource : Source[ByteString, _] =
userSource
.map(_.utf8String)
.filter(isValidUser)
.map(ByteString.apply)
complete(HttpResponse(entity=Chunked.fromData(ContentTypes.`text/plain(UTF-8)`,
responseSource)))
}
Client Response Processing
The client can similarly process the filtered users without reading them all into memory. We can, for example, dispatch the request and send all of the valid users to the console:
import akka.http.scaladsl.Http
Http()
.singleRequest(httpRequest)
.map { response =>
response
.entity
.dataBytes
.map(_.utf8String)
.foreach(System.out.println)
}

User destinations in a multi-server environment? (Spring WebSocket and RabbitMQ)

The documentation for Spring WebSockets states:
4.4.13. User Destinations
An application can send messages targeting a specific user, and Spring’s STOMP support recognizes destinations prefixed with "/user/" for this purpose. For example, a client might subscribe to the destination "/user/queue/position-updates". This destination will be handled by the UserDestinationMessageHandler and transformed into a destination unique to the user session, e.g. "/queue/position-updates-user123". This provides the convenience of subscribing to a generically named destination while at the same time ensuring no collisions with other users subscribing to the same destination so that each user can receive unique stock position updates.
Is this supposed to work in a multi-server environment with RabbitMQ as broker?
As far as I can tell, the queue name for a user is generated by appending the simpSessionId. When using the recommended client library stomp.js this results in the first user getting the queue name "/queue/position-updates-user0", the next gets "/queue/position-updates-user1" and so on.
This in turn means the first users to connect to different servers will subscribe to the same queue ("/queue/position-updates-user0").
The only reference to this I can find in the documentation is this:
In a multi-application server scenario a user destination may remain unresolved because the user is connected to a different server. In such cases you can configure a destination to broadcast unresolved messages to so that other servers have a chance to try. This can be done through the userDestinationBroadcast property of the MessageBrokerRegistry in Java config and the user-destination-broadcast attribute of the message-broker element in XML.
But this only makes the it possible to communicate with a user from a different server than the one where the web socket is established.
I feel I'm missing something? Is there anyway to configure Spring to be able to safely use MessagingTemplate.convertAndSendToUser(principal.getName(), destination, payload) in a multi-server environment?
If they need to be authenticated (I assume their credentials are stored in a database) you can always use their database unique user id to subscribe to.
What I do is when a user logs in they are automatically subscribed to two topics an account|system topic for system wide broadcasts and account|<userId> topic for specific broadcasts.
You could try something like notification|<userid> for each person to subscribe to then send messages to that topic and they will receive it.
Since user Ids are unique to each user you shouldn't have an issue within a clustered environment as long as each environment is hitting the same database information.
Here is my send method:
public static boolean send(Object msg, String topic) {
try {
String destination = topic;
String payload = toJson(msg); //jsonfiy the message
Message<byte[]> message = MessageBuilder.withPayload(payload.getBytes("UTF-8")).build();
template.send(destination, message);
return true;
} catch (Exception ex) {
logger.error(CommService.class.getName(), ex);
return false;
}
}
My destinations are preformatted so if i want to send a message to user with id of one the destinations looks something like /topic/account|1.
Ive created a ping pong controller that tests websockets for users who connect to see if their environment allows for websockets. I don't know if this will help you but this does work in my clustered environment.
/**
* Play ping pong between the client and server to see if web sockets work
* #param input the ping pong input
* #return the return data to check for connectivity
* #throws Exception exception
*/
#MessageMapping("/ping")
#SendToUser(value="/queue/pong", broadcast=false) // send only to the session that sent the request
public PingPong ping(PingPong input) throws Exception {
int receivedBytes = input.getData().length;
int pullBytes = input.getPull();
PingPong response = input;
if (pullBytes == 0) {
response.setData(new byte[0]);
} else if (pullBytes != receivedBytes) {
// create random byte array
byte[] data = randomService.nextBytes(pullBytes);
response.setData(data);
}
return response;
}

Ignite and Kafka Integration

I am trying the Ignite and Kafka Integration to bring kafka message into Ignite cache.
My message key is a random string(To work with Ignite, the kafka message key can't be null), and the value is a json string representation for Person(a java class)
When Ignite receives such a message, it looks that Ignite will use the message's key(the random string in my case) as the cache key.
Is it possible to change the message key to the person's id, so that I can put the into the cache.
Looks that streamer.receiver(new StreamReceiver) is workable
streamer.receiver(new StreamReceiver<String, String>() {
public void receive(IgniteCache<String, String> cache, Collection<Map.Entry<String, String>> entries) throws IgniteException {
for (Map.Entry<String, String> entry : entries) {
Person p = fromJson(entry.getValue());
//ignore the message key,and use person id as the cache key
cache.put(p.getId(), p);
}
}
});
Is this the recommended way? and I am not sure whether calling cache.put in StreamReceiver is a correct way, since it is only a pre-processing step before writing to cache.
Data streamer will map all your keys to cache affinity nodes, create batches of entries and send batches to affinity nodes. After it StreamReceiver will receive your entries, get Person's ID and invoke cache.put(K, V). Putting entry lead to mapping your key to corresponding cache affinity node and sending update request to this node.
Everything looks good. But result of mapping your random key from Kafka and result of mapping Person's ID will be different (most likely different nodes). As result your will get poor performance due to redundant network hops.
Unfortunately, current KafkaStreamer implementations doesn't support stream tuple extractors (see e.g. StreamSingleTupleExtractor class). But you can easily create your own Kafka streamer implementation using existing one as example.
Also you can try use KafkaStreamer's keyDecoder and valDecoder in order to extract Person's ID from Kafka message. I don't sure, but it can help.

Multitenant/Shared Application system, how to maintain multiple tentant-specific identifiers?

I have a multi-tenant system where each tenant shares the same instance of the codebase, but has their own databases.
I'm using RavenDB for persistence, with a standard c# facade/BLL backend wrapped with Asp.net WebAPI, and I'm finding that at every lower level operation (deep within my business logic classes) that touch the datbase, I need to pass in an identifier so that my RavenDb client session knows which database to operate against.
When the user authenticates, I resolve the appropriate database identifer, store it in the session manager. Every call against the Web API layer passes in a session ID which resolves the database ID in the backend, which is then used to pass into every single facade/BLL call.
All my dependencies are handled via an IoC container at the WebAPI level, but i can't pass in the database ID at this phase because it can be different for every user that is logged in.
this, of course is getting tedious.
can someone give me some guidance as to what I can do to alleviate this? Maybe perhaps some sort of policy injection/AOP solution?
a rough sample of my backend code looks like..
public class WidgetService()
{
private WidgetBLL _widgetBLL;
private ISessionManager _sessionManager;
public WidgetService(WidgetBLL _widgetBLL, ISessionManager sessionManager)
{
_widgetBLL = widgetBLL;
_sessionManager = sessionManager
}
public Widget getWidget(string sessionId, string widgetId)
{
string DbId = sessionManager.ResolveDbId(sessionId)
return _widgetBLL.GetWidget(string dbId, string widgetId);
}
}
public class WidgetManager()
{
public GetWidget(string dbId, string widgetId)
{
using (IDocumentSession session = documentStore.OpenSession(dbId)
{
var widget = session.load<Widget>(widgetid);
}
return widget;
}
}
the DBID is the identifier for that particular tenant that this particular user is a member of.
You need to change how you are using the session.
Instead of opening and closing the session yourself, do that in the IoC code.
Then you pass a session that is already opened for the right db.