Does an Azure VM throttle SignalR messages being sent? Running locally, the client receives every message, but when hosted on the VM, clients only receive messages 30% of the time!?!?
This question is about Microsoft.AspNet.SignalR nuget package for SignalR in .Net Core 3.1 API back end, with a VueJS SPA front-end, all being hosted on an Azure Windows Server 2016 VM using IIS 10.
On my local machine, SignalR works perfectly. Messages get sent/received all the time, instantaneously. Then I publish to the VM, and when (IF) the WebSocket connection is successful, the client can only receive messages for the first 5 or so seconds at most, then stops receiving messages.
I've set up a dummy page that sends a message to my API, which then sends said message back down to all connections. It's a simple input form and "Send" button. After the few seconds of submitting and receiving messages, I need to rapidly submit (even hold down the "enter" button to submit) the form and send what should be a constant stream of messages back, until, low and behold, several seconds later messages begin to be received again, but only for a few seconds.
I've actually held down the submit button for constant stream and timed how long it takes to start getting messages, then again how long it takes to stop receiving. My small sample shows ~30 messages get received, then skips (does not receive) the next ~70 messages until another ~30 messages come in .. then the pattern persists, no messages for several seconds, then (~30) messages for a few seconds.
Production Environment Continuously Sending 1000 messages:
Same Test in Local Environment Sending 1000 messages:
If I stop the test, not matter how long I wait, when I hold the enter button (repeated submit), it takes a few seconds to get back into the 3 second/2 second pattern. It's almost as if I need to keep pressuring the server to send the message back to the client, otherwise the server gets lazy and doesn't do any work at all. If I slow play the message submits, it's rare that the client receives any messages at all. I really need to persistently and quickly send messages in order to start receiving them again.
FYI, during the time that I am holding down submit, or rapidly submitting, I receive no errors for API calls (initiating messages) and no errors for Socket Connection or receiving messages. All the while, when client side SignalR log level is set to Trace, I see ping requests being sent and received successfully every 10 seconds.
Here is the Socket Config in .Net:
services.AddSignalR()
.AddHubOptions<StreamHub>(hubOptions => {
hubOptions.EnableDetailedErrors = true;
hubOptions.ClientTimeoutInterval = TimeSpan.FromHours(24);
hubOptions.HandshakeTimeout = TimeSpan.FromHours(24);
hubOptions.KeepAliveInterval = TimeSpan.FromSeconds(15);
hubOptions.MaximumReceiveMessageSize = 1000000;
})
.AddJsonProtocol(options =>
{
options.PayloadSerializerOptions.PropertyNamingPolicy = null;
});
// Adding Authentication
services.AddAuthentication(options =>
{
options.DefaultAuthenticateScheme = JwtBearerDefaults.AuthenticationScheme;
options.DefaultChallengeScheme = JwtBearerDefaults.AuthenticationScheme;
options.DefaultScheme = JwtBearerDefaults.AuthenticationScheme;
})
// Adding Jwt Bearer
.AddJwtBearer(options =>
{
options.SaveToken = true;
options.RequireHttpsMetadata = false;
options.TokenValidationParameters = new TokenValidationParameters()
{
ValidateIssuer = true,
ValidateAudience = true,
ValidAudience = Configuration["JWT:ValidAudience"],
ValidIssuer = Configuration["JWT:ValidIssuer"],
IssuerSigningKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes(Configuration["JWT:Secret"])),
ClockSkew = TimeSpan.Zero
};
// Sending the access token in the query string is required due to
// a limitation in Browser APIs. We restrict it to only calls to the
// SignalR hub in this code.
// See https://learn.microsoft.com/aspnet/core/signalr/security#access-token-logging
// for more information about security considerations when using
// the query string to transmit the access token.
options.Events = new JwtBearerEvents
{
OnMessageReceived = context =>
{
var accessToken = context.Request.Query["access_token"];
// If the request is for our hub...
var path = context.HttpContext.Request.Path;
if (!string.IsNullOrEmpty(accessToken) && (path.StartsWithSegments("/v1/stream")))
{
// Read the token out of the query string
context.Token = accessToken;
}
return Task.CompletedTask;
}
};
});
I use this endpoint to send back messages:
[HttpPost]
[Route("bitcoin")]
public async Task<IActionResult> SendBitcoin([FromBody] BitCoin bitcoin)
{
await this._hubContext.Clients.All.SendAsync("BitCoin", bitcoin.message);
return Ok(bitcoin.message);
}
Here is the Socket Connection in JS and the button click to call message API:
this.connection = new signalR.HubConnectionBuilder()
.configureLogging(process.env.NODE_ENV.toLowerCase() == 'development' ? signalR.LogLevel.None : signalR.LogLevel.None)
.withUrl(process.env.VUE_APP_STREAM_ROOT, { accessTokenFactory: () => this.$store.state.auth.token })
.withAutomaticReconnect({
nextRetryDelayInMilliseconds: retryContext => {
if(retryContext.retryReason && retryContext.retryReason.statusCode == 401) {
return null
}
else if (retryContext.elapsedMilliseconds < 3600000) {
// If we've been reconnecting for less than 60 minutes so far,
// wait between 0 and 10 seconds before the next reconnect attempt.
return Math.random() * 10000;
} else {
// If we've been reconnecting for more than 60 seconds so far, stop reconnecting.
return null;
}
}
})
.build()
// connection timeout of 10 minutes
this.connection.serverTimeoutInMilliseconds = 1000 * 60 * 10
this.connection.reconnectedCallbacks.push(() => {
let alert = {
show: true,
text: 'Data connection re-established!',
variant: 'success',
isConnected: true,
}
this.$store.commit(CONNECTION_ALERT, alert)
setTimeout(() => {
this.$_closeConnectionAlert()
}, this.$_appMessageTimeout)
// this.joinStreamGroup('event-'+this.event.eventId)
})
this.connection.onreconnecting((err) => {
if(!!err) {
console.log('reconnecting:')
this.startStream()
}
})
this.connection.start()
.then((response) => {
this.startStream()
})
.catch((err) => {
});
startStream() {
// ---------
// Call client methods from hub
// ---------
if(this.connection.connectionState.toLowerCase() == 'connected') {
this.connection.methods.bitcoin = []
this.connection.on("BitCoin", (data) => {
console.log('messageReceived:', data)
})
}
}
buttonClick() {
this.$_apiCall({url: 'bitcoin', method: 'POST', data: {message:this.message}})
.then(response => {
// console.log('message', response.data)
})
}
For the case when the Socket Connection fails:
On page refresh, sometimes the WebSocket Connection fails, but there are multiple calls to the Socket endpoint that are almost identical, where one returns 404 and another returns a 200 result
Failed Request
This is the request that failed, the only difference to the request that succeeded (below) is the content-length in the Response Headers (highlighted). The Request Headers are identical:
Successful request to Socket Endpoint
Identical Request Headers
What could be so different about the configuration on my local machine vs. the configuration on my Azure VM? Why would the client stop and start receiving messages like this? Might it be the configuration on my VM? Are the messages getting blocked somehow? I've exhausted myself trying to figure this out!!
Update:
KeepAlive messages are being sent correctly, but the continuous stream of messages sent (expecting received) only works periodically.
Here we see that the KeepAlive messages are being sent and received every 15 seconds, as expected.
Related
I have a web app using Angular and Express. I have a rest api for database updates and I am also using SocketIO to update the clients in realtime.
I am tracking a list of active socket IDs for each user but now I would like to have access to the clients socket id in my express route so that I can emit a message to all other users (equivalent of socket.broadcast)
I'm trying to store the socket ID in an expression session so I can access it in my route but i've not been able to get it working. In the code below i'm logging my session when the socket connects and this shows the socketio key i've added but in the /jobs request the socketio key is undefined..
My server.ts is something like this:
import * as expsession from 'express-session'
// create Express app variable
const app = express()
// create HTTP server
const server = http.createServer(app);
// set up CORS so it can work Angular application at another domain/port
app.use(cors({
origin: [ "http://localhost:4200" ],
credentials: true
}))
// install session middleware
const session = expsession({
secret: 'random secret',
saveUninitialized: true,
resave: true,
cookie: { secure: false }
});
// run session middleware for regular http connections
app.use(session);
// *** SOCKET IO *** //
const io = require('socket.io')(server);
// run session middleware for socket.io connections
io.use((socket, next) => {
session(socket.request, socket.request.res || {}, next);
});
// when a socket.io connection connects, put the socket.id into the session
// so it can be accessed from other http requests from that client
io.on('connection', (socket) => {
console.log(`socket.io connected: ${socket.id}`);
// save socket.io socket in the session
socket.request.session.socketio = socket.id;
socket.request.session.save();
console.log("socket.io connection, session:\n", socket.request.session);
});
app.get('/jobs', (req, res) => {
const session = req.session;
console.log("JOBS: session at '/jobs':\n", session);
Job.getJobs((err, jobs) => {
if (err) {
res.json({success: false, msg: err});
} else {
res.json(jobs);
}
});
});
I'm also including credentials in my angular service request, e.g.:
this.http.get(api_url + '/jobs', {withCredentials:true}).subscribe((jobs:IJob[]) => {
this.jobs$.next(jobs)
})
When a socket is connected an id is created(socket.id) and by default it joins to a room with that id. If you already have id for your client just send it to the server and join to a room with the client id.
socket.leave(socket.id);//leaving default room
socket.join(my_custom_id);//joining to custom id room
the custom id will appear in client side as the websocket7 id.
If you still don't know how to get the id in your express route; use a jsonwebtoken with the id and decoded in your route, thats it.
Example:
var bearer = req.headers.authorization.split(" ")[1];//this is node, get headers with express way
var auth_data = jwt.decodeToken( bearer );
decoding:
var jwt = require('jsonwebtoken');
var secret = 'your_secret';
function decodeToken(token){
try {
var decoded = jwt.verify( token , secret );
return decoded;
} catch (error) {
console.log('jwt', error);
return null;
}
}
encoding:
//data is an object with data you want to encode, id, etc
function createJsonWebToken(data){
var token = jwt.sign( data, secret );
return token
}
/*after encoding you send this token to the
client, then you send it back to the server
to the routes you need the data, then just
decoded it in that route*/
I need to add my track and send to other peers after having etablished the connection I've followed the MDN example
pc.onnegotiationneeded = async () => {
try {
makingOffer = true;
await pc.setLocalDescription();
signaler.send({ description: pc.localDescription });
} catch(err) {
console.error(err);
} finally {
makingOffer = false;
}
};
onnegotiationneeded is fired when we call pc.addTrack so I will remake the process offer -> answer -> ICE.
So I have a function which call getUserMedia and set the local video I've added a callback to apply addTrack to my peers
const handleActiveVideo = async (cb) => {
const devices = await getVideoDevices();
const localStream = await createLocalStream(devices[0].deviceId);
setLocalVideoEl(localStream, devices[0].deviceId);
cb((pc) => {
localStream.getTracks().forEach((track) => {
pc.connection.addTrack(track, localStream);
});
});
};
But if I do that with an etablished connection when I add my local stream to the track with one peer it's ok all work fine but with my second peer I have this error on Firefox
Remote description indicates ICE restart but offer did not request ICE
restart (new remote description changes either the ice-ufrag or
ice-pwd)
with Chrome
DOMException: Failed to execute 'setRemoteDescription' on
'RTCPeerConnection': Failed to set remote answer sdp: Failed to apply
the description for m= section with mid='0': Failed to set SSL role
for the transport.
To recapitulate
connect my 2 peers without any tracks on both sides
start the video with my Peer1 ok I can see the video with my Peer2
start the video with my Peer2 error on Peer2 in the answer
setRemoteDescription(description);
I have a question about MassTransit configuration.
There is a Main application and Microservice.
For example, the Main application sends a commands to the microservice(consumer) to write off funds from the account.
Configuration in the Main application:
var rabbitHost = new Uri("rabbitmq://localhost/app");
services.AddMassTransit(x => {
x.AddBus(provider => Bus.Factory.CreateUsingRabbitMq(cfg => {
var host = cfg.Host(rabbitHost, hostConfigurator => {
hostConfigurator.Username("user");
hostConfigurator.Password("password");
});
}));
});
EndpointConvention.Map<WithdrawFunds>(new Uri(rabbitHost + "/test-queue"));
Microservice configuration:
var rabbitHost = new Uri("rabbitmq://localhost/app");
services.AddMassTransit(x => {
x.AddBus(provider => Bus.Factory.CreateUsingRabbitMq(cfg => {
var host = cfg.Host(rabbitHost, hostConfigurator => {
hostConfigurator.Username("username");
hostConfigurator.Password("password");
});
cfg.ReceiveEndpoint(host, "test-queue", ep => {
ep.Consumer<WithdrawFundsConsumer>();
});
}));
});
Command executed in Main application like:
await _sendEndpointProvider.Send<WithdrawFunds>(new {
Amount = 100,
AccountId = "someId"
});
MassTransit creates a "test-queue" queue and if both applications are running, then the interaction works successfully. But if I stop the microservice, then a
"test-queue_skipped" queue is created in which the missed messages fall. However, if I start the Microservice, it will not receive missed messages.
How can I configure MassTransit so that "_skipped" is not created, and messages are waiting for the consumer to appear?
I am building an application with a node/express backend. The endpoint has a really long response time due to heavy computation (take 1 - 2 mins). Heroku times me out at 30 seconds.
I'm using Kue and redis in order to fulfill the computation in redis.
A little bit of pseudo
1. When a user hits the endpoint
2. Initiate the job
3. Send response after invoking job to not time out of heroku
4. Once Job is complete, respond/send completed job/data => which will go into redux.
Here is my code so far:
const redis = require('redis');
const client = redis
.createClient();
const kue = require('kue');
const queue = kue.createQueue();
router.post('/endpoint', (req, res) => {
const param = req.body.param;
var job = queue
.create('job', param)
.priority('high')
.save(err => {
if (err) {
console.log('failed');
process.exit(0);
return;
}
job.on('complete', result => {
res.send(result);
console.log('completed');
});
job.on('failed', errorMessage => {
console.log(errorMessage);
process.exit(0);
});
});
queue.process('job', async (job, done) => {
// heavy heavy computation
// heavy heavy heavy heavy
// heavy heavy heavy
//output is results => passed into promise
done(null, await Promise.all(results));
});
res.json({ message: 'job in progress' });
});
The problem with this code is that I can't send a header response once it has been set. I am very new to redis and very new kue and background jobs. I've read webhooks can help, but I don't have much experience with webhooks either.
Thank you in advance!
We are having a trouble with our refresh authentication token flow, where if multiple requests are made with the same expired access token at the same time, all of them would make backend to refresh the token and would have different refreshed tokens in their responses. Now when the next request is made, the refresh token of the last response would be used which may or may not be latest.
One solution to this problem is to have UI (which is using axios in vue) send only one request at a time. So if a token is refreshed, the following request would have the latest token.
Hence I am wondering if axios has a default option to disable simultaneous requests as I couldn't find anything online about it. Another solution would be to maintain a queue of requests, where each request is only sent when a response (either success or fail) is received for the previous request(manual option). I understand that this might decrease the performance as it may seem in the UI, but it would also take the load off backend.
One possible solution I found here is to use interceptor:
let isRefreshing = false;
let refreshSubscribers = [];
const instance = axios.create({
baseURL: Config.API_URL,
});
instance.interceptors.response.use(response => {
return response;
}, error => {
const { config, response: { status } } = error;
const originalRequest = config;
if (status === 498) { //or 401
if (!isRefreshing) {
isRefreshing = true;
refreshAccessToken()
.then(newToken => {
isRefreshing = false;
onRrefreshed(newToken);
});
}
const retryOrigReq = new Promise((resolve, reject) => {
subscribeTokenRefresh(token => {
// replace the expired token and retry
originalRequest.headers['Authorization'] = 'Bearer ' + token;
resolve(axios(originalRequest));
});
});
return retryOrigReq;
} else {
return Promise.reject(error);
}
});
subscribeTokenRefresh(cb) {
refreshSubscribers.push(cb);
}
onRrefreshed(token) {
refreshSubscribers.map(cb => cb(token));
}