I've got a web server written in Go.
tlsConfig := &tls.Config{
PreferServerCipherSuites: true,
MinVersion: tls.VersionTLS12,
CurvePreferences: []tls.CurveID{
tls.CurveP256,
tls.X25519,
},
CipherSuites: []uint16{
tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,
tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,
tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
},
}
s := &http.Server{
ReadTimeout: 5 * time.Second,
WriteTimeout: 10 * time.Second,
IdleTimeout: 120 * time.Second,
Handler: r, // where r is my router
TLSConfig: tlsConfig,
}
// redirect http to https
redirect := &http.Server{
ReadTimeout: 5 * time.Second,
WriteTimeout: 10 * time.Second,
IdleTimeout: 120 * time.Second,
Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Connection", "close")
url := "https://" + r.Host + r.URL.String()
http.Redirect(w, r, url, http.StatusMovedPermanently)
}),
}
go func() {
log.Fatal(redirect.ListenAndServe())
}()
log.Fatal(s.ListenAndServeTLS(certFile, keyFile))
Here is a screenshot from my Digital Ocean dashboard.
As you can see memory keeps growing and growing. So I started looking at https://github.com/google/pprof. Here is the output of top5.
Type: inuse_space
Time: Nov 7, 2018 at 10:31am (CET)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top5
Showing nodes accounting for 289.50MB, 79.70% of 363.24MB total
Dropped 90 nodes (cum <= 1.82MB)
Showing top 5 nodes out of 88
flat flat% sum% cum cum%
238.98MB 65.79% 65.79% 238.98MB 65.79% crypto/tls.(*block).reserve
20.02MB 5.51% 71.30% 20.02MB 5.51% crypto/tls.Server
11.50MB 3.17% 74.47% 11.50MB 3.17% crypto/aes.newCipher
10.50MB 2.89% 77.36% 10.50MB 2.89% crypto/aes.(*aesCipherGCM).NewGCM
The SVG shows the same huge amount of memory allocated by crypto/tls.(*block).reserve.
Here is the exact code.
I spent the last days reading every article, document, blog post, source code, help file I could find. However nothing helps. The code is running on a Ubuntu 17.10 x64 machine using Go 1.11 inside a Docker container.
It looks like the server doesn't close the connections to the client. I thought setting all the xyzTimeout would help but it didn't.
Any ideas?
Edit 12/20/2018:
fixed now https://github.com/golang/go/issues/28654#issuecomment-448477056
Adding an answer so this doesn't keep showing up in the list of upvoted and unanswered questions.
It appears that the memory leak was related to the gorilla context bug https://github.com/gorilla/sessions/commit/12bd4761fc66ac946e16fcc2a32b1e0b066f6177 and had nothing to do with tls in the stdlib.
Related
I hope you are well! First, I am new to the EIP world. I am trying to do a simple request reply with:
A Golang rabbitMQ client
An apache Camel route in Kotlin acting as a RabbitMQ server
I have tried to read all the docs I could and search for answers but I could't find nothing. I am basically desperate. Mainly I saw this and nothing has worked yet.
My goal is to do a sync request-reply as the image.
My Golang client looks like this:
func (r *RabbitMQConn) GetQueue(name string) *amqp.Queue {
ch := r.GetChannel()
defer ch.Close()
q, err := ch.QueueDeclare(
name,
false,
false,
true,
false,
nil,
)
if err != nil {
panic(err)
}
return &q
}
func (r *RabbitMQConn) PublishAndWait(routingKey string, correlationId string, event domain.SyncEventExtSend) (domain.SyncEventExtReceive, error) {
message, err := json.Marshal(event)
if err != nil {
return domain.SyncEventExtReceive{}, apperrors.ErrInternal
}
ch := r.GetChannel()
defer ch.Close()
q := r.GetQueue("response")
h, err := ch.Consume(
q.Name,
"",
true,
false,
false,
false,
nil,
)
if err != nil {
return domain.SyncEventExtReceive{}, err
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
err = ch.PublishWithContext(
ctx,
"",
routingKey,
false,
false,
amqp.Publishing{
ContentType: "application/json",
Body: message,
CorrelationId: correlationId,
ReplyTo: q.Name,
},
)
if err != nil {
return domain.SyncEventExtReceive{}, err
}
for d := range h {
fmt.Println("Received a message:", string(d.Body))
if d.CorrelationId == correlationId {
var event domain.SyncEventExtReceive
err = json.Unmarshal(d.Body, &event)
return event, err
}
}
return domain.SyncEventExtReceive{}, apperrors.ErrInternal
}
Basically, just consuming from the default exchange with a named response queue. Also, I send the queue name as the ReplyTo parameter and I give it a correlation id. The routing-key that is sent is daily-weather in this case.
On the server side, I tried to do the server with the default exchange, but Apache Camel forbids me to do nothing with that exchange.
from("rabbitmq:?queue=daily-weather&autoAck=true&autoDelete=false")
So, I assigned it the amq.direct exchange. However, that didn't also worked.
"rabbitmq:amq.direct?queue=daily-weather&autoAck=true&autoDelete=false"
Then, I added a second RabbitMQ endpoint to see if it would sent it, but nothing.
from("rabbitmq:amq.direct?queue=daily-weather&autoAck=true&autoDelete=false")
.log(LoggingLevel.INFO, "weather-daily", "Received message: \${body}")
.to("rabbitmq:amq.direct?queue=response&autoAck=true&autoDelete=false")
I ask if anybody has any simple example to do this with Apache Camel, because I am ultra lost. Any further detail can be shared if you contact me.
Thank you very much!!!! :)
SOLVED
Hi! After some time I decided to take a look to the spring-rabbitmq Camel component. I realised that Camel has exchange patterns, and rabbitmq, by default, sets it to inOut. This way, automatically returns the information back to the replyTo property.
val RABBIMQ_ROUTE =
"spring-rabbitmq:default?queues={{rabbitmq.weather.daily.routing_key}}"
default refers to the default exchange queue.
I was playing with this was able to get it working immediately using Node/Javascript, took a while using Golang (this is just generating the user/password to be sent to coturn.) Notice the secret should match the coturn configuration and in the API JS/Go side.
The configuration on coturn: /etc/turnserver.conf
listening-port=443
tls-listening-port=443
listening-ip=10.100.0.2
relay-ip=10.100.0.2
external-ip=123.456.78.9
min-port=10000
max-port=20000
verbose
fingerprint
lt-cred-mech
server-name=myserver
realm=myserver
cert=/etc/SSL/fullchain.pem
pkey=/etc/SSL/privkey.pem
log-file=/var/log/turnserver.log
use-auth-secret
static-auth-secret=MySecret
The following is Node/Js Implementation API (copied from elsewhere - worked):
var crypto = require('crypto');
var unixTimeStamp = parseInt(Date.now()/1000) + 24*3600, // this credential valid for 24 hours
TempUser = [unixTimeStamp, "SomeUser"].join(':'),
TempPassword,
hmac = crypto.createHmac('sha1', "MySecret");
hmac.setEncoding('base64');
hmac.write(TempUser);
hmac.end();
TempPassword = hmac.read();
The following is GOLANG Implementation API (took a while):
UserId := "SomeUser"
// This worked, returned the exact seconds
timestamp := strconv.FormatInt(time.Now().UTC().Unix()+24*3600, 10)
// Example: The above is 1602692130
secret := "MySecret"
TempUser := timestamp + ":" + UserId // For API Auth, coturn expects this format, the timestamp is the expiry date of the final temp user/password.
// Create a new HMAC by defining the hash type and the key (as byte array)
//h := hmac.New(sha256.New, []byte(secret)) // sha256 does not work, use sha1
h := hmac.New(sha1.New, []byte(secret))
h.Write([]byte(TempUser))
//sha := b64.URLEncoding.EncodeToString(h.Sum(nil)) // URLEncoding did not work
TempPassword := b64.StdEncoding.EncodeToString(h.Sum(nil)) // StdEncoding worked
The JS on the Webrtc client. Notice we are using the TempUser and TempPassword here to be sent to coturn.
...
const stunUrl = 'stun:mystun_server',
turnUsername = TempUser,
turnPassword = TempPassword,
...
'iceServers': [
{ 'url': stunUrl },
{
'urls': turnUrl1,
'username': turnUsername,
'credential': turnPassword
},
Now coturn will authenticate using the TempUser and TempPassword above. Hope someone will find this useful. Thanks!
The client program below receives messages from a WebSocket server.
It doesn't send any messages.
CLIENT
use v6;
use Cro::WebSocket::Client;
constant WS-URL = 'ws://localhost:20000/status';
constant TIMEOUT-TO-CONNECT = 5; # seconds
my $timeout;
my $connection-attempt;
await Promise.anyof(
$connection-attempt = Cro::WebSocket::Client.connect(WS-URL),
$timeout = Promise.in(TIMEOUT-TO-CONNECT));
if $timeout.status == Kept
{
say "* could not connect to server in ', TIMEOUT-TO-CONNECT, ' seconds";
exit 1;
}
if $connection-attempt.status != Kept
{
my $cause = $connection-attempt.cause;
say '"* error when trying to connect to server';
say '"* --------------------------------------';
# $cause is a long string, how do we get a simple numeric code ?
say $cause;
say '"* ======================================';
exit 1;
}
my $connection = $connection-attempt.result;
my $peer = 'localhost:20000';
say '* connected with ', $peer;
react
{
whenever $connection.messages -> $message
{
my $body = await $message.body;
say '* received message=[' ~ $body ~ '] from server';
LAST { say '* LAST'; done; }
QUIT { default { say '* QUIT'; done; }}
}
CLOSE { say '* CLOSE: leaving react block';}
} # react
SERVER
use Cro::HTTP::Router;
use Cro::HTTP::Server;
use Cro::HTTP::Router::WebSocket;
my $application =
route
{
get -> 'status'
{
web-socket -> $incoming
{
my $counter = 0;
my $timer = Supply.interval(1);
supply
{
whenever $incoming -> $thing
{
LAST { note '* LAST: client connection was closed'; done; }
QUIT { default { note '* QUIT: error in client connection'; done; } }
}
whenever $timer
{
$counter++;
say '* sending message ', $counter;
emit $counter.Str;
}
CLOSE { say '* CLOSE: leaving supply block'; }
} # supply
} #incoming
} # get -> status
}
my $server = Cro::HTTP::Server.new: :port(20000), :$application;
$server.start;
say '* serving on port 20000';
react whenever signal(SIGINT)
{
$server.stop;
exit;
}
Now, when the server goes out (say, by Ctrl+C) the client sees nothing.
Setting CRO_TRACE=1 in the client gives this:
TRACE(anon 2)] Cro::WebSocket::MessageParser EMIT WebSocket Message - Text
* received message=[4] from server
[TRACE(anon 1)] Cro::TCP::Connector DONE
[TRACE(anon 2)] Cro::WebSocket::FrameParser DONE
[TRACE(anon 2)] Cro::WebSocket::MessageParser DONE
[TRACE(anon 1)] Cro::HTTP::ResponseParser DONE
^C
The client showed nothing more (and then I cancelled it).
So, the question is: how should the client deal with this scenario ?
UPDATE
Edited the question, now showing the server code
Also, I'm in Fedora 28.
When I first cancel the server, netstat shows
$ netstat -ant | grep 20000
tcp6 0 0 ::1:20000 ::1:56652 TIME_WAIT
$
Tcpdump shows
IP6 ::1.20000 > ::1.56652: Flags [F.], seq 145, ack 194, win 350, options [nop,nop,TS val 1476681452 ecr 1476680552], length 0
IP6 ::1.56652 > ::1.20000: Flags [F.], seq 194, ack 146, win 350, options [nop,nop,TS val 1476681453 ecr 1476681452], length 0
IP6 ::1.20000 > ::1.56652: Flags [.], ack 195, win 350, options [nop,nop,TS val 1476681453 ecr 1476681453], length 0
It seems the last ACK from the client to the server is missing, I guess the client didn't close the connection.
Also, I'm curious as to why Cro chooses to work with IPv6 by default.
This is a bug that has been fixed since this question was posted, but I'm leaving an answer because of this part of the question that may confuse people when dealing with networking in Raku:
Also, I'm curious as to why Cro chooses to work with IPv6 by default.
localhost will resolve to an IPv6 address first if that's what the first address for localhost in your hosts file is. As of writing, IO::Socket::Async (which Cro uses internally) only allows PF_UNSPEC to be specified as a family, and the only address that will ever used from the results of hostname resolution is the first one in the list of addresses received. This will be changed at some point in the future as part of the work for my IP6NS grant and a problem solving issue to improve how DNS is handled, but for now, if you want to use IPv4/IPv6 only, you should specify 127.0.0.1/::1 instead of using localhost (or whichever addresses your machine resolves it to if they're different).
1) I setup a private ethereum network using the following command
$geth --genesis <genesis json file path> --datadir <some path to an empty
folder> --networkid 123 --nodiscover --maxpeers 0 console
2) Created an account
3) Then, started the miner using miner.start() command.
After a while ethers were getting added automatically to my account, but I don’t have any pending transaction in my private network. So from where could my miners are getting the ethers?
Even though, I didn’t instantiate any transactions in my network, I could see the some transaction being recorded in the logs, once I start the miner.
The log is as follows:
I0118 11:59:11.696523 9427 backend.go:584] Automatic pregeneration of ethash
DAG ON (ethash dir: /Users/minisha/.ethash)
I0118 11:59:11.696590 9427 backend.go:591] checking DAG (ethash dir:
/Users/minisha/.ethash)
I0118 11:59:11.696728 9427 miner.go:119] Starting mining operation (CPU=4
TOT=5)
true
> I0118 11:59:11.703907 9427 worker.go:570] commit new work on block 1 with 0
txs & 0 uncles. Took 7.109111ms
I0118 11:59:11.704083 9427 ethash.go:220] Generating DAG for epoch 0 (size
1073739904) (0000000000000000000000000000000000000000000000000000000000000000)
I0118 11:59:12.698679 9427 ethash.go:237] Done generating DAG for epoch 0, it
took 994.61107ms
I0118 11:59:15.163864 9427 worker.go:349]
And my genesis block code is as follows:
{
“nonce”: “0xdeadbeefdeadbeef”,
“timestamp”: “0x0”,
“parentHash”:
“0x0000000000000000000000000000000000000000000000000000000000000000”,
“extraData”: “0x0”,
“gasLimit”: “0x8000000”,
“difficulty”: “0x400”,
“mixhash”:
“0x0000000000000000000000000000000000000000000000000000000000000000”,
“coinbase”: “0x3333333333333333333333333333333333333333”,
“alloc”: {
}
}
Since my network is isolated and have only one node (no peers), I am quite confused with this behaviour. Any insights would be greatly appreciated.
Your client is mining empty blocks (containing no transactions) and getting rewards for mined blocks what is 5 ETH per block.
If you want to prevent empty block in your private blockchain you should consider using eth client (C++ implementation).
In case of geth client you can use a JavaScript script that will modify client's behavior. Any script can be loaded with js command: geth js script.js.
var mining_threads = 1
function checkWork() {
if (eth.getBlock("pending").transactions.length > 0) {
if (eth.mining) return;
console.log("== Pending transactions! Mining...");
miner.start(mining_threads);
} else {
miner.stop(0); // This param means nothing
console.log("== No transactions! Mining stopped.");
}
}
eth.filter("latest", function(err, block) { checkWork(); });
eth.filter("pending", function(err, block) { checkWork(); });
checkWork();
You can try EmbarkJS which can run geth client with mineWhenNeeded option on private network. It will only mine when new transactions come in.
I have working code where two peers are connecting over a relay server (coturn) and everything seems to be fine over pseudo-tcp. I've tested message exchange successfully with nice_agent_attach_recv() and nice_agent_get_io_stream().
But when I try to create a GTlsClientConnection I get back: 0:TLS support is not available
Here is some partial code:
if(!nice_agent_set_relay_info(agent, stream_id,
NICE_COMPONENT_TYPE_RTP,
"my.coturn.server",
5349, //tls-listener-port (I also tried the non tls port: 3478)
username.c_str(),
password.c_str(),
NICE_RELAY_TYPE_TURN_TCP))
{
printf("error setting up relay info\n");
}
...
//after state has changed to NICE_COMPONENT_STATE_READY
...
io_stream = nice_agent_get_io_stream (agent, stream_id, component_id);
input = g_io_stream_get_input_stream (G_IO_STREAM (io_stream));
output = g_io_stream_get_output_stream (G_IO_STREAM (io_stream));
GIOStream* tlsConnection = g_tls_client_connection_new
(G_IO_STREAM (io_stream), NULL, &error);
/////////////////////////
/// error == 0 (TLS support is not available)
I am new to libnice and glib. So, I may be missing something basic.
Probably need the glib-networking package installed.