I'm building a simple platform to allow Twitter users to post to their Twitter accounts on a time schedule of their choosing. So for example someone might want to make a post to his/her Twitter account at exactly 12:00 am midnight, but the platform allows them to create the post by 4:00 pm and automatically post to their timeline when it's the scheduled time. I'm confused as to how to implement this since the Api endpoint responsible for this action has to be called manually for this event to take place, I thought of using cron jobs to run every minute and check for all the scheduled posts, call the endpoint if it's time to create the post, but I feel this is cumbersome and error-prone. Are there any more programmatic ways of implementing this, please?
Ps: I'm trying to do this in golang...
You can use a Pub/Consumer model with a message queue. It will make your solution more robust and distributed. I always try not to use sleep in any kind of distributed environment.
Your system will have 3 components:
Producer: This will take care of scheduling and writing messages to a message queue.
Message Queue: You can use RabbitMQ(with delayMessageExchange) or ActiveMQ(it has in-built scheduling). Your message queue will take care of the scheduling problem. Because the message will be delivered only after a specific time delay to the consumer.
Consumer: The consumer will read data from the message queue and perform operations as the functionality
HLD will look like this:
You can use https://pkg.go.dev/github.com/streadway/amqp this package for RabbitMQ
and https://pkg.go.dev/github.com/go-stomp/stomp this package for ActiveMQ
Here is an simple example, I have used AWS ActiveMQ.
package main
import (
"crypto/tls"
"flag"
"fmt"
"github.com/go-stomp/stomp/v3"
"os"
"time"
)
const defaultPort = ":61613"
var serverAddr = flag.String("server", "b-50ad5529-0347-4308-af59-f6265d68d290-1.mq.us-east-1.amazonaws.com:61614", "STOMP server endpoint")
var messageCount = flag.Int("count", 2, "Number of messages to send/receive")
var queueName = flag.String("queue", "/queue/client_test", "Destination queue")
var helpFlag = flag.Bool("help", false, "Print help text")
var stop = make(chan bool)
// these are the default options that work with RabbitMQ
var options []func(*stomp.Conn) error = []func(*stomp.Conn) error{
stomp.ConnOpt.Login("activemquser", "activemqpassword"),
}
func main() {
flag.Parse()
if *helpFlag {
fmt.Fprintf(os.Stderr, "Usage of %s\n", os.Args[0])
flag.PrintDefaults()
os.Exit(1)
}
subscribed := make(chan bool)
go recvMessages(subscribed)
// wait until we know the receiver has subscribed
<-subscribed
go sendMessages()
<-stop
<-stop
}
func sendMessages() {
defer func() {
stop <- true
}()
netConn, err := tls.Dial("tcp", *serverAddr, &tls.Config{})
if err != nil {
println("cannot connect to server", err.Error())
}
conn, err := stomp.Connect(netConn, options...)
if err != nil {
println("cannot connect to server", err.Error())
}
for i := 1; i <= *messageCount; i++ {
text := fmt.Sprintf("Message #%d", i)
fmt.Println("sending message ", text, " ", time.Now())
// scheduling a message with 15 seconds delay
err = conn.Send(*queueName, "text/plain",
[]byte(text), stomp.SendOpt.Header("AMQ_SCHEDULED_DELAY", "15000"))
if err != nil {
println("failed to send to server", err)
return
}
// schedule each message after 3 secs
time.Sleep(3 * time.Second)
}
println("sender finished")
}
func recvMessages(subscribed chan bool) {
defer func() {
stop <- true
}()
netConn, err := tls.Dial("tcp", *serverAddr, &tls.Config{})
if err != nil {
println("cannot connect to server", err.Error())
}
conn, err := stomp.Connect(netConn, options...)
if err != nil {
println("cannot connect to server", err.Error())
}
sub, err := conn.Subscribe(*queueName, stomp.AckAuto)
if err != nil {
println("cannot subscribe to", *queueName, err.Error())
return
}
close(subscribed)
for i := 1; i <= *messageCount; i++ {
msg := <-sub.C
expectedText := fmt.Sprintf("Message #%d", i)
actualText := string(msg.Body)
fmt.Println("got message", actualText, " ", time.Now())
if expectedText != actualText {
println("Expected:", expectedText)
println("Actual:", actualText)
}
}
println("receiver finished")
}
Related
Implemented a context timeout in Go APIs using
context.WithTimeout(context,time)
Expectation is that if I set it to 10 seconds and if the API happens to run longer, it should stop and return Timeout error. However, if I add a sleep in between , the API still runs and gives 200 output. Is there any solution to this other than manually checking at random points for the time elapsed ? Looks like context.WithTimeout() doesnt solve its purpose.
So, this is a minimum code to exemplify how the context.WithTimeout can work:
server.go
package main
import (
"fmt"
"net/http"
"time"
)
func main() {
http.HandleFunc("/bad", badResponse)
http.HandleFunc("/nice", niceResponse)
http.ListenAndServe(":8099", nil)
}
// badResponse is a procedure that takes too much time
func badResponse(w http.ResponseWriter, r *http.Request) {
fmt.Println("got request, it may take too long to answer")
time.Sleep(3 * time.Second)
fmt.Fprintf(w, "bad response!")
}
// niceResponse is a procedure that respond in time
func niceResponse(w http.ResponseWriter, r *http.Request) {
fmt.Println("got request, will return fast")
fmt.Fprintf(w, "nice response!")
}
client.go
package main
import (
"context"
"io/ioutil"
"log"
"net/http"
"time"
)
func main () {
req, err := http.NewRequest(http.MethodGet, "http://localhost:8099/bad", nil)
if err != nil {
log.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Millisecond)
defer cancel()
req = req.WithContext(ctx)
c := &http.Client{}
res, err := c.Do(req)
if err != nil {
log.Fatal(err)
}
defer res.Body.Close()
out, err := ioutil.ReadAll(res.Body)
if err != nil {
log.Fatal(err)
}
log.Println(string(out))
}
In the example above the client must fail because the timeout is lower than the client server delay.
In my Go application I use crontab package to run Tracker function every minute. As you can notice from the code I call PostgreSQL function. To interact with the PostgreSQL database, I use the gorm package. Application worked several days without any problem but now I notice an error in logs: pq: sorry, too many clients already. I know that same questions was asked several times in StackOverflow before. For example in this post people advice to use Exec or Scan methods. In my case as you can see I use Exec method but anyway I have error. As far as I understand, each database request makes a separate connection and does not close it. I can't figure out what I'm doing wrong.
main.go:
package main
import (
"github.com/mileusna/crontab"
)
func main() {
database.ConnectPostgreSQL()
defer database.DisconnectPostgreSQL()
err = crontab.New().AddJob("* * * * *", controllers.Tracker); if err != nil {
utils.Logger().Fatal(err)
return
}
}
tracker.go:
package controllers
import (
"questionnaire/database"
"time"
)
var Tracker = func() {
err := database.DBGORM.Exec("CALL tracker($1)", time.Now().Format("2006-01-02 15:04:05")).Error; if err != nil {
utils.Logger().Println(err) // ERROR: pq: sorry, too many clients already
return
}
}
PostgreSQL.go:
package database
import (
"fmt"
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/postgres"
"github.com/joho/godotenv"
"questionnaire/utils"
)
var DBGORM *gorm.DB
func ConnectPostgreSQL() {
err := godotenv.Load(".env")
if err != nil {
utils.Logger().Println(err)
panic(err)
}
databaseUser := utils.CheckEnvironmentVariable("PostgreSQL_USER")
databasePassword := utils.CheckEnvironmentVariable("PostgreSQL_PASSWORD")
databaseHost := utils.CheckEnvironmentVariable("PostgreSQL_HOST")
databaseName := utils.CheckEnvironmentVariable("PostgreSQL_DATABASE_NAME")
databaseURL:= fmt.Sprintf("host=%s user=%s dbname=%s password=%s sslmode=disable", databaseHost, databaseUser, databaseName, databasePassword)
DBGORM, err = gorm.Open("postgres", databaseURL)
if err != nil {
utils.Logger().Println(err)
panic(err)
}
err = DBGORM.DB().Ping()
if err != nil {
utils.Logger().Println(err)
panic(err)
}
DBGORM.LogMode(true)
}
func DisconnectPostgreSQL() error {
return DBGORM.Close()
}
The following Golang test never exits. I suspect it has something to do with a channel deadlock but being a go-noob, I am not very certain.
const userName = "xxxxxxxxxxxx"
func TestSynchroninze(t *testing.T) {
c, err := channel.New(github.ChannelName, authToken)
if err != nil {
t.Fatalf("Could not create channel: %s", err)
return
}
state := channel.NewState(nil)
ctx := context.Background()
ctx = context.WithValue(ctx, "userId", userName)
user := api.User{}
output, errs := c.Synchronize(state, ctx)
if err = <-errs; err != nil {
t.Fatalf("Error performing synchronize: %s", err)
return
}
for o := range output {
switch oo := o.Data.(type) {
case api.User:
user = oo
glog.Infof("we have a USER %s\n", user)
default:
t.Errorf("Encountered unexpected data type: %T", oo)
}
}
}
Here are the methods being tested.
type github struct {
client *api.Client
}
func newImplementation(t auth.UserToken) implementation.Implementation {
return &github{client: api.NewClient(t)}
}
// -------------------------------------------------------------------------------------
const (
kLastUserFetch = "lastUserFetch"
)
type synchronizeFunc func(implementation.MutableState, chan *implementation.Output, context.Context) error
// -------------------------------------------------------------------------------------
func (g *github) Synchronize(state implementation.MutableState, ctx context.Context) (<-chan *implementation.Output, <-chan error) {
output := make(chan *implementation.Output)
errors := make(chan error, 1) // buffer allows preflight errors
// Close output channels once we're done
defer func() {
go func() {
// wg.Wait()
close(errors)
close(output)
}()
}()
err := g.fetchUser(state, output, ctx)
if err != nil {
errors <- err
}
return output, errors
}
func (g *github) fetchUser(state implementation.MutableState, output chan *implementation.Output, ctx context.Context) error {
var err error
var user = api.User{}
userId, _ := ctx.Value("userId").(string)
user, err = g.client.GetUser(userId, ctx.Done())
if err == nil {
glog.Info("No error in fetchUser")
output <- &implementation.Output{Data: user}
state.SetTime(kLastUserFetch, time.Now())
}
return err
}
func (c *Client) GetUser(id string, quit <-chan struct{}) (user User, err error) {
// Execute request
var data []byte
data, err = c.get("users/"+id, nil, quit)
glog.Infof("USER DATA %s", data)
// Parse response
if err == nil && len(data) > 0 {
err = json.Unmarshal(data, &user)
data, _ = json.Marshal(user)
}
return
}
Here is what I see in the console (most of the user details removed)
I1228 13:25:05.291010 21313 client.go:177] GET https://api.github.com/users/xxxxxxxx
I1228 13:25:06.010085 21313 client.go:36] USER DATA {"login":"xxxxxxxx","id":00000000,"avatar_url":"https://avatars.githubusercontent.com/u/0000000?v=3",...}
I1228 13:25:06.010357 21313 github.go:90] No error in fetchUser
==========EDIT=============
Here is the relevant portion of the api package.
package api
type Client struct {
authToken auth.UserToken
http *http.Client
}
func NewClient(authToken auth.UserToken) *Client {
return &Client{
authToken: authToken,
http: auth.NewClient(authToken),
}
}
// -------------------------------------------------------------------------------------
type User struct {
Id int `json:"id,omitempty"`
Username string `json:"login,omitempty"`
Email string `json:"email,omitempty"`
FullName string `json:"name,omitempty"`
ProfilePicture string `json:"avatar_url,omitempty"`
Bio string `json:"bio,omitempty"`
Website string `json:"blog,omitempty"`
Company string `json:"company,omitempty"`
}
And the channel package
package channel
type Channel struct {
implementation.Descriptor
imp implementation.Implementation
}
// New returns a channel implementation with a given name and auth token.
func New(name string, token auth.UserToken) (*Channel, error) {
if desc, ok := implementation.Lookup(name); ok {
if imp := implementation.New(name, token); imp != nil {
return &Channel{Descriptor: desc, imp: imp}, nil
}
}
return nil, ErrInvalidChannel
}
and the implementation package...
package implementation
import "golang.org/x/net/context"
// -------------------------------------------------------------------------------------
// Implementation is the interface implemented by subpackages.
type Implementation interface {
// Synchronize performs a synchronization using the given state. A context parameters
// is provided to provide cancellation as well as implementation-specific behaviors.
//
// If a fatal error occurs (see package error definitions), the state can be discarded
// to prevent the persistence of an invalid state.
Synchronize(state MutableState, ctx context.Context) (<-chan *Output, <-chan error)
// FetchDetails gets details for a given timeline item. Any changes to the TimelineItem
// (including the Meta value) will be persisted.
FetchDetails(item *TimelineItem, ctx context.Context) (interface{}, error)
}
======Edit #2=======
This is the original Synchronize method. I removed some details in my testing to try and simplify the problem. By removing a go func call, I believe I introduced a new problem which could be confusing things.
Here is the original Synchronize method. There are some things with Wait Groups and a function array containing a single function because this method will eventually be synchronizing multiple functions.
func (g *github) Synchronize(state implementation.MutableState, ctx context.Context) (<-chan *implementation.Output, <-chan error) {
wg := sync.WaitGroup{}
output := make(chan *implementation.Output)
errors := make(chan error, 1) // buffer allows preflight errors
// Close output channels once we're done
defer func() {
go func() {
wg.Wait()
close(errors)
close(output)
}()
}()
// Perform fetch functions in separate routines
funcs := []synchronizeFunc{
g.fetchUser,
}
for _, f := range funcs {
wg.Add(1)
go func(f synchronizeFunc) {
defer wg.Done()
if err := f(state, output, ctx); err != nil {
errors <- err
}
}(f)
}
glog.Info("after go sync...")
return output, errors
}
I think the two problems are in
output <- &implementation.Output{Data: user}
the channel does not have a buffer. It will block until some other goroutine reads from it. But in your code is the same goroutine so it will block.
and second:
// Close output channels once we're done
defer func() {
go func() {
// wg.Wait()
close(errors)
close(output)
}()
}()
you launch a go routine when the routine exits. The goroutine is scheduled, the function returns but it never calls the goroutine.
I would suggest to unify all that logic in one:
func (g *github) Synchronize(state implementation.MutableState, ctx context.Context) (<-chan *implementation.Output, <-chan error) {
output := make(chan *implementation.Output)
errors := make(chan error, 1) // buffer allows preflight errors
go func() {
defer close(output)
defer close(errors)
err := g.fetchUser(state, output, ctx)
if err != nil {
errors <- err
}
}()
return output, errors
}
I have a fleet of servers that I'm trying to establish SSH connections to, and I'm spawning a new goroutine for every new SSH connection I have to establish. I then send the results of that connection (along with the error(s) (if any)) down a channel, and then read from the channel. This program sort of works, but it freezes in the end even though I close the channel.
This is what I have so far:
package main
import (
"fmt"
"net"
"sync"
"github.com/awslabs/aws-sdk-go/aws"
"github.com/awslabs/aws-sdk-go/service/ec2"
)
// ConnectionResult container
type ConnectionResult struct {
host string
message string
}
func main() {
cnres := make(chan ConnectionResult)
ec2svc := ec2.New(&aws.Config{Region: "us-east-1"})
wg := sync.WaitGroup{}
params := &ec2.DescribeInstancesInput{
Filters: []*ec2.Filter{
&ec2.Filter{
Name: aws.String("instance-state-name"),
Values: []*string{
aws.String("running"),
},
},
},
}
resp, err := ec2svc.DescribeInstances(params)
if err != nil {
panic(err)
}
for _, res := range resp.Reservations {
for _, inst := range res.Instances {
for _, tag := range inst.Tags {
if *tag.Key == "Name" {
host := *tag.Value
wg.Add(1)
go func(hostname string, cr chan ConnectionResult) {
defer wg.Done()
_, err := net.Dial("tcp", host+":22")
if err != nil {
cr <- ConnectionResult{host, "failed"}
} else {
cr <- ConnectionResult{host, "succeeded"}
}
}(host, cnres)
}
}
}
}
for cr := range cnres {
fmt.Println("Connection to " + cr.host + " " + cr.message)
}
close(cnres)
defer wg.Wait()
}
What am I doing wrong? Is there a better way of doing concurrent SSH connections in Go?
The code above is stuck in the range cnres for loop. As pointed out in the excellent 'Go by Example', range will only exit on a closed channel.
One way to address that difficulty, is to run the range cnres iteration in another goroutine. You could then wg.Wait(), and then close() the channel, as such:
...
go func() {
for cr := range cnres {
fmt.Println("Connection to " + cr.host + " " + cr.message)
}
}()
wg.Wait()
close(cnres)
On a tangential note (independently of the code being stuck), I think the intention was to use hostname in the Dial() function, and subsequent channel writes, rather than host.
Thanks to Frederik, I was able to get this running successfully:
package main
import (
"fmt"
"net"
"sync"
"github.com/awslabs/aws-sdk-go/aws"
"github.com/awslabs/aws-sdk-go/service/ec2"
)
// ConnectionResult container
type ConnectionResult struct {
host string
message string
}
func main() {
cnres := make(chan ConnectionResult)
ec2svc := ec2.New(&aws.Config{Region: "us-east-1"})
wg := sync.WaitGroup{}
params := &ec2.DescribeInstancesInput{
Filters: []*ec2.Filter{
&ec2.Filter{
Name: aws.String("instance-state-name"),
Values: []*string{
aws.String("running"),
},
},
},
}
resp, err := ec2svc.DescribeInstances(params)
if err != nil {
panic(err)
}
for _, res := range resp.Reservations {
for _, inst := range res.Instances {
for _, tag := range inst.Tags {
if *tag.Key == "Name" {
host := *tag.Value
publicdnsname := *inst.PublicDNSName
wg.Add(1)
go func(ec2name, cbname string, cr chan ConnectionResult) {
defer wg.Done()
_, err := net.Dial("tcp", ec2name+":22")
if err != nil {
cr <- ConnectionResult{cbname, "failed"}
} else {
cr <- ConnectionResult{cbname, "succeeded"}
}
}(publicdnsname, host, cnres)
}
}
}
}
go func() {
for cr := range cnres {
fmt.Println("Connection to " + cr.host + " " + cr.message)
}
}()
wg.Wait()
}
Frederik's solution works fine but with some exceptions. If command group routines (from loop which write to to the channel) execute command with a bit longer response time, processing routine (Frederik's hint) will process and close the channel, before last command routine to finish, so some data loss may occur.
In my case I'm using it to execute remote SSH command to multiple servers and to print response. Working solution for me is to use 2 separate WaitGroups, one for command group routines and second for processing routine. This way, processing routine will wait all command routines to be completed, then process response and close channel to exit for loop:
// Create waitgroup, channel and execute command with concurrency (goroutine)
outchan := make(chan CommandResult)
var wg_command sync.WaitGroup
var wg_processing sync.WaitGroup
for _, t := range validNodes {
wg_command.Add(1)
target := t + " (" + user + "#" + nodes[t] + ")"
go func(dst, user, ip, command string, out chan CommandResult) {
defer wg_command.Done()
result := remoteExec(user, ip, cmdCommand)
out <- CommandResult{dst, result}
}(target, user, nodes[t], cmdCommand, outchan)
}
wg_processing.Add(1)
go func() {
defer wg_processing.Done()
for o := range outchan {
bBlue.Println(o.target, "=>", cmdCommand)
fmt.Println(o.cmdout)
}
}()
// wait untill all goroutines to finish and close the channel
wg_command.Wait()
close(outchan)
wg_processing.Wait()
I have written a UDP server and client in Go. I am not seeing any error messages when the server is running on port 1200, I also do not see any errors when client is trying to connect to port 1200 on the same machine (OS X 10.9.1)
The server is not printing "SOS...", that is the message the client is writing, in a infinite loop.
The client is able to send a message to the server though, however what the server is reading is 0 bytes.
Server Code
package main
import ( "net" "fmt" "time" )
func main() {
port := "127.0.0.1:1200"
udpAddress, err := net.ResolveUDPAddr("udp4",port)
if err != nil {
fmt.Println("error resolving UDP address on ", port)
fmt.Println(err)
return
}
conn ,err := net.ListenUDP("udp",udpAddress)
if err != nil {
fmt.Println("error listening on UDP port ", port)
fmt.Println(err)
return
}
defer conn.Close()
var buf []byte
for {
time.Sleep(100 * time.Millisecond)
n,address, err := conn.ReadFromUDP(buf)
if err != nil {
fmt.Println("error reading data from connection")
fmt.Println(err)
return
}
if address != nil {
fmt.Println("got message from ", address, " with n = ", n)
if n > 0 {
fmt.Println("from address", address, "got message:", string(buf[0:n]), n)
}
}
}
}
client code, running on same server, with command go run udp-client.go :1200 or go run udp-client.go 127.0.0.1:1200
package main
import (
"fmt"
"net"
"os"
"time"
)
func main() {
if len(os.Args) != 2{
fmt.Fprintf(os.Stderr, "Usage:%s host:port", os.Args[0])
os.Exit(1)
}
service := os.Args[1]
fmt.Println("Connecting to server at ", service)
conn, err := net.Dial("udp",service)
if err != nil {
fmt.Println("Could not resolve udp address or connect to it on " , service)
fmt.Println(err)
return
}
fmt.Println("Connected to server at ", service)
defer conn.Close()
fmt.Println("About to write to connection")
for {
time.Sleep(1000*time.Millisecond)
n, err := conn.Write([]byte("SOS ... \n"))
if err != nil {
fmt.Println("error writing data to server", service)
fmt.Println(err)
return
}
if n > 0 {
fmt.Println("Wrote ",n, " bytes to server at ", service)
}
}
}
The UDPConn.ReadFromUDP method reads data and puts it into the slice you provided.
In your case this slice is nil. Therefore your buffer does not provide enough space for data.
You can fix this by changing one line in the server code:
var buf []byte = make([]byte, 1500)
The buffer size should probably be chosen to fit your network protocol. Or you create a 64k buffer so that you can receive maximum sized udp packets. This however seems a bit wastefull :)