We are back again for the second part of this series. If you didn't check out part one yet make sure to do so from here.

In this part we'll cover the following:

1. An overview of the backend components used.

2. Starting up our Go server

3. Creating a TCP connection between the rust client and Golang server

4. Generating a URL using uuid for every Rust client

5. Downloading and installing dnsmasq which is a useful tool for a development environment to redirect DNS names using wildcards *.ngrnotok.app

Backend Design

We have 2 components in a docker-compose. Nginx is a web server to redirect different requests from and to the servers. And our Golang code consists of 2 servers running in different goroutines. You can be flexible with this I just did this because it took less time and worked as I wanted it to.

The end user requests a URL for example http://231fsa.ngrnotok.app and gets redirected to the Nginx server and then to the Go HTTP Server.

The Rust CLI user once he executes the init command he'll get to Nginx which will forward him to the TCP server where the connection will be maintained throughout his usage.

Our GoLang Server

Before diving into it. There is no specific reason why I decided to use Go here. Maybe it was because I was getting rusty from not using it that much so I decided to use it as a refresher.

Our Go server will be the server that our CLI connects to and achieves a TCP connection with. And it will also be the Server the client requests URLs from.

We will generate a new UUID for every new TCP connection we get from a CLI client and keep an In memory HashMap mapping every UUID to the TCP socket.

Before going any deeper note that this is a simple and working solution. But it doesn't cover every single corner case. It's just to understand what's going on nothing more nothing less.

//main.go
func main() {
    // Listen for incoming connections.
    // create a new mapping
    mapping := newMapping()
    id := uuid.New()
    wg := new(sync.WaitGroup)
    wg.Add(2)
    go spawnServer(8080, mapping, id, handleClientRequest, wg)
    go spawnHttpServer(8081, wg, mapping)

    // block the main thread with stdin
    wg.Wait()
}

In our main.go we create 2 servers as mentioned. One for the CLI client to have a TCP raw connection with. And the other is for the external client when they request a certain URL.

Let's go step by step:

1. newMapping is just a struct that returns an instance of an object mapping that has a Hashmap property. It will map our -to be generated- UUIDs to the TCP connection.

    type mapping struct {
        // map of unique id to connection
        m map[string]net.Conn
    }
   
    // create a new mapping
    func newMapping() *mapping {
        return &mapping{m: make(map[string]net.Conn)}
    }
   

2. We instantiate a uuid instance using the Go uuid package

3. Create a WaitGroup that will make the main thread wait on our two goroutines (both servers) until they terminate and exit gracefully.

4. We invoke SpawnServer which creates the raw TCP server.

    func spawnServer(port int, mapping *mapping, id uuid.UUID, fn func(net.Conn, *mapping, string), wg *sync.WaitGroup) {
        l, err := net.Listen("tcp", "0.0.0.0:" + strconv.Itoa(port))
        println("Listening on port " + strconv.Itoa(port) + "...")
        if err != nil {
            fmt.Println("Error listening:", err.Error())
            os.Exit(1)
        }
        // Close the listener when the application closes.
        defer l.Close()
        defer wg.Done()
        for {
            // Listen for an incoming connection.
            conn, err := l.Accept()
            if err != nil {
                fmt.Println("Error accepting: ", err.Error())
                os.Exit(1)
            }
            go fn(conn, mapping, id.String()) // new UUID generated
        }
    }
   

   Simply put. It's an infinite loop that gets blocked on Accept waiting on new connections. Once a connection is found it passes it to the callback function provided as an argument.

5. spawnHttpServer creates a Go native HTTP server using http package

    func spawnHttpServer(port int, wg *sync.WaitGroup, mapping *mapping) {
        mux := http.NewServeMux()
        mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
            handleProxyRequest(w, r, mapping) // Pass the desired value as an argument
        })
        http.ListenAndServe(":"+strconv.Itoa(port), mux)
        wg.Done()
    }
   

   It forwards the requests to the function handleProxyRequest

handleProxyRequest then will proceed to find the requested connection from the uuid provided in the URL. Request from the client CLI to communicate with its local host and fetch the required response.

Before diving into what handleClientRequest handleProxyRequest do, we'll take it back to the client side and see how the CLI connects to our TCP server.

Updating the Rust CLI client

pub struct Server {}
const LOCAL_ADDRESS: &str = "localhost";
impl Server {
    pub async fn new(address: String, port: String, local_port: String) -> Result<(), Error> {
        let lport: Arc<String> = Arc::new(local_port);
        let mut stream = TcpStream::connect(format!("{}:{}", address, port)).await?;
        println!("Connected to server!");
        let (mut reader, mut writer) = stream.split();
        writer.write_all(b"Client Hello").await?;
        let mut first = true;
        let client = reqwest::Client::new();
        loop {
            println!("Waiting for request");
            let mut buffer = [0; 4096];
            let n = reader.read(&mut buffer).await?;
            if first {
                println!("You can access the website at: {}.ngrnotok.local", String::from_utf8_lossy(&buffer[..n]));
                first = false;
            }
            else{
                // to be covered in part 3.
            }
            if n == 0 {
                break;
            }
        }
        Ok(())
    }

        match &cli.command {
            Some(Commands::Http(argss)) => {
                match &argss.port {
                    Some(port) => {
                        async {
                            match Server::new("ngrnotok.local".to_string(), "8080".to_string(), port.to_string()).await{
                                Ok(_) => {
                                    println!("fin!")
                                }
                                Err(e) => {
                                    println!("Error: {}", e)
                                }
                            }
                        }
                        .await;
                    }
                    None => {
                        println!("No port provided")
                    }
                }
            }
            None => {
                println!("No command provided")
            }
        }
    }

We create a struct Server in our Rust Client. It takes in address which is the address of our Go server, port which is the port of our Go server and local_port which was the argument we provided to the cli ./ngrnotok http -p 3000 for example our local_port here is 3000.

We then use TcpStream package and connect to our Go Server, and have an infinite loop listening for anything in the stream. Only the first time we get something from the Go server it's going to be the uuid so we print that to our CLI user. Then afterwards we only get requests to our local server from clients.

We then update our CLI part of the code (explained in the previous part) and instantiate an instance of the server.

Now you may not understand some of the code above (didn't explain the flow yet) but let's go back to our Go server and I'll explain handleClientRequest first.

Handling Client init requests

func handleClientRequest(conn net.Conn, mapping *mapping, uuid string) {
    mapping.add(uuid, conn)
    buf := make([]byte, 1024)
    _, err := conn.Read(buf)
    if err != nil {
        mapping.remove(uuid)
        return
    }
    conn.Write([]byte(uuid))
}

In our Rust client. When I accept a connection I send Client Hello to the server.

When the server receives this, it generates a new UUID for the connection. And adds it to our Map. Then it returns the uuid to the Client.

This takes us back to when I said that the first bytes the client receives are always the uuid. We simply check for that on the client side and print the message You can access the website at: {}.ngrnotok.local

Now the Go server keeps track of every connection along with its associated UUID.

In part 3 we'll discuss how handleProxyRequest works and how we identify the host from the request URL and proceed.

How to locally use wildcards for domain names

When developing locally, we'll face the problem of since we generate UUIDs on the fly. We need to redirect them all to the same server. But we can't add every uuid in /etc/hosts that are not very efficient and don't mimic what happens in real life.

There is this useful tool called dnsmasq which allows us to use wildcards to redirect to the same host. Here's a link to install it.

To be continued

In the next part, we'll cover how we forward HTTP requests/responses over the raw TCP connection. If you have any questions feel free to leave them in the comments and I'll answer as soon as I can. Till the next one!