Untitled Publication

How to Create a SOCKS5 Proxy with VPN Tunneling Using Docker

Georgios Komninos — Sat, 10 May 2025 05:29:46 GMT

This guide shows you how to set up a SOCKS5 proxy that routes all traffic through a VPN connection using Docker Compose. This setup provides an extra layer of privacy for your internet traffic.

Prerequisites

Docker and Docker Compose installed
A VPN subscription (this example uses NordVPN but works with other providers)

Step 1: Create Project Structure

First, create a directory for your project and a subdirectory for your VPN configuration:

mkdir -p proxy-vpn/vpn1
cd proxy-vpn

Step 2: Create the Docker Compose File

Create a file named docker-compose.yml in the project directory and paste the following configuration:

services:
  vpn1:
    image: dperson/openvpn-client:latest
    container_name: vpn-client1
    cap_add:
      - NET_ADMIN
    devices:
      - /dev/net/tun
    restart: unless-stopped
    volumes:
      - ./vpn1:/vpn
    environment:
      - PUID=1000
      - PGID=1000
    command: ["-f", '""', "-a", "${VPN_USERNAME};${VPN_PASSWORD}"]
    networks:
      - vpn_network1
    ports:
      - "1080:1080"
    dns:
      - 8.8.8.8
      - 8.8.4.4
    sysctls:
      - net.ipv6.conf.all.disable_ipv6=1

  socks-proxy1:
    image: serjs/go-socks5-proxy:latest
    container_name: socks5-proxy1
    restart: unless-stopped
    environment:
      - PROXY_USER=${SOCKS_USER}
      - PROXY_PASSWORD=${SOCKS_PASSWORD}
    network_mode: "service:vpn1"
    depends_on:
      - vpn1
    sysctls:
      - net.ipv6.conf.all.disable_ipv6=1

networks:
  vpn_network1:
    driver: bridge

Step 3: Set Up VPN Configuration

Download your VPN configuration file from your provider (for NordVPN, visit https://nordvpn.com/blog/nordvpn-config-files/)
Save the configuration file as vpn.conf in the vpn1 directory

Here's a sample of what a NordVPN configuration looks like (replace SERVER-IP with the server of your choice - pick one from the configs you downloaded)

client
dev tun
proto udp
remote  1194
resolv-retry infinite
remote-random
nobind
tun-mtu 1500
tun-mtu-extra 32
mssfix 1450
persist-key
persist-tun
ping 15
ping-restart 0
ping-timer-rem
reneg-sec 0
comp-lzo no
verify-x509-name CN=es137.nordvpn.com

remote-cert-tls server

auth-user-pass
verb 3
pull
fast-io
cipher AES-256-CBC
auth SHA512

-----BEGIN CERTIFICATE-----
MIIFCjCCAvKgAwIBAgIBATANBgkqhkiG9w0BAQ0FADA5MQswCQYDVQQGEwJQQTEQ
MA4GA1UEChMHTm9yZFZQTjEYMBYGA1UEAxMPTm9yZFZQTiBSb290IENBMB4XDTE2
MDEwMTAwMDAwMFoXDTM1MTIzMTIzNTk1OVowOTELMAkGA1UEBhMCUEExEDAOBgNV
BAoTB05vcmRWUE4xGDAWBgNVBAMTD05vcmRWUE4gUm9vdCBDQTCCAiIwDQYJKoZI
hvcNAQEBBQADggIPADCCAgoCggIBAMkr/BYhyo0F2upsIMXwC6QvkZps3NN2/eQF
kfQIS1gql0aejsKsEnmY0Kaon8uZCTXPsRH1gQNgg5D2gixdd1mJUvV3dE3y9FJr
XMoDkXdCGBodvKJyU6lcfEVF6/UxHcbBguZK9UtRHS9eJYm3rpL/5huQMCppX7kU
eQ8dpCwd3iKITqwd1ZudDqsWaU0vqzC2H55IyaZ/5/TnCk31Q1UP6BksbbuRcwOV
skEDsm6YoWDnn/IIzGOYnFJRzQH5jTz3j1QBvRIuQuBuvUkfhx1FEwhwZigrcxXu
MP+QgM54kezgziJUaZcOM2zF3lvrwMvXDMfNeIoJABv9ljw969xQ8czQCU5lMVmA
37ltv5Ec9U5hZuwk/9QO1Z+d/r6Jx0mlurS8gnCAKJgwa3kyZw6e4FZ8mYL4vpRR
hPdvRTWCMJkeB4yBHyhxUmTRgJHm6YR3D6hcFAc9cQcTEl/I60tMdz33G6m0O42s
Qt/+AR3YCY/RusWVBJB/qNS94EtNtj8iaebCQW1jHAhvGmFILVR9lzD0EzWKHkvy
WEjmUVRgCDd6Ne3eFRNS73gdv/C3l5boYySeu4exkEYVxVRn8DhCxs0MnkMHWFK6
MyzXCCn+JnWFDYPfDKHvpff/kLDobtPBf+Lbch5wQy9quY27xaj0XwLyjOltpiST
LWae/Q4vAgMBAAGjHTAbMAwGA1UdEwQFMAMBAf8wCwYDVR0PBAQDAgEGMA0GCSqG
SIb3DQEBDQUAA4ICAQC9fUL2sZPxIN2mD32VeNySTgZlCEdVmlq471o/bDMP4B8g
nQesFRtXY2ZCjs50Jm73B2LViL9qlREmI6vE5IC8IsRBJSV4ce1WYxyXro5rmVg/
k6a10rlsbK/eg//GHoJxDdXDOokLUSnxt7gk3QKpX6eCdh67p0PuWm/7WUJQxH2S
DxsT9vB/iZriTIEe/ILoOQF0Aqp7AgNCcLcLAmbxXQkXYCCSB35Vp06u+eTWjG0/
pyS5V14stGtw+fA0DJp5ZJV4eqJ5LqxMlYvEZ/qKTEdoCeaXv2QEmN6dVqjDoTAo
k0t5u4YRXzEVCfXAC3ocplNdtCA72wjFJcSbfif4BSC8bDACTXtnPC7nD0VndZLp
+RiNLeiENhk0oTC+UVdSc+n2nJOzkCK0vYu0Ads4JGIB7g8IB3z2t9ICmsWrgnhd
NdcOe15BincrGA8avQ1cWXsfIKEjbrnEuEk9b5jel6NfHtPKoHc9mDpRdNPISeVa
wDBM1mJChneHt59Nh8Gah74+TM1jBsw4fhJPvoc7Atcg740JErb904mZfkIEmojC
VPhBHVQ9LHBAdM8qFI2kRK0IynOmAZhexlP/aT/kpEsEPyaZQlnBn3An1CRz8h0S
PApL8PytggYKeQmRhl499+6jLxcZ2IegLfqq41dzIjwHwTMplg+1pKIOVojpWA==
-----END CERTIFICATE-----

key-direction 1

#
# 2048 bit OpenVPN static key
#
-----BEGIN OpenVPN Static key V1-----
e685bdaf659a25a200e2b9e39e51ff03
0fc72cf1ce07232bd8b2be5e6c670143
f51e937e670eee09d4f2ea5a6e4e6996
5db852c275351b86fc4ca892d78ae002
d6f70d029bd79c4d1c26cf14e9588033
cf639f8a74809f29f72b9d58f9b8f5fe
fc7938eade40e9fed6cb92184abb2cc1
0eb1a296df243b251df0643d53724cdb
5a92a1d6cb817804c4a9319b57d53be5
80815bcfcb2df55018cc83fc43bc7ff8
2d51f9b88364776ee9d12fc85cc7ea5b
9741c4f598c485316db066d52db4540e
212e1518a9bd4828219e24b20d88f598
a196c9de96012090e333519ae18d3509
9427e7b372d348d352dc4c85e18cd4b9
3f8a56ddb2e64eb67adfc9b337157ff4
-----END OpenVPN Static key V1-----

Step 4: Create Environment Variables File

Create a .env file in the project directory with your credentials:

VPN_USERNAME=username
VPN_PASSWORD=password
#SOCKS_USER=proxyuser # leave commented for no credentials in the socks proxy
#SOCKS_PASSWORD=proxypass # leave commented for no credentials in the socks proxy

Step 5: Start the Services

docker-compose up -d

Step 6: Test the Connection

curl --socks5 localhost:1080 ifconfig.me

This will display your VPN's IP address rather than your actual IP.

Using Your SOCKS5 Proxy

You can now configure applications to use your SOCKS5 proxy with these details:

Proxy host: localhost (or your server's IP)
Port: 1080
Username: proxyuser (or whatever you set in the .env file) (if you set this in .env - otherwise empty)
Password: proxypass (or whatever you set in the .env file) (if you set this in .env - otherwise empty)

How to generate PDF in Go

Georgios Komninos — Fri, 21 Jun 2024 05:00:40 GMT

In this article I am going to show you one way of generating PDF documents using Go. Generating a PDF document is a common need in many applications.

We are building an invoice management application and one of the requirements is the to generate a PDF.

This blog post is part of the Building a Web App with Golang .

From HTML to PDF

In order to create a PDF document we will first create an HTML document and use the weasyprint tool to generate the PDF from the HTML .

There are some golang libraries out there but I found that weasyprint works very nice and haven't run into any issue yet.

Let's first install weasyprint in our machine and see how it works.

Please see the detailed instructions in weasyprint's documentation .

Here is how you can install by utilizing python PIP:

pip install weasyprint

In the documentation there are instructions for many operating systems. Please check there if you run into issues with the above.

Now let's try a demo to verify that it works.

Writing our pdf generator

First let's create a new git branch

git checkout -b invoice-generation

Then create a folder named templates .

Ins

Create an HTML file named invoice.pdf.html

html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Invoice 1001/24title>
    <style>
        body {
            font-family: Arial, sans-serif;
            margin: 20px;
            display: flex;
            justify-content: center;
        }
        .container {
            width: 80%;
            max-width: 800px;
        }
        .header {
            display: flex;
            justify-content: space-between;
            margin-bottom: 20px;
        }
        .seller {
            text-align: right;
        }
        .invoice-info {
            text-align: right;
            margin-bottom: 20px;
        }
        .buyer {
            margin-top: 20px;
        }
        .line-items {
            margin-bottom: 20px;
        }
        .line-items table {
            width: 100%;
            border-collapse: collapse;
        }
        .line-items th, .line-items td {
            border: 1px solid #000;
            padding: 8px;
            text-align: left;
        }
        .totals {
            margin-bottom: 20px;
        }
        .totals table {
            width: 100%;
            border-collapse: collapse;
        }
        .totals td {
            padding: 8px;
            text-align: right;
        }
        .totals .total {
            font-weight: bold;
        }
        .footer {
            border-top: 1px solid #000;
            padding-top: 20px;
            text-align: left;
        }
    style>
head>
<body>

    <div class="container">
        <div class="header">
            <div class="buyer">
                <h3>Bill To:h3>
                <p>Buyer LTDp>
                <p>Buyer Addressp>
                <p>VAT: Vat-numberp>
            div>
            <div class="seller">
                <h2>Sellerh2>
                <p>Seller Addressp>
                <p>Seller Address2p>
                <p>T.I.C. No: Seller Tax Numberp>
                <p>V.A.T. No: Seller Vat Numberp>
            div>
        div>

        <div class="invoice-info">
            <h2>Tax Invoice No.: 1001/24h2>
            <p>Invoice Date: 02/01/2024p>
        div>

        <div class="line-items">
            <table>
                <tr>
                    <th>Descriptionth>
                    <th>Amount (€)th>
                tr>
                <tr>
                    <td>Service Descriptiontd>
                    <td>1,000.00td>
                tr>
            table>
        div>

        <div class="totals">
            <table>
                <tr>
                    <td>Feestd>
                    <td>€1,000.00td>
                tr>
                <tr>
                    <td>VATtd>
                    <td>€0td>
                tr>
                <tr class="total">
                    <td>Totaltd>
                    <td>€1,000.00td>
                tr>
            table>
        div>

        <div class="footer">
            <h3>Invoice Payable within 14 days to:h3>
            <p>Bank Namep>
            <p>IBAN: IBANp>
            <p>BIC: Bic Numberp>
            <p>Account Owner Namep>
        div>
    div>

body>
html>

DISCLAIMER: I used AI (ChatGPT) to assist creating the HTML

Let's see how it looks:

The HTML and CSS needs some polishing, but let's continue.

Now we try to create a pdf from the command line to verify that weasyprint works .

weasyprint templates/invoice.pdf.html invoice.pdf

this will create a pdf document. It looks like that:

Creating a Go package that wraps the weasyprint command

Create a folder pkg/pdfgen

mkdir -p pkg/pdfgen

and add a file pdfgen.go

Now in our pdfgen.go:

package pdfgen

import (
    "context"
    "io"
    "os"
    "os/exec"
)

func Generate(ctx context.Context, w io.Writer, html []byte) error {
    tempHTMLFile, err := os.CreateTemp("", "*.html")
    if err != nil {
        return err
    }

    defer os.Remove(tempHTMLFile.Name())

    if _, err = tempHTMLFile.Write(html); err != nil {
        return err
    }

    if err = tempHTMLFile.Close(); err != nil {
        return err
    }

    tempPDFFile, err := os.CreateTemp("", "*.pdf")
    if err != nil {
        return err
    }

    tempPDFFilePath := tempPDFFile.Name()
    defer os.Remove(tempPDFFilePath)

    if err = tempPDFFile.Close(); err != nil {
        return err
    }

    cmd := exec.CommandContext(ctx, "weasyprint", tempHTMLFile.Name(), tempPDFFilePath)

    err = cmd.Run()

    if err != nil {
        return err
    }

    pdfFile, err := os.Open(tempPDFFilePath)
    if err != nil {
        return err
    }

    defer pdfFile.Close()

    if _, err = io.Copy(w, pdfFile); err != nil {
        return err
    }

    return nil
}

Let's explain what we did:

the function Generate accepts as the context and

and io.Writer and a slice of bytes.

The slice of bytes contains the HTML we want to convert to PDF

The io.Writer is the "place" that we are going to write the contents of the PDF

We chose to use an io.Writer since this will make our package for versatile. We can pass in a file an http.ResponseWriter or everything that implements that interface

The above function works as follows:

creates a temporary file and creates a temporary file with our html contents
creates another temporary file (just to get the name) and removes it
invokes the weasyprint command
reads the file that weasyprint created (the PDF) and copies the contents to the io.Writer we passed as argument

Step 2 might be replaced by creating a random string. But then we have to compute the temporary directory which is not the same for all systems or configurations. Since Go handles these for us we just (ab)use that .

Now let's write a unit test.

let's run our tests:

go test -v ./...

Let's commit our work so far

git add pkg/ templates/
git commit -m "library that generates pdf from html"

We haven't used our new package in our freelance invoice management project yet. We only wrote a library and we will utilize it in the next article

Find the code in the article's branch in github

Conclusion

In this article we created a small go package that will help to generate PDFs in Golang. Basically, we just wrote a wrapper over a command line tool that is available in most platforms.

I found that creating PDF this way works very well in Go. There are some Go libraries like pdfcpu but for the purpose of creating a PDF out of an HTML I found that weasyprint works very well.

In the next article we will integrate our newly created pdfgen package in our application in order to generate PDFs from our invoices.

❤️ Follow me on X or LinkedIn

💡

Do you know a better way to generate PDF in Go?

Crafting a Web app in Golang: Implementing the Service Layer

Georgios Komninos — Thu, 20 Jun 2024 05:00:20 GMT

Tutorial: golang web application development

Hello all, today we will continue the development of our web application by implementing the part of invoice management.

We are going to implement the InvoiceService interface. We will learn also how we can use mocks to unit test our service.

The interface

For managing the invoices we have defined the following interface :

The interface has 3 methods:

Create: this method is responsible to create a new invoice
Get: returns the invoice with id
CreatePDF: Given an invoice id returns a PDF document

Let's get started

Create a new git branch

git checkout -b invoice-svc

Create a new package named invoices

mkdir invoices

and a file invoices/invoices.go .

and add the following contents in the newly created invoices.go .

package invoices

import (
    "context"
    "invoicehub"
)

var _ invoicehub.InvoiceService = (*invoiceSvc)(nil)

type invoiceSvc struct {
    repo invoicehub.InvoiceRepository
}

func New(repo invoicehub.InvoiceRepository) invoicehub.InvoiceService {
    return &invoiceSvc{repo}
}

func (svc *invoiceSvc) Create(ctx context.Context, invoice *invoicehub.Invoice) error {
    return nil
}

func (svc *invoiceSvc) Get(ctx context.Context, id int) (invoicehub.Invoice, error) {
    return invoicehub.Invoice{}, nil
}

func (svc *invoiceSvc) CreatePDF(ctx context.Context, id int) ([]byte, error) {
    return nil, nil
}

We haven't done much here actualy, just created a struct that implements the interface.

Let's now create the a file that we will add our tests:

and let's add placeholder for our tests:

package invoices_test

import "testing"

func Test_New(t *testing.T) {
}

func Test_invoiceSvc_Create(t *testing.T) {
}

func Test_invoiceSvc_Get(t *testing.T) {
}

func Test_invoiceSvc_CreatePDF(t *testing.T) {
}

How to use mocks

In order to write our tests we need to be able to provide a "mock" implementation of the InvoiceRepository . This will help us to test the behavior of our implementation without having to rely on a real database.

When we created our project layout we created a package called mocks that we haven't used so far.

We are going to utilize the package mock and go generate to create mocks for our interfaces.

Setting up mock generator

first install the required package:

go install go.uber.org/mock/mockgen@latest
go get go.uber.org/mock/mockgen/model

and add the following in our Makefile:

Now we can add a special comment line above the definition of the interfaces we want to mock.

The comment has the following format:

//go:generate mockgen -destination=mocks/mock_*.go -package=mocks . InternfaceName

So let's open the file invoices.go and above the definition of the InvoiceRepository:

Now run :

make gen

this will generate a file: mocks/mock_invoice_repo.go

If you open the file you will see that it generated a struct with name MockInvoiceRepository that implments the InvoiceRepository interface .

Each time you add a method in the InvoiceRepository you have to run the make gen command to re-generate the mock implementation.

Now let's open the file invoices/invoices_test.go` and add the following:

the above shows how we were able to initialize the a invoice service using a mock implementation of the InvoiceRepository.

Unit Tests and Implementation

We will now implement our interface's methods one by one.

We start from the Create method.

Our create method should work as following:

we will pass an Invoice struct as an argument and this method will be responsible to :

calculate the InvoiceNumber based on the IssueDate and the last invoice Number of the current year. If an invoice does not exist for the current year we start with invoice number 1001 😄 .

Then we should save the newly created invoice in the database .

Let's do that :

func (svc *invoiceSvc) Create(ctx context.Context, invoice *invoicehub.Invoice) error {
    issueYear := invoice.IssueDate.Year()

    lastInvoice, err := svc.repo.GetLastInvoiceForYear(ctx, issueYear)
    if err != nil {
        return err
    }

    parts := strings.Split(lastInvoice.InvoiceNumber, "/")
    if len(parts) != 2 {
        return errors.New("invalid invoice number")
    }

    lastInvoiceNumber, err := strconv.Atoi(parts[0])
    if err != nil {
        return err
    }

    invoice.InvoiceNumber = fmt.Sprintf("%d/%d", lastInvoiceNumber+1, issueYear)

    if _, err := svc.repo.Create(ctx, invoice); err != nil {
        return err
    }

    return nil
}

Now we have to write some tests to see if this works

in your invoices/invoices_test.go :

Here I have added some test cases . This is not 100% covers all the cases but it's good enough for now.

Let's first implement the first case:

    t.Run("when there is no last invoice for the current year", func(t *testing.T) {
        inv := &invoicehub.Invoice{
            IssueDate: time.Date(2024, 1, 2, 0, 0, 0, 0, time.UTC),
        }

        err := svc.Create(context.Background(), inv)
        require.NoError(t, err)
    })

Now let's run the test and see what it will happen:

go test -v ./... -run=Test_invoiceSvc_Create/when_there_is_no_last_invoice_for_the_current_year

and we get:

The error tells us that our method called the method:
*mocks.MockInvoiceRepository.GetLastInvoiceForYear

and this was not expected .

We need to instruct the test what the call to this method will return .

But wait a second...What our repository returns when a record is not found in the database?

It returns a gorm.ErrNotFound error . We could return that BUT this will leak implemetation details to our service layer. We have to define a custom error in our domain that we can utilize.

Open the invoice.go and add on top:

Now open the sqlite/invoice.go and return that error when an invoice is not found.
We have to do that in two places:

Let's also modify the InvoiceRepository's tests to check that when an invoice is not found it returns that error:

In the func Test_invoiceRepository(t *testing.T) {

when an invoice is not found assert that the error is of this type:

and in the bottom:

Now let's run that test to make sure it works:

go test -v ./... -run=Test_invoiceRepository

Now we can proceed with writing the test for the Create method of InvoiceRepository:

    t.Run("when there is no last invoice for the current year", func(t *testing.T) {
        inv := &invoicehub.Invoice{
            IssueDate: time.Date(2024, 1, 2, 0, 0, 0, 0, time.UTC),
        }

        repo.EXPECT().GetLastInvoiceForYear(gomock.Any(), 2024).Return(invoicehub.Invoice{}, invoicehub.ErrInvoiceNotFound)

        err := svc.Create(context.Background(), inv)
        require.NoError(t, err)
    })

But our test still fails:

Why is that?

in our implementation we do that:

    lastInvoice, err := svc.repo.GetLastInvoiceForYear(ctx, issueYear)
    if err != nil {
        return err
    }

But, when there is no invoice in the database we return an ErrInvoiceNotFound.

In our requirements we have that when a invoice for this year does not exist then we should generate a new invoice number starting with 1001.

Let's fix it:

So now we check for that error and create an invoiceNumber as we need.

But again the test still fail 😟

Ok , this is good actually . We know what is going on. We just call the repo.Create method but we don't mock it as above .

The final code for that test case becomes:

    t.Run("when there is no last invoice for the current year", func(t *testing.T) {
        inv := &invoicehub.Invoice{
            IssueDate: time.Date(2024, 1, 2, 0, 0, 0, 0, time.UTC),
        }

        repo.EXPECT().GetLastInvoiceForYear(gomock.Any(), 2024).Return(invoicehub.Invoice{}, invoicehub.ErrInvoiceNotFound)
        repo.EXPECT().Create(gomock.Any(), inv).Return(1, nil)

        err := svc.Create(context.Background(), inv)
        require.NoError(t, err)
    })

Now let's quickly complete the remaining test cases.
Our test now looks like:

func Test_invoiceSvc_Create(t *testing.T) {
    mctrl := gomock.NewController(t)
    defer mctrl.Finish()

    repo := mocks.NewMockInvoiceRepository(mctrl)

    svc := invoices.New(repo)

    t.Run("when there is no last invoice for the current year", func(t *testing.T) {
        inv := &invoicehub.Invoice{
            IssueDate: time.Date(2024, 1, 2, 0, 0, 0, 0, time.UTC),
        }

        repo.EXPECT().GetLastInvoiceForYear(gomock.Any(), 2024).Return(invoicehub.Invoice{}, invoicehub.ErrInvoiceNotFound)
        repo.EXPECT().Create(gomock.Any(), inv).Return(1, nil)

        err := svc.Create(context.Background(), inv)
        require.NoError(t, err)

        require.Equal(t, "1001/2024", inv.InvoiceNumber)
    })

    t.Run("when there is a last invoice for the current year", func(t *testing.T) {
        inv1 := &invoicehub.Invoice{
            IssueDate:     time.Date(2024, 1, 2, 0, 0, 0, 0, time.UTC),
            InvoiceNumber: "1001/2024",
        }

        invNew := &invoicehub.Invoice{
            IssueDate: time.Date(2024, 2, 5, 0, 0, 0, 0, time.UTC),
        }

        repo.EXPECT().GetLastInvoiceForYear(gomock.Any(), 2024).Return(*inv1, nil)

        repo.EXPECT().Create(gomock.Any(), invNew).Return(2, nil)

        err := svc.Create(context.Background(), invNew)
        require.NoError(t, err)

        require.Equal(t, "1002/2024", invNew.InvoiceNumber)
    })

    t.Run("when there is a database error while getting the last invoice", func(t *testing.T) {
        inv := &invoicehub.Invoice{
            IssueDate: time.Date(2024, 1, 2, 0, 0, 0, 0, time.UTC),
        }

        repo.EXPECT().GetLastInvoiceForYear(gomock.Any(), 2024).Return(invoicehub.Invoice{}, errors.New("something went wrong"))

        err := svc.Create(context.Background(), inv)
        require.Error(t, err)
    })

    t.Run("when we cannot save the invoice", func(t *testing.T) {
        invoice := &invoicehub.Invoice{
            IssueDate: time.Date(2024, 1, 2, 0, 0, 0, 0, time.UTC),
        }

        repo.EXPECT().GetLastInvoiceForYear(gomock.Any(), 2024).Return(invoicehub.Invoice{}, invoicehub.ErrInvoiceNotFound)
        repo.EXPECT().Create(gomock.Any(), invoice).Return(0, errors.New("something went wrong"))

        err := svc.Create(context.Background(), invoice)
        require.Error(t, err)
    })
}

Verify that the tests pass:

go test -v ./...

The Get method

We have to implement the Get method of the InvoiceService now.

We start again from the unit tests and then we will proceed with the implementation.

func Test_invoiceSvc_Get(t *testing.T) {
    mctrl := gomock.NewController(t)
    defer mctrl.Finish()

    repo := mocks.NewMockInvoiceRepository(mctrl)

    svc := invoices.New(repo)

    t.Run("when the invoice is found", func(t *testing.T) {
        expected := invoicehub.Invoice{
            ID:            1,
            InvoiceNumber: "1001/2024",
            IssueDate:     time.Date(2024, 1, 2, 0, 0, 0, 0, time.UTC),
            SellerID:      1,
            BuyerID:       2,
            DaysToPay:     14,
        }

        repo.EXPECT().Get(gomock.Any(), 1).Return(expected, nil)

        inv, err := svc.Get(context.Background(), 1)
        require.NoError(t, err)

        require.Equal(t, expected, inv)
    })

    t.Run("when the invoice is not found", func(t *testing.T) {
        repo.EXPECT().Get(gomock.Any(), 1).Return(invoicehub.Invoice{}, invoicehub.ErrInvoiceNotFound)

        _, err := svc.Get(context.Background(), 1)
        require.Error(t, err)
        require.ErrorIs(t, err, invoicehub.ErrInvoiceNotFound)
    })

    t.Run("when there is a database error", func(t *testing.T) {
        repo.EXPECT().Get(gomock.Any(), 1).Return(invoicehub.Invoice{}, errors.New("something went wrong"))

        _, err := svc.Get(context.Background(), 1)
        require.Error(t, err)
    })
}

The implementation is pretty straightforward here we just call the repo Get method.

Let's commit our work:

git add .
git commit -m "partial implementation of the InvoiceService"

Conclusion

In today's blog post we continued the implementation of our Invoice management web application. The new thing that we discussed today is how to use mocks in golang to assist you with unit testing.

Find all the code in the github repo .

In the next blog post we will continue the implementation of the InvoiceService by adding the required function to generate PDF using golang.

❤️ In case you found this article useful hit the Like button and follow me on X .

💡

How do you do mock in Go? Please leave a comment

Tutorial: Repository pattern in Golang with Test Driven Development

Georgios Komninos — Wed, 19 Jun 2024 05:00:40 GMT

In the last blog post I showed you how you can implement the repository pattern in Golang. We used SQLite and GORM to implement the CompanyRepository .

Today I am going to show you how to implement the InvoiceRepository .
The implementation will be very similar but today we are going to start first by writing unit tests and then the actual implementation.

Create a new branch

We will work in a new git branch. Let's create this:

git checkout -b invoice-repo

Our interface

We have to implement a struct that implements these three methods:

Create : Inserts an invoice in the database and returns it's id
Get: Fetches an invoices from the database using its id
GetLastInvoiceForYear: Returns the last invoice from the provided year.

Writing a test first

In which package are we going to write our tests?

In our project layout we have a package sqlite . This is where we are going to write our code.

Now create a new file named invoice_test.go .

Let's write our tests:

package sqlite_test

import (
    "testing"
)

func Test_invoiceRepository(t *testing.T) {
}

We don't test anything so far but let's verify that we can run our test:

go test -v ./... -run=Test_invoiceRepository

the output should be similar to:

Provide a dummy implementation of the interface

Create a file sqlite/invoice.go with contents:

package sqlite

import (
    "context"
    "invoicehub"

    "gorm.io/gorm"
)

var _ invoicehub.InvoiceRepository = &invoiceRepository{}

type invoiceRepository struct {
    db *gorm.DB
}

func NewInvoiceRepository(db *gorm.DB) invoicehub.InvoiceRepository {
    return &invoiceRepository{db}
}

func (r *invoiceRepository) Create(ctx context.Context, invoice *invoicehub.Invoice) (int, error) {
    return 0, nil
}

func (r *invoiceRepository) Get(ctx context.Context, id int) (invoicehub.Invoice, error) {
    return invoicehub.Invoice{}, nil
}

func (r *invoiceRepository) GetLastInvoiceForYear(ctx context.Context, year int) (invoicehub.Invoice, error) {
    return invoicehub.Invoice{}, nil
}

We only wrote the required methods but they are empty.

We now create our tests:

func compareInvoice(t *testing.T, a, b invoicehub.Invoice) {
    require.Equal(t, a.ID, b.ID, "ID mismatch")
    require.Equal(t, a.InvoiceNumber, b.InvoiceNumber, "InvoiceNumber mismatch")
    require.Equal(t, a.IssueDate, b.IssueDate, "IssueDate mismatch")
    require.Equal(t, a.SellerID, b.SellerID, "SellerID mismatch")
    require.Equal(t, a.BuyerID, b.BuyerID, "BuyerID mismatch")
    require.Equal(t, a.DaysToPay, b.DaysToPay, "DaysToPay mismatch")

    require.Len(t, a.LineItems, len(b.LineItems), "LineItems length mismatch")

    for i := range a.LineItems {
        require.Equal(t, a.LineItems[i].Description, b.LineItems[i].Description, "Description mismatch")
        require.Equal(t, a.LineItems[i].Amount.Currency, b.LineItems[i].Amount.Currency, "Currency mismatch")
        require.Equal(t, a.LineItems[i].Amount.Value.String(), b.LineItems[i].Amount.Value.String(), "Amount value mismatch")
    }
}


func Test_invoiceRepository(t *testing.T) {
    db, err := sqlite.SetupDB(":memory:")
    require.NoError(t, err)

    repo := sqlite.NewInvoiceRepository(db)
    require.NotNil(t, repo)

    ctx := context.Background()

    invoice := invoicehub.Invoice{
        InvoiceNumber: "1001/24",
        IssueDate:     time.Date(2024, 1, 2, 0, 0, 0, 0, time.UTC),
        SellerID:      1,
        BuyerID:       2,
        DaysToPay:     14,

        LineItems: []invoicehub.LineItem{
            {
                Description: "My awesome services",
                Amount: invoicehub.Amount{
                    Value:    decimal.NewFromFloat(100.00),
                    Currency: "EUR",
                },
                VatRate: decimal.NewFromFloat(0.19),
            },
        },
    }

    // Create a new invoice
    id, err := repo.Create(ctx, &invoice)
    require.NoError(t, err)
    require.NotZero(t, id)
    require.Equal(t, id, invoice.ID)

    // fetch the invoice

    invoice2, err := repo.Get(ctx, id)
    require.NoError(t, err)
    compareInvoice(t, invoice, invoice2)

    // fetch a non-existing invoice
    _, err = repo.Get(ctx, 999)
    require.Error(t, err)

    // add one more invoice
    invoice3 := invoicehub.Invoice{
        InvoiceNumber: "1002/24",
        IssueDate:     time.Date(2024, 2, 2, 0, 0, 0, 0, time.UTC),
        SellerID:      1,
        BuyerID:       2,
        DaysToPay:     14,

        LineItems: []invoicehub.LineItem{
            {
                Description: "web development",
                Amount: invoicehub.Amount{
                    Value:    decimal.NewFromFloat(100.00),
                    Currency: "EUR",
                },
                VatRate: decimal.NewFromFloat(0.19),
            },
            {
                Description: "web scraping",
                Amount: invoicehub.Amount{
                    Value:    decimal.NewFromFloat(150.00),
                    Currency: "EUR",
                },
                VatRate: decimal.NewFromFloat(0.19),
            },
        },
    }

    id3, err := repo.Create(ctx, &invoice3)
    require.NoError(t, err)
    require.NotZero(t, id3)
    require.Equal(t, id3, invoice3.ID)

    // fetch the last invoice for the year 2024

    lastInvoice, err := repo.GetLastInvoiceForYear(ctx, 2024)
    require.NoError(t, err)
    compareInvoice(t, invoice3, lastInvoice)
}

In the above test:

create an invoice
assert that our function returns a newly generated id
assert that the id generated is also set in our Invoice struct
try to fetch the invoice with that id
try to fetch a non existing invoice
create another invoice for a later date
assert that the last invoice of the year is the last created

We are using the package decimal.Decimal and testify package does not play well with it (bacause the way it uses reflection I think) . Thus, we created a custom compareInvoice function that performs the assertions

now run the test again:

As expected our test fails . We don't have any implementation yet.

Separate our domain entity (invoicehub.Invoice) from the database models

In sqlite/invoice.go let's define the required struct that reassembles our database table:

type dbinvoice struct {
    ID            int       `gorm:"primaryKey"`
    InvoiceNumber string    `gorm:"type:text"`
    IssueDate     time.Time `gorm:"type:datetime"`
    BuyerID       int
    SellerID      int
    DaysToPay     int

    LineItems datatypes.JSONSlice[invoicehub.LineItem]
}

above we defined the db model (the table will be generated from that).

In order for Gorm to take care of the table creation modify the sqlite/sqlite.go .

an attentive reader will have already noticed that the sellerID and buyerID fields do not have foreign keys that reference the dbcompanies table.

We will leave these out for now. However this might lead to inconsistencies in our data at some point.

Implement the Create method

func (r *invoiceRepository) Create(ctx context.Context, invoice *invoicehub.Invoice) (int, error) {
    dbInvoice := dbinvoice{
        InvoiceNumber: invoice.InvoiceNumber,
        IssueDate:     invoice.IssueDate,
        BuyerID:       invoice.BuyerID,
        SellerID:      invoice.SellerID,
        DaysToPay:     invoice.DaysToPay,
        LineItems:     datatypes.NewJSONSlice(invoice.LineItems),
    }

    if err := r.db.WithContext(ctx).Create(&dbInvoice).Error; err != nil {
        return 0, err
    }

    invoice.ID = dbInvoice.ID

    return invoice.ID, nil
}

Implement the Get method

func (r *invoiceRepository) Get(ctx context.Context, id int) (invoicehub.Invoice, error) {
    var dbitem dbinvoice
    if err := r.db.WithContext(ctx).First(&dbitem, id).Error; err != nil {
        return invoicehub.Invoice{}, err
    }

    invoice := invoicehub.Invoice{
        ID:            dbitem.ID,
        InvoiceNumber: dbitem.InvoiceNumber,
        IssueDate:     dbitem.IssueDate,
        BuyerID:       dbitem.BuyerID,
        SellerID:      dbitem.SellerID,
        DaysToPay:     dbitem.DaysToPay,
        LineItems:     dbitem.LineItems,
    }

    return invoice, nil
}

Implement the GetLastInvoiceForYear

func (r *invoiceRepository) GetLastInvoiceForYear(ctx context.Context, year int) (invoicehub.Invoice, error) {
    query := r.db.WithContext(ctx).
        Where("strftime('%Y', issue_date) = ?", strconv.Itoa(year)).
        Order("issue_date desc").
        Limit(1)

    var dbitem dbinvoice
    if err := query.First(&dbitem).Error; err != nil {
        return invoicehub.Invoice{}, err
    }

    invoice := invoicehub.Invoice{
        ID:            dbitem.ID,
        InvoiceNumber: dbitem.InvoiceNumber,
        IssueDate:     dbitem.IssueDate,
        BuyerID:       dbitem.BuyerID,
        SellerID:      dbitem.SellerID,
        DaysToPay:     dbitem.DaysToPay,
        LineItems:     dbitem.LineItems,
    }

    return invoice, nil
}

This one is a little bit more tricky.

We utilize the strftime of SQLite in order to fetch all invoices with issue date in the provided year.

We don't have an index in the IssueDate column. This query might be slow when we have a lot of data.

But, since this web application will be only for one user we will skip creating index beforehand. We will do this when it's required. But it's good to keep that in mind.

Now we run our tests again:

It looks we are good so let's commit

git add .
git commit -m "implements invoice repository"

You can find today's version in the github branch:

github invoice-repo branch

Conclusion

This blog posted shows the reader how to implement the repository pattern in Go using GORM and SQLite.

We seen how we can write unittests first for our database related code and then complete the implementation.

We once again flagged the need to separate your database models with your domain entities.

❤️ If you read the article until here and liked it please hit the like button.

👉 I would love you to follow me on X or LinkedIn

💡

What challenges have you faced while implementing the repository pattern in your projects?

Tutorial: Implementing Repository with GORM and SQLite

Georgios Komninos — Tue, 18 Jun 2024 05:00:47 GMT

In the previous part of the series we created the required interface for you invoice generation/management web application.

In this post we are going to provide and implementation of the CompanyRepository .

In our requirements in the first part of the series we decided that we are going to use SQLite as our DBMS. Let's implement together the integration with SQLite.

Implementing the CompanyRepository

We are going to use GORM to interact with the database. However, we are going to ensure that our database models and code will be decoupled so we can switch ORM/driver or even database in the future.

We are going to work in the sqlite package.

Let's first create a branch

 git checkout -b company-repo

and then install the dependencies:

go get gorm.io/gorm
go get gorm.io/driver/sqlite
go get gorm.io/datatypes"

then in our sqlite/sqlite.go add

package sqlite

import (
    "gorm.io/driver/sqlite"
    "gorm.io/gorm"
)

func SetupDB(path string) (*gorm.DB, error) {
    db, err := gorm.Open(sqlite.Open(path), &gorm.Config{})
    if err != nil {
        return nil, err
    }

    err = db.AutoMigrate(
        &dbcompany{},
    )
    if err != nil {
        return nil, err
    }

    return db, nil
}

Explanation:

SetupDB function creates a connection to the SQLite database and runs the migrations, meaning it creates the required tables if not exists or modifies them according to our schema.

and then create a file sqlite/company.go

package sqlite

import (
    "context"
    "invoicehub"

    "gorm.io/datatypes"
    "gorm.io/gorm"
)

var _ invoicehub.CompanyRepository = (*companyRepository)(nil)

type dbcompany struct {
    ID           int    `gorm:"primaryKey"`
    Name         string `gorm:"type:text"`
    Address      datatypes.JSONType[invoicehub.Address]
    Email        string `gorm:"type:text"`
    TaxID        string `gorm:"type:text"`
    VatID        string `gorm:"type:text"`
    BankAccounts datatypes.JSONSlice[invoicehub.BankAccount]
}

type companyRepository struct {
    db *gorm.DB
}

func NewCompanyRepository(db *gorm.DB) invoicehub.CompanyRepository {
    return &companyRepository{db: db}
}

func (r *companyRepository) Create(ctx context.Context, company *invoicehub.Company) (int, error) {
    // Convert invoicehub.Company to dbcompany
    dbCompany := dbcompany{
        Name:         company.Name,
        Address:      datatypes.NewJSONType(company.Address),
        Email:        company.Email,
        TaxID:        company.TaxID,
        VatID:        company.VatID,
        BankAccounts: datatypes.NewJSONSlice(company.BankAccounts),
    }

    // Save to database

    if err := r.db.WithContext(ctx).Create(&dbCompany).Error; err != nil {
        return 0, err
    }

    company.ID = dbCompany.ID

    return company.ID, nil
}

func (r *companyRepository) Update(ctx context.Context, company *invoicehub.Company) error {
    var existingCompany dbcompany
    if err := r.db.WithContext(ctx).First(&existingCompany, company.ID).Error; err != nil {
        return err
    }

    existingCompany.Name = company.Name
    existingCompany.Address = datatypes.NewJSONType(company.Address)
    existingCompany.Email = company.Email
    existingCompany.TaxID = company.TaxID
    existingCompany.VatID = company.VatID
    existingCompany.BankAccounts = datatypes.NewJSONSlice(company.BankAccounts)

    if err := r.db.WithContext(ctx).Save(&existingCompany).Error; err != nil {
        return err
    }

    return nil
}

func (r *companyRepository) Get(ctx context.Context, id int) (invoicehub.Company, error) {
    var dbCompany dbcompany
    if err := r.db.WithContext(ctx).First(&dbCompany, id).Error; err != nil {
        return invoicehub.Company{}, err
    }

    company := invoicehub.Company{
        ID:           dbCompany.ID,
        Name:         dbCompany.Name,
        Address:      dbCompany.Address.Data(),
        Email:        dbCompany.Email,
        TaxID:        dbCompany.TaxID,
        VatID:        dbCompany.VatID,
        BankAccounts: dbCompany.BankAccounts,
    }

    return company, nil
}

❓why I used a dbcompany structs that basically reassembles our invoicehub.Company struct

We want our repository implementation to be decoupled with the domain Entity. The reason is flexibility to change in the future. We don't want our domain Entity to include any gorm related code (see the struct tags).

Writing unit tests

We have to verify that our implementation works. Let's write a unit test for that.

For unit test I like to use the testify package

go get github.com/stretchr/testify

Create a file sqlite/company_test.go .

package sqlite_test

import (
    "context"
    "invoicehub"
    "invoicehub/sqlite"
    "testing"

    "github.com/stretchr/testify/require"
)

func Test_companyRepository(t *testing.T) {
    db, err := sqlite.SetupDB(":memory:")
    require.NoError(t, err)

    repo := sqlite.NewCompanyRepository(db)
    ctx := context.Background()

    company := invoicehub.Company{
        Name:  "Example Ltd",
        Email: "info@example.com",
        TaxID: "123456789",
        VatID: "987654321",
        Address: invoicehub.Address{
            Street:     "123 Example St",
            City:       "Limassol",
            PostalCode: "12345",
            Country:    "CY",
        },
        BankAccounts: []invoicehub.BankAccount{
            {
                BankName:      "Example Bank",
                AccountNumber: "1234567890",
                IBAN:          "EX12345678901234567890",
                BIC:           "EXBIC123",
            },
        },
    }

    // Create a new company
    id, err := repo.Create(ctx, &company)
    require.NoError(t, err)
    require.NotZero(t, id)
    require.Equal(t, id, company.ID)

    // Get the company
    company2, err := repo.Get(ctx, id)
    require.NoError(t, err)
    require.Equal(t, company, company2)

    // Update the company
    company.Name = "Example Ltd 2"
    err = repo.Update(ctx, &company)
    require.NoError(t, err)

    updatedCompany, err := repo.Get(ctx, id)
    require.NoError(t, err)
    require.Equal(t, company, updatedCompany)

    // not found
    _, err = repo.Get(ctx, 999)
    require.Error(t, err)
}

💥 We cover just the basics test cases so we have some confidence that it works.

Let's run the tests

go test -v ./...

and you should get something like:

Pass the dependency in our main.go

Open cmd/main.go and do the following changes:

Pay attention, that I set a default path that will work with our docker setup

Also add to the .gitignore the data.sqlite3 file.

echo "data.sqlite3" >> .gitignore

Now let's test our docker setup.

make dev

notice the created file:

Let's open the file with an sqlite client. On Linux run:

Let's now commit our code so far

As usual you can find the code so far in the relate github branch .

Deployment

We need to setup a new environment variable in our production .env .

FH_DB_PATH=invoices.sqlite3

Now you can go back to your machine and run

make deploy

After deployment check that the invoices.sqlite3 file was created .

Conclusion

In this tutorial, we successfully implemented the CompanyRepository using GORM and SQLite, ensuring our code remains flexible and decoupled from the domain entity. We also wrote unit tests to verify our implementation and made necessary adjustments to our main.go to integrate the repository. Finally, we configured our production environment and verified the deployment.

In the next article we are going to implement the InvoiceRepository .

❤️ Please follow me on X or LinkedIn

If you have any questions please reach out to me via a comment or a DM in X.

Find the rest of the tutorials in this series here

Tutorial: Defining the Domain entities

Georgios Komninos — Mon, 17 Jun 2024 05:00:27 GMT

This blog is the part of the series Building a Web App with Golang.

In today's article we are going to define the basic entities and the operations on them for our web application. In the first part we defined the scope of the application.

We are going to build a system that can issue issue invoices for clients.

Let's create a new git branch to work on:

git checkout -b basic-domain

Company

Every client and "us" (our company/the freelancer) will be represented by an entity Company .

Let's create a file named company.go in the root of our project

then we define a struct with the basic attributes we have:

package invoicehub

type Company struct {
    ID           int
    Name         string
    Address      Address
    Email        string
    TaxID        string
    VatID        string
    BankAccounts []BankAccount
}

type Address struct {
    Street     string
    City       string
    PostalCode string
    Country    string
}

type BankAccount struct {
    BankName      string
    AccountNumber string
    IBAN          string
    BIC           string
}

The idea is that we will represent each real life company using the above struct.

Invoice

Create an invoice.go in the root of the project

Each invoice will have at least the following:
- Seller
- Buyer
- Amount
- Issue Date
- Invoice Number
- Days (number of days after issuance that must be paid).

We need to install first one dependency to handle decimal numbers:

get github.com/shopspring/decimal

Now let's start with that:

package invoicehub

import (
    "time"

    "github.com/shopspring/decimal"
)

type Invoice struct {
    ID            int
    InvoiceNumber string
    IssueDate     time.Time
    SellerID      int
    BuyerID       int
    DaysToPay     int

    LineItems []LineItem
}

type LineItem struct {
    Description string
    Amount      Amount
    VatRate     decimal.Decimal
}

type Amount struct {
    Value    decimal.Decimal
    Currency string
}

The above structs should satisfy our needs for now.

Pay, attention that I am using ids for Buyer and Seller. We could have used a Company struct there.

Why I choose to use ids instead of a company struct?

Our application now is a "monolithic" application. In the future we might want to split into different microservices (maybe one microservice to handle the companies and one microservice to handle the invoices). Let's be proactive and try to decouple the Companies with the Invoices.

I does not look that we need other structs to represent our business domain. Of-course , we might forgotten something but then we will change.

Operations

Our application is pretty simple:

For a Company we need to be able to:

Create a company
Read a company
Update a company

Our requirements don't have a Delete so we skip for now.

Creating, Reading, Updating a company are straightforward.

First we need a way to do these operations in our storage layer.
For this we are going to create an interface CompanyRepository

Let's do that:

in your company.go add the following:

type CompanyRepository interface {
    Create(ctx context.Context, company *Company) (int, error)
    Update(ctx context.Context, company *Company) error
    Get(ctx context.Context, id int) (Company, error)
}

Now for our invoices:

We need to be able to :

Create Invoice
Read Invoice
Create a PDF from an invoice

we don't have full CRUD (Create Read Update Delete) since we don't want to Update or Delete invoices.

Let's define the repository also for invoices:

in the invoice.go add:

type InvoiceRepository interface {
    Create(ctx context.Context, invoice *Invoice) (int, error)
    Get(ctx context.Context, id int) (Invoice, error)
}

What about PDF generation here? This is not a database operation.
Additionally, what about creation of an invoice? It is a little bit more complex.

In the requirements we had that the InvoiceNumber looks like:

1031/24

so we have two parts:
1. a number that resets every year (1031)
2. the year (24)

So the next invoice should look like 1032/24 but if it is the first for 2025 should look like 1001/24 .

Notice that we start always with 1000.

This is just one way others, might do it differently.

It would be nice to have a generic way to compute that per customer. BUT, do not forget our requirements. We want to code for now and not for imaginary use cases. Let's get stuff done

Since invoices are a bit more complex let's also create a Service Layer that will handle the business logic .

open invoices.go and add:

type InvoiceService interface {
    Create(ctx context.Context, invoice *Invoice) error
    Get(ctx context.Context, id int) (Invoice, error)
    CreatePDF(ctx context.Context, id int) ([]byte, error)
}

We also need an extra method in the Repository that returns the last invoice of the year. We are going to use that to compute the invoice number

Add the GetLastInvoiceForYear in the repository

Adding the interfaces in our HTTP layer

Now that we have "described" the operations of our system via the interfaces let's see where we are going to use them.

in the http/router.go we defined a base handler :

Let's add the interfaces there:

type baseHandler struct {
    e         *echo.Echo
    companies invoicehub.CompanyRepository
    invoices  invoicehub.InvoiceService
}

so when we create the baseHandler we should inject the implementations of the interfaces:

Since we don't have any concrete implementation yet let's pass nil where we call the NewRouter function in our cmd/main.go .

Commit and Push

let's commit our code and push our branch

git add .
git commit -m "add basic domain entities"
git push origin basic-domain

And as usual, you can find today's code in the github branch

Summary and what is next

In today's blog post we started implementing our business logic. We first defined the required structs and the operations that we can do on these.
The operations are defined using interfaces. Utilizing interfaces allows us to think about our business logic without having to think about implementation details. Additionally, it will allow us to easily unit test our application later.

I would like to share a piece of advice here. When you have to develop an application no matter how complex it is try first to focus on what is your goal. Don't fall into the trap of questioning your self all the time like:

what if the user wants also that?
what if this changes ?
what if ... ?

These what if may never happen. If they do be confident that when it's required you will handle it.

I am not saying or implying do not think about scalability, changes of requirements etc . I am saying think about them try to code in a way that some decisions are deferred but don't waste your time trying to code something smart when it is not required.

The moments I am writing these lines I am thinking the following:

- How am I distinguish which company is the Seller when we don't have a User entity ?
- What if when we create an invoice someone else does it parallel (so the invoice number may come wrong?_
- How am i going to create an invoice number when I do it near the end of the year and I use UTC in the system but my current timezone is UTC+3 and I am in the next year?
- ...

and many many more.

Some of these are maybe valid concerns. But so far we don't have anything working yet. We don't have to aim for perfection, a working software is better that no software in the end.

Links to the other part of that series

https://blog.gkomninos.com/crafting-a-web-application-with-golang-a-step-by-step-guide

https://blog.gkomninos.com/setting-up-a-docker-development-enviroment-for-go

https://blog.gkomninos.com/building-a-robust-web-server-in-go-a-step-by-step-guide

https://blog.gkomninos.com/tutorial-deployment-of-golang-web-app-using-systemd

👍 Please leave a comment or like the article if you like it. This will help me gain visibility.

❤️ Following me on Github or X or LinkedIn motivates me keep writing.

Tutorial: Deployment of Golang web app using Systemd

Georgios Komninos — Sun, 16 Jun 2024 05:00:20 GMT

Today, I am going to show you a simple way of deploying a Golang web application.

We are going to use Systemd and a Makefile to deploy the code when we merge to main branch. In a future blog post we will revisit and show you how you can deploy
when you push to main by utilizing Github Pipelines.

Let's get started 🥳

Setting up Systemd

I assume that you already have a linux VPS. If not, then I recommend you create an account in Digital Ocean using this link . If you register using this link you will get $200 in credit and I will get $25 in case you continue using this. Basically, you can try this for free so give it a try .

The instructions will be for an Ubuntu VPS so if you create a new one now please use the Ubuntu image.

Setting up DNS

Configure your domain to point to your server's ip address.

then ssh into the VPS and do:

In your VPS

sudo touch /etc/systemd/system/invoice-hub.service
mkdir -p /home/giorgos/invoice-server
touch /home/giorgos/invoice-server/.env
sudo mkdir -p /.cache/.certs && sudo chown -R giorgos:giorgos /.cache/.certs

☢️ adjust the paths and the USER

Then open with an editor and paste:

[Unit]
Description=Freelance invoice hub service

[Install]
WantedBy=multi-user.target

[Service]
Type=simple
ExecStart=/home/giorgos/invoice-server/invoice-server
WorkingDirectory=/home/giorgos/invoice-server
EnvironmentFile=/home/giorgos/invoice-server/.env
Restart=always
RestartSec=5
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=%n

⚠️ modify the paths to match your servers.

❕this is not the "best" systemd config. However we want to deploy fast and this is good enough. We will revisit in the future

The being still in your VPS edit the .env file you created and add:

FIH_DOMAIN=

Obviously, replace with your domain.

Now let's reload the systemd and enable the service

sudo systemctl daemon-reload
sudo systemctl enable invoice-hub

In your local computer

Now in your local computer build the program:

GOOS=linux GOARCH=amd64 go build -o invoice-server cmd/main.go

This will create a file invoice-server .
If your VPS uses a different OS or architecture please use the correct enviroment variables. See here for a tutorial .

Now let's manually copy the file to the server

scp invoice-server @:~/invoice-server/invoice-server

The above command just copies the executable you build to the servers path.

If not already done I highly recommend you create an public/private key pair and use that to login to your server .

This tutorial is not about that, so if you don't know how to do it google or ask ChatGPT .

I like to add an entry in my ~/.ssh/config file like:

Host 
HostName 
User giorgos
IdentityFile ~/.ssh/id_rsa
Port 22

Highly recommend to do that.

Then you can ssh using ssh just-a-name .

Enough, with that. I assume you managed to copy your file to the server now.

and run:

sudo systemctl start invoice-hub

Now check the status:

And if you did everything correct then you can visit your new web app

https://YOUR-DOMAIN.com

and you should see the

🚀🚀🚀🚀🚀

Because it's a little bit tedious to manually build, copy the executable and the run stop and start the service let's automate a bit.

We are going to utilize our make file

Let's create a new branch:

git checkout -b deploy-Makefile-auth

and add the following in your Makefile

deploy: ## deploys to the remote server
    GOOS=linux GOARCH=amd64 go build -o invoice-server cmd/main.go
    scp invoice-server contabo-main:~/invoice-server/invoice-server.2
    ssh myserver sudo systemctl stop invoice-hub
    ssh myserver sudo mv invoice-server/invoice-server.2 invoice-server/invoice-server
    ssh myserver sudo systemctl start invoice-hub

Please replace myserver with your server Host (as you configured in your ~/.ssh/config OR use username@server-ip instead

let's do a change and deploy.

Instead of HELLO WORLD lets make our handler to return

THANK YOU

Change your http/router.go as in the image

and then run

make deploy

wait until the deployment finishes and go to you website

You should see the above .

Basic HTTP Auth

We don't want our web app to be accessible to everyone. Let's configure Basic HTTP Auth.

In our case this is fine since the app will be used only by one user and we are using TLS.

Echo has a middleware that does that for us.

Let's do the following:

go get "github.com/labstack/echo/v4/middleware"

and then in our router.go modify the new method to work like:

As you see in the image we need to store our username and password in the environment variables.

For local development add them in the dev.env

for production set them in your server invoice-server/.env file.

USE SOMETHING SECURE

Let's now test it

make dev

and then when you visit:

https://local.freelance-invoice-hub.com

since it works let's now commit and deploy:

git add .
git commit -m "simple deploy script and basic auth"
git push origin deploy-Makefile-auth

And let's deploy

make deploy

As usual you can find all the code in the related github branch

Summary and what's next

Today we learned a simple way to deploy our Golang applications using Systemd.

This is a really basic method and we can improve our deployments a LOT.

However, the scope of this series is to create a WORKING golang application and at the same time learn that our time is precious. We will do only what is required.
When we have an application working we will keep improving.

Additionally, we learned how to use the Basic Auth Middleware that echo providers

In the next blog we will start coding our application. We will define our domain models and the operations on them.

Until then have fun.

Please comment if something is not clear or does not work for you and I will try to help

Also, don't forget to follow me on X

Building a Robust Web Server in Go: A Step-by-Step Guide

Georgios Komninos — Sat, 15 Jun 2024 05:00:44 GMT

We continue our journey implementing a real world web application in Golang in this blog post.

In the previous post we setup our docker development environment that just prints hello world.

In this post we are going to setup a real webserver that it's going to serve our web application.

We are going to learn how to:

setup the golang http.Server
automatically obtain a TLS certificate using let's encrypt
create and HTTP handler using Echo Framework

Webserver

Golang offers a very capable HTTP server in the package http . We are going to utilize that.

Le't first create a new git branch:

git checkout -b basic-webserver

Let's see our current project layout:

We will put the code of our webserver in the http package. Let's not complicate things and create extra packages or abstractions for now.

In the http/http.go

package http

import (
    "context"
    "crypto/tls"
    "net/http"
    "time"
)

type ServerParams struct {
    Handler http.Handler
}

type Server struct {
    websrv *http.Server
}

func New(params *ServerParams) *Server {
    ans := Server{
        websrv: &http.Server{
            Addr:              ":443",
            Handler:           params.Handler,
            ReadTimeout:       5 * time.Second,
            WriteTimeout:      10 * time.Second,
            IdleTimeout:       5 * time.Second,
            ReadHeaderTimeout: 5 * time.Second,
            MaxHeaderBytes:    1 << 20,
        },
    }

    return &ans
}

func (s *Server) Start(ctx context.Context) error {
    return s.websrv.ListenAndServeTLS("", "")
}

As you noticed all the settings are hardcoded for the moment. However, we have a params struct passed as an argument to the function so we can customize.

An interesting part is that our server listens to port 443 (TLS) but we haven't configured any SSL certificate.

Let's see the behavior

in cmd/main.go :

package main

import (
    "context"

    "invoicehub/http"
)

func main() {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    params := http.ServerParams{}

    srv := http.New(¶ms)

    err := srv.Start(ctx)
    if err != nil {
        panic(err)
    }
}

and then:

we need a valid SSL certificate.

We have actually two things to do:

Make our server to automatically obtain the SSL certificate when we deploy to a server
Provide a valid certificate when we work locally

Valid local certificate

Create a folder certs and add to gitignore

mkdir certs
echo "certs/*" > .gitignore

Install the awesome mkcert tool for your OS .

then add in your Makefile

create-certs: ## generate self-signed certificates
    @mkcert -install
    @mkcert -cert-file ./certs/local-cert.crt \
       -key-file ./certs/local-cert.key \
       local.freelance-invoice-hub.com localhost 127.0.0.1 ::1

And run make certs

the above created 2 files:
./certs/local-cert.crt

./certs/local-cert.key

We need to modify our code to use these certs when we work locally.

We read an environment variable called FIH_DOMAIN , if this is empty then use the certificates we just created.

(also import the os package in your imports)

Let's try that:

So for localhost it works, let's also configure the domain
local.freelance-invoice-hub.com.

For this to work we need to add an entry on

On linux add an entry on /etc/hosts like:

for other operating systems do the equivalent as described here

and try again :

Of course the HTTP status code we get is perfectly valid since we haven't registered anything on our server yet.

Obtain a TLS certificate automatically using Auto TLS

To obtain automatically a valid TLS certificate using Golang we are going to install two packages:

When in your root directory do :

go get "golang.org/x/crypto/acme" "golang.org/x/crypto/acme/autocert"

This will install the 2 packages . Actually these two package are part of the Go project but they are in different repos. That's why they are prefixed with an /x/ .

Now add a method in the Server struct:

func (s *Server) setupAutoTLS(domains []string) {
    const defaultCertCache = "/.cache/.certs"
    autoTLSManager := autocert.Manager{
        Prompt:     autocert.AcceptTOS,
        Cache:      autocert.DirCache(defaultCertCache),
        HostPolicy: autocert.HostWhitelist(domains...),
    }
    // https://ssl-config.mozilla.org/#server=go&version=1.22.0&config=intermediate&guideline=5.7
    s.websrv.TLSConfig = &tls.Config{
        MinVersion:               tls.VersionTLS12,
        PreferServerCipherSuites: true,
        CipherSuites: []uint16{
            tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
            tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
            tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
            tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
            tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,
            tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,
        },
        CurvePreferences: []tls.CurveID{
            tls.CurveP256,
            tls.X25519,
        },
        GetCertificate: autoTLSManager.GetCertificate,
        NextProtos: []string{
            "h2", "http/1.1", // enable HTTP/2
            acme.ALPNProto, // enable tls-alpn ACME challenges
        },
    }
}

and don't forget to import on top

    "golang.org/x/crypto/acme"
    "golang.org/x/crypto/acme/autocert"

Important note:

When I created the TLS configuration I selected some CipherSuites and Elliptic Curves. I used the ones in the code since they are the recommended ones by

👉 mozilla.

In order to test that we need a domain and setup the Nameservers to point to our hosting server. In the next blog post we are going to deploy what we have so far in a real webserver. For now let's continue.

What is missing

Our server now does not gracefully exits when a SIGTERM is received.
We will leave it for now. It's good enough but we should revisit .

Let's finally make our server do something useful

We will finally add an HTTP handler that returns something super useful .

What's better from a HELLO WORLD that we can see in our browser 🚀

First install the echo framework.

Wait a second why Echo? Echo makes our life a little bit easier without being huge and does not go into our way a lot. Another very good choice is chi . Basically I don't know, there are so many frameworks and benchmarks out there just pick Echo.

In any case, the way we are going to code our application we should be able to switch frameworks without having to rewrite everything (but let's avoid that).

So, install echo:

go get github.com/labstack/echo/v4

and then create a file http/router.go

package http

import (
    "net/http"

    "github.com/labstack/echo/v4"
)

type Router struct {
    e *echo.Echo
}

func NewRouter() *Router {
    ans := Router{
        e: echo.New(),
    }

    ans.e.Debug = true

    baseHandler := baseHandler{
        e: ans.e,
    }

    handlers := []handler{
        &baseHandler,
    }

    for _, h := range handlers {
        h.RegisterRoutes()
    }

    return &ans
}

func (r *Router) Handler() http.Handler {
    return r.e
}

type handler interface {
    RegisterRoutes()
}

type baseHandler struct {
    e *echo.Echo
}

func (b *baseHandler) RegisterRoutes() {
    b.e.GET("/", func(c echo.Context) error {
        return c.String(http.StatusOK, "HELLO WORLD")
    })
}

Here we just created an echo.Echo instance and attached a function to run
when we visit the path / on the server .

If you follow the tutorial so far then visit:

make dev

and visit:

https://local.freelance-invoice-hub.com

https://localhost

🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥

Commit

As usual we should commit our branch and you can find it in :
the github branch

Summary and what's next

In today's blog we learned how to create a webserver in Go, how to attach a router (echo in our case). Additionally we learned how to utilize golang's features to obtain automatically TLS certificates using Let's Encrypt (I forgot to mention that Go takes care of certificate renewal automatically). We also created valid TSL certificates for local development.

So far in the series we are just building our foundation, kind of our own framework to ease and speedup our development procedure. Most of the code we wrote is the same for many web apps and we could extract some libraries out of it or a template that we use in other projects.

In the next blog post we are going to add Basic authentication and show you how you can deploy the application in a cheap VPS. We wil also configure github CI/CD when we merge to main branch.

That's all for today.

❓ If you have any questions or something is not running on your machine I am happy to help. Reach out via a comment and I will try to help

❤️ Please subscribe to my newsletter and follow me on X or LinkedIn .

Setting up a Docker development enviroment for Go

Georgios Komninos — Fri, 14 Jun 2024 05:00:08 GMT

Welcome back to our Golang web application series! If you're excited to learn web development with Golang, you're in the right place. In this guide, we'll set up a solid development environment. By the end, you'll have a well-organized project and be able to run your code locally with Docker Compose, complete with hot reloading for easy development. Let's get started!

Prerequisites

Ensure that you have:
- docker
- https://docs.docker.com/compose/
- golang

installed

Basic setup

Create a folder that will host your project and init a git repository

mkdir freelance-invoice-hub
cd freelance-invoice-hub
git init

Let's create a new git branch

git checkout -b project-skeleton

Now we need to create a go module:

go mod init invoicehub

Also let's create a .gitignore file and a README and LICENCE and Makefile .
Let's keep the empty for now

touch README.md
touch .gitignore
touch LICENCE
touch Makefile

Golang Project structure

We need to organize our project skeleton in a way that we can easily extend and test our code.
Below you can see a good starting point for your go projects.

Let's explain the project layout and what we will put in each folder

cmd: Here we will have our executable, the main function.
Right now it just prints hello world :)

http: In this package we are going to add all the code that is related with the HTTP server, the handlers etc. For now just create an empty file http.go

mocks: This folder will hold the mock implementation of our services. The mocks will be auto-generated. More on that later. for now just add a mocks.go which only declares that this is a go package.

sqlite: This folder will contain the implementation of our database layer. Since we are going to use sqlite, we name that package sqlite. Similarly just create a file just declaring the package name.

In the root folder we are going to create .go files that represent our domain entities and their operations. For now let's do nothing

Let's test that our main runs now:

go run cmd/main.go

This must print hello world .

If not please check the error message and your go installation.
In case you run into problems please reach to me either on Twitter or in a comment and I will assist.

Setting up Dockerfile and docker-compose

We want to be able to run our application in a docker container. Additionally, the docker container should reload when the code changes (hot reload).

Let's do that 🚀

touch dev.Dockerfile

and paste the following

FROM golang:1.22.1-bullseye

RUN apt-get update \
    && apt-get install -y ca-certificates curl gnupg \
    && mkdir -p /etc/apt/keyrings

WORKDIR /app

RUN go install go.uber.org/mock/mockgen@latest && \
    go install github.com/air-verse/air@latest


RUN git config --global --add safe.directory /app

CMD ["air", "-c", ".air.toml"]

Notes:

We install in the docker container 2 extra go packages:

mockgen: It will be used to generate mock implementation automatically from our interfaces.

air: This is a program that monitors your files for changes and restart the process. This is the tool that we are going to use for "hot" reloading the code.

Air requires a .air.toml file with it's configuration.
Create that file using touch .air.toml and paste the following:

# Config file for [Air](https://github.com/cosmtrek/air) in TOML format

# Working directory
# . or absolute path, please note that the directories following must be under root.
root = "."
tmp_dir = "tmp"

[build]
# Array of commands to run before each build
pre_cmd = ["go mod download"]
# Just plain old shell command. You could use `make` as well.
cmd = "go build -o ./tmp/freelance-invoice-hub cmd/main.go"
# Array of commands to run after ^C
#post_cmd = ["echo 'hello air' > post_cmd.txt"]
# Binary file yields from `cmd`.
bin = "tmp/freelance-invoice-hub"
# Customize binary, can setup environment variables when run your app.
#full_bin = "./tmp/main"
full_bin = "export $(grep -v '^#' dev.env | xargs);tmp/freelance-invoice-hub"
# Watch these filename extensions.
include_ext = ["go", "tpl", "tmpl", "html", "js", "css", "scss", "toml"]
# Ignore these filename extensions or directories.
exclude_dir = ["bin", "tmp", "vendor", "certs", "static", "uploads"]
# Watch these directories if you specified.
include_dir = []
# Watch these files.
include_file = []
# Exclude files.
exclude_file = []
# Exclude specific regular expressions.
exclude_regex = ["_test\\.go", "mock_.*\\.go", "gomock_.*\\.go"]
# Exclude unchanged files.
exclude_unchanged = true
# Follow symlink for directories
follow_symlink = true
# This log file places in your tmp_dir.
log = "air.log"
# Poll files for changes instead of using fsnotify.
poll = false
# Poll interval (defaults to the minimum interval of 500ms).
poll_interval = 500 # ms
# It's not necessary to trigger build each time file changes if it's too frequent.
delay = 1000 # ms
# Stop running old binary when build errors occur.
stop_on_error = true
# Send Interrupt signal before killing process (windows does not support this feature)
send_interrupt = true
# Delay after sending Interrupt signal
kill_delay = 500 # nanosecond
# Rerun binary or not
rerun = false
# Delay after each executions
rerun_delay = 500
# Add additional arguments when running binary (bin/full_bin). Will run './tmp/main hello world'.
#args_bin = ["hello", "world"]

[log]
# Show log time
time = false
# Only show main log (silences watcher, build, runner)
main_only = false

[color]
# Customize each part's color. If no color found, use the raw app log.
main = "magenta"
watcher = "cyan"
build = "yellow"
runner = "green"

[misc]
# Delete tmp directory on exit
clean_on_exit = true

[screen]
clear_on_rebuild = true
keep_scroll = true

Basically, you get that file and you modify it. We are not going to explain more here.
If you want to understand what it's changed from the original focus on the [Build] section. I just changed the paths more or less to match our project.

Finally add the tmp folder in your .gitignore file:

echo "tmp" >> .gitignore

Docker-compose

to run our project we are going to use docker compose.

create a file touch dev.docker-compose.yaml and paste the following:

services:
  app:
    build:
      context: .
      dockerfile: dev.Dockerfile
    env_file:
      - 'dev.env'
    ports:
      - '127.0.0.1:443:443'
      - '127.0.0.1:80:8080'
    extra_hosts:
        - "local.freelance-invoice-hub.com:127.0.0.1"
    volumes:
      - .:/app
      - freelance-invoice-hub_mod_cache:/go/pkg/mod

volumes:
  freelance-invoice-hub_mod_cache:

Finally create a file touch dev.env . We are going to add environment variables for development there later.

Let's test the setup

docker compose -f dev.docker-compose.yaml up

then it should start building the image and start the container.

you should see something like the below

[+] Running 1/0
 ✔ Container freelance-invoice-hub-app-1  Created                                                                                                                                                                                        0.0s
Attaching to app-1
app-1  |
app-1  |   __    _   ___
app-1  |  / /\  | | | |_)
app-1  | /_/--\ |_| |_| \_ v1.52.2, built with Go go1.22.1
app-1  |
app-1  | mkdir /app/tmp
app-1  | watching .
app-1  | watching cmd
app-1  | watching http
app-1  | watching mocks
app-1  | watching sqlite
app-1  | !exclude tmp
app-1  | > go mod download
app-1  | go: no module dependencies to download
app-1  | building...
app-1  | running...
app-1  | export GOLANG_VERSION='1.22.1'
app-1  | export GOPATH='/go'
app-1  | export GOTOOLCHAIN='local'
app-1  | export HOME='/root'
app-1  | export HOSTNAME='5a8638cf6ceb'
app-1  | export PATH='/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
app-1  | export PWD='/app'
app-1  | hello world
app-1  | Process Exit with Code 0

Notice the hello world in the end. This is what our program does.
Now let's test if it will reload when we do a change.

In cmd/main.go change the hello world string to hello hot reload .

Now the console that started the container automatically run the modified code:

         cmd/main.go has changed
app-1  | > go mod download
app-1  | go: no module dependencies to download
app-1  | building...
app-1  | running...
app-1  | export GOLANG_VERSION='1.22.1'
app-1  | export GOPATH='/go'
app-1  | export GOTOOLCHAIN='local'
app-1  | export HOME='/root'
app-1  | export HOSTNAME='5a8638cf6ceb'
app-1  | export PATH='/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
app-1  | export PWD='/app'
app-1  | hello hot reload

Makefile

Since it's tedious to write the docker compose -f dev.docker-compose.yaml up all the time
let's add a command for that in our Makefile

default: help

help: ## help information about make commands
    @grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'

dev: ## runs the application in development mode
    @docker compose -f dev.docker-compose.yaml up

Now run

make

and you should see:

And you can run your docker development container by using:

make dev

Commit

It's time for us to commit and merge to the main branch

git add .
git commit -m "Initial project setup"
git checkout main
git merge project-skeleton

Summary

In this blog post, we have successfully set up the basic skeleton for our Golang-based web application. We created a structured project layout, set up essential files, and configured Docker and Docker Compose for local development with hot reloading. With this foundation in place, we are now ready to start building and extending our application.

The code is accessible in the project-skeleton brach in github .

Stay tuned for the next part of the series, where we will dive deeper into implementing core functionalities. Happy coding!

💡

Please comment or ask on Twitter if you have issues or need extra help. Happy to help

💡

If you found this guide helpful, please consider subscribing to our newsletter for more tutorials and updates. And don't forget to share this article with others who might benefit from it!

Crafting a Web Application with Golang: A Step-by-Step Guide

Georgios Komninos — Thu, 13 Jun 2024 05:00:14 GMT

Introduction

This is the first part the series Build a Web App with Golang . I am going to show you how you can build a web application using Golang. In fact, we are going to build an application that I need.

I this blog post we will cover the following:

Scope, meaning we will define what we want to build

The goal of this blog post is to understand the problem and it's domain. This will help us to understand what we need to code.

I will try to keep each blog post in a separate git branch. I believe this will make it easier for you to follow the tutorial.

Scope

As a freelancer I want a web application in which I will login and I can create invoices for my clients.

I want to be able to view the invoices created and download a PDF version

Let's breakdown the above requirements one by one.

First, let's see what an invoice contains:

Freelancer's details which include:
Company Name
Company Address
Email
Tax Number
VAT Number
Bank Accounts
Client details which include:
Company Name
Company Address
Email
VAT number
- Invoice Details:
  Invoice Number
  Invoice Date
  Line Items
  
  Here a sample invoice:
  
  A Line Item, consists of:
  - Description
  - Disbursements
  - Fees
  
  A Bank Account, consists of:
  - Bank Name
  - Account No.
  - IBAN
  - BIC

We also need to have the VAT rate for our country and the amount of days that the invoice is payable within.

Let's now discuss the invoice number. This should be unique per invoice and in most of the cases it must be sequential. For my case the invoice number has the form for example 1030/24 which means that the invoice belongs to 2024 year and it has the number 1030. The next invoice should be 1031/24 and so on.
This generation should be automated.

Since this is an MVP and it should work only for one freelancer we will use HTTP Basic Authentication and the UI will be really minimal

Pages

Since we will build a web application let's define the pages that we want to build.
That's a draft but it will help us to get started.

- / : this is the homepage of the application. It should display a list of all the invoices created with pagination. Next to each invoice we want to have a button to view its details.
- /settings: Here we will display our company's details
-/settings/edit: A form to edit our company's details
- /clients: We will display a list of our clients
- /clients/id: It will display the details for the client with id id
- /clients/id/edit: a form to edit the client's details with id
- /clients/new: a form to add a new client
- /invoices/id/download: this will download the invoice with id id
- /invoices/id: we will view the details of an invoice
- /invoices/new: a form to create a new invoice

Technologies that we will use

We need to store our data somewhere. For simplicity and easy of deployment we are going to use the awesome SQLite DBMS.

This post is all about Golang so that's the back-end language. However we are going to use the Echo Framework and GORM to speedup development.

For the front-end I am not sure yet, most likely we are going to use Bootstrap . I am also pretty sure that we are going to use ChatGPT to write the basic HTML and CSS for us.

Conclusion

In the blog post we will dive into coding. We will create our git repository and setup some basics tooling that will save us a lot of time while we are developing.

I expect to release the next blog post of the tutorial every day.

If you have any questions or requests please write a comment.

Please subscribe to my newsletter so you get the updates.

Finally, just today I created an X Account and I would love to start having some followers.

Use context in your HTTP handlers

Georgios Komninos — Wed, 12 Jun 2024 06:09:32 GMT

Let's consider a simple webserver that runs a task. The task can be anything like a time consuming computation, a database query.

You need to be able to do two things:

when the client drops the connection while the task is running terminate the task so resources are released.
when the task takes too much time return a proper HTTP status code to the client

Let's see a practical example

package main

import (
    "context"
    "errors"
    "fmt"
    "math/rand"
    "net/http"
    "time"
)

func main() {
    mux := http.NewServeMux()

    // Register the handler with the middleware, setting the timeout to 5 seconds
    mux.HandleFunc("/", longRunningTaskHandler)

    const timeout = time.Second * 5
    wrappedMux := contextMiddleware(timeout)(mux)

    // Start the server
    http.ListenAndServe(":8080", wrappedMux)
}

// contextMiddleware is a middleware that sets a timeout for the request context.
// If the request takes longer than the timeout, the middleware will cancel the context.
func contextMiddleware(timeout time.Duration) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            // Set a timeout for the request context
            ctx, cancel := context.WithTimeout(r.Context(), timeout)
            defer cancel()

            // Create a new request with the updated context
            r = r.WithContext(ctx)

            // Call the next handler with the new context
            next.ServeHTTP(w, r)
        })
    }
}

func longRunningTaskHandler(w http.ResponseWriter, r *http.Request) {
    err := longRunningTask(r.Context())
    if err != nil {
        // if the context expired, return a 504 Gateway Timeout
        if errors.Is(err, context.DeadlineExceeded) {
            w.WriteHeader(http.StatusGatewayTimeout)
            return
        }

        // if the task failed for some other reason, return a 500 Internal Server Error
        w.WriteHeader(http.StatusInternalServerError)

        return
    }

    // if the task completed successfully, return a 200 OK
    w.WriteHeader(http.StatusOK)
}

// longRunningTask is a dummy function that simulates a long running task.
// the task will take 10 seconds to complete.
// If the context is cancelled before the task completes, it will return the error.
func longRunningTask(ctx context.Context) error {
    var dur time.Duration

    // for ~50% of the time, the task will take 10 seconds to complete
    if rand.Float64() > 0.5 {
        dur = 10 * time.Second
    } else {
        dur = 1 * time.Second // and for the other ~50% of the time, the task will take 1 second to complete
    }

    // simulate the task by sleeping for the duration
    select {
    case <-time.After(dur):
        return nil
    case <-ctx.Done():
        fmt.Println("task cancelled")
        return ctx.Err()
    }
}

Run the webserver:

go run main.go

and in another terminal:

curl -i http://localhost:8080

Curl command will return at most in 5 seconds, even though the task can take up to 10 seconds.

See some runs:

➜  ~ curl -i http://localhost:8080/
HTTP/1.1 200 OK
Date: Wed, 12 Jun 2024 05:52:18 GMT
Content-Length: 0

➜  ~ curl -i http://localhost:8080/
HTTP/1.1 504 Gateway Timeout
Date: Wed, 12 Jun 2024 05:52:24 GMT
Content-Length: 0

➜  ~ curl -i http://localhost:8080/
HTTP/1.1 200 OK
Date: Wed, 12 Jun 2024 05:52:26 GMT
Content-Length: 0

As you notice in the second request the server returned 504.

Please run this in your computer and hit CTL-C when the request is running.
Notice the output of the webserver and you will see a message like:

task cancelled

See the related github repo

Unveiling a Powerful Google Maps Scraping Tool

Georgios Komninos — Sun, 03 Dec 2023 11:39:14 GMT

Introduction

In a world where data is increasingly crucial, having the right tools to gather information is essential. This introduces a command-line Google Maps scraper, developed using the Scrapemate web crawling framework. Ideal for business analysts, data scientists, or anyone in need of detailed location data, this tool offers an efficient way to extract valuable information from Google Maps.

Try it now: https://s.gkomninos.com/s/Ww6eBR

What is the Google Maps Scraper?

The Google Maps scraper is an open source command-line tool designed to extract a wide array of data points from Google Maps. Users have the flexibility to utilize the tool as is or customize its code to fit specific needs.

Key Features

Data Extraction: Extracts numerous data points from Google Maps.
Export Options: Allows exporting data in CSV, JSON, or directly to PostgreSQL.
Performance: Extracts approximately data from 55 places per minute.
Customization: Extendable for writing custom exporters or code modifications.
Cross-Platform: Dockerized for compatibility across various platforms.
Scalability: Suitable for scaling across multiple machines.
Email Extraction: Optionally extracts emails from business websites.

Email Extraction

The email extraction feature is disabled by default to optimize performance. When enabled, the scraper visits the website of a listed business and tries to extract emails from the page, primarily from the main page registered on Google Maps. Future updates aim to include extraction from additional pages such as 'About Us' and 'Contact'.

Extracted Data Points

The tool is capable of extracting a comprehensive list of data points including:

Link, title, category, address
Open hours, popular times, website, phone
Plus code, review count, review rating
Latitude, longitude, and much more

Disclaimer

Please use this program responsibly. It is important to adhere to Google Maps' terms of service and local laws regarding data scraping and privacy. The tool is provided for legitimate purposes and should be used with ethical considerations in mind.

Conclusion

This Google Maps scraper tool offers an easy and efficient way to access detailed location data. Whether for professional data analysis or personal use, this tool is a valuable resource for anyone needing comprehensive data from Google Maps.

PostgreSQL window functions crash course

Georgios Komninos — Sat, 20 May 2023 10:24:50 GMT

Introduction

According to PostgreSQL documentation

Window functions provide the ability to perform calculations across sets of rows that are related to the current query row.

Window functions in PostgreSQL are a powerful tool that can significantly enhance your data analysis capabilities. They can perform calculations across a set of related rows, in a way traditional SQL queries can't. In this blog post, I'll walk you through the concept of window functions, their syntax, and several examples to better understand their practical use cases.

The primary goal of this post is that the reader after reading the post and trying all the examples will understand how to use PostgreSQL window functions.

Please also use the accompanying GitHub repository.

To follow the crash course I highly encourage you to try the examples.

Clone the github repository

First, clone the repository

got clone https://github.com/gosom/postgres-window-functions.git

Then enter the folder

cd postgres-window-functions

A docker-compose.yaml file is provided that will spin a PostgreSQL docker container that includes the example tables and the data.

To spin the container: docker-compose up -d

You can connect to the database from the command line using:

make psql

Alternatively, use your favorite PostgreSQL client using the below connection details.

Hostname: localhost
Port: 5432
Username: postgres
Password: postgres
Database: postgres

The Dataset

We'll use a hypothetical dataset from a library system, specifically a read_log table, which tracks the reading habits of various individuals:

CREATE TABLE read_log (
    id INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    reader_name TEXT,
    book_title TEXT,
    pages INT,
    read_date DATE
);

id: is an auto-increment integer which is unique per row
reader_name: is the name of the reader
book_title: is the title of the book
read_date: is the date that the reader finished the book

Now, let's populate our read_log table with sample data:

INSERT INTO read_log (reader_name, book_title, pages, read_date) VALUES 
('Giorgos', 'The Iliad', 711, '2023-01-01'),
('Giorgos', 'The Odyssey', 800, '2023-02-15'),
('Giorgos', 'The Republic', 400, '2023-03-30'),
('Emmanouela', 'Antigone', 150, '2023-01-20'),
('Emmanouela', 'Oedipus Rex', 100, '2023-02-26'),
('Emmanouela', 'The Symposium', 200, '2023-04-10'),
('Eleni', 'The Histories', 900, '2023-02-01'),
('Eleni', 'Works and Days', 150, '2023-03-15'),
('Eleni', 'Prometheus Bound', 80, '2023-04-25'),
('Eleni', 'Metamorphoses', 480, '2023-02-15'),
('Konstantina', 'The Iliad', 711, '2023-01-22'),
('Konstantina', 'The Odyssey', 800, '2023-03-18'),
('Konstantina', 'The Symposium', 200, '2023-04-30'),
('Andreas', 'Antigone', 150, '2023-01-30'),
('Andreas', 'Oedipus Rex', 100, '2023-02-20'),
('Andreas', 'The Republic', 400, '2023-04-18'),
('Andreas', 'Metamorphoses', 480, '2023-05-08');

Understanding Window Functions

A window function performs a calculation across a set of table rows that are related to the current row. It's like an advanced version of an aggregation function (like SUM(), COUNT(), etc.), but instead of collapsing all the rows into a single output row, it maintains the separate rows.

Syntax

window_function (expression) OVER (
    [PARTITION BY partition_expression]
    [ORDER BY sort_expression]
    [frame_clause]
)

Let's break this down:

window function (expression) : this part is similar to regular SQL function calls. The expression is the subject of the calculation. For example, if we are using the SUM() window function, the expression would be the column we want to sum.
OVER(...): A window function will always include an OVER clause. It defines the 'window' or set of rows the function operates on.
PARTITION BY partition_expression: This is used to break the data into smaller partitions. The window function is applied within each of these partitions. If it's not specified, the window function treats all rows of the result set as a single partition.
ORDER BY sort_expression: This clause determines the order in which rows are processed by the window function. Rows are ordered according to this expression before the function is applied.
frame_clause: This clause further refines the window within a partition that the window function operates on.
This is a more complex part, that I will cover later. For now, focus understanding the basic syntax and capabilities of window functions.

Let's see the syntax in action:

Let's suppose that we want to get all the rows from the read_log table but we additionally want in each row the tp, which is the total number of pages for all readers.

Without window functions:

One way to do it:

WITH cte AS (
    SELECT 
        sum(pages) as tp 
    FROM 
        read_log
)
SELECT 
    read_log.*, cte.tp 
FROM 
    read_log, cte;

Result set:

  id | reader_name |    book_title    | pages | read_date  |  tp  
----+-------------+------------------+-------+------------+------
  1 | Giorgos     | The Iliad        |   711 | 2023-01-01 | 6812
  2 | Giorgos     | The Odyssey      |   800 | 2023-02-15 | 6812
  3 | Giorgos     | The Republic     |   400 | 2023-03-30 | 6812
  4 | Emmanouela  | Antigone         |   150 | 2023-01-20 | 6812
  5 | Emmanouela  | Oedipus Rex      |   100 | 2023-02-26 | 6812
  6 | Emmanouela  | The Symposium    |   200 | 2023-04-10 | 6812
  7 | Eleni       | The Histories    |   900 | 2023-02-01 | 6812
  8 | Eleni       | Works and Days   |   150 | 2023-03-15 | 6812
  9 | Eleni       | Prometheus Bound |    80 | 2023-04-25 | 6812
 10 | Eleni       | Metamorphoses    |   480 | 2023-02-15 | 6812
 11 | Konstantina | The Iliad        |   711 | 2023-01-22 | 6812
 12 | Konstantina | The Odyssey      |   800 | 2023-03-18 | 6812
 13 | Konstantina | The Symposium    |   200 | 2023-04-30 | 6812
 14 | Andreas     | Antigone         |   150 | 2023-01-30 | 6812
 15 | Andreas     | Oedipus Rex      |   100 | 2023-02-20 | 6812
 16 | Andreas     | The Republic     |   400 | 2023-04-18 | 6812
 17 | Andreas     | Metamorphoses    |   480 | 2023-05-08 | 6812

Using Window Function

SELECT 
    *, sum(tp) OVER() 
FROM 
    read_log;

Notice, here that we are using an empty OVER clause, we want our window function to operate in all the rows.

Result set:

 id | reader_name |    book_title    | pages | read_date  | tp  
----+-------------+------------------+-------+------------+------
  1 | Giorgos     | The Iliad        |   711 | 2023-01-01 | 6812
  2 | Giorgos     | The Odyssey      |   800 | 2023-02-15 | 6812
  3 | Giorgos     | The Republic     |   400 | 2023-03-30 | 6812
  4 | Emmanouela  | Antigone         |   150 | 2023-01-20 | 6812
  5 | Emmaounela  | Oedipus Rex      |   100 | 2023-02-26 | 6812
  6 | Emmanouela  | The Symposium    |   200 | 2023-04-10 | 6812
  7 | Eleni       | The Histories    |   900 | 2023-02-01 | 6812
  8 | Eleni       | Works and Days   |   150 | 2023-03-15 | 6812
  9 | Eleni       | Prometheus Bound |    80 | 2023-04-25 | 6812
 10 | Eleni       | Metamorphoses    |   480 | 2023-02-15 | 6812
 11 | Konstantina | The Iliad        |   711 | 2023-01-22 | 6812
 12 | Konstantina | The Odyssey      |   800 | 2023-03-18 | 6812
 13 | Konstantina | The Symposium    |   200 | 2023-04-30 | 6812
 14 | Andreas     | Antigone         |   150 | 2023-01-30 | 6812
 15 | Andreas     | Oedipus Rex      |   100 | 2023-02-20 | 6812
 16 | Andreas     | The Republic     |   400 | 2023-04-18 | 6812
 17 | Andreas     | Metamorphoses    |   480 | 2023-05-08 | 6812

Our requirement has changed and now we need the tp to be the total pages by each reader_name

Without Window functions

WITH cte AS (
    SELECT 
    reader_name, sum(pages) AS tp 
    FROM 
        read_log 
    GROUP BY 
        reader_name
)
SELECT 
    read_log.*, cte.tp 
FROM 
    read_log, cte 
WHERE 
    read_log.reader_name = cte.reader_name;

Result Set:

 id | reader_name |    book_title    | pages | read_date  |  tp  
----+-------------+------------------+-------+------------+------
  1 | Giorgos     | The Iliad        |   711 | 2023-01-01 | 1911
  2 | Giorgos     | The Odyssey      |   800 | 2023-02-15 | 1911
  3 | Giorgos     | The Republic     |   400 | 2023-03-30 | 1911
  4 | Emmanouela  | Antigone         |   150 | 2023-01-20 |  450
  5 | Emmanouela  | Oedipus Rex      |   100 | 2023-02-26 |  450
  6 | Emmanouela  | The Symposium    |   200 | 2023-04-10 |  450
  7 | Eleni       | The Histories    |   900 | 2023-02-01 | 1610
  8 | Eleni       | Works and Days   |   150 | 2023-03-15 | 1610
  9 | Eleni       | Prometheus Bound |    80 | 2023-04-25 | 1610
 10 | Eleni       | Metamorphoses    |   480 | 2023-02-15 | 1610
 11 | Konstantina | The Iliad        |   711 | 2023-01-22 | 1711
 12 | Konstantina | The Odyssey      |   800 | 2023-03-18 | 1711
 13 | Konstantina | The Symposium    |   200 | 2023-04-30 | 1711
 14 | Andreas     | Antigone         |   150 | 2023-01-30 | 1130
 15 | Andreas     | Oedipus Rex      |   100 | 2023-02-20 | 1130
 16 | Andreas     | The Republic     |   400 | 2023-04-18 | 1130
 17 | Andreas     | Metamorphoses    |   480 | 2023-05-08 | 1130

With window functions

SELECT
    *, sum(pages) OVER(PARTITION BY reader_name) AS tp
FROM
    read_log;

Notice, here that we are using PARTITION BY in the OVER clause, since we want to calculate the SUM of pages per reader_name.

Result Set:

 id | reader_name |    book_title    | pages | read_date  |  tp  
----+-------------+------------------+-------+------------+------
 14 | Andreas     | Antigone         |   150 | 2023-01-30 | 1130
 17 | Andreas     | Metamorphoses    |   480 | 2023-05-08 | 1130
 16 | Andreas     | The Republic     |   400 | 2023-04-18 | 1130
 15 | Andreas     | Oedipus Rex      |   100 | 2023-02-20 | 1130
  9 | Eleni       | Prometheus Bound |    80 | 2023-04-25 | 1610
  7 | Eleni       | The Histories    |   900 | 2023-02-01 | 1610
  8 | Eleni       | Works and Days   |   150 | 2023-03-15 | 1610
 10 | Eleni       | Metamorphoses    |   480 | 2023-02-15 | 1610
  6 | Emmanouela  | The Symposium    |   200 | 2023-04-10 |  450
  5 | Emmanouela  | Oedipus Rex      |   100 | 2023-02-26 |  450
  4 | Emmanouela  | Antigone         |   150 | 2023-01-20 |  450
  1 | Giorgos     | The Iliad        |   711 | 2023-01-01 | 1911
  2 | Giorgos     | The Odyssey      |   800 | 2023-02-15 | 1911
  3 | Giorgos     | The Republic     |   400 | 2023-03-30 | 1911
 12 | Konstantina | The Odyssey      |   800 | 2023-03-18 | 1711
 11 | Konstantina | The Iliad        |   711 | 2023-01-22 | 1711
 13 | Konstantina | The Symposium    |   200 | 2023-04-30 | 1711

Note: Notice that the results are in different order. This is not an issue since we didn't have any order requirements.

Let's now change our requirement and we want the running total of pages read by each reader, ordered by the date when they finished each book.
So, for example for reader Andreas we need an output like:

14 | Andreas | Antigone | 150 | 2023-01-30 | 150 
15 | Andreas | Oedipus Rex | 100 | 2023-02-20 | 250 
16 | Andreas | The Republic | 400 | 2023-04-18 | 650 
17 | Andreas | Metamorphoses | 480 | 2023-05-08 | 1130

Without window functions this is not so trivial:

WITH A AS (                                      
    SELECT                                                                   
        read_date,
        reader_name,
        SUM(pages) AS tp
    FROM(
        SELECT
            t1.read_date,
            t1.reader_name,
            t2.pages
        FROM 
            read_log t1
        JOIN 
            read_log t2
        ON 
            t1.reader_name = t2.reader_name
        AND t1.read_date >= t2.read_date
    ) t3
    GROUP BY
        read_date,
        reader_name
)
SELECT 
    read_log.*,
    A.tp
FROM 
    read_log
JOIN A ON 
    read_log.read_date = A.read_date
    AND read_log.reader_name = A.reader_name
;

Result Set

 id | reader_name |    book_title    | pages | read_date  |  tp  
----+-------------+------------------+-------+------------+------
  1 | Giorgos     | The Iliad        |   711 | 2023-01-01 |  711
  2 | Giorgos     | The Odyssey      |   800 | 2023-02-15 | 1511
  3 | Giorgos     | The Republic     |   400 | 2023-03-30 | 1911
  4 | Emmanouela  | Antigone         |   150 | 2023-01-20 |  150
  5 | Emmanouela  | Oedipus Rex      |   100 | 2023-02-26 |  250
  6 | Emmanouela  | The Symposium    |   200 | 2023-04-10 |  450
  7 | Eleni       | The Histories    |   900 | 2023-02-01 |  900
  8 | Eleni       | Works and Days   |   150 | 2023-03-15 | 1530
  9 | Eleni       | Prometheus Bound |    80 | 2023-04-25 | 1610
 10 | Eleni       | Metamorphoses    |   480 | 2023-02-15 | 1380
 11 | Konstantina | The Iliad        |   711 | 2023-01-22 |  711
 12 | Konstantina | The Odyssey      |   800 | 2023-03-18 | 1511
 13 | Konstantina | The Symposium    |   200 | 2023-04-30 | 1711
 14 | Andreas     | Antigone         |   150 | 2023-01-30 |  150
 15 | Andreas     | Oedipus Rex      |   100 | 2023-02-20 |  250
 16 | Andreas     | The Republic     |   400 | 2023-04-18 |  650
 17 | Andreas     | Metamorphoses    |   480 | 2023-05-08 | 1130

Now using Window Function

SELECT 
    *, sum(pages) OVER (PARTITION BY reader_name ORDER BY read_date) AS tp
FROM
    read_log
;

Result Set

 id | reader_name |    book_title    | pages | read_date  |  tp  
----+-------------+------------------+-------+------------+------
 14 | Andreas     | Antigone         |   150 | 2023-01-30 |  150
 15 | Andreas     | Oedipus Rex      |   100 | 2023-02-20 |  250
 16 | Andreas     | The Republic     |   400 | 2023-04-18 |  650
 17 | Andreas     | Metamorphoses    |   480 | 2023-05-08 | 1130
  7 | Eleni       | The Histories    |   900 | 2023-02-01 |  900
 10 | Eleni       | Metamorphoses    |   480 | 2023-02-15 | 1380
  8 | Eleni       | Works and Days   |   150 | 2023-03-15 | 1530
  9 | Eleni       | Prometheus Bound |    80 | 2023-04-25 | 1610
  4 | Emmanouela  | Antigone         |   150 | 2023-01-20 |  150
  5 | Emmanouela  | Oedipus Rex      |   100 | 2023-02-26 |  250
  6 | Emmanouela  | The Symposium    |   200 | 2023-04-10 |  450
  1 | Giorgos     | The Iliad        |   711 | 2023-01-01 |  711
  2 | Giorgos     | The Odyssey      |   800 | 2023-02-15 | 1511
  3 | Giorgos     | The Republic     |   400 | 2023-03-30 | 1911
 11 | Konstantina | The Iliad        |   711 | 2023-01-22 |  711
 12 | Konstantina | The Odyssey      |   800 | 2023-03-18 | 1511
 13 | Konstantina | The Symposium    |   200 | 2023-04-30 | 1711

Please pay attention how the window function behaves differently comparing to the behavior without the ORDER BY in the previous example.
Rows are ordered according to this expression before the function is applied.

Now we want to know the moving average of pages each user reads.

Example output for reader_name Giorgos

Without window functions

WITH A AS (                                      
    SELECT                                                                   
        read_date,
        reader_name,
        AVG(pages) AS moving_avg
    FROM(
        SELECT
            t1.read_date,
            t1.reader_name,
            t2.pages
        FROM 
            read_log t1
        JOIN 
            read_log t2
        ON 
            t1.reader_name = t2.reader_name
        AND t1.read_date >= t2.read_date
    ) t3
    GROUP BY
        read_date,
        reader_name
)
SELECT 
    read_log.reader_name,
    read_log.read_date,
    A.moving_avg
FROM 
    read_log
JOIN A ON 
    read_log.read_date = A.read_date
    AND read_log.reader_name = A.reader_name
;

Result set:

 reader_name | read_date  |      moving_avg      
-------------+------------+----------------------
 Giorgos     | 2023-01-01 | 711.0000000000000000
 Giorgos     | 2023-02-15 | 755.5000000000000000
 Giorgos     | 2023-03-30 | 637.0000000000000000
 Emmanouela  | 2023-01-20 | 150.0000000000000000
 Emmanouela  | 2023-02-26 | 125.0000000000000000
 Emmanouela  | 2023-04-10 | 150.0000000000000000
 Eleni       | 2023-02-01 | 900.0000000000000000
 Eleni       | 2023-03-15 | 510.0000000000000000
 Eleni       | 2023-04-25 | 402.5000000000000000
 Eleni       | 2023-02-15 | 690.0000000000000000
 Konstantina | 2023-01-22 | 711.0000000000000000
 Konstantina | 2023-03-18 | 755.5000000000000000
 Konstantina | 2023-04-30 | 570.3333333333333333
 Andreas     | 2023-01-30 | 150.0000000000000000
 Andreas     | 2023-02-20 | 125.0000000000000000
 Andreas     | 2023-04-18 | 216.6666666666666667
 Andreas     | 2023-05-08 | 282.5000000000000000

Let's find the average pages read by each reader, considering only the current book and one book prior, so for reader Giorgos the first value should be 711, the second (711+800)/2 = 755, the third (800+400)/2=600

Without window functions

I cannot think of an obvious way to do that. I am pretty sure that this is possible although.

Using window functions

If you see the window functions syntax there is the frame_clause. It can be very helpful when you need more control over the rows that are being considered by the window function. We are going to utilize that here.

SELECT 
    reader_name, 
    read_date,
    AVG(pages) OVER(PARTITION BY reader_name ORDER BY read_date ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) as moving_avg
FROM 
    read_log
;

Notice now that we use a frame_clause .

ROWS BETWEEN 1 PRECEDING AND CURRENT ROW

which further refines the 'window' that the window functions operate on.
In our case, it controls the number of rows included in the calculation.

A frame_clause can be RANGE, ROWS, GROUPS . Here we used ROWS . I am not going to explain further the frame_clause.
You can search online or read the documentation for further examples.

Result Set

 reader_name | read_date  |      moving_avg      
-------------+------------+----------------------
 Andreas     | 2023-01-30 | 150.0000000000000000
 Andreas     | 2023-02-20 | 125.0000000000000000
 Andreas     | 2023-04-18 | 250.0000000000000000
 Andreas     | 2023-05-08 | 440.0000000000000000
 Eleni       | 2023-02-01 | 900.0000000000000000
 Eleni       | 2023-02-15 | 690.0000000000000000
 Eleni       | 2023-03-15 | 315.0000000000000000
 Eleni       | 2023-04-25 | 115.0000000000000000
 Emmanouela  | 2023-01-20 | 150.0000000000000000
 Emmanouela  | 2023-02-26 | 125.0000000000000000
 Emmanouela  | 2023-04-10 | 150.0000000000000000
 Giorgos     | 2023-01-01 | 711.0000000000000000
 Giorgos     | 2023-02-15 | 755.5000000000000000
 Giorgos     | 2023-03-30 | 600.0000000000000000
 Konstantina | 2023-01-22 | 711.0000000000000000
 Konstantina | 2023-03-18 | 755.5000000000000000
 Konstantina | 2023-04-30 | 500.0000000000000000

Window Functions in Action

Now that we understand the syntax and concept of window functions, let's take a look at some examples using different window functions on our read_log table.

Ranking Functions

Ranking functions are a category of functions that assign a unique rank to each row within the window partition. This is is useful when you want to find top-N or bottom-N rows.

RANK()

Rank() function assigns a unique rank to each distinct row.

We want to rank the readers by the total number of pages they have read:

SELECT 
    reader_name, 
    SUM(pages) as total_pages, 
    RANK() OVER(ORDER BY SUM(pages) DESC) as rank
FROM 
    read_log
GROUP BY 
reader_name
;

Result Set:

 reader_name | total_pages | rank 
-------------+-------------+------
 Giorgos     |        1911 |    1
 Konstantina |        1711 |    2
 Eleni       |        1610 |    3
 Andreas     |        1130 |    4
 Emmanouela  |         450 |    5

Now I would like you to insert this into our database

INSERT INTO read_log
(reader_name, book_title, pages, read_date) 
VALUES
('Konstantina', 'The Orestia', 200, '2023-05-15');

We would like readers Giorgos and Konstatina to have read the same amount of pages.

Run the query again:

SELECT 
    reader_name, 
    SUM(pages) as total_pages, 
    RANK() OVER(ORDER BY SUM(pages) DESC) as rank
FROM 
    read_log
GROUP BY 
reader_name
;

Notice the result now:

 reader_name | total_pages | rank 
-------------+-------------+------
 Giorgos     |        1911 |    1
 Konstantina |        1911 |    1
 Eleni       |        1610 |    3
 Andreas     |        1130 |    4
 Emmanouela  |         450 |    5

Giorgos and Konstantina have the same rank but there is gap in rank column (it goes from 1 to 3).

DENSE_RANK()

DENSE_RANK() works similarly to RANK() , but it doesn't leave gaps between groups of duplicate values.

SELECT 
    reader_name, 
    SUM(pages) as total_pages, 
    DENSE_RANK() OVER(ORDER BY SUM(pages) DESC) as rank
FROM 
    read_log
GROUP BY 
reader_name
;

Result:

 reader_name | total_pages | rank 
-------------+-------------+------
 Giorgos     |        1911 |    1
 Konstantina |        1911 |    1
 Eleni       |        1610 |    2
 Andreas     |        1130 |    3
 Emmanouela  |         450 |    4

Notice that now there is no gap and reader Eleni has rank 2.

ROW_NUMBER()

ROW_NUMBER() assigns a unique number to each row in the results. This function is useful when you need a unique value for each row belonging to the partition.

SELECT 
    reader_name, 
    book_title, 
    read_date,
    ROW_NUMBER() OVER(PARTITION BY reader_name ORDER BY read_date) as read_order
FROM read_log;

This query will give a unique row number to each book read by the same reader, ordered by the date they finished each book.

Aggregate Functions

I have already shown you a couple of examples of aggregate functions SUM() and AVG() .

SUM()

SUM() can be useful to calculate running totals.

AVG()

AVG() can be useful to calculate moving averages.

You can use any builtin aggregate function as a window function.

General Purpose Window Functions

FIRST VALUE() and LAST_VALUE()

FIRST_VALUE() and LAST_VALUE() are window functions that return the first and last value of an ordered set of values. Let's see which book each reader started and ended with:

SELECT 
    DISTINCT ON (reader_name) reader_name, 
    FIRST_VALUE(book_title) OVER(PARTITION BY reader_name ORDER BY read_date) as first_book,
    LAST_VALUE(book_title) OVER(PARTITION BY reader_name 
                                ORDER BY read_date 
                                ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) as last_book

Note from the docs:

Note that first_value, last_value, and nth_value consider only the rows within the “window frame”, which by default contains the rows from the start of the partition through the last peer of the current row. This is likely to give unhelpful results for last_value and sometimes also nth_value. You can redefine the frame by adding a suitable frame specification (RANGE, ROWS or GROUPS) to the OVER clause. See Section 4.2.8 for more information about frame specifications.

...

To obtain aggregation over the whole partition, omit ORDER BY or use ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING.

The ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING clause ensures that the window frame includes all rows in the partition.

Based on the docs we can also write the query like:

SELECT 
    DISTINCT ON (reader_name) reader_name, 
    FIRST_VALUE(book_title) OVER(PARTITION BY reader_name ORDER BY read_date) as first_book,
    LAST_VALUE(book_title) OVER(PARTITION BY reader_name )
FROM read_log;

Result:

 reader_name |  first_book   |    last_book     
-------------+---------------+------------------
 Andreas     | Antigone      | Metamorphoses
 Eleni       | The Histories | Prometheus Bound
 Emmanouela  | Antigone      | The Symposium
 Giorgos     | The Iliad     | The Republic
 Konstantina | The Iliad     | The Orestia

LAG() and LEAD()

LAG() and LEAD() functions allow you to fetch the value of a previous row or following row within your data partition.

For example, you can use LAG() to get the book each reader read before the current one:

SELECT 
    reader_name, 
    book_title, 
    LAG(book_title) OVER(PARTITION BY reader_name ORDER BY read_date) as previous_book
FROM 
read_log;

Result:

 reader_name |    book_title    | previous_book  
-------------+------------------+----------------
 Andreas     | Antigone         | 
 Andreas     | Oedipus Rex      | Antigone
 Andreas     | The Republic     | Oedipus Rex
 Andreas     | Metamorphoses    | The Republic
 Eleni       | The Histories    | 
 Eleni       | Metamorphoses    | The Histories
 Eleni       | Works and Days   | Metamorphoses
 Eleni       | Prometheus Bound | Works and Days
 Emmanouela  | Antigone         | 
 Emmanouela  | Oedipus Rex      | Antigone
 Emmanouela  | The Symposium    | Oedipus Rex
 Giorgos     | The Iliad        | 
 Giorgos     | The Odyssey      | The Iliad
 Giorgos     | The Republic     | The Odyssey
 Konstantina | The Iliad        | 
 Konstantina | The Odyssey      | The Iliad
 Konstantina | The Symposium    | The Odyssey
 Konstantina | The Orestia      | The Symposium

You can find a list of all the builtin window functions in the docs

Conclusion

In this blog post, we covered the basics of PostgreSQL window functions. I showed you the basic syntax of window functions and how you can use them to rank data, keep running totals, find averages and check data around each row

In order to master window functions the key is practice. Get the test dataset or create your dataset, look up the documentation and try them with different examples.

References

Distributed Google Maps scraping

Georgios Komninos — Sun, 14 May 2023 15:10:45 GMT

Introduction

In this post, I will show you how you can utilize the power of Kubernetes to scrape data from Google Maps without using an API key.

for the tutorial, I will use as an example deploying to

But this will work in any managed Kubernetes provider.

The whole procedure to get the scraper up and running won't take more than 20 minutes. So give it a try.

Prerequisites

Create a Digital Ocean Account . I recommend if you do not have an account to create it via the referral link . This way you get 200$ of credit and I may also get 25$ (depending if you continue using Digital Ocean). This way you can try the tutorial for Free .
Note: To get the 200$ credit you need to add a payment method.
Install kubectl in your local machine. Follow the official instructions .

Create a K8s Cluster

In the menu that popups select: Kubernetes

After clicking Kubernetes the Kubernetes page opens:

For the purposes of the tutorial leave the defaults.
In a real life scenario you need to pick the desired region and configure the nodes you like.

Don't change the defaults for now. If you registered in Digital Ocean via the referral link don't worry about costs for now. Additionally, keep in mind that since we are going to start headless web browser we need memory and CPU.

Please wait until the cluster initializes. This can take around 5 minutes.

Once the cluster is provisioned then you have to download the kubernetes configuration file:

Download the configuration file and take note of the location. For the purposes of the tutorial we assume that it is located at `/home/giorgos/k8s.config.yaml

Let's check that we can connect:

kubectl --kubeconfig=$HOME/k8s.config.yaml get pods && echo $?

You should get output like:

No resources found in default namespace.
0

Create a PostgreSQL database

In your Digital Ocean dashboard click on the left panel Databases or follow this create a database in Digital Ocean .

Select PostgresSQL database and in the next page leave the defaults (the lower tier).

Then click Create:

Again wait a bit until it is provisioned.

Once the database is ready we need to:

Create a User and a database

Create a User and a database

First, open a terminal (or your favorite GUI tool) and connect to your database

psql -p 25060 -h db-postgresql-sfo3-81615-do-user-14100026-0.b.db.ondigitalocean.com -U doadmin -d defaultdb

(Please replace host with yours)

If you managed to connect then we can move to the next step.

Create tables

    CREATE TABLE gmaps_jobs(
        id UUID PRIMARY KEY,
        priority SMALLINT NOT NULL,
        payload_type TEXT NOT NULL,
        payload BYTEA NOT NULL,
        created_at TIMESTAMP WITH TIME ZONE NOT NULL,
        status TEXT NOT NULL
    );

    CREATE TABLE results(
        id INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
        title TEXT NOT NULL,
        category TEXT NOT NULL,
        address TEXT NOT NULL,
        openhours TEXT NOT NULL,
        website TEXT NOT NULL,
        phone TEXT NOT NULL,
        pluscode TEXT  NOT NULL,
        review_count INT NOT NULL,
        rating NUMERIC NOT NULL
    );

Execute the above queries in your database client.

Google maps scraper deployment

First create a file with your queries. A sample is

bars in Athens
bars in Berlin
restaurants in Rome

Save this file in a file name queries.txt.

then:

docker run -v $PWD/queries.txt:/queries.txt gosom/google-maps-scraper:v0.9.3  -depth 5 -input /queries.txt  -dsn "postgres://doadmin:{yourPassword}@{yourHost}:25060/defaultdb" -produce -lang en

(Replace with your password and your host)

be patient because the image is around 1GB so it needs to be downloaded

Once, the command finishes verify that the jobs are inserted to the database:

select count(1) from gmaps_jobs

Run the above query in your database client. It should return 3 if you use my sample file.

We are now ready to start our scrapers.

Create a file with the kubernetes deployment configuration named gmaps.deployment.yaml and paste the following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: google-maps-scraper
spec:
  selector:
    matchLabels:
      app: google-maps-scraper
  replicas: 2
  template:
    metadata:
      labels:
        app: google-maps-scraper
    spec:
      containers:
      - name: google-maps-scraper
        image: gosom/google-maps-scraper:v0.9.3
        imagePullPolicy: IfNotPresent
        args: ["-c", "1", "-depth", "5", "-dsn", "postgres://doadmin:{YourPassword}@{YourHost}:25060/defaultdb"]

(Edit your password and your host)

Then apply the configuration:

kubectl --kubeconfig=$HOME/k8s.config.yaml apply -f gmaps.deployment.yaml

Give it some time, since the image needs to also get downloaded.

Check the status of the pods:

giorgos@gtp:~$ kubectl --kubeconfig=$HOME/k8s.config.yaml get pods
NAME                                   READY   STATUS    RESTARTS   AGE
google-maps-scraper-6489d96b84-7nltl   1/1     Running   0          68s
google-maps-scraper-6489d96b84-vvx6c   1/1     Running   0          116s
giorgos@gtp:~$

Meanwhile, check periodically the results table:

 select count(1) from results;

it will start slowly populating the results table.

defaultdb=> select * from results limit 5;
-[ RECORD 1 ]+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id           | 1
title        | Athens Sports Bar
category     | Sports bar
address      | Veikou 3a, Athina 117 42, Greece
openhours    | Sunday, 10 AM to 12 AM; Monday, 10 AM to 12 AM; Tuesday, 10 AM to 12 AM; Wednesday, 10 AM to 12 AM; Thursday, 10 AM to 12 AM; Friday, 10 AM to 12 AM; Saturday, 10 AM to 12 AM. Hide open hours for the week
website      | http://www.athenssportsbar.gr/
phone        | +302109235811
pluscode     | XP8H+V9 Athens, Greece
review_count | 1
rating       | 4.4
-[ RECORD 2 ]+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id           | 2
title        | 360 Cocktail bar
category     | Bar
address      | Ifestou 2, Athina 105 55, Greece
openhours    | Sunday, 9 AM to 4 AM; Monday, 9 AM to 3 AM; Tuesday, 9 AM to 3 AM; Wednesday, 9 AM to 3 AM; Thursday, 9 AM to 3 AM; Friday, 9 AM to 3 AM; Saturday, 9 AM to 4 AM. Hide open hours for the week
website      | http://www.three-sixty.gr/
phone        | +302103210006
pluscode     | XPGG+H6 Athens, Greece
review_count | 8
rating       | 4.4
-[ RECORD 3 ]+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id           | 3
title        | Teddy Boy
category     | Bar
address      | Taki 18, Athina 105 54, Greece
openhours    | 
website      | https://m.facebook.com/teddyboy.bar
phone        | +306951116651
pluscode     | XPHF+8F Athens, Greece
review_count | 489
rating       | 4.5
-[ RECORD 4 ]+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id           | 4
title        | Revolt street bar
category     | Bar
address      | Koletti 25-27, Athina 106 77, Greece
openhours    | Sunday, 11 AM to 2 AM; Monday, 11 AM to 2 AM; Tuesday, 11 AM to 2 AM; Wednesday, 11 AM to 2 AM; Thursday, 11 AM to 2 AM; Friday, 11 AM to 3 AM; Saturday, 11 AM to 3 AM. Hide open hours for the week
website      | https://www.facebook.com/Revoltstreetbar/
phone        | +302103800016
pluscode     | XPPM+85 Athens, Greece
review_count | 461
rating       | 4.5
-[ RECORD 5 ]+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id           | 5
title        | 42 Barstronomy Athens
category     | Cocktail bar
address      | Kolokotroni 3, Athina 105 62, Greece
openhours    | 
website      | https://42barstronomy.gr/
phone        | +302130052153
pluscode     | XPGM+Q8 Athens, Greece
review_count | 1
rating       | 4.5

defaultdb=>

Conclusion

In this tutorial, I showed you how you can use the google-maps-scraper in Kubernetes to automate and scale scraping Google Maps results.

Note: Please clean up the resources in your Digital Ocean account to avoid undesired charges once you are done with this tutorial

How to extract data from Google maps using Golang

Georgios Komninos — Sun, 07 May 2023 07:26:23 GMT

This post introduces a command-line application that allows you to extract data from Google Maps.

If you want to extract to collect some data for local businesses using Google Maps then this tool is for you.

The tools retrieve and export to a CSV file the following data:

Title: the title of the business
Category: the category of the business
Address: the address of the business
OpenHours: the opening hours of the business
Website: the website of the business
Phone: the phone number of the business
PlusCode: the plus code of the business
ReviewCount: the number of reviews
ReviewRating: the rating of the results

Getting started

create a file with your queries like:

 bars in Athens
 doctor in Berlin
 doctor in Bonn

Make sure you have Docker installed

 docker run -v $PWD/example-queries.txt:/example-queries -v $PWD:/results gosom/google-maps-scraper -depth 1 -input /example-queries -results /results/result-file.csv

wait for it to finish. The program does not exits automatically. When there are no updates in the console for some time you can hit CTRL-C. Meanwhile you will see that the file result-file.csv will be populated
Results will be to the file result-file.csv

Notes: Please adjust the filenames to the desired ones.

Conclusion

In this blog post, I introduced google-maps-scraper command line tool.

The tool is built using the Go programming language and it uses scrapemate .

All the code is on Github. Feel free to create an Issue or leave a comment here if you run into problems or you have some ideas for extension.

Golang web scraping using Scrapemate

Georgios Komninos — Sat, 15 Apr 2023 15:58:59 GMT

Introduction

In this blog post, we are going to use scrapemate to extract hockey teams form

from the website https://www.scrapethissite.com/pages/forms

This website contains sandboxes for testing your scrapers, so no real data.

You can find the full code on github

The previous post uses the low lever API from scrapemate, this one uses the high level API

Code Skeleton

Create a folder named scrapemate-highlevel-api-example

mkdir scrapemate-highlevel-api-example
cd scrapemate-highlevel-api-example

the initialize a go module

go mod init github.com/gosom/scrapemate-highlevel-api-example

Create 2 folders:

hockey
testdata

mkdir hockey testdata

Parser

Now we need to figure out how we are going to parse the data from the website.

Scrapemate high level API is using goquery and CSS-selectors. You can use another html parsing library by utilizing the low level API if you like.

Create a file team.go in the hockey directory

touch hockey/team.go

and copy the following:

package hockey

import (
    "strconv"
    "strings"

    "github.com/PuerkitoBio/goquery"
)

type Team struct {
    Name         string
    Year         int
    Wins         int
    Losses       int
    OTLosses     int
    WinPct       float64
    GoalsFor     int
    GoalsAgainst int
    GoalDiff     int
}

func (t Team) CsvHeaders() []string {
    return []string{
        "Name",
        "Year",
        "Wins",
        "Losses",
        "OTLosses",
        "WinPct",
        "GoalsFor",
        "GoalsAgainst",
        "GoalDiff",
    }
}

func (t Team) CsvRow() []string {
    return []string{
        t.Name,
        strconv.Itoa(t.Year),
        strconv.Itoa(t.Wins),
        strconv.Itoa(t.Losses),
        strconv.Itoa(t.OTLosses),
        strconv.FormatFloat(t.WinPct, 'f', 2, 64),
        strconv.Itoa(t.GoalsFor),
        strconv.Itoa(t.GoalsAgainst),
        strconv.Itoa(t.GoalDiff),
    }
}

func parseTeams(doc *goquery.Document) ([]Team, error) {
    sel := "table.table tr.team"
    var teams []Team
    doc.Find(sel).Each(func(i int, s *goquery.Selection) {
        teams = append(teams, parseTeam(s))
    })
    return teams, nil
}

func parseTeam(s *goquery.Selection) Team {
    var team Team
    team.Name = cleanText(s.Find("td.name").Text())
    team.Year = parseInt(s.Find("td.year").Text())
    team.Wins = parseInt(s.Find("td.wins").Text())
    team.Losses = parseInt(s.Find("td.losses").Text())
    team.OTLosses = parseInt(s.Find("td.ot-losses").Text())
    team.WinPct = parseFloat(s.Find("td.pct").Text())
    team.GoalsFor = parseInt(s.Find("td.gf").Text())
    team.GoalsAgainst = parseInt(s.Find("td.ga").Text())
    team.GoalDiff = parseInt(s.Find("td.diff").Text())
    return team
}

func parseNextLink(doc *goquery.Document) (string, map[string]string) {
    sel := "ul.pagination>li:last-child>a[aria-label=Next]"
    s := doc.Find(sel).AttrOr("href", "")
    if s == "" {
        return "", nil
    }
    s = "https://www.scrapethissite.com" + s
    parts := strings.Split(s, "?")
    nextLink := parts[0]
    params := make(map[string]string)
    for _, p := range strings.Split(parts[1], "&") {
        kv := strings.Split(p, "=")
        params[kv[0]] = kv[1]
    }
    return nextLink, params
}

func cleanText(s string) string {
    s = strings.TrimFunc(s, func(r rune) bool {
        return r == '\n'
    })
    return strings.TrimSpace(s)
}

func parseInt(s string) int {
    s = cleanText(s)
    if s == "" {
        return 0
    }
    ans, _ := strconv.Atoi(s)
    return ans
}

func parseFloat(s string) float64 {
    s = cleanText(s)
    if s == "" {
        return 0
    }
    ans, _ := strconv.ParseFloat(s, 64)
    return ans
}

The code above is straightforward.

The most important functions are:

parseTeams : returns a list of Team structs with the attributes populated
parseNextLink: returns the next link in two parts the url and the url params

We need to test that we parse properly. Let's create some unit tests.

But first download into testdata the website:

 curl -o testdata/teams.html 'https://www.scrapethissite.com/pages/forms/?page_num=1&per_page=100'

create a file hockey/team_test.go and paste the following:

package hockey

import (
    "os"
    "testing"

    "github.com/PuerkitoBio/goquery"
    "github.com/stretchr/testify/require"
)

func Test_parseTeams(t *testing.T) {
    fd, err := os.Open("../testdata/teams.html")
    require.NoError(t, err)
    defer fd.Close()
    doc, err := goquery.NewDocumentFromReader(fd)
    require.NoError(t, err)

    teams, err := parseTeams(doc)
    require.NoError(t, err)
    require.Equal(t, 100, len(teams))

    team := teams[0]
    require.Equal(t, "Boston Bruins", team.Name)
    require.Equal(t, 1990, team.Year)
    require.Equal(t, 44, team.Wins)
    require.Equal(t, 24, team.Losses)
    require.Equal(t, 0, team.OTLosses)
    require.Equal(t, 0.55, team.WinPct)
    require.Equal(t, 299, team.GoalsFor)
    require.Equal(t, 264, team.GoalsAgainst)
    require.Equal(t, 35, team.GoalDiff)
}

func Test_parseNextLink(t *testing.T) {
    fd, err := os.Open("../testdata/teams.html")
    require.NoError(t, err)
    defer fd.Close()
    doc, err := goquery.NewDocumentFromReader(fd)
    require.NoError(t, err)

    nextLink, params := parseNextLink(doc)
    require.Equal(t, "https://www.scrapethissite.com/pages/forms/", nextLink)
    require.Equal(t, "2", params["page_num"])
    require.Equal(t, "100", params["per_page"])
}

Run

go mod tidy

and then run the unit tests

go test -v ./...

Tests must pass.

Scraping Job definition

create a file hockey/collect.go

touch hockey/collect.go

and paste the following

package hockey

import (
    "context"
    "fmt"
    "net/http"
    "time"

    "github.com/PuerkitoBio/goquery"
    "github.com/google/uuid"
    "github.com/gosom/scrapemate"
)

type TeamCollectJob struct {
    scrapemate.Job
}

func NewTeamCollectJob(u string, params map[string]string) *TeamCollectJob {
    return &TeamCollectJob{
        Job: scrapemate.Job{
            // just give it a random id
            ID:        uuid.New().String(),
            Method:    http.MethodGet,
            URL:       u,
            UrlParams: params,
            Headers: map[string]string{
                "User-Agent": scrapemate.DefaultUserAgent,
            },
            Timeout:    10 * time.Second,
            MaxRetries: 3,
        },
    }
}

func (o *TeamCollectJob) Process(ctx context.Context, resp *scrapemate.Response) (any, []scrapemate.IJob, error) {
    doc, ok := resp.Document.(*goquery.Document)
    if !ok {
        return nil, nil, fmt.Errorf("invalid document type %T expected *goquery.Document", resp.Document)
    }
    teams, err := parseTeams(doc)
    if err != nil {
        return nil, nil, err
    }

    var nextJobs []scrapemate.IJob

    nextLink, params := parseNextLink(doc)
    if nextLink != "" {
        nextJobs = append(nextJobs, NewTeamCollectJob(nextLink, params))
    }

    return teams, nextJobs, nil
}

Here we define the scraping job (TeamCollectJob) and the Process method.

The Process method returns three things:

the result (here a slice of Teams)
the next jobs (the job defined by the next link in pagination)
an error if it occurs

Run

go mod tidy

the main function

create a file main.go

touch main.go

and paste the contents

package main

import (
    "context"
    "encoding/csv"
    "os"

    "github.com/gosom/scrapemate"
    "github.com/gosom/scrapemate-highlevel-api-example/hockey"
    "github.com/gosom/scrapemate/adapters/writers/csvwriter"
    "github.com/gosom/scrapemate/scrapemateapp"
)

func main() {
    if err := run(); err != nil {
        os.Stderr.WriteString(err.Error() + "\n")
        os.Exit(1)
        return
    }
    os.Exit(0)
}

func run() error {
    csvWriter := csvwriter.NewCsvWriter(csv.NewWriter(os.Stdout))

    writers := []scrapemate.ResultWriter{
        csvWriter,
    }

    cfg, err := scrapemateapp.NewConfig(writers)
    if err != nil {
        return err
    }
    app, err := scrapemateapp.NewScrapeMateApp(cfg)
    if err != nil {
        return err
    }
    params := map[string]string{
        "page_num": "1",
        "per_page": "100",
    }
    seedJobs := []scrapemate.IJob{
        hockey.NewTeamCollectJob("https://www.scrapethissite.com/pages/forms/", params),
    }
    return app.Start(context.Background(), seedJobs...)
}

Here we define a csvwriter that writes to stdout

csvWriter := csvwriter.NewCsvWriter(csv.NewWriter(os.Stdout))

writers := []scrapemate.ResultWriter{
    csvWriter,
}

Then we initialize our scraper:

cfg, err := scrapemateapp.NewConfig(writers)
    if err != nil {
        return err
    }
    app, err := scrapemateapp.NewScrapeMateApp(cfg)
    if err != nil {
        return err
    }

Finally, we create a seed job (the one that our scraper will use to start).

and start the scraper

params := map[string]string{
        "page_num": "1",
        "per_page": "100",
    }
    seedJobs := []scrapemate.IJob{
        hockey.NewTeamCollectJob("https://www.scrapethissite.com/pages/forms/", params),
    }
    return app.Start(context.Background(), seedJobs...)

Run

go mod tidy

Run the scraper

In order to run the scraper just do:

 go run main.go 1>hockey.csv

After all the 6 pages are crawled you may stop the scraper using CTRL-C

the results must be in hockey.csv

Summary

Scrapemate is a web scraping framework written in Golang. In this post, we demonstrated how easy it is to create a scraper and save the results in a CSV file.

Read the blog post for the low level API here and see how easier is to scrape using the high level API.

See the full example here and see another example on github examples

Getting Started with Web Scraping Using Golang and Scrapemate

Georgios Komninos — Fri, 14 Apr 2023 07:50:30 GMT

Introduction

Web scraping is the process of extracting data from websites, and it can be a powerful tool for collecting information for research, analysis, or automation.

In this tutorial, I will show you how to use Golang and the Scrapemate framework to scrape data from a website.

As an example, we will extract product information from https://scrapeme.live/shop/. Specifically, we'll extract

title
price
short_description
sku
categories

for each of the Pokemon products on the site.

Once we have this data, we'll create a CSV file containing it.

See the image below with highlighted the data we need to extract for each product

Prerequisites

Before you get started make sure that you have a Golang version >=1.20 installed. You can find installation instructions here

Step1: Inspecting the website

The first step in any web scraping project is to inspect the website you want to scrape. In our case, we want to extract data about Pokemon products from https://scrapeme.live/shop/.

Finding the css-selectors from the home page

Open https://scrapeme.live/shop/ in your browser. This opens the home page of the Pokemon e-shop.

This will be the starting page for our scraper.
Our scraper should try to find all the links to the products (pokemon in that case) and additionally, it should find the link to the next page.

The idea is that we should visit all the pages via pagination and for each page we extract the Pokemon product links. We also visit them and extract the information.

By using Chrome's Developer tools and by right-clicking the element we are interested in we can find a suitable CSS selector so we can extract the information in our parser.

In the image above I show you how can you do it in Chrome:

Hit F12 to open developer tools
Right click the next arrow that takes you to the next page and click inspect element
then in the developer's tools you can find the proper CSS selector

In our case to get the page to the next link we need to use the element

with the following CSS selector:

a.next.page-numbers

By right-clicking on a product (pokemon) image we can inspect the element and find the element that contains that lint that takes us to the detailed product page.
In our case it's

a.woocommerce-LoopProduct-link

finding the CSS selectors for the product details page

Now visit a product detail page like https://scrapeme.live/shop/Bulbasaur/

Similarly, we find the CSS selectors for the elements we are interested in.

In particular, we have:

title : h1.product_title
price: p.price
short_description: div.woocommerce-product-details__short-description>p
sku: span.sku
categories: div.product_meta > span.posted_in > a
tags: div.product_meta > span.tagged_as > a

Step2: Create project skeleton

You need to create a folder that will host your code:


mkdir scrapemelive
cd scrapemelive

initialize a go module using go mod init

go mod init scrapemelive

Then create the following folders:

mkdir scrapemelive
mkdir testdata

the contents of the folder should be:

├── go.mod
├── scrapemelive
└── testdata

Add also a main.go file which just prints 'hello world' for the moment.

touch main.go

open main.go into your editor and add the following:

package main

import "fmt"

func main() {
        fmt.Println("hello world")
}

Let's test that the code can run by running go run main.go .
You should see in the standard output the hello world printed.

The main.go file will contain the code that will start our scraper and the code that writes the results to CSV.

The folder scrapemelive will contain the necessary functions/types/code that the framework will use

The folder testdata will contain data that we need for our unit tests.

Step 3: Writing the Scraping Code

Let's first create a file in scrapemelive/product.go . Here we are going to add a struct that holds the Product data that we scrape.

package scrapemelive

// Product is a product scraped from the detail page
type Product struct {
        // Name is the name of the product
        Title string
        // Price is the price of the product
        Price string
        // ShortDescription is the short description of the product
        ShortDescription string
        // Sku is the sku of the product
        Sku string
        // Categories is the categories of the product
        Categories []string
        // Tags is the tags of the product
        Tags []string
}

Each scraped Product will be an instance of the Product struct.

Let's also create a scrapemelive/product_test.go file in which we will write our unit tests. For now, it will be empty

package scrapemelive

Our project folder should look like this

├── go.mod
├── main.go
├── scrapemelive
│   ├── product.go
│   └── product_test.go
└── testdata

The next step is to write the functions that extract the data from the detail page based on the css selectors we identified above.

Open the scrapemelive/product.go file and add the following


func parseTitle(doc *goquery.Document) string {
        return doc.Find("h1.product_title").Text()
}

func parsePrice(doc *goquery.Document) string {
        return doc.Find("p.price").Text()
}

func parseShortDescription(doc *goquery.Document) string {
        return doc.Find("div.woocommerce-product-details__short-description>p").Text()
}

func parseSku(doc *goquery.Document) string {
        return doc.Find("span.sku").Text()
}

func parseCategories(doc *goquery.Document) []string {
        var categories []string
        doc.Find("div.product_meta > span.posted_in > a").Each(func(i int, s *goquery.Selection) {
                categories = append(categories, s.Text())
        })
        return categories
}

func parseTags(doc *goquery.Document) []string {
        var tags []string
        doc.Find("div.product_meta > span.tagged_as > a").Each(func(i int, s *goquery.Selection) {
                tags = append(tags, s.Text())
        })
        return tags
}

We need the goquery library so let's also get it

go get github.com/PuerkitoBio/goquery

make sure you also import at the top of the file the import for goquery

 import "github.com/PuerkitoBio/goquery"

Each of the above functions accepts a *goquery.Document as an input, extracts from the document the data we are interested using goquery in and returns them.

We wrote some code but it's not tested yet. Let's write some tests then.

Before writing the tests let's download the HTML of a product page in our test data directory:

curl -o testdata/sample-product.html 'https://scrapeme.live/shop/Charmeleon/'

The command above saves the HTML for the Charmeleon product in the file testdata/sample-product.html.

Let's now add some tests.

For unit tests we use the testify library, so get it via go get

go get github.com/stretchr/testify/require

In scrapemelive/product_test.go add the following:

package scrapemelive

import (
        "os"
        "testing"

        "github.com/PuerkitoBio/goquery"
        "github.com/stretchr/testify/require"
)

func openTestFile(t *testing.T, filename string) *goquery.Document {
        t.Helper()
        file, err := os.Open(filename)
        require.NoError(t, err)
        defer file.Close()
        doc, err := goquery.NewDocumentFromReader(file)
        require.NoError(t, err)
        return doc
}

func Test_parseTitle(t *testing.T) {
        t.Parallel()
        doc := openTestFile(t, "../testdata/sample-product.html")
        require.Equal(t, "Charmeleon", parseTitle(doc))
}

func Test_parsePrice(t *testing.T) {
        t.Parallel()
        doc := openTestFile(t, "../testdata/sample-product.html")
        require.Equal(t, "£165.00", parsePrice(doc))
}

func Test_parseShortDescription(t *testing.T) {
        t.Parallel()
        doc := openTestFile(t, "../testdata/sample-product.html")
        require.Equal(t, "Charmeleon mercilessly destroys its foes using its sharp claws. If it encounters a strong foe, it turns aggressive. In this excited state, the flame at the tip of its tail flares with a bluish white color.", parseShortDescription(doc))
}

func Test_parseSku(t *testing.T) {
        t.Parallel()
        doc := openTestFile(t, "../testdata/sample-product.html")
        require.Equal(t, "6565", parseSku(doc))
}

func Test_parseCategories(t *testing.T) {
        t.Parallel()
        doc := openTestFile(t, "../testdata/sample-product.html")
        require.ElementsMatch(t, []string{"Pokemon", "Flame"}, parseCategories(doc))
}

func Test_parseTags(t *testing.T) {
        t.Parallel()
        doc := openTestFile(t, "../testdata/sample-product.html")
        require.ElementsMatch(t, []string{"Blaze", "charmeleon", "Flame"}, parseTags(doc))

Let's run our tests:

go test -v ./...

All tests must pass like:

?       scrapemelive    [no test files]
=== RUN   Test_parseTitle
=== PAUSE Test_parseTitle
=== RUN   Test_parsePrice
=== PAUSE Test_parsePrice
=== RUN   Test_parseShortDescription
=== PAUSE Test_parseShortDescription
=== RUN   Test_parseSku
=== PAUSE Test_parseSku
=== RUN   Test_parseCategories
=== PAUSE Test_parseCategories
=== RUN   Test_parseTags
=== PAUSE Test_parseTags
=== CONT  Test_parseTitle
=== CONT  Test_parseSku
=== CONT  Test_parseShortDescription
=== CONT  Test_parsePrice
=== CONT  Test_parseTags
=== CONT  Test_parseCategories
--- PASS: Test_parseShortDescription (0.00s)
--- PASS: Test_parseTitle (0.00s)
--- PASS: Test_parseSku (0.00s)
--- PASS: Test_parsePrice (0.00s)
--- PASS: Test_parseCategories (0.00s)
--- PASS: Test_parseTags (0.00s)
PASS
ok      scrapemelive/scrapemelive       0.005s

This verifies that our parsing functions work as we expect.

The next step is to add a function product.go that uses all these functions and returns a Product .

In scrapemelive/product.go add the following function:

func parseProduct(doc *goquery.Document) Product {
        return Product{
                Title:            parseTitle(doc),
                Price:            parsePrice(doc),
                ShortDescription: parseShortDescription(doc),
                Sku:              parseSku(doc),
                Categories:       parseCategories(doc),
                Tags:             parseTags(doc),
        }
}

Don't forget to add a unit test in scrapemelive/product_test.go also for this function

func Test_parseProduct(t *testing.T) {
        t.Parallel()
        doc := openTestFile(t, "../testdata/sample-product.html")
        product := parseProduct(doc)
        require.Equal(t, "Charmeleon", product.Title)
        require.Equal(t, "£165.00", product.Price)
        require.Equal(t, "Charmeleon mercilessly destroys its foes using its sharp claws. If it encounters a strong foe, it turns aggressive. In this excited state, the flame at the tip of its tail flares with a bluish white color.", product.ShortDescription)
        require.Equal(t, "6565", product.Sku)
        require.ElementsMatch(t, []string{"Pokemon", "Flame"}, product.Categories)
        require.ElementsMatch(t, []string{"Blaze", "charmeleon", "Flame"}, product.Tags)
}

Make sure your tests still pass by running go test -v ./... .

(Scrapemate)[https://github.com/gosom/scrapemate] accepts jobs that implement a specific interface scrapemate.IJob . Luckily, there is an implementation of that interface that we can use as a base (scrapemate.Job) .

We previously identified that we have two types of pages:

the listing pages, which contain multiple products and pagination. An example of a listing page is: https://scrapeme.live/shop/
the product or detail page, which contains the data for a specific product. An example of a detail page is: https://scrapeme.live/shop/Charmeleon/

IJob interface has a method with the following signature:

Process(ctx context.Context, resp scrapemate.Response) (any, []scrapemate.IJob, error)

It accepts:

context.Context: which is a context object
scrapemate.Response: the Response object that contains the goquery.Document and other fields.

it returns:

any: this is the result of the scraper
[]scrapemate.IJob: when we parse a web page we may need to instruct the scraper to visit other pages that we have discovered. So we return the next pages to visit there.
error: returns an error if there is one

Based on the above and in particular to the []scrapemate.IJob explanation we understand that we need two types of jobs.

ProductCollectJob: for the listing pages
ProductJob: for the product detail pages

The ProductCollectJob will extract the next page link and create another ProductCollectJob and will extract the product details and other ProductJob.

The ProductJob is responsible to return a Product and it does not have other jobs to return.

Install some dependencies:

go get github.com/gosom/scrapemate
go get github.com/gosom/kit/logging
go get github.com/google/uuid

ProductJob

Let's create first the ProductJob by creating a file scrapemelive/detail.go .

package scrapemelive

import (
        "context"
        "errors"

        "github.com/PuerkitoBio/goquery"
        "github.com/gosom/kit/logging"
        "github.com/gosom/scrapemate"
)

type ProductJob struct {
        scrapemate.Job
}

func (o *ProductJob) Process(ctx context.Context, resp *scrapemate.Response) (any, []scrapemate.IJob, error) {
        log := ctx.Value("log").(logging.Logger)
        log.Info("processing product job")
        doc, ok := resp.Document.(*goquery.Document)
        if !ok {
                return nil, nil, errors.New("failed to convert response to goquery document")
        }
        product := parseProduct(doc)
        return product, nil, nil
}

Let's explain a bit about what the above is doing:

We create a new struct called ProductJob in which we embed the scrapemate.Job and we implement the method Process to accommodate our needs.

The Process method for that job has to parse the document fetched and extract the data we need and create an instance of Product. We then need to return that newly scraped Product.

We explain a bit more almost line by line:

Scrapemate offers you a logger and you can get it from context via

ctx.Value("log").(logging.Logger)

In order to fetch the document we need to do:

doc, ok := resp.Document.(*goquery.Document)

Notice the typecasting, this is because scrapemate gives you the capability to configure the type of document parser you want to use. We will see that when we initialize the framework.

Once we have a document we can extract the information we need and create a product.

product := parseProduct(doc)

Notice, that this is the function we created before, nothing more to explain here.

Finally, we return

return product, nil, nil

The order of return is :

first: the data we parsed, here the product
second: the next jobs that the scraper should process - here nothing
third: an error if there is any - here no error

ProductCollectJob

The ProductCollectJob will extract the required links (for products and for the next page) and will return new jobs.

create the file scrapemelive/collect.go and add the following contents

package scrapemelive

import (
    "context"
    "errors"
    "time"

    "github.com/PuerkitoBio/goquery"
    "github.com/google/uuid"
    "github.com/gosom/kit/logging"
    "github.com/gosom/scrapemate"
)

type ProductCollectJob struct {
    scrapemate.Job
}

func (o *ProductCollectJob) Process(ctx context.Context, resp *scrapemate.Response) (any, []scrapemate.IJob, error) {
    log := ctx.Value("log").(logging.Logger)
    log.Info("processing collect job")
    doc, ok := resp.Document.(*goquery.Document)
    if !ok {
        return nil, nil, errors.New("failed to convert response to goquery document")
    }
    var nextJobs []scrapemate.IJob
    links := parseProductLinks(doc)
    for _, link := range links {
        nextJobs = append(nextJobs, &ProductJob{
            Job: scrapemate.Job{
                ID:     uuid.New().String(),
                Method: "GET",
                URL:    link,
                Headers: map[string]string{
                    "User-Agent": scrapemate.DefaultUserAgent,
                },
                Timeout:    10 * time.Second,
                MaxRetries: 3,
                Priority:   0,
            },
        })
    }
    nextPage := parseNextPage(doc)
    if nextPage != "" {
        nextJobs = append(nextJobs, &ProductCollectJob{
            Job: scrapemate.Job{
                ID:     uuid.New().String(),
                Method: "GET",
                URL:    nextPage,
                Headers: map[string]string{
                    "User-Agent": scrapemate.DefaultUserAgent,
                },
                Timeout:    10 * time.Second,
                MaxRetries: 3,
                Priority:   1,
            },
        })
    }

    return nil, nextJobs, nil
}

func parseProductLinks(doc *goquery.Document) []string {
    var links []string
    doc.Find("a.woocommerce-LoopProduct-link").Each(func(i int, s *goquery.Selection) {
        link, _ := s.Attr("href")
        links = append(links, link)
    })
    return links
}

func parseNextPage(doc *goquery.Document) string {
    return doc.Find("a.next.page-numbers").AttrOr("href", "")
}

Notice here that we have at the end of the file two functions

parseProductLinks
parseNextPage

They do what the name implies.
First, it extracts all the links for the products and returns a slice of strings that contains the links.

The second function extracts the links that take us to the next page in the pagination.

Let's see in more detail the Process method

var nextJobs []scrapemate.IJob
    links := parseProductLinks(doc)
    for _, link := range links {
        nextJobs = append(nextJobs, &ProductJob{
            Job: scrapemate.Job{
                ID:     uuid.New().String(),
                Method: "GET",
                URL:    link,
                Headers: map[string]string{
                    "User-Agent": scrapemate.DefaultUserAgent,
                },
                Timeout:    10 * time.Second,
                MaxRetries: 3,
                Priority:   0,
            },
        })
    }

The portion of the code above is responsible to :

parse the links out of the webpage
create a ProductJob for each link and append it to nextJobs slice

    nextPage := parseNextPage(doc)
    if nextPage != "" {
        nextJobs = append(nextJobs, &ProductCollectJob{
            Job: scrapemate.Job{
                ID:     uuid.New().String(),
                Method: "GET",
                URL:    nextPage,
                Headers: map[string]string{
                    "User-Agent": scrapemate.DefaultUserAgent,
                },
                Timeout:    10 * time.Second,
                MaxRetries: 3,
                Priority:   1,
            },

What we do here is that we first parse the next link.

Afterward, if the next link is not empty we create a ProductCollectJob and we append it to the nextJobs slice.

Let's write some tests for the parseProductLinks and parseNextPage functions.

Fetch a listing page and store its HTML to testdata folder

curl -o testdata/sample-category.html 'https://scrapeme.live/shop/'

Create a file scrapemelive/collect_test.go and add:

package scrapemelive

import (
        "testing"

        "github.com/stretchr/testify/require"
)

func Test_parseProductLinks(t *testing.T) {
        t.Parallel()
        doc := openTestFile(t, "../testdata/sample-category.html")
        links := parseProductLinks(doc)
        require.Len(t, links, 16)
}

func Test_parseNextPage(t *testing.T) {
        t.Parallel()
        doc := openTestFile(t, "../testdata/sample-category.html")
        nextPage := parseNextPage(doc)
        require.Equal(t, "https://scrapeme.live/shop/page/2/", nextPage)
}

Make sure tests pass:

go test -v ./...

Main function

Now it's time to write our main.go function.

Instal another dependency

go get github.com/gosom/scrapemate/adapters/cache/leveldbcache

Now in your main.go add the following

package main

import (
    "context"
    "errors"
    "net/http"
    "os"
    "time"

    "github.com/google/uuid"
    "github.com/gosom/scrapemate"
    "github.com/gosom/scrapemate/adapters/cache/leveldbcache"
    fetcher "github.com/gosom/scrapemate/adapters/fetchers/nethttp"
    parser "github.com/gosom/scrapemate/adapters/parsers/goqueryparser"
    provider "github.com/gosom/scrapemate/adapters/providers/memory"

    "scrapemelive/scrapemelive"
)

func main() {
    err := run()
    if err == nil || errors.Is(err, scrapemate.ErrorExitSignal) {
        os.Exit(0)
        return
    }
    os.Exit(1)
}

func run() error {
    ctx, cancel := context.WithCancelCause(context.Background())
    defer cancel(errors.New("deferred cancel"))

    provider := provider.New()

    go func() {
        job := &scrapemelive.ProductCollectJob{
            Job: scrapemate.Job{
                ID:     uuid.New().String(),
                Method: http.MethodGet,
                URL:    "https://scrapeme.live/shop/",
                Headers: map[string]string{
                    "User-Agent": scrapemate.DefaultUserAgent,
                },
                Timeout:    10 * time.Second,
                MaxRetries: 3,
            },
        }
        provider.Push(ctx, job)
    }()

    httpFetcher := fetcher.New(&http.Client{
        Timeout: 10 * time.Second,
    })

    cacher, err := leveldbcache.NewLevelDBCache("__leveldb_cache")
    if err != nil {
        return err
    }

    mate, err := scrapemate.New(
        scrapemate.WithContext(ctx, cancel),
        scrapemate.WithJobProvider(provider),
        scrapemate.WithHTTPFetcher(httpFetcher),
        scrapemate.WithConcurrency(10),
        scrapemate.WithHTMLParser(parser.New()),
        scrapemate.WithCache(cacher),
    )

    if err != nil {
        return err
    }

    resultsDone := make(chan struct{})
    go func() {
        defer close(resultsDone)
        if err := writeCsv(mate.Results()); err != nil {
            cancel(err)
            return
        }
    }()

    err = mate.Start()
    <-resultsDone
    return err
}

func writeCsv(results <-chan scrapemate.Result) error {
    // TODO
    return nil
}

Let's go over it slowly:

Initially, the function run is called and it returns an error.

scrapemate returns a special error ErrorExitSignal when the program exits because it captured a SIGINT. In such a case we want it to exist with status code 0.

In the other cases, we return exit code 1.

Now, the run function. A lot is happening here

Scrapemate requires as to first declare a job provider:

provider := provider.New()

    go func() {
        job := &scrapemelive.ProductCollectJob{
            Job: scrapemate.Job{
                ID:     uuid.New().String(),
                Method: http.MethodGet,
                URL:    "https://scrapeme.live/shop/",
                Headers: map[string]string{
                    "User-Agent": scrapemate.DefaultUserAgent,
                },
                Timeout:    10 * time.Second,
                MaxRetries: 3,
            },
        }
        provider.Push(ctx, job)
    }()

A provider is a data structure that provides jobs to the scraper. Here we want to start our crawler with the homepage of the e-shop. We create the initial job and we push to the provider.

We also have to define how we are going to fetch webpages. For this purpose, we need an instance of a fetcher

httpFetcher := fetcher.New(&http.Client{
        Timeout: 10 * time.Second,
    })

We also want to cache the responses so we initialize an instance of a Cacher.

cacher, err := leveldbcache.NewLevelDBCache("__leveldb_cache")
    if err != nil {
        return err
    }

Above, we are going to cache using leveldb and the database will be created in a folder named __leveldb_cache .

We can now initialize our scraper:

mate, err := scrapemate.New(
        scrapemate.WithContext(ctx, cancel),
        scrapemate.WithJobProvider(provider),
        scrapemate.WithHTTPFetcher(httpFetcher),
        scrapemate.WithConcurrency(10),
        scrapemate.WithHTMLParser(parser.New()),
        scrapemate.WithCache(cacher),
    )

    if err != nil {
        return err
    }

Notice the WithConcurrency , this configures the framework to use 10 parallel workers.

The WithHtml parser configures the html parser. We chose to use the default one, which uses goquery.

scrapemate.WithHtmlParser(parser.New()),

Once Scrapemate finishes a job it pushes the result into a channel. To acces that channel we can use the mate.Results() method.

resultsDone := make(chan struct{})
    go func() {
        defer close(resultsDone)
        if err := writeCsv(mate.Results()); err != nil {
            cancel(err)
            return
        }
    }()

The above snippet starts a new goroutine that is responsible to write the results in a CSV file.

In the last part

err = mate.Start()
<-resultsDone
return err

we just start the scraper and wait until all the results are written.

The scraper, even if it has no more jobs will still wait until you kill it via ctrl-c.

CSV writing

In our task description, we want to create a CSV with headers:

title,price,short_description,sku,categories,tags

Open scrapemelive.go and add the following:

func (o Product) CsvHeaders() []string {
    return []string{
        "title",
        "price",
        "short_description",
        "sku",
        "categories",
        "tags",
    }
}

func (o Product) CsvRow() []string {
    return []string{
        o.Title,
        o.Price,
        o.ShortDescription,
        o.Sku,
        strings.Join(o.Categories, ","),
        strings.Join(o.Tags, ","),
    }
}

Now open main.go and replace the function writeCsv with the following:

func writeCsv(results <-chan scrapemate.Result) error {
    w := csv.NewWriter(os.Stdout)
    defer w.Flush()
    headersWritten := false
    for result := range results {
        if result.Data == nil {
            continue
        }
        product, ok := result.Data.(scrapemelive.Product)
        if !ok {
            return fmt.Errorf("unexpected data type: %T", result.Data)
        }
        if !headersWritten {
            if err := w.Write(product.CsvHeaders()); err != nil {
                return err
            }
            headersWritten = true
        }
        if err := w.Write(product.CsvRow()); err != nil {
            return err
        }
        w.Flush()
    }
    return w.Error()
}

The code above just takes each result from the Results channel and writes it to the CSV file.

Run the scraper

Now that all the code is in place we can run our scraper.

go run main.go 1>pokemons.csv

This will take some time. Meanwhile, you will see the logs in your screen.

Once the logs stop updating wait a few seconds and hit CTRL-C.

{"level":"info","component":"scrapemate","job":"Job{ID: 4de24748-1e8e-4ab7-843f-8571ca8b2d49, Method: GET, URL: https://scrapeme.live/shop/Blacephalon/, UrlParams: map[]}","status":"success","duration":1323.897329,"time":"2023-04-14T07:41:41.911948749Z","message":"job finished"}
^C{"level":"info","component":"scrapemate","time":"2023-04-14T07:42:13.345446924Z","message":"received signal, shutting down"}
{"level":"info","component":"scrapemate","time":"2023-04-14T07:42:13.345757168Z","message":"scrapemate exited"}

You should see something like the above.

The results should be in pokemons.csv file.

Summary

In this tutorial, I have shown how to use Golang and Scrapemate to extract data from a website.

Specifically, I demonstrated how to scrape product information from the website https://scrapeme.live/shop/ by extracting the title, price, short_description, sku, tags, and categories for each of the Pokemon products on the site.

I used Scrapemate, a Golang-based scraping framework, to perform the web scraping, and then wrote the scraped data to a CSV file.

This example illustrates how web scraping can be a powerful tool for collecting information for research, analysis, or automation, and how Scrapemate can simplify the process of building scraping tools in Golang.

You can find all the code above in github

Managing Distributed Transactions in PostgreSQL and Golang using two phase commit

Georgios Komninos — Sat, 25 Mar 2023 20:04:09 GMT

If you're building a distributed system with PostgreSQL as the database backend, you might have encountered issues with managing transactions across multiple nodes. When a transaction spans multiple databases, ensuring atomicity and consistency can be a challenge. That's where gosql2pc comes in.

gosql2pc is a Golang library for implementing 2 phase commit transactions in PostgreSQL, ensuring atomicity and consistency across distributed systems. With gosql2pc, you can manage transactions across multiple databases, ensuring that all changes are committed or rolled back atomically.

Getting Started

To get started with gosql2pc, you need to have a working knowledge of Golang and PostgreSQL. You also need to have PostgreSQL installed, as gosql2pc relies on the PostgreSQL 2-phase commit protocol.

Once you have PostgreSQL installed, you can install gosql2pc using the following command:

go get github.com/gosom/gosql2pc

Then, use the library's API to create participants for your distributed transaction.

Here's an example of using gosql2pc to create a simple distributed transaction that inserts a new user and order into two separate databases:

// Create the participants for the 2 phase commit
p1 := twophase.NewParticipant(db1, func(ctx context.Context, tx *sql.Tx) error {
    _, err := tx.ExecContext(ctx, "INSERT INTO users (id, name) VALUES ($1, $2)", userID, name)
    return err
})

p2 := twophase.NewParticipant(db2, func(ctx context.Context, tx *sql.Tx) error {
    _, err := tx.ExecContext(ctx, "INSERT INTO orders (id, user_id, amount) VALUES ($1, $2, $3)", orderID, userID, amount)
    return err
})

// setup the parameters for the transaction
params := twophase.Params{
    Participants: []gosql2pc.Participant{p1, p2},
}

// run the transaction
if err := twophase.Do(context.Background(), params); err != nil {
    panic(err)
}

You can find more examples in the library's example directory.

Notes

It's worth noting that distributed transactions can be tricky to manage, and gosql2pc is no exception. PostgreSQL has disabled prepared transactions by default for a good reason, and enabling them can lead to orphaned transactions and data inconsistencies if not monitored carefully.

Nevertheless, gosql2pc provides a useful tool for simplifying the implementation of distributed transactions. If you find a bug or want to suggest a new feature, contributions are always welcome.

To learn more about gosql2pc, including how to enable prepared transactions and monitor for orphaned transactions, check out the library's README file and the accompanying blog posts.

Introducing Address Parser Go REST: A Simple Solution for Address Parsing

Georgios Komninos — Thu, 09 Mar 2023 20:16:58 GMT

Parsing addresses can be a complex and time-consuming task.

Different address formats and variations can make it challenging to extract accurate information from address strings.

Fortunately, Address Parser Go REST provides a simple solution to this problem.

This REST API allows you to parse addresses into their individual components quickly and easily.

What is Address Parser Go REST?

Address Parser Go REST is a REST API that uses the libpostal library to parse addresses into their individual components.

By submitting a request to the API with an address, you will receive a JSON response with the parsed components.

This API removes the need to include the libpostal library as a dependency in your project and allows you to parse addresses with ease.

How does Address Parser Go REST work?

Address Parser Go REST uses the libpostal library to parse addresses.

The library is a powerful natural language processing library for addresses that can handle different address formats, variations, and misspellings, ensuring accurate parsing results.

The API is built with Go, a fast and efficient programming language, which allows for speedy processing of requests.

Getting started with Address Parser Go REST

docker run -p 8080:8080 gosom/address-parser-go-rest:v1.0.1

This command will run the Address Parser Go REST API container and map port 8080 of the container to port 8080 of your machine.

Now you can start parsing addresses! Here's an example of how to parse an address using the API:

curl --location --request POST 'http://localhost:8080/parse' --header 'Content-Type: application/json' --data-raw '{
    "address": "1600 Amphitheatre Parkway, Mountain View, CA 94043",
    "title_case": true
}'

This command sends a POST request to the /parse endpoint of the API with a JSON payload containing an address string. The API will parse the address and return a JSON response with the individual components of the address:

{
  "house_number": "1600",
  "road": "Amphitheatre Parkway",
  "postcode": "94043",
  "city": "Mountain View",
  "state": "Ca"
}

The title_case parameter is optional. Default behavior of libpostal is to return all addresses in lower case and if you omit that then you will have the same behavior.

That's it! You've successfully parsed an address using Address Parser Go REST.

You can see the available options and the responses in swagger documentation.

Access it by visiting https://localhost:8080/docs/ .

Acknowledgements

We would like to acknowledge the contributors of the libpostal library and the Go bindings used in this project.

We appreciate their hard work and dedication to creating these useful tools.

Conclusion

Address Parser Go REST is a simple solution for parsing addresses that can save you time and effort.

With its powerful parsing capabilities and efficient processing, this REST API is a useful tool for any project that requires address parsing.

If you encounter any issues or have feedback to provide, please create a new issue on the GitHub repository.

FOSDEM 2023: My Top Highlights from the Open-Source Conference

Georgios Komninos — Sat, 11 Feb 2023 09:56:10 GMT

Introduction

FOSDEM 2023 (Free and Open Source Software Developers' European Meeting) is an annual event that takes place in Brussels, Belgium. The conference brings together a diverse range of open-source enthusiasts, developers, and community members from around the world to exchange ideas, share knowledge, and collaborate on the latest open-source technologies and projects. FOSDEM offers a wide range of sessions, talks, and workshops that cover a variety of topics, including programming, system administration, and security. With a strong focus on community building and collaboration, FOSDEM is a must-attend event for anyone interested in the open-source world.

Preparation

Preparing for the FOSDEM conference involved researching the conference schedule and speakers, and booking travel and accommodation. Since this was my fourth time there I knew what to expect and there were no surprises. This year I joined the event with two friends. We started the conference on Friday with a few beers :) .

Selecting which talks to attend at the FOSDEM conference can be a challenging task, as there are often a large number of interesting sessions and workshops to choose from. With a wide variety of topics covering everything from programming and system administration to security and community building, it can be difficult to decide which sessions will provide the most value. It's important to consider one's professional interests and goals when selecting talks, but it's also helpful to step outside of one's comfort zone and attend sessions that cover new and unfamiliar topics.

My Schedule

This year I decided to attend only two Dev rooms the one focused on Go and the one focused on PostgreSQL. Go talks were on Saturday and PostgreSQL talks on Sunday.

Saturday

The State of Go: This is a talk about the changes/new features in the latest go version. If you are a Go developer I highly recommend watching it or reading the slides
Recipes for reducing cognitive load: Interesting talk about how you can write code that is easier to understand. Federico gave some nice tips and examples.
Recommend watching
Reconciliation Pattern, Control Theory and Cluster API: A talk about the reconciliation pattern that is used in Kubernetes to ensure that the current state is the desired state.
Five Steps to Make Your Go Code Faster & More Efficient: Nice talk from Bartłomiej. It explains a methodology of how you can optimize your code.
Recommend watching
Headscale: How we are using integration testing to reimplement Tailscale: Interesting talk about Kristofer's and Juan's journey implementing headscale .
Recommend watching
Our Mad Journey of Building a Vector Database in Go: A talk about https://weaviate.io/ . This is a vector search database built in Go. The talk touches Go internals and I found it very interesting.
Recommend watching
Building a basic event-driven application in Go in 20 minutes: Talk about the watermill library. A nice talk which quickly demonstrates the main feature of the library

I learned something from all of the talks and if you have the time, watch all of them. But if I had to suggest only 2 then my picks are The State of Go, Our Mad Journey of Building a Vector Database in Go.

Sunday

Tour de Data Types: VARCHAR2 or CHAR(255)?: Very nice talk about data types in Postgres with nice tips. Andreas did a very nice presentation
Recommend watching
How to Give Your Postgres Blog Posts an Outsize Impact: This was out of my comfort zone. It's about how to write and promote your tech blog. I realized that it is a lot of work. Thanks Claire
Recommend watching
When it all GOes right: Talk about PGX driver, a PostgreSQL driver for Golang. I liked Pavlo's presentation, he has a style.
Recommend watching
Deep Dive Into Query Performance: Talk from Peter Zaitsev the co-founder or Percona.
Don't Do This: This talk has a clickbait title but in my opinion, was the most interesting talk I attended this year. Thanks Jimmy

Highly recommend watching

If you have the time to watch only one talk I recommend the "Don't Do This" talk. If you have the time to watch one more then watch Claire's talk.

Networking

It was good that I attended the conference with two friends. I haven't seen one of them in person since 2019. This was one of the highlights for me . Additionally, I met randomly an old colleague with whom we are in contact now and then. This was great.

Finally, I had the chance to chat with the author of Watermill and had the opportunity to ask advice for on work-related problems from PostgreSQL hackers.

Summary

In conclusion, the FOSDEM conference was a rich and rewarding experience that offered insights into the latest developments in the open-source world. Attending the conference provided the opportunity to connect with a diverse community of like-minded individuals, exchange ideas, and learn from experts in the field. The conference schedule was packed with informative sessions and workshops, and it was challenging to choose which talks to attend due to the variety of interesting topics. Nevertheless, the highlights of the conference provided new perspectives, fresh ideas, and valuable insights that will stay with me for a long time. I highly recommend FOSDEM to anyone interested in the open-source community and technology.

Tips

Be Selective: With so many high-quality sessions and workshops to choose from, it's important to prioritize and select the talks that align with your professional interests and goals
Be open-minded: FOSDEM provides a great opportunity to learn from others and exchange ideas, so be open-minded and willing to engage in discussions and debates with others.
Try Belgium Beer: You should try a few of the Tripel Beers from Belgium