From 508527f52de524a4fd174d386808e314b4138b11 Mon Sep 17 00:00:00 2001 From: alex emery Date: Sun, 3 Nov 2024 15:33:28 +0000 Subject: feat: static builds --- public/embed.go | 6 + public/ethos.html | 91 +++++++++++++ public/index.html | 97 ++++++++++++++ public/posts.html | 63 +++++++++ public/posts/2024-08-25-01.html | 117 +++++++++++++++++ public/posts/2024-08-25-02.html | 153 ++++++++++++++++++++++ public/posts/2024-08-25-03.html | 78 +++++++++++ public/posts/2024-08-26-01.html | 280 ++++++++++++++++++++++++++++++++++++++++ public/posts/2024-08-31-01.html | 96 ++++++++++++++ public/posts/2024-09-04-01.html | 165 +++++++++++++++++++++++ public/posts/2024-09-08-01.html | 265 +++++++++++++++++++++++++++++++++++++ 11 files changed, 1411 insertions(+) create mode 100644 public/embed.go create mode 100644 public/ethos.html create mode 100644 public/index.html create mode 100644 public/posts.html create mode 100644 public/posts/2024-08-25-01.html create mode 100644 public/posts/2024-08-25-02.html create mode 100644 public/posts/2024-08-25-03.html create mode 100644 public/posts/2024-08-26-01.html create mode 100644 public/posts/2024-08-31-01.html create mode 100644 public/posts/2024-09-04-01.html create mode 100644 public/posts/2024-09-08-01.html (limited to 'public') diff --git a/public/embed.go b/public/embed.go new file mode 100644 index 0000000..43a129b --- /dev/null +++ b/public/embed.go @@ -0,0 +1,6 @@ +package public + +import "embed" + +//go:embed * +var FS embed.FS diff --git a/public/ethos.html b/public/ethos.html new file mode 100644 index 0000000..f142062 --- /dev/null +++ b/public/ethos.html @@ -0,0 +1,91 @@ + + + + + + + + + + a73x + + + + + +

a73x

+ high effort, low reward + + +

ethos

+

Yes, this site has an ethos.

+ +

This site was born out of the belief that the current state of the internet is wrong. +As things currently stand, the internet is predominantly controlled by big tech, but it is a decentralised system, a series of interconnected networked devices that communicate over a standardised protocol.

+ +

Users flock to singular points on the modern internet, controlled and governed by megacorps. These points are where you go to give up your privacy and digital rights, in exchange for a service.

+ +

Social media platforms offer a pretense of community. Google pretends it can help you navigate the vast web, serving you results relevant to what you’re looking for, funnelling you to whoever is willing to pay the most to appear at the top of that list, and tracking you for possibilities of targeted advertising.

+ +

You’re digital data is farmed and harvested for surveillance capitalism because they couldn’t work out how else to squeeze out more money.

+ +

So this is my pièce de résistance. It represents my part of the internet, a small space subsumed by big tech’s centralised web.

+ +

I believe that a small amount of tech literacy would go a long way. We don’t need to commune on centralised platforms; rather, we can set up our own and own them.

+ +

This site runs on a single-node Kubernetes cluster hosted on an old Lenovo M910Q. It runs an i5-6500T (4 core) and has 8GB of RAM. +I bought it in 2021 for £189.99.

+ +

If I check this online against AWS, an equivalent machine might be a c6g.xlarge (4vcpu 8GiB) with an EC2 Instance Savings Plan; for three years, it would have cost $1597.82 upfront. I’ve ignored the cost of electricity. I have some breathing room, but this little device has already saved me considerable money. I’ve used it to host quite a few services over the years such as Tiny Tiny RSS,Miniflux, Ergochat, vaultwarden, Immich, NextCloud, Matrix, cigt, etc; the list goes on.

+ +

Currently, it only runs Miniflux, ergochat, cgit, and this site.

+ +

Someone somewhere might be angrily shaking their fist, declaring that this is no way to run a “production” environment. +I have no data backup or failover nodes — and I’d agree. +Miniflux exports its content as an XML, IRC is ephemeral, and this site is git versioned on sr.ht.

+ +

I don’t need my RSS reader to be highly available; if it’s broken, I’ll fix it.

+ +

For things to be cost-practical, I would have to sink down to a t2.micro (1 core 1GiB), which costs $152 for 3 years, but then I’d only have 1 core and 1GiB.

+ +

And whilst I tend to my flock of a single node server, I learn more about the system. Kubernetes, firewalls, DNS, DHCP, etc., a responsibility delegated away or hidden behind interfaces when using cloud services.

+ +

Using your equipment is cost-effective, leaving you room to trial services you’re interested in. You can also simply remove them when they’re no longer required.

+ +

As for the site itself I’ve chosen a minimal design, keeping javascript to a minimum and serving only static content. +I want the site to work consistently across all browsers, be readable on all devices, and continue to work into the future.

+ +

The aim here is to build a place to share the things I’ve learnt. This space serves as an attempt to counter the low-effort writings found on Medium and write more human content as we see more AI-generated drivel. Over time, I intend to curate my posts, merging, filtering, and ultimately removing irrelevant ones.

+ +

I work as a cloud developer, writing primarily in go (this might explain my chosen minimal design choice), so the content I write through that lens.

+ +

I intend to provide information about how one can go about setting up a site like this and administering it yourself. +As a go developer, I’ll also include things I’ve learned and am learning.

+ + + + + + + diff --git a/public/index.html b/public/index.html new file mode 100644 index 0000000..ecaf361 --- /dev/null +++ b/public/index.html @@ -0,0 +1,97 @@ + + + + + + + + + + a73x + + + + + +

a73x

+ high effort, low reward + + +

home

+

Welcome!

+ +

me

+ + + +

tidbits

+ +

#go

+ + + +

#git

+ + + +

resources

+ + + +

books

+ + + + + + + + + diff --git a/public/posts.html b/public/posts.html new file mode 100644 index 0000000..d6680fb --- /dev/null +++ b/public/posts.html @@ -0,0 +1,63 @@ + + + + + + + + + + a73x + + + + + +

a73x

+ high effort, low reward + + +

posts

+ + + + + + + + diff --git a/public/posts/2024-08-25-01.html b/public/posts/2024-08-25-01.html new file mode 100644 index 0000000..4bd4371 --- /dev/null +++ b/public/posts/2024-08-25-01.html @@ -0,0 +1,117 @@ + + + + + + + + + + a73x + + + + + +

a73x

+ high effort, low reward + + +

Go Benchmarking

+ +

The benchmark cycle:

+ +
    +
  1. write a benchmark
    2. +
  2. run a benchmark
    3. +
  3. get a profile
    4. +
  4. optimise
    5. +
  5. run your tests
    6. +
  6. goto 2.
    7. +
+ +

cpuprofile

+
go test -test=XXX -bench <regex> -cpuprofile <file>
+
+

memprofile

+
go test -test=XXX -bench <regex> -memprofile <file> -benchmem
+
+

pprof

+ +

pprof usage

+
go pprof -http=:8080 profile.pb.gz
+
+

will show a web UI for analysing the profile.

+ +

views

+ + + + + + + + + diff --git a/public/posts/2024-08-25-02.html b/public/posts/2024-08-25-02.html new file mode 100644 index 0000000..81e8f77 --- /dev/null +++ b/public/posts/2024-08-25-02.html @@ -0,0 +1,153 @@ + + + + + + + + + + a73x + + + + + +

a73x

+ high effort, low reward + + +

Go Project Layouts

+

Do you lay awake at night and consider how to optimally layout your Go project? +No…? what about recommending Windows to a friend or colleague?? +yeah me either…

+ +

I’ve seen a lot online that shows what I can only describe as endgame enterprise Go project layouts. These layouts are heavily folder-based and only make sense when your project has grown large enough to warrant the verbosity these folders provide. My only problem is that people often try to start there.

+ +

A lot of design tells you to think about your project in layers.

+ + + +

If you read The Clean Architecture +you get told the layers should be,

+ +
    +
  1. entities
    2. +
  2. use cases
    3. +
  3. interface adapters
    4. +
  4. frameworks and drivers.
    5. +
+ +

and that all dependencies should point in (yeah I know, I didn’t do a circle so “in” doesn’t make sense but I’m sure you can follow).

+ +

This is an excellent idea; separation of concerns is good.

+ +

So you make your folders.

+
.
+├── drivers
+├── entities
+├── interfaces
+└── usecases
+
+

aaand this is an awful idea. I don’t even want to go further into this hypothetical layout because it hurts too much.

+ +

Find me a project that actually creates these folders, and I’ll find you the medium article titled “Clean Code in Go” which spawned it.

+ +

The important parts of clean code, are the ideas presented, and how you apply them to a package orientated language. Creating a folder to represent each layer, doesn’t really carry much weight here.

+ +

As a package orientated language, we want to think and reason about things in terms of packages. Yes there will be a point where you may want to group your packages into some group, but that is mostly ceremonial. +Go doesn’t care if you’re accessing domain/bar or domain/foo/bar. Either will simply be accessed as bar. This means that what matters what’s in that package bar. Since everything will be read as bar.Thing i.e import bytes and bytes.Buffer.

+ +

So, the package name sets context and expectations. If I grab the json package, I expect that package to do things around json. I’d feel a bit confused if I was able to configure an smtp server.

+ +

If you cannot come up with a package name that’s a meaningful prefix for the package’s contents, the package abstraction boundary may be wrong

+ +

“but you’ve still not provided a good example?” +well +yes

+ +

I think the project should grow organically to some degree. What we want to do is write code, and refactoring in Go is fairly cheap.

+ +

Start with a main.go and make a Run function or some equivalent which it calls.

+
func Run() error {
+	// actual important stuff here
+}
+func main() {
+	if err := Run(); err != nil {
+		log.Fatal(err)
+	}
+}
+
+

This allows you to test your run function in a unit test, and keeps your main func minimal.

+ +

As your project grows, you can keep it flat inside the root directory

+
├── api.go
+├── go.mod
+├── go.sum
+├── main.go
+├── rss.go
+└── sqlite.go
+
+

Even just glancing at that, you can guess that this might be an RSS server, that uses sqlite to back it.

+ +

Who knows what

+
├── drivers
+├── entities
+├── interfaces
+└── usecases
+
+

does.

+ +

As things evolve you might want to put them in internal to hide them from being imported by other packages, or cmd as you develop multiple binaries. Placing things in internal means you’re free to mess around with it, without breaking any public contracts other users rely on. +I can’t be bothered rewriting my example, so here’s a random one I found online; it’s probably all right. +Server Project

+
project-root-directory/
+  go.mod
+  internal/
+    auth/
+      ...
+    metrics/
+      ...
+    model/
+      ...
+  cmd/
+    api-server/
+      main.go
+    metrics-analyzer/
+      main.go
+    ...
+  ... the project's other directories with non-Go code
+
+

My vague summary is that clean code gives you a north star to follow, an idea of how you want to separate and reason about the packages you create. You don’t need to create the entities of abstraction that are also presented. Think about what things do or relate to and create packages for them. You should allow your project to grow organically but don’t expect architecture to appear without following a north star.

+ + + + + + + diff --git a/public/posts/2024-08-25-03.html b/public/posts/2024-08-25-03.html new file mode 100644 index 0000000..07e48a0 --- /dev/null +++ b/public/posts/2024-08-25-03.html @@ -0,0 +1,78 @@ + + + + + + + + + + a73x + + + + + +

a73x

+ high effort, low reward + + +

Levels of Optimisation

+ +

This probably isn’t strictly true, but it makes sense to me. +We’ve got three levels of “optimisation” (assuming your actual design doesn’t suck and needs optimising).

+ +

Benchmark Optimisation

+ +

To begin with, we have benchmark optimisation; you create a benchmark locally, dump a profile of it, and optimise it. Then, you run your tests because the most optimal solution is “return nil” and make sure you didn’t break your tests. +This is the first and easiest optimisation because it only requires a function, nothing else, and can be done in isolation. You don’t need a working “application” here, just the function you’re trying to benchmark. There are different types of benchmarks, micro, macro, etc., but I’m leaving them out of scope for this conversation. Go read Efficient Go.

+ +

Profile guided optimisation

+ +

This is a mild step up from benchmark optimisation only because you need a live server load from which you use to pull a profile, but it is probably the most hands-off step. You import the net/http/pprof package into your service, call the debug/profile?seconds=30 to get a profile, and compile your binary with go build -pgo=profile.pgo. The compiler will make optimisations for you, and even if your profile is garbage, it shouldn’t cause any regressions.

+ +

You probably want to get a few profiles and merge them using go tool pprof -proto a.out b.out > merged. This will help provide optimisations that are more relevant to your overall system; instead of just a single 30s slice. +Also, if you have long-running calls that are chained together, a 30-second snapshot might not be enough, so try a sample with a longer window.

+ +

Runtime optimisation

+ +

This is where you expose /runtime/metrics and monitor them continuously. There’s a list of metrics that you might be interested in, a recommended set of metrics, and generally, you are looking to optimise your interactions with the go runtime. There are a few stats here: goroutine counts, goroutines waiting to run, heap size, how often garbage collection runs, how long garbage collection takes, etc. All useful information to use when optimising - when garbage collection is running, your program ain’t. It’s also useful for finding memory leaks; it becomes pretty obvious you are leaking goroutines when you graph the count and just watch it go up and never down. +It’s also just lowkey fun to look at the exposed data and understand what your system is doing.

+ + + + + + + diff --git a/public/posts/2024-08-26-01.html b/public/posts/2024-08-26-01.html new file mode 100644 index 0000000..acf51a6 --- /dev/null +++ b/public/posts/2024-08-26-01.html @@ -0,0 +1,280 @@ + + + + + + + + + + a73x + + + + + +

a73x

+ high effort, low reward + + +

Writing HTTP Handlers

+

I’m sharing how I write handlers in Go.

+ +

I write them like this for reasons that are probably fairly contextual. I’ve written a few applications and had to swap REST libraries or even swapped REST for GRPC, so things that make that easier speak to me a great deal.

+ +

I’ve used ints instead of the http.StatusXXXX and omitted JSON tags in an attempt to try save up screen space.

+ +

To begin with, you might have something like this:

+
package main
+
+import (
+	"fmt"
+	"log"
+	"net/http"
+)
+
+func handler(w http.ResponseWriter, r *http.Request) {
+	fmt.Fprintf(w, "Hello, World!")
+}
+
+func main() {
+	http.HandleFunc("/", handler)
+	log.Fatal(http.ListenAndServe(":8080", nil))
+}
+
+
+

Then you might get told off because you’ve just registered routes with the default mux, which isn’t very testable.

+ +

So you tweak it a little bit.

+
package main
+
+import (
+	"fmt"
+	"log"
+	"net/http"
+)
+
+func handler(w http.ResponseWriter, r *http.Request) {
+	fmt.Fprintf(w, "Hello, World!")
+}
+
+func newMux() *http.ServeMux {
+	mux := http.NewServeMux()
+	mux.HandleFunc("/", handler)
+
+	return mux
+}
+
+func Run() error {
+	mux := newMux()
+	return http.ListenAndServe(":8080", mux)
+}
+
+func main() {
+	if err := Run(); err != nil {
+		log.Fatal(err)
+	}
+}
+
+

newMux() gives you a mux to use when testing.

+ +

Run keeps main nice and clean, so you can just return errors as needed instead of going log.Fatal and just generally being messy.

+ +

But now you need to do something real, you want to store and fetch data.

+
package main
+
+import (
+	"encoding/json"
+	"fmt"
+	"log"
+	"net/http"
+	"strconv"
+)
+
+func NewMux() *http.ServeMux {
+	mux := http.NewServeMux()
+	s := Server{
+		data: make(map[int]Content),
+	}
+
+	s.Register(mux)
+	return mux
+}
+
+func Run() error {
+	mux := NewMux()
+	return http.ListenAndServe(":8080", mux)
+}
+
+type Server struct {
+	data map[int]Content
+}
+
+func (s *Server) Register(mux *http.ServeMux) {
+	mux.HandleFunc("GET /{id}", s.Get)
+	mux.HandleFunc("POST /", s.Post)
+}
+
+func (s *Server) Get(w http.ResponseWriter, r *http.Request) {
+	idStr := r.PathValue("id")
+	id, err := strconv.Atoi(idStr)
+	if err != nil {
+		w.WriteHeader(400)
+		w.Write([]byte(fmt.Sprintf("failed to parse id: %v", err)))
+		return
+	}
+	data, ok := s.data[id]
+	if !ok {
+		w.WriteHeader(404)
+		w.Write([]byte("not found"))
+		return
+	}
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(200)
+	json.NewEncoder(w).Encode(data)
+}
+
+type ContentPostReq struct {
+	Foo string
+}
+
+func (s *Server) Post(w http.ResponseWriter, r *http.Request) {
+	req := ContentPostReq{}
+	if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+		w.WriteHeader(400)
+		w.Write([]byte(fmt.Sprintf("failed to parse request: %v", err)))
+		return
+	}
+	id := len(s.data)
+	content := Content{
+		ID:  id,
+		Foo: req.Foo,
+	}
+	s.data[id] = content
+
+	w.WriteHeader(200)
+	json.NewEncoder(w).Encode(content)
+}
+
+type Content struct {
+	ID  int
+	Foo string
+}
+
+func main() {
+	if err := Run(); err != nil {
+		log.Fatal(err)
+	}
+}
+
❯ curl -X POST localhost:8080 --header "Content-Type: application/json" -d '{"foo":"bar"}'
+{"ID":0,"Foo":"bar"}
+❯ curl -X GET localhost:8080/0
+{"ID":0,"Foo":"bar"}
+
+

Erm, well, okay. Quite a bit has changed here, but I’m sure you can read it. We now save and fetch very, very safely from a map and return the response as JSON. I’ve done some things for brevity because I want to get to the main point.

+ +

This API is inconsistent. It sometimes returns JSON, and the others return strings. Overall, it’s just a mess.

+ +

So let’s try to standardise things. +First, let’s design some form of REST spec.

+
type JSONResp[T any] struct {
+	Resources []T
+	Errs      []ErrorResp
+}
+
+type ErrorResp struct {
+	Status int
+	Msg    string
+}
+
+

We want to be able to support fetching multiple resources at once, if we can only fetch some resources, let’s return them under resources and show the errors under errs

+ +

Now, add some helpful functions to handle things.

+
func Post[In any, Out any](successCode int, fn func(context.Context, In) ([]Out, []ErrorResp)) func(http.ResponseWriter, *http.Request) {
+	return func(w http.ResponseWriter, r *http.Request) {
+		var v In
+
+		if err := json.NewDecoder(r.Body).Decode(&v); err != nil {
+			writeJSONResp[Out](w, http.StatusBadRequest, nil, []ErrorResp{
+				{
+					Status: http.StatusBadRequest,
+					Msg:    fmt.Sprintf("failed to parse request: %v", err),
+				},
+			})
+
+			return
+		}
+
+		res, errs := fn(r.Context(), v)
+		writeJSONResp(w, successCode, res, errs)
+	}
+}
+
+func writeJSONResp[T any](w http.ResponseWriter, successCode int, res []T, errs []ErrorResp) {
+	body := JSONResp[T]{
+		Resources: res,
+		Errs:      errs,
+	}
+
+	status := successCode
+	for _, e := range errs {
+		if e.Status > status {
+			status = e.Status
+		}
+	}
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(status)
+	json.NewEncoder(w).Encode(body)
+}
+
+

And we’ve standardised all POST requests!

+ +

This function can be used by all POST requests, ensuring they adhere to the spec. It also removes the repetitive code around marshalling and unmarshalling to JSON and handles errors in a consistent manner. +The handler functions accept a context param and their expected struct input.

+
func (s *Server) Register(mux *http.ServeMux) {
+...
+	mux.HandleFunc("POST /", Post(201, s.Post))
+}
+
+func (s *Server) Post(ctx context.Context, req ContentPostReq) ([]Content, []ErrorResp) {
+	id := len(s.data)
+	content := Content{
+		ID:  id,
+		Foo: req.Foo,
+	}
+	s.data[id] = content
+
+	return []Content{content}, nil
+}
+
+

As you can see, the post function is fairly cleaner now.

+ +

You can extend this to all the other request types. If you have query or path parameters, you could either pass in the request, write a custom struct tag parser, or find someone else who has already done it: https://github.com/gorilla/schema.

+ + + + + + + diff --git a/public/posts/2024-08-31-01.html b/public/posts/2024-08-31-01.html new file mode 100644 index 0000000..68d8563 --- /dev/null +++ b/public/posts/2024-08-31-01.html @@ -0,0 +1,96 @@ + + + + + + + + + + a73x + + + + + +

a73x

+ high effort, low reward + + +

Go's unique pkg

+

https://pkg.go.dev/unique

+ +
+

The unique package provides facilities for canonicalising (“interning”) comparable values.1

+
+ +

oh yeah, thats obvious I fully understand what this package does now, great read, tune in for the next post.

+ +

Interning, is the re-use of an object of equal value instead of creating a new one. I’m pretending I knew this but really I’ve just reworded Interning.

+ +

So lets try again.

+ +

If you’re parsing out a csv of bank transactions, its very likely a lot of names will be repeated. Instead of allocating memory for each string representing a merchant, you can simply reuse the the same string.

+ +

So the dumbed down version might look like

+
var internedStrings sync.Map
+
+func Intern(s string) string {
+	if val, ok := internedStrings.Load(s); ok { 
+		return val.(string) 
+	} 
+	internedStrings.Store(s, s) 
+	return s 
+}
+
+

With a small demo here https://go.dev/play/p/piSYjCHIcLr

+ +

This implementation is fairly naive, it can only grow and it only works with strings, so naturally go’s implementation is a better.

+ +

It’s also worth noting, that since strings are a pointer under the hood

+ +
+

When comparing two strings, if the pointers are not equal, then we must compare their contents to determine equality. But if we know that two strings are canonicalized, then it is sufficient to just check their pointers.2

+
+ +

So to recap, goes unique package provides a way to reuse objects instead of creating new ones, if we consider the objects of equal value.

+ +
+ +
+ +
    +
  1. https://pkg.go.dev/unique [return]
    2. + +
  2. https://go.dev/blog/unique [return]
    3. +
+ +
+ + + + + + + diff --git a/public/posts/2024-09-04-01.html b/public/posts/2024-09-04-01.html new file mode 100644 index 0000000..2e1abae --- /dev/null +++ b/public/posts/2024-09-04-01.html @@ -0,0 +1,165 @@ + + + + + + + + + + a73x + + + + + +

a73x

+ high effort, low reward + + +

Kubernetes Intro

+ +

My crash course to Kubernetes. +You’re welcome.

+ +

Pods

+ +

In Kubernetes, if you wish to deploy an application the most basic component you would use to achieve that, is a pod. A Pod represents the smallest deployable unit in Kubernetes, encapsulating one or more containers that need to work together. While containers run the actual application code, Pods provide the environment necessary for these containers to operate, including shared networking and storage.

+ +

A Pod usually represents an ephemeral single instance of a running process or application. For example, a Pod might contain just one container running a web server. In more complex scenarios, a Pod could contain multiple containers that work closely together, such as a web server container and a logging agent container.

+ +

Additionally we consider a Pod as ephemeral because when a Pod dies, it can’t be brought back to life—Kubernetes would create a new instance instead. This behaviour reinforces the idea that Pods are disposable and should be designed to handle failures gracefully.

+ +

When you use Docker, you might build a image with docker build . -t foo:bar and run a container with docker run foo:bar. In Kubernetes, to run that same container, you place it inside a Pod, since Kubernetes manages containers through Pods.

+
apiVersion: v1
+kind: Pod
+metadata:
+  name: <my pod name here> 
+spec:
+  containers:
+  - name: <my container name here> 
+    image: foo:bar
+
+

In this YAML manifest, we define the creation of a Pod using the v1 API version. The metadata field is used to provide a name for identifying the Pod within the Kubernetes cluster. Inside the spec, the containers section lists all the containers that will run within that Pod.

+ +

Each container has its own name and image, similar to the --name and image parameters used in the docker run command. However, in Kubernetes, these containers are encapsulated within a Pod, ensuring that they are always co-scheduled, co-located, and share the same execution context.

+ +

As a result, containers within a Pod should be tightly coupled, meaning they should work closely together, typically as parts of a single application or service. This design ensures that the containers can efficiently share resources like networking and storage while providing a consistent runtime environment.

+ +

Why Multiple Containers in a Pod?

+ +

Sometimes, you might need multiple containers within a single Pod. Containers in a Pod share the same network namespace, meaning they can communicate with each other via localhost. They also share storage volumes, which can be mounted at different paths within each container. This setup is particularly useful for patterns like sidecars, where an additional container enhances or augments the functionality of the main application without modifying it.

+ +

For example, imagine your application writes logs to /data/logs. You could add a second container in the Pod running fluent-bit, a tool that reads in files and sends them to a user defined destination. fluent-bit reads these logs and forwards them to an external log management service, without changing the original application code. This separation also ensures that if the logging container fails, it won’t affect the main application’s operation.

+ +

When deciding what containers go in a pod, consider how they’re coupled. Questions like “how should these scale” might be helpful. If you have two containers, one for a web server and one for a database, as your web server traffic goes up, it doesn’t really make sense to start creating more instances of the database. So you would put your web server in one pod and your database in another, allowing Kubernetes to scale them independently. On the other hand a container which shares a volume with the web server would need to scale on a 1:1 basis, so they go in the same pod.

+ +

Pod Placement

+ +

When a Pod is created, Kubernetes assigns it to a Node—a physical or virtual machine in the cluster—using a component called the scheduler. The scheduler considers factors like resource availability, node labels, and any custom scheduling rules you’ve defined (such as affinity or anti-affinity) when making this decision. Affinity means the pods go together, anti-affinity means keep them on separate nodes. Other rules can be used to direct Pods to specific Nodes, such as ensuring that GPU-based Pods run only on GPU-enabled Nodes.

+ +

Scaling

+ +

In practise, you won’t be managing pods manually. If a pod crashes, manual intervention would be required to start a new Pod and clean up the old one. Fortunately, Kubernetes provides controllers to manage Pods for you: Deployments, StatefulSets, DaemonSets, and Jobs.

+ + + +

Deployments and StatefulSets also support scaling mechanisms, allowing you to increase or decrease the number of Pods to handle varying levels of traffic.

+ +

Services

+ +

As your application scales and you handle multiple Pods, you need a way to keep track of them for access. Since Pods can change names and IP addresses when they are recreated, you need a stable way to route traffic to them. This is where Kubernetes services come into play.

+ +

Services provide an abstraction layer that allows you to access a set of Pods without needing to track their individual IP addresses. You define a selector in the Service configuration and traffic reaching the Service is routed to one of the Pods matching the selector.

+ +

There are four types of services in Kubernetes: ClusterIP, NodePort, LoadBalancer, and ExternalName.

+ + + +

Volumes

+ +

Broadly speaking, Kubernetes offers two types of storage: ephemeral and persistent volumes.

+ + + +

Understanding storage in Kubernetes can be a bit complex due to its abstraction and reliance on third-party controllers. Kubernetes uses the Container Storage Interface (CSI), a standardised specification that allows it to request storage from different providers, which then manage the lifecycle of the underlying storage. This storage could be anything from a local directory on a node to an AWS Elastic Block Store (EBS) volume. Kubernetes abstracts the details and relies on the CSI-compliant controller to handle the specifics.

+ +

Key Components of Kubernetes Storage

+ +

There are three main components to understand when dealing with storage in Kubernetes: Storage Classes, PersistentVolumes (PVs), and PersistentVolumeClaims (PVCs).

+ +
    +
  1. Storage Class: A Storage Class defines the type of storage and the parameters required to provision it. Each Storage Class corresponds to a specific storage provider or controller. For example, a Storage Class might define a template for AWS EBS volumes or Google Cloud Persistent Disks.
    2. +
  2. PersistentVolume (PV): A PersistentVolume represents a piece of storage in the cluster that has been provisioned according to the specifications in a Storage Class. PVs can be either statically created by a cluster administrator or dynamically provisioned by a controller. For instance, when a Storage Class is associated with AWS, the controller might create an EBS volume when a PV is needed.
    3. +
  3. PersistentVolumeClaim (PVC): A PersistentVolumeClaim is a user’s request for storage. It specifies the desired size, access modes, and Storage Class. When a PVC is created, Kubernetes will find a matching PV or trigger the dynamic provisioning of a new one through the associated Storage Class and controller. Once a PV is provisioned, it becomes bound to the PVC, ensuring that the requested storage is dedicated to that specific claim.
    4. +
+ +

How It Works Together

+ +

The typical workflow involves a user creating a PersistentVolumeClaim to request storage. The CSI controller picks up this request and, based on the associated Storage Class, dynamically provisions a PersistentVolume that meets the user’s specifications. This PersistentVolume is then bound to the PersistentVolumeClaim, making the storage available to the Pod that needs it.

+ + + + + + + diff --git a/public/posts/2024-09-08-01.html b/public/posts/2024-09-08-01.html new file mode 100644 index 0000000..72cde61 --- /dev/null +++ b/public/posts/2024-09-08-01.html @@ -0,0 +1,265 @@ + + + + + + + + + + a73x + + + + + +

a73x

+ high effort, low reward + + +

Building a Static Site with Hugo and Docker

+ +

This will be a quick walkthrough of how to create a static site using Hugo, and use Nginx to serve it.

+ +

Prerequisites

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SkillDescription
Basic Terminal UsageFamiliarity with navigating directories and running basic commands in a terminal.
GitAbility to initialize a Git repository, commit changes, and interact with remote repositories.
MarkdownKnowledge of writing basic content in Markdown format (used for posts).
Docker BasicsUnderstanding of Docker commands for building images and running containers.
HTML/CSS BasicsGeneral awareness of HTML and CSS for customising static site content.
GoGo should be installed on your system to use Hugo with the go install method. Alternatively, you can download Hugo binaries directly or use a package manager.
+ +

Step 1: Installing Hugo

+ +

Hugo is a static site generator, meaning it builds HTML, CSS, and JavaScript that doesn’t need a backend server, since the website’s content is static.

+ +

You can install Hugo in multiple ways. If you already have Go installed, you can use

+
go install github.com/gohugoio/hugo@latest
+
+

Alternatively, you can install Hugo following the official install guide>

+ +

Step 2: Creating a New Hugo Site

+ +

To create a new Hugo site, run:

+
hugo new site website
+
+

This creates a new folder called website with the basic structure of a Hugo site.

+ +

Step 3: Setting Up a Theme

+ +

By default, Hugo doesn’t ship with any themes installed, so its likely you’ll want to add one. A list of pre-made themes exist here, which saves you from having to create one from scratch. Typically, this involves using a Git submodule to manage the theme:

+
cd website
+git init
+git submodule add --depth=1 https://github.com/adityatelange/hugo-PaperMod.git themes/PaperMod
+
+

A Git submodule is a way to link a separate repository (the theme) into your project without copying it directly. This keeps the theme up-to-date and lets you manage it separately from your main website’s content.

+ +

To use the theme, add it to your config.toml file:

+
echo 'theme = ["PaperMod"]' >> hugo.toml
+
+

Alternatively, you could manually download the theme and place it in the themes folder, but using submodules allows for easier updates.

+ +

Step 4: Personalising Your Site

+ +

Open config.toml in your favorite code editor (e.g., VS Code), and change the title line to peronsalise your site’s name:

+
title = "<insert name>'s blog"
+
+

This will update the title of your site, which you’ll see when we generate the site in a moment.

+ +

Step 5: Creating a New Post

+ +

To create a new post, run the following command:

+
hugo new content content/posts/my-first-post.md
+
+

This will create a new file in the content/posts directory, named my-first-post.md. When you open it, you’ll see:

+
+++
+title = 'My First Post'
+date = 2024-09-08T15:44:30+01:00
+draft = true
++++
+
+

The block of text wrapped in +++ is called “front matter” and acts as metadata for your post. It won’t be visible in your generated website, the actual content of your post goes below this.

+ +

Now, you can edit the file to include your first post:

+
+++
+title = 'My First Post'
+date = 2024-09-08T15:44:30+01:00
+draft = true
++++
+## Welcome! 
+
+This is my first post on my new site. It's written in markdown and utilises hugo for generating html which browsers understand and can parse.
+
+Visit the [Hugo](https://gohugo.io) website!
+
+

Step 6: Previewing Your Website

+ +

To preview your website locally, run the following command:

+
hugo server --buildDrafts
+
+

This will start a local server, and you can view your site by visiting http://localhost:1313 in your browser.

+ +

Step 7: Publishing the Website

+ +

Once you’re ready to publish, update your post by changing draft = true to draft = false (or just delete the draft property) and run:

+
hugo
+
+

This will build your site and place the generated files in the public folder. This folder contains all the HTML, CSS, and JavaScript that make up your static site.

+ +

From here you can deploy it following [Hugo’s own guide](https://gohugo.io/categories/hosting-and-deployment/. However, most of these options involve using someone else’s compute, and our goal here is self hosting.

+ +

Step 8: Dockerising your Site

+ +

Now that you have a static site in the public folder, let’s create a Docker image that will serve your site using Nginx, a lightweight web server. While you could install Nginx directly on a server and follow this guide to deploy your site, using Docker offers several advantages. Containers provide a reproducible, isolated environment that makes it easy to package, run, and deploy your site across different systems. So, let’s use Docker to handle serving your site instead.

+ +

Create a Dockerfile with the following content

+
FROM nginx:1.27.1
+COPY public /usr/share/nginx/html
+
+

This tells Docker to use the official Nginx image and copy the files from your public folder (which Hugo generated) into the default location that Nginx serves from.

+ +

Step 9: Building and Running the Docker Image

+ +

To proceed, you’ll need a container registry. We’ll use Docker hub for this example (you’ll need an account) but you can use whatever container registry you have access to.

+ +

To build your Docker image, run:

+
docker build . -t <dockerusername>website:0.0.1
+
+

Here:

+ + + +

Now that you have the Docker image locally, you can run it using

+
docker run -p 8080:80 <dockerusername>website:0.0.1
+
+

Here, -p 8080:80 maps port 8080 on your local machine to port 80 in the Docker container, where Nginx serves the content.

+ +

Now, open a browser and go to http://localhost:8080. You should see your static site served by Nginx!

+ +

But your server is not your local machine (potentially), so you need a method of getting the image from your local machine, to your server. We can use container registries as an intermediary.

+ +
    +
  1. First login to Docker Hub:
    2. +
+
docker login
+
+
    +
  1. Push your image to Docker Hub, by default this image will be public.
    2. +
+
`docker push <dockerusername>website:0.0.1
+
+
    +
  1. SSH into the server
    2. +
+
ssh user@server-ip
+
+
    +
  1. Pull the Image (we don’t need to login since the image is public):
    2. +
+
docker <dockerusername>website:0.0.1
+
+
    +
  1. Run the Docker Image
    2. +
+
docker run -d -p 80:80 <dockerusername>website:0.0.1
+
+ + +

You will be able to access the website by visiting http://server-ip:80

+ +

Conclusion

+ +

Congratulations on creating and running your first static website with Hugo and Docker!

+ + + + + + + -- cgit v1.2.3