Showing posts with label Go. Show all posts
Showing posts with label Go. Show all posts

Tuesday, 18 August 2026

OS core, OS thread, process, go process, go routine

 

 Binary vs process

A Go binary sitting on disk is just a file:

certgen-service

When you run it:

./certgen-service

the OS creates a process.

Go binary
↓ run
Go process

A process is mainly a container for things like memory, files, sockets, and threads.


2. Every running process needs at least one OS thread

A generic process can be as simple as:

Process
└── OS Thread
↓
Logical CPU
↓
Physical CPU Core

The thread is the actual execution unit that the OS schedules.

The process itself is not what directly runs on a core.


3. Go adds another layer: goroutines

A Go process has the Go runtime inside it.

Go Process
│
├── Go Runtime
│
├── main goroutine
├── Goroutine A
├── Goroutine B
├── Goroutine C
├── runtime goroutines
│
└── OS Threads

Important point:

main() itself runs as the main goroutine.

Even if you never write:

go someFunction()

Go still has internal goroutines for runtime work such as GC and scheduling.


4. Goroutine is not an OS thread

A goroutine is a lightweight user-space execution unit.

You can casually think:

goroutine ≈ lightweight thread

But technically:

Goroutine != OS Thread

Many goroutines are scheduled onto a smaller number of OS threads:

Go Process

Goroutine main ─┐
Goroutine A ────┤
Goroutine B ────┤
Goroutine C ────┤
Goroutine D ────┤
Goroutine E ────┘
↓
Go Scheduler
↓
┌───────────────┐
│ OS Thread 1 │
│ OS Thread 2 │
│ OS Thread 3 │
└───────────────┘

Creating:

go foo()

does not mean:

new goroutine
=
new OS thread

Usually it just creates another goroutine for the Go scheduler to manage.


5. One OS thread executes one goroutine at a time

Suppose several goroutines can use Thread 1:

OS Thread 1
├── main goroutine
├── Goroutine A
├── Goroutine B
└── Goroutine C

That does not mean all four run simultaneously on Thread 1.

It means over time:

Thread 1
↓
main runs
↓
A runs
↓
B runs
↓
main runs again
↓
C runs

Go switches which goroutine is running.

And goroutines are generally not permanently attached to a specific OS thread.

For example:

Time 1:

Thread 1 → Goroutine A
Thread 2 → Goroutine B


Time 2:

Thread 1 → Goroutine C
Thread 2 → Goroutine A

Goroutine A moved.


6. OS threads are tied to CPU execution

The OS schedules threads onto logical CPUs.

Goroutine
↓ Go scheduler
OS Thread
↓ OS scheduler
Logical CPU
↓
Physical Core

At one instant:

Logical CPU 1 → Thread A
Logical CPU 2 → Thread B
Logical CPU 3 → Thread C
Logical CPU 4 → Thread D

So if you have 4 logical CPUs, roughly 4 OS threads can actually execute instructions simultaneously.


7. Physical core vs logical CPU

Without SMT / Hyper-Threading:

Physical Core 1
└── Logical CPU 1
↓
one OS thread executing

With SMT / Hyper-Threading:

Physical Core 1
├── Logical CPU 1 → Thread A
└── Logical CPU 2 → Thread B

So a machine might say:

8 physical cores
16 logical CPUs

The OS generally schedules against those 16 logical CPUs.


Full Go picture

Putting everything together:

MACHINE
│
┌───────────┴───────────┐
│ │
Physical Core 1 Physical Core 2
├─ Logical CPU 1 ├─ Logical CPU 3
└─ Logical CPU 2 └─ Logical CPU 4
↑ ↑
│ OS Scheduler │
│ │
OS Thread 1 OS Thread 2
↑ ↑
│ Go Scheduler │
│ │
┌───────┴───────┐ ┌──────┴───────┐
│ │ │ │
main A B C
goroutine goroutine goroutine goroutine

all inside one Go process

The two schedulers are important:

Goroutines
↓
Go scheduler
↓
OS Threads
↓
OS scheduler
↓
Logical CPUs / cores

8. What happens when a goroutine launches OpenSSL

This is different because OpenSSL becomes a separate process.

Suppose you write:

go func() {
exec.Command("openssl", ...).Run()
}()

Conceptually:

Go Process
│
├── main goroutine
│
├── Goroutine A
│ │
│ └── exec.Command("openssl")
│ │
│ │ asks OS to launch
│ ↓
│
└── Go OS Threads


NEW PROCESS
↓

OpenSSL Process
└── OS Thread
↓
Logical CPU

The OpenSSL process has its own address space and own OS thread(s).

So:

Goroutine
↓
starts
↓
OpenSSL Process
↓
contains
↓
OpenSSL OS Thread
↓
CPU

The OpenSSL process is not an OS thread itself.


9. Your certificate service example

Say you have:

8 logical CPUs
1 Go certgen process
100 goroutines
20 OpenSSL commands running

You could conceptually have:

Machine
│
├── Logical CPU 1
├── Logical CPU 2
├── Logical CPU 3
├── Logical CPU 4
├── Logical CPU 5
├── Logical CPU 6
├── Logical CPU 7
└── Logical CPU 8
↑
│ OS schedules everything here
│
┌────────┴───────────────────────────────┐
│ │
│ Go Process │
│ ├── main goroutine │
│ ├── goroutine 1 │
│ ├── goroutine 2 │
│ ├── ... │
│ ├── goroutine 100 │
│ │ │
│ └── several Go OS threads │
│ │
├── OpenSSL Process #1 │
│ └── OS thread │
│ │
├── OpenSSL Process #2 │
│ └── OS thread │
│ │
├── OpenSSL Process #3 │
│ └── OS thread │
│ │
└── ... │
OpenSSL Process #20 │
└── OS thread │

All of those threads compete for the same CPU resources.

That's why creating:

1000 goroutines
+
1000 OpenSSL processes

does not mean you get 1000-way CPU parallelism.

If you only have 8 logical CPUs:

thousands of runnable threads/work
↓
OS scheduler
↓
8 logical CPUs

Only a limited amount can actually execute at once.


The shortest version to remember

Binary
↓ run
Process
↓ contains
OS Threads
↓ OS schedules onto
Logical CPUs
↓ provided by
Physical CPU cores

And for Go, insert one extra layer:

Go Process
↓ contains
Goroutines
↓ Go schedules onto
OS Threads
↓ OS schedules onto
Logical CPUs
↓
Physical CPU cores

And for your OpenSSL case:

Go goroutine
↓
exec.Command()
↓
NEW OpenSSL process
↓
OpenSSL OS thread(s)
↓
CPU

That entire model is the foundation for understanding why goroutine count, GOMAXPROCS, CPU cores, semaphores, and spawning many OpenSSL processes affect the performance of your certificate service.

Monday, 27 July 2026

GO OS process vs thread

## 1. Channels vs Goroutines vs Processes


These are 3 different things:


```javascript

┌─── OS Process (itemgen-service binary) ─────────────────────┐

│  ┌─ OS Thread 1 ────────────────────────────────────────┐   │

│  │  ┌─ Goroutine: HTTP handler ──┐                      │   │

│  │  │  ┌─ Goroutine: HTTP handler ┤                      │   │

│  │  │  │  ┌─ Goroutine: Worker ───┤                      │   │

│  │  │  │  │                         │                      │   │

│  │  │  │  │  ←── channel (in-memory) ──┘                  │   │

│  │  │  │  │      (just a queue, NOT a goroutine)          │   │

│  │  └──┴──┴────────────────────────┘                      │   │

│  └───────────────────────────────────────────────────────┘   │

│  ┌─ OS Thread 2 ────────────────────────────────────────┐   │

│  │  ┌─ Goroutine: scheduler ──┐                          │   │

│  │  └──────────────────────────┘                          │   │

│  └───────────────────────────────────────────────────────┘   │

└──────────────────────────────────────────────────────────────┘

```


- __Channel__: Just a thread-safe in-memory queue. No code runs "inside" it. It's a data structure, like a slice with a lock.

- __Goroutine__: A lightweight unit of execution (~2KB stack). NOT an OS process. Go runs thousands of goroutines on a few OS threads. Your app already uses goroutines (the scheduler, the item generation `sync.WaitGroup` workers, etc.)

- __Process__: The OS-level process — your running `itemgen-service` binary. There's only ONE process. Everything (channels, goroutines) lives inside it.



```javascript

$ ps aux | grep itemgen-service

jxiang  12345  ... itemgen-service    ← ONE process, always


$ htop (press H to toggle thread view)

12345 itemgen-service                 ← main thread

12346 itemgen-service                 ← OS thread (Go runtime)

12347 itemgen-service                 ← OS thread (Go runtime)

12348 itemgen-service                 ← OS thread (Go runtime)

```


They all share PID 12345 as the parent. The sub-IDs (12346, 12347...) are __threads__, not separate processes. They share the same memory space.


## Go's GMP model


```javascript

Goroutines (G)          OS Threads (M)         CPU Cores (P)

┌──────────────┐        ┌──────────────┐      ┌──────────┐

│ HTTP handler │──┐     │              │      │          │

│ HTTP handler │──┤     │  Thread 1    │◀────▶│  Core 1  │

│ Scheduler    │──┼──▶  │              │      │          │

│ DB sync      │──┤     ├──────────────┤      ├──────────┤

│ Worker       │──┤     │  Thread 2    │◀────▶│  Core 2  │

│ item gen #1  │──┘     │              │      │          │

│ item gen #2  │        └──────────────┘      └──────────┘

└──────────────┘

  ~1000 goroutines         ~4 threads (= CPU count)

```


- __Goroutines__: Your code spawns these with `go func()`. Thousands can exist. They're cheap (~2KB each).

- __OS Threads__: Go runtime creates these automatically. Default = number of CPU cores. These are what `htop` shows.

- __Processes__: Only ONE. Your binary. Always.


## How to verify


```bash

# Count OS processes (always 1):

ps -C itemgen-service -o pid= | wc -l

# Output: 1


# Count OS threads (several):

ps -C ite,gen-service -L -o tid= | wc -l

# Output: 4-8 (depending on CPU cores)


# Count goroutines (many):

# Run inside your Go code:

runtime.NumGoroutine()

# Output: 50-100+ (handlers, scheduler, worker, etc.)

```


## Why you might see "new process spawning"


Some monitoring tools (like Docker stats, certain dashboards) display __threads as if they were processes__. On Linux, `/proc/<pid>/task/` lists each thread with its own TID (thread ID), which can look like process IDs. But they're threads — they share memory, file descriptors, and the same PID.


Your app already spawns goroutines today (scheduler, item generation workers, DB sync) — you're just not noticing because the worker pattern is the same. Adding the notify worker is no different.


Shall I implement the SMTP notification plan now?



# Linux — count threads vs processes:

ps -eLf | grep itemgen-service | head

# Columns: UID PID PPID LWP C NLWP ...

# PID = process ID (same for all)

# LWP = thread ID (different for each)

# NLWP = number of threads


# Or simpler:

cat /proc/$(pgrep itemgen-service)/status | grep Threads

# Output: Threads: 7


# On Windows (your machine):

tasklist /fi "imagename eq itemgen-service.exe" /fo csv

# Shows 1 process


# PowerShell — count threads:

(Get-Process itemgen-service).Threads.Count

# Output: 7 (threads, not processes)


Thursday, 16 July 2026

timezone handling guide

 # Timezone Handling Guide


## 1. Best Practice: Store Time in UTC


**Keep all timestamps in UTC in the database.** This is the industry standard for servers.


| Component | Current Setting | Why It's Best |

|-----------|----------------|---------------|

| MySQL (Docker) | UTC (default) | No DST issues, universal, sync works across timezones |

| Go application | UTC (no `loc=` in DSN) | Consistent with MySQL, no conversion bugs |

| Source DB sync | UTC | `TIMESTAMP` stored as UTC internally, comparisons are correct |


**MySQL `TIMESTAMP` is always stored as UTC internally.** The display timezone only affects what you see, not what's stored.


---


## 2. Direct DB Query: Use `SET time_zone` Per Connection


When querying the database manually (MySQL CLI, Workbench, etc.):


```sql

-- Set timezone for this session

SET time_zone = 'America/Vancouver';


-- All TIMESTAMP columns now display in PST

SELECT * FROM sync_metadata;

-- Shows: 2026-07-16 11:31:37 (PST display)

-- Stored: 2026-07-16 18:31:37 (UTC — unchanged)


-- Query with PST times — MySQL converts automatically

SELECT * FROM sync_metadata WHERE last_run_at > '2026-07-16 11:00:00';

```


**Key points:**

- ✅ Display only — does NOT change stored UTC data

- ✅ Lasts for the entire session (all queries until you disconnect)

- ✅ Resets to UTC when connection closes

- ✅ Does NOT affect the Go application (separate connections)

- ⚠️ Only works for `TIMESTAMP` columns, not `DATETIME`


---


## 3. Why `SET time_zone` Can't Be Used in Go


Go uses a **connection pool** — connections are reused, not created/destroyed per query. If you `SET time_zone` on a connection, it **leaks** to the next function that gets the same connection.


### Connection Pool Diagram


```

┌─────────────────────────────────────────────────────────────────┐

│  Connection Pool (max 100 open, 50 idle, 300s lifetime)       │

│                                                                 │

│  ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐                 │

│  │ conn1  │ │ conn2  │ │ conn3  │ │  ...   │  ← 50 idle       │

│  │ (UTC)  │ │ (UTC)  │ │ (UTC)  │ │ (UTC)  │     (pre-warmed) │

│  └────────┘ └────────┘ └────────┘ └────────┘                 │

└─────────────────────────────────────────────────────────────────┘


Function A: SET time_zone on conn1

┌─────────────────────────────────────────────────────────────────┐

│                                                                 │

│  ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐                 │

│  │ conn1  │ │ conn2  │ │ conn3  │ │  ...   │                 │

│  │ (PST!) │ │ (UTC)  │ │ (UTC)  │ │ (UTC)  │                 │

│  └───┬────┘ └────────┘ └────────┘ └────────┘                 │

│      │                                                         │

│      └── Function A done, conn1 returned to pool              │

│          conn1 STILL has PST!                                  │

└─────────────────────────────────────────────────────────────────┘


Function B: gets conn1 from pool

┌─────────────────────────────────────────────────────────────────┐

│                                                                 │

│  ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐                 │

│  │ conn1  │ │ conn2  │ │ conn3  │ │  ...   │                 │

│  │ (PST!) │ │ (UTC)  │ │ (UTC)  │ │ (UTC)  │                 │

│  └───┬────┘ └────────┘ └────────┘ └────────┘                 │

│      │                                                         │

│      └── Function B gets conn1 — INHERITS PST! ← LEAKED!      │

│          B's query runs in PST, not UTC                        │

└─────────────────────────────────────────────────────────────────┘


conn1 stays PST until:

  - Connection recycled (after 300s lifetime)

  - Explicitly reset with SET time_zone = 'UTC'

  - Application restarts

```


### Pool Configuration (from `.env`)


```

DB_MAX_OPEN_CONNS=100       Max simultaneous connections

DB_MAX_IDLE_CONNS=50        Idle connections kept alive (pre-warmed)

DB_CONN_MAX_LIFETIME_SEC=300  Each connection recycled after 5 minutes

```


At rest: 50 idle connections waiting. Under load: up to 100 total. When load drops: extras closed back to 50.


### Why It Leaks


| What you think | What actually happens |

|---|---|

| Function ends → connection closed → timezone gone | Function ends → connection returned to pool → timezone persists |

| Next function gets fresh UTC connection | Next function might get the same PST connection |

| Each function is isolated | Timezone leaks for up to 300 seconds (connection lifetime) |


---


## 4. Solution: Convert Time in Go Code


### The Pattern


```

Input (PST) → Convert to UTC → Query DB (UTC) → Results (UTC) → Convert to PST → Display

```


### In Go Code


```go

// Load Vancouver timezone (handles PST/PDT automatically)

pstLoc, _ := time.LoadLocation("America/Vancouver")


// 1. Input: user wants logs after "2026-07-16 11:00:00" PST

pstTime, _ := time.ParseInLocation("2006-01-02 15:04:05", "2026-07-16 11:00:00", pstLoc)


// 2. Convert to UTC for query

utcTime := pstTime.UTC()  // 2026-07-16 18:00:00 UTC


// 3. Query DB with UTC — no SET time_zone needed, no pool contamination

rows, err := db.Query(`

    SELECT last_run_at, last_sync_time, last_run_status

    FROM sync_metadata

    WHERE last_run_at > ?

`, utcTime)


// 4. Read results (Go reads TIMESTAMP as UTC because no loc= in DSN)

for rows.Next() {

    var lastRunAt time.Time  // Go gives you UTC time

    var lastSyncTime time.Time

    var status string

    rows.Scan(&lastRunAt, &lastSyncTime, &status)


    // 5. Convert to PST for display

    pstRunAt := lastRunAt.In(pstLoc)  // 2026-07-16 11:31:37 PST


    fmt.Printf("Run at: %s, Status: %s\n",

        pstRunAt.Format("2006-01-02 15:04:05"), status)

}

```


### Why This Is Cost-Free


`time.Time.In(loc)` doesn't do any calculation — it just attaches the timezone location to the existing time value. The actual time moment doesn't change, only how it's displayed.


| Operation | Time per row | Comparison |

|---|---|---|

| `rows.Scan()` (parse from MySQL) | ~1-2 microseconds | 1,000x slower than conversion |

| Network I/O (fetching the row) | ~50-100 microseconds | 50,000x slower than conversion |

| `time.In(pstLoc)` (timezone conversion) | ~1-2 nanoseconds | Baseline — essentially free |


Even with 10,000 rows:

- Go conversion: 10,000 × 1ns = **0.01 milliseconds**

- MySQL fetching: 10,000 × 100μs = **1 second**


The timezone conversion is **0.001%** of the total time. Negligible.


### Why This Is Better Than Alternatives


| Approach | Safe? | MySQL Load | Pool Contamination | Complexity |

|---|---|---|---|---|

| `SET time_zone` in Go | ❌ No — leaks to pool | None | Yes | Low |

| `CONVERT_TZ()` in SQL | ✅ Safe | Adds CPU load | No | Medium |

| Convert in Go code | ✅ Safe | None | No | Low |


---


## 5. If You Ever Want to Change to PST (All or Nothing)


If you decide to move away from UTC, **all three changes must be made together**:


### Change 1: docker-compose.yml — both containers

```yaml

certgen:

  environment:

    - TZ=America/Vancouver    # Go's time.Now() returns PST


mysql:

  environment:

    - TZ=America/Vancouver    # MySQL's CURRENT_TIMESTAMP returns PST

```


### Change 2: db.go — local DB DSN (add `loc=Local`)

```go

dsn := "%s:%s@tcp(%s:%d)/%s?parseTime=true&loc=Local&multiStatements=true&timeout=10s"

```


### Change 3: db.go — source DB DSN (add `loc=Local`)

```go

dsn := "%s:%s@tcp(%s:%d)/%s?parseTime=true&loc=Local&timeout=10s"

```


### Why All Three


| If you change... | But not... | What breaks |

|---|---|---|

| `TZ` on both containers | DSN `loc=Local` | Go reads TIMESTAMP as UTC — mismatch with MySQL PST |

| DSN `loc=Local` | `TZ` on containers | Go uses container's UTC — `loc=Local` = UTC, no effect |

| MySQL `TZ` only | Go `TZ` + DSN | `CURRENT_TIMESTAMP` = PST, `time.Now()` = UTC — mismatch |

| Go `TZ` only | MySQL `TZ` + DSN | `time.Now()` = PST, `CURRENT_TIMESTAMP` = UTC — mismatch |


**All three or none. Partial changes cause mismatches.**


### Risks of Changing to PST


- ❌ Daylight saving time: March/November clock changes cause gaps or duplicates in sync

- ❌ Source DB must also be PST, or sync breaks

- ❌ If source DB is UTC, `modify_time` comparison is off by 7-8 hours


---


## 6. Quick Reference


| Scenario | What to do |

|---|---|

| **Keep current setup (recommended)** | Change nothing. Everything is UTC. |

| **View timestamps in PST manually** | `SET time_zone = 'America/Vancouver';` per MySQL session |

| **Query with PST time in Go** | Convert PST→UTC before query, convert UTC→PST after |

| **Change everything to PST** | Set `TZ` on both containers + `loc=Local` in both DSNs |

| **Check if a column auto-converts** | `TIMESTAMP` = yes, `DATETIME` = no |

Thursday, 19 June 2025

GO LANG MIGRATION with embed

 GO LANG migration init by default takes a string but when using embed, it wont work, there is a work aoround:



The error "cannot use migrationFS (variable of type embed.FS) as string value in argument to migrate.New" indicates a type mismatch when attempting to use Go's embed.FS with the migrate library. The migrate.New function, when used with a file system source, expects a string path or a fs.FS interface, not directly an embed.FS variable passed as a string.
To resolve this, two common approaches are available:
  • Using migrate.NewWithSource with io/fs.FS: This is the recommended and most direct approach for embed.FS. The migrate library provides migrate.NewWithSource which can accept an io/fs.FS interface. Since embed.FS implements io/fs.FS, it can be directly passed to this function.


    import (
        "embed"
        "io/fs"

        "github.com/golang-migrate/migrate/v4"
        _ "github.com/golang-migrate/migrate/v4/database/your_database_driver" // Replace with your database driver
        "github.com/golang-migrate/migrate/v4/source/iofs"
    )

    //go:embed migrations/*.sql
    var migrationFS embed.FS

    func main() {
        // ... database connection string ...
        driver, err := iofs.New(migrationFS, "migrations") // "migrations" is the sub-directory within embed.FS
        if err != nil {
            // handle error
        }

        m, err := migrate.NewWithSourceInstance("iofs", driver, "your_database_connection_string")
        if err != nil {
            // handle error
        }

        // ... run migrations ...
    }

GO add static files using Go embed * go by default exclude .go file during go build

 https://stackoverflow.com/questions/56608186/does-go-build-include-non-go-files-in-binary

Does go build includes non go files in binary?

By default, no: you need to embed them.


GO:EMBED:

When using go:embed to embed files or directories in a Go application, the path to access the embedded content within the embed.FS will be relative to the directory containing the source file where the //go:embed directive is declared.
Here's how the path is determined:
  • Relative to the package directory:
    The patterns specified in the //go:embed directive are interpreted as paths relative to the directory of the Go source file containing the directive.
  • Embedded structure:
    If you embed a directory, the files within that directory and its subdirectories will maintain their relative path structure within the embed.FS.
  • Accessing files:
    When you later access these embedded files using methods like embed.FS.ReadFile() or embed.FS.Open(), you will use paths that reflect this embedded structure.
Example:
Consider a file main.go in the root of your project, and a static directory with a css subdirectory containing style.css:
myproject/
├── main.go
└── static/
    └── css/
        └── style.css
main.go
package main

import (
"embed"
)

//go:embed static/*
var staticFS embed.FS

func main() {
// To access style.css, the path will be "static/css/style.css"
// relative to the embedded root.
content, err := staticFS.ReadFile("static/css/style.css")
if err != nil {
// Handle error
}
// Use content
}