Showing posts with label MYSQL. Show all posts
Showing posts with label MYSQL. Show all posts

Thursday, 16 July 2026

timezone handling guide

 # Timezone Handling Guide


## 1. Best Practice: Store Time in UTC


**Keep all timestamps in UTC in the database.** This is the industry standard for servers.


| Component | Current Setting | Why It's Best |

|-----------|----------------|---------------|

| MySQL (Docker) | UTC (default) | No DST issues, universal, sync works across timezones |

| Go application | UTC (no `loc=` in DSN) | Consistent with MySQL, no conversion bugs |

| Source DB sync | UTC | `TIMESTAMP` stored as UTC internally, comparisons are correct |


**MySQL `TIMESTAMP` is always stored as UTC internally.** The display timezone only affects what you see, not what's stored.


---


## 2. Direct DB Query: Use `SET time_zone` Per Connection


When querying the database manually (MySQL CLI, Workbench, etc.):


```sql

-- Set timezone for this session

SET time_zone = 'America/Vancouver';


-- All TIMESTAMP columns now display in PST

SELECT * FROM sync_metadata;

-- Shows: 2026-07-16 11:31:37 (PST display)

-- Stored: 2026-07-16 18:31:37 (UTC — unchanged)


-- Query with PST times — MySQL converts automatically

SELECT * FROM sync_metadata WHERE last_run_at > '2026-07-16 11:00:00';

```


**Key points:**

- ✅ Display only — does NOT change stored UTC data

- ✅ Lasts for the entire session (all queries until you disconnect)

- ✅ Resets to UTC when connection closes

- ✅ Does NOT affect the Go application (separate connections)

- ⚠️ Only works for `TIMESTAMP` columns, not `DATETIME`


---


## 3. Why `SET time_zone` Can't Be Used in Go


Go uses a **connection pool** — connections are reused, not created/destroyed per query. If you `SET time_zone` on a connection, it **leaks** to the next function that gets the same connection.


### Connection Pool Diagram


```

┌─────────────────────────────────────────────────────────────────┐

│  Connection Pool (max 100 open, 50 idle, 300s lifetime)       │

│                                                                 │

│  ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐                 │

│  │ conn1  │ │ conn2  │ │ conn3  │ │  ...   │  ← 50 idle       │

│  │ (UTC)  │ │ (UTC)  │ │ (UTC)  │ │ (UTC)  │     (pre-warmed) │

│  └────────┘ └────────┘ └────────┘ └────────┘                 │

└─────────────────────────────────────────────────────────────────┘


Function A: SET time_zone on conn1

┌─────────────────────────────────────────────────────────────────┐

│                                                                 │

│  ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐                 │

│  │ conn1  │ │ conn2  │ │ conn3  │ │  ...   │                 │

│  │ (PST!) │ │ (UTC)  │ │ (UTC)  │ │ (UTC)  │                 │

│  └───┬────┘ └────────┘ └────────┘ └────────┘                 │

│      │                                                         │

│      └── Function A done, conn1 returned to pool              │

│          conn1 STILL has PST!                                  │

└─────────────────────────────────────────────────────────────────┘


Function B: gets conn1 from pool

┌─────────────────────────────────────────────────────────────────┐

│                                                                 │

│  ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐                 │

│  │ conn1  │ │ conn2  │ │ conn3  │ │  ...   │                 │

│  │ (PST!) │ │ (UTC)  │ │ (UTC)  │ │ (UTC)  │                 │

│  └───┬────┘ └────────┘ └────────┘ └────────┘                 │

│      │                                                         │

│      └── Function B gets conn1 — INHERITS PST! ← LEAKED!      │

│          B's query runs in PST, not UTC                        │

└─────────────────────────────────────────────────────────────────┘


conn1 stays PST until:

  - Connection recycled (after 300s lifetime)

  - Explicitly reset with SET time_zone = 'UTC'

  - Application restarts

```


### Pool Configuration (from `.env`)


```

DB_MAX_OPEN_CONNS=100       Max simultaneous connections

DB_MAX_IDLE_CONNS=50        Idle connections kept alive (pre-warmed)

DB_CONN_MAX_LIFETIME_SEC=300  Each connection recycled after 5 minutes

```


At rest: 50 idle connections waiting. Under load: up to 100 total. When load drops: extras closed back to 50.


### Why It Leaks


| What you think | What actually happens |

|---|---|

| Function ends → connection closed → timezone gone | Function ends → connection returned to pool → timezone persists |

| Next function gets fresh UTC connection | Next function might get the same PST connection |

| Each function is isolated | Timezone leaks for up to 300 seconds (connection lifetime) |


---


## 4. Solution: Convert Time in Go Code


### The Pattern


```

Input (PST) → Convert to UTC → Query DB (UTC) → Results (UTC) → Convert to PST → Display

```


### In Go Code


```go

// Load Vancouver timezone (handles PST/PDT automatically)

pstLoc, _ := time.LoadLocation("America/Vancouver")


// 1. Input: user wants logs after "2026-07-16 11:00:00" PST

pstTime, _ := time.ParseInLocation("2006-01-02 15:04:05", "2026-07-16 11:00:00", pstLoc)


// 2. Convert to UTC for query

utcTime := pstTime.UTC()  // 2026-07-16 18:00:00 UTC


// 3. Query DB with UTC — no SET time_zone needed, no pool contamination

rows, err := db.Query(`

    SELECT last_run_at, last_sync_time, last_run_status

    FROM sync_metadata

    WHERE last_run_at > ?

`, utcTime)


// 4. Read results (Go reads TIMESTAMP as UTC because no loc= in DSN)

for rows.Next() {

    var lastRunAt time.Time  // Go gives you UTC time

    var lastSyncTime time.Time

    var status string

    rows.Scan(&lastRunAt, &lastSyncTime, &status)


    // 5. Convert to PST for display

    pstRunAt := lastRunAt.In(pstLoc)  // 2026-07-16 11:31:37 PST


    fmt.Printf("Run at: %s, Status: %s\n",

        pstRunAt.Format("2006-01-02 15:04:05"), status)

}

```


### Why This Is Cost-Free


`time.Time.In(loc)` doesn't do any calculation — it just attaches the timezone location to the existing time value. The actual time moment doesn't change, only how it's displayed.


| Operation | Time per row | Comparison |

|---|---|---|

| `rows.Scan()` (parse from MySQL) | ~1-2 microseconds | 1,000x slower than conversion |

| Network I/O (fetching the row) | ~50-100 microseconds | 50,000x slower than conversion |

| `time.In(pstLoc)` (timezone conversion) | ~1-2 nanoseconds | Baseline — essentially free |


Even with 10,000 rows:

- Go conversion: 10,000 × 1ns = **0.01 milliseconds**

- MySQL fetching: 10,000 × 100μs = **1 second**


The timezone conversion is **0.001%** of the total time. Negligible.


### Why This Is Better Than Alternatives


| Approach | Safe? | MySQL Load | Pool Contamination | Complexity |

|---|---|---|---|---|

| `SET time_zone` in Go | ❌ No — leaks to pool | None | Yes | Low |

| `CONVERT_TZ()` in SQL | ✅ Safe | Adds CPU load | No | Medium |

| Convert in Go code | ✅ Safe | None | No | Low |


---


## 5. If You Ever Want to Change to PST (All or Nothing)


If you decide to move away from UTC, **all three changes must be made together**:


### Change 1: docker-compose.yml — both containers

```yaml

certgen:

  environment:

    - TZ=America/Vancouver    # Go's time.Now() returns PST


mysql:

  environment:

    - TZ=America/Vancouver    # MySQL's CURRENT_TIMESTAMP returns PST

```


### Change 2: db.go — local DB DSN (add `loc=Local`)

```go

dsn := "%s:%s@tcp(%s:%d)/%s?parseTime=true&loc=Local&multiStatements=true&timeout=10s"

```


### Change 3: db.go — source DB DSN (add `loc=Local`)

```go

dsn := "%s:%s@tcp(%s:%d)/%s?parseTime=true&loc=Local&timeout=10s"

```


### Why All Three


| If you change... | But not... | What breaks |

|---|---|---|

| `TZ` on both containers | DSN `loc=Local` | Go reads TIMESTAMP as UTC — mismatch with MySQL PST |

| DSN `loc=Local` | `TZ` on containers | Go uses container's UTC — `loc=Local` = UTC, no effect |

| MySQL `TZ` only | Go `TZ` + DSN | `CURRENT_TIMESTAMP` = PST, `time.Now()` = UTC — mismatch |

| Go `TZ` only | MySQL `TZ` + DSN | `time.Now()` = PST, `CURRENT_TIMESTAMP` = UTC — mismatch |


**All three or none. Partial changes cause mismatches.**


### Risks of Changing to PST


- ❌ Daylight saving time: March/November clock changes cause gaps or duplicates in sync

- ❌ Source DB must also be PST, or sync breaks

- ❌ If source DB is UTC, `modify_time` comparison is off by 7-8 hours


---


## 6. Quick Reference


| Scenario | What to do |

|---|---|

| **Keep current setup (recommended)** | Change nothing. Everything is UTC. |

| **View timestamps in PST manually** | `SET time_zone = 'America/Vancouver';` per MySQL session |

| **Query with PST time in Go** | Convert PST→UTC before query, convert UTC→PST after |

| **Change everything to PST** | Set `TZ` on both containers + `loc=Local` in both DSNs |

| **Check if a column auto-converts** | `TIMESTAMP` = yes, `DATETIME` = no |

Friday, 27 February 2026

MYSQL Server VS SQL lite

 

FeatureSQLiteMySQL Server
ArchitectureEmbedded (no server)Client-server
SetupNo installation neededRequires installation & configuration
StorageSingle .db fileManaged database directories
ConcurrencyLimited (best for low write load)High concurrency support
ScalabilitySmall to medium appsMedium to very large systems
User ManagementNone (file-based access)Full user & role system
PerformanceVery fast for local/single-userOptimized for multi-user workloads
BackupCopy the fileDump tools / replication / snapshots


What is SQLite?

SQLite is a lightweight, embedded relational database engine.

Key characteristics:

  • 📦 Serverless — no separate database server process

  • 📁 File-based — entire database is stored in a single file

  • Zero configuration

  • 🔌 Runs inside your application (linked as a library)

Commonly used in:

  • Mobile apps (Android, iOS)

  • Desktop apps

  • Browsers (e.g., local storage engines)

  • Small tools and embedded systems


What is MySQL Server?

MySQL is a full client-server relational database management system (RDBMS).

Key characteristics:

  • 🖥 Runs as a separate database server process

  • 🌐 Supports multiple concurrent users

  • 🔐 Advanced security & user management

  • 📊 Designed for web apps and production systems

Commonly used in:

  • Web applications

  • Enterprise systems

  • APIs & backend services

  • Cloud-hosted platforms



  • A mobile Angular + Capacitor app → SQLite

  • A Node.js backend serving 10,000 users → MySQL Server


SQLite has a built-in .dump command.

Using CLI:

sqlite3 mydatabase.db .dump > dump.sql

This generates a file containing:

  • CREATE TABLE statements

  • INSERT statements

  • Indexes

  • Triggers

Restore from dump:

sqlite3 newdatabase.db < dump.sql

🔹 2️⃣ Simple Backup (Copy the File)

Because SQLite is just a single file:

cp mydatabase.db backup.db

⚠️ Important:
If the database is being written to, use the .backup command instead:

sqlite3 mydatabase.db ".backup backup.db"




🔹 2️⃣ Read Performance

SQLite is very fast for reads.

It can handle:

  • Thousands of SELECT queries per second

  • Many concurrent readers

  • Great performance for cached data

Reads do NOT block other reads.


🔹 3️⃣ Write Concurrency (Important Limitation)

SQLite allows:

🚨 Only ONE writer at a time.

That means:

  • Multiple users can read simultaneously

  • But writes are serialized (queued)

If many users try to write at the same time:

  • You’ll see database is locked errors

  • Performance drops under heavy write load


Rough Practical Numbers

These are approximate real-world observations:

Workload TypeWhat SQLite Can Handle
Reads10,000+ per second (depending on hardware)
Writes100–1,000 small writes/sec (serialized)
Concurrent UsersDozens fine, hundreds depends on workload
Database SizeMulti-GB very common



MySQL support concurrent writes?

Yes — especially when using the default engine InnoDB (the default in modern MySQL).

It supports:

  • Multiple concurrent writers

  • Row-level locking

  • Transactions (ACID compliant)

  • MVCC (Multi-Version Concurrency Control)

This is very different from SQLite, which allows only one writer at a time.

Thursday, 9 October 2025

MYSQL , restrict user from hosts, change restriction, restrict users from docker container hosts

 MYSQL Docker container created user root is root@%, so it can connect from any host


During MYSQL init set up , in init.sql, you might have


CREATE USER 'test'@'%' IDENTIFIED BY 'gitea';

GRANT ALL PRIVILEGES ON `gitea`.* TO 'test'@'%';


thats allowing user test to connect from any host, to enforce restriction for the user from a docker container service from any docker container within a docker subnet, its more safer to do :


CREATE USER 'test'@'169.255.255.%' IDENTIFIED BY 'gitea';

GRANT ALL PRIVILEGES ON `gitea`.* TO 'test'@'169.255.255.%';



Or if your init.sql already ran, you can do :

RENAME USER 'test'@'%' TO 'test'@'169.255.255.%';



You can do this with root user as well, 


If root@'localhost' already exists, drop the wide one:


sql

Copy code

DROP USER 'root'@'%';


you can use SQL statement to check:

SELECT user, host, plugin FROM mysql.user WHERE user='root';




Thursday, 28 August 2025

MYSQL show user permission and user@host

 MYSQL cmds to show user privileges

SELECT CONCAT('SHOW GRANTS FOR \'', user, '\'@\'', host, '\';') FROM mysql.user;

this produce a list of :

SHOW GRANTS FOR 'john'@'localhost';


then execute individual command to get permission of user


To create a new user in MySQL, execute the following SQL command within your MySQL client (e.g., MySQL Shell, phpMyAdmin, or a command-line interface):
Code
CREATE USER 'your_username'@'localhost' IDENTIFIED BY 'your_password';
  • Replace 'your_username' with the desired username.
  • Replace 'localhost' with the hostname or IP address from which the user will connect. Use '%' for any host.
  • Replace 'your_password' with a strong, secure password for the new user.

This host measn the source IP of the user that can use to connect to 

if its test@123.456.789.123

if test is connecting from 155.155.155.231 test will be rejected


MY SQL grant permission 

GRANT ALL PRIVILEGES ON `<mydatabase>`.* TO `test`@`10.121.150.150`;

FLUSH PRIVILEGES;




  • Host Restriction:
    host specifies the allowed source(s) from which the user can connect. This can be:
    • localhostAllows connections only from the same machine where the MySQL server is running (often using a Unix socket, or TCP/IP on 127.0.0.1).
    • A specific IP address (e.g., 192.168.1.100): Limits connections to that exact IP address.
    • A hostname (e.g., myclient.example.com): Restricts connections to that specific hostname.
    • A wildcard character (%): Allows connections from any host. This is often used for broad access but can pose security risks if not managed carefully.
    • An IP address range with wildcards (e.g., 10.0.%): Allows connections from any IP address within that specific range.
    ------------------------------------------------------------------------------------
    --------------------------------------------------------------------------------------
    What if mysql is running on server 172.168.11.110, the docker services running bridge network also in that machine, can docker user test@172.168.11.110 to conenct assume such user is created :




Yes. MySQL decides which account to use based on the source IP it sees, and Docker’s NAT usually makes your container’s traffic look like it’s coming from the host’s IP (10.0.0.205) when it connects to a service on the host. So the server accepts test@10.0.0.205.

Why this happens

  • Your container lives on a bridge subnet (e.g., 172.17.0.0/16).

  • When the container connects to the host’s MySQL at 10.0.0.205:3306, Docker’s iptables rules do MASQUERADE/SNAT for packets leaving the bridge.

  • Result: MySQL receives a connection from 10.0.0.205 → 10.0.0.205:3306 (hairpin NAT), which matches the MySQL user test@10.0.0.205.

How to confirm exactly what MySQL sees
Run this in your session after connecting from the container:

SELECT USER() AS client_says, CURRENT_USER() AS mysql_matched_account, SUBSTRING_INDEX(HOST, ':', 1) AS source_ip FROM information_schema.processlist WHERE ID = CONNECTION_ID();
  • source_ip shows what IP MySQL thinks you’re coming from.

  • CURRENT_USER() shows the precise user@host entry MySQL matched.

Tuesday, 28 January 2025

MYSQL how to get ERD

 

https://dev.to/alumassy/how-to-access-the-erd-entity-relationship-diagram-of-your-database-schema-in-mysql-workbench-5813


Below are steps you can follow to view the ERD (Entity-Relationship Diagram) of your database schema in MySQL Workbench:

  1. Open MySQL Workbench and open your database connection.
  2. In the top navigation bar, click on “Database” to expand the list of options.
  3. Select “Reverse Engineer” from the database context menu.

Tuesday, 24 December 2024

MYSQL set uuid id, text column cant have default value

 Good old auto increment

CREATE TABLE example (

    id INT AUTO_INCREMENT PRIMARY KEY,

    description TEXT NOT NULL

);

// TEXT column cant have default value ''
// use
CREATE TABLE example (
    id INT AUTO_INCREMENT PRIMARY KEY,
    description TEXT
);

// uuid
CREATE TABLE my_table (
    id CHAR(36) NOT NULL PRIMARY KEY,
    name VARCHAR(255) NOT NULL
);
Insert values manually:

sql
Copy code
INSERT INTO my_table (id, name) VALUES (UUID(), 'Sample Name');

MYSQL docker user permission & MYSQL need 3306

 For example, if you set:



MYSQL_DATABASE=mydb

MYSQL_USER=myuser

MYSQL_PASSWORD=mypassword

The myuser will have full privileges (such as SELECT, INSERT, UPDATE, DELETE, etc.) on mydb but will not have privileges on other databases unless explicitly granted later.


MYSQL need port 3306, you can map different port from 0.0.0.0 to 3306


You will need a ini file to grant global permission to docker mysql user if you want to 


GRANT ALL PRIVILEGES ON *.* TO 'myuser'@'%'; FLUSH PRIVILEGES;

Thursday, 5 December 2024

MYSQL unique constraint, vice versa are allowed

 UNIQUE (A, B)

This ensures that:

  • (1, 2) and (1, 2) are not allowed (duplicate values in the same order).
  • (2, 1) and (1, 2) are allowed (different order, treated as distinct combinations).

Friday, 25 October 2024

MYSQL how to store large binary object like file binary

  BLOB


https://stackoverflow.com/questions/13435187/what-is-difference-between-storing-data-in-a-blob-vs-storing-a-pointer-to-a-fi

According to MySQL manual page on Blob, A BLOB is a binary large object that can hold a variable amount of data.

Tuesday, 30 July 2024

MYSQL connectionr refused, MYSQL default root@Local host can change to % to all host, or create a user

 mysql connection issue

https://stackoverflow.com/questions/40561433/docker-mysql-2002-connection-refused

connection refused means connected but refused access

if mysql is run by docker 

access in two ways, same host, web app acces, for hostname use docker compose services name:

AKA db

docker-compose ps can tell you servie name


for mysql bench,

pick ssh option with credential first


then host use 127.0.0.1 or serivce name then credential


root user by default is local host

https://stackoverflow.com/questions/41645309/mysql-error-access-denied-for-user-rootlocalhost



createa a user in mysql grant permission in a table:

https://www.digitalocean.com/community/tutorials/how-to-create-a-new-user-and-grant-permissions-in-mysql


https://www.atlassian.com/data/admin/how-to-grant-all-privileges-on-a-database-in-mysql#:~:text=To%20GRANT%20ALL%20privileges%20to,GRANT%20ALL%20PRIVILEGES%20ON%20database_name.


CREATE USER 'sammy'@'localhost' IDENTIFIED BY 'password';


@ '%' means everywhere access


mysql> GRANT ALL PRIVILEGES ON database_name.* TO 'username'@'localhost';

Friday, 29 December 2023

MYSQL bench to connect to docker-compose db containers

MYSQL bench is visualization tool of db :

https://dev.mysql.com/downloads/workbench/

check out the MYSQL version it supports


older version of MYSQL workbench can be downloaded :

https://downloads.mysql.com/archives/workbench/


https://dev.mysql.com/doc/workbench/en/wb-intro.html#:~:text=MySQL%20Workbench%20fully%20supports%20MySQL,attempt%20to%20make%20a%20connection.

MySQL Workbench is a graphical tool for working with MySQL servers and databases. MySQL Workbench fully supports MySQL server version 5.7 and higher. Deprecated versions of MySQL Server (prior to version 5.7) are incompatible with MySQL Workbench and should be upgraded before you attempt to make a connection.


MYSQL workbench wont support  MYSQL 8.1.0

https://www.reddit.com/r/mysql/comments/15dzgac/mysql_workbench_not_compatible_with_mysql_810/


downgrade your Mysql server to 8.0.34


workaround(currently used)

I encounter the same message followed by a crash of MySQL Workbench, with version 6.3.7 (build 1199).

I didn't find a solution, but here is a work-around:

Once you press the button Continue anyway, just open a database use toto, and then wait a bit (a minute is enough in my case), and then you can call a query without a crash. I found this trick here.





Connection to docker


1. first database -> manage connections -> create a new one 

2.  connection method -> standard tcp/ip over ssh

3. specify ssh info to the remote server, private key needed if using public/private key method, 

4. then speicfy ur mysql ip and port, for docker-compose u should specify 127.0.0.1 it means its using your remote server's IP in 3,  then specify the port you used in docker-compose like 3333,

because docker uses different subnet, and maps your current host port to its subnet port, you can also find docker containers ip through docker ps, docker inspect <containerID> then u can use container IP 3306

  db:

    image: mysql:8

    environment:

....

    ports:

      - "3333:3306"