Stay Hungry

Friday, 21 December 2018

Updated Docker Compose setup

Docker useful commands
--------------------------------------

* Install docker && docker-compose(prewritten docker commands that did congfigs for container)
Docker: - https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-using-the-repository
Lateset docker-compose:
curl -L "https://github.com/docker/compose/releases/download/1.22.0/docker-compose-$(uname -s)-$(uname -m)" > ./docker-compose
sudo mv ./docker-compose /usr/bin/docker-compose
sudo chmod +x /usr/bin/docker-compose
(https://stackoverflow.com/questions/49839028/how-to-upgrade-docker-compose-to-latest-version)

* Config docker to use some IP not in conflict wiht company one
- vim /etc/docker/daemon.json
{
"bip": "192.168.40.1/28",
"ipv6": false,
"dns": ["8.8.8.8"]
}

* restart docker -
sudo systemctl restart docker.service

* Start docker service:
- Instructions
We'll start by creating a folder for this project: mkdir lamp-stack && cd lamp-stack

Create another subdirectory, /php which contains the following index.php file:



<html>
<head>
<title>Hello World</title>
</head>

<body>
<?php
echo "Hello, World!";
?>
</body>
</html>
Populate docker-compose.yml with the following configuration:
# ./docker-compose.yml

version: '3'

services:
db:
image: mysql:5.7
environment:
MYSQL_ROOT_PASSWORD: my_secret_pw_shh
MYSQL_DATABASE: test_db
MYSQL_USER: devuser
MYSQL_PASSWORD: devpass
ports:
- "9906:3306"
web:
image: php:7.2.2-apache
container_name: php_web
depends_on:
- db
volumes:
- ./php/:/var/www/html/
ports:
- "8100:80"
stdin_open: true
tty: true
Our directory structure should look as follows:
$ tree
.
├── docker-compose.yml
└── php
└── index.php
---DockerFile(use by docker to create an image(https://docs.docker.com/get-started/part2/#apppy))

DockerFile content:

FROM php:7.2.2-apache
MAINTAINER egidio docile
RUN docker-php-ext-install pdo pdo_mysql mysqli
---------------------(use php:7.2.2-apache image, use its built in command to install pdo, mysqli: https://linuxconfig.org/how-to-create-a-docker-based-lamp-stack-using-docker-compose-on-ubuntu-18-04-bionic-beaver-linux)

-----------------------------------------------
Execute docker-compose up -d in the terminal and load http://localhost(your url):8100/ in your browser.
Time to learn some PHP!

Notes
* image is from docker hub
* volume is the folder we want to retain when destroy and recreate same containers

We use port-forwarding to connect to the inside of containers from our local machine.
webserver: http://localhost:8100
db: mysql://devuser:devpass@localhost:9906/test_db
Our local directory, ./php, is mounted inside of the webserver container as /var/www/html/
The files within in our local folder will be served when we access the website inside of the container

(https://alysivji.github.io/php-mysql-docker-containers.html)
--------------------------------------------------------------------------------------------------
* show list of running containers:
- sudo docker-compose ps
* GO inside to edit contianer file:
- you have two ways to edit file (config, source code, any files):

* php_web is docker container name

1. docker exec -it php_web bash
2. if you have a volume, then you can edit the files inside the volume directly on the host machine.

or
vim ~/.bashrc
dexec() {
docker exec -it -e TERM=$TERM -e LINES=$LINES -e COLUMNS=$COLUMNS "$@"
}
$ source ~/.bashrc
Then, you can login into your container by running:

dexec CONTAINER_NAME CMD`

* exit a container
-
// If you don't want to add an editor just to make a few small changes (e.g., change the Tomcat configuration), you can just use:

docker cp <container>:/path/to/file.ext (if just want place in a folder go like var/www/html, if want place at a specific file in folder, go like var/www/html/index.php)

//which copies it to your local machine (to your current directory).

//Then edit the file locally using your favorite editor, and then do a

docker cp file.ext <container>:/path/to/file.ext

to replace the old file.

(https://stackoverflow.com/questions/30853247/how-do-i-edit-a-file-after-i-shell-to-a-docker-container)

This will first stop all the containers, next remove all the containers, and finally start them in the background as specified by the docker-compose.yml file.

First, cd to the directory where docker-compose.yml file is present, and then execute the following to restart.

cd /home/myapp (directory where docker-compose.yml file is )

docker-compose stop && docker-compose rm -f

docker-compose up -d

(https://www.thegeekstuff.com/2016/04/docker-compose-up-stop-rm/comment-page-1/)

----------------- parallel && Makefile

apt-get install parallel
apt-get install make

vim Makefile

anything:(TAB) try_curl say_hello
say_hello:
echo "hello world"
try_curl:
seq 4 | parallel -n0 -j2 "curl -H 'Content-Type:application/json' -X GET http://172.16.107.209/~jxiang/response_GET.php"

----------------------------ubuntu cat
* cat file (displays file)
* cat file1 > file2 (copy cotnent of 1 to 2(override), if 2 dne, it will be created)
* cat file1 >> file2 (copy cotnent of 1 to 2(attach), if 2 dne, it will be created)

Using Curl lib to retrieve CSRF token from server response set-cookie, set CSRF header, retrieve logged in session from server response set-cookie

// Init
$ch = curl_init();
$cookies = array();

/* Fetch CSRFTOKEN from Server response header by visting login page */
$request_headers = array(
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'Accept-Encoding: gzip, deflate, br',
'Accept-Language: en-US,en;q=0.9',
'Cache-Control: no-cache',
'Connection: keep-alive',
"Host:{$url}",
'Pragma: no-cache',
"Referer: https://{$url}/",
'Upgrade-Insecure-Requests: 1',
'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'
);

curl_setopt($ch, CURLOPT_URL, "https://{$url}/login/");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
// Set request header
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
// Get response header
curl_setopt($ch, CURLOPT_HEADER, TRUE);
$result = curl_exec($ch);
if (curl_errno($ch)){
throw new \Exception('Retreive CSRF token failed -> ' . curl_error($ch));
}
// Get cookie instruction from the server response
preg_match_all('/^Set-Cookie:\s*([^;]*)/mi', $result, $matches);
foreach ($matches[1] as $item) {
parse_str($item, $cookie);
$cookies = array_merge($cookies, $cookie);
}

if (empty($cookies['csrftoken'])) {
throw new \Exception('Unable to obtain CSRF token from server response.');
}

/* Login */
$request_headers = array(
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'Accept-Encoding: gzip, deflate, br',
'Accept-Language: en-US,en;q=0.9',
'Cache-Control: no-cache',
'Connection: keep-alive',
"Cookie: csrftoken={$cookies['csrftoken']}",
'Content-Type: application/x-www-form-urlencoded',
"Host: {$url}",
"Origin: https://{$url}",
'Pragma: no-cache',
"Referer: https://{$url}/",
'Upgrade-Insecure-Requests: 1',
'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'
);

$post_values = array(
'username' => $username,
'password' => $pwd,
'next' => '/'
);

curl_setopt($ch, CURLOPT_URL, "https://{$url}/login/" );
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($post_values));

$result = curl_exec($ch);
if (curl_errno($ch)){
throw new \Exception('Login to FortiAuthenticator failed -> ' . curl_error($ch));
}

preg_match_all('/^Set-Cookie:\s*([^;]*)/mi', $result, $matches);
foreach ($matches[1] as $item) {
parse_str($item, $cookie);
$cookies = array_merge($cookies, $cookie);
}

if ( ! empty($cookies['sessionid'])) {
$is_logged_in = TRUE;
}

SSH

SSH IP_ADDR -l USER_NAME

Thursday, 20 December 2018

PHP CURL parallel && write download content on the fly using curl

Apache parallel command:

apt-get install parallel

#execute 4 parallel requests in total, 2 parallel request to be executed at atime

seq 4 | parallel -n0 -j2 "curl -H 'Content-Type: application/json' -X GET http://172.16.107.209/~jxiang/response_GET.php"

#execute 4 parallel requests in total, 4 parallel request to be executed at atime

seq 4 | parallel -n0 "curl -H 'Content-Type: application/json' -X GET http://172.16.107.209/~jxiang/response_GET.php"

PHP parallel curl(curl multi):

/**

* Webkul Software.

* @category Webkul

* @author Webkul

* @license https://store.webkul.com/license.html

// array of curl handles

$multiCurl = array();

// data to be returned

$result = array();

// multi handle

$mh = curl_multi_init();

foreach ($ids as $i => $id) {

// URL from which data will be fetched

$fetchURL = 'https://webkul.com&customerId='.$id;

$multiCurl[$i] = curl_init();

curl_setopt($multiCurl[$i], CURLOPT_URL,$fetchURL);

curl_setopt($multiCurl[$i], CURLOPT_HEADER,0);

curl_setopt($multiCurl[$i], CURLOPT_RETURNTRANSFER,1);

curl_multi_add_handle($mh, $multiCurl[$i]);

}

$index=null;

do {

curl_multi_exec($mh,$index);

} while($index > 0);

// get content and remove handles

foreach($multiCurl as $k => $ch) {

$result[$k] = curl_multi_getcontent($ch);

curl_multi_remove_handle($mh, $ch);

}

// close

curl_multi_close($mh);

// Open file to write donwload content

$fp = fopen('fortipoc_export_all.poc', 'w+');

curl_setopt($this->_curl, CURLOPT_FILE, $fp);

curl_exec($this->_curl);

https://webkul.com/blog/simultaneous-curl-requests-in-php/

Tuesday, 18 December 2018

Go basic knowledge notes

github gin /? (light weight framework)

r := gin.default()

ss -an | grep 8080 (find out whether a port has been listened)

import {

classA ? or its just package

}

// object autpo

classA.function_in_classA();

import {

classA

}

import {

dir

}

dir.anypkg

// gin

c.DefaultQuery("Name", "Guest");

c.postForm('value');w

get name=xxx from url, or use default value guest

// move route handler to another module

// go auto compile, needs to set up a go project name after installation

// installation can be done in make file

// build explore?

~./vim (juns git vim file https://github.com/junxie6/config_centos_v2.git

- git clone, tools will be there )

// Then Juns handy tool will be ready to use

ctrl-j file system

ctrk-k opened file

GO reverse proxy:

https://github.com/wolfeidau/reverseproxy/blob/master/main.go

Go Installation

GO Installation:

1. CURL GET tar, unzip in /usr/local

curl 'https://dl.google.com/go/go1.10.3.linux-amd64.tar.gz' -o go.tar.gz \
&& sudo tar -zxvf go.tar.gz -C /usr/local \

&& sudo rm -f go.tar.gz

2. SET bash_profile, add to path, and to export $GO, $PATH variables to children programs like go

if bash never exists:

```
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

# User specific environment and startup programs

PATH="$PATH:$HOME/.local/bin:$HOME/bin"

export GOPATH="$HOME/go"
export PATH="$PATH:/usr/local/go/bin:$GOPATH/bin:/usr/local/bin" if already there: vim ~/.bash_profile
export GOPATH="$HOME/go"
export PATH="$PATH:/usr/local/go/bin:$GOPATH/bin:/usr/local/bin"

refresh

source ~/.bash_profile`

$ echo $GOPATH`

`$ echo $PATH`

go version
go version go1.10.3 linux/amd64 To build the binary:

```
$ go build main.go
```

To run the binary:

`$ ./main`

go-git

https://github.com/go-gitea/gitea