Friday, 21 December 2018

Updated Docker Compose setup

Docker useful commands
--------------------------------------

* Install docker && docker-compose(prewritten docker commands that did congfigs for container)
Docker: - https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-using-the-repository
Lateset docker-compose:
curl -L "https://github.com/docker/compose/releases/download/1.22.0/docker-compose-$(uname -s)-$(uname -m)" > ./docker-compose
sudo mv ./docker-compose /usr/bin/docker-compose
sudo chmod +x /usr/bin/docker-compose
(https://stackoverflow.com/questions/49839028/how-to-upgrade-docker-compose-to-latest-version)


* Config docker to use some IP not in conflict wiht company one
- vim /etc/docker/daemon.json
{
    "bip": "192.168.40.1/28",
    "ipv6": false,
    "dns": ["8.8.8.8"]
}

* restart docker -
   sudo systemctl restart docker.service

 * Start docker service:
  - Instructions
We'll start by creating a folder for this project: mkdir lamp-stack && cd lamp-stack

Create another subdirectory, /php which contains the following index.php file:

<!-- ./php/index.php -->

<html>
    <head>
        <title>Hello World</title>
    </head>

    <body>
        <?php
            echo "Hello, World!";
        ?>
    </body>
</html>
Populate docker-compose.yml with the following configuration:
# ./docker-compose.yml

version: '3'

services:
  db:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: my_secret_pw_shh
      MYSQL_DATABASE: test_db
      MYSQL_USER: devuser
      MYSQL_PASSWORD: devpass
    ports:
      - "9906:3306"
  web:
    image: php:7.2.2-apache
    container_name: php_web
    depends_on:
      - db
    volumes:
      - ./php/:/var/www/html/
    ports:
      - "8100:80"
    stdin_open: true
    tty: true
Our directory structure should look as follows:
$ tree
.
├── docker-compose.yml
└── php
    └── index.php
 ---DockerFile(use by docker to create an image(https://docs.docker.com/get-started/part2/#apppy))


 DockerFile content:

FROM php:7.2.2-apache
MAINTAINER egidio docile
RUN docker-php-ext-install pdo pdo_mysql mysqli
---------------------(use php:7.2.2-apache image, use its built in command to install pdo, mysqli: https://linuxconfig.org/how-to-create-a-docker-based-lamp-stack-using-docker-compose-on-ubuntu-18-04-bionic-beaver-linux)

-----------------------------------------------
Execute docker-compose up -d in the terminal and load http://localhost(your url):8100/ in your browser.
Time to learn some PHP!

Notes
* image is from docker hub
* volume is the folder we want to retain when destroy and recreate same containers


We use port-forwarding to connect to the inside of containers from our local machine.
webserver: http://localhost:8100
db: mysql://devuser:devpass@localhost:9906/test_db
Our local directory, ./php, is mounted inside of the webserver container as /var/www/html/
The files within in our local folder will be served when we access the website inside of the container



 (https://alysivji.github.io/php-mysql-docker-containers.html)
--------------------------------------------------------------------------------------------------
* show list of running containers:
- sudo docker-compose ps
* GO inside to edit contianer file:
- you have two ways to edit file (config, source code, any files):

* php_web is docker container name

1. docker exec -it php_web bash
2. if you have a volume, then you can edit the files inside the volume directly on the host machine.

or
vim ~/.bashrc
dexec() {
  docker exec -it -e TERM=$TERM -e LINES=$LINES -e COLUMNS=$COLUMNS "$@"
}
$ source ~/.bashrc
Then, you can login into your container by running:

dexec   CONTAINER_NAME  CMD`


* exit a container
-
// If you don't want to add an editor just to make a few small changes (e.g., change the Tomcat configuration), you can just use:

docker cp <container>:/path/to/file.ext  (if just want place in a folder go like var/www/html, if want place at a specific file in folder, go like var/www/html/index.php)

//which copies it to your local machine (to your current directory).

//Then edit the file locally using your favorite editor, and then do a

docker cp file.ext <container>:/path/to/file.ext

to replace the old file.

(https://stackoverflow.com/questions/30853247/how-do-i-edit-a-file-after-i-shell-to-a-docker-container)


This will first stop all the containers, next remove all the containers, and finally start them in the background as specified by the docker-compose.yml file.

First, cd to the directory where docker-compose.yml file is present, and then execute the following to restart.

cd /home/myapp (directory where docker-compose.yml file is )

docker-compose stop && docker-compose rm -f

docker-compose up -d

(https://www.thegeekstuff.com/2016/04/docker-compose-up-stop-rm/comment-page-1/)



----------------- parallel && Makefile


apt-get install parallel
apt-get install make


vim Makefile

anything:(TAB) try_curl say_hello
say_hello:
echo "hello world"
try_curl:
seq 4 | parallel -n0 -j2 "curl -H 'Content-Type:application/json' -X GET http://172.16.107.209/~jxiang/response_GET.php"


----------------------------ubuntu cat
* cat file (displays file)
* cat file1 > file2 (copy cotnent of 1 to 2(override), if 2 dne, it will be created)
* cat file1 >> file2 (copy cotnent of 1 to 2(attach), if 2 dne, it will be created)

Using Curl lib to retrieve CSRF token from server response set-cookie, set CSRF header, retrieve logged in session from server response set-cookie

   // Init
            $ch = curl_init();
            $cookies = array(); 

            /* Fetch CSRFTOKEN from Server response header by visting login page */
            $request_headers = array(
                'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
                'Accept-Encoding: gzip, deflate, br',
                'Accept-Language: en-US,en;q=0.9',
                'Cache-Control: no-cache',
                'Connection: keep-alive',
                "Host:{$url}",
                'Pragma: no-cache',
                "Referer: https://{$url}/",
                'Upgrade-Insecure-Requests: 1',
                'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'
            );

            curl_setopt($ch, CURLOPT_URL, "https://{$url}/login/");
            curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
            curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
            // Set request header
            curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
            // Get response header
            curl_setopt($ch, CURLOPT_HEADER, TRUE);
            $result = curl_exec($ch);
            if (curl_errno($ch)){
                throw new \Exception('Retreive CSRF token failed -> ' . curl_error($ch));
            }
            // Get cookie instruction from the server response
            preg_match_all('/^Set-Cookie:\s*([^;]*)/mi', $result, $matches);
            foreach ($matches[1] as $item) {
                parse_str($item, $cookie);
                $cookies = array_merge($cookies, $cookie);
            }       

            if (empty($cookies['csrftoken'])) {
                throw new \Exception('Unable to obtain CSRF token from server response.');           
            }

            /* Login */
            $request_headers = array(
                'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
                'Accept-Encoding: gzip, deflate, br',
                'Accept-Language: en-US,en;q=0.9',
                'Cache-Control: no-cache',
                'Connection: keep-alive',
                "Cookie: csrftoken={$cookies['csrftoken']}",
                'Content-Type: application/x-www-form-urlencoded',
                "Host: {$url}",
                "Origin: https://{$url}",
                'Pragma: no-cache',
                "Referer: https://{$url}/",
                'Upgrade-Insecure-Requests: 1',
                'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'
            );

            $post_values = array(
                              'username'            => $username,
                              'password'            => $pwd,
                              'next'                => '/'
                            );

            curl_setopt($ch, CURLOPT_URL, "https://{$url}/login/" );
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
            curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
            curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($post_values));

            $result = curl_exec($ch);
            if (curl_errno($ch)){
                throw new \Exception('Login to FortiAuthenticator failed -> ' . curl_error($ch));
            }

            preg_match_all('/^Set-Cookie:\s*([^;]*)/mi', $result, $matches);
            foreach ($matches[1] as $item) {
                parse_str($item, $cookie);
                $cookies = array_merge($cookies, $cookie);
            }

            if ( ! empty($cookies['sessionid'])) {
                $is_logged_in = TRUE;
            }

SSH

SSH IP_ADDR -l USER_NAME

Thursday, 20 December 2018

PHP CURL parallel && write download content on the fly using curl

Apache parallel command:

apt-get install parallel
#execute 4 parallel requests in total, 2 parallel request to be executed at atime
seq 4 | parallel -n0 -j2 "curl -H 'Content-Type: application/json'  -X GET http://172.16.107.209/~jxiang/response_GET.php"
#execute 4 parallel requests in total, 4 parallel request to be executed at atime
seq 4 | parallel -n0  "curl -H 'Content-Type: application/json'  -X GET http://172.16.107.209/~jxiang/response_GET.php"



PHP parallel curl(curl multi):
/**
* Webkul Software.
*
* @category Webkul
* @author Webkul
* @copyright Copyright (c) 2010-2016 Webkul Software Private Limited (https://webkul.com)
* @license https://store.webkul.com/license.html
*/
// array of curl handles
$multiCurl = array();
// data to be returned
$result = array();
// multi handle
$mh = curl_multi_init();
foreach ($ids as $i => $id) {
  // URL from which data will be fetched
  $fetchURL = 'https://webkul.com&customerId='.$id;
  $multiCurl[$i] = curl_init();
  curl_setopt($multiCurl[$i], CURLOPT_URL,$fetchURL);
  curl_setopt($multiCurl[$i], CURLOPT_HEADER,0);
  curl_setopt($multiCurl[$i], CURLOPT_RETURNTRANSFER,1);
  curl_multi_add_handle($mh, $multiCurl[$i]);
}
$index=null;
do {
  curl_multi_exec($mh,$index);
} while($index > 0);
// get content and remove handles
foreach($multiCurl as $k => $ch) {
  $result[$k] = curl_multi_getcontent($ch);
  curl_multi_remove_handle($mh, $ch);
}
// close
curl_multi_close($mh);


 // Open file to write donwload content
                                $fp = fopen('fortipoc_export_all.poc', 'w+');
                                curl_setopt($this->_curl, CURLOPT_FILE, $fp);

                                curl_exec($this->_curl);

Tuesday, 18 December 2018

Go basic knowledge notes

github gin /? (light weight framework)
 
r := gin.default()
 
ss -an | grep 8080 (find out whether a port has been listened)
 
 
import {
 
     classA ? or its just package
}
 
// object autpo
classA.function_in_classA();
 
 
  import {
     classA
 }
 
 
import {
 
     dir
}
 
 
dir.anypkg
 
// gin
c.DefaultQuery("Name", "Guest");
//
c.postForm('value');w
 
 
get name=xxx from url, or use default value guest
 
 
// move route handler to another module
 
// go auto compile, needs to set up a go project name after installation
// installation can be done in make file
 
// build explore?
~./vim (juns git vim file  https://github.com/junxie6/config_centos_v2.git
- git clone, tools will be there )
 
// Then Juns handy tool will be ready to use 
ctrl-j file system
ctrk-k opened file
 
 

GO reverse proxy:
https://github.com/wolfeidau/reverseproxy/blob/master/main.go

Go Installation

GO Installation:
 
 
1. CURL GET tar, unzip in /usr/local
 
 curl 'https://dl.google.com/go/go1.10.3.linux-amd64.tar.gz' -o go.tar.gz \
&& sudo tar -zxvf go.tar.gz -C /usr/local \
&& sudo rm -f go.tar.gz
 
2. SET bash_profile, add to path, and  to export $GO,  $PATH variables to children programs like go
 
if bash never exists:
```
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc

fi


# User specific environment and startup programs

PATH="$PATH:$HOME/.local/bin:$HOME/bin"

export GOPATH="$HOME/go"
export PATH="$PATH:/usr/local/go/bin:$GOPATH/bin:/usr/local/bin" if already there: vim ~/.bash_profile
export GOPATH="$HOME/go"
export PATH="$PATH:/usr/local/go/bin:$GOPATH/bin:/usr/local/bin"

 
 
refresh
source ~/.bash_profile` 
$ echo $GOPATH`

`$ echo $PATH` 
 
 go version
go version go1.10.3 linux/amd64 To build the binary:


```
$ go build main.go
```


To run the binary:

`$ ./main`

go-git

https://github.com/go-gitea/gitea