CK's IT blog: docker

Showing posts with label docker. Show all posts

Wednesday, May 12, 2021

How to install docker client and connect to docker desktop engine on another machine macOS or Windows.

Use Docker Desktop for Windows 10 / macOS as engine and docker client for linux/macos/Android to connect.
Why use docker client ? Because don't want to / cannot install docker engine in the client environment and just want to connect to the docker engine on local LAN.
(1) For Windows 10 Host, after installation of docker desktop
# C:\ProgramData\Docker\config\daemon.json and add

"hosts" : ["tcp://0.0.0.0:2375"],

# change port forwarding to docker wsl backend, run this in powershell admin mode

netsh interface portproxy add v4tov4 listenport=2375 listenaddress=192.168.64.1 connectaddress=127.0.0.1 connectport=2375

# change Windows defender firewall and add incoming rule to enable port 2375 # Refer to this for setting openssh server authorized_keys, with proper file permission https://superuser.com/questions/1445976/windows-ssh-server-refuses-key-based-authentication-from-client

(2) For macOS Host, after installation of docker desktop
# ssh-keygen in client and ssh-copy-id to host, e.g. remote host username with ip address of 192.168.64.1

ssh-keygen -t rsa
ssh-copy-id username@192.168.64.1

# edit sshd_config

sudo vi /private/etc/ssh/sshd_config

# and add

PermitUserEnvironment PATH,LANG

# edit ~/.docker/daemon.json and add

"hosts" : ["tcp://0.0.0.0:2375"],

# add .ssh/environment

PATH=$PATH:/usr/local/bin

# restarting sshd using macOS System Preferences -> Sharing -> File Sharing

(3) To install docker client for Linux using tcp

cd ~/
wget https://download.docker.com/linux/static/stable/x86_64/docker-20.10.6.tgz
tar -xzvf docker-20.10.6.tgz
cd docker
./docker -H tcp://<remote host ip address>:2375 images

(3.1) To install docker client for Termux app of Android and using tcp

wget https://download.docker.com/linux/static/stable/aarch64/docker-20.10.6.tgz

tar xzvf docker-20.10.6.tgz
mv docker/docker /data/data/com.termux/files/usr/bin/
./docker -H tcp://<remote host ip address>:2375 images

(4) To install docker client for macOS using ssh

cd ~/
wget https://download.docker.com/mac/static/stable/x86_64/docker-20.10.6.tgz
#or curl -OL https://download.docker.com/mac/static/stable/x86_64/docker-20.10.6.tgz
#or curl -OL https://download.docker.com/mac/static/stable/aarch64/docker-20.10.6.tgz
tar xzvf docker-20.10.6.tgz
xattr -rc docker
cd docker
sudo mkdir -p /usr/local/bin
sudo mv * /usr/local/bin/
docker -H ssh://username@<remote host ip address> images

(5) Or simply add the corresponding variables to ~/.bashrc

unset DOCKER_HOST
# for tcp connection to Windows 10 host
export DOCKER_HOST=tcp://192.168.64.1:2375

# for ssh connection to macOS host using SSH
export DOCKER_HOST=ssh://user@192.168.64.1

Sunday, February 28, 2021

How to cross compile using Docker Desktop experimental feature buildx

Another demo to show the building of packages using experiemtal feature of Docker desktop. Have to enable the experimental feature to cross compile in other architetcures different from the host machine. E.g. cross compile armv7 packages in amd64 or arm64 host of Mac or PC.

Shell script Select all

If using WSL2 docker CLI in Linux, should enable the experimental feature first.

export DOCKER_CLI_EXPERIMENTAL=enabled

docker buildx create --name mybuilder

docker buildx use mybuilder

docker buildx inspect --bootstrap

docker buildx build --platform linux/arm/v7 --memory="8g" --output "type=docker,push=false,dest=armv7_ql.tar" -f Dockerfile_ql_armv7_1.21 -t armv7/quantlib:1.21 .

Thursday, February 25, 2021

How to cross compile QuantLib in docker

This example demo using dockcross to build Quantlib armv7 library packages and cross compile in host X86_64 machine

Shell script Select all

# install qemu and run in host
sudo apt update
sudo apt install -y qemu-user
qemu-arm Bonds

# Or test run it in armv7 machine e.g. RaspberryPi 3B+
scp Bonds mypi3b:~/.
ssh mypi3b '~/Bonds'

Sunday, February 14, 2021

How to install Windows Subsystem for Linux 2 and docker on Windows 10

Required Windows 10 update 1903 or 1909 or above
(1) Use Windows Power Shell (run as Administrator) to install WSL2

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart

# for Windows update 2004 or later

dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

# for Windows update 1903 or 1909

Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform -NoRestart


# Restart Windows after enable Enable-WindowsOptionalFeature

curl -o $env:userprofile\Desktop\wsl_update_x64.msi https://wslstorestorage.blob.core.windows.net/wslblob/wsl_update_x64.msi

msiexec.exe /I $env:userprofile\Desktop\wsl_update_x64.msi /quiet

wsl --set-default-version 2

(2) Install Ubuntu 20.04 from Windows Store

(3) Install docker (Reference : https://docs.docker.com/engine/install/ubuntu/)

sudo apt update

sudo apt install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common 

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
   
sudo apt update

sudo apt install -y docker-ce docker-ce-cli containerd.io

sudo usermod -aG docker $USER



# Test docker

sudo /etc/init.d/docker start

docker run -e MYSQL_ROOT_PASSWORD=rootpassword -e MYSQL_USER=wpuser -e MYSQL_PASSWORD=wpuserpassword -e MYSQL_DATABASE=wordpressdb --name wordpressdb -d mariadb ; sudo docker run -e WORDPRESS_DB_USER=wpuser -e WORDPRESS_DB_PASSWORD=wpuserpassword -e WORDPRESS_DB_NAME=wordpressdb -p 8080:80 --link wordpressdb:mysql --name wordpress -d wordpress

# browser test enter address http://127.0.0.1:8080/

Test (4) Install docker-compose

sudo apt install python3.8 python3-pip

sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 10

sudo update-alternatives --install /usr/bin/pip pip /usr/bin/pip3 10

sudo pip3 -v install docker-compose

(5) For example, the creation Dockerfile to build for Quantlib juypter notebook server is as below.
P.S. You need 8G RAM to build thia dockerfile using gcc

Shell script Select all

(6) Install youtube-dl for WSL2

wget https://github.com/ytdl-org/youtube-dl/releases/download/2021.12.17/youtube-dl

sudo chmod +x youtube-dl
sudo mv youtube-dl /usr/local/bin/
sudo apt install -y openssl ffmpeg

sudo apt install python-is-python3

youtube-dl -U

youtube-dl -i -f m4a https://youtu.be/e0npW4WoGmc


# use ffmpeg to resize videp
https://ottverse.com/change-resolution-resize-scale-video-using-ffmpeg/

# use ffmpeg to resize image
https://stackoverflow.com/questions/28806816/use-ffmpeg-to-resize-image

(7) Install X Display Server for WSL gui app
https://techcommunity.microsoft.com/t5/windows-dev-appconsult/running-wsl-gui-apps-on-windows-10/ba-p/1493242

Run wsl app batch file
https://gist.githubusercontent.com/zarinfam/eb671a82340eadb5af9026ba6e0b666b/raw/c2d0d947a488e54eeb7fdaafae5464afc778de71/wsl-app-runner.bat

vbscript to start the batch file
https://gist.github.com/zarinfam/5dbc0e10662b79468a3da9a67c107217/raw/cf4d2d3b6095cb3c42fca0df9e86e4d3b2703671/linux-gui-app-runner.vbs

To install OpenSSH Server on Windows 10 https://virtualizationreview.com/articles/2020/05/21/ssh-server-on-windows-10.aspx

To enable ssh login default to wsl ubuntu. Should use powershell with admin authority yo rmable features. https://www.hanselman.com/blog/the-easy-way-how-to-ssh-into-bash-and-wsl2-on-windows-10-from-an-external-machine

In order not to interrupt the ssh server service, should consider to disable sleep when on power adapter. And disable automatic windows update and restart.

Wednesday, April 8, 2020

Swift 5.2 on Ubuntu for Windows 10 bash shell

Prerequisite:
Windows 10 64 bits Home or Pro Edition
Enable WSL via powershell as Administrator

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform

Install Ubuntu App from Microsoft Store

Swift 5.2 on Ubuntu (18.04 LTS) for Windows 10 bash shell

You need to download and extract the Release (Ubuntu 18.04) from swift.org and do these under bash shell of Ubuntu App.

shell script Select all

Follow this guide to install docker on Ubuntu 16.0.4 LTS. Currently 18.0.4 is not working for docker CE on WSL.
https://medium.com/faun/docker-running-seamlessly-in-windows-subsystem-linux-6ef8412377aa

To get Swift 5.2 on Ubuntu 16.0.4

wget https://swift.org/builds/swift-5.2-release/ubuntu1604/swift-5.2-RELEASE/swift-5.2-RELEASE-ubuntu16.04.tar.gz

tar xzvf swift-5.2-RELEASE-ubuntu16.04.tar.gz

sudo mv ~/swift-5.2-RELEASE-ubuntu16.04 /usr/share/swift-5.2

For docker on WSL, you might encounter error on apt-get update with Error in GPG signature. The temporary solution is to change storage driver to vfs as the default overlay2 driver does not work under wsl. But the performance of vfs is really poor.

shell script Select all

Click on the docker label below to see some previous examples on using docker.

For example using the quantlib-python3 docker image
docker run -t -i lballabio/quantlib-python3:latest bash

To test the quantlib-python3 docker image

shell script Select all

Export and Import of container

docker export CONTAINER_NAME | gzip > NAME.gz

zcat NAME.gz | docker import - IMAGE-NAME

Wednesday, August 3, 2016

Bash Shell Built Into Windows 10 Anniversary Update 1607 Build 14393.10

After Windows 10 Anniversary Update, you have to enable "Developer mode" on the "Update & Security" page.
Then go to Control Panel -> Program and Features -> Turn Windows Features on or off -> Windows SubSystem for Linux (Beta), and click OK.

Bash Shell is only for 64-bit versions of Windows 10.

Bash Shortcut Keys
There are a number of very useful shortcut keys you can use in the bash shell:

Ctrl + U: Clears the line from the cursor point back to the beginning.
Ctrl + A: Moves the cursor to the beginning of the line.
Ctrl + E: Moves the cursor to the end of the line.
Ctrl + Left / Right: Moves the cursor to one word Left / Right.
Ctrl + R: Allows you to search through the previous commands.

Batch Rename Files
rename –v 's/foo/bar/g' *

Customize command line prompt

function prompt

{

local WHITE="\[\033[1;37m\]"

local GREEN="\[\033[0;32m\]"

local CYAN="\[\033[0;36m\]"

local GRAY="\[\033[0;37m\]"

local BLUE="\[\033[0;34m\]"

export PS1="${GREEN}\u${CYAN}@${BLUE}\h ${CYAN}\w${GRAY}$ "

}

prompt

where
\u refers to the current username
\h refers to the host name
\w refers to the current file path.

or simply uncomment the line force_color_prompt=yes in ~/.bashrc

You can use Docker for Windows as engine and docker client for linux in bash. "Docker for Windows" requires Windows 10 Pro 64-bit and Hyper-V Package.

To install docker client for Linux under bash shell
cd ~/
wget https://get.docker.com/builds/Linux/x86_64/docker-1.12.0.tgz
tar -xzvf docker-1.12.0.tgz
cd docker
./docker -H tcp://0.0.0.0:2375 ps

Or simply add the corresponding variables to ~/.bashrc
export DOCKER_HOST=tcp://0.0.0.0:2375
export PATH=$PATH:~/docker

To detach the docker tty without exiting the shell, use the escape sequence Ctrl-p + Ctrl-q

Install docker-compose and try docker-compose example as per https://docs.docker.com/compose/wordpress/

shell script Select all

Swift 3 on Ubuntu (14.04 LTS trusty) for Windows 10 bash shell

You need to download and extract the development snapshot (Ubuntu 14.04) from swift.org and do these under bash shell.

shell script Select all

Install node.js and mongo shell for Ubuntu

shell script Select all

File System Access

Ubuntu filesystem from Windows Explorer
C:\Users\%USERNAME%\AppData\Local\Lxss\rootfs
Ubuntu 16.04 filesystem from Windows Explorer, Fall Creators Update (2017 Oct)
C:\Users\%USERNAME%\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc\LocalState\rootfs\

Windows filesystem from Ubuntu bash
/mnt/c/
/mnt/d/

#edit ~/.bashrc
#add export LS_COLORS=$LS_COLORS:'di=1;44:' ; # dir white on blue background

To upgrade to Ubuntu 16.04 after Windows 10's Creators Update (2017 April)
open the Bash shell and run the following command:

sudo do-release-upgrade
lsb_release -a

However, lxrun /uninstall then lxrun /install is faster.

The lastest installtion of WSL is documented at https://docs.microsoft.com/en-us/windows/wsl/install-win10
where different and multiple distros can be installed via Microsoft Store since Fall Creators Update (2017 Oct).

Installation of Swift 3.1.1 in Ubuntu 16.04

shell script Select all

Installation of Swift 4 for Cygwin64

shell script Select all

Tuesday, June 21, 2016

How to run Docker for macOS

1) Download from here -> https://docs.docker.com/docker-for-mac/

2) Drag to Application Folder and double click to start and then check version in Terminal

docker --version

docker info

3) Search and Pull Docker image
E.g.

docker search kitura
docker pull ibmcom/kitura-ubuntu:latest

4) Run image with port forwarding <local port : container port>
E.g.
docker run --cap-add sys_ptrace -p 8095:8095 -t -i ibmcom/kitura-ubuntu:latest /bin/bash

5) Run / Download (if none) additional images and run as daemon

docker run -d -p 8080:8080 --name nodejs node

docker run -d -p 80:80 --name webserver nginx

6) List all downloaded or committed images
docker images

Download old images from
http://mirror.yandex.ru/mirrors/download.openvz.org/template/precreated/
and using import

cat ~/Downloads/ubuntu-10.04-x86.tar.gz | docker import - ubuntu-x86:10.04

or

docker import ubuntu-10.04-x86.tar.gz ubuntu-x86:10.04



docker run -t -i ubuntu-x86:10.04 bash

7) Run image with port forwarding <local port : container port> and bash shell

docker run -p 8095:8095 -t -i ibmcom/kitura-ubuntu:latest bash

docker run -p 80:80 -t -i nginx:latest bash

docker run -p 3000:3000 -t -i node:latest bash

docker run -t -i lballabio/quantlib:latest bash

docker run -t -i lballabio/quantlib-python:latest bash

8) List all containers with <CONTAINER ID>

docker ps -a

#export container

docker export <CONTAINER ID> > ubuntu.tar

Execute container with bash shell
docker exec -t -i <CONTAINER ID> bash

9) Commit changes to image with <newname:newtag>
docker commit <CONTAINER ID> kitura-ubuntu:v2
docker run -p 8095:8095 -it kitura-ubuntu:v2 /bin/bash

This is useful when you cannot start a container manually after docker upgraded. e.g.

docker start <CONTAINER ID> && docker attach <CONTAINER ID>

10) copy docker file to local

docker start <CONTAINER ID>

docker cp <CONTAINER ID>:/root/myfile.tgz .

Docker for Mac OSX data is at
~/Library/Containers/com.docker.docker/Data

To test the kitura docker image for Swift web app

shell script Select all

To test the node docker image for node.js web app

shell script Select all

To test the quantlib or quantlib-python3 docker image

shell script Select all

Try docker-compose example as per https://docs.docker.com/compose/wordpress/

shell script Select all

CK's IT blog