Sunday, February 28, 2021

How to cross compile using Docker Desktop experimental feature buildx

Another demo to show the building of packages using experiemtal feature of Docker desktop. Have to enable the experimental feature to cross compile in other architetcures different from the host machine. E.g. cross compile armv7 packages in amd64 or arm64 host of Mac or PC.
Shell script   Select all
cd $HOME mkdir -p my-quantlib cd my-quantlib # get helloworld.ipynb wget https://raw.githubusercontent.com/lballabio/dockerfiles/master/quantlib-jupyter/Hello%20world.ipynb cat >$HOME/my-quantlib/Dockerfile_ql_armv7_1.21 <<'HEREEOF' # Dockerfile_ql_armv7_1.21 # docker buildx build --platform linux/arm/v7 -f Dockerfile_ql_armv7_1.21 -t armv7/quantlib:1.21 . # Build Quantlib libraries for armv7 in arm64/amd64 host ARG tag=latest FROM arm32v7/ubuntu:18.04 LABEL Description="Provide a building environment where the QuantLib Python jupyter-notebook" RUN apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y build-essential wget libbz2-dev vim git ENV boost_version=1.67.0 ENV boost_dir=boost_1_67_0 # Build boost RUN echo 'Building boost ...' RUN wget --no-check-certificate https://boostorg.jfrog.i o/artifactory/main/release/${boost_version}/source/${boo st_dir}.tar.gz \ && tar xfz ${boost_dir}.tar.gz \ && rm ${boost_dir}.tar.gz \ && cd ${boost_dir} \ && ./bootstrap.sh \ && ./b2 --without-python --prefix=/usr -j 4 link=shared runtime-link=shared install \ && ./b2 --prefix=/Staging/usr install \ && cd .. && rm -rf ${boost_dir} && ldconfig # Build Quantlib C++ RUN echo 'Building Quantlib C++ ...' ENV quantlib_version=1.21 RUN wget https://github.com/lballabio/QuantLib/releases/download/QuantLib-v${quantlib_version}/QuantLib-${quantlib_version}.tar.gz \ && tar xfz QuantLib-${quantlib_version}.tar.gz \ && rm QuantLib-${quantlib_version}.tar.gz \ && cd QuantLib-${quantlib_version} \ && ./configure --prefix=/usr --disable-static CXXFLAGS=-O3 \ && make -j 4 && make install \ && make DESTDIR=/Staging install \ && make clean \ && cd .. && ldconfig # && cd .. && rm -rf QuantLib-${quantlib_version} && ldconfig # Build Quantlib-Python RUN echo 'Build Quantlib-Python ...' RUN apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y swig python3 python3-pip python-dev libgomp1 # Build Quantlib for Python3 RUN echo 'Install Quantlib Python' ENV quantlib_swig_version=1.21 RUN wget https://github.com/lballabio/QuantLib-SWIG/releases/download/QuantLib-SWIG-v${quantlib_swig_version}/QuantLib-SWIG-${quantlib_swig_version}.tar.gz \ && tar xfz QuantLib-SWIG-${quantlib_swig_version}.tar.gz \ && rm QuantLib-SWIG-${quantlib_swig_version}.tar.gz \ && cd QuantLib-SWIG-${quantlib_swig_version} \ && ./configure CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768" PYTHON=/usr/bin/python3 \ && make -C Python && make -C Python check && make -C Python install \ && cd .. && rm -rf QuantLib-SWIG-${quantlib_swig_version} && ldconfig # Build jupyter-notebook server RUN python3 -c "print('\033[91m Building jupyter-notebook server ... \033[0m')" RUN pip3 install --no-cache-dir jupyter jupyterlab matplotlib numpy scipy pandas ipywidgets RISE RUN jupyter-nbextension install rise --py --sys-prefix RUN jupyter-nbextension install widgetsnbextension --py --sys-prefix \ && jupyter-nbextension enable widgetsnbextension --py --sys-prefix # Build Quantlib for Python2 RUN apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y python \ && apt-get clean RUN wget https://bootstrap.pypa.io/2.7/get-pip.py \ && python2 get-pip.py \ && rm get-pip.py RUN wget https://dl.bintray.com/quantlib/releases/QuantLib-SWIG-${quantlib_swig_version}.tar.gz \ && tar xfz QuantLib-SWIG-${quantlib_swig_version}.tar.gz \ && rm QuantLib-SWIG-${quantlib_swig_version}.tar.gz \ && cd QuantLib-SWIG-${quantlib_swig_version} \ && ./configure CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768" \ && make -C Python && make -C Python check && make -C Python install \ && cd .. && rm -rf QuantLib-SWIG-${quantlib_swig_version} && ldconfig RUN pip2 install --no-cache-dir numpy EXPOSE 8888 RUN mkdir /notebooks VOLUME /notebooks COPY *.ipynb /notebooks/ # Starting jupyter-notebook server RUN python3 -c "print('\033[92m Starting jupyter-notebook server at port 8888 \033[0m')" CMD jupyter notebook --no-browser --allow-root --ip=0.0.0.0 --port=8888 --notebook-dir=/notebooks HEREEOF # build and load image (after experimental feature of Docker Desktop is enabled docker buildx build --platform linux/arm/v7 --memory="8g" --output "type=docker,push=false,dest=armv7_ql.tar" -f Dockerfile_ql_armv7_1.21 -t armv7/quantlib:1.21 . docker load < armv7_ql.tar # run image docker run --platform linux/arm/v7 -d -p 8888:8888 --name myquantlibarmv7testing armv7/quantlib:1.21 # list the token of the jupyter-notebook server docker container exec -it myquantlibarmv7testing jupyter notebook list


If using WSL2 docker CLI in Linux, should enable the experimental feature first.
export DOCKER_CLI_EXPERIMENTAL=enabled
docker buildx create --name mybuilder
docker buildx use mybuilder
docker buildx inspect --bootstrap
docker buildx build --platform linux/arm/v7 --memory="8g" --output "type=docker,push=false,dest=armv7_ql.tar" -f Dockerfile_ql_armv7_1.21 -t armv7/quantlib:1.21 .


Thursday, February 25, 2021

How to cross compile QuantLib in docker

This example demo using dockcross to build Quantlib armv7 library packages and cross compile in host X86_64 machine
Shell script   Select all
cd $HOME mkdir -p dockcross cd dockcross cat >$HOME/dockcross/dockcross_ql_1.21 <<'HEREEOF' # dockcross_ql_1.21 # docker build -f dockcross_ql_1.21 -t dockcross/linux-armv7/quantlib:1.21 . # Build Quantlib libraries for armv7 ARG tag=latest FROM dockcross/linux-armv7 ENV DEFAULT_DOCKCROSS_IMAGE dockcross/linux-armv7/quantlib:1.21 ENV CROSS_PREFIX /usr/xcc/armv7-unknown-linux-gnueabi/armv7-unknown-linux-gnueabi/sysroot/usr LABEL Description="Provide a building environment dockcross/linux/armv7 for the QuantLib" RUN apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y build-essential wget vim git ENV boost_version=1.67.0 ENV boost_dir=boost_1_67_0 # Install boost dependencies RUN echo 'Building boost and required packages ...' RUN dpkg --add-architecture armhf \ && apt-get update \ && apt-get download libbz2-1.0:armhf libbz2-dev:armhf liblzma-dev:armhf \ && dpkg-deb -x libbz2-1*armhf.deb ./x \ && dpkg-deb -x libbz2-dev_*armhf.deb ./x \ && dpkg-deb -x liblzma-dev_*armhf.deb ./x \ && mv ./x/usr/lib/arm-linux-gnueabihf/* ${CROSS_PREFIX}/lib/ \ && mv ./x/usr/bin/* ${CROSS_PREFIX}/bin/ \ && mv ./x/usr/include/* ${CROSS_PREFIX}/include/ \ && mv ./x/usr/share/* ${CROSS_PREFIX}/share/ \ && mv ./x/lib/arm-linux-gnueabihf/* ${CROSS_PREFIX}/lib/ \ && rm -fr ./x *.deb && ldconfig RUN wget https://sourceforge.net/projects/libpng/files/zlib/1.2.11/zlib-1.2.11.tar.gz/download -O zlib-1.2.11.tar.gz \ && tar xfz zlib-1.2.11.tar.gz \ && rm zlib-1.2.11.tar.gz \ && cd zlib-1.2.11 \ && ./configure --prefix=${CROSS_PREFIX} && make && make install \ && cd .. && rm -rf zlib-1.2.11 && ldconfig # Build boost RUN wget https://dl.bintray.com/boostorg/release/${boost_version}/source/${boost_dir}.tar.gz \ && tar xfz ${boost_dir}.tar.gz \ && rm ${boost_dir}.tar.gz \ && cd ${boost_dir} \ && ./bootstrap.sh --with-toolset=gcc --prefix=${CROSS_PREFIX} \ && touch user-config.jam \ && echo "using gcc : armv7 : ${CXX} ;" > user-config.jam \ && echo "using mpi ;" >> user-config.jam \ && ./b2 --toolset=gcc-armv7 --address-model=32 --architecture=arm --user-config=./user-config.jam --without-python --prefix=${CROSS_PREFIX} -j 4 link=shared runtime-link=shared install \ && cd .. && rm -rf ${boost_dir} && ldconfig # Build Quantlib C++ RUN echo 'Building Quantlib C++ ...' ENV quantlib_version=1.21 RUN wget https://dl.bintray.com/quantlib/releases/QuantLib-${quantlib_version}.tar.gz \ && tar xfz QuantLib-${quantlib_version}.tar.gz \ && rm QuantLib-${quantlib_version}.tar.gz \ && cd QuantLib-${quantlib_version} \ && ./configure --host=x86_64-linux-gnu --target=armv7-unknown-linux-gnueabi --with-boost-include=${CROSS_PREFIX}/include/boost --with-boost-lib=${CROSS_PREFIX}/lib --prefix=${CROSS_PREFIX} --disable-static CXXFLAGS=-O3 \ && make -j 4 && make install \ && make clean \ && cd .. && ldconfig HEREEOF # build image docker build -f dockcross_ql_1.21 -t dockcross/linux-armv7/quantlib:1.21 . # Test dockcross compile docker run --rm dockcross/linux-armv7/quantlib:1.21 > dockcross-quantlib-armv7 chmod +x dockcross-quantlib-armv7 ./dockcross-quantlib-armv7 bash -c '$CXX -v' # Copy example cpp file from image docker run -v $PWD:/opt/mount --rm --entrypoint cp dockcross/linux-armv7/quantlib:1.21 /work/QuantLib-1.21/Examples/Bonds/Bonds.cpp /opt/mount/Bonds.cpp # Use this command to copy whole directory recursively # docker run -v $PWD:/opt/mount --rm --entrypoint cp dockcross/linux-armv7/quantlib:1.21 -r /work/QuantLib-1.21/Examples /opt/mount/QLExamples # Compile and check binary ./dockcross-quantlib-armv7 bash -c '$CXX Bonds.cpp -lQuantLib -o Bonds -static' file Bonds


# install qemu and run in host
sudo apt update
sudo apt install -y qemu-user
qemu-arm Bonds

# Or test run it in armv7 machine e.g. RaspberryPi 3B+
scp Bonds mypi3b:~/.
ssh mypi3b '~/Bonds'

Sunday, February 14, 2021

How to install Windows Subsystem for Linux 2 and docker on Windows 10

Required Windows 10 update 1903 or 1909 or above
(1) Use Windows Power Shell (run as Administrator) to install WSL2
dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
# for Windows update 2004 or later
dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart
# for Windows update 1903 or 1909
Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform -NoRestart
# Restart Windows after enable Enable-WindowsOptionalFeature curl -o $env:userprofile\Desktop\wsl_update_x64.msi https://wslstorestorage.blob.core.windows.net/wslblob/wsl_update_x64.msi
msiexec.exe /I $env:userprofile\Desktop\wsl_update_x64.msi /quiet
wsl --set-default-version 2


(2) Install Ubuntu 20.04 from Windows Store

(3) Install docker (Reference : https://docs.docker.com/engine/install/ubuntu/)
sudo apt update
sudo apt install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io
sudo usermod -aG docker $USER

# Test docker
sudo /etc/init.d/docker start
docker run -e MYSQL_ROOT_PASSWORD=rootpassword -e MYSQL_USER=wpuser -e MYSQL_PASSWORD=wpuserpassword -e MYSQL_DATABASE=wordpressdb --name wordpressdb -d mariadb ; sudo docker run -e WORDPRESS_DB_USER=wpuser -e WORDPRESS_DB_PASSWORD=wpuserpassword -e WORDPRESS_DB_NAME=wordpressdb -p 8080:80 --link wordpressdb:mysql --name wordpress -d wordpress
# browser test enter address http://127.0.0.1:8080/


Test (4) Install docker-compose
sudo apt install python3.8 python3-pip
sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 10
sudo update-alternatives --install /usr/bin/pip pip /usr/bin/pip3 10
sudo pip3 -v install docker-compose


(5) For example, the creation Dockerfile to build for Quantlib juypter notebook server is as below.
P.S. You need 8G RAM to build thia dockerfile using gcc

Shell script   Select all
cd $HOME mkdir -p my-quantlib cd my-quantlib # get helloworld.ipynb wget https://raw.githubusercontent.com/lballabio/dockerfiles/master/quantlib-jupyter/Hello%20world.ipynb cat >$HOME/my-quantlib/Dockerfile_ql_1.21 <<'HEREEOF' # Dockerfile_ql_1.21 # docker build -f Dockerfile_ql_1.21 -t quantlib:1.21 . # Build Quantlib libraries for amd64 ARG tag=latest FROM ubuntu:20.04 MAINTAINER Luigi Ballabio LABEL Description="Provide a building environment where the QuantLib Python jupyter-notebook" RUN apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y build-essential wget libbz2-dev vim git ENV boost_version=1.67.0 ENV boost_dir=boost_1_67_0 # Build boost RUN echo 'Building boost ...' RUN wget https://dl.bintray.com/boostorg/release/${boost_version}/source/${boost_dir}.tar.gz \ && tar xfz ${boost_dir}.tar.gz \ && rm ${boost_dir}.tar.gz \ && cd ${boost_dir} \ && ./bootstrap.sh \ && ./b2 --without-python --prefix=/usr -j 4 link=shared runtime-link=shared install \ && cd .. && rm -rf ${boost_dir} && ldconfig # Build Quantlib C++ RUN echo 'Building Quantlib C++ ...' ENV quantlib_version=1.21 RUN wget https://dl.bintray.com/quantlib/releases/QuantLib-${quantlib_version}.tar.gz \ && tar xfz QuantLib-${quantlib_version}.tar.gz \ && rm QuantLib-${quantlib_version}.tar.gz \ && cd QuantLib-${quantlib_version} \ && ./configure --prefix=/usr --disable-static CXXFLAGS=-O3 \ && make -j 4 && make check && make install \ && make clean \ && cd .. && ldconfig # && cd .. && rm -rf QuantLib-${quantlib_version} && ldconfig # Build Quantlib-Python RUN echo 'Build Quantlib-Python ...' RUN apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y swig python3 python3-pip python-dev libgomp1 # Build Quantlib for Python3 RUN echo 'Install Quantlib Python' ENV quantlib_swig_version=1.21 RUN wget https://dl.bintray.com/quantlib/releases/QuantLib-SWIG-${quantlib_swig_version}.tar.gz \ && tar xfz QuantLib-SWIG-${quantlib_swig_version}.tar.gz \ && rm QuantLib-SWIG-${quantlib_swig_version}.tar.gz \ && cd QuantLib-SWIG-${quantlib_swig_version} \ && ./configure CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768" PYTHON=/usr/bin/python3 \ && make -C Python && make -C Python check && make -C Python install \ && cd .. && rm -rf QuantLib-SWIG-${quantlib_swig_version} && ldconfig # Build jupyter-notebook server RUN python3 -c "print('\033[91m Building jupyter-notebook server ... \033[0m')" RUN pip3 install --no-cache-dir jupyter jupyterlab matplotlib numpy scipy pandas ipywidgets RISE RUN jupyter-nbextension install rise --py --sys-prefix RUN jupyter-nbextension install widgetsnbextension --py --sys-prefix \ && jupyter-nbextension enable widgetsnbextension --py --sys-prefix # Build Quantlib for Python2 RUN apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y python2 \ && apt-get clean RUN wget https://bootstrap.pypa.io/2.7/get-pip.py \ && python2 get-pip.py \ && rm get-pip.py RUN wget https://dl.bintray.com/quantlib/releases/QuantLib-SWIG-${quantlib_swig_version}.tar.gz \ && tar xfz QuantLib-SWIG-${quantlib_swig_version}.tar.gz \ && rm QuantLib-SWIG-${quantlib_swig_version}.tar.gz \ && cd QuantLib-SWIG-${quantlib_swig_version} \ && ./configure CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768" \ && make -C Python && make -C Python check && make -C Python install \ && cd .. && rm -rf QuantLib-SWIG-${quantlib_swig_version} && ldconfig RUN pip2 install --no-cache-dir numpy EXPOSE 8888 RUN mkdir /notebooks VOLUME /notebooks COPY *.ipynb /notebooks/ # Starting jupyter-notebook server RUN python3 -c "print('\033[92m Starting jupyter-notebook server at port 8888 \033[0m')" CMD jupyter notebook --no-browser --allow-root --ip=0.0.0.0 --port=8888 --notebook-dir=/notebooks HEREEOF # build image docker build -f Dockerfile_ql_1.21 -t quantlib:1.21 . # run image docker run -d -p 8888:8888 --name myquantlibtesting quantlib:1.21 # list the token of the jupyter-notebook server docker container exec -it myquantlibtesting jupyter notebook list


(6) Install youtube-dl for WSL2
wget https://github.com/ytdl-org/youtube-dl/releases/download/2021.12.17/youtube-dl
sudo chmod +x youtube-dl sudo mv youtube-dl /usr/local/bin/ sudo apt install -y openssl ffmpeg
sudo apt install python-is-python3
youtube-dl -U
youtube-dl -i -f m4a https://youtu.be/e0npW4WoGmc
# use ffmpeg to resize videp https://ottverse.com/change-resolution-resize-scale-video-using-ffmpeg/ # use ffmpeg to resize image https://stackoverflow.com/questions/28806816/use-ffmpeg-to-resize-image


(7) Install X Display Server for WSL gui app
https://techcommunity.microsoft.com/t5/windows-dev-appconsult/running-wsl-gui-apps-on-windows-10/ba-p/1493242

Run wsl app batch file
https://gist.githubusercontent.com/zarinfam/eb671a82340eadb5af9026ba6e0b666b/raw/c2d0d947a488e54eeb7fdaafae5464afc778de71/wsl-app-runner.bat

vbscript to start the batch file
https://gist.github.com/zarinfam/5dbc0e10662b79468a3da9a67c107217/raw/cf4d2d3b6095cb3c42fca0df9e86e4d3b2703671/linux-gui-app-runner.vbs



To install OpenSSH Server on Windows 10 https://virtualizationreview.com/articles/2020/05/21/ssh-server-on-windows-10.aspx

To enable ssh login default to wsl ubuntu. Should use powershell with admin authority yo rmable features. https://www.hanselman.com/blog/the-easy-way-how-to-ssh-into-bash-and-wsl2-on-windows-10-from-an-external-machine

In order not to interrupt the ssh server service, should consider to disable sleep when on power adapter. And disable automatic windows update and restart.