Showing posts with label Raspberry Pi. Show all posts
Showing posts with label Raspberry Pi. Show all posts

Monday, May 17, 2021

How to compile Quantlib-Python for Raspberry Pi 4B arm32 and arm64

Raspberry Pi has default gcc-8 and Python 3.7 for its 32 bit / 64 bit buster image. And compiling QuantLib-Python on this machine could have out of memeory error. Cross compiling on docker might have different python version which is not compatible. The trick to compile on Raspberry Pi is to setup swap say 2G and 4G Ram and turn off debug -g flag when compiling as Python package.
Shell script for building arm32 version   Select all
# install necessary packages for building sudo apt update sudo apt install -y build-essential wget libbz2-dev libboost-test1.67.0 libboost-test-dev # Get QuantLib-1.22 and build static library cd ${HOME} wget https://github.com/lballabio/QuantLib/releases/download/QuantLib-v1.22/QuantLib-1.22.tar.gz tar xzf QuantLib-1.22.tar.gz cd QuantLib-1.22/ ./configure --prefix=/usr --disable-shared CXXFLAGS=-O3 make -j 4 && make install sudo ldconfig # Setup and enable swap and check it for at least 2GB. sudo dphys-swapfile setup sudo dphys-swapfile swapon free -mh sudo apt install -y python3 python3-pip python-dev libgomp1 # Get QuantLib-SWIG-1.22 and compile it cd ${HOME} wget --no-check-certificate https://github.com/lballabio/QuantLib-SWIG/releases/download/QuantLib-SWIG-v1.22/QuantLib-SWIG-${quantlib_swig_version}.tar.gz tar xfz QuantLib-SWIG-1.22.tar.gz cd QuantLib-SWIG-1.22/ ./configure CXXFLAGS="-O2 --param ggc-min-expand=1 --param ggc-min-heapsize=32768 -Wno-deprecated-declarations -Wno-misleading-indentation" PYTHON=/usr/bin/python3 # manual compile it and remove the -g flag cd Python/ mkdir -p build/temp.linux-armv7l-3.7/QuantLib export CXX="echo gcc"; python3 setup.py bdist_wheel g++ -fwrapv -O2 -Wall -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DNDEBUG -I/usr/include/python3.7m -I/usr/include -c QuantLib/quantlib_wrap.cpp -o build/temp.linux-armv7l-3.7/QuantLib/quantlib_wrap.o -Wno-unused --param ggc-min-expand=1 --param ggc-min-heapsize=32768 -Wno-deprecated-declarations -Wno-misleading-indentation mkdir -p build/lib.linux-armv7l-3.7/QuantLib/ g++ -shared -Wl,-z,relro -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-armv7l-3.7/QuantLib/quantlib_wrap.o -lQuantLib -o build/lib.linux-armv7l-3.7/QuantLib/_QuantLib.cpython-37m-arm-linux-gnueabihf.so # create wheel file python3 setup.py bdist_wheel # Upgrade PIP and install the wheel file /usr/bin/python3 -m pip install --upgrade pip pip3 install dist/QuantLib-1.22-cp37-cp37m-linux_armv7l.whl # Or alternatively install as site-package sudo python3 setup.py install # Test examples after installation pip3 install pandas python3 examples/bonds.py . . . .


Compiling for Rapberry Pi arm64 is very similar but has to add -fPIC flag for the QuantLib when building static library
Shell script for building arm64 version   Select all
# install necessary packages for building sudo apt update sudo apt install -y build-essential wget libbz2-dev sudo apt install -y libboost-test1.67.0 libboost-test-dev cd ${HOME} wget https://github.com/lballabio/QuantLib/releases/download/1.22/QuantLib-1.22.tar.gz tar xzf QuantLib-1.22.tar.gz cd QuantLib-1.22/ # enable -fPIC flag for building static library ./configure --prefix=/usr --disable-shared CXXFLAGS="-O3 -fPIC" make -j 4 && make install sudo ldconfig # If Raspbeery Pi has 8GB Ram, no need to setup and enable swap sudo apt install -y python3 python3-pip python-dev libgomp1 # Get QuantLib-SWIG-1.22 and compile it cd {HOME} wget https://github.com/lballabio/QuantLib-SWIG/releases/download/QuantLib-SWIG-v1.22/QuantLib-SWIG-1.22.tar.gz tar xzf QuantLib-SWIG-1.22.tar.gz cd QuantLib-SWIG-1.22/ cd Python/ ./configure CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768 -fPIC -Wno-deprecated-declarations -Wno-misleading-indentation" PYTHON=/usr/bin/python3 # manual compile it and remove the -g flag cd Python/ mkdir -p build/temp.linux-aarch64-3.7/QuantLib/ g++ -fwrapv -O2 -Wall -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.7m -I/usr/include -c QuantLib/quantlib_wrap.cpp -o build/temp.linux-aarch64-3.7/QuantLib/quantlib_wrap.o -Wno-unused --param ggc-min-expand=1 --param ggc-min-heapsize=32768 -fno-strict-aliasing -Wno-unused -Wno-uninitialized -Wno-sign-compare -Wno-write-strings -Wno-deprecated-declarations -Wno-misleading-indentation mkdir -p build/lib.linux-aarch64-3.7/QuantLib/ g++ -shared -Wl,-z,relro -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-aarch64-3.7/QuantLib/quantlib_wrap.o -lQuantLib -o build/lib.linux-aarch64-3.7/QuantLib/_QuantLib.cpython-37m-aarch64-linux-gnu.so # create wheel file python3 setup.py bdist_wheel # Upgrade PIP and install the wheel file /usr/bin/python3 -m pip install --upgrade pip pip3 install dist/QuantLib-1.22-cp37-cp37m-linux_aarch64.whl # Or alternatively install as site-package sudo python3 setup.py install # Test examples after installation pip3 install pandas python3 examples/bonds.py


File Download QuantLib-1.22-cp37-cp37m-linux_armv7l.whl https://mega.nz/file/mtJSxZTT#fzDDHw0AIqz-2LIspBGNZLoyW4_MT9qjft_b-ITTA8w

File Download QuantLib-1.22-cp37-cp37m-linux_aarch64.whl https://mega.nz/file/WlAEXJCZ#UKFnlTrfQfRNzFW-OJbXHLFIHwzCw_189HvMa_xU4Oo

Thursday, May 21, 2020

Personal Installation Guide of Raspberry Pi 4B cluster

Why this project with Raspberry Pi cluster ? Because it is cheap and can be used for the many purposes as below.

Hardware

4 x Raspberry Pi 4B with heat sinks
Raspberry Pi Cluster Case 4 layers with Cooling Fan for each layer
4 x MicroSDHC SanDisk 32G Class 10
One MicroSD Adapter for installation of OS
4 x USB-C power cable
4 x Cat 6 LAN cable
USB power supply with 8 USB ports total max 10A
External USB fans connected to USB power supply, important to keep the CPU cool especially when overclock
4 x UPS Battery Case 5V max 3.3A (each with 3 x Panasonic 18650BD 3200mAH batteries)
8 ports Gigabit Ethernet Switch
External RAID-0 disks with 2 x 8TB (WD Ultrastar HC320 7200rpm) storage, USB-C to USB3.0 interface (HD is the most expensive item for this project)
cat /proc/cpuinfo    Select all
Reference : https://www.rs-online.com/designspark/raspberry-pi-3-model-b-vs-3-model-b processor : 0 BogoMIPS : 108.00 Features : fp asimd evtstrm crc32 cpuid CPU implementer : 0x41 CPU architecture: 8 CPU variant : 0x0 CPU part : 0xd08 CPU revision : 3 processor : 1 BogoMIPS : 108.00 Features : fp asimd evtstrm crc32 cpuid CPU implementer : 0x41 CPU architecture: 8 CPU variant : 0x0 CPU part : 0xd08 CPU revision : 3 processor : 2 BogoMIPS : 108.00 Features : fp asimd evtstrm crc32 cpuid CPU implementer : 0x41 CPU architecture: 8 CPU variant : 0x0 CPU part : 0xd08 CPU revision : 3 processor : 3 BogoMIPS : 108.00 Features : fp asimd evtstrm crc32 cpuid CPU implementer : 0x41 CPU architecture: 8 CPU variant : 0x0 CPU part : 0xd08 CPU revision : 3 Hardware : BCM2835 Revision : c03112 Serial : 100000003bc32951 Model : Raspberry Pi 4 Model B Rev 1.2


SD Card images for BerryBoot

BerryBoot(very flexible and allow to add custom OS images for multiboot)
https://www.berryterminal.com/doku.php/berryboot

OS : Ubuntu 18.04.3 LTS
Download from Ubuntu_Server_arm64_18.04.3.img
https://sourceforge.net/projects/berryboot/files/os_images/
or raspberry pi image from
https://wiki.ubuntu.com/ARM/RaspberryPi
or download the latest image for raspi4 and convert to berryboot format as below
shell script    Select all
nohup curl -OL http://cdimage.ubuntu.com/ubuntu/releases/18.04.4/release/ubuntu-18.04.4-preinstalled-server-arm64+raspi4.img.xz & # or download the 32 bits version # nohup curl -OL http://cdimage.ubuntu.com/ubuntu/releases/18.04.4/release/ubuntu-18.04.4-preinstalled-server-armhf+raspi4.img.xz & sudo apt update sudo apt install kpartx squashfs-tools unxz ubuntu-18.04.4-preinstalled-server-arm64+raspi4.img.xz # And follow this guide to create your own berry boot images. # https://www.berryterminal.com/doku.php/berryboot/adding_custom_distributions # Convert arm64+raspi4 to berryboot OS image sudo kpartx -av ubuntu-18.04.4-preinstalled-server-arm64+raspi4.img #sudo mount /dev/mapper/loop0p2 /mnt sudo mount /dev/mapper/loop1p2 /mnt sudo sed -i 's/^\/dev\/mmcblk/#\0/g' /mnt/etc/fstab sudo sed -i 's/^PARTUUID/#\0/g' /mnt/etc/fstab sudo rm -f /mnt/etc/console-setup/cached_UTF-8_del.kmap.gz sudo rm -f /mnt/etc/systemd/system/multi-user.target.wants/apply_noobs_os_config.service sudo rm -f /mnt/etc/systemd/system/multi-user.target.wants/raspberrypi-net-mods.service sudo rm -f /mnt/etc/rc3.d/S01resize2fs_once sudo mksquashfs /mnt Ubuntu_Server_arm64_18.04.4_raspi4.img -comp lzo -e lib/modules sudo umount /mnt sudo kpartx -d ubuntu-18.04.4-preinstalled-server-arm64+raspi4.img # Convert armhf+raspi4 to berryboot OS image unxz ubuntu-18.04.4-preinstalled-server-armhf+raspi4.img.xz sudo kpartx -av ubuntu-18.04.4-preinstalled-server-armhf+raspi4.img sudo mount /dev/mapper/loop1p2 /mnt sudo sed -i 's/^\/dev\/mmcblk/#\0/g' /mnt/etc/fstab sudo sed -i 's/^PARTUUID/#\0/g' /mnt/etc/fstab sudo rm -f /mnt/etc/console-setup/cached_UTF-8_del.kmap.gz sudo rm -f /mnt/etc/systemd/system/multi-user.target.wants/apply_noobs_os_config.service sudo rm -f /mnt/etc/systemd/system/multi-user.target.wants/raspberrypi-net-mods.service sudo rm -f /mnt/etc/rc3.d/S01resize2fs_once sudo mksquashfs /mnt Ubuntu_Server_armhf_18.04.4_raspi4.img -comp lzo -e lib/modules sudo umount /mnt sudo kpartx -d ubuntu-18.04.4-preinstalled-server-armhf+raspi4.img Download links for these 2 converted images and other updated Raspbian images are on the right hand sidebar of this blog. # BerryBoot way to change default OS images on reboot # Reference : https://www.raspberrypi.org/forums/viewtopic.php?t=37861 # Reference : https://yoursunny.com/t/2017/berryboot-reboot-into/


Ubuntu Server image setup

shell script    Select all
plug in the Ethernet cable before boot up login: ubuntu password: ubuntu change ubuntu password once login # Update Server Security # Reference : https://www.raspberrypi.org/documentation/configuration/security.md sudo apt install openssh-server # check hostname hostnamectl # check network interface ifconfig # update some packages sudo apt update sudo apt-get install dpkg sudo apt-get install --reinstall python3-minimal python3-lockfile sudo apt-get install --reinstall python3-twisted sudo apt-get install --reinstall python3 python3-pip sudo apt-get install --reinstall python-minimal python-lockfile sudo apt-get install --reinstall python python-pip # change hostname, change to pi01, pi02 ... # Reference : https://linuxize.com/post/how-to-change-hostname-on-ubuntu-18-04/ sudo hostnamectl set-hostname pi01 hostnamectl # change timezone sudo dpkg-reconfigure tzdata # change eth0 to Static IP # Reference : https://linuxconfig.org/how-to-configure-static-ip-address-on-ubuntu-18-04-bionic-beaver-linux cat /etc/netplan/01-netcfg.yaml network: version: 2 renderer: networkd ethernets: eth0: dhcp4: no addresses: - 10.0.1.XXX/24 gateway4: 10.0.0.1 nameservers: addresses: [8.8.8.8, 1.1.1.1] # Once ready apply changes with: sudo netplan apply # Mount NTFS external raid disk and Install NFS Server # Reference : https://www.tecmint.com/install-nfs-server-on-ubuntu/ # Reference : https://vitux.com/install-nfs-server-and-client-on-ubuntu/ # Check UUID or PARTUUID sudo blkid # add this in /etc/fstab, for example PARTUUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" /media/RAID0WD ntfs defaults,nls=utf8,dmask=0000,fmask=0022,uid=1000,gid=1000,windows_names 0 0 # reboot to check mounted disk sudo reboot # After reboot df -h # add this in /etc/exports, for example /media/RAID0WD 10.0.1.0/24(rw,sync,no_root_squash,no_subtree_check,insecure,anonuid=1000,anongid=1000) echo "/media/RAID0WD 10.0.1.0/24(rw,sync,no_root_squash,no_subtree_check,insecure,anonuid=1000,anongid=1000)" | sudo tee -a /etc/exports # Export and restart NFS Server sudo exportfs -a sudo systemctl restart nfs-kernel-server # Allow nfs on firewall sudo ufw allow from 10.0.1.0/24 to any port nfs showmount -e pi01 # Install Raspberry pi bin and check cpu temperature sudo add-apt-repository ppa:ubuntu-raspi2/ppa sudo apt-get update sudo apt-get install libraspberrypi-bin # My CPU temp=38.0'C vcgencmd measure_temp # Mount NFS from Mac OS X Connect to Server, enter nfs://10.0.1.101/media/RAID0WD # Mount NFS from other Ubuntu nodes # Reference : https://www.raspberrypi.org/documentation/configuration/nfs.md # add this in /etc/fstab, for example 10.0.1.101:/media/RAID0WD /mnt/RAID0WD nfs auto 0 0 echo "10.0.1.101:/media/RAID0WD /mnt/RAID0WD nfs auto 0 0" | sudo tee -a /etc/fstab ## Install Samba for Ubuntu Server # Reference https://linuxize.com/post/how-to-install-and-configure-samba-on-ubuntu-18-04/ # put this in /etc/samba/smb.conf, for example [raidshare] path = /media/RAID0WD browseable = yes guest ok = no read only = no force create mode = 0660 force directory mode = 2770 valid users = ubuntu @ubuntu ## Add Samba password for user ubuntu sudo smbpasswd -a ubuntu # Restart Samba Server sudo systemctl restart smbd # Allow samba on firewall sudo ufw allow 'Samba' # create images folder for berryboot images installation for other nodes cd $HOME ln -sf /media/RAID0WD smb_share cd $HOME/smb_share mkdir -p images # Download required berryboot os images nohup curl -L https://sourceforge.net/projects/berryboot/files/os_images/Ubuntu_Server_arm64_18.04.3.img/download -o Ubuntu_Server_arm64_18.04.3.img & # Check sha1 signature openssl sha1 Ubuntu_Server_arm64_18.04.3.img


Raspbian Buster image setup

shell script    Select all
plug in the Ethernet cable before boot up login: pi password: raspberry change password once login # Reference : https://www.raspberrypi.org/documentation/configuration/security.md sudo apt install openssh-server # check hostname hostname # check network interface ifconfig # change hostname, change to pi01, pi02 ... etc sudo raspi-config -> Select 2. Network Options -> Select N1 Hostname # Enable SSH at the command line using raspi-config sudo raspi-config -> Select 5. Interfacing Options -> Select P2 SSH -> Select Yes # VNC Server at the command line using raspi-config sudo raspi-config -> Select 5. Interfacing Options -> Select P3 VNC -> Select Yes # change timezone sudo dpkg-reconfigure tzdata -> Select the timezone # change locales sudo dpkg-reconfigure locales -> Select the locale # change eth0 to Static IP # Reference : https://pimylifeup.com/raspberry-pi-static-ip-address/ sudo vi /etc/dhcpcd.conf # Restart dhcp sudo service dhcpcd restart # you will have 2 IP addresses, which is good for mounting nfs at start hostname -I # if you have 2 ip addresses and would like to stop the dhcp for eth0, see discussions here. https://raspberrypi.stackexchange.com/questions/52010/set-static-ip-and-stop-dhcp-on-jessie-lite # Mount NTFS external raid disk and Install NFS Server for Raspbian Buster # Check UUID or PARTUUID sudo blkid # add this in /etc/fstab, for example PARTUUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" /media/RAID0WD ntfs defaults,nls=utf8,dmask=0000,fmask=0022,uid=1000,gid=1000,windows_names 0 0 # reboot to check mounted disk sudo reboot # After reboot df -h # Install NFS Server for Raspbian Buster sudo apt install nfs-kernel-server # add this in /etc/exports, for example /media/RAID0WD 10.0.1.0/24(rw,sync,no_root_squash,no_subtree_check,insecure,anonuid=1000,anongid=1000) echo "/media/RAID0WD 10.0.1.0/24(rw,sync,no_root_squash,no_subtree_check,insecure,anonuid=1000,anongid=1000)" | sudo tee -a /etc/exports # Export and restart NFS Server sudo exportfs -a sudo systemctl restart nfs-kernel-server showmount -e pi01 # Mount NFS from Mac OS X Connect to Server, enter nfs://10.0.1.101/media/RAID0WD # Mount NFS from other Raspbian nodes # Reference : https://www.raspberrypi.org/documentation/configuration/nfs.md # add this in /etc/fstab, for example 10.0.1.101:/media/RAID0WD /mnt/RAID0WD nfs auto 0 0 echo "10.0.1.101:/media/RAID0WD /mnt/RAID0WD nfs auto 0 0" | sudo tee -a /etc/fstab # Install Samba for Raspbian Buster sudo apt install samba # put this in /etc/samba/smb.conf, for example [raidshare] path = /media/RAID0WD browseable = yes guest ok = no read only = no force create mode = 0660 force directory mode = 2770 valid users = pi @pi # Add Samba password for user pi sudo smbpasswd -a pi # Restart Samba Server sudo systemctl restart smbd # create images folder for berryboot images installation for other nodes cd $HOME ln -sf /media/RAID0WD smb_share cd $HOME/smb_share mkdir -p images # Download required berryboot os images nohup curl -L https://sourceforge.net/projects/berryboot/files/os_images/Debian_Buster_Raspbian_FULL_2019.10.img/download -o Debian_Buster_Raspbian_FULL_2019.10.img & nohup curl -L https://sourceforge.net/projects/berryboot/files/os_images/Debian_Buster_Raspbian_2019.10.img/download -o Debian_Buster_Raspbian_2019.10.img & # Check sha1 signature openssl sha1 Debian_Buster_Raspbian_FULL_2019.10.img openssl sha1 Debian_Buster_Raspbian_2019.10.img


Nodes pi01 pi02 pi03 pi04 setup

shell script    Select all
Reference : https://magpi.raspberrypi.org/articles/build-a-raspberry-pi-cluster-computer Install OS images to other nodes, change hostname and assign fixed IP address for each node. # Mount NFS for other nodes # Reference : https://www.raspberrypi.org/documentation/configuration/nfs.md # Install package sudo apt install nfs-common # add this in /etc/fstab, for pi02, pi03, pi04 nodes and reboot to be effective 10.0.1.101:/media/RAID0WD /mnt/RAID0WD nfs auto 0 0 echo "10.0.1.101:/media/RAID0WD /mnt/RAID0WD nfs auto 0 0" | sudo tee -a /etc/fstab # Generate ssh key copy it to every other node in the cluster # Reference : http://www.linuxproblem.org/art_9.html # Login pi01 ssh pi@10.0.1.101 ssh-keygen -t rsa ssh-copy-id 10.0.1.102 ssh-copy-id 10.0.1.103 ssh-copy-id 10.0.1.104 exit # Login pi02 ssh pi@10.0.1.102 ssh-keygen -t rsa ssh-copy-id 10.0.1.101 ssh-copy-id 10.0.1.103 ssh-copy-id 10.0.1.104 exit # Login pi03 ssh pi@10.0.1.103 ssh-keygen -t rsa ssh-copy-id 10.0.1.101 ssh-copy-id 10.0.1.102 ssh-copy-id 10.0.1.104 exit # Login pi04 ssh pi@10.0.1.104 ssh-keygen -t rsa ssh-copy-id 10.0.1.101 ssh-copy-id 10.0.1.102 ssh-copy-id 10.0.1.103 exit Install MPI for each node pi01, pi02, pi03, pi04 nodes sudo apt install mpich python3-mpi4py sudo apt install python-mpi4py ##Test01 # Running in pi01 mpirun -n 4 -host 10.0.1.101,10.0.1.102,10.0.1.102,10.0.1.104 hostname ##Test02 # Running in pi01 mkdir -p /media/RAID0WD/Projects ln -sf /media/RAID0WD/Projects $HOME #Create shell script as temp.sh in $HOME/Projects folder cat > $HOME/Projects/temp.sh <<EOF #!/bin/sh echo "\$(hostname)" "\$(vcgencmd measure_temp)" EOF chmod +x $HOME/Projects/temp.sh # create a common Projects folder in all other nodes # all nodes pi02 to pi04 create folder link and assume mounted NFS from pi01 ln -sf /mnt/RAID0WD/Projects $HOME/ # Running in pi01 to pi04 ssh pi@10.0.1.101 mpirun -n 4 -host 10.0.1.101,10.0.1.102,10.0.1.102,10.0.1.104 $HOME/Projects/temp.sh ssh pi@10.0.1.102 mpirun -n 4 -host 10.0.1.101,10.0.1.102,10.0.1.102,10.0.1.104 $HOME/Projects/temp.sh ssh pi@10.0.1.103 mpirun -n 4 -host 10.0.1.101,10.0.1.102,10.0.1.102,10.0.1.104 $HOME/Projects/temp.sh ssh pi@10.0.1.104 mpirun -n 4 -host 10.0.1.101,10.0.1.102,10.0.1.102,10.0.1.104 $HOME/Projects/temp.sh ##Test03 # Add this in /etc/hosts for all nodes pi01, pi02, pi03, pi04 ssh pi@10.0.1.101 echo -e "10.0.1.101\tpi01\n10.0.1.102\tpi02\n10.0.1.103\tpi03\n10.0.1.104\tpi04" | sudo tee -a /etc/hosts ssh pi@10.0.1.102 echo -e "10.0.1.101\tpi01\n10.0.1.102\tpi02\n10.0.1.103\tpi03\n10.0.1.104\tpi04" | sudo tee -a /etc/hosts ssh pi@10.0.1.103 echo -e "10.0.1.101\tpi01\n10.0.1.102\tpi02\n10.0.1.103\tpi03\n10.0.1.104\tpi04" | sudo tee -a /etc/hosts ssh pi@10.0.1.104 echo -e "10.0.1.101\tpi01\n10.0.1.102\tpi02\n10.0.1.103\tpi03\n10.0.1.104\tpi04" | sudo tee -a /etc/hosts #Running on any node cd $HOME/Projects curl -OL https://raw.githubusercontent.com/mpi4py/mpi4py/master/demo/helloworld.py mpirun -n 4 -host pi01,pi02,pi03,pi04 python $HOME/Projects/helloworld.py ##Test04 #Running on exactly 2 processes only cd $HOME/Projects curl -OL https://raw.githubusercontent.com/mpi4py/mpi4py/master/demo/osu_bw.py mpirun -n 2 -host pi01,pi02 python $HOME/Projects/osu_bw.py # create a file $HOME/Projects/4bmachinelist with the list of available pi 4b nodes for mpi # It is not advised to mix pi 4b with 3b together to run mpi, as it will degrade the performance. mpirun -n 2 -machinefile $HOME/Projects/4bmachinelist python $HOME/Projects/osu_bw.py #On each node, launch 1 process only mpirun -npernode 1 -machinefile $HOME/Projects/4bmachinelist $HOME/Projects/temp.sh # -N is same as -npernode, -hostfile is same as -machinefile mpirun -N 1 -hostfile $HOME/Projects/4bmachinelist $HOME/Projects/temp.sh # or mpirun -N 1 -hostfile $HOME/Projects/4bmachinelist bash -c 'echo "$(hostname)" "$(vcgencmd measure_temp)"' | sort ##Test05 # Count how many processes for your cluster, you should get 4 x 4 nodes = 16 processes cd $HOME/Projects curl -L https://github.com/Apress/raspberry-pi-supercomputing/archive/master.zip -o supercomputing.zip unzip supercomputing.zip mpirun -hostfile 4bmachinelist -N 1 python3 $HOME/Projects/raspberry-pi-supercomputing-master/Codes/code/chapter08/prog01.py mpirun -hostfile 4bmachinelist -N 4 python3 $HOME/Projects/raspberry-pi-supercomputing-master/Codes/code/chapter08/prog03.py ##Test06 # Calculate primes # Get the source code from curl -OL https://people.sc.fsu.edu/~jburkardt/py_src/prime_mpi/prime_mpi.py # There are 2 errors to fix before using it # Line 41, there is a missing closing bracket ) at the end # Line 74, should be changed from comm.Reduce ( [ t, MPI.DOUBLE ], [ primes, MPI.INT ], op = MPI.SUM, root = 0 ) to primes = comm.reduce ( t, op = MPI.SUM, root = 0 ) # Test the time required among different processes by running the below # assuming maximum 4 processes (pi 4 CPU has 4 cores) per node cd $HOME/Projects mpirun -n 4 -hostfile 4bmachinelist python3 prime_mpi.py mpirun -n 8 -hostfile 4bmachinelist python3 prime_mpi.py mpirun -n 12 -hostfile 4bmachinelist python3 prime_mpi.py mpirun -n 16 -hostfile 4bmachinelist python3 prime_mpi.py #On each node, launch 3 and 4 processes, and time their differences cd $HOME/Projects time mpirun -N 3 -hostfile 4bmachinelist python3 prime_mpi.py time mpirun -N 4 -hostfile 4bmachinelist python3 prime_mpi.py # Please post your results of Test06 here in the comment ##Test07 # Collective communication using the scatter function example cd ~/Projects curl -L https://pythonprogramming.net/scatter-gather-mpi-mpi4py-tutorial/ | grep -A13 -B1 "from mpi4py" | sed '1d' > sct9.py time mpirun -N 4 -hostfile 4bmachinelist python sct9.py ##Test08 # Collective communication using the gather function example cd ~/Projects curl -L https://pythonprogramming.net/mpi-gather-command-mpi4py-python/ | grep -A19 -B1 "from mpi4py" | sed '1d' > sct10.py time mpirun -N 4 -hostfile 4bmachinelist python sct10.py ##Test09 # mpicc examples cd $HOME/Projects curl -L https://github.com/wesleykendall/mpitutorial/archive/gh-pages.zip -o mpitutorial.zip unzip mpitutorial.zip cd mpitutorial-gh-pages/tutorials/mpi-reduce-and-allreduce/code make time mpirun -N 4 -hostfile $HOME/Projects/4bmachinelist reduce_avg 100000000 time mpirun -N 4 -hostfile $HOME/Projects/4bmachinelist reduce_stddev 100000000


Optional Server or services setup

shell script    Select all
# Nginx (a lightweght webserver) and fast php plugin sudo apt-get install nginx cd; ln -s /usr/share/nginx/html . # check ip address and use browser connect to test web server hostname -I ifconfig eth0 ifconfig wlan0 # for wireless lan ip addr | grep -Po '(?!(inet 127.\d.\d.1))(inet \K(\d{1,3}\.){3}\d{1,3})' # install php in nginx sudo apt install php-fpm php-curl php-gd php-cli php7.3-opcache php-mbstring php-xml php-zip # link the html folder to home cd $HOME sudo chown pi:pi /var/www/html ln -sf /var/www/html . # create testing php page cat > ~/html/info.php <<EOF <?php phpinfo(); ?> EOF # add in /etc/php/7.3/fpm/pool.d/www.conf user = pi group = pi # enable php in nginx and edit this file sudo vi /etc/nginx/sites-enabled/default # and change or add the followings in server section: server { ... # Add index.php to the list if you are using PHP index index.html index.htm index.php index.nginx-debian.html; ... ## Begin - PHP location ~ \.php$ { # Choose either a socket or TCP/IP address fastcgi_pass unix:/var/run/php/php7.3-fpm.sock; # fastcgi_pass 127.0.0.1:9000; fastcgi_split_path_info ^(.+\.php)(/.+)$; fastcgi_index index.php; include fastcgi_params; fastcgi_param SCRIPT_FILENAME $document_root/$fastcgi_script_name; } ## End - PHP ## Begin - Security # deny all direct access for these folders location ~* /(.git|cache|bin|logs|backups|tests)/.*$ { return 403; } # deny running scripts inside core system folders location ~* /(system|vendor)/.*\.(txt|xml|md|html|yaml|php|pl|py|cgi|twig|sh|bat)$ { return 403; } # deny running scripts inside user folder location ~* /user/.*\.(txt|md|yaml|php|pl|py|cgi|twig|sh|bat)$ { return 403; } # deny access to specific files in the root folder location ~ /(LICENSE.txt|composer.lock|composer.json|nginx.conf|web.config|htaccess.txt|\.htaccess) { return 403; } ## End - Security ... } # # see instructions for php7 here -> https://getgrav.org/blog/raspberrypi-nginx-php7-dev # reload web server and test # check to ensure the /var/run/php/php7.3-fpm.sock file exists sudo service nginx restart sudo service php7.3-fpm restart ls -l /var/run/php/php7.3-fpm.sock # Use this command to check whether the web server is working or not curl -L http://127.0.0.1/ curl -L http://127.0.0.1/info.php # python cgi plugin for nginx see here # install minidlna as a media server sudo apt install minidlna # edit /etc/minidlna.conf and add the followings media_dir=V,/media/RAID0WD/MyMovie friendly_name=MyMovie # edit /etc/default/minidlna and add the followings USER="root" GROUP="root" # reload minidlna sudo service minidlna restart sudo service minidlna force-reload # see what network services are working on raspberry pi sudo netstat -ntlp # free ddns no-ip.com Free sign-up and have 3 Hostnames but need to Confirm Every 30 Days # After sign-up, manual update of IP address # Use this command to obtain your public IP address and update on their website host myip.opendns.com resolver1.opendns.com | grep myip # and then download and Install the dynamic update client for Linux https://www.noip.com/support/knowledgebase/installing-the-linux-dynamic-update-client/ You can have 3 host names and have to install virtual host in nginx web server. In server settings of /etc/nginx/sites-available/default, use this settings to map to different html subfolders for different hosts server { ... server_name ~^(.*)\.(.*)\.(.*)$; set $host_name $1; set $subdomain_name $2; set $domain_name $3; root /var/www/html/$host_name.$subdomain_name.$domain_name; ... } # Suppose myhostname1, myhostname2 and myhostname3 are the hostnames obtained from no-ip.com # Setup html root for mutli-hosts as above settings in nginx sudo chown -R pi:pi /var/www/html mkdir -p /var/www/html/10.0.1.101 mkdir -p /var/www/html/127.0.0.1 mkdir -p /var/www/html/myhostname1.ddns.net mkdir -p /var/www/html/myhostname2.ddns.net mkdir -p /var/www/html/myhoatname3.ddns.net # Restart web server to be effective sudo service nginx restart sudo service php7.3-fpm restart # Test web server after restart services cd /var/www/html/ cp index.nginx-debian.html 10.0.1.101 curl -L http://10.0.1.101/ cd /var/www/html/ cp info.php 127.0.0.1/ curl -L http://127.0.0.1/info.php # Test webserver from ddns cd /var/www/html/myhostname1.ddns.net curl -L https://getgrav.org/blog/raspberrypi-nginx-php7-dev -o index.html # make sure your hone router tcp port 80 has been forwarded to your internal Pi host and test with curl -L http://myhostname1.ddns.net/ # And also test the webserver from the browser on Phone # How to access the server and nodes from Android Phone # Recommend Termux from Google Play Store # It use the Volume Up key + keyboard to enter special control characters and can install packages # Reference : https://wiki.termux.com/wiki/Touch_Keyboard apt update apt upgrade # Install ssh and login server apt install openssh ssh-copy-id pi@10.0.1.101 ssh pi@10.0.1.101 # Assume ddns is setup ssh-copy-id pi@myhostname.ddns.net ssh pi@myhostname.ddns.net # Install python 3 apt search python apt install python # To improve command line history productivity, please refer Terminal history usage tips : https://www.howtogeek.com/howto/44997/how-to-use-bash-history-to-improve-your-command-line-productivity/amp/ # ssh forwarding from Android Device (better to have Android tablet with keyboard and mouse) # Reference: https://wiki.termux.com/wiki/Main_Page # Need to install VNC Viewer and Termux from Google Play Store # Install and start vncserver in Termux vncserver -localhost export DISPLAY=":1" # ssh forwarding and login pi ssh -Y pi@10.0.1.104 # install and run jupyter-notebook from pi (after installation of tensorflow) pip3 install jupyter jupyter-notebook & # run the juypter notebook example from https://colab.research.google.com/github/lmoroney/io19/blob/master/Zero%20to%20Hero/Rock-Paper-Scissors.ipynb # After session ended and kill vncserver in Termux vncserver -kill :1 # free ssl certificate for web server https://letsencrypt.org/getting-started/ # After you have your ddns host and nginx running on your Pi, follow the certbot instructions here # Reference : https://certbot.eff.org/lets-encrypt/debianbuster-nginx # set up certbot and obtain ssl certifcate for nginx sudo apt-get install certbot python-certbot-nginx sudo certbot --nginx crontab -e # add this entry to automate letencrypt certificate renewal 43 6 * * * certbot renew --renew-hook "systemctl reload nginx" Certificate and chain have been saved at: /etc/letsencrypt/live/myhostname1.ddns.net/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/myhostname1.ddns.net/privkey.pem Your cert will expire on (90 days after). To obtain a new or tweaked version of this certificate in the future, simply run certbot again with the "certonly" option. To non-interactively renew *all* of your certificates, run "certbot renew" sudo certbot certonly sudo certbot certificates sudo systemctl reload nginx # make sure your home router tcp port 443 has been forwarded to your internal Pi host and test with curl -L https://myhostname1.ddns.net/ # Install OpenVPN Server After you have the ddns hostname or fixed IP address, setup OpenVPN as per instructions here # Reference : https://www.pcmag.com/how-to/how-to-create-a-vpn-server-with-raspberry-pi curl -L https://install.pivpn.io | bash # Choose OpenVPN and use udp port 1194 # Also need to assign fixed IP to your Pi # After installation, sudo reboot to be effective # forward udp port 1194 to the internal IP address of your Pi # Client access # Create configuration file e.g. iphone, macbook etc. pivpn add # Client access Use "OpenVPN Connect App" for phone and notebook send the profile to the client by email and import, e.g. for iPhone # Makeuse of X11 forwarding to run graphical applications see https://kb.iu.edu/d/bdnt For macOS High Sierra or above, please see this guide to install xQuartz. https://www.unixtutorial.org/get-x11-forwarding-in-macos-high-sierra/
You can install it from https://www.xquartz.org or via "sudo port -v install xorg" in the terminal. For In Windows 10 WSL, and install XServer in Windows 10 such as xming -> https://sourceforge.net/projects/xming/ and set DISPLAT for WSL x11-apps export DISPLAY=localhost:0.0 export DISPLAY=:0 For Linux, there is built-in support. In Terminal, type ssh -Y pi@10.0.1.101 # After login Raspberry pi sudo apt-get install idle3 idle & # run scratch sudo apt-get install scratch scratch & # run codeblocks sudo apt install codeblocks codeblocks & If you get "cannot open display error", see discussions here. https://superuser.com/questions/310197/how-do-i-fix-a-cannot-open-display-error-when-opening-an-x-program-after-sshi # run browser chromium-browser & # Access Raspberry Pi Desktop Remotely Use Windows Remote Desktop Client to connect to the Raspberry Pi. # Install xrdp and reboot the pi after installation sudo apt install xrdp sudo systemctl restart xrdp # For macOS, there is "Microsoft Remote Desktop Connection Client for Mac" in the Mac App Store. # setup cron job to backup project data # Reference : https://www.raspberrypi.org/documentation/linux/usage/cron.md Create a shell script e.g. cat > /home/pi/backup.sh <<EOF #!/bin/sh cd /media/RAID0WD; tar --exclude='./Projects/Downloads' -zcf Projects-backup/"Projects$(date '+%Y%m%d').tar.gz" Projects EOF chmod a+x /home/pi/backup.sh mkdir -p /media/RAID0WD/Projects-backup # add this entry in the crontab 0 0 * * * /home/pi/backup.sh #List crontab crontab -l # Overclock # Reference : https://www.seeedstudio.com/blog/2020/02/12/how-to-safely-overclock-your-raspberry-pi-4-to-2-147ghz/ #Install docker # Not for berryboot images curl -sSL https://get.docker.com | sh sudo usermod -aG docker $USER sudo newgrp docker sudo apt install libffi-dev libssl-dev python3 python3-pip sudo apt-get remove python-configparser sudo pip3 -v install docker-compose # test docker hello-world docker run hello-world cd $HOME mkdir my-wordpress cd my-wordpress cat > docker-compose.yaml <<EOF version: '3.2' services: db: image: hypriot/rpi-mysql volumes: - "./.data/db:/var/lib/mysql" restart: always environment: MYSQL_ROOT_PASSWORD: wordpress MYSQL_DATABASE: wordpress MYSQL_USER: wordpress MYSQL_PASSWORD: wordpress wordpress: depends_on: - db image: wordpress:latest links: - db ports: - "8000:80" restart: always environment: WORDPRESS_DB_HOST: db:3306 WORDPRESS_DB_PASSWORD: wordpress EOF docker-compose up -d # then try browser http://localhost:8000/ # stop the docker-compose example docker-compose down # test nodejs cd $HOME git clone https://github.com/hypriot/rpi-node-haproxy-example cd rpi-node-haproxy-example docker-compose up curl http://localhost:80 curl http://localhost:70 docker-compose stop # test nodejs + mongodb cd $HOME git clone https://github.com/hagaik/easy-node-authentication.git cd easy-node-authentication cat > config/database.js <<EOF // config/database.js module.exports = { 'url' : 'mongodb://mongo:27017' // looks like mongodb://<user>:<pass>@mongo.onmodulus.net:27017/Mikha4ot }; EOF cat > Dockerfile <<EOF FROM hypriot/rpi-node # Create app directory WORKDIR /usr/src/app # Install app dependencies COPY package*.json ./ RUN npm install # Copy app source code COPY . . #Expose port and start application EXPOSE 8080 CMD [ "npm", "start" ] EOF cat > docker-compose.yaml <<EOF version: "3.2" services: db: image: dhermanns/rpi-mongo volumes: - "data-volume:/data/db" restart: always ports: - "27017:27017" expose: - "27017" webm: build: . depends_on: - db ports: - "8080:8080" expose: - "8080" volumes: data-volume: EOF # build and start as daemon docker-compose up --build -d # then try browser http://localhost:8080/ # stop and down docker-compose down -v # move docker data-root to nfs mounted drive e.g. /mnt/docker-data sudo service docker stop cat << EOF | sudo tee -a /etc/docker/daemon.json { "storage-driver": "overlay", "data-root": "/mnt/docker-data" } EOF sudo rsync -aP /var/lib/docker/ /mnt/docker-data sudo mv /var/lib/docker /var/lib/docker.old sudo service docker start


Install tensorflow and horovod for Pi 4 cluster

shell script    Select all
Install all these packages for every nodes in the Pi 4 cluster in order to run horovod with tensorflow # Install tenorflow 2.1.0 # Reference https://qengineering.eu/install-tensorflow-2.1.0-on-raspberry-pi-4.html # or https://qengineering.eu/install-tensorflow-2.2.0-on-raspberry-pi-4.html for tenorflow 2.2.0 cd ~/Projects sudo apt-get install gfortran sudo apt-get install libhdf5-dev libc-ares-dev libeigen3-dev sudo apt-get install libatlas-base-dev libopenblas-dev libblas-dev sudo apt-get install liblapack-dev cython sudo pip3 install pybind11 sudo apt-get install python3-h5py # upgrade pip3 and check pip3 version curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py python3 get-pip.py --force-reinstall # pip 20.1.1 from /home/pi/.local/lib/python3.7/site-packages/pip (python 3.7) python3 --version pip3 --version # download the wheel wget https://github.com/Qengineering/Tensorflow-Raspberry-Pi/raw/master/tensorflow-2.1.0-cp37-cp37m-linux_armv7l.whl # install TensorFlow 2.1.0 python3 -m pip install --user tensorflow-2.1.0-cp37-cp37m-linux_armv7l.whl # test run examples as in how-to-install-tensorflow-with-gpu python3 -m pip install --user matplotlib python3 -m pip install --user pandas python3 -m pip install --user keras scikit-learn cd ~/Projects curl -L https://tinyurl.com/tensorflowwin | grep -A7 tftest.py | sed '1,2d' > tftest.py python3 tftest.py curl -L https://tinyurl.com/tensorflowwin | grep -A129 irislearn.py | sed '1,8d' > irislearn.py curl -L https://tinyurl.com/tensorflowwin | grep -A150 iris.data.nbsp | sed '1d' > iris.data python3 irislearn.py curl -L https://tinyurl.com/tensorflowwin | grep -A37 keraslearn.py | sed '1,3d' > keraslearn.py curl -L https://tinyurl.com/tensorflowwin | grep -A768 pima-indians-diabetes.data.nbsp | sed '1d' > pima-indians-diabetes.data python3 keraslearn.py # install horovod # https://github.com/horovod/horovod#install python3 -m pip install cffi>=1.4.0 cloudpickle python3 -m pip install horovod # Reboot to make it effective sudo reboot # First test run the simple hellohorovod.py cd ~/Projects cat > hellohorovod.py <<EOF from mpi4py import MPI import horovod.tensorflow as hvd # Split COMM_WORLD into subcommunicators subcomm = MPI.COMM_WORLD.Split(color=MPI.COMM_WORLD.rank % 2, key=MPI.COMM_WORLD.rank) # Initialize Horovod hvd.init(comm=subcomm) print('COMM_WORLD rank: %d, Name: %s, Horovod rank: %d' % (MPI.COMM_WORLD.rank, MPI.Get_processor_name(), hvd.rank())) EOF # run it with cd ~/Projects horovodrun -np 1 -H localhost:1,pi02:1,pi03:1,pi04:4 python3 hellohorovod.py # Then try run the tensorflow v2 example as below # please be warned that and watch out the temperature of PIs. # https://github.com/horovod/horovod/blob/master/examples/tensorflow2_keras_mnist.py # For non-keras, the sample is https://github.com/horovod/horovod/blob/master/examples/tensorflow2_mnist.py cd ~/Projects wget https://raw.githubusercontent.com/horovod/horovod/master/examples/tensorflow2_keras_mnist.py time horovodrun -np 4 -H localhost:1,pi02:1,pi03:1,pi04:1 python3 tensorflow2_keras_mnist.py time horovodrun -np 8 -H localhost:2,pi02:2,pi03:2,pi04:2 python3 tensorflow2_keras_mnist.py time horovodrun -np 12 -H localhost:3,pi02:3,pi03:3,pi04:3 python3 tensorflow2_keras_mnist.py # 1 slot per node has the faster performance for the Pi cluster # As with the release of the new 8GB Pi4B model, it is possible to increase the number of slots for these 8GB machines. # append self public key to self authorized list ssh pi@10.0.1.101 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys exit ssh pi@10.0.1.102 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys exit ssh pi@10.0.1.103 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys exit ssh pi@10.0.1.104 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys exit # create ~/Projects/myhostfile cat > ~/Projects/myhostfile <<EOF pi01 slots=1 pi02 slots=1 pi03 slots=1 pi04 slots=1 EOF # running in background with nohup cd ~/Projects nohup horovodrun -np 4 -hostfile myhostfile python3 tensorflow2_keras_mnist.py & cat ~/Projects/nohup.out exit # create folder in non-nfs ext4 partition folder if running in nodes other than pi01 mkdir -p ~/horovod ssh pi01 'mkdir -p ~/horovod' ssh pi02 'mkdir -p ~/horovod' ssh pi03 'mkdir -p ~/horovod' ssh pi04 'mkdir -p ~/horovod' # When running in pi02, pi03 and pi04, it cannot start in the nfs shared folder if it is not an ext4 partition. # e.g. when start in pi03 ssh pi03 cd ~/horovod horovodrun -np 4 -hostfile ~/Projects/myhostfile python3 ~/Projects/tensorflow2_keras_mnist.py # run it with horovod and time and redirect outputfile to keras_mnist.np4.out cd ~/horovod nohup bash -c 'time horovodrun -np 4 -hostfile ~/Projects/myhostfile3 python3 ~/Projects/tensorflow2_keras_mnist.py' &> keras_mnist.np4.out & # The time for number of nodes for this testing nohup bash -c 'time horovodrun -np 4 -H pi01:1,pi02:1,pi03:1,pi04:1 --output-filename logs python3 ~/Projects/tensorflow2_keras_mnist.py' &> nohup.out.np4_1111 & nohup bash -c 'time horovodrun -np 3 -H pi01:1,pi02:1,pi03:1 --output-filename logs python3 ~/Projects/tensorflow2_keras_mnist.py' &> nohup.out.np3_111 & nohup bash -c 'time horovodrun -np 2 -H pi01:1,pi02:1 --output-filename logs python3 ~/Projects/tensorflow2_keras_mnist.py' &> nohup.out.np2_11 & nohup bash -c 'time horovodrun -np 1 -H pi01:1 --output-filename logs python3 ~/Projects/tensorflow2_keras_mnist.py' &> nohup.out.np1_1 & nohup.out.np4_1111:real not yet done (may be 50m) nohup.out.np3_111:real 88m13.692s nohup.out.np2_11:real 129m11.322s nohup.out.np1_1:real 207m28.119s


Some shortcuts

shell script    Select all
# Offending ECDSA key # Offending ECDSA key in $HOME/.ssh/known_hosts:5 # For Mac sed sed -i '' '5d' $HOME/.ssh/known_hosts # For GNU sed sed -i '5d' $HOME/.ssh/known_hosts # print line 5 to 6 of a text file sed -n '5,6p' $HOME/.ssh/known_hosts # print line 1 and Line 5 to 6 of a text file sed -n -e '1p' -e '5,6p' $HOME/.ssh/known_hosts # add all the hosts to the ~/.ssh/known_hosts file using ssh-keyscan # first login pi01 10.0.1.101 ssh pi@10.0.1.101 ssh-keyscan -t rsa,dsa pi02,pi01,pi04 > ~/.ssh/known_hosts






Saturday, May 16, 2020

Running python cgi scripts on the Raspberry Pi nginx

Basically, the setup of python plugin cgi is here at https://www.takaitra.com/running-python-cgi-scripts-on-the-raspberry-pi/
and the enhanced functions of uwsgi-cgi documentation here https://uwsgi-docs.readthedocs.io/en/latest/CGI.html
Except the followings:

# Build and install the uwsgi with the cgi plugin
wget https://projects.unbit.it/downloads/uwsgi-latest.tar.gz
tar zxvf uwsgi-latest.tar.gz 
cd uwsgi-2.0.18
# compile as cgi plugin
make PROFILE=cgi
sudo cp uwsgi /usr/local/bin/



# Create the file /etc/uwsgi.ini
plugins = cgi
# change to unix sock
socket = /tmp/uwsgi.sock
#socket = 127.0.0.1:9000
module = pyindex
cgi = /var/www/html/cgi-bin
#cgi = /usr/share/nginx/www
cgi-allowed-ext = .py
cgi-helper = .py=python
logger=file:/tmp/uwsgi-error.log
uid = www-data
gid = www-data



# Add a location to the /etc/nginx/sites-available/default
location ~ \.py$ {
  # uwsgi_pass 127.0.0.1:9000;
  # change to unix sock
  uwsgi_pass unix:/tmp/uwsgi.sock;
  include uwsgi_params;
  uwsgi_modifier1 9;
}


Test this python script to show the temperature of Raspberry Pi in a html web page

/var/www/html/cgi-bin/temp.py    Select all
#!/usr/bin/env python import os # Return CPU temperature as a character string def getCPUtemperature(): res = os.popen('vcgencmd measure_temp').readline() return(res.replace("temp=","").replace("'C\n","")) #We have to print a valid HTTP header first so the browser will know how to decode the data print "Content-type: text/html\n\n" temp1=getCPUtemperature() print temp1


/var/www/html/temp.html    Select all
<html> <head> <title>Pi Temp</title> <script src="http://code.jquery.com/jquery-1.10.1.min.js"></script> </head> <body> <h1>Temp from Pi</h1> <script> $(document).ready(function () { var interval = 500; //number of milli seconds between each call var refresh = function() { $.ajax({ url: "temp.py", cache: false, success: function(html) { $('#pi-temp-here').html(html); setTimeout(function() { refresh(); }, interval); } }); }; refresh(); }); </script> <div id="pi-temp-here"></div> </body> </html>


// sed 's/<\([^>]*\)>/\<\1\>/g;'


Shell script    Select all
# Append video Group to www-date user sudo usermod -aG video www-data # reboot the Raspberry Pi and test the python cgi script sudo reboot http://127.0.0.1/temp.html




To install FastCGI for php in ngnix, please follow this guide -> https://getgrav.org/blog/raspberrypi-nginx-php7-dev

If you use buster, it will install the latest php-7.3, so change everything from 7.2 from this guide to 7.3 and the installation of packages will be
sudo apt-get update
sudo apt-get install php php-curl php-gd php-fpm php-cli php-opcache php-mbstring php-xml php-zip


#add in /etc/php/7.3/fpm/pool.d/www.conf
user = pi
group = pi


#reload web server and test
#check to ensure the /var/run/php/php7.3-fpm.sock file exists
sudo service nginx restart
sudo service php7.3-fpm restart




Wednesday, April 15, 2020

How to create bootable DEBIAN PIXEL, STRETCH, BUSTER and Ubuntu 18.04 on one USB stick

This follows the previous post on "How to create bootable PIXEL USB stick for Mac


The Debian PIXEL is for x86 platforms. The PIXEL ISO, the latest is a 2.0GB download is here
https://downloads.raspberrypi.org/rpd_x86/images/rpd_x86-2017-06-23/2017-06-22-rpd-x86-jessie.iso

Raspberry Pi Desktop image Debian STRETCH latest is 2019-04-11-rpd-x86-stretch.iso (2.4G) is here
https://downloads.raspberrypi.org/rpd_x86/images/rpd_x86-2019-04-12/2019-04-11-rpd-x86-stretch.iso

The new RASPBERRY PI DESKTOP image Debian BUSTER 2020-02-12-rpd-x86-buster.iso (3.13GB) is here
https://downloads.raspberrypi.org/rpd_x86/images/rpd_x86-2020-02-14/2020-02-12-rpd-x86-buster.iso

And the first task is to create a USB stick for Mac/PC for three Debian OSs with persistent feature. Important: The Linux kernel might not be able to boot up for the modern MacBook Retina Display driver. You might need the original Ubuntu 18.04 image in later task for these MacBooks.

shell script    Select all
And here are the instructions to create EFI bootable USB stick on Mac # Running on Mac # list disk volumes diskutil list # assume format USB stick (128G) on /dev/disk1 with 2 partitions 16g and remaining 111g respectively # if for USB stick (64G) on /dev/disk1 with 2 partitions 8g and remaining 55g respectively # if for 32G USB stick, the 2 partition sizes can be divided into 4g and remaining 27g respectively sudo diskutil partitionDisk /dev/disk1 MBRFormat FAT32 LINUX 16g FAT32 PERSISTENCE 0b # for older Mac OSX 10.6, the partition type is "MS-DOS FAT32" # sudo diskutil partitionDisk /dev/disk1 MBRFormat "MS-DOS FAT32" LINUX 16g "MS-DOS FAT32" PERSISTENCE 0b # sudo diskutil partitionDisk /dev/disk1 MBRFormat "MS-DOS FAT32" LINUX 8g "MS-DOS FAT32" PERSISTENCE 0b mkdir -p /Volumes/LINUX/efi/boot # Download Enterprise-0.4.1.tar.gz to ~/Download # from https://github.com/SevenBits/Enterprise/releases cd ~/Downloads curl -OL https://github.com/SevenBits/Enterprise/releases/download/v0.4.1/Enterprise-0.4.1.tar.gz tar xzvf Enterprise-0.4.1.tar.gz Copy efi boot files to the USB stick cp ~/Downloads/Enterprise-0.4.1/*.efi /Volumes/LINUX/efi/boot/ Copy the jessie and buster iso to the USB stick cp ~/Downloads/2017-06-22-rpd-x86-jessie.iso /Volumes/LINUX/efi/boot/pixel.iso cp ~/Downloads/2019-04-11-rpd-x86-stretch.iso /Volumes/LINUX/efi/boot/stretch.iso cp ~/Downloads/2020-02-12-rpd-x86-buster.iso /Volumes/LINUX/efi/boot/buster.iso # create enterprise.cfg cat > /Volumes/LINUX/efi/boot/enterprise.cfg << EOF autoboot 0 entry Debian BUSTER non-persistence family Debian iso buster.iso initrd /live/initrd1.img kernel /live/vmlinuz1 findiso=/efi/boot/buster.iso boot=live config live-config quiet splash EOF # umount disk sudo diskutil unmountDisk disk1 # Reboot Mac and press Option key on restart and select EFI Boot for boot menu # Running on Debian PIXEL sudo fdisk -l #assume /dev/sdb is the USB stick sudo fdisk /dev/sdb # (d) (2) to delete partition 2 # and then (n) (p) (2) to recreate primary partition 2 for Linux in fdisk # (w) to write to partition table and quit fdisk # Reboot to let partition table effective # Running on Debian BUSTER # unmount /dev/sdb2 sudo umount /dev/sdb2 # format and label /dev/sdb2 sudo mkfs.ext4 -L persistence /dev/sdb2 # rename /dev/sdb2 if manually afterwards # sudo e2label /dev/sdb2 "persistence" # create persistence.conf sudo mkdir -p /mnt/persistence sudo mount -t ext4 /dev/sdb2 /mnt/persistence echo / union | sudo tee /mnt/persistence/persistence.conf #unmount /dev/sdb2 sudo umount /dev/sdb2 # Reboot # Running on Mac # recreate enterprise.cfg with persistence cat > /Volumes/LINUX/efi/boot/enterprise.cfg << EOF entry Debian BUSTER persistence family Debian iso buster.iso initrd /live/initrd1.img kernel /live/vmlinuz1 findiso=/efi/boot/buster.iso boot=live config live-config quiet splash persistence entry Debian STRETCH persistence family Debian iso stretch.iso initrd /live/initrd1.img kernel /live/vmlinuz1 findiso=/efi/boot/stretch.iso boot=live config live-config quiet splash persistence entry Debian PIXEL persistence family Debian iso pixel.iso initrd /live/initrd1.img kernel /live/vmlinuz1 findiso=/efi/boot/pixel.iso boot=live config live-config quiet splash persistence entry Debian BUSTER non-persistence family Debian iso buster.iso initrd /live/initrd1.img kernel /live/vmlinuz1 findiso=/efi/boot/buster.iso boot=live config live-config quiet splash EOF # umount disk sudo diskutil unmountDisk disk1 # Reboot and Running on Debian BUSTER to verify the persistence mounting df -h


For additional configuration and setting, or the Ubuntu images http://releases.ubuntu.com/16.04/ or http://releases.ubuntu.com/18.04/, and need to create a third partition casper-rw in Linux to enable persistence. please refer to the previous post.
For Ubuntu images, the related boot menu entries are as below, the non-persistent entry is used to setup the persistence and casper-rw partitions for the USB stick. The nomodeset is used when the video driver biit up failed with black screen. The Ubuntu 18.04 image should be more compatible with the modern Retina display
.
enterprise.cfg    Select all
entry Ubuntu 16.04.6 persistent family Ubuntu iso ubuntu-16.04.6.iso initrd /casper/initrd kernel /casper/vmlinuz findiso=/efi/boot/ubuntu-16.04.6.iso file=/cdrom/preseed/ubuntu.seed boot=casper persistent nomodeset --- entry Ubuntu 18.04.4 persistent family Ubuntu iso ubuntu-18.04.4.iso initrd /casper/initrd kernel /casper/vmlinuz findiso=/efi/boot/ubuntu-18.04.4.iso boot=casper persistent quiet splash --- entry Ubuntu 18.04.4 non-persistent family Ubuntu iso ubuntu-18.04.4.iso initrd /casper/initrd kernel /casper/vmlinuz findiso=/efi/boot/ubuntu-18.04.4.iso boot=casper quiet splash ---


Saturday, December 31, 2016

How to create bootable PIXEL USB stick for Mac

The Debian+PIXEL is for x86 platforms. The PIXEL ISO, which is a 1.3GB download.


The latest RASPBERRY PI DESKTOP image is DEBIAN STRETCH (add nomodeset in enterprise.cfg) https://downloads.raspberrypi.org/rpd_x86/images/rpd_x86-2017-12-01/2017-11-16-rpd-x86-stretch.iso

Reference : Fixing booting of the x86 image on Macs

Instructions to create EFI bootable USB stick for Mac. (tested working on my old MacBook Air 11-inch, Late 2010, Intel Core 2 Duo)

shell script    Select all
And here are the instructions to create EFI bootable USB stick on Mac # Running on Mac # list disk volumes diskutil list # assume format USB stick (64G) on /dev/disk1 with 2 partitions 8g and remaining 55g respectively # if for 32G USB stick, the 2 partition sizes can be divided into 4g and remaining 27g respectively sudo diskutil partitionDisk /dev/disk1 MBRFormat FAT32 LINUX 8g FAT32 PERSISTENCE 0b # for older Mac OSX 10.6, the partition type is "MS-DOS FAT32" # sudo diskutil partitionDisk /dev/disk1 MBRFormat "MS-DOS FAT32" LINUX 8g "MS-DOS FAT32" PERSISTENCE 0b mkdir -p /Volumes/LINUX/efi/boot # Download Enterprise-0.4.0.tar.gz to ~/Download # from https://github.com/SevenBits/Enterprise/releases cd ~/Downloads curl -OL https://github.com/SevenBits/Enterprise/releases/download/v0.4.0/Enterprise-0.4.0.tar.gz tar xzvf Enterprise-0.4.0.tar.gz cp ~/Downloads/Enterprise-0.4.0/*.efi /Volumes/LINUX/efi/boot/ cp ~/Downloads/2016-12-13-pixel-x86-jessie.iso /Volumes/LINUX/efi/boot/boot.iso # create enterprise.cfg cat > /Volumes/LINUX/efi/boot/enterprise.cfg << EOF autoboot 0 entry Debian family Debian initrd /live/initrd1.img kernel /live/vmlinuz1 findiso=/efi/boot/boot.iso boot=live config live-config quiet splash EOF # umount disk sudo diskutil unmountDisk disk1 # Reboot Mac and press Option key on restart and select EFI Boot for boot menu # Running on Debian PIXEL sudo fdisk -l #assume /dev/sdb is the USB stick sudo fdisk /dev/sdb # (d) (2) to delete partition 2 # and then (n) (p) (2) to recreate primary partition 2 for Linux in fdisk # (w) to write to partition table and quit fdisk # Reboot to let partition table effective # Running on Debian PIXEL # unmount /dev/sdb2 sudo umount /dev/sdb2 # format and label /dev/sdb2 sudo mkfs.ext4 -L persistence /dev/sdb2 # rename /dev/sdb2 if manually # sudo e2label /dev/sdb2 "persistence" # create persistence.conf sudo mkdir -p /mnt/persistence sudo mount -t ext4 /dev/sdb2 /mnt/persistence echo / union | sudo tee /mnt/persistence/persistence.conf #unmount /dev/sdb2 sudo umount /dev/sdb2 # Reboot # Running on Mac # recreate enterprise.cfg with persistence cat > /Volumes/LINUX/efi/boot/enterprise.cfg << EOF autoboot 0 entry Debian family Debian initrd /live/initrd1.img kernel /live/vmlinuz1 findiso=/efi/boot/boot.iso boot=live config live-config quiet splash persistence EOF # Reboot and Running on Debian PIXEL to verify the persistence mounting df -h


Additional configurations or installation for PIXEL using Terminal
shell script    Select all
# username of this image is pi # password is raspberry #keyboard configuration #Preferences -> Mouse and Keyboard Settings -> Keyboard -> Keyboard Layout... sudo dpkg-reconfigure keyboard-configuration #reload the keymap (need reboot to be effective) sudo invoke-rc.d keyboard-setup start #reset timezone sudo dpkg-reconfigure tzdata #generate locale sudo dpkg-reconfigure locales sudo locale-gen en_US.UTF-8 # requires reboot sudo update-locale LC_ALL="en_US.UTF-8" #scan private hidden SSID network sudo iwlist wlan0 scanning essid "yourSSID" # then edit /etc/wpa_supplicant/wpa_supplicant.conf and add scan_ssid=1 in the network block for auto scan #reset wireless interface sudo ifdown wlan0 sudo ifup wlan0 # or sudo service network-manager restart # For Buster or above sudo vi /etc/network/interfaces.d/wlan0 wpa-scan-ssid 1 wpa-ssid "Your SSID network" wpa-key-mgmt WPA-PSK wpa-psk "password" # then reboot #reinstall sshd on the system sudo apt-get update sudo apt-get purge openssh-server sudo apt-get install -y openssh-server ssh # List available upgrade sudo apt-get update sudo apt-get -V -u --assume-no upgrade #two fingers tap on touch pad for Right Click synclient tapbutton2=3 #node.js x86 (32 bit) download cd $HOME wget --no-check-certificate https://nodejs.org/dist/v6.9.2/node-v6.9.2-linux-x86.tar.xz tar xJvf node-v6.9.2-linux-x86.tar.xz # put this in $HOME/.bashrc export PATH=$HOME/node-v6.9.2-linux-x86/bin:$PATH #google app engine for python download cd $HOME wget --no-check-certificate https://storage.googleapis.com/appengine-sdks/featured/google_appengine_1.9.49.zip unzip google_appengine_1.9.49.zip # put this in $HOME/.bashrc export PATH=$HOME/google_appengine:$PATH #Traditional Chinese input method, need reboot to be effective and control+space to activate #Choose Auto in Preferences -> Input Method sudo apt-get install -y scim-tables-zh im-config # install additional Chinese fonts sudo apt-get install -y ttf-wqy-microhei ttf-wqy-zenhei xfonts-wqy # install vlc sudo apt-get install -y vlc browser-plugin-vlc # restart control panel lxpanelctl restart #install spotify #add repo and certificate see instructions here https://www.spotify.com/download/linux/ sudo apt-get update sudo apt-get install -y spotify-client Here is Visual Studio Code (32 bits) for Debian https://go.microsoft.com/fwlink/?LinkID=760680








EFI bootable USB stick for Ubuntu 16.10 Exton x64 platforms (PC or Mac)

exton-os-64bit-mate-refracta-1840mb-161231.iso (which is Ubuntu 16.10 for x64 platforms) can be downloaded from here https://sourceforge.net/projects/exton-os/ and copy to /Volumes/LINUX/efi/boot/ as boot.iso

or alternatively use the exton-os-light-64bit-isohybrid-970mb-161021.iso image from https://sourceforge.net/projects/exton-os/files/

The setup for this image for persistence in Mac is similar to the Debian PIXEL above
except the enterprise.cfg content should be
shell script    Select all
cat > /Volumes/LINUX/efi/boot/enterprise.cfg << EOF autoboot 0 entry Ubuntu family Ubuntu initrd /live/initrd.img kernel /live/vmlinuz findiso=/efi/boot/boot.iso boot=live username=live config live-config splash persistence EOF


Additional configuration and installation for Ubuntu 16.10 Exton x64 using Terminal
shell script    Select all
# additional installation for swift 3.0 for Exton OS Ubuntu 16.10 # username of this Ubuntu image is root # password is root sudo apt-get update sudo apt-get install -y libicu-dev clang-3.6 git cd $HOME # To download swift 3 release for Ubuntu 16.04 instead as the one for ubuntu1610 has bugs wget --no-check-certificate https://swift.org/builds/swift-3.0.2-release/ubuntu1604/swift-3.0.2-RELEASE/swift-3.0.2-RELEASE-ubuntu16.04.tar.gz tar xzvf $HOME/swift-3.0.2-RELEASE-ubuntu16.04.tar.gz export PATH=$HOME/swift-3.0.2-RELEASE-ubuntu16.04/usr/bin:$PATH #wget --no-check-certificate https://swift.org/builds/development/ubuntu1610/swift-DEVELOPMENT-SNAPSHOT-2017-01-05-a/swift-DEVELOPMENT-SNAPSHOT-2017-01-05-a-ubuntu16.10.tar.gz #tar xzvf $HOME/swift-DEVELOPMENT-SNAPSHOT-2017-01-05-a-ubuntu16.10.tar.gz #export PATH=$HOME/swift-DEVELOPMENT-SNAPSHOT-2017-01-05-a-ubuntu16.10/usr/bin:$PATH sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-3.6 100 sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-3.6 100 swift --version #node.js x64 (64 bit) download cd $HOME wget --no-check-certificate https://nodejs.org/dist/v6.9.3/node-v6.9.3-linux-x64.tar.xz tar xJvf node-v6.9.3-linux-x64.tar.xz # put this in $HOME/.bashrc export PATH=$HOME/node-v6.9.3-linux-x64/bin:$PATH node --version npm --version #install spotify sudo apt-get update sudo apt-get install -y spotify-client #Traditional Chinese input method sudo apt-get install -y ibus-cangjie ibus restart #then choose input method in System -> Preferences -> Other # install R # see insturctions here https://cloud.r-project.org/bin/linux/ubuntu/README.html cd $HOME echo deb https://cran.cnr.berkeley.edu//bin/linux/ubuntu yakkety/ | sudo tee /etc/apt/sources.list.d/r.list sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9 gpg --keyserver keyserver.ubuntu.com --recv-key E084DAB9 gpg -a --export E084DAB9 | sudo apt-key add - sudo apt-get update sudo apt-get install -y r-base # download R Studio from https://www.rstudio.com/products/rstudio/download/ and install sudo apt-get install -y libjpeg62 libgstreamer0.10-0 libgstreamer-plugins-base0.10-0 cd ~/Downloads wget --no-check-certificate https://download1.rstudio.org/rstudio-1.0.136-amd64.deb sudo dpkg -i rstudio-1.0.136-amd64.deb # install Quantlib 1.9.1 sudo apt-get update sudo apt-get install -y libboost1.60-all-dev cd $HOME wget https://ncu.dl.sourceforge.net/project/quantlib/QuantLib/1.9.1/QuantLib-1.9.1.tar.gz tar xzvf QuantLib-1.9.1.tar.gz cd QuantLib-1.9.1 ./autogen.sh ./configure --prefix=/usr make -j9 sudo make install # Install RQuantLib Package in R # install.packages("RQuantLib") # install Quantlib-Python cd $HOME wget https://jaist.dl.sourceforge.net/project/quantlib/QuantLib/1.9/other%20languages/QuantLib-SWIG-1.9.tar.gz tar xzvf QuantLib-SWIG-1.9.tar.gz cd QuantLib-SWIG-1.9 ./autogen.sh ./configure make -C Python sudo make -C Python install # upgrade pip sudo apt-get update sudo apt-get install -y python-pip pip install --upgrade pip sudo pip install numpy # install ipython sudo pip install ipython # install jupyter notebook sudo pip install jupyter # List available upgrade sudo apt-get update sudo apt list --upgradeable # test QuantLib and QuantLib-Python cd $HOME cat > $HOME/qlversion.cpp <<EOF #include <iostream> #include <ql/version.hpp> int main() { std::cout << "Current QL Version:" << QL_LIB_VERSION << std::endl; return 0; } EOF g++ qlversion.cpp -o qlversion ./qlversion cat > $HOME/swap.py <<EOF import numpy as np import QuantLib as ql # Set Evaluation Date today = ql.Date(31,3,2015) ql.Settings.instance().setEvaluationDate(today) # Setup the yield termstructure rate = ql.SimpleQuote(0.03) rate_handle = ql.QuoteHandle(rate) dc = ql.Actual365Fixed() disc_curve = ql.FlatForward(today, rate_handle, dc) disc_curve.enableExtrapolation() hyts = ql.YieldTermStructureHandle(disc_curve) discount = np.vectorize(hyts.discount) start = ql.TARGET().advance(today, ql.Period('2D')) end = ql.TARGET().advance(start, ql.Period('10Y')) nominal = 1e7 typ = ql.VanillaSwap.Payer fixRate = 0.03 fixedLegTenor = ql.Period('1y') fixedLegBDC = ql.ModifiedFollowing fixedLegDC = ql.Thirty360(ql.Thirty360.BondBasis) index = ql.Euribor6M(ql.YieldTermStructureHandle(disc_curve)) spread = 0.0 fixedSchedule = ql.Schedule(start, end, fixedLegTenor, index.fixingCalendar(), fixedLegBDC, fixedLegBDC, ql.DateGeneration.Backward, False) floatSchedule = ql.Schedule(start, end, index.tenor(), index.fixingCalendar(), index.businessDayConvention(), index.businessDayConvention(), ql.DateGeneration.Backward, False) swap = ql.VanillaSwap(typ, nominal, fixedSchedule, fixRate, fixedLegDC, floatSchedule, index, spread, index.dayCounter()) engine = ql.DiscountingSwapEngine(ql.YieldTermStructureHandle(disc_curve)) swap.setPricingEngine(engine) print(swap.NPV()) print(swap.fairRate()) EOF python swap.py sudo apt-get install -y git cd $HOME git clone git://github.com/mmport80/QuantLib-with-Python-Blog-Examples.git cd QuantLib-with-Python-Blog-Examples/ python blog_frn_example.py








If for Debian PIXEL-x86 and Ubuntu-16.10 Exton-x64 together in one USB stick with persistence
shell script    Select all
# Running on Mac # copy iso to USB stick rm -f /Volumes/LINUX/efi/boot/boot.iso cp ~/Downloads/2016-12-13-pixel-x86-jessie.iso /Volumes/LINUX/efi/boot/pixel.iso cp ~/Downloads/exton-os-64bit-mate-refracta-1840mb-161231.iso /Volumes/LINUX/efi/boot/exton.iso # set up enterprise.cfg in USB stick and assume persistence is already formatted as ext4 with persistence.conf cat > /Volumes/LINUX/efi/boot/enterprise.cfg << EOF entry Debian PIXEL family Debian iso pixel.iso initrd /live/initrd1.img kernel /live/vmlinuz1 findiso=/efi/boot/pixel.iso boot=live config live-config quiet splash persistence entry Debian STRETCH family Debian iso stretch.iso initrd /live/initrd1.img kernel /live/vmlinuz1 findiso=/efi/boot/stretch.iso boot=live config live-config quiet splash nomodeset persistence entry Ubuntu 16.10 Exton family Ubuntu iso exton.iso initrd /live/initrd.img kernel /live/vmlinuz findiso=/efi/boot/exton.iso boot=live username=live config live-config splash persistence EOF








Add the Original Ubuntu 16.04 from Ubuntu to the EFI bootable USB stick

For the Original ubuntu-16.04.1-desktop-amd64.iso image, add it by copying the downloaded iso image to the LINUX FAT32 partition and create the enterprise.cfg as below
shell script    Select all
# Running on Mac # copy iso to USB stick rm -f /Volumes/LINUX/efi/boot/boot.iso cp ~/Downloads/2016-12-13-pixel-x86-jessie.iso /Volumes/LINUX/efi/boot/pixel.iso cp ~/Downloads/exton-os-64bit-mate-refracta-1840mb-161231.iso /Volumes/LINUX/efi/boot/exton.iso cp ~/Downloads/ubuntu-16.04.1-desktop-amd64.iso /Volumes/LINUX/efi/boot/ubuntu.iso cat > /Volumes/LINUX/efi/boot/enterprise.cfg << EOF entry Debian PIXEL family Debian iso pixel.iso initrd /live/initrd1.img kernel /live/vmlinuz1 findiso=/efi/boot/pixel.iso boot=live config live-config quiet splash persistence entry Debian STRETCH family Debian iso stretch.iso initrd /live/initrd1.img kernel /live/vmlinuz1 findiso=/efi/boot/stretch.iso boot=live config live-config quiet splash nomodeset persistence entry Ubuntu 16.10 Exton family Ubuntu iso exton.iso initrd /live/initrd.img kernel /live/vmlinuz findiso=/efi/boot/exton.iso boot=live username=live config live-config splash persistence entry Ubuntu 16.04-1 family Ubuntu iso ubuntu.iso initrd /casper/initrd.lz kernel /casper/vmlinuz.efi findiso=/efi/boot/ubuntu.iso file=/cdrom/preseed/ubuntu.seed boot=casper persistent quiet splash --- EOF # Reboot and running in the new Ubuntu 16.04-1 # it requires a Linux partition named casper-rw in order to enable the persistent feature. # So we need to create a third partition in Linux and then format it. # Running on Ubuntu 16.04 # First remove the second partition and recreate second partition with smaller size and add a third partition to it sudo fdisk -l #assume /dev/sdb is the USB stick # umount the partition 2 in order to modify it sudo umount /dev/sdb2 # partition the USB stick sudo fdisk /dev/sdb # (d) (2) to delete partition 2 # and then (n) (p) (2) to recreate primary partition 2 for Linux in fdisk # choose size say 20G if for 64G USB stick # and then (n) (p) (3) to recreate primary partition 3 for Linux in fdisk # choose size say remaining sectors # (w) to write to partition table and quit fdisk # Reboot to let partition table effective # Running on Ubuntu 16.04-1 # format the second partition with label persistence and add persistence.conf sudo mkfs.ext4 -L persistence /dev/sdb2 sudo mkdir -p /media/ubuntu/persistence sudo mount /dev/sdb2 /media/ubuntu/persistence echo / union | sudo tee /media/ubuntu/persistence/persistence.conf # format the third partition with label casper-rw sudo mkfs.ext4 -L casper-rw /dev/sdb3 # Reboot to start other configuration and installation


Additional configuration and installation for Ubuntu 16.04-1 using Terminal
shell script    Select all
# additional installation for swift 3.0.2 for Ubuntu 16.04 # username of this Ubuntu image is ubuntu # password is ubuntu # Additional installation needs to add these lines in /etc/apt/sources.list echo "deb http://archive.ubuntu.com/ubuntu/ xenial universe" | sudo tee --append /etc/apt/sources.list echo "deb http://archive.ubuntu.com/ubuntu/ xenial-updates universe" | sudo tee --append /etc/apt/sources.list echo "deb http://security.ubuntu.com/ubuntu/ xenial-security universe" | sudo tee --append /etc/apt/sources.list # To install swift 3.0.2 release for Ubuntu 16.04-1 is sudo apt-get update # for AppStream cache update failed error sudo chmod 777 /var/cache/app-info/xapian/default -R sudo apt-get update sudo apt-get install -y libicu-dev clang-3.6 git cd $HOME wget --no-check-certificate https://swift.org/builds/swift-3.0.2-release/ubuntu1604/swift-3.0.2-RELEASE/swift-3.0.2-RELEASE-ubuntu16.04.tar.gz tar xzvf $HOME/swift-3.0.2-RELEASE-ubuntu16.04.tar.gz export PATH=$HOME/swift-3.0.2-RELEASE-ubuntu16.04/usr/bin:$PATH sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-3.6 100 sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-3.6 100 swift --version #node.js x64 (64 bit) download cd $HOME wget --no-check-certificate https://nodejs.org/dist/v6.9.3/node-v6.9.3-linux-x64.tar.xz tar xJvf node-v6.9.3-linux-x64.tar.xz # put this in $HOME/.bashrc export PATH=$HOME/node-v6.9.3-linux-x64/bin:$PATH node --version npm --version #install spotify sudo apt-get update sudo apt-get install -y spotify-client #Traditional Chinese input method sudo apt-get install -y ibus-cangjie ibus restart # install R # see insturctions here https://cloud.r-project.org/bin/linux/ubuntu/README.html cd $HOME echo deb https://cran.cnr.berkeley.edu//bin/linux/ubuntu xenial/ | sudo tee /etc/apt/sources.list.d/r.list sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9 gpg --keyserver keyserver.ubuntu.com --recv-key E084DAB9 gpg -a --export E084DAB9 | sudo apt-key add - sudo apt-get update sudo apt-get install -y r-base # download R Studio from https://www.rstudio.com/products/rstudio/download/ and install sudo apt-get install -y libjpeg62 libgstreamer0.10-0 libgstreamer-plugins-base0.10-0 cd ~/Downloads wget --no-check-certificate https://download1.rstudio.org/rstudio-1.0.136-amd64.deb sudo dpkg -i rstudio-1.0.136-amd64.deb # install Quantlib 1.9.1 sudo apt-get update sudo apt-get install -y libboost1.58-all-dev cd $HOME wget https://ncu.dl.sourceforge.net/project/quantlib/QuantLib/1.9.1/QuantLib-1.9.1.tar.gz tar xzvf QuantLib-1.9.1.tar.gz cd QuantLib-1.9.1 ./autogen.sh ./configure --prefix=/usr make -j9 # uninstall old library # cd ~/Downloads/QuantLib-1.9 # sudo make uninstall sudo make install # Install RQuantLib Package in R # install.packages("RQuantLib") # install Quantlib-Python cd $HOME wget http://jaist.dl.sourceforge.net/project/quantlib/QuantLib/1.9/other%20languages/QuantLib-SWIG-1.9.tar.gz tar xzvf QuantLib-SWIG-1.9.tar.gz cd QuantLib-SWIG-1.9 ./autogen.sh ./configure # if there is "out of memory" error use this below to configure # ./configure CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768" make -C Python # uninstall old QuantLib-Python package # sudo -H pip uninstall QuantLib-Python sudo make -C Python install # upgrade pip and install numpy sudo apt-get update sudo apt-get install -y python-pip sudo -H pip install --upgrade pip sudo -H pip install numpy # test QuantLib and QuantLib-Python cd $HOME cat > $HOME/qlversion.cpp <<EOF #include <iostream> #include <ql/version.hpp> int main() { std::cout << "Current QL Version:" << QL_LIB_VERSION << std::endl; return 0; } EOF g++ qlversion.cpp -o qlversion ./qlversion sudo apt-get install -y git cd $HOME git clone git://github.com/mmport80/QuantLib-with-Python-Blog-Examples.git cd QuantLib-with-Python-Blog-Examples/ python blog_frn_example.py # List available upgrade sudo apt-get update sudo apt list --upgradeable


install Emscripten SDK for WebAssembly
shell script    Select all
# need Debian Stretch 32 bits image or Ubuntu 64 bits # need cmake for building in Debian Stretch(32 bits) sudo apt-get update sudo apt-get install cmake libxml2-dev ocaml python-yaml # Get the emsdk repo git clone https://github.com/juj/emsdk.git # Enter that directory cd emsdk # Fetch the latest registry of available tools. ./emsdk update # Download and install the latest SDK tools. Need rebuilding libraries in Debian Stretch 32 bits ./emsdk install latest # Make the "latest" SDK "active" for the current user. (writes ~/.emscripten file) ./emsdk activate latest # Activate PATH and other environment variables in the current terminal source ./emsdk_env.sh # Test mkdir $HOME/hello cd $HOME/hello cat > $HOME/hello/hello_world.c <<EOF #include <stdio.h> int main() { printf("hello, world!\n"); return 0; } EOF # compile and test emcc hello_world.c node a.out.js emcc hello_world.c -s WASM=0 -o hello.html # start webserver and test http://localhost:8080/hello.html emrun --no_browser --port 8080 .


Wednesday, June 22, 2016

How to build Node.js REST APIs in Raspberry Pi 3

This project is based on the "Build a REST API with node.js" video tutorial from Udemy


The project file can be downloaded here : https://mega.nz/#!z9oFlTgJ!geCPHjoXJ3rpm1OdldU8s5RBSJxLKiAxh_axvTbODB4

The node.js ARMv7 version is v4.4.3 and npm 2.15.1
which (ARMv7) can be downloaded from https://nodejs.org/en/download/
The project also use packages like bcryptjs, body-parser, express, jsonwebtoken and mongoose and integrate with mongodb. - No-SQL db.

To install mongodb in Raspberrry Pi, use sudo apt-get install mongodb

The following enhancements were made to the original project code

1. add shell scripts and use curl for API testing (test1.sh ... test6.sh), as POSTMAN is only available in google chrome
2. add js code to avoid creating of duplicated user (test using test2.sh)
3. add test to user login and obtain Authorization sessionToken (test using test3.sh)
4. add js code to check Authorization before CRUD (create, read, update and delete definition record)
5. add js code to check existence of definition id before update and delete record (test using test5.sh)
6. shell script to get sessionToken in each test for instruction to further test on update and delete definition (test using test5.sh)
7. add js code to create log when create, update and delete definition record (test using test4.sh / test5.sh and then test6.sh)

To use mongo to admin user
mongo
> use workouts
> db.addUser({ user: "test", pwd: "test", roles:["readWrite", "dbAdmin"]});

or use createUser as below
> db.createUser({ user: "test", pwd: "test", roles:["readWrite", "dbAdmin"]});

To use mongo to display mongodb info
mongo
> use workouts
> show collections
> db.users.find()
> db.definitions.find().pretty()
> db.logs.find().pretty()

To use mongo to remove object
> db.definitions.remove({_id:ObjectId("573c29c0f11288dc9997d0")})

Instructions to use
-----------------------
1. First install node.js and mongodb and check that if mongod is running successfully in background. (If you use windows / OSX version of mongodb, please make sure the required data path is created before running mongod.exe / mongod)
2. use mongo to create admin user as above
3. Download the project file and unzip and install packages and start
bash    Select all
unzip nodeapi.zip cd nodeapi npm install npm start & # test ./test1.sh ./test2.sh ./test3.sh ... ./test6.sh # kill background job kill $!



Tuesday, March 15, 2016

Personal Installation Guide for Raspberry Pi 3 with Swift

Hardware List from RS:
Raspberry Pi 3 Model B SBC
Heatsink BGA 12x12 27K/W black
Official Pi 3 Power Supply Black
Official Pi 3 Black/Grey Case

Accessories:
MicroSDHC Card (with Adapter) 32G Class 10
USB SSD Harddisk 240G
USB Keyboard and Mouse

SD Card images for multiboot
1. NOOBS(easy to install but hard to reconfigure or add custom image for multiboot)
#Download NOOBS using Mac Terminal and write to MicroSDHC Card (with Adapter and mounted in Mac)
cd ~/Downloads
wget http://vx2-downloads.raspberrypi.org/NOOBS/images/NOOBS-2016-02-29/NOOBS_v1_8_0.zip
unzip NOOBS_v1_8_0.zip -d /Volumes/NO\ NAME/

#insert the SD card and power on the machine
#after boot into NOOBS and choose install Rasbian and OSMC (that is kodi)
#Rasbian username and password are pi and raspberry respectively
#OSMC username and password are all osmc


2. BerryBoot(very flexible and allow to add custom image for multiboot)
wget http://downloads.sourceforge.net/project/berryboot/berryboot-20160313-pi2-pi3.zip
# for reformat SD Card to FAT32 and label BERRYBOOT
# sudo diskutil eraseDisk FAT32 BERRYBOOT MBRFormat /dev/disk4

unzip berryboot-20160313-pi2-pi3.zip -d /Volumes/BERRYBOOT/

#insert the SD card and power on the machine
#after boot into BerryBoot and choose install Debian Rasbian, OpenELEC (that is kodi) or others
#OpenELEC username and password are root and openelec respectively.
#hold down mouse button on AddOS button of BerryBoot Menu and choose Copy OS from USB stick

#To install Ubuntu-trusty image for berryboot downloaded here below
https://mega.nz/#!PwYl1QBJ!Fu7A87qrAL4jTmv0RhlnksprhVCm4gsHi1xhQBELJGk
# login to Ubuntu-trusty, username and password are all ubuntu

#BerryBoot additional images are here below and can be downloaded in USB drive for offline installation
https://sourceforge.net/projects/berryboot/files/os_images/


It is possible to shrink the Linux partition of berryboot in order to create a third partition for the dphys-swapfile with about 2G size. The SD card for berryboot should be mounted as USB and Use  sudo umount /dev/sda2; sudo e2fsck -f /dev/sda2; sudo resize2fs /dev/sda2 27G; 
to check and shrink and use  sudo fdisk /dev/sda;  to delete the second partition, recreate the second partition with +27G and then create the third partition with remaining space for swap (about 2.5G if for 32G SD card).
And then  sudo mkfs.ext4 /dev/sda3;  to format as ext4 and run  sudo fsck.ext4 /dev/sda1; sudo fsck.ext4 /dev/sda2;  to check.

After booting into the berryboot SD card, add this in /etc/fstab
/dev/mmcblk0p3 /swap ext4 defaults,noatime 0 0

and then edit swap location to /swap/swap and mount the partition using  sudo mkdir -p /swap; sudo mount -t ext4 /dev/mmcblk0p3 /swap;  and then setup dphys-swapfile using  sudo vi /etc/dphys-swapfile;  to /swap/swap and run  sudo dphys-swapfile setup; sudo dphys-swapfile swapon; 


shell script    Select all
# change password passwd # su sudo su sudo apt-get update sudo apt-get upgrade sudo apt-get install vim zip unzip openssh-server git #Login to Ubuntu-trusty, username and password are ubuntu and ubuntu respectively # install clang and swift for Ubuntu-trusty only (but no swift package manager) sudo apt-get install libicu-dev clang-3.6 sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-3.6 100 sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-3.6 100 #install swift-2.2_1%3a2.2-0ubuntu11~trusty1_armhf.deb (Ubuntu-trusty only) or just download from here without adding source repo #https://mega.nz/#!fpYxzBgY!MLGiW8HAV6MllAD0gkzrSFlQ6TdubwBfFYGyezNVAXg wget -qO- http://dev.iachieved.it/iachievedit.gpg.key | sudo apt-key add - echo "deb [arch=armhf] http://iachievedit-repos.s3.amazonaws.com/ trusty main" | sudo tee --append /etc/apt/sources.list sudo apt-get update sudo apt-get install swift-2.2 swift --version #test simple swift compile cat > hello.swift <<EOF let device = "Raspberry Pi 3!" print("Hello from Swift on \(device)") EOF swiftc hello.swift ./hello #Login to Rasbian, username and password are pi and raspberry respectively #change password (recomended) passwd #Rasbian config to set locale and timezone sudo raspi-config # configure keyboard to US sudo dpkg-reconfigure keyboard-configuration #reload the keymap sudo invoke-rc.d keyboard-setup start #scan hidden SSID sudo iwlist wlan0 scanning essid "Your Hidden SSID" #vi ~/.bashrc and add export LC_ALL="en_US.UTF-8" #add export LS_COLORS=$LS_COLORS:'di=1;44:' ; # dir white on blue background sudo locale-gen en_US.UTF-8 # generate locale for en_US.UTF-8 sudo dpkg-reconfigure locales #command line reset timezone sudo dpkg-reconfigure tzdata sudo apt-get update sudo apt-get upgrade sudo apt-get install vim openssh-server # Forwarding X11 application using ssh please refer to this guide https://kb.iu.edu/d/bdnt # In Linux or Mac ssh -Y pi@10.0.1.XX # In Windows 10 WSL, and install XServer in Windows 10 such as xming # export DISPLAY=:0 (for WSL x11-apps) export DISPLAY=localhost:0.0 ssh -Y pi@10.0.1.XX # After login Raspberry pi sudo apt-get install idle3 idle & # run scratch sudo apt-get install scratch scratch & sudo apt install codeblocks codeblocks & # run browser chromium-browser & #support of HFS Plus format drives sudo apt-get install hfsplus hfsutils hfsprogs gdisk #setup RAID, please refer to this, please use usb hub (with power source) for external HD disks. https://pchelp.ricmedia.com/build-raspberry-pi3-raid-nas-server/ #install nginx (lightweght webserver) sudo apt-get install nginx cd; ln -s /usr/share/nginx/html . #check ip adress and use browser connect to test web server hostname -I ifconfig eth0 ifconfig wlan0 # for wireless lan ip addr | grep -Po '(?!(inet 127.\d.\d.1))(inet \K(\d{1,3}\.){3}\d{1,3})' # generation of ssh key to remote server ssh-keygen -t rsa # copy ssh key to Raspberry Pi ssh-copy-id pi@raspberrypi_ip_address #install php in nginx #sudo apt-get install php5-fpm php-apc #sudo apt-get install php-fpm php-apcu sudo apt install php-fpm php-curl php-gd php-cli php7.3-opcache php-mbstring php-xml php-zip # link the html folder to home cd #ln -s /usr/share/nginx/html . ln -sf /var/www/html . # create testing php page cat > ~/html/info.php <<EOF <?php phpinfo(); ?> EOF #add in /etc/php/7.3/fpm/pool.d/www.conf user = pi group = pi # enable php in nginx sudo vi /etc/nginx/sites-enabled/default #see instructions for php7 here -> https://getgrav.org/blog/raspberrypi-nginx-php7-dev #see instructions here https://www.raspberrypi.org/documentation/remote-access/web-server/nginx.md sudo touch /var/run/php/php7.3-fpm.sock #reload web server and test #check to ensure the /var/run/php/php7.3-fpm.sock file exists sudo service nginx restart sudo service php7.3-fpm restart #install samba as a file server sudo apt-get install samba samba-common-bin #edit /etc/samba/smb.conf and add the followings # assume USB drive is mounted on /media/pi/USBDRIVE #manual mount external drive sudo mkdir -p /media/pi/USBDRIVE sudo mount -o uid=pi,gid=pi,rw /dev/sda1 /media/pi/USBDRIVE [public_USBDRIVE] comment = Public USBDRIVE browseable = yes read only = no force user = "pi" guest ok = yes public = yes writable = yes path = /media/pi/USBDRIVE create mask = 0755 directory mask = 0755 # reload samba # sudo service samba reload sudo systemctl restart smbd #install minidlna as a media server sudo apt install minidlna #edit /etc/minidlna.conf and add the followings media_dir=V,/media/pi/PassportUltra/Video friendly_name=My-Movie #edit /etc/default/minidlna and add the followings USER="root" GROUP="root" # reload minidlna sudo service minidlna restart sudo service minidlna force-reload # see what network services are working on raspberry pi sudo netstat -ntlp #install google app engine for development wget --no-check-certificate https://storage.googleapis.com/appengine-sdks/featured/google_appengine_1.9.34.zip unzip google_appengine_1.9.34.zip export PATH=$HOME/google_appengine:$PATH #test GAE web page from localhost web browser dev_appserver.py ~/google_appengine/demos/python/guestbook #test GAE web app from other machine dev_appserver.py --host $(ip addr | grep -Po '(?!(inet 127.\d.\d.1))(inet \K(\d{1,3}\.){3}\d{1,3})') --port 8080 ~/google_appengine/demos/python/guestbook/ dev_appserver.py --host $(hostname -I) --port 8080 ~/google_appengine/demos/python/guestbook/ # download node-v6.9.4-linux-armv7l.tar.xz from https://nodejs.org/en/download/ wget --no-check-certificate https://nodejs.org/dist/v6.9.4/node-v6.9.4-linux-armv7l.tar.xz # extract node-v6.9.4-linux-armv7l.tar.xz cd $HOME tar xJvf $HOME/Download/node-v6.9.4-linux-armv7l.tar.xz # setup PATH export PATH=$HOME/node-v6.9.4-linux-armv7l/bin:$PATH # Compile and Install Visual Studio Code 1.9.1 sudo apt-get update sudo apt-get install libx11-dev build-essential cd $HOME git clone https://github.com/microsoft/vscode cd $HOME/vscode git tag -l #checkout 1.9.1 git checkout tags/1.9.1 ./scripts/npm.sh install --arch=armhf # run Visual Studio Code ./scripts/code.sh # Visual Studio Code - Desktop Entry sudo tee /usr/share/applications/code.desktop > /dev/null <<'EOF' [Desktop Entry] Name=Visual Studio Code Comment=Code Editing. Redefined. GenericName=Text Editor Exec=/home/pi/vscode/scripts/code.sh %U Icon=code Type=Application StartupNotify=true StartupWMClass=Code Categories=Utility;TextEditor;Development;IDE; MimeType=text/plain;inode/directory; Actions=new-window; Keywords=vscode; [Desktop Action new-window] Name=New Window Name[de]=Neues Fenster Name[es]=Nueva ventana Name[fr]=Nouvelle fenêtre Name[it]=Nuova finestra Name[ja]=新規ウインドウ Name[ko]=새 창 Name[ru]=Новое окно Name[zh_CN]=新建窗口 Name[zh_TW]=開新視窗 Exec=/home/pi/vscode/scripts/code.sh %U Icon=code EOF # Manual download and install Visual Studio Extension # please refer to how to get the download url as referenced from http://stackoverflow.com/questions/37071388/how-to-install-vscode-extensions-offline wget --no-check-certificate https://ms-vscode.gallery.vsassets.io:443/_apis/public/gallery/publisher/ms-vscode/extension/cpptools/0.10.1/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage mv Microsoft.VisualStudio.Services.VSIXPackage ms-vscode-cpptools.vsix # Then use "Install from VSIX" menu item in Visual Studio code Debug Setting #install squid + unblock cn and assume node.js npm are installed as above sudo apt-get install squid3 git npm install Unblocker/Unblock-Youku #install phantomjs for testing git clone --depth 1 https://github.com/gautamMalu/PhantomJs-armhf.git #optionally install casper tag 1.1.1, PhantomJs+Casper for web scraping git clone --branch '1.1.1' --single-branch --depth 1 https://github.com/casperjs/casperjs.git sudo apt-get install libfontconfig1 libjpeg8 libicu-dev cd PhantomJs-armhf/bin; sudo mv phantomjs /usr/local/bin/ cd ~/node_modules/ub.uku.js/ npm install npm test #cp ~/node_modules/ub.uku.js/shared/urls.js ~/node_modules/ub.uku.js/youku.rules #cat ~/node_modules/ub.uku.js/shared/urls.js | grep "^[ ']*http" | sed "s/^[ ']*/\^/;s/[,'][ ]*\/\/.*$//;s/\*'$//;s/\*',$//;s/',//;s/[*?]$//;s/\*/\.\*/g;s/\?/\\\?/g;" > ~/node_modules/ub.uku.js/youku.rules vi ~/node_modules/ub.uku.js/youku.rules curl -o ~/node_modules/ub.uku.js/youku.rules http://pac.uku.im/regex sudo vi /etc/squid3/squid.conf acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network http_access allow localnet # add acl rules acl uyouku url_regex -i "/home/pi/node_modules/ub.uku.js/youku.rules" never_direct allow uyouku cache_peer 127.0.0.1 parent 8888 0 no-query default cache_peer_access 127.0.0.1 allow uyouku cache_peer_access 127.0.0.1 deny all # enable cache_dir say 2048 (The default is 100 MB) cache_dir ufs /var/spool/squid3 2048 16 256 # 2016/03/09 https://github.com/Unblocker/Unblock-Youku/issues/618 # dns_nameservers 158.69.209.100 45.32.72.192 45.63.69.42 #restart squid3 sudo service squid3 reload #edit /home/pi/node_modules/ub.uku.js/server/server.js (near line 257) to change listening ip and port say 127.0.0.1 8888 # ubuku_server.listen('8888','127.0.0.1')... # auto run unblockcn and use http://ipservice.163.com/isFromMainland to test # sudo vi /etc/rc.local and add su pi -c '/usr/bin/nodejs /home/pi/node_modules/ub.uku.js/server/server.js --nolog --proxy=https://secure.uku.im:993 --bak_proxy=http://proxy.uku.im:443 --pac_proxy=http://proxy.uku.im:443 > /dev/null &' # test terminal run with external proxy sudo node /home/pi/node_modules/ub.uku.js/server/server.js --proxy=https://secure.uku.im:993 --bak_proxy=http://proxy.uku.im:443 --pac_proxy=http://proxy.uku.im:443 #support Chinese font 文鼎 PL 上海宋 Un and input method sudo apt-get install ttf-arphic-uming scim-tables-zh im-switch #support Chinese font 文泉驛微米黑體、文泉驛正黑體、文泉驛點陣宋體 sudo apt-get install ttf-wqy-microhei ttf-wqy-zenhei xfonts-wqy #ext4 TRIM command support using the discard mount option in fstab (or with tune2fs -o discard /dev/sdaX). sudo tune2fs -o discard /dev/sdb1 #install bitcoind #sudo apt-get install bitcoind # bitcoind deb package in Debian is obsolete and needs upgrade #build from source as below or download the precompiled version from here and use sudo dpkg -i to install https://mega.nz/#!KtJWTZ7A!RN4IOGtvflSc0lXuJB1__lYTkdYvsnoegk_JJ0JGtA8 #Build requirements sudo apt-get install build-essential libtool autotools-dev automake pkg-config libssl-dev libevent-dev bsdmainutils git sudo apt-get install libboost-system-dev libboost-filesystem-dev libboost-chrono-dev libboost-program-options-dev libboost-test-dev libboost-thread-dev libminiupnpc-dev libminiupnpc-dev libzmq3-dev libcrypto++9 minissdpd git clone -b 0.12 --depth 1 https://github.com/bitcoin/bitcoin.git #see here to build bitcoind with Berkeley DB script https://github.com/bitcoin/bitcoin/blob/master/doc/build-unix.md cd bitcoin mkdir -p db4 wget 'http://download.oracle.com/berkeley-db/db-4.8.30.NC.tar.gz' tar -xzvf db-4.8.30.NC.tar.gz cd db-4.8.30.NC/build_unix/ BDB_PREFIX=$(cd $(dirname "../../db4") && pwd -P)/$(basename "../../db4"); ../dist/configure --enable-cxx --disable-shared --with-pic --prefix=$BDB_PREFIX make install cd ../.. ./autogen.sh BDB_PREFIX="$(pwd)/db4"; ./configure LDFLAGS="-L${BDB_PREFIX}/lib/" CPPFLAGS="-I${BDB_PREFIX}/include/" CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768" --without-gui --with-miniupnpc --enable-upnp-default --disable-tests #Rasbian config to set boot option to CLI mode sudo raspi-config #compiling bitcoind is cpu-intensive and memory-hungry, better to reboot to CLI mode and kill unnecessary services then make -j2 sudo make install #setup bitcoind and let it create bitcoin.conf bitcoind -daemon bitcoin-cli stop # To enable TRIM, put discard option in fstab like this PARTUUID=6c887fff-01 /media/pi/SSDDRIVE ext4 defaults,discard 0 0 #create data directory in USB SSD drive mkdir -p /media/pi/SSDDRIVE/bitcoin cp ~/.bitcoin/bitcoin.conf /media/pi/SSDDRIVE/bitcoin/ cp ~/wallet.dat /media/pi/SSDDRIVE/bitcoin/ # if have old bitcoin wallet.dat #start bitcoind as daemon in SSD Drive bitcoind -datadir=/media/pi/SSDDRIVE/bitcoin -daemon bitcoind --help #while bitcoind is running as daemon, get new address and check other status # bitcoin-cli help bitcoin-cli getinfo bitcoin-cli listaccounts bitcoin-cli getnewaddress "worker1" # check tx history say https://blockchain.info/address/14uUbzJSi2t5MGgYH72PdTonD22V8drBVN bitcoin-cli getaccountaddress "worker1" bitcoin-cli getbalance #daemon will download the entire block chain data and can take over several days and total size > 60G bitcoin-cli getblockcount # the current blockcount should be over 1M https://blockchain.info/latestblock bitcoin-cli getconnectioncount #install cpuminer sudo apt-get install libcurl4-openssl-dev git make git clone --depth 1 git://github.com/pooler/cpuminer.git cd cpuminer ./autogen.sh ./configure CFLAGS="-O3" make sudo make install minerd --help minerd --url http://api.bitcoin.cz:8332 -a sha256d --userpass=yourid.worker1:pass1 #cgminer (but not working in Raspberry Pi 3 #OpenCL...............: Detection overrided. GPU mining support DISABLED git clone -b 3.7 https://github.com/ckolivas/cgminer.git cd cgminer ./autogen.sh ./configure --enable-opencl --enable-scrypt # configure failed due to absence of GPU and opencl in Raspberry Pi 3 #Build QuantLib 1.7.1 #libquantlib0-dev 1.4.2 is available in package repo #QuantLib 1.7.1 #or download the updated package from here https://mega.nz/#!PkYWBDLC!dC8klXxcouc1RrNQ2Bct_tSweaO6A3UwTNkGeXvGldo #Download QuantLib source and extract wget http://jaist.dl.sourceforge.net/project/quantlib/QuantLib/1.7.1/QuantLib-1.7.1.tar.gz tar -xzvf QuantLib-1.7.1.tar.gz #Build requirement sudo apt-get install build-essential libtool libboost-all-dev cd QuantLib-1.7.1 ./configure CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768" --disable-shared #Rasbian config to set boot option to CLI mode sudo raspi-config #compiling QuantLib is cpu-intensive and memory-hungry, better to reboot to CLI mode and kill unnecessary services then make -j2 # or just make if failed sudo make install #test compile QuantLib example cd ~/QuantLib-1.7.1/Examples/Bonds g++ Bonds.cpp -o bin/Bonds -lQuantLib # run example Bonds ./bin/Bonds #compiling litecoind is similar to bitcoind but it would conflict with bitcoind git clone --depth 1 https://github.com/litecoin-project/litecoin.git #build from source or download the precompiled version from here https://mega.nz/#!Ptwl0KZD!XVyO7DHugtoK2iEib2nj2jdyMRSkadlIMa1oPp3QSak #setup litecoind in raspberry pi litecoind #create data directory in USB SSD drive mkdir -p /media/pi/SSDDRIVE/litecoin cp ~/.litecoin/litecoin.conf /media/pi/SSDDRIVE/litecoin/ #start litecoind as daemon in SSD Drive litecoind -datadir=/media/pi/SSDDRIVE/litecoin -daemon litecoind --help #reindex litecoin database if corrupted litecoind -reindex -datadir=/media/pi/SSDDRIVE/litecoin #while litecoind is running as daemon, get new address and check other status # litecoin-cli help litecoin-cli getinfo litecoin-cli listaccounts litecoin-cli getnewaddress "worker2" # check tx history say https://block-explorer.com/address/LRwz2mJpoMz2FPkqE7hxKvcLnUyUHESW4Q litecoin-cli getaccountaddress "worker2" litecoin-cli getbalance litecoin-cli getblockcount # the current blockcount should be over 900000 see http://explorer.litecoin.net/ #setup local repo sudo apt-get install dpkg-dev sudo mkdir -p /usr/local/mydebs sudo cp -s /var/cache/apt/archives/*.deb /usr/local/mydebs/ cd /usr/local/mydebs sudo sh -c 'dpkg-scanpackages . /dev/null | gzip -9c > Packages.gz' echo "deb file:/usr/local/mydebs ./" | sudo tee --append /etc/apt/sources.list sudo apt-get update #setup swapfile in Rasbian sudo vi /etc/dphys-swapfile #edit the swapfile location and size (suggest min 2048) then save the file #for berryboot the swapfile should be in external USB drive sudo dphys-swapfile setup sudo dphys-swapfile swapon #check memory and swap free -h #turn off swap sudo dphys-swapfile swapoff #configure and make again for the projects #shutdown the machine sudo halt #restart the machine sudo reboot



Swift 2.2 for Linux package requires clang-3.6 and Ubuntu-trusty for Pi
This image https://wiki.ubuntu.com/ARM/RaspberryPi (2015-04-06-ubuntu-trusty.img) is for Raspberry Pi 2 and does not work on Raspberry Pi 3, the FAT32 partition content in the layout folder and the start*.elf, bcm2710-rpi-3-b.dtb should be upgraded/added for the new hardware for booting.
config.txt add these settings
disable_overscan=1
start=1
gpu_mem=64
max_usb_current=1
initramfs initrd7.img
kernel=kernel7.img


It is possible to install the Swift 2.2 in Rasbian other than Ubuntu-trusty for Pi
1. Install clang-3.7 and libicu-dev in Rasbian
sudo apt-get install libicu-dev clang-3.7
sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-3.7 100
sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-3.7 100

2. download the package from here without adding source repo
#https://mega.nz/#!fpYxzBgY!MLGiW8HAV6MllAD0gkzrSFlQ6TdubwBfFYGyezNVAXg

3. download a fake clang-3.6 package that install nothing but to meet dependency as a hack
https://mega.nz/#!voZ0yA6T!10lqNIhEKLfNpptmy-rlsEdRn15gIl14Fd0ibjmI_x8

4. install these deb files together
sudo dkpg -i clang-3.6-fake.deb swift-2.2_1%3a2.2-0ubuntu11~trusty1_armhf.deb

Swift 2.2 for Linux without Swift Package Manager. Here are the alternative ways:
shell script    Select all
#Method 1 : compile everything on build cd ${HOME} mkdir -p SwiftProjects cd ${HOME}/SwiftProjects # get dependency package git clone --depth 1 https://github.com/erica/SwiftString.git git clone --depth 1 https://github.com/haginile/SwiftDate.git #fix some errors in SwiftDate/SwiftDate/DateHelpers.swift and SwiftDate/SwiftDate/Term.swift simlar to previous post "What you need for Swift on Linux" vi SwiftDate/SwiftDate/DateHelpers.swift extension String { /* comment out these 2 func subscript (i: Int) -> String { return String(Array(self.characters)[i]) } subscript (r: Range<int>) -> String { get { let subStart = self.startIndex.advancedBy(r.startIndex, limit: self.endIndex) let subEnd = subStart.advancedBy(r.endIndex - r.startIndex, limit: self.endIndex) return self.substringWithRange(Range(start: subStart, end: subEnd)) } } */ vi +195 SwiftDate/SwiftDate/Term.swift #change from switch timeUnit { #to switch timeUnit! { // unwrapped optional cd ${HOME}/SwiftProjects mkdir -p Sources/Test1 cat > Sources/Test1/main.swift <<EOF #if os(Linux) import Glibc import Foundation #endif //import SwiftString //import SwiftDate import XCTest class SwiftDateTesting: XCTestCase { var allTests : [(String, () -> Void)] { return [ ("testCalendar", testCalendar), ("testDayCounter", testDayCounter), ] } func testCalendar() { var cal = USSettlementCalendar() XCTAssert(cal.isBizDay(Date(string: "2014-05-15")) == true, "Pass") XCTAssert(cal.isBizDay(Date(string: "2014-05-17")) == false, "Pass") XCTAssertEqual(cal.nextBizDay(Date(string: "2014-05-16")).serialNumber, Date(string: "2014-05-19").serialNumber, "Pass") XCTAssertEqual(cal.bizDaysBetween(Date(string: "2014-05-15"), toDate: Date(string : "2014-05-20")), 3, "Pass") } func testDayCounter() { var start_date = Date(string: "2000-01-15") var end_date = Date(string: "2000-05-31") var actual_365 = Actual365Fixed() XCTAssertEqual(actual_365.dayCountFraction(start_date, date2: end_date), 137.0 / 365, "Pass") var nl_365 = NL365() XCTAssertEqual(nl_365.dayCountFraction(start_date, date2: end_date), 136.0 / 365, "Pass") var act_360 = Actual360() XCTAssertEqual(act_360.dayCountFraction(start_date, date2: end_date), 137.0 / 360.0, "Pass") var thirty_360 = Thirty360() XCTAssertEqual(thirty_360.dayCountFraction(start_date, date2: end_date), 136.0 / 360.0, "Pass") var thirty_360e = Thirty360(convention: Thirty360.Convention.EurobondBasis) } } XCTMain([SwiftDateTesting()]) EOF cd Sources/Test1 swiftc -emit-executable ../../SwiftString/Sources/*.swift ../../SwiftDate/SwiftDate/*.swift main.swift -o test1 ./test1 #Method 2 : dynamic library link #build module and library for dynamic linking cd ${HOME}/SwiftProjects mkdir -p Sources/Test2 cat > Sources/Test2/main.swift <<EOF #if os(Linux) import Glibc import Foundation #endif import SwiftDateString let start_date = Date(string: "2000-01-15") var end_date = Date(string: "2000-05-31") var actual_365 = Actual365Fixed() print(actual_365.dayCountFraction(start_date, date2: end_date)) EOF cd Sources/Test2 #build module and dynamic library swiftc -emit-library ../../SwiftString/Sources/*.swift ../../SwiftDate/SwiftDate/*.swift -module-name SwiftDateString -emit-module-path SwiftDateString.swiftmodule -module-name SwiftDateString -module-link-name SwiftDateString #compile and link with dynamic library at current directory swiftc -emit-executable main.swift -o test2 -I. -L. -Xlinker -rpath -Xlinker "." -lSwiftDateString ./test2 #alternative way is to copy the shared library to /usr/lib then link sudo cp libSwiftDateString.so /usr/lib/ swiftc -emit-executable -I. -lSwiftDateString main.swift -o test2a ./test2a #Method 3: static library link #build module and static library for linking cd ${HOME}/SwiftProjects mkdir -p Sources/Test3 cat > Sources/Test3/main.swift <<EOF #if os(Linux) import Glibc import Foundation #endif import SwiftDateString public func timetest(block: () -> Void) { let date = NSDate() block() let timeInterval = NSDate().timeIntervalSinceDate(date) print("Elasped time: \(timeInterval)") } public func testit() { let start_date = Date(string: "2000-01-15") let end_date = Date(string: "2000-05-31") let actual_365 = Actual365Fixed() print("Function: \(__FUNCTION__), File: \(__FILE__)") print("2000-01-15 to 2000-05-31 dayCountFraction is ", terminator:"") print(actual_365.dayCountFraction(start_date, date2: end_date)) } timetest(testit) EOF #compile cd Sources/Test3 swiftc -module-name SwiftDateString -c ../../SwiftString/Sources/*.swift ../../SwiftDate/SwiftDate/*.swift -emit-module-path SwiftDateString.swiftmodule -module-link-name SwiftDateString PROJECT_PREFIX=$(cd $(dirname "../../") && pwd -P)/$(basename "../../"); mkdir -p obj; cd obj; swiftc -parse-as-library -module-name SwiftDateString -emit-object ${PROJECT_PREFIX}/SwiftString/Sources/*.swift ${PROJECT_PREFIX}/SwiftDate/SwiftDate/*.swift; cd .. #build static library ar -rcs libSwiftDateString.a obj/*.o #link with static library swiftc -emit-executable main.swift -o test3 -I. -L. -Xlinker libSwiftDateString.a #Method 4: Use makefile for static library link #makefile to build module and static library for linking cd ${HOME}/SwiftProjects mkdir -p Sources/tidyjson cd Sources/tidyjson git clone --depth 1 https://github.com/benloong/TidyJSON.git # edit TidyJSON/Sources/TidyJSON.swift #find dataUsingEncoding #//if let data = string.dataUsingEncoding(NSUTF8StringEncoding) { #and change it to if let data = NSString(string:string).dataUsingEncoding(NSUTF8StringEncoding) { # edit TidyJSON/Tests/Test.swift #add #if os(Linux) import Glibc #endif # find all lines with # var allTests : [(String, () throws -> ())] { # and change them to var allTests : [(String, () -> Void)] { # find all lines with let content = try String(contentsOfFile: "./Tests/TestCases/\(path).json", encoding: NSUTF8StringEncoding) # and change them to block code as below if let content = try? NSString(contentsOfFile: "./TidyJSON/Tests/TestCases/\(path).json", encoding: NSUTF8StringEncoding).bridge() { // no change here if let _ = try? JSON.parse(content) { ... //(no change here) } else { ... //(no change here) } } #create this Makefile cat <<'EOF' > Makefile APP=tidyjsonTest PROJECT_PREFIX=$(PWD) MODULENAME=TidyJSON OBJ_DIR=obj DEP_LIB_SRC_FOLDER1 = TidyJSON/Sources SWIFT_FILES = $(wildcard $(PROJECT_PREFIX)/$(DEP_LIB_SRC_FOLDER1)/*.swift) OBJ_FILES=$(addprefix, ./obj/, $(notdir $(SWIFT_FILES:.swift=.o))) testfile= $(PROJECT_PREFIX)/TidyJSON/Tests/Test.swift mainfile=$(PROJECT_PREFIX)/TidyJSON/Tests/main.swift all: $(APP) $(APP): $(testfile) $(mainfile) lib$(MODULENAME).a $(MODULENAME).swiftmodule ; \ swiftc -emit-executable $(testfile) $(mainfile) -o $(APP) -I. -L. -Xlinker lib$(MODULENAME).a ; lib$(MODULENAME).a: $(SWIFT_FILES) ; \ $(shell mkdir -p $(OBJ_DIR); cd $(OBJ_DIR); swiftc -parse-as-library -module-name $(MODULENAME) -emit-object $(SWIFT_FILES)) \ rm -f lib$(MODULENAME).* ; \ ar -rcs lib$(MODULENAME).a $(OBJ_DIR)/*.o ; $(MODULENAME).swiftmodule: $(SWIFT_FILES) ; \ rm -f $(MODULENAME).swiftmodule $(MODULENAME).swiftdoc ; \ swiftc -emit-module -module-name $(MODULENAME) -c $(SWIFT_FILES) -emit-module-path $(MODULENAME).swiftmodule -module-link-name $(MODULENAME) ; clean: ; rm -rf obj $(MODULENAME).swift* lib$(MODULENAME).* $(APP) .PHONY: all clean EOF # make then run to test make ./tidyjsonTest

To install docker, see this http://blog.hypriot.com/post/run-docker-rpi3-with-wifi/
The latest Kali Linux image for pentesting is here https://www.offensive-security.com/kali-linux-arm-images/



Swift 3.0 is out for Ubuntu 16 (Xenial Xerus)
see Raspberry Pi image for Ubuntu 16 here https://wiki.ubuntu.com/ARM/RaspberryPi


installation shell script    Select all
cd $HOME wget http://swift-arm.ddns.net/job/Swift-3.0-Pi3-ARM-Incremental/lastSuccessfulBuild/artifact/swift-3.0.tgz mkdir $HOME/swift-3.0 cd $HOME/swift-3.0 && tar -xzf ../swift-3.0.tgz export PATH=$HOME/swift-3.0/usr/bin:$PATH sudo apt-get update sudo apt-get install -y libicu-dev clang-3.6 sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-3.6 100 sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-3.6 100 mkdir $HOME/hello; cd $HOME/hello swift package init --type executable swift build .build/debug/hello