Hardware
4 x Raspberry Pi 4B with heat sinksRaspberry Pi Cluster Case 4 layers with Cooling Fan for each layer
4 x MicroSDHC SanDisk 32G Class 10
One MicroSD Adapter for installation of OS
4 x USB-C power cable
4 x Cat 6 LAN cable
USB power supply with 8 USB ports total max 10A
External USB fans connected to USB power supply, important to keep the CPU cool especially when overclock
4 x UPS Battery Case 5V max 3.3A (each with 3 x Panasonic 18650BD 3200mAH batteries)
8 ports Gigabit Ethernet Switch
External RAID-0 disks with 2 x 8TB (WD Ultrastar HC320 7200rpm) storage, USB-C to USB3.0 interface (HD is the most expensive item for this project)
- cat /proc/cpuinfo Select all
Reference : https://www.rs-online.com/designspark/raspberry-pi-3-model-b-vs-3-model-b
processor : 0
BogoMIPS : 108.00
Features : fp asimd evtstrm crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd08
CPU revision : 3
processor : 1
BogoMIPS : 108.00
Features : fp asimd evtstrm crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd08
CPU revision : 3
processor : 2
BogoMIPS : 108.00
Features : fp asimd evtstrm crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd08
CPU revision : 3
processor : 3
BogoMIPS : 108.00
Features : fp asimd evtstrm crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd08
CPU revision : 3
Hardware : BCM2835
Revision : c03112
Serial : 100000003bc32951
Model : Raspberry Pi 4 Model B Rev 1.2
SD Card images for BerryBoot
BerryBoot(very flexible and allow to add custom OS images for multiboot)https://www.berryterminal.com/doku.php/berryboot
OS : Ubuntu 18.04.3 LTS
Download from Ubuntu_Server_arm64_18.04.3.img
https://sourceforge.net/projects/berryboot/files/os_images/
or raspberry pi image from
https://wiki.ubuntu.com/ARM/RaspberryPi
or download the latest image for raspi4 and convert to berryboot format as below
- shell script Select all
nohup curl -OL http://cdimage.ubuntu.com/ubuntu/releases/18.04.4/release/ubuntu-18.04.4-preinstalled-server-arm64+raspi4.img.xz &
# or download the 32 bits version
# nohup curl -OL http://cdimage.ubuntu.com/ubuntu/releases/18.04.4/release/ubuntu-18.04.4-preinstalled-server-armhf+raspi4.img.xz &
sudo apt update
sudo apt install kpartx squashfs-tools
unxz ubuntu-18.04.4-preinstalled-server-arm64+raspi4.img.xz
# And follow this guide to create your own berry boot images.
# https://www.berryterminal.com/doku.php/berryboot/adding_custom_distributions
# Convert arm64+raspi4 to berryboot OS image
sudo kpartx -av ubuntu-18.04.4-preinstalled-server-arm64+raspi4.img
#sudo mount /dev/mapper/loop0p2 /mnt
sudo mount /dev/mapper/loop1p2 /mnt
sudo sed -i 's/^\/dev\/mmcblk/#\0/g' /mnt/etc/fstab
sudo sed -i 's/^PARTUUID/#\0/g' /mnt/etc/fstab
sudo rm -f /mnt/etc/console-setup/cached_UTF-8_del.kmap.gz
sudo rm -f /mnt/etc/systemd/system/multi-user.target.wants/apply_noobs_os_config.service
sudo rm -f /mnt/etc/systemd/system/multi-user.target.wants/raspberrypi-net-mods.service
sudo rm -f /mnt/etc/rc3.d/S01resize2fs_once
sudo mksquashfs /mnt Ubuntu_Server_arm64_18.04.4_raspi4.img -comp lzo -e lib/modules
sudo umount /mnt
sudo kpartx -d ubuntu-18.04.4-preinstalled-server-arm64+raspi4.img
# Convert armhf+raspi4 to berryboot OS image
unxz ubuntu-18.04.4-preinstalled-server-armhf+raspi4.img.xz
sudo kpartx -av ubuntu-18.04.4-preinstalled-server-armhf+raspi4.img
sudo mount /dev/mapper/loop1p2 /mnt
sudo sed -i 's/^\/dev\/mmcblk/#\0/g' /mnt/etc/fstab
sudo sed -i 's/^PARTUUID/#\0/g' /mnt/etc/fstab
sudo rm -f /mnt/etc/console-setup/cached_UTF-8_del.kmap.gz
sudo rm -f /mnt/etc/systemd/system/multi-user.target.wants/apply_noobs_os_config.service
sudo rm -f /mnt/etc/systemd/system/multi-user.target.wants/raspberrypi-net-mods.service
sudo rm -f /mnt/etc/rc3.d/S01resize2fs_once
sudo mksquashfs /mnt Ubuntu_Server_armhf_18.04.4_raspi4.img -comp lzo -e lib/modules
sudo umount /mnt
sudo kpartx -d ubuntu-18.04.4-preinstalled-server-armhf+raspi4.img
Download links for these 2 converted images and other updated Raspbian images are on the right hand sidebar of this blog.
# BerryBoot way to change default OS images on reboot
# Reference : https://www.raspberrypi.org/forums/viewtopic.php?t=37861
# Reference : https://yoursunny.com/t/2017/berryboot-reboot-into/
Ubuntu Server image setup
- shell script Select all
plug in the Ethernet cable before boot up
login: ubuntu
password: ubuntu
change ubuntu password once login
# Update Server Security
# Reference : https://www.raspberrypi.org/documentation/configuration/security.md
sudo apt install openssh-server
# check hostname
hostnamectl
# check network interface
ifconfig
# update some packages
sudo apt update
sudo apt-get install dpkg
sudo apt-get install --reinstall python3-minimal python3-lockfile
sudo apt-get install --reinstall python3-twisted
sudo apt-get install --reinstall python3 python3-pip
sudo apt-get install --reinstall python-minimal python-lockfile
sudo apt-get install --reinstall python python-pip
# change hostname, change to pi01, pi02 ...
# Reference : https://linuxize.com/post/how-to-change-hostname-on-ubuntu-18-04/
sudo hostnamectl set-hostname pi01
hostnamectl
# change timezone
sudo dpkg-reconfigure tzdata
# change eth0 to Static IP
# Reference : https://linuxconfig.org/how-to-configure-static-ip-address-on-ubuntu-18-04-bionic-beaver-linux
cat /etc/netplan/01-netcfg.yaml
network:
version: 2
renderer: networkd
ethernets:
eth0:
dhcp4: no
addresses:
- 10.0.1.XXX/24
gateway4: 10.0.0.1
nameservers:
addresses: [8.8.8.8, 1.1.1.1]
# Once ready apply changes with:
sudo netplan apply
# Mount NTFS external raid disk and Install NFS Server
# Reference : https://www.tecmint.com/install-nfs-server-on-ubuntu/
# Reference : https://vitux.com/install-nfs-server-and-client-on-ubuntu/
# Check UUID or PARTUUID
sudo blkid
# add this in /etc/fstab, for example
PARTUUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" /media/RAID0WD ntfs defaults,nls=utf8,dmask=0000,fmask=0022,uid=1000,gid=1000,windows_names 0 0
# reboot to check mounted disk
sudo reboot
# After reboot
df -h
# add this in /etc/exports, for example
/media/RAID0WD 10.0.1.0/24(rw,sync,no_root_squash,no_subtree_check,insecure,anonuid=1000,anongid=1000)
echo "/media/RAID0WD 10.0.1.0/24(rw,sync,no_root_squash,no_subtree_check,insecure,anonuid=1000,anongid=1000)" | sudo tee -a /etc/exports
# Export and restart NFS Server
sudo exportfs -a
sudo systemctl restart nfs-kernel-server
# Allow nfs on firewall
sudo ufw allow from 10.0.1.0/24 to any port nfs
showmount -e pi01
# Install Raspberry pi bin and check cpu temperature
sudo add-apt-repository ppa:ubuntu-raspi2/ppa
sudo apt-get update
sudo apt-get install libraspberrypi-bin
# My CPU temp=38.0'C
vcgencmd measure_temp
# Mount NFS from Mac OS X
Connect to Server, enter nfs://10.0.1.101/media/RAID0WD
# Mount NFS from other Ubuntu nodes
# Reference : https://www.raspberrypi.org/documentation/configuration/nfs.md
# add this in /etc/fstab, for example
10.0.1.101:/media/RAID0WD /mnt/RAID0WD nfs auto 0 0
echo "10.0.1.101:/media/RAID0WD /mnt/RAID0WD nfs auto 0 0" | sudo tee -a /etc/fstab
## Install Samba for Ubuntu Server
# Reference https://linuxize.com/post/how-to-install-and-configure-samba-on-ubuntu-18-04/
# put this in /etc/samba/smb.conf, for example
[raidshare]
path = /media/RAID0WD
browseable = yes
guest ok = no
read only = no
force create mode = 0660
force directory mode = 2770
valid users = ubuntu @ubuntu
## Add Samba password for user ubuntu
sudo smbpasswd -a ubuntu
# Restart Samba Server
sudo systemctl restart smbd
# Allow samba on firewall
sudo ufw allow 'Samba'
# create images folder for berryboot images installation for other nodes
cd $HOME
ln -sf /media/RAID0WD smb_share
cd $HOME/smb_share
mkdir -p images
# Download required berryboot os images
nohup curl -L https://sourceforge.net/projects/berryboot/files/os_images/Ubuntu_Server_arm64_18.04.3.img/download -o Ubuntu_Server_arm64_18.04.3.img &
# Check sha1 signature
openssl sha1 Ubuntu_Server_arm64_18.04.3.img
Raspbian Buster image setup
- shell script Select all
plug in the Ethernet cable before boot up
login: pi
password: raspberry
change password once login
# Reference : https://www.raspberrypi.org/documentation/configuration/security.md
sudo apt install openssh-server
# check hostname
hostname
# check network interface
ifconfig
# change hostname, change to pi01, pi02 ... etc
sudo raspi-config
-> Select 2. Network Options -> Select N1 Hostname
# Enable SSH at the command line using raspi-config
sudo raspi-config
-> Select 5. Interfacing Options -> Select P2 SSH -> Select Yes
# VNC Server at the command line using raspi-config
sudo raspi-config
-> Select 5. Interfacing Options -> Select P3 VNC -> Select Yes
# change timezone
sudo dpkg-reconfigure tzdata
-> Select the timezone
# change locales
sudo dpkg-reconfigure locales
-> Select the locale
# change eth0 to Static IP
# Reference : https://pimylifeup.com/raspberry-pi-static-ip-address/
sudo vi /etc/dhcpcd.conf
# Restart dhcp
sudo service dhcpcd restart
# you will have 2 IP addresses, which is good for mounting nfs at start
hostname -I
# if you have 2 ip addresses and would like to stop the dhcp for eth0, see discussions here.
https://raspberrypi.stackexchange.com/questions/52010/set-static-ip-and-stop-dhcp-on-jessie-lite
# Mount NTFS external raid disk and Install NFS Server for Raspbian Buster
# Check UUID or PARTUUID
sudo blkid
# add this in /etc/fstab, for example
PARTUUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" /media/RAID0WD ntfs defaults,nls=utf8,dmask=0000,fmask=0022,uid=1000,gid=1000,windows_names 0 0
# reboot to check mounted disk
sudo reboot
# After reboot
df -h
# Install NFS Server for Raspbian Buster
sudo apt install nfs-kernel-server
# add this in /etc/exports, for example
/media/RAID0WD 10.0.1.0/24(rw,sync,no_root_squash,no_subtree_check,insecure,anonuid=1000,anongid=1000)
echo "/media/RAID0WD 10.0.1.0/24(rw,sync,no_root_squash,no_subtree_check,insecure,anonuid=1000,anongid=1000)" | sudo tee -a /etc/exports
# Export and restart NFS Server
sudo exportfs -a
sudo systemctl restart nfs-kernel-server
showmount -e pi01
# Mount NFS from Mac OS X
Connect to Server, enter nfs://10.0.1.101/media/RAID0WD
# Mount NFS from other Raspbian nodes
# Reference : https://www.raspberrypi.org/documentation/configuration/nfs.md
# add this in /etc/fstab, for example
10.0.1.101:/media/RAID0WD /mnt/RAID0WD nfs auto 0 0
echo "10.0.1.101:/media/RAID0WD /mnt/RAID0WD nfs auto 0 0" | sudo tee -a /etc/fstab
# Install Samba for Raspbian Buster
sudo apt install samba
# put this in /etc/samba/smb.conf, for example
[raidshare]
path = /media/RAID0WD
browseable = yes
guest ok = no
read only = no
force create mode = 0660
force directory mode = 2770
valid users = pi @pi
# Add Samba password for user pi
sudo smbpasswd -a pi
# Restart Samba Server
sudo systemctl restart smbd
# create images folder for berryboot images installation for other nodes
cd $HOME
ln -sf /media/RAID0WD smb_share
cd $HOME/smb_share
mkdir -p images
# Download required berryboot os images
nohup curl -L https://sourceforge.net/projects/berryboot/files/os_images/Debian_Buster_Raspbian_FULL_2019.10.img/download -o Debian_Buster_Raspbian_FULL_2019.10.img &
nohup curl -L https://sourceforge.net/projects/berryboot/files/os_images/Debian_Buster_Raspbian_2019.10.img/download -o Debian_Buster_Raspbian_2019.10.img &
# Check sha1 signature
openssl sha1 Debian_Buster_Raspbian_FULL_2019.10.img
openssl sha1 Debian_Buster_Raspbian_2019.10.img
Nodes pi01 pi02 pi03 pi04 setup
- shell script Select all
Reference : https://magpi.raspberrypi.org/articles/build-a-raspberry-pi-cluster-computer
Install OS images to other nodes, change hostname and assign fixed IP address for each node.
# Mount NFS for other nodes
# Reference : https://www.raspberrypi.org/documentation/configuration/nfs.md
# Install package
sudo apt install nfs-common
# add this in /etc/fstab, for pi02, pi03, pi04 nodes and reboot to be effective
10.0.1.101:/media/RAID0WD /mnt/RAID0WD nfs auto 0 0
echo "10.0.1.101:/media/RAID0WD /mnt/RAID0WD nfs auto 0 0" | sudo tee -a /etc/fstab
# Generate ssh key copy it to every other node in the cluster
# Reference : http://www.linuxproblem.org/art_9.html
# Login pi01
ssh pi@10.0.1.101
ssh-keygen -t rsa
ssh-copy-id 10.0.1.102
ssh-copy-id 10.0.1.103
ssh-copy-id 10.0.1.104
exit
# Login pi02
ssh pi@10.0.1.102
ssh-keygen -t rsa
ssh-copy-id 10.0.1.101
ssh-copy-id 10.0.1.103
ssh-copy-id 10.0.1.104
exit
# Login pi03
ssh pi@10.0.1.103
ssh-keygen -t rsa
ssh-copy-id 10.0.1.101
ssh-copy-id 10.0.1.102
ssh-copy-id 10.0.1.104
exit
# Login pi04
ssh pi@10.0.1.104
ssh-keygen -t rsa
ssh-copy-id 10.0.1.101
ssh-copy-id 10.0.1.102
ssh-copy-id 10.0.1.103
exit
Install MPI for each node pi01, pi02, pi03, pi04 nodes
sudo apt install mpich python3-mpi4py
sudo apt install python-mpi4py
##Test01
# Running in pi01
mpirun -n 4 -host 10.0.1.101,10.0.1.102,10.0.1.102,10.0.1.104 hostname
##Test02
# Running in pi01
mkdir -p /media/RAID0WD/Projects
ln -sf /media/RAID0WD/Projects $HOME
#Create shell script as temp.sh in $HOME/Projects folder
cat > $HOME/Projects/temp.sh <<EOF
#!/bin/sh
echo "\$(hostname)" "\$(vcgencmd measure_temp)"
EOF
chmod +x $HOME/Projects/temp.sh
# create a common Projects folder in all other nodes
# all nodes pi02 to pi04 create folder link and assume mounted NFS from pi01
ln -sf /mnt/RAID0WD/Projects $HOME/
# Running in pi01 to pi04
ssh pi@10.0.1.101
mpirun -n 4 -host 10.0.1.101,10.0.1.102,10.0.1.102,10.0.1.104 $HOME/Projects/temp.sh
ssh pi@10.0.1.102
mpirun -n 4 -host 10.0.1.101,10.0.1.102,10.0.1.102,10.0.1.104 $HOME/Projects/temp.sh
ssh pi@10.0.1.103
mpirun -n 4 -host 10.0.1.101,10.0.1.102,10.0.1.102,10.0.1.104 $HOME/Projects/temp.sh
ssh pi@10.0.1.104
mpirun -n 4 -host 10.0.1.101,10.0.1.102,10.0.1.102,10.0.1.104 $HOME/Projects/temp.sh
##Test03
# Add this in /etc/hosts for all nodes pi01, pi02, pi03, pi04
ssh pi@10.0.1.101
echo -e "10.0.1.101\tpi01\n10.0.1.102\tpi02\n10.0.1.103\tpi03\n10.0.1.104\tpi04" | sudo tee -a /etc/hosts
ssh pi@10.0.1.102
echo -e "10.0.1.101\tpi01\n10.0.1.102\tpi02\n10.0.1.103\tpi03\n10.0.1.104\tpi04" | sudo tee -a /etc/hosts
ssh pi@10.0.1.103
echo -e "10.0.1.101\tpi01\n10.0.1.102\tpi02\n10.0.1.103\tpi03\n10.0.1.104\tpi04" | sudo tee -a /etc/hosts
ssh pi@10.0.1.104
echo -e "10.0.1.101\tpi01\n10.0.1.102\tpi02\n10.0.1.103\tpi03\n10.0.1.104\tpi04" | sudo tee -a /etc/hosts
#Running on any node
cd $HOME/Projects
curl -OL https://raw.githubusercontent.com/mpi4py/mpi4py/master/demo/helloworld.py
mpirun -n 4 -host pi01,pi02,pi03,pi04 python $HOME/Projects/helloworld.py
##Test04
#Running on exactly 2 processes only
cd $HOME/Projects
curl -OL https://raw.githubusercontent.com/mpi4py/mpi4py/master/demo/osu_bw.py
mpirun -n 2 -host pi01,pi02 python $HOME/Projects/osu_bw.py
# create a file $HOME/Projects/4bmachinelist with the list of available pi 4b nodes for mpi
# It is not advised to mix pi 4b with 3b together to run mpi, as it will degrade the performance.
mpirun -n 2 -machinefile $HOME/Projects/4bmachinelist python $HOME/Projects/osu_bw.py
#On each node, launch 1 process only
mpirun -npernode 1 -machinefile $HOME/Projects/4bmachinelist $HOME/Projects/temp.sh
# -N is same as -npernode, -hostfile is same as -machinefile
mpirun -N 1 -hostfile $HOME/Projects/4bmachinelist $HOME/Projects/temp.sh
# or
mpirun -N 1 -hostfile $HOME/Projects/4bmachinelist bash -c 'echo "$(hostname)" "$(vcgencmd measure_temp)"' | sort
##Test05
# Count how many processes for your cluster, you should get 4 x 4 nodes = 16 processes
cd $HOME/Projects
curl -L https://github.com/Apress/raspberry-pi-supercomputing/archive/master.zip -o supercomputing.zip
unzip supercomputing.zip
mpirun -hostfile 4bmachinelist -N 1 python3 $HOME/Projects/raspberry-pi-supercomputing-master/Codes/code/chapter08/prog01.py
mpirun -hostfile 4bmachinelist -N 4 python3 $HOME/Projects/raspberry-pi-supercomputing-master/Codes/code/chapter08/prog03.py
##Test06
# Calculate primes
# Get the source code from
curl -OL https://people.sc.fsu.edu/~jburkardt/py_src/prime_mpi/prime_mpi.py
# There are 2 errors to fix before using it
# Line 41, there is a missing closing bracket ) at the end
# Line 74, should be changed
from
comm.Reduce ( [ t, MPI.DOUBLE ], [ primes, MPI.INT ], op = MPI.SUM, root = 0 )
to
primes = comm.reduce ( t, op = MPI.SUM, root = 0 )
# Test the time required among different processes by running the below
# assuming maximum 4 processes (pi 4 CPU has 4 cores) per node
cd $HOME/Projects
mpirun -n 4 -hostfile 4bmachinelist python3 prime_mpi.py
mpirun -n 8 -hostfile 4bmachinelist python3 prime_mpi.py
mpirun -n 12 -hostfile 4bmachinelist python3 prime_mpi.py
mpirun -n 16 -hostfile 4bmachinelist python3 prime_mpi.py
#On each node, launch 3 and 4 processes, and time their differences
cd $HOME/Projects
time mpirun -N 3 -hostfile 4bmachinelist python3 prime_mpi.py
time mpirun -N 4 -hostfile 4bmachinelist python3 prime_mpi.py
# Please post your results of Test06 here in the comment
##Test07
# Collective communication using the scatter function example
cd ~/Projects
curl -L https://pythonprogramming.net/scatter-gather-mpi-mpi4py-tutorial/ | grep -A13 -B1 "from mpi4py" | sed '1d' > sct9.py
time mpirun -N 4 -hostfile 4bmachinelist python sct9.py
##Test08
# Collective communication using the gather function example
cd ~/Projects
curl -L https://pythonprogramming.net/mpi-gather-command-mpi4py-python/ | grep -A19 -B1 "from mpi4py" | sed '1d' > sct10.py
time mpirun -N 4 -hostfile 4bmachinelist python sct10.py
##Test09
# mpicc examples
cd $HOME/Projects
curl -L https://github.com/wesleykendall/mpitutorial/archive/gh-pages.zip -o mpitutorial.zip
unzip mpitutorial.zip
cd mpitutorial-gh-pages/tutorials/mpi-reduce-and-allreduce/code
make
time mpirun -N 4 -hostfile $HOME/Projects/4bmachinelist reduce_avg 100000000
time mpirun -N 4 -hostfile $HOME/Projects/4bmachinelist reduce_stddev 100000000
Optional Server or services setup
- shell script Select all
# Nginx (a lightweght webserver) and fast php plugin
sudo apt-get install nginx
cd; ln -s /usr/share/nginx/html .
# check ip address and use browser connect to test web server
hostname -I
ifconfig eth0
ifconfig wlan0 # for wireless lan
ip addr | grep -Po '(?!(inet 127.\d.\d.1))(inet \K(\d{1,3}\.){3}\d{1,3})'
# install php in nginx
sudo apt install php-fpm php-curl php-gd php-cli php7.3-opcache php-mbstring php-xml php-zip
# link the html folder to home
cd $HOME
sudo chown pi:pi /var/www/html
ln -sf /var/www/html .
# create testing php page
cat > ~/html/info.php <<EOF
<?php
phpinfo();
?>
EOF
# add in /etc/php/7.3/fpm/pool.d/www.conf
user = pi
group = pi
# enable php in nginx and edit this file
sudo vi /etc/nginx/sites-enabled/default
# and change or add the followings in server section:
server {
...
# Add index.php to the list if you are using PHP
index index.html index.htm index.php index.nginx-debian.html;
...
## Begin - PHP
location ~ \.php$ {
# Choose either a socket or TCP/IP address
fastcgi_pass unix:/var/run/php/php7.3-fpm.sock;
# fastcgi_pass 127.0.0.1:9000;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root/$fastcgi_script_name;
}
## End - PHP
## Begin - Security
# deny all direct access for these folders
location ~* /(.git|cache|bin|logs|backups|tests)/.*$ { return 403; }
# deny running scripts inside core system folders
location ~* /(system|vendor)/.*\.(txt|xml|md|html|yaml|php|pl|py|cgi|twig|sh|bat)$ { return 403; }
# deny running scripts inside user folder
location ~* /user/.*\.(txt|md|yaml|php|pl|py|cgi|twig|sh|bat)$ { return 403; }
# deny access to specific files in the root folder
location ~ /(LICENSE.txt|composer.lock|composer.json|nginx.conf|web.config|htaccess.txt|\.htaccess) { return 403; }
## End - Security
...
}
#
# see instructions for php7 here -> https://getgrav.org/blog/raspberrypi-nginx-php7-dev
# reload web server and test
# check to ensure the /var/run/php/php7.3-fpm.sock file exists
sudo service nginx restart
sudo service php7.3-fpm restart
ls -l /var/run/php/php7.3-fpm.sock
# Use this command to check whether the web server is working or not
curl -L http://127.0.0.1/
curl -L http://127.0.0.1/info.php
# python cgi plugin for nginx
see here
# install minidlna as a media server
sudo apt install minidlna
# edit /etc/minidlna.conf and add the followings
media_dir=V,/media/RAID0WD/MyMovie
friendly_name=MyMovie
# edit /etc/default/minidlna and add the followings
USER="root"
GROUP="root"
# reload minidlna
sudo service minidlna restart
sudo service minidlna force-reload
# see what network services are working on raspberry pi
sudo netstat -ntlp
# free ddns
no-ip.com
Free sign-up and have 3 Hostnames but need to Confirm Every 30 Days
# After sign-up, manual update of IP address
# Use this command to obtain your public IP address and update on their website
host myip.opendns.com resolver1.opendns.com | grep myip
# and then download and Install the dynamic update client for Linux
https://www.noip.com/support/knowledgebase/installing-the-linux-dynamic-update-client/
You can have 3 host names and have to install virtual host in nginx web server.
In server settings of /etc/nginx/sites-available/default, use this settings to map to different html subfolders for different hosts
server {
...
server_name ~^(.*)\.(.*)\.(.*)$;
set $host_name $1; set $subdomain_name $2; set $domain_name $3;
root /var/www/html/$host_name.$subdomain_name.$domain_name;
...
}
# Suppose myhostname1, myhostname2 and myhostname3 are the hostnames obtained from no-ip.com
# Setup html root for mutli-hosts as above settings in nginx
sudo chown -R pi:pi /var/www/html
mkdir -p /var/www/html/10.0.1.101
mkdir -p /var/www/html/127.0.0.1
mkdir -p /var/www/html/myhostname1.ddns.net
mkdir -p /var/www/html/myhostname2.ddns.net
mkdir -p /var/www/html/myhoatname3.ddns.net
# Restart web server to be effective
sudo service nginx restart
sudo service php7.3-fpm restart
# Test web server after restart services
cd /var/www/html/
cp index.nginx-debian.html 10.0.1.101
curl -L http://10.0.1.101/
cd /var/www/html/
cp info.php 127.0.0.1/
curl -L http://127.0.0.1/info.php
# Test webserver from ddns
cd /var/www/html/myhostname1.ddns.net
curl -L https://getgrav.org/blog/raspberrypi-nginx-php7-dev -o index.html
# make sure your hone router tcp port 80 has been forwarded to your internal Pi host and test with
curl -L http://myhostname1.ddns.net/
# And also test the webserver from the browser on Phone
# How to access the server and nodes from Android Phone
# Recommend Termux from Google Play Store
# It use the Volume Up key + keyboard to enter special control characters and can install packages
# Reference : https://wiki.termux.com/wiki/Touch_Keyboard
apt update
apt upgrade
# Install ssh and login server
apt install openssh
ssh-copy-id pi@10.0.1.101
ssh pi@10.0.1.101
# Assume ddns is setup
ssh-copy-id pi@myhostname.ddns.net
ssh pi@myhostname.ddns.net
# Install python 3
apt search python
apt install python
# To improve command line history productivity, please refer
Terminal history usage tips : https://www.howtogeek.com/howto/44997/how-to-use-bash-history-to-improve-your-command-line-productivity/amp/
# ssh forwarding from Android Device (better to have Android tablet with keyboard and mouse)
# Reference: https://wiki.termux.com/wiki/Main_Page
# Need to install VNC Viewer and Termux from Google Play Store
# Install and start vncserver in Termux
vncserver -localhost
export DISPLAY=":1"
# ssh forwarding and login pi
ssh -Y pi@10.0.1.104
# install and run jupyter-notebook from pi (after installation of tensorflow)
pip3 install jupyter
jupyter-notebook &
# run the juypter notebook example from https://colab.research.google.com/github/lmoroney/io19/blob/master/Zero%20to%20Hero/Rock-Paper-Scissors.ipynb
# After session ended and kill vncserver in Termux
vncserver -kill :1
# free ssl certificate for web server
https://letsencrypt.org/getting-started/
# After you have your ddns host and nginx running on your Pi, follow the certbot instructions here
# Reference : https://certbot.eff.org/lets-encrypt/debianbuster-nginx
# set up certbot and obtain ssl certifcate for nginx
sudo apt-get install certbot python-certbot-nginx
sudo certbot --nginx
crontab -e
# add this entry to automate letencrypt certificate renewal
43 6 * * * certbot renew --renew-hook "systemctl reload nginx"
Certificate and chain have been saved at:
/etc/letsencrypt/live/myhostname1.ddns.net/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/myhostname1.ddns.net/privkey.pem
Your cert will expire on (90 days after). To obtain a new or tweaked
version of this certificate in the future, simply run certbot again
with the "certonly" option. To non-interactively renew *all* of
your certificates, run "certbot renew"
sudo certbot certonly
sudo certbot certificates
sudo systemctl reload nginx
# make sure your home router tcp port 443 has been forwarded to your internal Pi host and test with
curl -L https://myhostname1.ddns.net/
# Install OpenVPN Server
After you have the ddns hostname or fixed IP address, setup OpenVPN as per instructions here
# Reference : https://www.pcmag.com/how-to/how-to-create-a-vpn-server-with-raspberry-pi
curl -L https://install.pivpn.io | bash
# Choose OpenVPN and use udp port 1194
# Also need to assign fixed IP to your Pi
# After installation, sudo reboot to be effective
# forward udp port 1194 to the internal IP address of your Pi
# Client access
# Create configuration file e.g. iphone, macbook etc.
pivpn add
# Client access
Use "OpenVPN Connect App" for phone and notebook
send the profile to the client by email and import, e.g. for iPhone
# Makeuse of X11 forwarding to run graphical applications
see https://kb.iu.edu/d/bdnt
For macOS High Sierra or above, please see this guide to install xQuartz. https://www.unixtutorial.org/get-x11-forwarding-in-macos-high-sierra/
You can install it from https://www.xquartz.org or via "sudo port -v install xorg" in the terminal.
For In Windows 10 WSL, and install XServer in Windows 10 such as xming -> https://sourceforge.net/projects/xming/
and set DISPLAT for WSL x11-apps
export DISPLAY=localhost:0.0
export DISPLAY=:0
For Linux, there is built-in support.
In Terminal, type
ssh -Y pi@10.0.1.101
# After login Raspberry pi
sudo apt-get install idle3
idle &
# run scratch
sudo apt-get install scratch
scratch &
# run codeblocks
sudo apt install codeblocks
codeblocks &
If you get "cannot open display error", see discussions here.
https://superuser.com/questions/310197/how-do-i-fix-a-cannot-open-display-error-when-opening-an-x-program-after-sshi
# run browser
chromium-browser &
# Access Raspberry Pi Desktop Remotely
Use Windows Remote Desktop Client to connect to the Raspberry Pi.
# Install xrdp and reboot the pi after installation
sudo apt install xrdp
sudo systemctl restart xrdp
# For macOS, there is "Microsoft Remote Desktop Connection Client for Mac" in the Mac App Store.
# setup cron job to backup project data
# Reference : https://www.raspberrypi.org/documentation/linux/usage/cron.md
Create a shell script e.g.
cat > /home/pi/backup.sh <<EOF
#!/bin/sh
cd /media/RAID0WD; tar --exclude='./Projects/Downloads' -zcf Projects-backup/"Projects$(date '+%Y%m%d').tar.gz" Projects
EOF
chmod a+x /home/pi/backup.sh
mkdir -p /media/RAID0WD/Projects-backup
# add this entry in the crontab
0 0 * * * /home/pi/backup.sh
#List crontab
crontab -l
# Overclock
# Reference : https://www.seeedstudio.com/blog/2020/02/12/how-to-safely-overclock-your-raspberry-pi-4-to-2-147ghz/
#Install docker
# Not for berryboot images
curl -sSL https://get.docker.com | sh
sudo usermod -aG docker $USER
sudo newgrp docker
sudo apt install libffi-dev libssl-dev python3 python3-pip
sudo apt-get remove python-configparser
sudo pip3 -v install docker-compose
# test docker hello-world
docker run hello-world
cd $HOME
mkdir my-wordpress
cd my-wordpress
cat > docker-compose.yaml <<EOF
version: '3.2'
services:
db:
image: hypriot/rpi-mysql
volumes:
- "./.data/db:/var/lib/mysql"
restart: always
environment:
MYSQL_ROOT_PASSWORD: wordpress
MYSQL_DATABASE: wordpress
MYSQL_USER: wordpress
MYSQL_PASSWORD: wordpress
wordpress:
depends_on:
- db
image: wordpress:latest
links:
- db
ports:
- "8000:80"
restart: always
environment:
WORDPRESS_DB_HOST: db:3306
WORDPRESS_DB_PASSWORD: wordpress
EOF
docker-compose up -d
# then try browser http://localhost:8000/
# stop the docker-compose example
docker-compose down
# test nodejs
cd $HOME
git clone https://github.com/hypriot/rpi-node-haproxy-example
cd rpi-node-haproxy-example
docker-compose up
curl http://localhost:80
curl http://localhost:70
docker-compose stop
# test nodejs + mongodb
cd $HOME
git clone https://github.com/hagaik/easy-node-authentication.git
cd easy-node-authentication
cat > config/database.js <<EOF
// config/database.js
module.exports = {
'url' : 'mongodb://mongo:27017' // looks like mongodb://<user>:<pass>@mongo.onmodulus.net:27017/Mikha4ot
};
EOF
cat > Dockerfile <<EOF
FROM hypriot/rpi-node
# Create app directory
WORKDIR /usr/src/app
# Install app dependencies
COPY package*.json ./
RUN npm install
# Copy app source code
COPY . .
#Expose port and start application
EXPOSE 8080
CMD [ "npm", "start" ]
EOF
cat > docker-compose.yaml <<EOF
version: "3.2"
services:
db:
image: dhermanns/rpi-mongo
volumes:
- "data-volume:/data/db"
restart: always
ports:
- "27017:27017"
expose:
- "27017"
webm:
build: .
depends_on:
- db
ports:
- "8080:8080"
expose:
- "8080"
volumes:
data-volume:
EOF
# build and start as daemon
docker-compose up --build -d
# then try browser http://localhost:8080/
# stop and down
docker-compose down -v
# move docker data-root to nfs mounted drive e.g. /mnt/docker-data
sudo service docker stop
cat << EOF | sudo tee -a /etc/docker/daemon.json
{
"storage-driver": "overlay",
"data-root": "/mnt/docker-data"
}
EOF
sudo rsync -aP /var/lib/docker/ /mnt/docker-data
sudo mv /var/lib/docker /var/lib/docker.old
sudo service docker start
Install tensorflow and horovod for Pi 4 cluster
- shell script Select all
Install all these packages for every nodes in the Pi 4 cluster in order to run horovod with tensorflow
# Install tenorflow 2.1.0
# Reference https://qengineering.eu/install-tensorflow-2.1.0-on-raspberry-pi-4.html
# or https://qengineering.eu/install-tensorflow-2.2.0-on-raspberry-pi-4.html for tenorflow 2.2.0
cd ~/Projects
sudo apt-get install gfortran
sudo apt-get install libhdf5-dev libc-ares-dev libeigen3-dev
sudo apt-get install libatlas-base-dev libopenblas-dev libblas-dev
sudo apt-get install liblapack-dev cython
sudo pip3 install pybind11
sudo apt-get install python3-h5py
# upgrade pip3 and check pip3 version
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3 get-pip.py --force-reinstall
# pip 20.1.1 from /home/pi/.local/lib/python3.7/site-packages/pip (python 3.7)
python3 --version
pip3 --version
# download the wheel
wget https://github.com/Qengineering/Tensorflow-Raspberry-Pi/raw/master/tensorflow-2.1.0-cp37-cp37m-linux_armv7l.whl
# install TensorFlow 2.1.0
python3 -m pip install --user tensorflow-2.1.0-cp37-cp37m-linux_armv7l.whl
# test run examples as in how-to-install-tensorflow-with-gpu
python3 -m pip install --user matplotlib
python3 -m pip install --user pandas
python3 -m pip install --user keras scikit-learn
cd ~/Projects
curl -L https://tinyurl.com/tensorflowwin | grep -A7 tftest.py | sed '1,2d' > tftest.py
python3 tftest.py
curl -L https://tinyurl.com/tensorflowwin | grep -A129 irislearn.py | sed '1,8d' > irislearn.py
curl -L https://tinyurl.com/tensorflowwin | grep -A150 iris.data.nbsp | sed '1d' > iris.data
python3 irislearn.py
curl -L https://tinyurl.com/tensorflowwin | grep -A37 keraslearn.py | sed '1,3d' > keraslearn.py
curl -L https://tinyurl.com/tensorflowwin | grep -A768 pima-indians-diabetes.data.nbsp | sed '1d' > pima-indians-diabetes.data
python3 keraslearn.py
# install horovod
# https://github.com/horovod/horovod#install
python3 -m pip install cffi>=1.4.0 cloudpickle
python3 -m pip install horovod
# Reboot to make it effective
sudo reboot
# First test run the simple hellohorovod.py
cd ~/Projects
cat > hellohorovod.py <<EOF
from mpi4py import MPI
import horovod.tensorflow as hvd
# Split COMM_WORLD into subcommunicators
subcomm = MPI.COMM_WORLD.Split(color=MPI.COMM_WORLD.rank % 2,
key=MPI.COMM_WORLD.rank)
# Initialize Horovod
hvd.init(comm=subcomm)
print('COMM_WORLD rank: %d, Name: %s, Horovod rank: %d' % (MPI.COMM_WORLD.rank, MPI.Get_processor_name(), hvd.rank()))
EOF
# run it with
cd ~/Projects
horovodrun -np 1 -H localhost:1,pi02:1,pi03:1,pi04:4 python3 hellohorovod.py
# Then try run the tensorflow v2 example as below
# please be warned that and watch out the temperature of PIs.
# https://github.com/horovod/horovod/blob/master/examples/tensorflow2_keras_mnist.py
# For non-keras, the sample is https://github.com/horovod/horovod/blob/master/examples/tensorflow2_mnist.py
cd ~/Projects
wget https://raw.githubusercontent.com/horovod/horovod/master/examples/tensorflow2_keras_mnist.py
time horovodrun -np 4 -H localhost:1,pi02:1,pi03:1,pi04:1 python3 tensorflow2_keras_mnist.py
time horovodrun -np 8 -H localhost:2,pi02:2,pi03:2,pi04:2 python3 tensorflow2_keras_mnist.py
time horovodrun -np 12 -H localhost:3,pi02:3,pi03:3,pi04:3 python3 tensorflow2_keras_mnist.py
# 1 slot per node has the faster performance for the Pi cluster
# As with the release of the new 8GB Pi4B model, it is possible to increase the number of slots for these 8GB machines.
# append self public key to self authorized list
ssh pi@10.0.1.101
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
exit
ssh pi@10.0.1.102
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
exit
ssh pi@10.0.1.103
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
exit
ssh pi@10.0.1.104
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
exit
# create ~/Projects/myhostfile
cat > ~/Projects/myhostfile <<EOF
pi01 slots=1
pi02 slots=1
pi03 slots=1
pi04 slots=1
EOF
# running in background with nohup
cd ~/Projects
nohup horovodrun -np 4 -hostfile myhostfile python3 tensorflow2_keras_mnist.py &
cat ~/Projects/nohup.out
exit
# create folder in non-nfs ext4 partition folder if running in nodes other than pi01
mkdir -p ~/horovod
ssh pi01 'mkdir -p ~/horovod'
ssh pi02 'mkdir -p ~/horovod'
ssh pi03 'mkdir -p ~/horovod'
ssh pi04 'mkdir -p ~/horovod'
# When running in pi02, pi03 and pi04, it cannot start in the nfs shared folder if it is not an ext4 partition.
# e.g. when start in pi03
ssh pi03
cd ~/horovod
horovodrun -np 4 -hostfile ~/Projects/myhostfile python3 ~/Projects/tensorflow2_keras_mnist.py
# run it with horovod and time and redirect outputfile to keras_mnist.np4.out
cd ~/horovod
nohup bash -c 'time horovodrun -np 4 -hostfile ~/Projects/myhostfile3 python3 ~/Projects/tensorflow2_keras_mnist.py' &> keras_mnist.np4.out &
# The time for number of nodes for this testing
nohup bash -c 'time horovodrun -np 4 -H pi01:1,pi02:1,pi03:1,pi04:1 --output-filename logs python3 ~/Projects/tensorflow2_keras_mnist.py' &> nohup.out.np4_1111 &
nohup bash -c 'time horovodrun -np 3 -H pi01:1,pi02:1,pi03:1 --output-filename logs python3 ~/Projects/tensorflow2_keras_mnist.py' &> nohup.out.np3_111 &
nohup bash -c 'time horovodrun -np 2 -H pi01:1,pi02:1 --output-filename logs python3 ~/Projects/tensorflow2_keras_mnist.py' &> nohup.out.np2_11 &
nohup bash -c 'time horovodrun -np 1 -H pi01:1 --output-filename logs python3 ~/Projects/tensorflow2_keras_mnist.py' &> nohup.out.np1_1 &
nohup.out.np4_1111:real not yet done (may be 50m)
nohup.out.np3_111:real 88m13.692s
nohup.out.np2_11:real 129m11.322s
nohup.out.np1_1:real 207m28.119s
Some shortcuts
- shell script Select all
# Offending ECDSA key
# Offending ECDSA key in $HOME/.ssh/known_hosts:5
# For Mac sed
sed -i '' '5d' $HOME/.ssh/known_hosts
# For GNU sed
sed -i '5d' $HOME/.ssh/known_hosts
# print line 5 to 6 of a text file
sed -n '5,6p' $HOME/.ssh/known_hosts
# print line 1 and Line 5 to 6 of a text file
sed -n -e '1p' -e '5,6p' $HOME/.ssh/known_hosts
# add all the hosts to the ~/.ssh/known_hosts file using ssh-keyscan
# first login pi01 10.0.1.101
ssh pi@10.0.1.101
ssh-keyscan -t rsa,dsa pi02,pi01,pi04 > ~/.ssh/known_hosts
