Google colab - keras-learn and Logistic Regression example

(1) Following the previous post, this demo the keras sample from how-to-install-tensorflow-with-gpu.html
Please take note that google only allow one active session for the free service. If you need faster GPU, more RAM and sessions, please consider to subscribe colab pro.

keraslearn.ipynb   Select all

# Step 1 mount google drive if data is from google drive import os from google.colab import drive drive.mount('/content/drive') # Step 2 if using tensorflow GPU #%tensorflow_version 2.x #import tensorflow as tf #print('TensorFlow: {}'.format(tf.__version__)) #tf.test.gpu_device_name() # Step 3 from keras.models import Sequential from keras.layers import Dense import numpy import time # fix random seed for reproducibility numpy.random.seed(7) # Step 4 # download pima indians dataset to google drive !curl -L https://tinyurl.com/tensorflowwin | grep -A768 pima-indians-diabetes.data.nbsp | sed '1d' > 'drive/MyDrive/Colab Notebooks/pima-indians-diabetes.data' # or download to local data directory !mkdir -p ./data !curl -L https://tinyurl.com/tensorflowwin | grep -A768 pima-indians-diabetes.data.nbsp | sed '1d' > './data/pima-indians-diabetes.data' # Step 5 load dataset from google drive dataset = numpy.loadtxt("drive/MyDrive/Colab Notebooks/pima-indians-diabetes.data", delimiter=",") # or load data from local data directory dataset = numpy.loadtxt("./data/pima-indians-diabetes.data", delimiter=",") # Step 6 # split into input (X) and output (Y) variables X = dataset[:,0:8] Y = dataset[:,8] # Step 7 # create model model = Sequential() model.add(Dense(12, input_dim=8, activation='relu')) model.add(Dense(1, activation='sigmoid')) # Step 8 # Compile model model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) # Step 9 start_time=time.time() # Fit the model model.fit(X, Y, batch_size=10, epochs=1500) # parameters for keras 1.2.2 # evaluate the model scores = model.evaluate(X, Y) print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100)) print("\nTraining took %.2f seconds\n" %(time.time()-start_time))

(2) For large training data set, consider to zip them and upload to google drive. Mount the google drive, then unzip it in local session. e.g.

!mkdir -p ./data
!unzip -o './drive/MyDrive/Colab Notebooks/mydata.zip' -d ./data/

(3) To stop the running cell in Google Colab use Ctrl-M I

(4) How to quickly run an ipynb example from github ?
4.1) Go to https://colab.research.google.com/, after login gmail and choose GitHub tab and enter search say "clareyan/From-Linear-to-Logistic-Regression-Explained-Step-by-Step"
4.2) In Step 2 cell box change the importing of dataset to
df = pd.read_csv('https://raw.githubusercontent.com/clareyan/From-Linear-to-Logistic-Regression-Explained-Step-by-Step/master/Social_Network_Ads.csv')
4.3) Then choose menu -> Runtime -> Run All. After that, use menu -> File -> Save a copy in Drive.

How to setup google colab and start linear regression with tensorflow.

(1) You only need a chrome browser, google gmail account and google drive account to start cloud tensorflow computing. And it is free to use and learn.

(2) Go to https://colab.research.google.com/
(3) Create a new notebook rename it and then Copy to Drive
(4) Type the following into notebook and run it step by step (Press Alt-Enter to run after each step)

LinearRegression.ipynb   Select all

#Step 1 # mount Google Drive, will ask for authorization code import numpy as np import os from google.colab import drive drive.mount('/content/drive') #Step 2 # choose the notebook settings to use GPU, via Menu -> Edit -> Notebook Settings. %tensorflow_version 2.x import tensorflow as tf ＃ will show GPU if successful tf.test.gpu_device_name() #Step 3 # load data import pandas as pd # either download the linear_data.csv and upload to google drive, or direct download it via the shell command as below !curl -L https://tinyurl.com/lineardatacsv | grep -A200 START_OF_LINEAR_DATA.CSV | sed '1d' | sed -n "/END_OF_LINEAR_DATA.CSV/q;p" | sed 's/&gt;/\>/g;s/&lt;/\</g' > 'drive/MyDrive/Colab Notebooks/linear_data.csv' df = pd.read_csv('drive/MyDrive/Colab Notebooks/linear_data.csv') df.head() #Step 4 # split into independent and dependent X = df[['X']].values y = df[['Y']].values X.shape, y.shape #Step 5 # visualize data import matplotlib.pyplot as plt %matplotlib inline plt.scatter(X,y) plt.xlabel('independent') plt.ylabel('dependent') plt.show() # Use Text box to enter ＃ # Linear Regression $ \hat y = a + b * X $ #Step 6 # Linear Regression # define regression model class regression(): def __init__(self): self.a = tf.Variable(initial_value=0,dtype=tf.float32) self.b = tf.Variable(initial_value=0,dtype=tf.float32) def __call__(self, X): x = tf.convert_to_tensor(X,dtype=tf.float32) y_est = tf.add(self.a, tf.multiply(self.b,x)) return y_est model = regression() # Use Text box to enter ＃ # loss = sum of square error (sse) = $ \sum (y_t - y_p) ^ 2 $ # step 7 # define loss function def loss_func(y_true, y_pred): # both values are in tensors sse = tf.reduce_sum(tf.square(tf.subtract(y_true,y_pred))) return sse # Use Text box to enter ＃ # Gradient Descent $ a = a_i - \nabla(sse) | a * LR $ $ b = b_i - \nabla(sse) | b * LR $ # step 8 # define train function def train(model, inputs, outputs, learning_rate): # convert outputs into tensor y_true = tf.convert_to_tensor(outputs,dtype=tf.float32) # GradientTape cal gradient distance with tf.GradientTape() as g: y_pred = model(inputs) current_loss = loss_func(y_true,y_pred) da,db = g.gradient(current_loss,[model.a,model.b]) # update the values model.a.assign_sub(da*learning_rate) model.b.assign_sub(db*learning_rate) # Step 9 def plot_scatter(x,y): plt.scatter(x,y) # scatter plt.plot(x,model(x),'r--') #line plot_scatter(X,y) # step 10 # model fitting model = regression() a_values = [] b_values = [] cost_values = [] # epochs, no of steps epochs = 100 # learning_rate learning_rate = 0.0001 for epoch in range(epochs): a_values.append(model.a) b_values.append(model.b) # prediction values and error y_pred = model(X) cost_value = loss_func(y,y_pred) cost_values.append(cost_value) # training train(model,X,y,learning_rate) # visual the scatter plot_scatter(X,y) plt.show #print the value print('Epoch: %d, Loss: %0.2f, a: %0.2f, b: %0.2f' %(epoch,cost_value,a_values[-1],b_values[-1])) # step 11 plt.plot(cost_values)

(5) Linear_data.csv , download and upload to google drive

linear_data.csv   Select all

X,Y 4,2 4,10 7,4 7,22 8,16 9,10 10,18 10,26 10,34 11,17 11,28 12,14 12,20 12,24 12,28 13,25 13,34 13,24 13,46 14,26 14,36 14,60 14,80 15,20 15,26 15,54 16,32 16,40 17,32 17,40 17,50 18,42 18,56 18,76 18,84 19,36 19,45 19,68 20,32 20,48 20,52

Personal Installation Guide of Raspberry Pi 4B cluster

Why this project with Raspberry Pi cluster ? Because it is cheap and can be used for the many purposes as below.

Hardware

4 x Raspberry Pi 4B with heat sinks
Raspberry Pi Cluster Case 4 layers with Cooling Fan for each layer
4 x MicroSDHC SanDisk 32G Class 10
One MicroSD Adapter for installation of OS
4 x USB-C power cable
4 x Cat 6 LAN cable
USB power supply with 8 USB ports total max 10A
External USB fans connected to USB power supply, important to keep the CPU cool especially when overclock
4 x UPS Battery Case 5V max 3.3A (each with 3 x Panasonic 18650BD 3200mAH batteries)
8 ports Gigabit Ethernet Switch
External RAID-0 disks with 2 x 8TB (WD Ultrastar HC320 7200rpm) storage, USB-C to USB3.0 interface (HD is the most expensive item for this project)

cat /proc/cpuinfo    Select all

Reference : https://www.rs-online.com/designspark/raspberry-pi-3-model-b-vs-3-model-b processor : 0 BogoMIPS : 108.00 Features : fp asimd evtstrm crc32 cpuid CPU implementer : 0x41 CPU architecture: 8 CPU variant : 0x0 CPU part : 0xd08 CPU revision : 3 processor : 1 BogoMIPS : 108.00 Features : fp asimd evtstrm crc32 cpuid CPU implementer : 0x41 CPU architecture: 8 CPU variant : 0x0 CPU part : 0xd08 CPU revision : 3 processor : 2 BogoMIPS : 108.00 Features : fp asimd evtstrm crc32 cpuid CPU implementer : 0x41 CPU architecture: 8 CPU variant : 0x0 CPU part : 0xd08 CPU revision : 3 processor : 3 BogoMIPS : 108.00 Features : fp asimd evtstrm crc32 cpuid CPU implementer : 0x41 CPU architecture: 8 CPU variant : 0x0 CPU part : 0xd08 CPU revision : 3 Hardware : BCM2835 Revision : c03112 Serial : 100000003bc32951 Model : Raspberry Pi 4 Model B Rev 1.2

SD Card images for BerryBoot

BerryBoot(very flexible and allow to add custom OS images for multiboot)
https://www.berryterminal.com/doku.php/berryboot

OS : Ubuntu 18.04.3 LTS
Download from Ubuntu_Server_arm64_18.04.3.img
https://sourceforge.net/projects/berryboot/files/os_images/
or raspberry pi image from
https://wiki.ubuntu.com/ARM/RaspberryPi
or download the latest image for raspi4 and convert to berryboot format as below

shell script    Select all

nohup curl -OL http://cdimage.ubuntu.com/ubuntu/releases/18.04.4/release/ubuntu-18.04.4-preinstalled-server-arm64+raspi4.img.xz & # or download the 32 bits version # nohup curl -OL http://cdimage.ubuntu.com/ubuntu/releases/18.04.4/release/ubuntu-18.04.4-preinstalled-server-armhf+raspi4.img.xz & sudo apt update sudo apt install kpartx squashfs-tools unxz ubuntu-18.04.4-preinstalled-server-arm64+raspi4.img.xz # And follow this guide to create your own berry boot images. # https://www.berryterminal.com/doku.php/berryboot/adding_custom_distributions # Convert arm64+raspi4 to berryboot OS image sudo kpartx -av ubuntu-18.04.4-preinstalled-server-arm64+raspi4.img #sudo mount /dev/mapper/loop0p2 /mnt sudo mount /dev/mapper/loop1p2 /mnt sudo sed -i 's/^\/dev\/mmcblk/#\0/g' /mnt/etc/fstab sudo sed -i 's/^PARTUUID/#\0/g' /mnt/etc/fstab sudo rm -f /mnt/etc/console-setup/cached_UTF-8_del.kmap.gz sudo rm -f /mnt/etc/systemd/system/multi-user.target.wants/apply_noobs_os_config.service sudo rm -f /mnt/etc/systemd/system/multi-user.target.wants/raspberrypi-net-mods.service sudo rm -f /mnt/etc/rc3.d/S01resize2fs_once sudo mksquashfs /mnt Ubuntu_Server_arm64_18.04.4_raspi4.img -comp lzo -e lib/modules sudo umount /mnt sudo kpartx -d ubuntu-18.04.4-preinstalled-server-arm64+raspi4.img # Convert armhf+raspi4 to berryboot OS image unxz ubuntu-18.04.4-preinstalled-server-armhf+raspi4.img.xz sudo kpartx -av ubuntu-18.04.4-preinstalled-server-armhf+raspi4.img sudo mount /dev/mapper/loop1p2 /mnt sudo sed -i 's/^\/dev\/mmcblk/#\0/g' /mnt/etc/fstab sudo sed -i 's/^PARTUUID/#\0/g' /mnt/etc/fstab sudo rm -f /mnt/etc/console-setup/cached_UTF-8_del.kmap.gz sudo rm -f /mnt/etc/systemd/system/multi-user.target.wants/apply_noobs_os_config.service sudo rm -f /mnt/etc/systemd/system/multi-user.target.wants/raspberrypi-net-mods.service sudo rm -f /mnt/etc/rc3.d/S01resize2fs_once sudo mksquashfs /mnt Ubuntu_Server_armhf_18.04.4_raspi4.img -comp lzo -e lib/modules sudo umount /mnt sudo kpartx -d ubuntu-18.04.4-preinstalled-server-armhf+raspi4.img Download links for these 2 converted images and other updated Raspbian images are on the right hand sidebar of this blog. # BerryBoot way to change default OS images on reboot # Reference : https://www.raspberrypi.org/forums/viewtopic.php?t=37861 # Reference : https://yoursunny.com/t/2017/berryboot-reboot-into/

Ubuntu Server image setup

shell script    Select all

plug in the Ethernet cable before boot up login: ubuntu password: ubuntu change ubuntu password once login # Update Server Security # Reference : https://www.raspberrypi.org/documentation/configuration/security.md sudo apt install openssh-server # check hostname hostnamectl # check network interface ifconfig # update some packages sudo apt update sudo apt-get install dpkg sudo apt-get install --reinstall python3-minimal python3-lockfile sudo apt-get install --reinstall python3-twisted sudo apt-get install --reinstall python3 python3-pip sudo apt-get install --reinstall python-minimal python-lockfile sudo apt-get install --reinstall python python-pip # change hostname, change to pi01, pi02 ... # Reference : https://linuxize.com/post/how-to-change-hostname-on-ubuntu-18-04/ sudo hostnamectl set-hostname pi01 hostnamectl # change timezone sudo dpkg-reconfigure tzdata # change eth0 to Static IP # Reference : https://linuxconfig.org/how-to-configure-static-ip-address-on-ubuntu-18-04-bionic-beaver-linux cat /etc/netplan/01-netcfg.yaml network: version: 2 renderer: networkd ethernets: eth0: dhcp4: no addresses: - 10.0.1.XXX/24 gateway4: 10.0.0.1 nameservers: addresses: [8.8.8.8, 1.1.1.1] # Once ready apply changes with: sudo netplan apply # Mount NTFS external raid disk and Install NFS Server # Reference : https://www.tecmint.com/install-nfs-server-on-ubuntu/ # Reference : https://vitux.com/install-nfs-server-and-client-on-ubuntu/ # Check UUID or PARTUUID sudo blkid # add this in /etc/fstab, for example PARTUUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" /media/RAID0WD ntfs defaults,nls=utf8,dmask=0000,fmask=0022,uid=1000,gid=1000,windows_names 0 0 # reboot to check mounted disk sudo reboot # After reboot df -h # add this in /etc/exports, for example /media/RAID0WD 10.0.1.0/24(rw,sync,no_root_squash,no_subtree_check,insecure,anonuid=1000,anongid=1000) echo "/media/RAID0WD 10.0.1.0/24(rw,sync,no_root_squash,no_subtree_check,insecure,anonuid=1000,anongid=1000)" | sudo tee -a /etc/exports # Export and restart NFS Server sudo exportfs -a sudo systemctl restart nfs-kernel-server # Allow nfs on firewall sudo ufw allow from 10.0.1.0/24 to any port nfs showmount -e pi01 # Install Raspberry pi bin and check cpu temperature sudo add-apt-repository ppa:ubuntu-raspi2/ppa sudo apt-get update sudo apt-get install libraspberrypi-bin # My CPU temp=38.0'C vcgencmd measure_temp # Mount NFS from Mac OS X Connect to Server, enter nfs://10.0.1.101/media/RAID0WD # Mount NFS from other Ubuntu nodes # Reference : https://www.raspberrypi.org/documentation/configuration/nfs.md # add this in /etc/fstab, for example 10.0.1.101:/media/RAID0WD /mnt/RAID0WD nfs auto 0 0 echo "10.0.1.101:/media/RAID0WD /mnt/RAID0WD nfs auto 0 0" | sudo tee -a /etc/fstab ## Install Samba for Ubuntu Server # Reference https://linuxize.com/post/how-to-install-and-configure-samba-on-ubuntu-18-04/ # put this in /etc/samba/smb.conf, for example [raidshare] path = /media/RAID0WD browseable = yes guest ok = no read only = no force create mode = 0660 force directory mode = 2770 valid users = ubuntu @ubuntu ## Add Samba password for user ubuntu sudo smbpasswd -a ubuntu # Restart Samba Server sudo systemctl restart smbd # Allow samba on firewall sudo ufw allow 'Samba' # create images folder for berryboot images installation for other nodes cd $HOME ln -sf /media/RAID0WD smb_share cd $HOME/smb_share mkdir -p images # Download required berryboot os images nohup curl -L https://sourceforge.net/projects/berryboot/files/os_images/Ubuntu_Server_arm64_18.04.3.img/download -o Ubuntu_Server_arm64_18.04.3.img & # Check sha1 signature openssl sha1 Ubuntu_Server_arm64_18.04.3.img

Raspbian Buster image setup

shell script    Select all

plug in the Ethernet cable before boot up login: pi password: raspberry change password once login # Reference : https://www.raspberrypi.org/documentation/configuration/security.md sudo apt install openssh-server # check hostname hostname # check network interface ifconfig # change hostname, change to pi01, pi02 ... etc sudo raspi-config -> Select 2. Network Options -> Select N1 Hostname # Enable SSH at the command line using raspi-config sudo raspi-config -> Select 5. Interfacing Options -> Select P2 SSH -> Select Yes # VNC Server at the command line using raspi-config sudo raspi-config -> Select 5. Interfacing Options -> Select P3 VNC -> Select Yes # change timezone sudo dpkg-reconfigure tzdata -> Select the timezone # change locales sudo dpkg-reconfigure locales -> Select the locale # change eth0 to Static IP # Reference : https://pimylifeup.com/raspberry-pi-static-ip-address/ sudo vi /etc/dhcpcd.conf # Restart dhcp sudo service dhcpcd restart # you will have 2 IP addresses, which is good for mounting nfs at start hostname -I # if you have 2 ip addresses and would like to stop the dhcp for eth0, see discussions here. https://raspberrypi.stackexchange.com/questions/52010/set-static-ip-and-stop-dhcp-on-jessie-lite # Mount NTFS external raid disk and Install NFS Server for Raspbian Buster # Check UUID or PARTUUID sudo blkid # add this in /etc/fstab, for example PARTUUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" /media/RAID0WD ntfs defaults,nls=utf8,dmask=0000,fmask=0022,uid=1000,gid=1000,windows_names 0 0 # reboot to check mounted disk sudo reboot # After reboot df -h # Install NFS Server for Raspbian Buster sudo apt install nfs-kernel-server # add this in /etc/exports, for example /media/RAID0WD 10.0.1.0/24(rw,sync,no_root_squash,no_subtree_check,insecure,anonuid=1000,anongid=1000) echo "/media/RAID0WD 10.0.1.0/24(rw,sync,no_root_squash,no_subtree_check,insecure,anonuid=1000,anongid=1000)" | sudo tee -a /etc/exports # Export and restart NFS Server sudo exportfs -a sudo systemctl restart nfs-kernel-server showmount -e pi01 # Mount NFS from Mac OS X Connect to Server, enter nfs://10.0.1.101/media/RAID0WD # Mount NFS from other Raspbian nodes # Reference : https://www.raspberrypi.org/documentation/configuration/nfs.md # add this in /etc/fstab, for example 10.0.1.101:/media/RAID0WD /mnt/RAID0WD nfs auto 0 0 echo "10.0.1.101:/media/RAID0WD /mnt/RAID0WD nfs auto 0 0" | sudo tee -a /etc/fstab # Install Samba for Raspbian Buster sudo apt install samba # put this in /etc/samba/smb.conf, for example [raidshare] path = /media/RAID0WD browseable = yes guest ok = no read only = no force create mode = 0660 force directory mode = 2770 valid users = pi @pi # Add Samba password for user pi sudo smbpasswd -a pi # Restart Samba Server sudo systemctl restart smbd # create images folder for berryboot images installation for other nodes cd $HOME ln -sf /media/RAID0WD smb_share cd $HOME/smb_share mkdir -p images # Download required berryboot os images nohup curl -L https://sourceforge.net/projects/berryboot/files/os_images/Debian_Buster_Raspbian_FULL_2019.10.img/download -o Debian_Buster_Raspbian_FULL_2019.10.img & nohup curl -L https://sourceforge.net/projects/berryboot/files/os_images/Debian_Buster_Raspbian_2019.10.img/download -o Debian_Buster_Raspbian_2019.10.img & # Check sha1 signature openssl sha1 Debian_Buster_Raspbian_FULL_2019.10.img openssl sha1 Debian_Buster_Raspbian_2019.10.img

Nodes pi01 pi02 pi03 pi04 setup

shell script    Select all

Reference : https://magpi.raspberrypi.org/articles/build-a-raspberry-pi-cluster-computer Install OS images to other nodes, change hostname and assign fixed IP address for each node. # Mount NFS for other nodes # Reference : https://www.raspberrypi.org/documentation/configuration/nfs.md # Install package sudo apt install nfs-common # add this in /etc/fstab, for pi02, pi03, pi04 nodes and reboot to be effective 10.0.1.101:/media/RAID0WD /mnt/RAID0WD nfs auto 0 0 echo "10.0.1.101:/media/RAID0WD /mnt/RAID0WD nfs auto 0 0" | sudo tee -a /etc/fstab # Generate ssh key copy it to every other node in the cluster # Reference : http://www.linuxproblem.org/art_9.html # Login pi01 ssh pi@10.0.1.101 ssh-keygen -t rsa ssh-copy-id 10.0.1.102 ssh-copy-id 10.0.1.103 ssh-copy-id 10.0.1.104 exit # Login pi02 ssh pi@10.0.1.102 ssh-keygen -t rsa ssh-copy-id 10.0.1.101 ssh-copy-id 10.0.1.103 ssh-copy-id 10.0.1.104 exit # Login pi03 ssh pi@10.0.1.103 ssh-keygen -t rsa ssh-copy-id 10.0.1.101 ssh-copy-id 10.0.1.102 ssh-copy-id 10.0.1.104 exit # Login pi04 ssh pi@10.0.1.104 ssh-keygen -t rsa ssh-copy-id 10.0.1.101 ssh-copy-id 10.0.1.102 ssh-copy-id 10.0.1.103 exit Install MPI for each node pi01, pi02, pi03, pi04 nodes sudo apt install mpich python3-mpi4py sudo apt install python-mpi4py ##Test01 # Running in pi01 mpirun -n 4 -host 10.0.1.101,10.0.1.102,10.0.1.102,10.0.1.104 hostname ##Test02 # Running in pi01 mkdir -p /media/RAID0WD/Projects ln -sf /media/RAID0WD/Projects $HOME #Create shell script as temp.sh in $HOME/Projects folder cat > $HOME/Projects/temp.sh <<EOF #!/bin/sh echo "\$(hostname)" "\$(vcgencmd measure_temp)" EOF chmod +x $HOME/Projects/temp.sh # create a common Projects folder in all other nodes # all nodes pi02 to pi04 create folder link and assume mounted NFS from pi01 ln -sf /mnt/RAID0WD/Projects $HOME/ # Running in pi01 to pi04 ssh pi@10.0.1.101 mpirun -n 4 -host 10.0.1.101,10.0.1.102,10.0.1.102,10.0.1.104 $HOME/Projects/temp.sh ssh pi@10.0.1.102 mpirun -n 4 -host 10.0.1.101,10.0.1.102,10.0.1.102,10.0.1.104 $HOME/Projects/temp.sh ssh pi@10.0.1.103 mpirun -n 4 -host 10.0.1.101,10.0.1.102,10.0.1.102,10.0.1.104 $HOME/Projects/temp.sh ssh pi@10.0.1.104 mpirun -n 4 -host 10.0.1.101,10.0.1.102,10.0.1.102,10.0.1.104 $HOME/Projects/temp.sh ##Test03 # Add this in /etc/hosts for all nodes pi01, pi02, pi03, pi04 ssh pi@10.0.1.101 echo -e "10.0.1.101\tpi01\n10.0.1.102\tpi02\n10.0.1.103\tpi03\n10.0.1.104\tpi04" | sudo tee -a /etc/hosts ssh pi@10.0.1.102 echo -e "10.0.1.101\tpi01\n10.0.1.102\tpi02\n10.0.1.103\tpi03\n10.0.1.104\tpi04" | sudo tee -a /etc/hosts ssh pi@10.0.1.103 echo -e "10.0.1.101\tpi01\n10.0.1.102\tpi02\n10.0.1.103\tpi03\n10.0.1.104\tpi04" | sudo tee -a /etc/hosts ssh pi@10.0.1.104 echo -e "10.0.1.101\tpi01\n10.0.1.102\tpi02\n10.0.1.103\tpi03\n10.0.1.104\tpi04" | sudo tee -a /etc/hosts #Running on any node cd $HOME/Projects curl -OL https://raw.githubusercontent.com/mpi4py/mpi4py/master/demo/helloworld.py mpirun -n 4 -host pi01,pi02,pi03,pi04 python $HOME/Projects/helloworld.py ##Test04 #Running on exactly 2 processes only cd $HOME/Projects curl -OL https://raw.githubusercontent.com/mpi4py/mpi4py/master/demo/osu_bw.py mpirun -n 2 -host pi01,pi02 python $HOME/Projects/osu_bw.py # create a file $HOME/Projects/4bmachinelist with the list of available pi 4b nodes for mpi # It is not advised to mix pi 4b with 3b together to run mpi, as it will degrade the performance. mpirun -n 2 -machinefile $HOME/Projects/4bmachinelist python $HOME/Projects/osu_bw.py #On each node, launch 1 process only mpirun -npernode 1 -machinefile $HOME/Projects/4bmachinelist $HOME/Projects/temp.sh # -N is same as -npernode, -hostfile is same as -machinefile mpirun -N 1 -hostfile $HOME/Projects/4bmachinelist $HOME/Projects/temp.sh # or mpirun -N 1 -hostfile $HOME/Projects/4bmachinelist bash -c 'echo "$(hostname)" "$(vcgencmd measure_temp)"' | sort ##Test05 # Count how many processes for your cluster, you should get 4 x 4 nodes = 16 processes cd $HOME/Projects curl -L https://github.com/Apress/raspberry-pi-supercomputing/archive/master.zip -o supercomputing.zip unzip supercomputing.zip mpirun -hostfile 4bmachinelist -N 1 python3 $HOME/Projects/raspberry-pi-supercomputing-master/Codes/code/chapter08/prog01.py mpirun -hostfile 4bmachinelist -N 4 python3 $HOME/Projects/raspberry-pi-supercomputing-master/Codes/code/chapter08/prog03.py ##Test06 # Calculate primes # Get the source code from curl -OL https://people.sc.fsu.edu/~jburkardt/py_src/prime_mpi/prime_mpi.py # There are 2 errors to fix before using it # Line 41, there is a missing closing bracket ) at the end # Line 74, should be changed from comm.Reduce ( [ t, MPI.DOUBLE ], [ primes, MPI.INT ], op = MPI.SUM, root = 0 ) to primes = comm.reduce ( t, op = MPI.SUM, root = 0 ) # Test the time required among different processes by running the below # assuming maximum 4 processes (pi 4 CPU has 4 cores) per node cd $HOME/Projects mpirun -n 4 -hostfile 4bmachinelist python3 prime_mpi.py mpirun -n 8 -hostfile 4bmachinelist python3 prime_mpi.py mpirun -n 12 -hostfile 4bmachinelist python3 prime_mpi.py mpirun -n 16 -hostfile 4bmachinelist python3 prime_mpi.py #On each node, launch 3 and 4 processes, and time their differences cd $HOME/Projects time mpirun -N 3 -hostfile 4bmachinelist python3 prime_mpi.py time mpirun -N 4 -hostfile 4bmachinelist python3 prime_mpi.py # Please post your results of Test06 here in the comment ##Test07 # Collective communication using the scatter function example cd ~/Projects curl -L https://pythonprogramming.net/scatter-gather-mpi-mpi4py-tutorial/ | grep -A13 -B1 "from mpi4py" | sed '1d' > sct9.py time mpirun -N 4 -hostfile 4bmachinelist python sct9.py ##Test08 # Collective communication using the gather function example cd ~/Projects curl -L https://pythonprogramming.net/mpi-gather-command-mpi4py-python/ | grep -A19 -B1 "from mpi4py" | sed '1d' > sct10.py time mpirun -N 4 -hostfile 4bmachinelist python sct10.py ##Test09 # mpicc examples cd $HOME/Projects curl -L https://github.com/wesleykendall/mpitutorial/archive/gh-pages.zip -o mpitutorial.zip unzip mpitutorial.zip cd mpitutorial-gh-pages/tutorials/mpi-reduce-and-allreduce/code make time mpirun -N 4 -hostfile $HOME/Projects/4bmachinelist reduce_avg 100000000 time mpirun -N 4 -hostfile $HOME/Projects/4bmachinelist reduce_stddev 100000000

Optional Server or services setup

shell script    Select all

# Nginx (a lightweght webserver) and fast php plugin sudo apt-get install nginx cd; ln -s /usr/share/nginx/html . # check ip address and use browser connect to test web server hostname -I ifconfig eth0 ifconfig wlan0 # for wireless lan ip addr | grep -Po '(?!(inet 127.\d.\d.1))(inet \K(\d{1,3}\.){3}\d{1,3})' # install php in nginx sudo apt install php-fpm php-curl php-gd php-cli php7.3-opcache php-mbstring php-xml php-zip # link the html folder to home cd $HOME sudo chown pi:pi /var/www/html ln -sf /var/www/html . # create testing php page cat > ~/html/info.php <<EOF <?php phpinfo(); ?> EOF # add in /etc/php/7.3/fpm/pool.d/www.conf user = pi group = pi # enable php in nginx and edit this file sudo vi /etc/nginx/sites-enabled/default # and change or add the followings in server section: server { ... # Add index.php to the list if you are using PHP index index.html index.htm index.php index.nginx-debian.html; ... ## Begin - PHP location ~ \.php$ { # Choose either a socket or TCP/IP address fastcgi_pass unix:/var/run/php/php7.3-fpm.sock; # fastcgi_pass 127.0.0.1:9000; fastcgi_split_path_info ^(.+\.php)(/.+)$; fastcgi_index index.php; include fastcgi_params; fastcgi_param SCRIPT_FILENAME $document_root/$fastcgi_script_name; } ## End - PHP ## Begin - Security # deny all direct access for these folders location ~* /(.git|cache|bin|logs|backups|tests)/.*$ { return 403; } # deny running scripts inside core system folders location ~* /(system|vendor)/.*\.(txt|xml|md|html|yaml|php|pl|py|cgi|twig|sh|bat)$ { return 403; } # deny running scripts inside user folder location ~* /user/.*\.(txt|md|yaml|php|pl|py|cgi|twig|sh|bat)$ { return 403; } # deny access to specific files in the root folder location ~ /(LICENSE.txt|composer.lock|composer.json|nginx.conf|web.config|htaccess.txt|\.htaccess) { return 403; } ## End - Security ... } # # see instructions for php7 here -> https://getgrav.org/blog/raspberrypi-nginx-php7-dev # reload web server and test # check to ensure the /var/run/php/php7.3-fpm.sock file exists sudo service nginx restart sudo service php7.3-fpm restart ls -l /var/run/php/php7.3-fpm.sock # Use this command to check whether the web server is working or not curl -L http://127.0.0.1/ curl -L http://127.0.0.1/info.php # python cgi plugin for nginx see here # install minidlna as a media server sudo apt install minidlna # edit /etc/minidlna.conf and add the followings media_dir=V,/media/RAID0WD/MyMovie friendly_name=MyMovie # edit /etc/default/minidlna and add the followings USER="root" GROUP="root" # reload minidlna sudo service minidlna restart sudo service minidlna force-reload # see what network services are working on raspberry pi sudo netstat -ntlp # free ddns no-ip.com Free sign-up and have 3 Hostnames but need to Confirm Every 30 Days # After sign-up, manual update of IP address # Use this command to obtain your public IP address and update on their website host myip.opendns.com resolver1.opendns.com | grep myip # and then download and Install the dynamic update client for Linux https://www.noip.com/support/knowledgebase/installing-the-linux-dynamic-update-client/ You can have 3 host names and have to install virtual host in nginx web server. In server settings of /etc/nginx/sites-available/default, use this settings to map to different html subfolders for different hosts server { ... server_name ~^(.*)\.(.*)\.(.*)$; set $host_name $1; set $subdomain_name $2; set $domain_name $3; root /var/www/html/$host_name.$subdomain_name.$domain_name; ... } # Suppose myhostname1, myhostname2 and myhostname3 are the hostnames obtained from no-ip.com # Setup html root for mutli-hosts as above settings in nginx sudo chown -R pi:pi /var/www/html mkdir -p /var/www/html/10.0.1.101 mkdir -p /var/www/html/127.0.0.1 mkdir -p /var/www/html/myhostname1.ddns.net mkdir -p /var/www/html/myhostname2.ddns.net mkdir -p /var/www/html/myhoatname3.ddns.net # Restart web server to be effective sudo service nginx restart sudo service php7.3-fpm restart # Test web server after restart services cd /var/www/html/ cp index.nginx-debian.html 10.0.1.101 curl -L http://10.0.1.101/ cd /var/www/html/ cp info.php 127.0.0.1/ curl -L http://127.0.0.1/info.php # Test webserver from ddns cd /var/www/html/myhostname1.ddns.net curl -L https://getgrav.org/blog/raspberrypi-nginx-php7-dev -o index.html # make sure your hone router tcp port 80 has been forwarded to your internal Pi host and test with curl -L http://myhostname1.ddns.net/ # And also test the webserver from the browser on Phone # How to access the server and nodes from Android Phone # Recommend Termux from Google Play Store # It use the Volume Up key + keyboard to enter special control characters and can install packages # Reference : https://wiki.termux.com/wiki/Touch_Keyboard apt update apt upgrade # Install ssh and login server apt install openssh ssh-copy-id pi@10.0.1.101 ssh pi@10.0.1.101 # Assume ddns is setup ssh-copy-id pi@myhostname.ddns.net ssh pi@myhostname.ddns.net # Install python 3 apt search python apt install python # To improve command line history productivity, please refer Terminal history usage tips : https://www.howtogeek.com/howto/44997/how-to-use-bash-history-to-improve-your-command-line-productivity/amp/ # ssh forwarding from Android Device (better to have Android tablet with keyboard and mouse) # Reference: https://wiki.termux.com/wiki/Main_Page # Need to install VNC Viewer and Termux from Google Play Store # Install and start vncserver in Termux vncserver -localhost export DISPLAY=":1" # ssh forwarding and login pi ssh -Y pi@10.0.1.104 # install and run jupyter-notebook from pi (after installation of tensorflow) pip3 install jupyter jupyter-notebook & # run the juypter notebook example from https://colab.research.google.com/github/lmoroney/io19/blob/master/Zero%20to%20Hero/Rock-Paper-Scissors.ipynb # After session ended and kill vncserver in Termux vncserver -kill :1 # free ssl certificate for web server https://letsencrypt.org/getting-started/ # After you have your ddns host and nginx running on your Pi, follow the certbot instructions here # Reference : https://certbot.eff.org/lets-encrypt/debianbuster-nginx # set up certbot and obtain ssl certifcate for nginx sudo apt-get install certbot python-certbot-nginx sudo certbot --nginx crontab -e # add this entry to automate letencrypt certificate renewal 43 6 * * * certbot renew --renew-hook "systemctl reload nginx" Certificate and chain have been saved at: /etc/letsencrypt/live/myhostname1.ddns.net/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/myhostname1.ddns.net/privkey.pem Your cert will expire on (90 days after). To obtain a new or tweaked version of this certificate in the future, simply run certbot again with the "certonly" option. To non-interactively renew *all* of your certificates, run "certbot renew" sudo certbot certonly sudo certbot certificates sudo systemctl reload nginx # make sure your home router tcp port 443 has been forwarded to your internal Pi host and test with curl -L https://myhostname1.ddns.net/ # Install OpenVPN Server After you have the ddns hostname or fixed IP address, setup OpenVPN as per instructions here # Reference : https://www.pcmag.com/how-to/how-to-create-a-vpn-server-with-raspberry-pi curl -L https://install.pivpn.io | bash # Choose OpenVPN and use udp port 1194 # Also need to assign fixed IP to your Pi # After installation, sudo reboot to be effective # forward udp port 1194 to the internal IP address of your Pi # Client access # Create configuration file e.g. iphone, macbook etc. pivpn add # Client access Use "OpenVPN Connect App" for phone and notebook send the profile to the client by email and import, e.g. for iPhone # Makeuse of X11 forwarding to run graphical applications see https://kb.iu.edu/d/bdnt For macOS High Sierra or above, please see this guide to install xQuartz. https://www.unixtutorial.org/get-x11-forwarding-in-macos-high-sierra/
You can install it from https://www.xquartz.org or via "sudo port -v install xorg" in the terminal. For In Windows 10 WSL, and install XServer in Windows 10 such as xming -> https://sourceforge.net/projects/xming/ and set DISPLAT for WSL x11-apps export DISPLAY=localhost:0.0 export DISPLAY=:0 For Linux, there is built-in support. In Terminal, type ssh -Y pi@10.0.1.101 # After login Raspberry pi sudo apt-get install idle3 idle & # run scratch sudo apt-get install scratch scratch & # run codeblocks sudo apt install codeblocks codeblocks & If you get "cannot open display error", see discussions here. https://superuser.com/questions/310197/how-do-i-fix-a-cannot-open-display-error-when-opening-an-x-program-after-sshi # run browser chromium-browser & # Access Raspberry Pi Desktop Remotely Use Windows Remote Desktop Client to connect to the Raspberry Pi. # Install xrdp and reboot the pi after installation sudo apt install xrdp sudo systemctl restart xrdp # For macOS, there is "Microsoft Remote Desktop Connection Client for Mac" in the Mac App Store. # setup cron job to backup project data # Reference : https://www.raspberrypi.org/documentation/linux/usage/cron.md Create a shell script e.g. cat > /home/pi/backup.sh <<EOF #!/bin/sh cd /media/RAID0WD; tar --exclude='./Projects/Downloads' -zcf Projects-backup/"Projects$(date '+%Y%m%d').tar.gz" Projects EOF chmod a+x /home/pi/backup.sh mkdir -p /media/RAID0WD/Projects-backup # add this entry in the crontab 0 0 * * * /home/pi/backup.sh #List crontab crontab -l # Overclock # Reference : https://www.seeedstudio.com/blog/2020/02/12/how-to-safely-overclock-your-raspberry-pi-4-to-2-147ghz/ #Install docker # Not for berryboot images curl -sSL https://get.docker.com | sh sudo usermod -aG docker $USER sudo newgrp docker sudo apt install libffi-dev libssl-dev python3 python3-pip sudo apt-get remove python-configparser sudo pip3 -v install docker-compose # test docker hello-world docker run hello-world cd $HOME mkdir my-wordpress cd my-wordpress cat > docker-compose.yaml <<EOF version: '3.2' services: db: image: hypriot/rpi-mysql volumes: - "./.data/db:/var/lib/mysql" restart: always environment: MYSQL_ROOT_PASSWORD: wordpress MYSQL_DATABASE: wordpress MYSQL_USER: wordpress MYSQL_PASSWORD: wordpress wordpress: depends_on: - db image: wordpress:latest links: - db ports: - "8000:80" restart: always environment: WORDPRESS_DB_HOST: db:3306 WORDPRESS_DB_PASSWORD: wordpress EOF docker-compose up -d # then try browser http://localhost:8000/ # stop the docker-compose example docker-compose down # test nodejs cd $HOME git clone https://github.com/hypriot/rpi-node-haproxy-example cd rpi-node-haproxy-example docker-compose up curl http://localhost:80 curl http://localhost:70 docker-compose stop # test nodejs + mongodb cd $HOME git clone https://github.com/hagaik/easy-node-authentication.git cd easy-node-authentication cat > config/database.js <<EOF // config/database.js module.exports = { 'url' : 'mongodb://mongo:27017' // looks like mongodb://<user>:<pass>@mongo.onmodulus.net:27017/Mikha4ot }; EOF cat > Dockerfile <<EOF FROM hypriot/rpi-node # Create app directory WORKDIR /usr/src/app # Install app dependencies COPY package*.json ./ RUN npm install # Copy app source code COPY . . #Expose port and start application EXPOSE 8080 CMD [ "npm", "start" ] EOF cat > docker-compose.yaml <<EOF version: "3.2" services: db: image: dhermanns/rpi-mongo volumes: - "data-volume:/data/db" restart: always ports: - "27017:27017" expose: - "27017" webm: build: . depends_on: - db ports: - "8080:8080" expose: - "8080" volumes: data-volume: EOF # build and start as daemon docker-compose up --build -d # then try browser http://localhost:8080/ # stop and down docker-compose down -v # move docker data-root to nfs mounted drive e.g. /mnt/docker-data sudo service docker stop cat << EOF | sudo tee -a /etc/docker/daemon.json { "storage-driver": "overlay", "data-root": "/mnt/docker-data" } EOF sudo rsync -aP /var/lib/docker/ /mnt/docker-data sudo mv /var/lib/docker /var/lib/docker.old sudo service docker start

Install tensorflow and horovod for Pi 4 cluster

shell script    Select all

Install all these packages for every nodes in the Pi 4 cluster in order to run horovod with tensorflow # Install tenorflow 2.1.0 # Reference https://qengineering.eu/install-tensorflow-2.1.0-on-raspberry-pi-4.html # or https://qengineering.eu/install-tensorflow-2.2.0-on-raspberry-pi-4.html for tenorflow 2.2.0 cd ~/Projects sudo apt-get install gfortran sudo apt-get install libhdf5-dev libc-ares-dev libeigen3-dev sudo apt-get install libatlas-base-dev libopenblas-dev libblas-dev sudo apt-get install liblapack-dev cython sudo pip3 install pybind11 sudo apt-get install python3-h5py # upgrade pip3 and check pip3 version curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py python3 get-pip.py --force-reinstall # pip 20.1.1 from /home/pi/.local/lib/python3.7/site-packages/pip (python 3.7) python3 --version pip3 --version # download the wheel wget https://github.com/Qengineering/Tensorflow-Raspberry-Pi/raw/master/tensorflow-2.1.0-cp37-cp37m-linux_armv7l.whl # install TensorFlow 2.1.0 python3 -m pip install --user tensorflow-2.1.0-cp37-cp37m-linux_armv7l.whl # test run examples as in how-to-install-tensorflow-with-gpu python3 -m pip install --user matplotlib python3 -m pip install --user pandas python3 -m pip install --user keras scikit-learn cd ~/Projects curl -L https://tinyurl.com/tensorflowwin | grep -A7 tftest.py | sed '1,2d' > tftest.py python3 tftest.py curl -L https://tinyurl.com/tensorflowwin | grep -A129 irislearn.py | sed '1,8d' > irislearn.py curl -L https://tinyurl.com/tensorflowwin | grep -A150 iris.data.nbsp | sed '1d' > iris.data python3 irislearn.py curl -L https://tinyurl.com/tensorflowwin | grep -A37 keraslearn.py | sed '1,3d' > keraslearn.py curl -L https://tinyurl.com/tensorflowwin | grep -A768 pima-indians-diabetes.data.nbsp | sed '1d' > pima-indians-diabetes.data python3 keraslearn.py # install horovod # https://github.com/horovod/horovod#install python3 -m pip install cffi>=1.4.0 cloudpickle python3 -m pip install horovod # Reboot to make it effective sudo reboot # First test run the simple hellohorovod.py cd ~/Projects cat > hellohorovod.py <<EOF from mpi4py import MPI import horovod.tensorflow as hvd # Split COMM_WORLD into subcommunicators subcomm = MPI.COMM_WORLD.Split(color=MPI.COMM_WORLD.rank % 2, key=MPI.COMM_WORLD.rank) # Initialize Horovod hvd.init(comm=subcomm) print('COMM_WORLD rank: %d, Name: %s, Horovod rank: %d' % (MPI.COMM_WORLD.rank, MPI.Get_processor_name(), hvd.rank())) EOF # run it with cd ~/Projects horovodrun -np 1 -H localhost:1,pi02:1,pi03:1,pi04:4 python3 hellohorovod.py # Then try run the tensorflow v2 example as below # please be warned that and watch out the temperature of PIs. # https://github.com/horovod/horovod/blob/master/examples/tensorflow2_keras_mnist.py # For non-keras, the sample is https://github.com/horovod/horovod/blob/master/examples/tensorflow2_mnist.py cd ~/Projects wget https://raw.githubusercontent.com/horovod/horovod/master/examples/tensorflow2_keras_mnist.py time horovodrun -np 4 -H localhost:1,pi02:1,pi03:1,pi04:1 python3 tensorflow2_keras_mnist.py time horovodrun -np 8 -H localhost:2,pi02:2,pi03:2,pi04:2 python3 tensorflow2_keras_mnist.py time horovodrun -np 12 -H localhost:3,pi02:3,pi03:3,pi04:3 python3 tensorflow2_keras_mnist.py # 1 slot per node has the faster performance for the Pi cluster # As with the release of the new 8GB Pi4B model, it is possible to increase the number of slots for these 8GB machines. # append self public key to self authorized list ssh pi@10.0.1.101 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys exit ssh pi@10.0.1.102 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys exit ssh pi@10.0.1.103 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys exit ssh pi@10.0.1.104 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys exit # create ~/Projects/myhostfile cat > ~/Projects/myhostfile <<EOF pi01 slots=1 pi02 slots=1 pi03 slots=1 pi04 slots=1 EOF # running in background with nohup cd ~/Projects nohup horovodrun -np 4 -hostfile myhostfile python3 tensorflow2_keras_mnist.py & cat ~/Projects/nohup.out exit # create folder in non-nfs ext4 partition folder if running in nodes other than pi01 mkdir -p ~/horovod ssh pi01 'mkdir -p ~/horovod' ssh pi02 'mkdir -p ~/horovod' ssh pi03 'mkdir -p ~/horovod' ssh pi04 'mkdir -p ~/horovod' # When running in pi02, pi03 and pi04, it cannot start in the nfs shared folder if it is not an ext4 partition. # e.g. when start in pi03 ssh pi03 cd ~/horovod horovodrun -np 4 -hostfile ~/Projects/myhostfile python3 ~/Projects/tensorflow2_keras_mnist.py # run it with horovod and time and redirect outputfile to keras_mnist.np4.out cd ~/horovod nohup bash -c 'time horovodrun -np 4 -hostfile ~/Projects/myhostfile3 python3 ~/Projects/tensorflow2_keras_mnist.py' &> keras_mnist.np4.out & # The time for number of nodes for this testing nohup bash -c 'time horovodrun -np 4 -H pi01:1,pi02:1,pi03:1,pi04:1 --output-filename logs python3 ~/Projects/tensorflow2_keras_mnist.py' &> nohup.out.np4_1111 & nohup bash -c 'time horovodrun -np 3 -H pi01:1,pi02:1,pi03:1 --output-filename logs python3 ~/Projects/tensorflow2_keras_mnist.py' &> nohup.out.np3_111 & nohup bash -c 'time horovodrun -np 2 -H pi01:1,pi02:1 --output-filename logs python3 ~/Projects/tensorflow2_keras_mnist.py' &> nohup.out.np2_11 & nohup bash -c 'time horovodrun -np 1 -H pi01:1 --output-filename logs python3 ~/Projects/tensorflow2_keras_mnist.py' &> nohup.out.np1_1 & nohup.out.np4_1111:real not yet done (may be 50m) nohup.out.np3_111:real 88m13.692s nohup.out.np2_11:real 129m11.322s nohup.out.np1_1:real 207m28.119s

Some shortcuts

shell script    Select all

# Offending ECDSA key # Offending ECDSA key in $HOME/.ssh/known_hosts:5 # For Mac sed sed -i '' '5d' $HOME/.ssh/known_hosts # For GNU sed sed -i '5d' $HOME/.ssh/known_hosts # print line 5 to 6 of a text file sed -n '5,6p' $HOME/.ssh/known_hosts # print line 1 and Line 5 to 6 of a text file sed -n -e '1p' -e '5,6p' $HOME/.ssh/known_hosts # add all the hosts to the ~/.ssh/known_hosts file using ssh-keyscan # first login pi01 10.0.1.101 ssh pi@10.0.1.101 ssh-keyscan -t rsa,dsa pi02,pi01,pi04 > ~/.ssh/known_hosts

How to install tensorflow gpu for Ubuntu 20.04 or Mac OSX

1. Boot into Ubuntu 20.04 or Ubuntu 18.04

2. Download Anaconda3-2020.02-Linux-x86_64.sh from https://repo.anaconda.com/archive/

3. Start installation in Terminal

shell script    Select all

wget https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh /bin/sh Anaconda3-2020.02-Linux-x86_64.sh # After completion of installation, and accept init exit

4. Start Terminal App again

5. Update Anaconda, create virtual environment and install packages

shell script    Select all

conda config --set auto_activate_base false conda deactivate # Exit conda shell conda update conda conda update anaconda conda update python conda update --all mkdir -p $HOME/Projects cd $HOME/Projects/ conda create --name tf-gpu conda activate tf-gpu # Now in (tf-gpu) shell conda install tensorflow-gpu conda install matplotlib pandas scikit-learn conda install keras

6. Run the examples in how-to-install-tensorflow-with-gpu

or run the TensorFlow2 tutorial
conda activate tf-gpu
git clone https://github.com/lambdal/TensorFlow2-tutorial.git
cd TensorFlow2-tutorial/01-basic-image-classification/
python resnet_cifar.py


7. It is not necessary to have the required GPU hardware in order to install the gpu package, For machine without the required GPU, just first install tensorflow_gpu package and then download and install the tensorflow wheels file here
pip install --upgrade ~/Downloads/tensorflow-2.1.0-cp37-cp37m-linux_x86_64.whl
, in order to optimise for CPU with AVX, AVX2, and FMA or to build from source here -> https://www.tensorflow.org/install/source

Install tensorflow-mkl for Mac OS X

The installation steps are very similar, except
Download and install Anaconda3-2020.02-MacOSX-x86_64.pkg
curl -OL https://repo.anaconda.com/archive/Anaconda3-2020.02-MacOSX-x86_64.pkg
And then follow step 4 to step 6 above. In step 5, additionally, install tensorflow-mkl from anaconda channel. tensorflow-mkl is optimized with Intel® MKL-DNN to use the following CPU instructions in performance critical operations: SSE4.1 SSE4.2 AVX AVX2 FMA (Specifically, Intel MKL-DNN is optimized for Intel® Xeon® processors and Intel® Xeon Phi™ processors). Package installation steps are
conda install tensorflow
conda install -c anaconda tensorflow-mkl
conda install matplotlib pandas scikit-learn keras


How to Install TensorFlow with GPU Support on Windows 10 Notebook

Need Windows 10, 64 bits (Home Edition is OK) and NVIDIA Graphics Card with minimum Cuda capability of 3.0(2GB Graphic RAM, or more is better for larger dataset), i5/i7 Notebook with 8GB RAM (more is better) are fine. The latest gaming i7 laptop typically has 16GB RAM or more and NVIDIA Card with 6GB graphics.

This installation guide works for my i7 SAMSUNG notebook with GT 650M.

How to Install TensorFlow with GPU Support on Windows 10


After the installation, and go into tf-gpu python environment
conda activate tf-gpu
test run the followings

python tftest.py    Select all

#!/usr/bin/env python import tensorflow as tf with tf.compat.v1.Session() as sess: hello = tf.constant('Hello, TensorFlow! '+ tf.__version__) print (sess.run(hello).decode()) sess.close()

The following irislearn.py requires the additional installation of python modules
pip install matplotlib pandas sklearn
and download the iris.data from
https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data
or create the iris.data file from the content below

python irislearn.py    Select all

#!/usr/bin/env python # Check the versions of libraries # Python version import sys print('Python: {}'.format(sys.version)) # scipy import scipy print('scipy: {}'.format(scipy.__version__)) # numpy import numpy print('numpy: {}'.format(numpy.__version__)) # matplotlib import matplotlib print('matplotlib: {}'.format(matplotlib.__version__)) # pandas import pandas print('pandas: {}'.format(pandas.__version__)) # scikit-learn import sklearn print('sklearn: {}'.format(sklearn.__version__)) # Load libraries import pandas from pandas.plotting import scatter_matrix import matplotlib.pyplot as plt from sklearn import model_selection from sklearn.metrics import classification_report from sklearn.metrics import confusion_matrix from sklearn.metrics import accuracy_score from sklearn.linear_model import LogisticRegression from sklearn.tree import DecisionTreeClassifier from sklearn.neighbors import KNeighborsClassifier from sklearn.discriminant_analysis import LinearDiscriminantAnalysis from sklearn.naive_bayes import GaussianNB from sklearn.svm import SVC # Load dataset #url = "https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data" url = "iris.data" names = ['sepal-length', 'sepal-width', 'petal-length', 'petal-width', 'class'] dataset = pandas.read_csv(url, names=names) # shape print(dataset.shape) # head print(dataset.head(20)) # descriptions print(dataset.describe()) # class distribution print(dataset.groupby('class').size()) # box and whisker plots dataset.plot(kind='box', subplots=True, layout=(2,2), sharex=False, sharey=False) plt.suptitle("Box and Whisker Plots for inputs") plt.show() # histograms dataset.hist() plt.suptitle('Histograms for inputs') plt.show() # scatter plot matrix scatter_matrix(dataset) plt.suptitle('Scatter Plot Matrix for inputs') plt.show() # Split-out validation dataset array = dataset.values X = array[:,0:4] Y = array[:,4] validation_size = 0.20 seed = 7 X_train, X_validation, Y_train, Y_validation = model_selection.train_test_split(X, Y, test_size=validation_size, random_state=seed) # Test options and evaluation metric seed = 7 scoring = 'accuracy' # Spot Check Algorithms models = [] models.append(('LR', LogisticRegression(max_iter=10000))) models.append(('LDA', LinearDiscriminantAnalysis())) models.append(('KNN', KNeighborsClassifier())) models.append(('CART', DecisionTreeClassifier())) models.append(('NB', GaussianNB())) models.append(('SVM', SVC())) # evaluate each model in turn results = [] names = [] for name, model in models: kfold = model_selection.KFold(n_splits=10, random_state=seed, shuffle=True) cv_results = model_selection.cross_val_score(model, X_train, Y_train, cv=kfold, scoring=scoring) results.append(cv_results) names.append(name) msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std()) print(msg) # Compare Algorithms fig = plt.figure() fig.suptitle('Algorithm Comparison') ax = fig.add_subplot(111) plt.boxplot(results) ax.set_xticklabels(names) plt.show() # Make predictions on validation dataset knn = KNeighborsClassifier() knn.fit(X_train, Y_train) predictions = knn.predict(X_validation) print(accuracy_score(Y_validation, predictions)) print(confusion_matrix(Y_validation, predictions)) print(classification_report(Y_validation, predictions)) print("Make predictions on LogisticRegression Model") model = LogisticRegression(max_iter=10000) model.fit(X_train, Y_train) predictions = model.predict(X_validation) print(accuracy_score(Y_validation, predictions)) print(confusion_matrix(Y_validation, predictions)) print(classification_report(Y_validation, predictions)) # print prediction results on test data for i, prediction in enumerate(predictions): print ('Predicted: %s, Target: %s %s' % (prediction, Y_validation[i], '' if prediction==Y_validation[i] else '(WRONG!!!)'))

iris.data

iris.data    Select all 5.1,3.5,1.4,0.2,Iris-setosa 4.9,3.0,1.4,0.2,Iris-setosa 4.7,3.2,1.3,0.2,Iris-setosa 4.6,3.1,1.5,0.2,Iris-setosa 5.0,3.6,1.4,0.2,Iris-setosa 5.4,3.9,1.7,0.4,Iris-setosa 4.6,3.4,1.4,0.3,Iris-setosa 5.0,3.4,1.5,0.2,Iris-setosa 4.4,2.9,1.4,0.2,Iris-setosa 4.9,3.1,1.5,0.1,Iris-setosa 5.4,3.7,1.5,0.2,Iris-setosa 4.8,3.4,1.6,0.2,Iris-setosa 4.8,3.0,1.4,0.1,Iris-setosa 4.3,3.0,1.1,0.1,Iris-setosa 5.8,4.0,1.2,0.2,Iris-setosa 5.7,4.4,1.5,0.4,Iris-setosa 5.4,3.9,1.3,0.4,Iris-setosa 5.1,3.5,1.4,0.3,Iris-setosa 5.7,3.8,1.7,0.3,Iris-setosa 5.1,3.8,1.5,0.3,Iris-setosa 5.4,3.4,1.7,0.2,Iris-setosa 5.1,3.7,1.5,0.4,Iris-setosa 4.6,3.6,1.0,0.2,Iris-setosa 5.1,3.3,1.7,0.5,Iris-setosa 4.8,3.4,1.9,0.2,Iris-setosa 5.0,3.0,1.6,0.2,Iris-setosa 5.0,3.4,1.6,0.4,Iris-setosa 5.2,3.5,1.5,0.2,Iris-setosa 5.2,3.4,1.4,0.2,Iris-setosa 4.7,3.2,1.6,0.2,Iris-setosa 4.8,3.1,1.6,0.2,Iris-setosa 5.4,3.4,1.5,0.4,Iris-setosa 5.2,4.1,1.5,0.1,Iris-setosa 5.5,4.2,1.4,0.2,Iris-setosa 4.9,3.1,1.5,0.1,Iris-setosa 5.0,3.2,1.2,0.2,Iris-setosa 5.5,3.5,1.3,0.2,Iris-setosa 4.9,3.1,1.5,0.1,Iris-setosa 4.4,3.0,1.3,0.2,Iris-setosa 5.1,3.4,1.5,0.2,Iris-setosa 5.0,3.5,1.3,0.3,Iris-setosa 4.5,2.3,1.3,0.3,Iris-setosa 4.4,3.2,1.3,0.2,Iris-setosa 5.0,3.5,1.6,0.6,Iris-setosa 5.1,3.8,1.9,0.4,Iris-setosa 4.8,3.0,1.4,0.3,Iris-setosa 5.1,3.8,1.6,0.2,Iris-setosa 4.6,3.2,1.4,0.2,Iris-setosa 5.3,3.7,1.5,0.2,Iris-setosa 5.0,3.3,1.4,0.2,Iris-setosa 7.0,3.2,4.7,1.4,Iris-versicolor 6.4,3.2,4.5,1.5,Iris-versicolor 6.9,3.1,4.9,1.5,Iris-versicolor 5.5,2.3,4.0,1.3,Iris-versicolor 6.5,2.8,4.6,1.5,Iris-versicolor 5.7,2.8,4.5,1.3,Iris-versicolor 6.3,3.3,4.7,1.6,Iris-versicolor 4.9,2.4,3.3,1.0,Iris-versicolor 6.6,2.9,4.6,1.3,Iris-versicolor 5.2,2.7,3.9,1.4,Iris-versicolor 5.0,2.0,3.5,1.0,Iris-versicolor 5.9,3.0,4.2,1.5,Iris-versicolor 6.0,2.2,4.0,1.0,Iris-versicolor 6.1,2.9,4.7,1.4,Iris-versicolor 5.6,2.9,3.6,1.3,Iris-versicolor 6.7,3.1,4.4,1.4,Iris-versicolor 5.6,3.0,4.5,1.5,Iris-versicolor 5.8,2.7,4.1,1.0,Iris-versicolor 6.2,2.2,4.5,1.5,Iris-versicolor 5.6,2.5,3.9,1.1,Iris-versicolor 5.9,3.2,4.8,1.8,Iris-versicolor 6.1,2.8,4.0,1.3,Iris-versicolor 6.3,2.5,4.9,1.5,Iris-versicolor 6.1,2.8,4.7,1.2,Iris-versicolor 6.4,2.9,4.3,1.3,Iris-versicolor 6.6,3.0,4.4,1.4,Iris-versicolor 6.8,2.8,4.8,1.4,Iris-versicolor 6.7,3.0,5.0,1.7,Iris-versicolor 6.0,2.9,4.5,1.5,Iris-versicolor 5.7,2.6,3.5,1.0,Iris-versicolor 5.5,2.4,3.8,1.1,Iris-versicolor 5.5,2.4,3.7,1.0,Iris-versicolor 5.8,2.7,3.9,1.2,Iris-versicolor 6.0,2.7,5.1,1.6,Iris-versicolor 5.4,3.0,4.5,1.5,Iris-versicolor 6.0,3.4,4.5,1.6,Iris-versicolor 6.7,3.1,4.7,1.5,Iris-versicolor 6.3,2.3,4.4,1.3,Iris-versicolor 5.6,3.0,4.1,1.3,Iris-versicolor 5.5,2.5,4.0,1.3,Iris-versicolor 5.5,2.6,4.4,1.2,Iris-versicolor 6.1,3.0,4.6,1.4,Iris-versicolor 5.8,2.6,4.0,1.2,Iris-versicolor 5.0,2.3,3.3,1.0,Iris-versicolor 5.6,2.7,4.2,1.3,Iris-versicolor 5.7,3.0,4.2,1.2,Iris-versicolor 5.7,2.9,4.2,1.3,Iris-versicolor 6.2,2.9,4.3,1.3,Iris-versicolor 5.1,2.5,3.0,1.1,Iris-versicolor 5.7,2.8,4.1,1.3,Iris-versicolor 6.3,3.3,6.0,2.5,Iris-virginica 5.8,2.7,5.1,1.9,Iris-virginica 7.1,3.0,5.9,2.1,Iris-virginica 6.3,2.9,5.6,1.8,Iris-virginica 6.5,3.0,5.8,2.2,Iris-virginica 7.6,3.0,6.6,2.1,Iris-virginica 4.9,2.5,4.5,1.7,Iris-virginica 7.3,2.9,6.3,1.8,Iris-virginica 6.7,2.5,5.8,1.8,Iris-virginica 7.2,3.6,6.1,2.5,Iris-virginica 6.5,3.2,5.1,2.0,Iris-virginica 6.4,2.7,5.3,1.9,Iris-virginica 6.8,3.0,5.5,2.1,Iris-virginica 5.7,2.5,5.0,2.0,Iris-virginica 5.8,2.8,5.1,2.4,Iris-virginica 6.4,3.2,5.3,2.3,Iris-virginica 6.5,3.0,5.5,1.8,Iris-virginica 7.7,3.8,6.7,2.2,Iris-virginica 7.7,2.6,6.9,2.3,Iris-virginica 6.0,2.2,5.0,1.5,Iris-virginica 6.9,3.2,5.7,2.3,Iris-virginica 5.6,2.8,4.9,2.0,Iris-virginica 7.7,2.8,6.7,2.0,Iris-virginica 6.3,2.7,4.9,1.8,Iris-virginica 6.7,3.3,5.7,2.1,Iris-virginica 7.2,3.2,6.0,1.8,Iris-virginica 6.2,2.8,4.8,1.8,Iris-virginica 6.1,3.0,4.9,1.8,Iris-virginica 6.4,2.8,5.6,2.1,Iris-virginica 7.2,3.0,5.8,1.6,Iris-virginica 7.4,2.8,6.1,1.9,Iris-virginica 7.9,3.8,6.4,2.0,Iris-virginica 6.4,2.8,5.6,2.2,Iris-virginica 6.3,2.8,5.1,1.5,Iris-virginica 6.1,2.6,5.6,1.4,Iris-virginica 7.7,3.0,6.1,2.3,Iris-virginica 6.3,3.4,5.6,2.4,Iris-virginica 6.4,3.1,5.5,1.8,Iris-virginica 6.0,3.0,4.8,1.8,Iris-virginica 6.9,3.1,5.4,2.1,Iris-virginica 6.7,3.1,5.6,2.4,Iris-virginica 6.9,3.1,5.1,2.3,Iris-virginica 5.8,2.7,5.1,1.9,Iris-virginica 6.8,3.2,5.9,2.3,Iris-virginica 6.7,3.3,5.7,2.5,Iris-virginica 6.7,3.0,5.2,2.3,Iris-virginica 6.3,2.5,5.0,1.9,Iris-virginica 6.5,3.0,5.2,2.0,Iris-virginica 6.2,3.4,5.4,2.3,Iris-virginica 5.9,3.0,5.1,1.8,Iris-virginica

The following keraslearn.py requires the additional installation of python module as below
pip install keras

python keraslearn.py    Select all#!/usr/bin/env python from keras.models import Sequential from keras.layers import Dense import numpy import time # fix random seed for reproducibility numpy.random.seed(7) # load pima indians dataset #dataset = numpy.loadtxt("pima-indians-diabetes.csv", delimiter=",") dataset = numpy.loadtxt("pima-indians-diabetes.data", delimiter=",") # split into input (X) and output (Y) variables X = dataset[:,0:8] Y = dataset[:,8] # create model model = Sequential() model.add(Dense(12, input_dim=8, activation='relu')) model.add(Dense(1, activation='sigmoid')) # Compile model model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) start_time=time.time() # Fit the model #model.fit(X, Y, epochs=150, batch_size=10) model.fit(X, Y, 10, 150) # parameters change to keras 1.2.2 # evaluate the model scores = model.evaluate(X, Y) print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100)) print("\nTraining took %.2f seconds\n" %(time.time()-start_time))

pima-indians-diabetes.data

pima-indians-diabetes.data    Select all 6,148,72,35,0,33.6,0.627,50,1 1,85,66,29,0,26.6,0.351,31,0 8,183,64,0,0,23.3,0.672,32,1 1,89,66,23,94,28.1,0.167,21,0 0,137,40,35,168,43.1,2.288,33,1 5,116,74,0,0,25.6,0.201,30,0 3,78,50,32,88,31.0,0.248,26,1 10,115,0,0,0,35.3,0.134,29,0 2,197,70,45,543,30.5,0.158,53,1 8,125,96,0,0,0.0,0.232,54,1 4,110,92,0,0,37.6,0.191,30,0 10,168,74,0,0,38.0,0.537,34,1 10,139,80,0,0,27.1,1.441,57,0 1,189,60,23,846,30.1,0.398,59,1 5,166,72,19,175,25.8,0.587,51,1 7,100,0,0,0,30.0,0.484,32,1 0,118,84,47,230,45.8,0.551,31,1 7,107,74,0,0,29.6,0.254,31,1 1,103,30,38,83,43.3,0.183,33,0 1,115,70,30,96,34.6,0.529,32,1 3,126,88,41,235,39.3,0.704,27,0 8,99,84,0,0,35.4,0.388,50,0 7,196,90,0,0,39.8,0.451,41,1 9,119,80,35,0,29.0,0.263,29,1 11,143,94,33,146,36.6,0.254,51,1 10,125,70,26,115,31.1,0.205,41,1 7,147,76,0,0,39.4,0.257,43,1 1,97,66,15,140,23.2,0.487,22,0 13,145,82,19,110,22.2,0.245,57,0 5,117,92,0,0,34.1,0.337,38,0 5,109,75,26,0,36.0,0.546,60,0 3,158,76,36,245,31.6,0.851,28,1 3,88,58,11,54,24.8,0.267,22,0 6,92,92,0,0,19.9,0.188,28,0 10,122,78,31,0,27.6,0.512,45,0 4,103,60,33,192,24.0,0.966,33,0 11,138,76,0,0,33.2,0.420,35,0 9,102,76,37,0,32.9,0.665,46,1 2,90,68,42,0,38.2,0.503,27,1 4,111,72,47,207,37.1,1.390,56,1 3,180,64,25,70,34.0,0.271,26,0 7,133,84,0,0,40.2,0.696,37,0 7,106,92,18,0,22.7,0.235,48,0 9,171,110,24,240,45.4,0.721,54,1 7,159,64,0,0,27.4,0.294,40,0 0,180,66,39,0,42.0,1.893,25,1 1,146,56,0,0,29.7,0.564,29,0 2,71,70,27,0,28.0,0.586,22,0 7,103,66,32,0,39.1,0.344,31,1 7,105,0,0,0,0.0,0.305,24,0 1,103,80,11,82,19.4,0.491,22,0 1,101,50,15,36,24.2,0.526,26,0 5,88,66,21,23,24.4,0.342,30,0 8,176,90,34,300,33.7,0.467,58,1 7,150,66,42,342,34.7,0.718,42,0 1,73,50,10,0,23.0,0.248,21,0 7,187,68,39,304,37.7,0.254,41,1 0,100,88,60,110,46.8,0.962,31,0 0,146,82,0,0,40.5,1.781,44,0 0,105,64,41,142,41.5,0.173,22,0 2,84,0,0,0,0.0,0.304,21,0 8,133,72,0,0,32.9,0.270,39,1 5,44,62,0,0,25.0,0.587,36,0 2,141,58,34,128,25.4,0.699,24,0 7,114,66,0,0,32.8,0.258,42,1 5,99,74,27,0,29.0,0.203,32,0 0,109,88,30,0,32.5,0.855,38,1 2,109,92,0,0,42.7,0.845,54,0 1,95,66,13,38,19.6,0.334,25,0 4,146,85,27,100,28.9,0.189,27,0 2,100,66,20,90,32.9,0.867,28,1 5,139,64,35,140,28.6,0.411,26,0 13,126,90,0,0,43.4,0.583,42,1 4,129,86,20,270,35.1,0.231,23,0 1,79,75,30,0,32.0,0.396,22,0 1,0,48,20,0,24.7,0.140,22,0 7,62,78,0,0,32.6,0.391,41,0 5,95,72,33,0,37.7,0.370,27,0 0,131,0,0,0,43.2,0.270,26,1 2,112,66,22,0,25.0,0.307,24,0 3,113,44,13,0,22.4,0.140,22,0 2,74,0,0,0,0.0,0.102,22,0 7,83,78,26,71,29.3,0.767,36,0 0,101,65,28,0,24.6,0.237,22,0 5,137,108,0,0,48.8,0.227,37,1 2,110,74,29,125,32.4,0.698,27,0 13,106,72,54,0,36.6,0.178,45,0 2,100,68,25,71,38.5,0.324,26,0 15,136,70,32,110,37.1,0.153,43,1 1,107,68,19,0,26.5,0.165,24,0 1,80,55,0,0,19.1,0.258,21,0 4,123,80,15,176,32.0,0.443,34,0 7,81,78,40,48,46.7,0.261,42,0 4,134,72,0,0,23.8,0.277,60,1 2,142,82,18,64,24.7,0.761,21,0 6,144,72,27,228,33.9,0.255,40,0 2,92,62,28,0,31.6,0.130,24,0 1,71,48,18,76,20.4,0.323,22,0 6,93,50,30,64,28.7,0.356,23,0 1,122,90,51,220,49.7,0.325,31,1 1,163,72,0,0,39.0,1.222,33,1 1,151,60,0,0,26.1,0.179,22,0 0,125,96,0,0,22.5,0.262,21,0 1,81,72,18,40,26.6,0.283,24,0 2,85,65,0,0,39.6,0.930,27,0 1,126,56,29,152,28.7,0.801,21,0 1,96,122,0,0,22.4,0.207,27,0 4,144,58,28,140,29.5,0.287,37,0 3,83,58,31,18,34.3,0.336,25,0 0,95,85,25,36,37.4,0.247,24,1 3,171,72,33,135,33.3,0.199,24,1 8,155,62,26,495,34.0,0.543,46,1 1,89,76,34,37,31.2,0.192,23,0 4,76,62,0,0,34.0,0.391,25,0 7,160,54,32,175,30.5,0.588,39,1 4,146,92,0,0,31.2,0.539,61,1 5,124,74,0,0,34.0,0.220,38,1 5,78,48,0,0,33.7,0.654,25,0 4,97,60,23,0,28.2,0.443,22,0 4,99,76,15,51,23.2,0.223,21,0 0,162,76,56,100,53.2,0.759,25,1 6,111,64,39,0,34.2,0.260,24,0 2,107,74,30,100,33.6,0.404,23,0 5,132,80,0,0,26.8,0.186,69,0 0,113,76,0,0,33.3,0.278,23,1 1,88,30,42,99,55.0,0.496,26,1 3,120,70,30,135,42.9,0.452,30,0 1,118,58,36,94,33.3,0.261,23,0 1,117,88,24,145,34.5,0.403,40,1 0,105,84,0,0,27.9,0.741,62,1 4,173,70,14,168,29.7,0.361,33,1 9,122,56,0,0,33.3,1.114,33,1 3,170,64,37,225,34.5,0.356,30,1 8,84,74,31,0,38.3,0.457,39,0 2,96,68,13,49,21.1,0.647,26,0 2,125,60,20,140,33.8,0.088,31,0 0,100,70,26,50,30.8,0.597,21,0 0,93,60,25,92,28.7,0.532,22,0 0,129,80,0,0,31.2,0.703,29,0 5,105,72,29,325,36.9,0.159,28,0 3,128,78,0,0,21.1,0.268,55,0 5,106,82,30,0,39.5,0.286,38,0 2,108,52,26,63,32.5,0.318,22,0 10,108,66,0,0,32.4,0.272,42,1 4,154,62,31,284,32.8,0.237,23,0 0,102,75,23,0,0.0,0.572,21,0 9,57,80,37,0,32.8,0.096,41,0 2,106,64,35,119,30.5,1.400,34,0 5,147,78,0,0,33.7,0.218,65,0 2,90,70,17,0,27.3,0.085,22,0 1,136,74,50,204,37.4,0.399,24,0 4,114,65,0,0,21.9,0.432,37,0 9,156,86,28,155,34.3,1.189,42,1 1,153,82,42,485,40.6,0.687,23,0 8,188,78,0,0,47.9,0.137,43,1 7,152,88,44,0,50.0,0.337,36,1 2,99,52,15,94,24.6,0.637,21,0 1,109,56,21,135,25.2,0.833,23,0 2,88,74,19,53,29.0,0.229,22,0 17,163,72,41,114,40.9,0.817,47,1 4,151,90,38,0,29.7,0.294,36,0 7,102,74,40,105,37.2,0.204,45,0 0,114,80,34,285,44.2,0.167,27,0 2,100,64,23,0,29.7,0.368,21,0 0,131,88,0,0,31.6,0.743,32,1 6,104,74,18,156,29.9,0.722,41,1 3,148,66,25,0,32.5,0.256,22,0 4,120,68,0,0,29.6,0.709,34,0 4,110,66,0,0,31.9,0.471,29,0 3,111,90,12,78,28.4,0.495,29,0 6,102,82,0,0,30.8,0.180,36,1 6,134,70,23,130,35.4,0.542,29,1 2,87,0,23,0,28.9,0.773,25,0 1,79,60,42,48,43.5,0.678,23,0 2,75,64,24,55,29.7,0.370,33,0 8,179,72,42,130,32.7,0.719,36,1 6,85,78,0,0,31.2,0.382,42,0 0,129,110,46,130,67.1,0.319,26,1 5,143,78,0,0,45.0,0.190,47,0 5,130,82,0,0,39.1,0.956,37,1 6,87,80,0,0,23.2,0.084,32,0 0,119,64,18,92,34.9,0.725,23,0 1,0,74,20,23,27.7,0.299,21,0 5,73,60,0,0,26.8,0.268,27,0 4,141,74,0,0,27.6,0.244,40,0 7,194,68,28,0,35.9,0.745,41,1 8,181,68,36,495,30.1,0.615,60,1 1,128,98,41,58,32.0,1.321,33,1 8,109,76,39,114,27.9,0.640,31,1 5,139,80,35,160,31.6,0.361,25,1 3,111,62,0,0,22.6,0.142,21,0 9,123,70,44,94,33.1,0.374,40,0 7,159,66,0,0,30.4,0.383,36,1 11,135,0,0,0,52.3,0.578,40,1 8,85,55,20,0,24.4,0.136,42,0 5,158,84,41,210,39.4,0.395,29,1 1,105,58,0,0,24.3,0.187,21,0 3,107,62,13,48,22.9,0.678,23,1 4,109,64,44,99,34.8,0.905,26,1 4,148,60,27,318,30.9,0.150,29,1 0,113,80,16,0,31.0,0.874,21,0 1,138,82,0,0,40.1,0.236,28,0 0,108,68,20,0,27.3,0.787,32,0 2,99,70,16,44,20.4,0.235,27,0 6,103,72,32,190,37.7,0.324,55,0 5,111,72,28,0,23.9,0.407,27,0 8,196,76,29,280,37.5,0.605,57,1 5,162,104,0,0,37.7,0.151,52,1 1,96,64,27,87,33.2,0.289,21,0 7,184,84,33,0,35.5,0.355,41,1 2,81,60,22,0,27.7,0.290,25,0 0,147,85,54,0,42.8,0.375,24,0 7,179,95,31,0,34.2,0.164,60,0 0,140,65,26,130,42.6,0.431,24,1 9,112,82,32,175,34.2,0.260,36,1 12,151,70,40,271,41.8,0.742,38,1 5,109,62,41,129,35.8,0.514,25,1 6,125,68,30,120,30.0,0.464,32,0 5,85,74,22,0,29.0,1.224,32,1 5,112,66,0,0,37.8,0.261,41,1 0,177,60,29,478,34.6,1.072,21,1 2,158,90,0,0,31.6,0.805,66,1 7,119,0,0,0,25.2,0.209,37,0 7,142,60,33,190,28.8,0.687,61,0 1,100,66,15,56,23.6,0.666,26,0 1,87,78,27,32,34.6,0.101,22,0 0,101,76,0,0,35.7,0.198,26,0 3,162,52,38,0,37.2,0.652,24,1 4,197,70,39,744,36.7,2.329,31,0 0,117,80,31,53,45.2,0.089,24,0 4,142,86,0,0,44.0,0.645,22,1 6,134,80,37,370,46.2,0.238,46,1 1,79,80,25,37,25.4,0.583,22,0 4,122,68,0,0,35.0,0.394,29,0 3,74,68,28,45,29.7,0.293,23,0 4,171,72,0,0,43.6,0.479,26,1 7,181,84,21,192,35.9,0.586,51,1 0,179,90,27,0,44.1,0.686,23,1 9,164,84,21,0,30.8,0.831,32,1 0,104,76,0,0,18.4,0.582,27,0 1,91,64,24,0,29.2,0.192,21,0 4,91,70,32,88,33.1,0.446,22,0 3,139,54,0,0,25.6,0.402,22,1 6,119,50,22,176,27.1,1.318,33,1 2,146,76,35,194,38.2,0.329,29,0 9,184,85,15,0,30.0,1.213,49,1 10,122,68,0,0,31.2,0.258,41,0 0,165,90,33,680,52.3,0.427,23,0 9,124,70,33,402,35.4,0.282,34,0 1,111,86,19,0,30.1,0.143,23,0 9,106,52,0,0,31.2,0.380,42,0 2,129,84,0,0,28.0,0.284,27,0 2,90,80,14,55,24.4,0.249,24,0 0,86,68,32,0,35.8,0.238,25,0 12,92,62,7,258,27.6,0.926,44,1 1,113,64,35,0,33.6,0.543,21,1 3,111,56,39,0,30.1,0.557,30,0 2,114,68,22,0,28.7,0.092,25,0 1,193,50,16,375,25.9,0.655,24,0 11,155,76,28,150,33.3,1.353,51,1 3,191,68,15,130,30.9,0.299,34,0 3,141,0,0,0,30.0,0.761,27,1 4,95,70,32,0,32.1,0.612,24,0 3,142,80,15,0,32.4,0.200,63,0 4,123,62,0,0,32.0,0.226,35,1 5,96,74,18,67,33.6,0.997,43,0 0,138,0,0,0,36.3,0.933,25,1 2,128,64,42,0,40.0,1.101,24,0 0,102,52,0,0,25.1,0.078,21,0 2,146,0,0,0,27.5,0.240,28,1 10,101,86,37,0,45.6,1.136,38,1 2,108,62,32,56,25.2,0.128,21,0 3,122,78,0,0,23.0,0.254,40,0 1,71,78,50,45,33.2,0.422,21,0 13,106,70,0,0,34.2,0.251,52,0 2,100,70,52,57,40.5,0.677,25,0 7,106,60,24,0,26.5,0.296,29,1 0,104,64,23,116,27.8,0.454,23,0 5,114,74,0,0,24.9,0.744,57,0 2,108,62,10,278,25.3,0.881,22,0 0,146,70,0,0,37.9,0.334,28,1 10,129,76,28,122,35.9,0.280,39,0 7,133,88,15,155,32.4,0.262,37,0 7,161,86,0,0,30.4,0.165,47,1 2,108,80,0,0,27.0,0.259,52,1 7,136,74,26,135,26.0,0.647,51,0 5,155,84,44,545,38.7,0.619,34,0 1,119,86,39,220,45.6,0.808,29,1 4,96,56,17,49,20.8,0.340,26,0 5,108,72,43,75,36.1,0.263,33,0 0,78,88,29,40,36.9,0.434,21,0 0,107,62,30,74,36.6,0.757,25,1 2,128,78,37,182,43.3,1.224,31,1 1,128,48,45,194,40.5,0.613,24,1 0,161,50,0,0,21.9,0.254,65,0 6,151,62,31,120,35.5,0.692,28,0 2,146,70,38,360,28.0,0.337,29,1 0,126,84,29,215,30.7,0.520,24,0 14,100,78,25,184,36.6,0.412,46,1 8,112,72,0,0,23.6,0.840,58,0 0,167,0,0,0,32.3,0.839,30,1 2,144,58,33,135,31.6,0.422,25,1 5,77,82,41,42,35.8,0.156,35,0 5,115,98,0,0,52.9,0.209,28,1 3,150,76,0,0,21.0,0.207,37,0 2,120,76,37,105,39.7,0.215,29,0 10,161,68,23,132,25.5,0.326,47,1 0,137,68,14,148,24.8,0.143,21,0 0,128,68,19,180,30.5,1.391,25,1 2,124,68,28,205,32.9,0.875,30,1 6,80,66,30,0,26.2,0.313,41,0 0,106,70,37,148,39.4,0.605,22,0 2,155,74,17,96,26.6,0.433,27,1 3,113,50,10,85,29.5,0.626,25,0 7,109,80,31,0,35.9,1.127,43,1 2,112,68,22,94,34.1,0.315,26,0 3,99,80,11,64,19.3,0.284,30,0 3,182,74,0,0,30.5,0.345,29,1 3,115,66,39,140,38.1,0.150,28,0 6,194,78,0,0,23.5,0.129,59,1 4,129,60,12,231,27.5,0.527,31,0 3,112,74,30,0,31.6,0.197,25,1 0,124,70,20,0,27.4,0.254,36,1 13,152,90,33,29,26.8,0.731,43,1 2,112,75,32,0,35.7,0.148,21,0 1,157,72,21,168,25.6,0.123,24,0 1,122,64,32,156,35.1,0.692,30,1 10,179,70,0,0,35.1,0.200,37,0 2,102,86,36,120,45.5,0.127,23,1 6,105,70,32,68,30.8,0.122,37,0 8,118,72,19,0,23.1,1.476,46,0 2,87,58,16,52,32.7,0.166,25,0 1,180,0,0,0,43.3,0.282,41,1 12,106,80,0,0,23.6,0.137,44,0 1,95,60,18,58,23.9,0.260,22,0 0,165,76,43,255,47.9,0.259,26,0 0,117,0,0,0,33.8,0.932,44,0 5,115,76,0,0,31.2,0.343,44,1 9,152,78,34,171,34.2,0.893,33,1 7,178,84,0,0,39.9,0.331,41,1 1,130,70,13,105,25.9,0.472,22,0 1,95,74,21,73,25.9,0.673,36,0 1,0,68,35,0,32.0,0.389,22,0 5,122,86,0,0,34.7,0.290,33,0 8,95,72,0,0,36.8,0.485,57,0 8,126,88,36,108,38.5,0.349,49,0 1,139,46,19,83,28.7,0.654,22,0 3,116,0,0,0,23.5,0.187,23,0 3,99,62,19,74,21.8,0.279,26,0 5,0,80,32,0,41.0,0.346,37,1 4,92,80,0,0,42.2,0.237,29,0 4,137,84,0,0,31.2,0.252,30,0 3,61,82,28,0,34.4,0.243,46,0 1,90,62,12,43,27.2,0.580,24,0 3,90,78,0,0,42.7,0.559,21,0 9,165,88,0,0,30.4,0.302,49,1 1,125,50,40,167,33.3,0.962,28,1 13,129,0,30,0,39.9,0.569,44,1 12,88,74,40,54,35.3,0.378,48,0 1,196,76,36,249,36.5,0.875,29,1 5,189,64,33,325,31.2,0.583,29,1 5,158,70,0,0,29.8,0.207,63,0 5,103,108,37,0,39.2,0.305,65,0 4,146,78,0,0,38.5,0.520,67,1 4,147,74,25,293,34.9,0.385,30,0 5,99,54,28,83,34.0,0.499,30,0 6,124,72,0,0,27.6,0.368,29,1 0,101,64,17,0,21.0,0.252,21,0 3,81,86,16,66,27.5,0.306,22,0 1,133,102,28,140,32.8,0.234,45,1 3,173,82,48,465,38.4,2.137,25,1 0,118,64,23,89,0.0,1.731,21,0 0,84,64,22,66,35.8,0.545,21,0 2,105,58,40,94,34.9,0.225,25,0 2,122,52,43,158,36.2,0.816,28,0 12,140,82,43,325,39.2,0.528,58,1 0,98,82,15,84,25.2,0.299,22,0 1,87,60,37,75,37.2,0.509,22,0 4,156,75,0,0,48.3,0.238,32,1 0,93,100,39,72,43.4,1.021,35,0 1,107,72,30,82,30.8,0.821,24,0 0,105,68,22,0,20.0,0.236,22,0 1,109,60,8,182,25.4,0.947,21,0 1,90,62,18,59,25.1,1.268,25,0 1,125,70,24,110,24.3,0.221,25,0 1,119,54,13,50,22.3,0.205,24,0 5,116,74,29,0,32.3,0.660,35,1 8,105,100,36,0,43.3,0.239,45,1 5,144,82,26,285,32.0,0.452,58,1 3,100,68,23,81,31.6,0.949,28,0 1,100,66,29,196,32.0,0.444,42,0 5,166,76,0,0,45.7,0.340,27,1 1,131,64,14,415,23.7,0.389,21,0 4,116,72,12,87,22.1,0.463,37,0 4,158,78,0,0,32.9,0.803,31,1 2,127,58,24,275,27.7,1.600,25,0 3,96,56,34,115,24.7,0.944,39,0 0,131,66,40,0,34.3,0.196,22,1 3,82,70,0,0,21.1,0.389,25,0 3,193,70,31,0,34.9,0.241,25,1 4,95,64,0,0,32.0,0.161,31,1 6,137,61,0,0,24.2,0.151,55,0 5,136,84,41,88,35.0,0.286,35,1 9,72,78,25,0,31.6,0.280,38,0 5,168,64,0,0,32.9,0.135,41,1 2,123,48,32,165,42.1,0.520,26,0 4,115,72,0,0,28.9,0.376,46,1 0,101,62,0,0,21.9,0.336,25,0 8,197,74,0,0,25.9,1.191,39,1 1,172,68,49,579,42.4,0.702,28,1 6,102,90,39,0,35.7,0.674,28,0 1,112,72,30,176,34.4,0.528,25,0 1,143,84,23,310,42.4,1.076,22,0 1,143,74,22,61,26.2,0.256,21,0 0,138,60,35,167,34.6,0.534,21,1 3,173,84,33,474,35.7,0.258,22,1 1,97,68,21,0,27.2,1.095,22,0 4,144,82,32,0,38.5,0.554,37,1 1,83,68,0,0,18.2,0.624,27,0 3,129,64,29,115,26.4,0.219,28,1 1,119,88,41,170,45.3,0.507,26,0 2,94,68,18,76,26.0,0.561,21,0 0,102,64,46,78,40.6,0.496,21,0 2,115,64,22,0,30.8,0.421,21,0 8,151,78,32,210,42.9,0.516,36,1 4,184,78,39,277,37.0,0.264,31,1 0,94,0,0,0,0.0,0.256,25,0 1,181,64,30,180,34.1,0.328,38,1 0,135,94,46,145,40.6,0.284,26,0 1,95,82,25,180,35.0,0.233,43,1 2,99,0,0,0,22.2,0.108,23,0 3,89,74,16,85,30.4,0.551,38,0 1,80,74,11,60,30.0,0.527,22,0 2,139,75,0,0,25.6,0.167,29,0 1,90,68,8,0,24.5,1.138,36,0 0,141,0,0,0,42.4,0.205,29,1 12,140,85,33,0,37.4,0.244,41,0 5,147,75,0,0,29.9,0.434,28,0 1,97,70,15,0,18.2,0.147,21,0 6,107,88,0,0,36.8,0.727,31,0 0,189,104,25,0,34.3,0.435,41,1 2,83,66,23,50,32.2,0.497,22,0 4,117,64,27,120,33.2,0.230,24,0 8,108,70,0,0,30.5,0.955,33,1 4,117,62,12,0,29.7,0.380,30,1 0,180,78,63,14,59.4,2.420,25,1 1,100,72,12,70,25.3,0.658,28,0 0,95,80,45,92,36.5,0.330,26,0 0,104,64,37,64,33.6,0.510,22,1 0,120,74,18,63,30.5,0.285,26,0 1,82,64,13,95,21.2,0.415,23,0 2,134,70,0,0,28.9,0.542,23,1 0,91,68,32,210,39.9,0.381,25,0 2,119,0,0,0,19.6,0.832,72,0 2,100,54,28,105,37.8,0.498,24,0 14,175,62,30,0,33.6,0.212,38,1 1,135,54,0,0,26.7,0.687,62,0 5,86,68,28,71,30.2,0.364,24,0 10,148,84,48,237,37.6,1.001,51,1 9,134,74,33,60,25.9,0.460,81,0 9,120,72,22,56,20.8,0.733,48,0 1,71,62,0,0,21.8,0.416,26,0 8,74,70,40,49,35.3,0.705,39,0 5,88,78,30,0,27.6,0.258,37,0 10,115,98,0,0,24.0,1.022,34,0 0,124,56,13,105,21.8,0.452,21,0 0,74,52,10,36,27.8,0.269,22,0 0,97,64,36,100,36.8,0.600,25,0 8,120,0,0,0,30.0,0.183,38,1 6,154,78,41,140,46.1,0.571,27,0 1,144,82,40,0,41.3,0.607,28,0 0,137,70,38,0,33.2,0.170,22,0 0,119,66,27,0,38.8,0.259,22,0 7,136,90,0,0,29.9,0.210,50,0 4,114,64,0,0,28.9,0.126,24,0 0,137,84,27,0,27.3,0.231,59,0 2,105,80,45,191,33.7,0.711,29,1 7,114,76,17,110,23.8,0.466,31,0 8,126,74,38,75,25.9,0.162,39,0 4,132,86,31,0,28.0,0.419,63,0 3,158,70,30,328,35.5,0.344,35,1 0,123,88,37,0,35.2,0.197,29,0 4,85,58,22,49,27.8,0.306,28,0 0,84,82,31,125,38.2,0.233,23,0 0,145,0,0,0,44.2,0.630,31,1 0,135,68,42,250,42.3,0.365,24,1 1,139,62,41,480,40.7,0.536,21,0 0,173,78,32,265,46.5,1.159,58,0 4,99,72,17,0,25.6,0.294,28,0 8,194,80,0,0,26.1,0.551,67,0 2,83,65,28,66,36.8,0.629,24,0 2,89,90,30,0,33.5,0.292,42,0 4,99,68,38,0,32.8,0.145,33,0 4,125,70,18,122,28.9,1.144,45,1 3,80,0,0,0,0.0,0.174,22,0 6,166,74,0,0,26.6,0.304,66,0 5,110,68,0,0,26.0,0.292,30,0 2,81,72,15,76,30.1,0.547,25,0 7,195,70,33,145,25.1,0.163,55,1 6,154,74,32,193,29.3,0.839,39,0 2,117,90,19,71,25.2,0.313,21,0 3,84,72,32,0,37.2,0.267,28,0 6,0,68,41,0,39.0,0.727,41,1 7,94,64,25,79,33.3,0.738,41,0 3,96,78,39,0,37.3,0.238,40,0 10,75,82,0,0,33.3,0.263,38,0 0,180,90,26,90,36.5,0.314,35,1 1,130,60,23,170,28.6,0.692,21,0 2,84,50,23,76,30.4,0.968,21,0 8,120,78,0,0,25.0,0.409,64,0 12,84,72,31,0,29.7,0.297,46,1 0,139,62,17,210,22.1,0.207,21,0 9,91,68,0,0,24.2,0.200,58,0 2,91,62,0,0,27.3,0.525,22,0 3,99,54,19,86,25.6,0.154,24,0 3,163,70,18,105,31.6,0.268,28,1 9,145,88,34,165,30.3,0.771,53,1 7,125,86,0,0,37.6,0.304,51,0 13,76,60,0,0,32.8,0.180,41,0 6,129,90,7,326,19.6,0.582,60,0 2,68,70,32,66,25.0,0.187,25,0 3,124,80,33,130,33.2,0.305,26,0 6,114,0,0,0,0.0,0.189,26,0 9,130,70,0,0,34.2,0.652,45,1 3,125,58,0,0,31.6,0.151,24,0 3,87,60,18,0,21.8,0.444,21,0 1,97,64,19,82,18.2,0.299,21,0 3,116,74,15,105,26.3,0.107,24,0 0,117,66,31,188,30.8,0.493,22,0 0,111,65,0,0,24.6,0.660,31,0 2,122,60,18,106,29.8,0.717,22,0 0,107,76,0,0,45.3,0.686,24,0 1,86,66,52,65,41.3,0.917,29,0 6,91,0,0,0,29.8,0.501,31,0 1,77,56,30,56,33.3,1.251,24,0 4,132,0,0,0,32.9,0.302,23,1 0,105,90,0,0,29.6,0.197,46,0 0,57,60,0,0,21.7,0.735,67,0 0,127,80,37,210,36.3,0.804,23,0 3,129,92,49,155,36.4,0.968,32,1 8,100,74,40,215,39.4,0.661,43,1 3,128,72,25,190,32.4,0.549,27,1 10,90,85,32,0,34.9,0.825,56,1 4,84,90,23,56,39.5,0.159,25,0 1,88,78,29,76,32.0,0.365,29,0 8,186,90,35,225,34.5,0.423,37,1 5,187,76,27,207,43.6,1.034,53,1 4,131,68,21,166,33.1,0.160,28,0 1,164,82,43,67,32.8,0.341,50,0 4,189,110,31,0,28.5,0.680,37,0 1,116,70,28,0,27.4,0.204,21,0 3,84,68,30,106,31.9,0.591,25,0 6,114,88,0,0,27.8,0.247,66,0 1,88,62,24,44,29.9,0.422,23,0 1,84,64,23,115,36.9,0.471,28,0 7,124,70,33,215,25.5,0.161,37,0 1,97,70,40,0,38.1,0.218,30,0 8,110,76,0,0,27.8,0.237,58,0 11,103,68,40,0,46.2,0.126,42,0 11,85,74,0,0,30.1,0.300,35,0 6,125,76,0,0,33.8,0.121,54,1 0,198,66,32,274,41.3,0.502,28,1 1,87,68,34,77,37.6,0.401,24,0 6,99,60,19,54,26.9,0.497,32,0 0,91,80,0,0,32.4,0.601,27,0 2,95,54,14,88,26.1,0.748,22,0 1,99,72,30,18,38.6,0.412,21,0 6,92,62,32,126,32.0,0.085,46,0 4,154,72,29,126,31.3,0.338,37,0 0,121,66,30,165,34.3,0.203,33,1 3,78,70,0,0,32.5,0.270,39,0 2,130,96,0,0,22.6,0.268,21,0 3,111,58,31,44,29.5,0.430,22,0 2,98,60,17,120,34.7,0.198,22,0 1,143,86,30,330,30.1,0.892,23,0 1,119,44,47,63,35.5,0.280,25,0 6,108,44,20,130,24.0,0.813,35,0 2,118,80,0,0,42.9,0.693,21,1 10,133,68,0,0,27.0,0.245,36,0 2,197,70,99,0,34.7,0.575,62,1 0,151,90,46,0,42.1,0.371,21,1 6,109,60,27,0,25.0,0.206,27,0 12,121,78,17,0,26.5,0.259,62,0 8,100,76,0,0,38.7,0.190,42,0 8,124,76,24,600,28.7,0.687,52,1 1,93,56,11,0,22.5,0.417,22,0 8,143,66,0,0,34.9,0.129,41,1 6,103,66,0,0,24.3,0.249,29,0 3,176,86,27,156,33.3,1.154,52,1 0,73,0,0,0,21.1,0.342,25,0 11,111,84,40,0,46.8,0.925,45,1 2,112,78,50,140,39.4,0.175,24,0 3,132,80,0,0,34.4,0.402,44,1 2,82,52,22,115,28.5,1.699,25,0 6,123,72,45,230,33.6,0.733,34,0 0,188,82,14,185,32.0,0.682,22,1 0,67,76,0,0,45.3,0.194,46,0 1,89,24,19,25,27.8,0.559,21,0 1,173,74,0,0,36.8,0.088,38,1 1,109,38,18,120,23.1,0.407,26,0 1,108,88,19,0,27.1,0.400,24,0 6,96,0,0,0,23.7,0.190,28,0 1,124,74,36,0,27.8,0.100,30,0 7,150,78,29,126,35.2,0.692,54,1 4,183,0,0,0,28.4,0.212,36,1 1,124,60,32,0,35.8,0.514,21,0 1,181,78,42,293,40.0,1.258,22,1 1,92,62,25,41,19.5,0.482,25,0 0,152,82,39,272,41.5,0.270,27,0 1,111,62,13,182,24.0,0.138,23,0 3,106,54,21,158,30.9,0.292,24,0 3,174,58,22,194,32.9,0.593,36,1 7,168,88,42,321,38.2,0.787,40,1 6,105,80,28,0,32.5,0.878,26,0 11,138,74,26,144,36.1,0.557,50,1 3,106,72,0,0,25.8,0.207,27,0 6,117,96,0,0,28.7,0.157,30,0 2,68,62,13,15,20.1,0.257,23,0 9,112,82,24,0,28.2,1.282,50,1 0,119,0,0,0,32.4,0.141,24,1 2,112,86,42,160,38.4,0.246,28,0 2,92,76,20,0,24.2,1.698,28,0 6,183,94,0,0,40.8,1.461,45,0 0,94,70,27,115,43.5,0.347,21,0 2,108,64,0,0,30.8,0.158,21,0 4,90,88,47,54,37.7,0.362,29,0 0,125,68,0,0,24.7,0.206,21,0 0,132,78,0,0,32.4,0.393,21,0 5,128,80,0,0,34.6,0.144,45,0 4,94,65,22,0,24.7,0.148,21,0 7,114,64,0,0,27.4,0.732,34,1 0,102,78,40,90,34.5,0.238,24,0 2,111,60,0,0,26.2,0.343,23,0 1,128,82,17,183,27.5,0.115,22,0 10,92,62,0,0,25.9,0.167,31,0 13,104,72,0,0,31.2,0.465,38,1 5,104,74,0,0,28.8,0.153,48,0 2,94,76,18,66,31.6,0.649,23,0 7,97,76,32,91,40.9,0.871,32,1 1,100,74,12,46,19.5,0.149,28,0 0,102,86,17,105,29.3,0.695,27,0 4,128,70,0,0,34.3,0.303,24,0 6,147,80,0,0,29.5,0.178,50,1 4,90,0,0,0,28.0,0.610,31,0 3,103,72,30,152,27.6,0.730,27,0 2,157,74,35,440,39.4,0.134,30,0 1,167,74,17,144,23.4,0.447,33,1 0,179,50,36,159,37.8,0.455,22,1 11,136,84,35,130,28.3,0.260,42,1 0,107,60,25,0,26.4,0.133,23,0 1,91,54,25,100,25.2,0.234,23,0 1,117,60,23,106,33.8,0.466,27,0 5,123,74,40,77,34.1,0.269,28,0 2,120,54,0,0,26.8,0.455,27,0 1,106,70,28,135,34.2,0.142,22,0 2,155,52,27,540,38.7,0.240,25,1 2,101,58,35,90,21.8,0.155,22,0 1,120,80,48,200,38.9,1.162,41,0 11,127,106,0,0,39.0,0.190,51,0 3,80,82,31,70,34.2,1.292,27,1 10,162,84,0,0,27.7,0.182,54,0 1,199,76,43,0,42.9,1.394,22,1 8,167,106,46,231,37.6,0.165,43,1 9,145,80,46,130,37.9,0.637,40,1 6,115,60,39,0,33.7,0.245,40,1 1,112,80,45,132,34.8,0.217,24,0 4,145,82,18,0,32.5,0.235,70,1 10,111,70,27,0,27.5,0.141,40,1 6,98,58,33,190,34.0,0.430,43,0 9,154,78,30,100,30.9,0.164,45,0 6,165,68,26,168,33.6,0.631,49,0 1,99,58,10,0,25.4,0.551,21,0 10,68,106,23,49,35.5,0.285,47,0 3,123,100,35,240,57.3,0.880,22,0 8,91,82,0,0,35.6,0.587,68,0 6,195,70,0,0,30.9,0.328,31,1 9,156,86,0,0,24.8,0.230,53,1 0,93,60,0,0,35.3,0.263,25,0 3,121,52,0,0,36.0,0.127,25,1 2,101,58,17,265,24.2,0.614,23,0 2,56,56,28,45,24.2,0.332,22,0 0,162,76,36,0,49.6,0.364,26,1 0,95,64,39,105,44.6,0.366,22,0 4,125,80,0,0,32.3,0.536,27,1 5,136,82,0,0,0.0,0.640,69,0 2,129,74,26,205,33.2,0.591,25,0 3,130,64,0,0,23.1,0.314,22,0 1,107,50,19,0,28.3,0.181,29,0 1,140,74,26,180,24.1,0.828,23,0 1,144,82,46,180,46.1,0.335,46,1 8,107,80,0,0,24.6,0.856,34,0 13,158,114,0,0,42.3,0.257,44,1 2,121,70,32,95,39.1,0.886,23,0 7,129,68,49,125,38.5,0.439,43,1 2,90,60,0,0,23.5,0.191,25,0 7,142,90,24,480,30.4,0.128,43,1 3,169,74,19,125,29.9,0.268,31,1 0,99,0,0,0,25.0,0.253,22,0 4,127,88,11,155,34.5,0.598,28,0 4,118,70,0,0,44.5,0.904,26,0 2,122,76,27,200,35.9,0.483,26,0 6,125,78,31,0,27.6,0.565,49,1 1,168,88,29,0,35.0,0.905,52,1 2,129,0,0,0,38.5,0.304,41,0 4,110,76,20,100,28.4,0.118,27,0 6,80,80,36,0,39.8,0.177,28,0 10,115,0,0,0,0.0,0.261,30,1 2,127,46,21,335,34.4,0.176,22,0 9,164,78,0,0,32.8,0.148,45,1 2,93,64,32,160,38.0,0.674,23,1 3,158,64,13,387,31.2,0.295,24,0 5,126,78,27,22,29.6,0.439,40,0 10,129,62,36,0,41.2,0.441,38,1 0,134,58,20,291,26.4,0.352,21,0 3,102,74,0,0,29.5,0.121,32,0 7,187,50,33,392,33.9,0.826,34,1 3,173,78,39,185,33.8,0.970,31,1 10,94,72,18,0,23.1,0.595,56,0 1,108,60,46,178,35.5,0.415,24,0 5,97,76,27,0,35.6,0.378,52,1 4,83,86,19,0,29.3,0.317,34,0 1,114,66,36,200,38.1,0.289,21,0 1,149,68,29,127,29.3,0.349,42,1 5,117,86,30,105,39.1,0.251,42,0 1,111,94,0,0,32.8,0.265,45,0 4,112,78,40,0,39.4,0.236,38,0 1,116,78,29,180,36.1,0.496,25,0 0,141,84,26,0,32.4,0.433,22,0 2,175,88,0,0,22.9,0.326,22,0 2,92,52,0,0,30.1,0.141,22,0 3,130,78,23,79,28.4,0.323,34,1 8,120,86,0,0,28.4,0.259,22,1 2,174,88,37,120,44.5,0.646,24,1 2,106,56,27,165,29.0,0.426,22,0 2,105,75,0,0,23.3,0.560,53,0 4,95,60,32,0,35.4,0.284,28,0 0,126,86,27,120,27.4,0.515,21,0 8,65,72,23,0,32.0,0.600,42,0 2,99,60,17,160,36.6,0.453,21,0 1,102,74,0,0,39.5,0.293,42,1 11,120,80,37,150,42.3,0.785,48,1 3,102,44,20,94,30.8,0.400,26,0 1,109,58,18,116,28.5,0.219,22,0 9,140,94,0,0,32.7,0.734,45,1 13,153,88,37,140,40.6,1.174,39,0 12,100,84,33,105,30.0,0.488,46,0 1,147,94,41,0,49.3,0.358,27,1 1,81,74,41,57,46.3,1.096,32,0 3,187,70,22,200,36.4,0.408,36,1 6,162,62,0,0,24.3,0.178,50,1 4,136,70,0,0,31.2,1.182,22,1 1,121,78,39,74,39.0,0.261,28,0 3,108,62,24,0,26.0,0.223,25,0 0,181,88,44,510,43.3,0.222,26,1 8,154,78,32,0,32.4,0.443,45,1 1,128,88,39,110,36.5,1.057,37,1 7,137,90,41,0,32.0,0.391,39,0 0,123,72,0,0,36.3,0.258,52,1 1,106,76,0,0,37.5,0.197,26,0 6,190,92,0,0,35.5,0.278,66,1 2,88,58,26,16,28.4,0.766,22,0 9,170,74,31,0,44.0,0.403,43,1 9,89,62,0,0,22.5,0.142,33,0 10,101,76,48,180,32.9,0.171,63,0 2,122,70,27,0,36.8,0.340,27,0 5,121,72,23,112,26.2,0.245,30,0 1,126,60,0,0,30.1,0.349,47,1 1,93,70,31,0,30.4,0.315,23,0