Termux pkg update error on Android devices

If you have pkg update error on your old installation of Android devices

pkg install termux-tools
termux-change-repo

Personal Installation Guide for new Chromebook

(1) For new chromebook, there is free Google One 100GB and DropBox 100GB both for 1 year. See here. https://www.google.com/chromebook/perks/

(2) To enable google drive on Files app of Chrome OS, Bottom right clock -> Settings Icon -> Advanced -> Files -> Disconnect Google Drive account set to off.

(3) To enable DropBox on Files app of Chrome OS, Install DropBox app from Android Play Store and login dropbox.

(4) To enable smb share on Files app of Chrome OS, Open Files App and select the 3 dots menu on top right and Choose "Service" and then "SMB File Share".

(5) To enable OneDrive (readonly) on Files app of Chrome OS, Install OneDrive App from Google Play Store.

(5.1) How to share files or Google Drive with Linux apps on Chrome OS

    Select all

(a) Open the My Files app. (b) From within that app, locate the directory housing the documents you want accessible by the installed Linux apps. (c) Right-click the folder in question and then select Share With Linux. The sharing happens immediately. (d) Google Drive is in /mnt/chromeos/GoogleDrive/MyDrive/ (e) MyFiles is in /mnt/chromeos/MyFiles/

(6) If you cannot install Chrome OS Linux Beta Crostini now, install Termux App as a temporary solution, Download the apk file from here https://f-droid.org/en/packages/com.termux/ (Google Play Version was outdated).

(7) How to install Linux Web Development Environment on Chrome OS, see here https://www.youtube.com/watch?v=3CWUAisN-vo

shellscript.sh    Select all

# nodejs dependencies sudo apt install gnupg2 # install nodejs v14 and npm v6 curl -sL https://deb.nodesource.com/setup_14.x | sudo bash - sudo apt-get install -y nodejs # try it out! Sweep for mines sudo apt install libnss3 git clone https://github.com/GoogleChromeLabs/proxx.git cd proxx npm install npm run build npm run serve # Install React npm install create-react-app # Create a React app npx create-react-app hello-world-react cd hello-world-react npm start # Install Angular npm install @angular/cli # Create an Angular app ng new my-angular-project cd my-angular-project ng serve # Docker Dependencies sudo apt install ca-certificates software-properties-common # Docker repository curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" # Install Docker-ce sudo apt update sudo apt install docker-ce docker-ce-cli containerd.io # Install docker client only and connect to docker desktop of windows machine cd $HOME wget https://download.docker.com/linux/static/stable/x86_64/docker-20.10.7.tgz tar xzvf docker-20.10.7.tgz ./docker/docker -H tcp://10.0.1.78:2375 images # Or simply add the corresponding variables to ~/.bashrc export DOCKER_HOST=tcp://192.168.1.78:2375 export PATH=$PATH:~/docker # # To detach the docker tty without exiting the shell, use the escape sequence Ctrl-p + Ctrl-q + Ctrl-c # # for docker desktop on mac, see this solution to open tcp port https://stackoverflow.com/questions/39411126/access-docker-daemon-remote-api-on-docker-for-mac # on Mac machine brew install socat socat TCP-LISTEN:2375,reuseaddr,fork UNIX-CONNECT:/var/run/docker.sock &

(8) System requirement for Android Studio for Chrome OS is 8GB RAM and Intel i5 CPU or above.

(9) Flatpak applications can be installed on Chrome OS with the Crostini Linux compatibility layer. See here https://flatpak.org/setup/Chrome%20OS/

(10) Popular shortcuts https://support.google.com/chromebook/answer/183101?hl=en

(11) How to access onedrive in linux partition

shellscript.sh    Select all

# download latest deb package from https://launchpad.net/~jstaf/+archive/ubuntu/onedriver/+packages wget https://launchpad.net/~jstaf/+archive/ubuntu/onedriver/+files/onedriver_0.11.1-1_amd64.deb # install the deb package in linux sudo apt install ./onedriver_0.11.1-1_amd64.deb # reference from https://github.com/jstaf/onedriver # create mount point mkdir -p ~/onedrive # determine the service name export SERVICE_NAME=$(systemd-escape --template onedriver@.service --path ~/onedrive) # mount onedrive systemctl --user daemon-reload systemctl --user start $SERVICE_NAME # automatically mount onedrive when you login systemctl --user enable $SERVICE_NAME # query the status of onedriver systemctl --user status $SERVICE_NAME # check onedriver's logs for the current day journalctl --user -u $SERVICE_NAME --since today

P.S. if you cannot install Chrome OS Linux Beta Crostini, please refer to the reason in this article https://chromeunboxed.com/google-pauses-chrome-os-update-breaks-linux-container/

How to install Windows 11 WSL2 GUI App

(1) If you have unspported hardware, please follow this guide to install Dev Channel Preview of Windows 11 -> update-any-pc-to-windows11

(2) After installed Windows 11 Preview, Start Windows Terminal (install from Microsoft Store) in Administrative Role. The following commands are entered using Windows Terminal with elevated admin rights.

(3) wslconfig /u ubuntu # if wsl already installed previously in Windows 10 and want to remove old ubuntu distro

(4.1) wsl --update # if wsl already installed previously in Windows 10 and needs update to WSLg

(4.2) wsl --shutdown # wsl needs to be restarted after update

(5) wsl --install -d Ubuntu # this will install WSLg on Windows 11 and the new Ubuntu 20.04 instance

(6) sudo apt install audacity # this will install audacity linux gui app for ubuntu

(7) Use Windows search function to search for Audacity (Ubuntu) App to start the linux GUI App under Windows 11.

(8) You can even install and run Chrome, Edge, or Teams under Ubuntu on Windows 11 with WSLg.

(9) To install flatpak for GUI App

Shellscript   Select all

sudo apt install flatpak # add repo flatpak --user remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo # search app and install flatpak search gimp flatpak install flathub org.gimp.GIMP

Hello swift async and await for Swift 5.5

(1) This is to test the new async and await feature for Swift 5.5 dev on Linux. First Download and install packages in Ubuntu 20.04.

Shell script for setup Swift 5.5   Select all

sudo apt-get install -y binutils git gnupg2 libc6-dev libcurl4 libedit2 libgcc-9-dev libpython2.7 libsqlite3-0 libstdc++-9-dev libxml2 libz3-dev pkg-config tzdata zlib1g-dev cd ${HOME} wget https://swift.org/builds/swift-5.5-branch/ubuntu2004/swift-5.5-DEVELOPMENT-SNAPSHOT-2021-06-02-a/swift-5.5-DEVELOPMENT-SNAPSHOT-2021-06-02-a-ubuntu20.04.tar.gz tar xzvf swift-5.5-DEVELOPMENT-SNAPSHOT-2021-06-02-a-ubuntu20.04.tar.gz export PATH=${HOME}/swift-5.5-DEVELOPMENT-SNAPSHOT-2021-06-02-a-ubuntu20.04/usr/bin:$PATH

(2) Create swift package for executable

Shell script testing async and await   Select all

cd ${HOME} mkdir -p ${HOME}/AsyncSwift cd ${HOME}/AsyncSwift swift package init --type executable

(3) Amend ${HOME}/AsyncSwift/Package.swift to have swift setting flags

Package.swift   Select all

// swift-tools-version:5.5 // The swift-tools-version declares the minimum version of Swift required to build this package. import PackageDescription let package = Package( name: "AsyncSwift", products: [ .executable(name: "AsyncSwift", targets: ["AsyncSwift"]) ], dependencies: [ // none for now.. ], targets: [ .executableTarget(name: "AsyncSwift", swiftSettings: [ .unsafeFlags([ "-parse-as-library", "-Xfrontend", "-disable-availability-checking", "-Xfrontend", "-enable-experimental-concurrency", ]) ] ), .testTarget(name: "AsyncSwiftTests", dependencies: ["AsyncSwift"]), ] )

(4) Edit ${HOME}/AsyncSwift/Sources/AsyncSwift/main.swift

Sources/AsyncSwift/main.swift   Select all

import Foundation func calculateFirstNumber() async -> Int { print("First number is now being calculated...") return await withUnsafeContinuation { c in DispatchQueue.main.asyncAfter(deadline: .now() + 2) { print("First number is now ready.") c.resume(returning: 42) } } } func calculateSecondNumber() async -> Int { print("Second number is now being calculated...") return await withUnsafeContinuation { c in DispatchQueue.main.asyncAfter(deadline: .now() + 1) { print("Second number is now ready.") c.resume(returning: 6) } } } func calculateThirdNumber() async -> Int { print("Third number is now being calculated...") return await withUnsafeContinuation { c in DispatchQueue.main.asyncAfter(deadline: .now() + 3) { print("Third number is now ready.") c.resume(returning: 69) } } } func trace(task: Int) async { // Generates a random integer in the [0, task] range print("Task \(task) started") sleep( UInt32.random(in: 0...UInt32(task)) ) print("Task \(task) completed") } @main struct MyProgram { static func main() async { print("Hello, swift async!\n") print("\nSerial queue, asynchronous execution\n") for i in 5...10 { print("Submitting task \(i)") async { await trace(task: i) } } async let x = calculateFirstNumber() let y = await calculateSecondNumber() let z = await calculateThirdNumber() await print(x + y + z) sleep(2) await print("Program ended") } } /* [6/6] Build complete! Hello, swift async! First number is now being calculated... First number is now ready. Second number is now being calculated... Second number is now ready. Third number is now being calculated... Third number is now ready. 117 */

(5) build and run executable

Shell script for building and running   Select all

cd ${HOME}/AsyncSwift swift package clean swift build swift run

How to compile Quantlib-Python for Raspberry Pi 4B arm32 and arm64

Raspberry Pi has default gcc-8 and Python 3.7 for its 32 bit / 64 bit buster image. And compiling QuantLib-Python on this machine could have out of memeory error. Cross compiling on docker might have different python version which is not compatible. The trick to compile on Raspberry Pi is to setup swap say 2G and 4G Ram and turn off debug -g flag when compiling as Python package.

Shell script for building arm32 version   Select all

# install necessary packages for building sudo apt update sudo apt install -y build-essential wget libbz2-dev libboost-test1.67.0 libboost-test-dev # Get QuantLib-1.22 and build static library cd ${HOME} wget https://github.com/lballabio/QuantLib/releases/download/QuantLib-v1.22/QuantLib-1.22.tar.gz tar xzf QuantLib-1.22.tar.gz cd QuantLib-1.22/ ./configure --prefix=/usr --disable-shared CXXFLAGS=-O3 make -j 4 && make install sudo ldconfig # Setup and enable swap and check it for at least 2GB. sudo dphys-swapfile setup sudo dphys-swapfile swapon free -mh sudo apt install -y python3 python3-pip python-dev libgomp1 # Get QuantLib-SWIG-1.22 and compile it cd ${HOME} wget --no-check-certificate https://github.com/lballabio/QuantLib-SWIG/releases/download/QuantLib-SWIG-v1.22/QuantLib-SWIG-${quantlib_swig_version}.tar.gz tar xfz QuantLib-SWIG-1.22.tar.gz cd QuantLib-SWIG-1.22/ ./configure CXXFLAGS="-O2 --param ggc-min-expand=1 --param ggc-min-heapsize=32768 -Wno-deprecated-declarations -Wno-misleading-indentation" PYTHON=/usr/bin/python3 # manual compile it and remove the -g flag cd Python/ mkdir -p build/temp.linux-armv7l-3.7/QuantLib export CXX="echo gcc"; python3 setup.py bdist_wheel g++ -fwrapv -O2 -Wall -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DNDEBUG -I/usr/include/python3.7m -I/usr/include -c QuantLib/quantlib_wrap.cpp -o build/temp.linux-armv7l-3.7/QuantLib/quantlib_wrap.o -Wno-unused --param ggc-min-expand=1 --param ggc-min-heapsize=32768 -Wno-deprecated-declarations -Wno-misleading-indentation mkdir -p build/lib.linux-armv7l-3.7/QuantLib/ g++ -shared -Wl,-z,relro -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-armv7l-3.7/QuantLib/quantlib_wrap.o -lQuantLib -o build/lib.linux-armv7l-3.7/QuantLib/_QuantLib.cpython-37m-arm-linux-gnueabihf.so # create wheel file python3 setup.py bdist_wheel # Upgrade PIP and install the wheel file /usr/bin/python3 -m pip install --upgrade pip pip3 install dist/QuantLib-1.22-cp37-cp37m-linux_armv7l.whl # Or alternatively install as site-package sudo python3 setup.py install # Test examples after installation pip3 install pandas python3 examples/bonds.py . . . .

Compiling for Rapberry Pi arm64 is very similar but has to add -fPIC flag for the QuantLib when building static library

Shell script for building arm64 version   Select all

# install necessary packages for building sudo apt update sudo apt install -y build-essential wget libbz2-dev sudo apt install -y libboost-test1.67.0 libboost-test-dev cd ${HOME} wget https://github.com/lballabio/QuantLib/releases/download/1.22/QuantLib-1.22.tar.gz tar xzf QuantLib-1.22.tar.gz cd QuantLib-1.22/ # enable -fPIC flag for building static library ./configure --prefix=/usr --disable-shared CXXFLAGS="-O3 -fPIC" make -j 4 && make install sudo ldconfig # If Raspbeery Pi has 8GB Ram, no need to setup and enable swap sudo apt install -y python3 python3-pip python-dev libgomp1 # Get QuantLib-SWIG-1.22 and compile it cd {HOME} wget https://github.com/lballabio/QuantLib-SWIG/releases/download/QuantLib-SWIG-v1.22/QuantLib-SWIG-1.22.tar.gz tar xzf QuantLib-SWIG-1.22.tar.gz cd QuantLib-SWIG-1.22/ cd Python/ ./configure CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768 -fPIC -Wno-deprecated-declarations -Wno-misleading-indentation" PYTHON=/usr/bin/python3 # manual compile it and remove the -g flag cd Python/ mkdir -p build/temp.linux-aarch64-3.7/QuantLib/ g++ -fwrapv -O2 -Wall -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.7m -I/usr/include -c QuantLib/quantlib_wrap.cpp -o build/temp.linux-aarch64-3.7/QuantLib/quantlib_wrap.o -Wno-unused --param ggc-min-expand=1 --param ggc-min-heapsize=32768 -fno-strict-aliasing -Wno-unused -Wno-uninitialized -Wno-sign-compare -Wno-write-strings -Wno-deprecated-declarations -Wno-misleading-indentation mkdir -p build/lib.linux-aarch64-3.7/QuantLib/ g++ -shared -Wl,-z,relro -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-aarch64-3.7/QuantLib/quantlib_wrap.o -lQuantLib -o build/lib.linux-aarch64-3.7/QuantLib/_QuantLib.cpython-37m-aarch64-linux-gnu.so # create wheel file python3 setup.py bdist_wheel # Upgrade PIP and install the wheel file /usr/bin/python3 -m pip install --upgrade pip pip3 install dist/QuantLib-1.22-cp37-cp37m-linux_aarch64.whl # Or alternatively install as site-package sudo python3 setup.py install # Test examples after installation pip3 install pandas python3 examples/bonds.py

File Download QuantLib-1.22-cp37-cp37m-linux_armv7l.whl https://mega.nz/file/mtJSxZTT#fzDDHw0AIqz-2LIspBGNZLoyW4_MT9qjft_b-ITTA8w

File Download QuantLib-1.22-cp37-cp37m-linux_aarch64.whl https://mega.nz/file/WlAEXJCZ#UKFnlTrfQfRNzFW-OJbXHLFIHwzCw_189HvMa_xU4Oo

How to install docker client and connect to docker desktop engine on another machine macOS or Windows.

Use Docker Desktop for Windows 10 / macOS as engine and docker client for linux/macos/Android to connect.
Why use docker client ? Because don't want to / cannot install docker engine in the client environment and just want to connect to the docker engine on local LAN.
(1) For Windows 10 Host, after installation of docker desktop
# C:\ProgramData\Docker\config\daemon.json and add

"hosts" : ["tcp://0.0.0.0:2375"],

# change port forwarding to docker wsl backend, run this in powershell admin mode

netsh interface portproxy add v4tov4 listenport=2375 listenaddress=192.168.64.1 connectaddress=127.0.0.1 connectport=2375

# change Windows defender firewall and add incoming rule to enable port 2375 # Refer to this for setting openssh server authorized_keys, with proper file permission https://superuser.com/questions/1445976/windows-ssh-server-refuses-key-based-authentication-from-client

(2) For macOS Host, after installation of docker desktop
# ssh-keygen in client and ssh-copy-id to host, e.g. remote host username with ip address of 192.168.64.1

ssh-keygen -t rsa
ssh-copy-id username@192.168.64.1

# edit sshd_config

sudo vi /private/etc/ssh/sshd_config

# and add

PermitUserEnvironment PATH,LANG

# edit ~/.docker/daemon.json and add

"hosts" : ["tcp://0.0.0.0:2375"],

# add .ssh/environment

PATH=$PATH:/usr/local/bin

# restarting sshd using macOS System Preferences -> Sharing -> File Sharing

(3) To install docker client for Linux using tcp

cd ~/
wget https://download.docker.com/linux/static/stable/x86_64/docker-20.10.6.tgz
tar -xzvf docker-20.10.6.tgz
cd docker
./docker -H tcp://<remote host ip address>:2375 images

(3.1) To install docker client for Termux app of Android and using tcp

wget https://download.docker.com/linux/static/stable/aarch64/docker-20.10.6.tgz

tar xzvf docker-20.10.6.tgz
mv docker/docker /data/data/com.termux/files/usr/bin/
./docker -H tcp://<remote host ip address>:2375 images

(4) To install docker client for macOS using ssh

cd ~/
wget https://download.docker.com/mac/static/stable/x86_64/docker-20.10.6.tgz
#or curl -OL https://download.docker.com/mac/static/stable/x86_64/docker-20.10.6.tgz
#or curl -OL https://download.docker.com/mac/static/stable/aarch64/docker-20.10.6.tgz
tar xzvf docker-20.10.6.tgz
xattr -rc docker
cd docker
sudo mkdir -p /usr/local/bin
sudo mv * /usr/local/bin/
docker -H ssh://username@<remote host ip address> images

(5) Or simply add the corresponding variables to ~/.bashrc

unset DOCKER_HOST
# for tcp connection to Windows 10 host
export DOCKER_HOST=tcp://192.168.64.1:2375

# for ssh connection to macOS host using SSH
export DOCKER_HOST=ssh://user@192.168.64.1

How to install openssh server in wsl2

(1) This is the guide to install openssh server and connect to wsl2 shell when ssh login from other machines in LAN network.

https://www.hanselman.com/blog/the-easy-way-how-to-ssh-into-bash-and-wsl2-on-windows-10-from-an-external-machine

(2) However, if still cannot connect from external machines from LAN network after setting all the firewall rules. Do this in powershell with admin right.

Set-ExecutionPolicy Unrestricted -Force

(3) If want to setup jupyter-notebook server in wsl2 and connect from other machines in the LAN network. Follow this guide.
https://medium.com/swlh/how-to-set-up-a-jupiter-notebook-server-and-access-it-from-a-local-or-remote-network-on-windows-d335c5ba490d

(4) The important setup steps are to open the Windows firewall rule and the script to port forward to the vm of the wsl2 as in.

wget https://gist.githubusercontent.com/david-littlefield/f45999c069e0b6b68bdae829d8616727/raw/80a60968f1bdda598eaf275bf1300bb3451d45ab/jupyter_notebook_port_wsl2.ps1

(5) It is advised to login from external internet to local LAN machines via VPN Server only rather than exposing or forwarding the local ports directly via the router.

(6) Refer to this for setting openssh server authorized_keys, with proper file permission https://superuser.com/questions/1445976/windows-ssh-server-refuses-key-based-authentication-from-client

How to use Chrome browser to scrape website using javascript

(1) This is to demo how to scrape website using Chrome Browser and save the json text results to local drive.

(2) First launch Desktop version of Chrome Browser and goto this website "http://aastocks.com/en/stocks/market/calendar.aspx?type=5"

(3) Install "JQuery Inject" as Chrome Extension and enable it in current browser session.

(4) Open Chrome Developer tools (Ctrl-Shift-I) and select console tab to enter the following code. Enter the code in 3 steps.

Chrome Browser console code : Step 1   Select all

// Step 1 // define arrays for scraped objects and should be store as global variable var scrapeResults = [];

Console Code : Step 2   Select all

//Step 2 // function to scrape page $("table.CalendarResultTable > tbody > .crtRow").each((index, element) => { const tds = $(element).find("td"); if (index===0) { previousDate = $(tds[0]).text(); } if ($(tds[0]).text().trim()==='') { } else { previousDate = $(tds[0]).text(); } const date = previousDate; const namecell = $(tds[1]).find("a"); const name = $(tds[1]).text().replace('\n','').split(/[0-9]+.HK/)[0].trim(); const stockcode = $(namecell).text(); const stockurl = $(namecell).attr("href"); const dividend = $(tds[2]).text().trim().split('D:')[1]; const dividenddate = $(tds[3]).text().trim().split(/Ex-Date: | Payable: | Book Close: /); const exdate = dividenddate[1] const payable = dividenddate[2] const bookclose = dividenddate[3] const scrapeResult = { date, name, stockcode, stockurl, dividend, exdate, payable, bookclose }; //console.log(scrapeResult); if (!scrapeResults.find(({stockcode}) => stockcode === scrapeResult.stockcode)) { scrapeResults.push(scrapeResult); } }); // copy to clipboard. copy(scrapeResults);

If there are more than 1 page, click to goto next page and Repeat Step 2 to scrape again. After finished with all the pages then run Step 3 code to download to local drive. scrapeResults should be stored as Global variable for this to work,

Console Code : Step 3   Select all

// Step 3 // define download function for webAPI function download(content, fileName, contentType) { var a = document.createElement("a"); var file = new Blob([content], {type: contentType}); a.href = URL.createObjectURL(file); a.download = fileName; a.click(); } // download json to local folder var jsonData = JSON.stringify(scrapeResults); var currentdate = new Date(); download(jsonData, currentdate.getFullYear() + ('0'+(currentdate.getMonth()+1)).slice(-2) + ('0'+currentdate.getDate()).slice(-2) + '_' + ('0'+currentdate.getHours()).slice(-2) + ('0'+currentdate.getMinutes()).slice(-2) + ('0'+currentdate.getSeconds()).slice(-2)+'_stockjson.txt', 'text/plain');

(5) The same jquery code function above can be used in nodejs script for automation. Just add "request request-promise cheerio" packages to the project

(6) For nodejs, the save function should be

nodejs script   Select all

var fs = require('fs'); fs.writeFile("json.txt", jsonData, function(err) { if (err) { console.log(err); } });

(7) For browser console code without jQuery inject or don't want to import the jQuery library, and have to use querySelectorAll() function and use Object.values to convert to object as demo below.

console code   Select all

// Goto http://www.aastocks.com/en/stocks/market/calendar.aspx?type=1 and then open Browser (Chrome, Firefox, Safari) developer tools using ( Cmd + Opt + I in mac or Ctrl + Shift + I in win) and enter the following console code to run. var scrapeResults = ''; document.querySelectorAll('tr.crtRow').forEach(function(item) { const first = Object.values(item.querySelectorAll('.first'))[0]; if (typeof first !== 'undefined' && first !== null) { console.log(first.textContent??first.textContent.trim()); scrapeResults = scrapeResults.concat(first.textContent??first.textContent.trim(), ' '); }; const second = Object.values(item.querySelectorAll('td.second'))[0]; if (typeof second !== 'undefined' && second !== null) { console.log(second.textContent??second.textContent.trim()); scrapeResults = scrapeResults.concat(second.textContent??second.textContent.trim(), ' '); }; const third = Object.values(item.querySelectorAll('td.minw4'))[0]; if (typeof third !== 'undefined' && third !== null) { console.log(third.textContent??third.textContent.trim()); scrapeResults = scrapeResults.concat(third.textContent??third.textContent.trim(), ' '); }; const last = Object.values(item.querySelectorAll('td.last.minw1'))[0]; if (typeof last !== 'undefined' && last !== null) { console.log(last.textContent??last.textContent.trim()); scrapeResults = scrapeResults.concat(last.textContent??last.textContent.trim(), '\n'); }; }); // copy results to clipboard. copy(scrapeResults);

(7.1) For browser console code and use Array.from and map function to return json data.

console code   Select all

// //// Goto http://www.aastocks.com/en/stocks/market/calendar.aspx?type=1 and then open Browser (Chrome, Firefox, Safari) developer tools using ( Cmd + Opt + I in mac or Ctrl + Shift + I in win) and enter the following console code to run. var data = Array.from( document.querySelectorAll('.crtRow') ).map( row => Array.from(row.children).map(node => node.textContent.trim()) ).map( (row) => row[0].length === 0 ? [...row.slice(1)] : row ).map( (row, idx, arr) => { if (row.length === 1) return null; const getLastMatch = (idx, arr) => arr[idx].length === 4 ? arr[idx] : getLastMatch(idx - 1, arr); const match = getLastMatch(idx, arr); const isSameDate = row.length === 3; console.log(''.concat(match[0],' ',row[1 - isSameDate *1],' ',row[2 - isSameDate *1],' ',row[3 - isSameDate *1],'\n')); return { date:match[0], stock:row[1- isSameDate *1], code:row[1 - isSameDate *1].slice(-8).slice(0,5), industry:row[2 - isSameDate *1], period:row[3 - isSameDate *1 -3] } }).filter(Boolean); console.log(data); copy(data);

(8) Another example for nodejs scrapping code as demo below.

ronaldo.js   Select all

// need to install npm install request // run with node ronaldo.js const request = require("request-promise"); const url = "https://www.transfermarkt.com/cristiano-ronaldo/alletore/spieler/8198/plus/1" headers = { 'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36' } ; const jsdom = require("jsdom"); const { JSDOM } = jsdom; const options = { url: url, timeout: 300000, headers:headers }; async function scrape() { try { const htmlResult = await request.get(options); const dom = new JSDOM(htmlResult); const { document } = dom.window; var data = Array.from( document.querySelectorAll('.responsive-table table tbody tr') ).map( row => Array.from(row.children).map(node => node.textContent.trim()) ).map( (row) => row.length === 15 ? [...row.slice(0, 5), ...row.slice(6)] : row ).map( (row, idx, goals) => { if (row.length === 1) return null; const getLastMatch = (idx, goals) => goals[idx].length === 14 ? goals[idx] : getLastMatch(idx - 1, goals); const match = getLastMatch(idx, goals); const isSameMatch = row.length === 14; return { competition: match[1], matchday: match[2], date: match[3], venue: match[4], opponent: match[7], result: match[8], position: match[9], minute: row[1 + isSameMatch * 9], atScore: row[2 + isSameMatch * 9], goalType: row[3 + isSameMatch * 9], assist: row[4 + isSameMatch * 9], } } ).filter(Boolean) // filter null .filter(x => (new Date(x.date)).getFullYear() >= 2021) // filter year console.log(data); } catch (err) { // try catch console.error(err); } } scrape();

Google colab - keras-learn and Logistic Regression example

(1) Following the previous post, this demo the keras sample from how-to-install-tensorflow-with-gpu.html
Please take note that google only allow one active session for the free service. If you need faster GPU, more RAM and sessions, please consider to subscribe colab pro.

keraslearn.ipynb   Select all

# Step 1 mount google drive if data is from google drive import os from google.colab import drive drive.mount('/content/drive') # Step 2 if using tensorflow GPU #%tensorflow_version 2.x #import tensorflow as tf #print('TensorFlow: {}'.format(tf.__version__)) #tf.test.gpu_device_name() # Step 3 from keras.models import Sequential from keras.layers import Dense import numpy import time # fix random seed for reproducibility numpy.random.seed(7) # Step 4 # download pima indians dataset to google drive !curl -L https://tinyurl.com/tensorflowwin | grep -A768 pima-indians-diabetes.data.nbsp | sed '1d' > 'drive/MyDrive/Colab Notebooks/pima-indians-diabetes.data' # or download to local data directory !mkdir -p ./data !curl -L https://tinyurl.com/tensorflowwin | grep -A768 pima-indians-diabetes.data.nbsp | sed '1d' > './data/pima-indians-diabetes.data' # Step 5 load dataset from google drive dataset = numpy.loadtxt("drive/MyDrive/Colab Notebooks/pima-indians-diabetes.data", delimiter=",") # or load data from local data directory dataset = numpy.loadtxt("./data/pima-indians-diabetes.data", delimiter=",") # Step 6 # split into input (X) and output (Y) variables X = dataset[:,0:8] Y = dataset[:,8] # Step 7 # create model model = Sequential() model.add(Dense(12, input_dim=8, activation='relu')) model.add(Dense(1, activation='sigmoid')) # Step 8 # Compile model model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) # Step 9 start_time=time.time() # Fit the model model.fit(X, Y, batch_size=10, epochs=1500) # parameters for keras 1.2.2 # evaluate the model scores = model.evaluate(X, Y) print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100)) print("\nTraining took %.2f seconds\n" %(time.time()-start_time))

(2) For large training data set, consider to zip them and upload to google drive. Mount the google drive, then unzip it in local session. e.g.

!mkdir -p ./data
!unzip -o './drive/MyDrive/Colab Notebooks/mydata.zip' -d ./data/

(3) To stop the running cell in Google Colab use Ctrl-M I

(4) How to quickly run an ipynb example from github ?
4.1) Go to https://colab.research.google.com/, after login gmail and choose GitHub tab and enter search say "clareyan/From-Linear-to-Logistic-Regression-Explained-Step-by-Step"
4.2) In Step 2 cell box change the importing of dataset to
df = pd.read_csv('https://raw.githubusercontent.com/clareyan/From-Linear-to-Logistic-Regression-Explained-Step-by-Step/master/Social_Network_Ads.csv')
4.3) Then choose menu -> Runtime -> Run All. After that, use menu -> File -> Save a copy in Drive.

How to setup google colab and start linear regression with tensorflow.

(1) You only need a chrome browser, google gmail account and google drive account to start cloud tensorflow computing. And it is free to use and learn.

(2) Go to https://colab.research.google.com/
(3) Create a new notebook rename it and then Copy to Drive
(4) Type the following into notebook and run it step by step (Press Alt-Enter to run after each step)

LinearRegression.ipynb   Select all

#Step 1 # mount Google Drive, will ask for authorization code import numpy as np import os from google.colab import drive drive.mount('/content/drive') #Step 2 # choose the notebook settings to use GPU, via Menu -> Edit -> Notebook Settings. %tensorflow_version 2.x import tensorflow as tf ＃ will show GPU if successful tf.test.gpu_device_name() #Step 3 # load data import pandas as pd # either download the linear_data.csv and upload to google drive, or direct download it via the shell command as below !curl -L https://tinyurl.com/lineardatacsv | grep -A200 START_OF_LINEAR_DATA.CSV | sed '1d' | sed -n "/END_OF_LINEAR_DATA.CSV/q;p" | sed 's/&gt;/\>/g;s/&lt;/\</g' > 'drive/MyDrive/Colab Notebooks/linear_data.csv' df = pd.read_csv('drive/MyDrive/Colab Notebooks/linear_data.csv') df.head() #Step 4 # split into independent and dependent X = df[['X']].values y = df[['Y']].values X.shape, y.shape #Step 5 # visualize data import matplotlib.pyplot as plt %matplotlib inline plt.scatter(X,y) plt.xlabel('independent') plt.ylabel('dependent') plt.show() # Use Text box to enter ＃ # Linear Regression $ \hat y = a + b * X $ #Step 6 # Linear Regression # define regression model class regression(): def __init__(self): self.a = tf.Variable(initial_value=0,dtype=tf.float32) self.b = tf.Variable(initial_value=0,dtype=tf.float32) def __call__(self, X): x = tf.convert_to_tensor(X,dtype=tf.float32) y_est = tf.add(self.a, tf.multiply(self.b,x)) return y_est model = regression() # Use Text box to enter ＃ # loss = sum of square error (sse) = $ \sum (y_t - y_p) ^ 2 $ # step 7 # define loss function def loss_func(y_true, y_pred): # both values are in tensors sse = tf.reduce_sum(tf.square(tf.subtract(y_true,y_pred))) return sse # Use Text box to enter ＃ # Gradient Descent $ a = a_i - \nabla(sse) | a * LR $ $ b = b_i - \nabla(sse) | b * LR $ # step 8 # define train function def train(model, inputs, outputs, learning_rate): # convert outputs into tensor y_true = tf.convert_to_tensor(outputs,dtype=tf.float32) # GradientTape cal gradient distance with tf.GradientTape() as g: y_pred = model(inputs) current_loss = loss_func(y_true,y_pred) da,db = g.gradient(current_loss,[model.a,model.b]) # update the values model.a.assign_sub(da*learning_rate) model.b.assign_sub(db*learning_rate) # Step 9 def plot_scatter(x,y): plt.scatter(x,y) # scatter plt.plot(x,model(x),'r--') #line plot_scatter(X,y) # step 10 # model fitting model = regression() a_values = [] b_values = [] cost_values = [] # epochs, no of steps epochs = 100 # learning_rate learning_rate = 0.0001 for epoch in range(epochs): a_values.append(model.a) b_values.append(model.b) # prediction values and error y_pred = model(X) cost_value = loss_func(y,y_pred) cost_values.append(cost_value) # training train(model,X,y,learning_rate) # visual the scatter plot_scatter(X,y) plt.show #print the value print('Epoch: %d, Loss: %0.2f, a: %0.2f, b: %0.2f' %(epoch,cost_value,a_values[-1],b_values[-1])) # step 11 plt.plot(cost_values)

(5) Linear_data.csv , download and upload to google drive

linear_data.csv   Select all

X,Y 4,2 4,10 7,4 7,22 8,16 9,10 10,18 10,26 10,34 11,17 11,28 12,14 12,20 12,24 12,28 13,25 13,34 13,24 13,46 14,26 14,36 14,60 14,80 15,20 15,26 15,54 16,32 16,40 17,32 17,40 17,50 18,42 18,56 18,76 18,84 19,36 19,45 19,68 20,32 20,48 20,52

HelloWorld Assembler Code for x86_64, arm64 and for linux or macOS

(1) Following the previous post, this post demo the assembler code for command line program HelloWorld for x86_64, arm64 and for linux or macOS.

HelloWorld.S   Select all

// // Assembler program to print "Hello World!" // to stdout. For amr64, x86_64, linux and macOS // #define STDIN 0 // standard input device #define STDOUT 1 // standard output device #ifdef __APPLE__ #define SYS_read 0x2000003 // system call to read input macOS #define SYS_write 0x2000004 // system call to write message macOS #define SYS_exit 0x2000001 // system call to terminate program macOS #define SVC_write 4 // SVC write arm64 macOS #define SVC_exit 1 // SVC exit arm64 macOS #endif #ifdef __linux__ #define SYS_read 0 // system call to read input #define SYS_write 1 // system call to write message #define SYS_exit 60 // system call to terminate program #define SVC_write 64 // SVC write arm64 linux #define SVC_exit 93 // SVC exit arm64 linux #endif #define EXIT_OK 0 // OK exit status .globl _start // Provide program starting address to linker #ifdef __APPLE__ .align 4 #endif .text _start: #if defined __arm64__ || defined __ARM_ARCH_ISA_A64 mov X0, #STDOUT // 1 = StdOut #ifdef __linux__ ldr X1, =helloworld // string to print mov X8, #SVC_write // linux write system call #endif #ifdef __APPLE__ // adr X1, helloworld // string to print //(adr calculates an address from the PC plus an offset, but for local) adrp X1, helloworld@PAGE // adrp can be used to access relative address of 4GB range add X1, X1, helloworld@PAGEOFF // string to print mov X16, #SVC_write // linux write system call #endif ldr X2, =len // length of our string svc #0 // Call linux to output the string // Setup the parameters to exit the program // and then call Linux to do it. mov X0, #0 // Use 0 return code #ifdef __linux__ mov X8, #SVC_exit // Service command code 93 terminates this program #endif #ifdef __APPLE__ mov X16, #1 // Service command terminates this program #endif svc #0 // Call linux to terminate the program #endif #if defined __x86_64__ movq $STDOUT, %rdi #ifdef __linux__ movq $helloworld, %rsi // char * #endif #ifdef __APPLE__ leaq helloworld(%rip), %rsi #endif movq $len, %rdx // length of our string movq $SYS_write, %rax // write system call syscall movq $EXIT_OK, %rdi // Use 0 return code movq $SYS_exit, %rax // exit system call syscall #endif .data helloworld: .ascii "Hello World!\n" len = . - helloworld // len = start - end

(2) To compile and debug for different systems

shell scripts   Select all

# To download the above code using command line. curl -L https://tinyurl.com/helloworld-gas | grep -A200 START_OF_HELLOWORLD.S | sed '1d' | sed -n "/END_OF_HELLOWORLD.S/q;p" | sed 's/&gt;/\>/g;s/&lt;/\</g' > HelloWorld.S # To compile with debug symbols under linux, e.g. Win10 WSL2 or Linux or Android Termux App clang -g -c HelloWorld.S -o HelloWorld.o ; ld HelloWorld.o -o HelloWorld # To compile under macOS (e.g. with M1 cpu) clang -g HelloWorld.S -o HelloWorld_x86_64 -e _start -arch x86_64 clang -g HelloWorld.S -o HelloWorld_arm64 -e _start -arch arm64

(3) To debug using lldb

shell scripts   Select all

# To start program debug lldb HelloWorld_x86_64 # or lldb HelloWorld_arm64 # lldb debug session for arm64 - useful commands (lldb) breakpoint set --name _start (lldb) breakpoint list (lldb) run (lldb) step (lldb) reg read x0 x1 x2 x8 lr pc (lldb) reg read -f t cpsr # lldb debug session for x86_64 - useful commands (lldb) reg read -f d rax rdi rsi rdx rflags (lldb) reg read -f t rflags # print the address value in the stackpointer for x86_64 (lldb) p *(int **)$sp # hint: to search lldb command history use ctrl-r

(4) Summary of differences
4.1) In order to preprocess the assembler file using clang compiler, the filename extension should be capital letter S in linux. Subroutine name between C and global asm labels should prefix by underscore for macOS.
4.2) A64 (arm64) parameter/ results registers are X0-7. If the function has a return value, it will be stored in X0.
4.3) x86_64 parameter registers for integer or pointer are %rdi. %rsi, %rdx, %rcx, %r8, %r9. If the function has a return value, it will be stored in %rax.
4.4) Linux and macOS has different syscall number (x86_64) or Service call number (for arm64). They are defined in this source code.
4.5) Absolute addressing is not allowed for arm64. For macOS, adr instruction can be used for accessing readonly local data. But for non-local data section (which is a buffer in RAM), adrp instruction and @PAGE and @PAGEOFF operators should be used as demo in the code.

Mixing C and Assembler for x86_64 and arm64, major differences.

(1) These demo the mixing of C and Assembler Language for x86_64 and arm64 and show the differences in linux and macOS environment.

callsum.c   Select all

/* * callsum.c * * Illustrates how to call the sum function in assembly language. */ #include <stdio.h> double sum(double[], unsigned); int main() { double test[] = { 40.5, 26.7, 21.9, 1.5, -40.5, -23.4 }; printf("%20.7f\n", sum(test, 6)); printf("%20.7f\n", sum(test, 2)); printf("%20.7f\n", sum(test, 0)); printf("%20.7f\n", sum(test, 3)); printf("I am "); #ifdef __ARM_ARCH_ISA_A64 printf(" __ARM_ARCH_ISA_A64 "); #endif #ifdef __arm64__ printf(" __arm64__ "); #endif #ifdef __x86_64__ printf(" __x86_64__ "); #endif #ifdef __linux__ printf(" __linux__ "); #endif #ifdef __APPLE__ printf(" __APPLE__ "); #endif printf("\n"); return 0; }

sum.S   Select all

# --------------------------------------------------------------- # A 64-bit function that returns the sum of the elements in a # floating-point array. The function has prototype: # # double sum(double[] array, unsigned length) # ----------------------------------------------------------------------- #ifdef __linux__ .global sum #endif #ifdef __APPLE__ .global _sum #endif .text #ifdef __ARM_ARCH_ISA_A64 .align 4 #endif #ifdef __linux__ sum: #endif #ifdef __APPLE__ _sum: #endif #ifdef __x86_64__ xorpd %xmm0, %xmm0 // initialize the sum to 0 cmp $0, %rsi // special case for length = 0 je done #endif #if defined __arm64__ || defined __ARM_ARCH_ISA_A64 movi d0, #0 // initialize the sum to 0 // floats in s0-7 and doubles in the d0-7 registers. cmp x1, #0 // special case for length = 0 b.eq done #endif next: #ifdef __x86_64__ addsd (%rdi), %xmm0 // add in the current array element add $8, %rdi // move to next array element dec %rsi // count down jnz next // if not done counting, continue #endif #if defined __arm64__ || defined __ARM_ARCH_ISA_A64 ldr d16, [x0] // load the float into d16 // floats in s0-7 and doubles in the d0-7 registers. fadd d0, d0, d16 // add in the current array element add x0, x0, #8 // move to next array element subs x1, x1, #1 // count down cbnz w1, next // if not done counting, continue #endif done: ret

callfactorial.c   Select all

/* * An application that illustrates calling the factorial function defined elsewhere. */ #include <stdio.h> #include <inttypes.h> #ifdef __USE_C_FUNCTION uint64_t factorial(unsigned n) { return (n <= 1) ? 1 : n * factorial(n-1); } #else uint64_t factorial(unsigned n); #endif int main() { for (unsigned i = 0; i < 20; i++) { #ifdef __linux__ printf("factorial(%2u) = %lu\n", i, factorial(i)); #endif #ifdef __APPLE__ printf("factorial(%2u) = %llu\n", i, factorial(i)); #endif } printf("I am "); #ifdef __ARM_ARCH_ISA_A64 printf(" __ARM_ARCH_ISA_A64 "); #endif #ifdef __arm64__ printf(" __arm64__ "); #endif #ifdef __x86_64__ printf(" __x86_64__ "); #endif #ifdef __linux__ printf(" __linux__ "); #endif #ifdef __APPLE__ printf(" __APPLE__ "); #endif printf("\n"); }

factorial.S   Select all

# ---------------------------------------------------------------------------- # A 64-bit recursive implementation of the function # # uint64_t factorial(unsigned n) # # implemented recursively # ---------------------------------------------------------------------------- #ifdef __linux__ .globl factorial #endif #ifdef __APPLE__ .globl _factorial #endif .text #ifdef __ARM_ARCH_ISA_A64 .align 4 #endif #ifdef __linux__ factorial: #endif #ifdef __APPLE__ _factorial: #endif #if defined __arm64__ || defined __ARM_ARCH_ISA_A64 cmp x8, #1 //# n > 1? b.gt L1 //# if yes, go do a recursive call mov x0, #1 //# otherwise return 1 ret #endif #ifdef __x86_64__ cmp $1, %rdi # n <= 1? jnbe L1 # if not, go do a recursive call mov $1, %rax # otherwise return 1 ret #endif L1: #if defined __arm64__ || defined __ARM_ARCH_ISA_A64 STP X8, LR, [SP, #-16]! //# push x8 and LR(x30) // LR is used to return from subroutine subs x8, x8, #1 //# n-1 #ifdef __linux__ bl factorial //# factorial(n-1), result goes in x0 #endif #ifdef __APPLE__ bl _factorial //# factorial(n-1), result goes in x0 #endif LDP X8, LR, [SP], #16 //# pop x8 and LR(x30) mul x0, x0, x8 //# n * factorial(n-1), stored in x0 ret #endif #ifdef __x86_64__ push %rdi # save n on stack (also aligns %rsp!) dec %rdi # n-1 #ifdef __linux__ call factorial # factorial(n-1), result goes in %rax #endif #ifdef __APPLE__ call _factorial # factorial(n-1), result goes in %rax #endif pop %rdi # restore n imul %rdi, %rax # n * factorial(n-1), stored in %rax ret #endif

callmaxofthree.c   Select all

/* * callmaxofthree.c * * A small program that illustrates how to call the maxofthree function we wrote in * assembly language. */ #include <stdio.h> #include <inttypes.h> int64_t maxofthree(int64_t, int64_t, int64_t); int main() { #ifdef __linux__ printf("%ld\n", maxofthree(1, -4, -7)); printf("%ld\n", maxofthree(2, -6, 1)); printf("%ld\n", maxofthree(2, 3, 1)); printf("%ld\n", maxofthree(-2, 4, 3)); printf("%ld\n", maxofthree(2, -6, 5)); printf("%ld\n", maxofthree(2, 4, 6)); #endif #ifdef __APPLE__ printf("%lld\n", maxofthree(1, -4, -7)); printf("%lld\n", maxofthree(2, -6, 1)); printf("%lld\n", maxofthree(2, 3, 1)); printf("%lld\n", maxofthree(-2, 4, 3)); printf("%lld\n", maxofthree(2, -6, 5)); printf("%lld\n", maxofthree(2, 4, 6)); #endif printf("I am "); #ifdef __ARM_ARCH_ISA_A64 printf(" __ARM_ARCH_ISA_A64 "); #endif #ifdef __arm64__ printf(" __arm64__ "); #endif #ifdef __x86_64__ printf(" __x86_64__ "); #endif #ifdef __linux__ printf(" __linux__ "); #endif #ifdef __APPLE__ printf(" __APPLE__ "); #endif printf("\n"); return 0; }

maxofthree.S   Select all

# ----------------------------------------------------------------------------- # A 64-bit function that returns the maximum value of its three 64-bit integer # arguments. The function has signature: # # int64_t maxofthree(int64_t x, int64_t y, int64_t z) # # Note that the parameters for x86_64 have already been passed in rdi, rsi, and rdx. We # Note that the parameters for arm64 have already been passed in x0, x1, x2. We # just have to return the value in rax(x86_64), x0(arm64). # ----------------------------------------------------------------------------- #ifdef __linux__ .globl maxofthree #endif #ifdef __APPLE__ .globl _maxofthree #endif .text #ifdef __ARM_ARCH_ISA_A64 .align 4 #endif #ifdef __linux__ maxofthree: #endif #ifdef __APPLE__ _maxofthree: #endif #if defined __arm64__ || defined __ARM_ARCH_ISA_A64 cmp x0, x1 //# is x0 > x1 csel x0, x0, x1, GT // if GT, x0 = x0 else x0 = x1 cmp x0, x2 //# is x0 > x2 csel x0, x0, x2, GT // if GT, x0 = x0 else x0 = x2 ret //# the max will be in x0 #endif #ifdef __x86_64__ mov %rdi, %rax # result (rax) initially holds x cmp %rsi, %rax # is x less than y? cmovl %rsi, %rax # if so, set result to y cmp %rdx, %rax # is max(x,y) less than z? cmovl %rdx, %rax # if so, set result to z ret # the max will be in eax #endif

chaskey.h   Select all

#ifndef CHASKEY_H #define CHASKEY_H #define CHASKEY_ENCRYPT 1 #define CHASKEY_DECRYPT 0 #ifdef __cplusplus extern "C" { #endif void chas_encrypt(int, void*, void*); void chaskey(void*, void*); void chas_encryptx(void*, void*); #ifdef __cplusplus } #endif #endif

testckey.c   Select all

// test unit for chaskey #include <stdio.h> #include <string.h> #include <inttypes.h> #include "chaskey.h" uint8_t plain[16]= { 0xb8, 0x23, 0x28, 0x26, 0xfd, 0x5e, 0x40, 0x5e, 0x69, 0xa3, 0x01, 0xa9, 0x78, 0xea, 0x7a, 0xd8 }; uint8_t key[16] = { 0x56, 0x09, 0xe9, 0x68, 0x5f, 0x58, 0xe3, 0x29, 0x40, 0xec, 0xec, 0x98, 0xc5, 0x22, 0x98, 0x2f }; uint8_t cipher[16] = { 0xd5, 0x60, 0x8d, 0x4d, 0xa2, 0xbf, 0x34, 0x7b, 0xab, 0xf8, 0x77, 0x2f, 0xdf, 0xed, 0xde, 0x07 }; int main(void) { uint8_t t[16]; int e; memcpy(t, plain, 16); chaskey(key, t); e = memcmp(t, cipher, 16)==0; printf("\nCHASKEY Encryption: %s\n", e ? "OK" : "FAILED"); printf("I am "); #ifdef __ARM_ARCH_ISA_A64 printf(" __ARM_ARCH_ISA_A64 "); #endif #ifdef __arm64__ printf(" __arm64__ "); #endif #ifdef __x86_64__ printf(" __x86_64__ "); #endif #ifdef __linux__ printf(" __linux__ "); #endif #ifdef __APPLE__ printf(" __APPLE__ "); #endif printf("\n"); return 0; }

ckey.S   Select all

// CHASKEY in ARM64 assembly // Chaskey-LTS Block Cipher in AMD64 assembly (Encryption only) .text #ifdef __x86_64__ .intel_syntax noprefix #endif .globl chaskey .globl _chaskey #ifdef __ARM_ARCH_ISA_A64 .align 4 #endif // chaskey(void*mk, void*data); chaskey: _chaskey: #if defined __arm64__ || defined __ARM_ARCH_ISA_A64 // load 128-bit key ldp w2, w3, [x0] ldp w4, w5, [x0, 8] // load 128-bit plain text ldp w6, w7, [x1] ldp w8, w9, [x1, 8] // xor plaintext with key eor w6, w6, w2 // x[0] ^= k[0]; eor w7, w7, w3 // x[1] ^= k[1]; eor w8, w8, w4 // x[2] ^= k[2]; eor w9, w9, w5 // x[3] ^= k[3]; mov w10, 16 // i = 16 #endif #ifdef __x86_64__ // .intel_syntax noprefix push rbx push rbp push rsi # load plaintext lodsd xchg eax, ebp lodsd xchg eax, ebx lodsd xchg eax, edx lodsd xchg eax, ebp # pre-whiten xor eax, [rdi ] xor ebx, [rdi+ 4] xor edx, [rdi+ 8] xor ebp, [rdi+12] push 16 pop rcx #endif L0: #if defined __arm64__ || defined __ARM_ARCH_ISA_A64 add w6, w6, w7 // x[0] += x[1]; eor w7, w6, w7, ror 27 // x[1]=R(x[1],27) ^ x[0]; add w8, w8, w9 // x[2] += x[3]; eor w9, w8, w9, ror 24 // x[3]=R(x[3],24) ^ x[2]; add w8, w8, w7 // x[2] += x[1]; ror w6, w6, 16 add w6, w9, w6 // x[0]=R(x[0],16) + x[3]; eor w9, w6, w9, ror 19 // x[3]=R(x[3],19) ^ x[0]; eor w7, w8, w7, ror 25 // x[1]=R(x[1],25) ^ x[2]; ror w8, w8, 16 // x[2]=R(x[2],16); subs w10, w10, 1 // i-- bne L0 // i > 0 // xor cipher text with key eor w6, w6, w2 // x[0] ^= k[0]; eor w7, w7, w3 // x[1] ^= k[1]; eor w8, w8, w4 // x[2] ^= k[2]; eor w9, w9, w5 // x[3] ^= k[3]; // save 128-bit cipher text stp w6, w7, [x1] stp w8, w9, [x1, 8] ret #endif #ifdef __x86_64__ // .intel_syntax noprefix # x[0] += x[1]# add eax, ebx # x[1]=ROTR32(x[1],27) ^ x[0] ror ebx, 27 xor ebx, eax # x[2] += x[3]# add edx, ebp # x[3]=ROTR32(x[3],24) ^ x[2] ror ebp, 24 xor ebp, edx # x[2] += x[1]# add edx, ebx # x[0]=ROTR32(x[0],16) + x[3] ror eax, 16 add eax, ebp # x[3]=ROTR32(x[3],19) ^ x[0] ror ebp, 19 xor ebp, eax # x[1]=ROTR32(x[1],25) ^ x[2] ror ebx, 25 xor ebx, edx # x[2]=ROTR32(x[2],16) ror edx, 16 loop L0 # post-whiten xor eax, [rdi ] xor ebx, [rdi+ 4] xor edx, [rdi+ 8] xor ebp, [rdi+12] pop rdi # save ciphertext stosd xchg eax, ebx stosd xchg eax, edx stosd xchg eax, ebp stosd pop rbp pop rbx ret #endif

speck.h   Select all

#ifndef SPECK_H #define SPECK_H #ifdef __cplusplus extern "C" { #endif void speck64(void*, void*); void speck128(void*, void*); #ifdef __cplusplus } #endif #endif

testspk.c   Select all

// test unit for speck #include <stdio.h> #include <string.h> #include <inttypes.h> #include "speck.h" void print_bytes(char *s, void *p, int len) { int i; printf("%s : ", s); for (i=0; i<len; i++) { printf ("%02x ", ((uint8_t*)p)[i]); } putchar('\n'); } // SPECK64/128 test vectors // // p = 0x3b7265747475432d uint8_t plain64[]= { 0x74, 0x65, 0x72, 0x3b, 0x2d, 0x43, 0x75, 0x74 }; // c = 0x8c6fa548454e028b uint8_t cipher64[]= { 0x48, 0xa5, 0x6f, 0x8c, 0x8b, 0x02, 0x4e, 0x45 }; // key = 0x03020100, 0x0b0a0908, 0x13121110, 0x1b1a1918 uint8_t key64[]= { 0x00, 0x01, 0x02, 0x03, 0x08, 0x09, 0x0a, 0x0b, 0x10, 0x11, 0x12, 0x13, 0x18, 0x19, 0x1a, 0x1b }; // SPECK128/256 test vectors // uint8_t key128[]= { 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f }; uint8_t plain128[]= { 0x70, 0x6f, 0x6f, 0x6e, 0x65, 0x72, 0x2e, 0x20, 0x49, 0x6e, 0x20, 0x74, 0x68, 0x6f, 0x73, 0x65}; uint64_t cipher128[2] = {0x4eeeb48d9c188f43, 0x4109010405c0f53e}; #define R(v,n)(((v)>>(n))|((v)<<(64-(n)))) #define F(n)for(i=0;i<n;i++) typedef unsigned long long W; void speck128x(void*mk,void*in){ W i,t,k[4],r[2]; memcpy(r,in,16); memcpy(k,mk,32); F(34) r[1]=(R(r[1],8)+*r)^*k, *r=R(*r,61)^r[1], t=k[3], k[3]=(R(k[1],8)+*k)^i, *k=R(*k,61)^k[3], k[1]=k[2],k[2]=t; memcpy(in,r,16); } int main (void) { uint64_t buf[4]; int equ; // copy plain text to local buffer memcpy (buf, plain64, sizeof(plain64)); speck64(key64, buf); equ = memcmp(cipher64, buf, sizeof(cipher64))==0; printf ("\nSPECK64/128 encryption %s\n", equ ? "OK" : "FAILED"); print_bytes("CT result ", buf, sizeof(plain64)); print_bytes("CT expected", cipher64, sizeof(cipher64)); print_bytes("K ", key64, sizeof(key64)); print_bytes("PT", plain64, sizeof(plain64)); // copy plain text to local buffer memcpy (buf, plain128, sizeof(plain128)); #ifdef __USE_C_FUNCTION speck128x(key128, buf); #else speck128(key128, buf); #endif equ = memcmp(cipher128, buf, sizeof(cipher128))==0; printf ("\nSPECK128/256 encryption %s\n", equ ? "OK" : "FAILED"); print_bytes("CT result ", buf, sizeof(plain128)); print_bytes("CT expected", cipher128, sizeof(cipher128)); print_bytes("K ", key128, sizeof(key128)); print_bytes("PT", plain128, sizeof(plain128)); printf("I am "); #ifdef __ARM_ARCH_ISA_A64 printf(" __ARM_ARCH_ISA_A64 "); #endif #ifdef __arm64__ printf(" __arm64__ "); #endif #ifdef __x86_64__ printf(" __x86_64__ "); #endif #ifdef __linux__ printf(" __linux__ "); #endif #ifdef __APPLE__ printf(" __APPLE__ "); #endif printf("\n"); return 0; }

spk64.S   Select all

// SPECK64/128 in ARM64 assembly // SPECK-64/128 Block Cipher in x86 assembly (Encryption only) #ifdef __x86_64__ .intel_syntax noprefix #endif .text .globl speck64 .globl _speck64 #ifdef __ARM_ARCH_ISA_A64 .align 4 #endif // speck64(void*mk, void*data); speck64: _speck64: #if defined __arm64__ || defined __ARM_ARCH_ISA_A64 // load 128-bit key // k0 = k[0]; k1 = k[1]; k2 = k[2]; k3 = k[3]; ldp w5, w6, [x0] ldp w7, w8, [x0, 8] // load 64-bit plain text ldp w2, w4, [x1] // x0 = x[0]; x1 = k[1]; mov w3, wzr // i=0 #endif #ifdef __x86_64__ push rbx push rbp push rsi # save lodsd xchg eax, ebx # ebx = in[0] lodsd xchg eax, edx # edx = in[1] push rdi pop rsi lodsd xchg eax, edi # edi = key[0] lodsd xchg eax, ebp # ebp = key[1] lodsd xchg eax, ecx # ecx = key[2] lodsd xchg eax, esi # esi = key[3] xor eax, eax # i = 0 #endif L0: #if defined __arm64__ || defined __ARM_ARCH_ISA_A64 ror w2, w2, 8 add w2, w2, w4 // x0 = (R(x0, 8) + x1) ^ k0; eor w2, w2, w5 // eor w4, w2, w4, ror 29 // x1 = R(x1, 3) ^ x0; mov w9, w8 // backup k3 ror w6, w6, 8 add w8, w5, w6 // k3 = (R(k1, 8) + k0) ^ i; eor w8, w8, w3 // eor w5, w8, w5, ror 29 // k0 = R(k0, 3) ^ k3; mov w6, w7 // k1 = k2; mov w7, w9 // k2 = t; add w3, w3, 1 // i++; cmp w3, 27 // i < 27; bne L0 // save result stp w2, w4, [x1] // x[0] = x0; x[1] = x1; ret #endif #ifdef __x86_64__ # ebx = (ROTR32(ebx, 8) + edx) ^ edi; ror ebx, 8 add ebx, edx xor ebx, edi # edx = ROTR32(edx, 29) ^ ebx; ror edx, 29 xor edx, ebx # ebp = (ROTR32(ebp, 8) + edi) ^ i; ror ebp, 8 add ebp, edi xor ebp, eax # edi = ROTR32(edi, 29) ^ ebp; ror edi, 29 xor edi, ebp xchg esi, ecx xchg esi, ebp # i++ inc al cmp al, 27 jnz L0 pop rdi xchg eax, ebx stosd xchg eax, edx stosd pop rbp pop rbx ret #endif

spk128.S   Select all

// SPECK128/256 in ARM64 assembly // SPECK-128/256 Block Cipher in AMD64 assembly (Encryption only) #ifdef __x86_64__ .intel_syntax noprefix #endif .text .global speck128 .global _speck128 #ifdef __ARM_ARCH_ISA_A64 .align 4 #endif // speck128(void*mk, void*data); speck128: _speck128: #if defined __arm64__ || defined __ARM_ARCH_ISA_A64 // load 256-bit key // k0 = k[0]; k1 = k[1]; k2 = k[2]; k3 = k[3]; ldp x5, x6, [x0] ldp x7, x8, [x0, 16] // load 128-bit plain text ldp x2, x4, [x1] // x0 = x[0]; x1 = k[1]; mov x3, xzr // i=0 #endif #ifdef __x86_64__ push rbp push rbx push rdi push rsi # load 128-bit plaintext mov rbp, [rsi ] mov rsi, [rsi+8] # load 256-bit key mov rbx, [rdi ] # k0 mov rcx, [rdi+ 8] # k1 mov rdx, [rdi+16] # k2 mov rdi, [rdi+24] # k3 # i = 0 xor eax, eax #endif L0: #if defined __arm64__ || defined __ARM_ARCH_ISA_A64 ror x4, x4, 8 add x4, x4, x2 // x1 = (R(x1, 8) + x0) ^ k0; eor x4, x4, x5 // eor x2, x4, x2, ror 61 // x0 = R(x0, 61) ^ x1; mov x9, x8 // backup k3 ror x6, x6, 8 add x8, x5, x6 // k3 = (R(k1, 8) + k0) ^ i; eor x8, x8, x3 // eor x5, x8, x5, ror 61 // k0 = R(k0, 61) ^ k3; mov x6, x7 // k1 = k2; mov x7, x9 // k2 = t; add x3, x3, 1 // i++; cmp x3, 34 // i < 34; bne L0 // save result stp x2, x4, [x1] // x[0] = x0; x[1] = x1; ret #endif #ifdef __x86_64__ # x[1] = (R(x[1], 8) + x[0]) ^ k[0]; ror rsi, 8 add rsi, rbp xor rsi, rbx # x[0] = R(x[0], 61) ^ x[1]; ror rbp, 61 xor rbp, rsi # k[1] = (R(k[1], 8) + k[0]) ^ i; ror rcx, 8 add rcx, rbx xor cl, al # k[0] = R(k[0], 61) ^ k[3]; ror rbx, 61 xor rbx, rcx # X(k3, k2), X(k3, k1); xchg rdi, rdx xchg rdi, rcx # i++ inc al cmp al, 34 jnz L0 pop rax push rax # save 128-bit result mov [rax ], rbp mov [rax+8], rsi pop rsi pop rdi pop rbx pop rbp ret #endif

(2) Compile and Linking

shell script   Select all

# use clang to compile and link # To compile and link the above in Linux (e.g. tested in Android arm64 Termux App or Windows 10 WSL2 and clang package should be installed) clang callsum.c sum.S -o callsum; clang callfactorial.c factorial.S -o callfactorial; clang maxofthree.S callmaxofthree.c -o callmaxofthree; clang testckey.c ckey.S -o testckey; clang testspk.c spk64.S spk128.S -o testspk; #To compile and link the above in macOS (new M1 machine is capable to run x86_64 and arm64 binaries with Rosetta 2 installed). To compile on macOS, XCode, Command Line Utility and Rosetta 2 should be installed. clang callsum.c sum.S -o callsum_x86_64 -arch x86_64; clang callfactorial.c factorial.S -o callfactorial_x86_64 -arch x86_64; clang maxofthree.S callmaxofthree.c -o callmaxofthree_x86_64 -arch x86_64; clang testckey.c ckey.S -o testckey_x86_64 -arch x86_64; clang testspk.c spk64.S spk128.S -o testspk_x86_64 -arch x86_64; clang callsum.c sum.S -o callsum_arm64 -arch arm64; clang callfactorial.c factorial.S -o callfactorial_arm64 -arch arm64; clang maxofthree.S callmaxofthree.c -o callmaxofthree_arm64 -arch arm64; clang testckey.c ckey.S -o testckey_arm64 -arch arm64; clang testspk.c spk64.S spk128.S -o testspk_arm64 -arch arm64; In order to debug say using lldb, add -g option when compile and, in addition, macOS has to codesign with enttlements

(3) Summary of differences
3.1) In order to preprocess the assembler file using clang compiler, the filename extension should be capital letter S in linux. Subroutine name between C and global asm labels should prefix by underscore for macOS.
3.2) A64 (arm64) instruction set does not include an explicit stack push instruction. Functions can use the stp and ldp (load pair of registers) to carry out the push and pop operations as demo in factorial.S source code above.
3.3) Most Armv8-64 platforms (e.g. macOS) require quadword (16-byte) alignment of the SP register.
3.4) A64 (arm64) parameter/ results registers are X0-7.   X8 is designated as the Indirect Result Location Parameter and X30 (LR) is the Link Register. If the function has a return value, it will be stored in X0.  A64 (arm64) floating point result registers are S0 or D0  as demo in sum.S
3.5) x86_64 parameter registers for integer or pointer are %rdi. %rsi, %rdx, %rcx, %r8, %r9. If the function has a return value, it will be stored in %rax.  x86_64 floating point result registers are %xmm0.  as demo in sum.S
3.6) by using the directive .intel_syntax noprefix, the x86_64 intel syntax assembly code can be used where the first assembler operand usually is the destination operand where the order is similar to that of arm64 code. In addition the prefix % can be omitted when using noprefix.

(4) To download the above source code using command line
curl -L https://tinyurl.com/mixcasm | grep -A200 START_OF_CALLSUM.C | sed '1d' | sed -n "/END_OF_CALLSUM.C/q;p" | sed 's/&gt;/\>/g;s/&lt;/\</g' > callsum.c
curl -L https://tinyurl.com/mixcasm | grep -A200 START_OF_SUM.S | sed '1d' | sed -n "/END_OF_SUM.S/q;p" | sed 's/&gt;/\>/g;s/&lt;/\</g' > sum.S

curl -L https://tinyurl.com/mixcasm | grep -A200 START_OF_CALLFACTORIAL.C | sed '1d' | sed -n "/END_OF_CALLFACTORIAL.C/q;p" | sed 's/&gt;/\>/g;s/&lt;/\</g' > callfactorial.c
curl -L https://tinyurl.com/mixcasm | grep -A200 START_OF_FACTORIAL.S | sed '1d' | sed -n "/END_OF_FACTORIAL.S/q;p" | sed 's/&gt;/\>/g;s/&lt;/\</g' > factorial.S

curl -L https://tinyurl.com/mixcasm | grep -A200 START_OF_CALLMAXOFTHREE.C | sed '1d' | sed -n "/END_OF_CALLMAXOFTHREE.C/q;p" | sed 's/&gt;/\>/g;s/&lt;/\</g' > callmaxofthree.c
curl -L https://tinyurl.com/mixcasm | grep -A200 START_OF_MAXOFTHREE.S | sed '1d' | sed -n "/END_OF_MAXOFTHREE.S/q;p" | sed 's/&gt;/\>/g;s/&lt;/\</g' > maxofthree.S

curl -L https://tinyurl.com/mixcasm | grep -A200 START_OF_CHASKEY.H | sed '1d' | sed -n "/END_OF_CHASKEY.H/q;p" | sed 's/&gt;/\>/g;s/&lt;/\</g' > chaskey.h
curl -L https://tinyurl.com/mixcasm | grep -A200 START_OF_TESTCKEY.C | sed '1d' | sed -n "/END_OF_TESTCKEY.C/q;p" | sed 's/&gt;/\>/g;s/&lt;/\</g' > testckey.c
curl -L https://tinyurl.com/mixcasm | grep -A200 START_OF_CKEY.S | sed '1d' | sed -n "/END_OF_CKEY.S/q;p" | sed 's/&gt;/\>/g;s/&lt;/\</g' > ckey.S

curl -L https://tinyurl.com/mixcasm | grep -A200 START_OF_SPECK.H | sed '1d' | sed -n "/END_OF_SPECK.H/q;p" | sed 's/&gt;/\>/g;s/&lt;/\</g' > speck.h
curl -L https://tinyurl.com/mixcasm | grep -A200 START_OF_TESTSPK.C | sed '1d' | sed -n "/END_OF_TESTSPK.C/q;p" | sed 's/&gt;/\>/g;s/&lt;/\</g' > testspk.c
curl -L https://tinyurl.com/mixcasm | grep -A200 START_OF_SPK64.S | sed '1d' | sed -n "/END_OF_SPK64.S/q;p" | sed 's/&gt;/\>/g;s/&lt;/\</g' > spk64.S
curl -L https://tinyurl.com/mixcasm | grep -A200 START_OF_SPK128.S | sed '1d' | sed -n "/END_OF_SPK128.S/q;p" | sed 's/&gt;/\>/g;s/&lt;/\</g' > spk128.S

Differences of Mac and Linux Assembly Language

This example demo the differences of Assembly Language in Mac and Linux for x86_64 architecture. For arm64 assembly language differences, see these 2 codes in github. https://github.com/below/HelloSilicon and https://github.com/Apress/programming-with-64-bit-ARM-assembly-language

The assembly code has read and write functions. The differences of Linux and macOS are using the %ifdef MACHO64 which is the code for macOS %else is for Linux. For example the system call numbers are different and macho64 cannot use absolute addressing.

01_05-solution1.asm   Select all

; solution - answer to Chapter 1 challenge ; Read decimal input, add 10, generate output section .data prompt db "How old are you? ",0 result db "In 10 years, you will be ",0 newline db 10 ; newline, \n ten dq 10 ; used for multiplication size equ 4 ; buffer size azero equ 0x30 ; ASCII zero, '0' anine equ 0x39 ; ASCII nine, '9' nullchar equ 0x0 ; null terminator, \0 STDIN equ 0 ; standard input device STDOUT equ 1 ; standard output device %ifdef MACHO64 SYS_read equ 0x2000003 ; system call to read input macOS SYS_write equ 0x2000004 ; system call to write message macOS SYS_exit equ 0x2000001 ; system call to terminate program macOS %else SYS_read equ 0 ; system call to read input SYS_write equ 1 ; system call to write message SYS_exit equ 60 ; system call to terminate program %endif EXIT_OK equ 0 ; OK exit status TENS dq 10000 ; tens table dq 1000 dq 100 dq 10 dq 1 ; uninitialized data section .bss input_buf resb size ; age input buffer age resq 1 ; binary age value output_buf resb size ; modified age string ; code section .text global _start _start: ; output the prompt mov rsi, prompt ; prompt string output call print_string ; read input mov rbx, input_buf ; age string storage call input ; translate decimal string to binary mov rsi, input_buf ; where string value is stored call decimal2binary cmp al, 0 ; check for bogus input jz start_exit ; bail if so add rax, 10 ; add 10 to the value %ifdef MACHO64 mov [rel age], rax ; store result %else mov [age], rax ; store result %endif ; output the final prompt mov rsi, result ; First part of the string call print_string ; convert the value in 'age' to a string %ifdef MACHO64 mov rbx, [rel age] ; value %else mov rbx, [age] ; value %endif mov rsi, output_buf ; modified string call binary2decimal ; output the age-value string mov rsi, output_buf call print_value ; use this function to strip leading ; zeros ; exit program start_exit: mov rax, SYS_exit ; system exit call mov rdi, EXIT_OK syscall ; end of _start ; functions ;--------; ; output a null terminated string in rsi print_string: cmp byte [rsi], nullchar ; end-of-string test je print_string_exit ; bail on null char mov rdx, 1 ; number of characters to write mov rdi, STDOUT ; to standard output mov rax, SYS_write ; Write characters(s) syscall inc rsi ; next character jmp print_string ; keep looping print_string_exit: ret ;--------; ; grab standard input in the buffer in rbx input: mov r12, 1 ; buffer position read_char: mov rsi, rbx ; input char storage mov rdx, 1 ; character count mov rdi, STDIN ; from standard input mov rax, SYS_read ; read into rsi (rbx) syscall cmp byte [rbx], 10 ; is character read newline? je input_exit ; finish, don't store newline inc rbx ; next byte in the buffer inc r12 ; up the character count cmp r12, size ; check buffer size jl read_char ; keep looping if room ; otherwise, fall through: input_exit: mov byte [rbx], 0 ; cap the string ret ;--------; ; translate string input at rsi into binary value in rax ; if garbage input, returned value is zero decimal2binary: mov rax, 0 ; initial value d2b0: mov rbx, 0 ; initialize bax mov bl, byte [rsi] ; character, digit ; filter out non-digit values cmp bl, azero jl d2b_exit ; exit on character < '0' cmp bl, anine jg d2b_exit ; exit on character > '9' sub bl, azero ; convert from ASCII to binary %ifdef MACHO64 mul qword [rel ten] ; multiply rax by 10 %else mul qword [ten] ; multiply rax by 10 %endif add rax, rbx ; add new value inc rsi ; next char cmp byte [rsi], nullchar ; end of string? jnz d2b0 ; if not, keep looping d2b_exit: ret ;--------; ; generate a string at rsi representing the value in rbx binary2decimal: mov rdi, TENS ; reference comparision table ; values here are subtracted from ; rbx to calculate base 10 digits b2d0: xor al, al ; zero out al; al stores the character mov rcx, [rdi] ; get power of ten b2d1: or al, al ; clear carry bit (for jb) sub rbx, rcx ; subtract power of ten jb b2d2 ; if <0, the count in al is the value inc al ; decimal value++ jmp b2d1 ; keep subtracting b2d2: add al, azero ; make al ASCII add rbx, rcx ; recover from last subtraction mov byte [rsi], al ; add character to the string inc rsi ; next string position add rdi, 8 ; next value in TENS table cmp rcx, 1 ; end of table? jnz b2d0 ; loop again if not mov byte [rsi], nullchar ; terminate string ret ;--------; ; output a null-terminated string in rsi ; strip any leading zero characters print_value: cmp byte [rsi], nullchar ; always check for null char je print_value_exit ; and exit; string empty cmp byte [rsi], azero ; ASCII zero jne pv1 ; if the char isnt zero, continue inc rsi ; otherwise, check next character jmp print_value ; keep looping pv1: ; process the remaining non-zero digits cmp byte [rsi], nullchar ; null char terminator je print_value_exit ; if true, exit mov rdx, 1 ; chars to write mov rdi, STDOUT ; standard output mov rax, SYS_write ; write characters syscall inc rsi ; next char jmp pv1 ; loop until null char print_value_exit: mov rsi, newline ; newline defined as \n mov rdx, 1 ; write 1 char mov rax, SYS_write ; write character mov rdi, STDOUT ; to standard output syscall ret ; end

makefile for Linux

makefile   Select all

#makefile for Linux filename_prefix := 01_05- src_files := $(wildcard *.asm) obj_files := $(src_files:.asm=.o) prog_files := $(subst $(filename_prefix), ,$(basename $(src_files))) all: $(prog_files) $(prog_files): % : $(filename_prefix)%.o ld -o $@ $< -arch x86_64 $(filter %.o,$(obj_files)): %.o : %.asm nasm -g -f elf64 $< -o $@ .PHONY: clean clean: rm -f $(obj_files) $(prog_files)

makefile for macOS

makefile    Select all

#makefile for macOS filename_prefix := 01_05- src_files := $(wildcard *.asm) obj_files := $(src_files:.asm=.o) prog_files := $(subst $(filename_prefix), ,$(basename $(src_files))) all: $(prog_files) $(prog_files): % : $(filename_prefix)%.o ld -no_pie -macosx_version_min 11.0.0 -o $@ $< -lSystem -syslibroot `xcrun -sdk macosx --show-sdk-path` -e _start -arch x86_64 codesign --entitlement entitlements --force -s - $@ $(filter %.o,$(obj_files)): %.o : %.asm nasm -g -f macho64 -dMACHO64 $< -o $@ .PHONY: clean clean: rm -f $(obj_files) $(prog_files)

In order to debug in macOS, codesign entitlement file is needed.

entitlements   Select all

<?xml version="1.0" encoding="UTF-8"> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>com.apple.security.get-task-allow</key> <true/> </dict> </plist>

For linux, it is required to install these packages "sudo apt install build-essentials nasm gdb lldb"

For macOS, it is required to install Xcode and Command Line Tools and Homebrew package "brew install nasm"
It is possible to build, run and debug x86_64 program on M1 Macs, if Rosetta is installed.

How to install Swift 5 compiler for Android Termux App

(1) Installation of packages
pkg install clang
pkg install swift


(2) Test Grand Central Dispatch - concurrent queue

Shell script   Select all

cd $HOME cat >$HOME/concurrentqueue.swift <<'HEREEOF' /* # Concurrent Programming with Grand Central Dispatch in Swift 5 */ import Foundation let globalQueue = DispatchQueue.global(qos: .userInitiated) func trace(task: Int) { // Generates a random integer in the [0, task] range print("Task \(task) started") sleep( UInt32.random(in: 0...UInt32(task)) ) print("Task \(task) completed") } print("Concurrent queue, synchronous execution") for i in 1...5 { print("Submitting task \(i)") globalQueue.sync { trace(task: i) } } print("\nConcurrent queue, asynchronous execution") for i in 6...9 { print("Submitting task \(i)") globalQueue.async { trace(task: i) } } sleep(10) print("Program ended") HEREEOF # test swiftc concurrentqueue.swift ./concurrentqueue

(3) Test Grand Central Dispatch - serial queue

Shell script   Select all

cd $HOME cat >$HOME/serialqueue.swift <<'HEREEOF' /* # Concurrent Programming with Grand Central Dispatch in Swift 5 */ import Foundation let serialQueue = DispatchQueue(label: "com.mycompany.demo.serial") func trace(task: Int) { // Generates a random integer in the [0, task] range print("Task \(task) started") sleep( UInt32.random(in: 0...UInt32(task)) ) print("Task \(task) completed") } print("\nSerial queue, synchronous execution") for i in 1...5 { print("Submitting task \(i)") serialQueue.sync { trace(task: i) } } print("\nSerial queue, asynchronous execution") for i in 6...9 { print("Submitting task \(i)") serialQueue.async { trace(task: i) } } sleep(25) print("Program ended") HEREEOF # test swiftc serialqueue.swift ./serialqueue

How to use m1 Mac mini to do mining

(1) download m1 ethminer here
curl -OL https://github.com/gyf304/ethminer-m1/releases/download/v0.19.0-alpha.0-m1/ethminer-m1
chmod +x ethminer-m1

(2) Download a wallet app (e.g. Blockchain) and obtain Ether Wallet Address

(3) start to mine by joining pool
./ethminer-m1 -P stratum1+tcp://0x601913a35B0f5A599271506773ae25BF1858e92b@asia1.ethermine.org:4444

If you like this article pls consider Ethereum Donations:
0x601913a35B0f5A599271506773ae25BF1858e92b

How to cross compile using Docker Desktop experimental feature buildx

Another demo to show the building of packages using experiemtal feature of Docker desktop. Have to enable the experimental feature to cross compile in other architetcures different from the host machine. E.g. cross compile armv7 packages in amd64 or arm64 host of Mac or PC.

Shell script   Select all

cd $HOME mkdir -p my-quantlib cd my-quantlib # get helloworld.ipynb wget https://raw.githubusercontent.com/lballabio/dockerfiles/master/quantlib-jupyter/Hello%20world.ipynb cat >$HOME/my-quantlib/Dockerfile_ql_armv7_1.21 <<'HEREEOF' # Dockerfile_ql_armv7_1.21 # docker buildx build --platform linux/arm/v7 -f Dockerfile_ql_armv7_1.21 -t armv7/quantlib:1.21 . # Build Quantlib libraries for armv7 in arm64/amd64 host ARG tag=latest FROM arm32v7/ubuntu:18.04 LABEL Description="Provide a building environment where the QuantLib Python jupyter-notebook" RUN apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y build-essential wget libbz2-dev vim git ENV boost_version=1.67.0 ENV boost_dir=boost_1_67_0 # Build boost RUN echo 'Building boost ...' RUN wget --no-check-certificate https://boostorg.jfrog.i o/artifactory/main/release/${boost_version}/source/${boo st_dir}.tar.gz \ && tar xfz ${boost_dir}.tar.gz \ && rm ${boost_dir}.tar.gz \ && cd ${boost_dir} \ && ./bootstrap.sh \ && ./b2 --without-python --prefix=/usr -j 4 link=shared runtime-link=shared install \ && ./b2 --prefix=/Staging/usr install \ && cd .. && rm -rf ${boost_dir} && ldconfig # Build Quantlib C++ RUN echo 'Building Quantlib C++ ...' ENV quantlib_version=1.21 RUN wget https://github.com/lballabio/QuantLib/releases/download/QuantLib-v${quantlib_version}/QuantLib-${quantlib_version}.tar.gz \ && tar xfz QuantLib-${quantlib_version}.tar.gz \ && rm QuantLib-${quantlib_version}.tar.gz \ && cd QuantLib-${quantlib_version} \ && ./configure --prefix=/usr --disable-static CXXFLAGS=-O3 \ && make -j 4 && make install \ && make DESTDIR=/Staging install \ && make clean \ && cd .. && ldconfig # && cd .. && rm -rf QuantLib-${quantlib_version} && ldconfig # Build Quantlib-Python RUN echo 'Build Quantlib-Python ...' RUN apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y swig python3 python3-pip python-dev libgomp1 # Build Quantlib for Python3 RUN echo 'Install Quantlib Python' ENV quantlib_swig_version=1.21 RUN wget https://github.com/lballabio/QuantLib-SWIG/releases/download/QuantLib-SWIG-v${quantlib_swig_version}/QuantLib-SWIG-${quantlib_swig_version}.tar.gz \ && tar xfz QuantLib-SWIG-${quantlib_swig_version}.tar.gz \ && rm QuantLib-SWIG-${quantlib_swig_version}.tar.gz \ && cd QuantLib-SWIG-${quantlib_swig_version} \ && ./configure CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768" PYTHON=/usr/bin/python3 \ && make -C Python && make -C Python check && make -C Python install \ && cd .. && rm -rf QuantLib-SWIG-${quantlib_swig_version} && ldconfig # Build jupyter-notebook server RUN python3 -c "print('\033[91m Building jupyter-notebook server ... \033[0m')" RUN pip3 install --no-cache-dir jupyter jupyterlab matplotlib numpy scipy pandas ipywidgets RISE RUN jupyter-nbextension install rise --py --sys-prefix RUN jupyter-nbextension install widgetsnbextension --py --sys-prefix \ && jupyter-nbextension enable widgetsnbextension --py --sys-prefix # Build Quantlib for Python2 RUN apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y python \ && apt-get clean RUN wget https://bootstrap.pypa.io/2.7/get-pip.py \ && python2 get-pip.py \ && rm get-pip.py RUN wget https://dl.bintray.com/quantlib/releases/QuantLib-SWIG-${quantlib_swig_version}.tar.gz \ && tar xfz QuantLib-SWIG-${quantlib_swig_version}.tar.gz \ && rm QuantLib-SWIG-${quantlib_swig_version}.tar.gz \ && cd QuantLib-SWIG-${quantlib_swig_version} \ && ./configure CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768" \ && make -C Python && make -C Python check && make -C Python install \ && cd .. && rm -rf QuantLib-SWIG-${quantlib_swig_version} && ldconfig RUN pip2 install --no-cache-dir numpy EXPOSE 8888 RUN mkdir /notebooks VOLUME /notebooks COPY *.ipynb /notebooks/ # Starting jupyter-notebook server RUN python3 -c "print('\033[92m Starting jupyter-notebook server at port 8888 \033[0m')" CMD jupyter notebook --no-browser --allow-root --ip=0.0.0.0 --port=8888 --notebook-dir=/notebooks HEREEOF # build and load image (after experimental feature of Docker Desktop is enabled docker buildx build --platform linux/arm/v7 --memory="8g" --output "type=docker,push=false,dest=armv7_ql.tar" -f Dockerfile_ql_armv7_1.21 -t armv7/quantlib:1.21 . docker load < armv7_ql.tar # run image docker run --platform linux/arm/v7 -d -p 8888:8888 --name myquantlibarmv7testing armv7/quantlib:1.21 # list the token of the jupyter-notebook server docker container exec -it myquantlibarmv7testing jupyter notebook list

If using WSL2 docker CLI in Linux, should enable the experimental feature first.

export DOCKER_CLI_EXPERIMENTAL=enabled

docker buildx create --name mybuilder

docker buildx use mybuilder

docker buildx inspect --bootstrap

docker buildx build --platform linux/arm/v7 --memory="8g" --output "type=docker,push=false,dest=armv7_ql.tar" -f Dockerfile_ql_armv7_1.21 -t armv7/quantlib:1.21 .

How to cross compile QuantLib in docker

This example demo using dockcross to build Quantlib armv7 library packages and cross compile in host X86_64 machine

Shell script   Select all

cd $HOME mkdir -p dockcross cd dockcross cat >$HOME/dockcross/dockcross_ql_1.21 <<'HEREEOF' # dockcross_ql_1.21 # docker build -f dockcross_ql_1.21 -t dockcross/linux-armv7/quantlib:1.21 . # Build Quantlib libraries for armv7 ARG tag=latest FROM dockcross/linux-armv7 ENV DEFAULT_DOCKCROSS_IMAGE dockcross/linux-armv7/quantlib:1.21 ENV CROSS_PREFIX /usr/xcc/armv7-unknown-linux-gnueabi/armv7-unknown-linux-gnueabi/sysroot/usr LABEL Description="Provide a building environment dockcross/linux/armv7 for the QuantLib" RUN apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y build-essential wget vim git ENV boost_version=1.67.0 ENV boost_dir=boost_1_67_0 # Install boost dependencies RUN echo 'Building boost and required packages ...' RUN dpkg --add-architecture armhf \ && apt-get update \ && apt-get download libbz2-1.0:armhf libbz2-dev:armhf liblzma-dev:armhf \ && dpkg-deb -x libbz2-1*armhf.deb ./x \ && dpkg-deb -x libbz2-dev_*armhf.deb ./x \ && dpkg-deb -x liblzma-dev_*armhf.deb ./x \ && mv ./x/usr/lib/arm-linux-gnueabihf/* ${CROSS_PREFIX}/lib/ \ && mv ./x/usr/bin/* ${CROSS_PREFIX}/bin/ \ && mv ./x/usr/include/* ${CROSS_PREFIX}/include/ \ && mv ./x/usr/share/* ${CROSS_PREFIX}/share/ \ && mv ./x/lib/arm-linux-gnueabihf/* ${CROSS_PREFIX}/lib/ \ && rm -fr ./x *.deb && ldconfig RUN wget https://sourceforge.net/projects/libpng/files/zlib/1.2.11/zlib-1.2.11.tar.gz/download -O zlib-1.2.11.tar.gz \ && tar xfz zlib-1.2.11.tar.gz \ && rm zlib-1.2.11.tar.gz \ && cd zlib-1.2.11 \ && ./configure --prefix=${CROSS_PREFIX} && make && make install \ && cd .. && rm -rf zlib-1.2.11 && ldconfig # Build boost RUN wget https://dl.bintray.com/boostorg/release/${boost_version}/source/${boost_dir}.tar.gz \ && tar xfz ${boost_dir}.tar.gz \ && rm ${boost_dir}.tar.gz \ && cd ${boost_dir} \ && ./bootstrap.sh --with-toolset=gcc --prefix=${CROSS_PREFIX} \ && touch user-config.jam \ && echo "using gcc : armv7 : ${CXX} ;" > user-config.jam \ && echo "using mpi ;" >> user-config.jam \ && ./b2 --toolset=gcc-armv7 --address-model=32 --architecture=arm --user-config=./user-config.jam --without-python --prefix=${CROSS_PREFIX} -j 4 link=shared runtime-link=shared install \ && cd .. && rm -rf ${boost_dir} && ldconfig # Build Quantlib C++ RUN echo 'Building Quantlib C++ ...' ENV quantlib_version=1.21 RUN wget https://dl.bintray.com/quantlib/releases/QuantLib-${quantlib_version}.tar.gz \ && tar xfz QuantLib-${quantlib_version}.tar.gz \ && rm QuantLib-${quantlib_version}.tar.gz \ && cd QuantLib-${quantlib_version} \ && ./configure --host=x86_64-linux-gnu --target=armv7-unknown-linux-gnueabi --with-boost-include=${CROSS_PREFIX}/include/boost --with-boost-lib=${CROSS_PREFIX}/lib --prefix=${CROSS_PREFIX} --disable-static CXXFLAGS=-O3 \ && make -j 4 && make install \ && make clean \ && cd .. && ldconfig HEREEOF # build image docker build -f dockcross_ql_1.21 -t dockcross/linux-armv7/quantlib:1.21 . # Test dockcross compile docker run --rm dockcross/linux-armv7/quantlib:1.21 > dockcross-quantlib-armv7 chmod +x dockcross-quantlib-armv7 ./dockcross-quantlib-armv7 bash -c '$CXX -v' # Copy example cpp file from image docker run -v $PWD:/opt/mount --rm --entrypoint cp dockcross/linux-armv7/quantlib:1.21 /work/QuantLib-1.21/Examples/Bonds/Bonds.cpp /opt/mount/Bonds.cpp # Use this command to copy whole directory recursively # docker run -v $PWD:/opt/mount --rm --entrypoint cp dockcross/linux-armv7/quantlib:1.21 -r /work/QuantLib-1.21/Examples /opt/mount/QLExamples # Compile and check binary ./dockcross-quantlib-armv7 bash -c '$CXX Bonds.cpp -lQuantLib -o Bonds -static' file Bonds

# install qemu and run in host
sudo apt update
sudo apt install -y qemu-user
qemu-arm Bonds

# Or test run it in armv7 machine e.g. RaspberryPi 3B+
scp Bonds mypi3b:~/.
ssh mypi3b '~/Bonds'