Install options: select only the “Windows 10 SDK” (assuming the computer is Windows 10). Optionally, if you want to use MSVC cl.exe C/C++ compiler from the command line, additionally select the MSVC 2019 C++ build tools, which takes an additional 1.5 GB disk space. If you need VS2017 C++ Build Tools, this is another 2.5 GB disk space.
Windows Python needs Visual C++ libraries installed via the SDK to build code, such as via setuptools.extension.Extension or numpy.distutils.core.Extension.
For example, building
modules in Windows with Python requires Visual C++ SDK as installed above.
On Linux and Mac, the C++ libraries are installed with the compiler.
This was the former link to Microsoft Visual C++ Build Tools–no longer active:
Travis-CI switches to
default Ubuntu 16.04
as of April 23, 2019.
This change was long overdue, but at least didn’t take as long as their default switch to Ubuntu 14.04 from 12.04, which was almost scandalous in how long it took.
Travis-CI has many strengths, but one of its weaknesses is their out of date Ubuntu configurations.
We have been doing some tasks in AppVeyor alone for some time, since
Ubuntu 18.04 has been available on AppVeyorimage: ubuntu1804 since September 2018.
Dozens of “free” compilers exist for C, C++ and Fortran.
However, only five compiler families available at no charge support the modern features of these popular compiled languages.
This list covers four compiler families available at no charge that support most of:
GCC has broad support of modern standards on a very wide range of computing platforms.
GCC’s downside in some cases can be slower runtime performance than compilers having less broad language and platform support.
IBM XL compilers are currently for POWER CPUs only e.g. ppc64le.
IBM XL compilers do not work with a typical x86-based computer.
If you have a $2000 Raptor IBM POWER9 desktop, then IBM XL may be for you.
The IBM XL compilers are high-performance compilers that have a free community edition.
IBM XL Fortran has wide support for Fortran 2008.
The procedure described here is for a new user to Docker.
There are certainly more efficient ways to do these tasks that are perhaps less easy for less experienced Docker users.
Please let us know if you have a suggested improvement suitable for novice Docker users.
Install Docker image
add docker group and yourself to this group, to avoid needing sudo for every docker command:
adduser $(whoami) docker
then reboot (not just logout, an actual system restart)
install Docker on your laptop (the example here is for a Linux laptop)
snap install docker
after this step, sudo should no longer be required.
try the Hello World images, which should auto-download and print a welcome message
docker run hello-world
Search for a desired image.
Consider those images listed as “official”.
You might need to widen your terminal to 155 columns or so to render properly.
Let’s use ubuntu as it’s quite popular for Docker and in general.
docker search ubuntu
get the desired image
docker pull ubuntu
verify the images on your computer
ubuntu takes just under 100 MB to start.
start the Docker container on your laptop
docker run -it ubuntu
where -it means interactive session
verify the version running, it will have a # superuser prompt:
These commands are issued NOT from within the Docker container itself (open a new terminal)
where are containers stored: docker info | grep "Root Dir"
How much space is Docker using altogether: du -sh /var/snap/docker
use previous command to find where your Docker stuff is
list images: docker images -a
list containers (running and stopped): docker ps -a
stop a Docker container: docker stop container_name
start a Docker container: docker start container_name
login to a running Docker container: docker exec -it container_name bash
get container environment variables: docker exec container_name env
cleanup unused containers docker system prune
delete image: docker image rm ubuntu:19.04
Configure Docker image
At least through April 2019, Travis-CI only had up to Ubuntu 16.04 dist: xenial, which does not have OpenCoarrays or other contemporary software that’s in Ubuntu 18.04 Bionic.
Travis-CI (and countless other online, server and computer resources) can use Docker images.
For research reproducability, a Docker image gives a known software state that should be usable for many years.
Interactive setup method tends to make the final image larger than using RUN commands.
So let’s start off by using a Dockerfile as that helps maintain smaller images with repeatable builds.
The Docker container can be turned into a new image that is usable from Travis-CI or any other suitable Docker host.
We do not claim this Dockerfile to be optimal.
Please let us know of suggested improvements.
# based on https://github.com/naftulikay/docker-bionic-vm/blob/master/DockerfileFROM ubuntu:18.04ENV container=docker TERM=xterm LC_ALL=en_US LANGUAGE=en_US LANG=en_US.UTF-8# stop some annoying interactive prompts, also need "-yq" on apt-getENV DEBIAN_FRONTEND=noninteractive# localeRUN apt-get update -q > /dev/null &&\
apt-get install --no-install-recommends -yq apt-utils locales language-pack-en dialog > /dev/null &&\
locale-gen $LANGUAGE $LANG# add sudo commmandRUN apt-get -yq install sudo > /dev/null# create and switch to a non-priviliged (but sudo-enabled) user, arbitrary nameRUN echo "nonprivuser ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoersRUN useradd --no-log-init --home-dir /home/nonprivuser --create-home --shell /bin/bash nonprivuser && adduser nonprivuser sudoUSER nonprivuserWORKDIR /home/nonprivuser# Git/Curl -- don't disable recommends here or you won't have Certification Authority certificates and will failRUN sudo apt-get install -yq git curl > /dev/null# packages specific to your needsRUN sudo apt-get install --no-install-recommends -yq make gfortran libcoarrays-dev libopenmpi-dev open-coarrays-bin > /dev/null &&\
sudo apt-get clean -q# latest cmakeRUN git clone --depth 1 https://github.com/scivision/cmake-utils &&\
mkdir -v /home/nonprivuser/.local &&\
cd cmake-utils && PREFIX=/home/nonprivuser/.local ./cmake_setup.sh > /dev/null &&\
mv -v /home/nonprivuser/.local/cmake* /home/nonprivuser/.local/cmakeENV PATH=$PATH:/home/nonprivuser/.local/cmake/bin# other optional installs# RUN sudo apt-get install --no-install-recommends -yq octave
create the file above named Dockerfile.
build the Docker image:
docker build -t opencoarrays_fortran .
assuming you’re in the same directory as Dockerfile
check the image exists, for me it was about 350 MB:
Before committing for upload, you must invoke the container.
docker run opencoarrays_fortran
This will almost immediately start and stop, as you didn’t give it a command to run or persist.
get the hexadecimal container ID by:
docker ps -a
the container will have an auto-assigned name like foo_bar.
Note the hexadecimal “container ID”.
upload Docker image
Once you have configured a Docker container on your laptop suitable for your purposes, you may wish to share this container and use it on other hosts such as Travis-CI.
This can be done for free with DockerHub.
Once you are ready to upload the image, note the “container ID”, which is the hexadecimal number at the terminal prompt of the Docker container, under docker ps.
The container hex ID must appear under docker ps, just being in docker images is not enough.
The changes to the image the container made are gathered into a new image.
It may take a minute or two if your image is large.
Ideally with a small image it will take only a couple seconds.
The new image has not yet left your computer, it will show up under
Once uploaded, your Docker image is visible to the world by default.
If your image is very large > 1 GB, be aware it will take a while to upload, and for the CI system to download.
This is a telltale that it’s important to keep Docker images small.
Setup Travis-CI to use this image
env: Dimg=scivision/opencoarrays_fortran; Dname=fortran-ubuntu
# Travis UID does not necessarily match 1000:1000 of Docker FIXME will Docker always be 1000:1000
- sudo chown -R 1000:1000 $TRAVIS_BUILD_DIR
- docker run -d --name $Dname -v $TRAVIS_BUILD_DIR:/home/nonprivuser/travis -w /home/nonprivuser/travis $Dimg tail -f /dev/null# prepend $Dcmd to run commands inside the Docker container
- export Dcmd="docker exec -t $Dname bash -c"
# for this example, I already installed everything I need in the image previously created by the Dockerfile
# all commands must be quoted
- $Dcmd "cd build && cmake .. && cmake --build . && ctest -V"
each docker exec command is a new shell instance. So changing directories in one docker exec has no effect on subsequent commands for example. Note the CMake one-liner in .travis.yml that configures, builds and tests in one command.
One can very rapidly find files with arbitrary criteria on systems with
This includes Linux, MacOS and Windows Subsystem for Linux.
Just about any criteria one could think of can be used to rapidly find files.
If working on a remote filesystem mounted over
we suggest SSHing into that system and running the find command that–it will be orders of magnitude faster.
Most examples use home directory ~ as a starting point just for convenience.
Appending 2>/dev/null to the end of a command removes nuisance messages about file permissions.
If piping the find command, put 2>/dev/null before the pipe.
Find files with “report” in the filename, case-insensitive
find ~ -iname "*report*"
Symbolic soft links
Suppose ~/data is a symbolic link to another directory, either on your computer, a USB hard drive or a server.
By default, find will not search this resource unless you
“resolve” the symbolic link
to a directory by putting a final / like ~/data/:
find ~/data/ -iname "*report*"
symbolic link section for more details, in particular the -H and -L options.
Compiling with Intel compilers on Windows is a distinctive task from MacOS or Linux.
The Intel C, C++ and Fortran compilers masquerade as Visual Studio compilers, especially in their command line options.
For the Intel Fortran compiler on Windows, the build options one normally uses for Intel Fortran on MacOS and Linux do not work, and even error out the compiler.
For C and C++, one must use the icl compiler (there is no icc or icpc on Windows) as if it were MSVC.
That can be perplexing at first for those not coming from the MSVC world.
The scientific computing world and most software engineering tasks not originating in the Microsoft domain use GCC-like compilers, which have completely different sets of compiler options.
Loading Intel compiler environment
It’s important that you load the compilervars.bat script to enable the Intel compilers for each session and NOT the psxevars.bat.
For convenience, make a batch script like %HOMEPATH%\intel.bat containing:
C:\"Program Files (x86)"\IntelSWTools\parallel_studio_xe_2019\compilers_and_libraries_2019\windows\bin\compilervars.bat intel64`set FC=ifort
Another distinctive feature of using MPI on Windows is that a username and password are required, even on a simple standalone laptop.
Good security practices may include creating a separate unprivileged user account that is only used for MPI jobs.
If this is suitable for your security environment, run MPI jobs in that other user account by from Command Prompt:
runas /user:mympiusername cmd
This opens a new Command Prompt window.
The environment variables are not passed to this new windows, so you may need to run Intel compilervars.bat again.
the user credential into the Windows registry, if appropriate for your environment.
If doing so, we would again urge to consider using a separate user account that’s only used for MPI runs.
If you don’t sign in, errors will be like:
Unable to manage jobs using credentials with a blank password. Please enter another account.