Lossless convert PNG image stack to PDF

Converting an image to a PDF file with ImageMagick can be as simple as:

convert in.png out.pdf

For single images -page letter can leave big margins, which may not be desirable.

convert image stack to PDF

This ImageMagick command puts one PNG image per page

convert -page letter -adjoin *.png joined.pdf
-page letter
does not upsize images, so tiny images will be tiny on the page, one image per page.

Print PDF from any Linux program

Related: convert image stack to PDF


On Linux, the CUPS PDF program saves printed PDFs to the ~/PDF directory from any program.

apt install cups-pdf

The output directory can be configured by editing the “Out” directory in /etc/cups/cups-pdf.conf

However, many times we print from Matlab, GNU Octave or Matplotlib using the APIs built-in as described below.

Matlab / GNU Octave

Example to save figure to PDF: this will create a color PDF test.pdf in your current directory from Matlab or Octave.

imagesc(rand(10))

print(gcf, '-dpdf', 'test.pdf')

or

imagesc(rand(10))

saveas(gcf, 'test.pdf')

Comparing print() vs. saveas():

  • print() allows controlling image DPI via the -rDPI option, while saveas() is fixed at 150 DPI
  • saveas() is an “easier” version of print(), since saveas() is actually making calls to print() for both Matlab and Octave.

Matplotlib

This can be accomplished by either

  • save to PDF from the Matplotlib figure GUI
  • use figure.savefig() to save to PDF from Matplotlib.

For situations where local display of figures is not desired, that is, you wish to save figures to disk without displaying them onscreen first, which is generally significantly faster, first do:

import matplotlib
matplotlib.use('Agg')

... (plotting commands)

The Agg Matplotlib backend is commonly used in web/cloud services.

Example:

from matplotlib.pyplot import figure

fg = figure()
ax = fg.gca()
ax.plot(range(10))

fg.savefig('line.pdf',bbox_inches='tight',dpi=300)

Fix Python 3 on Windows error Microsoft Visual C++ 14.0 is required

Related: Fix Python 2 error Visual C++ 10.0 missing vcvarsall.bat


Fix the error for Python 3.6, 3.7 and 3.8 on Windows:

error Microsoft Visual C++ 14.0 is required

as follows. Visual Studio Studio 2019 Build Tools requires about 3 GB of disk space.

  1. Install using any ONE of these choices:
  2. Select: Workloads → C++ build tools.
  3. Install options: select only the “Windows 10 SDK” (assuming the computer is Windows 10). Optionally, if you want to use MSVC cl.exe C/C++ compiler from the command line, additionally select the MSVC 2019 C++ build tools, which takes an additional 1.5 GB disk space. If you need VS2017 C++ Build Tools, this is another 2.5 GB disk space.

Notes

Python versionVisual Studio
2.7, 3.42010
3.52015 / 2017
3.6, 3.7, 3.82017 / 2019

Why Visual C++?

Windows Python needs Visual C++ libraries installed via the SDK to build code, such as via setuptools.extension.Extension or numpy.distutils.core.Extension. For example, building f2py modules in Windows with Python requires Visual C++ SDK as installed above. On Linux and Mac, the C++ libraries are installed with the compiler.

Reference

This was the former link to Microsoft Visual C++ Build Tools–no longer active: https://landinghub.visualstudio.com/visual-cpp-build-tools

Travis-CI default Ubuntu version

Travis-CI switches to default Ubuntu 16.04 as of April 23, 2019. This change was long overdue, but at least didn’t take as long as their default switch to Ubuntu 14.04 from 12.04, which was almost scandalous in how long it took. Travis-CI has many strengths, but one of its weaknesses is their out of date Ubuntu configurations. We have been doing some tasks in AppVeyor alone for some time, since Ubuntu 18.04 has been available on AppVeyor image: ubuntu1804 since September 2018.

Also keep in mind that Travis-CI defaults to Python 3.6 for lang: python builds since April 16, 2019.

OpenMPI 3 Fortran 2008 interface update

In legacy Fortran MPI programs, near the top of the procedure we would have:

use mpi

or

include 'mpi.h'

It is typically recommended to use the OpenMPI Fortran 2008 interface:

use mpi_f08

Intel MPI

Intel MPI supports MPI Fortran 2008 use mpi_f08 since Intel 16.0 EXCEPT for Windows Intel MPI, where even Intel 19.0 did not have mpi_f08.

Interfacing MPI 3 with legacy Fortran code

You will notice for example that constants like mpi_comm_world and mpi_real are no longer integer data type, but rather custom types.

Many legacy and even current libraries (such as MUMPS 5) have not yet updated to be polymorphic for this enhanced variable type.

Here is a minimal working example for MUMPS using the backwards compatible OpenMPI 3 interface, grabbing the integer via the %mpi_val property.

program mumps_mpi3

use mpi_f08
include 'dmumps_struc.h'  ! per MUMPS 5 manual

type(dmumps_struc) mumps_par

mumps_par%comm = mpi_comm_world%mpi_val   ! %mpi_var emits the legacy integer


end program

Notes

OpenMPI 3

Five free C C++ Fortran compiler families

Dozens of “free” compilers exist for C, C++ and Fortran. However, only five compiler families available at no charge support the modern features of these popular compiled languages. This list covers four compiler families available at no charge that support most of:

  • C17
  • C++17
  • Fortran 2008

GCC

GCC has broad support of modern standards on a very wide range of computing platforms. GCC’s downside in some cases can be slower runtime performance than compilers having less broad language and platform support.

LLVM

LLVM Clang and Flang have significant industry support, including from Nvidia, and are known for high performance, but somewhat less language feature support and less broad platform support than GCC.

PGI

PGI Community Edition is free to use, and has performance comparable to Intel compilers in many cases. PGI ≥ 19 should be used for modern Fortran. PGI supports CUDA Fortran.

Intel

Intel compilers are a paid product in general. For non-commercial purposes, including:

  • educators
  • students
  • open-source projects

the Intel compilers are available at no charge. The Intel performance libraries like MKL, IPP, TBB and more are available at no cost with more liberal use terms.

IBM XL

IBM XL compilers are currently for POWER CPUs only e.g. ppc64le. IBM XL compilers do not work with a typical x86-based computer. If you have a $2000 Raptor IBM POWER9 desktop, then IBM XL may be for you.

The IBM XL compilers are high-performance compilers that have a free community edition. IBM XL Fortran has wide support for Fortran 2008.

Setup Docker Fortran image for Travis-CI

The procedure described here is for a new user to Docker. There are certainly more efficient ways to do these tasks that are perhaps less easy for less experienced Docker users. Please let us know if you have a suggested improvement suitable for novice Docker users.

Install Docker image

  1. add docker group and yourself to this group, to avoid needing sudo for every docker command:

    addgroup docker
    
    adduser $(whoami) docker

    then reboot (not just logout, an actual system restart)

  2. install Docker on your laptop (the example here is for a Linux laptop)

    snap install docker

    after this step, sudo should no longer be required.

  3. try the Hello World images, which should auto-download and print a welcome message

    docker run hello-world
  4. Search for a desired image. Consider those images listed as “official”. You might need to widen your terminal to 155 columns or so to render properly. Let’s use ubuntu as it’s quite popular for Docker and in general.

    docker search ubuntu
  5. get the desired image

    docker pull ubuntu
  6. verify the images on your computer

    docker images

    ubuntu takes just under 100 MB to start.

  7. start the Docker container on your laptop

    docker run -it ubuntu

    where -it means interactive session verify the version running, it will have a # superuser prompt:

    cat /etc/lsb-release

manage containers

These commands are issued NOT from within the Docker container itself (open a new terminal)

  • where are containers stored: docker info | grep "Root Dir"
  • How much space is Docker using altogether: du -sh /var/snap/docker
    • use previous command to find where your Docker stuff is
  • list images: docker images -a
  • list containers (running and stopped): docker ps -a
  • stop a Docker container: docker stop container_name
  • start a Docker container: docker start container_name
  • login to a running Docker container: docker exec -it container_name bash

  • get container environment variables: docker exec container_name env

  • cleanup unused containers docker system prune

  • delete image: docker image rm ubuntu:19.04

Configure Docker image

At least through April 2019, Travis-CI only had up to Ubuntu 16.04 dist: xenial, which does not have OpenCoarrays or other contemporary software that’s in Ubuntu 18.04 Bionic. Travis-CI (and countless other online, server and computer resources) can use Docker images. For research reproducability, a Docker image gives a known software state that should be usable for many years.

Interactive setup method tends to make the final image larger than using RUN commands. So let’s start off by using a Dockerfile as that helps maintain smaller images with repeatable builds. The Docker container can be turned into a new image that is usable from Travis-CI or any other suitable Docker host.

We do not claim this Dockerfile to be optimal. Please let us know of suggested improvements.

# based on https://github.com/naftulikay/docker-bionic-vm/blob/master/Dockerfile

FROM ubuntu:18.04
ENV container=docker TERM=xterm LC_ALL=en_US LANGUAGE=en_US LANG=en_US.UTF-8

# stop some annoying interactive prompts, also need "-yq" on apt-get
ENV DEBIAN_FRONTEND=noninteractive

# locale
RUN apt-get update -q > /dev/null && \
  apt-get install --no-install-recommends -yq apt-utils locales language-pack-en dialog > /dev/null && \
  locale-gen $LANGUAGE $LANG

# add sudo commmand
RUN apt-get -yq install sudo > /dev/null

# create and switch to a non-priviliged (but sudo-enabled) user, arbitrary name
RUN echo "nonprivuser ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
RUN useradd --no-log-init --home-dir /home/nonprivuser --create-home --shell /bin/bash nonprivuser && adduser nonprivuser sudo
USER nonprivuser
WORKDIR /home/nonprivuser

# Git/Curl -- don't disable recommends here or you won't have Certification Authority certificates and will fail
RUN sudo apt-get install -yq git curl > /dev/null

# packages specific to your needs
RUN sudo apt-get install --no-install-recommends -yq make gfortran libcoarrays-dev libopenmpi-dev open-coarrays-bin > /dev/null && \
  sudo apt-get clean -q

# latest cmake
RUN git clone --depth 1 https://github.com/scivision/cmake-utils && \
  mkdir -v /home/nonprivuser/.local && \
  cd cmake-utils && PREFIX=/home/nonprivuser/.local ./cmake_setup.sh > /dev/null  && \
  mv -v /home/nonprivuser/.local/cmake* /home/nonprivuser/.local/cmake

ENV PATH=$PATH:/home/nonprivuser/.local/cmake/bin

# other optional installs

# RUN sudo apt-get install --no-install-recommends -yq octave
  1. create the file above named Dockerfile.
  2. build the Docker image:

    docker build -t opencoarrays_fortran .

    assuming you’re in the same directory as Dockerfile

  3. check the image exists, for me it was about 350 MB:

    docker images
  4. Before committing for upload, you must invoke the container.

    docker run opencoarrays_fortran

    This will almost immediately start and stop, as you didn’t give it a command to run or persist.

  5. get the hexadecimal container ID by:

    docker ps -a

    the container will have an auto-assigned name like foo_bar. Note the hexadecimal “container ID”.

upload Docker image

Once you have configured a Docker container on your laptop suitable for your purposes, you may wish to share this container and use it on other hosts such as Travis-CI. This can be done for free with DockerHub.

Once you are ready to upload the image, note the “container ID”, which is the hexadecimal number at the terminal prompt of the Docker container, under docker ps. The container hex ID must appear under docker ps, just being in docker images is not enough.

docker commit -m "basic fortran OpenCoarrays setup" hex_id dockerhub_username/opencoarrays_fortran

The changes to the image the container made are gathered into a new image. It may take a minute or two if your image is large. Ideally with a small image it will take only a couple seconds. The new image has not yet left your computer, it will show up under

docker images

Once uploaded, your Docker image is visible to the world by default. {: .alert-box.warning}

  1. Login to DockerHub

    docker login -u dockerhub_username
  2. Push (upload) this image. Note this image is world-visible by default!

    docker push dockerhub_username/opencoarrays_fortran

If your image is very large > 1 GB, be aware it will take a while to upload, and for the CI system to download. This is a telltale that it’s important to keep Docker images small.

Setup Travis-CI to use this image

Create .travis.yml:

language: minimal

services: docker

git:
  depth: 3
  quiet: true

env: Dimg=scivision/opencoarrays_fortran; Dname=fortran-ubuntu

before_install:
# Travis UID does not necessarily match 1000:1000 of Docker FIXME will Docker always be 1000:1000
- sudo chown -R 1000:1000 $TRAVIS_BUILD_DIR

- docker run -d --name $Dname -v $TRAVIS_BUILD_DIR:/home/nonprivuser/travis -w /home/nonprivuser/travis $Dimg tail -f /dev/null

# prepend $Dcmd to run commands inside the Docker container
- export Dcmd="docker exec -t $Dname bash -c"

install:
# for this example, I already installed everything I need in the image previously created by the Dockerfile

script:
# all commands must be quoted
- $Dcmd "cd build && cmake .. && cmake --build . && ctest -V"

Notes

  • each docker exec command is a new shell instance. So changing directories in one docker exec has no effect on subsequent commands for example. Note the CMake one-liner in .travis.yml that configures, builds and tests in one command.

References

Find files from the command line

One can very rapidly find files with arbitrary criteria on systems with GNU Findutils. This includes Linux, MacOS and Windows Subsystem for Linux.

Examples

Just about any criteria one could think of can be used to rapidly find files. If working on a remote filesystem mounted over SSHFS, we suggest SSHing into that system and running the find command that–it will be orders of magnitude faster.

Most examples use home directory ~ as a starting point just for convenience. Appending 2>/dev/null to the end of a command removes nuisance messages about file permissions. If piping the find command, put 2>/dev/null before the pipe.

Find files with “report” in the filename, case-insensitive

find ~ -iname "*report*"

Suppose ~/data is a symbolic link to another directory, either on your computer, a USB hard drive or a server. By default, find will not search this resource unless you “resolve” the symbolic link to a directory by putting a final / like ~/data/:

find ~/data/ -iname "*report*"

See the findutils manual symbolic link section for more details, in particular the -H and -L options.

Intel MPI on Windows

Compiling with Intel compilers on Windows is a distinctive task from MacOS or Linux. The Intel C, C++ and Fortran compilers masquerade as Visual Studio compilers, especially in their command line options. For the Intel Fortran compiler on Windows, the build options one normally uses for Intel Fortran on MacOS and Linux do not work, and even error out the compiler. For C and C++, one must use the icl compiler (there is no icc or icpc on Windows) as if it were MSVC. That can be perplexing at first for those not coming from the MSVC world. The scientific computing world and most software engineering tasks not originating in the Microsoft domain use GCC-like compilers, which have completely different sets of compiler options.

Loading Intel compiler environment

It’s important that you load the compilervars.bat script to enable the Intel compilers for each session and NOT the psxevars.bat. For convenience, make a batch script like %HOMEPATH%\intel.bat containing:

C:\"Program Files (x86)"\IntelSWTools\parallel_studio_xe_2019\compilers_and_libraries_2019\windows\bin\compilervars.bat intel64`

set FC=ifort
set CC=icl
set CXX=icl

MPI

Another distinctive feature of using MPI on Windows is that a username and password are required, even on a simple standalone laptop. Good security practices may include creating a separate unprivileged user account that is only used for MPI jobs. If this is suitable for your security environment, run MPI jobs in that other user account by from Command Prompt:

runas /user:mympiusername cmd

This opens a new Command Prompt window. The environment variables are not passed to this new windows, so you may need to run Intel compilervars.bat again.

You can register the user credential into the Windows registry, if appropriate for your environment. If doing so, we would again urge to consider using a separate user account that’s only used for MPI runs.

If you don’t sign in, errors will be like:

Unable to manage jobs using credentials with a blank password. Please enter another account.

Intel MPI on Windows is only for Intel compiler

Unlike for Linux Intel MPI that may be used with:

  • GNU gcc and gfortran out of the box
  • PGI via $MKLROOT/../mpi/binding/intel-mpi-binding-kit.tar.gz

Windows Intel MPI is only for the Intel Fortran compiler and none other at this time.

MPI 2008

Intel 19 MPI does not support use mpi_f08 the Fortran 2008 bindings. To keep compatibility with Intel MPI on Windows, at this time one would consider the legacy Fortran MPI interface:

use mpi

call mpi_init(ierr)

call mpi_finalize(ierr)

end program

The Fortran 77 include mpif.h should not be used in current Fortran programs as it breaks the fundamental Fortran 90 paradigm of modularity.