Compilation Guide

DFTBaby includes Fortran extensions for performance-critical calculations. This guide explains how to compile them.

Overview

DFTBaby has three main Fortran extension modules:

1. Core Extensions (DFTB/extensions/) - Essential for most calculations

2. Force Field (DFTB/ForceField/src/) - Required for QM/MM simulations

3. Multiple Scattering (DFTB/MultipleScattering/src/) - Advanced electronic structure methods

Prerequisites

Required Tools

• Python 3.12+ with NumPy 2.0+

• Fortran compiler: gfortran (recommended) or Intel ifort

• f2py: Included with NumPy

• BLAS/LAPACK: Linear algebra libraries

• make: Build automation tool

Install on Ubuntu/Debian

sudo apt-get update
sudo apt-get install gfortran libblas-dev liblapack-dev make

Install on macOS

brew install gcc make openblas lapack

Install on Fedora/RHEL

sudo dnf install gcc-gfortran blas-devel lapack-devel make

Install in Conda Environment (Recommended for Python 3.12)

IMPORTANT: If you’re using conda with Python 3.12 and NumPy 2.0+, use conda-forge packages to avoid meson/BLAS linking issues.

# Create conda environment
conda create -n dftbaby python=3.12 -y
conda activate dftbaby

# Install all dependencies
conda install -c conda-forge numpy scipy openblas liblapack gfortran make

# Verify installation
python3 -c "import numpy; print(f'NumPy {numpy.__version__}')"
gfortran --version

For conda users experiencing BLAS linking errors, see the detailed troubleshooting guide: ➜ :doc:`compilation_fixes` (Error 7: Meson backend BLAS linking)

Verify Installation

Check that required tools are available:

# Check gfortran
gfortran --version

# Check f2py
python3 -c "import numpy.f2py; print(numpy.f2py.__version__)"

# Check make
make --version

Core Extensions (Essential)

These extensions are required for most DFTBaby calculations.

Location

DFTB/extensions/

Modules Compiled

• thomson.so - Thomson charge distribution

• tddftb.so - TD-DFTB calculations

• mulliken.so - Mulliken population analysis

• slako.so - Slater-Koster transformations

• grad.so - Analytical gradients

• cosmo.so - COSMO solvation model

Compilation Steps

Method 1: Simple compilation (GNU gfortran)

For system Python or conda with working BLAS:

cd DFTB/extensions/
make clean
make

This compiles all extensions with default settings (GNU compilers, OpenMP parallelization).

Method 1b: Conda environment (if regular Makefile fails)

If you’re using conda and get cannot find -lblas errors, use the conda-specific Makefile:

cd DFTB/extensions/

# Use Makefile.conda which lets meson auto-detect libraries
make -f Makefile.conda clean
make -f Makefile.conda

# Or create a symlink to use regular make commands
mv Makefile Makefile.system
ln -s Makefile.conda Makefile
make clean && make

Why use Makefile.conda?

• NumPy 2.0+ uses meson/ninja build backend (not distutils)

• Meson handles library linking differently - explicit -lblas -lgomp may fail

• Makefile.conda removes explicit library flags and lets meson auto-detect

• Works with conda-installed BLAS/LAPACK packages

Method 2: Intel compilers (for HPC clusters)

If you have Intel compilers and MKL:

cd DFTB/extensions/

# Load Intel modules (on HPC systems)
module load compiler/intel/icc
source $(dirname $(which icc))/compilervars.sh intel64
source $(dirname $(which icc))/../mkl/bin/mklvars.sh intel64

# Edit Makefile: change SYS=GNU to SYS=INTEL
sed -i 's/SYS=GNU/SYS=INTEL/' Makefile

# Compile
make clean
make

Method 3: Manual f2py (if make fails)

cd DFTB/extensions/

# Compile each module individually
python3 -m numpy.f2py -c thomson.f90 -m thomson --fcompiler=gfortran --opt="-O3"
python3 -m numpy.f2py -c tddftb.f90 -m tddftb --fcompiler=gfortran --opt="-O3"
python3 -m numpy.f2py -c mulliken.f90 -m mulliken --fcompiler=gfortran --opt="-O3"
python3 -m numpy.f2py -c slako.f90 -m slako --fcompiler=gfortran --opt="-O3"
python3 -m numpy.f2py -c grad.f90 -m grad --fcompiler=gfortran --opt="-O3"
python3 -m numpy.f2py -c cosmo.f90 -m cosmo --fcompiler=gfortran --opt="-O3"

Verification

Test that modules can be imported:

cd DFTB/extensions/
python3 << 'EOF'
import thomson
import tddftb
import mulliken
import slako
import grad
import cosmo
print("✓ All core extensions compiled successfully!")
EOF

Performance Settings

For optimal performance with OpenMP parallelization:

# Set number of threads (adjust based on your CPU)
export OMP_NUM_THREADS=4

# Increase stack size (prevents segfaults in parallel code)
export OMP_STACKSIZE=1g
ulimit -s unlimited

Add these lines to your ~/.bashrc to make them permanent.

Force Field Extensions (QM/MM)

These extensions are required only for QM/MM simulations with periodic systems.

Location

DFTB/ForceField/src/

Module Compiled

• ff.so - UFF/DREIDING force fields with periodic boundary conditions

Compilation Steps

cd DFTB/ForceField/src/
make clean
make

This compiles the C-based force field module.

Verification

cd DFTB/ForceField/
python3 << 'EOF'
import ff
print("✓ Force field extension compiled successfully!")
EOF

Troubleshooting

Issue: lapack not found

# Install LAPACK development files
sudo apt-get install liblapack-dev

Issue: setup.py errors

Check that setup.py exists in DFTB/ForceField/src/:

ls -la DFTB/ForceField/src/setup.py

Multiple Scattering Extensions (Advanced)

These extensions are optional and used for advanced electronic structure calculations.

Location

DFTB/MultipleScattering/src/

Modules Compiled

• Wigner3j.so - Wigner 3j symbols

• coul90.so - Coulomb and spherical Bessel functions

• mod_bessel.so - Modified spherical Bessel functions

• sphharm.so - Spherical harmonics

• cms.so - Continuum multiple scattering

• ms.so - Bound orbital multiple scattering

• photo.so - Photoionization cross sections

• numerov.so - Numerov solver for radial Schrödinger equation

Compilation Steps

cd DFTB/MultipleScattering/src/
make clean
make

This compiles all multiple scattering modules. Compilation may take several minutes.

Verification

cd DFTB/MultipleScattering/
python3 << 'EOF'
import Wigner3j
import coul90
import mod_bessel
import sphharm
import cms
import ms
import photo
import numerov
print("✓ All multiple scattering extensions compiled successfully!")
EOF

Complete Compilation Script

To compile everything at once:

#!/bin/bash
# compile_all.sh - Compile all DFTBaby Fortran extensions

set -e  # Exit on error

echo "Compiling DFTBaby Fortran Extensions"
echo "===================================="

# Set performance options
export OMP_NUM_THREADS=4
export OMP_STACKSIZE=1g

# 1. Core Extensions (Required)
echo ""
echo "[1/3] Compiling core extensions..."
cd DFTB/extensions/
make clean
make
cd ../..
echo "✓ Core extensions compiled"

# 2. Force Field (for QM/MM)
echo ""
echo "[2/3] Compiling force field extensions..."
cd DFTB/ForceField/src/
make clean
make
cd ../../..
echo "✓ Force field extensions compiled"

# 3. Multiple Scattering (optional)
echo ""
echo "[3/3] Compiling multiple scattering extensions..."
cd DFTB/MultipleScattering/src/
make clean
make
cd ../../..
echo "✓ Multiple scattering extensions compiled"

echo ""
echo "===================================="
echo "All extensions compiled successfully!"
echo ""
echo "Performance settings:"
echo "  OMP_NUM_THREADS=$OMP_NUM_THREADS"
echo "  OMP_STACKSIZE=$OMP_STACKSIZE"
echo ""
echo "To use parallelization, run:"
echo "  export OMP_NUM_THREADS=4"
echo "  export OMP_STACKSIZE=1g"
echo "  ulimit -s unlimited"

Save this as compile_all.sh and run:

chmod +x compile_all.sh
./compile_all.sh

Common Issues and Solutions

Compilation Errors

Error: f2py: command not found

Solution: Install NumPy with f2py:

pip3 install --upgrade numpy

Error: gfortran: command not found

Solution: Install gfortran compiler:

sudo apt-get install gfortran

Error: cannot find -lblas or cannot find -llapack

Solution: Install BLAS and LAPACK:

sudo apt-get install libblas-dev liblapack-dev

Error: Segmentation fault when importing modules

Solution: Increase stack size:

export OMP_STACKSIZE=1g
ulimit -s unlimited

Error: Module version mismatch

Solution: Recompile with the same Python version:

make clean
make

Import Errors

Error: ImportError: cannot import name 'thomson'

Solution: Ensure you’re in the correct directory or extensions are in Python path:

# Check if .so files exist
ls -la DFTB/extensions/*.so

# Test from DFTB parent directory
cd /path/to/DFTBaby
python3 -c "from DFTB.extensions import thomson"

Error: ImportError: undefined symbol: __kmpc_fork_call

Solution: Link with OpenMP library:

# Recompile with explicit OpenMP flag
cd DFTB/extensions/
make clean
F2PY_OPTIONS="--f90flags='-fopenmp' -lgomp" make

Performance Issues

Slow calculations despite compilation

1. Check OpenMP is enabled:

   # Should show your thread count
   echo $OMP_NUM_THREADS
   

2. Verify parallel regions in code:

   nm DFTB/extensions/tddftb.so | grep -i omp
   

3. Profile with different thread counts:

   for i in 1 2 4 8; do
       export OMP_NUM_THREADS=$i
       echo "Threads: $i"
       time python3 your_calculation.py
   done
   

Platform-Specific Notes

Linux (Intel MKL)

For maximum performance on Intel CPUs with MKL:

# Load Intel MKL
source /opt/intel/mkl/bin/mklvars.sh intel64

# Compile with Intel compilers
cd DFTB/extensions/
sed -i 's/SYS=GNU/SYS=INTEL/' Makefile
make clean
make

# Set MKL threads
export MKL_NUM_THREADS=4

macOS (Apple Silicon)

For M1/M2/M3 Macs:

# Install Homebrew GCC (not Apple's gcc)
brew install gcc

# Use Homebrew's gfortran
cd DFTB/extensions/
make clean
FC=gfortran-13 make  # Version may vary

Windows (WSL)

Use Windows Subsystem for Linux (WSL2):

# In WSL Ubuntu
sudo apt-get update
sudo apt-get install gfortran libblas-dev liblapack-dev

# Then follow Linux instructions
cd DFTB/extensions/
make

HPC Clusters

On SLURM-based clusters:

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=8
#SBATCH --time=00:30:00

# Load modules
module load python/3.12
module load gcc/11.3.0
module load openblas

# Compile
cd $SLURM_SUBMIT_DIR/DFTB/extensions/
make clean
make

# Set threads
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

Testing Compilation

Quick Test Suite

#!/usr/bin/env python3
"""Test all compiled extensions"""

import sys

def test_core_extensions():
    """Test core extensions"""
    print("Testing core extensions...")
    try:
        from DFTB.extensions import thomson, tddftb, mulliken, slako, grad, cosmo
        print("  ✓ All core extensions loaded")
        return True
    except ImportError as e:
        print(f"  ✗ Core extension import failed: {e}")
        return False

def test_forcefield():
    """Test force field extension"""
    print("Testing force field extension...")
    try:
        from DFTB.ForceField import ff
        print("  ✓ Force field extension loaded")
        return True
    except ImportError as e:
        print(f"  ✗ Force field import failed: {e}")
        return False

def test_multiple_scattering():
    """Test multiple scattering extensions"""
    print("Testing multiple scattering extensions...")
    try:
        from DFTB.MultipleScattering import (
            Wigner3j, coul90, mod_bessel, sphharm,
            cms, ms, photo, numerov
        )
        print("  ✓ All multiple scattering extensions loaded")
        return True
    except ImportError as e:
        print(f"  ✗ Multiple scattering import failed: {e}")
        return False

if __name__ == "__main__":
    print("DFTBaby Fortran Extensions Test")
    print("=" * 40)

    results = []
    results.append(test_core_extensions())
    results.append(test_forcefield())
    results.append(test_multiple_scattering())

    print("=" * 40)
    if all(results):
        print("SUCCESS: All extensions working!")
        sys.exit(0)
    else:
        print("FAILURE: Some extensions failed")
        sys.exit(1)

Save as test_extensions.py and run:

python3 test_extensions.py

Performance Benchmarking

Simple benchmark for parallelization:

#!/usr/bin/env python3
"""Benchmark OpenMP performance"""

import time
import numpy as np
from DFTB.extensions import tddftb

def benchmark(threads):
    """Run benchmark with specified thread count"""
    import os
    os.environ['OMP_NUM_THREADS'] = str(threads)

    # Create test data
    n = 1000
    A = np.random.rand(n, n)
    B = np.random.rand(n, n)

    # Time matrix operation
    start = time.time()
    C = np.dot(A, B)
    elapsed = time.time() - start

    return elapsed

print("OpenMP Performance Benchmark")
print("Threads | Time (s) | Speedup")
print("-" * 35)

baseline = None
for threads in [1, 2, 4, 8]:
    t = benchmark(threads)
    if baseline is None:
        baseline = t
    speedup = baseline / t
    print(f"   {threads:2d}   | {t:7.3f}  | {speedup:6.2f}x")

NumPy 2.0 Migration Issues

IMPORTANT: If you experience compilation errors, especially after upgrading from NumPy 1.x to 2.0+, see the comprehensive troubleshooting guide:

➜ :doc:`compilation_fixes`

This includes:

• NumPy 2.0 breaking changes (removed -fexternal-blas, numpy.distutils)

• Fixed Makefile changes

• Common error messages and solutions

• Verified working configurations

• Step-by-step migration checklist

Further Information

• Makefile details: Check comments in each Makefile

• f2py documentation: https://numpy.org/doc/stable/f2py/

• OpenMP tuning: https://www.openmp.org/

• Compiler options: gfortran --help or ifort --help

For compilation issues, check:

1. Python version: python3 --version

2. NumPy version: python3 -c "import numpy; print(numpy.__version__)

3. f2py availability: python3 -m numpy.f2py --help

4. Compiler version: gfortran --version

5. Library paths: ldconfig -p | grep -E "(blas|lapack)"