Below are the materials from my today’s Installfest talk. https://pretalx.installfest.cz/installfest-2022/talk/Z8TP3D/

1. Characteristics

https://julialang.org/

  • New language – announced 2012
  • Goal: Solve the “two language” problem
    • Prototype in Python, rewrite in C++ for speed
  • Compiled language (LLVM)
  • Good for interactive work (JIT)
  • Dynamic types + type inference
  • Garbage collected
  • Multiple dispatch paradigm
    • one function can have different implementations based on type of all arguments
  • Lisp-like macros
  • Target domain:
    • Originally: Scientific computations
    • Nowadays: General purpose, Data Science, Visualisation, …

1.1. Speed

1.2. My success story

  • Load & process 2GB CSV file
  • Julia: 40s, 5 GB of memory
  • Python Pandas: 5 min, OUT OF MEMORY
  • Python Pandas + big machine: 15 min, 36 GB of memory

2. Installation

  • distribution package managers
  • download archive, unzip/untar, run
  • console application julia-terminal.png

2.1. IDE: Julia plugin for VS Code

3. Calculator

1 + 1
3 * 5

4. REPL modes

Special characters at the start of the line switch modes:

  • Help

    ?atan
    
  • Shell

    ;ls
    
  • Package manager

    ]add IJulia
    

Return with backspace

5. Variables

x = 1
x

# Julia likes Unicode :-)
π  # Enter as \pi<TAB>
α = atan(1/2)

3α + 100 # You can skip multiply operator

6. Vectors, Arrays

a=[10,20,30]
a[1]  # one-based indexing
a[2:3]
a[2:end]

b=[1 2; 3 4]

c=[5 6
   7 8]

6.1. Element-wise operations – broadcasting

a=rand(10_000_000)
b=rand(10_000_000)

a*b                 # 
a.*b
sin(a)              # 
sin.(a)             # Broadcasting
[sin(i) for i in a] # Comprehension

# Broadcasting is faster – no need to
# allocate intermediate memory
@time sin.(a) .+ cos.(b);
@time [sin(i) for i in a] .+ [cos(i) for i in b];

# In-place assignment (no memory allocation)
c = zeros(size(a))
@time c .= sin.(a).^2 .+ cos.(b).^2;
# Tired of writing dots?
@. c = sin(a)^2 + cos(b)^2

6.2. GPU arrays

https://juliagpu.org/

  • Available APIs
    • CUDA.jl
    • AMDGPU.jl
    • oneAPI.jl (Intel)
  • High-level (array) API:

    using oneAPI
    oneAPI.versioninfo()
    a=rand(100_000_000)
    b=rand(100_000_000)
    c=zeros(size(a))
    @time c .= a .* b;
    ga = oneArray(a)
    gb = oneArray(b)
    gc = oneArray(c)
    @time gc .= ga .* gb;
    
  • Possibility to write GPU kernels in Julia

7. Types

Julia uses dynamic types + type inference.

typeof(1)
typeof(1.0)
typeof(Int8(1))
typeof("Hello world")

7.1. Parametric types

1//3
typeof(ans)    # ans = result of previous line
typeof(Int8(1)//Int8(3))
typeof([1,2,3])
typeof([1 2 3])
typeof(Int8[1,2,3])
typeof([1 2;3 4.1])

7.2. Abstract types and type hierarchy

“is subtype” operator: <:

Integer <: Number
Int8 <: Integer
Int8 <: Number
Float32 <: Number
Float32 <: Integer

7.3. User-defined types

mutable struct MyType
    id::Int64
    text::String
end

x = MyType(10, "Hello")

# Define addition of our type
Base.:+(a::MyType, b::MyType) = MyType(a.id + b.id, "Sum")
x + MyType(1, "xxx")

8. Functions

# Mathematics-like definition
square(x) = x*x
square(4)

# Programming-like definition
function hello(name)
    return "Hello " * name
end
hello("Michal")

square("Hello")

# Operators are functions too
+(1,2)
(a,b) = (a+b)/(a-b)
3→1

8.1. Anonymous functions

Functional programming features.

x->x^2
ans(5)  # call

a=rand(10)
# Return elements > 0.5
filter(x->x>0.5, a)
[x for x in a if x > 0.5]
a[a .> 0.5]

8.2. Methods and multiple dispatch

Functions can have multiple methods (implementations). The method to call is selected based on types of all arguments.

fun(x::Number) = "Number "*string(x)
methods(fun)
fun(1)
fun(1.1)
fun("1.1")
fun(x::String) = "String "*x
methods(fun)
fun("1.1")

# Functions can have many methods
methods(+)
methods(show)

8.3. Multimedia I/O (show examples)

Values can be presented in different formats:

@doc atan
typeof(@doc atan)
show(stdout, MIME("text/plain"), @doc atan)
show(stdout, MIME("text/html"), @doc atan)

9. Macros

  • Inspired by Lisp
  • Work at abstract syntax tree level.
  • Can rewrite/generate programs.
    • Often used for domain specific languages
  • No separate macro language, macros are written in Julia itself.

9.1. Examples

9.1.1. Unit testing

using Test
@test 1 + 1 == 2

9.1.2. Code simplification

sqrt(abs(sin(1)))

# Pipe syntax (for unary functions only)
1 |> sin |> abs |> sqrt

rnd = rand(10)
sort(rnd, rev=true) .+ 1

# Pipes with higher-arity functions ⇒ lambdas
rnd |>  x -> sort(x, rev=true) |> x -> x .+ 1
using Pipe

# Piped value represented by underscore
@pipe rnd |> sort(_, rev=true) |> _ .+ 1
  • Similar: Chains.jl, DataFramesMeta.jl, …
@chain df begin
  dropmissing
  filter(:id => >(6), _)
  groupby(:group)
  combine(:age => sum)
end

9.2. Benchmarking, code inspection, optimization

@time rand(Int(1e6));

using BenchmarkTools
@benchmark rand(Int(1e6))

@code_native sum(1:5)

a = [1, 2, 3]
# Don’t perform bounds checking
@inbounds a[2]

10. Showcase

Example of what is possible with the language. This is not builtin functionality. Everything is programmed in Julia.

10.1. Measurements

Computation with confidence intervals

using Measurements
a = 5.2 ± 1.0
typeof(a)
b = 3.7 ± 1.0
a + b
a * b
a / b

10.2. Unitful

using Unitful
using Unitful.DefaultSymbols
using Unitful: hr

1m + 3cm |> cm |> float
sin(90)
sin(90°)
sin(π/2)

15m/3s

10km/hr |> m/s
10km/hr |> m/s |> float
0°C |> K |> float

11. Plotting

  • Plots supports multiple plotting backends (e.g. Python matplotlib).
  • Infamous “Time to first plot” – much better today
using Plots
plot(sin.(0:0.1:2π))
  • One of several available interfaces to Gnuplot.
  • Faster than Plots
using Gnuplot
@gp sin.(0:0.1:2π) "with lines lw 5" "set grid"

12. Package management

  • Fast development, breaking changes (packages with version < 1.0)
  • Reproducible environments (projects)
    • Which packages and versions
    • Project.toml, Manifest.toml
  • Needs manual setup

12.1. Package manager

pwd()
# Switch to package manager
]
?
# create a new project or activate existing
activate .
add Pipe
;cat Project.toml
cat Manifest.toml

12.2. Using packages, modules

  • Module = namespace

    module MyMod
    export x
    x=1
    y=2
    end
    x
    MyMod.x
    
    • Package is a git repo with certain structure
    # introduce exported symbols to current namespace
    using Package
    # introduce just the symbol Package  to current namespace
    import Package
    

13. Tasks & Channels

  • Easy to use parallelism
  • Similar to goroutines in Go

14. Interfacing other languages

Direct call to a function from shared library:

# libc call
ccall(:clock, Int32, ())
# using other libraries
ccall((:zlibVersion, "libz"), Cstring, ()) |> unsafe_string

Python code can be called transparently from Julia:

using PyCall
math = pyimport("math")
math.sin(math.pi / 4) # returns ≈ 1/√2 = 0.70710678...

15. Interactive notebooks

  • Jupyter vs. Pluto.jl
    • Pluto is something between Jupyter and Excel

15.1. Jupyter

using IJulia
notebook()

15.2. Pluto.jl

import Pluto
Pluto.run()

See ./installfest-pluto.html

15.3. Feature comparison

Feature Jupyter Pluto.jl
Languages many Julia
File format JSON Julia script with comments
Results Stored in JSON Available only at runtime
Execution order Top-down/manual Dependency-based
Cell updates Manual Automatic
Package management No Yes, reproducible

16. Dataframes.jl

  • Work with tabular data, named columns
  • Easy import from CSV (CSV.jl)

17. Conclusion

  • ➕ “Simple”, fast, versatile language
  • ➕ A lot of packages available
  • ➕ Active community
  • ➖ Some packages are not mature, breaking changes
  • ➖ Compilation can be slow (new session)
Posted Sun 12 Jun 2022 10:55:57 PM CEST Tags:

Below are notes for my Nix workshop at Installfest. https://pretalx.installfest.cz/installfest-2022/talk/QDTPQL/

1. Why Nix?

https://nixos.org/

  • Reproducible
    • Build exactly the same software on different systems.
  • Declarative
    • Store your development environment in a simple file/git.
  • “Reliable”
    • Nix ensures that installing or upgrading one package cannot break other packages.

2. Basics

2.1. Commands

  • Classic: nix-build, nix-shell, nix-env
    • older, incoherent, …
  • Experimental: nix
    • nix build, nix develop, nix profile
    • needs to be explicitly enabled:
      • nix --experimental-features nix-command flake
      • config file: ~/.config/nix/nix.conf

2.1.1. Package management

apt Nix classic Nix (flakes see later)
apt install mc nix-env -iA nixpkgs.mc nix profile install nixpkgs#mc
dpkg -l nix-env -q nix profile list
apt remove mc nix-env -e mc nix profile remove …
apt upgrade mc nix-env -uA nixpkgs.mc nix profile upgrade ..
  • Features:
    • No root required
    • Installed into ~./nix-profile/bin
ls -l ~/.nix-profile
ls -lH ~/.nix-profile
  • Updating

    nix-channel --list   # channel = package repository
    nix-channel --update # similar to apt update
    nix-env --upgrade    # upgrade installed packages to channel version
    

2.1.2. Searching

apt Nix classic Nix (flakes see later)
apt-cache search … nix-index + nix-locate nix search nixpkgs …
  (3rd party tool)  

https://search.nixos.org/packages

3. Under the hood

3.1. Nix store

ls -l /nix/store | head
ls -lh $(which bash)
tree $(dirname $(readlink $(which bash)))/.. | head

3.2. Profiles

https://nixos.org/manual/nix/stable/package-management/profiles.html

ls -l /nix/var/nix/profiles/per-user/$USER
  • Easy to switch to older versions
nix-env --rollback
nix-env --list-generations
nix-env --switch-generation 43
  • Comparing generations (experimental nix)

    nix profile diff-closures
    

4. Nixpkgs

4.1. Structure

  • pkgs – packages (programs/libraries/…)
  • nixos – Linux distribution based on nixpkgs
  • lib – helper functions
  • pkgs/top-level/all-packages.nix – list of top-level “attributes”
  • Searching: search for “attrname<space>=”

    cd nixpkgs
    grep -r 'gnuplot =' .
    

4.2. Building/Modifying packages

nix-build '<nixpkgs>' -A gnuplot
git clone https://github.com/NixOS/nixpkgs/
nix-build ./nixpkgs -A gnuplot
edit ./nixpkgs/pkgs/tools/graphics/gnuplot/default.nix
nix-build ./nixpkgs -A gnuplot
ls -l result
nix log result/  # show build log
nix log /nix/store/mr9l6qjimaarcak7s5rk6j1h4b16rpw9-gnuplot-5.4.3
# See the log from “distro” build
nix log $(dirname $(readlink $(which bash)))/..
cd nixpkgs
# Rebuild all packages depending on
nixpkgs-review wip

4.2.1. Nix output monitor

Let’s have better understanding what happens during building:

NIXPKGS_ALLOW_UNFREE=1 \
nix-build $HOME/.cache/nixpkgs-review/rev-877f0e97eb2eff62e675b75dfe19940400b2ed09-dirty/build.nix \
          --arg pkgs 'import <nixpkgs> {}' |& nom

Unfortunately does not work with experimental nix command, but will probably implement this in the future.

4.2.2. Examining dependencies

nix-tree ./result/

5. Development environments

Task Nix classic Nix
Temporary package installation nix-shell -p nix shell
Enter build environment of a package nix-shell nix develop

5.1. Install packages temporarily

mc --version
echo $PATH
echo $PATH | tr : \\n
nix-shell -p mc
mc --version
echo $PATH | tr : \\n
exit
nix-shell --pure -p mc  # only explicitly mentioned dependencies are available
echo $PATH | tr : \\n
exit
nix-shell -p mc -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/d1c3fea7ecbed758168787fe4e4a3157e52bc808.tar.gz

5.1.1. Experimental nix, flakes

  • Discoverability, CLI completion
nix shell nixpkgs#gnuplot
nix run nixpkgs#gnuplot

5.2. Existing package development/hacking

  • Traditionally:

    ./configure
    sudo apt install ...
    ./configure
    sudo apt install ...
    ./configure
    sudo apt install ...
    
  • With nix:

    git clone https://github.com/MidnightCommander/mc.git
    cd mc
    ./autogen.sh
    nix-shell '<nixpkgs>' -A mc
    ./autogen.sh
    ./configure
    make -j$(nproc)
    

5.3. Per-project development environments

  • One does not need to relay only on packages in nixpkgs
  • Source can contain Nix files (e.g. default.nix, shell.nix, flake.nix)
git clone https://github.com/wentasah/boardproxy
cd boardproxy
make  # fails due to missing dependencies
nix-shell  # make deps from shell.nix or default.nix available
make  # success
  • Just building the package:

    nix-build
    
    • Drawback: every single change in the source code rebuilds everything

5.3.1. Entering the environment automatically

  • direnv, lorri, nix-direnv
  • Needs to be installed and configured (simpler on NixOS)

    nix-env -i -A direnv -A lorri
    cd my-project
    lorri init  # creates shell.nix template unless it exists
    direnv allow
    cd ..
    cd -
    edit shell.nix
    

6. How are Nix derivations built?

  • Nix expression is evaluated → low-level derivation

    nix-build '<nixpkgs>' -A gnuplot
    ls -l result
    nix-store --query --deriver result/
    nix show-derivation result/  # pretty format
    cat "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
    
  • Important attributes: builder, args
  • Most attributes (and their values) are propagated to environment variables
nix-shell '<nixpkgs>' -A gnuplot
printenv
type genericBuild
type buildPhase
type configurePhase
unpackPhase
cd gnuplot-5.4.3
configurePhase
buildPhase
  • invalidace hash (aby se projevil update)

7. Generating Docker images from Nix packages

7.1. Running

nix-build docker-image.nix
docker load < result
docker run -it gnuplot:nmg0lnkf1dl7a0bsv152a6xphkxm9sq9

7.2. What’s inside?

mkdir tmp && cd tmp
tar xf ../result
ls -lR
tar tf */layer.tar

7.3. Image from custom package

{ pkgs ? import <nixpkgs> {}
, boardproxy ? import ./boardproxy/default.nix { inherit pkgs; }
}:

pkgs.dockerTools.buildImage {
  name = "boardproxy";
  config = {
    Entrypoint = "${boardproxy}/bin/boardproxy";
  };
}
nix-build docker-boardproxy.nix
docker load < result
docker run -it boardproxy:60j25jpwggspx1mg694143ddchsynq1q --help

7.4. Smaller image needed?

https://nixos.org/#asciinema-demo-cover

{ pkgs ? import <nixpkgs> {}
}:
pkgs.redis.overrideAttrs (old: {
  # no need for systemd support in our docker image
  makeFlags = old.makeFlags ++ ["USE_SYSTEMD=no"];
  # build static binary with musl
  preBuild = ''
     makeFlagsArray=(PREFIX="$out"
                     CC="${pkgs.musl.dev}/bin/musl-gcc -static"
                      CFLAGS="-I${pkgs.musl.dev}/include"
                      LDFLAGS="-L${pkgs.musl.dev}/lib")
    '';
  # Let's remove some binaries which we don't need
  postInstall = "rm -f $out/bin/redis-{benchmark,check-*,cli}";
})

8. Flakes intro

8.1. Problems of clasic Nix

  • What is inside a repo? Difficult to figure out without reading all *.nix files.
  • No enforcment of naming .nix files (default.nix, shell.nix, …)
  • What does <nixpkgs> mean? Depends on the value of $NIXPATH env. var.
  • nix-env operation depends on the content of ~/.nix-defexpr/.
  • Evaluation of Nix expressions can read arbitrary files on your disk. Harms reproducibility.

8.2. How do flakes solve it?

  • Hermetic isolation of Nix evaluation (no NIXPATH, no arbitrary files).
    • Only files from the same git repo can be read.
    • => Makes it possible to cache results of evaluation.
  • Defines a structure of repository “entry point” (flake.nix).
    • Predefined attribute names for packages, overlays, tests, …
  • Explicit references with versions to other sources, e.g. nixpkgs.
    • lock files

8.3. Drawbacks of flakes

  • One more thing to learn.
  • Lower motivation to include software into nixpkgs (can be separate flake)

9. Other notes

  • nixGL – running OpenGL apps from nixpkgs on other distros
  • Remote builds
    • slow laptop?
    • nix --builders ssh://server --max-jobs 0
      • build on server, no local builds (except simple ones, i.e. runCommdnLocal)

10. Pros and cons

  • Nix vs. Docker
    • Docker solves problems of traditional package managers by building on top of them
    • Nix changes (fixes) the core of package management
  • Nix is (largely) reproducible
  • Declarative specifications of build/development/production environments
  • Nix is difficult to learn
  • Using unpackaged applications (especially proprietary) is more difficult
    • buildFHSUserEnv
    • nix-ld
    • autopatchelf-hook
Posted Sun 12 Jun 2022 11:03:50 PM CEST Tags: