Table of Contents
Below are the materials from my today’s Installfest talk. https://pretalx.installfest.cz/installfest-2022/talk/Z8TP3D/
1 Characteristics
- New language – announced 2012
- Goal: Solve the “two language” problem
- Prototype in Python, rewrite in C++ for speed
- Compiled language (LLVM)
- Good for interactive work (JIT)
- Dynamic types + type inference
- Garbage collected
- Multiple dispatch paradigm
- one function can have different implementations based on type of all arguments
- Lisp-like macros
- Target domain:
- Originally: Scientific computations
- Nowadays: General purpose, Data Science, Visualisation, …
1.1 Speed
Source: https://youtu.be/LT4AP7CUMAw
1.2 My success story
- Load & process 2GB CSV file
- Julia: 40s, 5 GB of memory
- Python Pandas: 5 min, OUT OF MEMORY
- Python Pandas + big machine: 15 min, 36 GB of memory
2 Installation
- distribution package managers
- download archive, unzip/untar, run
- console application
2.1 IDE: Julia plugin for VS Code
3 Calculator
1 + 1 3 * 5
4 REPL modes
Special characters at the start of the line switch modes:
Help
?atan
Shell
;ls
Package manager
]add IJulia
Return with backspace
5 Variables
x = 1 x # Julia likes Unicode :-) π # Enter as \pi<TAB> α = atan(1/2) 3α + 100 # You can skip multiply operator
6 Vectors, Arrays
a=[10,20,30] a[1] # one-based indexing a[2:3] a[2:end] b=[1 2; 3 4] c=[5 6 7 8]
6.1 Element-wise operations – broadcasting
a=rand(10_000_000) b=rand(10_000_000) a*b # ❌ a.*b sin(a) # ❌ sin.(a) # Broadcasting [sin(i) for i in a] # Comprehension # Broadcasting is faster – no need to # allocate intermediate memory @time sin.(a) .+ cos.(b); @time [sin(i) for i in a] .+ [cos(i) for i in b]; # In-place assignment (no memory allocation) c = zeros(size(a)) @time c .= sin.(a).^2 .+ cos.(b).^2; # Tired of writing dots? @. c = sin(a)^2 + cos(b)^2
6.2 GPU arrays
- Available APIs
- CUDA.jl
- AMDGPU.jl
- oneAPI.jl (Intel)
High-level (array) API:
using oneAPI oneAPI.versioninfo() a=rand(100_000_000) b=rand(100_000_000) c=zeros(size(a)) @time c .= a .* b; ga = oneArray(a) gb = oneArray(b) gc = oneArray(c) @time gc .= ga .* gb;
- Possibility to write GPU kernels in Julia
7 Types
Julia uses dynamic types + type inference.
typeof(1) typeof(1.0) typeof(Int8(1)) typeof("Hello world")
7.1 Parametric types
1//3 typeof(ans) # ans = result of previous line typeof(Int8(1)//Int8(3)) typeof([1,2,3]) typeof([1 2 3]) typeof(Int8[1,2,3]) typeof([1 2;3 4.1])
7.2 Abstract types and type hierarchy
“is subtype” operator: <:
Integer <: Number Int8 <: Integer Int8 <: Number Float32 <: Number Float32 <: Integer
7.3 User-defined types
mutable struct MyType id::Int64 text::String end x = MyType(10, "Hello") # Define addition of our type Base.:+(a::MyType, b::MyType) = MyType(a.id + b.id, "Sum") x + MyType(1, "xxx")
8 Functions
# Mathematics-like definition square(x) = x*x square(4) # Programming-like definition function hello(name) return "Hello " * name end hello("Michal") square("Hello") # Operators are functions too +(1,2) →(a,b) = (a+b)/(a-b) 3→1
8.1 Anonymous functions
Functional programming features.
x->x^2 ans(5) # call a=rand(10) # Return elements > 0.5 filter(x->x>0.5, a) [x for x in a if x > 0.5] a[a .> 0.5]
8.2 Methods and multiple dispatch
Functions can have multiple methods (implementations). The method to call is selected based on types of all arguments.
fun(x::Number) = "Number "*string(x) methods(fun) fun(1) fun(1.1) fun("1.1") fun(x::String) = "String "*x methods(fun) fun("1.1") # Functions can have many methods methods(+) methods(show)
8.3 Multimedia I/O (show examples)
Values can be presented in different formats:
@doc atan typeof(@doc atan) show(stdout, MIME("text/plain"), @doc atan) show(stdout, MIME("text/html"), @doc atan)
9 Macros
- Inspired by Lisp
- Work at abstract syntax tree level.
- Can rewrite/generate programs.
- Often used for domain specific languages
- No separate macro language, macros are written in Julia itself.
9.1 Examples
9.1.1 Unit testing
using Test @test 1 + 1 == 2
9.1.2 Code simplification
sqrt(abs(sin(1))) # Pipe syntax (for unary functions only) 1 |> sin |> abs |> sqrt rnd = rand(10) sort(rnd, rev=true) .+ 1 # Pipes with higher-arity functions ⇒ lambdas rnd |> x -> sort(x, rev=true) |> x -> x .+ 1 using Pipe # Piped value represented by underscore @pipe rnd |> sort(_, rev=true) |> _ .+ 1
- Similar: Chains.jl, DataFramesMeta.jl, …
@chain df begin dropmissing filter(:id => >(6), _) groupby(:group) combine(:age => sum) end
9.2 Benchmarking, code inspection, optimization
@time rand(Int(1e6)); using BenchmarkTools @benchmark rand(Int(1e6)) @code_native sum(1:5) a = [1, 2, 3] # Don’t perform bounds checking @inbounds a[2]
10 Showcase
Example of what is possible with the language. This is not builtin functionality. Everything is programmed in Julia.
10.1 Measurements
Computation with confidence intervals
using Measurements
a = 5.2 ± 1.0
typeof(a)
b = 3.7 ± 1.0
a + b
a * b
a / b
10.2 Unitful
using Unitful using Unitful.DefaultSymbols using Unitful: hr 1m + 3cm |> cm |> float sin(90) sin(90°) sin(π/2) 15m/3s 10km/hr |> m/s 10km/hr |> m/s |> float 0°C |> K |> float
11 Plotting
- Plots supports multiple plotting backends (e.g. Python matplotlib).
- Infamous “Time to first plot” – much better today
using Plots
plot(sin.(0:0.1:2π))
- One of several available interfaces to Gnuplot.
- Faster than Plots
using Gnuplot @gp sin.(0:0.1:2π) "with lines lw 5" "set grid"
12 Package management
- Fast development, breaking changes (packages with version < 1.0)
- Reproducible environments (projects)
- Which packages and versions
- Project.toml, Manifest.toml
- Needs manual setup
12.1 Package manager
pwd() # Switch to package manager ] ? # create a new project or activate existing activate . add Pipe ;cat Project.toml cat Manifest.toml
12.2 Using packages, modules
Module = namespace
module MyMod export x x=1 y=2 end x MyMod.x
- Package is a git repo with certain structure
# introduce exported symbols to current namespace using Package # introduce just the symbol Package to current namespace import Package
13 Tasks & Channels
- Easy to use parallelism
- Similar to goroutines in Go
14 Interfacing other languages
Direct call to a function from shared library:
# libc call ccall(:clock, Int32, ()) # using other libraries ccall((:zlibVersion, "libz"), Cstring, ()) |> unsafe_string
Python code can be called transparently from Julia:
using PyCall math = pyimport("math") math.sin(math.pi / 4) # returns ≈ 1/√2 = 0.70710678...
15 Interactive notebooks
- Jupyter vs. Pluto.jl
- Pluto is something between Jupyter and Excel
15.1 Jupyter
using IJulia
notebook()
15.2 Pluto.jl
import Pluto
Pluto.run()
15.3 Feature comparison
Feature | Jupyter | Pluto.jl |
---|---|---|
Languages | many | Julia |
File format | JSON | Julia script with comments |
Results | Stored in JSON | Available only at runtime |
Execution order | Top-down/manual | Dependency-based |
Cell updates | Manual | Automatic |
Package management | No | Yes, reproducible |
16 Dataframes.jl
- Work with tabular data, named columns
- Easy import from CSV (CSV.jl)
17 Conclusion
- ➕ “Simple”, fast, versatile language
- ➕ A lot of packages available
- ➕ Active community
- ➖ Some packages are not mature, breaking changes
- ➖ Compilation can be slow (new session)
Table of Contents
Below are notes for my Nix workshop at Installfest. https://pretalx.installfest.cz/installfest-2022/talk/QDTPQL/
1 Why Nix?
- Reproducible
- Build exactly the same software on different systems.
- Declarative
- Store your development environment in a simple file/git.
- “Reliable”
- Nix ensures that installing or upgrading one package cannot break other packages.
2 Basics
2.1 Commands
- Classic:
nix-build
,nix-shell
,nix-env
- older, incoherent, …
- Experimental:
nix
nix build
,nix develop
,nix profile
- needs to be explicitly enabled:
nix --experimental-features nix-command flake
- config file:
~/.config/nix/nix.conf
2.1.1 Package management
apt | Nix classic | Nix (flakes see later) |
---|---|---|
apt install mc | nix-env -iA nixpkgs.mc | nix profile install nixpkgs#mc |
dpkg -l | nix-env -q | nix profile list |
apt remove mc | nix-env -e mc | nix profile remove … |
apt upgrade mc | nix-env -uA nixpkgs.mc | nix profile upgrade .. |
- Features:
- No root required
- Installed into
~./nix-profile/bin
ls -l ~/.nix-profile ls -lH ~/.nix-profile
Updating
nix-channel --list # channel = package repository nix-channel --update # similar to apt update nix-env --upgrade # upgrade installed packages to channel version
2.1.2 Searching
apt | Nix classic | Nix (flakes see later) |
---|---|---|
apt-cache search … | nix-index + nix-locate | nix search nixpkgs … |
(3rd party tool) |
3 Under the hood
3.1 Nix store
ls -l /nix/store | head ls -lh $(which bash) tree $(dirname $(readlink $(which bash)))/.. | head
3.2 Profiles
https://nixos.org/manual/nix/stable/package-management/profiles.html
ls -l /nix/var/nix/profiles/per-user/$USER
- Easy to switch to older versions
nix-env --rollback nix-env --list-generations nix-env --switch-generation 43
Comparing generations (experimental nix)
nix profile diff-closures
4 Nixpkgs
- Collection of Nix expressions for building software
- Whole Linux distribution in one repository
- Ranks well in https://repology.org/ top repositories
- https://github.com/NixOS/nixpkgs/
- https://nixos.org/manual/nixpkgs/unstable/
4.1 Structure
pkgs
– packages (programs/libraries/…)nixos
– Linux distribution based on nixpkgslib
– helper functionspkgs/top-level/all-packages.nix
– list of top-level “attributes”Searching: search for “attrname<space>=”
cd nixpkgs grep -r 'gnuplot =' .
4.2 Building/Modifying packages
nix-build '<nixpkgs>' -A gnuplot git clone https://github.com/NixOS/nixpkgs/ nix-build ./nixpkgs -A gnuplot edit ./nixpkgs/pkgs/tools/graphics/gnuplot/default.nix nix-build ./nixpkgs -A gnuplot ls -l result nix log result/ # show build log nix log /nix/store/mr9l6qjimaarcak7s5rk6j1h4b16rpw9-gnuplot-5.4.3 # See the log from “distro” build nix log $(dirname $(readlink $(which bash)))/.. cd nixpkgs # Rebuild all packages depending on nixpkgs-review wip
4.2.1 Nix output monitor
Let’s have better understanding what happens during building:
NIXPKGS_ALLOW_UNFREE=1 \ nix-build $HOME/.cache/nixpkgs-review/rev-877f0e97eb2eff62e675b75dfe19940400b2ed09-dirty/build.nix \ --arg pkgs 'import <nixpkgs> {}' |& nom
Unfortunately does not work with experimental nix
command, but will
probably implement this in the future.
4.2.2 Examining dependencies
nix-tree ./result/
5 Development environments
Task | Nix classic | Nix |
---|---|---|
Temporary package installation | nix-shell -p | nix shell |
Enter build environment of a package | nix-shell | nix develop |
5.1 Install packages temporarily
mc --version echo $PATH echo $PATH | tr : \\n nix-shell -p mc mc --version echo $PATH | tr : \\n exit nix-shell --pure -p mc # only explicitly mentioned dependencies are available echo $PATH | tr : \\n exit
nix-shell -p mc -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/d1c3fea7ecbed758168787fe4e4a3157e52bc808.tar.gz
5.1.1 Experimental nix, flakes
- Discoverability, CLI completion
nix shell nixpkgs#gnuplot nix run nixpkgs#gnuplot
5.2 Existing package development/hacking
Traditionally:
./configure sudo apt install ... ./configure sudo apt install ... ./configure sudo apt install ...
With nix:
git clone https://github.com/MidnightCommander/mc.git cd mc ./autogen.sh nix-shell '<nixpkgs>' -A mc ./autogen.sh ./configure make -j$(nproc)
5.3 Per-project development environments
- One does not need to relay only on packages in nixpkgs
- Source can contain Nix files (e.g.
default.nix
,shell.nix
,flake.nix
)
git clone https://github.com/wentasah/boardproxy cd boardproxy make # fails due to missing dependencies nix-shell # make deps from shell.nix or default.nix available make # success
Just building the package:
nix-build
- Drawback: every single change in the source code rebuilds everything
5.3.1 Entering the environment automatically
- https://direnv.net/, https://github.com/nix-community/lorri, https://github.com/nix-community/nix-direnv
Needs to be installed and configured (simpler on NixOS)
nix-env -i -A direnv -A lorri cd my-project lorri init # creates shell.nix template unless it exists direnv allow cd .. cd - edit shell.nix
6 How are Nix derivations built?
Nix expression is evaluated → low-level derivation
nix-build '<nixpkgs>' -A gnuplot ls -l result nix-store --query --deriver result/ nix show-derivation result/ # pretty format cat "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
- Important attributes:
builder
,args
- Most attributes (and their values) are propagated to environment variables
nix-shell '<nixpkgs>' -A gnuplot printenv type genericBuild type buildPhase type configurePhase unpackPhase cd gnuplot-5.4.3 configurePhase buildPhase
- invalidace hash (aby se projevil update)
7 Generating Docker images from Nix packages
- https://nix.dev/tutorials/building-and-running-docker-images
https://nixos.org/manual/nixpkgs/unstable/#sec-pkgs-dockerTools
{ pkgs ? import <nixpkgs> {} }: pkgs.dockerTools.buildImage { name = "gnuplot"; config = { Cmd = [ "${pkgs.gnuplot}/bin/gnuplot" ]; }; }
7.1 Running
nix-build docker-image.nix docker load < result docker run -it gnuplot:nmg0lnkf1dl7a0bsv152a6xphkxm9sq9
7.2 What’s inside?
mkdir tmp && cd tmp
tar xf ../result
ls -lR
tar tf */layer.tar
7.3 Image from custom package
{ pkgs ? import <nixpkgs> {} , boardproxy ? import ./boardproxy/default.nix { inherit pkgs; } }: pkgs.dockerTools.buildImage { name = "boardproxy"; config = { Entrypoint = "${boardproxy}/bin/boardproxy"; }; }
nix-build docker-boardproxy.nix docker load < result docker run -it boardproxy:60j25jpwggspx1mg694143ddchsynq1q --help
7.4 Smaller image needed?
https://nixos.org/#asciinema-demo-cover
{ pkgs ? import <nixpkgs> {} }: pkgs.redis.overrideAttrs (old: { # no need for systemd support in our docker image makeFlags = old.makeFlags ++ ["USE_SYSTEMD=no"]; # build static binary with musl preBuild = '' makeFlagsArray=(PREFIX="$out" CC="${pkgs.musl.dev}/bin/musl-gcc -static" CFLAGS="-I${pkgs.musl.dev}/include" LDFLAGS="-L${pkgs.musl.dev}/lib") ''; # Let's remove some binaries which we don't need postInstall = "rm -f $out/bin/redis-{benchmark,check-*,cli}"; })
8 Flakes intro
8.1 Problems of clasic Nix
- What is inside a repo? Difficult to figure out without reading all *.nix files.
- No enforcment of naming .nix files (default.nix, shell.nix, …)
- What does
<nixpkgs>
mean? Depends on the value of $NIXPATH env. var. nix-env
operation depends on the content of~/.nix-defexpr/
.- Evaluation of Nix expressions can read arbitrary files on your disk. Harms reproducibility.
8.2 How do flakes solve it?
- Hermetic isolation of Nix evaluation (no NIXPATH, no arbitrary files).
- Only files from the same git repo can be read.
- => Makes it possible to cache results of evaluation.
- Defines a structure of repository “entry point” (flake.nix).
- Predefined attribute names for packages, overlays, tests, …
- Explicit references with versions to other sources, e.g. nixpkgs.
- lock files
8.3 Drawbacks of flakes
- One more thing to learn.
- Lower motivation to include software into nixpkgs (can be separate flake)
9 Other notes
- https://github.com/guibou/nixGL – running OpenGL apps from nixpkgs on other distros
- Remote builds
- slow laptop?
nix --builders ssh://server --max-jobs 0
- build on
server
, no local builds (except simple ones, i.e. https://nixos.org/manual/nixpkgs/unstable/#trivial-builder-runCommandLocal)
- build on
10 Pros and cons
- Nix vs. Docker
- Docker solves problems of traditional package managers by building on top of them
- Nix changes (fixes) the core of package management
- Nix is (largely) reproducible
- Declarative specifications of build/development/production environments
- Nix is difficult to learn
- Using unpackaged applications (especially proprietary) is more difficult
- buildFHSUserEnv
- nix-ld
- autopatchelf-hook