When you are not able to explain it with words, you may have to add pictures. And when you even do not manage it with pictures, you can still make a video.

In the year 1970 Prof. Niklaus Wirth invented the Pascal programming language to teach his students the fundamentals of computer programming. While the initial core Pascal language was designed for teaching purposes only, it was soon expanded by commercial vendors and got some popularity. Later Wirth presented the language Modula-2 with improved syntax and support of modules for larger projects, and the Oberon language family with additional support for Object Oriented Programming.

The Nim programming language can be seen in this tradition, as it is basically an easy language suited for beginners with no prior programming experience, but at the same time is not restricted in any way. Nim offers all the concepts of modern and powerful programming languages in combination with high performance and some sort of universality — Nim can be used to create programs for tiny micro-controller as well as for large desktop apps and web applications.

Most books about programming languages concentrate on the language itself and assume that the reader is already familiar with the foundations of computer hardware and already has some programming experience. This is generally a valid approach, as today most people are taught this fundamental knowledge, sometimes called Computer Science (CS) in school. But still, there are people who missed this introduction in school for various reasons and decide later that they need some programming skills, maybe for a technical job. And there may exist some children that are not satisfied with the introduction to computer science taught at school. So we have decided to start this book with a short introduction to fundamental concepts — most people can skip that part. In part II we explain the basics of computer programming step by step in a way which should enable even children to learn independently. Part III will give you an overview of the Nim standard library and part IV will introduce some useful external packages. Part V will introduce advanced concepts like asynchronous operations, threading and parallel processing, and macros and meta-programming. Nim macros are very powerful but difficult at first. Part VI may finally present some advanced examples.

While Nim has a JavaScript backend and so supports web related development well, this book concentrates on native code generation using the C and C++ backends. We will discuss some peculiarities of the JavaScript backend in the second half of the book, and we may give some complete examples for the use of the JavaScript backend in the final part VI of the book. If you are strongly interested in web development and the JavaScript backend, then you may also consult the book Nim in Action of Dominik Picheta, which gives some detailed examples for the development of web based software with the Nim programming language, including a simple chat application and the skeleton of a microblogging and social networking service. And you may consult the tutorials and manuals of Nim web packages like Karax, Jester or basolato.

This book is based on the Nim reference implementation (Nim v1.4) of the team around Mr. A. Rumpf — other implementations were not available in 2020, when the initial version of the book was created. If this should change later and Nim implementations with significantly different behaviour may become available, then we may add remarks where necessary.

Part I: Introduction

For using computers and writing computer programs, you initially do not have to know many details. It is basically like driving a car: Although a car is a powerful and complicated tool, kids generally can drive it after a 3-minute introduction. Still, good racing drivers generally need a much broader insight into the inner working of all the technical components, and finally, a lot of practice.

What is a Computer?

A computer is primarily a device which can run computer programs, by following instructions about how to manipulate data.

Nearly all of the computers currently in use, from the tiny ones integrated in electronic gadgets, the well known desktop computers (PCs), to large and powerful super computers filling out entire rooms, work internally with digital data only.[1] Digital data are basically integer (whole) numbers encoded in binary form, which are represented by sequences of the symbols 0 and 1. We will discuss the term digital in the next section in more detail.

The most important part of a digital computer is the CPU, the Central Processing Unit. That tiny device is built of digital electronic circuits and can perform very basic mathematical and logical operations on numbers, like adding two numbers or deciding if a number is larger or smaller than another number. Most computer CPU’s can only store very few numbers internally, and forget the numbers when the power is switched off. So the CPU is generally electrically connected to a RAM module, a Random Access Memory, which can store many more numbers and allow fast access to these numbers, and to a Harddisk or SSD device which can permanently store the numbers, but does not allow such fast access. The stored numbers are most often called just data — basically that data is nothing more than numbers, but it can be interpreted in many different ways, such as pictures, sounds and much more.

Nearly all of today’s desktop computers, and even most notebooks and cellphones contain not only a single CPU, but multiple CPUs, also called "Cores", so they can run different programs in parallel, or a single program can run parts of it on different CPUs, to increase performance or reduce total execution time. The so called super computers can contain thousands of CPUs. Beside CPUs most computers have also at least one GPU, a Graphic Processing Unit, that can be used to display data on a screen or monitor, maybe for doing animations in games or for playing video. The distinction between CPU and GPU is not really sharp; generally a CPU can also display data on screens and monitors, and GPUs can do also some data processing that CPUs can do. But GPUs are optimized for the data display task.

More visible to the ordinary computer user are the peripheral devices like keyboard, mouse, screen and maybe a printer. These enable human interaction with the computer, but are in no way a core component of it; the computer can run well without them. In notebook or laptop computers or in cellphones, the peripheral devices are closely integrated with the core components. All the physical parts of a computer are also called hardware, while the programs running on that hardware are called software.

A less visible but also very important class of computers are micro-controller and so called embedded devices, tiny pieces with generally a hull of black plastic with some electrical contacts. The devices generally contain all necessary elements, that is the CPU, some RAM and a persistent storage that can store programs when no electric power supply is available. These devices may be restricted in computing power and the amount of data that they can store and process, but they are contained in many devices. They control your washing machine, refrigerator, television and radio and much more. Some devices in your home may even contain multiple micro-controllers and often the micro-controllers can already communicate with each other by RF (Radio-Frequency), or access by WLAN the internet, which is sometimes called Internet of Things (IoT).

Another class of large and very powerful digital computers are called mainframe computers or super computers, which are optimized to process large amount of data very fast. The key to their gigantic computing power is that many fast CPUs work in parallel — the problem or task is split into many small parts that are solved by one CPU each, and the final result is then the combination of all the solved sub-tasks. Unfortunately it is not always possible to split large problems into smaller sub-tasks.

Digital computers are generally driven by a clock signal that pulses at a certain frequency. The CPU can do simple operations like the addition of two integers at each pulse of the clock signal. For more complicated operations like a multiplication or a division it may need more clock pulses.

So a rough measure for the performance of a computer is the clock rate, that is the number of clock pulses per second, divided by the number of pulses that the CPU needs to perform a basic operation, multiplied by the number of CPUs or Cores that the computer can use.

A totally different kind of computers are Quantum Computers, large, expensive high-tech devices, which use the rules of quantum mechanics to calculate many computations in parallel. Today only a few of them exist, for research at universities and some large commercial institutes. Quantum computers may at some time in the future fundamentally change computing and our whole world, but they are not the topic of this book.

Analog and Digital

Whenever we measure a quantity based on one tiny base unit, then we work in the digital area, we measure with some granularity. Our ordinary money is digital in some way, as the cent is the smallest base unit; you will never pay a fraction of a cent for something. Time can be seen as a digital quantity as long as we accepts the second as the smallest unit. Even on so called analogue watches the second hand will generally jump forwards in steps of a second, so you can not measure fractions of a seconds with that watch.

An obvious analogue property is the thermodynamic temperature and its classic measurement device is the well known capillary thermometer consisting of a glass capillary filled with alcohol or liquid mercury. When temperature increases the liquid in a reservoir expand more than the surrounding glass and partly fills the capillary. That filling rate is an analogue measure for the temperature.

While the hourglass works digitally (you can count the tiny sand stones), the sundial does not.

Most of the quantities in our real world seem analog, and digital quantities seem to be some sort of arbitrary approximation.

But quantum mechanics has taught us that many quantities in our world really have a granularity. Physically quantities like energy or momentum are indeed multiplies of the tiny planck constant. Or consider electric charge, which is always a multiple of the elementary charge unit of one electron. Whenever an electrical current is flowing through an electrically conducting wire, an ionized gas or an electrolyte like salt water, there are flowing multiplies of the elementary charge only, never fractions of it. And of course light and electromagnetic radiation also has some form of granularity, which the photoelectric effect as well as compton scattering proves.

An important and useful property of digital signals and digital data, is that they map directly to integral numbers.

The simplest form of digital data is binary data, which can have only two distinct values. When you use a mechanical switch to turn the light bulb in your house on, or of, you change the binary state of the bulb. And your neighbor, when watching your house, receives binary signals.[2]

Digital computers are generally using binary electric states internally — voltage or current on or off. Such a on/off state is called a bit. We will learn more about bits and binary logic later. One bit can store obviously only two states, which we may map to the numbers 0 and 1. Larger integer numbers can be represented by a sequence of multiple bits.

The morse code was an early application to transmit messages encoded in binary form.

A very important property of digital encoded numbers (data) is that they can be copied and transmitted exactly. The reason for this is that digital numbers have a well defined clean state, there is no noise which overlays the data and may accumulate when the data is copied multiple times. Well, that statement is not really true — under bad conditions the noise can become so large that it changes the binary state of signals. Imagine we try to transfer some whole numbers encoded in binary form, maybe by binary states encoded as voltage level 0 Volt and 5 Volt, over an electric wire and a long distance. It is clear that the long wire can pick up some electromagnetic noise that can change the true 0 Volt data to a voltage that is closer to 5 Volt than to the true 0 Volt level, so it is received incorrectly. To catch such types of transmission errors checksums are added to the actual data. A checksum is derived by a special mathematical formula from the original data and transferred with it. The receiver applies the same formula to the received data and compares the result with the received checksum. If it does not match, then it is clear that data transmission is corrupted, and a resend is requested.

The opposite of digital is generally called analogue, a term which is used for data which have or seems to have no granularity. For example we speak of an analogue voltage when the voltage can have each value in a given range and when the voltage does not "jump" but change continuous.[3] For observing analogue voltages or currents, one can use a moving coil meter, a device where the current flows through a coil in a magnetic field and the magnetic force moves the hand/pointer.

We said in the previous section that nearly all of our current computers work with digital data only. Basically that is that they work internally with integer numbers, stored in sequences of binary bits. All input for computers must have the form of integer numbers, and all output has the form of integer numbers. Whenever we want to feed computers with some sort of analogue data, like an analogue voltage, we have to convert it into a digital approximation. For that task special devices called analog to digital converters (ADC) exists. And in some cases we have to convert the digital output data of computers to analogue signals, like when a computer plays music: The computer output in form of digital data is then converted by a device called digital to analog converter (DAC) into an analogue voltage, that generates an analogue current through a coil in the speakers of our sound box, and that electric current in the coil generates a magnetic field which exercise mechanical forces and moves the membrane of the speaker, resulting in oscillating motions, which generates air pressure variations that our ear can detect and that we finally hear as sound.

What is an Operating System?

Most computers, from cellphones to large super computers, use an operating system (OS). A well known OS is the GNU/Linux kernel. Operating Systems can be seen as the initial program that is loaded and started when we switch the computer on and that works as some kind of supervisor:[4] it can load other programs and it distributes resources like CPU cores or RAM between multiple running programs. It also controls user input by keyboard and mouse, displays output data on the screen — as text or graphics, controls how data is loaded and stored to nonvolatile storage media like hard-disk or SSD, manages all the network traffic and many more tasks. An important task of the OS is to allow user programs to access all the various hardware components from different vendors in a uniform high level manner. An OS can be seen as an intermediate layer between user programs like a text processor or a game, and the hardware of the computer. The OS allows user programs to work on a higher level of abstraction, so they do not need to know much about the low level hardware details.

Current Linux kernel version 5.13 has 30 million lines of source code!

Small micro-controllers and embedded devices often do not need to use an operating system, as they generally run only one single user program and because they usually do not have a large variety of hardware components to support.

What is Computer Programming?

Computer programming includes the creation, testing and optimizing of computer programs.

What is a Computer Program?

A computer program is basically a sequence of numbers, which make some sense to a computer CPU, in such a way that the CPU recognizes the numbers as so called instructions or numeric machine code, maybe the instruction to add two numbers.

The first computers, built in the 1950’s, were indeed programmed by feeding sequences of plain numbers to the device. The numbers were stored on so called punch cards, consisting of strong paper where the numbers were encoded by holes in the cards. The holes could be recognized by electrical contacts to feed the numbers into the CPU. As plain numbers do not match well human thinking, soon more abstract codes where used. A very direct code, which matches numerical instructions to symbols, is the assembly language. In that language for example the character sequence "add A0, $8" may map directly to a sequence of numbers which instructs the CPU to add the constant integer number 8 to CPU register A0, where A0 is a storage area in the CPU where numbers can be stored. As there exist many different types of CPUs, all with their own instruction sets, there exists many different assembly instruction sets, with similar, but not identical instructions. The rules that describe how these basic instructions have to look are called the syntax of the assembly language.

The numerical machine code, or the corresponding assembly language, is the most basic instruction set for a CPU. Every instruction which a CPU can execute maps to a well-defined assembly instruction. So each operation that a computer may be able to perform can be expressed in a sequence of assembly instructions. But complicated tasks may require millions of assembly instructions, which would take humans very long to write, and even much longer to modify, proof and debug.[5]

Just a few years after the invention of the first computers, people recognized that they would need even more abstract instruction sets, like repeated execution, composed conditionals, or other data types than plain numbers as operands. So higher level programming languages like Algol, Fortran, C, Pascal or Basic were created.

What is an Algorithm?

An algorithm is a detailed sequence of more or less abstract instructions to solve a specific task, or to reach a goal. Cooking recipe books and car repair instructions are examples of algorithms. They are a generalized, only symbolically representative pattern of all the necessary steps required to perform a certain task.

The basic math operations kids learn in school — to add, multiply or divide two numbers with paper and pencil — are algorithms too. Even starting a car follows an algorithm — when the temperature is below zero, and snow covers the vehicle, than you first have to clean the windows and lights. And when you first drive again after a longer break, you would have to check the tires before you start the engine. Algorithm can be carried out by strictly following the instructions — it is not necessary to really understand how and why it works.

So an algorithm is a perfect fit for a computer, as computers are really good at following instructions without really understanding what they are trying to accomplish.

A math algorithm to sum up the first 100 natural numbers may look like:

use two integer variables called i and sum
assign the value 0 to both variables

while i is less than 100 do:
  increase i by one
  add value of i to sum

optionally print the final value of sum

What is a Programming Language?

Most traditional programming languages were created to map algorithms to elementary CPU instructions. Algorithms generally contain nested conditionals, repetition, math operations, recovery from errors and maybe plausibility checks. A more complicated algorithm generally can be split into various separate logical parts, which may include reading in data at one point, multiple processing steps at another, and storing, or displaying data as plain text, graphic or animation at yet another point. This splitting into parts is mapped to programming languages by grouping tasks into subroutines, functions or procedures which accept a set of input parameters and can return a result.

As algorithms often work not only with numbers, but also with text, it makes sense to have a form of textual data type in a programming language too. And all the data types can be grouped in various ways, for example, as sequences of multiple data of the same type, like lists of numbers or names. Or as collections of different types, like name, age and profession of a citizen in an income tax database. For all these use cases programming languages provide some sort of support.

Compilers and Interpreters

We already learned that the CPU in the computer can execute only simple instructions, which we call numeric machine code or assembly instructions.

To run a program written in a high level language with many abstractions, we need some sort of a converter to transform that program into the basic instructions that the CPU can execute. For the conversion process we have basically two options: We can convert the whole program into machine code, store it to disk, and than run it on the CPU. Or we can convert it in small portions, maybe line by line, and run each portion whenever we have converted it. Tools that convert the whole program first are called compilers. Compilers process the program that we have written, include other source code like needed library modules, check the code for obvious errors and then generate and store the machine code that we then can run.

Tools that process the source code in small portions, like single statements, are called interpreters. They read in a line of source code, investigate it to check if it is a valid statement, and then feed the CPU with corresponding instructions to execute it. It is similar to picking strawberries: you can pick one and eat it at once, or you can put them all into a basket and eat them later. Both interpreters and compilers have advantages and disadvantages for special use cases. Compilers can already detect errors before the program is run, and compiled programs generally run fast, as all the instructions are already available when the programs runs. The compiling step takes some time of course, at least a few seconds, but for some languages and large programs it may take much longer. That can make the software development process slow because as you add or change code, you have to compile it before you can execute and test your program. That may be inconvenient for unskilled programmers as they may have to do much testing. Some use a programming style that is: change a tiny bit of the source code, then run it and see what is does. But a more common practice is that you think about the problem first and then write the code, that then in most cases does nearly that what you intended. For this style of programming, you do not have to compile and execute your code that often. Compilers have one important benefit: they can detect many bugs, mostly typing errors, already in the compile phase, and they give you a detailed error message. Interpreters have the advantage that you can modify your code and immediately execute it without delay.

That is nice for learning a new language and for some fast tests, but even simple typing errors can only be detected when they are encountered while running the program. If your test does not try to run a faulty statement, there will be no error, but it may occur later. Generally interpreted program execution is much slower than running compiled executables, as the interpreter has to continually process the source code in real-time as it’s being run, while the compiler does it only once before the program is run. At the end of this section, a few additional notes:

Compilers are sometimes supported by so called linkers. In that case the compiler converts the source code, that can be stored in multiple text files, each in a sequence of machine code instructions, and finally the linker joins all these machine code files to the final executable. Some compilers do not need the linking step or call the linker automatically. And some interpreters convert the textual source code in one very fast, initial pre-processing step ("on the fly") to so called byte code, that can then be interpreted faster. The languages Ruby and Python do that. Some languages, like Java, can compile and optimize the source code while the program is running. For that process a so called virtual machine is used, which builds an intermediate layer between the hardware and the user program.

Types of Programming Languages

We already mentioned the assembly languages, which provide only the basic operations that the CPU can perform. Assembly languages provide no abstractions, so maybe we should not even call them programming languages at all. Then there are low level languages like Fortran or C, with some basic abstractions which still work close to the hardware and which are mostly designed for high performance and low resource consumption (RAM) but not to detect and prevent programming errors or to make life easy for programmers. These languages already support some higher order data types, like floating point numbers or text (strings), and homogeneous, fixed size containers (called arrays in C) or heterogeneous fixed size containers (called structs in C).

A different approach is taken by languages like Python or Ruby, which try to make writing code easier by offering many high level abstractions and which have better protection against errors, but are not as efficient. These languages support also dynamic containers which can grow and shrink, or advanced data structures like hash tables (maps) or support for textual pattern matching by regular expressions (regex).

Another way to differentiate programming languages is if they are statically, or dynamically typed. Ruby, Python and JavaScript are all examples of dynamically typed languages, that is, they use variables which can store any data type, so the variable’s type of data that it accepts can therefore dynamically change during program execution. That seems comfortable for the user, and sometimes it is, especially for short programs which may be written for one-time use only and are sometimes called scripts. But dynamic typing makes discovery of logical errors harder — an illegal addition of a number to a letter may be detected only at run-time. And dynamically typed languages generally waste a lot of memory and their performance is not that great. It is as you would own many moving boxes and you store all your goods in it, each piece in one box.

For statically typed languages, each variable has a well defined data type like integer number, real number, a single letter, a text element and many more. The data type is either assigned by the author of the program with a type declaration, or is detected by the compiler itself when processing the program source code, called type inference, and the variable’s type does never change. In this way, the compiler can check for logical errors early in the compile process, and the compiler can reserve memory blocks exactly customized to the variables that we want to store, so total memory consumption and performance can be optimized. Referring again to our boxes example, statically typing is like using customized boxes for all your goods.

All these types of programming languages are often called imperative programming languages, as the program describes detailed what to do. There are other types of programming languages too, for example languages like Prolog, which try to give only a set of rules and then let the computer try to solve a problem with these rules. And of course there are the new concepts of artificial intelligence (AI) and machine learning (ML), which are less based on algorithms and more on neural nets which are trained with a lot of data until it can provide the desired results. Nim, the computer language this book is about, is an imperative language, so we will focus on the imperative programming style in this book. But of course, Nim can be used to create AI applications.

Further still, we can differentiate between languages like C, C++, Rust, Nim and many more that compile to native executables and can run directly on the hardware of the computer, contrasted with languages like Java, Scala, Julia and some more, that use a large Virtual Machine (VM) as an intermediate layer between the program and the hardware, and interpreted languages like Ruby and Python. Languages using a virtual machine generally need some startup time when a program is invoked, as the VM must be loaded and initialized, and interpreted languages are generally not very fast.[6] The distinction between languages that compile to native executables, and those that are executed on a virtual machine, is not really sharp. For example Kotlin and Julia where executed on a virtual machine initially, but now can compile the source code to native executables.

An important class of programming languages are the so called Object-Oriented-Programming (OOP) languages, which uses inheritance and dynamic dispatch, and become popular in the 1990’s. For some time it was assumed that Object-Oriented-Programming was the ultimate solution to manage and structure really large programs. Java was the most prominent example of the OOP languages. Java forces the programmer to use OOP design, and languages like C++, Python or Ruby strongly push programmer to use OPP design. Practice has shown that OOP design is not the ultimate solution for all computing problems, and OPP design may prevent optimal performance. So newer languages like Go, Rust and Nim support some form of OOP programming, but use it only as one paradigm among many other.

Another popular and important class of programming languages is JavaScript and its more modern cousins like TypeScript, Kotlin or Dart and others. JavaScript was designed to run in web browsers to support interactive web pages and programs and games running in the browser. In this way the program became nearly independent from the native operating system of the computer. Note that unlike the name may indicate, JavaScript is not closely related to the Java language. Nim can compile to a JavaScript backend, so it supports web development well.

Sometimes source code written in one programming language is converted into another one. A prominent target for such conversions is JavaScript, as JavaScript enables execution of programs in web browsers. Another important target language is C or C++. Creating intermediate C code, which is then compiled by a C compiler to native executables has some advantages compared to direct compilation to native executables: C compilers exists for nearly all computer systems including {mc}s and embedded systems, so the use of a language is not restricted to systems for which a native compiler backend is provided. And C as intermediate code simplifies the use of system libraries which generally provide a C compatible interface. Due to decades of development C compilers generally can do better code optimizations than young languages may manage to do. Some people fear that intermediate C code carries the problems of the C language, like verbosity, confusing and error-prone code or undefined behavior to the source languages. But these well known problems of C occur only when humans write C code directly, in the same way when humans write assembly code directly. Automatic conversions are well defined and well tested, which means they are free of errors to the same degree as direct machine code generation would be. But indeed there are some small drawbacks when C or C++ is used as a backend of a programming language: C does not always allow direct access to all CPU instructions, which may make it difficult to generate optimal code for some special constructs like exceptions. And C uses wrap around arithmetic for unsigned integer types, which may not be what modern languages desire. The current Nim implementation provides a JavaScript and a C and C++ backend. While the JavaScript backend is a design decision to enable web development, the C and C++ backends are more a pragmatic decision, and may be later replaced or at least supported by direct native code generation or use of the popular LLVM backend. [7] When computer languages are converted from one language to another, then sometimes the term transpiler is used to differentiate the translation process to a direct compilation to a binary executable. When program code is converted between very similar languages with nearly the same level of abstractions, then the term transpiler may be justified. But Nim is very different from C and has a higher abstraction level, and the Nim compiler performs many advanced optimizations. So it should be not called a transpiler, even when compiling to JavaScript or to the C++ backend.

Why Nim?

In this section we are using a lot of new Computer Science (CS) expressions but do not explain them. That is intentional — when you already know them you may get a better feeling of what Nim is, and when you do not know them you will at least learn that we can describe Nim with fancy-sounding terms.

Three well known traditional programming languages are C, Java and Python. C is basically a simple, close-to-the-hardware language created in 1972, for which compilers can generate fast, highly optimized native machine code, but it has cryptic syntax, some strange semantics, and is missing higher concepts of modern languages. Java, created in 1995, forces you strongly to the object-orientated style of programming (OOP) and runs on a virtual machine, which makes it unsuitable for embedded systems and micro-controllers. Python, created in 1991, is generally interpreted instead of compiled, which makes program execution not very fast, and it does not really allow writing low level code which operates close to the hardware. Of course there are many more programming languages, each with its own advantages and disadvantages, with some optimized for special use cases.

Nim is a state-of-the-art programming language well suited for systems and application programming. Its clean Python-like syntax makes programming easy and fun for beginners, without applying any restrictions to experienced systems programmers. Nim combines successful concepts from mature languages like Python, Ada and Modula with a few established features of the latest research. It offers high performance with type and memory safety while keeping the source code short and readable. The compiler itself and the generated executables support all major platforms including Windows, Linux, BSD and Mac OS X. The custom package managers, Nimble or Nimph, makes use and redistribution of programs and libraries easy and secure. Nim supports various "backends" to generate the final code. The C, C++ and LLVM-based backends allow easy OS library calls without additional glue code, while the JavaScript backend generates high quality code for web applications. The integrated "Read/Eval/Print Loop" (REPL), "Hot code reloading", incremental compilation, and support of various development environments including debugging and language server protocols makes working with Nim productive and enjoyable.

Some Facts About Nim

  • The generated executables are dependence free and small: a simple chess program with a plain GTK-based graphical user interface is only 100 kB in size and the size of the Nim compiler executable itself is about 5 MB. It is possible to shrink the executable size of "Hello World" programs to about 10 kB for use on tiny micro-controllers.

  • Nim is fast. Generally performance is very close to other high-performance languages such as C or C++. There are some exceptions still: other languages may have libraries or applications that have been tuned for performance for many years, while similar Nim applications are so far less tuned for performance, or maybe are more written with a priority of short and clean code or run-time safety.

  • Clean syntax with significant white-space, no need for block delimiters like {} or begin/end keywords, and no need for statement delimiters like ;

  • Safety: Nim programs are type- and memory-safe — memory corruption is prevented by the compiler as long as unsafe low level constructs like cast and the addr operator are not used.

  • Fast compiler. The Nim compiler can compile itself and other medium-size packages in less than 10 seconds, and upcoming incremental compilation will increase that speed further.

  • Nim is statically typed: each object and each variable has a well-defined type, which catches most programming errors already at compile time, prevents run-time errors, and ensures highest performance.

  • Nim supports various memory management strategies, including manually allocations for critical low-level tasks as well as various garbage collectors including a state-of-the-art, fully deterministic memory manager.

  • Nim produces native, highly optimized executables and can also generate JavaScript output for web applications.

  • Nim has a clean module concept which helps to structure large projects.

  • Nim has a well-designed library which supports many basic programming tasks. The full source code of the library is included and can be viewed easily from within the HTML-based API documentation.

  • That library is supported by more than 1000 external packages for a broad range of use cases.

  • Asynchronous operation, threading and parallel processing is supported.

  • Nim supports all popular operating systems like Linux, Windows, MacOS and Android.

  • Usage of external libraries written in C is easy and and occurs directly without any glue code, and Nim can even work together with code written in other languages, for example there are some Nim <-> Python interfaces available.

  • Many popular editors have support for Nim syntax highlighting and other IDE functionality like on-the-fly checking for errors and displaying detailed information about imported functions.

  • In the last few years Nim has reached some important milestones: Version 1.0 with some stability promises was released, and with the ARC and ORC memory management strategies and full destructor support fully deterministic memory management comparable to memory management in C++ or Rust is available. So problems of conventional garbage collectors like delayed memory deallocation or longer pausing of programs due to the GC process are gone. And some larger companies have started using Nim in production, the most important may be currently the Status Corp. with their Etherium client development.

Nim is Efficient

Nim is a compiled and statically-typed language. While for interpreted, dynamically-typed languages like Python we have to run every statement to check even for trivial errors, the Nim compiler checks for most errors during the compile process. The static typing together with the well-designed Nim type system allows the compiler to catch most errors already in the compile phase, like the undefined addition of a number and a letter, and to report the errors in the terminal window or directly in the editor or IDE. When no errors are found or all errors have been fixed, then the compiler generates highly optimized dependency free executables. And this compilation process is generally really fast, for example the compiler compiles itself in maybe 10 to 30 seconds on a typical modern PC.[8]

Modern concepts like zero-overhead iterators, compile-time evaluation of user-defined functions and cross-module inlining in combination with the preference of value-based, stack-located data types leads to extremely efficient code. Multi-threading, asynchronous input/output operations (async IO), parallel processing and SIMD instructions including GPU execution are supported. Various memory management strategies exists: selectable and tuneable high performance Garbage Collectors (GC), including a new fully deterministic destructor based GC, are supported for automatic memory management. These can be disabled for manual memory management. This makes Nim a good choice for application development and close-to-the-hardware system programming at the same time. The unrestricted hardware access, small executables and optional GC will make Nim a perfect solution for embedded systems, hardware driver and (OS) development.

Nim is Expressive and Elegant

Nim offers a modern type system with templates, generics and type inference. Built-in advanced data types like dynamic containers, sets, and strings with full UTF support are completed by a large collection of library types like hash tables and regular expressions. While the traditional Object-Oriented-Programming programming style with inheritance and dynamic dispatch is supported, Nim does not enforce this programming paradigm and offers modern concepts, like procedural and functional programming. The powerful AST-based hygienic macro system offers nearly unlimited possibilities for the advanced programmer. This macro and meta-programming system allows compiler-guided code generation at compile time, so the Nim core language can be kept small and compact, while many advanced features are enabled by user defined macros. For example the support of asynchronous IO operations has been created with these forms of meta-programming, as well as many Domain Specific Language (DSL) extensions.

Nim is Open and Free

The Nim compiler and all of the standard library are implemented in Nim. All source code is available under the less restricted MIT license.

Nim has a friendly and helpful growing community

The Nim forum is hosted at:

and the software running the forum is coded in Nim.

Real-time chat is supported by IRC, Gitter and others.

Nim has a encouraging future

Started more than 12 years ago as a small community project of some bright CS students led by Mr. A. Rumpf, it is now considered as one of the most interesting and promising programming languages, supported by countless individuals and leading companies of the computer industry, for instance, it’s actively used in the areas of game, web and crypto-currency development. Nim has made a large amount of progress in the last few years: it reached version 1.4 with some stability guarantees and a new deterministic memory management system was introduced, which will improve support of parallel processing and the use of Nim in the area of embedded systems development.

Nim was created by Mr. A. Rumpf in 2008, supported by a few volunteers. Finally, in 2018 Nim got some significant monetary support by Status Corp. and in 2019 the stable Nim version 1.0 was released. But still Nim is developed by a small core team and some volunteers, while some other languages like Java, C#, Go, or Rust are supported by large companies, or like C and C++ have a very long history and well-trained users. And finally there are many competing languages, some with a longer history, and some maybe better suited for special purposes, like JavaScript, Dart or Kotlin for web development, Julia or R for numeric applications, or C and Assembly for the tiny 8-bit micro-controllers with a small amount of RAM.

Nim is already supported by more than 1000 external packages which cover many application areas, but that number is still small compared to really popular languages like Python, Java or JavaScript. And some Nim packages can currently not really compare with the libraries of other languages, which have been optimized for years by hundreds or thousands of full-time developers.

Indeed the future of Nim is not really secure. Core developers may vanish, financial support may stop, or maybe a better language may appear. But even if the development of Nim should stop some day, you will still be able to use it, and many concepts that you may have learned with Nim can be used with other modern languages too.

Is Nim a good choice as first language for a Beginner?

When you use C as your first language, you may learn well how computers really work, but the learning experience is not that nice, progress is slow and C lacks many concepts of modern programming languages. C++, Rust or Haskell are really too difficult for beginners. So currently many starts with Python. While you can learn high level concepts well with Python and you get useful results fast, you learn not much about the internal working of computers. So you may never understand why your code is slow and consumes so much resources, and you will have no idea how to improve the program or how you could run it successfully on restricted hardware. It’s like learning to drive a car, without any knowledge about how a combustion engine, the transmission, or the brakes really work. Nim has none of these restrictions, as we have high level concepts available like in Python, but we have access to low level stuff too, so we can really understand the internal workings, if we want. Learning resources for Nim are still not that good as for mainstream languages, but there exists some nice tutorials already, and hopefully this book will help beginners also a bit.

Is Nim really a good teaching language?

Generally yes, in the same way as Pascal was in the 1980’s, and Modula/Oberon were at the end of the last century. But Nim still has the same problems as the wirthian languages: They do not really help with finding a job. When we teach the kids some Javascript or C, they may find at least a simple employment when they have to leave the intended education path early for some reason. With niche languages this is unfortunately not the case, so teachers should know about their responsibility. And of course teaching against the interests of the kids makes not much sense. When they want to learn some Javascript to make some visual effects or whatever easily, then it is hard to teach another language which may be not immediately available on the PC at home or their smartphone.

So is Nim really the best start for me?

Maybe not. When you intend to learn a programming language today and make a great video game tomorrow, then definitely not. This is just not possible. While there are nice libs for making games with Nim already available, there exists easier solutions in other languages. With some luck you may find some source code for that languages, so that you can patch a few strings and maybe modify some colors and the background music and call it your game.

After learning Nim, will I still have to learn other programming languages?

Nim is a quite universal language, so it is a good candidate for someone who intends to learn only one single language. But of course it is always a good idea to learn a few other languages later. Generally we can not really avoid learning C, as so much C code exists world wide. Most algorithm that have ever been invented are available as a C implementation somewhere, and most libraries are written in C or have at least a C API, which you can use from other languages including Nim. As C is a small language without difficult language constructs, some minimal C knowledge is generally sufficient to convert a C program to another language. Often that conversion process is supported by tools, like the Nim c2nim tool. So learning some C later is really a good idea, and when you have some basic understanding of Nim and CS in general, learning some C is an easy task. Learning C before Nim would be an option still, as for C more learning resources exists. So years ago some people recommended learning C or Python before Nim. But Nim has enough learning resources now, so we recommend indeed starting with Nim directly.

Why should I not use Nim?

Maybe it is just not the ideal solution for you. A racing bicycle or a mountain bike are both great devices, but for cycling a few hundred meters to the bakers shop both may be not the perfect solution. A plain old bicycle would do better. Even as Nim seems to join the benefits of a racing bicycle and a mountain bike well — high performance and robust design — and is not expensive, it is just not the optimal solution for everybody. People who write only tiny scripts and have not to care about performance can continue using Python. People who are only interested in special applications, maybe only in web development or only in tiny 8 bit micro-controllers may not really need Nim. Nim can do this and much more well, but for special use cases better suited languages may still exist. And someone who has managed to learn C++ really well over a period of many years may decide to continue with C++ also. Currently another possible reason for not using Nim can be missing libraries. When you need some important libraries for your project, and these are currently not available for Nim, this can be of course a big problem in the case that you have not the skills or the time to write them from scratch or at least create high level bindings to a C library.

Our first Nim Program

To keep our motivation, we will present a first tiny Nim program now. Actually we should have delayed this section until we have installed the Nim compiler on our computer, but we can already run and test the program by just copying it into one of the available Nim online playgrounds like

In the section What is an Algorithm? we described an algorithm to sum up the first 100 natural numbers. Converting that algorithm into a Nim program is straightforward and results in the text file below. You can copy it into the playground and run it now if you want. The program is built using some elementary Nim instructions for which we will give only a very short description here. Everything is explained in much more detail in the next part of this book.

var sum: int
var i: int
sum = 0
i = 0
while i < 100:
  inc(i, 1)
  inc(sum, i)
echo sum

We write Nim programs in the form of plain text files, and you will learn how to create them soon. We call these text files the source code of the program. The source code is the input for the compiler. The compiler processes the source code, checks it for obvious errors and then generates an executable file, which contains the final CPU instructions and can be run. Executable files are sometimes called executables or binary files. The term binary is misleading, as all files on computers are indeed stored as binary data, but the expression "binary" is used to differentiate the executable program from text files like the Nim source code which we can read, print and edit in an editor. Don’t try to load the executable files generated by the Nim compiler into a text editor, as the content is not plain text, but numeric machine code that may confuse the editor. On the Windows OS, executable files generally get a special name extension .exe, but on Linux no special name extensions are used.

Elementary entities of computer programs are variables, which are basically named storage areas in the computer. As Nim is a compiled and statically-typed language, we have to declare each variable before we can use it. We do that by choosing a meaningful name for the variable and specifying its data type. To tell the compiler about our intention to declare a variable, we start the line with the var keyword, followed by the chosen name, a colon and the data type of our variable. The first line of our program declares a new variable named sum of data type int. Int is short for integer and indicates, that our variable should be able to store negative or positive integer numbers. The var at the start of the line is a keyword. Keywords are reserved symbols which have a special meaning for the compiler. Var indicates that we want to introduce a new variable. The compiler will recognize that and will reserve a memory location in the RAM of the computer which can store the actual value of the variable.

The second line is nearly identical to the first line: we declare another variable again with int type and plain name i. Variable names like i, j, k are often used when we have no idea for a meaningful name and when we intend to use that variable as a counter in a loop.

In the lines 3 and 4 of our program we initialize the variables, that is, we give them a well-defined initial value. To do that we use the = operator to assign it a value. Operators are special symbols like +, -, * or / to indicate our desire to do an addition, a subtraction, a multiplication or a division. Note that the = operator is used in Nim like in many other programming languages for assignment, and not like in traditional mathematics as an equality test. The reason for that is that in computer programming, assignments occur more often than equality tests. Some early languages like Pascal use the compound := operator for assignment, which may be closer to mathematics use, but is more difficult to type on a keyboard and looks not too nice for most people. An expression like x = y assigns the content of variable y to x, that is, x gets the value of y, the former value of x is overwritten and lost, and the content of y remains unchanged. After that assignment, x and y contain the same value. In the above example we do not assign the content of a variable to the destination, but instead use a literal numeric constant with value 0. When the computer has executed lines 3 and 4 the variables sum and i each contain the start value 0.

Line 5 is much more interesting: it contains a while condition. The line starts with the term while, which is again a reserved keyword, followed by the logical expression i < 100 and a colon. An expression in Nim is something which has a result, like a math expression as 2 + 2 which has the result 4 of type integer. A logical expression has no numerical result, but a logical one, which can be true or false. The logical expression i < 100 depends on the actual content of variable i. The two lines following the line with the while keyword are each indented by two spaces, meaning that these lines start with two spaces more than the line before. That form of indentation is used in Nim to indicate blocks. Blocks are grouped statements. The complete while loop consists of the line containing the while keyword followed by a block of statements. The block after the while condition is executed as long as the while condition evaluates to true. For the first iteration i has the initial value 0, the condition i < 100 evaluates to true and the block after the while condition is executed for the first time. In the following block we have the inc() instruction. inc is short for increment. inc(a, b) increases the value of a by b, b is unchanged. So in the above block i is increased by one, and after that sum is increased by the current value of i. So when that block is executed for the first time i has the value 1 and sum also has the value 1. At the end of that block execution starts again at the line with the while condition, now testing the expression i < 100 with i containing the value 1. Again it evaluates to true, the block is executed again, i gets the new value 2, and sum gets the value 3. This process continues until i has the value 100, so the condition i < 100 evaluates to false and execution proceeds with the first instruction after the while block. That instruction is an echo statement, which is used in Nim to write values to the terminal or screen of the computer. Some other languages use the term print or put instead of echo.

Don’t worry if you have not understood much of this short explanation, we will explain all that in much more detail later.

Binary Numbers

When we write numbers in ordinary life we generally use the decimal system with base 10 and the 10 available digits 0, 1, …​ 9. To get the value of a decimal number we multiply each digit with powers of 10 depending on the position of the digit and sum the individual terms. The rightmost digit is multiplied with 10^0, the next digit with 10^1, and so on. A literal decimal number like 7382 has then the numerical value 2 * 10^0 + 8 * 10^1 + 3 * 10^2 + 7 * 10^3. We have used here the exponential operator ^ — with 10^3 = 10 * 10 * 10. Current computers use binary representation internally for numbers. Generally we do not care much about that fact, but it is good to know some facts about binary numbers. Binary numbers work nearly identically to decimal numbers. The distinction is that we have only two available digits, which we write as 0 and 1. A number in binary representation is a sequence of these two digits. Like in the decimal system, the numerical value results from the individual digits and their position: The binary number 1011 has the numerical value 1 * 2^0 + 1 * 2^1 + 0 * 2^2 + 1 * 2^3, which is 11 in decimal notation. For binary numbers the base is 2, so we multiply the binary digits by powers of two. Formally addition of two binary numbers works like we know it from the decimal system: we add the matching digits and take carry into account: 1001 + 1101 = 10110 because we start by adding the two least significant digits of each number, which are both 1. That addition 1+1 results in a carry and result 0. The next two digits are both zero, but we have to take the carry from the former operation into account, so result is 1. For the next position we have to add 0 and 1, which is just 1 without a carry. And finally we have 1 + 1, which results in 0 with a carry. The carry generates one more digit, and we are done. In the decimal system with base 10 a multiplication with 10 is easily calculated by just shifting all digits one place to the left and writing a 0 at the now empty rightmost position. For binary numbers it is very similar: a multiplication by the base, which is two in the binary system, is just a shift left, with the rightmost position getting digit 0.[9]

In the binary system we call the digits often bits, and we number the bits from right to left, starting with 0 for the rightmost bit — we say that the binary number 10010101 is an 8-bit number because writing that number in binary representation needs 8 digits. Often we imagine the individual bits as small bulbs, a 1 bit is imagined as a lit bulb, and a 0 bit is imagined as a dark bulb. For lit bulbs we say also that the bit is set, meaning that in the binary number 10010101, bits 0, 2, 4 and 7 are set, and the other bits are unset or cleared.

Groups of 8 bits are called a byte, and sometimes 4 bits are called a nibble.

One, two, four or 8 bytes are sometimes called a word, where a word is an entity which the computer can process in one single instruction. When we have a CPU with 8 byte word size this means that the computer can for example add two variables, each 8 byte in size, in one single instruction.

Let us investigate some basic properties of binary numbers. Let us assume that we have an 8-bit word (a byte). An 8-bit word can have 2^8 different states, as each bit can be set or unset independently from the other bits. That corresponds to numbers 0 up to 255 — we assume that we work with positive numbers only for now, we will come to negative numbers soon. An important property of binary numbers in computers is the wrapping around, which is a consequence of the fact that we have only a limited set of bits available to store the number. So when we continuously add 1 to a number, at some point all bits are set, which corresponds to the largest number that can be stored with that number of bits. When we then add again 1, we get an overflow. The run-time system may catch that overflow, so we get an overflow error, or the number is just reset to zero, as it may happen in our car when we manage to drive one million miles, or when the ordinary clock jumps from 23:59 to 00:00 of the next day. An useful property of binary numbers is the fact that we can easily invert all bits, that is replace set bits by unset ones and vice versa. Let us use the prefix ! to indicate the operation of bit inversion, then !01001100 is 10110011. It is an obvious and useful fact that for each number x we get a number with all bits set when we add x and !x. That is x + !x = 11111111 when we consider a 8 bit word. And when we ignore overflow, then it follows that x + !x + 1 = 0 for each number x. That is a useful property, which we can use when we consider negative numbers.

Now let us investigate how we can encode negative numbers in binary form. In the binary representation we have only two states available, 0 or 1, a set bit or an unset bit. But we have no unitary minus sign. We could encode the sign of a number in the topmost bit of a word — when the topmost bit is set that indicates that the number is regarded negative. Generally a modified version of this encoding is used, called two’s complement: a negative number is constructed by first inverting all the bits — a 0 bit is transferred into a 1 bit and vice versa — and finally the number 1 is added. That encoding simplifies the CPU construction, as subtraction can be replaced by addition in this way:

Consider the case that we want to do a subtraction of two binary encoded numbers. The operation has the symbolic notation A - B for arbitrary numbers A and B. The subtraction is by definition the inverse operation of the addition, that is A + B - B = A for each number A and B, or in other words, B - B = 0 for each number B.

Assume we have a CPU that can do additions and that can invert all the bits of a number. Can we do subtraction with that CPU? Indeed we can. Remember the fact that for each number X X + !X + 1 = 0 as long as we ignore overflow. If that relation is true for each number, than it is obviously true for each B in the expression A - B, and we can write A - B = A + (B + !B + 1) - B = A + (!B + 1) when we use the fact that in mathematics addition and subtraction is associative, that is we can group the terms as we want. But the term in the parenthesis is just the two’s complement, which we get when we invert all bits of B and add 1. So to do a subtraction we have to invert the bits of B, and then add A and !B and 1 ignoring overflow. That may sound complicated, but bit inversion is a very cheap operation in a CPU, which is always available, and adding 1 is also a very simple operation. The advantage is that we do not need separate hardware for the subtraction operation. Generally subtraction in this way is not slower than addition because the bit inversion and the addition of 1 can be performed at the same time in the CPU as an ordinary addition.

From the equation above indicating A - B = A + (!B + 1) it is obvious that we consider the two’s complement (!B + 1) as the negative of B. Note that the two’s complement of zero is again zero, and two’s complement of 00000001 is 11111111. All negative numbers in this system have a bit set to 1 at the leftmost position. This restrict all positive numbers to all the bit combinations where the leftmost bit is unset. For an 8-bit word this means that positive numbers are restricted to the bits 00000000 to 01111111, which is the range 0 to 127 in decimal notation. The two’s complement of decimal 127 is 10000001. Seems to be fine so far, but note there exists also the bit pattern 10000000 which is -128 in decimal. For that bit pattern there exists no positive value. If we try to build the two’s complement of that bit pattern, we would get the same pattern again. This is an asymmetry of two’s complement representation, which can not be avoided. It generally is no problem, with one exception. We can never invert the sign of the smallest available integer; that operation would result in a run-time error.[10]

Summary: when we work only with positive numbers, we can store in an 8-bit word, which is generally called a byte, numbers from 0 up to 255. In a 16-bit word we could store values from 0 up to 2^16 - 1, which is 65535. When we need numbers which can be also negative we have for 8-bit words the range from -128 to 127 available, which is -2^7 up to 2^7 - 1. For a signed 16-bit word the range would be -2^15 up to 2^15 - 1.

While we can work with 8 or 16-bit words, for PC programming the CPU usually supports 32 or 64 bit words, so we have a much larger number range available. But when we program micro-controllers or embedded devices we may indeed have only 8 or 16-bits words available, or we may use such small words size intentionally on a PC to fit all of our data into a smaller memory area.

One important note at the end of this section: whenever we have a word with a specific bit pattern stored in the memory of our computer, then we can not decide from the bit pattern directly what type of data it is. It can be a positive or a negative number, but maybe it is not a number at all but a letter or maybe something totally different. As an example consider this 8 bit word: 10000001. It could be 129 if we have stored intentionally positive numbers in that storage location, or could be -127 if we intentionally stored a negative value. Or it could be not a number at all. Is that a problem? No it is not as long as we use a programming language like Nim which use static typing. Whenever we are using variables we declare their type first, and so the compiler can do bookkeeping about the type of each variable stored in the computer memory. The benefit is, that we can use all the available bits to encode our actual data, and we do not have to reserve a few bits to encode the actual data type of variables. For languages without static typing that is not the case. In languages like Python or Ruby we can use variables without a static type, so we can assign whatever we want to it. That seems to be comfortable at first, but can be confusing when we write larger programs and the Python or Ruby interpreter has to do all the bookkeeping at run-time, which is slow and wastes memory for the bookkeeping.

To say it again in other words: for deciding if an operation is valid, it is generally sufficient to know the data type of the operands only. We do not have to know the actual content. The only exception is if we invert the sign of the most negative integer number or if we do an operation with causes an overflow, as there are not enough bits available to store the result — we may get a run-time error for that case.[11] In a statically-typed language each variable has a well-defined type, and the compiler can ensure at compile time that all operations on that variables are valid. If an operation is not valid then the compiler will give an error message. Then when these operations are executed at run-time they are always valid operations, and the actual content, like the actual numeric value, does not matter.

Hexadecimal Numbers

These number type with base 16 is by far not that important than the binary numbers, and it has not really a technical justification to exist, but you may get in touch with these numbers from time to time. Hexadecimal numbers are mostly a legacy from early days of computers, where computer programming was done not in real programming languages but with numeric codes. To represent the 16 hexadecimal digits the 10 decimal digits are supported by the characters 'A' .. 'F'. The most important property of a decimal digit is that it can represent four bits, a unit halve of a byte which is called sometimes a nibble. In old times when it was necessary to type in binary numbers it was sometimes easier to encode a nibble with a hexadecimal digit:

Decimal Binary Hexadecimal

0

0000

00

1

0001

01

2

0010

02

3

0011

03

4

0100

04

5

0101

05

6

0110

06

7

0111

07

8

1000

08

9

1001

09

10

1010

0A

11

1011

0B

12

1100

0C

13

1101

0D

14

1110

0E

15

1111

0F

The only location where we hear about hexadecimal characters again in this book should be when we introduce the character and string data types — there control characters like a newline character are sometimes specified in hexadecimal form like "\x0A" for a newline character.

Installation of the Compiler

We will not describe in too much detail how you can install the Nim compiler, because that strongly depends on your operating system, and because the install instructions may change in the future. We assume that you have a computer with an installed operating system and internet access, and you are able to do at least very basic operations with your computer, such as switching it on, log in and and opening a web browser or a terminal window. If that is not the case then you really should ask someone for help for this basic step, and maybe for some more help for other basic tasks.

Detailed installation instructions are available on the Nim internet homepage at To visit and read that page, you have to enter this string in the address input field of your internet browser. Try to follow those instructions, and when they are not sufficient, then please ask at the Nim forum for help: https://forum.nim-lang.org/

If you are using a Linux operating system, then your system generally provides a package manager, which should make the installation very easy.

For example for a Gentoo Linux system you would open a root terminal and simple type "emerge -av nim". That command would install Nim including all necessary dependencies for you. It may take a few minutes as Gentoo compiles all packages fresh from source code, but then you are done. Similar commands exist for most other Linux distributions.

Another solution, which is preferable when you want to ensure that you get the most recent Nim compiler, is compiling directly from the latest git sources. That process is also easy and is described here: https://github.com/nim-lang/Nim. But before you can follow those instructions you have to ensure that the git software and a working C compiler is available on your computer.

Creation of Source Code Files

Nim source code, as most source code of other programming languages, is based on text files. Text files are documents saved on your computer that contain only ordinary letters which you can type on your keyboard. No images or videos, no HTML content with fancy CSS styling. Generally source code should contain only ordinary ASCII text, that is no umlauts or unicode characters.

To create source code we generally use a text editor, which is a tool designed for creating and modifying of plain text files. If you do not have a text editor yet you may also use a word processor for writing some source code, but then you have to ensure that the file is finally saved as plain ASCII text. Editors generally support syntax highlighting, that is keywords, numbers and such are displayed with a unique color or style to make it easier to recognize the content. Some editors support advanced features like checking for errors while you type the program source code.

A list of recommended editors is available at https://nim-lang.org/faq.html

If you do not want to use a special editor now, then for Linux gedit or at least nano should be available. For Windows maybe something like notepad.

Generally we store our Nim source code files in its own directory, that is a separate section of your hard-disk. If you work on Linux in a terminal window, then you can type

cd
mkdir mynimfiles
cd mynimfiles
gedit test.nim

You type these commands in the terminal window and press the return key after each of the above lines — that is you type cd on your keyboard and then press the return key to execute that command. The same for the next three commands. What you have done is this: you go to your default working area (home directory), then create a subarea named mynimfiles, then you go into that subarea and finally you launch the gedit editor — the argument test.nim tells gedit that you want to create a new file called test.nim. If gedit is not available, or if you work on a computer without a graphical user interface, then you may replace the gedit command by nano. While gedit opens a new window with a graphical interface, nano opens only a very simple interface in the current terminal. An interesting editor without a GUI is vim or neovim. That is a very powerful editor, but it is difficult to learn and it is a bit strange as you have a command mode and an ordinary text input mode available. For neovim there is very good Nim support available.

If you do not want to work from a terminal, or if you are using Windows or MAC OS, then you should have a graphical user interface which enables you also to create a directory and to launch an editor.

When the editor is opened, you can type in the Nim source code from our previous example and save it to a file named test.nim. Then you can terminate the editor.

Note that the return key behaves differently in editors than in the terminal window: In the terminal window you type in a command and finally press the return key to "launch" or execute the command. In an editor the return key is not that special: if you press ordinary keys in your editor, than that key is inserted and the cursor moves one position to the right. And when you press the return key then an invisible newline character is inserted and the cursor moves to the start of the next line.

Launching the compiler and running the program

If you are working from a Linux terminal then you can type

ls -lt
cat test.nim

That is you first show the content of your directory with the ls command and then display the content of the Nim source code file that you just have typed in with the cat command.

Now type

nim c test.nim

That invokes the Nim compiler and instructs it to compile your source code. The "c" letter is called an option, it tells the Nim compiler to compile your program and to use the C backend to generate an executable.

The compiler should display nearly immediately a success message. If it displays some error messages instead, then you launch gedit or nano again, fix your typing error, save the modified file and call the compiler again.

Finally, when the source text is successfully compiled, you can run your program by typing

./test

In your terminal window you see a number now, which is the sum of the numbers 1 to 100.

You may wonder why you have to type the prefix ./ in front of the name of your generated executable program, as you can launch most other executables on your computer without such a prefix. The prefix is generally needed to protect you and your computer from erroneously launching a program in the current directory while you intended to launch a system command. Imagine you downloaded a zip file from internet, extract it, cd into the extracted directory and type ls to see the directory content. Imagine now that the directory contains an executable named ls which is executed instead of system ls. That foreign ls command may damage your system. So to execute non system commands, you generally have to use the prefix ./ where the period refers to the current directory. Of course you can install your own programs in a way that you don’t need such a prefix any more — just ask your Mom or Grandma if you don’t know yourself already.

If you have not managed to open a terminal where you can invoke the compiler — well maybe then you should install some of the advanced editors like VS-Code. They should be able to launch the compiler and run the program from within the editor directly.

The command

nim c test.nim

is the most basic compiler invocation. The extension .nim is optional, the compiler can infer that option. This command compiles our program in default debug mode, it uses the C compiler back end and generates a native executable. Debug mode means, that the generated executable contains a lot of checks, like array index checks, range checks, nil dereference checks and many more. The generated executable will run not very fast and it will be large, but when your program has bugs then the program will give you a meaningful error message in most cases. Only after you have tested your program carefully you may consider compiling it without debug mode. You may do that with

nim c -d:release test.nim

nim c -d:danger test.nim

The compiler option -d:release removes most checks and debugging code and enables the backend optimization by passing the option "-O3" to the C compiler backend, giving a very fast and small executable file. The option -d:danger removes all checks, it includes -d:release. You should be aware that compiling with -d:danger means that your program may crash without any useful information, or even bad, may run, but contain uncatched errors like overflows and so may give you wrong results. Generally you should compile your program with plain nim c first. When you have tested it well and you may need the additional performance, you may switch to -d:release option. For games, benchmarks or other uncritical stuff you may try -d:danger.

There exists many more compiler options, you can find them explained in the Nim manual or you may use the command nim --help and nim --fullhelp to get them displayed. One important new option is --gc:arc to enable the new deterministic memory management. You may combine --gc:arc with -d:useMalloc to disable Nims own memory allocator, this reduces the executable size and enables the use of Valgrind to detect memory leaks. Similar to --gc:arc is the option --gc:orc which can deal with cyclic data structures. Finally a very powerful option is --passC:-flto. This option is for the C compiler backend and enables link time optimization (LTO). LTO enables inlining for all procedure calls and can significantly reduce the final program size. We should mention that you can also try the C++ compiler backend with the cpp command instead of plain c command, and that you may compile with clang backend instead of default gcc backend with the --cc:clang option. You can additional specify the option -r to immediately run the program after successful build. For testing small scripts the compiler invocation in the form "nim r myfile.nim" can be used to compile and run a program without generation of a permanent executable file. Here is an example how we use all these options:

nim c -d:release --gc:arc -d:useMalloc --passC:-flto --passC:-march=native board.nim

In this example we additional pass -march=native to the C compiler backend to enable use of the most efficient CPU instructions of our computer, which may result in an executable that will not run on older hardware. Of course we can save all these parameters in configuration files, so that we don’t have to actual type then for each compiler invocation. You may find more explanations to all the compiler options in the Nim manual or in later sections of this book, this includes the options for the JavaScript backend.

Part II: The Basics

In this part we will introduce the most important constructs of the Nim programming language, like statements and expression, conditional and repeated execution, functions and procedures, iterators, templates, exceptions and we will discuss various basic data types including the basic container types array, sequence and string.

Declarations

We can declare constants, variables, procedures or our custom data types. Declarations are used to give information to the compiler, for example about the type of a variable that we intend to use.

We will explain type and procedure declarations in later sections. Currently only constant and variable declarations are important.

A constant declaration in its simplest form maps a symbolic name to a value, like

const Pi = 3.1415

We use the reserved word const to tell the compiler that we want to declare a constant which we have named Pi and we assign it the numeric decimal value 3.1415. Nim has a small set of reserved words like var, const, proc, while and others, to tell the compiler that we want to declare a variable, a constant, a procedure or that we want to use a while loop for some repeated execution. The = is the assignment operator in Nim, it assigns the value or expression on the right side of it to the symbol on the left. You have to understand that it is different from the equal sign we may use in mathematics. Some languages like Pascal initially used the compound operator := for assignments, but that is not easy to type on the keyboard and looks a bit angry for sensible people. And source code usually contains a lot of assignments, so use of = makes some sense. We call = an operator. Operators are symbols which perform some basic operation, like + for the addition of two numbers, or = for the assignment of a value to a symbol. With the above constant declaration we can use the symbol Pi in our program’s source code and don’t have to remember or retype the exact sequence of digits. Using named constants like our Pi above makes it easy to modify the value — if we notice that we need more precision, we can look up the exact value of Pi and change the constant at one place in our source code, we don’t have to search for the digit sequence 3.14 in all our source code files.

For numeric constants like our Pi value the compiler will do a substitution in the source code when the program is compiled, so where we write the symbol Pi the actual numeric value is used.

For constant declarations it must be possible to determine its value at compile time. Expressions assigned to constants can contain simple operations like basic math, but some functions calls may be not allowed.

Variable declarations are more complicated, as we ask the compiler to reserve a named storage location for us:

var velocity: int

Here we put the reserved keyword var at the beginning of the line to tell the compiler that we want to declare a variable, then we give our chosen name for that variable followed by a colon and the data type of the variable. The int type is a predefined numeric type indicating a signed integer type. The storage capacity of an integer variable depends on the operating system of your computer. On 32-bit systems 32 bits are used, and on 64-bit systems 64 bits are used to store one single integer variable. That is enough for even large signed integer numbers: the range is -2^31 up to 2^31 - 1 for 32 bit systems and -2^63 up to 2^63 - 1 for 64-bit systems.

For variables we generally use lower case names, but names of constants may start with an upper case letter.

Statements

Statements or instructions are a core component of Nim programs: they tell the computer what it shall do. Often statements are procedure calls, like the call of the echo() or inc() procedure which we have already seen in part I of the book. What procedures exactly are we will learn in later sections. For now we just regard procedures as entities that perform a well defined task for us when we call them. We call them by writing their name in our source file, optionally followed by a list of parameters, also called arguments. When we write echo 7 then echo is the procedure which we call, and 7 is the argument, an integer literal in this case. The effect of our procedure call is that the decimal number 7 is written to the terminal when we run the program after compilation. A special form of procedures are functions, that are procedures that can return a value or result. In mathematics, sin() or cos() would be functions — we pass an angle as argument and get the sine or cosine as a result.

Let’s look at this minimal Nim program:

var a: int
a = 2 + 3
echo a

The Nim program above consists of a variable declaration and two statements: in the first line we declare the variable we want to use. In the next line we assign the value 2 + 3 to it, and finally in line 3 we use the procedure echo() to display the content of our variable in the terminal window.

Nim programs are generally processed from top to bottom by the compiler, and when we execute the program after successful compilation, then it also executes from top to button. A consequence of this is that we have to write the lines of above program exactly in that order. If we moved the variable declaration down, then the compiler would complain about an undeclared variable because the variable is used before it has been declared. If we exchanged lines 2 and 3, then the compiler would be still satisfied, and we would be able to compile and run the program. But we would get a very different result, because we would first try to display the value of variable a, and later assign a value to it.

When we have to declare multiple constants or variables, then we can use a block, that is we write the keyword var or const on its own line, followed by the actual declarations like in

const
  Pi = 3.1415
  Year = 2020
var
  sum: int
  age: int

Note the indentation — the lines after const and var start with some space characters, so they build a block which allows the compiler to detect where the declaration ends. Generally we use two spaces for each level of indentation. Other numbers would work also, but the indentation scheme should be consistent. Two spaces is the general recommendation, as it is clearly recognizable for humans in the source code, and because it doesn’t waste too much space, that is, it would not generate long lines which may not fit onto the screen.

Also note that in Nim we generally write each statement onto its own line. The line break indicates to the compiler that the statement has ended. There are a few exceptions — long mathematical expressions can continue on the next line (see the Nim manual for details). We can also put multiple statements on a single line when we separate them by a semicolon.

We can also declare multiple variables of the same type in one single declaration, like

var
  sum, age: int

or we can assign an initial start value to a variable like in

var
  year: int = 1900

Finally, for variable declarations we can use type inference when we assign an initial start value, that is we can write

var
  year = 1900

The compiler recognizes in this case that we assign an integer literal to that variable and so silently gives the variable the int type for us. Type inference can be comfortable, but may make it harder for readers to understand the code, or the type inference may not always do exactly what we want. For example in the above code year gets the type int, which is a signed 4 or 8 byte number. But maybe we would prefer an unsigned number, or a number which occupies only two bytes in memory. So use type inference with some caution.

Note: For integral data we mostly use the int data type in Nim, which is a signed type with 4 or 8-byte size. It usually does not make sense to use many different integral types — signed, unsigned, and types of different byte size. Mixing them in numerical expressions can be confusing and maybe even decrease performance, because the computer may have to do type conversion before it can do the math operation. For unsigned types, another problem is that math operations on unsigned operands could have a negative result. Consider the following example where we use a hypothetical data type "unsigned int" to indicate unsigned integers:

var a, b: unsigned int
a = 3
b = 7
a = a - b

The true result would be -4, but a is of unsigned type and can never contain a negative content. So what should happen — an incorrect result or a program termination?

Related to variable declarations is the initial start value of variables. Nim clears for us all the bits of our variables when we declare them, that is, numbers get always the initial start value zero if we do not assign a different value in the variable declaration.

In this declaration

var
  a: int = 0
  b: int

both variables get the initial value zero.

There exists a variant for variable declarations which uses the let keyword instead of the var keyword. Let is used when we need a variable which only once gets a value assigned, while var is used when we want to change the content of the variable during program execution. Let seems to be similar to const, but in const declarations we can use only values that are known at compile time. Let allows us to assign to variables values that are only available at program run time, maybe because the value is a result of a prior calculation. But let indicates, at the same time, that the assignment occurs only once, the content does not change later, during the program’s execution. We say that the variable is immutable. Use of the let keyword may help the human reader of the source code with understanding what is going on, and it may also help the compiler doing optimizations to get faster, or more compact code. For now, we can just ignore let declarations and use var instead — later we may use let where appropriate, and the compiler will tell us when let will not work and we have to use var.

The way how we declare constants, variables, types and procedures in Nim is very similar as it was done in the wirthian languages Pascal, Modula and Oberon. People coming from languages like C argue sometimes that C uses a shorter and better variable declaration of the form int velocity; instead Nim’s var velocity: int. Indeed that declaration is shorter in this case. And some people like it better that the data type is written first, they consider the data type more important than the name of the variable. That is a matter of taste, and the C notation would not work well for var/let/const distinction and for type declarations.

With what we have learned in this section we can rewrite our initial Nim example from part I in this form:

const
  Max = 100
var
  sum, i: int
while i < Max:
  inc(i)
  inc(sum, i)
echo sum

In the code above we declare both variables of type int in a single line and take advantage of the fact that the compiler will initialize them with 0 for us. And we use a named constant for the upper loop boundary. Another tiny fix is that we write inc(i) instead of inc(i, 1). We can do that because there exists multiple procedures with the name inc() — one which takes two arguments, and one which takes only one argument and always increases that argument by one. Instead of inc(i) we could have written also i = i + 1 and instead of inc(sum, i) we could write sum = sum + i. That would generate identical code in the executable, so we can use whatever we like better.

Input and Output

We have already used the echo() procedure for displaying output in the terminal window. For our experiments we may want to have some user input also. As we do not know much about procedures currently, let us keep things simple for now and read in a textual user input from the terminal window. We use a procedure with name readLine() for this task.

echo "enter some text"
var mytext = readLine(stdin)
echo "you entered:", mytext

Note that you have to press the return key after you have entered your text.

The first line of our program is not really necessary; we just print some message. In the second line we use the readLine() procedure to read textual user input. The readLine() procedure needs one parameter to know from where it should read — from the terminal window or from a file for example. The stdin parameter indicates that it should read from the current terminal window — stdin is a global variable of the system (io) module and indicates the standard input stream. Finally in line 3 we use again the echo() procedure to print some text. In this case we pass two arguments to echo(), a literal text enclosed in quotes, and then separated by a comma, the mytext variable. The mytext variable has the data type string. We used type inference in this example to declare that type: the readLine() procedure always returns a string, the compiler knows that, so our mytext variable is automatically declared with type string. We will learn more about data type string and other useful predefined data types in the next section.

Nim supports the method call syntax, which was earlier called Uniform Function Call Syntax in the D language. With that syntax we can write procedure calls in the form a.f instead of f(a). We will discuss that syntax in more detail when we explain procedures and functions. For now it is enough that you know about the existence of that syntax, as we may use it at some places in the following sections. For example for the length of text strings we generally write myTextString.len instead of len(myTextString). Both notations are fully equivalent.

Data types

Integer types

We have already used the int data type, which indicates a signed integer type of 4 or 8-byte size, depending on the operating system. The reason the type depends on the word size of the OS will become clear later when we explain what references and pointers are.

Beside the int data type, Nim has some more data types for signed and unsigned integers: int8, int16, int32 and int64 are signed types with well-defined bit and byte size, and uint8, uint16, uint32 and uint64 are the unsigned equivalents. The number at the end of the type name is the bit size; we get the byte size when we divide that value by 8. Additional we have the type uint, which corresponds to int and has same size, but stores unsigned numbers only. [12] Generaly we should try to use the int type for all integral numbers, but sometimes it can make sense to use the other types. For example, when you have to work with a large collection of numbers, you know that each number is not very big, and your RAM is not really that large, then you may decide for example to use int16 for all your numbers. Or when you know that your numbers will be really big and will not fit in a 4 byte integer, then you may use the int64 type to ensure that the numbers fit in that type even when your program is compiled and executed on a computer with a 32 bit OS.

For integer numbers we have the predefined operators +, - and * available for addition, subtraction and multiplication. Basically these operations works as we may expect, but we have to remember that we may get overflows. For signed ints we get compile- or run-time errors in that case, while unsigned ints just wrap around, see example at the end of this section. For division of integers we have the operators div, mod, and / available. The div operator does an integer division ignoring the remainder, mod is short for modulus and gives us the remainder of the division, and / finally is currently only predefined for the signed int type and gives us a fractional result of data type float. That type is introduced in the next section.

Remembering how div and mod behaves when the divisor or dividend are negative can be confusing, and it may differ for other programming languages. You may find a detailed justified explanation in the Nim manual and at Wikipedia.

Result of i div j
   -4 -3 -2 -1  0  1  2  3  4
-4  1  1  2  4    -4 -2 -1 -1
-3  0  1  1  3    -3 -1 -1  0
-2  0  0  1  2    -2 -1  0  0
-1  0  0  0  1    -1  0  0  0
 0  0  0  0  0     0  0  0  0
 1  0  0  0 -1     1  0  0  0
 2  0  0 -1 -2     2  1  0  0
 3  0 -1 -1 -3     3  1  1  0
 4 -1 -1 -2 -4     4  2  1  1

Result of i mod j
   -4 -3 -2 -1  0  1  2  3  4
-4  0 -1  0  0     0  0 -1  0
-3 -3  0 -1  0     0 -1  0 -3
-2 -2 -2  0  0     0  0 -2 -2
-1 -1 -1 -1  0     0 -1 -1 -1
 0  0  0  0  0     0  0  0  0
 1  1  1  1  0     0  1  1  1
 2  2  2  0  0     0  0  2  2
 3  3  0  1  0     0  1  0  3
 4  0  1  0  0     0  0  1  0

When performance matters we generally should try to use the "CPU native" number type what for Nim is the int type. And we should try to avoid using math expressions with different types, as the CPU may have to do type conversion in that case before the math operation can be applied. Adding two int8 types can on some CPU’s be slower then adding two ints, because the CPU may have to size extend the operands before the math operation is performed. But this depends on the actual CPU, and there are important exceptions: Multiplying two ints would result in an int128 result if int size is 64 bit, which can be slow when the CPU does not support that operation well. Another important point to consider for maximum performance is the cache usage. If you are performing operations on a large set of data, then you may get a significant performance gain when large fractions of your data fits in the caches of your computer, as cache access is much faster than ordinary RAM access. So using smaller data types, i.e. int32 instead Nim’s default int which is int64 on a 64 bit OS may increase performance in this special application.

When we use Nim on tiny micro-controllers, maybe even on 8-bit controllers like the popular AVR devices, it may be best to use only integers of well defined size like int8.

An important property of the current Nim implementation of A. Rumpf used with the C backend is the fact that unsigned integers does not generate overflow errors but simple wrap around:

var x: int8 = 0

while true:
  inc(x)
  echo x

Above code would print the numbers 0 up to 127 and then terminate program execution due to an overflow error. But when we change the data type to uint8 we would get a continues sequence of the numbers 0 up to 255. After the value 255 is reached, the value wraps around to 0 again and the process continues. This behavior can lead to strange bugs and is the reason that the Nim team generally recommends to avoid unsigned integers.

For compatibility with external libraries Nim has also the integer types cint and cuint, which exactly match the C types int and uint when we compile for the C or C++ backend. For the javascript backend, the LLVM backend or other backends these types may be also avalable, for details you should consult the compiler documentation. For most operating system and C compilers the int and uint types in C are 4 bytes in size, but there can be exceptions, so we better should not write code that depends on the actual byte size of the types. The Nim types cint and cuint are generally only used for parameter lists of © library functions. To match other integer types like C char, short, long, longlong Nim supports these types when we put a c letter in front of the name like clong. Again you should consult the Nim compiler manual when you need more details, ie. when you create bindings to external libraries.

Floating point types

Another important numeric data type is float, for floating point numbers. Floats are an approximation of real numbers. They can also store fractions, and are most often printed in the decimal system with a decimal point, or in scientific notation with an exponent. Examples for floats are

var
  mean = 3.0 / 7.9
  x: float = 12
  y = 1.2E3

The variable mean is the result of a float division. If we printed the result there would be a decimal point and some digits following it. For variable x we specify the float type explicitly and assign the value 12. We could use type inference if we assigned 12.0, because the compiler can recognize by the decimal point that we want a float, not an int. In line 3 we use scientific notation for the float literal that we assign to y, and the value is 1.2 * 10^3 = 1200.0. Literal values like 2E3 are also valid float literals — the value would be 2000.0. But literals with a decimal point and no digits before or after the point — 1. or .2 — are not valid.

In the current Nim implementation float variables occupy 64 bits. Nim has also the data type float64 which is currently identical to plain float and float32 which can store only smaller numbers and has less precision.[13] Floats can store values up to a magnitude of approximately 1E308 with a positive or negative sign, and floats have a typical precision of 16 digits. That is, when you do a division of two arbitrary floats and print the result, you will get up to 16 valid digits.

Generally we use floats whenever integers are not sufficient for some reason. For example when we have to do complicated mathematical operations which include fractional operands like Pi, or when we have to do divisions and need the exact fractional value.

The float, float32 and float64 data types provides the +, -, * and \ operators for addition, subtraction, multiplication and division. Unlike for the int types, for the float types we get never overflow or underflow errors, and also no error for a division by zero. But the result of an operation of two float operands can be a special value like system.Inf, system.NegInf or system.NaN. The first two indicate an over- or underflow, and NaN indicates that the result of an operation is not a valid number at all, for example the result of a division by zero or the result of calculating the square root of a negative number. This behaviour is sometimes called saturated arithmetic. When a variable has one of these special values, and we apply further math operations, then this value is kept. So we can detect at the end of a longer mathematical calculation if something went wrong — we have not to check after each single operation.[14]

Two important properties of floats are that not all numbers can be represented exactly and that math operations are not absolutely accurate. When you do an addition like 1.0 + 2.0, the result will not be exactly 3.0, and 1.0 + 2.0 - 3.0 will not be exactly zero, but a tiny floating point number. Maybe in the case when you use the above literal values the compiler may be able to recognize the expression and deliver the exact value, but when you assign the literals to variables and then do the math, you can be sure that it is not absolutely accurate. That is no real problem: the accuracy is in most cases much better than needed. But you should know about the minimal numeric errors and should avoid testing two floats for exact equality, because they may not be equal due to rounding errors, even when in theory the values should exactly match. Instead of a test for equality sometimes expressions like abs(a - b) < 0.000001 are used, ie, we take the absolute value of the difference and compare it with a tiny epsilon. If you ever should use a test like that, think carefully about it — sometimes it may be better to make an even more complicated test regarding not the absolute, but the relative difference of the numbers.

For floats we have the operators +, -, * and / for addition, subtraction, multiplication and division. For powers with integral exponents you can use the ^ operator, but you have to import it from the math module. x ^ 3 is the same as x * x * x. The math module contains many more functions like sin() or cos(), sqrt() and pow(). sqrt() is short for square-root, pow() for power, so pow(x, y) is x to the power of y, when both operands have type float.

The operators +, -, * and / can be used also when one operand is a float variable and the other operand is an int literal. But when one operand is a float variable and the other is an int variable, then a type conversion is necessary like in float(myIntVal) * myFloatVal.

As for integer variables Nim supports also the compatible types cfloat and cdouble which match the C types float and double when the C backend is enabled. For most C compilers C float matches Nim’s float32 and C double matches Nim’s float64.

Some CPUs and C compilers support also other floating point types beside the common float32 and float64 types. Intel x86 compatible CPUs generally support float80 math operations, and the gcc C compiler may support float128. But these types are not yet supported by the Nim compiler of A. Rumpf.

Distinct types

Before we continue with subrange types we should introduce the distinct types. In the real world we have a lot of quantities for which the set of meaningful math operations is restricted and which should not be mixed with quantities of other types. For example we may have the quantities time and distance measured in seconds and meters and mapped to the float or int data type. While adding seconds and adding meters is a valid operation, adding seconds to meters makes no sense and would be a program bug if it should occur in the program code. But again dividing a distance by a time period resulting in the average speed would be a valid operation. Nim provides the distinct keyword which allows us to define new data types that are based on existing types, but that are not compatible with them or with other distinct types. And the new defined distinct types have no predefined operations, we have to define all desired operations our self.

type
  Time = distinct float # in seconds
  Distance = distinct float # in meters

var t: time = 0.2 # not allowed
var t: Time = Time(0.2)

For distinct types we have to define all the allowed operations our self. We can convert distinct types to the base types and then use operations of the base type or we can borrow operations from the base type by use of the {.borrow.} pragma. Using distinct types can be complicated when the new type should support many operations, but it can make our code more save. For some data type with a very limited set of operations distinct types can be used easily. Distinct types are explained in detail in the Nim manual, we may explain then in more detail in later sections. For now it is enough that we know about their existence.

Subrange types

Sometimes it makes sense to limit the range of numeric variables to only a sub-range. For this Nim uses the range keyword with this notation: range[LowVal .. HighVal]. Values of this type can never be smaller than LowVal or larger than HighVal.

type
  Year = range[2020 .. 2023] # software update required at least for 2024!
  Month = range[1 .. 12]
  Day = range[1 .. 31]

var d: Day = 0 #Error: conversion from int literal(0) to Day is invalid

For the above example the base type of the defined ranges is int, so the ranges are compatible with the predefined int type, we can assign values of int type to our range types and vice versa. The size of the range types is the size of the int type. If we try to assign to a range type a value that falls not into the allowed range then we get a compile-time or run-time range error. This can help us to prevent or to discover errors in our programs. We can also declare subrange types with float base types like type Probability = range[0.0 .. 1.0].

Note that we can still mix different sub-range type:

var d: Day = 13
var m: Month = 3
d = d + m

Such an operation is generally a bug, to prevent it we can put the distinct keyword in from of our ranges. But then again we have to define the allowed operations our self.

Enums

While enums in C are nothing more than integers with some special syntax for creation, Nim’s enums are more complex.

In Nim enums can be used whenever some form of symbols are needed like the colors red, yellow and green of a traffic light or the directions north, south, east and west for a map or a game.

Most of the time we declare an enum type and the corresponding values by simple listing them like

type
  TrafficLight = enum
    red, yellow, green

We can use variables of type TrafficLight then like

var tl: TrafficLight
tl = green
if tl == red:
  tl = ...

Enums support assignment, plain tests for (in)-equality and for smaller or greater. Additional the functions succ() and pred() are defined for enums to get the successor or predecessor of an enum, ord() or int() deliver the corresponding integer number and the $ operator can be used to get the name of an enum. We can also iterate over enums, so we can print all the colors of our TrafficLight by

for el in TrafficLight
  echo el.ord, ' ', $el

Ordinary enums start at 0 and uses continues numbers for the internal numeric value, so that enums can be used as array indices.

type
  A = array[TrafficLight, string]

var a: A
a[red] = "Rot"
echo a[red]

But we can also assign custom numbers like

type
  TrafficLigth = enum
    red = -1, yello = 3, green = 8

We should avoid that, as these "enums with holes" generate some problems for the compiler and may be later deprecated. For example array indexing or iterating is obviously not possible for enums with holes.

It is also possible to set the string that the stringify operator $ returns, like in

type
  TrafficLigth = enum
    red = "Stop"
    yellow = (2, "Caution")
    green = ("Go")

Here the assigned numerical values should be 0, 2 and 3. Currently the enums values must be specified in ascending order always.

When we have many enums in a program then name conflicts may occur, for example we may have an additional enum type named BaseColor which also has red and green members. For that case the {.pure.} pragma exists:

type
  BaseColor {.pure.} = enum
    red, green, blue

With the pure pragma applied we can use the full qualified enum name when necessary, like BaseColor.red. But we can still use unqualified names like blue when there is no name conflict.

Boolean types

Boolean types are used to store the result of logic operations. The type is called bool in Nim and can store only two values, false and true. Although we have only two distinct states for a boolean variable and so one single bit would suffice to store a bool, generally a whole byte (8 bits) is used for storing a boolean variable. Most other programming languages including C do the same. The reason is that most CPU’s can not access single bits in the RAM — the smallest entity that can be directly accessed in RAM is a byte. The default initial state of a boolean variable is false, corresponding to a byte with all bits cleared.

var
  age = 17
  adult: bool = age > 17
  iLikeNim = true
  iLikeOtherLangaugeBetter = false.

In line three we assign to the variable adult the result of a logical comparison. The next two lines assign the boolean constants true and false to the variables, with their type bool inferred.

Variables of type bool support the operators not, and, or and xor. Not inverts the logic value, a and b is only true when both values are true, and false otherwise. And a or b is true when at least one of the values is true, and only false when both values are false. xor is not used that often. It is called exclusive or, a xor b is false when both values have the same logic state, that is when both are true, or both are false. When the values are not the same, than the result of the xor operator is true.

Characters

The data type for single characters is called char in Nim. A variable of type char has 8 bits and can store single characters. Indeed it stores 8-bit integers which are mapped to characters. The mapping is described by the ASCII table. For example the integer value 65 in decimal is mapped to the character A. When we use single character literals, then we have to enclose the letter in single quotes. As only 8 bits are used to store characters, we only have 256 different values, including upper and lower case letters, punctuation characters and some characters with a special meaning like a newline character to move the cursor in the terminal to the next line, or a backspace character to move the cursor one position backwards. Single characters are not used that often since we generally group them in sequences called strings to build text.

The initial ASCII table contains only the characters with numbers 0 up to 127, here is an overview generated with the small program listed in the appendix:

Visible ASCII Characters

      +0   +1   +2   +3   +4   +5   +6   +7   +8   +9  +10  +11  +12  +13  +14  +15
  0
 16
 32        !    "    #    $    %    &    '    (    )    *    +    ,    -    .    /
 48   0    1    2    3    4    5    6    7    8    9    :    ;    <    =    >    ?
 64   @    A    B    C    D    E    F    G    H    I    J    K    L    M    N    O
 80   P    Q    R    S    T    U    V    W    X    Y    Z    [    \    ]    ^    _
 96   `    a    b    c    d    e    f    g    h    i    j    k    l    m    n    o
112   p    q    r    s    t    u    v    w    x    y    z    {    |    }    ~

The position in the table is the sum of the number on the left and the number on the top, ie, character A has position 64+1=65, which is the value the Nim standard function ord('A') or int('A') would return. The characters with a decimal value less than 32 can not be printed and are called control characters, like linefeed, carriage return, backspace, audible beep and such. Character 127 is also not printable, and is called DEL. An important property of this table is the fact that decimal digits and upper- and lower-case letters form contiguous blocks. So to test for example if a characters is an uppercase letter we can use this simple condition: c >= 'A' and c <= 'Z'.

Characters with ord() > 127 are so called umlauts, exotic characters of other languages, and some special characters. But these characters may be different on different computers, as the characters depend on the active code-page, which maps position to actual character, and there are multiple code pages. When we need more than the plain ASCII characters, then we use strings in Nim, which display many more glyphs by using UTF-8 encoding.

The control characters with a decimal value less than 32 can not be typed on the keyboard directly and for some characters with decimal value greater than 126 it can be difficult to enter them on some keyboards. For these characters as well as for all other characters escape sequences can be used. Escape sequences start with the backslash character, and the following characters are interpreted in a special way: The backslash can follow a numeric value in decimal or hexadecimal encoding, or a letter which is interpreted in a special way. We mentioned already that the character 'A' is mapped to the decimal value 65, which is its position in the ASCII table. So instead of 'A' we could use the escape sequence '\65' for this character. Or, as decimal 65 is 41 in hexadecimal notation (4 * 16^1 + 1 * 16^0) we can use '\x41' where the x indicates that the following digits are hexadecimal. For common, often used control characters it is not easy to remember their numeric value, so another notation with a letter following the backslash can be used. For the important newline character we can use the decimal numeric value '\10', the hexadecimal value '\xA' or the symbolic form '\n'. Here the letter n stands for newline. The following table lists a few important control characters:

Decimal Hexadecimal Symbolic Meaning

10

xA

\n, \l

newline or linefeed — move cursor one position down

12

xC

\f

formfeed

9

x9

\t

tabulator

11

xB

\v

vertical tabulator

92

x5C

\\

backslash

39

x27

\'

single quote, apostrophe

7

x7

\a

alert, audible beep

8

x8

\b

backspace

27

x1B

\e

Escape, [ESC]

13

xD

\r, \c

return or carriage return — move cursor at the beginning of the line

The hexadecimal numbers after the \x character can by upper or lower case and can have one or two hexadecimal digits. For symbolic control characters like '\a' for alert the upper case variant '\A' seems to be identical currently. The single quote entered as ''' does give an error message, so you have to escape it as '\''. Unfortunately by supporting this form of escaping it becomes impossible to enter a backslash character directly, so we have to escape the backslash character as '\\' to print a single backslash.

For Nim the most important control character is '\n' which is used to start the output in a terminal window at the beginning of a new line. But '\n' is generally not used as a single character but embedded in strings, that is sequences of characters. We will learn more about strings soon. Note that the echo() function inserts a newline character automatically after each printed line, but the write() function does not:

echo 'N', 'i', 'm'
stdout.write 'N', 'i', 'm', '\n'

What may be a bit confusing is the fact that we use the backslash character as escape symbol, and at the same time above table has an entry '\e' which is also called [ESC]. These '\e' control character with decimal value 27 is fully unrelated to the backslash character that we use to type in control characters. [ESC] is a special character to start control sequences, it was used in the past to send special commands to printers or modems, and can be used to control font style or colors in terminal windows.

Nim’s control characters should, with few exceptions, be identical with control characters of the C language, so you may also consult C literature for more details.

Ordinal types

In Nim integers, enumerations, characters and the boolean types are ordinal types. Ordinal types are countable and ordered, and for each of these types a lowest and largest member exists. The integer ordinal types supports the inc() and dec() operations to get the next larger or next smaller value, and the other ordinal types use succ() and pred() for this operation. These operations can produce overflow or underflow like errors if applied to largest or smallest value. The function ord() can be used on ordinal types to get the corresponding integer value. Note that unsigned integers are currently not called ordinal types in Nim, and that these unsigned types wrap around instead of generation overflow and underflow errors.

Sets

From mathematics we know that sets are some form of unordered collection for which we can test membership (x is included in mySet) and we can perform general set operations like union of multiple sets. In Nim we can have sets of all the ordinal types. All elements in a set must have the same base type. A set can be empty, or it can contain one or multiple elements. For a specific element it can be contained in a given set or it can be not contained, but it can be never contained multiple times. One very basic set operation is the test if an element is contained in a set or is not contained in it. Sets are unordered data types, that is sets containing the same elements are always equal, it does not matter in which sequence we added the elements. Important set operations are building the union and building the difference: The union of set a and set b is a set which contains all the elements that are contained in set a or in set b (or in both). The intersection of set a and b is a set which contains only elements which are contained in set a and in set b.

The mathematical concept of sets maps well to words and bits of computers, as most CPU’s has instructions to set and clear single bits and to test if a bit is set or unset. And CPU’s can do and, or and xor operations which corresponds to the union and intersection operation in mathematical set.

Nim supports sets with base type bool, enum, char, int8, uint8, int16 and uint16. Note that we need a bit in the computer memory for each member of the base type. The types char, int8 and uint8 are 8 bit types and can have 2^8 = 256 distinct values, so we need 256 bits in the computer memory to represent such a set. That would be 8 bytes or four 64 bit words. To represent a set of the base type uint16 or int16 we need already 2^16 bits, that is 2^13 bytes or 2^10 words on a 64 bit CPU. So it becomes clear that supporting base types with more than 16 bit makes not much sense.

We will start our explanation with sets with character base type as these sets are very easy to understand and at the same time very useful. Let us assume that we have a variable of character type and we want to test if that variable is alphanumeric, that is if it is a lower or upper case letter or a digit. A traditional test would be (x >= a and x ⇐z) or (x >= A and x ⇐ Z) or (x >= 0 and x ⇐ 9). Using Nim’s set notation we can write that in a simpler form:

const
  AlphaNum: set[char] = {'a' .. 'z', 'A' .. 'Z', '0' .. '9'}

var x: char = 's'
echo x in AlphaNum

Here we defined a constant of set[char] types which contains lower and upper case letters and the decimal digits. We used the range notation to save us a lot of typing ({'a', 'b', 'c', …​}). It works in this case only, as we know that all the lowercase letters, the upper case letters and the decimal digits built an uninterrupted range in the ASCII table.

With that definition we can use a simple test with the in keyword. These test is equivalent to the procedure call AlphaNum.contains(x). In compiled languages set operations are generally very fast as they map well to CPU instructions.

Older languages like C have not a dedicated set data type, but as sets are so useful and efficient, C emulates these operations by using logical and and or operations in conjunction with bit shifts.

Two important operations for sets are building the union and the intersection:

const
  AlphaNum: set[char] = {'a' .. 'z', 'A' .. 'Z', '0' .. '9'}
  MathOp = {'+', '-', '*', '/'} # set[char]

  ANMO = AlphaNum + MathOp # union
  Empty = AlphaNum * MathOp # intersection

The constant ANMO would now contain all the characters from AlphaNum and from MathOp, that is letters, digits and math operators. The constant Empty would get all the characters that are at the same time contained in set AlphaNum and in set MathOp. As there is not a single common characters, the set Empty is indeed empty. Remembering the two operators + and * for union and intersection is not easy. For the intersection operator * it may help when we imagine the set members as bits, and we assume that we multiply the bits of both operands bitwise, that is we multiply the set or uset bits at corresponding position each. The resulting bit pattern would get set bits only for positions where both arguments have set bits.

We can use the functions incl() and excl() to add or remove single set members:

var s: set[char]
s = {} # empty set
s = {'a' .. 'd', '_'}
s.excl('d')
s.incl('?')

The result is a set with letters a, b, c and the characters _ and ?. Note that calling incl() has no effect when the value is already included in the set, and calling excl() has no effect when the value is not contained in the set at all.

Another operation is the difference of two sets — a - b is a set which contains only the elements of a which are not contained in b. In Nim there is currently no operator for the complement or the symmetric difference of sets available. We can produce a set complement by using a fully filled set and then removing the elements of which we want the complement. For a character set that would look like {'\0'..'\255'} - s, where s is the set to complement. And the symmetric difference of set a and set b can be generated by the operation (a+b) - (a*b) or by (a-b) + (b-a).

As the not operator binds more tightly than the in operator, we have to use brackets for the inverted membership test like not(x in a) or we can use the notin operator and write x notin a. We can test for equality of sets a and b like a == b and for subset relation a < b or a ⇐ b. a ⇐ b indicates that b contains at least all members of a, and a < b that b contains all members of a and at least one more element.

Finally we can use the function card() to get the cardinality of a set variable, that is the number of contained members.

We should also mention that we can have character sets which are restricted to a range of characters:

type
  CharRange = set['a' .. 'f']

# var y: CharRange = {'x'} #invalid

var y: CharRange = {'b', 'd'}
echo 'c' in y

In the code above the compiler detects the assignment to variable y as invalid.

Set of numbers work in principle in the same way as sets of characters. One problem is that in Nim integer numbers are generally 4 or 8 bytes large, but sets can contain only numbers with 1 or 2 byte size. So we have to specify the type of set members explicitly:

type
  ChessPos = set[0'i8 .. 63'i8]

var baseLine: ChessPos = {0.int8 .. 7.int8}
var p: int8
echo p in baseLine

In the code above we defined a set type which can contain int8 numbers in the range 0 to 63.

We can use also another notation for numeric sets when we define an explicit range type like in

type
  ChessSquare = range[0..63]
  ChessSquares = set[ChessSquare]

const baseLine = {0.ChessSquare .. 7.ChessSquare}
# or
const baseLineExplicit: ChessSquares = {0.ChessSquare .. 7.ChessSquare}
assert baseLine == baseLineExplicit

What may be a bit surprising is the fact that Nim’s sets work also for negative numbers:

type
  XPos = set[-3'i8 .. +2'i8]

var xp: XPos = {-3.int8 .. 1.int8}
var pp: int8 = -1

Enum sets are also very useful and can be used to represent multiple boolean properties in a single set variable instead of using multiple boolean variables for this purpose:

type
  CompLangFlags = enum
    compiled, interpreted, hasGC, isOpenSource, isSelfHosted
   CompLangProp = set[CompLangFlags]

const NimProp:  CompLangProp = {compiled, hasGC, isOpenSource, isSelfHosted}

Enum sets can be used to interact with functions of C libraries where for flag variables often or’ed ints where used. For example for the gintro C bindings there is this definition:

type
  DialogFlag* {.size: sizeof(cint), pure.} = enum
    modal = 0
    destroyWithParent = 1
    useHeaderBar = 2

  DialogFlags* {.size: sizeof(cint).} = set[DialogFlag]

Here the {.size.} pragma is used to ensure that the byte size of that set type matches the size of integers in C languages.

When we define set of enums in this way to generate bindings to C libraries, then we have to ensure that the enum values start with zero, otherwise Nim’s definition will not match with the C definition. For example in the gdk.nim module we have

type
  AxisFlag* {.size: sizeof(cint), pure.} = enum
    ignoreThisDummyValue = 0
    x = 1
    y = 2
    pressure = 3
    xtilt = 4
    ytilt = 5
    wheel = 6
    distance = 7
    rotation = 8
    slider = 9

  AxisFlags* {.size: sizeof(cint).} = set[AxisFlag]

The first enum with ordinal value zero was automatically added by the bindings generator script to ensure type matching. Nim devs sometimes recommend to use plain (distinct) integer constants for C enums. That may be easier, but integer constants provide no name spaces, so names may be aFlagWheel instead of AxisFlag.wheel or plain wheel when there is no name conflict for pure enums. And with integer constants we have to combine flags by or operation like (aFlagWheel or aFlagSlider) instead of clean {AxisFlag.wheel, slider}.

Can we print sets easily? As sets are an unordered type, it is not fully trivial, but we can iterate over the full base type and check if the element is contained in our set like

var s: set[char] = {'d' .. 'f', '!'}

for c in 0.char .. 255.char:
  if c in s:
    stdout.write(c, ' ')
echo ' '
! d e f

We will learn how the for loop work soon. Note that the sequence in which the set members are printed is determined by our query loop, not by the set content itself, as sets are unordered types.

Strings

The string data type is a sequence of characters. It is used whenever a textual input or output operation is performed. Usually it is a sequence of ASCII-only characters, but multiple characters in the string can be interpreted as so called utf-8 unicode characters, that allow displaying nearly unlimited symbols as long as all the needed fonts are installed on your computer and you manage to enter them — unicode characters may be not accessible by a simple keystroke. For now we will only use ASCII characters, as they are simpler and work everywhere. String literals must be enclosed in double quotes. Nim strings are similar to the Nim seq data types: both are homogeneous variable-size containers. That means that a string or a seq expands automatically when you append or insert characters or other strings. Don’t confuse short strings with only one character with single characters: A string is an non trivial entity with internal state like data buffer (the actual contained characters), length and storage capacity why a character is nothing more than a single byte interpreted in a special way. So a string like "x" is fully different from 'x'.

var
  str: string = "Hello"
  name: string
echo "Please tell me your name"
name = readLine(stdin)
add(str, ' ')
echo str, name

In above example code we declare a variable called str and assign it the initial literal value "Hello". We use the echo() procedure to ask the user for his name, and use the readLine() procedure to read the user input from the terminal. To show how we can add characters to existing string variables we call the add() procedure to append a space character to our str variable, and finally call the echo() procedure to print the hello message and the name to the screen. Note that the echo() procedure automatically terminates each output operation with a jump to the next line. If you want an output operation without a newline, you can use the similar write() procedure. But write() needs an additional first parameter, for which we use the special variable stdout when we want to write to the terminal window.

So we could substitute the last two lines of the above code by

write(stdout, str)
write(stdout, ' ')
echo name

The Nim standard library provides a lot of functions for creating and modifying strings, most of these functions are collected in the system and in the strutils module. The most important procedures for strings are len() and high(). The len() procedure returns the length of a string, that is the number of ASCII characters or bytes that the string currently contains. The empty string "" has length zero. Note that the plain len() function returns the number of 8-bit characters, not the number of unicode glyphs when the string should be interpreted as unicode text. To determine the number of glyphs of unicode strings you should use some of the unicode modules. The high() function is very similar to the len() function, it returns the index of the last character in the string. For each string s high(s) == len(s) -1. Remember that Nim supports method call syntax, so we can also write s.len instead len(s).

The most important operators for strings are the subscript operator [] which allows access to individual characters of strings, and the slice operator which allows access to sub-strings. The first character in a string has always the index zero. For concatenation of string literals or string variables Nim uses the & operator.

var s = "We hate " & "Nim?"
s[3 .. 6] = "like"
s[s.high] = '!'

In the example above we define the string s by use of two literal strings to show the use of the concatenation operator. In line two we use the slice operator to replace the substring "hate", that is the characters 3 up to 6, with the string "like". In this case the replacement has exactly that many characters as the text to replace, but that is not necessary: We can replace substrings with longer or shorter strings, which includes the empty string "" to delete a text area. In the last line of above example we use the subscript operator [] to replace the single character '?' at the end of our string with an exclamation mark. For subscript and slice operators Nim supports also a special notation which indicates indexing from the end of the string. Python and Ruby use negative integers for this purpose, while Nim uses the ^ character. So ^1 is the last character, ^2 the one before the last. So we could have written s[^1] = '!' for the last line of our code fragment above. The reason that Nim does not use negative integers for this purpose is that Nim arrays don’t have to start at index zero, but can start with an arbitrary index including negative indices, so for negative indices it may be not always clear if a regular index or a position from the end of the string is desired. The term s[^x] is equivalent to s[s.len - x]. We will learn some more details about the slice operator in a later section when we have introduced arrays and sequences.

Another important operator for strings is the "toString" or stringify operator $. It can be applied to nearly all data types and converts them to strings which can then be printed. Some procedures like echo() apply this operator on its arguments automatically. When we define our own data types then it can make some sense to define the $ for them, in case we need a textual representation of our data — maybe only for debugging purpose. Note that applying the $ operator on a string directly has no effect and is ignored, as the result would not change.

Strings can contain all characters of the char data type including the control characters. The most important control character for strings is the newline character '\n' which is used at the end or sometimes also in strings to start a new line. For strings Nim also supports the virtual character "\p" to encode an OS dependent line break. When compiled for Windows, "\p" is automatically converted to "\r\n", and to plain '\n' on Linux. Note that "\p" can be used in strings, but not as single character, as it is two byte on Windows. "\p" is only needed to support very old Windows version or maybe other exotic operating system, as modern Windows recognizes plain '\n' well.

As strings support utf-8 unicode, an escape sequence starting with "\u" is supported to insert unicode codepoints. The "\u" follows exactly 4 hexadecimal digits or an arbitrary number of hex digits enclosed in curly braces {}.

As string literals are enclosed in quotation marks, it follows that strings can not directly contain this character, we have to escape it as in "\"Hello\", she said".

Maybe we should mention that Nim strings use copy semantics for assignment. As we have not yet introduced ref objects or pointers, copy semantics is what you should expect — strings behave just like all the other simple data types we used before like integer or float numbers or enums and characters:

var
  s1: string
  s2: string
s1 = "Nim"
s2 = s1
s1.add(" is easy!")
echo s1 & "\n" & "s2"

The output is

Nim is easy!
Nim

The assignment s2 = s1 creates a copy of s1, so the subsequent add() operation does only modify s1 but not s2. Probably not surprising for you, but other programming languages may behave differently, ie. the assignment may not copy the textual content but create only a reference to the first string, so that modifying one of then also effect the other. We will learn more about the concept of references when we introduce the object data type.

Entering Unicode Characters

UTF-8 is a variable-width character encoding. To cite the introducing section from https://en.wikipedia.org/wiki/UTF-8:

UTF-8 is capable of encoding all 1,112,064[nb 1] valid character code points in Unicode using one to four one-byte (8-bit) code units. Code points with lower numerical values, which tend to occur more frequently, are encoded using fewer bytes. It was designed for backward compatibility with ASCII: the first 128 characters of Unicode, which correspond one-to-one with ASCII, are encoded using a single byte with the same binary value as ASCII, so that valid ASCII text is valid UTF-8-encoded Unicode as well. Since ASCII bytes do not occur when encoding non-ASCII code points into UTF-8, UTF-8 is safe to use within most programming and document languages that interpret certain ASCII characters in a special way, such as "/" (slash) in filenames, "\" (backslash) in escape sequences, and "%" in printf.

In Nim we have four ways to enter unicode characters: As hexadecimal digits following the "\x", as unicode codepoint following the "\u" or we can type the unicode sequence directly on our keyboard, as one single keystroke when our keyboard layout supports that, or as a special OS dependent sequence of keystrokes:

echo "\xe2\x99\x9A \xe2\x99\x94"
echo "\u265A \u2654"
echo "\u{265A} \u{2654}" # {} is only necessary for more than 4 hex digits
echo "♚ ♔"

The code above shows three was to print the symbol for a black and a white king of a chess game. In the first line we typed the unicode sequence directly as hexadecimal digits, this method is rarely used today. In the second line we used "\u" to enter the codepoint directly, we get the code from https://en.wikipedia.org/wiki/List_of_Unicode_characters. And finally we entered the glyph directly in an editor. For some Linux editors like gedit we can hold down shift and control key and then type u, release all keys and type the unicode digits like 265a and a space. See https://en.wikipedia.org/wiki/Unicode_input for details and other operating system.

The CString data type

In the C language strings are just pointers to sequences of characters of fixed length.[15] The end of such a C string is generally marked with the character '\x0'. C functions like printf() needs this "\x0" characters to determine the end of the C string. While Nim strings are entities that store its current size and other properties, and can grow dynamically, the character sequence of Nim strings has also a hidden terminating '\0' character at the end to make them compatible with C strings. Nim has also the data type cstring, called "compatible string" in modern Nim, which matches the strings in C language if we compile as usual with C backend. Cstrings are used in bindings definitions for C libraries, but as cstrings can not grow and do support only few string operations, they are only used in rare cases in ordinary Nim source code. The Nim compiler passes automatically the zero terminated data buffer of Nim strings to C libraries whenever we call a C library, so there is no expensive type conversion involved. But the other way is much more expensive: When you have an existing cstring and need a Nim string with same content, then a simple conversion is not possible as a Nim string is a different, more complex entity. So we have to create a Nim string and copy the content, you can use the stringify operator $ for this like in myNimStr = $myCString. Generally string creation is an expensive operation compared to plain operations like adding two numbers, so when performance matters one should try to avoid unnecessary string creation and also unnecessary string operations. This is mostly important in loops, which are executed often. We will explain more about the internal of strings and why string creation and dynamically allocating memory is expensive in later sections of the book.

When we access text ranges with the slice operator or single characters with the subscript operator we should never access indices below the currently last index, which is the index mystr.high or ^1. If we do that we get an exception, as that index would contain undefined data or would not exist at all. We said earlier that Nim strings grow automatically if we insert or append data. But that does not mean that we can use the subscript or slice operator to access characters after the current end of the string. Such an operation would really make not much sense: Imagine we have a string var str = "Nim" and now use the subscript operator and assign a character at position 10 with str[10] = '!'. What should become the content of characters 4 .. 9? Well maybe spaces would make some sense, but in fact such an access after the currently last valid character of the string is forbidden. You could do str.add(" !") for this purpose.

Escape Sequences in Strings

We learned about control characters already in the section about characters, and earlier in this section we mentioned that strings can also contain control characters. As the use of control characters may be not really easy to understand, we will explain their use in strings in some more detail and give a concrete example.

The most important control characters for strings is the newline character, which moves the cursor in the terminal window to the beginning of the next line. The echo() procedure prints that character automatically after each output operation. Indeed it can be important to terminate each output operation with that character, as the output can be buffered, and writing just a string without a termination newline may not appear at once on the screen, but can be delayed. That is bad when the user is asked something and should respond, but the message is still buffered and not yet visible.

The problem with special characters like backspace or newline is that we can not enter them directly with the keyboard.[16] To solve that problem, escape sequences were introduced for most programming languages. An escape sequence is a special sequence of characters, that the compiler can discover in strings and then replace with a single special character. Whenever we want a newline in a string we type it as "\n", that is, the backslash character followed by an ordinary letter n, n for newline.

echo "\n"
echo "Hello\nHello\nHello"

The first line prints two empty lines — two because the \n generates a jump to next line, and because echo() always adds one more jump to next line. The second line prints three lines which each contains the word Hello, and the cursor is moved below the last Hello, because echo() automatically adds one more newline character.

Older Windows versions used generally a sequence of two control characters to start a new line, one '\r' (carriage-return) to move to the start of the line, and one '\l' (linefeed) to move down. You may still find these two characters in old Windows text files at the end of each line. Old printers used these combination too, so it was possible to send that text files to old printers directly. Nim also has the special escape sequence "\p" which is called platform dependent newline and maps to "\c\l" on Windows. That is when we compile our program on Windows, then the compiler replaces "\p" in our strings with a carriage-return and a linefeed character, and when we compile on Linux then the compiler replaces "\n" only with a newline character. But modern Windows supports '\n', so we generally can use that.

Raw Strings and multi-line Strings

In rare situations you may want to print exactly what you have typed, so you do not want the compiler to replaces a \n by a newline character. You can do that in two ways: You can escape the escape character, that is you put in front of the backslash one more backslash. When you print the string "\\n" you will get a backslash and the n character in your terminal. Or you can use so called raw strings, that is you put the character r immediately in front of your string like

echo r"\n"
echo "\\n"

Multi-line strings are also raw strings, that is contained escape-sequences are not interpreted by the compiler, and additional multi-line strings, as the name implies, can extend over multiple lines of the source text. Multi-line texts starts and ends with three quotes like in

echo """this is
three lines
of text"""

echo "this is\nthree lines\nof text"

Both echo() commands above generates exactly the same machine code!

Comments

Comments are not really a data type, but they are also important. Ordinary comments starts with the hashtag character # and extend to the end of the line. The # character itself and all following characters up to the line end are ignored by the compiler. You can also start the comment with ##, then it is a documentation comment. It is also ignored by the compiler, but can be processed when you use later tools to generate documentation for your code. Documentation comments are only allowed at certain places, often they are inserted at the beginning of a procedure body to explain its use. There are also multi-line comments, which starts with the two characters #[ and ends with ]#. These form of comment can extend over multiple lines and can be nested, that is multi-line comments can contain plain or multi-line comments.

# this is comment
## important note for documentation
#[ a longer
but useless comment
#]

Multi-line documentation comments also exists and can also be nested.

proc even(i: int): bool =
  ##[ This procedure
  returns true if the integer argument is
  even and false otherwise.
  ]##
  return i mod 2 == 0

You can also use the #[ comment ]# notation to insert comments everywhere in the source code at places where a whitspace character is allowed, but these form of in source comments is rarely used.

Other data types

There exists some more predefined types like the container types array and seq, which can contain multiple objects of the same type, or the tuple type which can contain different types. And we have sets, and enums and finally objects. Nim objects are similar to C structs, they are not so verbose as Java classes. We will learn more about all these types in later sections of the book.

Nim Source Code

You have already seen a few examples of simple Nim source code. The code is basically a plain text file consisting of ASCII characters, that is the ordinary characters which you can type on your keyboard. Generally Nim source code can also contain unicode utf-8 characters, so instead of using names consisting of ASCII characters for your symbol names, you could just use single unicode characters or sequences of unicode characters. But generally that makes not much sense, entering unicode is not that easy with a keyboard, and it is displayed only correctly on the screen or in the terminal when the editor or terminal supports unicode properly and when all necessary fonts are installed. That may be the case for your local computer, but what when someone other may edit your source code?

Nim currently doess not allow to insert tabular characters (tabs) in your source code, so you have to do the indentation of blocks by spaces only. Generally we use two spaces for each indentation level. Other numbers work also but you should stick to a fixed value.

Names in Nim, as used for modules, variables, constants, procedures, user defined types and other symbols may contain lower and upper case letters, digits, unicode characters and additional underscores. But the names are not allowed to start with digits or to start or end with an underscore, and one underscore may not follow directly after another underscore.

var
  pos2: int # OK
  leftMargin: int # OK
  next_right_margin: int # OK
  _privat: int # illegal
  custom_: int # illegal
  strange__error: int # illegal

Generally we use camel case like leftMargin for variable names, not snake case like left_margin.

Current Nim has the special property that names are case insensitive and that underscores are simple ignored by the compiler. The only exception is the first letter of a name, that letter is case sensitive. So the names leftMargin, leftmargin and left_margin are identical for the compiler. But LeftMargin is different to all the others, because it starts with a capital letter. This may sound a bit strange at first, but works well in practice. One advantage is, that a library author may use snake case in his library for names, but the users of the library can freely decide if they prefer camelCase. But still you may think that all this generates confusion. In practice it does not, it prevents confusion. Imagine a conventional programming language, fully case sensitive and not ignoring underscores: In a larger program we may then have names like nextIteration and next_Iteration or keymap and keyMap. What when both names are visible in current scope, and we type the wrong one. The compiler may not detect it when types match, but the program may do strange things. Nim would not allow that similar looking names, as the compiler would regard them as identical and would complain about a name redefinition.

You may ask why the first letter is case sensitive. That is to allow for user defined types to use capital type names and then write something like var window: Window. So we can declare a variable named window of a user defined data type named Window. That is a common practice.

The case insensitivity and the ignoring of underscores may be not the greatest invention of Nim, but it does not really hurt. The only exception is when we make bindings to C libraries, where leading or trailing underscores are used, that can make some renamings necessary.

The only minor disadvantage of Nim’s fuzzy names is when you use tools like grep or your editor search functionality: You could not be sure if a search for "KdTree" would give you all results, you would have to try "Kd_Tree" or "KDTree" and maybe some more variants too. For that task Nim provides a tool called nimgrep that does a case- and style-insensitive search. And maybe your editor supports that type of search also. You can also enforce a consistent naming scheme when you call the compiler with the command line argument --styleCheck:error or --styleCheck:hint.

Languages like C uses curly braces to mark blocks, while other languages like Pascal uses begin/end keywords for this purpose. At the same time blocks are generally indented by tabs or spaces to make it easier for the programmer to recognize the extent of the block. This is some redundancy which is not always helpful — block marks and indentation range can contradict each other and can generate strange bugs. Like Python or Haskell Nim does not need additional block markers, the indent level is enough to mark the block extend for the compiler and the human programmer. This style looks clean and compact and was used in pseudo-code of textbooks for decades already. Some people still argue that this style is less "safe", as the behavior of the code depends on invisible white-space. But this is a strange argumentation — the white-space is always visible due to the fact that there are visible characters on the right. Of course changing the indention of the last line of a block would effect the behavior of the code. But that change is well visible. And program code contains many locations where changing one character breaks it. All numeric literals would suffer from adding a digit or deleting a digit. Or the operators like ++ or += from C — the code may compile well after deleting the leading +, but it would be wrong. Computer programming is working carefully! Indeed use of curly braces for blocks has some advantages, eg. many editors can highlight such blocks well, editor may support jumping back and forth between the braces, and for really large blocks it may be indeed simpler to discover the whole block range. But practice has shown that marking blocks with indentation only works fine, most people who have used it for some time just prefer it.

Control Structures

The most important control structures of Nim are the if statement for conditional execution, the related case statement and the while loop for repetitions.

All these structures can be nested in arbitrary ways, so we can have in one if branch other if conditions or while loops, and in while loops again other control structures including other loops.

If Statement and If Expression

The if statement with the optional elif and else parts evaluates a sequence of conditions. As soon as one condition evaluates as true the corresponding statement block is executed, and after that the program execution continues after the whole if construct. That is at most one branch is executed. If none of the conditions after the if or elif keywords evaluates to true, then the else branch is executed if it exists.

if condition1:
  statement1a
  statement1b
  ...
elif condition2:
  statement2a
  statement2b
  ...
elif condition3:
  statement3a
  statement3b
  ...
elif ...:
  ...
else:
  statementa
  statementb
  ...

The most simple form of an if statement is

if condition:
  statement
if age > 17:
  echo "you may drink and smoke, but better avoid it!"

Note that the statements are indented by spaces, we use two spaces generally, but other numbers work as well. And note that it is elif, not elsif like in Ruby, and that there is a colon after the condition. Instead of a single statement we can use multiple each, all on its own line and all indented in the same way.

No, the terminating colon is not really necessary for the compiler, the compiler could determine the end of the condition without it, as the following statement is indented. But it looks better with colon, the colon makes it for humans easier to understand the structure of the whole if statement. So the compiler expects the colons and complains otherwise currently.

We can also have if/else expressions which returns a value like in

var speed: float = if time > 0: delta / time else: 0.0 # prevent div by zero error

In C for a similar construct the ternary ? operator is used.

In languages like C or Ruby the assignment operator = is an expression which returns the value on the left, so we can write code like

while (char c = getChar()) { process(c)}

In Nim the assignment operator is not an expression with a result, but we can group multiple statements in round brackets separated by semicolon, and when the last statement in the bracket is an expression, than the whole bracket has the same value. So we can use conditional terms like

while (let c = getChar(); c):
  process(c)

If we declare a variable in this way using the var or let keyword then that variable is only visible in the bracket expression itself and in the following indented block.

Note that if-expressions must always return a well defined value, so they must contain an else. A plain if, without else, or an if/elif without an else does not work. And as Nim is a statically typed language and all variables have a strictly well defined type, the if-expression must return the same type for all branches!

var a: int
var b: bool
a = if b: 1 elif a > 0: 7 else: 0 # OK
a = if b: 1 elif a > 0: 7 # invalid
a = if b: 1 # invalid
a = if b: 1 else: 0.0 # invalid, different types!

The When Statement

The when condition is similar to the if statement, but while the if condition is evaluated at program runtime, the when condition is already evaluated at compile time. In ordinary program code the when condition is not used that often, but it is very important and useful when we write bindings to C libraries and low level code. An important condition for the when statement is the isMainModule condition and the test for defined symbols like defined(windows):

when not defined(gcDestructors):
  echo "You may try to compile your code with option --gc:arc"
when isMainModule:
  doAllTheTests()

We will discuss the use of when statements, the isMainModule condition and defined symbols in more detail later.

The Case Statement

The case statement is not used that often, but it can be useful when we have many similar conditions:

case inputChar
of 'x': deleteWord()
of 'v': pastWord()
of 'q', 'e': quitProgram()
else: echo "unknown keycode"

To enable optimizations the case construct has some restrictions compared to a more flexible if/elif statement:

The variable after the case keyword must have a so called ordinal type like int, char or string, while float would not work. And the values after each of keyword must be constant, that is a single constant value, multiple constant values or a constant range like 'a' .. 'd' for the 4 first lower case letters. Of course these constants must have a type compatible to the type of the variable after the case keyword. A case statement must cover all possible cases, so most of the time an else branch is necessary.

Unless the similar switch statement in C the case statement needs no break after each branch. If a condition after a of keyword is true, then the corresponding statement or statement sequence is executed, and after that program execution continues after the whole case construct.

The case construct can also be used as an expression like in

var j: int
var i: int =
  case j
    of 0 .. 3: 1
    of 4, 5: 2
    of 9: 7
    else: 0

Here an else is necessary to cover all cases. And as you see we can also indent the block after the case keyword if we want.

The While Loop

The while loop is used when we want to do conditional repetitions, that is if we want to check a condition and want to execute a block of statements only as long as the condition is true. If the condition is false in advance or becomes false after some repetitions then the program execution proceeds after the indented block.

A basic while loop has this shape:

while condition:
  statement1
  statementN
firstStatementAfterTheWhileLoop
var repetitions = 3
while repetitions > 0:
  echo "Nim is easy!"
  repetitions = repetitions - 1

That loop would print the message three times. Like the condition in the if clause the condition is terminated with a colon. Note that the condition must change during execution of the loop, otherwise, when the condition is true for the first iteration, it would remain true and the loop would never terminate. We decrease the loop counter repetitions in the loop, so at some point the condition will become false and the loop will terminate and program execution will continue with the first statement after the loop body. Note how we decrement the loop counter: The right site of the assignment operator is evaluated, after that is done, the new value is assigned to the counter.

There exists two rarely used variants of a while loop: the loop body can contain a break or a continue statement, which each consists only of this single keyword. A break in the body stops execution of the loop immediately and continues execution after the loop body. And a continue statement in the body skips the following statements in the body and starts at the top again, the while condition is evaluated again.

var input = ""
while input != "quit":
  input = readLine(stdin)
  if input == "":
    continue
  if input == "exit":
    break

Above code used the == and the != operator. The == operator does a test for equality, and != test for inequality. Both operator work for most data types like integer, floats, characters and strings. The literal value of an empty string is written "". In line 2 we test if the variable named input has not the value "quit", and in line 4 we test of that variable is empty, that it contains no text at all.

Using of break and continue destroys the expected flow in loops, it can make understanding loops harder. So we generally avoid their use, but sometimes break or continue are really helpful. For example when an unexpected error occurs, maybe by invalid user input.

There in no repeat loop as in Pascal in Nim, which does the first check at the end of the loop when it was executed already for the first time. Repeat loops are not used that much in Pascal, and they are some sort of dangerous, because they check the condition after the first execution of the body, so maybe the body is executed with invalid data for the first iteration. Later we will see how we can use Nim macros to extend Nim by a repeat loop that can be used as it would be part of Nim core functionality.

For Loops and Iterators

These are very useful and important in Nim and other languages. For loops are most often used to iterate over containers or collections. We have not discussed the important array and seq containers yet, but we know already the string container.

A string contains characters, the characters are numbered starting with 0, and we can access single characters of a string with the subscript operator [] which gets the position of the desired character as argument. So we could print the single characters of a string, in this way:

var
  s = "Nim is not always that easy?"
pos = 0
while s[pos] != '?':
  echo "-->", s[pos]
  inc(pos)

It is obvious that the pos variable is some sort of annoying here — we want to process all the characters in the string in sequence, so why would we have to use a position variable to do that. And this way is susceptible to errors, maybe we forget increasing the pos variable in the loop body. So most modern languages provide us with iterators for this purpose:

var
  s = "Nim is not always that easy?"
for ch in items(s):
  echo "-->", ch

That is obvious shorter. The for construct may appear a bit strange, and it is indeed, but it is a common way to write iterators, it is used in Python too. Ruby uses something like s.each{|ch| …​} instead.

For loops in Nim iterates over containers or collections, and pics each element in sequence in this process. The variable after the for keyword is used to access or to reference the single elements. That variable has automatically the right type, which is the type of the elements in the container, and get in each iteration the value of the next element in the container, starting by the first element in the container and stopping when there is no element left. Items() is here the actual iterator, which allows us to access the individual characters in sequence. It exists the convention in Nim that an items() iterator is automatically called in a for construct when no iterator name is explicitly given, so we could also write shorter for ch in s: in this use case.

You may recognize that the output of the above for loop is not identical to the output of the previous while loop. The while loops stops when the last character, that is '?', is reached, while the for loop processes this last character still. That is intended for the for loop, its general purpose is to process all the elements in containers or collections.

The above for loop does a read access to the string, that is we get basically a copy of each character, and we can not modify the actual string in this way. When we want to modify the string, there is a variant available.

var
  s = "Nim is not always that easy?"
for ch in mitems(s):
  if ch == '?':
    ch = '!'

Here we use mitems() instead of the plain items(), the leading m stands for mutable. In the loop body we can assign different values to the actual content.

Objects

We have worked with basic data types like numbers, characters and strings already. Often it makes sense to join some variables of these basic data types to more complex entities. Assume you want to build an online store to sell computers, and you want to built a database for them. The database should contain the most important data of each device type, like type of CPU, RAM and SSD size, power consumption, manufacturer, quantity available, and actual selling price.

We can create a custom object data type with fields containing the desired data for this purpose:

type
  Computer = object
    manufacturer: string
    cpu: string
    powerConsumption: float
    ram: int # GB
    ssd: int # GB
    quantity: int
    price: float

We have to use the type keyword to tell the compiler that we want to define a new custom type. Writing the type keyword on its own line begins a type section where we can declare one or more custom data types. All type declarations in a type section must be indented. In the next line we write our type name, an equal sign and the keyword object. That indicates that we want to declare a new object type named Computer. Here Computer is a type name, in Nim we use the convention that user defined type names start with a capital letter. In the following indented block we specify the desired fields, each line contains the name of a field, and a colon followed by the needed data type. That is similar like a plain variable declaration.

Objects in Nim are similar to structs in C. Unlike classes in Java Nim objects contain only the fields, sometimes also called member variables, but no procedures, functions or methods, and no initializers or destructors as in C++. In Nim we keep the data objects, and the procedures, functions, methods and also optional initializers and destructors that work with that data objects separated.

Now that we have defined our own new object type, we can declare variables of that type and store content in its fields.

var
  computer: Computer

computer.manufacturer = "bananas"
computer.cpu = "x7"
computer.powerConsumption = 17
computer.ram = 32
computer.ssd = 1024
computer.quantity = 3
computer.price = 499.99

Of course in real applications we would fill the fields not in this way, but we would maybe read the data from a file, from terminal or maybe from a graphical user interface.

It may look a bit ugly that we have to write computer. before each field when we access the fields. Indeed in recent Nim versions that is not necessary, you may use the with construct now instead.

import std/with
var
  computer: Computer
with computer:
	manufacturer = "bananas"
	cpu = "x7"
	powerConsumption = 17
	ram = 32
	ssd = 1024
	quantity = 3
	price = 499.99

We can use the fields like ordinary variables:

computer.quantity = computer.quantity - 1 # we sold one piece
echo computer.quantity

As you already know, the right side of the assignment operator is evaluated first, then the result is stored in the variable on the left side. But we can also just write computer.quantity -= 1 or dec(computer.quantity).

Generally a computer store would offer many different types of computers, so it would make sense to store all the different devices in a container like a sequence, called short seq in Nim.

Arrays and Sequences

Sequences and arrays are homogeneous containers, they can contain multiple other elements of the same data type, while a plain variable like a float or an int only contains a single value. In some way we could regard objects also as containers, because objects contain multiple fields. The same holds for tuples — tuples are a very simple, restricted form of objects and also contain fields. But more typical container data types are the built in arrays and sequences, or for example hash tables which are provided by the Nim standard library. Arrays, sequences and hash tables can contain multiple elements, but all elements must have the same data type, which we call the base type.[17] The data type of the base type is not restricted, it can be even again array or sequence types, so we can built multidimensional matrices in this way. Arrays have a fixed, predefined size, they can not grow or shrink during runtime of our program. Sequences and hash tables can grow and shrink.

Arrays and sequences appear very similar, a sequence appears even more powerful because it can change its size, that is the number of elements that it contains, at runtime, while an array has a fixed size. So why do we have arrays at all? The reason is mostly efficiency and performance. An array is a plain block of memory in the RAM of the computer, which can be accessed very fast and needs not much care by the runtime system. Sequences take much more effort, especially when we add elements and the sequence has to grow. When we create sequences, we can specify how many elements should fit in it at least and the runtime system reserves a block of RAM of the appropriate size. But when our estimation was too small, and we want to append or insert even more elements, then the runtime system may have to allocate a larger block of memory first, copy the already existing elements at the new location, and then release the old, now unnecessary memory block. And this is an relative slow operation. The reason why this process can be necessary is, that the initially allocated memory block may not increase in size because the neighborhood in the RAM is already occupied by other data. Now let us see what we can do with arrays and sequences:

var
  a: array[8, int]
  v = 1
for el in mitems(a):
  el = v
  inc(v)
for el in mitems(a)
  el = el * el
for square in a:
  echo square

In the second line of the code above we declare a variable named a of array type — we want to use an array with exactly 8 elements, and each element should have the data type int. To declare a variable of array data type we use the array keyword followed in square brackets by the number of the elements, and separated by a comma, the data type of the elements. We can also specify the range of the indices explicitly by specifying a range like array[0 .. 7, int] or array[-4 .. 3, int]. The first specification is identical to the one in above example program, and the second one would allow us to access array elements with index positions from -4 up to 3.[18] or [int, 8]. It may help to remember that for plain variables the data type comes last also like in var i: int.]

The first for loop of above program fills our array — that is for each of the 8 storage places in the array we fill in some well defined data. We use the mitems() iterator here, because we want to modify the content of our array — we fill in numbers 1 .. 8. In the next for loop we square each storage location, and finally we print the content. In the last for loop we do not modify the content, so a plain items() instead of mitems() would work, but we already learned that we have not to write the plain items at all in this case.

Sequences work very similar like arrays, but they can grow:

var
  s: seq[int]
  v = 0
while v < 8:
  inc(v)
  add(s, v)
for el in mitems(s)
  el = el * el
for square in s:
  echo square

We start with an empty seq here, and use the add() procedure to append elements. After that we can iterate over the seq as we did for the array.

In the same way as we accessed single characters of a string with the subscript operator, we can use that operator to access single elements like in a[myPos]. The slice operator is available for arrays and sequences too and can be used to extract sub-ranges or to replace multiple elements. As arrays have a fixed length, the slice operator can only replace elements in arrays, but not remove or insert ranges. The first element position is generally 0 for arrays and sequences. Arrays can even be defined in a way that the index position starts with an arbitrary value, but that is not used that often. Whenever you use the subscript or slice operator you have to ensure that you access only valid positions, that is positions that really exists. a[8] or s[8] would be invalid in our above example — the array has only places numbered 0 .. 7, and for the seq we have added 8 values which now occupy positions 0 .. 7 also, position 8 in the seq is still undefined. We would get a runtime error if we would try to access position 8, as well when we would try to access negative positions. You might think that an assignment for a seq like s[s.length] = 9 is the same as s.add(9), but only the add() operation works in this case.

Note that in some languages like Julia arrays start at position 1. Nim array can have an arbitrary integral start position, including negative start positions, but start position as well as highest subscript position are determined in the program source code and can not change at runtime. We say that arrays have fixed compile-time bounds. Seqs start always at position 0, we can specify an initial size, and we can always add more elements at runtime.

Arrays and sequences allow fast access to its elements: All the elements are stored in a continues memory block in RAM, and the start location of that memory block is well known. As all the elements have the same byte size, it is an easy operation to find the memory location of each element. The compiler uses the start location of the array or seq, and add the product of subscript index and element byte size. The result is the memory location of the desired element, which was selected by the index used in the subscript operator. When the array should not start at position 0, then the compiler would have to adjust the index, by subtraction of the well known start index. This operation takes not much time, but still arrays starting at position 0 are a bit faster. We said that the compiler has to do a multiplication of index and element size — that is an integer multiplication, which is very fast. When the element size is a power of two, then the compiler can even optimize the multiplication by using simple shift operation, which may be even faster, depending on your CPU.

It should be not surprising that the internal structure of seqs are a bit more involved than arrays. Arrays are indeed nothing more than a block of memory, generally allocated on the stack for local data or allocated in the BSS segment for global data. Don’t worry when you have not yet an idea what the stack, the heap and a BSS segment is, we will learn that soon. The Nim seq data type has a variable size, so it is clear that it needs not only a storage location for its elements, but also a counter to store how many elements it currently contains, and another counter how many it could contain at most. The element counter must be updated when we add or delete elements, and when the counters tells that there is currently no more space available for more elements, then a new block of memory must be allocated, and the existing elements must be copied from the old location into the newly allocated memory region, before the old memory region can be released.[19] Due to this additional effort appending elements to a seq by using the add() procedure is not extremely fast. You may wonder why we have not to save a size information for arrays. Well arrays have fixed size, so it is obvious that we never have to adjust something like a size counter, simple why it would never change. But would we have to save the desired initial size of the array? Well, in some way yes. But it is a constant value. During the compile process the compiler can catch some errors already for us — when we have an array as above with size 8, then the compiler would be able already at compile time to recognize some invalid access to array elements — a[9] would be a compile time error for sure. But at runtime, when we execute our program, access to not existing index position may occur, for example by constructs like var i = 9; a[i] = 1 when the array is declared as var a: array[8, int]. For catching that type of error the compiler has to store the fixed array size somewhere and to check against that value when an array access by using the subscript operator with a non constant argument occurs, as the a[i] above. One related remark: Accessing array elements is as fast as ordinary variable access when we use a constant value as index, that is a constant literal or a named constant. The reason for this is, that when the index is a constant, then the compiler just knows the exact position of that array element in memory, just as it knows the address of plain variables, so there is no need for address calculations at runtime.

We said that appending elements to seqs is not extremely fast — indeed it is a few times slower than access to an array element by its index using the subscript operator. So when we know that our seq will have to contain at least an initial amount of elements, then it can be useful for maximum performance, that we allocate the seq from the beginning for this size and than fill in the content by use of the subscript operator instead that we append all the elements one by one. Here is one example:

var s: seq[int] = newSeq[int](8)
var i: int
while i < 8:
  s[i] = i * i
  inc(i)

We use the newSeq() procedure to initialize the seq for us, the content of the square brackets tells the newSeq() procedure that we want a sequence with base type int, and the number 8 as argument tells it that the newly created seq should have 8 elements with default value (0). This procedure is a so called generic procedure, it needs additional information, which is the data type that the elements should have. Don’t confuse the square bracket in the newSeq[int]() call with the subscript operator a[i] which we have used for array access, both are completely unrelated. Note that the initialization of the seq above does not restrict its use in any way, we can still use it like an uninitialized seq, that is we can use the add() operator to add more elements, we can insert or delete elements and all that.

Deleting elements from an array or from a sequence can be very slow. It is slow when we use the naive approach and move all the elements located after the element one position forward. This would obtain the order in the container, so sometimes this is the only solution, but of course moving all the entries is expensive for large containers. Nim standard library provides the delete() function for this order maintaining delete operation. A much faster way to delete an entry in a seq or array is to remove the last entry and replace the one that should be deleted with that last entry. This operation moves the last entry to the front, so order of elements in not maintained. Nim standard library provides the del() function for this faster, but order changing delete operation. Of course, whenever the order is not important, we should use del().[20]

In the section about strings we said that strings have value semantic, that is that an assignment like str1 = str2 creates a copy of str2 and that after that assignment str1 and str2 are fully independent entities — modifying one does not change the content of the other one. Array and sequences behave in the same way, both have value semantic too. Indeed arrays are true value types in Nim, as they live on the stack in the same way as plain variables like integers, floats or characters. Sequences have a dynamic data buffer which is allocated on the heap, so it would be possible that an assignment like seq1 = seq2 would not copy the data buffer but reuse the old one. In that case both sequences would be not independent, seq2 would be an alias for seq1. This is called reference semantic, some languages like Ruby behave in this way. But in Nim arrays, strings and sequences have value semantic, an assignment creates an independent copy. We will learn more details about reference semantic and the use of the stack or heap to store data soon when we discuss references to objects.

Some details

Let us investigate at the end of this section some internal details about arrays and sequences. Beginners not yet familiar with the concept of pointers should better skip this subsection, and maybe come back later. We could consult the Nim language manual or the compiler source code to learn more details about arrays and sequences. Or we can write some code to test properties and behavior. Let us start investigating an array:

proc main =
  var a: array[4, uint64]
  echo sizeof(a)
  a[0] = 7
  echo a[0]
  echo cast[int](addr a)
  echo cast[int](addr a[0])

  var a2 = a
  a[0] = 3
  echo a2[0]

main()

When we run this program we get this output:

32
7
140734216410384
140734216410384
7

The size of the whole array is 32, as we have 4 elements each 8 byte in size. And the address of the array itself as well as the address of its first element is identical. Remember that the actual address values will be different for each run of our program, and they may be totally different on different computers, as it is some random choice of the OS which free memory area is used to run our program. This result is expected as the array is a plain block of memory stored on the stack. And indeed the array has copy semantic, when we create a copy called a2 and later modify a, then the content of a2 is unchanged. That was not really surprising, so let us investigate a sequence:

proc main =
  var dummy: int
  var s: seq[int64]
  echo sizeof(seq)
  echo sizeof(s)
  s.add(7)
  echo s[0]
  echo cast[int](addr dummy)
  echo cast[int](addr s)
  echo cast[int](addr s[0])

  var s2 = s
  s[0] = 3
  echo s2[0]

main()

When we run above code we get:

8
8
7
140732171249104
140732171249112
140463681433696
7

The first two lines of the output may confuse us, as a size of only 8 byte may indicate a plain pointer value on a 64 bit system. Indeed the sequence is not a large object that contains size and capacity fields, but only a tiny object that contains a single pointer to the data storage of that sequence. We know that it is not a plain pointer or ref by the fact that we can not assign nil or test for nil for sequences. (But an object which contains only a pointer is basically identically to a plain pointer, as Nim objects have no overhead as long as we do not use inheritance and when no padding to word size is needed for tiny fields like int8.) Capacity and length are stored also in the memory block that is allocated for the elements as long as the sequence is not empty. So empty sequences don’t wast much memory when we have a lot of them, i.e. arrays or sequences of sequences (matrices). We use the dummy int variable in the code above as we know that plain ints are stored on the stack, and when we compare the addresses of our dummy variable and our sequence, then we see that the addresses indicate close neighborhood, so the seq object is also stored on the stack. But the address of s[0] is very different, indicating that the data buffer is stored in a different memory region, which is the heap. If we would continuously add elements to the seq, then the address s[0] would change at some point, while the address of s would remain always unchanged. That is while the capacity of the data buffer would become exhausted at some point and a new data buffer with a different address would be used. Finally we see again that the sequence has also copy semantic, as the content of the copy s2 remains unchanged when we modify the initial sequence s. We could try to discover some more details of the internals of Nim’s sequences, i.e. we could try to detect where the capacity and size is stored. But that are internal details which should not really interest us and which may change with new compiler version or different compilers.

But OK, you may still not believe what we said, so let us go one layer deeper. We strongly assume that a seq needs a length and a capacity field. And we assume that its data type should be int. We said that both fields should be adjacent to the buffer of the seq elements, that means at the start or at the end. Obviously we can not access the end as long as we do not know the capacity, so capacity field should be at the start, and then length field also. We may find out which one is which by observing the content when seq grows. So let us write some code:

proc main =
  var
    s: seq[int64] = newSeqOfCap[int64](4)
    s2: seq[int64]
    p: ptr int

  var h = cast[ptr int](addr s2) # prove that an uninitialized seq is indeed a pointer with nil (0) value
  echo cast[int](h) # address on stack
  echo h[] # value (0)
  echo ""

  for i in 0 .. 8:
    s.add(i)
    echo cast[int](addr s[0])
    p = cast[ptr int](cast[int](addr s[0]) - 8) # capacity
    echo p[]
    p = cast[ptr int](cast[int](addr s[0]) - 16) # length
    echo p[]

main()

Output when we run the program is:

140725732630192
0

140251431497824
4
1
140251431497824
4
2
140251431497824
4
3
140251431497824
4
4
140251431506016
8
5
140251431506016
8
6
140251431506016
8
7
140251431506016
8
8
140251431510112
16
9

Don’t worry when you do not understand the program and the output yet. You will better understand it when you have read the sections about references, pointers and memory management. The first two output lines shows us that an uninitialized seq is just a pointer pointing to nil. And the remaining output lines show us the address of the first seq element, the capacity and the length of the seq whenever we add an element. We started with a seq with initial capacity of 4, so address and capacity is constant while we add the first 4 elements. Then the capacity of the allocated buffer is exhausted. A new buffer with different address and doubled capacity is allocated, the already contained elements are silently copied to the start of the new buffer and so on.

Slices

Nim slices are objects of type Slice that contain a lower bound a and an upper bound b. Slice objects are used by functions of the standard library or by user defined functions to access sub-ranges of strings, arrays and sequences. Applied on these container data types slices look syntactically like sub-ranges:

var
  m = "Nim programming is difficult."
m[19 .. 28] = "not easy."
echo m
echo "Indeed " & m[0 .. 18] & "is much fun!"

In line three we use the slice to replace the sub-string "is difficult." which starts at position 19 with another string. Note that the replacement can be a longer or a shorter string, that is the slice supports not only overwriting characters, but also inserting or deleting operations. In the last line we use the slice to access a sub-string and create a new string with it. As we learned earlier in the strings section already, we can use the ^ operator to access elements counted from the end of the container, so we could have written line three also as m[19 .. ^1] = "not easy.".

Slices can be used in a similar way for arrays, strings and sequences. But we have to remember that slices are only objects with a lower and an upper bound, so there must be always a procedure that accepts the container and the slice as arguments to do the real work.

When we care for utmost performance, then we have to be a bit carefully with slices, as slices can generate copies. Consider this example:

type
  O = object
    i: int

proc main =
  var
    s = newSeq[O](1000000)
  for i in 0 .. (1000000 - 1):
    s[i] = O(i: i)

  var sum = 0
  for x in s[1 .. ^1]:
    sum += x.i

Here we use the slice operator to exclude the first element from our summing operation. Unfortunately in current Nim v1.4 use of the slice operation in this way creates a copy of our sequence, which increases the run-time and memory consumption. We may try to use the new toOpenArray() expression and try a construct like

  for x in items(s.toOpenArray(1, s.high)):

but that does currently not compile.

One option is currently that we create a custom iterator like

iterator span*[T](a: openArray[T]; j, k: Natural): T {.inline.} =
  assert k < a.len
  var i: int = j
  while i <= k:
    yield a[i]
    inc(i)

and use

for x in s.span(1, s.high):

Or we may do the summing in a procedure and pass that proc an openArray created with toOpenArray() like

proc sum(x: openArray[O]): int =
  for el in x:
    inc(result, el.i)
echo sum(s.toOpenArray(1, s.high))

But this is work in progress, so the situation may improve, see

Value Objects and References

We have already used different types of variables — integers, floats, or the custom Computer object, and some more. We said that variables are named memory regions, where the content of our variables is stored. We call this type of variables also value objects.

Value objects always implies copies when we do an assignment

var i, j: int
i = 7
j = i
i = 3
echo i, j

Here we have 3 assignments, first we assign the integer literal 7 to variable i, then we assign the content of variable i to variable j, and finally we overwrite the old content of variable i with the new literal value 3. The output of the echo statement should be 3 and 7, because in line 3 we copy the content of variable i, which is currently the value 7, into variable j. The new assignment in line 4 in no way touched the content of variable j.

Maybe that is not too surprising, but when we would have references instead of plain variables, then the situation would be different, as we will see soon.

Whenever possible we should use this simple form of variables, as they are fast and easy to use.

But there exist situations where we need some sort of indirection, and then references and pointers come into play. For example when the data entities depend in some form on each other, the elements may build linked lists, threes or other structures. The entities may have some neighborhood relation, also called some one to many relation.

Indeed value objects and references occur in real life also:

Imagine you have baked a cake for your family, and you know that your friendly neighbor loves cakes too. As you have still a lot of all necessary ingredients and because the oven is still hot, you make one more identical cake to give it later to your neighbor. We can think of the cake as a value type, and your second cake can be considered as a copy. When you give the copy to your neighbor, then you have still your own, and when you or the neighbor eats the cake, then the other one still exist.

Now imagine that you know a good car repair shop. You can give the telephone number or location of that car repair shop to your neighbor, so he can use that shop too. So you gave him a reference to the shop, but you gave him not a copy. You can also give some of your other friends each a reference to that shop, which is nearly no effort for you. While backing a cake for all of them would be some effort.

You can regard names of persons as some sort of reference too. Imagine you have a list with the names of all the people you intend to invite to your birthday party, and another list with names of people who owe you money. Some names may be on both list, this is it refers to the same person.

In computers the dynamic storage, called RAM, consists of consecutive, numbered storage locations, called words. Each individual word has its address, which is a number generally starting at zero and extending to a value which is defined by the amount of memory available in your computer. These addresses can be used to access the storage locations, that is to store a value at that address, or to read the content again. Reading generally does not modify the content, you can read it many times and will always get the same value. When your write another value to that storage location, then reads will give you that new value.

Basically for all the data that you use in your program you need in some form its address in the RAM, without the address you can not access it. But what is with all the plain, value object variables we have used before, we have never used addresses? That is true — we used only names to access our variables, and the compiler mapped our chosen name to the actual address of the variables in memory whenever we accessed the variable. For most simple cases this is the best way to access variables. Now let us assume we have such value object type of variable declared in our program, can we access it without using its name? When we have declared it, it should reside somewhere in the RAM when the program is executed. Well, when we do really not want to access it by variable name, then there is still one chance: We can search in the whole RAM for the desired content. In practice we would never do that, as it is stupid and would take very long, but we could do. But how can we detect our variable? How can we be sure that it is indeed ours? Generally we can not. Even when we are sure that the variable must reside somewhere in the RAM, generally the variable is marked in no way of course. Even when we would know the value which is stored in that variable, we would only know what bit pattern it should have, so for most words of the RAM with a different bit pattern we could say for sure that it can not be our variable, but whenever we find the expected bit pattern than it can be just a coincidence, there can be many more words in RAM with that content. In some way it is as you would search a person and you know that that person lives in a long road with numbered houses. If you only know that the person wears brown shoes but you know not the number of the house nor the name of the person and no other unique property of that person, then you have not much luck.

References and Pointers

Introduction to Pointers

In Nim references are some form of smart or managed pointers, we will learn more about references later. The plain pointer data type is nothing more than a memory address, it is similar to a (unsigned) integer number. We say that a pointer points to an entity when the pointer contains the memory address of that entity.

Beside the pointer data type, which is only some RAM address, we have also the ptr entity. Ptr is not a datatype for its own, it is always used in conjunction with another data type:

var
 p: pointer
 ip: ptr int

Here the variable p is of type pointer, we could use it to point to some arbitrary memory address. The variable ip is of type ptr int, which indicates that it should only point to memory addresses where a variable with data type int resides. So a ptr is a pointer that is bound to a specific data type. Generally we speak only about pointers, if we are referring to an untyped pointer or a typed ptr is generally clear from the context.

When we only declare pointers but do not assign a value then the pointers have the value nil, what indicates that they are regarded to point to nothing. Exactly speaking a pointer can never point to nothing in the same way as an integer variable can not contain no number. As an integer variable always contains a bit pattern, a pointer also always contains a bit pattern. But we are free to define a special pattern as nil, and whenever a pointer has this special value, then we know that it does not really point to something useful. In C instead of nil NULL was chosen for the same purpose. In practice nil and NULL are generally mapped to 0, that is a word with all bits cleared. But that is more or less an arbitrary decision.

So how can we give our pointers above a useful value?

One possibility would be to use Nim’s addr() function, which gives us the memory address of each ordinary variable.

var
 number: int = 7
 p: pointer
 ip: ptr int
echo cast[int](p)
echo cast[int](ip)
p = addr(number)
ip = addr(number)
echo cast[int](p)
echo cast[int](ip)

First we declare an ordinary integer variable called number which will reside somewhere in memory when we execute the program, and then we use the addr() function to assign the address of that variable to p and ip. The addr() function is a low level function provided by the compiler, it can be used to determine the memory address of variables and some other entities known to the compiler.[21] We used the echo() procedure to show us the numeric decimal value of the addresses in the terminal. As it generally makes not too much sense to print addresses, echo() would refuse to print it, so we have used the construct cast[int](someValue) to tell that echo() should regard our pointers as plain integer and print it. That operation is called casting, we generally should avoid it, as it destroys type safety, but for learning purposes it is OK to use it. We will learn more about casts and related type conversion later.

The first two echo statements should print the decimal value 0, as the pointers have the initial default value nil.

The echos in the last two lines should print a value different from 0, as we have assigned the valid address of an ordinary variable that resides somewhere in the RAM when the program is executed. Both outputs should be identical, as we have assigned for both pointers addr(number) each.

Maybe a funny fact is, that when you run the program multiple times the output of the last two echo() statements print different values. But that is not really surprising — whenever you launch the program, then for our variable number a storage location in RAM is reserved. And that location can differ for each new program execution. For your next holiday in the same hotel, you may get a different room also.

So when we have the pointer ip pointing to a valid address, can we recover the content of that memory region? Sure, we use the de-reference operator [] for that purpose. Whenever we have a typed pointer x we can use x[] to get the content of the memory location where the pointer is pointing to. Note that the operator [] is not really related to the subscript operator [pos] which we used earlier for array, seq and string access. Nim uses ASCII characters for its operators, and that set is not very large. And maybe it would even be confusing when we would have a different symbol for each operator. We can consider [] as some form of content access operator — mystring[pos] gives us the character at that position, and ip[] gives us the content of the memory location where ip points to.

var
 number: int = 7
 ip: ptr int
echo cast[int](ip)
ip = addr(number)
echo cast[int](ip)
echo ip[]

What do you expect as output for the last echo statement? Note that for the last echo statement we do not need a cast, as ip[] has a well defined type: ip has type ptr int, so ip[] is of well defined type int and echo() can print the content.

Now let us investigate how we can use pointers to modify the content of variables:

var
 number: int = 7
 ip: ptr int
ip = addr(number)
echo ip[]
ip[] = 3
echo ip[]
echo number

What do you expect for the output of the last echo() statement? Well remember, ip points to the location where variable number is stored in RAM. So echo ip[] gave us the content of number. Now ip[] = 3 is an assignment, the right site of the assignment operator is the literal number 3, which is a value type. Earlier we said that for value types an assignment is a copy operation, the right site of the assignment operator is copied into the variable on the left site. Now ip[] stands exactly for the same content as the name number, and so assigning to ip[] is the same as assigning to number.

Pointer Arithmetic

In low level programming languages pointer arithmetic can be useful. For example old C code often iterates with pointer arithmetic over arrays by use of constructs like sum += *(myIntPtr++). This was done to maximize performance. Modern C compiler generally understands statements like sum += el[i]; i++ well and generates very good assembly instructions for it. So pointer arithmetic is not necessary in C that often today.

Nim does not provide math operations for pointers directly, but we can always cast pointers to integers and do arbitrary math. And of course we could define our own operators for that purpose, but generally we should avoid that, as it is dangerous, error prone and generally not necessary. As an example let us sum up some array elements:

proc main =
  var
    a: array[8, int] = [0, 1, 2, 3, 4, 5, 6, 7]
    sum = 0
  var p: ptr int = addr(a[0])
  for i in a.low .. a.high:
    echo p[]
    sum += p[]
    echo cast[int](p)
    var h = cast[int](p); h += sizeof(a[0]); p = cast[ptr int](h)
    #cast[var int](p) += sizeof(a[0]) # this compiles but does not work currently

  echo sum
  echo typeof(sizeof(a[0]))

main()

When we do pointer arithmetic or similar math to calculate the address of variables in the computer memory, then memory addresses are used like integer numbers, and so it makes same sense that Nim’s integers have the same byte size as pointers.

Allocating Objects

In the previous section we learned the basics about pointers. We used the addr() operator to initialize the pointer by assigning the address of an already existing object. This is in practice not that often done, and it can be a bit dangerous, as it is not always guaranteed that the variable on which we applied addr() will exist as long as our pointer exist. So the pointer may point later to a memory location that is already freed or used by a totally different object already. So the use of addr() is more reserved for advanced programmers who know well what they do, and most of the time addr() is not necessary at all or is only necessary for really low level code, maybe when interfacing with external libraries written in C. Instead of using addr() to assigning to pointers a valid address, often procedures like alloc() or create() are used to reserve a block of memory:

var ip: ptr int
ip = create(int)
ip[] = 13
echo ip[] * 3
var ip2: ptr int
ip2 = ip
echo ip2[] * 3
dealloc(ip)

Here the procedure create() is used to reserve a block of memory, the int parameter ensures that the block has the size of an integer value. After ip has a valid value, we can store a value in that memory location and read it again. Note that multiple pointers can point to the same memory location: We declared one more int ptr called ip2. But for that pointer we do not allocate a new block, but we assign the old block that we allocated for ip to ip2. Now both pointers points to the same object, the int value 13. We may call ip2 an alias, as it is a different way to access the same object.

When we use alloc() or create() to allocate memory blocks, then we have to deallocate them when we need them not any more. Otherwise that memory blocks couldn’t be reused. If we would continuously allocate memory blocks and never deallocate, that is free them, then at some point in time all memory would be occupied — not only for our own program, but for all programs running currently on the same computer. We had to terminate our program — when a program is terminated then all resources get freed automatically by the OS.

The use of procedure pairs like alloc() and dealloc() is common practice in low level programming languages like C, but it is inconvenient and dangerous: We can forget to call dealloc() and waste resources, or we may even deallocate memory blocks but still use it by our pointers. The later would at some point of time crash our program, as we would use memory blocks which are already released and may be used for other variables — from our own program or from other programs. Note that in the source code above there is only one single dealloc() call. The reason for that is, that we only allocated one single memory block in one single create call, ip2 is only one more pointer that points to that block. If we would have used an addition dealloc(ip2) call, then that would be a so called double free error.

As you see, using pointers is inconvenient and dangerous. But still there are situations where plain value type variables do not suffice. The solution of many higher level programming languages to this problem is a Garbage-Collector (GC). The GC does the dangerous and inconvenient task of deallocating unused memory blocks for us automatically.

To distinct the GC managed "pointers" cleanly from the manually managed ones, we call them in Nim references, in some other languages they are called traced pointers. References are always typed like ptr, there is no equivalent to the untyped pointer type for references.

For References we have still to do the allocation our self, then we can use the references, and when we are not using them any more, then the GC frees the corresponding memory block. A typical scenario is that we use references in a procedure or in an otherwise limited block of code: We declare the reference in that code block, allocated and use it, and when the code block is left the GC frees the allocated memory for us. You may think that the fact that we still have to allocate the memory for our references our self is still a problem, as we may forget that step. Well it is not that dangerous, when we forget the allocation step, we would use a reference with value nil, which would immediate result in a runtime error. So we would see the problem immediately. Other pointer errors, like missing de-allocation or use after free are not that obvious and more dangerous.

With references we can rewrite our previous example code in this way:

var ip: ref int
new(ip)
ip[] = 13
echo ip[] * 3
var ip2: ref int
ip2 = ip
echo ip2[] * 3

We have replaced ptr by ref, and instead of alloc() or create() we are using the new() proc which gets the uninitialized ref as a parameter and allocates a managed memory block for it, that is after the new() call ip has a well defines value referring to a managed memory block that can store an integer value. Again, we can use one more ref and assign that ref the value of the other, so now both references the same memory block. The advantage here is that we don’t have to care about freeing that block, the GC will do that when appropriate.

To verify that in the example code above both references really reference the same object in memory, we could add two more lines of code:

ip2[] = 7
echo ip[]
echo ip2[]

Here we are using the reference ip2 to assign to the memory block the literal value 7. After that assignment both echo statements would display that new content.

Using references and pointers to store basic data types like integers is not done that often, in most cases we work with larger objects, and we create some relations between the objects. We will try that in the next section.

References to Objects

You should still wonder for what references are really useful — they seem to be only a more complicated version of plain value type variables.

Now let us assume we want to create a list of things or persons, maybe a list of our previously used Computer data type, or maybe a list of persons we will invite to our next party. We will create the party list for now, as the Computer data type we used before has already many fields, and filling all the fields would be some effort, so let us use a new Friend data type which should store only the friends name for the beginning — we may add more fields later when necessary. So we may have

type
  Friend = object
    name: string

With that declaration we could declare a few friends variables like

  var harry, clint, eastwood: Friend

But that is not what we want, we would need a list with all of our friends that we would like to invite to our party, we would want to add friends to the list, and maybe we would want to delete friends also. You may think we could use Nim’s sequence data type for that, and you are right. But let us assume we could not use that predefined Nim data type for some reason. Then we could create a list of linked references to Person.

type
  Friend = ref object
    name: string
    next: Friend

Now our Friend data type is a reference to an object, and the object itself has an additional next field which is again of type friend.

That is some sort of recursion. If that should appear as too strange, then imagine you have some numbered paper cards, each with two fields: One field name, one field next: In the name field you can fill in a name of a friend, in the next field you fill in the number of the next card. The last card in the chain gets no entry in the next field.

In languages like Nim or C lists, also called linked lists, are dynamically created data structures consisting of elements (called nodes), where each node has a field which is a references or pointer to its successor or predecessor. When the nodes have only a successor field, then we call the list a single linked list, and when it also has a predecessor field, then we call is a double linked list. Contrary to arrays and Nim’s sequences lists do not allow access to arbitrary elements, we can only traversal the list starting from its first element for single linked lists, or also from its last elements for double linked lists. The first element of a list is also called its head, and the last element is called its tail. Often the head and the tail elements are just plain nodes, but the head can be also an extended node object with additional fields carrying information for the whole list, maybe an additional string field for the list name. In this section we use the simplest form of a list, which is a single linked list where the head is just an ordinary node. If the head has the value nil then the list is empty.

Now we create a small Nim program which reads in names of our friends from the terminal, creates a list of all friends, and finally prints the list.

type
  Friend = ref object
    name: string
    next: Friend

var
  f: Friend # the head of our list
  n: string # name or "quit" to terminate the input process

while true:
  write(stdout, "Name of friend: ")
  n = readline(stdin)
  if n == "" or n == "quit":
    break
  var node: Friend (1)
  new(node)
  node.name = n
  node.next = f
  f = node

while f != nil:
  echo f.name
  f = f.next
1 The actual name for this temporary variable is arbitrary, we could have used el for element maybe

This example code seems to be not that easy. But it is not really difficult, and when you have understood it, you can call yourself a Nim programmer already. Maybe you should think about the code above for a few minutes before reading the explanations below.

First let us summarize what our program should do: It should read in some names of our friends which we would like to invite to our next party. Of course when entering the names, we would need a way to tell that we are done. In our program we can do that in two ways, we can enter an empty name by just pressing the return key, or we can enter the text "quit" to stop the loop. Unfortunately that means that we can never invite a friend with that name to our parties. When we have terminated the input loop, then the next loop prints all the entries to the terminal.

Let us start with the type and variable declarations: We use a user defined type named Friend which is a reference to an object, that object type has a field name of type string, and a field next which is again a reference to the same data type.

We are using two variables, one called n of type string to read in a name or the quit command from terminal, and a variable called f of type Friend. The variable f seems to match only to one single friend, but as the type of f has a next field it can be a whole list of friends, with f being the start or head of that list.

In the code above we are using a special while loop — special because the construct while true: and because the loop contains a break statement. Earlier we said that we should avoid the break statement in loops, because it interrupts the control flow and can make it more difficult to understand and proof the flow. But in this case that form makes some sense: For the first loop we have to first read in a name from terminal and then we can decide what to do, so we can not really evaluate a condition after the while statement at the top. So we use the simple constant condition true, which would never terminate the loop. We need a break inside the loop body to terminate the loop.

Let us investigate the second loop first as it is really easy: In the while condition we check if current value of f is nil, that is if there are no more entries in our list. For that case we terminate the loop, as we are done. If f has not the value nil, than f points to a valid content, that is there is at least a valid name, which we access by the field access operator and print it with echo f.name. Note that in Nim the field access operator . works in the same way for value objects types as well as for ref objects types. For ref objects types we could also write f[].name instead of plain f.name, that is we first apply [] to f to get the content, and then use . to access the name field. In some other languages like C we would have to use a special operator -> to access fields of pointer or reference types.

The most interesting statement in the output loop is f = f.next. We assign the content of f.next to f and proceed with that new content. The content could be a valid reference to one more friend object, or it could be nil, indicating that our loop should terminate.

The input loop is also not that complicated: To make the process of adding more friends to the list easy, we always add the new names at the beginning. First we ask the user to enter a name. We use write(stdout) for this, as echo() always generates a newline, but we want to read in the name on the same line. If the name is empty or has the special value "quit" then we terminate the input loop. In the loop we are using a temporary variable called node of type Friend, we allocate a memory block for it with new().[22] Then we assign the read in friend’s name n to the name field. The last two statements of the loop body are a bit demanding: First we assign to node.next the value of f. Now node is basically the start of our list, and it next field refers to the first element of the current list. Fine, but we said that the node variable is only a temporary variable, we do not intend to use it longer as necessary. But currently node is so useful, it is the head of our list. On the other hand, the former list start f is now useless, current f is identical with node.next. So the trick is, we just assign to f the value of node. Now f is the complete list, and we do not need node any more. The node variable can be used in the next loop iteration again, but we have to allocate a new memory block for the node reference, as the previous memory block is still in use, it contains the name which we just entered and also a reference to the next object in the list.

Note that we add the new elements at the top of the list in this way. We have done it that way because it is very easy in this way. For adding at the end of the list, we would have to use one more reference variable which allows us always access to the current end of the list, or we would have to traversal the list from head to tail whenever we would like to add elements at the tail.

For one more exercise let us consider deleting entries in our list. Basically that operation is very easy, we would just skip one entry. Lets adds this code to the program above:

while f != nil:
  write(stdin, "Name to delete: ")
  n = readline(stdin)
  if n == "" or n == "quit":
    break
  if f.name == n:
    f = f.next
  else:
    while f.next != nil:
      if f.next.name == n:
        f.next = f.next.next
        break
      f = f.next

Here we are using again an outer while loop to read in the names which we want to delete. That loop uses the condition while f != nil: because when the list is empty we should stop of course.

In the loop body we have an if statement, and in the else branch of the if statement we have one more loop. The reason why we need the if statement is, that the case that our name to delete is the first in the list is some sort of special. Let us investigate the inner loop first. That loop assumes that there are at least 2 elements in the list, f and f.next. We compare the name of the next entry with n. If they match then we would have to skip the next entry. We can do that by the statement f.next = f.next.next. That is we replace the reference from the current element f to the next list entry, that is f.next, by the next entry of the next element, which is (n.next).next. We do not have to write the parenthesis. The n.next.next entry can be nil, in that case it is the end of the list. If we found a matching name then we terminate the inner loop with a break statement, and we are done. Otherwise we assign to f the value of f.next and continue the loop execution. Now to the special case that the name to delete is the first in the list. We need the first if branch for that — if already the first element matches the name to delete than we just skip the first element by setting the head of the list to the next entry, which may or may not be nil.

This is one way to solve the task, for operations on lists there exist in most cases various solutions, some optimized to easy or short code, some for performance. You may copy the code segment above to the end of the former code, and maybe add one more copy of our printing loop at the end again. Then you should have a program that reads in a list, prints the contents, then ask for names to delete, and finally prints the resulting list. Maybe you can improve the code, or maybe you can detect special corner cases where it may fail. What is for example when some of your friends have the same name? May the program fail in that case? Or you may add more fields to your Friend data type. Maybe a textual field with content male or female, and you can report the ratio of male to female. And maybe remove males from the list when we have more males then females?

For references to objects the assignment operator = copies the references, but not the object. In the same way the operator == for equality test compares the references, but not the content of the objects to which the references point. If you want to compare the content of the objects, you can apply the dereference operator [] on both references:

type
  RO = ref object
    i: int

var
  ro1 = RO(i: 1)
  ro2 = RO(i: 1)
  ro3 = ro1

echo ro1 == ro2 # false
echo ro1[] == ro2[] # true
echo ro1 == ro3 # true

Procedures and Functions

Introduction

Procedures and functions, called proc and func in Nim, are the most common way to structure larger programs in smaller parts with a dedicated task. The terms procedure and function where used in Pascal and the other languages of Wirth already, while C uses the term function only, and Fortran generally uses the term subroutine instead. And Python and Ruby finally are using the really strange names def and fun for it.

We call or invoke a proc by just writing its name followed by a parameter list enclosed in parenthesis. The parameter list can be empty. When we call a proc, then the program execution continues with that procedure, and when the execution of the procedure terminates, then the next statement after that proc call is executed. Sometimes we say that we jump into a procedure and jump back when that procedure terminates.

In Nim functions are a special form of procedures that return a result and do not modify the current state of the program. Modifying a global variable or an output operation would be examples for modifying the state. We have already used some predefined procedures like echo() for output operations, add() for appending single characters to strings, and readLine() for reading in textual user input. And we talked about math functions like sin(), cos(), pow() — these are functions as they accept one or two arguments and return a result but do not change a state — calling them again with the same arguments would always give the same result. ReadLine() is only a proc, not a function, as the result may be different for each call, and as we pass a file variable as argument, which may change its state for each call, maybe because the file end is reached. A function is only a special subtype of a procedure, the func keyword indicates to the reader of the code and to the compiler some special properties, that is that a result is returned and that global state is not changed. Whenever the func keyword is used a proc would do as well, and in this text we generally speak about procedures, even when a function would do.

Let us start with a very simple function called sqr() for square.

func sqr(i: int): int =
  i * i

A procedure declaration consists of the keyword proc, a user selected name, a optional parameter list enclosed in parenthesis and an optional colon followed by a data type.

Note that this is only a declaration so far — the compiler could recognize the construct, its parameters and its result type. Sometimes we call this construct a procedure-header.

Generally we do not only declare a function, but we define it, that is we add a equal sign to the procedure header and add an indented procedure body that contains the code that is performed for each invocation.

Pure proc declarations can be necessary in rare situations, maybe when two procedures call each other. In this case the procedure defined first would call the other procedure, which is not already defined, so the compiler may complain about a unknown procedure. We could solve that problem by first declaring the second procedure, so that the compiler would know about it existence. We would then define that second procedure later, that is closer to the end of the program file.

The sqr() proc above accepts an integer argument and returns its square of same data type. We would call that proc like

var j: int
j = 7
echo sqr(j)

Earlier in this book we said that the compiler processes our source code from top to bottom, and that the final program is executed from top to bottom too. The first statement is indeed true, for that reason it can be necessary to declare a function at the top, and define it below, as we can not call a proc before it is declared or defined.

For the program execution we have to know that procs are only executed when we call them. That is, when we write a proc at the top of our source code, then that proc is processed by the compiler, but it is not executed during program runtime before we call it.

The procedure body builds a new scope. We can declare entities like variables, constants, types or other procedures and functions in that scope. These entities are only visible in the procedure body but not outside of the proc. Will will learn more about scopes and visibility soon.

Procedure Parameters of Var Type

Our sqr() function above accepts only one parameter and that parameter is a value type, which indicates that we can not modify it in the procedure body. That fact is useful to know for the caller of a proc, as one can be sure that the passed parameter is not modified and is available unchanged after the proc call. But of course there are situations where we may want that a passed parameter is modified. Let us assume we want to "frame" a passed string, for example we want to pass in the string "Hello" and want to change it to "* Hello *". Further let us assume that we may sometimes want to use other characters instead of the asterisk, maybe a + sign.

proc frame(s: var string; c: char = '*') =
  var cs = newString(2)
  cs[0] = c
  cs[1] = ' '
  insert(s, cs)
  add(s, ' ')
  add(s, c)

# we can call that proc like
var message = "Hello World"
frame(message)
echo message

The frame proc above accepts two parameters and returns no result. The first parameter has the type string, it is not a value parameter but a var parameter, which is indicated by the var keyword between the colon and the type of the parameter. Note that we use here again the keyword var that we used earlier to declare variables. The main reason that we use again the same keyword is that we do not want to use a new one — var proc parameters are different from var declarations. Var parameters can be modified in the procedure body, and that modification is visible after the proc call.[23] The second proc parameter is a plain value type, it is a character which has the default value '*'. Default values are useful for parameters that have in most cases the same value, but can be different sometimes. The advantage is, that when calling that proc we can just leave that parameter out. For default values we have to be a bit careful, only value parameter can have default values, and when we call a proc with many parameters with default values it may be not always clear which parameter we pass and for which parameter we want a default value.

To generate the frame around the passed in string we have to insert two characters at the front of the string, and to append two characters. Inserting in strings is not a very cheap operation, as it involves moving all following characters. So we try not to insert two single characters, but we first create a short string consisting of the passed c character and a whitespace character, and then insert that two character string at the front of the passed string. We use the standard procedure newString() with parameter 2 to create a new string of length 2 with undefined content, and then fill in the content by using the subscript operator. We could have used the add() proc to add that two characters to an empty string, but that is a bit slower. Then we use the standard procedure insert() to insert our two character string at the front of our passed string. Finally we add a whitespace and the c character to the passed string. The passed string is now modified, it is 4 characters longer. That modification is noticeable for the caller of that proc, that is echo() will print the modified version.

When we call a proc or function with multiple arguments, then we have to pass the arguments in the same order as they are specified in the proc declaration.

Our frame() proc above modifies the passed string. We could have instead decided that the proc should not modify the string, but should return a new string consisting of the frame and the passed string in the center. Generally when creating procs we have to decide what is more useful — modifying a passed value or returning a modified copy. And sometimes we have to regard efficiency too. Returning newly created large data types like strings may be expensive. A string is not a trivial structure, as it contains the dynamic buffer for the string content, which has to be allocated. On the other hand, for the passed var string we inserted characters, which involves moving characters and is also not a really cheap operation, and maybe when we insert a lot, the string buffer must be even enlarged, which is again expensive. So for this use case it is not really clear what approach is better — we used the var parameter mainly to introduce var parameters. OK, let us investigate how a function that returns a modified string may look:

func framed(s: string; c: char = '*'): string =
  var res = newStringOfCap(s.len + 4)
  add(res, c)
  add(res, ' ')
  add(res, s)
  add(res, ' ')
  add(res, c)
  return res

# we can call that proc like
echo framed("Hello World")
echo framed("Hello World", '#')

Above code is one possible solution. We can use the keyword func instead of proc here as we only return a result but modify no states. We pass the initial string and the character for the frame both as plain value parameters and return a newly created framed string. In the function body we start with an optimized version of the procedure newString() from the system module, called newStringOfCap(). Like newString() that procedure creates an empty string variable, but it ensures that the data buffer of the new string has exactly the specified size. That is an optimization, which makes sense in our use case, as we know that our newly created string will have 4 characters more than the passed string. So we can avoid that the result string has to be enlarged while we add characters or the initial string, and we ensure at the same time that no space is wasted — the data buffer size of the new string will be a perfect fit for the desired result. The rest of the function body is not really interesting, we just add() what is needed and return the result. Well, earlier we said that add() is not extremely fast. So when you have to frame millions of strings each day you may consider avoiding add(), and you know already enough about Nim to do it. Just try it. You may start with an string of right size containing undefined content created by newString(s.len + 4) and then you may copy in the required data in a loop chararcter for character. Or you may use the slice operator to insert the passed string into the new string.

Click to see a possible solution
func framed(s: string; c: char = '*'): string =
  var res = newString(s.len + 4)
  res[0] = c
  res[1] = ' '
  res[2 .. s.high + 2] = s # we may insert the string by using the slice operator or
  # for p in 0 .. s.high: # we can use a for loop and
  #   res[p + 2] = s[p] # the subscript operator
  res[^2] = ' '
  res[^1] = c
  return res

The situation that we may need a procedure that works on a var parameter in one case and returns a modified copy in another case is not that rare. So for example Nim’s standard library contains a procedure called sort() which can sort container data types in place, and a procedure called sorted() which returns a sorted copy. This code duplication is not really that nice. Of course sorted() is the more universal solution, as we can always replace sort(data) with data = sorted(data). But the later creates a temporary copy, which may not be optimal for performance. Since Nim version 1.2 a dup() macro is available from sugar module which creates copies of variables and then applies one or multiple in place procs on the copy. So the procs sorted() or our proc framed() would be unnecessary. We can use dup() as in this example:

from sugar import dup

proc frame(s: var string; c: char = '*') =
  var cs = newString(2)
  cs[0] = c
  cs[1] = ' '
  insert(s, cs)
  add(s, ' ')
  add(s, c)

echo "Hello World".dup(frame)
echo "Hello World".dup(frame, frame)
echo "Hello World".dup(frame('#'))

Note that we apply frame() two times in the line before the last one — in the same way we could apply a sequence of different procs. The result of above program is

* Hello World *
* * Hello World * *
# Hello World #

The execution of a procedure terminates when the last statement of the procedure body has been processed. We can also terminate a procedure earlier when we specify a return statement somewhere.

Functions and procedures which return a result can also terminate with the last expression of the procedure body, or earlier with a return expression like return i * i. Functions and procedures with a result declare automatically a result variable for us, which we may use or just ignore. So for our previous sqr() function we have various ways to write it:

func sqr1(i: int): int =
  i * i

func sqr2(i: int): int =
  result = i * i

func sqr3(i: int): int =
  return i * i

For short and simple procedures the first form is often used. For longer procedures where the result is constructed in multiple steps, like some string operations, using the result variable makes sense. And finally, when there exist multiple points where we may jump back using return statements may make sense. One use case is an early error check, maybe we want to return -1 as some form of error indication when we write a procedure that should calculate the square root of an integer value. (Well in Nim we have other and sometimes better ways to catch errors, we will learn about that later.)

Generally we should avoid writing something like

func sqr(i: int): int =
  result = i
  i * i

as it may be unclear in this case if the expression i * i is returned or the result variable with value i. For Nim v1.4 we will get a warning or an error message in such a case.

Note that we used the proc names sqr1, sqr2 and sqr3 above. Using the same name with the same argument types multiple times would result in a redefinition error, as the compiler could not know what proc body should be executed when that proc name is called.

But Nim supports so called proc overloading, that is we can use the same name when the parameter list is different, as the compiler can select from the parameters in the proc call which proc has to be called:

func sqr(i: real): real =
  i * i

We have only changed the parameter and result data type. Now there is no conflict with the proc with same name which we defined for integers. Note that Nim use only the parameter list for overload resolution, but not the result type of a proc or function. The reason for that is that Nim supports type inference, and that would not work when we would have two procs with same name each accepting an int parameter but one returning an int and one returning a float number.

Nim does also support named arguments in proc calls, that is we could invoke the proc above with sqr(i = 2.0). Named arguments can be useful when procs or functions have many arguments, maybe some with default values, and we do not remember the order of parameters or when we want to specify only a few.

Objects and Ref Objects as Procedure Parameters

In the previous section we learned that we have to pass var parameters when the procedure should be able to mutate the variable permanently. This is also valid when the parameters are objects. When a procedure should modify fields of an object parameter, then we have to pass that object as a var parameter. In the following example proc t1 gives a compiler error because that procedure tries to modify a field of an object while the object instance is not passed as a var parameter. If we remove proc t1 then we can compile and run the example:

type O = object
  i: int

proc t1(o: O) =
  o.i = 7 # Error: 'o.i' cannot be assigned to

proc t2(o: var O) =
  o.i = 13

proc main =
  var x = O(i: 3)
  echo x.repr
  t2(x)
  echo x.repr

main()

The output is:

[i = 3]
[i = 13]

Proc t2 gets a var parameter and can modify fields of the passed object. Here we used the expression echo x.repr to print the whole object. Strings and sequences are value objects in Nim, so you have to pass them as var parameters when you want to change their length or when you want to modify elements. This code would give you compile errors, unless you add the var keyword to make the proc parameters mutable:

proc t1(s: string) =
  s.setLen(7)
  s[0] = 'x'

proc t2(s: seq[int]) =
  s.setLen(7)
  s[0] = 13

This was not really surprising. But what when we use a reference to an object and pass that to procedures as value and as var parameter? In the code below proc t1 gets a variable of type ref object and the procedure can modify fields of the passed instance. That can be indeed surprising. In this case passing the ref object without use of the var keyword means only that we can not mutate the ref value itself in the procedure, but we are allowed to modify the fields of the object. For proc t2 we pass a var parameter. As always we can modify a var parameter in the procedure, so we can assign it a newly created instance.

type O = ref object
  i: int

proc t1(o: O) =
  o.i = 7

proc t2(o: var O) =
  o = O(i : 11)

proc main =
  var x = O(i : 3)
  echo x.repr
  t1(x)
  echo x.repr
  t2(x)
  echo x.repr

main()

When we compile and run above code we get:

ref 0x7f054a904050 --> [i = 3]

ref 0x7f054a904050 --> [i = 7]

ref 0x7f054a904070 --> [i = 11]

For a ref object the repr function gives us the address of the object instance in memory and the contents of its fields. The first two echo statements shows the same address, indicating that proc t1 has modified only a field of our instance, the instance itself (its address in memory) was not changed. But proc t2 has created a new instance and assigned that value to the variable x in the main procedure. We notice this as the address of variable x has changed. The old instance variable with address 0x7f054a904050 is now unused and will be freed by the Nim memory management.

Special Argument Types: OpenArray and Varargs

The openArray and varargs data types can be used only in parameter lists. OpenArray is a type which allows to pass arrays and sequences to the procedure or function. Allowing that makes sense as arrays as well as seqs store their content in a block of memory, which can be processed uniformly. Although arrays generally do not have to start with index number 0, when passed as openArray the first element is mapped to index 0, and the index of the last element is available by using the high() function on the passed array parameter. Whenever we write a procedure that accepts an array or a sequence, we should consider using the openArray parameter type to allow passing in both data types. Strings can be passed also to procs accepting openArrays with char base type. Note that a proc with openArray parameter type can not change the length of a passed seq, as for the openArray parameter type sequences are handled like arrays. So in the code below proc t1 generates a compiler error while t2 compiles and works fine.

proc t1(x: var openarray[int]) =
  x.setLen(7)

proc t2(x: var seq[int]) =
  x.setLen(7)

The varargs parameter type is similar to the openArray type, but it additional allows passing an arbitrary number of single arguments. The compiler automatically collects the single arguments into an array for us, so in the proc body we can use it like an array, ie. iterating over it.

proc print(s: varargs[string]) =
  for el in s:
    stdout.write(el)
    stdout.write(", ")
  stdout.write('\n')

print("Hello", "World") # compiler builds the array for us
print(["Hello", "World"]) # we generate the array our self

There exists a variant of the varargs argument type that performs a type conversion automatically by applying a proc on all arguments. For example varargs[string, $] would apply the stringify operation on the passed arguments automatically. That is what echo does.

Varargs arguments may be only allowed for the last argument in a parameter list.

Finally we may wonder if it makes sense to specify a parameter of type var varargs. If we try to pass a constant string this will obviously not work, and if the compiler generates an array for us it does also not work, the automatically generated array seems to behave like a constant array. But may we pass an array variable? Let us try:

proc print(s: var varargs[string]) =
  s[0] = "Goodbye"
  for el in s:
    stdout.write(el)
    stdout.write(", ")
  stdout.write('\n')

var msg = ["Hello", "World"]
print(msg)

Surprisingly that does not compile, while it works when we replace varargs with openArray.

Procedures bound to a Data Type

In some other programming languages like Python or Ruby we can define class methods or static methods which are bound to a class or type and can be called as MyType.myProc. In Nim we can do something similar by use of the typedesc proc parameter type:

type
    Factory = object
        name: string

proc start(t: typedesc[Factory]) =
    echo "Factory.start"

Factory.start

Here we used the method call syntax instead of start(Factory). We will learn more about the typedesc data type later.

Scoping , Visibility and Locality

Scoping, visibility and locality is an important concept in computer programming to keep the source code clean. Imagine that a variable which we declare at some point in our program would be visible everywhere. That would even for medium size programs generate a lot of confusion — whenever we would need a variable we would have to carefully check which names are already in use. And for performance it would be bad also, as all variables declared somewhere would reside permanently in memory.

So most programming languages including Nim support the concept of locality — names declared inside of a procedure body or inside another form of block are only visible there and can only be used there. We say that they are only visible in that scope. For Nim we can say that whenever Nim’s syntax requires a new level of indentation, that is a new statement block, then all symbols declared in that block are only visible in that block and in sub-blocks of this block, but not outside of that block. Nim has another important concept of visibility, which is called modules and allows separation our code in logically separated text files with well defined visibility rules, we will discuss modules later.

Visibility is really a simple concept, let us regard this useless example:

var e: float = 2.7

proc p1 =
  var x: float = 3.1415
  if x > 1.0:
    var y = 2.0 * x
    echo y # OK
  echo x # OK
  echo y # compile error, y is not visible
  echo e # OK, e is declared globally, so it is visible everywhere

echo e # OK
echo x # ?
echo y # ?

In line one we declare a so called global variable, that one is visible after declaration, that is below the line where it is declared, in the whole program. The variables declared in the proc p1 are called local variables, they are not visible outside of that proc p1. The variable x is declared at the start of the proc body and is visible in the whole proc everywhere, while variable y is declared in the if block and is visible only there. So it should be clear if the last two echo statements for x and y compile fine?

Related to visibility of variables is their lifetime, that is the duration how long they exist and how long they can store a value. Global variables exist for the whole program runtime — when you have assigned a value to it that value can be used everywhere as long as the program runs, and as long as you do not assign a different value of course. Global variables are generally stored in a special memory region that is called the BSS region.

Variables of value type defined locally inside a procedure or function do exist only for the execution time of that proc, that is they are created when the proc is invoked and vanish when the proc terminates, that is when execution continues with the statement following on the proc call.

Local variables declared in a proc reside in a special memory region of the RAM which is called the stack. The stack is nothing more than an arbitrary part of the hole RAM that is used in some clever fashion: The memory words in it are used in consecutive order. A so called stack pointer is used to indicate the address of the first free area in that stack. So when a proc is called, which may have n bytes of local variables, then the compiler can use the area where the stack pointer points to for that variables, and when the proc is called then the stack pointer is increased by that size. So the stack pointer points again to the next free area of the stack, and another proc can be called in the same way from within the current proc. Whenever a proc terminates, the stack pointer is set back to the value which it had when the proc starts execution. This method of memory management is simple and fast, but it does only work when the total amount of memory that the local variables in a proc needs is known at compile time, so that the compiler can adjust the stack pointer accordingly. It does not work for dynamically sized data types like strings or sequences.

Note that pointers and references are value types itself, we can regard pointers and references as a plain integer variable interpreted in a special way — as a memory location. But the memory blocks to which the pointers and references may point and that is allocated by alloc() or new() is different: That memory blocks are not allocated on the stack, but in the ordinary RAM which we call heap to separate it from the stack.

So why can the stack not be used for memory blocks which alloc() or new() provides for us: An important fact for the use of the stack to store variables is that the total size which is needed by a proc for all the static variables must be a compile time constant. The stack pointer is adjusted by that amount when the proc starts and all the local variables are accessed with a fixed offset to that stack pointer then. When we use alloc() or new() in a proc, then we may call that multiple times like we did in our previous list example, and for alloc() an additional fact is that the byte size that alloc() should reserve can be a runtime value. So the total amount of ram that alloc() or new() would allocate is a runtime value, and we can not use the stack for it. Instead alloc() and new() allocates block of memory in a more dynamic fashion, which is basically that they ask the OS for a free block of right size somewhere in the available RAM. That block is later given back to the OS for reuse by functions like dealloc() or automatically by the GC.

Let us at the end of this section investigate some special cases:

While in languages like C we have always a well defined main() function and all program code is contained in this function or in other functions which are called from this main function, in Nim we have also global code as in scripting languages Ruby or Python:

var i: int
while i < 100:
  var j: int
  j = i * i
  echo j
  inc(i)

It should be clear that the global variable i resides in the BSS segment. But what is with the variable j declared in the body of the while statement? It is clear that that variable is only visible inside of the body of the while statement. But does j reside on a stack? There seems to be no proc involved so there may be no stack? The variable j may reside in the BSS segment too? That is not really clear and may be different for different Nim compilers maybe. But why should we care for that detail at all? Well it may be important for performance. Local proc variables allocated on the stack are generally optimal for performance, and they are optimized by the compiler very well. We will learn more about the reasons for that later when we discuss the data cache. For now we should only remember that it may be a good idea to avoid global code and put all code in procs. We may have an arbitrary named main() proc then and call that from global scope only. At least for the current Nim v1.4 that seems to be a good idea, maybe later versions or other implementations will automatically move all global code into a hidden proc for us.

For optimal performance put all your code in procedures or functions and avoid global code and when possible global variables.

Let us discuss above while loop again, but this time in the body of a proc:

proc p =
  var i: int
  while i < 100:
    let j: int = i * i
    echo j
    inc(i)

When we carefully investigate that proc within the while loop we may wonder about two points. First we said earlier that we can and should use the let keyword instead of var when there is only one assignment to a variable, so the variable can be regarded as immutable. But the loop is executed 100 times, so how can we say there is only a single assignment to variable j? The trick is, that j is locally to the while loop, and that j is virtually new created and initialized to 0 for each iteration. So let is OK and the compiler does not complain.

We can test that fact with this simple program:

proc main =
  var i: int
  while i < 10:
    var a: int
    a = a + 1
    echo a
    inc(i)
main()

The output is 1 for each loop iteration, as variable a is virtually newly created for each loop iteration.

We said virtually newly created, because we can not be sure how the compiler may handle it internally. Is storage for variable a already allocated when the proc is invoked, that is in the same way as storage for the loop counter variable i is allocated on the stack when the proc is called. Or is storage for variable a reserved for each loop iteration by increasing the stack pointer at the start of the loop and resetting it at the end of the loop. We can not be sure without reading the compiler source code, but finally we should not care, as it does not really matter.

Generics

In the previous section we defined a sqr() proc for ints and one for float numbers. Both procs look nearly identical, only the data types differ. For that case we can use so called generic procedures.

func sqr[T](v: T): T =
  var p: T
  p = v * v
  return p

echo sqr(2)
echo sqr(3.1415)

We put a square bracket after the function name which includes a symbolic name, and that name is then used instead of concrete types in the proc header or in the proc body.

We can now call that proc with parameters of different types including int and float types. You may wonder why that works — Nim is a statically typed language, so how can the parameter of function sqr() as well accept an integer as an float number? Is there a hidden type conversion involved? No, the trick is that whenever we call that generic proc with a different type, than a new proc or function is instantiated. As we called the generic sqr() proc with an int and a float parameter, during compile time the compiler creates machine code for two separate function, one which is called when an int is passed as parameter, and one which is called when a float is passed. If we would call that proc name again with an int or float parameter, than one of the two existing procs would be used. But for a different, still unused data type like float32 again a new proc would be instantiated. In this way generics procs can lead to some code bloat. Note that calling the generic function with a data type like a character or a string would fail, as that types do not support multiplication with itself.

A slightly different notation is available by so called or types:

func sqr(v: int or float): auto =
  var p: typeof(v)
  p = v * v
  return p

echo sqr(2)
echo sqr(3.1415)

Here we have limited the parameter types to the int type or the float type. We could have defined also a custom type first, like type MyNum = int or float and use that type for the type of our sqr proc. These or types are also called type classes. Instead of keyword or the | character can be used for defining type classes. Again the compiler would instantiate two separate functions for the both data types. As we had not the symbolic type T available here, we have used the keyword auto as return type, and for the type of variable p we used the macro typeof(). The type auto for the return type works as long as the function returns a well defined type. Note that we can not decide at runtime what a type the function should return, so a construct like if cond: return 2 else: return 3.1415 would not work, at least not when the values are variables of different type. For the literal value it may work, as the compiler may be smart and guess that we want to return the float literal 2.0.

A bit care is needed when we define procs for mutable or types:

# proc t(s: var seq[uint8] | var seq[char]) =
proc t(s: var (seq[uint8] | seq[char])) =

Here we try to define a proc called t which should accept a mutable seq[uint8] or a mutable seq[char] as parameter. While the first line compiles fine, the seq[char] would be immutable. The correct notation is shown in the second line. This behavior was labeled "won’t fix" in github issue tracker, so we have to remember this case, see https://github.com/nim-lang/Nim/issues/15063#issue-665553657.

Let us assume that you want to define a proc that accepts two numbers of type int or float and that returns a float. You may write it in one of this ways:

proc sqrsum(x, y: int | float): float =
  (x * x).float + (y * y).float

proc sqrsum2[T](x, y: T): float =
  (x * x).float + (y * y).float

proc sqrsum3[T1, T2](x: T1; y: T2): float =
  (x * x).float + (y * y).float

var i: int = 2
var x:float = 3.0

echo sqrsum(i, x)
#echo sqrsum2(i, x)
echo sqrsum2(x, 2)
#echo sqrsum2(2, x)
echo sqrsum3(i, x)

The commented out lines would give you a compiler error. The reason for this is that the proc sqrsum2[T] defines a generic proc, but the compiler enforces that both parameters have the same type. The expression sqrsum2(x, 2) compiles fine, as due to the first parameter x the compiler instantiates a proc for a float parameter type, and then converts the second parameter, which is an integer literal, to float automatically. This automatic conversion is only done for literal numbers, not for variables. The expression sqrsum2(2, x) does not compile, as due to the first parameter, which is an integer literal, a proc for integer parameters is instantiated, and the second x parameter of float type is not compatible with the instantiated proc.

Generics can become a bit complicated, as we may use multiple different generic types for different proc parameters. And we can use generics also for object types, we may for example create lists like we did for our names list that work not only for strings, but that can work with other data types like numbers or sequences in a very similar way. We may explain that in more detail later.

Example for the use of Generics

Generics are used a lot in Nim’s standard library. Most container types like sequences or tables accept generic types, and generic procedures like sort() are provided which can easily sort arbitrary data types and objects. We have only to provide a cmp() proc for our user defined data types which sort() can call to compare the values during the sorting process.

We will demonstrate the use of generics for library modules with a few tiny examples: Assume we create a library which should be able to store and process arbitrary data types. The stored values may have well defined relations, which enables ordering or much more complicated spatial relations. Triangulation of spatial data points or grouping the data in structures like RTrees for fast point location as well as geometric processing with algorithm like finding the convex hull are some examples. To make our example simple and compact, we define a generic container type which can store only two values of arbitrary data type. The container allows to sort the elements by size. The following code example defines a generic container called MyGenericContainer, a proc to add() data object into the container instance and a sortBySize() proc to sort the two elements:

type
  MyGenericContainer[T] = object
    storage: array[2, T]

proc add[T](c: var MyGenericContainer[T]; x, y: T) =
  c.storage[0] = x
  c.storage[1] = y

# sort by direct field access
proc sortBySize[T](c: var MyGenericContainer[T]) =
  if c.storage[0].size > c.storage[1].size:
    swap(c.storage[0], c.storage[1])

# a simple stringify proc for our container data type
proc `$`[T](c: MyGenericContainer[T]): string =
  `$`(c.storage[0]) & ", " & `$`(c.storage[1])

type
  TestObj1 = object
    name: string
    size: int

proc main =
  var c: MyGenericContainer[TestObj1]
  var a = TestObj1(name: "Alice", size: 162)
  var b = TestObj1(name: "Bob", size: 184)

  add(c, b, a)
  echo c
  c.sortBySize
  echo c

main()

The sortBySize() proc of the above examples accesses the size field of our data objects directly, so we can use the container for arbitrary data types as long as the data types have a size field and as long as a > proc is defined for the data type of the size field. In the above example we have defined a $ procedure to convert instances of our container to a string, which allows us to call the echo function on it. The output of our program looks like

(name: "Bob", size: 184), (name: "Alice", size: 162)
(name: "Alice", size: 162), (name: "Bob", size: 184)

We can avoid the restriction of a matching field name when we provide getter and setter procedures which the library procs can use to access the important fields:

type
  MyGenericContainer[T] = object
    storage: array[2, T]

proc add[T](c: var MyGenericContainer[T]; x, y: T) =
  c.storage[0] = x
  c.storage[1] = y

proc sortBySize[T](c: var MyGenericContainer[T]) =
  if c.storage[0].size > c.storage[1].size:
    swap(c.storage[0], c.storage[1])

proc `$`[T](c: MyGenericContainer[T]): string =
  `$`(c.storage[0]) & ", " & `$`(c.storage[1])

type
  TestObj1 = object # arbitrary field names
    name: string
    length: int

# this getter proc enables sorting
proc size(t: TestObj1): int =
  t.length

proc main =
  var c: MyGenericContainer[TestObj1]
  var a = TestObj1(name: "Alice", length: 162)
  var b = TestObj1(name: "Bob", length: 184)

  add(c, b, a)
  echo c
  c.sortBySize
  echo c

main()

In the example above our TestObj1 data type has no field with a name matching for the sortBySize() proc, but we define a size() proc for our data type which that library function can use. This solution is more flexible, and when we add the inline pragma to the used size() proc or when we compile with link time optimization (LTO) enabled, then the overhead should be negligible. The two examples above are located in a single file each, but of course for practical use we would use separate modules for the library and the application part as in

#module t3.nim
type
  MyGenericContainer*[T] = object
    storage: array[2, T]

proc add*[T](c: var MyGenericContainer[T]; x, y: T) =
  c.storage[0] = x
  c.storage[1] = y

proc sortBySize*[T](c: var MyGenericContainer[T]) =
  if c.storage[0].size > c.storage[1].size:
    swap(c.storage[0], c.storage[1])

proc `$`*[T](c: MyGenericContainer[T]): string =
  `$`(c.storage[0]) & ", " & `$`(c.storage[1])
import t3

type
  TestObj1 = object # arbitrary field names
    name: string
    length: int

proc size(t: TestObj1): int =
  t.length

proc main =
  var c: MyGenericContainer[TestObj1]
  var a = TestObj1(name: "Alice", length: 162)
  var b = TestObj1(name: "Bob", length: 184)

  add(c, b, a)
  echo c
  c.sortBySize
  echo c

main()

The example with direct field access would look for different modules like this:

# module t4.nim
type
  MyGenericContainer*[T] = object
    storage: array[2, T]

proc add*[T](c: var MyGenericContainer[T]; x, y: T) =
  c.storage[0] = x
  c.storage[1] = y

proc sortBySize*[T](c: var MyGenericContainer[T]) =
  if c.storage[0].size > c.storage[1].size:
    swap(c.storage[0], c.storage[1])

proc `$`*[T](c: MyGenericContainer[T]): string =
  `$`(c.storage[0]) & ", " & `$`(c.storage[1])
import t4

type
  TestObj1 = object
    name: string
    size: int

proc main =
  var c: MyGenericContainer[TestObj1]
  var a = TestObj1(name: "Alice", size: 162)
  var b = TestObj1(name: "Bob", size: 184)

  add(c, b, a)
  echo c
  c.sortBySize
  echo c

main()

You may wonder why we do not have to export the size field of our TestObj1 (or maybe the object itself also) as it is used from code defined in a different module. The reason why we do not need export markers is that the sortBySize() is defined in the library module, but as it is a generic procedure, it is instantiated and executed in the application module. For the same reason we had not to export the size() getter procedure before.

Finally one more way to use generic library modules is by passing procedure variables to the library functions. The passed in procedures may provide access to properties or attributes of the stored objects, or they may offer relations between the objects. The later is often used for sorting purposes:

# module tx.nim
type
  MyGenericContainer*[T] = object
    storage: array[2, T]

proc add*[T](c: var MyGenericContainer[T]; x, y: T) =
  c.storage[0] = x
  c.storage[1] = y

proc sortBy*[T](c: var MyGenericContainer[T]; smaller: proc(a, b: T): bool) =
  if smaller(c.storage[1], c.storage[0]):
    swap(c.storage[0], c.storage[1])

proc `$`*[T](c: MyGenericContainer[T]): string =
  `$`(c.storage[0]) & ", " & `$`(c.storage[1])
import tx

type
  TestObj1 = object
    name: string
    size: int

proc smaller(a, b: TestObj1): bool =
  a.size < b.size

proc main =
  var c: MyGenericContainer[TestObj1]
  var a = TestObj1(name: "Alice", size: 162)
  var b = TestObj1(name: "Bob", size: 184)

  add(c, b, a)
  echo c
  c.sortBy(smaller)
  echo c

main()

Here we have modified the sort() proc of our library module in a way that it takes an additional procedure parameter. In this case we use a procedure signature that takes two object instances and returns a boolean value indicating if the first parameter is smaller than the second. In our application module we define a matching procedure and pass that one to the sortBy() procedure. Again we get the desired sorted output:

(name: "Bob", size: 184), (name: "Alice", size: 162)
(name: "Alice", size: 162), (name: "Bob", size: 184)

This last method is used often in Nim’s standard library, e.g. for sorting sequences with custom objects. Unfortunately this way can introduce some performance regression, as the procedure variable has to be passed to the called proc and so inlining of that passed proc is not possible for the compiler. [24]

Method Call Syntax

A useful coding style introduced by OPP languages is the method call syntax, which was initially used in OOP programming style for objects, and later applied by languages like Ruby to all data types. Ruby in some way regards all data as objects.

Method call syntax means, that for example for a variable s of data type string we do write s.add(c) instead of add(s, c). Or for an integer variable i we may write i.abs instead of abs(i). That is we put the first parameter of the proc parameter list in front of the proc name, and separate that parameter from the proc name by a period. The compiler regards both notation as equivalent. The advantage of the method call syntax is that we may save a character and that it is more clear with what "object" we are working, as it stands in front of the expression.

Most OOP languages allows that notation only for a class, for example the string class may declare all possible operations that can be done with strings, and the method call syntax is used for that operations. One problem is, that it can be difficult to add more operations which can be used in that style, as often all that operations are defined in the class scope. Ruby fixed that restriction by allowing so called reopen of classes, that is user can later add more operations.

Nim simple allows that notation generally, as did the D language, but D used the tern Uniform Function Call Syntax (UFCS) for it.

Procedure Variables

Procedures and functions are not always fully static entities. We can assign procedures and functions to variables, and we can pass whole procedures or functions as parameters to other procedures or functions. And functions can even generate and return new functions. Let us investigate how procedure variables work:

var
  p: proc(i: int): int

proc p1(i: int): int =
  i + i

proc p2(i: int): int =
  i * i

p = p1
echo p(7)
p = p2
echo p(7)

The output of the two echo statements should be 14 and 49 — we called in both cases the same proc variable with the same parameter, but the proc variable p was in the first call an alias for p1, and in the second call an alias for p2. Note, when we assign a proc to a proc variable we do only write the name of that proc, there is no () involved. That is because we assign that proc to the proc variable, but we do not call the proc in this case. Of course, when we assign a proc to a proc variable then the proc signatures have to match, that is the parameter list and the result have to be compatible.

Now we use a function as a proc argument.

type
  EchoProc = proc (x: float)

proc t(ep: EchoProc; x: float) =
  echo "The value is"
  ep(x)

proc ep1(x: float) =
  echo "==> ", x

proc ep2(x: float) =
  echo x

t(ep1, 3.1415)
t(ep2, 3.1415)

A common use case for a function as a proc parameter is sorting. We can use the same sort procedure for different data types when we provide cmp proc that can compare that data type.

from algorithm import sort

proc cmp(a, b: int): int =
  if a < b:
    -1
  elif a == b:
    0
  else:
    1

proc main =
  var a = [2, 3, 1]
  a.sort(cmp)
  for i in a:
    echo i

main()

The sort procedure is provided by the algorithm module. The sort proc accepts an array or a sequence, and a cmp proc that gets two parameters of the same type as the elements in the passed array, and that returns -1, 0, or 1 as the result of the comparison. We could easily sort other data types like strings or our custom objects by an arbitrary key, as long as we can provide a matching cmp() proc. For the cmp proc it is important that it returns a well defined result based on the input, and when both parameters are equal it should really return 0. If you would exchange the return values 1 and -1 in the cmp proc above, you would invert the sort order.

Nested Procedures and Closures

While in C all functions must be defined in top level scope and nesting of functions is not allowed, in Nim procedures can contain other procedures. A special case occurs when the sub-procedures do access variables of the outer scope. In this case the sub-procedure is called a closure:

proc digitScanner(s: string) =

  var pos = 0
  proc nextDigit: char =
    while pos < s.len and s[pos] notin {'0' .. '9'}:
      inc(pos)
    if pos == s.len:
      return '\x0'
    result = s[pos]
    inc(pos)

  var c: char
  while true:
    c = nextDigit()
    if c == '\x0':
      break
    stdout.write(c)
  stdout.write('\n')

digitScanner("ad5f2eo73q9st")

When you run this program the output should be

52739

This program is not that easy, but when you think about it a bit you should be able to understand it. The task is to extract from a string all the digits and to ignore the other characters.

To get the digits, we use a local procedure that uses the pos variable of the enclosing procedure, and also access the parameter s of the enclosing procedure. The closure nextDigit() checks if the position in the string is still valid, that is if it is still smaller than the length of the string, and also checks if the current character is a digit. The first check uses the standard procedure len() which return the length of a passed string parameter, that is how many characters the string contains. We have used the method call syntax here instead of using the ordinary proc call len(s). The next check test if the current character is not a decimal digit. For that test we could use a series of compares like if c == '0' or c == '1' or …​ or c == '9'. But to make such tests easier and faster, Nim offers one more data type, the set type. And the notin operator test if an value is not contained in a set constant. An important point for the expression after the while statement is, that it is processed from left to right. That fact is here important, because we have first to check if pos is still a valid position, before we can use the subscript operator [] to access the current character and test if it is not contained in the set. If the check for the valid position would not come first, then we may access an invalid position in the string and we would get a runtime range error.

While the position is still valid but the current character is not a digit we increase the position. The while loop can end by two conditions: Either the current character is a digit, or we have reached the end of the string and we have to stop. For the last case we use a special stop mark, we return a special character which we have entered in escape notation as '\x0'. That is a very special character, that is used in C to mark the end of strings. It is the first character in the ASCII table and has the decimal value 0. We said earlier that characters are encoded in 8 bit and correspond to the unsigned integer numbers 0 up to 255. '\x0' is just a special notation for the first character which corresponds to integer value 0. Well, when the end of the string is reached then we return that character. Otherwise we return the current character. Remember, from the while condition we know that string end is reached, or current character is a digit. As we tested for string end before, we can only have the case that current character is a digit now. But can we return that character immediately now? If we would, s[pos] would be a digit, and we would get exactly the same character for the next proc call! So we have to move to the next character by increasing pos before we return that character. For this the pre-declared result variable is useful. We assign the current character to the result variable, and then increase pos. As the last statement in our proc is not a expression but a plain inc() statement, the content of the result variable is returned. The other while loop in the outer procedure is very simple, we just call the closure in the body of the while loop and terminate the loop when we get the special Null character.

And finally an example where one proc returns another procedure:

proc addN(n: int): auto = (proc(x: int): int = x + n)

let add2 = addN(2)
echo add2(7)

The output of echo would be 9 in this case. This construct is sometimes named currying.

Converters

Nim’s converters are a special variant of functions that are called automatically by the compiler when argument types do not match.

converter myIntToBool(i: int): bool =
  if i == 0:
    false
  else:
    true

proc processBool(b: bool) =
  if b:
    echo "true"
  else:
    echo "false"

var i = 7
processBool(i)
if i:
  echo "true"
else:
  echo "false"

With above converter we can pass an integer to a proc that expects a bool parameter, and we can even use an integer as a logical expression in an if condition in the same way as it is done in C language. Converters do work only in a direct way, that is automatic chaining is not supported: If we have one converter from character to int and one from int to boolean, that does not mean that we can pass a character to a proc that expects a boolean. We would have to declare one more converter that directly converts character to bool.

Whenever we do consider using converters we should think twice — converters may be confusing, may have some strange effects and may increase compile time.

Maybe you wondered why we wrote above converter in such a verbose way? Well it was done intentionally, but you are right of course, we can write it just as

converter myIntToBool(i: int): bool =
  i != 0

Object Orientated Programming and Inheritance

Object Orientated Programming and Inheritance became very popular in the early nineties of the last century. Java is a prominent representative of that programming paradigm, but most languages created in the nineties of the last century support it, like C++, Ruby and Python.

The idea of OOP is that objects and procedures working on that objects are grouped to classes, and that classes can be extended with additional data fields and with additional procedures. In OOP procedures and function are often called methods and data fields are called members. Sometimes the members are completely hidden and are accessed only by so called getter and setter methods. That is called encapsulation. Encapsulation allows hiding implementation details, so that that details may change when necessary, without that the change of internal details become visible to users of that class, so that the users can use the class without noticing the change. Getters and setters also help hiding internal details and they ensure that the class is always in a consistent and valid state.

An important property of OOP is dynamic dispatch: When we create various sub-classes of a common parent class, and we have defined methods for all the sub-classes, then we can have collections of instances of different sub-classes, and the compiler can automatically ensure that always the matching method for each instance is called.

A classical example is a drawing program, where we have different geometrical shapes like rectangle, circle and many more. All the geometrical objects are stored is some form of list, and when we want to draw all of them on the screen then we have to call only an unspecific draw() method, and the compiler ensures that for each shape the matching draw method is called. In Nim that may look like

type
  Shape = ref object of RootRef

  Rectangle = ref object of Shape
    x, y, width, height: float

  Circle = ref object of Shape
    x, y, radius: float

  LineSegment = ref object of Shape
    x1, y1, x2, y2: float

method draw(s: Shape) {.base.} =
  # override this base method
  quit "to override!"

method draw(r: Rectangle) =
  echo "drawing a rectangle"

method draw(r: Circle) =
  echo "drawing a circle"

method draw(r: LineSegment) =
  echo "drawing a line segment"

proc main =
  var l: seq[Shape]
  l.add(Rectangle(x: 0, y: 0, width: 100, height: 50))
  l.add(Circle(x: 60, y: 20, radius: 50))
  l.add(LineSegment(x1: 20, y1: 20, x2: 50, y2: 50))

  for el in l:
    draw(el)

main()

The output of that program is

drawing a rectangle
drawing a circle
drawing a line segment

So we can have a sequence of the base type, add various sub-types and then iterate over the list to draw all the various sub-types. Of course in the same way we could do many more task like moving, rotating or storing all the objects in one call. The compiler does the right dynamic dispatching for us, we have just to provide all necessary methods. The need of the base method seems to be a bit strange, some other OOP languages do not need that. The base method is marked by a {.base.} pragma, we will discuss the purpose of pragmas later. In the example we have used only one level of sub-classing, but of course we can use many levels, for example we can again subclass the Circle by a FilledCircle with a color field.

The OOP coding style can be very convenient for some tasks, one important use case could be graphical user interfaces where the graphical elements like labels, buttons, frames built in natural way a hierarchical structure. Another typical use case are drawing applications with code similar to our basic example.

Not that the OOP style only works with ref objects, but not with value objects. The obvious reason is that we can have collections of different sub-types stored in arrays or seqences only for ref objects, as in arrays and sequences all element types have to have equal size. For references that it the case, as references are basically pointers. But different value types would have different size. Linked lists would be no better solution, as again we can not built lists with value objects.

For maximum performance OOP code with ref objects is generally not optimal, as the dispatching itself needs some time, and as the ref objects are not contained in a single block of memory, but are distributed in the whole RAM, which is not cache friendly.

Other Builtin Data Types

Tuples

Tuples can be regarded as some form of lightweight objects without inheritance support, but mostly tuples are used when functions has to return multiple arguments.

type
  Move = tuple
    from: int
    to: int
    check: bool

proc findBestNextMove(): (dest: int; check: bool) =
  ...

let (dst, check) = findBestNextMove()

Here we define a custom Move type similar as we would do for a custom object, but we do not use that type at all but return an anonymous tuple from our proc, that is a pair of an int and a bool.

Due to type inference our dst and check variables gets the data types int and bool.

Tuples are also useful when a function has to return a value and also an error state, or if it may not be able to return something at all in a special case. For references we could return nil then, but for results like int or float we may not have a well defined error indicating constant, so we can return a tuple with an additional bool. But we could use exceptions instead, or we could use Nim’s option type instead. We will learn more about that later.

Object Variants

Nim’s object variants, sometimes also called sum types or abstract data types (ADT), are an advanced and type save variant of the union type known from C. The basic idea is that we may use value types that may store similar, but not identical data. Untyped languages like Ruby or Python allow that of course, and we can do it in Nim with ref types and inheritance too, as we showed in a previous section with our Shape base type and various geometric shapes. We could store that ref types in arrays or sequences or linked list and use dynamic dispatch for processing the various sub-types. That is convenient but gives not maximum performance due to dynamic dispatch at runtime and due to bad cache use. So we may like to have a value type with different content, so that we can store all the value types in a seq and all entities reside in a compact block of memory for good cache use.

type
  ShapeKind = enum
    line, rect, circ

  Shape = object
    visible: bool
    case kind: ShapeKind
    of line:
      x1, y1, x2, y2: float
    of rect:
      x, y, width, height: float
    of circ:
      x0, y0, radius: float

proc draw(el: Shape) =
  if el.kind == line:
    echo "process line segment"
  elif el.kind == rect:
    echo "process rectangle"
  elif el.kind == circ:
    echo "process circle"
  else:
    echo "unknown shape"

var
  s: seq[Shape]
s.add(Shape(kind: circ, x0: 0, y0:0, radius: 100, visible: true))
for el in s:
  draw(el)

Objects variants can have common fields like the boolean state visible above, but the other fields are not allowed to use the same names, so we had to use x0 and y0 for the names of the center coordinates in the cycle variant.

As you can see we can store all the different object variants as value objects in a sequence and iterate over it. Note that object variants may waste some storage, as all variants are silently enlarged to have the exact same size, so that all variant types can be stored in an array or sequences and can be passed as proc parameters in the same way to the same proc.

Iterators

We have already used some of Nim’s standard iterators to iterate over the characters of a string or the content of a sequence. Declaring iterators is similar to declaring procedures, but we use the yield keyword to return a single value back to the caller of the iterator. Iterators are used in for loops, and the iteration continues until the end of the body of the iterator declaration is reached the iterator terminates. In an earlier section we declared a procedure that extracted the decimal digits from a string. We can do the same with a iterator:

iterator decDigits(s: string) =
  var pos = 0
  while pos < s.len and s[pos] notin {'0' .. '9'}:
    inc(pos)
  if pos < s.len:
    inc(pos)
    yield(s[pos - 1])

for d in decDigits("df4j6dr78sd31tz"):
  stdout.write(d)
stdout.write('\n')

Nim supports closure-iterators and inline-iterators. The later have to be marked with the {.inline.} pragma in the header of the declaration. Inline-iterators are inserted like templates at the location where they are called and offer maximum performance, but each call increases the total code size. Closure iterators are invoked like ordinary procedures, the invocation eats some performance. Note: Iterators do not provide a predefined result variable as procedures do. To avoid the need of declaring a res variable of type character, in above code we increase pos by one first and yield s[pos - 1].

Templates

Nim templates are a more powerful variant of the macros in C. While in C macros are processed by the pre-processor and are mostly just a plain text substitution, in Nim templates are a fundamental part of the languages and can be used in a similar way like procedures. But while procedures built a single block of code that is then called multiple times, templates work like C macros as textual substitution. Wherever we call a template the template source code is inserted.

template sqr(i: int): int =
  i * i

var j = 7
echo sqr(j)

Casts and Type Conversion

While we have various types of casts in C++, we have only one cast and type conversions in Nim. In Nim casts just reinterpret the same bit pattern for another data type. For example the boolean value false is internally encoded as a byte with all bits cleared, and true is encoded as a byte with all bit but the least significant one cleared. We could cast a bool to an int8 of same size and would get a decimal number 0 or 1. Casting is not a real operation at all, as nothing is really done. We watch the same bit pattern just from a different perspective. But casting is dangerous, it violates the safe type system of the language, and it can go very wrong: Can we cast between float64 and int64? Well they have same size, and both are numbers. We can cast, but the result would be far away from what we may expect. While int64 has the well known and simple value encoding, that is rightmost bit stands for 2^0, next bit for 2^1 and so far, the encoding of floats is much more difficult and has not such a simple scheme. In floats some bits represent the so called mantissa and some bits represent the exponent. When we cast we may again get a number, but the value is not really predictable. We have to be very carefully when we cast between types of different size. Nim may permit that but we have to think what may really happen. When we cast between a bool and a int64, in one direction 7 bytes have to be ignored, and in the other direction for 7 missing bytes there is some padding necessary. We do a cast by writing after the keyword cast in square brackets the desired type followed by parenthesis enclosing the source variable:

var i: uint8 = cast[unint8](myBoolVar)

Totally different to casting is type conversion. We can convert integers to floats without problem, for the conversion we use the type like a proc call, that is int(myfloat) or float(myInt) — of course we could use method call syntax like myInt.float instead. Type conversion is some effort for the CPU, but most advanced CPU’s should have fast instructions for basic conversion.

Nim generally only allows type conversions that involve not too much effort. So we should not expect var i ="1234"; echo i.int * 7 to be available. Such a conversion is expensive, at runtime it costs many CPU cycles as we would have to extract the digits, multiply with it weight and sum them up. So for that operation functions like parseInt() are available from the Nim standard library that accept a string as argument and return an int. There exists different variants of parseInt(), one may raise exceptions for invalid input, the other may return a boolean.

Exceptions

When we execute our code, sometimes something can go wrong: We may have an unexpected division by zero or an overflow, or we get some invalid user input. There exists various strategies to handle such situations. One is to terminate our program, we may do that by a plain assert() or quit() statements. If we have absolutely no idea how to recover from an error then that may be our best option. The user may restart the program, or the program may be restarted by some sort of supervisor program. For more expectable errors some sort of error indicator may be a better solution, for example a parseInt() procedure may return a boolean value for success. As we have to return the result for success also, the parseInt() proc may return a tuple, or may get a var parameter in which the result is returned. Another solutions are exceptions as known from C++.

proc charToInt(c: char): int =
  if c in {'0' .. '9'}:
    return ord(c) - ord('0')
  raise newException(OSError, "parse error")

proc main =
  while true:
    stdout.write("Please enter a single decimal digit: ")
    let s = stdin.readline
    try:
      echo "Fine, the number is: ", charToInt(s[0])
    except:
      if s.len == 0:
        break
      echo "Try again"

main()

This section has to be extended…​

Destructors

Destructors and finalizers are used for automatic resource management. For example files can be closed automatically when a file variable goes out of scope, or when we create high level Nim bindings to C libraries we can use finalizers or destructors to deallocate entities of the C libs when a corresponding Nim (proxy) object is freed. Libraries like the gintro GTK bindings make use of this.

Finalizers are procedures that can be passed as a second optional parameter to the system new() proc. That way the finalizer proc is attached to the data type of the variable which we pass as first parameter to new() and that finalizer proc is automatically called whenever that variable is freed by the Nim memory management system. As finalizers are passed as a parameter to a new() call, and new() is only used for references, finalizers work only for ref data types.

Destructors do not have this restriction. We define the destructor for a value type, but it is also called for reference types by the compiler.

Starting with version 1.4 Nim got scope based resource management when the program is compiled with --gc:arc or --gc:orc. In that case variables are immediately deallocated when they go out of scope, and if a destructor was defined for the data type of that variable it is called automatically.

For the programming language C++ it is a common practice that resources like files are closed and released automatically by destructors when they go out of scope, and now this is also possible for Nim. To make use of destructors for our own data types we have to define a proc called =destroy which gets an instance of our data type passed as a var value object:

type
  O = object
    i: int

proc `=destroy`(o: var O) =
  echo "destroying O"

import random

proc test =
  for i in 0 .. 5:
    if rand(9) > 1:
      var o: O
      o.i = rand(100)
      echo o.i * o.i

randomize()
test()

In the for loop we enter a new scope when the if condition is evaluated to true, and at the end of the if block we leaf the scope and the destructor is called automatically. Inside the destructor proc we could do some cleanup tasks, close files and release resources. Destructors are also called when ref objects go out of scope:

type
  O = ref object of RootRef
    i: int

proc `=destroy`(o: var typeof(O()[])) =
  echo "destroying O"

import random

proc test =
  for i in 0 .. 5:
    if rand(9) > 1:
      var o: O = O() # new O
      o.i = rand(100)
      echo o.i * o.i

randomize()
test()

To use destructors we have to compile with the option --gc:arc or --gc:orc, otherwise the specified destructor procs are just ignored. In our code we can test for working destructors with a construct like "when defined(gcDestructors):".

Note that destructors do not work for plain pointer types:

type
  O = object
    i: int
  OP = ptr O

proc `=destroy`(o: var O) =
  echo "destroying O"

import random

proc test =
  for i in 0 .. 5:
    if rand(9) > 1:
      var o: OP = create(O) # new O
      o.i = rand(100)
      echo o.i * o.i

randomize()
test()

So using destructors to release data from C libraries directly is not possible. But at least for Nim >= v1.6 destructors work for distinct pointer types:

type
  O = object
    i: int
  OP1 = ptr O
  OP = distinct ptr O

proc `=destroy`(o: var OP) =
  echo "destroying OP"

import random

proc test =
  for i in 0 .. 5:
    if rand(9) > 1:
      var o: OP = OP(create(O)) # new O
      OP1(o).i = rand(100)
      echo OP1(o).i * OP1(o).i

randomize()
test()
81
destroying OP
3600
destroying OP
2401
destroying OP
9025
destroying OP

So using destructors to destroy data from C libraries should be possible now.

Destructors and Inheritance

When we use OOP style programming with subclassing of ref objects, then it is useful to know that for subclassed ref objects the destructor of the parent class is automatically invoked when we do not define our own one for our subclassed type. This works also when we import the parent type from another module, at least since Nim v1.6:

# module tt.nim
type
  O1* = ref object of Rootref
    i*: int

when defined(gcDestructors): # check not really needed, as =destroy call is just ignored when condition is false
  proc `=destroy`*(o1: var typeof(O1()[])) =
    echo "destroy O1 ", typeof(o1)
# module t.nim
import tt

type
  O2 = ref object of tt.O1
    j: int

type
  O3 = ref object
    o1: tt.O1

type
  O4 = object
    o1: tt.O1

type
  O5 = ref object of tt.O1
    x: float

when defined(gcDestructors):
  proc `=destroy`(o5: var typeof(O5()[])) =
    echo "destroy O5 ", typeof(o5)
    tt.`=destroy`(o5)

proc main =
  var o1: tt.O1
  new o1
  echo o1.i

  var o2: O2
  new o2
  echo o2.j

  var o3: O3
  new o3
  new o3.o1

  var o4: O4
  new o4.o1

  var o5: O5 = O5(x: 3.1415)
  echo o5.x

main()

When we compile module t.nim with --gc:arc or --gc:orc and run it, we get this output:

0
0
3.1415
destroy O5 O5:ObjectType
destroy O1 O1:ObjectType
destroy O1 O1:ObjectType
destroy O1 O1:ObjectType
destroy O1 O1:ObjectType
destroy O1 O1:ObjectType

So when our variables o1 to o5 go out of scope, then the destructors are called. Module tt.nim defines a ref object type, but the destructor proc takes a var value parameter. The destructor is called when a value object or a ref object goes out of scope. Our variable o1 has type tt.O1, so it was indeed expected that its destructor from module tt.nim is called. Variable o2 is a ref object with parent O1, as we define no destructor for this type, the destructor of the parent type is called. The variabales o3 and o4 are of ref object and of value object types, each with a field of type O1, and for that field the destructor for O1 is called. Finally for type O5 we define our own destructor, which then additional calls the destructor of module tt.

Destructors are mostly used for library implementations, e.g. for a file data type which is automatically closed when a file variable goes out of scope. As you may never have to use destructors yourself, it is not necessary to remember all these details. But it is good to know that destructors behave in a way as we may have expected, and when you later wants to use a destructor in your own code, you can consult again this section or maybe better the Nim manual.

References:

Finalizers

In Nim finalizers are procedures, that we can specify as an optional second parameter when we call the system new() proc to allocate heap memory for a reference type variable. That specified finalizer proc is then later called by the Nim memory management system when the ref variable is freed:

type
  O = ref object of RootRef
    i: int

proc finO(o: O) =
  echo "finalize O"

proc newO: O =
  new(result, finO)

proc main =
  var o = newO()
  var o2 = new(O)
  var o3 = O(i: 7)

main()
finalize O
finalize O
finalize O

The output of above program is really surprising at first: Only for variable o we call the proc newO() to initialize it, which then calls new() by passing a finalizer proc named finO(). For o2 and o3 we allocate memory as usual, without use of a finalizer proc. But when o2 and o3 goes out of scope, even for these two variables the finalizer proc finO() is called. The reason for this is, that the system proc new() binds the optional finalizer proc to the data type of the passed ref variable. This binding process occurs for the first call with a passed finalizer proc and can not be reverted. We can later call new() without finalizer or use the similar O() call to initialize the ref variable, but that can not undo the binding. And using a different finalizer proc for the same data type would not work any more. Passing the same finalizer proc multiple times is OK and may be a common use case, but it has no real effect, as the first call did the binding already.

This behaviour of finalizers in Nim is indeed a bit confusing and error prone. Maybe somewhere in a large program we pass a finalizer proc to new() and forget about it. Later we use new () without a finalizer or use the O() notation to reserve the memory for our ref variable. So we think that no finalizer is involved, but as a finalizer was used somewhere at least once, it is now bound to all of our allocations of that data type. That can easily lead to bugs as the unintended called finalizers may do things that it should not do with our data.

Finalizer procs have to be defined always in the same module as the type for which the finalizer shall be used is defined:

# module tt.nim
type
  O* = ref object of RootRef
    i: int

proc fin*[T](o: T) =
  echo "finalize T"

proc newO*: O =
  new(result, fin)
import tt

type
  OO = ref object of tt.O
    x: float

proc finn[T](o: T) =
  echo "finalize O"

proc main =
  var oo: OO
  new(oo, finn)

main()

We import module tt.nim and subclass the ref object type tt.O. While module tt.nim defines a generic finalizer proc fin(), we can not use that one for our subclased type OO, but have to copy it from module tt.nim into our main module and we may have even to use a different proc name. Otherwise we get the compiler message

Error: type bound operation `fin` can be defined only in the same module with its type (OO:ObjectType)

Whenever we should really need a finalizer or a destructor, we should prefer destructors when we can compile our code with the compiler options --gc:arc or --gc:orc.

Modules

Modules are Nim’s way to divide multiple source codes in clearly separated units and to hide implementation details. Nim use a module concept which is very similar to Modula-2 or Oberon. All the Nim standard libraries are divided into modules which collect and logically group data types and the related procedures. In some way modules are Nim’s classes.

In Nim each module directly corresponds to one text file. Sub-modules as known from Ruby, that divide a single text file in multiple modules are not supported by Nim currently.

Every textfile with Nim source code is basically a module, and that module can be imported and used by other modules. But all symbols like data types or procedures have to be exported to make them visible and usable by other modules. That is done like in Oberon by appending an asterisk character to all names that should be exported. These restricted exports allows to hide implementation details — all symbols not exported are private to that module and can be changed and improved at any time without noticing the importing module.

We can import whole modules, that is all symbols that are marked for export by the asterisk, or we can import only the symbols that we need by specifying their names. Let us create a module that declares a single procedure to remove all characters from a string that are not letters:

# save this textfile with name mystrops.nim
proc remNoneLetters*(s: string): string =
  result = newString(s.len)
  var pos = 0
  for c in s:
    if c in {'a' .. 'z', 'A' .. 'Z'}:
      result[pos] = c
      inc(pos)
  result.setLen(pos)

We save above text file with our Nim source code with name mystrops.nim and can import it like

import mystrops

echo remNoneLetters("3h7.5g:8h")

Module names should be lower case and may as other Nim symbols only contain letters, decimal digits and the underscore character.

Note the export marker after the proc name. Instead of importing whole modules we can import only single symbols with the from x import y, z syntax like

from mystrops import remNoneLetters

echo remNoneLetters("3h7.5g:8h")

Both forms are an unqualified import, that is we can refer to the proc by only its name, we do not need the qualified form with module name prefix like mystrops.remNoneLetters() as long as there are no name conflicts. But whenever we want we can use the qualified form also.

Nim programmer generally prefer importing whole modules and use unqualified names, while that is considered bad style in some other languages like Python. In untyped languages like Python unqualified imports may indeed pollute the name space and generate many name conflicts, but in statically typed languages like Nim unqualified import seems to generate name conflicts only in very rare cases. Procedures with the same name generally have different parameter lists, so the overload resolution of the compiler can decide what proc is to be used. And when really a name conflict occurs then the compiler will tell us, and we can easily fix it by prefixing the proc name with its module name.

For data types, constants or enums chances for name conflicts may be not that tiny, so we may have to use qualified names.

We can also enforce a fully qualified import in Nim by a notation like

from mystrops import nil

In this case we can use all symbols from that module only in qualified form. But that does not always work that well in Nim, as Nim has not classes like Java, so a qualified use of method call syntax or qualified use of user defined or overloaded operators is difficult. Imagine strutils.add(s, '\n'), how should that look with method call syntax?

For imports we have also the except keyword, so we may do something like

import strutils except toUpper

The except keyword can be used to prevent possible name conflicts without having to use qualified names.

Note that the system module is imported automatically, we should not import it directly. Also note that Nim always imports only what is really needed in the final executable, so importing only a few symbols from a module has no code size benefit over importing the whole module. Still it may improve readability of your code when you import only single symbols when you are sure that you need not more. Maybe like from math import Pi. Note that you can even in that case access other symbols of that module by fully qualified names like math.sin().

With the growing standard library it may occur that module names of the standard library interfere with your own module names. So Nim now allows and recommends qualified import of modules from the standard library like import std/strutils. And for external packages installed by the nimble package manager imports in the form import package/[mod1, mod2, mod3] are permitted.

Finally you can also import modules under another name using the as keyword like

import tables as maps

With the recent Nim compiler you can also enforce full qualified import and use of an alternate module name by using an import statement like

from tables as maps import nil

With this import statement you could access symbols from the tables module only by use of the maps module prefix like maps.newTable().

Include

The include statement should be not confused with the import statement. Include just insert a text file at the position where the include statement occurs. Include can be used to split very large modules in smaller entities.

Part III: The Nim Standard Library

In this part of the book we will introduce you to some of the most important modules of the Nim standard library. This includes modules for common operations like the serialization of Nim data types to write them to external nonvolatile storage and read them back into the program later, or handling command line options and parameters for programs launched from within a terminal window. Further we will introduce you to important container data types like hash tables (sometimes called hash maps in other programming languages) and and various kinds of set data types. We will also introduce modules for working with regular expressions, and we will show how simple modules like the times or the random module can be used. Most modules mentioned in this part will be from the Nim standard library, so that you will not have to install external packages for that. But there may be some exceptions, e.g. for some external nimble packages with a very useful functionality and an easy user interface.

Serialization — storing data permanently on external storage

When you start writing larger programs, these programs may create data which you may want to store permanently on external nonvolatile storage like SSDs or traditional hard disks of your computer. For textual data this is very easy, as you basically only has to write and read a stream of unstructured bytes. But when your program deals with object instances, container data types like sequences or references, things become more complicated. Writing the data is always easy — you can just convert all the fields of your object data type to strings and write them to a stream or a file. But the reading back part is much more difficult: You would have to read in the data as strings and then process each string — maybe converting to a float number — and then assign to the matching field of an object instance.

When your data consists only of value objects and no references, then you may consider just writing that data in plain binary form to a file and read it back. This strategy seems to be simple and it is very fast, as no type conversion steps are involved. But at the same time it has some drawbacks: The stored data can not be checked with tools like a text editor, it can generally not be used from other programs, and when you should change the data types used in your program, you could not read back stored files any more.

So we will explain how you can store Nim data types in a human readable text format first. Two popular text formats are JSON and YAML. JSON is a simple format, which is easy to parse, but not very good readable for humans. YAML is more complicated, but more flexible and is very good readable for humans.

For Nim we have already many modules which we can use for storing data in JSON or YAML format available. The Nim standard library includes the marshal and the json module. The marshal module uses like the json module the json data file format, is easy to use and simple, but is not really designed to generate human readable data files, as the stored data is not stored as a sequence of individual lines. So we will describe and use the json module in this section, which is also easy to use, but has some larger set of functionality and can generate real human readable text files by use of the pretty() function.

Other available external packages for data serialization are the nim-serialization module set from (https://github.com/status-im/nim-serialization) and the very powerful but complicated NimYaml implementation (https://nimyaml.org/). We may describe these packages in part V of the book.

When we have to store and read back Nim data to nonvolatile storage media, we have some serious points to consider: First we have to handle various data types like integers, floats, strings, objects — and even the container types like sequences. And we may have to support reference types and maybe also inherited types and containers filled with heterogeneous, subclassed reference objects. The json module supports all Nim data types including containers and references, but not heterogeneous sequences.

For our first json example let us assume that we have written a small tool that let the user create some geometrical shapes, and we want to store the shapes to a file and read it back. For that we generally use an intermediate step, which converts the data to a string and the string back to the data object. The string is then written to a file or stream and read back. Let us start with the string conversion — storing that string and reading it back from the file will be explained soon.

import json

type
  Line = object
    x1, y1, x2, y2: float

  Circ = ref object of RootRef
    x0, y0: float
    radius: float

  Data = object
    lines: seq[Line]
    circs: seq[Circ]

var
  l1 = Line(x1: 0, y1: 0, x2: 5, y2: 10)
  c1 = Circ(x0: 7, y0: 3, radius: 20)
  d1, d2: Data

d1.lines.add(l1)
d1.circs.add(c1)
d1.lines.add(Line(x1: 3, y1: 2, x2: 7, y2: 9))
d1.circs.add(Circ(x0: 9, y0: 7, radius: 2))

let str1 = pretty(%* d1) # convert the content of variable d1 to a string
echo str1 # let us see how the strings looks
d2 = to(parseJson(str1), Data) # read the string back into a data instance
let str2 = pretty(%* d2) # and verify that we got back the original content
echo str2

# assert d1 == d2 would fail
assert str1 == str2

When we run the program we would get this output:

{
  "lines": [
    {
      "x1": 0.0,
      "y1": 0.0,
      "x2": 5.0,
      "y2": 10.0
    },
    {
      "x1": 3.0,
      "y1": 2.0,
      "x2": 7.0,
      "y2": 9.0
    }
  ],
  "circs": [
    {
      "x0": 7.0,
      "y0": 3.0,
      "radius": 20.0
    },
    {
      "x0": 9.0,
      "y0": 7.0,
      "radius": 2.0
    }
  ]
}
{
  "lines": [
    {
      "x1": 0.0,
      "y1": 0.0,
      "x2": 5.0,
      "y2": 10.0
    },
    {
      "x1": 3.0,
      "y1": 2.0,
      "x2": 7.0,
      "y2": 9.0
    }
  ],
  "circs": [
    {
      "x0": 7.0,
      "y0": 3.0,
      "radius": 20.0
    },
    {
      "x0": 9.0,
      "y0": 7.0,
      "radius": 2.0
    }
  ]
}

As you can see we converted the instance d1 of type Data to a string and then we convert that string back to variable d2 with matching content. We have made intentionally the Circ a ref object, so we can see that the conversion works for value and reference objects. In the example program we applied the %* macro to our data instance d1 to get a JsonNode, and finally use the pretty() function to get a nice multi-line string. To fill the variable d2 with the content stored in str1, we first have to apply parseJson() on the string and then use to() to unmarshal the json node into the matching object type.

Now let us investigate what happens when we try to use the json module with a container with heterogeneous ref objects. For that we subclass the Disc type creating a new Arc type:

import json
from math import PI

type
  Line = object
    x1, y1, x2, y2: float

  Circ = ref object of RootRef
    x0, y0: float
    radius: float

  Arc = ref object of Circ
    startAngle, endAngle: float

  Data = object
    lines: seq[Line]
    circs: seq[Circ]

var
  d1, d2: Data

d1.lines.add(Line(x1: 0, y1: 0, x2: 5, y2: 10))
d1.circs.add(Circ(x0: 7, y0: 3, radius: 20))
d1.lines.add(Line(x1: 3, y1: 2, x2: 7, y2: 9))
d1.circs.add(Arc(x0: 9, y0: 7, radius: 2, startAngle: 0, endAngle: PI))

echo d1.circs[1] of Arc, " ", Arc(d1.circs[1]).endAngle

let str1 = pretty(%* d1)
d2 = to(parseJson(str1), Data)
let str2 = pretty(%* d2)
echo str2
echo d2.circs[1] of Arc

The output of that program looks like this:

true 3.141592653589793
{
  "lines": [
    {
      "x1": 0.0,
      "y1": 0.0,
      "x2": 5.0,
      "y2": 10.0
    },
    {
      "x1": 3.0,
      "y1": 2.0,
      "x2": 7.0,
      "y2": 9.0
    }
  ],
  "circs": [
    {
      "x0": 7.0,
      "y0": 3.0,
      "radius": 20.0
    },
    {
      "x0": 9.0,
      "y0": 7.0,
      "radius": 2.0
    }
  ]
}
false

While our initial instance d1 contains a run-time value of Arc type and so we can access the endAngle field, we get false as result for the of Arc test for the d2 instance. So run-time type information is lost.

When we have to store different data types in one container, then one solution is to use object variants, which should work with the json module. Another obvious possibility is to just copy the data into containers with the appropriate static type before storing to an external medium and copy them back when we read the data back from external storage. Will will show an example for that now:

import json
from math import PI

type
  Line = ref object of RootRef
    x1, y1, x2, y2: float

  Circ = ref object of RootRef
    x0, y0: float
    radius: float

  Arc = ref object of Circ
    startAngle, endAngle: float

  Data = object
    elements: seq[RootRef]

  Storage = object
    lines: seq[Line]
    circs: seq[Circ]
    arcs: seq[Arc]

const
  DataFileName = "MyJsonTest.json"

var
  d1, d2: Data
  storage1, storage2: Storage
  outFile, inFile: File

d1.elements.add(Line(x1: 0, y1: 0, x2: 5, y2: 10))
d1.elements.add(Circ(x0: 7, y0: 3, radius: 20))
d1.elements.add(Line(x1: 3, y1: 2, x2: 7, y2: 9))
d1.elements.add(Arc(x0: 9, y0: 7, radius: 2, startAngle: 0, endAngle: PI))

for el in d1.elements:
  if el of Arc:
    storage1.arcs.add(Arc(el))
  elif el of Circ:
    storage1.circs.add(Circ(el))
  elif el of Line:
    storage1.lines.add(Line(el))
  else:
    assert(false)

let str1 = pretty(%* storage1)

if not open(outFile, DataFilename, fmWrite):
  echo "Could not open file for storing data"
  quit()
outFile.write(str1)
outFile.close

if not open(inFile, DataFilename, fmRead):
  echo "Could not open file for recovering data"
  quit()
let str2 = inFile.readAll()
inFile.close

assert str1 == str2

storage2 = to(parseJson(str2), Storage)

for el in storage2.lines:
  d2.elements.add(el)
for el in storage2.circs:
  d2.elements.add(el)
for el in storage2.arcs:
  d2.elements.add(el)

for el in d2.elements:
  if el of Arc:
    echo "found arc with endAngle: ", Arc(el).endAngle

For this example program we use OOP programming style and keep all the geometric object instances as references in a single sequence. Note that doing this is not always a good idea, as this OOP style with the use of references and dynamic run-time dispatch can be slower due to many small heap allocations for each ref object and due to the dynamic dispatch (if el of …​) overhead. Using multiple, homogeneous sequences with value types for each of our data types can be a better solution, and in that way you have more control whenever you process the data, for drawing them on the screen or user interaction for example. Maybe you want to draw all the lines first? But there can be situations where we really need to have all the objects as references in a single container. A typical situation is, that we use an RTree for fast object location. RTrees are data structures, that can store two dimensional or multidimensional geometric object and their rectangular bounding boxes in a tree-like fashion for fast object location. This may be used in a drawing program, so that coordinates of a user mouse click can be fast matched to an object. For such a use case we would really prefer to have all the object instances available in one single RTree, and not use one RTree data structure for each object shape.

Our program defines an additional Storage data type, which contains homogeneous sequences for each possible geometric shape. We then copy all our ref objects from the elements sequence in the matching sequences of the storage object using the dynamic of type query to select the exactly matching sequence.

After that we can use the already known json functions to serialize the storage object into a string, store the string to a file, read it back and deserialize the data again into a different variable of Storage data type. Finally we use a simple for loop to copy the ref objects from the temporary storage object into a Data variable called d2. For storing the data to a external nonvolatile medium we use the File data type and the related functions open(), close, write() and read(). Their use should be obvious: We pass a uninitialized variable of File data type, a file name and a file mode to open(), use write() to write the whole string, and use readAll() to read the data back. When done with each file we use close() to close the file. The File data type is part of the io module, which is again part of module system, so we don’t have to import these modules. We could have used as an alternative also the streams module. We will learn some more details about the File data type and the streams module in later sections of the book.

We should mention that unfortunately live is not always that easy, as sometimes we can not freely select the textual output format. Imagine that you create a CAD (computer aided design) tool that should be compatible with another existing tool. In this case the textual storage format is already defined by the existing tool, and generally that format does not match the json or yaml file format. Even when the format should be one of these, matching it exactly would be difficult. While writing out own data in that foreign format is still not really difficult, as we can just write single matching strings, reading in the textual data is more complicated: Generally we would read the input file line by line and we would have to inspect and interpret each input string, maybe by use of regular expressions or a custom parser. That generally includes handling of missing or invalid data.

References:

Streams and Files

In the previous section we learned how we can store structured data like a sequence of objects in human readable form to nonvolatile media by use of the json module.

Text in form of a single string or in form of a container holding multiple strings is some kind of unstructured data which we can write directly to nonvolatile storage, and later read it back. We can do the same with containers of basic, unstructured data types like integer or floating point numbers, and with some restrictions we can even write tuples or objects directly as raw bits and bytes to external storage and read it back later. Of course this way the stored data is a binary blob, which can not be read or modified by other tools like a text editor. But that may be not intended or advantageous at all, maybe we do scientific data processing with a single tool, and we just want to temporary store the data and continue with the processing later.

Files

For storing unstructured data Nim provides the io module with the File data type and related procs, and the streams module with the Stream data type and related procs. While a File in Nim is currently only a pointer to a C file, the streams module has a higher abstraction level. Although the Nim language does not directly support interfaces, the Stream data type of the streams module is some form of an interface, which is implemented by a StringStream and a FileStream data type. Internally this interface concept is realized by storing a set of function pointers in the Stream instance.

When we have to store unstructured data like text it is not always clear if we better should use Files or Streams. Streams may be the better choice when we (also) want to use a string as data source like a file or when we need the peek() functions of the streams module to access data without advancing the position in the stream.

We will use the File data type of the io module first. As the io module is part of the system module, we do not have to import it before we can use it. The principle usage of files is, that we call the function open() to open a file with given name, call some procs to write or read data, and finally close() the file. While Nim support destructors when we compile with --gc:arc or --gc:orc, the io module does not yet use them, so we should actually call close() to close the file.

Historically a file is a one dimensional data type which is accessed in sequential order. Up to the end of the twenty century it was not uncommon that large files where stored on magnetic tapes, which could be read or written only slowly in sequential order. Read or write operations could take place only at the actual position, and available functions like f.setFilePos() where very slow as it involves moving the tape. The introduction of hard disks and solid state disks removed this restriction, and modern operating system often buffers files in RAM for longer time periods, so that files may have actually similar performance as arrays or sequences. The funny fact is, that with the modern CPU caches the ordinary RAM storage can look similar slow and sequential compared to the extreme fast cache as magnetic tapes in the past.

from os import fileExists
proc main =
  const FN = "NoImportantData"
  if os.fileExists(FN):
    echo "File exists, we may overwrite important data"
    quit()
  var f: File = open(FN, fmWrite)
  f.write("Hello ")
  f.writeLine("World!")
  f.writeLine(3.1415)
  f.close
main()

Running that program will create a text file with this content in the current working directory:

Hello World!
3.1415

At the start of our function we check if a file with that name already exists in the current working directory by using the function os.fileExists() to ensure that we do not overwrite important data.

Module io provides multiple overloaded open() procs. We use here a variant which returns a file, and raises an exception for the unlikely case of an error. We provide a file name and a file mode as parameters. We use mode fmWrite as we want to create a new file. Note that fmWrite would clear the content of an existing file, so we can not use fmWrite to append data to an existing file. We would have to use fmReadWriteExisting or fmAppend to append data to an already existing file. As this open() proc can raise an exception, it may make sense to enclose it in a try/except block, or we could use a open() variant which returns a boolean value to indicate success instead. When the file is successfully opened, we can use procs like write() or writeLine() to write text strings to the file. Both procs accept multiple arguments and apply the stringify operator $ on them before writing the content. WriteLine() writes a '\n' after the last argument to start a new line. When done we call close() to close the file. The operating system would close the file for us when our program terminates, so calling close is not that important, but when we open many files without closing them we may get errors from the operating system finally about too many open files and our program may fail or terminate.

The close() proc gets passed the file not as a var parameter, so it can not set the file to value nil. When the file has the value nil, then the close() call is ignored, but when we would call close() multiple times with a non nil argument we get a program crash. We may use the try/finally or the defer construct to ensure that we really close the file when done.

The io module provides some procs like writeBuffer(), writeBytes() or writeChars() which gives us as return value the actual number of bytes written. This return value should generally match the requested number of bytes to write, but can be smaller when the write operation fully or partially failed, e.g. because the storage medium had no capacity left.

When performance really matters, we should note that passing non-string arguments to write() or writLine() procs using their optional auto-stringify for us, involves allocation of new strings and cost some performance. When we have in our program already a string variable available, it can be faster to convert our data into that variable first and then pass that variable to the write() or writeLine() procs.

Reading strings from a file works very similar:

proc main =
  var f: File
  try:
    f = open("NoImportantData", fmRead)
    echo f.readLine
    echo f.readLine
  finally:
    if f != nil: # test for nil not really necessary, close() would ignore the call for f == nil
      f.close
main()

The readLine() proc reads in a line of text. The LF, CR or CRLF line end markers are not part of the returned text string. Of course we may get an empty string with length zero back when we read a line which immediately starts with LF, CR or CRLF, or we may get back a string with no visible characters but only a few spaces or tabulator characters '\t' when a line contains only white space. When our read() operations have moved the actual file io position to the end of the file, and we try to read more content, then an exception is raised.

The io module provides a readLine() proc that returns a newly allocated string, and one that takes an existing string as var parameter. The later may be a bit faster, as it can avoid the allocation of a new buffer when the passed string has already enough capacity.

The io module provides a function called endOfFile() with a boolean result which we can use to check if the end of file position is already reached. The provided functions readBuffer(), readBytes() or readChars() return the actual number of bytes read, which can be smaller than the requested value when the end of the file is reached earlier. Currently readChars() checks if the passed openArray[char] has enough capacity for the request, but readBytes() does no check!

We can use also the lines () iterator to iterate over the lines of a text file, or use the readLines() proc to read the content line by line.

proc main =
  var f: File
  f = open("NoImportantData", fmRead)
  for str in f.lines: # iterator
    echo str
  f.setFilePos(0) # read again from start index 0
  var s: string
  while f.readLine(s): # proc
    echo s
  f.close
  var sq = readLines("NoImportantData", 2) # read lines to seq of strings
  echo sq
main()

As iterating over the whole file line by line moves the actual file position to the end of the file we called setFilePos() to move again to the start position. The readLines() proc takes a filename and the number of lines to read as parameters and returns a seq of strings. When the file does not contain at least the number of requested lines an EOF exception is raised. Another provided proc is readAll() which reads the whole file content into a returned string variable. For readAll() to work the actual file position has to be the the start of the file. In case of an error an exception is raised.

We can also write and read binary data directly to a file, without converting it to (human readable) strings first:

proc main =
  var f: File
  f = open("NoImportantData", fmWrite)
  var i: int = 123
  var x: float = 3.1415
  assert f.writeBuffer(addr(x), sizeof(x)) == sizeof(x)
  assert f.writeBuffer(addr(i), sizeof(i)) == sizeof(i)
  f.close
  f = open("NoImportantData", fmRead)
  assert f.readBuffer(addr(x), sizeof(x)) == sizeof(x)
  assert f.readBuffer(addr(i), sizeof(i)) == sizeof(i)
  f.close
  echo i, " ", x
main()

Of course these are low level, dangerous operations. While writeBuffer() should never crash our program, readBuffer() can do that easily when we specify wrong sizes or destination addresses, as that may overwrite other data unintentionally. So we would generally not use these procs directly but write more safe helper procs, when we really need or want this form of binary file access. Fast storing big data sets with restricted hardware may be an use case, e.g. storing a float32 takes only 4 bytes on the storage medium and file io is fast, while that number as human readable digits may need more than 8 bytes (1.234567E3) and converting to string and and parsing back costs some time.

In the same way we can use writeBuffer() and readBuffer() to store tuples, objects and arrays or sequences of these directly in binary form:

type
  O = object
    x: float
    i: int
    b: bool

proc main =
  var s: seq[O]
  s.add(O(x: 3.1415, i: 12, b: true))
  var f: File
  f = open("NoImportantData", fmWrite)
  assert f.writeBuffer(addr(s[0]), sizeof(O) * s.len) == sizeof(O) * s.len
  f.close
  f = open("NoImportantData", fmRead)
  var s2 = newSeq[O](1)
  assert f.readBuffer(addr(s2[0]), sizeof(O) * s2.len) == sizeof(O) * s2.len
  f.close
  echo s2[0]
main()

Output should look like

(x: 3.1415, i: 12, b: true)

But of course this is dangerous and fragile. We just show that example as beginner generally ask about it, and may want to try it at least once. Obviously this can only work when the tuples or objects contain only plain data types, that is no string, no references and of course no other nested container types like sequences or tables. And reading back data may fail when we use a different OS or a different compiler version.

The io module provides the File variables stdin, stdout and stderr, which are the standard input, output and error streams. Sometimes we use stdout.write() instead of the common echo() proc when we want to write something to the terminal window without moving the cursor to the next line already.

An important function of the io module is flushFile(), which is used to ensure that all buffer content of buffered files is actually written to the file. This is important when we use the stdout File variable, maybe to ask the user a question in the terminal window. We would call sdtout.flushFile() to ensure that the user really sees the text on the screen immediately. The echo() proc calls flushFile() automatically after each output operation. When we close a file flushFile() should be called automatically, but when our program is terminated without calling close() it may depend on the actual implementation and operating system

The io module provides some more useful procedures, but we will stop this introducing section here and continue with the streams module in the next section.

References:

Streams

A stream is an abstract interface for performing certain I/O operations, which was introduced by languages like C or Modula-2 decades ago. The streams module of the Nim standard library provides a FileStream and a StringStream implementation, which behaves very similar. Nim’s streams module provides similar functions as the io module with its File data type, but it can operate on strings instead of on Files, and it provides a set of peek() functions to access data at the current read position without moving forward. And some functions are more robust, for example closing a stream multiple times does not crash the program, as the first close() call sets the file variable of file streams to nil, so that following close() calls are ignored. Currently the streams module does not support automatically closing of streams when they go out of scope.

We can create a new FileStream by calling the overloaded procs newFileStream() with an already opened file or a filename as parameter, or we can use openFileStream(). The later raises an exception when the stream can not be opened, while the former procs just return nil. We can write and read textual data with the streams module in a very similar way as we did it with the io module and the File data type:

from os import fileExists
import streams

proc main =
  const SN = "NoImportantData" # stream name
  if os.fileExists(SN):
    echo "File exists, we may overwrite important data"
    quit()
  var fstream = newFileStream(SN, fmReadWrite)
  if fstream != nil:
    fstream.write(123, ' ')
    fstream.writeLine(3.1415)
    fstream.setPosition(0)
    let l = fstream.readLine()
    fstream.close()
    assert l == "123 3.1415"
main()

We test again if a file with that name already exists. Then we try to create a new FileStream by using file mode fmReadWrite, so that we can write and read from that file. Finally we write two numbers, which are automatically converted to strings, set the file position back to the beginning and verify what we wrote by reading it in again, before we close the stream.

In a very similar way we can write to and read from string streams

import streams
proc main =
  var stream = newStringStream()
  stream.write(123, ' ')
  stream.writeLine(3.1415)
  stream.setPosition(0)
  let l = stream.readLine()
  stream.close()
  assert l == "123 3.1415"
main()

In the example above we do not test if the stream variable is not nil, as newStringStream() should never fail.

For buffered streams we can call flush() to ensure that the buffer content (of file streams) is written, similar as we can do it for plain Files of the io module. Instead of io.endofFile() we use the proc atEnd() to test if the current stream position is already at the end of the stream. Functions getPosition() and setPosition are available to query or set the actual position in the stream. While the io module with its File data type supports for io.setFilePos() although position modes relative to the actual position or relative to the file end, streams.setPosition() use always absolute values, that is positions measured from the beginning of the stream. The streams module provides also the low level procs readData() and readDataStr() which reads data to a memory region or into a string and returns the actual number of bytes read to indicate success. And as for the io module a proc readAll() is available to read all data of a stream into the returned string variable.

The procs writeLine() writes the passed arguments always as strings. The overloaded write() procs with varargs arguments write the passed values as strings and apply the stringify operator $ if necessary. The same does the writeLine() proc, but it writes a newline character when all passed variables have been written. One more overloaded write proc for single string parameters exist.

But for single non string arguments a generic write() proc is used, which writes numbers (and other data types like boolean types or single characters) directly in binary form without converting them to strings.

To read the binary numbers back we can use functions like readFloat64() which have a well defined return type and read a fixed number of bytes. Or we can use the generic read() proc which accepts a var parameter which defines the data type that we intend to read in binary form. Additional to the various read() procs the streams module provides a set of peek() procs which reads data in without moving the actual position in the stream forward. This may be useful for parsing of files, as we can read the same information multiple times easily. Internally the peek() functions uses a call of setPosition() to save the current position and one more call of setPosition() to set back the old position to the initial value, so peek() has some overhead.

import streams
from os import getFileSize
proc main =
  const SN = "NoImportantData" # stream name
  var fstream = newFileStream(SN, fmReadWrite)
  if fstream != nil:
    fstream.write("012") # write a 3 byte string
    var pi: float64 = 3.1415
    fstream.write(pi) # write as 8 byte binary blob
    fstream.setPosition(0) # prepare for reading from start
    var i16: int16
    i16 = fstream.peekint16 # read first 2 bytes as int16, do not change actual position
    assert i16 == '0'.ord + '1'.ord * 256
    var i8: int8
    i8 = fstream.readInt8 # read back one byte
    # fstream.read(i8) # does work also
    assert i8 == '0'.ord # char(0) has position 48 in ASCII table
    assert i8 == 48
    var buffer: array[2, char]
    fstream.read(buffer)
    let x = fstream.readFloat64 # read back in binary form
    assert x == 3.1415
    fstream.close()
    assert buffer == ['1', '2']
    assert os.getFileSize(SN) == 3 + 8 # 3 byte string and a float64
main()

In the example above we write a three byte long string and a float64 to the file stream. We call setPosition(0) to read the stream from the beginning again,and then read in a int16 with the function peekint16() without moving the actual position forward, followed from readInt8(), which moves the actual position one byte forward. (Instead of readInt8() we could also call read() with variable i8 as passed var parameter.) Then we read in two bytes and finally the float64 value at the end of the stream. Finally we check by use of the function getFileSize() from the os module if the file has really the expected size.

The streams module provides many functions, and the possible writing data as strings or in binary form can make using that module a bit daunting at first. But most procs have examples in the API docs which helps you using it.

For reading strings and whole lines the streams module provides functions like readLine(), peekLine(), readStr() and peekStr() each in a variant which returns a newly allocated string and one that uses a passed var parameter to return the string. The variants with var parameters may be a bit faster, as they can avoid allocating a new string when the passed in var parameter has already enough capacity.

References:

String Processing

String processing is a wide area. Nim’s standard library provides various modules like strutils, parseutils and strscans for supporting this task, and external packages supports more advanced operation like string pattern matching with regular expressions (regex). We will start with the strutils module, which is one of the mostly used modules of the Nim standard library. Then we will introduce some more specialized modules like strscans, parsecvs, parseutils, strformat, and finally we will give an introduction to use of regular expressions (regex) and to the use of the parsing expression grammar (PEG). While we describe generally only modules of Nim’s standard library in this part III of the book, we will make an exception for regex and PEG. The reason for this is, that regex and PEG use is closely related to basic string processing and that very powerful external packages exists, which are not too difficult to use and so needs not too much introductory explanations.

Whenever we do string processing in Nim we should care a bit for performance, as some string operations can be slow by design. For simple tasks we should prefer to use functions from the simple modules like strutils when possible, and use regex or PEG only when really necessary or when performance is uncritical. And even when we use elementary simple functions like a string split, it is generally good to have a feeling how the requested operations may work. Whenever string functions return a string as a result, this implies an allocation, which takes some time and consumes some memory. An example is the split() operation which return a sequence of multiple strings. The split() function is easy to use, so it is often the first choice when we read in lines of text from files and want to process it. But as for each section of the split line a string is allocated, it may be not as fast as desired. In some cases the compiler may be able to optimize the slitting process, but it may be also a good idea to think about other ways to extract the data, maybe by applying procs from the strscans module which can parse lines directly into passed var parameters avoiding unnecessary allocations.

Remember that Nim strings are value types and have value semantic. String assignment copies the string content and does not create just a reference as in some other programming languages. Nim defines also a string variant called TaintedString, which is generally just an alias for an ordinary string as long as the taint mode is not turned on. Functions like io.readLine() return tainted strings which generally can be used like ordinary strings.

Basic string operations

We discussed Nim’s string data type already in part II of the book. Remember that a string in Nim is a variable size container for ASCII characters. Strings can be plain ASCII strings, or the bytes of the string can be interpreted as unicode glyphs. Nim has also a cstring data type, which was initially introduced to be compatible with the character arrays used in C libraries as strings, but is now called compatible string as cstrings do also support the Javascript backend. Nim strings can be passed directly to C libs, as a Nim string contains a Null terminated buffer for the actual data, which is identical to a C language string. So whenever we convert a Nim string to a C language string or pass a Nim string to a C library, this is free of costs, while converting a C string to a Nim string always means allocating a new Nim string and coping the data content. Technically for a Nim string s addr s[0] is the C string pointer, called * char in C language. Whenever we pass strings to C libraries we have to care for the fact that Nim’s garbage collector may deallocate the string automatically. Most C libs create copies of passed strings when the libs use the string for a longer time span. GTK for example does this with text for its widgets. But when the C lib does not copy the string but use it directly for a longer time, then it can occur that the Nim code frees the string, as the only one Nim variable referring to the string goes out of scope, but the C library still uses the string. For that rare case we may call GC_ref() on the string to prevent garbage collection, but that may generate memory leaks then. For the case that C libs create strings, they provide generally also a function to deallocate the string. When we use such a C function, it is generally the best solution that we copy the string from the C lib to a Nim string and immediately deallocate the C string by a call of the provided free/dealloc() function. For most C libs there exist good high level bindings which do not have this issues, so we generally can use the C libs like pure Nim libs.

Nim' s system module provides already some basic string operation like accessing single characters by the subscript operator [], accessing slices of multiple adjacent characters, or joining multiple strings with the & operator. The overloaded add() functions to append single characters or other strings to existing string variables are also provided by the system module.

var s: string = "I like"
s &= " Nim."
s[^1] = '!'
s[0 .. 4] = "We low" # result is: "We love Nim!"

We start above example by assigning to the string variable s a string literal, then we append one more string literal, and finally replace the last character and the first five characters by another character and by another string. Note that by using the slice operator we can not only replace character ranges, but we can also replace slices of different length. This way we can also delete ranges in the string by replacing it with the empty string "".

The system module also defines the stringify operator $, which converts expressions to the string presentation when we put it in from of it. Procs like echo() apply the stringify operator automatically on all of its arguments when necessary. And the system module provides the contains() function which we can use to test if a string contains a character. Instead of contains we can also use the in operator.

echo $7
echo "Nim".contains('i')
echo 'i' in "Nim"

The system module also provides the procs newString() and newStringOfCap() which are mostly used for optimizing purposes. The function newString(n) creates a string of length n, but with uninitialized content. We would have to assign characters to the positions 0 .. n-1 to create a valid string. Function newStringOfCap() creates a string with length zero, but with a buffer capacity of n characters. When we know the needed buffer capacity, or at least a lower bound of it, it makes sense to create the string with newStringOfCap() with optimal buffer size to avoid reallocations. Of course we could still append more data, Nim would allocate a larger buffer and copy content.

var s1 = newString('z'.ord - 'a'.ord + 1)
for c in items('a' .. 'z'):
  s1[c.ord - 'a'.ord] = c

var s2 = newStringOfCap(32) # we intend to append not more than 32 characters, but we could do.
for c in 'a' .. 'z':
  s2.add(c)
# s1, s2 is abcdefghijklmnopqrstuvwxyz

Filling in characters into an existing string by use of the subscript operator [] is faster than appending single characters with the add() function, because the add() function has to check if the string has still enough capacity and because add() has to increase the actual string length by one for each call.

Note that a single character like 'a' is very different from a string with only one character like "a". A character in Nim is nothing more than a single byte, while a string — even one with only one character or an empty one — is an opaque entity with length, capacity and a pointer to a data buffer. When a single character is sufficient, we should use that and not a string containing a single character. A function call like s.add("a") may produce less optimized code than s.add('a'), but maybe the compiler optimized the former for us. When we consider optimization we may wonder if in

var s = "Hello!"
echo s
s = "Bye."
echo s

line 3 allocates a new string or just copies the string literal "Bye." in the existing data area. Well we would hope for the later of course.

Another interesting question is if in

var s = "Result is: "
var x = 12.34
echo s, x
echo s & $x

we should better use line 3 or line 4. Line 3 looks more clear and we assume that it also would produce better code, as the actual append operation is avoided.

Often used functions are len() and setLen() to query and set the length of a string. While len() looks like a function call, it is a compiler intern function, so calls are fully optimized. So it is OK to write

for i in 0 .. s.len():
  echo '*'

It is not necessary to introduce a temporary variable like let l = s.len() to avoid many function calls. In C this is different, as in C a call of function strLen() would not only imply a function call, but that function would really have to count all characters up to the terminating '\x0' as C strings have no length field. The function setLen() is generally used to truncate strings. A call like s.setLen(0) makes s look like a newly allocated string, but its data buffer can be reused. Reusing strings is generally better for performance than allocating many new strings. The setLen() function is rarely used to increase the string length — in that case an allocation of a larger data buffer can occur, and the string would still look the same as initially all the new string positions would still contain the default binary zero content. We would have to fill in actual characters by use of the [] subscript operator. To get the lowest and highest character index of a string we can use s.low and s.high. s.low should be always return zero, and high() is identical to len() - 1, so high() is -1 for an empty string. Note that while calling high() and len() on a Nim string has no costs, this may be different to C strings as these have no length field.

The system module provides the overloaded & operator which we can use to concatenate chars and strings, the &= operator to append a new string to an existing string, and the add() functions to append characters or strings to an existing string. For best performance we should try to use always the most simple, "native" operations, at least as that does not make the code ugly or we know for sure that the compiler optimizes it for us.

var s = "Ni"
s = s & "m" # maybe not a good idea for optimal performance
s.add('m') # better

The function add() can be also used to add a cstring to a string, and the JS backend allows even to append one cstring to another one by add() calls.

Module system also exports a substr() proc which copies and return a slice of a string. Overloads with optional first index with default 0 and optional last index with default s.high exists.

var s = "Hello!"
assert s.substr(2, 3) == s[2 .. 3]

s = "ABC"
echo s[0 .. 5] # fail
echo s.substr(0, 5) # index is clipped to s.high
s[0 .. 5] = "" # fail

And of course == and != operators for string comparison are provided. To test if a string is empty we can compare with an empty string literal or test if len() is zero. The later is guaranteed to have best performance, but the former is a bit shorter and should be not bad performance wise. Some other languages provide an empty() test function for this, we may define our own when we really want.

if s != "":
if s.len != 0:

When we pass a string to a proc that does no operations with it, maybe it only calls echo() or stdout.write() to print it, then it may have a tiny performance advantage to pass it as cstring. This is similar as we may pass sequences as openArray to functions, which also avoids one level of indirection. Also note, that while a Nim string is a value type, so we can not test it for nil or return nil from a proc that shall return a string, this restriction does not hold for cstrings.

Module stringutils

Module stringutils provides a set of functions (120 currently) and a few iterators for simple string operations. Using that functions and iterators is simple in most cases and generally well explained in the API docs. Remembering which functions exists, their exact name and the function arguments can be a bit difficult at first. We will introduce in this section some of the most used or more difficult functions, and give some warnings when the actual performance may be not as good as expected.

Performance critical operations are generally that one which has to allocate new strings or that has to shift many characters, like text inserting operations. Note that some functions of this module like toUpperAscii() work only with the lower and upper ASCII letters. For unicode operations we may need the unicode module.

In this section we use the term whitespace, which refers to the invisible characters {' ', '\t', '\v', '\c', '\n', '\f'} (space, tab, vertical tab, carriage return, new line, form feed). Note that we have two possible newline characters {'\c', '\n'} that starts a new line, and that older windows text files may still use the two character string "\c\n" to start a new line. The character set {'A'..'Z', 'a'..'z'} is called (ASCII) letters, the set {'0'..'9'} (decimal) digits and the set {'0'..'9', 'A'..'F', 'a'..'f'} hexdigits used to represent hexadecimal numbers.

The stringutils module support string interpolation by use of the % operator.

echo "The $1 programming language is $2." % ["best", "Nim"]
echo "The $# programming languages are $#." % ["most difficult", "C++, Rust and Haskell"]
echo "A $adj programming language is $lang." % ["adj", "low level", "lang", "assembly"]
echo "Let's learn $#" % "Nim!"
echo format("I know $# programming languages.", 2)

We can use $1 up to $9 to mark positions where string n from the array should be inserted, or just $# to insert the strings in the order as they appear in the array. We can also use named insert markers and specify name-value string pairs in the array. For a single string we can omit the array and pass just as string, and finally we can use the format proc to enable stringify for the parameters.

We mentioned already the useful but performance critical set of split() functions:

import strutils
let str = "Zero, first, second, third"
var s: seq[string] = str.split(", ")
echo s
echo str.split(", ", 0)
echo str.split(", ", 1)
echo str.split(", ", 3)
echo str.split(", ", 4)
@["Zero", "first", "second", "third"]
@["Zero, first, second, third"]
@["Zero", "first, second, third"]
@["Zero", "first", "second", "third"]
@["Zero", "first", "second", "third"]

We used the split() variant which accepts a string as split marker. This function accepts one optional parameter for the number of splits to execute and returns a sequence containing the single strings. The split marker string, also called separator, is removed from the strings. The default value for the number of splits is -1 indicating that we want a split at each separator position. If we specify a positive number n, then only n splits are executed and the last element of the returned sequence will contain the remainder of the string. When we specify for the intended splits a value which is larger than the number of contained split markers, then we get a full split.

The reason why this function is not very fast is that it has to allocate a sequence for the return value and a new string for each split and one more for the last string or the remainder. For the case that we do need only the first few strings of the split, it is a good idea to specify the number of actual splits to increase performance.

The strutils module provides overloaded functions which use single characters as separators or which accepts a set of characters as separators, so we may split at space, tabulator, comma or semicolon with {' ', '\t', ',', ';'}. And a function splitWhitespace() is available to split at whitespace like spaces and tabulators, which removes all the whitespace between the strings. Notice that the split() function for strings or single characters does one split for each separator, so we can get empty strings as result as in

import strutils
let str = "Zero___first_second__third"
let str2 = "Zero first\tsecond           third"
echo str.split('_')
echo str2.splitWhiteSpace()
# @["Zero", "", "", "first", "second", "", "third"]
# @["Zero", "first", "second", "third"]

An interesting behavior of splitWhiteSpace() is that whitespace at the start or end of a string is just ignored, while the split() function returns additional empty strings when the string to split starts or ends with the separator:

import strutils
let str = "_Zero_first_second_third_"
let str2 = "   Zero first\tsecond           third   "
echo str.split('_')
echo str2.splitWhiteSpace()
# @["", "Zero", "first", "second", "third", ""]
# @["Zero", "first", "second", "third"]

Another function is splitLines() which splits a string at CR, LF or CR-LF characters.

For these splitting functions also iterator variants exists, which behave like the functions with same name. When we limit for the iterators the number of splits to perform, we may get as last returned value the remained string. You may consult the API docs for details if necessary.

Functions with names rsplit() are also available which behave like split but start the splitting process from the end of the string, so we can get the file extension of a file name with something like filename.rsplit(1)[^1].

The functions removePrefix() and removeSuffix() can sometimes help to avoid expensive split operations. There is an overloaded function that removes all single characters, all characters from a set or a single string:

import strutils
var s = "NNimmm Language"
s.removePrefix('N'); echo s # immm Language
s.removePrefix({'N', 'i', 'm', ' '}); echo s # Language
s.removePrefix("Langu"); echo s # age
s. removeSuffix("ge"); echo s # a

Other useful functions are startsWith() and endsWith() which accept a single character or a string and return a boolean value:

import strutils
var s = "Nim Language"
echo s.startsWith('N') # true
echo s.startsWith("Nim") # true
echo s.endsWith('e') # true
echo s.endsWith("Programming") # false

An efficient function similar to find() is continuesWith() use like "Nim Language".continuesWith("Lang", 4) which tests is a string contains a substring at position n.

We can use the join() proc to join strings in arrays or sequences to single strings with an optional glue string. Join() works also when the elements are not already strings — in that case the stringify operator is applied first:

import strutils
var a = ["Nim", "Rust", "Julia"]
var s = a.join(", ")
echo s
echo s.split(", ").join("; ")
echo [1, 2, 3].join("; ")
# Nim, Rust, Julia
# Nim; Rust; Julia
# 1; 2; 3

The overloaded find() functions accepts optional start and end positions and returns the index position of the first match or -1 when the search gave no result:

import strutils
var s = "Nim Language"
echo s.find('L') # 4
echo s.find({' ', 'a'}) # 3
echo s.find("age") # 9
echo s.find("Rust") # -1

The contains() function can be used to test if a character, a set of characters or a substring in contained in a string. Instead of conatains(s, sub) we can write sub in s. Note that the a function variant for single characters is defined in the system module.

The replace() function can be used to replace all occurrences of a character or a substring in a string and to return the new string. We can use replace() with an empty replacement to delete a substring. The also available delete() function is used to delete a range specified by two indices in place.

Additional a replaceWord() function exists, which does only replaces whole words, i.e. words that are surrounded by word boundary characters like spaces, tabulators or newlines.

The function multiReplace() is a variant that can replace multiple substring/replacement pairs passed as tuples in one pass.

The function strip() can be used to remove multiple characters from a set at the start and the end of a string. The default character set is whitespace, and per default strip removes characters at both ends of the string. The function stripLineEnd() removes a single line end marker as \r, \n, \r\n, \f, \v from the end of the string, but only once.

Sometimes useful is the boolean function isEmptyOrWhitespace() which checks if a string is empty or contains only whitespace. Also useful can be the function repeat() which returns a string that contains the passed character or the passed string n times, and the function spaces() which returns a string containing only n spaces.

For single character tests we have functions like isDigit(), isUpperAscii, isLowerAscii, isSpaceAscii, isAlphaNumeric. Function isDigit() test for characters '0'..'9', isUpperAscii() for 'A'..'Z', islowerAscii() for 'a'..'z', isSpaceAsccii() for ASCII witetespace (' ', '\t') and is AlphaNumeric() test for lower or upper case ASCII letter or a decimal digit.

Function toLowerAscii() convert all the characters 'A'..'Z' to lower case, and toUpperAscii() converts all the characters 'a'..'z' to upper case. The function argument can be a string or just a single character. With capitalizeAscii() we can convert the first ASCII character of a string to upper case.

The overloaded functions count() can be used to count the characters or substrings in a string, and countLines() is available to count the number of lines, where lines are separated by CR, LF or CR-LF.

Sometimes we may also need functions like formatFloat(), formatBiggestFloat() or formatEng() to format float numbers for output purposes. You would have to consult the strutils API docs for all the format details. An intToStr() function with an argument to specify the minimal string length is also available. The string may get leading zeros for alignment.

Finally some important functions are the parsing functions like parseFloat() or parseInt() which converts strings to float or integer numbers. Both raises an exception when the string does not contain a valid number.

The strutils module contains some more not that often used function, like functions to convert data to hexadecimal, octal or binary representation, or to parse numbers back from that string representation into numbers. Other functions like align(), center() and indent() are available for string positioning. We will not try to describe these seldom used functions here, as it is hard to remember the detailed behavior. You should skim the API docs and consult them when you need one of the exotic functions or when you have forgotten how to use a concrete function.[25]

Module parseutils

The module parseutils provides a set of functions for efficient and fast parsing of strings. The functions avoids the allocation of new strings by passing back results in var string parameters and by returning the number of processed characters. The module parseutils is a good choice when we need efficient parsing of strings and the input strings have a simple structure. For more complicated input data we may have to use RegEx or PEGs. Let us assume that we have a set of library names which includes the version numbers, but we need the plain names. The function parseWhile() is a good candidate for this task:

import parseutils
var libs = ["libdconf.so.1.0.0", "libnice.so.10.11.0", "libwebkit2gtk-5.0.so.0.0.0"]
var l = newStringOfCap(128)
for s in libs:
  echo s.parseUntil(l, {'.', '-'})
  echo l
  echo s.parseWhile(l, {'a'..'z'})
  echo l

First we allocate a string with enough capacity, so that the parse() functions can use it without having to do allocations. As we want to receive the plain names, using parseWhile() with a charset as last parameter may be a possible solution. But as we see this will not really work for webkit2gtk which contains a digit in its name:

8
libdconf
8
libdconf
7
libnice
7
libnice
13
libwebkit2gtk
9
libwebkit

We can fix this by passing the extended char set {'a'..'z', '0'..'9'} to parseWhile() or by use of parseUntil() with a character set that does not belong to a name. Both functions return the number of processed characters and provide the captured string in the passed var parameter. Not the we can use the slice operator .. to specify character ranges for the charset parameter when the characters build a continues sequence in the ASCII table.

A related function is skipUntil(), which we may use when we are more interested in the version numbers after the name:

  let p = s.skipUntil({'.', '-'})
  echo s[p .. ^1]

All these functions accept an optional start parameter as last argument. A common use case is to use an integer position variable initialized with zero, which we increase by the returned value so that the parsing can continue at the current position in the string. The next example will use this strategy. For parseUntil() overloaded functions are available which gets not a char set but a single character or a substring as parameter. These functions stops parsing when the character or the substring is found and return that position.

Functions like parseInt() and parseFloat() can be used to extract numbers from strings:

import parseutils
var s = "In the year 2020 I gain 2.5 kg more fat."
var year: int
var value: float
var p = s.skipUntil({'0'..'9'})
p += parseUtils.parseInt(s, year, p)
p += s.skipUntil({'0'..'9'}, p)
p += parseUtils.parseFloat(s, value, p)
echo year, ": ", value # 2020: 2.5

In above example we used the module prefix as strutils contains also a parseInt() and a parseFloat() function. The functions parseBin(), parseOct() and parseHex() behave in a similar way. Returned is the number of processed characters. We add the returned value to the start position so that parsing can continue at the new position.

There are some more functions available in this module, which we will not discuss further. It is enough that you know that this module exists and provides some efficient parsing functions. Whenever you should really need one of these procs you would have to consult the API documentation for details.

Module strscans

The strscans module provides a scanf() macro which can be used to extract substrings from textual user input. The content of the substrings is automatically converted to Nim variables of matching data types.

Processing of well defined strings is easy, and with user defined matcher functions even text input with less strict shape can be processed.

Let us start with a simple example: We may have to create a program where the user should be able to create rectangles by entering the coordinates of two opposite corners in the form

Rect x1,y1,x2,y2
import std/strscans

var x1, y1, x2, y2: float
var name: string

let input = "Rect 10.0,20.0,100,200"

if scanf(input, "$w $f,$f,$f,$f", name, x1, y1, x2, y2):
  echo name, ' ',  x1, ' ', y1, ' ',  x2, ' ',  y2 # Rect 10.0 20.0 100.0 200.0

The first parameter for the scanf() macro is the user input string, and the second parameter is a pattern string that specifies how the input string should be processed. The following parameters are variables that gets the results of the input evaluation. The second parameter has some similarity with a regular expression. The letters after the dollar sign specifies the data type of the substrings, $i or $f ask to process an integer or a floating point number, and $w requests to process an ASCII identifier. Other characters are captured verbatim, that is the space character after $w has to match a space in the input string, and the comma characters that separates the $f has to match commas in the input string. The scanf() macro supports capturing of some more data types, i.e. $c for an arbitrary character or $s for optional white space. The optional white space is not captured, just ignored. With the use of $s our program allows already a more flexible input string:

let input = "Rect10.0,    20.0,100,200"

if scanf(input, "$w$s$f,$s$f,$s$f,$s$f", name, x1, y1, x2, y2):
  echo name, ' ',  x1, ' ', y1, ' ',  x2, ' ',  y2 # Rect 10.0 20.0 100.0 200.0

To allow to process even more flexible input strings it is possible to use user definable matchers in form of Nim procs with a well defined parameter signature. There are two different types of matcher procs supported — matchers to just skip a part of the input string, and capturing matchers. For the next example we will use a proc which can skip various separators like comma, semicolon or white space. And we will use a capturing matcher proc for the object name.

import std/strscans

proc sep(input: string; start: int; seps: set[char] = {' ',',',';'}): int =
  while start + result < input.len and input[start + result] in {' ','\t'}:
    inc(result)
  if start + result < input.len and input[start + result] in {';',','}:
    inc(result)
  while start + result < input.len and input[start + result] in {' ','\t'}:
    inc(result)

proc stt(input: string; strVal: var string; start: int; n: int): int =
  if input[start .. start + "Rect".high] == "Rect":
    strVal = "Rect"
    result = "Rect".len

var x1, y1, x2, y2: float
var name: string

let input = "Rect 10.0    ;20.0,100  ,  200"

if scanf(input, "${stt(0)}$s$f$[sep]$f$[sep]$f$[sep]$f", name, x1, y1, x2, y2):
  echo name, ' ',  x1, ' ', y1, ' ',  x2, ' ',  y2 # Rect 10.0 20.0 100.0 200.0

The use of our user definable matcher sep() allows to separate the four numbers with a colon or a semicolon with arbitrary leading or trailing white space, or with only white space. Multiple colons or semicolons between two numbers resulting from a typo would be not permitted.

The signature for this matcher proc has this shape:

proc sep(input: string; start: int; seps: set[char] = {' ',',',';'}): int =

The first parameter is the string to process, and the second parameter is the position at which the processing should start. (Or in other words, the second parameter of integer type is the actual position in the input string, that position moves during the whole capering process from the start of the input string to its end. The end may not be reached if the capering fails at some point.) The last parameter has always the type set[char] with a default value indication which characters that proc can process. Actually a default value seems to be necessary, but the actual value seems not to matter. The proc returns the number of characters that should be skipped. Zero is a valid return value, so we can support optional separators. In most cases separators are necessary to process the input string, but we can imagine input formats where separators are optional, e.g. when an integer number is followed by a name. A name never starts with a digit, so the boundary between the two values is well defined. This none capturing matcher proc is called by use of $[sep] in the pattern string.

The signature of the capturing matcher proc has this shape:

proc stt(input: string; strVal: var string; start: int; n: int): int =

That proc also gets as parameters the input string and the start position, and has to return the number of processed characters. But additional a var parameter of arbitrary data type is used to return the result of the capture, and the last parameter with arbitrary data type can influence the capturing process. One possible use of the last parameter is to use an integer value to limit the maximum number of characters to process. This proc is called in the pattern string by using curly braces like ${stt(0)}.

Scanf() returns true when all the parameters match, for that case all the passed in variables get assigned a value. Currently scanf() does not support the capturing of optionally data, as the whole processing stops when one capture fails, i.e. when $i is used to request the capture of an integer value but the input string does not contain decimal digits at the current capture position. In the same way the whole capturing process stops when a user defined capturing matcher returns zero as no capturing is possible. So intermediate optional arguments are currently not supported. When the processing stops due to missing arguments, scanf() returns false, but the already processed captures still have a valid value assigned. In this way we can use at least optional trailing arguments.

As next example for the use of the scanf() macro we will give a real world example: A simple CAD (Computer Aided Design) program has a PCB (Printed Circuit Board) mode, in which the user can create new PCB pads by entering the pad data in a text entry widget. A PCB pad is a rectangular shaped copper field which has an associated number and a name and maybe rounded corners. The user should be able to enter two 2D coordinates, the corner radius, an optional x/y translation for the next pad of same size, followed by the number of pads to create and the pad number and name. That is

pad x1 y1 x2 y2 r dx dy n num name

The first five arguments are mandatory, the rest is optional with default values. The user should be able to separate the arguments with white space or with a colon or a semicolon. Additional the values x2 and y2 can be preceded with a '+' character to indicate that the x2, y2 tuple is not an absolute coordinate value but the width and high of the pad.

A program fragment to process this form of user input may look like

import std/strscans

proc jecho(x: varargs[string, `$`]) =
  for el in x:
    stdout.write(el & " ")
  stdout.write('\n')
  stdout.flushfile

proc stt(input: string; strVal: var string; start: int; n: int): int =
  if input[start .. start + "pad".high] == "pad":
    strVal = "pad"
    result = "pad".len

proc pls(input: string; plusVal: var int; start: int; n: int): int =
  if input[start] == '+':
    plusVal = 1 # bool
    result = 1

proc sep(input: string; start: int; seps: set[char] = {' ',',',';'}): int =
  while start + result < input.len and input[start + result] in {' ','\t'}:
    inc(result)
  if start + result < input.len and input[start + result] in {';',','}:
    inc(result)
  while start + result < input.len and input[start + result] in {' ','\t'}:
    inc(result)

proc plus(input: string; plusVal: var int; start: int; n: int): int =
  result = sep(input, start)
  if input[start + result] == '+':
    plusVal = 1 # bool
    result += 1

var st: string
var x1, y1, x2, y2, dx, dy: float
var px2, py2: int # bool
var n: int
var number, name: string

(st, x1, y1, px2, x2, py2, y2, dx, dy, n, number, name) = ("pad", NaN, NaN, 0, NaN, 0, NaN, NaN, NaN, 0, "", "") # defaults

var res: bool
var input = "pad 10.0, 10   12 +12.0 ;20 0 8 Num Name"

# unfortunately the input start with "pad" is needed for unpatched strscan!

# using the pls matcher, this fails when there is no '+'
res = scanf(input, "${stt(0)}$[sep]$f$[sep]$f$[sep]${pls(0)}$f$[sep]${pls(0)}$f$[sep]$f$[sep]$f$[sep]$i$[sep]$w$[sep]$w", st, x1, y1, px2, x2, py2, y2, dx, dy, n, number, name)
jecho(res, st, x1, y1, px2, x2, py2, y2, dx, dy, n, number, name)

# using the plus matcher, so the '+' is optional
res = scanf(input, "${stt(0)}$[sep]$f$[sep]$f${plus(0)}$f${plus(0)}$f$[sep]$f$[sep]$f$[sep]$i$[sep]$w$[sep]$w", st, x1, y1, px2, x2, py2, y2, dx, dy, n, number, name)
jecho(res, st, x1, y1, px2, x2, py2, y2, dx, dy, n, number, name)

input = "pad 10.0, 10   12 +12.0" # test with missing optional values
(st, x1, y1, px2, x2, py2, y2, dx, dy, n, number, name) = ("pad", NaN, NaN, 0, NaN, 0, NaN, NaN, NaN, 0, "", "") # defaults
# using the plus matcher, so the '+' is optional
res = scanf(input, "${stt(0)}$[sep]$f$[sep]$f${plus(0)}$f${plus(0)}$f$[sep]$f$[sep]$f$[sep]$i$[sep]$w$[sep]$w", st, x1, y1, px2, x2, py2, y2, dx, dy, n, number, name)
jecho(res, st, x1, y1, px2, x2, py2, y2, dx, dy, n, number, name)

When we compile and run this program we get this output:

false pad 10.0 10.0 0 nan 0 nan nan nan 0
true pad 10.0 10.0 0 12.0 1 12.0 20.0 0.0 8 Num Name
false pad 10.0 10.0 0 12.0 1 12.0 nan nan 0

The first scanf() call uses the sequence $[sep]${pls(0)} which fails when the float value has no leading '' sign, so this call is of no real use. The second and third scanf() call uses instead a ${plus(0)} call, where the plus() proc processes the separators as well as the optional '' character, so that proc has never to return zero, and the capturing process continues. For the last scanf() call we give as input only five values, so scanf() returns false, but the first five values gets assigned values, and the rest has default values. One restriction of above code is, that we have always to start the input string with "pad", otherwise the processing stops immediately. As scanf() does not support the capture of boolean values, we uses the integer data type for the variables px2 and py2. The value zero means that there is no '+' prefix, and 1 indicates that there is a plus prefix.[26]

The next tiny example shows how we can use the last parameter of the user defined matcher proc to control the matching process:

import strscans

proc ndigits(input: string; intVal: var int; start: int; n: int): int =
  var i, x: int
  while i < n and i + start < input.len and input[i + start] in {'0'..'9'}:
    x = x * 10 + input[i + start].ord - '0'.ord
    inc(i)
  # only overwrite if we had a match
  if i == n:
    result = n
    intVal = x

var input = "1234"
var a, b: int

if scanf(input, "${ndigits(2)}$s${ndigits(2)}$.", a, b):
  echo "Input is OK:", a, " ", b

We want to capture two integer values, each with one or two decimal digits. By passing the upper limit of digits to the ndigits() proc, we get the intended result even when the user does not separate the two numbers with white space. Additional we have used $. at the end of the pattern string. The $. matches only when the end of the input string is reached, so that scanf() would return false if there are more characters in the input string left.

The latest version of the strscans module provides also a variant of the scanf() macro called scanTuple() which returns a tuple. We could use it in this way in our example above:

let (res, a, b) = scanTuple(input, "${ndigits(2)}$s${ndigits(2)}$.", int, int)
echo res, " ", a, " ", b

So we have not to declare the capering variables in advance. The final result of the scan process is returned in an additional boolean variable. When we use user defined matchers as above, we have to specify the data types of the returned values as additional parameters after the pattern string.

Additional the strscans module provides a scanp() macro which works somewhat similar as PEG or RegEx libraries. We will not try to explain the scanp() macro, as its use may be too difficult for a beginner book. And when we really have to process text strings with regular expression grammars, then we can use the available RegEx or PEG modules which have no restrictions and work for Nim in a similar way as for other programming languages. We will introduce the Nim RegEx and PEG modules later in the book — maybe we will compare the scanp() macro there.

Module strformat

With the fmt() macro from the strformat module we can format and interpolate strings similar as Python3 with its f-strings.

import strformat
from strutils import `%`
var lang = "C"
var year = 1972
stdout.write "The programming language ", $lang, " was created in ", $year, ".\n"
echo "The programming language " & $lang & " was created in " & $year, "."
echo "The programming language $# was created in $#." % [$lang, $year]
echo fmt"The programming language {lang} was created in {year}."
# The programming language C was created in 1972.

From the four ways to print some text the last one with fmt() is the shortest and maybe the cleanest. As fmt() is a macro which is processed at compile time there is no unnecessary run-time overhead involved. A small restriction of fmt() is, that it’s argument is regarded as a generalized raw string literal. So we can not use escape sequences like "\n" in the string literal. But the strformat API docs mention various solutions for this: We can use the unary & operator instead of the fmt() call, or we can use the notations {'\n'}, fmt() or "".fmt.

import strformat
var lang = "Fortran"
var year = 1957
stdout.write &"The programming language {lang} was created in {year}.\n"
stdout.write fmt"The programming language {lang} was created in {year}.{'\n'}"
stdout.write fmt("The programming language {lang} was created in {year}.\n")
stdout.write "The programming language {lang} was created in {year}.\n".fmt
# The programming language Fortran was created in 1957.

The fmt() macro works also with multi-line raw strings:

import strformat
echo fmt"""This is a {1 + 1 + 1} lines
multiline
string."""

The fmt() macro accept a few directives for the formating of integer and floating point numbers like

import strformat
var pi = 3.1415
var people = 123
echo fmt"The number{pi:>8.2f} is called PI by {people:>8} people."

The basic pattern is, that the numeric variable is followed by a colon and the total number of desired characters. We can precede the number with a < or > to indicate left or right alignment, and for floating point numbers we can use a notation similar as used for printf() in the C language: n.mf stands for a float formated with n characters total and m decimal places. As in C we can use e instead of f to indicate scientific notation. If the value which specifies the total number of characters starts with a zero digit, then the formated number uses zeros instead spaces for leading digits. And #X after the colon generates hexadecimal value for integer numbers.

A useful property of the fmt() macro is, that we can put an equal sign into the curly braces to get the initial expression both as string and as interpolated value, as in

import strformat
var pi = 3.1415
echo fmt"{2 * pi = }" # 2 * pi = 6.283

This is similar as the dump() macro from the sugar module and is mostly used for debugging purposes.

To use curly braces as literals in a fmt() argument we can use character literals with a backslash as we did to include a newline character, or we can use an extended fmt() macro with two additional arguments which specifies the two characters that should be used instead of {} to mark the expression which should be interpolated:

import strformat
echo fmt"{2} curly braces {'\{'} {'\}'}." # 2 curly braces { }.
echo "three time three is <3 * 3>".fmt('<', '>') # three time three is 9

We will not try to explain all these various formating options in detail, as it is really hard to remember. It is enough that you know that these options exists, so you can consult the API docs for details when you would need it.

References:

Arrays and Sequences

Together with strings arrays and sequences are the most important built-in containers for the Nim language. While arrays have a fixed size defined already at compile time, sequences are like strings of dynamic size and can grow when we append more elements. As arrays have fixed size they can be allocated on the stack, while due to the dynamic size of sequences the actual data buffer has to be allocated on the heap. We explained some details about sequences already in part II of the book. One important aspect was that sequences use a continues data buffer with a fixed capacity to store the actual elements. When that data buffer is fully occupied, and we try to add more elements, then a new larger buffer is allocated on the heap and the contained elements have to been copied from the old to the new larger buffer before the old buffer can be deallocated.

When we pass arrays or sequences to procs, then we can use the special data type openArray when we define the proc to allow passing both arrays and sequences. Note that this is very different from generic procs: When we define a generic proc, then the compiler creates a new proc instance for each of the generic data type that we use, so when we call a generic proc which accepts floats and signed and unsigned integers, and we call it a few times with float and with signed integer arguments, then the compiler has to create two distinct instances of the proc. For openArray parameters all the time only one proc instance is necessary as sequences behave like arrays in many ways. Both use a continues block of memory where the elements are stored, and the position of an entry is given by the start address of this memory block and an offset given by the index multiplied with the size of an array element. So when passing the actual parameter to the proc, the compiler passes the array and the data section of the sequence in the same manner. Both can be passed by copy or by address. The compiler passes also the actual size, the lower index is always zero for openArrays. Of course when we pass a seq as openArray, there are some restrictions, e.g. we could not add elements in the proc as the passed variable behaves like an array.

Memory layout of sequences and strings is very similar, both have length, capacity and a data buffer on the heap, and some procs that work on the data structure has the same names like add(), len() and setlen() and both support operators like [], & and .. for access to single elements, and for concatenation and slicing.

Some often used functions and operators for sequences and arrays are defined in the system module, like creating new sequences, converting arrays to sequences, joining sequences or adding elements to it. Other important functions, operators and iterators are defined in the sequtils module which we describe in the next section.

var s0: seq[int]
var s1: seq[int] = newSeqOfCap[int](2)
var s2: seq[int] = newSeq[int](2)

s0.add(3)
s0.add(5)
s1.add(3)
s1.add(5)
s2[0] = 3
s2[1] = 5
echo s0; echo s1; echo s2 # @[3, 5] for each

We can initialize sequences by a call of newSeq(), newSeqOfCap() or not at all. When we use newSeq(n) we get a seq with n elements initialized to binary zero each, and we then can just overwrite the elements by use of the subscript operator [] which is faster than appending elements with add() to an empty seq. With newSeqOfCap() we can allocate a seq of size zero but with a buffer size of that specified capacity. We can append elements by calling the add() function, and as long we append not more elements as specified in the newSeqOfCap() call we can avoid reallocations of the internal seq buffer. When performance in not that critical we can just use a uninitialized seq and add() elements — when the default capacity is exhausted a reallocation occurs, generally with doubled data buffer size.

We can use the overloaded add() proc to append single elements or to append an whole array to a seq, and the & operator is available to join two sequences:

s0.add([7, 9])
s0 &= s1
s1 = s0 & s2
echo s1 # @[3, 5, 7, 9, 3, 5, 3, 5]

We can use len() to query the length of a seq and setLen() to set a new length. In most cases setLen(n) is used to shorten a seq, that is to keep the first n elements, but we can also use setLen() to increase the length of a seq. In that case the new entries get the value binary zero as default and we can use the subscript operator [] to fill in actual content. Increasing the length with setLen() may cause a reallocation if the current capacity is not sufficient. Functions low() and high() are available to get the lowest and the highest index position of an array or a seq. While arrays can have negative indices, low can be less than zero for arrays, but for sequences and openArray proc parameters low is always zero.

Module system provides also the @ array to seq operator:

var i = 7
var j = 9
var s0 = @[1, 2]
s0 = s0 & @[i, j] # don't use this variant!
s0.add([i, j]) # faster
echo s0 # @[1, 2, 7, 9, 7, 9]

Note that the code in line 4 would be really slow as that would have to allocate a temporary seq. Using add to add a temporary array to the seq should be faster.

Slicing is also supported by the system module:

var s0 = @[1, 3, 5]
var s1 = s0[1 .. 2] # @[3, 5]
for el in s0[1 .. 2]: # may create a copy of the seq
  echo el

Note that line three may create a temporary copy of the sequence, which may be not that nice for optimal performance. We discussed that topic already in part II of the book, Nim 2.0 may improve the situation further by introducing views which create no copies. Beside the s[a .. b] slice operator which includes the elements at position a and b, there is s[a ..< b] which does not include position b and and s[a .. ^b] where position b is taken from the end of the seq or array, e.g. ^1 is the last position, ^2 the second last.

For deleting elements from a sequence we have del() which replaces the element at the specified position with the last element of the seq and reduces the seq length by one, and the delete() function which shifts all elements after the specified position one step forward. Obviously the later is slower, but it preserves the order of elements. The function pop() deletes and returns the last item of a seq. For using pop() on an empty seq we may expect a raised exception. With insert we can insert an item at the specified position by moving all the elements after this position upwards. Note that del(), delete(), pop() and insert() are not available for arrays.

Comparison of two arrays or sequences by the == operator returns true when the length as well as all contained items matches.

With the function contains() we can test if an item is contained in a seq or an array. We can also use the operators a in b and a notin b instead. The elements are tested from the start of the container until a match is found or the last position in the container is reached, so this is a O(n) operation.

Module sequtils

This module defines some useful procs, iterators and templates for working with arrays and sequences. Some functions of module sequtils use an generic openArray parameter and so can be used for strings as well. While the max() and min() procs are available from the system module, the minIndex() and maxIndex() procs are provided by sequtils:

import sequtils
var s = @[7, 3, 5]
echo s.min, " ", s.max # 3 7
echo s.minIndex, " ", s.maxIndex # 1 0

Sometimes we may need a minmax() proc which gives us both values, but that one is currently not available. We have to create it our self if needed, when performnce is not that critical we can call min() and max() separately.[27]

The functions minIndex() and maxIndex() as well as count() or deduplicate can work with strings also:

import sequtils
var s = @[3, 5, 1, 7, 3, 3, 5]
echo s.count(5) # 2
echo s.deduplicate # @[3, 5, 1, 7]
echo "abc".maxIndex() #2

The function name deduplicate may be irritating, as the function does not work in place, we may expect the name deduplicated() as for sort() and sorted(). Other programming languages use the name uniq() instead. Deduplication is easy when the elements are sorted, as for that case we can just iterate over the seq and ignore equal adjacent items. That is why deduplicate() accepts an optional boolean parameter indicating if the seq is sorted. If the seq is not sorted, it may be necessary to create a temporary set to store the already seen items so that they can be ignored the next time when they occur again in the seq.

The function concat() can join multiple sequences (yes only sequences, it does not work with arrays currently), maybe that is more efficient than using the & operator. And insert() can insert a new value at a position by shifting following items, and delete() allows to remove a range of items from the seq when we specify two index positions:

import sequtils
var s: seq[int]
s = concat(@[1, 2], @[3, 4], @[5, 6])
echo s # @[1, 2, 3, 4, 5, 6]
echo @[1, 2] & @[3, 4] & @[5, 6] # @[1, 2, 3, 4, 5, 6]
s.insert(7, 2)
s.delete(3, s.high)
echo s # @[1, 2, 7]

Proc repeat() is used to create a seq which contains a single value multiple times, and cycle() repeats the items of an existing seq multiple times:

import sequtils
echo 2.repeat(4) # @[2, 2, 2, 2]
echo @[1, 2].cycle(3) # @[1, 2, 1, 2, 1, 2]

A bit more complicated but really useful are functions like map(), filter(), keep() and the corresponding …​It() templates:

import sequtils, sugar
var s = (0 .. 9).toSeq
echo s.map(proc(x: int): int = x * x) # always @[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
echo s.map(x => x * x) # from sugar module
echo s.mapIt(it * it)

echo s.mapIt("*" & $it & "*") # @["*0*", "*1*", "*2*", "*3*", "*4*", "*5*", "*6*", "*7*", "*8*", "*9*"]

echo s.filter(proc(x: int): bool = (x and 1) == 0) # both @[0, 2, 4, 6, 8]
echo s.filterIt((it and 1) == 0)

The map variants return a new seq, with an operation performed on all items. The returned seq can have a different base type. In line 4 we used the ⇒ operator from the sugar module for a simpler notation

The filter() variants apply a proc with boolean result type on the seq items and return the items for which the result is true. Remembering if the elements for which the procs gives a true result are returned or removed from the initial seq may be not easy. It helps to remember that filter() behaves like the keepIf() procs — items with positive result survive.

import sequtils, sugar
var s = (0 .. 9).toSeq
var s1 = s
s1.apply(x => x * x) # @[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
s1 = s
s1.keepIf(proc(x: int): bool = (x and 1) == 0) # @[0, 2, 4, 6, 8]

In above code we used toSeq() with a slice argument to create an initial sequence with continues integers from the slice. The apply() function performs a transformation operation on all the items, and keepIf() preserve only the items for which a boolean predicate evaluates to true.

Two useful predicate functions are any() and all() to check if at least one item fulfills a condition or if all items fulfill a condition:

import sequtils
var s = (0 .. 9).toSeq
echo s.all(proc(x: int): bool = x < 10) # true
echo s.any(proc(x: int): bool = x * x == 25) # true

With zip() we can join the items of two sequences to tuples, and with unzip() we can separate the tuple items again in two separate sequences:

import sequtils
var s = (0 .. 9).toSeq
var s1 = s.mapIt(it * it)
var z = zip(s, s1)
echo z
echo z.unzip
# @[(0, 0), (1, 1), (2, 4), (3, 9), (4, 16), (5, 25), (6, 36), (7, 49), (8, 64), (9, 81)]
# (@[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], @[0, 1, 4, 9, 16, 25, 36, 49, 64, 81])

Finally sometimes the templates foldl() and foldr() can be useful for folding a sequence, that is to generate one final value from all the items. The fold templates uses the variables a and b to generate the result, a + b would sum all the items. Foldl() performs the operation from left to right, returning the accumulation and accepts an optional start value, while foldr() starts from right, i.e. with the item at the end of the seq.

import sequtils
var s = (0 .. 9).toSeq
echo s.foldl(a + b, 100) # 145
echo s.foldr(a + b) #45

The sequtils module contains some more procs, templates and macros that are not needed that often. It would not make much sense to mention all of them here, as it is already not easy to remember the ones that we have introduced above. You should skim the sequtils API docs from time to time to remember what is available.

Maybe you have missed the difference of two sequences which some other programming languages provide:

[1, 2, 3, 4, 1,  5] - [2, 4] # [1, 3, 5]

Well that is an expensive operation, O(n^2) if implemented in a naive way, as it may iterate for all items in the second seq over the whole first seq to remove the unwanted items. A better approach is to convert the items in the second seq to a temporary (hash) set to allow faster query:

A fast on the fly solution is this, as suggested by someone in the Nim forum:

import sequtils, sets, sugar
let a = [1, 2, 5, 2, 9, 7, 0]
let b = [7, 4, 1, 10, 7]
let bSet = b.toHashSet()
echo a.filter((x) => x notin bSet) # @[2, 5, 2, 9, 0]

When we should need this operation often, we may define our own proc like

# https://ruby-doc.org/core-2.6/Array.html#method-i-2D
# Array Difference
import sets

proc `-`*[T](a, b: openArray[T]): seq[T] =
  let s = b.toHashSet
  result = newSeq[T](a.len)
  var i = 0
  for el in a:
    if el notin s:
      result[i] = el
      inc(i)
  result.setLen(i)

proc `-=`*[T](a: var seq[T]; b: openArray[T]) =
  let s = b.toHashSet
  var i = 0
  var j = 0
  while i < a.len:
    if a[i] notin s:
      a[j] = a[i]
      inc(j)
    inc(i)
  a.setLen(a.len - (i - j))

proc main =
  let a = [1, 2, 5, 2, 9, 7, 0]
  let b = [7, 4, 1, 10, 7]
  echo a - b
  echo b - a

  var x = @a
  x -= b
  echo x

main()
@[2, 5, 2, 9, 0]
@[4, 10]
@[2, 5, 2, 9, 0]

Note that this preserves the order in the first seq, which is often requested. If order is not important, then we could convert both sequences to a set and build the set difference — but when order does not matter we may use sets instead of sequences from the beginning.

Maybe you missed also a shift() function which other container types or programming languages may provide, that is used similar to pop(), but deletes and returns the the first item of a seq? Well it should be obvious why a shift() is not provided by default and why such a function would generally be avoided — the function name gives you already a good hint. And if you should need such a function it should be no problem to implement one if efficiency is really not important. But maybe in that case it would be better to use a different container type, maybe the double-ended queue provided by the deques module.

References:

Random Numbers

Most computer programs work fully deterministic, that is one input data set generates exactly one well defined output data set. This behavior is not desired when we create games or simulations: The actions of computer controlled characters should not do exactly the same again and again when we restart the game, and maybe the computer generated landscape in the game should look different when we restart the game too.

To generate such an unpredictable behavior we use random number generators, which can generate sequences of random number of integer or float type. The most important property of true random numbers is that we can not predict the next one from the sequence of all values seen before. Random number sources have no memory! Children often think they have. If a child got value six in a dice game tree times in sequence, its often assumes that it is extremely unlikely that next roll will give again the value 6. But as the dice has no memory, chance to get a specific number is always 1/6, as we have 6 possible values all with same probability. At least when the dice is not manipulated. Another important property of random numbers is the distribution of the possible values. For most random number sources we would expect a uniform distribution of all possible values: For a dice with numbers 1 .. 6 we would expect that we get all these numbers with nearly the same total quantity when we roll the dice for a long time again and again. But of course not all random quantities are distributes uniformly. The distribution can have different shapes. An important non uniform distribution is the Gauss distribution, where the final result depends on many random decisions. So an average value is more likely than extreme values. You may know the marble nail board with multiple slots as an example: Marbles are thrown in at the top, and whenever they hit a nail, they get distracted to the left or right.

To built a perfect random generator we would have to use some physical noise sources like photons emitted by a thermal light source falling on a light detector with single photon resolution (Photo-multiplier), radioactive decay, thermal noise or similar physical entropy sources. But using real physically sources for random numbers is difficult and the random number generation is slow. So in computer programming we generally use so called pseudo-random-numbers, which are sequences of numbers calculated based on a given starting number. A mathematical function gets the last n numbers seen before and generates the next one from that. If that function uses a smart mathematical expression, its results looks really like random numbers. For games the generated sequences are generally good enough, for cryptographic applications they may be not good enough. So what we need for a random number generator is a sequence of start numbers, and a mathematical function with internal state. For each call of that function a new random number is returned and the internal stage is changed so that next call will result in a different number. When we use always the same sequence of starting numbers, then our generator would generate always the same sequence of random numbers. Sometimes this is desired, e.g. when we want a behavior that looks random, but is reproducible, maybe for debugging tasks. But in most cases we would use starting numbers that are different for each program start. To get well suited start numbers we can just use the current time with nanosecond resolution which most computer hardware do provide.

Most simple and fast random number generators use for its internal state two integer numbers. From these two numbers the next random value is calculated and then the numbers representing the internal state are modified also, to ensure that the next generated number is again different.

Nim uses in its random module an implementation of the xoroshiro128+ (xor/rotate/shift/rotate) library. A Rand object with two integer fields is used to store the actual state, and some simple and fast logic operations as bit shift, logical xor and addition is used to update state and to generate the next number:

type
when defined(js):
  type Ui = uint32
  const randMax = 4_294_967_295u32
else:
  type Ui = uint64

  Rand* = object # State of a random number generator.
    a0, a1: Ui

proc rotl(x, k: Ui): Ui =
  result = (x shl k) or (x shr (Ui(64) - k))

proc next*(r: var Rand): uint64 =
  let s0 = r.a0
  var s1 = r.a1
  result = s0 + s1
  s1 = s1 xor s0
  r.a0 = rotl(s0, 55) xor s1 xor (s1 shl 14) # a, b
  r.a1 = rotl(s1, 36) # c

The Rand object stores the internal state, rotl() is a helper function which updates the state for each call, and next() is the actual generator procedure returning an uint64 value. Note that the addition used in next() does wraps around instead of giving an overflow error as unsigned integers are used. The numbers returned by the rand() proc are the foundation for all the other random number types provided by the random module. To get integers with reduced numeric range, we can just use the modulo operation, and to get float results we may convert the integer value to float and apply some basic mathematical operations like division for range reduction.

The most basic functions provided by the random module are the overloaded rand() functions. Rand() called with an integer parameter n gives us integer random numbers in the range from 0 up to n, and rand() called with a float parameter x will give us random float numbers in the range 0 ..x. When we just use the rand() functions in this way we would get the same sequence of numbers for each run of our program, as the generator always starts with the same well defined initial state. We can call the proc randomize() before calling rand() to initialize the generator to a different state based on the current time. Then rand() will provide us with different number sequences for each start of our program.

Generally it is a good idea to not use the one internal global state of the random module for generation our random number, but to use our own state variable. That way we prevent conflicts with other modules which may use the random module as well. Imagine that we want to get the same sequence of random numbers for each run of our program as we are debugging our game, but another module initialize the internal state of module random with a value based on the current time.

So the module random provides overloaded rand() functions that gets a state variable:

from times import getTime, toUnix, nanosecond
import random

let now = getTime()
var rstate = initRand(now.toUnix * 1_000_000_000 + now.nanosecond)

for i in 0 .. 5:
  echo rstate.rand(5) + 1 # dice roll

for i in 0 .. 2:
  echo rstate.rand(100.0) # float random number in range 0.0 .. 100.0

Unfortunately the initRand() call to initialize it with a current time value is a bit complicated, as we have to provide the current time value directly. Note that you generally should call initRand() only once in your program. A common mistake of beginners is to call initRand() each time directly in from of the rand() call. That is not only not needed and slows down the generation process, it also can lead to strange number sequences.

At the end of this section we will discuss the problem of filling a container with random but unique numbers. For example assume that we want to generate a sequence of 100 random numbers in the range 1 .. 100, with the restriction that each number should occur exactly once in the sequence. Of course a code segment like s[i] = (rand(99) + 1) would not work, as the same numbers could be generated multiple times or not at all. The obvious solution for this task is to fill an array first with consecutive numbers 1 to 100 and then exchange the initial positions with destination positions determined by (rand(99)) The random module provides the shuffle() function for this shaking of a container. A related function is sample(), which is used to randomly select an element from an openArray or a set.

References:

Timers

Sometimes we may want to measure the execution time of a code segment of our program. For this the Nim standard library provides various modules including the larger times module and the monotimes module. The times module provides many functions and data types for handling dates and times, while the small monotimes module is more specialized for measuring of time intervals. For our first test we will use the times.cpuTime() function and the monotimes.getMonoTime() function. The former gives us time values as seconds in float format, while the later returns an int64 nanosecond value. To measure execution times of code segments we ask for the current time at the start and at the end of that segment and build the difference. Actually we will try to measure the time needed for a float square root calculation. In the past calculating square roots was considered a relatively slow operation, slow compared to a plain floating point math operation like multiplication or division. But on most modern hardware square root calculation is really fast actually as we will see.

import std/[random, times, monotimes]
from std/math import sqrt

proc warmup =
  var x: float
  for i in 1 .. 1e10.int:
    x += 1.0 / i.float
  echo x

proc main1 =
  let x = rand(3.0)
  let y = rand(7.0)
  let start = cpuTime()
  let res = sqrt(x) + y
  let stop = cpuTime()
  echo stop - start
  echo res

proc main2 =
  let x = rand(3.0)
  let y = rand(7.0)
  let start = getMonoTime()
  let res = sqrt(x) + y
  let stop = getMonoTime()
  echo stop - start
  echo res

randomize()
warmup()
main1()
main2()

To get meaningful results we have to take some care: Most important is, that the code segment that we want to measure is really executed. This may sound odd, but assume that the code produces no noticeable result at all. In that case the compiler may just remove that code fragment from the generated executable, as it is not needed for correct program execution. Or assume that the code fragment uses only data which is already known at compile time. Then the compiler may do all of the calculations already at compile time, and the whole code fragment is removed again and replaced by the pre-calculated values. And finally we have to remember that our computer may execute other task at the same time or may be in various power saving states with reduced CPU clock frequency, from which it takes some time to wake up. To take care of this we try to execute some warmup code before our actual timing task, and we do use the rand() function from the random module to provide input values for our code that are not known during compile time. Finally we output the result of the calculation by use of an echo statement to make clear to the compiler that the result of the calculation is really needed. Now let us compile and run this program. We compile with option -d:release or -d:danger to enable optimizations and avoid generation of debugging code that may distort our timing. The result is still a bit surprising:

$ ./t1
23.6030665949975
4.98999998654881e-07
4.676567857718999
42 nanoseconds
8.527354349689125

Lines tree and five are the results of our timing attempt. Both values are obvious too large and do not match. The reason for the wrong results is the overhead by the function calls itself. That overhead seems to be much larger for the cpuTime() call, as we get result of about 500 nanoseconds. Maybe the reason for this is that cpuTime() works with floats internally. At least we see that getMonoTime() can measure time intervals in the range of a few hundred nanoseconds. When we run the program a few times the printed time intervals may vary. The reason for that are internal processes in the CPU like clock rate and state changes. Generally the smallest time value of multiple program executions is the most important for us, as that is the minimal time which is actually needed for the program execution. With this example we have learned how we can measure time differences in our program, and that measuring really small time intervals is difficult.

Fortunately measuring such tiny time intervals is supported by the criterion package, which we may describe in later sections of the book.

For now we will present another example program, where we measure program code with longer running times. For that we create a loop that is executed many times. So the offset of the timer functions calls can be neglected compared to the actual running time of the loop, and due to the longer running time the printed time values get more reliable with low variations:

import std/[random, times, monotimes]
from std/math import sqrt

proc main3 =
  var s: array[64 * 1024, float]
  var res: float
  for i in 0 .. s.high:
    s[i] = rand(100.0)
  let start = getMonoTime()
  for i in 0 .. s.high:
    res += sqrt(s[i])
  let stop = getMonoTime()
  echo stop - start
  echo res

proc main4 =
  var s: array[64 * 1024, float]
  var res: float
  for i in 0 .. s.high:
    s[i] = rand(100.0)
  let start = cpuTime()
  for i in 0 .. s.high:
    res += sqrt(s[i])
  let stop = cpuTime()
  echo stop - start
  echo res

randomize()
main3()
main4()

For this example program we first fill an array with 64k random float numbers and than sum the square root of these numbers. As the total running time of our loops is not that tiny, we do not need a special warmup function which is executed in front of the timed code. The output of our program show that both timing functions match well for longer time periods:

$ ./t2
112 microseconds and 701 nanoseconds
436909.89897942
0.0001135580000000001 # 114 microseconds
437222.3001038401

The float result in line 4 is 114 microseconds which matches well with the 112 microseconds from line two. When you run this program multiple times you may notice that you may get sometimes much lager results for both or for only one of the two values. That is not surprising, as the computer is processing not only our program but many more, and due to task switching our program may be suspended for some time. When we divide that 114 microseconds by the number of loop iterations (64 * 1024) we get 1.7 nanoseconds, which is really surprising fast for a square root calculation. The concrete value is for a modern Intel I7 CPU. Of course these 1.7 nanoseconds is not only the time needed for the square root calculation, but it includes the operations with the loop counter and the time needed to fetch and to access the actual array elements.

As timing code segments is not an uncommon use case, there exists some external package which improve or simplify these operations, like the criterion or benchy package. A related task is profiling our program to find the part which takes the most CPU time, so that we can concentrate on these parts to improve the total performance of our program. For profiling various tools like the Linux perf tool are available, which we will discuss in more detail later in this book.

References:

Hash Tables

A common task in computer programming is the storing and retrieving of data records. The situation is very easy whenever each data record is directly mapped to continuous integer number n0, n0+1, n0+2, …​, n0+M. In that case we can use that numbers as index keys to access the data records and store the data as objects or references to objects in a sequence, or maybe when the data set in small and the maximal number of entries is known at compile time in an array.

In the past it was a common practice to give hardware parts in a shop and even customers an unique id number from a continues range, so that storing and fast access in indexed containers is possible. This works well when we really use the numbers as keys. But actually we generally work with data which is already labeled by names expressed as sequences of ASCII character: Customers in a hardware store, food in a super marked. Assigning id numbers to people is possible, but generally people do not like to have to remember the assigned id number when they want to buy something in a online shop.

So let us investigate how we can store and retrieve data objects without the use of continues numbers as key. Assume we have a customer data base

type
  Customer = object
    lastName: string
    firstName: string
    age: int
    postalAdress: string
    phone: string
    credit: float

Of course we can store the customers just in a seq, and do a linear search when we want to access a person by name:

customers: seq[Customer]

# ...

var found = false
for c in customers:
  if c.lastName == queryName:
    found = true
    echo "Person ", queryName, " has a credit limit of ", c.credit
if not found:
  echo queryName,  "not found"

Such a plain linear search is not very fast of course. An obvious improvement would be to sort the customers by names, as we can do a so called binary search in that case as we did long time ago in printed telephone registers: Open the telephone book somewhere in the middle, and when the names on that page are all greater than our friends name, then continue the search in the first half of the book, otherwise in the last half. We continue the halving strategy until we find the name. As we half the data set in each step this way, we say that the algorithm has log2(N) cost, where log2 is the logarithm with base two and N is the size of the data set.

A similar solution would be to use some form of an ordered binary tree, which has also log2(N) costs for retrieving operations. We will learn more about sorting sequences, doing a binary search in a sorted seq and about tree structures later in the book.

A hash table, called only table in Nim, is a homogeneous resizable container that behaves similar as the Nim sequences, but releases the restriction that for direct accessing an element its position in the sequence has to be known.

The idea of a hash table is to use an arbitrary data type to directly access objects stored in a container in a similar way as we can do it for arrays and sequences with integer keys. The first step is to use a so called hash function to map key objects, which are not already of integer type, to the integer type. We would try to use a hash function which can be evaluated fast and which maps our data to integer values distributed to the whole integer value space without clustering. The integers generated by a hash function look in some way like random numbers, they are distributed over the full integer value range without an obvious order or systematic. Mapping arbitrary objects to integers is generally not difficult. For as string a first attempt would be to use the characters of a string in the same way as we calculate the value of a number literal by summing up the digits each multiplied with powers of ten given by the position. For a string that may look like

intVal = uint(s[0]) * 256^0 + uint(s[1]) * 256^1 + uint(s[2]) * 256^2 + ...

We multiply with powers of 256 as we have 256 different ASCII characters.

That would be not really a good hash function, as all short strings would be mapped to low integer values, not distributed over the full value range. But similar, smarter has functions are available.

So a hash function can map arbitrary data types to integers. But what we really want is a sequence of continues integer values, which the hash functions does not provide by design. But that is no real problem: In the same way as we can do a range reduction for a random number generator function generating random numbers using the full integer range by just applying a modulo operation, we can apply the modulo operation on the value returned by a hash function.

Range reduction by modulo gives us smaller integer numbers, which may already be some form of index values for an array or a seq. With two restrictions: Index collisions can occur, as applying the hash function and the modulo range reduction on different strings may give us the same index value. And some index values may be never generated. The later is not that serious, some positions in the container would be remain unpopulated. Collisions are much more serious of course, we have to handle them somehow. One solution is, that we make each storage location in our container again a sequence, which can store all the colliding data sets. That way our hash table would be a sequence, where each element is again a short sequence containing all the colliding data record. For a data retrieving operation applying the hash function with modulo data reduction would give us the position in the larger seq, and we then would have to check all the elements in the short seq to find the actual record. In best case the short seq contains only one entry, when no collision has occurred. For a customer data base this strategy may be indeed the best solution, as in rare cases multiple different customers may have exactly the same name. So it would be nice if for query operations in that case a list of all customers with exactly that name is returned.

In practice often a modified strategy is applied preventing the seq in seq container type: We use only one loosely populated seq, and whenever a collision occur, we just put the colliding data record at some free index position after the position determined by the hash index value. That way data retrieving starts by the position given by the hash key and then checks the data record at that position and the following positions, until a matching record or an empty position is found, the later case indicates that the queried data record is not contained in the data base. Storing data records works similar: When the position given by the hash key is void, then the new entry is stored at that position. If the index position is already occupied, then following positions are examined until a void one is found and the data is stored there.

Hash tables work well generally when they are not too dense populated. Generally we make the number of available index position double the size as the number of expected entries. Then the change of collisions is not too large, and when a collisions occurs, then chances are high that one of the next positions are still unpopulated.

When by inserting more and more data records the population density becomes too high, then generally a new, larger table is allocated and the data records are moved from the old to the new table, in a similar way as it is done for plain sequences when all capacity is occupied.

Now let us see how we can us the tables module of the Nim standard library to store the customer record we introduced above:

import std/tables

type
  Customer = object
    lastName: string
    firstName: string
    yearOfBirth: int
    postalAdress: string
    phone: string
    credit: float

var customers: Table[string, Customer]

proc addNewCustomers =
  var c: Customer
  c = Customer(lastName: "Turing", firstName: "Alan")
  c.postalAdress = "England"
  c.yearOfBirth = 1912
  customers["Turing, Alan"] = c

  c = Customer(lastName: "Zuse", firstName: "Konrad")
  c.postalAdress = "Germany"
  c.yearOfBirth = 1910
  customers["Zuse, Konrad"] = c

proc queryCustomer(key: string) =
  if customers.hasKey(key):
    echo "known customer:"
    echo customers[key]
  else:
    echo "customer key not found in data base"

addNewCustomers()

queryCustomer("Zuse, Konrad")
queryCustomer("Gates, Bill")

The basic usage of Nim tables is very similar to the use of sequences. While we have to specify only the base type for a Nim seq, we have to specify the type of the key and the type of the stored entities for a hash table. The line var customers: Table[string, Customer] defines a variable with a generic table type. The table uses strings as keys and stores Customer objects. We can then use the [] subscript operator to store Customer objects in the Table. As we created a table with string key type, we have to specify strings when we use the subscript operator or other functions to access entries of our table. For the query operation we first call the function hasKey() to check if the customer with that name is contained in the data base and then use again the subscript operator to access the data record.

The tables module of the Nim standard library provides many more functions for interacting with tables. Most are easy to understand and use. When you inspect the API docs of the tables module you will discover that beside the Table data type also a TableRef exists. The Table type has value semantics, that is if you copy a whole table instance, then the whole content is copied. TableRef instances have reference semantics, the content is not copied when you assign one instance of a TableRef to another variable.

In the example above we called hasKey() to check if a data record is available before we accessed that record. Access with the subscript operator [] would raise an exception when an entity is not available in the table. HasKey() and [] both would have to locate the data record. A faster way to access data record when we are not sure if they exists in the table is the getOrDefault() proc:

let dummy = Customer()
let query = customers.getOrDefault(name, dummy)
if query.lastName.len == 0: # we know all entries in the database have a lastName, so we got the dummy default value
  echo name, "not found"
else:
  process(query)

A useful variant of the Table data type is the CountTable, which we can use to count data object, maybe words in a text:

import tables
var ct: CountTable[string]
ct.inc("Nim")
ct.inc("Rust")
ct.inc("Nim")

echo ct["Nim"]
for k, v in ct:
  echo k, ": ", v

A table instance stores entries not in the order of insertion. When we iterate over the table, we get the results not back in the order of insertion. If we really should need to preserve the insertion order we may use the OrderedTable variant. Note that an OrderedTable does not sort its entries, it remembers insertion order. Ordered Tables have some internal overhead, so we should use them only when necessary.

For all the various table variants we can use procs like clear(), len() or del() to remove all entries from a table, to check for the number of entries or to delete entries. Note that some functions may throw exceptions when we try to access entries that are not available. And note that the subscript operator =[] overwrites already existing entries.

For our initial customer data base the current table implementation may still be not optimal, as it is not clear how to handle different customers with the same name. But customer data bases are really special cases, in most cases different things have different names.

User defined hash values

The tables module uses the hashes module to calculate the hash value for the keys that we use to access table content. For many data types the hashes module already defines a hash function. When we would like to use tuples or object data types as keys for table access, then we would have to define a hash function for that key objects first. The API documentation of the tables module contains an example for this, where as key an object data type with firstName and lastName fields is used to store salary entries in the table. While firstName and lastName are strings, and for single strings a predefined hash function is available, we have to declare another hash function for objects with two strings:

import tables, hashes

type
  Person = object
    firstName, lastName: string

proc hash(x: Person): Hash =
  ## Piggyback on the already available string hash proc.
  ##
  ## Without this proc nothing works!
  result = x.firstName.hash !& x.lastName.hash
  result = !$result

var
  salaries = initTable[Person, int]()
  p1, p2: Person

p1.firstName = "Jon"
p1.lastName = "Ross"
salaries[p1] = 30_000

The hash generation is a bit cryptic: First we mix various existing hash values using the !& operator, and finally we use the !$ operator to generate the final hash value. For details please see the API documentation of the hashes module.

Hash tables can be seen as a way to attach arbitrary data to other data. The above example attaches a "salary" to a person object. In most cases we would just create one more object field when we have to store more data, but sometimes that is not easily possible. One example is when we use a low level C library, which gives us some C objects back. Maybe we use an advanced C or C math library like CGAL, and we get some abstract low level objects from it, maybe circles with center coordinate and diameter. As that objects are not Nim object, but C or C entities, we can not easily subclass them to attach more properties like a color attribute. But as each entity has a unique address, we can just use a table with key type address and all the needed values like color as data. That would be some overhead of course, as each color lookup would mean a table access, but it is a simple solution.

We can even attach properties to plain data types this way:

import tables

var t: Table[float, string]

let PI = 3.1415
t[PI] = "Pi"
t[2.0] = "two"

echo t[PI]
echo t[2.0]
echo t[5.0 - 3.0]

For floats a predefined hash function is available, so the code above should work. But floats as keys are a bit fragile due to the fact that float math is not really exact. So the last line in above code may raise an exception due to access of a non existent entry, as the difference 5.0 - 3.0 may not exactly be identical to the value 2.0.

Hash tables can be even useful containers, when we already have numeric data as possible keys for indices in a sequence: In mathematics we could have a two dimensional array, that is an array of array in Nim, to store matrices. This is OK when the matrix is really populated, that is most entries have meaningful non trivial values. But for very large, loosely populated matrices, with mostly just zero or one entries, storing the whole matrix as a table using a row, column tuple as key, may save a lot of RAM.

Equality and Identity

When we use objects or references to objects as keys for tables, we have to remember how Nim compares value and reference types:

import tables, hashes

type

  O = object
    i: int

  R = ref object
    j: int

proc hash(o: O): Hash = hash(o.i)

proc hash(r: R): Hash = hash(cast[int](addr(r[])))

var o1 = O(i: 7)
var o2 = O(i: 7)
var r1 = R(j: 13)
var r2 = R(j: 13)

echo o1 == o2
echo r1 == r2

var t1: Table[O, float]
t1[o1] = 3.1415
echo t1.hasKey(o2)

var t2: Table[R, float]
t2[r1] = 2.7
echo t2.hasKey(r2)

The output of above program is

true
false
true
false

By default the == operator compares content for value objects, but the instance addresses for references. Because of this it makes sense to define hash functions for object types and ref object types in a compatible way: We use the hash value of the single integer field of our value object as hash result for the whole object, and we use the address of the instance for the hash value of the reference object. As different instances of ref objects have always different addresses, the hasKey() does return false when we use as argument a different instance variable, independent on the content of its fields.

For special use cases we may redefine the == operator, but we have to ensure that the defined hash function matches the == operator: When a == b is true, then hash(a) has to be identical to hash(b)! The reason is that tables first compare the hash value of query key with key of entities in the table, and only for matching hash value do comparison of the actual data content.

Performance

Hash table lookup is fast. We say that hash table lookup is a O(1) operation, which shall indicate that the time needed for doing a table lookup does not depend on the total number of entries stored in the table. The reason for that is that for a lookup it is necessary to calculate the hash value, do the modulo operation and access the table content and maybe a few of the following table entries in the case that the first entry is not a match. Storing data is also an O(1) operation, as it works very similar, as long as the table is not already too dense populated so that a recreation is necessary. In that case that single storing operation is obviously very slow, but that occurs very rarely, and maybe not at all when we use a large enough table from the beginning. Still small tables are much faster than larger tables due to cache effects. For small tables all data may fit into the caches of the CPU, while for large tables most data is located outside of caches in RAM, and RAM access is magnitudes slower than cache access.

And hash table lookup is slower than array or seq access. To access an element from an array or seq we have only to multiply the index value with the byte size of the stored elements type and maybe to add an offset when the array does not start at index 0. For tables we have to calculate the hash value, do the modulo operation, access some elements at the calculated position, and most importantly, to compare the content at that positions with the actual key data. If the key is a string, a few string comparisons (at least one) are necessary to determine if the query element is available in the table. So while array access may take less than a nanosecond on modern hardware, table lookup may take a few dozens of nanoseconds. Lookup with string keys is generally slower than for other key types likes integer, as for string comparison it may be necessary to compare many characters to get a result and because strings generates some memory indirection by the fact that string content is stored somewhere in the heap outside of any cache.

Tuples or other containers as Keys

At the end of our introduction to hash tables we will present a very useful, but maybe not that obvious property of hash tables: The keys used for table access don’t have to be simple data types, but can be container types like tuples or arrays. Imagine you have a map in 2d with a set of points on that map each presented by an x, y coordinate pair. That points could be cities, and some cities may have a direct connection by a road. So how can we test if two cities are directly connected and get the distance between the two cities? With a hash table using a tuple of two city coordinates as key it is easy:

import tables, math

const
  InvalidFloat = 1e30 # arbitrary marker that in not a valid value
  InvalidCoord = (InvalidFloat, InvalidFloat)

type
  Coord = tuple
    x: float
    y: float

  Cities = Table[string, Coord]
  Distances = Table[(Coord, Coord), float]

var cities: Cities
var distances: Distances

proc insertCity(name: string; coord: Coord) =
  cities[name] = coord

proc insertDist(a, b: string) =
  var (a, b) = (a, b)
  if a > b: swap(a, b)
  let ca = cities[a] # caution, will raise an exception if name is not a know city
  let cb = cities[b]
  distances[(ca, cb)] = math.hypot(ca.x - cb.x, ca.y - cb.y)

proc checkDirectConnection(a, b: string): float =
  var (a, b) = (a, b)
  if a > b: swap(a, b)
  let ca = cities.getOrDefault(a, InvalidCoord)
  let cb = cities.getOrDefault(b, InvalidCoord)
  if ca == InvalidCoord or cb == InvalidCoord:
    return -1 # marker when cities are unknown
  result = distances.getOrDefault((ca, cb), InvalidFloat)

insertCity("aTown", (2.0, 7.0))
insertCity("bTown", (2.0, 11.0))
insertCity("cTown", (17.0, 23.0))

insertDist("aTown", "bTown")

var d = checkDirectConnection("aTown", "bTown")
if d == -1:
  echo "query for unknown town"
elif d == InvalidFloat:
  echo "Cities have no direct connection"
else:
  echo "Distance: ", d

echo checkDirectConnection("bTown", "aTown") # 4, same as above
echo checkDirectConnection("aTown", "cTown") # 1e30
echo checkDirectConnection("aTown", "xTown") # -1

For the procs insertDist() and checkDirectConnection() we use a trick to get the same results when we exchange the names: We sort the names alphabetically when we insert the distances and also when we query the distances. So we get the same result. Of course we could insert the tuple also twice instead, but as distances is the same in both direction sorting and inserting only ones makes some sense. Note that we used tuples for the coordinate pairs in the distances tables. Maybe the more obvious data type would be an array with two entries, as the array type is a container for homogeneous data, while a tuple can also contain different data types. But currently Nim supports tuples better in some situations, e.g. for automatic tuple unpacking. So often we use tuples when array would be the first choice maybe. Array type would have the benefit of iteration over elements at runtime, while tuples have the benefit that we can access elements by names and by an integer constant. For performance array or tuple should make no difference here.

References:

Part IV: Some Programming Tasks

In this section we will present a few simple programming exercises.

Sorting

Sorting a sequence or an array of numbers is generally a component of each computer programming course. While we would not really code sorting algorithm for the actual software that we write, but use the generic sorting algorithm from the standard library, sorting algorithm can teach us some basic programming skills. When we sort a small number of items manually, we would generally use selection or insertion sort intuitively: For selection sort we pick the smallest element and move it at position one, then pick next smallest item and move it at position two. This strategy is easy to implement and works not bad for small quantities. For larger containers algorithm like quicksort or mergesort gives better performance.

Selection Sort

#[ <--s.len
5
7 <-- i
4 <-- k, x == 4
6 <-- j
3
2 first tree entries are already sorted
1
]#

proc selectionSort(s: var seq[int]) =
  var i: int # used to step through the still unsorted range
  var j = 0 # lower bound for still unsorted range
  var k: int # position of currently smallest candidate
  var x: int # and its value

  while j < s.len: # while there is an unsorted section left
    i = j # start with i one above the already sorted range
    x = s[i]; k = i # assume first element is the smallest
    inc(i) # continue with next one in the still unsoprted range
    while i < s.len: # while there are unchecked candidates
      if s[i] < x: # that one is smaller than current candidate
        x = s[i] # remember its value
        k = i # and remember its position
      inc(i) # examine next candidate
    swap(s[j], s[k]) # exchange smallest value with the one currently at position k
    inc(j) # sorted range increased by one

import random
proc main =
  var s: seq[int]
  for i in 0 .. 9:
    s.add(rand(100))
  s.selectionSort
  echo s

main()

The comment on the top of the above example shows a partly sorted list of 7 integer numbers. The lowest three positions already contains the sorted numbers 1 to 3. The next four positions are still unsorted. For the sorting process we need the three indices i, j, k and the variable x to store an actual value for the comparison. The index j is the lower bound for the still unsorted range, j starts at zero obviously. The variable x stores the currently smallest value of the still unsorted range, and k is the index position of that value. Finally i is a counter that is used to step through all the values of the unsorted range. The outer loop is executed as long as j is smaller than the length of the sequence that we want to sort. We set i to the value of j, k to the same value and assume initially that s[i] is the smallest value from the still unsorted range. That value is stored in x. Then we execute the inner while loop until we have processed all elements of the still unsorted range. Whenever we find an element that is smaller than x, we store that position in k and the value in x. When the inner loop has finished, we exchange the smallest value with the first element of the still unsorted range. This way the sorted range increases, and the unsorted range decreases by one.

To test our sorting procedure we generate some random numbers, sort them and print the result. Selection sort is said to be of order O(n^2), with n being the number of values to sort. So the effort increases quadratic with the number of values. This is because we have to test all the still unsorted values just to increase the number of sorted values by one. Selection sort has a natural behavior, that is for an already sorted array the test s[i] < x would be always false and we would have to do no movement of values in that case. So performance is best for an already sorted or partly sorted list, and the sorting is stable in the sense that we do not move elements when it is not really necessary. In one of the following sections we will discuss the quicksort algorithm, which is not a stable sorting method: Elements with equal value may be move with quicksort. For plain numbers that does not really matter for the result, as numbers are indistinguishable. But when we sort objects, maybe persons by age, persons of same age would be exchanged by quicksort, which may not be desired.

Note that the code above is not really optimized for performance yet. One possible improvement may be to iterate the two loops not from zero to s.len, but in the opposite direction. In that way comparison of loop indices with a constant value, zero in this case, could be used to terminate the loop. Comparison with constants can be faster than comparison with actual variables, and comparison with zero is generally fastest. Note that we compared indices with s.len() in above code, which is not that bad, as len is a field in the seq data structure, so the compiler should be smart and replace s.len with just a field access without proc call overhead.

We started this section with thinking about how we would sort a small quantity of items lying on the table manually. This strategy is generally a a good one. Sometimes it can even help to ask how a child would solve a problem to find a way how to do it with the computer.

Insertion Sort

Insertion sort is another simple sorting method, which some card players like to use: They hold two sets of cards in their hand, one unsorted set, and one sorted which is initially empty. They pick one card from the unsorted set and insert it at the right position in the already sorted set. That action is repeated until the unsorted set is empty. For our next example we sort our data not in place as we did it in the previous example, but we generate a new sorted copy:

#[
unsorted     sorted
<-- k
4
7                     3 <-- result.high
5                     2
6                     1
]#

proc insertionSort(s: seq[int]): seq[int] =
  var j: int # current position in the new, sorted range
  var k = s.len # index one above the still unprocessed range
  var x: int # the value we have to insert next
  while k > 0: # as long as we have still unprocesed entries
    dec(k)
    x = s[k]
    j = result.high # top of sorted range
    result.setLen(result.len + 1) # reserve space for one more entry
    while j >= 0 and x < result[j]: # move the already sorted entries up
      result[j + 1] = result[j]
      dec(j)
    result[j + 1] = x # insert x

import random
proc main =
  var s: seq[int]
  for i in 0 .. 9:
    s.add(rand(100))
  echo s.insertionSort

main()

The commented code in front of the above program code shows at the left the still unsorted numbers and at the right 3 already sorted numbers. For the sorting process we need two index variables j and k, and one variable to store the actual value that we have to insert called x. The variable k is used as the index of the top entry of the still unsorted range. To insert the value x in the already sorted result, we first reserve space for one more entry by calling setLen() and then iterate over the sorted values and move them one place to the top. We do that moving to the top as long as we have not already reached the bottom of the sorted range and as long as the current entry is larger than the value x which we want to insert. We take the values from the top of the unsorted range, as that is convenient, but of course we could pick an arbitrary element from the unsorted range.

Insertion sort has O(n^2) cost, as for each element that we take from the unsorted range we have to iterate over the sorted range to insert it. As we have to move elements before we can insert an element, insertion sort is slow for larger containers. Selection sort, which is also of O(n^2) should be faster, as it does not use an expensive shift of many elements.

When you look at the example code you may immediately find two possible improvements: We do not really need the variable x, as we can just use s[k] instead. The compiler should optimize the code so that the subscript operator is not executed multiple time for the same index k, so use of s[k] or x should make no difference for performance. And we call setLen() in the outer loop to increase the capacity of the result sequence by one each. Of course setting capacity only one time to the value of s would suffer, as obviously result has the same length as out input data. Another possible optimization would be to take advantage of the fact that the destination sequence is sorted, so that we would not have to do linear search to find the insertion position, but we could use a binary search. But that would be more complicated and the benefit would be not large.

Quick Sort

As the name implies this sorting method is one of the fastest. We will explain it in some detail with various variants as it can teach us two important concepts: Recursion and avoiding recursion by use of a stack container.

The idea of the QuickSort algorithm is simple, and the code is also simple and short, but we have to care for some details like exact index stop positions. Generally sorting seems to be a O(n^2) operation, at least from the two traditional sorting methods, Insertion- and Selection-Sort it seems to be the case. So doubling the container size seems to increase the needed sorting time by a factor of four, which is really bad for large arrays or sequences. The trick of QuickSort is, that instead of sorting a container with n elements, we just sort the first half and the second half separately, each with approx n / 2 entries. This would be faster, as 2 * (n/2)^2 is only half of (n)^2. And we do apply this trick in a recursive manner on each halved range, until the range is reduced to only one or two entries. But to make this work we have to partition the full range in the first half r1 so that all entries of range r1 are smaller or equal to a median value x, and so that all entries in the second half r2 are all greater or equal to the median x. Let us consider an example with six numbers:

5 3 2 8 1 7
1 3 2 8 5 7

To partition that set of numbers we have only to exchange the numbers 5 and 1, with the value 5 or 4 being a possible median. Exchanging numbers in an array or a seq is a fast O(n) operation, we have to iterate the container only once. The problem is finding the median. For picking a perfect median we would need a sorted container so that we could pick the center entry. But of course our container is unsorted, if it would be sorted our work would be done already. Note that even summing up all entries and dividing by n would give only the average value, not the median. Average and median can be very different, e.g. for many small numbers and a few very large ones. But in practice picking an estimation for the median, maybe picking one by random from the whole range or picking the center entry is good enough. That choice will not really half the whole range in each step, but on average it splits the range in two parts with not too different size. This works really well when the input data looks like random numbers, but it may work bad in some unlikely cases when all the input numbers are equal or are already sorted. Already sorted can indeed occur — for that case picking the center elements of the range gives the perfect median, so we will choose that strategy.

Partitioning the full range is basically very simple: We move from the left to right with index i and stop when we find a value s[i] that is not smaller than our median x. And we do the same from the right to the left with index j until we find a value that is not greater than median x. After we have done that, we can just exchange the values at s[i] and s[j] and continue. We continue until i is close to j. The difficult part is to handle this terminating condition exactly, that is to stop exactly at the right position so that the first half really contains all entries with value less or equal to median x, and that the second half contains only entries with value equal or greater than median x. To make it more clear: Such a partition decouples the two ranges. Sorting the whole range would result in the same state as sorting first and second range on its own.

For writing our actual sorting function we use the fact that Nim like most modern programming languages support recursion, that is a function can call itself again. We saw at the beginning of the book a few examples for that. So we can pass our function the initial full container, then the function can partition the container in first and second half and call itself again on the two parts. This way the actual size of the ranges to sort decreases, and finally recursion stops when the range contains only one or two elements. For size one we have to do nothing at all, but for size two we may have to exchange the two elements if order is wrong.

A naive implementation may create for each function call two new sequences for the two parts, partition the initial sequence by inserting the values in one of the new shorter sequences, call itself on both parts and finally join the parts and return it — as result or as var parameter. But that would be really slow. A much simpler and faster solution is, when we work all the time on the same container, and just tell the function which range the function has to work on. So we pass to the function the whole sequence and two integers a and b which tell it the range to process: s[a], s[a +1] .. s[b].

from algorithm import isSorted, sort
import std/monotimes

proc qsort(s: var seq[int]; a, b: int) =
  assert a >= 0 # a .. b is the range that we have to sort
  assert b < s.len
  assert b - a > 1 # it may work for smaller intervals, but this is the intended use case
  let x = s[(a + b) div 2] # use element from center of range
  # var x = s[a] div 2 + s[b] div 2 # bad, x may be smaller than smallest entry in range
  # x = s[a .. b].min # worst case test!
  var i = a
  var j = b
  while true:
    while s[i] < x:
      inc(i)
    while s[j] > x:
      dec(j)
    if i < j:
      swap(s[i], s[j])
      inc(i)
      dec(j)
    else:
      break
  dec(i)
  inc(j)
  assert i >= a
  assert j <= b
  if i - a > 1: # still more than 2 entries
    qsort(s, a, i)
  elif i - a > 0: # two entries
    if s[i] < s[a]: # wrong order
      swap(s[i], s[a])
  if b - j > 1: # and the same for the other half
    qsort(s, j, b)
  elif b - j > 0:
    if s[b] < s[j]:
      swap(s[b], s[j])

proc quickSort(s: var seq[int]) =
  if s.len == 2 and s[0] > s[1]: swap(s[0], s[1])
  if s.len > 2:
    qsort(s, 0, s.high)

import random
proc main =
  var s: seq[int]
  var start: MonoTime
  randomize() # give us different random numbers for each program run
  for i in 0 .. 1e5.int:
    s.add(rand(1e8.int))
  for i in 0 .. 9:
    s.shuffle
    s.quickSort
    assert(isSorted(s))
  for i in 0 .. 1e7.int:
    s.add(rand(1e8.int))
  start = getMonotime()
  s.quickSort
  echo getMonotime() - start
  assert(isSorted(s))
  s.shuffle
  start = getMonotime()
  s.sort()
  echo getMonotime() - start
main()

The function qsort() does the whole work. It is called from function quicksort() passing it the whole sequence and the interval to sort. For the first call the interval is the whole content of the seq, from s.low to s.high. Function qsort() first asserts that the range is valid, that is that b > a and that both indices are valid positions in the sequence s. That check makes it easier to find stupid errors, the assert is automatically removed when we compile finally with -d:release. We set the iterating indices i and j to the interval boundaries a and b and enter an outer loop. In that outer loop we let run i and j to the center of the interval as long as the actual entry at the position i or j belongs in the range. If both inner loops have stopped, we swap the entries at position i and j. As positions i and j contain now again valid entries, we can move both indices one step further to the center. If i and j becomes the same we are done. Unfortunately both may stop too late, so we move both one position back after the outer loop has terminated. You may create a small example with pen and paper to recognize how the indices behave in detail and why fixing by one is necessary. The remainder of the qsort proc is really easy: For both partitions we check if the interval size is still larger than two entries, in that case we call qsort() again to continue with partition and sorting. But if the size is two entries, then we just swap() them if the order is wrong. If the size of the range is just one, we have nothing to do at all.

We test our quicksort proc by calling it from another proc called main. In that main() we fill a seq with random integer values, and shuffle() and sort() it a few times. Shuffle() reorders the entries by random. After our call of quicksort() we call isSorted() from Nim’s standard library to check the success of our sorting. After these tests, which does some warmup of the CPU for us, we add more random entries and again sort and test it, while we record the needed time with module monotimes as we did before in the Timers section. To get a feeling about the performance of our sorting proc we shuffle() again and sort this time with the sort() proc from Nim’s algorithm module. Sort() from algorithm module uses currently another sorting method called merge sort, which has the advantage that it is a stable sorting algorithm, but it may be a bit slower than quicksort. And sort() from algorithm may pass a cmp() proc around, which may cost some performance, while our plain, non generic proc compares entries directly with < and > operators. So it is not surprising that our proc is a bit faster.

You may wonder if it is really necessary to pass the sequence s for each call of qsort(), as for all time the same seq is used. Indeed Nim support nested procs, so we could just make qsort() proc local to quicksort() proc and let qsort work (as a closure) on the s variable of proc quicksort. But this does not compile currently, sequences can not be used by closure procs. But actually passing the sequence to proc qsort() should be only a minimal overhead.

One general problem of QuickSort is that the sort is not stable. When we sort an already sorted sequence again, entries with same value may move. For plain numbers that is not really a problem, we do not really notice it, as we can’t mark a number in some way, plain numbers are indistinguishable just as elementary particles like electrons and protons are. But when we sort a container with objects by some field, then we notice that objects with same value for sorting may move. The other problem of QuickSort is a generally problem of recursive algorithm: Each new call of a proc generates some stack usage, as proc parameters may be passed on the stack and because the proc may allocate its local data variables on the stack. So many nested calls may need a very large stack, the program may fail with a stack overflow error. Generally, we have no real problem with stack overflow, as for each partition the size of the two new partition is nearly halved, so that process stops soon. But imagine someone prepares a special data set for our sort proc. That data may be prepared in such a fashion that at the center of each range, where we pick the estimated median value from, always an extreme values is stored. So our partition would work very badly, in each step we would get a new range with only one element, and one with n - 1 elements. So the recursion dept would go very deep, and the performance would be very bad also. Preparing such a data set would be difficult, but possible in theory. One way to protect us from that attack would be to select the median by random. But unfortunately all strategies different from picking the leftmost, the center, or the rightmost entry as median are not very fast and make the whole sorting significantly slower. Note that the strategy of not picking a single element as median, but calculating a median value, works generally, but has some shortcomings: s[a] div 2 + s[b] div 2 would not work when both values are odd, as we then would get a value that can be smaller than all of our entries and our function would fail. We would have to add one to the average value when both summands are odd, and that fix cost performance again. And calculating the average by (s[a] + s[b]) div 2 could generate an overflow when both summands are large.

Because of the stack size restrictions we have a good motivation to show how we can replace recursion with plain iteration, when we provide a "buffer" variable that acts as a data stack. For each new partition of our data we have to put only the two bounds a and b on that data stack, which is not that much as a recursive proc would put on the real computer stack. The modifications to our code from above are tiny:[28]

from algorithm import isSorted, sort
import std/monotimes

proc qsort(s: var seq[int]) =
  var stack: seq[(int, int)]
  var maxStackLen: int
  stack.add((s.low, s.high))
  while stack.len > 0:
    if stack.len > maxStackLen:
      maxStackLen = stack.len
    var (a, b) = stack.pop
    assert(a >= 0 and b < s.len and b - a > 1)
    let x = s[(a + b) div 2]
    var (i, j) = (a, b)
    while true:
      while s[i] < x:
        inc(i)
      while s[j] > x:
        dec(j)
      if i < j:
        swap(s[i], s[j])
        inc(i) ; dec(j)
      else:
        break
    dec(i); inc(j)
    # assert(i >= a and j <= b) caution, this is not always true!
    if i - a > 1:
      stack.add((a, i))
    elif i - a > 0:
      if s[i] < s[a]:
        swap(s[i], s[a])
    if b - j > 1:
      stack.add((j, b))
    elif b - j > 0:
      if s[b] < s[j]:
        swap(s[b], s[j])
  echo "Max Stack Length: ", maxStackLen

proc quickSort(s: var seq[int]) =
  if s.len == 2 and s[0] > s[1]: swap(s[0], s[1])
  if s.len > 2:
    qsort(s)

import random
proc main =
  var s: seq[int]
  var start: MonoTime
  randomize()
  for i in 0 .. 1e5.int:
    s.add(rand(1e8.int))
  for i in 0 .. 9:
    s.shuffle
    s.quickSort
    assert(isSorted(s))
  for i in 0 .. 1e7.int:
    s.add(rand(1e8.int))
  start = getMonotime()
  s.quickSort
  echo getMonotime() - start
  assert(isSorted(s))
  s.shuffle
  start = getMonotime()
  s.sort()
  echo getMonotime() - start
main()

We added a variable called stack which is a seq which stores integer tuples. The qsort proc first stores the borders of the whole seq on the stack, and then executes a loop which takes in each iteration a set of two borders from the stack and processes that range. It may sound a bit strange that we start by putting the whole range onto the stack and then took it from stack immediately at the start of the loop. But that makes more sense when we look at the bottom of the qsort() proc. Instead of recursively calling qsort() again, we just put the borders of the two new partitions on the stack and continue. The whole process terminates when the stack becomes empty, as then all partitions are processed. Note that the actual partition code and the main() proc are still unchanged. We added a maxStackLen variable to get a feeling how large our stack has to be. Actually not that large, as the partition size shrinks in a logarithmic way. So we could replace the seq that we use now as stack with a plain array, as sequences have same overhead and the add() is slower than plain index access. But how can we prepare for worst case attacks? Indeed there exists a simple solution: Worst case occurs, when first we put a tiny one element range on the stack and then the large one, as we would continue with the large one in the same way in the next loop iteration. The other way round would be fine. If we put the tiny range on the stack last, next iteration would pick that one and iteration would stop immediately or at least very soon, as ranges drops to two or one entries. When an iteration for a range stops, all ranges pushed to the stack are removed already again, so total stack size will never become large. So the trick is to just sort the partitions in a way that we put the larger partition first on the stack, and the smaller partition second. So next iteration picks the smaller one and the whole process stops soon. This way a stack array of 64 entries should be enough, as max needed stack size should be log2(2^64) to sort a seq with 2^64 entries.

from algorithm import isSorted, sort
import std/monotimes

proc qsort(s: var seq[int]) =
  var stack: array[64, (int, int)]
  var stackPtr: int
  var maxStackLen: int
  stack[0] = (s.low, s.high)
  while stackPtr >= 0:
    if stackPtr > maxStackLen:
      maxStackLen = stackPtr
    let (a, b) = stack[stackPtr]; dec(stackPtr)
    assert(a >= 0 and b < s.len and b - a > 1)
    let x = s[(a + b) div 2]
    var (i, j) = (a, b)
    while true:
      while s[i] < x:
        inc(i)
      while s[j] > x:
        dec(j)
      if i < j:
        swap(s[i], s[j])
        inc(i); dec(j)
      else:
        break
    dec(i); inc(j)
    # assert(i >= a and j <= b) caution, this is not always true!
    var c, d: int
    for u in 0 .. 1:
      if (i - a > b - j) == (u == 0):
        (c, d) = (a, i)
      else:
        (c, d) = (j, b)
      if d - c > 1:
        inc(stackPtr) # inc before push!
        stack[stackPtr] = (c, d)
      elif d - c > 0:
        if s[c] > s[d]:
          swap(s[c], s[d])
  echo "Max Stack Length: ", maxStackLen

proc quickSort(s: var seq[int]) =
  if s.len == 2 and s[0] > s[1]: swap(s[0], s[1])
  if s.len > 2:
    qsort(s)

import random
proc main =
  var s: seq[int]
  var start: MonoTime
  randomize()
  for i in 0 .. 1e5.int:
    s.add(rand(1e8.int))
  for i in 0 .. 9:
    s.shuffle
    s.quickSort
    assert(isSorted(s))
  for i in 0 .. 1e7.int:
    s.add(rand(1e8.int))
  start = getMonotime()
  s.quickSort
  echo getMonotime() - start
  assert(isSorted(s))
  s.shuffle
  start = getMonotime()
  s.sort()
  echo getMonotime() - start
main()

Instead of processing the two new partitions at the end of the qsort() proc each, we apply only one processing code block now, which we execute in a loop that is executed two times. At the start of that loop we assign the actual interval boundaries to the variables c and d. That assignment depends on the actual loop index u, so that we push the larger range always first on the stack. You may modify the condition u == 0 to u != 0 and observe what happens to the maximum used stack dept. We could write that condition also with a boolean loop variable and a xor operator like

    for u in [false, true]:
      if (i - a > b - j) xor u:

We should not believe all what we think

And what seems to be correct. Our nonrecursive function seems to be fine, and indeed inverting the == (u == 0) condition makes a difference for random data, so it is correct? Well when we think about it again the next day we may get some doubts. The outer loop pops one entry from the stack, but in the loop we may push two new entries. Pushing the smaller interval helps, as we continue with the smaller interval in the next iteration and remove it so from the stack. But the net effect is still that we push one interval onto the stack for each iteration, and for the worst case that interval shrinks only by one in each iteration. So it should still not work.

But well, there are rumors that solutions exists. When we think about it, we may ask our self if we can just continue with one interval in a loop and push only the other one on the stack. And indeed, that is possible, and this time we did testing for worst case scenario:

from algorithm import isSorted, sort
import std/monotimes

proc qsort(s: var seq[int]) =
  var stack: array[64, (int, int)]
  var stackPtr: int = -1 # empty
  var maxStackLen: int
  var a = s.low
  var b = s.high
  while true:
    if b - a == 1: # done with actual interval, but we may have to swap()
      if s[a] > s[b]:
        swap(s[a], s[b])
    if b - a > 1: # interval has still more than two entries, so continue
      discard
    elif stackPtr >= 0: # get next interval from stack
      (a, b) = stack[stackPtr]; dec(stackPtr)
    else:
      break # all done
    if stackPtr > maxStackLen:
      maxStackLen = stackPtr
    assert(a >= 0 and b < s.len and b - a > 1)
    let x = s[(a + b) div 2]
    # let x = s[a .. b].max # worst case test! Slow, test with smaller container size.
    var (i, j) = (a, b)
    while true:
      while s[i] < x:
        inc(i)
      while s[j] > x:
        dec(j)
      if i < j:
        swap(s[i], s[j])
        inc(i); dec(j)
      else:
        break
    dec(i); inc(j)
    # assert(i >= a and j <= b) caution, this is not always true!
    if (i - a < b - j): # put large interval on stack and cont. directly with the small
      swap(i, b)
      swap(a, j)
    if i - a > 1: # interval has more than two entries, needs further processing
      inc(stackPtr) # inc before push!
      stack[stackPtr] = (a, i)
    elif i - a > 0: # two entries, we may have to swap()
      if s[a] > s[i]:
        swap(s[a], s[i])
    (a, b) = (j, b) # the smaller interval, we continue with that one
  echo "Max Stack Length: ", maxStackLen

proc quickSort(s: var seq[int]) =
  if s.len == 2 and s[0] > s[1]: swap(s[0], s[1])
  if s.len > 2:
    qsort(s)

import random
proc main =
  var s: seq[int]
  var start: MonoTime
  randomize()
  for i in 0 .. 1e5.int:
    s.add(rand(1e8.int))
  for i in 0 .. 9:
    s.shuffle
    s.quickSort
    assert(isSorted(s))
  for i in 0 .. 1e7.int:
    s.add(rand(1e8.int))
  start = getMonotime()
  s.quickSort
  echo getMonotime() - start
  assert(isSorted(s))
  s.shuffle
  start = getMonotime()
  s.sort()
  echo getMonotime() - start
main()

The modifications to the code are again tiny. We use an outer while true: loop, which continues with interval a .. b until its size is less than three entries. Then its pops() a new interval from the stack. At the end of that outer loop, we put only one interval on the stack and directly continue with the other interval. But which interval should we push on the stack and which one should we process directly further in the outer loop? The solution is to process the smaller interval in the outer loop further, as we are soon done with it. For processing that smaller interval, we may push some more ranges onto the stack, but we come to interval sizes of less than three soon and then we start popping intervals from the stack. And when we are done with that, we pop the larger interval again from the stack. This way the worst case, where we pick each time a min or max value as median, has the smallest stack consumption, that is one entry. We push the large interval with size n - 1 on the stack, continue with the tiny one entry range, which signals that we are done with that interval in the next loop iteration and so the just pushed n - 1 interval is popped from the stack again. This continues in this way. Slow, but minimal stack consumption.

From the above code it becomes clear that for our initial recursive qsort() function changing the order in which we process the partitions would not really help, as we continue the recursion until all is processed. There is no intermediate pop() involved.

Maybe you still wonder why the tiny inner loops use the conditions while s[i] < x: and not while s[i] <= x: as we said that both partitions are allowed to contain the median element. Well with <= there would be no guaranteed stop condition for the interval, so indices could run out of the interval. Using an additional condition like and i <= b would make it slower. Another possible modification would be to not use inner while loops at all. Tiny while loops with only one simple termination condition are fast, but the inner while loops would always terminate fast for random data. So we may try instead something like

    while true:
      if s[i] < x:
        inc(i)
      elif s[j] > x:
        dec(j)
      else:
        if i < j:
          swap(s[i], s[j])
          inc(i); dec(j)
        else:
          break
    dec(i); inc(j)

You may try that variant yourself, or maybe look for other variants in internet sources or textbooks. Our intention in this section was not to present a perfect sorting function, but to teach you some basic coding strategies and related traps.

Merge Sort

Initially we did not intend to discuss the actual MergeSort algorithm at all, as it is a bit more complicated and whenever we may have seen a sketch of it somewhere, it is generally not easy to remember details.[29] But MergeSort is indeed an important algorithm, it is used by default in Nim’s standard library and as we have discussed QuickSort already in some detail, we should be prepared for MergeSort now. When we regard the name Merge, which is some form of joining multiple sources to one destination, we may begin to remember the idea of MergeSort: The trick of QuickSort was, that we tried to split in a recursive manner the set of all container elements into two subsets, which we can process separately. That improves performance, as sorting is basically an O(n^2) process, and 2 * (n/2)^2 is only half of n^2. For QuickSort we partitioned the initial range into two ranges a and b, where all elements of range a are less or equal to a median element x, and all elements of range b are greater or equal to the median x. That way we decoupled the two ranges, we can sort a and b independently and get a fully sorted range. MergeSort starts also with splitting the full range into two parts, but it really only splits, without any form of rearrangement. Then it continues with sorting each part independently. That sounds strange at first, as we get two sorted parts a and b, but of course we can not simple append one to the other. The idea of the whole algorithm become clear immediately when we think about how we can find the smallest elements of the joined content from a and b. That one is obviously the smallest value of a, or the smallest value of b, min(a, b) = min(min(a), min(b)). But when a and b are already sorted, then the minimal value of each is the first element, and so one of these elements at index position zero is the smallest one for a and b joined. And this condition holds even when we pick and remove the smallest element from a or b.

So the basic algorithm is this: Split the whole container in parts a and b, sort them separately. Then create a new, sorted container by iterative picking the first elements from a or b, which ever is actually the smaller one.

Unfortunately it seems to be impossible to sort the initially container in place in this way, as we would take always elements from the front from both and put them at the front of the destination, so that newly inserted elements could overwrite still unprocessed elements. So we will try to give a sketch of an very slow unoptimized algorithm which creates and returns a new sorted container first to lean the fundamental idea of the algorithm:

proc msort(s: seq[int]; a, b: int): seq[int] =
  assert(b - a >= 0)
  if b - a == 0:
    result.add(s[a])
    return
  elif b - a == 1:
    var (a, b) = (s[a], s[b])
    if a > b:
      swap(a, b)
    result.add(a)
    result.add(b)
    return
  result = newSeq[int](b - a + 1)
  var sl = result.len
  assert b - a > 1
  var m = (a + b) div 2
  assert m >= a
  assert m < b
  var s1, s2: seq[int]
  s1 = msort(s, a, m)
  var (i, j) = (a, m)
  s2 = msort(s, m + 1, b)
  assert s1.len + s2.len == result.len
  var (k, l) = (m + 1, b)
  var l1 = s1.high
  var l2 = s2.high
  while sl > 0:
    dec(sl)
    if l1 >= 0 and l2 >= 0: # merge
      if s1[l1] > s2[l2]:
        result[sl] = s1[l1]
        dec(l1)
      else:
        result[sl] = s2[l2]
        dec(l2)
    else: # plain copy
      while l1 >= 0:
        result[sl] = s1[l1]
        dec(sl); dec(l1)
      while l2 >= 0:
        result[sl] = s2[l2]
        dec(sl); dec(l2)

proc mergeSort(s: seq[int]): seq[int] =
  if s.len < 2:
    return s
  msort(s, s.low, s.high)

import random, algorithm
proc main =
  var s, st: seq[int]
  randomize()
  for i in 0 .. 9:
    s.add(rand(100))
  st = s
  echo s.mergeSort
  assert(s.mergeSort == st.sorted)

main()

The example above is indeed not very complicated, the most daunting task is to get all the indices right. Due to the involved recursion and the fact that we have to do the sorting of the two parts first, before we can join them, debugging would be not easy. So we added many asserts to early find stupid errors.

The function mSort() starts by checking if the range to sort has only one or two entries and handles this simple case directly. Then we allocate the result sequence, find the center position m of the interval a and b and sort the intervals a .. m and m + 1 .. b each into a new sequence s1 and s2. We have decided that we will do the merging of s1 and s2 into the result sequence from the back, starting with the largest elements. This way we can count down to zero, which is a bit faster and simpler. We do an actual merging as long as s1 and s2 have still elements left. An we do the merge from end to start, we have to always pick the largest element from s1 or s2. If we have used all the elements from at least one of the sequences s1 or s2, then there is nothing more to merge, we can just copy the remaining elements from the other seq that has elements left. Of course the above example is very slow, as we allocate the result and additional the sequences s1 and s2, and as we have to copy many elements.

When we now think again about the problem, we get the feeling that allocating three sequences is really too much. When we regard both, in place sorting and sorting with a return value, we may discover that in place sorting allows us to partly reuse the passed container, and we have only to allocate one additional seq with half the size of the passed container. We copy the second half of the passed container into a newly allocated seq, and then can merge values from the new seq and from the first half of the passed seq to positions starting at the end of the passed seq, without overwriting values that we still have to process.

This way our code becomes even shorter and basically simpler — we have only to care very exactly to use the right index positions.

proc msort(s: var seq[int]; a, b: int) =
  assert(b - a >= 0)
  if b - a == 0:
    return
  elif b - a == 1:
    if s[a] > s[b]:
      swap(s[a], s[b])
    return
  var m = (a + b) div 2
  assert(m >= a and m < b and b - a > 1)
  var sh: seq[int] = s[(m + 1) .. b]
  var ls = b + 1
  msort(s, a, m)
  msort(sh, sh.low, sh.high)
  var lh = sh.high
  var lm = m
  while ls > a:
    dec(ls)
    if lh < 0:
      assert ls == lm
      break
    elif  lm < a:
      while lh >= 0:
        s[ls] = sh[lh]
        dec(ls); dec(lh)
    else:
      if sh[lh] > s[lm]:
        s[ls] = sh[lh]
        dec(lh)
      else:
        s[ls] = s[lm]
        dec(lm)

proc mergeSort(s: var seq[int]) =
  msort(s, s.low, s.high)

import random, algorithm, std/monotimes
proc main =
  var s: seq[int]
  var start: MonoTime
  randomize()
  for i in 0 .. 1e5.int:
    s.add(rand(1e8.int))
  for i in 0 .. 9:
    s.shuffle
    s.mergeSort
    assert(isSorted(s))
  for i in 0 .. 1e7.int:
    s.add(rand(1e8.int))
  var st = s
  start = getMonotime()
  s.mergeSort
  echo getMonotime() - start
  start = getMonotime()
  st.sort()
  echo getMonotime() - start
  assert s == st
main()

For the merging process, we first test if one of the source areas is already exhausted. If all entries from the newly allocated seq sh are consumed, then we can just stop, as continuing would only copy elements of the passed container with equal index positions. And if all elements from the first half of the passed container are consumed, we can just copy the elements from sh into the passed var container. Only if both sources have elements to process left, we have to do the actual merging.

When we run the program above, we may find that it is about 50% slower than our QuickSort functions. The reason for that may be that we have to allocate the temporary seq sh, fill it with values, and merge it back. And the total memory consumption is high, our recursive function calls consumes for the buffers sh totally the same amount as the initial container. The advantages of MergeSort is that there is no worst case as for QiuckSort, as we have not to select a median but can split the range just at the center, and that the sort is stable, that is merging does not exchange the position of elements with same value.footnote[ Well to ensure this, we may have to check the actual merge condition, do we have to test for < or ⇐. Currently we do not really care for that, but is is clear and well known that merge sort can be stable.]

When we think a bit more about the algorithm above and maybe try to sketch the recursive steps for a short sequence with pencil and paper we get the strong feeling that the additional buffer sh is only needed for the merging step, and as the merging process occurs from bottom to top (containers with only one or two entries are returned immediately, which are merged to a larger section, and these larger section is again merged …​) one single buffer could be used. So we have modified our example again. Now the quicksort() proc allocates the buffer seq with half the size of the actual data container, and we pass that buffer recursively to the qsort proc and use it for the merging only. We call recursively qsort() on the first and second half of the full range that we have to sort, then copy the sorted second half into the buffer and merge the fist half and the buffer into the final location.

proc msort(s, sh: var openArray[int]; a, b: int) =
  assert(b - a >= 0)
  if b - a == 0:
    return
  elif b - a == 1:
    if s[a] > s[b]:
      swap(s[a], s[b])
    return
  var m = (a + b) div 2
  assert (m >= a and m < b and b - a > 1)
  msort(s, sh, a, m)
  msort(s, sh, m + 1, b)

  var ls = b + 1
  var lh = b - m - 1
  var lm = m
  #sh[sh.low .. lh] = s[m + 1 .. b]
  for i in 0 .. lh: # a bit faster
    sh[i] = s[m + 1 + i]
  while ls > a:
    dec(ls)
    if lh < 0:
      assert ls == lm
      break
    elif  lm < a:
      while lh >= 0:
        s[ls] = sh[lh]
        dec(ls); dec(lh)
    else:
      if sh[lh] > s[lm]:
        s[ls] = sh[lh]
        dec(lh)
      else:
        s[ls] = s[lm]
        dec(lm)

proc mergeSort(s: var seq[int]) =
  if s.len == 0: return
  var sh = newSeq[int](s.len div 2)
  msort(s, sh, s.low, s.high)

import random, algorithm, std/monotimes
proc main =
  var s: seq[int]
  var start: MonoTime
  randomize()
  for i in 0 .. 1e5.int:
    s.add(rand(1e8.int))
  for i in 0 .. 9:
    s.shuffle
    s.mergeSort
    assert(isSorted(s))
  for i in 0 .. 1e7.int:
    s.add(rand(1e8.int))
  var st = s
  start = getMonotime()
  s.mergeSort
  echo getMonotime() - start
  start = getMonotime()
  st.sort()
  echo getMonotime() - start
  assert s == st
main()

An additional tiny performance improvement results from the fact that we now pass the seq s and the buffer sh as openArrays. This is generally a good idea, as we can so sort arrays also with the same sorting proc, and it improves performance, as this way the actual data buffer is directly passed to the qsort() proc, while passing a seq means that we pass the opaque seq structure which contains a pointer to the actual data. In the example above we do not only call isSorted() to prove our result, but we really sort a copy of our data with a sorting routine from Nim’s standard lib to ensure that our result is not only sorted data, but that it is indeed based on the actual values. That is a good idea, because although the algorithm is simple, getting some indices wrong may give us wrong results.

Our recursive merge sort routine is really not that bad. It does a fast stable sort and needs only a single buffer of half the size of our actual data. As the interval size is halved in each recursion step, the max recursion depth should be only 64 for a gigantic container with 2^64 elements. As the recursion occurs in a dept first fashion, that is msort() calls itself until range size is only one or two elements, then recursion continues in other branches of the whole sorting three, there should be never more than log2(n) actual recursion steps stored on the CPU stack. Non recursive, iterative merge sort algorithm exists, but converting the recursive algorithm in an iterative one is not that simple as for the QuickSort case. The reason is that qsort() has first to partition the input range, then call msort() on both subranges and finally do the merging. We will not try to present in this book an iterative msort(), which you may find in textbooks or somewhere in the internet, as that would be a bit too much for an introducing course.

We did the QuickSort and the MergeSort in a top down fashion, that is we split the initial container in two subpartitions and continue in this way until we have only ranges with one or two entries, and than do the merging from bottom to top. For MergeSort we could just start from the bottom joining single adjacent elements to sorted tuples of two entries, when done with that merging the tuples of two to sorted tuples of 4 and so on. This would work really well when the initial number of elements in our container is a power of two, and it would work well iterative without recursion. Unfortunately in most cases the container size is not a power of two, so such a bottom up merge sort needs some math to get all the ranges sizes right. But the bottom up process has a big disadvantage on modern hardware, as it has no locality for element access operations: We would iterate repeatedly over all the container entries in sequential order, so the CPU cache can not support our element access operation that well.

Other known sorting algorithms are the easy, funny and slow BubbleSort, or Shell- and Shaker-Sort. But these are not used in practice. As an exercise you can try to make our QuickSort or MergeSort generic and pass a cmp() proc, and make it work for sorting in ascending and descending order. Or, you may try to fall back to selection sort when the partitions become small. In theory SelectionSort is faster for ranges of only a few dozen elements, but when we have to do a decision which one to use inside of the qsort() or msort() proc, then this decision compensates generally the advantages again, so that the net benefit is tiny. Of course we would have to test all of our sorting procs for special cases, that is for seqs of length 0, 1 or two, and for sequences with all entries equal, all inversely sorted or presorted. And we would have to check how performance is when we sort not containers containing plain data like numbers, but containers which elements are objects, strings or again arrays or sequences. String sorting is special for various reasons: Strings in Nim are an opaque object with a pointer to the actual data. This is some indirection, and the actual data can be located somewhere in the RAM in a cache unfriendly manner, so the actual comparison process can be slow. Swapping of strings is also special, as swap() generally just does a pointer exchange for the data areas, and does not have to copy the actual data. For sorting containers where each entry is an array (of characters), swap would have to copy the data content.

Finally we should mention that the Python language uses a complicated sorting algorithm called TimSort which is a smart mix of various sorting algorithm.

References:

Part V: Advanced Nim

In this part of the book we will try to explain the more difficult parts of the Nim programming language: Macros and meta-programming, asynchronous code, threading and parallel processing, and finally the use of Nim’s concepts. We will start with macros and meta-programming, as that seems to be a really stable part of Nim’s advanced features. Nim’s concepts just got a redesign, and for the use of asynchronous code, threading and parallel processing there exists currently various implementations, and all that may change again when the Nim core devs should decide to actually use the CPS (Continuation-Passing Style) based programming style for the implementation of this.

Macros and Meta-Programming

Introduction

In computer science a macro (short for "macro instruction") is a rule or pattern that specifies how a certain input should be mapped to a replacement output. Meta-programming is a programming technique in which computer programs have the ability to treat other programs as their data. It means that a program can read, generate, analyze or transform other programs, and even modify itself while running.

Legacy programming languages like C or assembly languages support already some form of macros, which generally work directly on the textual representation of the source code.

A common use of textual macros in assembly languages was to group sequences of instructions, like reading data from a file or from the keyboard, to make that operations easily accessible. The C programming language uses the #define pre-processor directive to introduce textual macros. Macros in C are generally single line text substitutions which are processed by a pre-processor program before the actual compiling process. Some examples of common C macros are

#define PI 3.1415
#define sqr(x) (x)*(x)

The basic C macro syntax is that the first continues character sequence after the #define directive is replaced by the C pre-processor with the rest of that line. The #define directive has some basic parameter support which was used for the sqr() macro above. C macros have the purpose to support named constants and to support simple parameterized expressions like the sqr() from above, avoiding the need to create actual functions. The C pre-processor would substitute each occurrence of the string PI in the C source file with the float literal 3.1415 and the term sqr(1+2) with (1+2)*(1+2).

A Nim macro is a function that is executed at compile-time and transforms a Nim syntax tree into a different tree. This can be used to add custom language features and implement domain-specific languages (DSL). While macros enable advanced compile-time code transformations, they cannot change Nim’s syntax.

The macro keyword is used similar to proc, func and template to define a parameterized code block which is executed at compile time and consists of ordinary Nim code and meta-programming instructions. The meta-programming instructions are imported from the macros module and are used to construct an Abstract Syntax Tree (AST) of the Nim language. This AST is created by the macro body at compile time and is returned by the macro as untyped data type. The parameter list of macros accept ordinary (static) Nim data types and additional the data types typed and untyped, which we already used for templates. We will explain the differences of the various possible data types for macro parameters later in more detail, after we have given a first simple macro example. Note that Nim macros are hygienic by default, that is symbols defined inside the macro body are local and do not pollute the name space of the environment. As macros are executed at compile time, the use of macros may increase the compile time, but their use does not impact the performance of the final executable — in some cases the use of clever macros may even improve the performance of our final program.

Macros are by far the most difficult part of the Nim programming language. While in languages like Lisp macros integrate very well into the language, for Nim the meta-programming with macros is very different to the use of the language itself. Nim macros are very powerful — the current Nim async implementation is based on Nim macros, and some advanced libraries for threading, parallel processing or data serialization using JSON or YAML file formats make heavily use of Nim macros. And many modules of the Nim standard library provide some macros which extends the power of the Nim core language. The famous with macro of the equality named module is only one example for the usefulness of Nim’s macros. And same small but important parts of the high level GTK bindings are created with macros, e.g. the code to connect GTK callback functions to GTK signals. But this means not, that each Nim user really has to use macros. For a few use cases we really need macros, for other use cases macros may make our code shorter, maybe even cleaner. But at the same time the use of macros can make it for other people harder to understand the code, at least when we use exotic or our own complicated macros. And learning advanced Nim macro programming is not that easy. Nim macros have some similarity to the programming language C++: When we follow the explanations in a C++ text book then the C++ language seems to be not extremely difficult and even seems to follow a more or less logical design. But then, when we later try to write some actual code in C++ we notice that actually using the languages is hard as long as we have not a lot of practice. For Nim macros it is similar — when we follow a talk of an experienced Nim programmer about macro programming or when we read the code of an existing macro written by the Nim core devs, then all seems to be not that hard. But when we try to create macros our own for the first time, it can be frustrating. Strange error messages, or even worse no idea at all how we can solve a concrete task. So maybe the best start with macros is to read the code of existing macros, to study the macros module to see what is available, and maybe to follow some of the various tutorials listed at the end of this section. And finally you would have to ask for help in the Nim forum, on IRC or the other Nim help channels.

To verify that macros are really executed at compile time, we will start with a tiny macro that contains only an echo statement in its body:

import macros

macro m1(s: string): untyped =
  echo s

proc main =
  echo "calling macro m1"
  m1("Macro argument")

main()

When we compile above code the compiler prints the message "Macro argument", as it processes the macro body. When we run the program, we get only the output "calling macro m1" from the main() proc, as the macro m1() does return an empty AST only. The carefully reader may wonder, why the echo statement in the macro body above works at all, as the parameter of macro m1() is specified as ordinary string, not as static[string]. So the type of s in the macro body should be a NimNode. Well, maybe an echo overload exists that can work with NimNodes, or maybe, as we pass a string constant to macro m1(), in this concrete case s is indeed an ordinary string in the macro body. Maybe we should have used s: static[string] as parameter type, which would give us the exact same results.

We said that macros have to always return an untyped result. This is true, but as untyped is the only possible result type, that type can currently be omitted. So you may see in the code of the Nim standard lib a few macros which seems to return nothing. For our own macros we really should use always untyped as result. And sometimes you may even see macros where for parameters no data type is specified at all. In that case the data type has the default untyped type.

As macros are executed at compile time, we can not really pass runtime variables to it. When we try, we would expect a compiler error:

import macros

macro m1(s: string): untyped =
  echo s

proc main =
  var str = "non static string"
  m1(str)

main()

But with the current compiler version 1.5.1 that code compiles and prints the message "str", which is a bit surprising. To fix this, we can change the parameter type to static[string], which guarantees that we can indeed pass only compile time constants. Our last example would give a compile error in this case, while the one before with the string constant would work as expected.

Now let us create macros which actually creates an AST which is returned by the macro and executed when we run our program. For creating an AST in the macro body we have various options: we can use the parseStmt() function or the "quote do:" notation to generate the AST from regular program code in text form, or we can create the syntax tree directly with expressions provided by the macros module, e.g. by calls like newTree() or newLit() and such. The later gives us the best control over the AST generation process, but is not easy for beginners. The good news is that Nim now provides a set of helper functions like dumpTree() or dumpAstGen() which shows us the AST representation of a Nim source code block as well as the commands which we can use to create that AST. This makes it for beginners much easier to learn the basic instructions necessary to create valid syntax trees and to create useful macros.

We will start with the simple parseStmt() function which generates the syntax tree from the source code text string that we pass it as argument. This seems to be very restricted, and maybe even useless, as we can write the source code just as ordinary program text outside of the macro body. That is true, but we can construct the text string argument that we pass to the parseStmt() function with regular Nim code at compile time. That is similar as having one program, which generates a new source code string, saves that string to disk, and finally compiles and runs that created program. Let us check with a fully static string that parseStmt() actually works:

import macros

macro m1(s: static[string]): untyped =
  result = parseStmt(s)

proc main =

  const str = "echo \"We like Nim\""
  m1(str)

main()

When we compile and run above program, we get the output "We like Nim". The macro m1() is called at compile time with the static parameter str and returns an AST which represents the passed program code fragment. That AST is inserted into our program at the location of the macro call, and when we run our program the compiled AST is executed and produces the output.

Of course executing a fully static string this way is useless, as we could have used regular program code instead. Now let us investigate how we can construct some program code at compile time. Let us assume that we have an object with multiple fields, and we want to print the field contents. A sequence of echo statements would do that for us, or we may use only one echo statement, when we separate the field arguments each by "\n". The with module may further simplify our task. But as we have to print multiple fields, not an array or a seq, we can not directly iterate over the values to process them. Let us see how a simple text string based macro can solve the task:

import macros

type
  O = object
    x, y, z: float

macro m1(objName: static[string]; fields: varargs[untyped]): untyped =
  var s: string
  for x in fields:
    s.add("echo " & objName & "." & x.repr & "\n")
  echo s # verify the constructed string
  result = parseStmt(s)

proc main =
  var o = O(x: 1.0, y: 2.0, z: 3.0)
  m1("o", x, y, z)

main()

In this example we pass the name of our object instance as static string to the macro, while we pass the fields not as string, but as list of untyped values. The passed static string is indeed an ordinary Nim string inside the macro, we can apply sting operations on it. But the field names passed as untyped parameters appear as so called NimNodes inside the macro. We can use the repr() function to convert the NimNodes to ordinary strings, so that we can use string operations on them. We iterate with a for loop over all the passed field names, and generate echo statements from the object instance name and the field names, each separated by a newline character. Then all the statements are collected in a multi-line string s and are finally converted to the final AST by the parseStmt() function. In the macro body we use the echo statement to verify the content of that string. As the macro is executed during compile time, we get this output when we compile our program:

echo o.x
echo o.y
echo o.z

And when we run it we get:

1.0
2.0
3.0

Well, not a really great result for this concrete use case: We have replaced three echo commands with a five lines macro. But at least you got a feeling what macros can do for use.

Types of macro parameters

As Nim is a statically typed programming language, all variables and proc parameters have a well defined data type. There is some form of exception from this rule for OR-types, object variants and object references: OR-types are indeed no real exception, as whenever we use an OR-type as the type of a proc parameter, multiple instances of the proc with different parameter types are created when necessary. That is very similar to generic procs. Object variants and object references built indeed some form of exception, as instances of these types can have different runtime types that we can query with the case or with the of keyword at runtime. Note that object variants and references (the managed pointers itself, not the actual data allocated on the heap) occupy always the same amount of RAM, independent of the actual runtime type. (That is why we can store object variants with diffeent content or references to objects of different runtime types using inheritance in arrays and sequences.)

For the C sqr() macro from the beginning of this section, there is no real restriction for the argument data types. The sqr() C macro would work for all numeric types that support the multiply operation, from char data type over various int types to float, double and long double. This behaviour is not really surprising, as C macros are only a text substitution — by the * multiply operator for our sqr() macro. Actually the C pre-processor would even accept all data types and even undefined symbols for its substitution process. But then the C compiler would complain later.

Nim macros and Nim templates do also some form of code substitution, so it is not really surprising that they accept not only well defined data types, but also the relaxed types typed and untyped.

As parameters for Nim’s macros we can use ordinary Nim data types like int or string, compile time constants denoted with the static keyword like static[int], or the typed and untyped data types. When we call macros then the data types of the parameters are used in the same way for overload resolution as it is done for procs and templates. For example, if a macro defined as foo(arg: int) is called as foo(x), then x has to be of a type compatible to int.

What may be surprising at first is, that inside the macro body all parameter types have not the data type of the actual argument that we have passed to the macro, but the special macro data type NimNode which is defined in the macros module. The predefined result variable of the macro has the type NimNode as well. The only exception are macro parameters which are explicitly marked with the static keyword to be compile time constants like static[string], these parameters are not NimNodes in the macro body, but have their ordinary data types in the macro body. Variables that we define inside the macro body have exactly that type that we give to then, e.g. when we define a variable as s: string then this is an ordinary Nim string variable, for which we can use the common string operations. But of course we have always to remember that macros are executed at compile time, and so the operations on variables defined in the macro body occur at compile time, which may restrict a few operations. Currently macros are evaluated at compile time by the Nim compiler in the NimVM (Vitual Machine) and so share all the limitations of the NimVM: Macros have to be implemented in pure Nim code and can currently not call C functions except those that are built in the compiler.

In the Nim macros tutorial the static, typed and untyped macro parameters are described in some detail. We will follow that description, as it is more detailed as the current description in the Nim compiler manual. As these descriptions are very abstract, we will give some simple examples later.

Static Macro Parameters

Static arguments are a way to pass compile time constants not as a NimNode but as an ordinary value to a macro. These values can then be used in the macro body like ordinary Nim variables. For example, when we have a macro defined as m1(num: static[int]), then we can pass it constants values compatible with the int data type, and in the macro body we can use that parameter as ordinary integer variable.

Untyped Macro Parameters

Untyped macro arguments are passed to the macro before they are semantically checked. This means that the syntax tree that is passed down to the macro does not need to make sense for the Nim compiler yet, the only limitation is that it needs to be parsable. Usually, the macro does not check the argument either but uses it in the transformation’s result somehow. The result of a macro expansion is always checked by the compiler, so apart from weird error messages, nothing bad can happen. The downside for an untyped macro argument is that these do not play well with Nim’s overloading resolution. The upside for untyped arguments is that the syntax tree is quite predictable and less complex compared to its typed counterpart.[30]

Typed Macro Parameters

For typed arguments, the semantic checker runs on the argument and does transformations on it, before it is passed to the macro. Here identifier nodes are resolved as symbols, implicit type conversions are visible in the tree as calls, templates are expanded, and probably most importantly, nodes have type information. Typed arguments can have the type typed in the arguments list. But all other types, such as int, float or MyObjectType are typed arguments as well, and they are passed to the macro as a syntax tree.[31]

Code Blocks as Arguments

In Nim it is possible to pass the last argument of a proc, template or macro call as an indented code block following a colon, instead of an ordinary argument enclosed in the parentheses following the function name. For example instead of echo("1 + 2 = ", 1 + 2) we can also write

echo("1 + 2 = "):
  1 + 2

For procs this notation makes not much sense, but for macros this notation can be useful, as syntax trees of arbitrary complexity can be passed as arguments.

Now let us investigate in some more detail which data types a macro accepts. This way we hopefully get more comfortable with all these strange macro stuff. For our test we create a few tiny macros with only one parameter which does noting more than printing a short message when we compile our program:

import macros

macro m1(x: static[int]): untyped =
  echo "executing macro body"

m1(3)

This code should compile fine and print the message "executing macro body" during the compile process, and indeed it does. The next example is not that easy:

import macros

macro m1(x: int): untyped =
  echo "executing macro body"
  echo x
  echo x.repr

var y: int
y = 7
m1(y)

This compiles, but as the assignment y = 7 is executed at program runtime, while the macro body is already executed at compile time, we should not expect that the echo statement in the macro body prints the value 7. Instead we get just y for both echo calls. Now let us investigate what happens when we use typed instead of int for the macro parameter:

import macros

macro m1(x: typed): untyped =
  echo "executing macro body"
  echo x
  echo x.repr

var y: int
y = 7
m1(y)

We get the same result again, both echo statements prints y. The advantage of the use of typed here is, that we can change the data type of y from int to float and our program still compiles. So the typed parameter type just enforces that the parameter has a well defined type, but it does not restrict the actual data type to a special value. The previous macro with int parameter type would obviously not accept a float value.

Now let us see what happens when we pass an undefined symbol to this macro with typed parameter:

import macros

macro m1(x: typed): untyped =
  echo "executing macro body"
  echo x
  echo x.repr

m1(y)

This will not compile, as the macro expects a parameter with a well defined type. But we can make it compile by replacing typed with untyped:

import macros

macro m1(x: untyped): untyped =
  echo "executing macro body"
  echo x
  echo x.repr

m1(y)

So untyped macro parameters are the most flexible ones, and actually they are the most used. But in some situations it is necessary to use typed parameters, e.g. when we need to know the parameter type in the macro body.

Quote and the quote do: construct

In the section before we learned about the parseStmt() function which is used in a macro body to compile Nim code represented as a multi-line string to an abstract syntax tree representation. Macros uses as return type the "untyped" data type, which is compatible with the NimNode type returned by the parseStmt() funcion.

The quote function and the quote do: construct has some similarity with the parseStmt() function: It accepts an expression or a block of Nim code as argument and compiles that Nim code to an abstract syntax tree representation. The advantage of quote() is that the passed Nim code can contain NimNode expressions from the surrounding scope. The NimNode expressions have to be quoted using backticks.

As a first very simple example for the use of the quote do: construct we will present a way to print some debugging output.

Assume we have a larger Nim program which works not in the way that we expected, so we would add some echo statements like

var currentSpeed: float = calcSpeed(t)
echo "currentSpeed: ",  currentSpeed

Instead of the echo statement we would like to just write show(currentSpeed) to get exactly the same output. For that we need access not only to the actual value of a variable, but also to its name. Nim macros can give us this information, and by using the quote do: construct it is very easy to create our desired showMe() macro:

import macros

macro show(x: untyped): untyped =
  let n = x.toStrLit
  result = quote do:
    echo `n`,": ", `x`

import math
var a = 7.0
var b = 9.0
show(a * sqrt(b))

When we compile and run that code we get:

a * sqrt(b): 21.0

In the macro body we use the proc toStrLit() from the macros module which is described with this comment: "Converts the AST n to the concrete Nim code and wraps that in a string literal node" So our local variable n in the macro body is a NimNode that now contains the string representation of the macro argument x. We use the NimNode n enclosed with backtics in the quote do: construct. It seems that writing this macro was indeed not that difficult, but actually it was only that easy because we have basically copied the dump() macro from the sugar module of Nim’s standard library.

Let us investigate our show() macro in some more detail to learn more about the inner working of Nim macros. First recall that macros always have a return value of data type untyped, which is actually a NimNode. The quote do: construct gives us a result which we can use as return value of our macro. Sometimes we may see macros with no result type at all, which is currently identical to the untyped result type. As the macro body is executed at compile time, the quote do: construct is executed at compile time as well, that is that the code block which we pass to the quote do: construct is processed at compile time and the quoted NimNodes in the block are interpolated at compile time. For our program from above the actual echo statement in the block is then finally executed at program runtime. To prove how this final echo statement looks we may add as last line of our macro the statement "echo result.repr" and we would then get the string "echo "a * sqrt(b)", ": ", a * sqrt(b)" when we compile our program again.

Building the AST manually

In the two sections before we used the functions parseStmt() and quote() to build the AST from a textual representation of Nim code. That can be convenient, but is not very flexible. In this section we will learn how we can build a valid AST from scratch by calling functions of the macros module. That is not that easy, but this way we have the full power of the Nim meta-programming available.

Luckily the macros module provides some macros like dumpTree() and dumpAstGen() which can help us get started. We will create again a macro similar to the show() macro that we created before with the quote do: construct, but now with elementary instructions from the macros module. This may look a bit boring, but this plain example is already complicated enough for the beginning, and it shows use the basics to construct much more powerful macros later.

The core code of our debug macro would look in textual representation like

var a, b:int
echo "a + b", ": ", a + b

That is for debugging we would like to print an expression first in its string representation, and divided by a colon the evaluated expression. The dumpTree() macro can show us how the Nim syntax tree for such a print debug statement should look:

import macros

var a, b: int

dumptree:
  echo "a + b", ": ", a + b

When we compile this code we get as output:

 StmtList
  Command
    Ident "echo"
    StrLit "a + b"
    StrLit ": "
    Infix
      Ident "+"
      Ident "a"
      Ident "b"

So the Nim syntax tree for the echo statement from above is a statement list consisting of an echo command with two string literal arguments and a last argument which is built with the infix + operator and the two arguments a and b. So we can see how the AST that we would have to construct would have to look, but we still have no idea how we could construct such an AST in detail. Well, the macros module would contain the functions what we need for that, but it is not easy to find the right functions there. The dumpAstGen() macro can list us exactly the needed functions:

import macros

var a, b: int

dumpAstGen:
  echo "a + b", ": ", a + b
Compiling that code gives us:
 nnkStmtList.newTree(
  nnkCommand.newTree(
    newIdentNode("echo"),
    newLit("a + b"),
    newLit(": "),
    nnkInfix.newTree(
      newIdentNode("+"),
      newIdentNode("a"),
      newIdentNode("b")
    )
  )
)

This is a nested construct. The most outer instruction constructs a new tree of Nim Nodes with the node type statement list. The next construct creates a tree with node kind command, which again contains the ident node with name echo, which again contains two literals and the infix + operator.

Indeed we can use the output of the dumpAstGen() macro directly to create a working Nim program:

import macros

var a, b: int

#dumpAstGen:
#  echo "a + b", ": ", a + b

macro m(): untyped =
  nnkStmtList.newTree(
    nnkCommand.newTree(
      newIdentNode("echo"),
      newLit("a + b"),
      newLit(": "),
      nnkInfix.newTree(
        newIdentNode("+"),
        newIdentNode("a"),
        newIdentNode("b")
      )
    )
  )

m()

When we compile and run that code, we get the output:

a + b: 0

So the AST from above is fully equivalent to the one line echo statement. But now we would have to investigate how we can pass an actual expression to our macro and how we can use that passed argument in the macro body — first print its textual form, and then the evaluated value separated by a colon. And there is one more problem: That nested macro body from above is not really useful for our final dump() macro, as we would like to be able to construct the NimNde, that is returned by the dump macro step wise: Add the echo command, then the passed expression in string form, and finally the evaluated expression. So let us first rewrite above macro in a form where the AST is constructed step by step. That may look difficult, but when we know that we can call the newTree() function with only one node kind parameter to create a empty tree of that kind, and that we can later use the overloaded add() proc to add new nodes to that tree, then it is easy to guess how we can construct the macro body:

 import macros

var a, b: int

#dumpAstGen:
#  echo "a + b", ": ", a + b

macro m(): untyped =
  nnkStmtList.newTree(
    nnkCommand.newTree(
      newIdentNode("echo"),
      newLit("a + b"),
      newLit(": "),
      nnkInfix.newTree(
        newIdentNode("+"),
        newIdentNode("a"),
        newIdentNode("b")
      )
    )
  )

macro m2(): untyped =
  result = nnkStmtList.newTree()
  let c = nnkCommand.newTree()
  let i = nnkInfix.newTree()
  i.add(newIdentNode("+"))
  i.add(newIdentNode("a"))
  i.add(newIdentNode("b"))
  c.add(newIdentNode("echo"))
  c.add(newLit("a + b"))
  c.add(newLit(": "))
  c.add(i)
  result.add(c)

m2()

First we create the tree empty three structures of node kinds statement list, command and infix operator. Then we use the overloaded add() proc to populate the threes, using procs like newIdentNode() or newLit() to create the nodes of matching types as before. When we run our program with the modified macro version m2() we get again the same output:

a + b: 0

The next step to create our actual dump() macro is again easy — we pass the expression to dump as an untyped macro parameter to the macro, convert it to a NimNode of string type and use that instead of the newLit("a + b") from above. In our second macro, where we used the quote do: construct, we applied already toStrLit() on an untyped macro parameter, so we should be able to reuse that to get the string NimNode. Instead we would have to apply the stringify operator additional on that value. But a simpler way is to just apply repr() on the untyped macro argument to get a NimNode of string type. And finally, to get the value of the evaluated expression in our dump macro, we add() the untyped macro parameter directly in the command three — that value is evaluated when we run the macro generated code.

import macros

var a, b: int

macro m2(x: untyped): untyped =
  var s = x.toStrLit
  result = nnkStmtList.newTree()
  let c = nnkCommand.newTree()
  c.add(newIdentNode("echo"))
  c.add(newLit(x.repr))
  #c.add(newLit($s))
  c.add(newLit(": "))
  c.add(x)
  result.add(c)

m2(a + b)

Again, we get the desired output:

 a + b: 0

So our dump() macro called still m2() is complete and can be used to debug arbitrary expression. Note that this macro works for arbitrary expressions, not only for numerical ones. We may use it like

m2(a + b)
let what = "macros"
m2("Nim " & what & " are not that easy")
and get the output
a + b: 0
"Nim " & what & " are not that easy": Nim macros are not that easy

Now let us extend our debug macro so that it can accept multiple arguments. The needed modifications are tiny, we just pass instead of a single untyped argument an argument of type varargs[untyped] to the debug macro, and iterate in the macro body with a for loop over the varargs argument:

import macros

macro m2(args: varargs[untyped]): untyped =
  result = nnkStmtList.newTree()
  for x in args:
    let c = nnkCommand.newTree()
    c.add(newIdentNode("echo"))
    c.add(newLit(x.repr))
    c.add(newLit(": "))
    c.add(x)
    result.add(c)

var
  a = 2
  b = 3
m2(a + b, a * b)

When we compile and run that code we get:

a + b: 5
a * b: 6

The Assert Macro

As some more simple macro example we will show how we can create our own assert macro. The assert() has only one argument, which is a expression with a boolean result. If the expression evaluates to true at program runtime, then the assert macro should do nothing. But when the expression evaluates to false, then this indicates a serious error and the macro shall print the expression which evaluated to false, and then terminate the program execution. This is basically what the assert() macro in the Nim standard library already does, and the official Nim macros tutorial contains such an assert() macro as well.

Arguments for our assert macro may look like "x == 1 +2", containing one infix operator and one left-hand and one right-hand operand. We will show how we can use subscript [] operators on the NimNode argument to access each operand.

As first step we use the treeRepr() function from the macros module to show us the Nim tree structure of a boolean expression with an infix operator:

import std/macros

macro myAssert(arg: untyped): untyped =
  echo arg.treeRepr

let a = 1
let b = 2

myAssert(a != b)

When we compile that program, then the output of the treeRepr() function shows us, that we have passed as argument an infix operator with two operands at index position 1 and 2.

Infix
  Ident "!="
  Ident "a"
  Ident "b"

Now let us create an assert macro which accept such a boolean expression with an infix operator and two operands:

import std/macros

macro myAssert(arg: untyped): untyped =
  arg.expectKind(nnkInfix) # NimNodeKind enum value
  arg.expectLen(3)
  let op  = newLit(" " & arg[0].repr & " ") # operator as string literal NimNode
  let lhs = arg[1] # left hand side as NimNode
  let rhs = arg[