Nim Roadmap 2023

[Araq]

See, I am sending an angel ahead of you to guard you along the way and to bring you to the place I have prepared. -- Exodus 23:20

Version 2.0

• Fix the remaining ORC bugs and release the final 2.0 version early in 2023.

Version 2.2

Incremental compilation (IC) / nimsuggest / IDE tooling

This is a single point as for me it's a single thing. If we had a compiler that only recompiles the module that was touched the "nimsuggest" tool can be a simple command line tool that answers a single question like "where is symbol X used?".

We have this tool already, it's nim check --navigate, but the implementation of IC needs to be finished. In order to implement this feature in a disciplined manner that avoids complexity and hard to track down subtle bugs the following strategy will be followed:

With --ic:stress IC is enabled for every single module and the compiler always uses the same non-conditional logic:

The Nim compiler compiles a single a.nim module at a time. It is compiled to C(++). Let b.nim be a dependency of a.nim. If b.nim has been compiled to b.rod already and b.nim did not change since then, import b.rod and proceed.

Otherwise, compile b.nim via spawning another Nim compiler process that compiles b.nim to b.rod. Then proceed by importing b.rod. This rather simple setup ensures that by construction we stress test the mechanism completely, all the time. The downside is that all the required IO slows down the compiler a lot, so that development and testing will be slow.

• Finish the implementation of IC.

Stretch goals

Allow recursive module dependencies

• We should remove the requirement for forward declarations and allow for recursive module dependencies. Then we can remove these experimental features:

1. "Package level objects"
2. "code reordering"

see also
nim-lang/Nim#18818
nim-lang/Nim#18822

• parse the whole module at one time => fixes #19795; fixes #11852; fixes #19974; remove parsing pipeline, Nim now parses the whole module at one time Nim#21379
• simplify compilation pipeline => replaces implicit passes array registered at runtime with explicit function calls; simplify compilation pipeline Nim#21444

Language

• Patch the full compiler to use "definite assignment analysis" itself => use strictdefs for compiler Nim#22365

• "Void type" -- We should optimize away the "empty tuple" type in the backend and remove this feature.

• "AST based overloading" -- Should be removed.

• "Parallel & Spawn" -- Should be removed or overhauled. Currently it is better done by external packages.
  deprecate std/threadpool; use malebolgia, weave, nim-taskpool instead Nim#22576

• With our refined idea of "strict funcs", the algorithm for proving borrow checking correct can be much simpler and more precise. In other words, "view types" can be made stable for Nim 2.x.

Library

JSON

The JSON module is in a poor shape, the API exposes public fields, sometimes is not intuitive (the %* operator) and has poor performance. It could be split up into:
jsonbuilder, jsonparser, jsonlexer, jsontree, jsonmapper.

Async

The higher level async modules should be moved into their own Nimble package. The required building blocks like selectors.nim remain in the standard library, they served Nim well.

Less concrete goals

1. Find a better design for "Term rewriting macros". Once done, deprecate the old feature.
2. Find a better design for "Special Operators". Once done, deprecate the old feature.
3. Implement "Aliasing restrictions in parameter passing". The compiler can always produce warnings first.
4. Concepts: Find a way of how to infer static values and stabilize the feature.
5. Find a way of allowing a syntax like of Value(let x, var y) and embrace an official pattern matching solution that makes use of this syntax.

Atlas tool

• Refine atlas so that it supports dependencies in subdirectories of the workspace.
• Add an alternative to project.nimble like project.lock so that lock files and reproducible builds are supported.

How can you help?

The standard library cleanups should be rather easy to do. Atlas is also still a very simple tool, only about 600 lines of Nim code.

[Varriount]

Regarding incremental compilation:

The downside is that all the required IO slows down the compiler a lot, so that development and testing will be slow.
I don't know the specifics of how one would do this on Linux, but on Windows one could use something like a RAM disk to reduce IO latency.

For the JSON module(s), would it be possible to go with json/lexer, json/parser, etc.?

For term-rewriting macros (and I might write up an RFC for this), I feel that a better mechanism would be a way for a Nim module to implement AST rewriting. This could be something like plugins for the compiler, or an interface exposed by a module. This way the community can experiment with various forms of AST rewriting, and an "ideal" solution can emerge.

[Araq]

I don't know the specifics of how one would do this on Linux, but on Windows one could use something like a RAM disk to reduce IO latency.

Interesting, thanks!

For the JSON module(s), would it be possible to go with json/lexer, json/parser, etc.?

No. We don't have directories in Nim code (except in import, of course), only module names.

For term-rewriting macros (and I might write up an RFC for this), I feel that a better mechanism would be a way for a Nim module to implement AST rewriting.

AST rewriting via compiler plugins sounds interesting.

[c-blake]

I could not see where the snippet @Varriount quoted came from, but on Linux, you can just use --nimcache:/dev/shm/.... to use a RAM disk -- no license required. (By default 50% of system RAM is available there, but this is adjustable.) Or have $HOME/.cache/nim be a symbolic link into /dev/shm/nim-$LOGNAME. (I've been doing this since I began using Nim, actually..just to economize on writes to my NVMe drives.)

EDIT: and it seems like this is also pretty easy on OSX -- https://www.techjunkie.com/how-to-create-a-4gbs-ram-disk-in-mac-os-x/ but it is not "just there by default".

[arnetheduck]

Speaking of caches, LLVM supports caching during lto, something I'll probably enable in nlvm "soon": https://clang.llvm.org/docs/ThinLTO.html#id11 - I imagine nim could enable it too when used with clang

[PMunch]

For the JSON module(s), would it be possible to go with json/lexer, json/parser, etc.?

No. We don't have directories in Nim code (except in import, of course), only module names.

I'm pretty sure they meant the module names. So that you could do something like import std/json/[parser, lexer] for example. IMO this seems like a nice clean design.

[Araq]

Yeah but what if he does import std / json / parser; import std / xml / parser? Then he has to disambiguate via import as and nothing of the nice clean design remains.

[PMunch]

True, but I feel like it's less often that you'll need to import low-level parsers for two separate languages. I hope we'll still get a import std/json which exposes a similar API to what we have today. And if you indeed had to implement some weird dual parsing library then import std/json/parser as jsonparser doesn't seem like too bad a tradeof.

[metagn]

Sorry to be vague, but I think json/parser should only happen when you aren't meant to import json/parser normally. As in, it should be like a component/submodule of the json module, instead of a module of the json subpackage of the standard library. If it's an independent library, then it can be jsonparser, but otherwise it may pollute the std namespace

[Clonkk]

@ringabout
@Araq

Are there (rough) estimates or update regarding this roadmap ?

[Araq]

I just updated it and @ringabout is now working on "Patch the full compiler to use "definite assignment analysis" itself."

[ringabout]

Update: with nim-lang/Nim#22365 merged, the compiler now enables strictdefs in itself.

[blackmius]

what does "IDE tooling" in "Incremental compilation (IC) / nimsuggest / IDE tooling" means?

[Araq]

Continued here: #543