Migration fixes

[Lun4m]

This PR contains a lot of stuff (probably too much!):

1. I reworked a bit the code organization in the migration directory:

   • Now dump and import packages define their own set of tables/data they act on
   • Indices are dropped and created in a separate package, which I found easier to manage in case something goes wrong during the import step
   • Switched to zerolog

2. Added labels for kdvh and kvalobs data, so it's easier to delete specific timeseries in case we need to migrate again.

3. Added Frost quality codes (which are not exhaustive, but what can I do?)

4. Added handling of restricted data for migrations

5. Renamed SQL files so that we can automate database setup without having to update the file names every time

6. Significantly improved the just file, so it's a bit less of a pain to use

7. Figured out we have to set GOMEMLIMIT (or should I say GOMEM*E*LIMIT, I love golang) to avoid OOM when importing data

I don't expect you to check everything, but if you quickly go through the changes and spot something weird, please let me know!

Edit: I need to add an ansible script to disable/enable the replication before/after the migration

[intarga]

To be clear: remaining work on migrations is just testing restricted data?

We should try to be better about structuring PR's. Migrations are already hard to review because the code is not heavily trafficked, and when semantic changes are buried under refactoring + tooling changes, it becomes even harder to separate things and identify what needs attention. This ideally should have been a series of 3+ PRs, though I appreciate that may have been hard to do as you were fixing issues as they came up.

[Lun4m]

To be clear: remaining work on migrations is just testing restricted data?

Yes, which I actually am doing right now.
There are two other problems, but I would not qualify them as pressing issues:

• Importing data requires connections to both KDVH and Kvalobs, so we definitely need to dump all the auxiliary metadata
• There are other tables in KDVH that I haven't touched (I've left a TODO somewhere), but I feel like we want to dump everything we can

We should try to be better about structuring PR's. Migrations are already hard to review because the code is not heavily trafficked, and when semantic changes are buried under refactoring + tooling changes, it becomes even harder to separate things and identify what needs attention. This ideally should have been a series of 3+ PRs, though I appreciate that may have been hard to do as you were fixing issues as they came up.

I agree, a 5000+ changes PR is not easily reviewable. I can try to break this down into smaller PRs if you are okay with it!

[intarga]

I can try to break this down into smaller PRs if you are okay with it!

Not sure if it would be worth the effort, I meant the comment more as something to keep in mind in the future, structuring work with PRs a bit in mind, and filing them more frequently

[Lun4m]

Closing because somehow Github is refusing to merge even though the branch is already rebased on trunk 😕