"Invalid key" error while trying to restore re-encrypted database in WhatsApp

[kingbtcvl]

Hey guys, I have the following problem concerning the encryption function:

I have and old database msgstore_old.db.crypt14 and the corresponding key file. Using the decryption function wadecrypt I can successfully obtain the database msgstore_old.db.
Moreover I have my recent database msgstore_new.db.crypt15 which is end-to-end encrypted with a 64 characters key a69f... that I know. Again using wadecrypt with this key I can successfully obtain the database msgstore_new.db.
Now I want to merge this two database. Using Julia and its SQLite package, I can execute a bunch of SQL commands which yields a merged database msgstore_merged.db. After achieving that I want to restore this merged database in WhatsApp, of course.
So I use waencrypt with my key a69f... to get msgstore_merged.db.crypt15. I checked that everything is fine by decrypting this database with wadecrypt again as a little test and it worked, so I felt I was ready to restore this merged and encrypted database in WhatsApp.

Consequently I deinstalled WhatsApp completely, put my msgstore_merged.db.crypt15 as msgstore.db.crypt15 in newly created WhatsApp\Database folder and also copied the old WhatsApp\Media folder where I put the new media, too. Now I installed WhatsApp again, opened it, regestired wiht my phone number until I took me to the backup dialog. Since I am on Android WhatsApp wanted me to backup from my Google Account, but no problem. I closed WhatsApp, switched to airplane mode, opened WhatsApp again, waited a bit, skipped the error connection message and there we go, WhatsApp trying to find a local backup. My backup was recoginized by WhatsApp, too. So far, so good.

Now WhatsApp asks me to provide a password for encryption. I tell it that I have used a 64 character key instead in it leads me to the dialogue where I am finally supposed to put in my key. I typed that key a69f... in and it failed telling me it would be invalid :-(
I tried that a second time, double and triple checking that there was no typo and after being absoltely sure I again failed and WhatsApp told me that the key I put in was wrong again.
(I even made a third attempt decrypting msgstore_merged.db with a key 5555.... that consits of 64 times a 5, but this did not work either while encrypting it in WhatsApp).

So, it seems like altough I can successfully redecrypt my encrypted file msgstore_merged.db.crypt15 using the wadecrypt function from this repo, WhatsApp still hates the encrypted file.
I read through the issues in this repo and reading issue #9 I found out that WhatsApp obiously requires the correct header in the encrypted file to be able to read it successfully.
This means that I see some light at the end of the tunnel. However I haven't figured out yet, how to get the correct header into my encrypted file.
Due to the comments in #9 I gained some hope that the command
waencrypt a69f... msgstore_merged.db msgstore_merged.db.crypt13 --reference msgstore_new.db.crypt13 --type 13
would do the job, but looking at the current source code I feel that the --reference option is just an empty dummy at the moment?! Am I correct in this assumption or did I simply miss the part of the code where this option effectively does something?

I mean I could simply try it and see what happens, of course, but I don't want to waste a try here because WhatsApp already started to complain about me regestering the same phone number to often in a row. Therefore I had to swicht from SMS verification to get call verfication already and I am a little afraid of getting banned permanently :D

So any clues on how to make my merged database get accepted by WhatsApp would be highly appreciated!

[ElDavoo]

Hi,
As written here (I don't know if it printed), waencrypt is still a work in progress that needs to be completed. That's why:

1. Encryption support #9 is open
2. Documentation has not been written (and what I say is difficult to understand)
3. The pip release (the one that people were supposed to get?) is not updated

Probably the feature list is incorrect? As you correctly guessed, you should copy it from the backup file you want to modify and re-encrypt it using the same feature list with the --reference that unfortunately as you guessed it's still not implemented.
Your case is also particular because you want to merge two DB files.
So to give a rough idea of what should be done:

1. Hack wadecrypt to let it print the feature table (this too will be done in the future), decrypt your old DB files and get their feature list.
2. Merge the lists (how? no idea? A logical OR should work?)
3. Use waencrypt with --feature-list (also --max-feature if necessary) to insert the table you got previously.

This is assuming the feature list is incorrect. It might also be:

1. The encryption process is wrong (bug)
2. The header format is wrong (bug)
3. You didn't set --wa-version and --jid
4. Your merged DB is wrong in some way (especially if the schema changed)

Because unfortunately the application won't tell you what exactly is wrong.

Unfortunately, waencrypt requires lots more parameters to make a working backup, so explaining and letting everything work becomes difficult both technically and from a user experience view. At this state it's a little more than a proof of concept as it's difficult to test (you need to open the app and re-register to observe the restore process...)

[kingbtcvl]

Thanks a lot for you detailed explanation, @ElDavoo

So to give a rough idea of what should be done:

1. Hack wadecrypt to let it print the feature table (this too will be done in the future), decrypt your old DB files and get their feature list.
2. Merge the lists (how? no idea? A logical OR should work?)
3. Use waencrypt with --feature-list (also --max-feature if necessary) to insert the table you got previously.

I think I am able to try that and see what happens. The first point does not sound too difficult to achieve and the third one is easy when you already have the feature list. However, the second point worries me a little already. Altough using a logical OR does seem to be the natural choice at first glance, it might cause the backup process to fail again.
From what I have figured out about the features so far, the older the database the less features exist because new features are added in newer versions of WhatsApp. This might suggest that old features might have already become obsolete and pernanently deactivated for newer backups. If that's the case then a logical OR might not work for those features. You could argue to simply take all new features, but that might also be wrong because the new database is very small compared to the old one. There are especially quite a few tables in the old database that contain data, which are completely empty in the new database. If any of the features refers to such a table, the feature is active in the old backup and not in the new one, but must be active in the merged backup. 🤔

Could you maybe elaborate a little more on what is known about the features?
Where can I find more information about them to understand what they really represent as far as the databse is concerned.
I mean, I see from backup_expiry.proto that there are 35 features (f_5 to f_39), which are all boolean values except for the unused backup size f_38. However, what I don't get is how we know that for example feautre 5 corresponds to the call log, whereas feature 18 corresponds to the message system. What I also don't understand exactly either is what it means when a feature is active compared to what it means when a featue is not active in a database.

---

This is assuming the feature list is incorrect. It might also be:

1. The encryption process is wrong (bug)
2. The header format is wrong (bug)
3. You didn't set --wa-version and --jid
4. Your merged DB is wrong in some way (especially if the schema changed)

Well, hopefully point one and two are not the case for my database (or anybody's database). But concerning point one, I am confident that the encryption process itself is fine, since I could decrypt the re-encrypted database successfully.
Concerning point two, I believe that it is not too problematic because I saw that you and some other guy have already managed to restore a re-encrypted backup.
Yes, concerning point three I haven't used that options so far, but I will for my next test since these information is obviously crucial for the header.
And finally, concerning the last point four, I do think that even if the database was wrong in some way, WhatsApp will at least start to restore the database instead of telling me that the encryption key is invalid right from the beginning. So that is something I expect to figure out only after I made WhatsApp decrypt my re-encrypted database. However, I was quite carefully here and especially ensured that the shema of the database remained unchanged.

[ElDavoo]

Could you maybe elaborate a little more on what is known about the features?

Nothing except their names unfortunately.

However, what I don't get is how we know that for example feature 5 corresponds to the call log, whereas feature 18 corresponds to the message system.

They're taken from the official app, but I didn't study their effects on the DB files as it goes out of the scope of this project.

I believe that it is not too problematic because I saw that you and some other guy have already managed to restore a re-encrypted backup.

I only tested re-encrypting the exact same DB without modification, so going further is untested.

Yes, concerning point three I haven't used that options so far, but I will for my next test since these information is obviously crucial for the header.

Well, the jid option is mandatory, as it identifies the phone number linked to your account. If it doesn't match, WA should recognize it's not your backup and not restore it. The WA version could change without the restore failing (unless the version is more recent than your current app version (?).

---

For the other things you said, I know as much as you.

[kingbtcvl]

Could you maybe elaborate a little more on what is known about the features?

Nothing except their names unfortunately.

However, what I don't get is how we know that for example feature 5 corresponds to the call log, whereas feature 18 corresponds to the message system.

They're taken from the official app, but I didn't study their effects on the DB files as it goes out of the scope of this project.

Where exactly can one find those names? I would like to have a closer look at it because I am quite unsure whether or not I have all features now.

I mean I now added these lines of code to get the feature vector by inspecting all implemented features. The result is that my new database supports all of them and my old database supports all except for f_34, which is supposed to correspond to message_main_verification.
But now, I wonder how to figure out if there aren't any more new features that have not been implemented here yet.
I thought that I could kind of calculate that, finding out that in my case protobuf_size = 132. So the header consists of 132 bytes in total. But since I don't know the number of bytes of the unsued device_model string, I canot calculate how many of those 132 bytes must be left for the features, can I?

[kingbtcvl]

I only tested re-encrypting the exact same DB without modification, so going further is untested.

So, I am happy to help testing 😆

Well, the jid option is mandatory, as it identifies the phone number linked to your account. If it doesn't match, WA should recognize it's not your backup and not restore it. The WA version could change without the restore failing (unless the version is more recent than your current app version (?).

Actually I think, it's pretty strange that the jid option is mandatory. I mean, imagine you have a backup from on old phone and you bought a new one with a new number. Why should it be forbidden to load a backup. That does not make too much sense, since you would need the key anyway, assuming it is end to end encrypted.

[ElDavoo]

Where exactly can one find those names?

By decompiling the app with jadx and looking at the decompiled source code.

I thought that I could kind of calculate that, finding out that in my case protobuf_size = 132.

I don't think it's the right way to do that, I would change the proto message

[kingbtcvl]

By decompiling the app with jadx and looking at the decompiled source code.

Interesting. If I am unsuccessful with the current approach, I might have a look at that, too.

---

I thought that I could kind of calculate that, finding out that in my case protobuf_size = 132.

I don't think it's the right way to do that, I would change the proto message

Yeah, unfortuantely it seems that it is not that easy to sum up all the bytes that the header contains up to feature f_39 and then see if there are any bytes left for potentially new features. However, changing the proto message works great. I simply added one new optional test feature and then checked if this test feature had explicitlly given a new value. It turns out not, so I guess I can conclude that there are no new features right now.

---

So, I think I am close to start a new try for making WhatsApp restore my merged database.
Just two little more things.
First, I considered to create the head for msgstore_merged.db.crypt15 using the same IV that my msgstore_new.db.crypt15 contains, ie. I also want to apply the --iv option since I assume that this can only make things more likely to work, but not less likely. Would you agree here?
Second, I figured out that with current version of the code the backup_version in the header will always have the default value of C.DEFAULT_BACKUP_VERSION = 0. I think there should be a new option --backup-verson for that because in my case for example the backup version turn out to be 1.

[kingbtcvl]

Ladies and gentlemen,

I can happily report that my test was successful and I have just made WhatsApp recoginize my re-encrypted merged (i.e. modified) database. 🥳
Hence I can confirm that setting the correct header for a crypt15 file (and probably also for a crypt14 file, but I did not test that) while encrypting with waencrypt leads to an encrypted file, which WhatsApp accepts. Therefore I would personally consider this issue as solved.

For those of you who want to follow my steps, here is what I did:

• I started with my merged database msgstore_merged.db, my latest WhatsApp Backup msgstore_new.db.crypt15 and the corresponding 64-byte-key 8a359fb873c488f521aee87e3003f27ab376b9799cc39a70142d23e67a90b15c
• I used wadecrypt from the PR Added: header-info option #112 with the command
  wadecrypt 8a359fb873c488f521aee87e3003f27ab376b9799cc39a70142d23e67a90b15c msgstore_new.db.crypt15 msgstore_new.db --header-info
  to get the header information of my latest WhatsApp Backup. The output looked like this:

←[38;20mINFO - wadecrypt.py:222 : Header information in your crypt15 file:←[0m
←[38;20mINFO - wadecrypt.py:223 : IV: a902fb72341e7c7201f3dd2866a6a01e←[0m
←[38;20mINFO - wadecrypt.py:231 : Key type: 1←[0m
←[38;20mINFO - wadecrypt.py:232 : WhatsApp version: 2.24.3.81←[0m
←[38;20mINFO - wadecrypt.py:234 : The last two numbers of the user's Jid: 73←[0m
←[38;20mINFO - wadecrypt.py:235 : Backup version: 1←[0m
←[38;20mINFO - wadecrypt.py:238 : Features: [5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 39]←[0m
←[38;20mINFO - wadecrypt.py:239 : Max feature number: 39←[0m

• Next I used waencrypt from the PR Added: backup-version option #111 with the command
  waencrypt 8a359fb873c488f521aee87e3003f27ab376b9799cc39a70142d23e67a90b15c msgstore_merged.db msgstore.db.crypt15 --iv a902fb72341e7c7201f3dd2866a6a01e --wa-version 2.24.3.81 --jid 73 --backup-version 1 --enable-features 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 39 --max-feature 39
  to get my correctly encrypted database msgstore.db.crypt15.
• Then after uninstalling WhatsApp completely and reinstalling it, I simply used this output msgstore.db.crypt15 to let WhatsApp restore this database as a local backup.
  (I only provided the the Database folder com.whatsapp/WhatsApp/DataBases containing only the encrypted msgstore.db.crypt15 file and the Media folder com.whatsapp/WhatsApp/Media with all my merged media, but nothing else)
• And that's all. It worked like a charm and now my WhatsApp looks as if my merged database has always been there for ages. 😎

---

@ElDavoo
So as already said, the problem is solved now from my point of view and you can merge my PRs and/or close this issue if you like. That's up to you, of course. Anyway, thanks a lot for your support and this repo again!

[ElDavoo]

Well done! I will take a look at that after the 22nd

[haare123]

Ladies and gentlemen,

I can happily report that my test was successful and I have just made WhatsApp recoginize my re-encrypted merged (i.e. modified) database. 🥳 Hence I can confirm that setting the correct header for a crypt15 file (and probably also for a crypt14 file, but I did not test that) while encrypting with waencrypt leads to an encrypted file, which WhatsApp accepts. Therefore I would personally consider this issue as solved.

For those of you who want to follow my steps, here is what I did:

• I started with my merged database msgstore_merged.db, my latest WhatsApp Backup msgstore_new.db.crypt15 and the corresponding 64-byte-key 8a359fb873c488f521aee87e3003f27ab376b9799cc39a70142d23e67a90b15c
• I used wadecrypt from the PR Added: header-info option #112 with the command
  wadecrypt 8a359fb873c488f521aee87e3003f27ab376b9799cc39a70142d23e67a90b15c msgstore_new.db.crypt15 msgstore_new.db --header-info
  to get the header information of my latest WhatsApp Backup. The output looked like this:

←[38;20mINFO - wadecrypt.py:222 : Header information in your crypt15 file:←[0m
←[38;20mINFO - wadecrypt.py:223 : IV: a902fb72341e7c7201f3dd2866a6a01e←[0m
←[38;20mINFO - wadecrypt.py:231 : Key type: 1←[0m
←[38;20mINFO - wadecrypt.py:232 : WhatsApp version: 2.24.3.81←[0m
←[38;20mINFO - wadecrypt.py:234 : The last two numbers of the user's Jid: 73←[0m
←[38;20mINFO - wadecrypt.py:235 : Backup version: 1←[0m
←[38;20mINFO - wadecrypt.py:238 : Features: [5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 39]←[0m
←[38;20mINFO - wadecrypt.py:239 : Max feature number: 39←[0m

• Next I used waencrypt from the PR Added: backup-version option #111 with the command
  waencrypt 8a359fb873c488f521aee87e3003f27ab376b9799cc39a70142d23e67a90b15c msgstore_merged.db msgstore.db.crypt15 --iv a902fb72341e7c7201f3dd2866a6a01e --wa-version 2.24.3.81 --jid 73 --backup-version 1 --enable-features 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 39 --max-feature 39
  to get my correctly encrypted database msgstore.db.crypt15.
• Then after uninstalling WhatsApp completely and reinstalling it, I simply used this output msgstore.db.crypt15 to let WhatsApp restore this database as a local backup.
  (I only provided the the Database folder com.whatsapp/WhatsApp/DataBases containing only the encrypted msgstore.db.crypt15 file and the Media folder com.whatsapp/WhatsApp/Media with all my merged media, but nothing else)
• And that's all. It worked like a charm and now my WhatsApp looks as if my merged database has always been there for ages. 😎

@ElDavoo So as already said, the problem is solved now from my point of view and you can merge my PRs and/or close this issue if you like. That's up to you, of course. Anyway, thanks a lot for your support and this repo again!

Hey there,
this is my first post and im very new to "programming" anything at all. Yesterday i installed python+sql...
My Problem is as follows:
I have saved an entire "com.whatsapp" folder from my phone (1st March 2024) with a key file and a corresponding decrypted msgstore.db,without the added crypt12,14,15... (no idea if the key file was from a crypt12,14,15...).
Now i had to redo my phone and i tried to sell whatsapp the msgstore.db file but without success.
so i spend the entire yesterday evening trying to find a program to re-ecrypt my msgstore.db file with the key until i stumbled upon this project here (i tried the "whapa"-project aswell and whatsapp can "read" the backup file but gets stuck on "preparing" -.-)

Then after all the installing and figuring out how to actually type stuff in cmd i managed to test my abilites with a newly created backupfile from whatsapp. (got backup from com.whatsapp\Whatsapp\Databases and noted the 64-digit keyfile in a textfile.)
It successfully decrypted and i could open the file via SQlite (no idea if you can see the header-info there :/ )
Then i saw your post here and tried the stuff with the --header-info, after downloading all of your changed files and replacing the originals in the corresponding wa-crypt-tools folders with yours.
Sadly this happened:
wadecrypt: error: unrecognized arguments: --header-info

do i have to somehow reinstall the wa-crypt-tools with your changed stuff and if so, how? :D
I have the same problem with your changed "waencrypt" command and all of the headerinfo you put in there, doesnt work here aswell :/

Also is the step with the merging msgstore.db files necessary to "convert" my old msgstore.db into a newer version of ".db" so it can correctly apply the crypt15 thing?

I am very sorry if you read this and have to facepalm at the usage of my "progamming terms" and also at my english (i'm german)

I look forward to a response, TY

[kingbtcvl]

I am very sorry if you read this and have to facepalm at the usage of my "progamming terms" and also at my english (i'm german)

No worries, we all have different skill levels and I guess I can understand you. Moreover your english is pretty decent.

I have saved an entire "com.whatsapp" folder from my phone (1st March 2024) with a key file and a corresponding decrypted msgstore.db,without the added crypt12,14,15... (no idea if the key file was from a crypt12,14,15...).

Well, if you have a copy of the entire "com.whatsapp" folder, then you also have your encrypted database msgstore.db.crypt14, don't you? I mean you should find your encrtypted database in the subfolder "com.whatsapp/WhatsApp/DataBases". Because if that's the case, why would you bother with re-encrypting your decrypted msgstore.db at all instead of simply letting WhatsApp restore from your msgstore.db.crypt14 file? So I would reccomend to check this out first.

It successfully decrypted and i could open the file via SQlite (no idea if you can see the header-info there :/ )

I would assume that you can't see the header infos here because the header is technically no part of the actual database.

Then i saw your post here and tried the stuff with the --header-info, after downloading all of your changed files and replacing the originals in the corresponding wa-crypt-tools folders with yours. Sadly this happened: wadecrypt: error: unrecognized arguments: --header-info

do i have to somehow reinstall the wa-crypt-tools with your changed stuff and if so, how? :D

Yes, just replacing the files won't help, You need to rebuild the executable files wadecrypt and waencrypt.
I would recommend you to uninstall wacrypt-tools completely via python -m pip uninstall wa-crypt-tools and then reinstall the version from the PR #112 via python -m pip install . (after navigating to the folder where you saved the PR repository).
No need to manually download the files separately and replacing the originals ones by the way. Just go to the repository of the PR (https://github.com/kingbtcvl/wa-crypt-tools/tree/headerinfo) and download the whole thing there. Good luck!

[kingbtcvl]

And that was the information I needed. I could encrypt my merged DB and WhatsApp recognized and restored it. There are only minor issues like missing preview in the chats overview that will be fixed by updating the merge-code and then everything should be working just as I wanted it.

Sounds as if you nearly made it. Congratulations! 🥳
Making WhatsApp recognize your encrypted DB was probably your last real obstacle.
I mean if you have more or less code to merge the crucial parts of the database, i.e. the messages, then those minor issues should not be too hard to fix, too. I would recommend you to go step by step through every table of the databases and then merge them like you merged the stuff you already successfully merged. There quite many tables, but the principle of merging is always the same, so it's just a matter of time.

[Marvolo1590]

A couple of minor issues are left, such as the missing preview in the chats' overview or that the search function doesn't always work properly for all the merged parts. Did you encounter similiar issues or did your merge work 100% for the first time? I feel it's a lot of trial & error. The disadvantage of using the re-encryption of the merged database is that you cannot afford to trial & error too much because you're wasting verification codes each time. But the root method of placing the unencrypted db right into the root directory doesn't work for some reason.

[kingbtcvl]

Did you encounter similiar issues or did your merge work 100% for the first time?

Well, it took me four tries in total. I wasted two unsuccessfull tries where my merged database was not even loaded due to the incorrect header and my first try with the correct header that was successfully loaded by WA turned out to be a little messy, too, because some preview messages in the chat overview were wrong and seemed completely random.
However, I could correct all that with my fourth try and now everything looks and works as if I had always used the same WA on the same phone for ages.

I feel it's a lot of trial & error.

I guess that if one completely knew which values in which clolums of which tables of the database need to be changed then it would not be trial and error. So the more one really knows about the structure of the database the more one knows what to change for a perfectly merged database.
The problem is of course that there is no official documentation so that we have to do a lot of guesswork here. On the other hand studying the database closely and haveing some experiences with databases in general one can teach oneself concerning the structure of the WA database because a lot of things are a kind of self-explaining due to their names.

The disadvantage of using the re-encryption of the merged database is that you cannot afford to trial & error too much because you're wasting verification codes each time.

And yes the re-encryption method has the disadavantage that you need to register your phone number again and again.
That is a little risky in my opinion  because one might get blocked permanently if one tries too often since WA might start to assume that something is fishy. I mean they start to block you temporarly for a few hours after more than one try. I at least felt the threat of a permanent ban the whole time eventough I just did not try too many times. So be careful here! Having your phone number blocked forever would be a damn anoying thing (for most people).

[Marvolo1590]

And yes the re-encryption method has the disadavantage that you need to register your phone number again and again.
That is a little risky in my opinion because one might get blocked permanently if one tries too often since WA might start to assume that something is fishy

I'm having the same worries, but I checked the terms & services and there is no clause stating that you could be perma-banned if you re-register your number too many times. As long as I'm able to enter the correct verification code and my 2FA code, I hope everything's fine. I mean, what if my phone doesn't work properly and I just HAVE TO re-login several times? Would WhatsApp demand I have to buy a new phone or repair it just so I will stop re-logging in so many times? That shouldn't be any of their business....

That said, I have now managed to completely avoid having to re-login as I can now easily switch between pre-merged Database and (experimental/testing) merged database (but WiFi disabled, so that no new messages will be coming in to the merged DB which then I will again have to consider for merging) simply by exchanging the entire com.whatsapp folder in the ROOT directory. It's kinda like restoring a backup of the entire WhatsApp system folder and depending on which copy I place there, once I open WhatsApp it will show either the merged session or the pre-merged session. So now I can just experiment inside the merged session until everything is ready for the truly final merge.

---

I have another (hopefully final) question:
So my merge is more or less sucessful. Everything works now except for one tiny problem which both of us couldn't solve. I was wondering if you ran into the same problem or if you even know how to solve it.

It's about the chat search function. After the merge, chat search function only works if I use the global search function (which is NOT inside of an individual chat but outside in the chat overview list). However, if I enter an individual chat and use the search function there, it will not find words that are part from the merged section of the chats. Only those that are part of the original "pre-merged" database.

What we found out so far is that the table "message_ftsv2_content" is responsible for chat searches. But since it is an index table, it is kind of hard to merge or at least it didn't quite merge for us.
Inside the "message_ftsv2_content" table, "docid" corresponds to _id from message-table. "c0content" corresponds to text_data from message-table.
c1fts_jid corresponds to _id from jid (1 message from me, 0 from the other person. If neither 0 or 1 is first, then it's a jid _id of a sender in a group chat and second is jid _id of the group itself).

Apparently, WhatsApp seems to be using a coding system here which links certain letters of this c1fts_jid table to the actual JID. We haven't figured out the algorithm for this JID-to-letter-coding here but this seems to be crucial to make the chat search function work after the merge.

I'm wondering if it immediately worked for you with your merge or perhaps if you know how to fix this?

Thanks a lot, man!

PS: I checked your merge script. I didn't find any instance where you're dealing with the "message_ftsv2_content" table, so I'm really wondering if chat search function works for you 100%.

[kingbtcvl]

I'm having the same worries, but I checked the terms & services and there is no clause stating that you could be perma-banned if you re-register your number too many times. As long as I'm able to enter the correct verification code and my 2FA code, I hope everything's fine. I mean, what if my phone doesn't work properly and I just HAVE TO re-login several times? Would WhatsApp demand I have to buy a new phone or repair it just so I will stop re-logging in so many times? That shouldn't be any of their business....

😂 Just because they don't mention that in their terms & services, doesn't mean they can't and won't perma-ban anybody. In fact I read more than once that some usere exprienced perma-bans when I read through forums trying to figure out all the stuff. So watch out and don't feel to save here.

So now I can just experiment inside the merged session until everything is ready for the truly final merge.

It's nice when you found a way to be able to expriment as much as you want to. This should help you not to get into trouble concerning a perma-ban then I guess.

I have another (hopefully final) question: So my merge is more or less sucessful. Everything works now except for one tiny problem which both of us couldn't solve. I was wondering if you ran into the same problem or if you even know how to solve it.

It's about the chat search function. After the merge, chat search function only works if I use the global search function (which is NOT inside of an individual chat but outside in the chat overview list). However, if I enter an individual chat and use the search function there, it will not find words that are part from the merged section of the chats. Only those that are part of the original "pre-merged" database.

I'm wondering if it immediately worked for you with your merge or perhaps if you know how to fix this?

I just checked that since I rarely use the search function: On my database everything works like the charm, i.e. the global search as well as the individual search for specific chat finds every term in the old part as well as in the new part of the merged database. Absolutely no problems here.

PS: I checked your merge script. I didn't find any instance where you're dealing with the "message_ftsv2_content" table, so I'm really wondering if chat search function works for you 100%.

Yes, I did not do anything with that table and I just looked at my old project and found out why: This table simply did not exist when I merged my databases. When I merged my database I definitely handled all non-empty tables. However, since I could remember something with fts I tried to find out where I came accross this.
So I even had a closer look and finally found out that in my props table there was an entry like 830,fts_ready,0, which kind of suggestin that whatever this table message_ftsv2_content does now is something that they started to prepare a few months ago and what they have ready with the current version of WA now. In fact fstv2 seems to suggest that this is already the second verion of it (v2). So sorry, I can absolutely not help you here.

[haare123]

Hey, thank you first of all for your fast reply! <3

Well, if you have a copy of the entire "com.whatsapp" folder, then you also have your encrypted database msgstore.db.crypt14, don't you? I mean you should find your encrtypted database in the subfolder "com.whatsapp/WhatsApp/DataBases". Because if that's the case, why would you bother with re-encrypting your decrypted msgstore.db at all instead of simply letting WhatsApp restore from your msgstore.db.crypt14 file? So I would reccomend to check this out first.

I only have the com.whatsapp from i guess was the install directory. there is no .crypt file anywhere to be seen, since i did not make a "backup" via whatsapp apparently :/ the Media Folder by itself is empty now ...

It successfully decrypted and i could open the file via SQlite (no idea if you can see the header-info there :/ )

I would assume that you can't see the header infos here because the header is technically no part of the actual database.

Then i saw your post here and tried the stuff with the --header-info, after downloading all of your changed files and replacing the originals in the corresponding wa-crypt-tools folders with yours. Sadly this happened: wadecrypt: error: unrecognized arguments: --header-info
do i have to somehow reinstall the wa-crypt-tools with your changed stuff and if so, how?

Yes, just replacing the files won't help, You need to rebuild the executable files wadecrypt and waencrypt. I would recommend you to uninstall wacrypt-tools completely via python -m pip uninstall wa-crypt-tools and then reinstall the version from the PR #112 via python -m pip install . (after navigating to the folder where you saved the PR repository). No need to manually download the files separately and replacing the originals ones by the way. Just go to the repository of the PR (https://github.com/kingbtcvl/wa-crypt-tools/tree/headerinfo) and download the whole thing there. Good luck!

so i successfully re-installed this package with the PR#112 update and the decrypted --header-info worked out well.
now i got my old msgstore.db with the key file and the header from the "new" backup from whatsapp .crypt15 and the correspondeing 64-key.

now all i have to do is this "merging" part you were talking about, so i convert my msgstore.db into the new .db from to encrypt it with the 64-key or does the old msgstore.db work with the 64 key by itself?

Thanks in Advance <3

[haare123]

Edit:
i tried to recrypt my old msgstore.db with the 64 key and the header info i received from the new backup and i received this error message:

waencrypt: error: unrecognized arguments: --backup-version 1

what to do? :/

[kingbtcvl]

so i successfully re-installed this package with the PR#112 update and the decrypted --header-info worked out well.
now i got my old msgstore.db with the key file and the header from the "new" backup from whatsapp .crypt15 and the correspondeing 64-key.

Great, the sounds good!

now all i have to do is this "merging" part you were talking about, so i convert my msgstore.db into the new .db from to encrypt it with the 64-key or does the old msgstore.db work with the 64 key by itself?

Well no, actually you don't really want to do the merging process as far as I can see.
I did that because I wanted to combine two databases to get a merged database that holds all the data of the two single ones that I had. But you only have one database msgstore.db that already holds all data you want to have and you want to re-encrypt it and then restore that one in WhatsApp. Right? So essentially what you want to do is basically something like
waencrypt 8a359fb873c488f521aee87e3003f27ab376b9799cc39a70142d23e67a90b15c msgstore.db msgstore.db.crypt15 --iv a902fb72341e7c7201f3dd2866a6a01e --wa-version 2.24.3.81 --jid 73 --backup-version 1 --enable-features 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 39 --max-feature 39
of course, using not these parameters, but those ones you just got from reading your header infos.

i tried to recrypt my old msgstore.db with the 64 key and the header info i received from the new backup and i received this error message:
``
waencrypt: error: unrecognized arguments: --backup-version 1

what to do? :/

Well, you are very close, bro. For using the waencrypt with the --backup-version option you must uninstall your current wacrypt-tools again and use the one from PR #111 (i.e. the one from the repo https://github.com/kingbtcvl/wa-crypt-tools/tree/backupversion). That's because I created two different pull requests.

[haare123]

Hey, thanks for your kind words and im really proud of myself already hehe

Well, you are very close, bro. For using the waencrypt with the --backup-version option you must uninstall your current wacrypt-tools again and use the one from PR #111 (i.e. the one from the repo https://github.com/kingbtcvl/wa-crypt-tools/tree/backupversion). That's because I created two different pull requests.

ok, can u please show me step by step how to properly uninstall + install this pull request, because i cant seem to get it working :(

C:>git clone https://github.com/kingbtcvl/wa-crypt-tools/tree/headerinfo
Cloning into 'headerinfo'...
fatal: repository 'https://github.com/kingbtcvl/wa-crypt-tools/tree/headerinfo/' not found

C:>git clone https://github.com/kingbtcvl/wa-crypt-tools/tree/headerinfo.git
Cloning into 'headerinfo'...
fatal: repository 'https://github.com/kingbtcvl/wa-crypt-tools/tree/headerinfo.git/' not found

this is the stuff i get from cd...

[ElDavoo]

I wonder why noone ever wrote such a thing just as they did for Skype...

That's how open source works. Nobody pays open-source developers, they work in their free time. I developed wa-crypt-tools because of personal necessity and it's has been somewhat easy, but dealing with a complex DB schema that will change at every update is something nobody is going to do for free, for free time or for hobby.

The guys that developed the proprietary tools got paid for doing that and worked 8 hours a day, and still as you noticed their tools have bugs. For someone that has 1-2 hours of free time a week, it will take a lot before an open-source program would work.

Even when it works, it's going to be a hassle to maintain. For example, in whapa there is a CLI tool whamerge, it doesn't look maintained, but you can try it and see what happens.

[Marvolo1590]

Yes, I understand. I didn't mean to sound accusing or anything, I didn't even mean you guys. I was just generally wondering why something that sophisticated such as Skyperious exists for Skype but not yet for WhatsApp. As I understand it, Skype is not any less complex than WhatsApp I suppose?

Well, anyway. I'm still hoping there might be a way to fix this without needing to ask their support and giving them my chats. I mean, both on this thread as well on the XDA super thread there have been successful attempts merging databases and getting them to work even though the methodology seems to be varying over the years and with every WA update.

[Marvolo1590]

PS: I just contacted MobileTrans via Chat... Don't know how tech-savy those support chat people are but it doesn't seem to be going anywhere. They just asked if I had tried to run the transfer progress again.. Which I cant do because I don't even have the iPhone anymore as it was only for testing. I sent them the tables from above with my markings, now they just remain silent.

[kingbtcvl]

At first, it seemed to have worked again. All the chats were there, all the media.. But then, when I scrolled back to the point BEFORE I first transfered to iOS, all the chats were kind of damaged in a way that only my own responses remained visible in the chats but not the ones from the respective chat partners. So it looked as if I just wrote endless monologues, several answers with hardly ever a reply from the chat partners. The only time chat partners' content was visible was when it was tied to a quoted message or media subtitle. Then I could see their answers, but not on the left side of the UI as it would be the case, but strangely centered in the middle, seemingly not belonging to either the left or the right side of the chat.

As we can see by comparing the corrupted table with the correct one, the missing chats from the respective chat partners are also showing in the corrupted tables based on several values. I have no idea what has caused this.

Well, looking at your screenshots and remembering the things I leanred when I dealt with my databases I have a rough idea what happened here. The converting process completely messed up and did obviously convert some columns incorrectly leaving out values. Where this is rather harmless for some columns like receipt_server_timestamp it causes brutally wrong outputs on columns like sender_jid_row_id because this is as explained above the id for the telephone number. Hence, altough the messages some people send you are in the database the missing information of which number send this message to you, i.e. the missing value in the column sender_jid_row_id, WA has no chance of displaying the messages in the correct chats.
My guess is that their software is not up to date with the latest structure of the database, which is generally the biggest problem. I beleive that because I have found out the way call logs were represented was quite recentyl changed by WA and the fact that you lost them when you origianlly converted your database to ios is a hint that hey have not dealt with theses changes.

Having said this, the earlier part of your database is completely messed up and useless. Theres is no realistic chance to repair that. But since you have your original android backup for that part, this is not that bad. However chances are that the other part from January to April was also converted poorly are not too low eventough everything seemed fine in WA with that part.

Anyway, if there is a chance to merge a completely flawless database at all, you best chances are starting with the old android backup and append that one with the (hopefully) uncoruppted part of your from-ios-to-android-converted backup. In my opinion trying this only makes sense if the ids of the ios-to-android-converted backup remained unchanged. To check this compare the tables jid and chat as descripted above. If the ids are different then I have to honestly tell you that I don't think that someone who is unexperienced can make it. That's because the merging process will involve identifiyng the correct ids, converting them by comparision with the old database and therefore appending modified data to the database. That would be so complicated that it cannot be done manually in an efficient way anymore and you would need to write a programm for that.

No-one made a good open source DB manager so you're stuck with proprietary payware tools. Since you paid them, it might be a good idea to contact their support.

@ElDavoo made a great comment here. You paid for their tool and therefore have some right for being supported by them. Tough you should definitely not send them you database for privacy reasons I would bug them so long until someone looks at the code and makes it up to date. It is not good sign though that they already stopped reacting, but bein persistent often leads to success.

[Marvolo1590]

But since you have your original android backup for that part, this is not that bad. However chances are that the other part from January to April was also converted poorly are not too low eventough everything seemed fine in WA with that part.

Thanks for still making me at least a bit of hope there!! I was all discouraged now. Well, the iPhone chat part (everything from January 5th up to the end of testing in late March) was displayed fine in WhatsApp, even AFTER I restored it back to Android. My assumption is that everything made on iOS was good and was transfered back to Android correctly, but then everything that was made before going to iOS was messed up upon returning back to Android.
So it really only seems to be affecting everything before the first / initial transfer to iOS. Even though I'm currently not using the iOS (Jan-March) database any longer, I have it decrypted so I can view it in DB Browser. The chats made on iOS up until the end of testing the iPhone look like this (were displayed correctly in WA as far as I remember). I didn't bother to censor the chats as there is nothing important written there anyway (I'm German btw :P):

And here we can see the threshold or border between the faulty chats leading up to Jan. 5th and "good" chats, then produced with the iPhone after the transfer to iOS. As far as I remember the OLD chats prior to the first transfer were displayed right too on the iPhone, so the error must have happened during the SECOND transfer (back to Android). Left table is the correct database, right side the faulty one which is then followed by the correct ones after the switch to iOS:

I'm lacking the knowledge on whether this is serious or not, but my amateurish attempt would be to just copy everything that is correct after Jan. 5 and extend it to the correct database which stops on Jan 5. But I don't know HOW. I know that on the XDA thread they did it with the DB Browser software somehow, but I'm not familiar with that one either. I don't know how to cut out all these columns and insert them in a different database. Then I would have to repeat that because in the meantime I've been using WA of course and new data has been produced...

You paid for their tool and therefore have some right for being supported by them. Tough you should definitely not send them you database for privacy reasons I would bug them so long until someone looks at the code and makes it up to date. It is not good sign though that they already stopped reacting, but bein persistent often leads to success.

They answered back and asked me to send the log files of the software, which I did. I also sent them the screenshots from above with my markings as well as an SQL export of both the databases in DB Browser (I exported "scheme only" so as not to give them the chats). Don't know if only the schemes might be useful to them without the chats, but whatever.

I'm not counting on them though and even if they can somehow fix it, I'd still have to merge some databases myself as I have been producing continous data on WhatsApp of course.

[Marvolo1590]

Anyway, if there is a chance to merge a completely flawless database at all, you best chances are starting with the old android backup and append that one with the (hopefully) uncoruppted part of your from-ios-to-android-converted backup. In my opinion trying this only makes sense if the ids of the ios-to-android-converted backup remained unchanged. To check this compare the tables jid and chat as descripted above. If the ids are different then I have to honestly tell you that I don't think that someone who is unexperienced can make it.

I have opened both the CHAT and the JID table in DB browser. But I don't really know what I should be looking out for. Both look rather similiar to me. Since no chat-text is being displayed in both of these tables, I have a hard time navigating and finding the "threshold" between faulty chats and good chats just based on displayed phone numbers and some strange values.
Here's a screenshot of the CHAT table displaying _id & jid_row_id for both the faulty database and the correct one:

Correct Database:

Faulty Database:

As we can see, the FAULTY Database displays an orderly count beginning from 1, whereas the CORRECT database begins at 11 and has a more chaotic numbering in the jid_row_id column.

But both show the beginning of the database. Like I said, I have difficulties finding that "threshold" between faulty chats and good chats on Jan. 05.

As for the JID tables: both show about the same kind of content (sometimes phone numbers, sometimes other numbers I can't recognize). The only difference I can see is that the corrupt database has a shorter list inside the JID table whereas the GOOD database has much more content to scroll down (331 vs. 1137 entries), but I have been using the "good" database for chatting up until today, like I said so naturally there would be more entries..

I still have a tiny bit of hope that something can be done about this problem. I just haven't figured it all out yet.

[kingbtcvl]

Well, at first glance it looks as if you are in trouble and just copying the data starting from January will not work and yield a flawed database again. (Besides that the process of copying unmodified (or even simpliy deleting) data from one WA database to another is way more complicated then you think because you need to handle the triggers of the database and will need to work with SQL commands. It's not an easy ctrl+c and ctrl+v 😂)

However in order to give a final judgement of what is going on in your database, I would need to see the first lines of thejidtables, too. So if you could upload two images of them, too, I could probably tell you how complicated the merging process would be. Of course, censor the telephone numbers in that images, but leave the last three digits visible because otherwise one can't really analyze this table.

[Marvolo1590]

Besides that the process of copying unmodified (or even simpliy deleting) data from one WA database to another is way more complicated then you think because you need to handle the triggers of the database and will need to work with SQL commands. It's not an easy ctrl+c and ctrl+v 😂)

Oh yes, I'm aware of this. I've been trying to work my way through this SQL merging stuff and couldn't make any sense of it.

However in order to give a final judgement of what is going on in your database, I would need to see the first lines of thejidtables, too. So if you could upload two images of them, too, I could probably tell you how complicated the merging process would be. Of course, censor the telephone numbers in that images, but leave the last three digits visible because otherwise one can't really analyze this table.

Thanks so much in advance for all the effort you're putting into my silly little problem!!

So here's the faulty database, one is the beginning (top), one is the ending (bottom):

JID Beginning (Faulty Database):

JID Ending (Faulty Database):

The ones for the correct database will follow in a few minutes!

[Marvolo1590]

Here is the "correct database" - beginning and ending:

Beginning:

Ending:

[kingbtcvl]

No good news, unfortunately. My guess that just copying things will not work out seems to be correct. 😒

So let me try to explain you how WA reads all this data in order to display it and then you might understand why things don't look good and just copying data will not work and you would be forced to also make modifications concerning the IDs of the tables if you wanted to have a flawless merged database.

WA starts by looking at the chat table since this represents all the chats. For every _id in the chat table it will show one chat in the the chat overview screen. In order to know which phone number (or WA group) this chat _id belongs to it reads the jid_row_id in the same row of the chat table and memorizes it. Then WA looks at the jid table, goes through all rows until it finds the row where the _id in the jid table is equal to this memorized value. Examining this row it can get the phone number by evaluating the value of the user and/or raw_string column. So now WA can link each chat in the chat overview to a phone number (or group of phone numbers).
Next, when you open a chat in the chat overview to read the messages of this specific chat, WA needs to display the messages of that chat, of course. In order to do so, it looks at the message table and goes through all rows. Since WA already knows the _id of the chat you opened it compares that number with the value of chat_row_id in the message table. Whenever it finds a row where the known _id from the chat table is equal to the number given in the chat_row_id WA knows that the message in this row belongs to the chat that was opened. Then it only needs to identify who sent that message, so it looks at the value given in the send_jid_row_id column and you probably already guessed it, goes back to the jid table to identify the telephone number who send this message. Having fetched this information it can display the message in the opened chat.

Altough this is an extremly simplified explanation it should give you a clue of how the database is read by WA.
Now that we know this, let us see what this means looking at your two databases and its contents.
So in the correct database the chat with the _id = 16 is linked with jid_row_id = 7, which correspondends to a phone number ending with 946 as given by the user column of the jid table of the correct database. On the contrary, in the faulty database the chat with _id = 16 is linked with jid_row_id = 16, which correspondends to a phone number ending with 350 as given by the user column of the jid table of the faulty database. So far so good.
Now let us assume you wrote some messages on your iPhone to the person whose numer ends with 350 in January. Then the message table of the faulty database will contain some rows in which chat_row_id = 16 and sender_jid_row_id = 16. These rows will be part of the uncorrupted entries of the faulty databse you want to merge to the correct database.
Now assume you just delete all corrupted entires of the faulty database and simpliy copy all uncorrupted entries of the message table of the faulty database (so the entries starting in January which also contain the conversation with the person whose phone number ends with 350) to the message table of the correct database.
What happens? Well, for those messages send by the person with the phone number ending with 350 the rows in the message table will still contain chat_row_id = 16 and sender_jid_row_id = 16 since you simply copied them. However, in the correct databse according to the user column of the jid table sender_jid_row_id = 16 does not correspond to the phone numer ending with 350 anymore, but rather corresponds to the phone number ending with 551. Besides, the chat table of the correct databse says that the chat _id = 16 is linked with jid_row_id = 7, which yields an inconsistency.

Do you see the problem here?!
I don't know how the output would like or if WA would output anything at all. But even if WA ignores some inconsistencies and outputs something, you might end up with some weird stuff like messages which were orignally send by the person whose phone number ends with 350 being put in your chat with the person whose phone number ends with 551.

Therefore I am quite sorry to tell you that the merging process here is way harder the you think because you need to take care and modify the IDs of the uncorrupted part of the faulty database so that they are consistent with the correct database.
Not sure if this is worth the effort, especially when you don't have much porgramming experience.

[Marvolo1590]

Wow, thanks a LOT for all the effort you put into trying to help me. Even though the outcome is rather sad. I could understand everything you said and also could make sense of it!

And some of the things you described were mentioned exactly by users on the XDA thread:

But even if WA ignores some inconsistencies and outputs something, you might end up with some weird stuff like messages which were orignally send by the person whose phone number ends with 350 being put in your chat with the person whose phone number ends with 551.

I've read that a couple of times on the XDA thread, that messages turned up in the wrong chats or groups and such...

I'm quite astonished at how badly this software messed up my chats. I haven't heard back from them - the support guy said he forwarded everything to their software engineers. But what can they possibly do about it? They'd need access to my databases to fix anything, right? And then again, I've already started new chats using the January (good database) backup.. So there's now a gap between January and today in my current chats that they would have to consider.

What would be left for ME to do? Do you think there would be an automated way / script to fix these errors? Maybe if I tried to find someone who's familiar with SQL and this stuff? I'd be even willing to pay something.

I mean: you're partly right - those Jan-Apr. chats are not suuuuuuuper important and yet there are some messages that I'd like to keep. I mean, I have the decrypted database, but as you can see, DB Browser is no good for reading messages.
"Skyperious" has a nice feature embedded which lets you export your Skype "main.db" databases in HTML and it turns out beautifully: you'll even get the Skype chat design with those different coloured text bubbles and all that. Media files via hyperlinks...

If something similar existed for WhatsApp databases, I could probably live with that. I'd then export the rest of my working chats and try to merge them all via HTML for viewing on my computer. Then, they'd be reunited there. But I haven't come across any good solution. The old "WhatsApp Viewer" is really basic and doesn't even show media files.

---

On a related note:
You've now explained to me in amazing detail how WhatsApp works. I'm amazed at how much you seem to know about it.
What do you say about the following endeavour:

On my other WA account (running on a different number) I have a couple of private chats. That number used to be my private number but it's now my business number, but there are still some private chats on there.

Do you see any way to take certain chats from that number and move it over to my actual account on a different number and there reunite/merge them with the respective chats?

Even though I must say that the number of the respective chat partner is not the same anymore. So the task would be: taking chats from a person made with number A from my 2nd account to my actual accound where this person now uses number B.

So not only my number is different but also that person's number is now different.

It's probably not possible to merge all that, right? Though I think I read something on this thread here about someone taking chats from an old number to a new one by changing the headers or something.

Anyway, thanks a LOT again for your effort!! I'm still amazed.

[kingbtcvl]

I'm quite astonished at how badly this software messed up my chats. I haven't heard back from them - the support guy said he forwarded everything to their software engineers. But what can they possibly do about it? They'd need access to my databases to fix anything, right?

Yeah, probably they won't be able to help you. But then again they might be able to idtenify what went wrong when you converted your database back from ios to android, fix it and you could then convert it again assuming you still have the ios database, of course. There is also some chance they identify it without access to your database, though chance are not too high to be honest.

What would be left for ME to do? Do you think there would be an automated way / script to fix these errors? Maybe if I tried to find someone who's familiar with SQL and this stuff? I'd be even willing to pay something.

Well, yes, I do think it's very likely possible that a good programmer who is also familiar with SQL would be able to write an automated script that puts together your chat history pieces so that the database works flawlessy. I mean you seem to have them all, the part before January is in your correct old database, the part between January and March seems to be uncourrupted in your ios converted database (though having different IDs) and the very latest chats are in your current database with the gap between Januaray and Februrary.
In fact when I merged my stuff I had to tackle quite similar problems, too. However since my code is very specififc to my problem one would need to adjust that to your problem, i.e. you can't simply take my script run it on your problem and expect everything to work. One would likely need to add things, take away some other stuff and carefully study your databases while doing so to identify all other problems that I did not have and therefore not treated. Since you are unexperienced with all that stuff giving you my code would be like if you had a broken car and someome gave you the key to his garage and would allow you to use his tools and repair the car yourslef, but you are far from being a mechanic. But of course, if you think it might help you in any way I will upload my code.

If something similar existed for WhatsApp databases, I could probably live with that. I'd then export the rest of my working chats and try to merge them all via HTML for viewing on my computer. Then, they'd be reunited there. But I haven't come across any good solution. The old "WhatsApp Viewer" is really basic and doesn't even show media files.

Unfortunately I don't konw of any good WA viewing programm either and I don't think that any really useful exists at the moment.

---

Concerning your related note not my guess is that this would be possible in principle, too. I mean as I have explained to you the telephone numbers are kind of masked by the _id given in jid anyway. So one would "simpliy" just need to itendify the ids in both databases and correlate the persons. That would require some manual work since the number are not the same anymore. But having identified which numbers belong to which persons one could write a programm and merge the stuff. But again that needs a certain level of erperience.

[Marvolo1590]

Thanks.
In fact, based on your information so far, I have started working on it myself. Since I don't know any automated way to do it, I've started doing it manually. I have identified the chats in the corrupted database that now have different ids and JIDs and by looking at the good database, I know, what IDs and JIDs they are supposed to have.

In DB Browser I can manually just change those numbers. The problem is: it's too many entries to do it by hand. If DB Browser had, just like Excel, an option to "replace all 5s with 7s" that would already help, as I could then change all the IDs at once.
The other problem is that this method only applies to the messages-table.

But there are so many other tables of which I don't know IF and WHAT needed to be changed accordingly.

So my guess was: use the good database and ALL its tables as a starting point and then only copy everything after Jan. 5 into the good table, leaving the past as it is.

That would mainly include the messages table with the correct IDs and JIDs. The JID table probably doesn't have to be altered - the contacts and all their numbers stayed the same on iOS after all.
For the Chat table I'm not sure if changes have to be done. Probably, since they display the current chats.

But then there's so much more: call-logs, quoted-messages etc. Luckily, most of those tables are empty and don't have any entries.
Also, with many tables I have a hard time knowing my way through time, as many don't have chats or UNIX time codes to help me know where the "good" database from January onwards starts.

So in theory, I think I know how to fix at least the messages table by just changing the IDs and JIDs back to the correct number. Everything else (the message, time stamps, hash values and all that) should remain the same. But I'd need a way in DB Browser to quickly replace one value with another (change all 5s to 16s), because by hand it's not possible.

PS: I really hope it's not asking too much and I don't want to sound demanding or anything - but I can see that you've seemed to have delved into all this pretty well and know a lot about how WA works. If I offered you an appropriate compensation, could you imagine looking at my case and fix it, since you already know all the key points of my case? I could understand though if you've got too much to do as it is and I would of course never mind.
I just don't know where I should start looking for someone who knows a lot in this very specific tech environment.

Thanks a lot! :)

[kingbtcvl]

Well sorry, I don't think I have enough time for that. It took me about 100 hours to merge and then re-encrypt that stuff successfully and I have no concrete idea how deep your rabbit hole is, but I don't expect it to be less deep as mine was

[Marvolo1590]

No worries, it's all fine =)

[kingbtcvl]

I have started working on it myself. Since I don't know any automated way to do it, I've started doing it manually.

Dude, that's complete madness. 😂
Honestly, I only gave you a rough idea of what to do, but there are so many more things to do and consider. In my humble opinion trying that manually is as if you would try to cut down a thick tree using a buttered bread knife. Chances are you will never succeed wasting many hours. Don't say I did not warn you.

So my guess was: use the good database and ALL its tables as a starting point and then only copy everything after Jan. 5 into the good table, leaving the past as it is.

That would mainly include the messages table with the correct IDs and JIDs. The JID table probably doesn't have to be altered - the contacts and all their numbers stayed the same on iOS after all.

Yes, that's the smartest way to go. The chat table would need modifications, too tough in that case.
Moreover the jid appears in quite a number of tables, which all would need to be adjusted. Also the chat ids appear in other table.

And one more warning. The darabase contains a lot of triggers. Whenever something is added or updated in the database some triggers might get active, moifying, updating or deleting other data in the database. That happens automatically wihout you having any real control. Since there were too many I never had a closer look at them, but my gut feeling is that they might mess up things even more when you try to correct things. To put it differently when you modify your database to become correct you definitely do not want the behaviour of the triggern altough many might be harmless.
The way I handeled this in my programm was fetching the creation commamds of all triggers from the database, saving them, deleting all triggers, modifying the data and then recreate the triggers using the saved commands.

[Marvolo1590]

Update:
The support team just wrote me an e-mail asking to "please confirm if the most recent backup in the app (Mobile Trans App) shows the complete chat data.
If the backup data is incomplete and there is no original device to create a new backup, it will be impossible to analyze the problem. Sorry about it."

I opened up the Mobile Trans Software on my computer, hit "restore" and yes, the iOS backup as of March 2024 is still there. I can open it there and it shows me my contacts and all the chats (like some miniature WhatsApp). I'm now scrolling up all my chats to check and see if "the past" is displayed correctly which seems to be the case.

Like I assumed, something must have gone wrong DURING the transfer from iOS back to Android. The transfer from Android TO iOS seems to be correct (with the exception of missing call logs because for some reason they don't copy call logs. I wonder if I could insert them myself...)

So yes, it seems as if the iOS backup right before I switched back to Android has working chats, no mess-ups. I wonder what they're gonna do with that piece of information now...

But interestingly, this software also offers an option to "merge" chats. Apparently, you can copy one WhatsApp to another and hit the option to "keep existing chats". I wonder if that might help me.. But I'm too suspicious because it has already messed up everything for me.

[kingbtcvl]

Nice, it seems as if they care about your problem. It's also good news that you could restore the iso backup as of March 2024.

Have you already told them that you also used their software for originally converting the Android chats to ios?! If not, I would recommend you to do that since this might allow them to reproduce the fauly result with one of their own model databases.

[Marvolo1590]

It's also good news that you could restore the iso backup as of March 2024.

Well, I didn't exactly restore anything. Their software just allows me to look into the backup and it then opens some sort of miniature WhatsApp where I can browse through the chats.

Have you already told them that you also used their software for originally converting the Android chats to ios?!

Yes, I told them the entire process, probably too many times lol. I hope I'm not bugging them lol. Also, since English is not my native language, I'm sometimes insecure how to explain things properly, that's why I usually resort to explaining everything suuuuper detailed in very easy language as if I was explaining it to a 3-year-old. I hope they'll understand what I mean lol.

But then again, the first transfer (from Android to iOS) didn't seem to have messed up anything, except that they're losing call logs. As far as I browse through these iOS backups, I can't see any messed up chats where they are messed up after the final transfer back to Android.

So both my backups are there, the privat one and the business one (run on WhatsApp business). Both accounts got messed up after the final transfer back to Android. I'm surprised that both seem to have EXACTLY the same file size which cannot be true, actually. The private account is a lot larger than the business one.

Also, looking through the backup within their program I found an "export" option. It seems as if WA backups made with their software offer the option to export all chats either to PDF, HTML or TXT. :O That would be a feasible way to make a second backup of all the chats in HTML on my computer, just as Skyperious does with Skype.

I didn't try it as I'm scared to mess with those backups and my subscription has run out anyway (I wonder if they'll ask me to pay again to keep using the program).

Anyway: what exactly was the problem YOU had which as you say was as complicated as mine?

[kingbtcvl]

Also, looking through the backup within their program I found an "export" option. It seems as if WA backups made with their software offer the option to export all chats either to PDF, HTML or TXT. :O That would be a feasible way to make a second backup of all the chats in HTML on my computer, just as Skyperious does with Skype.

Maybe that's in the end the best solution for you and I guess they will likely renew your subscription for free considering it's likely a bug of their programm that caused all this mess.

Anyway: what exactly was the problem YOU had which as you say was as complicated as mine?

My problem was that I did not use the original WA but GBWhatsApp (a clone with more functionality), which I got banned from since Meta seems to have detected that I was using an unwanted app which connected to thier servers. Anyway, I had to switch to WA therefore, but there was no way to restore a GBWhatsApp backup in the original WA and even worse GBWhatsApp backups look a little different. So I started using WA for a few month when I finally started to wonder if I could somehow merge the old GBWhatsApp backup with my latest backup in WA. Well, one of the ostacles was that the Ids used in the GBWhatsApp backup were completely different then those generated in the WA backup, which is quite similar to your problem now.

[Marvolo1590]

@kingbtcvl

On an unrelated note that has nothing to do with the above-mentioned problem:
would you know anything about WhatsApp keys and backups?

I've been trying to restore older backups but as I never copied their key file (my phone was never rooted and also I never knew about that until recently), I now ended up having over 10 years of "backups" on my computer (databases & media files) and as I could always restore them in a relatively short time period after they were created (for example after a phone reset, new phone etc), this led me to believe, these backups could be restored ANYTIME as long as the phone number is the same for identification.

Well, how wrong I was.. The other day I tried to restore a backup from 2015 (same phone number, but missing key file) and WhatsApp just did not even detect the backup let alone offer me an option to restore.

I tested this several times with countless backups and ended up making the following observation:

The last backup WA detects is from December 2023 (!). A backup from February 2023 isn't detected anymore as well as all the other backups further back.

Even though the phone number always stayed the same over the years. It seems as if WA backups only work for a very small period of time (enough time for a phone reset or change of devices), but if you wait for too long, somehow, they cant be detected and restored anymore.

Apparently WA changes or deletes the keys for these backups on their servers. That's how I tried to explain it to myself, because why else would they now stop working even though the phone number stayed the same all the time?

Is there now any way left to get old backups restored or are they now lost forever?

[kingbtcvl]

The last backup WA detects is from December 2023 (!). A backup from February 2023 isn't detected anymore as well as all the other backups further back.

I read different things concerning that question. Some people reported that they could make WA read their old backups athough they knew they had a different key file then their latest ones and hence concluded that WA must have saved old key files on their server. On the other hand some other guys reported what you have just said, that they were unable to restore their old backups in WA which leads to the conjecture that WA does not know theses keys anymoer.

It seems as if WA backups only work for a very small period of time (enough time for a phone reset or change of devices), but if you wait for too long, somehow, they cant be detected and restored anymore.

It's hard to say if that's true in general, but critical points are persumably when WA makes big changes. However, now that we have the possibility to decrypt the databases without a key file end-to-end using a 64 byte key, which you (hopefully) complete control alone such things should not happen in the futute anymore for those who use end-to-end decryption.

Apparently WA changes or deletes the keys for these backups on their servers. That's how I tried to explain it to myself, because why else would they now stop working even though the phone number stayed the same all the time?

Yes, that's the most likely explanation.

Is there now any way left to get old backups restored or are they now lost forever?

Well, assuming that WA has deleted those keys and assuming that you don't have your own copy of them those backups will likely never be encrypted again. Your only hope would be that WA did not delete those old keys and saved them somewhere else. I mean we all know Meta loves data and they tend to not delte anything other then those things that they are forced to 😂

[Marvolo1590]

My problem was that I did not use the original WA but GBWhatsApp (a clone with more functionality), which I got banned from since Meta seems to have detected that I was using an unwanted app which connected to thier servers.

Yes, I shortly used one of those modded Fouad WA versions too, and the funny thing is: they ALL detect my old backups and offer me to restore them after I enter the phone number unlike the official version. Strangely, even those backups were detected which were almost certainly made on a different phone number, which I thought was weird.
However, as soon as restoring finished at 100% it tells me that it could not restore anything.
It's weird that they detect the backups on the phone when the official version doesn't.

Having used one of those modded versions, too, but only for like a couple minutes, I now hope that WA hasn't already flagged my number or account and that they won't ban me from using my official account even... I read a lot of "ban threads" on r/WhatsApp on reddit where people seem to get banned using their official accounts for apparently no reason.

Anyway, I had to switch to WA therefore, but there was no way to restore a GBWhatsApp backup in the original WA and even worse GBWhatsApp backups look a little different. So I started using WA for a few month when I finally started to wonder if I could somehow merge the old GBWhatsApp backup with my latest backup in WA. Well, one of the ostacles was that the Ids used in the GBWhatsApp backup were completely different then those generated in the WA backup, which is quite similar to your problem now.

Oh damn it :D
This is probably just identical (in a way) to the problem I'm having.. And you would know how to fix it. Such a pity that it's so time-consuming to fix it.
Now I just need to find someone who knows the same stuff about it like you do but I haven't been successful yet.
You mentioned earlier that you might be able to provide your code or some information about it? Maybe that might help once I've finally found someone who thinks they can do it. They'll probably be able to make sense of your code and your information, so if you could share what you've learned about how to fix this god-damn problem, it would be much appreciated :D I doubt I will be able to do it myself - or maybe I will be, if I'm spending enough time to learn everything about it. I dont know. I haven't dealt with coding and all that before though.

However, now that we have the possibility to decrypt the databases without a key file end-to-end using a 64 byte key, which you (hopefully) complete control alone such things should not happen in the futute anymore for those who use end-to-end decryption.

And that's the next question I'm having: that 64-letters-key - is it really truly the new "MASTER KEY", meaning it replaces the former key file?? Or is it (and that's what I read many times) only an ADDITIONAL protection of the regular / former key file? Meaning, just with the 64-letters-key alone, you can NEVER restore a database, as you would always need the regular key file as foundation??

Most stuff I read about it is that the 64-letters-key only serves as some sort of key-protected, "first entrance" to get to the database for which you would still need the regular key file to decrypt it.

If it wasn't like that, then there would not be any use for the regular / former key file once you enable E2EE-backup encryption, HOWEVER, even with that enabled, the regular key file is still stored in the WhatsApp directory, as I found out by extracting it.

Well, assuming that WA has deleted those keys and assuming that you don't have your own copy of them those backups will likely never be encrypted again.

A thought that was just coming to my mind how to detect if WA still saves old keys on their servers:
Would there be a way to change the information/header/footer of the old backups based on a most recent backup file, so that WhatsApp can be tricked into recognizing it and offering to restore?

As far as I understand it, the key for a backup (if available on the server) is re-assigned DURING or AFTER restoring the backup. Since I never saved any keys and all those backups always worked on a new device shortly after, this means, WA gets a new (or same) key from its servers during or after restoring the backup.

So that seems to be the only way to find out, if they're still there on their server. In order to do that, though, I would need to get WhatsApp to "see" and "detect" the old backup in the first place, which currently it doesn't.

Reading posts on here where a user even (successfully) switched to an entirely new number using an old backup from a different number, I believe there kind of must be a way to get those old backups running again, even if my phone number wasn't the same anymore?

Since I still have the same number though, this would probably make things a bit easier in the beginning: so changing the header/footer of an old backup with the information of the most recent one and then hoping that this will trigger WhatsApp to "see" the backup. If they still have a key for that backup, then restoring should work just as it always did for me, never having saved the actual local keys, right?

Or am I missing something?

[kingbtcvl]

This is probably just identical (in a way) to the problem I'm having.. And you would know how to fix it. Such a pity that it's so time-consuming to fix it.

Well, I am not sure if I could really fix your problem because without closely analyzing all tables in your databases I would not know how similar your problem actaully is to the one that I had altough you seem to face similar obstacles here.

You mentioned earlier that you might be able to provide your code or some information about it? Maybe that might help once I've finally found someone who thinks they can do it. They'll probably be able to make sense of your code and your information, so if you could share what you've learned about how to fix this god-damn problem, it would be much appreciated :D I doubt I will be able to do it myself - or maybe I will be, if I'm spending enough time to learn everything about it. I dont know. I haven't dealt with coding and all that before though.

Well, I can completely upload all my code since I don't mind it being public. So here you are: WhatsApp Database Merger
However, I am not sure how much this is going to help you, but as I said it might be quite useful when you find someone with some experience. Or if you are very lucky and your problem actually is very similar to mine you might even be able to modify it yourself with some efforts after learning some coding yourself. Who knows...

And that's the next question I'm having: that 64-letters-key - is it really truly the new "MASTER KEY", meaning it replaces the former key file?? Or is it (and that's what I read many times) only an ADDITIONAL protection of the regular / former key file? Meaning, just with the 64-letters-key alone, you can NEVER restore a database, as you would always need the regular key file as foundation??

Well crypt14 files use a key file for encryption whereas crypt15 files require the 64 bytes key. So in that sense the 64-letter-key kind of replaces the key file. I am writing "kind of" here because you can still deactivate end-to-end encryption in WA again and WA will start to create crypt14 backups again which will require the key file again. So if you have a crypt15 file you will need your 64 bytes key and if you have a crypt14 file you will need your key file, but you will never need both. This means your key file is useless for crypt15 files and your 64 bytes key is useless for crypt14 files as far as decryption is concerned.

Most stuff I read about it is that the 64-letters-key only serves as some sort of key-protected, "first entrance" to get to the database for which you would still need the regular key file to decrypt it.

Alough I only know very little details about how WA internally handles these thing I can tell you for sure that this is wrong. I decrypted all my crypt15 files without any key file and just the 64 byte character. That's the great thing here. You can simply switch to end-to-end encryption, note down the 64 characters, create a new backup and then decrypt it without the key file, which means no rooting (or jailbreaking) of your phone is necessary any longer.

If it wasn't like that, then there would not be any use for the regular / former key file once you enable E2EE-backup encryption, HOWEVER, even with that enabled, the regular key file is still stored in the WhatsApp directory, as I found out by extracting it.

Yes and that makes sense because otherwise there was no way to deactivate end-to-end encryption again.

A thought that was just coming to my mind how to detect if WA still saves old keys on their servers:
Would there be a way to change the information/header/footer of the old backups based on a most recent backup file, so that WhatsApp can be tricked into recognizing it and offering to restore?

That might work in principal I guess, but you would need some coding skills and can't rely on the program of this respository.
That's because you would need to modify the crypt14 file itself, but this programm here just allows to set a new header while re-encrypting. Nevertheless the code in the repsoitory can severe as basis because it contains all important parts to do something like that.

Reading posts on here where a user even (successfully) switched to an entirely new number using an old backup from a different number, I believe there kind of must be a way to get those old backups running again, even if my phone number wasn't the same anymore?

Since I still have the same number though, this would probably make things a bit easier in the beginning: so changing the header/footer of an old backup with the information of the most recent one and then hoping that this will trigger WhatsApp to "see" the backup. If they still have a key for that backup, then restoring should work just as it always did for me, never having saved the actual local keys, right?

Or am I missing something?

As I said my guess is yes, that's true. However I don't really know since I am not familiar with the internal details of WhatsApp. But it might be worth a try.

[Marvolo1590]

You are AWESOME, dude!!
You've helped me so much on here, even though it doesn't actually immediately solve my problem. But you've done so much to get me on track! Thanks so much!

Also you've answered one of my most important questions concerning the 64 digit key and backup restoration. I have now immediately activated the 64 digit key file so that I'll never have to deal with this stupid root extraction script again and whatelse not. It's been getting on my nerves, trying to extract the stupid keyfile and I dont want to root my new S24 ultra.

Thanks a ton, also, for providing your code / script. I'm in touch with some guys in a German forum and although they've written down some code queries from the top of their heads which could solve the problem, I still don't know what to do. So I'll be showing them your code and hopefully someone will understand :D

Finally, in my despair, I asked ChatGPT if it can give me a code which enables me to merge WhatsApp databases and it did. I just don't know if it is any good, but it sounded very convincing. lool :D

UPDATE
In the meantime, WonderShare logged into my computer this morning via TeamViewer and tried to fix their own problem. We restored the backup which is correct within their software and they would even be able to MERGE it with my existing database - so it's exactly what I've been trying to do all along. HOWEVER... after we restored the backup on my phone, the same thing happened: only my own messages were showing in some chats, not all of them.

They wanted my database so that they could analyze further, but I didn't want to give them my private chats. So I opened up my extracted databases and showed them what you had analysed: that the JIDs were different in the faulty restoration compared to the non-faulty one. They took the log file from the WonderShare software and I exported them my databases, the correct one and the faulty one, but "schema only". So there wont be any messages in them. No idea if that helps them, but oh well...

[Marvolo1590]

@kingbtcvl

Update:

So I finally found someone who claims to know about WA databases and SQL and who said he'd merged several databases himself before.
Unfortunately, all the scripts and codes he has written for me (he's using Termux on his phone to do all the scripts) keep working for him but for some reason not for me.

For example: I was trying to merge a few chats WITHIN THE SAME database together (it's not the main task that I've been outlining above with the messed up JIDs and all that, but a small side quest). A contact of mine has changed numbers 3 times which led to WA creating new chats for the same contact with every number.
I figured that I had to change the Chat_Row_Ids of the last 2 (old) numbers to be identical with the most recent number that this contact is presently using.

On Termux I navigated to the path where I placed the database on the phone and entered:

sqlite3 "/sdcard/msgstore.db" "update message set chat_row_id=38 where chat_row_id=39;"

So I wanted to replace all 39s with 38s and it did work. However, this only applied to the message-table and not to all the other tables containing chat_row_ids.

So that guy helping me wrote a script that would affect ALL tables containing chat-row-Ids, looking like this:

dir="/storage/emulated/0/msgstore.db"

sqlite3 $dir "delete from chat where _id=39;"
tbls=($(sqlite3 $dir ".tables"))
for tbl in ${tbls[@]}; do
columns=($(sqlite3 $dir "select name from pragma_table_info('$tbl');"))
for column in ${columns[@]}; do
if [[ "$column" == *"chat_row_id"* ]]; then
sqlite3 $dir "update $tbl set chat_row_id=38 where chat_row_id=39"
fi
done
done

Luckily, I could root an old phone of mine so I didn't have to go through the troubles of re-encrypting the databases. I simply put the modified database into ROOT data/data/whatsapp.com/Databases where whatsapp stores all the databases unencrypted.

However, after opening WhatsApp I got an error message claiming that there has been a problem with my database and it asked me whether I want to restore or skip. By tapping restore it will restore the stage as it was before I put the modified version. Probably it's using the encrypted crypt15 backup in the database folder to restore. If I tap on skip, it will reset the entire database and all the chats are gone - only the group chats are left, but empty.

And now the funny thing:
if I run this modified database through the HTML chat exporter (which exports all WA chats into an HTML file with all the media included) to get a visual representation of the chats, the chats are beautifully merged together just like I want them to be.
So apparently, it must be working for the HTML script.

But WhatsApp itself doesn't want to accept the modified database in the ROOT directory.

The guy who's helping me tested it for himself and merged his own chats and for him it's working, with the exact same script and by putting the modified db into the ROOT directory (he's rooted too).

He said I should try and make very minor changes to the original DB just to test if WA still comes up with an error. So I opened up the original db and made tiny changes to 1 text message in Text_Data. I only changed one letter or two. Then saved it with DB Browser and inserted it into the ROOT directory.

And yes.. even those tiny letter changes (with all the rest of the DB being original) was enough for WA to again show me the message and asking me to either skip or restore the chat history as there has been a problem.

So it seems as if even the tiniest changes to my original DB would not be accepted by WA. For him, it seems to work, though.

Would you know what the problem might be? He's almost on the same WA version like me. Did you encounter the same problems while modifying your database? Did we overlook something?

Thanks a lot for your thoughts and help!

[kingbtcvl]

On Termux I navigated to the path where I placed the database on the phone and entered:

sqlite3 "/sdcard/msgstore.db" "update message set chat_row_id=38 where chat_row_id=39;"

So I wanted to replace all 39s with 38s and it did work. However, this only applied to the message-table and not to all the other tables containing chat_row_ids.

😂 No wonder because the command nearly literally says
update (the table) message (by) set(ting) char_row_id=38 (on every row) where (you can find the value) chat_row_id=39,
but this is just a side note to help you enhance your understanding.

---

And yes.. even those tiny letter changes (with all the rest of the DB being original) was enough for WA to again show me the message and asking me to either skip or restore the chat history as there has been a problem.

So it seems as if even the tiniest changes to my original DB would not be accepted by WA. For him, it seems to work, though.

That sounds strange. I can't think of a good reason why it worked for the other guy, but not for you when you essentailly did the same. A reasonable explanation for that might be that "He's almost on the same WA version like me. " makes the difference here.
So my guess would be that WA has changed something internally again. I mean they came up with some new stuff lately. But that's just a pure guess, I haven't investigated that at all, of course.

Would you know what the problem might be?

Well, I might be that WA lately came up with kind of a protection mechanism to avoid the modification of the unencrypted database on a rooted phone by tracking whenever it changes. This would explain why your very tiny changes get you into trouble on WA, but are no problem for a HTML chat exporter. However that's no more than pure speculation on my side.

Did you encounter the same problems while modifying your database?

Well, since I did not try to avoid re-encrypting by rooting my phone, I could obviously not encounter the same problem. I just encountered the case that WA would fail to load my re-encrypted database when my header was still incorrect.

Did we overlook something?

Not really. I mean trying to modify the unencrypted database in data/data/whatsapp.com/Databases on a rooted phone hast not been tested here, neither by me nor anybody else. But if you get into trouble (for whatever reason) doing it that way, I would reccomend you to go for re-encrypting the modified database and trying to make WA load the re-encrypted database since this has been done successfully by a few people here.