feat: Added in-memory storage for testing purposes #59

Harshdev098 · 2025-10-12T08:40:04Z

Have added in_memory store for testing purpose.
We can edit config file to use specific store either postgresql or memory

ldk-reviews-bot · 2025-10-12T08:40:06Z

👋 Thanks for assigning @tnull as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

Harshdev098 · 2025-10-12T10:19:04Z

Hey @tnull @tankyleo Can you please review it

tnull

Thanks for looking into this!

Generally goes into the right direction, but we def. need to avoid re-allocating everything on every operation.

rust/Cargo.lock

rust/README.md

rust/impls/src/lib.rs

rust/impls/src/memory_store.rs

Harshdev098 · 2025-10-14T06:17:50Z

@tnull Have done the required changes

tnull

Looks much better, but I think we still need to handle global_version properly, even if we're currently not using it client-side.

rust/impls/src/in_memory_store.rs

ldk-reviews-bot · 2025-10-14T08:50:52Z

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2025-10-16T08:51:33Z

🔔 2nd Reminder

Hey @tnull @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2025-10-16T16:44:43Z

🔔 1st Reminder

Hey @tnull @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

tnull

One comment, will take another look once @tankyleo also had a chance to do a review round here.

rust/impls/src/in_memory_store.rs

Harshdev098 · 2025-10-17T09:15:56Z

@tankyleo Can you please review it!

ldk-reviews-bot · 2025-10-18T08:52:14Z

🔔 3rd Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2025-10-20T08:52:50Z

🔔 4th Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2025-10-22T08:53:27Z

🔔 5th Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

tankyleo

Sorry for the delay !

rust/impls/src/in_memory_store.rs

rust/impls/src/migrations.rs

Harshdev098 · 2025-10-26T01:53:09Z

@tankyleo Have done with the required changes! Can you please review it

tnull

When testing integration with LDK Node locally I found that the tests are currently failing. I now opened #62 to add LDK Node integration tests to our CI here. It would be great if that could land first, and we could also add a CI job for the in-memory store as part of this PR then, ensuring the implementation actually works as expected.

ldk-reviews-bot · 2025-10-28T01:53:31Z

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2025-10-30T01:54:18Z

🔔 2nd Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

tnull · 2025-10-31T12:09:51Z

@Harshdev098 Please rebase now that #62 landed to make use of the new CI checks here.

ldk-reviews-bot · 2025-11-05T01:55:32Z

🔔 5th Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

tnull

Right, and now you're hitting the errors that I originally found when running LDK Node against the in-memory store. Very likely, they are due to the in-memory implementation behaving in an unexpected manner. We need to fix this before this PR can get merged. Please take a look, but let me know if you end up getting stuck or need some assistance figuring this out.

ldk-reviews-bot · 2025-11-08T00:01:03Z

🔔 6th Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

Harshdev098 · 2025-11-08T09:36:32Z

Hey @tnull Have updated the code and the unit test are passing against the ldk node tests but didn't understand what is the cause of integration failures

Harshdev098 · 2025-11-09T03:53:12Z

Hey @tnull @tankyleo Have tested it locally, the test are working correctly now, don't know why its stuck in CI! Can you please review it

ldk-reviews-bot · 2025-11-10T00:01:47Z

🔔 7th Reminder

Hey @tnull @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2025-11-11T03:54:09Z

🔔 1st Reminder

Hey @tnull @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2025-11-12T00:02:20Z

🔔 8th Reminder

Hey @tnull @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

tankyleo

Sorry for the delay again I have cleared all the other priorities, this is now top priority :) Here are some comments, mostly on the types we pass to the different functions.

Will continue review tomorrow 100% ! Thank you again.

tankyleo · 2025-11-12T02:35:23Z

rust/impls/src/postgres_store.rs

-			ErrorKind::Other,
-			format!("Failed to drop database {}: {}", db_name, e),
-		)
+		Error::new(ErrorKind::Other, format!("Failed to drop database {}: {}", db_name, e))


Sorry the formats in this file were done in a CI-specific PR we merged, go ahead and rebase and drop them thank you :)

@Harshdev098 sorry i take it back, don't rebase to a new commit just yet :) if you can just drop these changes, keep the same base commit on main, will rebase to a newer base commit as necessary thank you

Ohh sorry @tankyleo , but I have done it already, my branch is now up to date with main

@Harshdev098 want to make sure you know about git reflog ? Would have allowed you to revert it back easily before pushing but all good now !

rust/impls/src/in_memory_store.rs

tankyleo · 2025-11-12T02:43:19Z

rust/impls/src/in_memory_store.rs

+		let vss_delete_records: Vec<VssDbRecord> = request
+			.delete_items
+			.into_iter()
+			.map(|kv| build_vss_record(user_token.clone(), store_id.clone(), kv))
+			.collect();


We also do not need to build VssDbRecords for the delete_items, so let's delete those.

rust/impls/src/in_memory_store.rs

Harshdev098 · 2025-11-12T04:50:07Z

rust/impls/src/postgres_store.rs

-			ErrorKind::Other,
-			format!("Failed to drop database {}: {}", db_name, e),
-		)
+		Error::new(ErrorKind::Other, format!("Failed to drop database {}: {}", db_name, e))


Ohh sorry @tankyleo , but I have done it already, my branch is now up to date with main

Harshdev098 · 2025-11-12T05:15:57Z

@tankyleo Have updated the code

tankyleo

Some more comments focusing on in_memory_store thank you

tankyleo · 2025-11-12T19:14:00Z

rust/impls/src/in_memory_store.rs

+
+/// In-memory implementation of the VSS Store.
+pub struct InMemoryBackendImpl {
+	store: Arc<RwLock<HashMap<String, VssDbRecord>>>,


Can you share why you picked an RwLock here ? I always default to Mutex unless it will be read-heavy / have good reasons to use RwLock.

VSS is rather even-handed / write-heavy, so my gut instinct is to go for Mutex.

Actually have used Mutex earlier but have changed to RwLock for debugging the integration test that was failing!

sounds good ok would you consider switching back to Mutex ? or are there some bugs that come up specifically when using Mutex ?

Switching back to Mutex, it was deadlock issue at that time!

Also I found that testing the ldk node integration test against memory_store I need to restart the server to test after every updation, because I think starting the integration test it needs fresh data but it gives error when runned multiple times without restarting the server, which can make bad experience as a developer @tankyleo!?

@Harshdev098 does this have to do with the choice between Mutex and RwLock ? I'll look into this

Probably not, you can reproduce it by just running the ldk node integration test multiple times with the same server without restarting it. I think it could be a problem with test because every time we run the test and its failing while setting up the first node and it's giving on Error value
ReadFailed

have you had success with passing the ldk integration test against in_memory just once ? I've passed against postgres all good, but in_memory_store seems to hang.

tankyleo · 2025-11-12T19:14:37Z

rust/impls/src/in_memory_store.rs

+
+/// In-memory implementation of the VSS Store.
+pub struct InMemoryBackendImpl {
+	store: Arc<RwLock<HashMap<String, VssDbRecord>>>,


How about we use a BTreeMap here to avoid having to resort all the keys on every subsequent call to list_key_versions further down below ?

tankyleo · 2025-11-12T19:15:07Z

rust/impls/src/in_memory_store.rs

+}
+
+impl InMemoryBackendImpl {
+	/// Creates an in-memory instance.


Let's not add comments that are obvious. I prefer we use them sparingly, only for non-obvious code.

But its giving this missing documentation for an associated function when not added the comment, should I allow missing documentation for this #[allow(missing_docs)] ?

ah thanks yes keep those docs then my bad !

rust/impls/src/in_memory_store.rs

tankyleo · 2025-11-12T19:35:19Z

rust/server/src/main.rs

-	if args.len() != 2 {
-		eprintln!("Usage: {} <config-file-path>", args[0]);
+	if args.len() < 2 {
+		eprintln!("Usage: {} <config-file-path> [--in-memory]", args[0]);


Was this specific flag requested by a user ? I'd be in favor of deleting it, and keeping the in_memory configuration to a single spot, as we currently have it in the configuration file.

But let me know what you think, curious to hear your thoughts.

I added --in-memory to make it easy for external projects like Fedimint to run integration tests with our VSS server

sounds good ok and the configuration file would be too much of a hassle for them ?

Yah, exactly

or we could change the "default" database to "in_memory" store so people can clone the repository, and run CI directly right ?

Also another question: could Fedimint just run against postgres ? We've done some work in the past to make setting that up much easier, as close to "just run it". Thank you for the context. You can see our own build-and-deploy-rust.yml in this repo runs test against a postgres service without too much trouble.

yah they can run it in postgres but the in-memory simplifies their devimint CI like no DB service needed, just spawn the binary,
And the in-memory flag, it's low-effort and keeps config as the single source while enabling overrides. I don't think to make the in_memory the default one!

ok sound sgood let's keep things as you've got them now thank you

rust/server/src/main.rs

rust/server/vss-server-config.toml

ldk-reviews-bot · 2025-11-13T03:55:15Z

🔔 2nd Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot requested a review from tankyleo October 12, 2025 08:50

tnull self-requested a review October 13, 2025 07:10

tnull reviewed Oct 13, 2025

View reviewed changes

Harshdev098 force-pushed the memory_store branch 2 times, most recently from 4980a75 to 25d57e3 Compare October 14, 2025 06:16

Harshdev098 requested a review from tnull October 14, 2025 06:17

tnull reviewed Oct 14, 2025

View reviewed changes

rust/impls/src/in_memory_store.rs Show resolved Hide resolved

Harshdev098 force-pushed the memory_store branch from 25d57e3 to 9012e95 Compare October 14, 2025 12:24

Harshdev098 requested a review from tnull October 14, 2025 16:44

tnull reviewed Oct 17, 2025

View reviewed changes

rust/impls/src/in_memory_store.rs Outdated Show resolved Hide resolved

Harshdev098 force-pushed the memory_store branch from 9012e95 to 3b434d0 Compare October 17, 2025 09:14

tankyleo reviewed Oct 23, 2025

View reviewed changes

Harshdev098 force-pushed the memory_store branch from 3b434d0 to e0c31bb Compare October 26, 2025 01:51

Harshdev098 requested a review from tankyleo October 26, 2025 01:53

Harshdev098 force-pushed the memory_store branch from 8003119 to 5898609 Compare October 26, 2025 02:18

tnull reviewed Oct 27, 2025

View reviewed changes

Harshdev098 force-pushed the memory_store branch from 81ca808 to 1f18bd8 Compare November 4, 2025 11:41

Harshdev098 force-pushed the memory_store branch 2 times, most recently from 84cfed6 to 939559f Compare November 5, 2025 03:34

tnull reviewed Nov 5, 2025

View reviewed changes

Harshdev098 force-pushed the memory_store branch 4 times, most recently from 51a791d to 0168750 Compare November 8, 2025 05:48

Harshdev098 force-pushed the memory_store branch 3 times, most recently from 92bd1e3 to bd3eca4 Compare November 9, 2025 03:41

Harshdev098 requested a review from tnull November 9, 2025 03:53

Harshdev098 force-pushed the memory_store branch 2 times, most recently from 9ec0e79 to 45fecd1 Compare November 9, 2025 10:18

tankyleo reviewed Nov 12, 2025

View reviewed changes

Harshdev098 added 3 commits November 12, 2025 09:26

chore: formatted and pub(crate) visibility

0bf37eb

feat: Added in-memory storage for testing purposes

02a1b4e

feat: Added CI for in_memory_testing

c2a140e

Harshdev098 commented Nov 12, 2025

View reviewed changes

Harshdev098 force-pushed the memory_store branch from 45fecd1 to c2a140e Compare November 12, 2025 05:10

tankyleo reviewed Nov 12, 2025

View reviewed changes

feat: Added in-memory storage for testing purposes #59

Are you sure you want to change the base?

feat: Added in-memory storage for testing purposes #59

Conversation

Harshdev098 commented Oct 12, 2025

Uh oh!

ldk-reviews-bot commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Harshdev098 commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tnull left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Harshdev098 commented Oct 14, 2025

Uh oh!

tnull left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ldk-reviews-bot commented Oct 14, 2025

Uh oh!

ldk-reviews-bot commented Oct 16, 2025

Uh oh!

ldk-reviews-bot commented Oct 16, 2025

Uh oh!

tnull left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Harshdev098 commented Oct 17, 2025

Uh oh!

ldk-reviews-bot commented Oct 18, 2025

Uh oh!

ldk-reviews-bot commented Oct 20, 2025

Uh oh!

ldk-reviews-bot commented Oct 22, 2025

Uh oh!

tankyleo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Harshdev098 commented Oct 26, 2025

Uh oh!

tnull left a comment

Choose a reason for hiding this comment

Uh oh!

ldk-reviews-bot commented Oct 28, 2025

Uh oh!

ldk-reviews-bot commented Oct 30, 2025

Uh oh!

tnull commented Oct 31, 2025

Uh oh!

ldk-reviews-bot commented Nov 5, 2025

Uh oh!

tnull left a comment

Choose a reason for hiding this comment

Uh oh!

ldk-reviews-bot commented Nov 8, 2025

Uh oh!

Harshdev098 commented Nov 8, 2025

ldk-reviews-bot commented Oct 12, 2025 •

edited

Loading

Harshdev098 commented Oct 12, 2025 •

edited

Loading