feat(zero-client): expo-sqlite storage - #4669
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
|
Bencher Report| Branch | 0xcadams/expo |
| Testbed | Linux |
Click to view all benchmark results
| Benchmark | Throughput | Benchmark Result operations / second (ops/s) x 1e3 (Result Δ%) | Lower Boundary operations / second (ops/s) x 1e3 (Limit %) |
|---|---|---|---|
| src/client/custom.bench.ts > big schema | 📈 view plot 🚷 view threshold | 386.77 ops/s x 1e3(-0.78%)Baseline: 389.83 ops/s x 1e3 | 357.67 ops/s x 1e3 (92.48%) |
| src/client/zero.bench.ts > basics > All 1000 rows x 10 columns (numbers) | 📈 view plot 🚷 view threshold | 1.63 ops/s x 1e3(-0.71%)Baseline: 1.64 ops/s x 1e3 | 1.60 ops/s x 1e3 (98.50%) |
| src/client/zero.bench.ts > pk compare > pk = N | 📈 view plot 🚷 view threshold | 30.21 ops/s x 1e3(+0.58%)Baseline: 30.04 ops/s x 1e3 | 28.88 ops/s x 1e3 (95.60%) |
| src/client/zero.bench.ts > with filter > Lower rows 500 x 10 columns (numbers) | 📈 view plot 🚷 view threshold | 2.55 ops/s x 1e3(-0.93%)Baseline: 2.58 ops/s x 1e3 | 2.51 ops/s x 1e3 (98.24%) |
|
| Branch | 0xcadams/expo |
| Testbed | Linux |
Click to view all benchmark results
| Benchmark | File Size | Benchmark Result kilobytes (KB) (Result Δ%) | Upper Boundary kilobytes (KB) (Limit %) |
|---|---|---|---|
| zero-package.tgz | 📈 view plot 🚷 view threshold | 1,251.66 KB(+0.53%)Baseline: 1,245.05 KB | 1,269.95 KB (98.56%) |
| zero.js | 📈 view plot 🚷 view threshold | 202.56 KB(0.00%)Baseline: 202.56 KB | 206.62 KB (98.04%) |
| zero.js.br | 📈 view plot 🚷 view threshold | 56.77 KB(0.00%)Baseline: 56.77 KB | 57.91 KB (98.04%) |
|
Exciting! I'm on vacation until beginning of August so I will not be able to provide any good feedback on this until then. |
tantaman
left a comment
There was a problem hiding this comment.
Thanks for figuring this out
| await withWrite(walStore, async wt => { | ||
| await wt.put('foo1', 'bar1'); |
There was a problem hiding this comment.
what is the bench attempting to measure? Just raw transactions per second? Or do you want more information on writes per second?
Writes per second should be roughly on the order of transactions_per_second * writes_per_transaction so 1 write per transaction will give you the same writes per second as transactions per second.
There was a problem hiding this comment.
Yeah this is a good question. I was using this benchmark to improve performance of the SQLite store w/ better-sqlite3, as a way to test performance of the store as I was making changes to schema/PRAGMAs. I updated this to be simpler and compare WAL modes. Let me know if this doesn't fit into existing benchmarks or any ways you see improving this.
|
|
||
| return Promise.resolve(write); | ||
| } catch (e) { | ||
| return Promise.reject(e); |
There was a problem hiding this comment.
same question as with read.
It does feel weird to be doing locking given the interface we expose to SQLite is technically synchronous.
|
I benchmarked performance (with TLDR; the read pooling is roughly 8-14x faster. The last test, bench(
`plain read`,
async () => {
const readP1 = withRead(store, async rt => {
expect(await rt.get('foo')).equal('bar');
});
const readP2 = withRead(store, async rt => {
expect(await rt.get('foo')).equal('bar');
});
const readP3 = withRead(store, async rt => {
expect(await rt.get('foo')).equal('bar');
});
const readP4 = withRead(store, async rt => {
expect(await rt.get('foo')).equal('bar');
});
const readP5 = withRead(store, async rt => {
expect(await rt.get('foo')).equal('bar');
});
await Promise.all([readP1, readP2, readP3, readP4, readP5]);
},
{
throws: true,
setup: async () => {
await withWrite(store, async wt => {
await wt.put('foo', 'bar');
});
},
},
);With a read connection pool of 2: 
With a new connection on every read: 
|
arv
left a comment
There was a problem hiding this comment.
LGTM
I know Matt is reviewing this already but I was very curious.
I looked at the locking structure and it seems to have the same semantics as the IDBStore.
| } | ||
|
|
||
| put(key: string, value: ReadonlyJSONValue): Promise<void> { | ||
| this._preparedStatements.put.run([key, JSON.stringify(value)]); |
There was a problem hiding this comment.
is there a reason why the type of run takes rest args but the callers always wrap in a single array? Consider removing one array allocation.
There was a problem hiding this comment.
Good catch - moved to rest args
| new SQLiteDatabaseManager({ | ||
| open: name => { | ||
| const filename = path.resolve(__dirname, `${name}.db`); | ||
| // this cannot be :memory: because multiple read connections must access |
There was a problem hiding this comment.
it's technically doable via memory with some fancy options. But yeah, better to just keep it file based.
|
OK on more thought I understand the reason for the design here. I was wondering why we cannot rely on RWLock within the context and thereby share a single connection across the context. But the problem is that RWLock wants to allow read tx to overlap. And we cannot represent overlapped, separate read tx with a single connection (at least not cleanly?). LGTM too. |
|
Unfortunately the tests keep failing and I don't have permissions to override the PR requirements |
Adds a new
@rocicorp/zero/expopackage to add a SQLiteStoreProviderfor Expo/React Native.Builds on @austinm911's work.
SQLiteDatabaseManagerwhich handles schema setup, PRAGMAs, and connection handling.SQLiteStorewhich works with bare SQLite and uses prepared statements withRWLock.Note:
iOS and Android both had issues with WAL journal mode - they hang on COMMIT indefinitely. According to Expo, this is recommended, but it doesn't seem to work, so it is disabled by default, with a performance hit. Here is the performance difference with
better-sqlite3:Expo Demo
expo-sqlite.mp4