Roman Zeyde
|
2852020c58
|
Fix leftovers at examples/load.rs
|
2018-07-13 22:24:14 +03:00 |
|
Roman Zeyde
|
daee3fbe4b
|
Replace "Vec::new()" by shorter "vec![]"
|
2018-07-13 21:30:30 +03:00 |
|
Roman Zeyde
|
1a9b977dad
|
Update TODOs
- implemented Daemon::gettransactions() API
- implemented multithreaded indexing workers
- SSTable injection requires non-overlapping ranges (= pre-sorting = full compaction)
- HTTP parsing can be improved
|
2018-07-13 21:12:26 +03:00 |
|
Roman Zeyde
|
48cd929a99
|
Remove StoreOptions and enable manual compactions explicitly
|
2018-07-13 14:23:10 +03:00 |
|
Roman Zeyde
|
9a89e6a2d8
|
Update README with latest performance metrics
|
2018-07-12 23:26:37 +03:00 |
|
Roman Zeyde
|
03790d9977
|
Add logging for bulk indexing
|
2018-07-12 21:30:33 +03:00 |
|
Roman Zeyde
|
4d322d4819
|
Substract signed integers
|
2018-07-12 19:38:39 +03:00 |
|
Roman Zeyde
|
1e258f4b3d
|
Allow vectorized fetch for mempool transactions
|
2018-07-12 19:38:16 +03:00 |
|
Roman Zeyde
|
cdacf451a9
|
Log delta of mempool transactions
|
2018-07-12 18:58:38 +03:00 |
|
Roman Zeyde
|
e09ac477bb
|
Skip bulk indexing if it's already over
|
2018-07-12 16:52:25 +03:00 |
|
Roman Zeyde
|
b9a4cb4318
|
Move bulk reader and indexer to separate module
|
2018-07-12 16:47:39 +03:00 |
|
Roman Zeyde
|
4c7413db9d
|
Increase open files limit explicitly
Otherwise, full compaction may fail.
|
2018-07-12 14:44:10 +03:00 |
|
Roman Zeyde
|
c7b8f57228
|
Keep default parallism settings
|
2018-07-12 12:47:05 +03:00 |
|
Roman Zeyde
|
125bfe0e15
|
Log latest indexed block
|
2018-07-12 10:47:25 +03:00 |
|
Roman Zeyde
|
62a78f18ad
|
Put "finish" marker after compaction is over
|
2018-07-12 10:46:33 +03:00 |
|
Roman Zeyde
|
f92dfc808d
|
Log indexed blk*.dat file path
|
2018-07-12 09:25:17 +03:00 |
|
Roman Zeyde
|
7b5679db70
|
Panic if DB exists before running bulk load benchmark
|
2018-07-11 21:26:12 +03:00 |
|
Roman Zeyde
|
b32e503db2
|
Rename regular indexing benchmark
|
2018-07-11 21:26:12 +03:00 |
|
Roman Zeyde
|
dd70a1cc8a
|
Persist last indexed block marker
|
2018-07-11 21:26:12 +03:00 |
|
Roman Zeyde
|
6609d6e008
|
Store indexed blocks' hashes
|
2018-07-11 21:01:54 +03:00 |
|
Roman Zeyde
|
2158fa1961
|
Sort blk*.dat files
|
2018-07-11 21:00:57 +03:00 |
|
Roman Zeyde
|
10919035bd
|
Rename bulk load benchmark
|
2018-07-11 20:44:05 +03:00 |
|
Roman Zeyde
|
4da521b3cd
|
Refactor bulk load benchmark
- bulk writes take ~1h (reading blk*.dat files at ~48MB/s)
- full compaction takes ~45m (writing 37GB of SSTables)
|
2018-07-11 20:42:57 +03:00 |
|
Roman Zeyde
|
1f979baf8f
|
Increase compaction readahead to 2MB
This should improve compaction throughput on HDDs.
|
2018-07-11 20:33:09 +03:00 |
|
Roman Zeyde
|
219a962145
|
Use sort_unstable_by() instead of implementing Ord for Row
|
2018-07-11 20:31:44 +03:00 |
|
Roman Zeyde
|
ef7a7eac8a
|
Sort rows before writing to DB
Can happen in parallel, before sending the rows to the DB writer thread.
|
2018-07-11 16:26:06 +03:00 |
|
Roman Zeyde
|
49ec4d1c38
|
Support SSTable ingestion into DB
|
2018-07-11 13:48:57 +03:00 |
|
Roman Zeyde
|
e69686f988
|
Use multiple bulk index workers
|
2018-07-10 23:52:26 +03:00 |
|
Roman Zeyde
|
4b83c6536f
|
Add SSTableWriter to store module
|
2018-07-10 16:09:31 +03:00 |
|
Roman Zeyde
|
fb372b2915
|
Split reading from indexing blk*.dat files
Also, use a single Vec<Row> for write operation
|
2018-07-10 15:37:56 +03:00 |
|
Roman Zeyde
|
56229409a0
|
Simplify SIGINT polling
|
2018-07-10 15:29:56 +03:00 |
|
Roman Zeyde
|
314978f30a
|
Simplify bulk indexing flow
Use separate thread per blk*.dat file.
|
2018-07-10 15:28:04 +03:00 |
|
Roman Zeyde
|
45f3c7d178
|
Make store::Row sortable (by key) and clonable
|
2018-07-09 19:07:33 +03:00 |
|
Roman Zeyde
|
baf2d8624d
|
Rename bulk_load() -> bulk_index()
|
2018-07-09 18:02:49 +03:00 |
|
Roman Zeyde
|
e88f075c50
|
Use 1MB block for RocksDB
|
2018-07-09 17:10:49 +03:00 |
|
Roman Zeyde
|
4c182a0650
|
Fix formatting
|
2018-07-09 17:10:24 +03:00 |
|
Roman Zeyde
|
f4ac913c39
|
Pass blk*.dat file path during bulk indexing process
|
2018-07-09 16:53:40 +03:00 |
|
Roman Zeyde
|
89551f3e5d
|
Don't keep too much LOG.old.* files
|
2018-07-09 12:59:10 +03:00 |
|
Roman Zeyde
|
be20b5bd05
|
Update store to use pingcap RocksDB API
|
2018-07-08 23:24:21 +03:00 |
|
Roman Zeyde
|
586d6132d6
|
Use 'pingcap' RocksDB fork
|
2018-07-08 23:24:21 +03:00 |
|
Roman Zeyde
|
da570a9182
|
Fix benchmarks
|
2018-07-08 23:23:36 +03:00 |
|
Roman Zeyde
|
c82ceb6ad7
|
Update usage documentation for JSONRPC cookie
|
2018-07-08 22:22:32 +03:00 |
|
Roman Zeyde
|
57c67163dd
|
Fail connection in case of HTTP error
|
2018-07-08 22:17:35 +03:00 |
|
Roman Zeyde
|
1762523e1a
|
Allow setting JSONRPC cookie via command-line
|
2018-07-08 22:17:35 +03:00 |
|
Roman Zeyde
|
d1e03f33a9
|
Add Gitter badge
|
2018-07-08 14:11:48 +03:00 |
|
Roman Zeyde
|
a797a3864e
|
Move usage instructions to doc/ subdirectory
|
2018-07-07 12:20:04 +03:00 |
|
Roman Zeyde
|
6035bd164b
|
Read cookie as a trimmed String
|
2018-07-07 11:47:19 +03:00 |
|
Roman Zeyde
|
1475fc949d
|
Refactor bitcoind cookie handling
|
2018-07-07 11:25:25 +03:00 |
|
Roman Zeyde
|
93f1b85cda
|
Use ms precision for logging
|
2018-07-07 11:02:29 +03:00 |
|
Roman Zeyde
|
53c0c02732
|
Use explicit integer types for BlockchainInfo
|
2018-07-07 10:41:24 +03:00 |
|