I've a typical parent ID based file system in database and files are physically stored on disks. What I want is to store files in blocks (4096K) of fixed size.

Chunking can be used I found, but how does a fixed size block storage file system may look like at high level in database?

Suppose I also want to update the specific blocks as per new changes inside files which is located remotely. How do I sync both the files? I found Rabin fingerprint and fixed blocks revision can be used. But, how would I implement the revision inside database that adjusts with my existing file system in database?

| cite | improve this question | | | | |
  • $\begingroup$ Why do you want to store the files in that way? BTW, a 4096K block is very large -- do you maybe mean 4096 bytes? $\endgroup$ – D.W. Mar 22 '19 at 4:24
  • $\begingroup$ en.wikipedia.org/wiki/Rsync $\endgroup$ – D.W. Mar 22 '19 at 4:25
  • $\begingroup$ @D.W. I meant 4 Megbytes. I've explored rsync, I guess it provides new changes (I can figure what blocks are changed by comparing remote block hashes). But how do I store new / edited blocks (block pointers actually) in database, how'd the schema design look like? As I don't want to replicate existing block pointer again. There will be new file record as a version. Thanks... $\endgroup$ – Abhigyan Tiwari Mar 22 '19 at 4:29
  • $\begingroup$ I would have an agent responsible for monitoring the FileSystem event stream in order to detect candidate changes. It would be responsible for constructing updated thumbprints/detecting real changes and on forwarding to the database agent. As this agent is a custom DB, I would architect a Bish-Tree storing thumbprint and block-id. A second tree storing file-id, version, offset, block-id. A third tree containing file details and version information. As for syncing, send the block itself instead of the thumb print from the FileSystem event stream observing agent. $\endgroup$ – Kain0_0 Mar 22 '19 at 6:27
  • $\begingroup$ @Kain0_0 Thank you for making it clearer. I see the event stream monitoring is done on the client side. Changes are streamed to custom DB agent as you said and blocks being updated there. For the later part that is using multiple trees, I get the idea, but can you provide this as an answer showing them in a diagram or structured form and how they are linked together which I don't get. I also want them normalized so want an expert's hand in that to learn from. Thanks... $\endgroup$ – Abhigyan Tiwari Mar 22 '19 at 6:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.