Distributed data storage

If you have many nodes and want to store data - how do you store data?

YAML Interest

This category is to explore the various approaches to distributed data storage. One popular technique is the Blockchain - used by bitcoin, Ethereum and the dfinity internet computer.

There are other techniques available to us too! I think it is unfair on Mindey has to host 0oo.li by himself and take the cost of maintaining it by himself. We should be able to distribute arbitrary data across machines.






Vote (Optional) (suppress notifications) (Optional)
Please, log in.

There's ipfs. Anyone can host ipfs node. Don't know how much work to refactor oo to use ipfs


Yeah, I'm thinking of this problem, and I think the right path would be --

  1. Researching the database models as an optimal ontology for what is described here. (If the ontology optimizes cooperative decision-making and economic activity, then it makes sense to look for ways to scale it and make nodes decentralize it.)

  2. Decentralize it.

The "how" of decentralization is, as I see -- searching for dApps that replace functionality, and providing a common UI for all those dApps, that would collectively be equivalent to the monolith application that the Infinity is.

Another way would be to build a new decentralized system from ground up, using things like gun.js, where each of users would be seeders at the same time.

// skihappy: There's ipfs.

This falls under the search for "dApps" that replace functionality. For example, IPFS is such a dApp for file storage. However, infinity as application requires the relational database as a dApp :) is there anything like that? (including schema migrations)


Relational links can be a layer above data storage, forming a graph. Graphs are much better to represent relational database then relational database. Relational links can be part of data set. That's how graph db works. Perhaps, there's already an ipfs graph solution like that already


Mindey the simplest thing that people can do to help is to backup the database with the dump URL which I have a script that runs daily to take a backup of the data. My main computer has been switched off unfortunately so I haven't been running it for the next 4 months.

I like Graph databases like Neo4j and Dgraph. I have even written a simple one that uses matrix multiplication to do Breadth first search like Graphblas.


Hidden in this category is the idea that the easiest way to get P2P behaviour is to make the data storage layer P2P and everything else is traditional.



    : Mindey
    :  -- 
    :  -- 
    

chronological,

// make the data storage layer P2P and everything else is traditional

It makes perfect sense -- most of software systems are clients of database systems.



    : chronological
    :  -- 
    :  -- 
    

Mindey,

I just found the project hyper Hyper space it's an interesting project. Rather than files you have spaces and they are backed by a merkel directed acyclic graph that somehow merges seamlessly. Assuming it uses CRDTs.

They say you can build almost any app with it, forum, chat, website etc



    :  -- 
    : Mindey
    :  -- 
    

chronological,

The problem with Hyperhyperspace is that I would never use it because it takes over the objects you use to store data to implement its functionality. So it takes a bad approach.

To create simple object graphs in memory you have to use a class called a MutableReference to link objects together. Feels wrong to me.



    :  -- 
    : Mindey
    :  -- 
    

chronological,

When the Hyper Hyper Space data layer synchronizes a space contents with other peers, it needs to validate that the received changes respect the data assumptions of the application. And yeah, anything that's mutable needs to be wrapped in a type that knows how to marshal the changes as CRDT-like operations.

The project is experimental, maybe we need more iterations before we figure out how to remove that coupling. The model is already a considerable simplification over previous ones :)


sbazerque,

One small clarification about Hyper Hyper Space: you don't need to use MutableReference to link objects together, you need it only if you want to be able to replace the object you linked with another one later!



    :  -- 
    : Mindey
    :  -- 
    

sbazerque,