Fireproof on Hacker News

03 Dec 2024 by Meghan Sinnott

Fireproof's founder, Chris Anderson, posted about Fireproof on Hacker News and we have had a spike in users, so we wanted to formally welcome everyone and share a comprehensive post on the capabilities, architecture, and how to effectively integrate Fireproof into applications.

About Fireproof

Fireproof is a lightweight embedded database that provides local-first live ledgers using object data storage and Git-like encrypted synchronization. It offers cryptographic causal consistency across browsers, edge functions, and any environment where TypeScript or JavaScript runs. The database is open source and available under the Apache/MIT licenses.

Why People Care

Fireproof simplifies app development by providing an easy-to-use API, making it accessible for coders of any level to build enterprise-critical applications.

What Makes Fireproof Unique?

Below are some core features of Fireproof, an outline of the developer experience, an introduction to the technical architecture, a few use cases and examples of Fireproof in the wild, a list of resources to get started, and a transparent acknowledgement of our current limitations along with future goals and a call to action.

Core Features

Data Model and Storage

• Uses a Git-like data model with content-addressed storage, enhancing data integrity and supporting deterministic merges.
• Implements Merkle Conflict-Free Replicated Data Types (CRDTs) to provide multi-writer safety, tamperproof data provenance, conflict tracking, and deterministic merging of data changes.
• Stores data as content-addressed encrypted blobs locally using IndexedDB in browsers or the filesystem in Node.js environments.
• Supports offline-first functionality, allowing applications to operate without a network connection and sync changes when reconnected.
• Employs ledgers as the unit of data sharing, access control, and maintaining a provable history of changes.

Synchronization and Collaboration

• Offers live synchronization of data across clients, with automatic UI updates through reactive APIs.
• Synchronization is achieved via encrypted sync, ensuring that data remains secure during transmission.
• Supports deterministic, idempotent, associative, and commutative merges, ensuring consistent state across all clients regardless of the order of updates.
• Uses eager decryption during sync operations, currently requiring data to fit in memory but with plans to support larger datasets.
• Provides real-time collaboration features, making it suitable for building collaborative applications like chat systems or shared documents.

Security and Access Control

• Data is stored and transmitted in an encrypted form, enhancing security and privacy.
• Implements end-to-end encryption, with plans to support key rotation and mature key management.
• Uses UCAN (User Controlled Authorization Networks) capability delegation for authorization, allowing for embedded authorization without relying on centralized resources.
• Access control changes are treated as data manipulations, enabling offline authorization changes and secure delegation.
• The system uses non-extractable key pairs generated by client devices for secure authentication and authorization.
• Fireproof leverages modern browser APIs like Passkeys and the Origin Private Filesystem to support embedded databases.

APIs and Developer Experience

Simple and Intuitive APIs

• Provides a minimalistic API for data operations, such as db.put({ hello: "world" }) for adding data and connect(db, remote) for syncing.
• Offers reactive APIs like db.subscribe(), enabling automatic UI updates in response to data changes.
• Includes query APIs with support for pagination and limits, helping manage large datasets and preventing data avalanches.
• Designed to make front-end development easier by handling live data sync and collaboration features internally.
• The database can function as an RPC bus, a pattern inherited from CouchDB, facilitating backend processes to subscribe to ledger updates.

Platform and Environment Support

• Runs anywhere JavaScript runs, including browsers, Node.js, edge functions, and mobile apps.
• Compatible with React, providing hooks and utilities like useFireproof, useLiveQuery, and useDocument for seamless integration.
• Can be embedded in iframed applications and is being adapted for React Native and Socket Supply for mobile app support.
• Designed with the philosophy of putting "the smarts at the edge," emphasizing client-side processing over heavy backend reliance.
• Supports edge functions by configuring the database to read and write directly with the cloud store.

Integration and Extensibility

• Supports multiple backend implementations, including S3, Cloudflare R2, AWS SAM templates, Netlify, and custom backends.
• Synchronization endpoints can use various protocols like polling and WebSockets.
• The sync endpoint uses minimal blob key/value and register APIs, which can be built on top of any raw key-value store with check-and-set semantics.
• Plans to offer a REST API in Fireproof Cloud for raw data queries without requiring local replication.
• Fireproof Cloud uses Cloudflare's R2 and Durable Objects for its backend infrastructure.
• Working on transparent replication to PostgreSQL JSON, facilitating easier adoption in existing infrastructures.

Technical Architecture

Data Synchronization and Conflict Resolution

• Each ledger replicates entirely during sync operations, with no sharding within a ledger.
• Synchronization can be configured via a centralized endpoint or potentially peer-to-peer, although P2P setups may require additional development.
• The merge algorithm ensures that once all updates are applied, all clients reach the same state, regardless of update order.
• The system handles multi-writer concurrency safely, making it suitable for collaborative applications.

Storage and Encryption

• Uses content-addressed storage, where data is stored based on its content hash, similar to how Git operates.
• The storage engine writes encrypted blobs, and the backend does not need to parse or understand the data, enhancing security.
• Employs Merkle trees in its CRDT implementation to efficiently manage and synchronize data.
• Encourages a local-first approach to data handling, improving performance and reliability in web applications.

Performance and Optimization

• Underlying data structures utilize asynchronous iterators, allowing for future support of streaming data and handling datasets larger than memory.
• Currently uses eager decryption, which may impact performance with very large datasets, but optimizations are planned.
• Supports limit parameters in queries to help manage and paginate large datasets.

Use Cases and Examples

• Suitable for applications requiring data integrity, privacy, and security, such as financial applications.
• Already used in projects like AI character chatbots, personal finance applications, and by hedge funds.
• Can act as a Retrieval Augmented Generation (RAG) system for Large Language Models (LLMs) by storing and searching across vector embeddings.
• Provides an easy way to add live collaboration features to legacy dashboards or build new interactive tools.

Getting Started and Resources

Installation and Setup

• Available via npm: npm install @fireproof/core.
• Initial setup is minimal, allowing developers to start using the database with just a few lines of code.
• We provide code samples and tutorials, including a React tutorial and examples on CodePen.

Community and Support

• Maintained by a team including notable contributors like Damien Katz, Meno Abels, Mikeal Rogers, and Alan Shaw.
• We encourage community contributions and collaboration.
• We have support channels including GitHub issues and a Discord server for community engagement.

Limitations and Future Work

Current Limitations

• Filtering data by specific criteria (e.g., messages since time T) is currently limited due to end-to-end encryption constraints.
• Revocation in access control is an area identified for further development.
• Subset synchronization is not yet implemented but is recognized as a necessary feature.
• Handling data larger than memory is currently limited but planned to be addressed with optimizations in decryption and data handling.

Planned Features

• Implementing read/write access control on a per-ledger basis.
• Enhancing key management, including support for key rotation.
• Developing a PostgreSQL backend for more flexible and efficient data storage options.
• Exploring subset synchronization using research like G-Trees to optimize data sync operations.

Want to roll up your sleeves and pitch in to Fireproof? Our project is open source and growing quickly. Find us on GitHub or log on to our Discord to chat with others eager to develop a realtime database that runs anywhere.

Tagged: database