Revised article

Introduction to Bitcoin for developers

Sonny Alves Dias17 Jan 202210 min read

If you are going to work on software using Bitcoin or Liquid, there are several concepts you need to be familiar with. This article aims at developers and is based on a presentation I held for the Pixelmatic team.

Disclaimer: this is an introduction and it does not cover everything obviously. And I simplified some explanations on purpose.

Summary

Bitcoin comes in the form of a daemon, a background process. Which is composed essentially of these components: - 🔗 Blockchain - a distributed ledger of transactions - 🤝🏼 Consensus protocol - a peer to peer network with PoW consensus - 👛 Wallet(s) - a tool to manage your funds - 🤖 Script - a programming language to state the behavior of the transactions - 📃 API - an RPC interface to interact with the bitcoin node

We will go through these concepts and I will add some other important notions on top of these subjects.

🔗 Blockchain

In essence, the Bitcoin blockchain is a linked list where a node has a link to its parent.

Bitcoin blockchain structure

Source: Wikipedia

A node, or called block too, contains also information: * A Merkle tree indexing and containing the transactions * A timestamp to state at what time the block was added to the blockchain * A nonce which is manipulated in the step of mining

There is a bit more information in the Bitcoin blocks, but these are the essentials.

FAQ: What is a ledger?

To me, the best analogy or even illustration of the concept is the poker game.

Imagine you are having a poker night with some friends.

That night, maybe you forgot to bring your poker coins.

So one of you is designated to write down on a piece of paper the results of every match.

The thing is after ten matches, for anybody, it's hard to remember what happened.

So when you check the notes from your friend at the end of the night, you can't tell if your friend cheated or maybe just made a mistake on any particular match.

It is where the public ledger comes to play.

Now imagine instead of having only one friend keeping the books, all of your friends, including you, keep a copy of that ledger.

At every match, all of you write down the results in your ledger.

You understand now that it's going to be harder to make a mistake as you can compare with your friends at any time!

It is what the bitcoin public distributed ledger is all about. Everybody gets a copy so that you can verify and double-check with everybody your copy.

Keeping copies of a ledger is only part of the problem. Participants also need validation and consensus rules to resolve conflicting histories; replication alone does not establish which transactions are valid.

💱 UTXOs

Before looking at the consensus protocol, I think it's important to understand the transaction model used in Bitcoin and the concept of UTXO.

Example of transactions

Source: Medium

In an ordinary transaction, the sum of the inputs equals the sum of the outputs plus the transaction fee. The coinbase transaction that pays the miner follows different rules.
UTXO stands for Unspent Transaction Output. This is the actual money you can spend and are deemed to become inputs.
In the example, Alice sends 0.5 BTC to Bob and sends the rest back to her. To maximize privacy, the rest of the input's owner is conventionally sent back to another address different from the original UTXO, called change address.
You can use multiple UTXO as inputs to fund a transaction.

🤝🏼 Consensus protocol

The consensus protocol is made for securing the funds on Bitcoin, and validate what is the source of truth.

In order to do that, Bitcoin nodes are part of a peer-to-peer network and the rights of appending a block in the blockchain are somehow randomized.

The consensus happens on two steps: - When signaling a transaction to the network

The direct peers of the bitcoin node broadcasting a transaction will evaluate the transaction and if it is valid will add it into their mempool and continue the propagation of the information.

When submitting a new block to the network

The direct peers of the bitcoin node submitting the block will evaluate the block and if it is valid will append it into their blockchain and broadcast it to their other peers.

The mempool

Each node keeps its own mempool of unconfirmed transactions. Nodes relay transactions, but their mempools can differ because of policy, configuration and the transactions they have seen. Acceptance into one node's mempool is not network consensus.

A node can evict a transaction when its mempool fills or the transaction expires. That does not cancel it globally: another node may still have it, and it may be broadcast again. Miners generally consider fee rate (fees relative to transaction size), not just the absolute fee, when choosing transactions.

See the Bitcoin developer guide to transactions.

Mining and Proof-of-work

To earn the right to append a block to the blockchain, the bitcoin nodes use the concept of proof of work.

On the bitcoin network, there is a variable synchronized on every node called the difficulty. That variable corresponds to the mining difficulty. And to earn the right to append a block, the goal is to try different nonces in a block template provided by the bitcoin daemon until the value of the hash is lower than a certain target hash extrapolated from the difficulty.

It is visible by looking at the hashes of the blocks, for example: - at block 2, the difficulty was 1, and the block hash: 000000006a625f06636b8bb6ac7b960a8d03705d1ace08b1a19da3fdcc99ddbd, - at block 718023, the difficulty was 24371874614345, and the block hash: 000000000000000000024d24499352253749af74e93c4c50553c1ec747375953.

The higher the difficulty, the lesser chance to find a block, the smaller the target number will be, thus the higher number of leading zeroes there will be.

In the first blocks, the difficulty was 1. And every 2016 block (about two weeks), the difficulty is adjusted up or down in order to make sure we get blocks every 10 minutes on average. Because this operation of trying nonce and hashing is actually predictable (on average).

Network splits

If the network splits, each side can see a different valid chain. A node follows the valid chain with the greatest accumulated proof of work, not simply the most blocks.

Difficulty on Bitcoin mainnet adjusts at block intervals of 2,016 blocks, not when two weeks of wall-clock time have passed. If a partition has very little mining power, reaching the next adjustment can take much longer. When connectivity returns, blocks on the lower-work branch may be reorganized away; applications must account for that possibility.

See block chain selection and difficulty.

👛 Wallets

Many Bitcoin wallets use Hierarchical Deterministic (HD) key derivation; this is a wallet design, not a requirement imposed on every wallet by the network.

HD Wallets BIP32 Hierarchical Deterministic Wallet

Source: Medium

The goal of the wallet is to simplify the management of your funds and UTXOs. Wallet software can select UTXOs for you, balancing fees, change and other constraints; the selection strategy depends on the wallet and is not simply largest-first.
From an initial seed, we create a key pair of a public and private key.
Those keys can be derived to obtain child keys.
Initially, the derivation was thought to possibly manage multiple accounts and for multiple coins. What happens in practice is often you don't mix coins in your wallets, but it's possible, for instance, Ethereum also uses HD Wallets.
The first use case of having multiple accounts is for change addresses. That way you make sure that the addresses you generate on the fly to receive payments do not overlap the change addresses. Reducing address reuse can improve privacy, but it does not guarantee that transactions cannot be linked.
Having multiple accounts can be helpful to split balance for managing different budgets.

FAQ: What is a wallet?

The first thing to understand is that the common digital wallet (a.k.a. Hierarchical Deterministic Wallet) nowadays is already an evolution of the concept. In the sense, that they supersede the original definition of a digital wallet with extra capabilities. But these super wallets are still called wallets.

So what is a bitcoin wallet? To answer I am going to use an analogy that I like.

Your wallet consists of two things:

📃 an index document that lists all your (bank) safes number,
🔑 and a keychain with all the keys to open your (bank) safes.

Now imagine Bitcoin being a bank vault with an infinite amount of safe.

Three things to note on these safes:

They all have a unique number written on their door.
As we've seen above, things are public on Bitcoin, so they are completely transparent. One could glance at it, and see how much Bitcoin is sitting in it. But anyhow, one cannot open it without its key.
Unlike a bank safe, these safes have an opening to freely slid deposits in them.

Now let's do the mind experiment of sending some money to a friend!

First, you would essentially go into the Bitcoin vault with your document and your keychain.

Then:

find one of your safes, where you have money in, thanks to your index,
open it with your keychain,
take out everything from it,
find a quiet place to make the count,
split the money into two parts, with the part for your friend:
- you would search for his safe first, with the number he gave you,
- slid his part into his safe,
then the rest of your money, you slid it into one of your other empty safe (called change address IRL).

Using a fresh change address reduces address reuse, but does not guarantee anonymity: observers may infer relationships from the transaction structure and other information.

Some links to go further on the subject:

https://www.investopedia.com/terms/h/hd-wallet-hierarchical-deterministic-wallet.asp
https://bitcoinbriefly.com/ultimate-guide-to-bitcoin-wallets-seeds-private-keys-public-keys-and-addresses/
https://medium.com/@harshagoli/hd-wallets-explained-from-high-level-to-nuts-and-bolts-9a41545f5b0
https://en.bitcoinwiki.org/wiki/Deterministic_wallet

Where the analogy stops

A wallet manages keys and tracks spendable transaction outputs; the coins are recorded on the blockchain, not stored inside the wallet file. Addresses describe payment destinations rather than private bank accounts. A transaction can reveal links between its inputs and outputs, even when it uses a new change address. See the Bitcoin transaction guide.

🧑‍🤝‍🧑 Multisig Wallets

Wallets can be also shared among multiple owners, and we call those multisignature wallets.

Multisignature Wallets Justin, Vittie, and Craig each hold one of the three keys needed to unlock the multisig wallet.

Source: Coindesk

There are two major use cases for the multisignature wallets: - Share funds across multiple people. Imagine a company owning a bunch of Bitcoins like a crypto exchange, having a multisignature wallet will make sure that if someone is not available other people on the team can access the funds.

Real-life example:

https://edition.cnn.com/2019/02/05/tech/quadriga-gerald-cotten-cryptocurrency/index.html - Have a backup of your wallet. Imagine you could have a signature on your computer, on your phone, and the third one on an old device hidden. If you lose any one of your devices, then you still have a chance to access your funds and move them to a new wallet.

🤖 Script

Bitcoin offers a simple stack-based programming language, non-Turing complete (on purpose). Below find the page about it from the Bitcoin Wiki.

Open reference documentation ↗

Source: Bitcoin Wiki

Bitcoin is sometimes called programmable money because of this language.
It describes how UTXOs should be validated.
I would invite you to read some Script examples from the Bitcoin wiki to quickly understand the nature of the language
FYI The opcode OP_RETURN can be used to leave a message on the blockchain.

📃 API

To interact with the bitcoin daemon, it exposes an RPC interface. This is what the bitcoin CLI tool uses, but you could also use curl or any HTTP client to communicate with a bitcoin node.

Open reference documentation ↗

Source: Bitcoin.org

You can do about everything with the interface, so it's not safe to expose it on the internet.
Keep RPC private. Authentication is necessary, but credentials alone are not a reason to expose the interface publicly; use a restricted local connection or a controlled private network.
Some examples (the send command spends funds; use regtest for experimentation and check bitcoin-cli help sendtoaddress for your installed version):
To send 1 BTC to a friend: bitcoin-cli sendtoaddress "${YOUR_FRIEND_ADDRESS}" 1
To get your balance: bitcoin-cli getbalance
To submit a block to the network: bitcoin-cli submitblock ${HEX_BLOCK}
To check if your local blockchain is fully synchronized: bitcoin-cli getblockchaininfo

🗣️ Bitcoin Improvement Proposal (BIP)

Major updates and changes in Bitcoin are always first discussed and properly specified in a document labeled BIP. All the BIPs and work-in-progress BIP are stored there:

https://github.com/bitcoin/bips

This is where the developer community consensus happens somehow.
Examples of BIPs that defined the wallets as we know them now:
BIP32 https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki
BIP44 https://github.com/bitcoin/bips/blob/master/bip-0044.mediawiki
One of the BIP in discussion to be included in one of the next major versions of Bitcoin (as of early 2022):
BIP119 https://github.com/bitcoin/bips/blob/master/bip-0119.mediawiki

🎇 Conclusion

As of 2022 and several years from now, if you understand all the concepts cited in that article, you are good enough to start with your development using Bitcoin.

Depending on what part of Bitcoin you will be working with or using, here are some hints on where you should start digging: - Regarding mining, blocks contain more information than I quoted, - Regarding transactions and Script, there are different types of transactions using Script: p2sh, p2pkh, p2wpkh, but also an interesting thing called Hashed TimeLock Contract (HTLC), - Regarding wallets, you probably want to look at the latest updates with Taproot that allows for more privacy with multisignature wallets, - More generally there are Lightning as layer 2 of Bitcoin solution for payments settlement, and Liquid a sidechain of Bitcoin with confidential transactions and assets issuance.

Continue with Introduction to Liquid for developers.