Page MenuHomePhorge
Files F112306

No OneTemporary
Actions

Size
10 KB
Referenced Files
None
Subscribers
None

View Options

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9f71e5c..f810f11 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,30 +1,31 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
## Unreleased
### Changed
* `FlakeId.get/0` function now returns `{:ok, flake}` or `{:error, some_atom}`.
* `FlakeId.Ecto.Type.autogenerate/0` will throw an error when a flake cannot be generated.
* Split up build functions in a separate `FlakeId.Builder` module.
### Fixed
* Use time from `os:system_time` which follows OS time changes.
* Errors when the sequence became too big for 16bit integer (when clock is not advancing).
### Added
* `FlakeId.get_local/0` local (in-process) generation strategy.
* `FlakeId.backdate/2`.
-* `FlakeId.keyfilter/1`.
+* `FlakeId.notional/1`.
* Configurable worker-id strategy (random, mac, node name, configurable integer).
* Persistable time-hint file to protect against clock going backwards between restarts.
+* Multiple `FlakeId.Worker` may be started and `FlakeId.get/1` allows to pass a worker registered name.
## [0.1.0] - 2019-09-25
Initial release
diff --git a/README.md b/README.md
index 570470d..bde0ccc 100644
--- a/README.md
+++ b/README.md
@@ -1,98 +1,87 @@
# ❄ FlakeId
> Decentralized, k-ordered ID generation service
Flake Ids are 128-bit identifiers, sortable by time, and can be generated safely from a multi-node system without
coordination.
Flake Ids consist of a 64 bit timestamp, 48 bit worker identifier, and 16 bit sequence. They are usually
represented as a Base 62 string and can be sorted either by bit or by their string representation.
The sort precision is limited to a millisecond, which is probably fine enough for most usages.
## Installation
Add `flake_id` to your list of dependencies in `mix.exs`:
```elixir
def deps do
[
{:flake_id, "~> 0.1.0"}
]
end
```
## Configuration
+## Time Warp
+
+Uniqueness of ID depends a lot on the accuracy of the clock. It is recommended to use the `+C multi_time_warp` BEAM ERTS
+option to use a more secure clock. Otherwise, Flake will use your system clock and will refuse to generate identifiers
+if it goes backwards.
+
+[More information on handling time in
+Erlang](https://adoptingerlang.org/docs/development/hard_to_get_right/#handling-time).
+
+## Time Hint
+
+If enabled, flake will periodically save the current time. If flake detects on startup that this file contains timestamps
+in the future or the distant past, it will refuse to start up.
+
+This can be enabled with:
+
+```
+config :flake_id,
+ time_hint: true
+```
+
+By default, it will save in flake_id's priv directory. You can set this to another path by setting `:time_hint_file`.
+
### Worker ID
The Worker ID is a very important part of a flake. It guarantees that individual nodes within the same system
can generate IDs without risks of collision. Take care to choose Worker IDs that are guaranteed to be unique
for your system.
You have multiple strategies available depending on your setup:
* random byte-string, generated at node boot
* erlang node name
* mac address (either random, or manually specified)
* fixed string or integer
It is configured as `:worker` in the `:flake_id` app environement and is picked up at boot time.
Worker: `:mac | {:mac, iface} | :node | :random | String.t | Integer.t`
## Usage
-### get/0
-
-The most common usage will be via `get/0`:
-
-```elixir
-iex> FlakeId.get()
-{:ok, "9n3171dJZpdD77K3DU"}
-```
-
-### get_local/0
-
-You can also generate flakes in-process without hitting a node-wide worker. This is especially useful when you have
-long-running processes that do a lot of insertions/ID generation. Note that the first call will be slowest as it
-has to setup the Flake state.
-
-Each process can have its Flake state stored in the process dictionary and the Worker ID is derived from the global
-Worker ID and the current PID:
-
-```elixir
-iex> FlakeId.get_local()
-{:ok, "9uAiavFlMUeyVxbNGT"}
-```
-
-### backdate/2
-
-If you wish to insert historical data while preserving sorting, you can achieve this using a backdated flake. In that
-case you have to provide yourself a **UNIQUE** Worker ID (for the given time). Be very careful to ensure the uniqueness
-of the Worker ID otherwise you may generate the same flake twice.
-
-```elixir
-iex> FlakeId.backdate(~U[2019-12-05 07:15:00.793018Z], "https://flakesocial.com/status/390929393992"
-{:ok, "9peCDV9GeXwzM15BE9"}
-```
-
-See [https://hexdocs.pm/flake_id](https://hexdocs.pm/flake_id) for the complete documentation.
+See [`FlakeId` module documentation](https://hexdocs.pm/flake_id/FlakeId.html).
### With Ecto
It is possible to use `FlakeId` with [`Ecto`](https://github.com/elixir-ecto/ecto/).
See [`FlakeId.Ecto.Type`](https://hexdocs.pm/flake_id/FlakeId.Ecto.Type.html) documentation for detailed instructions.
If you wish to migrate from integer serial ids to Flake, see [`FlakeId.Ecto.CompatType`](https://hexdocs.pm/flake_id/FlakeId.Ecto.Type.html) for instructions.
## Prior Art
* [flaky](https://github.com/nirvana/flaky), released under the terms of the Truly Free License,
* [Flake](https://github.com/boundary/flake), Copyright 2012, Boundary, Apache License, Version 2.0
## Copyright and License
Copyright © 2017-2020 [Pleroma Authors](https://pleroma.social/)
FlakeId source code is licensed under the GNU LGPLv3 License.
diff --git a/lib/flake_id.ex b/lib/flake_id.ex
index 34fbc45..1548aa3 100644
--- a/lib/flake_id.ex
+++ b/lib/flake_id.ex
@@ -1,154 +1,157 @@
# FlakeId: Decentralized, k-ordered ID generation service
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: LGPL-3.0-only
defmodule FlakeId do
@moduledoc """
Decentralized, k-ordered ID generation service.
"""
import Kernel, except: [to_string: 1]
alias FlakeId.Builder
alias FlakeId.Worker
alias FlakeId.Local
@type error ::
{:error, :clock_running_backwards} | {:error, :clock_stuck} | {:error, :invalid_state}
@doc """
Converts a binary Flake to a String
## Examples
iex> FlakeId.to_string(<<0, 0, 1, 109, 67, 124, 251, 125, 95, 28, 30, 59, 36, 42, 0, 0>>)
"9n2ciuz1wdesFnrGJU"
"""
def to_string(<<_::integer-size(64), _::integer-size(48), _::integer-size(16)>> = binary_flake) do
<<integer::integer-size(128)>> = binary_flake
Base62.encode(integer)
end
def to_string(string), do: string
@doc """
Converts a String to a binary Flake
## Examples
iex> FlakeId.from_string("9n2ciuz1wdesFnrGJU")
<<0, 0, 1, 109, 67, 124, 251, 125, 95, 28, 30, 59, 36, 42, 0, 0>>
"""
@spec from_string(String.t()) :: nil | <<_::128>>
def from_string(string)
def from_string(<<_::integer-size(128)>> = flake), do: flake
def from_string(string) when is_binary(string) and byte_size(string) < 18, do: nil
def from_string(string), do: string |> Base62.decode!() |> from_integer
@doc """
Converts a binary Flake to an integer
## Examples
iex> FlakeId.to_integer(<<0, 0, 1, 109, 67, 124, 251, 125, 95, 28, 30, 59, 36, 42, 0, 0>>)
28939165907792829150732718047232
"""
@spec to_integer(<<_::128>>) :: non_neg_integer
def to_integer(binary_flake)
def to_integer(<<integer::integer-size(128)>>), do: integer
@doc """
Converts an integer to a binary Flake
## Examples
iex> FlakeId.from_integer(28939165907792829150732718047232)
<<0, 0, 1, 109, 67, 124, 251, 125, 95, 28, 30, 59, 36, 42, 0, 0>>
"""
@spec from_integer(integer) :: <<_::128>>
def from_integer(integer) do
<<_time::integer-size(64), _node::integer-size(48), _seq::integer-size(16)>> =
<<integer::integer-size(128)>>
end
@doc """
Generates a string with Flake
"""
@spec get(atom()) :: {:ok, String.t()} | error()
def get(name \\ Worker) do
case Worker.get(name) do
{:ok, flake} -> {:ok, to_string(flake)}
error -> error
end
end
@doc """
- Generates a local Flake (in-process state). Worker ID will be derived from global worker-id and the process pid.
+ Generate a in-process flake, without hitting a node-wide worker. This is especially useful when you have
+ long-running processes that do a lot of insertions/ID generation. Note that the first call will be slowest as it
+ has to setup the Flake state.
- First call will be slower but subsequent calls will be faster than worker based `get/0` and is quite useful for long running processes.
+ Each process can have its Flake state stored in the process dictionary and the Worker ID is derived from the global
+ Worker ID and the current PID.
"""
@spec get_local :: {:ok, String.t()} | error()
def get_local do
case Local.get() do
{:ok, flake} -> {:ok, to_string(flake)}
error -> error
end
end
@doc """
- Generates a predictable and back-dated FlakeId.
-
- This can be useful when you want to insert historical data without messing with sorting: be **very careful** at choosing a `unique` value that is _really unique_ for the given `datetime`, otherwise, collisons will happen.
+ If you wish to insert historical data while preserving sorting, you can achieve this using a backdated flake. In that
+ case you have to provide yourself a **UNIQUE** Worker ID (for the given time). Be very careful to ensure the uniqueness
+ of the Worker ID otherwise you may generate the same flake twice.
"""
@spec backdate(any(), DateTime.t()) :: {:ok, String.t()} | error()
def backdate(datetime, unique) do
hash = :erlang.phash2(unique)
time = DateTime.to_unix(datetime, :millisecond)
case Builder.get(time, %Builder{node: hash}) do
{:ok, flake, _} -> {:ok, to_string(flake)}
error -> error
end
end
@doc """
Generates a notional FlakeId (with an empty worker id) given a timestamp.
This is useful for generating a lexical range of values that could have been generated in a span of time.
"""
def notional(datetime) do
time = DateTime.to_unix(datetime, :millisecond)
case Builder.get(time, %Builder{node: 0}) do
{:ok, flake, _} -> {:ok, to_string(flake)}
error -> error
end
end
@doc """
Checks that ID is a valid FlakeId
## Examples
iex> FlakeId.flake_id?("9n2ciuz1wdesFnrGJU")
true
iex> FlakeId.flake_id?("#cofe")
false
iex> FlakeId.flake_id?("pleroma.social")
false
"""
@spec flake_id?(String.t()) :: boolean
def flake_id?(id), do: flake_id?(String.to_charlist(id), true)
defp flake_id?([c | cs], true) when c >= ?0 and c <= ?9, do: flake_id?(cs, true)
defp flake_id?([c | cs], true) when c >= ?A and c <= ?Z, do: flake_id?(cs, true)
defp flake_id?([c | cs], true) when c >= ?a and c <= ?z, do: flake_id?(cs, true)
defp flake_id?([], true), do: true
defp flake_id?(_, _), do: false
end

File Metadata

Mime Type
text/x-diff
Expires
Sat, Nov 23, 8:01 AM (1 d, 9 m)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
38909
Default Alt Text
(10 KB)

Event Timeline