Page MenuHomePhorge
Files F112352

No OneTemporary
Actions

Size
36 KB
Referenced Files
None
Subscribers
None

View Options

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3396a54..9f71e5c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,29 +1,30 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
## Unreleased
### Changed
* `FlakeId.get/0` function now returns `{:ok, flake}` or `{:error, some_atom}`.
* `FlakeId.Ecto.Type.autogenerate/0` will throw an error when a flake cannot be generated.
* Split up build functions in a separate `FlakeId.Builder` module.
### Fixed
* Use time from `os:system_time` which follows OS time changes.
* Errors when the sequence became too big for 16bit integer (when clock is not advancing).
### Added
* `FlakeId.get_local/0` local (in-process) generation strategy.
-* (Still WIP) Configurable worker-id strategy (random, mac, node name, configurable integer).
-* `FlakeId.backdate/0`.
-* (WIP) Possibility to persist a time-hint file to protect against clock going backwards between restarts.
+* `FlakeId.backdate/2`.
+* `FlakeId.keyfilter/1`.
+* Configurable worker-id strategy (random, mac, node name, configurable integer).
+* Persistable time-hint file to protect against clock going backwards between restarts.
## [0.1.0] - 2019-09-25
Initial release
diff --git a/README.md b/README.md
index e8126fb..570470d 100644
--- a/README.md
+++ b/README.md
@@ -1,108 +1,98 @@
# ❄ FlakeId
> Decentralized, k-ordered ID generation service
Flake Ids are 128-bit identifiers, sortable by time, and can be generated safely from a multi-node system without
coordination.
Flake Ids consist of a 64 bit timestamp, 48 bit worker identifier, and 16 bit sequence. They are usually
represented as a Base 62 string and can be sorted either by bit or by their string representation.
The sort precision is limited to a millisecond, which is probably fine enough for most usages.
## Installation
Add `flake_id` to your list of dependencies in `mix.exs`:
```elixir
def deps do
[
{:flake_id, "~> 0.1.0"}
]
end
```
## Configuration
### Worker ID
The Worker ID is a very important part of a flake. It guarantees that individual nodes within the same system
can generate IDs without risks of collision. Take care to choose Worker IDs that are guaranteed to be unique
for your system.
You have multiple strategies available depending on your setup:
* random byte-string, generated at node boot
* erlang node name
* mac address (either random, or manually specified)
* fixed string or integer
It is configured as `:worker` in the `:flake_id` app environement and is picked up at boot time.
Worker: `:mac | {:mac, iface} | :node | :random | String.t | Integer.t`
-### Persisted Time Hint
-
-Flake's time is dependent on your OS system clock. At runtime Flake will refuse to generate IDs if the system clock goes
-backwards. In some cases after an OS reboot the clock may be incorrectly set to a time in the past which will trigger
-this condition. Enabling a time hint file will prevent this by refusing to start the Flake worker.
-
-Persisted Time hint is configured as `:time_hint` in the `:flake_id` app environement. By default it is disabled (`false`).
-Other accepted values are `true` which stores the time hint file in flake_id priv dir or it can be set to a string pointing
-to a file.
-
## Usage
### get/0
The most common usage will be via `get/0`:
```elixir
iex> FlakeId.get()
{:ok, "9n3171dJZpdD77K3DU"}
```
### get_local/0
You can also generate flakes in-process without hitting a node-wide worker. This is especially useful when you have
long-running processes that do a lot of insertions/ID generation. Note that the first call will be slowest as it
has to setup the Flake state.
Each process can have its Flake state stored in the process dictionary and the Worker ID is derived from the global
Worker ID and the current PID:
```elixir
iex> FlakeId.get_local()
{:ok, "9uAiavFlMUeyVxbNGT"}
```
### backdate/2
If you wish to insert historical data while preserving sorting, you can achieve this using a backdated flake. In that
case you have to provide yourself a **UNIQUE** Worker ID (for the given time). Be very careful to ensure the uniqueness
of the Worker ID otherwise you may generate the same flake twice.
```elixir
-iex> FlakeId.backdate("https://flakesocial.com/status/390929393992", ~U[2019-12-05 07:15:00.793018Z])
+iex> FlakeId.backdate(~U[2019-12-05 07:15:00.793018Z], "https://flakesocial.com/status/390929393992"
{:ok, "9peCDV9GeXwzM15BE9"}
```
See [https://hexdocs.pm/flake_id](https://hexdocs.pm/flake_id) for the complete documentation.
### With Ecto
It is possible to use `FlakeId` with [`Ecto`](https://github.com/elixir-ecto/ecto/).
See [`FlakeId.Ecto.Type`](https://hexdocs.pm/flake_id/FlakeId.Ecto.Type.html) documentation for detailed instructions.
If you wish to migrate from integer serial ids to Flake, see [`FlakeId.Ecto.CompatType`](https://hexdocs.pm/flake_id/FlakeId.Ecto.Type.html) for instructions.
## Prior Art
* [flaky](https://github.com/nirvana/flaky), released under the terms of the Truly Free License,
* [Flake](https://github.com/boundary/flake), Copyright 2012, Boundary, Apache License, Version 2.0
## Copyright and License
Copyright © 2017-2020 [Pleroma Authors](https://pleroma.social/)
FlakeId source code is licensed under the GNU LGPLv3 License.
diff --git a/lib/flake_id.ex b/lib/flake_id.ex
index cf1f8d0..1107a0e 100644
--- a/lib/flake_id.ex
+++ b/lib/flake_id.ex
@@ -1,140 +1,154 @@
# FlakeId: Decentralized, k-ordered ID generation service
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: LGPL-3.0-only
defmodule FlakeId do
@moduledoc """
Decentralized, k-ordered ID generation service.
"""
import Kernel, except: [to_string: 1]
alias FlakeId.Builder
alias FlakeId.Worker
alias FlakeId.Local
- @type error :: {:error, :clock_running_backwards} | {:error, :clock_stuck} | {:error, :invalid_state}
+ @type error ::
+ {:error, :clock_running_backwards} | {:error, :clock_stuck} | {:error, :invalid_state}
@doc """
Converts a binary Flake to a String
## Examples
iex> FlakeId.to_string(<<0, 0, 1, 109, 67, 124, 251, 125, 95, 28, 30, 59, 36, 42, 0, 0>>)
"9n2ciuz1wdesFnrGJU"
"""
def to_string(<<_::integer-size(64), _::integer-size(48), _::integer-size(16)>> = binary_flake) do
<<integer::integer-size(128)>> = binary_flake
Base62.encode(integer)
end
def to_string(string), do: string
@doc """
Converts a String to a binary Flake
## Examples
iex> FlakeId.from_string("9n2ciuz1wdesFnrGJU")
<<0, 0, 1, 109, 67, 124, 251, 125, 95, 28, 30, 59, 36, 42, 0, 0>>
"""
@spec from_string(String.t()) :: nil | <<_::128>>
def from_string(string)
def from_string(<<_::integer-size(128)>> = flake), do: flake
def from_string(string) when is_binary(string) and byte_size(string) < 18, do: nil
def from_string(string), do: string |> Base62.decode!() |> from_integer
@doc """
Converts a binary Flake to an integer
## Examples
iex> FlakeId.to_integer(<<0, 0, 1, 109, 67, 124, 251, 125, 95, 28, 30, 59, 36, 42, 0, 0>>)
28939165907792829150732718047232
"""
@spec to_integer(<<_::128>>) :: non_neg_integer
def to_integer(binary_flake)
def to_integer(<<integer::integer-size(128)>>), do: integer
@doc """
Converts an integer to a binary Flake
## Examples
iex> FlakeId.from_integer(28939165907792829150732718047232)
<<0, 0, 1, 109, 67, 124, 251, 125, 95, 28, 30, 59, 36, 42, 0, 0>>
"""
@spec from_integer(integer) :: <<_::128>>
def from_integer(integer) do
<<_time::integer-size(64), _node::integer-size(48), _seq::integer-size(16)>> =
<<integer::integer-size(128)>>
end
@doc """
Generates a string with Flake
"""
@spec get :: {:ok, String.t()} | error()
def get do
case Worker.get() do
{:ok, flake} -> {:ok, to_string(flake)}
error -> error
end
end
@doc """
Generates a local Flake (in-process state). Worker ID will be derived from global worker-id and the process pid.
First call will be slower but subsequent calls will be faster than worker based `get/0` and is quite useful for long running processes.
"""
@spec get_local :: {:ok, String.t()} | error()
def get_local do
case Local.get() do
{:ok, flake} -> {:ok, to_string(flake)}
error -> error
end
end
@doc """
- Generates a predictable and back-dated Flake.
+ Generates a predictable and back-dated FlakeId.
- This can be useful when you want to insert historical data without messing with sorting.
-
- Be **very careful** at choosing a `unique` value that is _really unique_ for the given `datetime`, otherwise, collisons will happen.
+ This can be useful when you want to insert historical data without messing with sorting: be **very careful** at choosing a `unique` value that is _really unique_ for the given `datetime`, otherwise, collisons will happen.
"""
@spec backdate(any(), DateTime.t()) :: {:ok, String.t()} | error()
- def backdate(unique, datetime) do
+ def backdate(datetime, unique) do
hash = :erlang.phash2(unique)
time = DateTime.to_unix(datetime, :millisecond)
- case Builder.get(time, %Builder{node: hash, time: time, sq: 0}) do
+
+ case Builder.get(time, %Builder{node: hash}) do
+ {:ok, flake, _} -> {:ok, to_string(flake)}
+ error -> error
+ end
+ end
+
+ @doc """
+ Generates a notional FlakeId (with an empty worker id) given a timestamp.
+
+ This is useful for generating a lexical range of values that could have been generated in a span of time.
+ """
+ def notional(datetime) do
+ time = DateTime.to_unix(datetime, :millisecond)
+
+ case Builder.get(time, %Builder{node: 0}) do
{:ok, flake, _} -> {:ok, to_string(flake)}
error -> error
end
end
@doc """
Checks that ID is a valid FlakeId
## Examples
iex> FlakeId.flake_id?("9n2ciuz1wdesFnrGJU")
true
iex> FlakeId.flake_id?("#cofe")
false
iex> FlakeId.flake_id?("pleroma.social")
false
"""
@spec flake_id?(String.t()) :: boolean
def flake_id?(id), do: flake_id?(String.to_charlist(id), true)
defp flake_id?([c | cs], true) when c >= ?0 and c <= ?9, do: flake_id?(cs, true)
defp flake_id?([c | cs], true) when c >= ?A and c <= ?Z, do: flake_id?(cs, true)
defp flake_id?([c | cs], true) when c >= ?a and c <= ?z, do: flake_id?(cs, true)
defp flake_id?([], true), do: true
defp flake_id?(_, _), do: false
end
diff --git a/lib/flake_id/application.ex b/lib/flake_id/application.ex
index c87f562..61fbc50 100644
--- a/lib/flake_id/application.ex
+++ b/lib/flake_id/application.ex
@@ -1,22 +1,27 @@
# FlakeId: Decentralized, k-ordered ID generation service
# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: LGPL-3.0-only
defmodule FlakeId.Application do
- # See https://hexdocs.pm/elixir/Application.html
- # for more information on OTP Applications
@moduledoc false
use Application
+ require Logger
def start(_type, _args) do
+ unless :erlang.system_info(:time_warp_mode) == :multi_time_warp do
+ Logger.error("""
+ FlakeId: Time consistency is not optimal, using system time.
+ More information at: https://hexdocs.pm/flake_id/readme.html#time-warp
+ """)
+ end
+
children = [
+ FlakeId.TimeHint,
FlakeId.Worker
]
- # See https://hexdocs.pm/elixir/Supervisor.html
- # for other strategies and supported options
opts = [strategy: :one_for_one, name: FlakeId.Supervisor]
Supervisor.start_link(children, opts)
end
end
diff --git a/lib/flake_id/builder.ex b/lib/flake_id/builder.ex
index b75fa03..1306634 100644
--- a/lib/flake_id/builder.ex
+++ b/lib/flake_id/builder.ex
@@ -1,91 +1,130 @@
# FlakeId: Decentralized, k-ordered ID generation service
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: LGPL-3.0-only
defmodule FlakeId.Builder do
@moduledoc false
- defstruct node: nil, time: 0, sq: 0
+ defstruct node: nil, time: 0, sq: 0, clock: nil
- @type state :: %__MODULE__{
+ @type t :: %__MODULE__{
node: non_neg_integer,
time: non_neg_integer,
sq: non_neg_integer
- }
+ }
+
+ @type options :: [worker_id_option() | time_hint_option() | clock_option()]
+ @type worker_id_option :: {:worker_id, any()}
+ @type time_hint_option :: {:time_hint, false | true | atom()}
+ @type clock_option :: {module(), atom()}
+
+ @spec new(options()) :: t()
+ def new(options \\ []) do
+ worker_id = worker_id(Keyword.get(options, :worker_id))
+
+ time =
+ case Keyword.get(options, :time_hint) do
+ false -> 0
+ name when is_atom(name) -> FlakeId.TimeHint.get(name)
+ _ -> FlakeId.TimeHint.get()
+ end
+
+ clock = Keyword.get(options, :clock, default_clock())
+ %__MODULE__{node: worker_id, time: time, clock: clock}
+ end
- def new(worker_id \\ nil) do
- %__MODULE__{node: worker_id || worker_id(), time: time()}
+ def backwards?(%__MODULE__{time: time, clock: clock}) do
+ time > time(clock)
end
- def time do
- :os.system_time(:millisecond)
+ def time() do
+ time(default_clock())
end
- def worker_id do
- worker_id(Application.get_env(:flake_id, :worker_id, :random))
+ def time(%__MODULE__{clock: clock}) do
+ time(clock)
+ end
+
+ def time({mod, fun}) do
+ apply(mod, fun, [:millisecond])
end
- @spec get(state) :: {:ok, <<_::128>>, state} | FlakeId.error()
- def get(state) do
- get(time(), state)
+ @spec get(t) :: {:ok, <<_::128>>, t} | FlakeId.error()
+ def get(state = %__MODULE__{clock: clock}) do
+ get(time(clock), state)
end
- @spec get(non_neg_integer, state) :: {:ok, <<_::128>>, state} | FlakeId.error()
+ @spec get(non_neg_integer, t) :: {:ok, <<_::128>>, t} | FlakeId.error()
- # Matches when time didn't change and sequence is too big for 16bit int.
- def get(newtime, %__MODULE__{time: time, sq: sq}) when newtime == time and sq >= 65_535 do
+ # Error when time didn't change and sequence is too big for 16bit int.
+ def get(time, %__MODULE__{time: time, sq: sq}) when sq >= 65_535 do
{:error, :clock_stuck}
end
# Matches when the calling time is the same as the state time. Incr. sq
- def get(time, %__MODULE__{node: node, time: time, sq: seq} = state) when is_integer(time) and is_integer(node) and is_integer(seq) do
- new_state = %__MODULE__{state | sq: state.sq + 1}
- {:ok, gen_flake(new_state), new_state}
- end
-
- # Matches when the times are different, reset sq
- def get(newtime, %__MODULE__{node: node, time: time, sq: seq} = state) when is_integer(time) and is_integer(node) and is_integer(seq) and newtime > time do
- new_state = %__MODULE__{state | time: newtime, sq: 0}
+ def get(time, %__MODULE__{node: node, time: time, sq: seq} = state)
+ when is_integer(time) and is_integer(node) and is_integer(seq) do
+ new_state = %__MODULE__{state | sq: seq + 1}
{:ok, gen_flake(new_state), new_state}
end
# Error when clock is running backwards
def get(newtime, %__MODULE__{time: time}) when newtime < time do
{:error, :clock_running_backwards}
end
+ # Matches when the times are different, reset sq
+ def get(newtime, %__MODULE__{node: node, time: time, sq: seq} = state)
+ when is_integer(time) and is_integer(node) and is_integer(seq) and newtime > time do
+ new_state = %__MODULE__{state | time: newtime, sq: 0}
+ {:ok, gen_flake(new_state), new_state}
+ end
+
def get(_, _) do
{:error, :invalid_state}
end
- @spec gen_flake(state) :: <<_::128>>
+ @spec gen_flake(t) :: <<_::128>>
defp gen_flake(%__MODULE__{time: time, node: node, sq: seq}) do
<<time::integer-size(64), node::integer-size(48), seq::integer-size(16)>>
end
- defp worker_id(:node) do
+ defp default_clock() do
+ if :erlang.system_info(:time_warp_mode) == :multi_time_warp do
+ {:erlang, :system_time}
+ else
+ {:os, :system_time}
+ end
+ end
+
+ def worker_id(setting \\ nil)
+
+ def worker_id(nil) do
+ worker_id(Application.get_env(:flake_id, :worker_id, :random))
+ end
+
+ def worker_id(:node) do
:erlang.phash2(node())
end
- defp worker_id(:mac) do
- "derp derp derp"
+ def worker_id({:mac, iface}) do
+ {:ok, addresses} = :inet.getifaddrs()
+ proplist = :proplists.get_value(iface, addresses)
+ hwaddr = Enum.take(:proplists.get_value(:hwaddr, proplist), 6)
+ <<worker::integer-size(48)>> = :binary.list_to_bin(hwaddr)
+ worker
end
- defp worker_id(:random) do
+ def worker_id(:random) do
<<worker::integer-size(48)>> = :crypto.strong_rand_bytes(6)
worker
end
- defp worker_id(integer) when is_integer(integer) do
- integer
+ def worker_id(integer) when is_integer(integer) do
+ 42
end
- defp worker_id(binary) when is_binary(binary) do
- #<<worker::integer-size(48)>> = :erlang.phash2(binary)
- #worker
+ def worker_id(binary) when is_binary(binary) do
:erlang.phash2(binary)
end
-
-
end
-
diff --git a/lib/flake_id/local.ex b/lib/flake_id/local.ex
index 1817035..e624a96 100644
--- a/lib/flake_id/local.ex
+++ b/lib/flake_id/local.ex
@@ -1,29 +1,29 @@
# FlakeId: Decentralized, k-ordered ID generation service
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: LGPL-3.0-only
defmodule FlakeId.Local do
@moduledoc false
alias FlakeId.Builder
@spec get() :: {:ok, String.t()} | FlakeId.error()
def get() do
case Builder.get(get_state()) do
{:ok, flake, state} ->
Process.put(__MODULE__, state)
{:ok, flake}
+
error = {:error, _} ->
error
end
end
defp get_state() do
- Process.get(__MODULE__, Builder.new(worker_id()))
+ Process.get(__MODULE__, Builder.new(worker_id: worker_id()))
end
defp worker_id() do
:erlang.phash2({Builder.worker_id(), self()})
end
-
end
diff --git a/lib/flake_id/time_hint.ex b/lib/flake_id/time_hint.ex
new file mode 100644
index 0000000..08846f2
--- /dev/null
+++ b/lib/flake_id/time_hint.ex
@@ -0,0 +1,121 @@
+# FlakeId: Decentralized, k-ordered ID generation service
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: LGPL-3.0-only
+
+defmodule FlakeId.TimeHint do
+ @moduledoc false
+ use GenServer
+ require Logger
+
+ @tick :timer.minutes(1)
+ @default_env false
+ @default_file "flake_id_time"
+ @clock {FlakeId.Builder, :time}
+
+ @spec get() :: 0 | non_neg_integer
+ @doc "Returns the latest time hint, or zero if disabled"
+ def get(name \\ nil) do
+ if name || Application.get_env(:flake_id, :time_hint, @default_env) do
+ GenServer.call(name || __MODULE__, :get)
+ else
+ 0
+ end
+ end
+
+ def start_link(options)
+
+ def start_link([]) do
+ if Application.get_env(:flake_id, :time_hint, false) do
+ file =
+ Application.get_env(
+ :flake_id,
+ :time_hint_file,
+ Application.app_dir(:flake_id, ["priv", @default_file])
+ )
+
+ tick = Application.get_env(:flake_id, :time_hint_tick, @tick)
+ clock = Application.get_env(:flake_id, :time_hint_clock, @clock)
+ GenServer.start_link(__MODULE__, [file, tick, clock], name: __MODULE__)
+ else
+ :ignore
+ end
+ end
+
+ def start_link(options) do
+ name = Keyword.fetch!(options, :name)
+ file = Keyword.fetch!(options, :file)
+ tick = Keyword.get(options, :tick, @tick)
+ clock = Keyword.get(options, :clock, @clock)
+ GenServer.start_link(__MODULE__, [file, tick, clock], name: name)
+ end
+
+ @impl true
+ def init([file, tick, clock]) do
+ Process.send_after(self(), :tick, tick)
+ {:ok, {file, tick, clock, read_and_update(file, clock)}}
+ end
+
+ @impl true
+ def handle_call(:get, _, state = {_, _, _, time}) do
+ {:reply, time, state}
+ end
+
+ @impl true
+ def handle_info(:tick, {file, tick, clock, prev_time}) do
+ Process.send_after(self(), :tick, tick)
+ {:noreply, {file, tick, clock, update(file, clock, prev_time)}}
+ end
+
+ if Mix.env() == :test do
+ @impl true
+ def handle_info(:stop, state) do
+ {:stop, :normal, state}
+ end
+ end
+
+ @impl true
+ def terminate(_, {file, _, clock, prev_time}) do
+ update(file, clock, prev_time)
+ end
+
+ defp read_and_update(file, clock) do
+ update(file, clock, read(file))
+ end
+
+ defp update(file, {mod, fun}, prev_time) do
+ time = :erlang.apply(mod, fun, [])
+
+ if time > prev_time do
+ :ok = write(file, time)
+ time
+ else
+ prev_time
+ end
+ end
+
+ defp read(file) do
+ case File.read(file) do
+ {:ok, data} ->
+ {__MODULE__, time} = :erlang.binary_to_term(data)
+ time
+
+ {:error, :enoent} ->
+ 0
+
+ error ->
+ Logger.error("#{__MODULE__}: could not read file #{file}: #{inspect(error)}")
+ throw(error)
+ end
+ end
+
+ defp write(file, time) do
+ case File.write(file, :erlang.term_to_binary({__MODULE__, time})) do
+ :ok ->
+ :ok
+
+ error ->
+ Logger.error("#{__MODULE__}: could not write file #{file}: #{inspect(error)}")
+ error
+ end
+ end
+end
diff --git a/lib/flake_id/worker.ex b/lib/flake_id/worker.ex
index c0ada20..e253594 100644
--- a/lib/flake_id/worker.ex
+++ b/lib/flake_id/worker.ex
@@ -1,34 +1,45 @@
# FlakeId: Decentralized, k-ordered ID generation service
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: LGPL-3.0-only
defmodule FlakeId.Worker do
@moduledoc false
use GenServer
+ require Logger
alias FlakeId.Builder
- def start_link(_) do
- GenServer.start_link(__MODULE__, [], name: __MODULE__)
+ @type options :: [{:name, atom()} | Builder.options()]
+
+ def start_link(options \\ []) do
+ name = Keyword.get(options, :name, __MODULE__)
+ GenServer.start_link(__MODULE__, [options], name: name)
end
@spec get :: {:ok, String.t()} | FlakeId.error()
- def get, do: GenServer.call(__MODULE__, :get)
+ def get(name \\ __MODULE__), do: GenServer.call(name, :get)
@impl true
- @spec init([]) :: {:ok, Builder.t}
- def init([]) do
- {:ok, Builder.new()}
+ @spec init([]) :: {:ok, Builder.t()} | {:stop, :clock_is_backwards}
+ def init([options]) do
+ state = Builder.new(options)
+
+ if Builder.backwards?(state) do
+ Logger.error("#{__MODULE__} Current time is backwards, not starting")
+ {:stop, :clock_is_backwards}
+ else
+ {:ok, Builder.new()}
+ end
end
@impl true
def handle_call(:get, _from, state) do
case Builder.get(state) do
{:ok, flake, new_state} ->
{:reply, {:ok, flake}, new_state}
+
error ->
{:reply, error, state}
end
end
-
end
diff --git a/mix.exs b/mix.exs
index 9138539..34f3308 100644
--- a/mix.exs
+++ b/mix.exs
@@ -1,52 +1,56 @@
defmodule FlakeId.MixProject do
use Mix.Project
def project do
[
app: :flake_id,
version: "0.1.0",
description: "Decentralized, k-ordered ID generation service",
elixir: "~> 1.8",
+ elixirc_paths: elixirc_paths(Mix.env()),
start_permanent: Mix.env() == :prod,
deps: deps(),
package: package(),
# Docs
name: "FlakeId",
source_url: "https://git.pleroma.social/pleroma/flake_id",
homepage_url: "https://git.pleroma.social/pleroma/flake_id",
docs: [
main: "readme",
extras: ["README.md"],
source_url_pattern:
"https://git.pleroma.social/pleroma/flake_id/blob/master/%{path}#L%{line}"
]
]
end
# Run "mix help compile.app" to learn about applications.
def application do
[
extra_applications: [:logger],
mod: {FlakeId.Application, []}
]
end
# Run "mix help deps" to learn about dependencies.
defp deps do
[
{:benchee, "~> 1.0", only: :dev},
{:credo, "~> 1.1.0", only: [:dev, :test], runtime: false},
{:ecto, ">= 2.0.0", optional: true},
{:ex_doc, "~> 0.21", only: :dev, runtime: false},
{:base62, "~> 1.2"}
]
end
+ defp elixirc_paths(:test), do: ["lib", "test/support"]
+ defp elixirc_paths(_), do: ["lib"]
+
defp package do
[
licenses: ["LGPLv3"],
links: %{"GitLab" => "https://git.pleroma.social/pleroma/flake_id"}
]
end
end
diff --git a/test/flake_id/builder_test.exs b/test/flake_id/builder_test.exs
index 6a380cf..500d6b8 100644
--- a/test/flake_id/builder_test.exs
+++ b/test/flake_id/builder_test.exs
@@ -1,40 +1,69 @@
defmodule FlakeId.BuilderTest do
use ExUnit.Case, async: true
alias FlakeId.Builder
describe "get/2" do
test "increment `:sq` when the calling time is the same as the state time" do
- time = Builder.time()
- node = Builder.worker_id()
- state = %Builder{time: time, node: node, sq: 0}
+ builder = Builder.new()
+ time = builder.time
- assert {:ok, <<_::integer-size(128)>>, %Builder{node: ^node, time: ^time, sq: 1}} =
- Builder.get(time, state)
+ assert {:ok, <<_::integer-size(128)>>, %Builder{time: ^time, sq: 1}} =
+ Builder.get(time, builder)
end
test "reset `:sq` when the times are different" do
time = Builder.time()
node = Builder.worker_id()
state = %Builder{time: time - 1, node: node, sq: 42}
assert {:ok, _, %Builder{time: ^time, sq: 0}} = Builder.get(time, state)
end
test "error when clock is running backwards" do
time = Builder.time()
state = %Builder{time: time + 1}
assert Builder.get(time, state) == {:error, :clock_running_backwards}
end
test "error when clock is stuck and `sq` overflows" do
time = Builder.time()
state = %Builder{node: 0, time: time, sq: 65534}
assert {:ok, _, %Builder{sq: sq}} = Builder.get(time, state)
state = %Builder{state | node: 0, sq: sq}
assert Builder.get(time, state) == {:error, :clock_stuck}
end
+ end
+
+ test "uses initial time from TimeHint" do
+ file = "TEST-flake-id-builder-initial-time-hint"
+ on_exit(fn -> File.rm(file) end)
+ FakeClock.set(42)
+
+ {:ok, _} =
+ FlakeId.TimeHint.start_link(name: TestTimeHint, file: file, clock: {FakeClock, :time})
+
+ builder = Builder.new(time_hint: TestTimeHint)
+ assert builder.time == FlakeId.TimeHint.get(TestTimeHint)
+ end
+
+ describe "worker_id" do
+ test "random" do
+ builder1 = Builder.new(worker_id: :random)
+ builder2 = Builder.new(worker_id: :random)
+ refute builder1.node == builder2.node
+ end
+ test "integer" do
+ builder = Builder.new(worker_id: 42)
+ assert builder.node == 42
+ end
+
+ test "binary" do
+ hash = :erlang.phash2("hello")
+ builder = Builder.new(worker_id: "hello")
+ assert builder.node == hash
+ end
end
end
diff --git a/test/flake_id/time_hint_test.exs b/test/flake_id/time_hint_test.exs
new file mode 100644
index 0000000..1f9e3a8
--- /dev/null
+++ b/test/flake_id/time_hint_test.exs
@@ -0,0 +1,93 @@
+defmodule FlakeId.TimeHintTest do
+ use ExUnit.Case, async: false
+
+ @testfile "TEST-flake-id-time-hint"
+
+ setup do
+ File.rm(@testfile)
+ on_exit(fn -> File.rm(@testfile) end)
+ :ok
+ end
+
+ alias FlakeId.TimeHint
+
+ test "start_link/1 default starts if enabled" do
+ Application.put_env(:flake_id, :time_hint, true)
+ Application.put_env(:flake_id, :time_hint_file, @testfile)
+ assert {:ok, _} = TimeHint.start_link([])
+ assert TimeHint.get()
+ end
+
+ test "start_link/1 default doesn't start if disabled" do
+ Application.put_env(:flake_id, :time_hint, false)
+ assert :ignore = TimeHint.start_link([])
+ end
+
+ test "get/0 returns 0 if disabled" do
+ Application.put_env(:flake_id, :time_hint, false)
+ assert 0 = TimeHint.get()
+ end
+
+ test "it doesn't start when file cannot exist" do
+ file = "/path/to/nowhere"
+ assert {:error, _} = TimeHint.start_link(name: NoFile, file: file)
+ end
+
+ test "it saves time in file" do
+ FakeClock.set(42)
+
+ {:ok, _pid} =
+ TimeHint.start_link(name: TestTimeHint, file: @testfile, clock: {FakeClock, :time})
+
+ assert {:ok, binary} = File.read(@testfile)
+ assert {TimeHint, 42} = :erlang.binary_to_term(binary)
+ end
+
+ test "it updates on tick" do
+ {:ok, _} = TimeHint.start_link(name: TestTimeHint, file: @testfile, tick: 200)
+ time1 = TimeHint.get(TestTimeHint)
+ Process.sleep(250)
+ time2 = TimeHint.get(TestTimeHint)
+ assert time2 > time1
+ end
+
+ test "it save file on tick" do
+ {:ok, _} = TimeHint.start_link(name: TestTimeHint, file: @testfile, tick: 200)
+ time1 = TimeHint.get(TestTimeHint)
+ Process.sleep(250)
+ assert {:ok, binary} = File.read(@testfile)
+ assert {TimeHint, time2} = :erlang.binary_to_term(binary)
+ assert time2 > time1
+ end
+
+ test "it save file on terminate" do
+ FakeClock.set(42)
+
+ {:ok, pid} =
+ TimeHint.start_link(name: TestTimeHint, file: @testfile, clock: {FakeClock, :time})
+
+ FakeClock.set(43)
+ send(pid, :stop)
+ Process.sleep(100)
+
+ assert {:ok, binary} = File.read(@testfile)
+ assert {TimeHint, 43} = :erlang.binary_to_term(binary)
+ end
+
+ test "it returns previously saved time if current time is backwards after a restart" do
+ FakeClock.set(42)
+
+ {:ok, pid} =
+ TimeHint.start_link(name: TestTimeHint, file: @testfile, clock: {FakeClock, :time})
+
+ assert TimeHint.get(TestTimeHint) == 42
+ send(pid, :stop)
+
+ FakeClock.set(10)
+
+ {:ok, _pid} =
+ TimeHint.start_link(name: TestTimeHint, file: @testfile, clock: {FakeClock, :time})
+
+ assert TimeHint.get(TestTimeHint) == 42
+ end
+end
diff --git a/test/flake_id_test.exs b/test/flake_id_test.exs
index fa94af5..499b932 100644
--- a/test/flake_id_test.exs
+++ b/test/flake_id_test.exs
@@ -1,78 +1,112 @@
defmodule FlakeIdTest do
use ExUnit.Case, async: true
doctest FlakeId
@flake_string "9n2ciuz1wdesFnrGJU"
@flake_binary <<0, 0, 1, 109, 67, 124, 251, 125, 95, 28, 30, 59, 36, 42, 0, 0>>
@flake_integer 28_939_165_907_792_829_150_732_718_047_232
test "flake_id?/1" do
assert FlakeId.flake_id?(@flake_string)
refute FlakeId.flake_id?("http://example.com/activities/3ebbadd1-eb14-4e20-8118-b6f79c0c7b0b")
refute FlakeId.flake_id?("#cofe")
end
test "get/0" do
assert {:ok, flake} = FlakeId.get()
assert String.valid?(flake)
assert FlakeId.flake_id?(flake)
end
describe "get_local/0" do
test "generates a flake" do
assert {:ok, flake} = FlakeId.get_local()
assert String.valid?(flake)
assert FlakeId.flake_id?(flake)
end
test "worker id is different from get/0, and different between processes" do
{:ok, flake1} = FlakeId.get()
me = self()
- spawn(fn() -> send(me, FlakeId.get_local()) end)
+ spawn(fn -> send(me, FlakeId.get_local()) end)
assert_receive {:ok, flake2}
- spawn(fn() -> send(me, FlakeId.get_local()) end)
+ spawn(fn -> send(me, FlakeId.get_local()) end)
assert_receive {:ok, flake3}
- <<_::integer-size(64), worker_id1::integer-size(48), _::integer-size(16)>> = FlakeId.from_string(flake1)
- <<_::integer-size(64), worker_id2::integer-size(48), _::integer-size(16)>> = FlakeId.from_string(flake2)
- <<_::integer-size(64), worker_id3::integer-size(48), _::integer-size(16)>> = FlakeId.from_string(flake3)
+
+ <<_::integer-size(64), worker_id1::integer-size(48), _::integer-size(16)>> =
+ FlakeId.from_string(flake1)
+
+ <<_::integer-size(64), worker_id2::integer-size(48), _::integer-size(16)>> =
+ FlakeId.from_string(flake2)
+
+ <<_::integer-size(64), worker_id3::integer-size(48), _::integer-size(16)>> =
+ FlakeId.from_string(flake3)
+
refute worker_id1 == worker_id2 == worker_id3
end
end
describe "to_string/1" do
test "with binary" do
assert FlakeId.to_string(@flake_binary) == @flake_string
bin = <<1::integer-size(64), 2::integer-size(48), 3::integer-size(16)>>
assert FlakeId.to_string(bin) == "LygHa16ApeN"
end
test "does nothing with other types" do
assert FlakeId.to_string("cofe") == "cofe"
assert FlakeId.to_string(42) == 42
end
end
describe "from_string/1" do
test "with a flake string" do
assert FlakeId.from_string(@flake_string) == @flake_binary
end
test "with a flake binary" do
assert FlakeId.from_string(@flake_binary) == @flake_binary
end
test "with a non flake string" do
assert FlakeId.from_string("cofe") == nil
end
end
test "to_integer/1" do
assert FlakeId.to_integer(@flake_binary) == @flake_integer
end
test "from_integer/1" do
assert FlakeId.from_integer(@flake_integer) == @flake_binary
end
+
+ test "backdate/2" do
+ at = ~U[2019-12-05 07:15:00.793018Z]
+ unique = :erlang.phash2("unique")
+ timestamp = DateTime.to_unix(at, :millisecond)
+ assert {:ok, flake} = FlakeId.backdate(at, "unique")
+
+ assert <<ts::integer-size(64), worker::integer-size(48), seq::integer-size(16)>> =
+ FlakeId.from_string(flake)
+
+ assert ts == timestamp
+ assert worker == unique
+ assert seq == 0
+ end
+
+ test "notional/1" do
+ at = ~U[2019-12-05 07:15:00.793018Z]
+ timestamp = DateTime.to_unix(at, :millisecond)
+ assert {:ok, flake} = FlakeId.notional(at)
+
+ assert <<ts::integer-size(64), worker::integer-size(48), seq::integer-size(16)>> =
+ FlakeId.from_string(flake)
+
+ assert ts == timestamp
+ assert worker == 0
+ assert seq == 0
+ end
end
diff --git a/test/samples/benchmark.exs b/test/samples/benchmark.exs
index 59b214c..3cdac22 100644
--- a/test/samples/benchmark.exs
+++ b/test/samples/benchmark.exs
@@ -1,27 +1,26 @@
list = Enum.to_list(1..10_000)
fun = fn i -> [i, i * i] end
-setenv = fn(worker_id) ->
+setenv = fn worker_id ->
Application.stop(:flake_id)
Application.put_env(:flake_id, :worker_id, worker_id)
Application.start(:flake_id)
end
Benchee.run(
%{
- "get (worker=node)" => {fn(_) -> FlakeId.get() end,
- before_scenario: fn(_) -> setenv.(:node) end},
- "get (worker=random)" => {fn(_) -> FlakeId.get() end,
- before_scenario: fn(_) -> setenv.(:random) end},
- "get (worker=string)" => {fn(_) -> FlakeId.get() end,
- before_scenario: fn(_) -> setenv.("helloworldhellohello") end},
- "get (worker=integer)" => {fn(_) -> FlakeId.get() end,
- before_scenario: fn(_) -> setenv.(988488949832) end},
+ "get (worker=node)" =>
+ {fn _ -> FlakeId.get() end, before_scenario: fn _ -> setenv.(:node) end},
+ "get (worker=random)" =>
+ {fn _ -> FlakeId.get() end, before_scenario: fn _ -> setenv.(:random) end},
+ "get (worker=string)" =>
+ {fn _ -> FlakeId.get() end, before_scenario: fn _ -> setenv.("helloworldhellohello") end},
+ "get (worker=integer)" =>
+ {fn _ -> FlakeId.get() end, before_scenario: fn _ -> setenv.(988_488_949_832) end},
"get (local)" => fn -> FlakeId.Local.get() end,
"backdate" => fn ->
FlakeId.backdate("some-random-stuff", ~U[2020-03-15 16:32:09.474798Z])
end
},
- [
- ]
+ []
)
diff --git a/test/samples/timebench.exs b/test/samples/timebench.exs
deleted file mode 100644
index 8065fe9..0000000
--- a/test/samples/timebench.exs
+++ /dev/null
@@ -1,16 +0,0 @@
-Benchee.run(%{
- "os:timestamp hack" => fn() ->
- {mega_seconds, seconds, micro_seconds} = :os.timestamp()
- 1_000_000_000 * mega_seconds + seconds * 1000 + :erlang.trunc(micro_seconds / 1000)
- end,
- "os:system_time" => fn() ->
- :os.system_time(:millisecond)
- end,
- "erlang:timestamp hack" => fn() ->
- {mega_seconds, seconds, micro_seconds} = :erlang.timestamp()
- 1_000_000_000 * mega_seconds + seconds * 1000 + :erlang.trunc(micro_seconds / 1000)
- end,
- "datetime to unix" => fn() ->
- DateTime.to_unix(DateTime.utc_now(), :millisecond)
- end
-})
diff --git a/test/support/fake_clock.ex b/test/support/fake_clock.ex
new file mode 100644
index 0000000..3c84038
--- /dev/null
+++ b/test/support/fake_clock.ex
@@ -0,0 +1,4 @@
+defmodule FakeClock do
+ def set(time), do: :persistent_term.put(__MODULE__, time)
+ def time(), do: :persistent_term.get(__MODULE__)
+end
diff --git a/test/test_helper.exs b/test/test_helper.exs
index 869559e..6a0af57 100644
--- a/test/test_helper.exs
+++ b/test/test_helper.exs
@@ -1 +1 @@
-ExUnit.start()
+ExUnit.start(capture_log: true)

File Metadata

Mime Type
text/x-diff
Expires
Sat, Nov 23, 10:06 AM (1 d, 2 h)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
38940
Default Alt Text
(36 KB)

Event Timeline