Page MenuHomePhorge
Files F113354

No OneTemporary
Actions

Size
28 KB
Referenced Files
None
Subscribers
None

View Options

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 58f5624..f753032 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,9 +1,19 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
+## unreleased
+
+* Use time from `os:system_time` which follows OS time changes
+* Errors when the sequence became too big for 16bit integer (when clock is not advancing)
+* Split up build functions in a separate module
+* Local (in-process) generation strategy
+* (Still WIP) Configurable worker-id strategy (random, mac, node name, configurable integer)
+* Backdate
+* (WIP) Possibility to persist a time-hint file to protect against clock going backwards between restarts
+
## [0.1.0] - 2019-09-25
Initial release
diff --git a/README.md b/README.md
index 5bf0245..e2dc261 100644
--- a/README.md
+++ b/README.md
@@ -1,39 +1,104 @@
# ❄ FlakeId
> Decentralized, k-ordered ID generation service
+Flake Ids are 128-bit identifiers, sortable by time, and can be generated safely from a multi-node system without
+coordination.
+
+It consists of a 64 bit timestamp, 48 bit worker identifier, and 16 bit sequence. They are usually
+represented as a Base 62 string; and can be sorted either by bit or by their string representation.
+
+The sort precision is limited to a millisecond, which is probably fine enough for most usages.
+
## Installation
Add `flake_id` to your list of dependencies in `mix.exs`:
```elixir
def deps do
[
{:flake_id, "~> 0.1.0"}
]
end
```
+## Configuration
+
+### Worker ID
+
+The worker ID is a very important part of a flake. It is what guarantees that individual node part of the same system
+can generate IDs without risks of collision, so be sure of picking ones that are guaranteed to be unique for your
+system.
+
+You have multiple strategies available, depending on your setup:
+
+* random byte-string, generated at node boot
+* erlang node name
+* mac address (either random, or manually specified)
+* fixed string or integer
+
+It is configured as `:worker` in the `:flake_id` app environement, and is picked up at boot time.
+
+Worker: `:mac | {:mac, iface} | :node | :random | String.t | Integer.t`
+
+### Persisted Time hint
+
+Flake's time is dependent on your OS system clock. At runtime, Flake will refuse to generate IDs if the time went
+backwards. However, in some cases, after a system reboot, your system clock may be backwards than the last time Flake
+ran.
+
+Enabling a time hint file will prevent this by refusing to start the Flake worker if current time is backwards than the
+last run time.
+
+This configured as `:time_hint` in `:flake_id` app environement. By default, it is disabled (`false`). Possible values
+are `true` in which case the time hint file will be stored in flake_id priv dir, or can be set to a string pointing to a
+file.
+
## Usage
+For most uses, just use `get/0`:
+
```elixir
iex> FlakeId.get()
"9n3171dJZpdD77K3DU"
```
+In complement, you can also generate flakes in-process, without hitting a node-wide worker. This is especially useful
+when you have long-running processes that does a lot of insertions/ID generation. The first call will be slower than the
+subsequent ones, as it have to set-up the Flake state.
+
+Each process would have its Flake state stored in the process dictionary, and the worker-id is derived from the global
+worker id and the current PID:
+
+```elixir
+iex> FlakeId.get_local()
+```
+
+If you wish to insert historical data while preservering sorting, you can achieve this using a backdated flake. In that
+case, you have to provide yourself a UNIQUE worker-id (for the given time). Be very careful at the unique value being
+really unique, otherwise you may generate the same flake twice.
+
+```elixir
+iex> FlakeId.backdate("https://flakesocial.com/status/390929393992", ~U[2019-12-05 07:15:00.793018Z])
+""
+```
+
See [https://hexdocs.pm/flake_id](https://hexdocs.pm/flake_id) for the complete documentation.
### With Ecto
-It is possible to use `FlakeId` with [`Ecto`](https://github.com/elixir-ecto/ecto/). See [`FlakeId.Ecto.Type`](https://hexdocs.pm/flake_id/FlakeId.Ecto.Type.html) documentation for detailed instructions.
+It is possible to use `FlakeId` with [`Ecto`](https://github.com/elixir-ecto/ecto/).
+See [`FlakeId.Ecto.Type`](https://hexdocs.pm/flake_id/FlakeId.Ecto.Type.html) documentation for detailed instructions.
+
+If you wish to migrate from integer serial ids to Flake, see [`FlakeId.Ecto.CompatType`](https://hexdocs.pm/flake_id/FlakeId.Ecto.Type.html) for instructions.
## Prior Art
* [flaky](https://github.com/nirvana/flaky), released under the terms of the Truly Free License,
* [Flake](https://github.com/boundary/flake), Copyright 2012, Boundary, Apache License, Version 2.0
## Copyright and License
Copyright © 2017-2019 [Pleroma Authors](https://pleroma.social/)
FlakeId source code is licensed under the GNU LGPLv3 License.
diff --git a/lib/flake_id.ex b/lib/flake_id.ex
index 8253ee7..197daf4 100644
--- a/lib/flake_id.ex
+++ b/lib/flake_id.ex
@@ -1,100 +1,128 @@
# FlakeId: Decentralized, k-ordered ID generation service
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: LGPL-3.0-only
defmodule FlakeId do
@moduledoc """
Decentralized, k-ordered ID generation service.
"""
import Kernel, except: [to_string: 1]
+ alias FlakeId.Builder
+ alias FlakeId.Worker
+ alias FlakeId.Local
+
+ @type error :: {:error, :clock_running_backwards} | {:error, :clock_stuck} | {:error, :invalid_state}
@doc """
Converts a binary Flake to a String
## Examples
iex> FlakeId.to_string(<<0, 0, 1, 109, 67, 124, 251, 125, 95, 28, 30, 59, 36, 42, 0, 0>>)
"9n2ciuz1wdesFnrGJU"
"""
def to_string(<<_::integer-size(64), _::integer-size(48), _::integer-size(16)>> = binary_flake) do
<<integer::integer-size(128)>> = binary_flake
Base62.encode(integer)
end
def to_string(string), do: string
@doc """
Converts a String to a binary Flake
## Examples
iex> FlakeId.from_string("9n2ciuz1wdesFnrGJU")
<<0, 0, 1, 109, 67, 124, 251, 125, 95, 28, 30, 59, 36, 42, 0, 0>>
"""
@spec from_string(binary()) :: nil | <<_::128>>
def from_string(string)
def from_string(<<_::integer-size(128)>> = flake), do: flake
def from_string(string) when is_binary(string) and byte_size(string) < 18, do: nil
def from_string(string), do: string |> Base62.decode!() |> from_integer
@doc """
Converts a binary Flake to an integer
## Examples
iex> FlakeId.to_integer(<<0, 0, 1, 109, 67, 124, 251, 125, 95, 28, 30, 59, 36, 42, 0, 0>>)
28939165907792829150732718047232
"""
@spec to_integer(<<_::128>>) :: non_neg_integer
def to_integer(binary_flake)
def to_integer(<<integer::integer-size(128)>>), do: integer
@doc """
Converts an integer to a binary Flake
## Examples
iex> FlakeId.from_integer(28939165907792829150732718047232)
<<0, 0, 1, 109, 67, 124, 251, 125, 95, 28, 30, 59, 36, 42, 0, 0>>
"""
@spec from_integer(integer) :: <<_::128>>
def from_integer(integer) do
<<_time::integer-size(64), _node::integer-size(48), _seq::integer-size(16)>> =
<<integer::integer-size(128)>>
end
@doc """
Generates a string with Flake
"""
- @spec get :: String.t()
- def get, do: FlakeId.Worker.get() |> to_string()
+ @spec get :: String.t() | error()
+ def get, do: Worker.get() |> to_string()
+
+ @doc """
+ Generates a local Flake (in-process state). Worker ID will be derived from global worker-id and the process pid.
+
+ First call will be slower but subsequent calls will be faster than worker based `get/0` and is quite useful for long running processes.
+ """
+ @spec get_local :: String.t() | error()
+ def get_local, do: Local.get() |> to_string()
+
+ @doc """
+ Generates a predictable and back-dated Flake.
+
+ This can be useful when you want to insert historical data without messing with sorting.
+
+ Be **very careful** at choosing a `unique` value that is _really unique_ for the given `datetime`, otherwise, collisons will happen.
+ """
+ @spec backdate(any(), DateTime.t()) :: String.t() | error()
+ def backdate(unique, datetime) do
+ hash = :erlang.phash2(unique)
+ time = DateTime.to_unix(datetime, :millisecond)
+ {flake, _} = Builder.get(time, %Builder{node: hash, time: time, sq: 0})
+ to_string(flake)
+ end
@doc """
Checks that ID is a valid FlakeId
## Examples
iex> FlakeId.flake_id?("9n2ciuz1wdesFnrGJU")
true
iex> FlakeId.flake_id?("#cofe")
false
iex> FlakeId.flake_id?("pleroma.social")
false
"""
@spec flake_id?(String.t()) :: boolean
def flake_id?(id), do: flake_id?(String.to_charlist(id), true)
defp flake_id?([c | cs], true) when c >= ?0 and c <= ?9, do: flake_id?(cs, true)
defp flake_id?([c | cs], true) when c >= ?A and c <= ?Z, do: flake_id?(cs, true)
defp flake_id?([c | cs], true) when c >= ?a and c <= ?z, do: flake_id?(cs, true)
defp flake_id?([], true), do: true
defp flake_id?(_, _), do: false
end
diff --git a/lib/flake_id/builder.ex b/lib/flake_id/builder.ex
new file mode 100644
index 0000000..3ab2a5d
--- /dev/null
+++ b/lib/flake_id/builder.ex
@@ -0,0 +1,91 @@
+# FlakeId: Decentralized, k-ordered ID generation service
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: LGPL-3.0-only
+
+defmodule FlakeId.Builder do
+ @moduledoc false
+
+ defstruct node: nil, time: 0, sq: 0
+
+ @type state :: %__MODULE__{
+ node: non_neg_integer,
+ time: non_neg_integer,
+ sq: non_neg_integer
+ }
+
+ def new(worker_id \\ nil) do
+ %__MODULE__{node: worker_id || worker_id(), time: time()}
+ end
+
+ def time do
+ :os.system_time(:millisecond)
+ end
+
+ def worker_id do
+ worker_id(Application.get_env(:flake_id, :worker_id, :random))
+ end
+
+ @spec get(state) :: {<<_::128>>, state} | FlakeId.error()
+ def get(state) do
+ get(time(), state)
+ end
+
+ @spec get(non_neg_integer, state) :: {<<_::128>>, state} | FlakeId.error()
+
+ # Matches when time didn't change and sequence is too big for 16bit int.
+ def get(newtime, %__MODULE__{time: time, sq: sq}) when newtime == time and sq >= 65_535 do
+ {:error, :clock_stuck}
+ end
+
+ # Matches when the calling time is the same as the state time. Incr. sq
+ def get(time, %__MODULE__{node: node, time: time, sq: seq} = state) when is_integer(time) and is_integer(node) and is_integer(seq) do
+ new_state = %__MODULE__{state | sq: state.sq + 1}
+ {gen_flake(new_state), new_state}
+ end
+
+ # Matches when the times are different, reset sq
+ def get(newtime, %__MODULE__{node: node, time: time, sq: seq} = state) when is_integer(time) and is_integer(node) and is_integer(seq) and newtime > time do
+ new_state = %__MODULE__{state | time: newtime, sq: 0}
+ {gen_flake(new_state), new_state}
+ end
+
+ # Error when clock is running backwards
+ def get(newtime, %__MODULE__{time: time}) when newtime < time do
+ {:error, :clock_running_backwards}
+ end
+
+ def get(_, _) do
+ {:error, :invalid_state}
+ end
+
+ @spec gen_flake(state) :: <<_::128>>
+ defp gen_flake(%__MODULE__{time: time, node: node, sq: seq}) do
+ <<time::integer-size(64), node::integer-size(48), seq::integer-size(16)>>
+ end
+
+ defp worker_id(:node) do
+ :erlang.phash2(node())
+ end
+
+ defp worker_id(:mac) do
+ "derp derp derp"
+ end
+
+ defp worker_id(:random) do
+ <<worker::integer-size(48)>> = :crypto.strong_rand_bytes(6)
+ worker
+ end
+
+ defp worker_id(integer) when is_integer(integer) do
+ integer
+ end
+
+ defp worker_id(binary) when is_binary(binary) do
+ #<<worker::integer-size(48)>> = :erlang.phash2(binary)
+ #worker
+ :erlang.phash2(binary)
+ end
+
+
+end
+
diff --git a/lib/flake_id/ecto/compat_type.ex b/lib/flake_id/ecto/compat_type.ex
index 8e6d8a6..4b6105f 100644
--- a/lib/flake_id/ecto/compat_type.ex
+++ b/lib/flake_id/ecto/compat_type.ex
@@ -1,68 +1,80 @@
# FlakeId: Decentralized, k-ordered ID generation service
# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: LGPL-3.0-only
defmodule FlakeId.Ecto.CompatType do
@moduledoc """
Provides a compatibility Ecto type for someone wishes to migrate from integer ids to flakes.
## Schema Example
@primary_key {:id, FlakeId.Ecto.CompatType, autogenerate: true}
schema "posts" do
add :body, :string
end
+ ## Migrating
+
+ The compat type allows to keep your old ids as they used to look before, while the newest ones will be fully fledged flakes.
+
+ This works by altering your ids columns from serial/int/bigint to an UUID, while rewriting the UUID to look like a zero-padded plus the old integer number.
+
+ Such a feat can be accomplished in your migration by using:
+
+ ALTER COLUMN user_id SET DATA TYPE uuid USING CAST( LPAD( TO_HEX(user_id), 32, '0' ) AS uuid)
+
+ A full example of a migration can be found on [Pleroma's code](https://git.pleroma.social/pleroma/pleroma/-/blob/develop/priv/repo/migrations/20181218172826_users_and_activities_flake_id.exs).
+
"""
import Kernel, except: [to_string: 1]
@behaviour Ecto.Type
def embed_as(_), do: :self
def equal?(term1, term2), do: term1 == term2
def type, do: :uuid
defdelegate autogenerate, to: FlakeId, as: :get
def cast(value) do
{:ok, to_string(value)}
end
def load(value) do
{:ok, to_string(value)}
end
def dump(value) do
{:ok, from_string(value)}
end
defp to_string(<<0::integer-size(64), id::integer-size(64)>>), do: Kernel.to_string(id)
defp to_string(binary_flake), do: FlakeId.to_string(binary_flake)
# zero or -1 is a null flake
for i <- [-1, 0] do
defp from_string(unquote(i)), do: <<0::integer-size(128)>>
defp from_string(unquote(Kernel.to_string(i))), do: <<0::integer-size(128)>>
end
defp from_string(int) when is_integer(int) do
int
|> Kernel.to_string()
|> from_string()
end
defp from_string(<<_::integer-size(128)>> = flake), do: flake
defp from_string(string) when is_binary(string) and byte_size(string) < 18 do
case Integer.parse(string) do
{id, ""} -> <<0::integer-size(64), id::integer-size(64)>>
_ -> nil
end
end
defp from_string(string), do: FlakeId.from_string(string)
end
diff --git a/lib/flake_id/local.ex b/lib/flake_id/local.ex
new file mode 100644
index 0000000..2711189
--- /dev/null
+++ b/lib/flake_id/local.ex
@@ -0,0 +1,27 @@
+# FlakeId: Decentralized, k-ordered ID generation service
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: LGPL-3.0-only
+
+defmodule FlakeId.Local do
+ @moduledoc false
+
+ alias FlakeId.Builder
+
+ def get() do
+ case Builder.get(get_state()) do
+ error = {:error, _} -> error
+ {flake, state} ->
+ Process.put(__MODULE__, state)
+ flake
+ end
+ end
+
+ defp get_state() do
+ Process.get(__MODULE__, Builder.new(worker_id()))
+ end
+
+ defp worker_id() do
+ :erlang.phash2({Builder.worker_id(), self()})
+ end
+
+end
diff --git a/lib/flake_id/worker.ex b/lib/flake_id/worker.ex
index c675255..9547c14 100644
--- a/lib/flake_id/worker.ex
+++ b/lib/flake_id/worker.ex
@@ -1,70 +1,30 @@
# FlakeId: Decentralized, k-ordered ID generation service
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: LGPL-3.0-only
defmodule FlakeId.Worker do
@moduledoc false
use GenServer
-
- defstruct node: nil, time: 0, sq: 0
-
- @type state :: %__MODULE__{
- node: non_neg_integer,
- time: non_neg_integer,
- sq: non_neg_integer
- }
+ alias FlakeId.Builder
def start_link(_) do
GenServer.start_link(__MODULE__, [], name: __MODULE__)
end
+ @spec get :: binary
+ def get, do: GenServer.call(__MODULE__, :get)
+
@impl true
- @spec init([]) :: {:ok, state}
+ @spec init([]) :: {:ok, Builder.t}
def init([]) do
- {:ok, %__MODULE__{node: worker_id(), time: time()}}
+ {:ok, Builder.new()}
end
@impl true
def handle_call(:get, _from, state) do
- {flake, new_state} = get(time(), state)
+ {flake, new_state} = Builder.get(state)
{:reply, flake, new_state}
end
- @spec get :: binary
- def get, do: GenServer.call(__MODULE__, :get)
-
- # Matches when the calling time is the same as the state time. Incr. sq
- @spec get(non_neg_integer, state) ::
- {<<_::128>>, state} | {:error, :clock_running_backwards}
- def get(time, %__MODULE__{time: time} = state) do
- new_state = %__MODULE__{state | sq: state.sq + 1}
- {gen_flake(new_state), new_state}
- end
-
- # Matches when the times are different, reset sq
- def get(newtime, %__MODULE__{time: time} = state) when newtime > time do
- new_state = %__MODULE__{state | time: newtime, sq: 0}
- {gen_flake(new_state), new_state}
- end
-
- # Error when clock is running backwards
- def get(newtime, %__MODULE__{time: time}) when newtime < time do
- {:error, :clock_running_backwards}
- end
-
- @spec gen_flake(state) :: <<_::128>>
- def gen_flake(%__MODULE__{time: time, node: node, sq: seq}) do
- <<time::integer-size(64), node::integer-size(48), seq::integer-size(16)>>
- end
-
- def time do
- {mega_seconds, seconds, micro_seconds} = :erlang.timestamp()
- 1_000_000_000 * mega_seconds + seconds * 1000 + :erlang.trunc(micro_seconds / 1000)
- end
-
- def worker_id do
- <<worker::integer-size(48)>> = :crypto.strong_rand_bytes(6)
- worker
- end
end
diff --git a/mix.exs b/mix.exs
index 3438a20..9138539 100644
--- a/mix.exs
+++ b/mix.exs
@@ -1,51 +1,52 @@
defmodule FlakeId.MixProject do
use Mix.Project
def project do
[
app: :flake_id,
version: "0.1.0",
description: "Decentralized, k-ordered ID generation service",
elixir: "~> 1.8",
start_permanent: Mix.env() == :prod,
deps: deps(),
package: package(),
# Docs
name: "FlakeId",
source_url: "https://git.pleroma.social/pleroma/flake_id",
homepage_url: "https://git.pleroma.social/pleroma/flake_id",
docs: [
main: "readme",
extras: ["README.md"],
source_url_pattern:
"https://git.pleroma.social/pleroma/flake_id/blob/master/%{path}#L%{line}"
]
]
end
# Run "mix help compile.app" to learn about applications.
def application do
[
extra_applications: [:logger],
mod: {FlakeId.Application, []}
]
end
# Run "mix help deps" to learn about dependencies.
defp deps do
[
+ {:benchee, "~> 1.0", only: :dev},
{:credo, "~> 1.1.0", only: [:dev, :test], runtime: false},
{:ecto, ">= 2.0.0", optional: true},
{:ex_doc, "~> 0.21", only: :dev, runtime: false},
{:base62, "~> 1.2"}
]
end
defp package do
[
licenses: ["LGPLv3"],
links: %{"GitLab" => "https://git.pleroma.social/pleroma/flake_id"}
]
end
end
diff --git a/mix.lock b/mix.lock
index 3fef1d4..54d7d9b 100644
--- a/mix.lock
+++ b/mix.lock
@@ -1,14 +1,16 @@
%{
"base62": {:hex, :base62, "1.2.1", "4866763e08555a7b3917064e9eef9194c41667276c51b59de2bc42c6ea65f806", [:mix], [{:custom_base, "~> 0.2.1", [hex: :custom_base, repo: "hexpm", optional: false]}], "hexpm"},
+ "benchee": {:hex, :benchee, "1.0.1", "66b211f9bfd84bd97e6d1beaddf8fc2312aaabe192f776e8931cb0c16f53a521", [:mix], [{:deep_merge, "~> 1.0", [hex: :deep_merge, repo: "hexpm", optional: false]}], "hexpm"},
"bunt": {:hex, :bunt, "0.2.0", "951c6e801e8b1d2cbe58ebbd3e616a869061ddadcc4863d0a2182541acae9a38", [:mix], [], "hexpm"},
"credo": {:hex, :credo, "1.1.4", "c2f3b73c895d81d859cec7fcee7ffdb972c595fd8e85ab6f8c2adbf01cf7c29c", [:mix], [{:bunt, "~> 0.2.0", [hex: :bunt, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm"},
"custom_base": {:hex, :custom_base, "0.2.1", "4a832a42ea0552299d81652aa0b1f775d462175293e99dfbe4d7dbaab785a706", [:mix], [], "hexpm"},
"decimal": {:hex, :decimal, "1.8.0", "ca462e0d885f09a1c5a342dbd7c1dcf27ea63548c65a65e67334f4b61803822e", [:mix], [], "hexpm"},
+ "deep_merge": {:hex, :deep_merge, "1.0.0", "b4aa1a0d1acac393bdf38b2291af38cb1d4a52806cf7a4906f718e1feb5ee961", [:mix], [], "hexpm"},
"earmark": {:hex, :earmark, "1.4.0", "397e750b879df18198afc66505ca87ecf6a96645545585899f6185178433cc09", [:mix], [], "hexpm"},
"ecto": {:hex, :ecto, "3.2.1", "a0f9af0fb50b19d3bb6237e512ac0ba56ea222c2bbea92e7c6c94897932c76ba", [:mix], [{:decimal, "~> 1.6", [hex: :decimal, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}], "hexpm"},
"ex_doc": {:hex, :ex_doc, "0.21.2", "caca5bc28ed7b3bdc0b662f8afe2bee1eedb5c3cf7b322feeeb7c6ebbde089d6", [:mix], [{:earmark, "~> 1.3.3 or ~> 1.4", [hex: :earmark, repo: "hexpm", optional: false]}, {:makeup_elixir, "~> 0.14", [hex: :makeup_elixir, repo: "hexpm", optional: false]}], "hexpm"},
"jason": {:hex, :jason, "1.1.2", "b03dedea67a99223a2eaf9f1264ce37154564de899fd3d8b9a21b1a6fd64afe7", [:mix], [{:decimal, "~> 1.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm"},
"makeup": {:hex, :makeup, "1.0.0", "671df94cf5a594b739ce03b0d0316aa64312cee2574b6a44becb83cd90fb05dc", [:mix], [{:nimble_parsec, "~> 0.5.0", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm"},
"makeup_elixir": {:hex, :makeup_elixir, "0.14.0", "cf8b7c66ad1cff4c14679698d532f0b5d45a3968ffbcbfd590339cb57742f1ae", [:mix], [{:makeup, "~> 1.0", [hex: :makeup, repo: "hexpm", optional: false]}], "hexpm"},
"nimble_parsec": {:hex, :nimble_parsec, "0.5.1", "c90796ecee0289dbb5ad16d3ad06f957b0cd1199769641c961cfe0b97db190e0", [:mix], [], "hexpm"},
}
diff --git a/test/flake_id/builder_test.exs b/test/flake_id/builder_test.exs
new file mode 100644
index 0000000..cb8a207
--- /dev/null
+++ b/test/flake_id/builder_test.exs
@@ -0,0 +1,40 @@
+defmodule FlakeId.BuilderTest do
+ use ExUnit.Case, async: true
+
+ alias FlakeId.Builder
+
+ describe "get/2" do
+ test "increment `:sq` when the calling time is the same as the state time" do
+ time = Builder.time()
+ node = Builder.worker_id()
+ state = %Builder{time: time, node: node, sq: 0}
+
+ assert {<<_::integer-size(128)>>, %Builder{node: ^node, time: ^time, sq: 1}} =
+ Builder.get(time, state)
+ end
+
+ test "reset `:sq` when the times are different" do
+ time = Builder.time()
+ node = Builder.worker_id()
+ state = %Builder{time: time - 1, node: node, sq: 42}
+
+ assert {<<_::integer-size(128)>>, %Builder{time: ^time, sq: 0}} = Builder.get(time, state)
+ end
+
+ test "error when clock is running backwards" do
+ time = Builder.time()
+ state = %Builder{time: time + 1}
+
+ assert Builder.get(time, state) == {:error, :clock_running_backwards}
+ end
+
+ test "error when clock not advancing and seq overflows" do
+ time = Builder.time()
+ state = %Builder{node: 0, time: time, sq: 65534}
+ assert {<<_::integer-size(128)>>, %Builder{sq: sq}} = Builder.get(time, state)
+ state = %Builder{state | node: 0, sq: sq}
+ assert Builder.get(time, state) == {:error, :clock_stuck}
+ end
+
+ end
+end
diff --git a/test/flake_id/worker_test.exs b/test/flake_id/worker_test.exs
deleted file mode 100644
index e4d8b6b..0000000
--- a/test/flake_id/worker_test.exs
+++ /dev/null
@@ -1,38 +0,0 @@
-defmodule FlakeId.WorkerTest do
- use ExUnit.Case, async: true
-
- alias FlakeId.Worker
-
- test "get/1" do
- flake = Worker.get()
-
- assert is_binary(flake)
- assert <<_::integer-size(128)>> = flake
- end
-
- describe "get/2" do
- test "increment `:sq` when the calling time is the same as the state time" do
- time = Worker.time()
- node = Worker.worker_id()
- state = %Worker{time: time, node: node, sq: 0}
-
- assert {<<_::integer-size(128)>>, %Worker{node: ^node, time: ^time, sq: 1}} =
- Worker.get(time, state)
- end
-
- test "reset `:sq` when the times are different" do
- time = Worker.time()
- node = Worker.worker_id()
- state = %Worker{time: time - 1, node: node, sq: 42}
-
- assert {<<_::integer-size(128)>>, %Worker{time: ^time, sq: 0}} = Worker.get(time, state)
- end
-
- test "wrror when clock is running backwards" do
- time = Worker.time()
- state = %Worker{time: time + 1}
-
- assert Worker.get(time, state) == {:error, :clock_running_backwards}
- end
- end
-end
diff --git a/test/flake_id_test.exs b/test/flake_id_test.exs
index c796377..43bc475 100644
--- a/test/flake_id_test.exs
+++ b/test/flake_id_test.exs
@@ -1,57 +1,78 @@
defmodule FlakeIdTest do
use ExUnit.Case, async: true
doctest FlakeId
@flake_string "9n2ciuz1wdesFnrGJU"
@flake_binary <<0, 0, 1, 109, 67, 124, 251, 125, 95, 28, 30, 59, 36, 42, 0, 0>>
@flake_integer 28_939_165_907_792_829_150_732_718_047_232
test "flake_id?/1" do
assert FlakeId.flake_id?(@flake_string)
refute FlakeId.flake_id?("http://example.com/activities/3ebbadd1-eb14-4e20-8118-b6f79c0c7b0b")
refute FlakeId.flake_id?("#cofe")
end
test "get/0" do
flake = FlakeId.get()
assert String.valid?(flake)
assert FlakeId.flake_id?(flake)
end
+ describe "get_local/0" do
+ test "generates a flake" do
+ flake = FlakeId.get_local()
+ assert String.valid?(flake)
+ assert FlakeId.flake_id?(flake)
+ end
+
+ test "worker id is different from get/0, and different between processes" do
+ flake1 = FlakeId.get()
+ me = self()
+ spawn(fn() -> send(me, FlakeId.get_local()) end)
+ assert_receive flake2
+ spawn(fn() -> send(me, FlakeId.get_local()) end)
+ assert_receive flake3
+ <<_::integer-size(64), worker_id1::integer-size(48), _::integer-size(16)>> = FlakeId.from_string(flake1)
+ <<_::integer-size(64), worker_id2::integer-size(48), _::integer-size(16)>> = FlakeId.from_string(flake2)
+ <<_::integer-size(64), worker_id3::integer-size(48), _::integer-size(16)>> = FlakeId.from_string(flake3)
+ refute worker_id1 == worker_id2 == worker_id3
+ end
+ end
+
describe "to_string/1" do
test "with binary" do
assert FlakeId.to_string(@flake_binary) == @flake_string
bin = <<1::integer-size(64), 2::integer-size(48), 3::integer-size(16)>>
assert FlakeId.to_string(bin) == "LygHa16ApeN"
end
test "does nothing with other types" do
assert FlakeId.to_string("cofe") == "cofe"
assert FlakeId.to_string(42) == 42
end
end
describe "from_string/1" do
test "with a flake string" do
assert FlakeId.from_string(@flake_string) == @flake_binary
end
test "with a flake binary" do
assert FlakeId.from_string(@flake_binary) == @flake_binary
end
test "with a non flake string" do
assert FlakeId.from_string("cofe") == nil
end
end
test "to_integer/1" do
assert FlakeId.to_integer(@flake_binary) == @flake_integer
end
test "from_integer/1" do
assert FlakeId.from_integer(@flake_integer) == @flake_binary
end
end
diff --git a/test/samples/benchmark.exs b/test/samples/benchmark.exs
new file mode 100644
index 0000000..59b214c
--- /dev/null
+++ b/test/samples/benchmark.exs
@@ -0,0 +1,27 @@
+list = Enum.to_list(1..10_000)
+fun = fn i -> [i, i * i] end
+
+setenv = fn(worker_id) ->
+ Application.stop(:flake_id)
+ Application.put_env(:flake_id, :worker_id, worker_id)
+ Application.start(:flake_id)
+end
+
+Benchee.run(
+ %{
+ "get (worker=node)" => {fn(_) -> FlakeId.get() end,
+ before_scenario: fn(_) -> setenv.(:node) end},
+ "get (worker=random)" => {fn(_) -> FlakeId.get() end,
+ before_scenario: fn(_) -> setenv.(:random) end},
+ "get (worker=string)" => {fn(_) -> FlakeId.get() end,
+ before_scenario: fn(_) -> setenv.("helloworldhellohello") end},
+ "get (worker=integer)" => {fn(_) -> FlakeId.get() end,
+ before_scenario: fn(_) -> setenv.(988488949832) end},
+ "get (local)" => fn -> FlakeId.Local.get() end,
+ "backdate" => fn ->
+ FlakeId.backdate("some-random-stuff", ~U[2020-03-15 16:32:09.474798Z])
+ end
+ },
+ [
+ ]
+)
diff --git a/test/samples/timebench.exs b/test/samples/timebench.exs
new file mode 100644
index 0000000..8065fe9
--- /dev/null
+++ b/test/samples/timebench.exs
@@ -0,0 +1,16 @@
+Benchee.run(%{
+ "os:timestamp hack" => fn() ->
+ {mega_seconds, seconds, micro_seconds} = :os.timestamp()
+ 1_000_000_000 * mega_seconds + seconds * 1000 + :erlang.trunc(micro_seconds / 1000)
+ end,
+ "os:system_time" => fn() ->
+ :os.system_time(:millisecond)
+ end,
+ "erlang:timestamp hack" => fn() ->
+ {mega_seconds, seconds, micro_seconds} = :erlang.timestamp()
+ 1_000_000_000 * mega_seconds + seconds * 1000 + :erlang.trunc(micro_seconds / 1000)
+ end,
+ "datetime to unix" => fn() ->
+ DateTime.to_unix(DateTime.utc_now(), :millisecond)
+ end
+})

File Metadata

Mime Type
text/x-diff
Expires
Mon, Nov 25, 4:33 AM (1 d, 12 h)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
39578
Default Alt Text
(28 KB)

Event Timeline