Page MenuHomePhorge
Files F114847

No OneTemporary
Actions

Size
12 KB
Referenced Files
None
Subscribers
None

View Options

diff --git a/README.md b/README.md
index d89e486..30bab53 100644
--- a/README.md
+++ b/README.md
@@ -1,130 +1,141 @@
# GenMagic
**GenMagic** provides supervised and customisable access to [libmagic](http://man7.org/linux/man-pages/man3/libmagic.3.html) using a supervised external process.
With this library, you can start an one-off process to run a single check, or run the process as a daemon if you expect to run many checks.
## Installation
The package can be installed by adding `gen_magic` to your list of dependencies in `mix.exs`:
```elixir
def deps do
[
{:gen_magic, "~> 1.0.0"}
]
end
```
You must also have [libmagic](http://man7.org/linux/man-pages/man3/libmagic.3.html) installed locally with headers, alongside common compilation tools (i.e. build-essential). These can be acquired by apt-get, yum, brew, etc.
Compilation of the underlying C program is automatic and handled by [elixir_make](https://github.com/elixir-lang/elixir_make).
## Usage
Depending on the use case, you may utilise a single (one-off) GenMagic process without reusing it as a daemon, or utilise a connection pool (such as Poolboy) in your application to run multiple persistent GenMagic processes.
To use GenMagic directly, you can use `GenMagic.Helpers.perform_once/1`:
```elixir
iex(1)> GenMagic.Helpers.perform_once "."
{:ok,
%GenMagic.Result{
content: "directory",
encoding: "binary",
mime_type: "inode/directory"
}}
```
To use the GenMagic server as a daemon, you can start it first, keep a reference, then feed messages to it as you require:
```elixir
{:ok, pid} = GenMagic.Server.start_link([])
{:ok, result} = GenMagic.Server.perform(pid, path)
```
See `GenMagic.Server.start_link/1` and `t:GenMagic.Server.option/0` for more information on startup parameters.
See `GenMagic.Result` for details on the result provided.
## Configuration
When using `GenMagic.Server.start_link/1` to start a persistent server, or `GenMagic.Helpers.perform_once/2` to run an ad-hoc request, you can override specific options to suit your use case.
| Name | Default | Description |
| - | - | - |
| `:startup_timeout` | 1000 | Number of milliseconds to wait for client startup |
| `:process_timeout` | 30000 | Number of milliseconds to process each request |
| `:recycle_threshold` | 10 | Number of cycles before the C process is replaced |
| `:database_patterns` | `[:default]` | Databases to load |
See `t:GenMagic.Server.option/0` for details.
### Use Cases
### Ad-Hoc Requests
For ad-hoc requests, you can use the helper method `GenMagic.Helpers.perform_once/2`:
```elixir
iex(1)> GenMagic.Helpers.perform_once(Path.join(File.cwd!(), "Makefile"))
{:ok,
%GenMagic.Result{
content: "makefile script, ASCII text",
encoding: "us-ascii",
mime_type: "text/x-makefile"
}}
```
### Supervised Requests
-The Server should be run under a pool which provides concurrency *and* resiliency.
+The Server should be run under a supervisor which provides resiliency.
Here we run it under a supervisor:
```elixir
iex(1)> {:ok, pid} = Supervisor.start_link([{GenMagic.Server, name: :gen_magic}], strategy: :one_for_one)
{:ok, #PID<0.199.0>}
```
Now we can ask it to inspect a file:
```elixir
iex(2)> GenMagic.Server.perform(:gen_magic, Path.expand("~/.bash_history"))
{:ok, [mime_type: "text/plain", encoding: "us-ascii", content: "ASCII text"]}
```
Note that in this case we have opted to use a named process.
+### Pool
+
+For concurrency *and* resiliency, you may start the `GenMagic.Pool`. By default, it will start a `GenMagic.Server`
+worker per online scheduler:
+
+```elixir
+iex(1)> GenMagic.Pool.start_link([])
+iex(2)> GenMagic.Pool.perform(GenMagic.Pool, Path.expand("~/.bash_history"))
+{:ok, [mime_type: "text/plain", encoding: "us-ascii", content: "ASCII text"]}
+```
+
### Check Uploaded Files
If you use Phoenix, you can inspect the file from your controller:
```elixir
def upload(conn, %{"upload" => %{path: path}}) do,
{:ok, result} = GenMagic.Helpers.perform_once(:gen_magic, path)
text(conn, "Received your file containing #{result.content}")
end
```
Obviously, it will be more ideal if you have wrapped `GenMagic.Server` in a pool such as Poolboy, to avoid constantly starting and stopping the underlying C program.
## Notes
### Soak Test
Run an endless cycle to prove that the program is resilient:
```bash
find /usr/share/ -name *png | xargs mix run test/soak.exs
find . -name *ex | xargs mix run test/soak.exs
```
## Acknowledgements
During design and prototype development of this library, the Author has drawn inspiration from the following individuals, and therefore thanks all contributors for their generosity:
- Mr [James Every](https://github.com/devstopfix)
- Enhanced Elixir Wrapper (based on GenServer)
- Initial Hex packaging (v.0.22)
- Soak Testing
diff --git a/lib/gen_magic/pool.ex b/lib/gen_magic/pool.ex
new file mode 100644
index 0000000..fac4cd6
--- /dev/null
+++ b/lib/gen_magic/pool.ex
@@ -0,0 +1,55 @@
+defmodule GenMagic.Pool do
+ @behaviour NimblePool
+
+ def start_link(options, pool_size \\ nil) do
+ pool_size = pool_size || System.schedulers_online()
+ NimblePool.start_link(worker: {__MODULE__, options}, pool_size: pool_size)
+ end
+
+ def perform(pool, path, opts \\ []) do
+ pool_timeout = Keyword.get(opts, :pool_timeout, 5000)
+ timeout = Keyword.get(opts, :timeout, 5000)
+
+ NimblePool.checkout!(pool, :checkout, fn _from, server ->
+ {GenMagic.perform(server, path, timeout), server}
+ end, pool_timeout)
+ end
+
+ @impl NimblePool
+ def init_pool(options) do
+ {name, options} = case Keyword.pop(options, :name) do
+ {name, options} when is_atom(name) -> {name, options}
+ {nil, options} -> {__MODULE__, options}
+ {_, options} -> {nil, options}
+ end
+ if name, do: Process.register(self(), atom)
+ {:ok, options}
+ end
+
+ @impl NimblePool
+ def init_worker(options) do
+ {:ok, server} = GenMagic.Server.start_link(options)
+ {:ok, server, nil}
+ end
+
+ @impl NimblePool
+ def handle_checkout(:checkout, _, server, state) do
+ {:ok, server, server, pool_state}
+ end
+
+ @impl NimblePool
+ def handle_checkin(server, _from, _old_server, state) do
+ {:ok, server, state}
+ end
+
+ @impl NimblePool
+ def terminate_worker(_reason, _worker, state) do
+ {:ok, state}
+ end
+
+ @impl NimblePool
+ def terminate(_reason, _conn, state) do
+ {:ok, state}
+ end
+
+end
diff --git a/mix.exs b/mix.exs
index 11ebdaf..83c1597 100644
--- a/mix.exs
+++ b/mix.exs
@@ -1,66 +1,67 @@
defmodule GenMagic.MixProject do
use Mix.Project
if :erlang.system_info(:otp_release) < '21' do
raise "GenMagic requires Erlang/OTP 21 or newer"
end
def project do
[
app: :gen_magic,
version: "1.0.0",
elixir: "~> 1.7",
elixirc_paths: elixirc_paths(Mix.env()),
start_permanent: Mix.env() == :prod,
compilers: [:elixir_make] ++ Mix.compilers(),
package: package(),
deps: deps(),
dialyzer: dialyzer(),
name: "GenMagic",
description: "File introspection with libmagic",
source_url: "https://github.com/evadne/gen_magic",
docs: docs()
]
end
def application do
[extra_applications: [:logger]]
end
defp elixirc_paths(:test), do: ["lib", "test/support"]
defp elixirc_paths(_), do: ["lib"]
defp dialyzer do
[
plt_add_apps: [:mix, :iex, :ex_unit],
flags: ~w(error_handling no_opaque race_conditions underspecs unmatched_returns)a,
ignore_warnings: "dialyzer-ignore-warnings.exs",
list_unused_filters: true
]
end
defp deps do
[
{:credo, "~> 1.4.0", only: [:dev, :test], runtime: false},
{:dialyxir, "~> 1.0.0-rc.6", only: :dev, runtime: false},
{:ex_doc, ">= 0.0.0", only: :dev, runtime: false},
- {:elixir_make, "~> 0.4", runtime: false}
+ {:elixir_make, "~> 0.4", runtime: false},
+ {:nimble_pool, "~> 0.1"}
]
end
defp package do
[
files: ~w(lib/gen_magic/* src/*.c Makefile),
licenses: ["Apache 2.0"],
links: %{"GitHub" => "https://github.com/evadne/packmatic"},
source_url: "https://github.com/evadne/packmatic"
]
end
defp docs do
[
main: "readme",
extras: ["README.md", "CHANGELOG.md"]
]
end
end
diff --git a/mix.lock b/mix.lock
index 3ebe0c6..3317183 100644
--- a/mix.lock
+++ b/mix.lock
@@ -1,15 +1,16 @@
%{
"bunt": {:hex, :bunt, "0.2.0", "951c6e801e8b1d2cbe58ebbd3e616a869061ddadcc4863d0a2182541acae9a38", [:mix], [], "hexpm", "7af5c7e09fe1d40f76c8e4f9dd2be7cebd83909f31fee7cd0e9eadc567da8353"},
"credo": {:hex, :credo, "1.4.0", "92339d4cbadd1e88b5ee43d427b639b68a11071b6f73854e33638e30a0ea11f5", [:mix], [{:bunt, "~> 0.2.0", [hex: :bunt, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm", "1fd3b70dce216574ce3c18bdf510b57e7c4c85c2ec9cad4bff854abaf7e58658"},
"dialyxir": {:hex, :dialyxir, "1.0.0", "6a1fa629f7881a9f5aaf3a78f094b2a51a0357c843871b8bc98824e7342d00a5", [:mix], [{:erlex, ">= 0.2.6", [hex: :erlex, repo: "hexpm", optional: false]}], "hexpm", "aeb06588145fac14ca08d8061a142d52753dbc2cf7f0d00fc1013f53f8654654"},
"earmark": {:hex, :earmark, "1.4.4", "4821b8d05cda507189d51f2caeef370cf1e18ca5d7dfb7d31e9cafe6688106a4", [:mix], [], "hexpm", "1f93aba7340574847c0f609da787f0d79efcab51b044bb6e242cae5aca9d264d"},
"elixir_make": {:hex, :elixir_make, "0.6.0", "38349f3e29aff4864352084fc736fa7fa0f2995a819a737554f7ebd28b85aaab", [:mix], [], "hexpm", "d522695b93b7f0b4c0fcb2dfe73a6b905b1c301226a5a55cb42e5b14d509e050"},
"erlex": {:hex, :erlex, "0.2.6", "c7987d15e899c7a2f34f5420d2a2ea0d659682c06ac607572df55a43753aa12e", [:mix], [], "hexpm", "2ed2e25711feb44d52b17d2780eabf998452f6efda104877a3881c2f8c0c0c75"},
"erlexec": {:hex, :erlexec, "1.10.0", "cba7924cf526097d2082ceb0ec34e7db6bca2624b8f3867fb3fa89c4cf25d227", [:rebar3], [], "hexpm"},
"ex_doc": {:hex, :ex_doc, "0.21.3", "857ec876b35a587c5d9148a2512e952e24c24345552259464b98bfbb883c7b42", [:mix], [{:earmark, "~> 1.4", [hex: :earmark, repo: "hexpm", optional: false]}, {:makeup_elixir, "~> 0.14", [hex: :makeup_elixir, repo: "hexpm", optional: false]}], "hexpm", "0db1ee8d1547ab4877c5b5dffc6604ef9454e189928d5ba8967d4a58a801f161"},
"exexec": {:hex, :exexec, "0.2.0", "a6ffc48cba3ac9420891b847e4dc7120692fb8c08c9e82220ebddc0bb8d96103", [:mix], [{:erlexec, "~> 1.10", [hex: :erlexec, repo: "hexpm", optional: false]}], "hexpm"},
"jason": {:hex, :jason, "1.2.0", "10043418c42d2493d0ee212d3fddd25d7ffe484380afad769a0a38795938e448", [:mix], [{:decimal, "~> 1.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm", "116747dbe057794c3a3e4e143b7c8390b29f634e16c78a7f59ba75bfa6852e7f"},
"makeup": {:hex, :makeup, "1.0.1", "82f332e461dc6c79dbd82fbe2a9c10d48ed07146f0a478286e590c83c52010b5", [:mix], [{:nimble_parsec, "~> 0.5.0", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "49736fe5b66a08d8575bf5321d716bac5da20c8e6b97714fec2bcd6febcfa1f8"},
"makeup_elixir": {:hex, :makeup_elixir, "0.14.0", "cf8b7c66ad1cff4c14679698d532f0b5d45a3968ffbcbfd590339cb57742f1ae", [:mix], [{:makeup, "~> 1.0", [hex: :makeup, repo: "hexpm", optional: false]}], "hexpm", "d4b316c7222a85bbaa2fd7c6e90e37e953257ad196dc229505137c5e505e9eff"},
"nimble_parsec": {:hex, :nimble_parsec, "0.5.3", "def21c10a9ed70ce22754fdeea0810dafd53c2db3219a0cd54cf5526377af1c6", [:mix], [], "hexpm", "589b5af56f4afca65217a1f3eb3fee7e79b09c40c742fddc1c312b3ac0b3399f"},
+ "nimble_pool": {:hex, :nimble_pool, "0.1.0", "ffa9d5be27eee2b00b0c634eb649aa27f97b39186fec3c493716c2a33e784ec6", [:mix], [], "hexpm", "343a1eaa620ddcf3430a83f39f2af499fe2370390d4f785cd475b4df5acaf3f9"},
}
diff --git a/test/gen_magic/pool_test.exs b/test/gen_magic/pool_test.exs
new file mode 100644
index 0000000..f336e12
--- /dev/null
+++ b/test/gen_magic/pool_test.exs
@@ -0,0 +1,9 @@
+defmodule GenMagic.PoollTest do
+ use GenMagic.MagicCase
+
+ test "pool" do
+ {:ok, _} = GenMagic.Pool.start_link([name: TestPool])
+ assert {:ok, _} = GenMagic.Pool.perform(TestPool, absolute_path("Makefile"))
+ end
+
+end

File Metadata

Mime Type
text/x-diff
Expires
Tue, Nov 26, 5:08 PM (1 d, 10 h)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
40440
Default Alt Text
(12 KB)

Event Timeline