{"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1238873948", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1238873948, "node_id": "IC_kwDOCGYnMM5J17dc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-07T03:46:26Z", "updated_at": "2022-09-07T03:46:26Z", "author_association": "OWNER", "body": "> Is it still nfortunately slow and tricky when playing with floats ?\r\n\r\nNot sure what you mean here?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229430228", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229430228, "node_id": "IC_kwDOCGYnMM5JR53U", "user": {"value": 4312421, "label": "stonebig"}, "created_at": "2022-08-28T10:43:35Z", "updated_at": "2022-08-28T10:43:35Z", "author_association": "NONE", "body": "Is it still nfortunately slow and tricky when playing with floats ?\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229125114", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229125114, "node_id": "IC_kwDOCGYnMM5JQvX6", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T05:08:58Z", "updated_at": "2022-08-27T05:08:58Z", "author_association": "OWNER", "body": "Testing `bulk --functions`:\r\n\r\n```\r\n% sqlite-utils create-table chickens.db chickens id integer name text name_upper text \r\n% echo 'id,name\r\n1,Blue\r\n2,Snowy\r\n3,Azi\r\n4,Lila\r\n5,Suna\r\n6,Cardi' | sqlite-utils bulk chickens.db '\r\ninsert into chickens (id, name, name_upper) values (:id, :name, myupper(:name))\r\n' - --functions '\r\ndef myupper(s):\r\n return s.upper()\r\n' --csv\r\n% sqlite-utils rows chickens.db chickens\r\n[{\"id\": 1, \"name\": \"Blue\", \"name_upper\": \"BLUE\"},\r\n {\"id\": 2, \"name\": \"Snowy\", \"name_upper\": \"SNOWY\"},\r\n {\"id\": 3, \"name\": \"Azi\", \"name_upper\": \"AZI\"},\r\n {\"id\": 4, \"name\": \"Lila\", \"name_upper\": \"LILA\"},\r\n {\"id\": 5, \"name\": \"Suna\", \"name_upper\": \"SUNA\"},\r\n {\"id\": 6, \"name\": \"Cardi\", \"name_upper\": \"CARDI\"}]\r\n```", "reactions": "{\"total_count\": 2, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229124549", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229124549, "node_id": "IC_kwDOCGYnMM5JQvPF", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T05:03:39Z", "updated_at": "2022-08-27T05:03:39Z", "author_association": "OWNER", "body": "I don't think I need separate documentation for `sqlite-utils memory` and `sqlite-tils bulk` since they work the same, and the `--help` text provides the necessary hints.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229124379", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229124379, "node_id": "IC_kwDOCGYnMM5JQvMb", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T05:02:21Z", "updated_at": "2022-08-27T05:02:21Z", "author_association": "OWNER", "body": "Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#defining-custom-sql-functions", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229120899", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229120899, "node_id": "IC_kwDOCGYnMM5JQuWD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T04:31:35Z", "updated_at": "2022-08-27T04:32:38Z", "author_association": "OWNER", "body": "I should add this `--functions` feature to the `memory` and `bulk` commands too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229120653", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229120653, "node_id": "IC_kwDOCGYnMM5JQuSN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T04:29:49Z", "updated_at": "2022-08-27T04:30:03Z", "author_association": "OWNER", "body": "Found a fix for that!\r\n\r\nI replaced this:\r\n\r\n```python\r\n locals = {}\r\n globals = {}\r\n exec(functions, globals, locals)\r\n # Register all callables in the locals dict:\r\n for name, value in locals.items():\r\n if callable(value):\r\n db.register_function(value, name=name)\r\n```\r\nWith this:\r\n```python\r\n globals = {}\r\n exec(functions, globals)\r\n # Register all callables in the globals dict:\r\n for name, value in globals.items():\r\n if callable(value):\r\n db.register_function(value, name=name)\r\n```\r\nBecause https://docs.python.org/3/library/functions.html#exec says:\r\n\r\n> If only *globals* is provided, it must be a dictionary (and not a subclass of dictionary), which will be used for both the global and the local variables. If *globals* and *locals* are given, they are used for the global and local variables, respectively.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229120104", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229120104, "node_id": "IC_kwDOCGYnMM5JQuJo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T04:25:39Z", "updated_at": "2022-08-27T04:25:39Z", "author_association": "OWNER", "body": "This works:\r\n```\r\nsqlite-utils :memory: 'select extract_domain(\"https://www.google.com/blah\")' --functions '\r\nfrom urllib.parse import urlparse\r\n\r\ndef extract_domain(url):\r\n global urlparse\r\n return urlparse(url).netloc\r\n'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229119999", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229119999, "node_id": "IC_kwDOCGYnMM5JQuH_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T04:24:58Z", "updated_at": "2022-08-27T04:24:58Z", "author_association": "OWNER", "body": "I've encountered this problem before: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-convert-complex\r\n\r\n> ```\r\n> $ sqlite-utils convert content.db articles score '\r\n> import random\r\n> random.seed(10)\r\n> \r\n> def convert(value):\r\n> global random\r\n> return random.random()\r\n> '\r\n> ```\r\n> Note the `global random` line here. Due to the way the tool compiles Python code, this is necessary to ensure the `random` module is available within the `convert()` function. If you were to omit this you would see a `NameError: name 'random' is not defined` error.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229119171", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229119171, "node_id": "IC_kwDOCGYnMM5JQt7D", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T04:18:28Z", "updated_at": "2022-08-27T04:18:28Z", "author_association": "OWNER", "body": "I tried this:\r\n```\r\nsqlite-utils :memory: 'select extract_domain(\"https://www.google.com/blah\")' --functions '\r\nfrom urllib.parse import urlparse\r\n\r\ndef extract_domain(url):\r\n from urllib.parse import urlparse\r\n return urlparse(url).netloc\r\n'\r\n```\r\nAnd got:\r\n```\r\nNameError: name 'urlparse' is not defined\r\nError: user-defined function raised exception\r\n```\r\nBut this worked OK:\r\n```\r\n% sqlite-utils :memory: 'select extract_domain(\"https://www.google.com/blah\")' --functions '\r\ndef extract_domain(url):\r\n from urllib.parse import urlparse\r\n return urlparse(url).netloc\r\n'\r\n[{\"extract_domain(\\\"https://www.google.com/blah\\\")\": \"www.google.com\"}]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229118619", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229118619, "node_id": "IC_kwDOCGYnMM5JQtyb", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T04:14:52Z", "updated_at": "2022-08-27T04:14:52Z", "author_association": "OWNER", "body": "Quick prototype:\r\n```diff\r\ndiff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py\r\nindex 43e76fa..5dee4f6 100644\r\n--- a/sqlite_utils/cli.py\r\n+++ b/sqlite_utils/cli.py\r\n@@ -1633,6 +1633,9 @@ def drop_view(path, view, ignore, load_extension):\r\n type=(str, str),\r\n help=\"Named :parameters for SQL query\",\r\n )\r\n+@click.option(\r\n+ \"--functions\", help=\"Python code defining one or more custom SQL functions\"\r\n+)\r\n @load_extension_option\r\n def query(\r\n path,\r\n@@ -1649,6 +1652,7 @@ def query(\r\n raw,\r\n param,\r\n load_extension,\r\n+ functions,\r\n ):\r\n \"\"\"Execute SQL query and return the results as JSON\r\n \r\n@@ -1665,6 +1669,16 @@ def query(\r\n _load_extensions(db, load_extension)\r\n db.register_fts4_bm25()\r\n \r\n+ # Register any Python functions as SQL functions:\r\n+ if functions:\r\n+ locals = {}\r\n+ globals = {}\r\n+ exec(functions, globals, locals)\r\n+ # Register all callables in the locals dict:\r\n+ for name, value in locals.items():\r\n+ if callable(value):\r\n+ db.register_function(value, name=name)\r\n+\r\n _execute_query(\r\n db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols\r\n )\r\n```\r\nDemo:\r\n```bash\r\n % sqlite-utils :memory: 'select 1 + dog()' --functions '\r\nquote> def dog():\r\nquote> return 2\r\nquote> '\r\n[{\"1 + dog()\": 3}]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229116423", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471", "id": 1229116423, "node_id": "IC_kwDOCGYnMM5JQtQH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-27T04:00:52Z", "updated_at": "2022-08-27T04:00:52Z", "author_association": "OWNER", "body": "Alternative design would be `--function name definition` - like this:\r\n\r\n```\r\nsqlite-utils data.db 'update images set domain = extract_domain(url)' --function extract_domain '\r\n from urllib.parse import urlparse\r\n return urlparse(url).netloc\r\n'\r\n```\r\nI like the `--functions` design better because it leaves space for import statements at the top of the code block, and allows more than one function to be defined in a single go.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1352932716, "label": "sqlite-utils query --functions mechanism for registering extra functions"}, "performed_via_github_app": null}