{"html_url": "https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1229438242", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/420", "id": 1229438242, "node_id": "IC_kwDOCGYnMM5JR70i", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-08-28T11:34:21Z", "updated_at": "2022-08-28T11:34:37Z", "author_association": "OWNER", "body": "I found a fix that makes that `global` workaround unnecessary:\r\n\r\n- #472", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1178546862, "label": "Document how to use a `--convert` function that runs initialization code first"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1081047053", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/420", "id": 1081047053, "node_id": "IC_kwDOCGYnMM5Ab3gN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-28T19:22:37Z", "updated_at": "2022-03-28T19:22:37Z", "author_association": "OWNER", "body": "Wrote about this in my weeknotes: https://simonwillison.net/2022/Mar/28/datasette-auth0/#new-features-as-documentation", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1178546862, "label": "Document how to use a `--convert` function that runs initialization code first"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1080141111", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/420", "id": 1080141111, "node_id": "IC_kwDOCGYnMM5AYaU3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-28T03:25:57Z", "updated_at": "2022-03-28T03:54:37Z", "author_association": "OWNER", "body": "So now this should solve your problem:\r\n```\r\necho '[{\"name\": \"notaword\"}, {\"name\": \"word\"}]\r\n' | python3 -m sqlite_utils insert listings.db listings - --convert '\r\nimport enchant\r\nd = enchant.Dict(\"en_US\")\r\n\r\ndef convert(row):\r\n    global d\r\n    row[\"is_dictionary_word\"] = d.check(row[\"name\"])\r\n'\r\n```", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1178546862, "label": "Document how to use a `--convert` function that runs initialization code first"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079404281", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/420", "id": 1079404281, "node_id": "IC_kwDOCGYnMM5AVmb5", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-25T20:19:50Z", "updated_at": "2022-03-25T20:19:50Z", "author_association": "OWNER", "body": "Now documented here: https://sqlite-utils.datasette.io/en/latest/cli.html#using-a-convert-function-to-execute-initialization", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1178546862, "label": "Document how to use a `--convert` function that runs initialization code first"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079384771", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/420", "id": 1079384771, "node_id": "IC_kwDOCGYnMM5AVhrD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-25T19:51:34Z", "updated_at": "2022-03-25T19:53:01Z", "author_association": "OWNER", "body": "This works:\r\n```\r\n% sqlite-utils insert dogs.db dogs dogs.json --convert '\r\nimport random\r\nprint(\"seeding\")\r\nrandom.seed(10)\r\nprint(random.random())\r\n\r\ndef convert(row):\r\n    global random\r\n    print(row)\r\n    row[\"random_score\"] = random.random()\r\n'\r\nseeding\r\n0.5714025946899135\r\n{'id': 1, 'name': 'Cleo'}\r\n{'id': 2, 'name': 'Pancakes'}\r\n{'id': 3, 'name': 'New dog'}\r\n(sqlite-utils) sqlite-utils % sqlite-utils rows dogs.db dogs\r\n[{\"id\": 1, \"name\": \"Cleo\", \"random_score\": 0.4288890546751146},\r\n {\"id\": 2, \"name\": \"Pancakes\", \"random_score\": 0.5780913011344704},\r\n {\"id\": 3, \"name\": \"New dog\", \"random_score\": 0.20609823213950174}]\r\n```\r\nHaving to use `global random` inside the function is frustrating but apparently necessary. https://stackoverflow.com/a/56552138/6083", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1178546862, "label": "Document how to use a `--convert` function that runs initialization code first"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079376283", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/420", "id": 1079376283, "node_id": "IC_kwDOCGYnMM5AVfmb", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-25T19:39:30Z", "updated_at": "2022-03-25T19:43:35Z", "author_association": "OWNER", "body": "Actually this doesn't work as I thought. This demo shows that the initialization code is run once per item, not a single time at the start of the run:\r\n```\r\n% sqlite-utils insert dogs.db dogs dogs.json --convert '\r\nimport random\r\nprint(\"seeding\")\r\nrandom.seed(10)\r\nprint(random.random())\r\n\r\ndef convert(row):\r\n    print(row)\r\n    row[\"random_score\"] = random.random()\r\n'\r\nseeding\r\n0.5714025946899135\r\nseeding\r\n0.5714025946899135\r\nseeding\r\n0.5714025946899135\r\nseeding\r\n0.5714025946899135\r\n```\r\nAlso that `print(row)` line is not being printed anywhere that gets to the console for some reason.\r\n\r\n... my mistake, that happened because I changed this line in order to try to get local imports to work:\r\n```python\r\n    try:\r\n        exec(code, globals, locals)\r\n        return globals[\"convert\"]\r\n    except (AttributeError, SyntaxError, NameError, KeyError, TypeError):\r\n```\r\nIt should be `locals[\"convert\"]`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1178546862, "label": "Document how to use a `--convert` function that runs initialization code first"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079243535", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/420", "id": 1079243535, "node_id": "IC_kwDOCGYnMM5AU_MP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-25T17:25:12Z", "updated_at": "2022-03-25T17:25:12Z", "author_association": "OWNER", "body": "That documentation is split across a few places. This is the only bit that talks about `def convert()` pattern right now:\r\n\r\n- https://sqlite-utils.datasette.io/en/stable/cli.html#converting-data-in-columns\r\n\r\nBut that's for `sqlite-utils convert` - the documentation for `sqlite-utils insert --convert` at https://sqlite-utils.datasette.io/en/stable/cli.html#applying-conversions-while-inserting-data doesn't mention it.\r\n\r\nSince both `sqlite-utils convert` and `sqlite-utils insert --convert` apply the same rules to the code, they should link to a shared explanation in the documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1178546862, "label": "Document how to use a `--convert` function that runs initialization code first"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078343231", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/420", "id": 1078343231, "node_id": "IC_kwDOCGYnMM5ARjY_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-24T21:16:10Z", "updated_at": "2022-03-24T21:17:20Z", "author_association": "OWNER", "body": "Aha! This may be possible already: https://github.com/simonw/sqlite-utils/blob/396f80fcc60da8dd844577114f7920830a2e5403/sqlite_utils/utils.py#L311-L316\r\n\r\nAnd yes, this does indeed work - you can do something like this:\r\n\r\n```\r\necho '{\"name\": \"harry\"}' | sqlite-utils insert db.db people - --convert '\r\nimport time\r\n# Simulate something expensive\r\ntime.sleep(1)\r\n\r\ndef convert(row):\r\n    row[\"upper\"] = row[\"name\"].upper()\r\n'\r\n```\r\nAnd after running that:\r\n```\r\nsqlite-utils dump db.db \r\nBEGIN TRANSACTION;\r\nCREATE TABLE [people] (\r\n   [name] TEXT,\r\n   [upper] TEXT\r\n);\r\nINSERT INTO \"people\" VALUES('harry','HARRY');\r\nCOMMIT;\r\n```\r\nSo this is a documentation issue - there's a trick for it but I didn't know what the trick was!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1178546862, "label": "Document how to use a `--convert` function that runs initialization code first"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078328774", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/420", "id": 1078328774, "node_id": "IC_kwDOCGYnMM5ARf3G", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-24T21:12:33Z", "updated_at": "2022-03-24T21:12:33Z", "author_association": "OWNER", "body": "Here's how the `_compile_code()` mechanism works at the moment: https://github.com/simonw/sqlite-utils/blob/396f80fcc60da8dd844577114f7920830a2e5403/sqlite_utils/utils.py#L308-L342\r\n\r\nAt the end it does this:\r\n```python\r\n    return locals[\"fn\"]\r\n```\r\nSo it's already building and then returning a function.\r\n\r\nThe question is if there's a sensible way to allow people to further customize that function by executing some code first, in a way that's easy to explain.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1178546862, "label": "Document how to use a `--convert` function that runs initialization code first"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078322301", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/420", "id": 1078322301, "node_id": "IC_kwDOCGYnMM5AReR9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-24T21:10:52Z", "updated_at": "2022-03-24T21:10:52Z", "author_association": "OWNER", "body": "I can think of three ways forward:\r\n\r\n- Figure out a pattern that gets that local file import workaround to work\r\n- Add another option such as `--convert-init` that lets you pass code that will be executed once at the start\r\n- Come up with a pattern where the `--convert` code can run some initialization code and then return a function which will be called against each value\r\n\r\nI quite like the idea of that third option - I'm going to prototype it and see if I can work something out.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1178546862, "label": "Document how to use a `--convert` function that runs initialization code first"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078315922", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/420", "id": 1078315922, "node_id": "IC_kwDOCGYnMM5ARcuS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-24T21:09:27Z", "updated_at": "2022-03-24T21:09:27Z", "author_association": "OWNER", "body": "Yeah, this is WAY harder than it should be.\r\n\r\nThere's a clumsy workaround you could use which looks something like this: create a file `my_enchant.py` containing:\r\n\r\n```python\r\nimport enchant\r\nd = enchant.Dict(\"en_US\")\r\n\r\ndef check(word):\r\n    return d.check(word)\r\n```\r\nThen run `sqlite-utils` like this:\r\n\r\n```\r\nPYTHONPATH=. cat items.json | jq '.data' | sqlite-utils insert listings.db listings - --convert 'my_enchant.check(value)' --import my_enchant\r\n```\r\nExcept I tried that and it doesn't work! I don't know the right pattern for getting `--import` to work with modules in the same directory.\r\n\r\nSo yeah, this is definitely a big feature gap.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1178546862, "label": "Document how to use a `--convert` function that runs initialization code first"}, "performed_via_github_app": null}