{"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869071435", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869071435, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA3MTQzNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:59:26Z", "updated_at": "2021-06-26T22:59:26Z", "author_association": "OWNER", "body": "The other alternative is to finish the work to build a `_metadata` internal table, see #1168. The idea there was that if we want to support efficient pagination and search across the metadata for thousands of attached tables powering it with a plugin hook doesn't work well - we don't want to call the hook once for every one of 1,000+ tables just to implement the homepage.\r\n\r\nSo instead, all metadata for all attached databases would be loaded into an in-memory database called `_metadata`. Plugins that want to modify stored metadata could then do so by directly writing to that table.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/860#issuecomment-869071236", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/860", "id": 869071236, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA3MTIzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:56:28Z", "updated_at": "2021-06-26T22:56:28Z", "author_association": "OWNER", "body": "This work is continuing in #1384.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642651572, "label": "Plugin hook for instance/database/table metadata"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869071167", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869071167, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA3MTE2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:55:36Z", "updated_at": "2021-06-26T22:55:36Z", "author_association": "OWNER", "body": "Just realized I already have an issue open for this, at #860. I'm going to close that and continue work on this in this issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869070941", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869070941, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA3MDk0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:53:34Z", "updated_at": "2021-06-26T22:53:34Z", "author_association": "OWNER", "body": "The `await` thing is worrying me a lot - it feels like this plugin hook is massively less useful if it can't make it's own DB queries and generally do asynchronous stuff - but I'd also like not to break every existing plugin that calls `datasette.metadata(...)`.\r\n\r\nOne solution that could work: introduce a new method, maybe `await datasette.get_metadata(...)`, which uses this plugin hook - and keep the existing `datasette.metadata()` method (which doesn't call the hook) around. This would ensure existing plugins keep on working.\r\n\r\nThen, upgrade those plugins separately - with the goal of deprecating and removing `.metadata()` entirely in Datasette 1.0 - having upgraded the plugins in the meantime.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869070348", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869070348, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA3MDM0OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:46:18Z", "updated_at": "2021-06-26T22:46:18Z", "author_association": "OWNER", "body": "Here's where the plugin hook is called, demonstrating the `fallback=` argument: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/datasette/app.py#L426-L472\r\n\r\nI'm not convinced of the use-case for passing `fallback=` to the hook here - is there a reason a plugin might care whether fallback is `True` or `False`, seeing as the `metadata()` method already respects that fallback logic on line 459?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869070076", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869070076, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA3MDA3Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:42:21Z", "updated_at": "2021-06-26T22:42:21Z", "author_association": "OWNER", "body": "Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.\r\n\r\n@brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869069926", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869069926, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA2OTkyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:40:15Z", "updated_at": "2021-06-26T22:40:53Z", "author_association": "OWNER", "body": "The documentation says:\r\n\r\n> **datasette**: You can use this to access plugin configuration options via `datasette.plugin_config(your_plugin_name)`, or to execute SQL queries.\r\n\r\nThat's not accurate: since the plugin hook is a regular function, not an awaitable, you can't use it to run `await db.execute(...)` so you can't execute SQL queries.\r\n\r\nI can fix this with the await-me-maybe pattern, used for other plugin hooks: https://simonwillison.net/2020/Sep/2/await-me-maybe/\r\n\r\nBUT... that requires changing the `ds.metadata()` function to be awaitable, which will affect every existing plugn that uses that documented internal method!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869069768", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869069768, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA2OTc2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:37:53Z", "updated_at": "2021-06-26T22:37:53Z", "author_association": "OWNER", "body": "The documentation doesn't describe the ``fallback`` argument at the moment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1384#issuecomment-869069655", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1384", "id": 869069655, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA2OTY1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:36:14Z", "updated_at": "2021-06-26T22:37:37Z", "author_association": "OWNER", "body": "Documentation for the new hook is now live at https://docs.datasette.io/en/latest/plugin_hooks.html#get-metadata-datasette-key-database-table-fallback\r\n\r\nLink to the current snapshot of that documentation: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/docs/plugin_hooks.rst#get-metadata-datasette-key-database-table-fallback", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 930807135, "label": "Plugin hook for dynamic metadata"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1368#issuecomment-869068554", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1368", "id": 869068554, "node_id": "MDEyOklzc3VlQ29tbWVudDg2OTA2ODU1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-26T22:23:57Z", "updated_at": "2021-06-26T22:23:57Z", "author_association": "OWNER", "body": "The only test failure is Black. I'm going to merge this and then reformat.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 913865304, "label": "DRAFT: A new plugin hook for dynamic metadata"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/37#issuecomment-868881190", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/37", "id": 868881190, "node_id": "MDEyOklzc3VlQ29tbWVudDg2ODg4MTE5MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-25T23:24:28Z", "updated_at": "2021-06-25T23:24:28Z", "author_association": "OWNER", "body": "Maybe I could release a separate Python package `types-sqlite-utils-numpy` which adds an over-ridden type definition that includes the `numpy` types?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 465815372, "label": "Experiment with type hints"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/37#issuecomment-868881033", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/37", "id": 868881033, "node_id": "MDEyOklzc3VlQ29tbWVudDg2ODg4MTAzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-25T23:23:49Z", "updated_at": "2021-06-25T23:23:49Z", "author_association": "OWNER", "body": "Twitter conversation about how to add types to the `.create_table(columns=)` parameter: https://twitter.com/simonw/status/1408532867592818693\r\n\r\n> Anyone know how to write a mypy type definition for this?\r\n>\r\n> {\"id\": int, \"name\": str, \"image\": bytes, \"weight\": float}\r\n>\r\n> It's a dict where keys are strings and values are one of int/str/bytes/float (weird API design I know, but I designed this long before I was thinking about mypy)\r\n\r\nLooks like this could work:\r\n\r\n```python\r\n def create_table(\r\n self,\r\n name,\r\n columns: Dict[str, Union[Type[int], Type[bytes], Type[str], Type[float]]],\r\n pk=None,\r\n foreign_keys=None,\r\n column_order=None,\r\n not_null=None,\r\n defaults=None,\r\n hash_id=None,\r\n extracts=None,\r\n ):\r\n```\r\nExcept... that method can optionally also accept `numpy` types if `numpy` is installed. I don't know if it's possible to dynamically change a signature based on an import, since `mypy` is a static type analyzer and doesn't ever execute the code.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 465815372, "label": "Experiment with type hints"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868728092", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/293", "id": 868728092, "node_id": "MDEyOklzc3VlQ29tbWVudDg2ODcyODA5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-25T17:39:35Z", "updated_at": "2021-06-25T17:39:35Z", "author_association": "OWNER", "body": "Here's more about this problem: https://github.com/numpy/numpy/issues/15947", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 929748885, "label": "Test against Python 3.10-dev"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868134040", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/293", "id": 868134040, "node_id": "MDEyOklzc3VlQ29tbWVudDg2ODEzNDA0MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-25T01:49:44Z", "updated_at": "2021-06-25T01:50:33Z", "author_association": "OWNER", "body": "Test failed on 3.10 with `numpy` on macOS:\r\n```\r\nsqlite_utils/__init__.py:1: in \r\n11\r\n from .db import Database\r\n12\r\nsqlite_utils/db.py:48: in \r\n13\r\n import numpy as np # type: ignore\r\n14\r\n../../../hostedtoolcache/Python/3.10.0-beta.3/x64/lib/python3.10/site-packages/numpy/__init__.py:391: in \r\n15\r\n raise RuntimeError(msg)\r\n16\r\nE RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, more information is available at https://numpy.org/doc/stable/user/building.html#accelerated-blas-lapack-libraries Otherwise report this to the vendor that provided NumPy.\r\n17\r\nE RankWarning: Polyfit may be poorly conditioned\r\n18\r\nError: Process completed with exit code 4.\r\n```", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 1}", "issue": {"value": 929748885, "label": "Test against Python 3.10-dev"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868125750", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/293", "id": 868125750, "node_id": "MDEyOklzc3VlQ29tbWVudDg2ODEyNTc1MA==", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2021-06-25T01:42:43Z", "updated_at": "2021-06-25T01:42:43Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#293](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ae0f46a) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/747be6057d09a4e5d9d726e29d5cf99b10c59dea?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (747be60) will **not change** coverage.\n> The diff coverage is `n/a`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/293/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n```diff\n@@ Coverage Diff @@\n## main #293 +/- ##\n=======================================\n Coverage 96.03% 96.03% \n=======================================\n Files 4 4 \n Lines 1994 1994 \n=======================================\n Hits 1915 1915 \n Misses 79 79 \n```\n\n\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [747be60...ae0f46a](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 929748885, "label": "Test against Python 3.10-dev"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/290#issuecomment-868021624", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/290", "id": 868021624, "node_id": "MDEyOklzc3VlQ29tbWVudDg2ODAyMTYyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-24T23:17:38Z", "updated_at": "2021-06-24T23:17:38Z", "author_association": "OWNER", "body": "The new documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#executing-queries", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 926777310, "label": "`db.query()` method (renamed `db.execute_returning_dicts()`)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1377#issuecomment-867209791", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1377", "id": 867209791, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NzIwOTc5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-23T22:51:32Z", "updated_at": "2021-06-23T22:51:32Z", "author_association": "OWNER", "body": "Documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#skip-csrf-datasette-scope", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 920884085, "label": "Mechanism for plugins to exclude certain paths from CSRF checks"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1368#issuecomment-867102944", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1368", "id": 867102944, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NzEwMjk0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-23T19:32:01Z", "updated_at": "2021-06-23T19:32:01Z", "author_association": "OWNER", "body": "Yes, let's move ahead with getting this into an alpha.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 913865304, "label": "DRAFT: A new plugin hook for dynamic metadata"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/291#issuecomment-866466388", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/291", "id": 866466388, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NjQ2NjM4OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-23T02:10:24Z", "updated_at": "2021-06-23T02:10:24Z", "author_association": "OWNER", "body": "This already helped me spot a bug in a test: https://github.com/simonw/sqlite-utils/commit/90e211e3e2f36d2ff911ecf1afe4470ff45c7c0d#diff-4e8715c7a425ee52e74b7df4d34efd32e8c92f3e60bd51bc2e1ad5943b82032eL1160-L1161", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 927766296, "label": "Adopt flake8"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/291#issuecomment-866461926", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/291", "id": 866461926, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NjQ2MTkyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-23T01:59:57Z", "updated_at": "2021-06-23T01:59:57Z", "author_association": "OWNER", "body": "That shouldn't be failing: it's complaining about these: https://github.com/simonw/sqlite-utils/blob/02898bf7af4a4e484ecc8ec852d5fee98463277b/tests/test_register_function.py#L56-L67\r\n\r\nBut I added `# noqa: F811` to them.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 927766296, "label": "Adopt flake8"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/289#issuecomment-866241836", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/289", "id": 866241836, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NjI0MTgzNg==", "user": {"value": 857609, "label": "adamchainz"}, "created_at": "2021-06-22T18:44:36Z", "updated_at": "2021-06-22T18:44:36Z", "author_association": "NONE", "body": "Great!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925677191, "label": "Mypy fixes for rows_from_file()"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865511810", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/290", "id": 865511810, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NTUxMTgxMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-22T04:07:34Z", "updated_at": "2021-06-22T18:26:21Z", "author_association": "OWNER", "body": "That documentation section is pretty weak at the moment - here's the whole thing:\r\n\r\n> ### Executing queries\r\n>\r\n> The `db.execute()` and `db.executescript()` methods provide wrappers around `.execute()` and `.executescript()` on the underlying SQLite connection. These wrappers log to the tracer function if one has been registered.\r\n> ```python\r\n> db = Database(memory=True)\r\n> db[\"dogs\"].insert({\"name\": \"Cleo\"})\r\n> db.execute(\"update dogs set name = 'Cleopaws'\")\r\n> ```\r\n> You can pass parameters as an optional second argument, using either a list or a dictionary. These will be correctly quoted and escaped.\r\n> ```python\r\n> # Using ? and a list:\r\n> db.execute(\"update dogs set name = ?\", [\"Cleopaws\"])\r\n> # Or using :name and a dictionary:\r\n> db.execute(\"update dogs set name = :name\", {\"name\": \"Cleopaws\"})\r\n> ```\r\n\r\n- Talks about `.execute()` - I want to talk about `.query()` instead\r\n- Doesn't clarify that `.execute()` returns a `Cursor` - and assumes you know what to do with one\r\n- Doesn't show an example of a `select` query at all\r\n- The \"tracer function\" bit is confusing (should at least link to docs further down)\r\n- For `UPDATE` should show how to access the number of rows modified (probably using `.execute()` there)\r\n\r\nIt does at least cover the two types of parameters, though that could be bulked out.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 926777310, "label": "`db.query()` method (renamed `db.execute_returning_dicts()`)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/37#issuecomment-509681590", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/37", "id": 509681590, "node_id": "MDEyOklzc3VlQ29tbWVudDUwOTY4MTU5MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-07-09T15:07:12Z", "updated_at": "2021-06-22T18:17:53Z", "author_association": "OWNER", "body": "Here's a magic incantation for generating types detected through running the tests with https://github.com/Instagram/MonkeyType\r\n\r\n```\r\npip install pytest-monkeytype\r\npytest --monkeytype-output=./monkeytype.sqlite3\r\nmonkeytype list-modules\r\nmonkeytype apply sqlite_utils.utils\r\nmonkeytype apply sqlite_utils.cli\r\nmonkeytype apply sqlite_utils.db\r\n```\r\n\r\nHere's the result: https://github.com/simonw/sqlite-utils/commit/d18c694fc25b7dd3d76e250c77ddf56d10ddf935", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 465815372, "label": "Experiment with type hints"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/289#issuecomment-866219755", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/289", "id": 866219755, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NjIxOTc1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-22T18:13:26Z", "updated_at": "2021-06-22T18:13:26Z", "author_association": "OWNER", "body": "Thanks @adamchainz - `mypy` now has a foothold on this project (and runs in CI).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925677191, "label": "Mypy fixes for rows_from_file()"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/267#issuecomment-866184260", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/267", "id": 866184260, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NjE4NDI2MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-22T17:26:18Z", "updated_at": "2021-06-22T17:27:27Z", "author_association": "OWNER", "body": "If an`.update()` method doesn't work because it collides with an existing dictionary method a `.pk` property could still be nice:\r\n```python\r\nfor row in db[\"sometable\"].rows:\r\n db[\"sometable\"].update(row.pk, {\"modified\": 1})\r\n```\r\n \r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 915421499, "label": "row.update() or row.pk"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/267#issuecomment-866182655", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/267", "id": 866182655, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NjE4MjY1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-22T17:24:03Z", "updated_at": "2021-06-22T17:24:03Z", "author_association": "OWNER", "body": "I'm re-opening this as a research task because it may be possible to cleanly implement this using a `dict` subclass - some notes on that here: https://treyhunner.com/2019/04/why-you-shouldnt-inherit-from-list-and-dict-in-python/\r\n\r\nSince this would just be for adding methods (and maybe a property for returning the primary keys for a row) the usual disadvantages of subclassing `dict` described in that article shouldn't apply.\r\n\r\nOne catch: dictionaries already have a `.update()` method! So would have to pick another name.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 915421499, "label": "row.update() or row.pk"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865510796", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/290", "id": 865510796, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NTUxMDc5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-22T04:04:40Z", "updated_at": "2021-06-22T04:04:48Z", "author_association": "OWNER", "body": "Still needs documentation, which will involve rewriting the whole [Executing queries](https://sqlite-utils.datasette.io/en/3.11/python-api.html#executing-queries) section.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 926777310, "label": "`db.query()` method (renamed `db.execute_returning_dicts()`)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865497846", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/290", "id": 865497846, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NTQ5Nzg0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-22T03:21:38Z", "updated_at": "2021-06-22T03:21:38Z", "author_association": "OWNER", "body": "The Python docs say: https://docs.python.org/3/library/sqlite3.html\r\n\r\n> To retrieve data after executing a SELECT statement, you can either treat the cursor as an iterator, call the cursor\u2019s `fetchone()` method to retrieve a single matching row, or call `fetchall()` to get a list of the matching rows.\r\n\r\nLooking at the C source code, both `fetchmany()` and `fetchall()` work under the hood by assembling a Python list: https://github.com/python/cpython/blob/be1cb3214d09d4bf0288bc45f3c1f167f67e4514/Modules/_sqlite/cursor.c#L907-L972 - see calls to `PyList_Append()`\r\n\r\nSo it looks like the most efficient way to iterate over a cursor may well be `for row in cursor:` - which I think calls this C function: https://github.com/python/cpython/blob/be1cb3214d09d4bf0288bc45f3c1f167f67e4514/Modules/_sqlite/cursor.c#L813-L876", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 926777310, "label": "`db.query()` method (renamed `db.execute_returning_dicts()`)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865495370", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/290", "id": 865495370, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NTQ5NTM3MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-22T03:14:30Z", "updated_at": "2021-06-22T03:14:30Z", "author_association": "OWNER", "body": "One small problem with the existing method:\r\nhttps://github.com/simonw/sqlite-utils/blob/8cedc6a8b29180e68326f6b76f249d5e39e4b591/sqlite_utils/db.py#L362-L365\r\n\r\nIt returns a full list, but what if the user would rather have a generator they can iterate over without loading the results into memory in one go?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 926777310, "label": "`db.query()` method (renamed `db.execute_returning_dicts()`)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865491922", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/290", "id": 865491922, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NTQ5MTkyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-22T03:05:35Z", "updated_at": "2021-06-22T03:05:35Z", "author_association": "OWNER", "body": "Potential names:\r\n\r\n- `db.query(sql)` - it's weird to have both this and `db.execute()` but it is at least short and memorable\r\n- `db.sql(sql)`\r\n- `db.execute_d(sql)` - ugly\r\n- `db.execute_dicts(sql)` - confusing\r\n- `db.execute_sql(sql)` - easily confused with `db.execute(sql)`\r\n\r\nI think `db.query(sql)` may be the best option here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 926777310, "label": "`db.query()` method (renamed `db.execute_returning_dicts()`)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1368#issuecomment-865204472", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1368", "id": 865204472, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NTIwNDQ3Mg==", "user": {"value": 2670795, "label": "brandonrobertz"}, "created_at": "2021-06-21T17:11:37Z", "updated_at": "2021-06-21T17:11:37Z", "author_association": "CONTRIBUTOR", "body": "If this is a concept ACK then I will move onto fixing the tests (adding new ones) and updating the documentation for the new plugin hook.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 913865304, "label": "DRAFT: A new plugin hook for dynamic metadata"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1368#issuecomment-865160132", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1368", "id": 865160132, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NTE2MDEzMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-21T16:07:06Z", "updated_at": "2021-06-21T16:08:48Z", "author_association": "OWNER", "body": "A few tests failed - Black, the test that checks the docs mention the new hook - the most interesting failing test looks like this one:\r\n```\r\n updated_metadata[\"databases\"][\"fixtures\"][\"queries\"][\"magic_parameters\"][\r\n \"allow\"\r\n ] = (allow if \"query\" in permissions else deny)\r\n> cascade_app_client.ds._metadata = updated_metadata\r\nE AttributeError: can't set attribute\r\n```\r\nFrom https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/tests/test_permissions.py#L439-L467\r\n\r\nThis test is directly manipulating `_metadata` purely for the purposes of simulating different permissions - I think updating it to manipulate `_local_metadata` instead would fix that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 913865304, "label": "DRAFT: A new plugin hook for dynamic metadata"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864621099", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/278", "id": 864621099, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDYyMTA5OQ==", "user": {"value": 601708, "label": "mcint"}, "created_at": "2021-06-20T22:39:57Z", "updated_at": "2021-06-20T22:39:57Z", "author_association": "CONTRIBUTOR", "body": "Fair. I looked into it, it looks like it could be done, but it would be _a bit ugly_. I can upload and link a gist of my exploration. **Click** can parse a first argument while still recognizing it as a sub-command keyword. From there, the program could:\r\n1. ignore it preemptively if it matches a sub-command\r\n2. and/or check if a (db) file exists at the path.\r\n\r\nIt would then also need to set a shared db argument variable.\r\n\r\nClick also makes it easy to parse arguments from environment variables. If you're amenable, I may submit a patch for only that, which would update each sub-command to check for a DB/SQLITE_UTILS_DB environment variable. The goal would be usage that looks like: `DB=./convenient.db sqlite-utils [operation] [args]`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 923697888, "label": "Support db as first parameter before subcommand, or as environment variable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/289#issuecomment-864609271", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/289", "id": 864609271, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDYwOTI3MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-20T20:42:07Z", "updated_at": "2021-06-20T20:42:07Z", "author_association": "OWNER", "body": "Wow, thank you! I didn't know about `typing.cast()`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925677191, "label": "Mypy fixes for rows_from_file()"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/286#issuecomment-864594956", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/286", "id": 864594956, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDU5NDk1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-20T18:38:05Z", "updated_at": "2021-06-20T18:38:05Z", "author_association": "OWNER", "body": "3.10 is out in Homebrew now (they turn that around so fast): https://formulae.brew.sh/formula/sqlite-utils", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925487946, "label": "Add installation instructions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1382#issuecomment-864480051", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1382", "id": 864480051, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQ4MDA1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-20T00:20:06Z", "updated_at": "2021-06-20T00:21:02Z", "author_association": "OWNER", "body": "Yes you can - thanks for pointing this out, I've added a comment to the `install.sh` script in the `datasette-csvs` Glitch project:\r\n\r\n```bash\r\npip3 install -U --no-cache-dir -r requirements.txt --user && \\\r\n mkdir -p .data && \\\r\n rm .data/data.db || true && \\\r\n for f in *.csv\r\n do\r\n # Add --encoding=latin-1 to the following if your CSVs use a different encoding:\r\n sqlite-utils insert .data/data.db ${f%.*} $f --csv\r\n done\r\n```\r\n\r\nSo if you edit that file in your own project and change the line to this:\r\n\r\n sqlite-utils insert .data/data.db ${f%.*} $f --csv --encoding=iso-8859-1\r\n\r\nIt should fix this for you.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925406964, "label": "Datasette with Glitch - is it possible to use CSV with ISO-8859-1 encoding?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-864476167", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 864476167, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQ3NjE2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T23:36:48Z", "updated_at": "2021-06-19T23:36:48Z", "author_association": "OWNER", "body": "Wrote this up on my blog here: https://simonwillison.net/2021/Jun/19/sqlite-utils-memory/ - with a video demo here: https://www.youtube.com/watch?v=OUjd0rkc678", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864419283", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/284", "id": 864419283, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxOTI4Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T15:15:34Z", "updated_at": "2021-06-19T15:15:34Z", "author_association": "OWNER", "body": "I think this code is at fault: https://github.com/simonw/sqlite-utils/blob/5b257949d996fe43dc5d218d4308b88796a90740/sqlite_utils/db.py#L1017-L1023\r\n\r\nIt's using `.pks` which adds `rowid` if it's missing.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925320167, "label": ".transform(types=) turns rowid into a concrete column"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864418795", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/285", "id": 864418795, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxODc5NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T15:11:05Z", "updated_at": "2021-06-19T15:11:14Z", "author_association": "OWNER", "body": "Actually I'm going to go with `use_rowid` instead - because the table doesn't inherently use a rowid itself, but you should use one if you want to query it in a way that gives you back a primary key.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925410305, "label": "Introspection property for telling if a table is a rowid table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864418188", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/285", "id": 864418188, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxODE4OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T15:05:53Z", "updated_at": "2021-06-19T15:05:53Z", "author_association": "OWNER", "body": "```python\r\n @property\r\n def uses_rowid(self):\r\n return not any(column for column in self.columns if column.is_pk)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925410305, "label": "Introspection property for telling if a table is a rowid table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417808", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/285", "id": 864417808, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxNzgwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T15:03:00Z", "updated_at": "2021-06-19T15:03:00Z", "author_association": "OWNER", "body": "I think I like `table.uses_rowid` best - it reads well.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925410305, "label": "Introspection property for telling if a table is a rowid table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417765", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/285", "id": 864417765, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxNzc2NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T15:02:42Z", "updated_at": "2021-06-19T15:02:42Z", "author_association": "OWNER", "body": "Some options:\r\n\r\n- `table.rowid_only`\r\n- `table.rowid_as_pk`\r\n- `table.no_pks`\r\n- `table.no_pk`\r\n- `table.uses_rowid`\r\n- `table.use_rowid`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925410305, "label": "Introspection property for telling if a table is a rowid table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417493", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/285", "id": 864417493, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxNzQ5Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T15:00:43Z", "updated_at": "2021-06-19T15:00:43Z", "author_association": "OWNER", "body": "I have to be careful about the language I use here. Here's the official definition: https://www.sqlite.org/rowidtable.html\r\n\r\n> A \"rowid table\" is any table in an SQLite schema that\r\n> \r\n> - is *not* a [virtual table](https://www.sqlite.org/vtab.html), and\r\n> - is *not* a [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html) table.\r\n> \r\n> Most tables in a typical SQLite database schema are rowid tables.\r\n> \r\n> Rowid tables are distinguished by the fact that they all have a unique, non-NULL, signed 64-bit integer [rowid](https://www.sqlite.org/lang_createtable.html#rowid) that is used as the access key for the data in the underlying [B-tree](https://www.sqlite.org/fileformat2.html#btree) storage engine.\r\n\r\nSo it's not correct to call a table a \"rowid table\" only if it is missing its own primary keys.\r\n\r\nMaybe `table.has_rowid` is the right language to use here? No, that's no good - because tables with their own primary keys usually also have a rowid.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925410305, "label": "Introspection property for telling if a table is a rowid table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417133", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/285", "id": 864417133, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxNzEzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T14:57:36Z", "updated_at": "2021-06-19T14:57:36Z", "author_association": "OWNER", "body": "So the logic is:\r\n```python\r\n[column.name for column in self.columns if column.is_pk]\r\n```\r\nI need to decide on a property name. Existing names are documented here: https://sqlite-utils.datasette.io/en/stable/python-api.html#introspection", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925410305, "label": "Introspection property for telling if a table is a rowid table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417031", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/285", "id": 864417031, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxNzAzMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T14:56:45Z", "updated_at": "2021-06-19T14:56:45Z", "author_association": "OWNER", "body": "```pycon\r\n>>> db = sqlite_utils.Database(memory=True)\r\n>>> db[\"rowid_table\"].insert({\"name\": \"Cleo\"})\r\n\r\n>>> db[\"regular_table\"].insert({\"id\": 1, \"name\": \"Cleo\"}, pk=\"id\")\r\n
\r\n>>> db[\"rowid_table\"].pks\r\n['rowid']\r\n>>> db[\"regular_table\"].pks\r\n['id']\r\n```\r\nBut that's because the `.pks` property hides the difference: https://github.com/simonw/sqlite-utils/blob/dc94f4bb8cfe922bb2f9c89f8f0f29092ea63133/sqlite_utils/db.py#L805-L810\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925410305, "label": "Introspection property for telling if a table is a rowid table"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864416911", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/284", "id": 864416911, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxNjkxMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T14:55:45Z", "updated_at": "2021-06-19T14:55:45Z", "author_association": "OWNER", "body": "https://github.com/simonw/sqlite-utils/blob/dc94f4bb8cfe922bb2f9c89f8f0f29092ea63133/sqlite_utils/db.py#L805-L810\r\n\r\nSo I can indeed detect a `rowid` table by looking for no `is_pk` columns.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925320167, "label": ".transform(types=) turns rowid into a concrete column"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864416785", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/284", "id": 864416785, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxNjc4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T14:54:41Z", "updated_at": "2021-06-19T14:54:41Z", "author_association": "OWNER", "body": "```pycon\r\n>>> db = sqlite_utils.Database(memory=True)\r\n>>> db[\"rowid_table\"].insert({\"name\": \"Cleo\"})\r\n
\r\n>>> db[\"regular_table\"].insert({\"id\": 1, \"name\": \"Cleo\"}, pk=\"id\")\r\n
\r\n>>> db[\"rowid_table\"].pks\r\n['rowid']\r\n>>> db[\"regular_table\"].pks\r\n['id']\r\n```\r\nI think I need an introspection property for working out if a table is a `rowid` table or not.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925320167, "label": ".transform(types=) turns rowid into a concrete column"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/283#issuecomment-864416086", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/283", "id": 864416086, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDQxNjA4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T14:49:06Z", "updated_at": "2021-06-19T14:49:13Z", "author_association": "OWNER", "body": "Once again, this is difficult because of the use of a generator here - `rows_from_file()` only yields rows, so there is no obvious mechanism for it to communicate back to the wrapping code that the detected format was CSV or TSV as opposed to JSON.\r\n\r\nI'm going to change `rows_from_file()` to return a `(generator, detected_format)` tuple.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925319214, "label": "memory: Shouldn't detect types for JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864358951", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/284", "id": 864358951, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDM1ODk1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T05:30:00Z", "updated_at": "2021-06-19T05:30:00Z", "author_association": "OWNER", "body": "If this can be fixed it will be in the `transform_sql()` method.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925320167, "label": ".transform(types=) turns rowid into a concrete column"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864358680", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/284", "id": 864358680, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDM1ODY4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T05:27:13Z", "updated_at": "2021-06-19T05:27:13Z", "author_association": "OWNER", "body": "How easy is it to detect a `rowid` table? Is it as simple as `.pks` returning `None`? If so the documentation should mention that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925320167, "label": ".transform(types=) turns rowid into a concrete column"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864354627", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/282", "id": 864354627, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDM1NDYyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T04:42:03Z", "updated_at": "2021-06-19T04:42:03Z", "author_association": "OWNER", "body": "Demo:\r\n\r\n curl -s 'https://api.github.com/users/simonw/repos?per_page=100' | \\\r\n sqlite-utils memory - 'select sum(size), sum(stargazers_count) from stdin limit 1'\r\n [{\"sum(size)\": 2042547, \"sum(stargazers_count)\": 6769}]\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925305186, "label": "Automatic type detection for CSV data"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864350407", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/282", "id": 864350407, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDM1MDQwNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T03:52:20Z", "updated_at": "2021-06-19T03:52:20Z", "author_association": "OWNER", "body": "I'll have an environment variable for `--detect-types` so users who really want that as the default option can turn it on.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925305186, "label": "Automatic type detection for CSV data"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864349123", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/282", "id": 864349123, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDM0OTEyMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T03:36:54Z", "updated_at": "2021-06-19T03:36:54Z", "author_association": "OWNER", "body": "I may change the default for `sqlite-utils insert` to detect types if I release `sqlite-utils` 4.0, as a backwards-incompatible change.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925305186, "label": "Automatic type detection for CSV data"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/179#issuecomment-864349066", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/179", "id": 864349066, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDM0OTA2Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T03:36:04Z", "updated_at": "2021-06-19T03:36:04Z", "author_association": "OWNER", "body": "This work is going to happen in #282.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 709577625, "label": "sqlite-utils transform/insert --detect-types"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864348954", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/282", "id": 864348954, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDM0ODk1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T03:34:42Z", "updated_at": "2021-06-19T03:35:46Z", "author_association": "OWNER", "body": "I built some prototype code here for something which looks at every row in a CSV import and records the likely types: https://gist.github.com/simonw/465f9356f175d1cf86957947dff501d4\r\n\r\nThis could be used by the command-line tools to figure out what `table.transform(types=...)` method to use at the end.\r\n\r\nThis is a different approach to the pure SQL version I tried building in https://github.com/simonw/sqlite-utils/issues/179 - I think this is a better approach though, it's less prone to weird idiosyncrasies of SQLite types, and it's also easy for us to add on to the existing CSV import code in a way that won't require scanning the data twice.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 925305186, "label": "Automatic type detection for CSV data"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864330508", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/279", "id": 864330508, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDMzMDUwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T00:34:24Z", "updated_at": "2021-06-19T00:34:24Z", "author_association": "OWNER", "body": "Got this working:\r\n\r\n % curl 'https://api.github.com/repos/simonw/datasette/issues' | sqlite-utils memory - 'select id from stdin' ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 924990677, "label": "sqlite-utils memory should handle TSV and JSON in addition to CSV"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864328927", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/279", "id": 864328927, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDMyODkyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-19T00:25:08Z", "updated_at": "2021-06-19T00:25:17Z", "author_association": "OWNER", "body": "I tried writing this function with type hints, but eventually gave up:\r\n```python\r\ndef rows_from_file(\r\n fp: BinaryIO,\r\n format: Optional[Format] = None,\r\n dialect: Optional[Type[csv.Dialect]] = None,\r\n encoding: Optional[str] = None,\r\n) -> Generator[dict, None, None]:\r\n if format == Format.JSON:\r\n decoded = json.load(fp)\r\n if isinstance(decoded, dict):\r\n decoded = [decoded]\r\n if not isinstance(decoded, list):\r\n raise RowsFromFileBadJSON(\"JSON must be a list or a dictionary\")\r\n yield from decoded\r\n elif format == Format.CSV:\r\n decoded_fp = io.TextIOWrapper(fp, encoding=encoding or \"utf-8-sig\")\r\n yield from csv.DictReader(decoded_fp)\r\n elif format == Format.TSV:\r\n yield from rows_from_file(\r\n fp, format=Format.CSV, dialect=csv.excel_tab, encoding=encoding\r\n )\r\n elif format is None:\r\n # Detect the format, then call this recursively\r\n buffered = io.BufferedReader(fp, buffer_size=4096)\r\n first_bytes = buffered.peek(2048).strip()\r\n if first_bytes[0] in (b\"[\", b\"{\"):\r\n # TODO: Detect newline-JSON\r\n yield from rows_from_file(fp, format=Format.JSON)\r\n else:\r\n dialect = csv.Sniffer().sniff(first_bytes.decode(encoding, \"ignore\"))\r\n yield from rows_from_file(\r\n fp, format=Format.CSV, dialect=dialect, encoding=encoding\r\n )\r\n else:\r\n raise RowsFromFileError(\"Bad format\")\r\n```\r\nmypy said:\r\n```\r\nsqlite_utils/utils.py:157: error: Argument 1 to \"BufferedReader\" has incompatible type \"BinaryIO\"; expected \"RawIOBase\"\r\nsqlite_utils/utils.py:163: error: Argument 1 to \"decode\" of \"bytes\" has incompatible type \"Optional[str]\"; expected \"str\"\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 924990677, "label": "sqlite-utils memory should handle TSV and JSON in addition to CSV"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/281#issuecomment-864323438", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/281", "id": 864323438, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDMyMzQzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T23:55:06Z", "updated_at": "2021-06-18T23:55:06Z", "author_association": "OWNER", "body": "The `-:json` idea is flawed: Click thinks that's the syntax for an option called `:json`.\r\n\r\nI'm going to do `stdin:json` - which means you can't open a file called `stdin` - but you could use `cat stdin | sqlite-utils memory stdin:json ...` instead which is an OK workaround.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 924992318, "label": "Mechanism for explicitly stating CSV or JSON or TSV for sqlite-utils memory"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864208476", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/279", "id": 864208476, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDIwODQ3Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T18:30:08Z", "updated_at": "2021-06-18T23:30:19Z", "author_association": "OWNER", "body": "So maybe this is a function which can either be told the format or, if none is provided, it detects one for itself.\r\n```python\r\ndef rows_from_file(fp, format=None):\r\n # ...\r\n yield from rows\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 924990677, "label": "sqlite-utils memory should handle TSV and JSON in addition to CSV"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864207841", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/279", "id": 864207841, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDIwNzg0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T18:28:40Z", "updated_at": "2021-06-18T18:28:46Z", "author_association": "OWNER", "body": "```python\r\ndef detect_format(fp):\r\n # ...\r\n return \"csv\", fp, dialect\r\n # or\r\n return \"json\", fp, parsed_data\r\n # or\r\n return \"json-nl\", fp, docs\r\n```\r\nThe mixed return types here are ugly. In all of these cases what we really want is to return a generator of `{...}` objects. So maybe it returns that instead.\r\n```python\r\ndef filepointer_to_documents(fp):\r\n # ...\r\n yield from documents\r\n```\r\nI can refactor `sqlite-utils insert` to use this new code too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 924990677, "label": "sqlite-utils memory should handle TSV and JSON in addition to CSV"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864206308", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/279", "id": 864206308, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDIwNjMwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T18:25:04Z", "updated_at": "2021-06-18T18:25:04Z", "author_association": "OWNER", "body": "Or... since I'm not using a streaming JSON parser at the moment, if I think something is JSON I can load the entire thing into memory to validate it.\r\n\r\nI still need to detect newline-delimited JSON. For that I can consume the first line of the input to see if it's a valid JSON object, then maybe sniff the second line too?\r\n\r\nThis does mean that if the input is a single line of GIANT JSON it will all be consumed into memory at once, but that's going to happen anyway.\r\n\r\nSo I need a function which, given a file pointer, consumes from it, detects the type, then returns that type AND a file pointer to the beginning of the file again. I can use `io.BufferedReader` for this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 924990677, "label": "sqlite-utils memory should handle TSV and JSON in addition to CSV"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864129273", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/279", "id": 864129273, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDEyOTI3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T15:47:47Z", "updated_at": "2021-06-18T15:47:47Z", "author_association": "OWNER", "body": "Detecting valid JSON is tricky - just because a stream starts with `[` or `{` doesn't mean the entire stream is valid JSON. You need to parse the entire stream to determine that for sure.\r\n\r\nOne way to solve this would be with a custom state machine. Another would be to use the `ijson` streaming parser - annoyingly it throws the same exception class for invalid JSON for different reasons, but the `e.args[0]` for that exception includes human-readable text about the error - if it's anything other than `parse error: premature EOF` then it probably means the JSON was invalid.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 924990677, "label": "sqlite-utils memory should handle TSV and JSON in addition to CSV"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864128489", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/278", "id": 864128489, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDEyODQ4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T15:46:24Z", "updated_at": "2021-06-18T15:46:24Z", "author_association": "OWNER", "body": "A workaround could be to define a bash or zsh alias of some sort.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 923697888, "label": "Support db as first parameter before subcommand, or as environment variable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864126781", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/278", "id": 864126781, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDEyNjc4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T15:43:19Z", "updated_at": "2021-06-18T15:43:19Z", "author_association": "OWNER", "body": "I don't think it's possible to do this without breaking backwards compatibility, unfortunately.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 923697888, "label": "Support db as first parameter before subcommand, or as environment variable"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864103005", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/279", "id": 864103005, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDEwMzAwNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T15:04:15Z", "updated_at": "2021-06-18T15:04:15Z", "author_association": "OWNER", "body": "To detect JSON, check to see if the stream starts with `[` or `{` - maybe do something more sophisticated than that.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 924990677, "label": "sqlite-utils memory should handle TSV and JSON in addition to CSV"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-864101267", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 864101267, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDEwMTI2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T15:01:41Z", "updated_at": "2021-06-18T15:01:41Z", "author_association": "OWNER", "body": "I'll split the remaining work out into separate issues.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/273#issuecomment-864099764", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/273", "id": 864099764, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDA5OTc2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T14:59:27Z", "updated_at": "2021-06-18T14:59:27Z", "author_association": "OWNER", "body": "I'm going to merge this as-is and work on the JSON/TSV support in a separate issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 922099793, "label": "sqlite-utils memory command for directly querying CSV/JSON data"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/277#issuecomment-864092515", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/277", "id": 864092515, "node_id": "MDEyOklzc3VlQ29tbWVudDg2NDA5MjUxNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-18T14:47:57Z", "updated_at": "2021-06-18T14:47:57Z", "author_association": "OWNER", "body": "This is a neat improvement.", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 923612361, "label": "add -h support closes #276"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1378#issuecomment-863230355", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1378", "id": 863230355, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MzIzMDM1NQ==", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2021-06-17T13:16:17Z", "updated_at": "2021-06-17T13:16:17Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#1378](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0c132d1) into [main](https://codecov.io/gh/simonw/datasette/commit/83e9c8bc7585dcc62f200e37c2daefcd669ee05e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (83e9c8b) will **not change** coverage.\n> The diff coverage is `n/a`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1378/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n```diff\n@@ Coverage Diff @@\n## main #1378 +/- ##\n=======================================\n Coverage 91.68% 91.68% \n=======================================\n Files 34 34 \n Lines 4340 4340 \n=======================================\n Hits 3979 3979 \n Misses 361 361 \n```\n\n\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [83e9c8b...0c132d1](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 923910375, "label": "Update pytest-xdist requirement from <2.3,>=2.2.1 to >=2.2.1,<2.4"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/277#issuecomment-863205049", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/277", "id": 863205049, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MzIwNTA0OQ==", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2021-06-17T12:40:49Z", "updated_at": "2021-06-17T12:40:49Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#277](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (abbd324) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/a19ce1a4d0048d389411cfe11a5dbe4c503720e1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a19ce1a) will **increase** coverage by `0.00%`.\n> The diff coverage is `100.00%`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/277/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n```diff\n@@ Coverage Diff @@\n## main #277 +/- ##\n=======================================\n Coverage 96.06% 96.06% \n=======================================\n Files 4 4 \n Lines 1828 1829 +1 \n=======================================\n+ Hits 1756 1757 +1 \n Misses 72 72 \n```\n\n\n| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [sqlite\\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/277/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `94.03% <100.00%> (+<0.01%)` | :arrow_up: |\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a19ce1a...abbd324](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 923612361, "label": "add -h support closes #276"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862817185", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/273", "id": 862817185, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjgxNzE4NQ==", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2021-06-17T00:15:34Z", "updated_at": "2021-06-17T00:15:34Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> :exclamation: No coverage uploaded for pull request base (`main@78aebb6`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#section-missing-base-commit).\n> The diff coverage is `n/a`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/273/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n```diff\n@@ Coverage Diff @@\n## main #273 +/- ##\n=======================================\n Coverage ? 96.10% \n=======================================\n Files ? 4 \n Lines ? 1873 \n Branches ? 0 \n=======================================\n Hits ? 1800 \n Misses ? 73 \n Partials ? 0 \n```\n\n\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [78aebb6...df7a37b](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 922099793, "label": "sqlite-utils memory command for directly querying CSV/JSON data"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/275#issuecomment-862617165", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/275", "id": 862617165, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjYxNzE2NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T18:34:51Z", "updated_at": "2021-06-16T18:34:51Z", "author_association": "OWNER", "body": "Also use this: https://github.com/simonw/datasette/blob/83e9c8bc7585dcc62f200e37c2daefcd669ee05e/codecov.yml\r\n\r\nAnd add a badge, as seen on https://github.com/simonw/asgi-csrf", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 922955697, "label": "Enable code coverage"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862605436", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/273", "id": 862605436, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjYwNTQzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T18:19:05Z", "updated_at": "2021-06-16T18:19:05Z", "author_association": "OWNER", "body": "`--attach` documentation: https://github.com/simonw/sqlite-utils/blob/192dc2c5b73bd836ab8e2e5fed4b36c6ea02f250/docs/cli.rst#joining-in-memory-data-against-existing-databases-using-attach", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 922099793, "label": "sqlite-utils memory command for directly querying CSV/JSON data"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/270#issuecomment-862574390", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/270", "id": 862574390, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjU3NDM5MA==", "user": {"value": 4068, "label": "frafra"}, "created_at": "2021-06-16T17:34:49Z", "updated_at": "2021-06-16T17:34:49Z", "author_association": "NONE", "body": "Sorry, I got confused because SQLite has a JSON column type, even if it is treated as TEXT, and I though automatic facets were available for JSON arrays stored as JSON only :)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 919314806, "label": "Cannot set type JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/267#issuecomment-862494864", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/267", "id": 862494864, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjQ5NDg2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T15:51:28Z", "updated_at": "2021-06-16T16:26:15Z", "author_association": "OWNER", "body": "I did add a slightly clumsy mechanism recently to help a bit here though: the `pks_and_rows_where()` method: https://sqlite-utils.datasette.io/en/stable/python-api.html#listing-rows-with-their-primary-keys\r\n\r\nMore details in the issue for that feature: #240\r\n\r\nThe idea here is that if you want to call update you need the primary key for the row - so you can do this:\r\n\r\n```python\r\nfor pk, row in db[\"sometable\"].pks_and_rows_where():\r\n db[\"sometable\"].update(pk, {\"modified\": 1}\")\r\n```\r\nThe `pk` may end up as a single value or a tuple depending on if the table has a compound primary key - but you don't need to worry about that if you use this method as it will return the correct primary key value for you.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 915421499, "label": "row.update() or row.pk"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/131#issuecomment-862495803", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/131", "id": 862495803, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjQ5NTgwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T15:52:33Z", "updated_at": "2021-06-16T15:52:33Z", "author_association": "OWNER", "body": "I like `-t` or `--type` for this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 675753042, "label": "sqlite-utils insert: options for column types"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/267#issuecomment-862493179", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/267", "id": 862493179, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjQ5MzE3OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T15:49:13Z", "updated_at": "2021-06-16T15:49:13Z", "author_association": "OWNER", "body": "The big challenge here is that the rows returned by this library aren't objects, they are Python dictionaries - so adding methods to them isn't possible without changing the type that is returned by these methods.\r\n\r\nPart of the philosophy of the library is that it should make it as easy as possible to round-trip between Python dictionaries and SQLite table data, so I don't think adding methods like this is going to fit.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 915421499, "label": "row.update() or row.pk"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/270#issuecomment-862491721", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/270", "id": 862491721, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjQ5MTcyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T15:47:06Z", "updated_at": "2021-06-16T15:47:06Z", "author_association": "OWNER", "body": "SQLite doesn't have a JSON column type - it has JSON processing functions, but they operate against TEXT columns - so there's nothing I can do here unfortunately.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 919314806, "label": "Cannot set type JSON"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862491016", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 862491016, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjQ5MTAxNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T15:46:13Z", "updated_at": "2021-06-16T15:46:13Z", "author_association": "OWNER", "body": "Columns from data imported from CSV in this way is currently treated as `TEXT`, which means numeric sorts and suchlike won't work as people might expect. It would be good to do automatic type detection here, see #179.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862485408", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 862485408, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjQ4NTQwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T15:38:58Z", "updated_at": "2021-06-16T15:39:28Z", "author_association": "OWNER", "body": "Also `sqlite-utils memory` reflects the existing `sqlite-utils :memory:` mechanism, which is a point in its favour.\r\n\r\nAnd it helps emphasize that the file you are querying will be loaded into memory, so probably don't try this against a 1GB CSV file.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862484557", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 862484557, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjQ4NDU1Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T15:37:51Z", "updated_at": "2021-06-16T15:38:34Z", "author_association": "OWNER", "body": "I wonder if there's a better name for this than `sqlite-utils memory`?\r\n\r\n- `sqlite-utils memory hello.csv \"select * from hello\"`\r\n- `sqlite-utils mem hello.csv \"select * from hello\"`\r\n- `sqlite-utils temp hello.csv \"select * from hello\"`\r\n- `sqlite-utils adhoc hello.csv \"select * from hello\"`\r\n- `sqlite-utils scratch hello.csv \"select * from hello\"`\r\n\r\nI think `memory` is best. I don't like the others, except for `scratch` which is OK.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862479704", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 862479704, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjQ3OTcwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T15:31:31Z", "updated_at": "2021-06-16T15:31:31Z", "author_association": "OWNER", "body": "Plus, could I make this change to `sqlite-utils query` without breaking backwards compatibility? Adding a new `sqlite-utils memory` command is completely safe from that perspective.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862478881", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 862478881, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjQ3ODg4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T15:30:24Z", "updated_at": "2021-06-16T15:30:24Z", "author_association": "OWNER", "body": "But... `sqlite-utils my.csv \"select * from my\"` is a much more compelling initial experience than `sqlite-utils memory my.csv \"select * from my\"`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862475685", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 862475685, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjQ3NTY4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T15:26:19Z", "updated_at": "2021-06-16T15:29:38Z", "author_association": "OWNER", "body": "Here's a radical idea: what if I combined `sqlite-utils memory` into `sqlite-utils query`?\r\n\r\nThe trick here would be to detect if the arguments passed on the command-line refer to SQLite databases or if they refer to CSV/JSON data that should be imported into temporary tables.\r\n\r\nDetecting a SQLite database file is actually really easy - they all start with the same binary string:\r\n```pycon\r\n>>> open(\"my.db\", \"rb\").read(100)\r\nb'SQLite format 3\\x00...\r\n```\r\n(Need to carefully check that a CSV file with`SQLite format 3` as the first column name doesn't accidentally get interpreted as a SQLite DB though).\r\n\r\nSo then what would the semantics of `sqlite-utils query` (which is also the default command) be?\r\n\r\n- `sqlite-utils mydb.db \"select * from x\"`\r\n- `sqlite-utils my.csv \"select * from my\"`\r\n- `sqlite-utils mydb.db my.csv \"select * from mydb.x join my on ...\"` - this is where it gets weird. We can't import the CSV data directly into `mpdb.db` - it's suppose to go into the in-memory database - so now we need to start using database aliases like `mydb.x` because we passed at least one other file?\r\n\r\nThe complexity here is definitely in the handling of a combination of SQLite database files and CSV filenames. Also, `sqlite-utils query` doesn't accept multiple filenames at the moment, so that will change.\r\n\r\nI'm not 100% sold on this as being better than having a separate `sqlite-utils memory` command, as seen in #273.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862046009", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/273", "id": 862046009, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjA0NjAwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T05:15:38Z", "updated_at": "2021-06-16T05:15:38Z", "author_association": "OWNER", "body": "I'm going to add a `--encoding` option - it will affect ALL CSV input files, so if you have CSV files with different encodings you'll need to sort that mess out yourself (likely by importing each CSV file separately into a database using `sqlite-utils insert` with different `--encoding` values).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 922099793, "label": "sqlite-utils memory command for directly querying CSV/JSON data"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862045639", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/273", "id": 862045639, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjA0NTYzOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T05:14:38Z", "updated_at": "2021-06-16T05:14:38Z", "author_association": "OWNER", "body": "Can't share much code though since a bunch of that `insert` stuff is specific to that command - showing progress bars, returning errors on illegal option combinations etc.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 922099793, "label": "sqlite-utils memory command for directly querying CSV/JSON data"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862045438", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/273", "id": 862045438, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjA0NTQzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T05:14:00Z", "updated_at": "2021-06-16T05:14:00Z", "author_association": "OWNER", "body": "I should probably refactor the CSV/JSON/loading stuff into a function in `utils.py` in order to share some of the implementation with the existing `sqlite-utils insert` code: https://github.com/simonw/sqlite-utils/blob/287cdcae8908916687f2ecccc87c38549d004ac6/sqlite_utils/cli.py#L691-L734", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 922099793, "label": "sqlite-utils memory command for directly querying CSV/JSON data"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862043974", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/273", "id": 862043974, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjA0Mzk3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T05:10:12Z", "updated_at": "2021-06-16T05:10:12Z", "author_association": "OWNER", "body": "I can stop promoting `:memory:` here and promote `memory` instead:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/c7234cae8336b8525034e8f917d82dd0699abd42/docs/cli.rst#L83-L86", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 922099793, "label": "sqlite-utils memory command for directly querying CSV/JSON data"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862042110", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/273", "id": 862042110, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjA0MjExMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T05:05:51Z", "updated_at": "2021-06-16T05:06:11Z", "author_association": "OWNER", "body": "Initial documentation is here: https://github.com/simonw/sqlite-utils/blob/c7234cae8336b8525034e8f917d82dd0699abd42/docs/cli.rst#running-queries-directly-against-csv-data\r\n\r\nIt only talks about CSV at the moment - needs to be updated to mention JSON too once that is implemented.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 922099793, "label": "sqlite-utils memory command for directly querying CSV/JSON data"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862040971", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 862040971, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjA0MDk3MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T05:02:56Z", "updated_at": "2021-06-16T05:02:56Z", "author_association": "OWNER", "body": "Moving this to a PR.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862040906", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 862040906, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjA0MDkwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T05:02:47Z", "updated_at": "2021-06-16T05:02:47Z", "author_association": "OWNER", "body": "Got a prototype working!\r\n```\r\n % curl -s 'https://fivethirtyeight.datasettes.com/polls/president_approval_polls.csv?_size=max&_stream=1' | sqlite-utils memory - 'select * from t limit 5' --nl \r\n{\"rowid\": \"1\", \"question_id\": \"139304\", \"poll_id\": \"74225\", \"state\": \"\", \"politician_id\": \"11\", \"politician\": \"Donald Trump\", \"pollster_id\": \"568\", \"pollster\": \"YouGov\", \"sponsor_ids\": \"352\", \"sponsors\": \"Economist\", \"display_name\": \"YouGov\", \"pollster_rating_id\": \"391\", \"pollster_rating_name\": \"YouGov\", \"fte_grade\": \"B\", \"sample_size\": \"1500\", \"population\": \"a\", \"population_full\": \"a\", \"methodology\": \"Online\", \"start_date\": \"1/16/21\", \"end_date\": \"1/19/21\", \"sponsor_candidate\": \"\", \"tracking\": \"\", \"created_at\": \"1/20/21 10:18\", \"notes\": \"\", \"url\": \"https://docs.cdn.yougov.com/y9zsit5bzd/weeklytrackingreport.pdf\", \"source\": \"538\", \"yes\": \"42.0\", \"no\": \"53.0\"}\r\n{\"rowid\": \"2\", \"question_id\": \"139305\", \"poll_id\": \"74225\", \"state\": \"\", \"politician_id\": \"11\", \"politician\": \"Donald Trump\", \"pollster_id\": \"568\", \"pollster\": \"YouGov\", \"sponsor_ids\": \"352\", \"sponsors\": \"Economist\", \"display_name\": \"YouGov\", \"pollster_rating_id\": \"391\", \"pollster_rating_name\": \"YouGov\", \"fte_grade\": \"B\", \"sample_size\": \"1155\", \"population\": \"rv\", \"population_full\": \"rv\", \"methodology\": \"Online\", \"start_date\": \"1/16/21\", \"end_date\": \"1/19/21\", \"sponsor_candidate\": \"\", \"tracking\": \"\", \"created_at\": \"1/20/21 10:18\", \"notes\": \"\", \"url\": \"https://docs.cdn.yougov.com/y9zsit5bzd/weeklytrackingreport.pdf\", \"source\": \"538\", \"yes\": \"44.0\", \"no\": \"55.0\"}\r\n{\"rowid\": \"3\", \"question_id\": \"139306\", \"poll_id\": \"74226\", \"state\": \"\", \"politician_id\": \"11\", \"politician\": \"Donald Trump\", \"pollster_id\": \"23\", \"pollster\": \"American Research Group\", \"sponsor_ids\": \"\", \"sponsors\": \"\", \"display_name\": \"American Research Group\", \"pollster_rating_id\": \"9\", \"pollster_rating_name\": \"American Research Group\", \"fte_grade\": \"B\", \"sample_size\": \"1100\", \"population\": \"a\", \"population_full\": \"a\", \"methodology\": \"Live Phone\", \"start_date\": \"1/16/21\", \"end_date\": \"1/19/21\", \"sponsor_candidate\": \"\", \"tracking\": \"\", \"created_at\": \"1/20/21 10:18\", \"notes\": \"\", \"url\": \"https://americanresearchgroup.com/economy/\", \"source\": \"538\", \"yes\": \"30.0\", \"no\": \"66.0\"}\r\n{\"rowid\": \"4\", \"question_id\": \"139307\", \"poll_id\": \"74226\", \"state\": \"\", \"politician_id\": \"11\", \"politician\": \"Donald Trump\", \"pollster_id\": \"23\", \"pollster\": \"American Research Group\", \"sponsor_ids\": \"\", \"sponsors\": \"\", \"display_name\": \"American Research Group\", \"pollster_rating_id\": \"9\", \"pollster_rating_name\": \"American Research Group\", \"fte_grade\": \"B\", \"sample_size\": \"990\", \"population\": \"rv\", \"population_full\": \"rv\", \"methodology\": \"Live Phone\", \"start_date\": \"1/16/21\", \"end_date\": \"1/19/21\", \"sponsor_candidate\": \"\", \"tracking\": \"\", \"created_at\": \"1/20/21 10:18\", \"notes\": \"\", \"url\": \"https://americanresearchgroup.com/economy/\", \"source\": \"538\", \"yes\": \"29.0\", \"no\": \"67.0\"}\r\n{\"rowid\": \"5\", \"question_id\": \"139298\", \"poll_id\": \"74224\", \"state\": \"\", \"politician_id\": \"11\", \"politician\": \"Donald Trump\", \"pollster_id\": \"1528\", \"pollster\": \"AtlasIntel\", \"sponsor_ids\": \"\", \"sponsors\": \"\", \"display_name\": \"AtlasIntel\", \"pollster_rating_id\": \"546\", \"pollster_rating_name\": \"AtlasIntel\", \"fte_grade\": \"B/C\", \"sample_size\": \"5188\", \"population\": \"a\", \"population_full\": \"a\", \"methodology\": \"Online\", \"start_date\": \"1/15/21\", \"end_date\": \"1/19/21\", \"sponsor_candidate\": \"\", \"tracking\": \"\", \"created_at\": \"1/19/21 21:52\", \"notes\": \"\", \"url\": \"https://projects.fivethirtyeight.com/polls/20210119_US_Atlas2.pdf\", \"source\": \"538\", \"yes\": \"44.6\", \"no\": \"53.9\"}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862018937", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 862018937, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MjAxODkzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T03:59:28Z", "updated_at": "2021-06-16T04:00:05Z", "author_association": "OWNER", "body": "Mainly for debugging purposes it would be useful to be able to save the created in-memory database back to a file again later. This could be done with:\r\n\r\n sqlite-utils memory blah.csv --save saved.db\r\n\r\nCan use `.iterdump()` to implement this: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.iterdump\r\n\r\nMaybe instead (or as-well-as) offer `--dump` which dumps out the SQL from that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861989987", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 861989987, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MTk4OTk4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T02:34:21Z", "updated_at": "2021-06-16T02:34:21Z", "author_association": "OWNER", "body": "The documentation already covers this\r\n```\r\n$ sqlite-utils :memory: \"select sqlite_version()\"\r\n[{\"sqlite_version()\": \"3.29.0\"}]\r\n```\r\nhttps://sqlite-utils.datasette.io/en/latest/cli.html#running-queries-and-returning-json\r\n\r\n`sqlite-utils memory \"select sqlite_version()\"` is a little bit more intuitive than that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861987651", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 861987651, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MTk4NzY1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T02:27:20Z", "updated_at": "2021-06-16T02:27:20Z", "author_association": "OWNER", "body": "Solution: `sqlite-utils memory -` attempts to detect the input based on if it starts with a `{` or `[` (likely JSON) or if it doesn't use the `csv.Sniffer()` mechanism. Or you can use `sqlite-utils memory -:csv` to specifically indicate the type of input.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861985944", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 861985944, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MTk4NTk0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T02:22:52Z", "updated_at": "2021-06-16T02:22:52Z", "author_association": "OWNER", "body": "Another option: allow an optional `:suffix` specifying the type of the file. If this is missing we detect based on the filename.\r\n\r\n sqlite-utils memory somefile:csv \"select * from somefile\"\r\n\r\nOne catch: how to treat `-` for standard input?\r\n\r\n cat blah.csv | sqlite-utils memory - \"select * from stdin\"\r\n\r\nThat's fine for CSV, but what about TSV or JSON or nl-JSON? Maybe this:\r\n\r\n cat blah.csv | sqlite-utils memory -:json \"select * from stdin\"\r\n\r\nBit weird though. The alternative would be to support this:\r\n\r\n cat blah.csv | sqlite-utils memory --load-csv -\r\n\r\nBut that's verbose compared to the version without the long `--load-x` option.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861984707", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 861984707, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MTk4NDcwNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-16T02:19:48Z", "updated_at": "2021-06-16T02:19:48Z", "author_association": "OWNER", "body": "This is going to need to be a separate command, for relatively non-obvious reasons.\r\n\r\n sqlite-utils blah.db \"select * from x\"\r\n\r\nIs equivalent to this, because `query` is the default sub-command:\r\n\r\n sqlite-utils query blah.db \"select * from x\"\r\n\r\nBut... this means that making the filename optional doesn't actually work - because then this is ambiguous:\r\n\r\n sqlite-utils --load-csv blah.csv \"select * from blah\"\r\n\r\nSo instead, I'm going to add a new sub-command. I'm currently thinking `memory` to reflect that this command operates on an in-memory database:\r\n\r\n sqlite-utils memory --load-csv blah.csv \"select * from blah\"\r\n\r\nI still think I need to use `--load-csv` rather than `--csv` because one interesting use-case for this is loading in CSV and converting it to JSON, or vice-versa.\r\n\r\nAnother option: allow multiple arguments which are filenames, and use the extension (or sniff the content) to decide what to do with them:\r\n\r\n sqlite-utils memory blah.csv foo.csv \"select * from foo join blah on ...\"\r\n\r\nThis would require the last positional argument to always be a SQL query, and would treat all other positional arguments as files that should be imported into memory.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861944202", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 861944202, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MTk0NDIwMg==", "user": {"value": 25778, "label": "eyeseast"}, "created_at": "2021-06-16T01:41:03Z", "updated_at": "2021-06-16T01:41:03Z", "author_association": "CONTRIBUTOR", "body": "So, I do things like this a lot, too. I like the idea of piping in from stdin. Something like this would be nice to do in a makefile:\r\n\r\n```sh\r\ncat file.csv | sqlite-utils --csv --table data - 'SELECT * FROM data WHERE col=\"whatever\"' > filtered.csv\r\n```\r\n\r\nIf you assumed that you're always piping out the same format you're piping in, the option names don't have to change. Depends how much you want to change formats.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891835", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 861891835, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MTg5MTgzNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-15T23:09:31Z", "updated_at": "2021-06-15T23:09:31Z", "author_association": "OWNER", "body": "`--load-csv` and `--load-json` and `--load-nl` and `--load-tsv` are unambiguous.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891693", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 861891693, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MTg5MTY5Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-15T23:09:08Z", "updated_at": "2021-06-15T23:09:08Z", "author_association": "OWNER", "body": "Problem: `--csv` and `--json` and `--nl` are already options for `sqlite-utils query` - need new non-conflicting names.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891272", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272", "id": 861891272, "node_id": "MDEyOklzc3VlQ29tbWVudDg2MTg5MTI3Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-15T23:08:02Z", "updated_at": "2021-06-15T23:08:02Z", "author_association": "OWNER", "body": "`--csv -` should work though, for reading from stdin. The table can be called `stdin`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 921878733, "label": "Idea: import CSV to memory, run SQL, export in a single command"}, "performed_via_github_app": null}