{"id": 1173023272, "node_id": "I_kwDOCGYnMM5F6uoo", "number": 416, "title": "Options for how `r.parsedate()` should handle invalid dates", "user": {"value": 638427, "label": "mattkiefer"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 11, "created_at": "2022-03-17T23:29:55Z", "updated_at": "2022-05-03T21:36:49Z", "closed_at": "2022-03-21T04:01:39Z", "author_association": "NONE", "pull_request": null, "body": "Exceptions are normal expected behavior when typecasting an invalid format. However, r.parsedate() is really just re-formatting strings and keeping the type as text. So it may be better to print-and-pass on exception so the user can see a complete list of invalid values -- while also allowing for the parser to reformat the remaining valid values. \r\n```\r\nsqlite-utils convert idfpr.db license \"Expiration Date\" \"r.parsedate(value)\"\r\n  [#######-----------------------------]   21%  00:01:57Traceback (most recent call last):\r\n  File \"/usr/local/lib/python3.9/dist-packages/sqlite_utils/db.py\", line 2336, in convert_value\r\n    return fn(v)\r\n  File \"<string>\", line 2, in fn\r\n  File \"/usr/local/lib/python3.9/dist-packages/sqlite_utils/recipes.py\", line 8, in parsedate\r\n    parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat()\r\n  File \"/usr/lib/python3/dist-packages/dateutil/parser/_parser.py\", line 1374, in parse\r\n    return DEFAULTPARSER.parse(timestr, **kwargs)\r\n  File \"/usr/lib/python3/dist-packages/dateutil/parser/_parser.py\", line 652, in parse\r\n    raise ParserError(\"String does not contain a date: %s\", timestr)\r\ndateutil.parser._parser.ParserError: String does not contain a date:   /  /    \r\n```\r\nIn this case, I had just one variation of an invalid date: '  /  /  '. But theoretically there could be many values that would have to be fixed one at a time with the current exception handling. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/416/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 340396247, "node_id": "MDU6SXNzdWUzNDAzOTYyNDc=", "number": 339, "title": "Expose SANIC_RESPONSE_TIMEOUT config option in a sensible way", "user": {"value": 12617395, "label": "bsilverm"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2018-07-11T20:38:06Z", "updated_at": "2022-03-21T22:22:40Z", "closed_at": "2022-03-21T22:22:34Z", "author_association": "NONE", "pull_request": null, "body": "Is it possible to configure the sql_time_limit_ms beyond 60 seconds?  It seems queries are still timing out at 60 seconds when sql_time_limit_ms is set to 180000.  We have a very large data set and often encounter timeouts when testing new queries from the datasette UI.  We are optimizing our database as much as we can, but still may require more than 60 seconds for complex queries.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/339/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 324835838, "node_id": "MDU6SXNzdWUzMjQ4MzU4Mzg=", "number": 276, "title": "Handle spatialite geometry columns better", "user": {"value": 45057, "label": "russss"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 21, "created_at": "2018-05-21T08:46:55Z", "updated_at": "2022-03-21T22:22:20Z", "closed_at": "2022-03-21T22:22:20Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I'd like to see spatialite geometry columns rendered more sensibly - at the moment they come through as well-known-binary unless you use custom SQL, and WKB isn't of much use to anyone on the web.\r\n\r\nIn HTML: they should be shown either as simple lat/long (if it's just a point, for example), or as a sensible placeholder if they're more complex geometries.\r\n\r\nIn JSON: they should be GeoJSON geometries, (which means they can be automatically fed into a leaflet map with no further messing around).\r\n\r\nIn CSV: they should be WKT.\r\n\r\nI briefly wondered if this should go into a plugin, but I suspect it needs hooking in at a deeper level than the plugin architecture will support any time soon.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/276/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1175854982, "node_id": "I_kwDOBm6k_c5GFh-G", "number": 1679, "title": "Research: how much overhead does the n=1 time limit have?", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 11, "created_at": "2022-03-21T19:27:46Z", "updated_at": "2022-03-21T21:55:57Z", "closed_at": "2022-03-21T21:55:56Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/datasette/blob/1a7750eb29fd15dd2eea3b9f6e33028ce441b143/datasette/utils/__init__.py#L181-L200\r\n\r\n```python\r\n@contextmanager\r\ndef sqlite_timelimit(conn, ms):\r\n    deadline = time.perf_counter() + (ms / 1000)\r\n    # n is the number of SQLite virtual machine instructions that will be\r\n    # executed between each check. It's hard to know what to pick here.\r\n    # After some experimentation, I've decided to go with 1000 by default and\r\n    # 1 for time limits that are less than 50ms\r\n    n = 1000\r\n    if ms < 50:\r\n        n = 1\r\n\r\n    def handler():\r\n        if time.perf_counter() >= deadline:\r\n            return 1\r\n\r\n    conn.set_progress_handler(handler, n)\r\n    try:\r\n        yield\r\n    finally:\r\n        conn.set_progress_handler(None, n)\r\n```\r\nHow often do I set a time limit of 50 or less? How much slower does it go thanks to this code?", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1679/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1170144879, "node_id": "I_kwDOBm6k_c5Fvv5v", "number": 1660, "title": "Refactor and simplify Datasette routing and views", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 8, "created_at": "2022-03-15T19:56:56Z", "updated_at": "2022-03-21T19:19:12Z", "closed_at": "2022-03-21T19:19:01Z", "author_association": "OWNER", "pull_request": null, "body": "While working on:\n- https://github.com/simonw/datasette/issues/1657\n- https://github.com/simonw/datasette/issues/1439\n\nIt became very clear that the least maintainable part of Datasette at the moment is the way routing to the database, table and row views work - in particular the subclassing mechanism with BaseView and DataView, but also the complex variety of ways in which the URL routes capture different named regular expression groups.", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1660/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1175715988, "node_id": "I_kwDOBm6k_c5GFACU", "number": 1678, "title": "Make `check_visibility()` a documented API", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 1, "created_at": "2022-03-21T17:30:34Z", "updated_at": "2022-03-21T19:04:03Z", "closed_at": "2022-03-21T19:01:46Z", "author_association": "OWNER", "pull_request": null, "body": "Spotted this while working on:\r\n- #1677\r\n\r\nhttps://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/utils/__init__.py#L1005-L1021", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1678/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1175694248, "node_id": "I_kwDOBm6k_c5GE6uo", "number": 1677, "title": "Remove `check_permission()` from `BaseView`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 3268330, "label": "Datasette 1.0"}, "comments": 1, "created_at": "2022-03-21T17:18:18Z", "updated_at": "2022-03-21T18:45:04Z", "closed_at": "2022-03-21T18:45:03Z", "author_association": "OWNER", "pull_request": null, "body": "Follow-on from:\r\n- #1675\r\n\r\nRefs:\r\n- #1660", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1677/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1175648453, "node_id": "I_kwDOBm6k_c5GEvjF", "number": 1675, "title": "Extract out `check_permissions()` from `BaseView", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2022-03-21T16:39:46Z", "updated_at": "2022-03-21T17:14:31Z", "closed_at": "2022-03-21T17:13:21Z", "author_association": "OWNER", "pull_request": null, "body": "> I'm going to refactor this stuff out and document it so it can be easily used by plugins:\r\n\r\nhttps://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L69-L103\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1660#issuecomment-1074136176_", "repo": {"value": 107914493, "label": "datasette"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/datasette/issues/1675/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1091819089, "node_id": "I_kwDOCGYnMM5BE9ZR", "number": 360, "title": "MemoryError", "user": {"value": 559453, "label": "nzaar9"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-01-01T13:39:17Z", "updated_at": "2022-03-21T04:22:46Z", "closed_at": "2022-03-21T04:22:46Z", "author_association": "NONE", "pull_request": null, "body": "HI, when dealing with large json file (~170GB) i got the following error \r\n```\r\nTraceback (most recent call last):\r\n  File \"/usr/local/bin/sqlite-utils\", line 8, in <module>\r\n    sys.exit(cli())\r\n  File \"/usr/lib/python3/dist-packages/click/core.py\", line 1126, in __call__\r\n    return self.main(*args, **kwargs)\r\n  File \"/usr/lib/python3/dist-packages/click/core.py\", line 1051, in main\r\n    rv = self.invoke(ctx)\r\n  File \"/usr/lib/python3/dist-packages/click/core.py\", line 1657, in invoke\r\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n  File \"/usr/lib/python3/dist-packages/click/core.py\", line 1393, in invoke\r\n    return ctx.invoke(self.callback, **ctx.params)\r\n  File \"/usr/lib/python3/dist-packages/click/core.py\", line 752, in invoke\r\n    return __callback(*args, **kwargs)\r\n  File \"/usr/local/lib/python3.9/dist-packages/sqlite_utils/cli.py\", line 1300, in memory\r\n    rows, format_used = rows_from_file(csv_fp, format=format, encoding=encoding)\r\n  File \"/usr/local/lib/python3.9/dist-packages/sqlite_utils/utils.py\", line 185, in rows_from_file\r\n    return rows_from_file(buffered, format=Format.JSON)\r\n  File \"/usr/local/lib/python3.9/dist-packages/sqlite_utils/utils.py\", line 156, in rows_from_file\r\n    decoded = json.load(fp)\r\n  File \"/usr/lib/python3.9/json/__init__.py\", line 293, in load\r\n    return loads(fp.read(),\r\nMemoryError\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/360/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1171599874, "node_id": "I_kwDOCGYnMM5F1TIC", "number": 415, "title": "Convert with `--multi` and `--dry-run` flag does not work", "user": {"value": 3976183, "label": "dotcs"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-03-16T21:59:46Z", "updated_at": "2022-03-21T04:18:24Z", "closed_at": "2022-03-21T04:18:24Z", "author_association": "NONE", "pull_request": null, "body": "It's not possible to combine `--multi` and `--dry-run` flag in the `convert` command.\r\n\r\nLet's first create a simple database from JSON string\r\n\r\n```console\r\n$ echo '[{\"foo\": \"abc\"}]' | sqlite-utils insert demo.db demo -\r\n$ sqlite-utils query demo.db \"SELECT * FROM demo\"             \r\n[{\"foo\": \"abc\"}]\r\n```\r\n\r\nand then try to convert the \"foo\" column with a static value \"bar\" (see docs [Converting a column into multiple columns](https://sqlite-utils.datasette.io/en/stable/cli.html#converting-a-column-into-multiple-columns))\r\n\r\n```console\r\n$ sqlite-utils convert demo.db demo foo '{\"foo\": \"bar\"}' --multi --dry-run\r\nTraceback (most recent call last):\r\n  File \"/home/dotcs/anaconda3/envs/tools/bin/sqlite-utils\", line 8, in <module>\r\n    sys.exit(cli())\r\n  File \"/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py\", line 1128, in __call__\r\n    return self.main(*args, **kwargs)\r\n  File \"/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py\", line 1053, in main\r\n    rv = self.invoke(ctx)\r\n  File \"/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py\", line 1659, in invoke\r\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n  File \"/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py\", line 1395, in invoke\r\n    return ctx.invoke(self.callback, **ctx.params)\r\n  File \"/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py\", line 754, in invoke\r\n    return __callback(*args, **kwargs)\r\n  File \"/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/sqlite_utils/cli.py\", line 2686, in convert\r\n    for row in db.conn.execute(sql, where_args).fetchall():\r\nsqlite3.OperationalError: user-defined function raised exception\r\n```\r\n\r\nBut without the `--dry-run` flag it does work as expected:\r\n\r\n```console\r\n$ sqlite-utils convert demo.db demo foo '{\"foo\": \"bar\"}' --multi\r\n$ sqlite-utils query demo.db \"SELECT * FROM demo\"               \r\n[{\"foo\": \"bar\"}]\r\n```\r\n\r\n```console\r\n$ sqlite-utils --version\r\nsqlite-utils, version 3.25.1\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/415/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}