github
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | pull_request | body | repo | type | active_lock_reason | performed_via_github_app | reactions | draft | state_reason |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1175744654 | I_kwDOCGYnMM5GFHCO | 417 | insert fails on JSONL with whitespace | 9954 | closed | 0 | 3 | 2022-03-21T17:58:14Z | 2022-03-25T21:19:06Z | 2022-03-25T21:17:13Z | NONE | Any JSON that is newline-delimited and has whitespace (newlines) between the start of a JSON object and an attribute fails due to a parse error. e.g. given the valid JSONL: ```{ "attribute": "value" } { "attribute": "value2" } ``` I would expect that `sqlite-utils insert --nl my.db mytable file.jsonl` would properly import the data into `mytable`. However, the following error is thrown instead: `json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 1 (char 2)` It makes sense that since the file is intended to be newline separated, the thing being parsed is "{" (which obviously fails), however the default newline-separated output of `jq` isn't compact. Using `jq -c` avoids this problem, but the fix is unintuitive and undocumented. Proposed solutions: 1. Default to a "loose" newline-separated parse; this could be implemented internally as [the equivalent of] a `jq -c` filter ahead of the insert step. 2. Catch the JSONDecodeError (or pre-empt it in the case of a record === "{\n") and give the user a "it looks like your json isn't _actually_ newline-delimited; try running it through `jq -c` instead" error message. It might just have been too early in the morning when I was playing with this, but running pipes of data through sqlite-utils without the 'knack' of it led to some false starts. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/417/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1175854982 | I_kwDOBm6k_c5GFh-G | 1679 | Research: how much overhead does the n=1 time limit have? | 9599 | closed | 0 | 3268330 | 11 | 2022-03-21T19:27:46Z | 2022-03-21T21:55:57Z | 2022-03-21T21:55:56Z | OWNER | https://github.com/simonw/datasette/blob/1a7750eb29fd15dd2eea3b9f6e33028ce441b143/datasette/utils/__init__.py#L181-L200 ```python @contextmanager def sqlite_timelimit(conn, ms): deadline = time.perf_counter() + (ms / 1000) # n is the number of SQLite virtual machine instructions that will be # executed between each check. It's hard to know what to pick here. # After some experimentation, I've decided to go with 1000 by default and # 1 for time limits that are less than 50ms n = 1000 if ms < 50: n = 1 def handler(): if time.perf_counter() >= deadline: return 1 conn.set_progress_handler(handler, n) try: yield finally: conn.set_progress_handler(None, n) ``` How often do I set a time limit of 50 or less? How much slower does it go thanks to this code? | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1679/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1175715988 | I_kwDOBm6k_c5GFACU | 1678 | Make `check_visibility()` a documented API | 9599 | closed | 0 | 3268330 | 1 | 2022-03-21T17:30:34Z | 2022-03-21T19:04:03Z | 2022-03-21T19:01:46Z | OWNER | Spotted this while working on: - #1677 https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/utils/__init__.py#L1005-L1021 | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1678/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1175694248 | I_kwDOBm6k_c5GE6uo | 1677 | Remove `check_permission()` from `BaseView` | 9599 | closed | 0 | 3268330 | 1 | 2022-03-21T17:18:18Z | 2022-03-21T18:45:04Z | 2022-03-21T18:45:03Z | OWNER | Follow-on from: - #1675 Refs: - #1660 | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1677/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1175648453 | I_kwDOBm6k_c5GEvjF | 1675 | Extract out `check_permissions()` from `BaseView | 9599 | closed | 0 | 7 | 2022-03-21T16:39:46Z | 2022-03-21T17:14:31Z | 2022-03-21T17:13:21Z | OWNER | > I'm going to refactor this stuff out and document it so it can be easily used by plugins: https://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L69-L103 _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1660#issuecomment-1074136176_ | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1675/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed |