github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991386841 | https://api.github.com/repos/simonw/sqlite-utils/issues/355 | 991386841 | IC_kwDOCGYnMM47F1zZ | 9599 | 2021-12-11T00:14:11Z | 2021-12-11T00:15:15Z | OWNER | Relevant code: https://github.com/simonw/sqlite-utils/blob/7a43af232e4bc00bd227307665163614e225948b/sqlite_utils/cli.py#L2128-L2135 One way to implement this would be to look to see if the code starts with `def ...` - but that's not going to work for proper module that start with a docstring or imports. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077322009 | |
https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991387044 | https://api.github.com/repos/simonw/sqlite-utils/issues/355 | 991387044 | IC_kwDOCGYnMM47F12k | 9599 | 2021-12-11T00:14:45Z | 2021-12-11T00:14:45Z | OWNER | Maybe attempt to compile their code, and if it fails try again after adding `def fn(value):` to the start? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077322009 | |
https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684 | https://api.github.com/repos/simonw/sqlite-utils/issues/355 | 991393684 | IC_kwDOCGYnMM47F3eU | 9599 | 2021-12-11T00:42:19Z | 2021-12-11T00:49:49Z | OWNER | Ideally I'd like to show the perfect syntax error messages to the user - but I don't know if it's possible to do this cleanly because the error might occur with their originally entered code OR it might occur after I add `def fn(value)` to it. I'm going to punt on that for the moment and tolerate slightly confusing syntax errors. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077322009 | |
https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991395494 | https://api.github.com/repos/simonw/sqlite-utils/issues/355 | 991395494 | IC_kwDOCGYnMM47F36m | 9599 | 2021-12-11T00:50:22Z | 2021-12-11T00:51:15Z | OWNER | Here's an example of the new (slightly confusing) error message: ```bash sqlite-utils convert fixtures.db roadside_attractions name ' def foo(value) bar baz ' Error: Syntax error in code: def foo(value) invalid syntax ``` Another: ``` sqlite-utils convert fixtures.db roadside_attractions name '$' Error: Syntax error in code: return $ invalid syntax ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077322009 | |
https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991395919 | https://api.github.com/repos/simonw/sqlite-utils/issues/354 | 991395919 | IC_kwDOCGYnMM47F4BP | 9599 | 2021-12-11T00:52:31Z | 2021-12-11T00:52:31Z | OWNER | It turns out `rebuild` does indeed work against content tables, so I can put that in the test instead. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077243232 | |
https://github.com/simonw/sqlite-utils/pull/347#issuecomment-991397907 | https://api.github.com/repos/simonw/sqlite-utils/issues/347 | 991397907 | IC_kwDOCGYnMM47F4gT | 9599 | 2021-12-11T01:01:40Z | 2021-12-11T01:01:40Z | OWNER | The change I made to that test in #354 might help with this. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1066603133 | |
https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991398367 | https://api.github.com/repos/simonw/sqlite-utils/issues/354 | 991398367 | IC_kwDOCGYnMM47F4nf | 9599 | 2021-12-11T01:03:14Z | 2021-12-11T01:03:14Z | OWNER | The new test: https://github.com/simonw/sqlite-utils/blob/ee13f98c2c7ca3b819bd0fc55da3108cb6a6434a/tests/test_fts.py#L270-L277 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077243232 | |
https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991399604 | https://api.github.com/repos/simonw/sqlite-utils/issues/354 | 991399604 | IC_kwDOCGYnMM47F460 | 9599 | 2021-12-11T01:08:46Z | 2021-12-11T01:08:46Z | OWNER | That passed! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077243232 | |
https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991399782 | https://api.github.com/repos/simonw/sqlite-utils/issues/353 | 991399782 | IC_kwDOCGYnMM47F49m | 9599 | 2021-12-11T01:09:37Z | 2021-12-11T01:09:37Z | OWNER | OK, this is implemented. Updated documentation is here: https://sqlite-utils.datasette.io/en/latest/cli.html#converting-data-in-columns | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077102934 | |
https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991400016 | https://api.github.com/repos/simonw/sqlite-utils/issues/353 | 991400016 | IC_kwDOCGYnMM47F5BQ | 9599 | 2021-12-11T01:10:52Z | 2021-12-11T01:11:02Z | OWNER | This won't be in a release for a little while, but you can install it to try it out using: pip install https://github.com/simonw/sqlite-utils/archive/ee13f98c2c.zip | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077102934 | |
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-991517209 | https://api.github.com/repos/simonw/sqlite-utils/issues/356 | 991517209 | IC_kwDOCGYnMM47GVoZ | 9599 | 2021-12-11T07:46:41Z | 2021-12-11T07:46:41Z | OWNER | By default this will accept single lines, but maybe there could be a `--all` option which instead grabs all of stdin into a single string against which the conversion function runs - like `git-history file`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077431957 | |
https://github.com/simonw/datasette/issues/1549#issuecomment-991752486 | https://api.github.com/repos/simonw/datasette/issues/1549 | 991752486 | IC_kwDOBm6k_c47HPEm | 9599 | 2021-12-11T19:09:15Z | 2021-12-11T19:09:15Z | OWNER | That's what this option does: ![EAB1B9E8-38E9-4C6D-8854-BD1935F163D9](https://user-images.githubusercontent.com/9599/145688531-668bafa1-e287-4bbd-84d6-157241fb1f68.jpeg) The usability of this is pretty terrible though (including "stream all rows" - how are people meant to understand what that does?) so it can definitely do with some rethinking. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077620955 | |
https://github.com/simonw/datasette/issues/1549#issuecomment-991754794 | https://api.github.com/repos/simonw/datasette/issues/1549 | 991754794 | IC_kwDOBm6k_c47HPoq | 9599 | 2021-12-11T19:16:33Z | 2021-12-11T19:16:33Z | OWNER | Good call! I'm doing a refactor #1518 right now which will hopefully bring the functionality of those two much closer - I'll make a note to consider this there too. | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077620955 | |
https://github.com/simonw/datasette/issues/617#issuecomment-991755013 | https://api.github.com/repos/simonw/datasette/issues/617 | 991755013 | IC_kwDOBm6k_c47HPsF | 9599 | 2021-12-11T19:17:11Z | 2021-12-11T19:17:11Z | OWNER | This work is now happening in #1518 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
519613116 | |
https://github.com/simonw/datasette/issues/1549#issuecomment-991755245 | https://api.github.com/repos/simonw/datasette/issues/1549 | 991755245 | IC_kwDOBm6k_c47HPvt | 9599 | 2021-12-11T19:17:54Z | 2021-12-11T19:17:54Z | OWNER | Also relevant: - #1062 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077620955 | |
https://github.com/simonw/datasette/issues/1550#issuecomment-991761635 | https://api.github.com/repos/simonw/datasette/issues/1550 | 991761635 | IC_kwDOBm6k_c47HRTj | 9599 | 2021-12-11T19:39:01Z | 2021-12-11T19:39:01Z | OWNER | I wonder if this could work for public instances too with some kind of queuing mechanism? I really need to use benchmarking to figure out what the right number of maximum SQLite connections is. I'm just guessing at the moment. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077628073 | |
https://github.com/simonw/datasette/issues/1550#issuecomment-991805516 | https://api.github.com/repos/simonw/datasette/issues/1550 | 991805516 | IC_kwDOBm6k_c47HcBM | 9599 | 2021-12-11T23:43:24Z | 2021-12-11T23:43:24Z | OWNER | I built a tiny Starlette app to experiment with this a bit: ```python import asyncio import janus from starlette.applications import Starlette from starlette.responses import JSONResponse, HTMLResponse, StreamingResponse from starlette.routing import Route import sqlite3 from concurrent import futures executor = futures.ThreadPoolExecutor(max_workers=10) async def homepage(request): return HTMLResponse( """ <html> <head><title>SQL CSV Server</title> <style>body { width: 40rem; font-family: helvetica; margin: 2em auto; }</style> <body> <h1>SQL CSV Server</h1> <form action="/csv"> <label style="display: block">SQL query: <textarea style="width: 90%; height: 20em" name="sql"></textarea> </label> <input type="submit" value="Run query"> </form> </head> """ ) def run_query_in_thread(sql, sync_q): db = sqlite3.connect("../datasette/covid.db") cursor = db.cursor() cursor.arraysize = 100 # Default is 1 apparently? cursor.execute(sql) columns = [d[0] for d in cursor.description] sync_q.put([columns]) # Now start putting batches of rows while True: rows = cursor.fetchmany() if rows: sync_q.put(rows) else: break # Let queue know we are finished\ sync_q.put(None) async def csv_query(request): sql = request.query_params["sql"] queue = janus.Queue() loop = asyncio.get_running_loop() async def csv_generator(): loop.run_in_executor(None, run_query_in_thread, sql, queue.sync_q) while True: rows = await queue.async_q.get() if rows is not None: for row in rows: yield ",".join(map(str, row)) + "\n " queue.async_q.task_done() else: # Cleanup queue.close() await queue.wait_closed() … | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077628073 | |
https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991405755 | https://api.github.com/repos/simonw/sqlite-utils/issues/353 | 991405755 | IC_kwDOCGYnMM47F6a7 | 536941 | 2021-12-11T01:38:29Z | 2021-12-11T01:38:29Z | CONTRIBUTOR | wow! that's awesome! thanks so much, @simonw! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077102934 | |
https://github.com/simonw/datasette/issues/1549#issuecomment-991754237 | https://api.github.com/repos/simonw/datasette/issues/1549 | 991754237 | IC_kwDOBm6k_c47HPf9 | 536941 | 2021-12-11T19:14:39Z | 2021-12-11T19:14:39Z | CONTRIBUTOR | that option is not available on [custom queries](https://labordata.bunkum.us/odpr-962a140?sql=with+local_union_filings+as+%28%0D%0A++select+*+from+lm_data+%0D%0A++where%0D%0A++++yr_covered+%3E+cast%28strftime%28%27%25Y%27%2C+%27now%27%2C+%27-5+years%27%29+as+int%29%0D%0A++++and+desig_name+%3D+%27LU%27%0D%0A++order+by+yr_covered+desc%0D%0A%29%2C%0D%0Amost_recent_filing+as+%28%0D%0A++select%0D%0A++++*%0D%0A++from+local_union_filings%0D%0A++group+by%0D%0A++++f_num%0D%0A%29%0D%0Aselect%0D%0A++*%0D%0Afrom%0D%0A++most_recent_filing%0D%0Awhere%0D%0A++next_election+%3E%3D+strftime%28%27%25Y-%25m%27%2C+%27now%27%29%0D%0A++and+next_election+%3C+strftime%28%27%25Y-%25m%27%2C+%27now%27%2C+%27%2B1+year%27%29%0D%0Aorder+by%0D%0A++members+desc%3B). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1077620955 | |
https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982123183 | https://api.github.com/repos/simonw/sqlite-utils/issues/347 | 982123183 | IC_kwDOCGYnMM46igKv | 22429695 | 2021-11-29T23:20:35Z | 2021-12-11T01:02:19Z | NONE | # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#347](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (71b6c38) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/213a0ff177f23a35f3b235386366ff132eb879f1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (213a0ff) will **increase** coverage by `0.00%`. > The diff coverage is `100.00%`. > :exclamation: Current head 71b6c38 differs from pull request most recent head 1a7ef2f. Consider uploading reports for the commit 1a7ef2f to get more accurate results [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/347/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #347 +/- ## ======================================= Coverage 96.51% 96.52% ======================================= Files 5 5 Lines 2270 2271 +1 ======================================= + Hits 2191 2192 +1 Misses 79 79 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/347?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/347/diff?src=pr&el=tree&utm_medium=referral&utm_sou… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1066603133 |