html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/datasette/issues/1550#issuecomment-991805516,https://api.github.com/repos/simonw/datasette/issues/1550,991805516,IC_kwDOBm6k_c47HcBM,9599,2021-12-11T23:43:24Z,2021-12-11T23:43:24Z,OWNER,"I built a tiny Starlette app to experiment with this a bit:
```python
import asyncio
import janus
from starlette.applications import Starlette
from starlette.responses import JSONResponse, HTMLResponse, StreamingResponse
from starlette.routing import Route
import sqlite3
from concurrent import futures
executor = futures.ThreadPoolExecutor(max_workers=10)
async def homepage(request):
return HTMLResponse(
""""""
SQL CSV Server
SQL CSV Server
""""""
)
def run_query_in_thread(sql, sync_q):
db = sqlite3.connect(""../datasette/covid.db"")
cursor = db.cursor()
cursor.arraysize = 100 # Default is 1 apparently?
cursor.execute(sql)
columns = [d[0] for d in cursor.description]
sync_q.put([columns])
# Now start putting batches of rows
while True:
rows = cursor.fetchmany()
if rows:
sync_q.put(rows)
else:
break
# Let queue know we are finished\
sync_q.put(None)
async def csv_query(request):
sql = request.query_params[""sql""]
queue = janus.Queue()
loop = asyncio.get_running_loop()
async def csv_generator():
loop.run_in_executor(None, run_query_in_thread, sql, queue.sync_q)
while True:
rows = await queue.async_q.get()
if rows is not None:
for row in rows:
yield "","".join(map(str, row)) + ""\n ""
queue.async_q.task_done()
else:
# Cleanup
queue.close()
await queue.wait_closed()
break
return StreamingResponse(csv_generator(), media_type='text/plain')
app = Starlette(
debug=True,
routes=[
Route(""/"", homepage),
Route(""/csv"", csv_query),
],
)
```
But.. if I run this in a terminal window:
```
/tmp % wget 'http://127.0.0.1:8000/csv?sql=select+*+from+ny_times_us_counties'
```
it takes about 20 seconds to run and returns a 50MB file - but while it is running no other requests can be served by that server - not even the homepage! So something is blocking the event loop.
Maybe I should be using `fut = loop.run_in_executor(None, run_query_in_thread, sql, queue.sync_q)` and then awaiting `fut` somewhere, like in the Janus documentation? Don't think that's needed though. Needs more work to figure out why this is blocking.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077628073,
https://github.com/simonw/datasette/issues/1550#issuecomment-991761635,https://api.github.com/repos/simonw/datasette/issues/1550,991761635,IC_kwDOBm6k_c47HRTj,9599,2021-12-11T19:39:01Z,2021-12-11T19:39:01Z,OWNER,"I wonder if this could work for public instances too with some kind of queuing mechanism?
I really need to use benchmarking to figure out what the right number of maximum SQLite connections is. I'm just guessing at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077628073,
https://github.com/simonw/datasette/issues/1549#issuecomment-991755245,https://api.github.com/repos/simonw/datasette/issues/1549,991755245,IC_kwDOBm6k_c47HPvt,9599,2021-12-11T19:17:54Z,2021-12-11T19:17:54Z,OWNER,"Also relevant:
- #1062 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,
https://github.com/simonw/datasette/issues/617#issuecomment-991755013,https://api.github.com/repos/simonw/datasette/issues/617,991755013,IC_kwDOBm6k_c47HPsF,9599,2021-12-11T19:17:11Z,2021-12-11T19:17:11Z,OWNER,This work is now happening in #1518 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",519613116,
https://github.com/simonw/datasette/issues/1549#issuecomment-991754794,https://api.github.com/repos/simonw/datasette/issues/1549,991754794,IC_kwDOBm6k_c47HPoq,9599,2021-12-11T19:16:33Z,2021-12-11T19:16:33Z,OWNER,Good call! I'm doing a refactor #1518 right now which will hopefully bring the functionality of those two much closer - I'll make a note to consider this there too.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,
https://github.com/simonw/datasette/issues/1549#issuecomment-991752486,https://api.github.com/repos/simonw/datasette/issues/1549,991752486,IC_kwDOBm6k_c47HPEm,9599,2021-12-11T19:09:15Z,2021-12-11T19:09:15Z,OWNER,"That's what this option does:
The usability of this is pretty terrible though (including ""stream all rows"" - how are people meant to understand what that does?) so it can definitely do with some rethinking.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,
https://github.com/simonw/sqlite-utils/issues/356#issuecomment-991517209,https://api.github.com/repos/simonw/sqlite-utils/issues/356,991517209,IC_kwDOCGYnMM47GVoZ,9599,2021-12-11T07:46:41Z,2021-12-11T07:46:41Z,OWNER,"By default this will accept single lines, but maybe there could be a `--all` option which instead grabs all of stdin into a single string against which the conversion function runs - like `git-history file`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,
https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991400016,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991400016,IC_kwDOCGYnMM47F5BQ,9599,2021-12-11T01:10:52Z,2021-12-11T01:11:02Z,OWNER,"This won't be in a release for a little while, but you can install it to try it out using:
pip install https://github.com/simonw/sqlite-utils/archive/ee13f98c2c.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,
https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991399782,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991399782,IC_kwDOCGYnMM47F49m,9599,2021-12-11T01:09:37Z,2021-12-11T01:09:37Z,OWNER,"OK, this is implemented. Updated documentation is here: https://sqlite-utils.datasette.io/en/latest/cli.html#converting-data-in-columns","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,
https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991399604,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991399604,IC_kwDOCGYnMM47F460,9599,2021-12-11T01:08:46Z,2021-12-11T01:08:46Z,OWNER,That passed!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,
https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991398367,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991398367,IC_kwDOCGYnMM47F4nf,9599,2021-12-11T01:03:14Z,2021-12-11T01:03:14Z,OWNER,The new test: https://github.com/simonw/sqlite-utils/blob/ee13f98c2c7ca3b819bd0fc55da3108cb6a6434a/tests/test_fts.py#L270-L277,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,
https://github.com/simonw/sqlite-utils/pull/347#issuecomment-991397907,https://api.github.com/repos/simonw/sqlite-utils/issues/347,991397907,IC_kwDOCGYnMM47F4gT,9599,2021-12-11T01:01:40Z,2021-12-11T01:01:40Z,OWNER,The change I made to that test in #354 might help with this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,
https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991395919,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991395919,IC_kwDOCGYnMM47F4BP,9599,2021-12-11T00:52:31Z,2021-12-11T00:52:31Z,OWNER,"It turns out `rebuild` does indeed work against content tables, so I can put that in the test instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,
https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991395494,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991395494,IC_kwDOCGYnMM47F36m,9599,2021-12-11T00:50:22Z,2021-12-11T00:51:15Z,OWNER,"Here's an example of the new (slightly confusing) error message:
```bash
sqlite-utils convert fixtures.db roadside_attractions name '
def foo(value)
bar
baz
'
Error: Syntax error in code:
def foo(value)
invalid syntax
```
Another:
```
sqlite-utils convert fixtures.db roadside_attractions name '$'
Error: Syntax error in code:
return $
invalid syntax
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,
https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991393684,IC_kwDOCGYnMM47F3eU,9599,2021-12-11T00:42:19Z,2021-12-11T00:49:49Z,OWNER,"Ideally I'd like to show the perfect syntax error messages to the user - but I don't know if it's possible to do this cleanly because the error might occur with their originally entered code OR it might occur after I add `def fn(value)` to it.
I'm going to punt on that for the moment and tolerate slightly confusing syntax errors.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,
https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991386841,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991386841,IC_kwDOCGYnMM47F1zZ,9599,2021-12-11T00:14:11Z,2021-12-11T00:15:15Z,OWNER,"Relevant code: https://github.com/simonw/sqlite-utils/blob/7a43af232e4bc00bd227307665163614e225948b/sqlite_utils/cli.py#L2128-L2135
One way to implement this would be to look to see if the code starts with `def ...` - but that's not going to work for proper module that start with a docstring or imports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,
https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991387044,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991387044,IC_kwDOCGYnMM47F12k,9599,2021-12-11T00:14:45Z,2021-12-11T00:14:45Z,OWNER,"Maybe attempt to compile their code, and if it fails try again after adding `def fn(value):` to the start?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,