issue_comments
8,883 rows where user = 9599 sorted by updated_at descending
This data as json, CSV (advanced)
user 1
- simonw · 5,699 ✖
id | html_url | issue_url | node_id | user | created_at | updated_at ▲ | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
973635157 | https://github.com/simonw/datasette/issues/878#issuecomment-973635157 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46CH5V | simonw 9599 | 2021-11-19T01:07:08Z | 2021-11-19T01:07:08Z | OWNER | This exercise is proving so useful in getting my head around how the enormous and complex Here's where I've got to now - I'm systematically working through the variables that are returned for HTML and for JSON copying across code to get it to work: ```python from datasette.database import QueryInterrupted from datasette.utils import escape_sqlite from datasette.utils.asgi import Response, NotFound, Forbidden from datasette.views.base import DatasetteError from datasette import hookimpl from asyncinject import AsyncInject, inject from pprint import pformat class Table(AsyncInject): @inject async def database(self, request, datasette): # TODO: all that nasty hash resolving stuff can go here db_name = request.url_vars["db_name"] try: db = datasette.databases[db_name] except KeyError: raise NotFound(f"Database '{db_name}' does not exist") return db
@hookimpl def register_routes(): return [ (r"/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", Table().view), ] async def check_permissions(datasette, request, permissions): """permissions is a list of (action, resource) tuples or 'action' strings""" for permission in permissions: if isinstance(permission, str): action = permission resource = None elif isinstance(permission, (tuple, list)) and len(permission) == 2: action, resource = permission else: assert ( False ), "permission should be string or tuple of two items: {}".format( repr(permission) ) ok = await datasette.permission_allowed( request.actor, action, resource=resource, default=None, ) if ok is not None: if ok: return else: raise Forbidden(action) async def columns_to_select(datasette, database, table, request): table_columns = await database.table_columns(table) pks = await database.primary_keys(table) columns = list(table_columns) if "_col" in request.args: columns = list(pks) _cols = request.args.getlist("_col") bad_columns = [column for column in _cols if column not in table_columns] if bad_columns: raise DatasetteError( "_col={} - invalid columns".format(", ".join(bad_columns)), status=400, ) # De-duplicate maintaining order: columns.extend(dict.fromkeys(_cols)) if "_nocol" in request.args: # Return all columns EXCEPT these bad_columns = [ column for column in request.args.getlist("_nocol") if (column not in table_columns) or (column in pks) ] if bad_columns: raise DatasetteError( "_nocol={} - invalid columns".format(", ".join(bad_columns)), status=400, ) tmp_columns = [ column for column in columns if column not in request.args.getlist("_nocol") ] columns = tmp_columns return columns ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
973568285 | https://github.com/simonw/datasette/issues/878#issuecomment-973568285 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46B3kd | simonw 9599 | 2021-11-19T00:29:20Z | 2021-11-19T00:29:20Z | OWNER | This is working! ```python from datasette.utils.asgi import Response from datasette import hookimpl import html from asyncinject import AsyncInject, inject class Table(AsyncInject): @inject async def database(self, request): return request.url_vars["db_name"]
@hookimpl
def register_routes():
return [
(r"/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", Table().view),
]
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
973564260 | https://github.com/simonw/datasette/issues/878#issuecomment-973564260 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46B2lk | simonw 9599 | 2021-11-19T00:27:06Z | 2021-11-19T00:27:06Z | OWNER | Problem: the fancy @hookimpl def register_routes(): return [ (r"/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", Table().view), ] ``` This failed with error: "Table.view() takes 1 positional argument but 2 were given" So I'm going to use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
973554024 | https://github.com/simonw/datasette/issues/878#issuecomment-973554024 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46B0Fo | simonw 9599 | 2021-11-19T00:21:20Z | 2021-11-19T00:21:20Z | OWNER | That's annoying: it looks like plugins can't use async def table(request): return Response.html("Hello from {}".format( html.escape(repr(request.url_vars)) )) @hookimpl
def register_routes():
return [
(r"/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", table),
]
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
973542284 | https://github.com/simonw/datasette/issues/878#issuecomment-973542284 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46BxOM | simonw 9599 | 2021-11-19T00:16:44Z | 2021-11-19T00:16:44Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
973527870 | https://github.com/simonw/datasette/issues/878#issuecomment-973527870 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46Bts- | simonw 9599 | 2021-11-19T00:13:43Z | 2021-11-19T00:13:43Z | OWNER | New plan: I'm going to build a brand new implementation of It will reuse the existing HTML template but will be a completely new Python implementation, based on I'm going to start by just getting the table to show up on the page - then I'll add faceting, suggested facets, filters and so-on. Bonus: I'm going to see if I can get it to work for arbitrary SQL queries too (stretch goal). |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
971209475 | https://github.com/simonw/datasette/issues/878#issuecomment-971209475 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c4543sD | simonw 9599 | 2021-11-17T05:41:42Z | 2021-11-17T05:41:42Z | OWNER | I'm going to build a brand new implementation of the I can maybe even run the tests against old |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
971057553 | https://github.com/simonw/datasette/issues/878#issuecomment-971057553 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c454SmR | simonw 9599 | 2021-11-17T01:40:45Z | 2021-11-17T01:40:45Z | OWNER | I shipped that code as a new library, |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
971056169 | https://github.com/simonw/datasette/pull/1512#issuecomment-971056169 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c454SQp | simonw 9599 | 2021-11-17T01:39:44Z | 2021-11-17T01:39:44Z | OWNER | Closing this PR because I shipped the code in it as a separate library instead. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
971055677 | https://github.com/simonw/datasette/pull/1512#issuecomment-971055677 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c454SI9 | simonw 9599 | 2021-11-17T01:39:25Z | 2021-11-17T01:39:25Z | OWNER | https://github.com/simonw/asyncinject version 0.1a0 is now live on PyPI: https://pypi.org/project/asyncinject/ |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
971010724 | https://github.com/simonw/datasette/pull/1512#issuecomment-971010724 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c454HKk | simonw 9599 | 2021-11-17T01:12:22Z | 2021-11-17T01:12:22Z | OWNER | I'm going to extract out the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
970861628 | https://github.com/simonw/datasette/pull/1512#issuecomment-970861628 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c453iw8 | simonw 9599 | 2021-11-16T23:46:07Z | 2021-11-16T23:46:07Z | OWNER | I made the changes locally and tested them with Python 3.6 like so:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
970857411 | https://github.com/simonw/datasette/pull/1512#issuecomment-970857411 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c453hvD | simonw 9599 | 2021-11-16T23:43:21Z | 2021-11-16T23:43:21Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
970855084 | https://github.com/simonw/datasette/issues/1513#issuecomment-970855084 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453hKs | simonw 9599 | 2021-11-16T23:41:46Z | 2021-11-16T23:41:46Z | OWNER | Conclusion: using a giant convoluted CTE and UNION ALL query to attempt to calculate facets at the same time as retrieving rows is a net LOSS for performance! Very surprised to see that. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970853917 | https://github.com/simonw/datasette/issues/1513#issuecomment-970853917 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453g4d | simonw 9599 | 2021-11-16T23:41:01Z | 2021-11-16T23:41:01Z | OWNER | One very interesting difference between the two: on the single giant query page:
The single big query takes 376ms total to render the page, spending 370ms in 5 queries Those 5 queries, if you're interested```sql select database_name, schema_version from databases PRAGMA schema_version PRAGMA schema_version explain with cte as (\r\n select rowid, date, county, state, fips, cases, deaths\r\n from ny_times_us_counties\r\n),\r\ntruncated as (\r\n select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths\r\n from cte order by date desc limit 4\r\n),\r\nstate_facet as (\r\n select 'state' as _facet, state as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\nfips_facet as (\r\n select 'fips' as _facet, fips as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\ncounty_facet as (\r\n select 'county' as _facet, county as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n)\r\nselect * from truncated\r\nunion all select * from state_facet\r\nunion all select * from fips_facet\r\nunion all select * from county_facet with cte as (\r\n select rowid, date, county, state, fips, cases, deaths\r\n from ny_times_us_counties\r\n),\r\ntruncated as (\r\n select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths\r\n from cte order by date desc limit 4\r\n),\r\nstate_facet as (\r\n select 'state' as _facet, state as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\nfips_facet as (\r\n select 'fips' as _facet, fips as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\ncounty_facet as (\r\n select 'county' as _facet, county as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n)\r\nselect * from truncated\r\nunion all select * from state_facet\r\nunion all select * from fips_facet\r\nunion all select * from county_facet ```All of that additional non-SQL overhead must be stuff relating to Python and template rendering code running on the page. I'm really surprised at how much overhead that is! This is worth researching separately. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970845844 | https://github.com/simonw/datasette/issues/1513#issuecomment-970845844 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453e6U | simonw 9599 | 2021-11-16T23:35:38Z | 2021-11-16T23:35:38Z | OWNER | I tried adding
Compared to: Which is 353ms total. The separate queries ran faster! Really surprising result there. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970828568 | https://github.com/simonw/datasette/issues/1513#issuecomment-970828568 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453asY | simonw 9599 | 2021-11-16T23:27:11Z | 2021-11-16T23:27:11Z | OWNER | One last experiment: I'm going to try running an expensive query in the CTE portion. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970827674 | https://github.com/simonw/datasette/issues/1513#issuecomment-970827674 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453aea | simonw 9599 | 2021-11-16T23:26:58Z | 2021-11-16T23:26:58Z | OWNER | With trace. https://covid-19.datasettes.com/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2&_trace=1 shows the following:
It didn't run a count because that's the homepage and the count is cached. So I dropped the count from the query and ran it: https://covid-19.datasettes.com/covid?sql=with+cte+as+(%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A)%2C%0D%0Atruncated+as+(%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A)%2C%0D%0Astate_facet+as+(%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count()+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Afips_facet+as+(%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count()+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Acounty_facet+as+(%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count()+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%0D%0Aselect++from+truncated%0D%0Aunion+all+select++from+state_facet%0D%0Aunion+all+select++from+fips_facet%0D%0Aunion+all+select+*+from+county_facet&_trace=1 Shows 649.4359889999259 ms for the query - compared to 755.78843400001ms for the separate. So it saved about 100ms. Still not a huge difference though! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970780866 | https://github.com/simonw/datasette/issues/1513#issuecomment-970780866 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453PDC | simonw 9599 | 2021-11-16T23:01:57Z | 2021-11-16T23:01:57Z | OWNER | One disadvantage to this approach: if you have a SQL time limit of 1s and it takes 0.9s to return the rows but then 0.5s to calculate each of the requested facets the entire query will exceed the time limit. Could work around this by catching that error and then re-running the query just for the rows, but that would result in the user having to wait longer for the results. Could try to remember if that has happened using an in-memory Python data structure and skip the faceting optimization if it's caused problems in the past? That seems a bit gross. Maybe this becomes an opt-in optimization you can request in your What if we kept the query that returns the rows to be displayed on the page separate from the facets, but then executed all of the facets together using this method such that the Maybe a better optimization would be to move facets to happening via |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970766486 | https://github.com/simonw/datasette/issues/1513#issuecomment-970766486 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453LiW | simonw 9599 | 2021-11-16T22:52:56Z | 2021-11-16T22:56:07Z | OWNER | https://covid-19.datasettes.com/covid is 805.2MB https://covid-19.datasettes.com/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2 Equivalent SQL: https://covid-19.datasettes.com/covid?sql=with+cte+as+%28%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A%29%2C%0D%0Astate_facet+as+%28%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Afips_facet+as+%28%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Acounty_facet+as+%28%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Atotal_count+as+%28%0D%0A++select+%27COUNT%27+as+_facet%2C+%27%27+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte%0D%0A%29%0D%0Aselect++from+truncated%0D%0Aunion+all+select++from+state_facet%0D%0Aunion+all+select++from+fips_facet%0D%0Aunion+all+select++from+county_facet%0D%0Aunion+all+select+*+from+total_count
_facet | facet_name | facet_count | rowid | date | county | state | fips | cases | deaths -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | | | 1917344 | 2021-11-15 | Autauga | Alabama | 1001 | 10407 | 154 | | | 1917345 | 2021-11-15 | Baldwin | Alabama | 1003 | 37875 | 581 | | | 1917346 | 2021-11-15 | Barbour | Alabama | 1005 | 3648 | 79 | | | 1917347 | 2021-11-15 | Bibb | Alabama | 1007 | 4317 | 92 state | Texas | 148028 | | | | | | | state | Georgia | 96249 | | | | | | | state | Virginia | 79315 | | | | | | | fips | | 17580 | | | | | | | fips | 53061 | 665 | | | | | | | fips | 17031 | 662 | | | | | | | county | Washington | 18666 | | | | | | | county | Unknown | 15840 | | | | | | | county | Jefferson | 15637 | | | | | | | COUNT | | 1920593 | | | | | | | |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970770304 | https://github.com/simonw/datasette/issues/1513#issuecomment-970770304 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453MeA | simonw 9599 | 2021-11-16T22:55:19Z | 2021-11-16T22:55:19Z | OWNER | (One thing I really like about this pattern is that it should work exactly the same when used to facet the results of arbitrary SQL queries as it does when faceting results from the table page.) |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970767952 | https://github.com/simonw/datasette/issues/1513#issuecomment-970767952 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453L5Q | simonw 9599 | 2021-11-16T22:53:52Z | 2021-11-16T22:53:52Z | OWNER | It's going to take another 15 minutes for the build to finish and deploy the version with |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970758179 | https://github.com/simonw/datasette/issues/1513#issuecomment-970758179 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453Jgj | simonw 9599 | 2021-11-16T22:47:38Z | 2021-11-16T22:47:38Z | OWNER | Trace now enabled: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet_size=3&_size=2&_nocount=1&_trace=1 Here are the relevant traces:
I modified the query to include the total count as well: https://global-power-plants.datasettes.com/global-power-plants?sql=with+cte+as+%28%0D%0A++select+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+cte+order+by+rowid+limit+4%0D%0A%29%2C%0D%0Acountry_long_facet+as+%28%0D%0A++select+%27country_long%27+as+_facet%2C+country_long+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aowner_facet+as+%28%0D%0A++select+%27owner%27+as+_facet%2C+owner+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aprimary_fuel_facet+as+%28%0D%0A++select+%27primary_fuel%27+as+_facet%2C+primary_fuel+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Atotal_count+as+%28%0D%0A++select+%27COUNT%27+as+_facet%2C+%27%27+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte%0D%0A%29%0D%0Aselect++from+truncated%0D%0Aunion+all+select++from+country_long_facet%0D%0Aunion+all+select++from+owner_facet%0D%0Aunion+all+select++from+primary_fuel_facet%0D%0Aunion+all+select+*+from+total_count&_trace=1
To my huge surprise, this convoluted optimization only shaves the sum query time down from 37.8ms to 34.8ms! That entire database file is just 11.1 MB though. Maybe it would make a meaningful difference on something larger? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970742415 | https://github.com/simonw/datasette/issues/1513#issuecomment-970742415 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453FqP | simonw 9599 | 2021-11-16T22:37:14Z | 2021-11-16T22:37:14Z | OWNER | The query takes 42.794ms to run. Here's the equivalent page using separate queries: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet_size=3&_size=2&_nocount=1 Annoyingly I can't disable facet suggestions but keep facets. I'm going to turn on tracing so I can see how long the separate queries took. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970738130 | https://github.com/simonw/datasette/issues/1513#issuecomment-970738130 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453EnS | simonw 9599 | 2021-11-16T22:32:19Z | 2021-11-16T22:32:19Z | OWNER | I came up with the following query which seems to work!
Results look like this: _facet | facet_name | facet_count | rowid | country | country_long | name | owner | primary_fuel -- | -- | -- | -- | -- | -- | -- | -- | -- | | | 1 | AFG | Afghanistan | Kajaki Hydroelectric Power Plant Afghanistan | | Hydro | | | 2 | AFG | Afghanistan | Kandahar DOG | | Solar | | | 3 | AFG | Afghanistan | Kandahar JOL | | Solar | | | 4 | AFG | Afghanistan | Mahipar Hydroelectric Power Plant Afghanistan | | Hydro country_long | United States of America | 8688 | | | | | | country_long | China | 4235 | | | | | | country_long | United Kingdom | 2603 | | | | | | owner | | 14112 | | | | | | owner | Lightsource Renewable Energy | 120 | | | | | | owner | Cypress Creek Renewables | 109 | | | | | | primary_fuel | Solar | 9662 | | | | | | primary_fuel | Hydro | 7155 | | | | | | primary_fuel | Wind | 5188 | | | | | | This is a neat proof of concept. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970718337 | https://github.com/simonw/datasette/pull/1512#issuecomment-970718337 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c452_yB | simonw 9599 | 2021-11-16T22:02:30Z | 2021-11-16T22:02:30Z | OWNER | I've decided to make the clever class Simple(AsyncBase): def init(self): self.log = []
class Complex(AsyncBase): inject_all = True
``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
970712713 | https://github.com/simonw/datasette/issues/878#issuecomment-970712713 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c452-aJ | simonw 9599 | 2021-11-16T21:54:33Z | 2021-11-16T21:54:33Z | OWNER | I'm going to continue working on this in a PR. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970705738 | https://github.com/simonw/datasette/issues/878#issuecomment-970705738 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c4528tK | simonw 9599 | 2021-11-16T21:44:31Z | 2021-11-16T21:44:31Z | OWNER | Wrote a TIL about what I learned using |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970673085 | https://github.com/simonw/datasette/issues/878#issuecomment-970673085 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c4520u9 | simonw 9599 | 2021-11-16T20:58:24Z | 2021-11-16T20:58:24Z | OWNER | New test: ```python class Complex(AsyncBase): def init(self): self.log = []
@pytest.mark.asyncio
async def test_complex():
result = await Complex().go()
# 'c' should only be called once
assert tuple(result) in (
# c and d could happen in either order
("c", "d", "b", "a", "go"),
("d", "c", "b", "a", "go"),
)
try: import graphlib except ImportError: from . import vendored_graphlib as graphlib class AsyncMeta(type): def new(cls, name, bases, attrs): # Decorate any items that are 'async def' methods registry = {} new_attrs = {"_registry": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.__name__ == "resolve": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value # Gather graph for later dependency resolution graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != "self" and not p.startswith("") } for key, method in _registry.items() } new_attrs["_graph"] = graph return super().new(cls, name, bases, new_attrs) def make_method(method): parameters = inspect.signature(method).parameters.keys()
class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names, results=None): print("\n resolve: ", names) if results is None: results = {}
``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970660299 | https://github.com/simonw/datasette/issues/878#issuecomment-970660299 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c452xnL | simonw 9599 | 2021-11-16T20:39:43Z | 2021-11-16T20:42:27Z | OWNER | But that does seem to be the plan that ts = TopologicalSorter(graph)
ts.prepare()
while ts.is_active():
nodes = ts.get_ready()
print(nodes)
ts.done(*nodes)
ts = TopologicalSorter(graph)
ts.prepare()
while ts.is_active():
nodes = ts.get_ready()
print(nodes)
ts.done(nodes)
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970657874 | https://github.com/simonw/datasette/issues/878#issuecomment-970657874 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c452xBS | simonw 9599 | 2021-11-16T20:36:01Z | 2021-11-16T20:36:01Z | OWNER | My goal here is to calculate the most efficient way to resolve the different nodes, running them in parallel where possible. So for this class: ```python class Complex(AsyncBase): async def d(self): pass
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970655927 | https://github.com/simonw/datasette/issues/878#issuecomment-970655927 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c452wi3 | simonw 9599 | 2021-11-16T20:33:11Z | 2021-11-16T20:33:11Z | OWNER | What should be happening here instead is it should resolve the full graph and notice that So maybe the algorithm I'm inheriting from https://docs.python.org/3/library/graphlib.html isn't the correct algorithm? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970655304 | https://github.com/simonw/datasette/issues/878#issuecomment-970655304 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c452wZI | simonw 9599 | 2021-11-16T20:32:16Z | 2021-11-16T20:32:16Z | OWNER | This code is really fiddly. I just got to this version: ```python import asyncio from functools import wraps import inspect try: import graphlib except ImportError: from . import vendored_graphlib as graphlib class AsyncMeta(type): def new(cls, name, bases, attrs): # Decorate any items that are 'async def' methods registry = {} new_attrs = {"_registry": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.__name__ == "resolve": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value # Gather graph for later dependency resolution graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != "self" and not p.startswith("") } for key, method in _registry.items() } new_attrs["_graph"] = graph return super().new(cls, name, bases, new_attrs) def make_method(method): @wraps(method) async def inner(self, _results=None, kwargs): print("inner - _results=", _results) parameters = inspect.signature(method).parameters.keys() # Any parameters not provided by kwargs are resolved from registry to_resolve = [p for p in parameters if p not in kwargs and p != "self"] missing = [p for p in to_resolve if p not in self._registry] assert ( not missing ), "The following DI parameters could not be found in the registry: {}".format( missing ) results = {} results.update(kwargs) if to_resolve: resolved_parameters = await self.resolve(to_resolve, _results) results.update(resolved_parameters) return_value = await method(self, results) if _results is not None: _results[method.name] = return_value return return_value
class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names, results=None): print("\n resolve: ", names) if results is None: results = {}
@pytest.mark.asyncio async def test_complex(): result = await Complex().go() # 'c' should only be called once assert result == ["c", "b", "a", "go"] ``` This test sometimes passes, and sometimes fails! Output for a pass: ``` tests/test_asyncdi.py inner - _results= None resolve: ['a'] ts.get_ready() returned nodes: ('c', 'b') resolve_nodes ('c', 'b') (current results = {}) awaitables: [<coroutine object Complex.c at 0x1074ac890>, <coroutine object Complex.b at 0x1074ac820>] inner - _results= {} LOG: c inner - _results= {'c': None} resolve: ['c']
ts.get_ready() returned nodes: ('c',)
resolve_nodes ('c',)
(current results = {'c': None})
awaitables: []
End of resolve(), returning {'c': None}
LOG: b
ts.get_ready() returned nodes: ('a',)
resolve_nodes ('a',)
(current results = {'c': None, 'b': None})
awaitables: [<coroutine object Complex.a at 0x1074ac7b0>]
inner - _results= {'c': None, 'b': None}
LOG: a
End of resolve(), returning {'c': None, 'b': None, 'a': None}
LOG: go
resolve: ['a'] ts.get_ready() returned nodes: ('b', 'c') resolve_nodes ('b', 'c') (current results = {}) awaitables: [<coroutine object Complex.b at 0x10923c890>, <coroutine object Complex.c at 0x10923c820>] inner - _results= {} resolve: ['c'] ts.get_ready() returned nodes: ('c',) resolve_nodes ('c',) (current results = {}) awaitables: [<coroutine object Complex.c at 0x10923c6d0>] inner - _results= {} LOG: c inner - _results= {'c': None} LOG: c End of resolve(), returning {'c': None} LOG: b ts.get_ready() returned nodes: ('a',) resolve_nodes ('a',) (current results = {'c': None, 'b': None}) awaitables: [<coroutine object Complex.a at 0x10923c6d0>] inner - _results= {'c': None, 'b': None} LOG: a End of resolve(), returning {'c': None, 'b': None, 'a': None} LOG: go F =================================================================================================== FAILURES =================================================================================================== _______________ test_complex _________________
tests/test_asyncdi.py:48: AssertionError ================== short test summary info ================================ FAILED tests/test_asyncdi.py::test_complex - AssertionError: assert ['c', 'c', 'b', 'a', 'go'] == ['c', 'b', 'a', 'go'] ``` I figured out why this is happening.
The code decides to run If If |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970624197 | https://github.com/simonw/datasette/issues/878#issuecomment-970624197 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c452ozF | simonw 9599 | 2021-11-16T19:49:05Z | 2021-11-16T19:49:05Z | OWNER | Here's the latest version of my weird dependency injection async class: ```python import inspect class AsyncMeta(type): def new(cls, name, bases, attrs): # Decorate any items that are 'async def' methods _registry = {} new_attrs = {"_registry": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.name == "resolve": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value
def make_method(method): @wraps(method) async def inner(self, kwargs): parameters = inspect.signature(method).parameters.keys() # Any parameters not provided by kwargs are resolved from registry to_resolve = [p for p in parameters if p not in kwargs and p != "self"] missing = [p for p in to_resolve if p not in self._registry] assert ( not missing ), "The following DI parameters could not be found in the registry: {}".format( missing ) results = {} results.update(kwargs) results.update(await self.resolve(to_resolve)) return await method(self, results)
bad = [0] class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names): print(" resolve({})".format(names)) results = {} # Resolve them in the correct order ts = TopologicalSorter() ts2 = TopologicalSorter() print(" names = ", names) print(" self._graph = ", self._graph) for name in names: if self._graph[name]: ts.add(name, self._graph[name]) ts2.add(name, self._graph[name]) print(" static_order =", tuple(ts2.static_order())) ts.prepare() while ts.is_active(): print(" is_active, i = ", bad[0]) bad[0] += 1 if bad[0] > 20: print(" Infinite loop?") break nodes = ts.get_ready() print(" Do nodes:", nodes) awaitables = [self._registryname for name in nodes] print(" awaitables: ", awaitables) awaitable_results = await asyncio.gather(*awaitables) results.update({ p[0].name: p[1] for p in zip(awaitables, awaitable_results) }) print(results) for node in nodes: ts.done(node)
foo = Foo()
await foo.other()
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970554697 | https://github.com/simonw/datasette/issues/782#issuecomment-970554697 | https://api.github.com/repos/simonw/datasette/issues/782 | IC_kwDOBm6k_c452X1J | simonw 9599 | 2021-11-16T18:32:03Z | 2021-11-16T18:32:03Z | OWNER | I'm going to take another look at this: - https://github.com/simonw/datasette/issues/878 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
970553780 | https://github.com/simonw/datasette/issues/782#issuecomment-970553780 | https://api.github.com/repos/simonw/datasette/issues/782 | IC_kwDOBm6k_c452Xm0 | simonw 9599 | 2021-11-16T18:30:51Z | 2021-11-16T18:30:58Z | OWNER | OK, I'm ready to start working on this today. I'm going to go with a default representation that looks like this:
I'll implement |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
970544733 | https://github.com/simonw/datasette/issues/1509#issuecomment-970544733 | https://api.github.com/repos/simonw/datasette/issues/1509 | IC_kwDOBm6k_c452VZd | simonw 9599 | 2021-11-16T18:22:32Z | 2021-11-16T18:22:32Z | OWNER | This is mainly happening here: - https://github.com/simonw/datasette/issues/782 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Datasette 1.0 JSON API (and documentation) 1054243511 | |
969621662 | https://github.com/simonw/datasette/issues/448#issuecomment-969621662 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45y0Ce | simonw 9599 | 2021-11-16T01:32:04Z | 2021-11-16T01:32:04Z | OWNER | Tests are failing and I think it's because the facets come back in different orders, need a tie-breaker. https://github.com/simonw/datasette/runs/4219325197?check_suite_focus=true |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969616626 | https://github.com/simonw/datasette/issues/1176#issuecomment-969616626 | https://api.github.com/repos/simonw/datasette/issues/1176 | IC_kwDOBm6k_c45yyzy | simonw 9599 | 2021-11-16T01:29:13Z | 2021-11-16T01:29:13Z | OWNER | I'm inclined to create a Sphinx reference documentation page for this, as I did for |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Policy on documenting "public" datasette.utils functions 779691739 | |
969613166 | https://github.com/simonw/datasette/issues/1012#issuecomment-969613166 | https://api.github.com/repos/simonw/datasette/issues/1012 | IC_kwDOBm6k_c45yx9u | simonw 9599 | 2021-11-16T01:27:25Z | 2021-11-16T01:27:25Z | OWNER | Requested here: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
For 1.0 update trove classifier in setup.py 718540751 | |
969602825 | https://github.com/simonw/datasette/issues/1012#issuecomment-969602825 | https://api.github.com/repos/simonw/datasette/issues/1012 | IC_kwDOBm6k_c45yvcJ | simonw 9599 | 2021-11-16T01:21:14Z | 2021-11-16T01:21:14Z | OWNER | I'd been wondering how to get new classifiers into Trove - thanks, I'll give this a go. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
For 1.0 update trove classifier in setup.py 718540751 | |
969600859 | https://github.com/simonw/datasette/issues/1511#issuecomment-969600859 | https://api.github.com/repos/simonw/datasette/issues/1511 | IC_kwDOBm6k_c45yu9b | simonw 9599 | 2021-11-16T01:20:13Z | 2021-11-16T01:20:13Z | OWNER | See: - #830 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Review plugin hooks for Datasette 1.0 1054246919 | |
969582098 | https://github.com/simonw/datasette/issues/448#issuecomment-969582098 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45yqYS | simonw 9599 | 2021-11-16T01:10:28Z | 2021-11-16T01:10:28Z | OWNER | Also note that this demo data is using a SQL view to create the JSON arrays - the view is defined as such:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969578466 | https://github.com/simonw/datasette/issues/448#issuecomment-969578466 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45ypfi | simonw 9599 | 2021-11-16T01:08:29Z | 2021-11-16T01:08:29Z | OWNER | Actually with the cache warmed up it looks like the facet query is taking 150ms which is good enough. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969572281 | https://github.com/simonw/datasette/issues/448#issuecomment-969572281 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45yn-5 | simonw 9599 | 2021-11-16T01:05:11Z | 2021-11-16T01:05:11Z | OWNER | I tried this and it seems to work correctly:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969557008 | https://github.com/simonw/datasette/issues/448#issuecomment-969557008 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45ykQQ | simonw 9599 | 2021-11-16T00:56:09Z | 2021-11-16T00:59:59Z | OWNER | This looks like it might work:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969557972 | https://github.com/simonw/datasette/issues/448#issuecomment-969557972 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45ykfU | simonw 9599 | 2021-11-16T00:56:58Z | 2021-11-16T00:56:58Z | OWNER | It uses a CTE which were introduced in SQLite 3.8 - and AWS Lambda Python 3.9 still provides 3.7 - but I've checked and I can use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969449772 | https://github.com/simonw/datasette/issues/448#issuecomment-969449772 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45yKEs | simonw 9599 | 2021-11-15T23:48:37Z | 2021-11-15T23:48:37Z | OWNER | Given this query: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads?sql=select%0D%0A++j.value+as+value%2C%0D%0A++count%28*%29+as+count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++id%2C%0D%0A++++++file%2C%0D%0A++++++clicks%2C%0D%0A++++++impressions%2C%0D%0A++++++text%2C%0D%0A++++++url%2C%0D%0A++++++spend_amount%2C%0D%0A++++++spend_currency%2C%0D%0A++++++created%2C%0D%0A++++++ended%2C%0D%0A++++++target_names%0D%0A++++from%0D%0A++++++ads_with_targets%0D%0A++++where%0D%0A++++++%3Ap0+in+%28%0D%0A++++++++select%0D%0A++++++++++value%0D%0A++++++++from%0D%0A++++++++++json_each%28%5Bads_with_targets%5D.%5Btarget_names%5D%29%0D%0A++++++%29%0D%0A++%29%0D%0A++join+json_each%28target_names%29+j%0D%0Agroup+by%0D%0A++j.value%0D%0Aorder+by%0D%0A++count+desc%2C%0D%0A++value%0D%0Alimit%0D%0A++31&p0=people_who_match%3Ainterests%3AAfrican-American+culture
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969446972 | https://github.com/simonw/datasette/issues/448#issuecomment-969446972 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45yJY8 | simonw 9599 | 2021-11-15T23:46:13Z | 2021-11-15T23:46:13Z | OWNER | It looks like the problem here is that some of the tags occur more than once in the documents: So they get counted more than once, hence the 182 count for something that couldn't possibly return more than 172 documents. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969442215 | https://github.com/simonw/datasette/issues/448#issuecomment-969442215 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45yIOn | simonw 9599 | 2021-11-15T23:42:03Z | 2021-11-15T23:42:03Z | OWNER | I think this code is wrong in the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969440918 | https://github.com/simonw/datasette/issues/448#issuecomment-969440918 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45yH6W | simonw 9599 | 2021-11-15T23:40:17Z | 2021-11-15T23:40:35Z | OWNER | Applied that fix to the Should never get 182 results on a page that faceting against only 172 items. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969436930 | https://github.com/simonw/datasette/issues/448#issuecomment-969436930 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45yG8C | simonw 9599 | 2021-11-15T23:31:58Z | 2021-11-15T23:31:58Z | OWNER | I think this SQL recipe may work instead:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969433734 | https://github.com/simonw/datasette/issues/519#issuecomment-969433734 | https://api.github.com/repos/simonw/datasette/issues/519 | IC_kwDOBm6k_c45yGKG | simonw 9599 | 2021-11-15T23:26:11Z | 2021-11-15T23:26:11Z | OWNER | I'm happy with this as the goals for 1.0. I'm going to close this issue and create three tracking tickets for the three key themes: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Decide what goes into Datasette 1.0 459590021 | |
968470212 | https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968470212 | https://api.github.com/repos/simonw/sqlite-utils/issues/329 | IC_kwDOCGYnMM45ua7E | simonw 9599 | 2021-11-15T02:49:28Z | 2021-11-15T02:49:28Z | OWNER | I was going to replace all of the I'll keep the This avoids the whole issue of needing to rewrite parameters, and solves the immediate problem which is consuming CSV files with bad column names. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Rethink approach to [ and ] in column names (currently throws error) 1005891028 | |
968458837 | https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968458837 | https://api.github.com/repos/simonw/sqlite-utils/issues/329 | IC_kwDOCGYnMM45uYJV | simonw 9599 | 2021-11-15T02:21:15Z | 2021-11-15T02:21:15Z | OWNER | I'm not going to implement a fix that rewrites the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Rethink approach to [ and ] in column names (currently throws error) 1005891028 | |
968453129 | https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968453129 | https://api.github.com/repos/simonw/sqlite-utils/issues/329 | IC_kwDOCGYnMM45uWwJ | simonw 9599 | 2021-11-15T02:07:46Z | 2021-11-15T02:07:46Z | OWNER | If I replace What should the following do?
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Rethink approach to [ and ] in column names (currently throws error) 1005891028 | |
968451954 | https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968451954 | https://api.github.com/repos/simonw/sqlite-utils/issues/329 | IC_kwDOCGYnMM45uWdy | simonw 9599 | 2021-11-15T02:05:29Z | 2021-11-15T02:05:29Z | OWNER |
I'm not going to do this, it's unnecessary extra complexity and it means the function that fixes the column names needs to have access to the current |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Rethink approach to [ and ] in column names (currently throws error) 1005891028 | |
968450579 | https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968450579 | https://api.github.com/repos/simonw/sqlite-utils/issues/339 | IC_kwDOCGYnMM45uWIT | simonw 9599 | 2021-11-15T02:02:34Z | 2021-11-15T02:02:34Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`table.lookup()` option to populate additional columns when creating a record 1053122092 | ||
968435041 | https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968435041 | https://api.github.com/repos/simonw/sqlite-utils/issues/339 | IC_kwDOCGYnMM45uSVh | simonw 9599 | 2021-11-15T01:44:42Z | 2021-11-15T01:44:42Z | OWNER |
Maybe this:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`table.lookup()` option to populate additional columns when creating a record 1053122092 | |
968434594 | https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968434594 | https://api.github.com/repos/simonw/sqlite-utils/issues/339 | IC_kwDOCGYnMM45uSOi | simonw 9599 | 2021-11-15T01:43:10Z | 2021-11-15T01:43:10Z | OWNER | What should I call this parameter? Django has a similar feature where it calls them |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`table.lookup()` option to populate additional columns when creating a record 1053122092 | |
968434425 | https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968434425 | https://api.github.com/repos/simonw/sqlite-utils/issues/339 | IC_kwDOCGYnMM45uSL5 | simonw 9599 | 2021-11-15T01:42:36Z | 2021-11-15T01:42:36Z | OWNER | Here's the current signature of I'm going to add a second positional argument which can provide a dictionary of column->value to use when creating the original table and populating the initial row. If the row already exists, those columns will be ignored entirely. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`table.lookup()` option to populate additional columns when creating a record 1053122092 | |
968401459 | https://github.com/simonw/sqlite-utils/pull/322#issuecomment-968401459 | https://api.github.com/repos/simonw/sqlite-utils/issues/322 | IC_kwDOCGYnMM45uKIz | simonw 9599 | 2021-11-15T00:26:42Z | 2021-11-15T00:26:42Z | OWNER | This relates to the fact that dictionaries, lists and tuples get special treatment and are converted to JSON strings, using this code: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L2937-L2947 So the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add dict type to be mapped as TEXT in sqllite 979612115 | |
968384988 | https://github.com/simonw/sqlite-utils/pull/324#issuecomment-968384988 | https://api.github.com/repos/simonw/sqlite-utils/issues/324 | IC_kwDOCGYnMM45uGHc | simonw 9599 | 2021-11-14T23:25:16Z | 2021-11-14T23:25:16Z | OWNER | Yes this was absolutely the intention! Thanks, I wonder how often I've made that mistake in other projects? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Use python-dateutil package instead of dateutils 988013247 | |
968384005 | https://github.com/simonw/sqlite-utils/issues/331#issuecomment-968384005 | https://api.github.com/repos/simonw/sqlite-utils/issues/331 | IC_kwDOCGYnMM45uF4F | simonw 9599 | 2021-11-14T23:19:29Z | 2021-11-14T23:20:32Z | OWNER | Tested it like this, against a freshly built My if name == "main":
db = sqlite_utils.Database(memory=True)
table = cast(sqlite_utils.db.Table, db["foo"])
table.insert({"id": 5})
print(list(db.query("select * from foo")))
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mypy error: found module but no type hints or library stubs 1026794056 | |
968381939 | https://github.com/simonw/sqlite-utils/issues/331#issuecomment-968381939 | https://api.github.com/repos/simonw/sqlite-utils/issues/331 | IC_kwDOCGYnMM45uFXz | simonw 9599 | 2021-11-14T23:06:20Z | 2021-11-14T23:06:20Z | OWNER | Thanks - I didn't know this was needed! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mypy error: found module but no type hints or library stubs 1026794056 | |
968380675 | https://github.com/simonw/sqlite-utils/issues/332#issuecomment-968380675 | https://api.github.com/repos/simonw/sqlite-utils/issues/332 | IC_kwDOCGYnMM45uFED | simonw 9599 | 2021-11-14T22:57:56Z | 2021-11-14T22:57:56Z | OWNER | This is a great idea. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`sqlite-utils memory --flatten` option to flatten nested JSON 1028056713 | |
968380387 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968380387 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45uE_j | simonw 9599 | 2021-11-14T22:55:56Z | 2021-11-14T22:55:56Z | OWNER | OK, this should fix it. |
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968371112 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968371112 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45uCuo | simonw 9599 | 2021-11-14T21:57:43Z | 2021-11-14T22:21:31Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968361671 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968361671 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45uAbH | simonw 9599 | 2021-11-14T20:54:53Z | 2021-11-14T21:01:14Z | OWNER | I'm leaning towards Or "ignore" sounds like it might not create the index if the name exists, but we want to still create the index but pick a new name. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968362285 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968362285 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45uAkt | simonw 9599 | 2021-11-14T20:59:44Z | 2021-11-14T20:59:44Z | OWNER | I think I'll attempt to create the index and re-try if it fails with that error. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968362214 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968362214 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45uAjm | simonw 9599 | 2021-11-14T20:59:15Z | 2021-11-14T20:59:15Z | OWNER | How to figure out if an index name is already in use?
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968361409 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968361409 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45uAXB | simonw 9599 | 2021-11-14T20:52:55Z | 2021-11-14T20:52:55Z | OWNER | Looking at the method signature: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L1518-L1524
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968361285 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968361285 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45uAVF | simonw 9599 | 2021-11-14T20:51:57Z | 2021-11-14T20:51:57Z | OWNER | SQLite will happily create multiple identical indexes on a table, using more disk space each time: ```pycon <Table t (id)> # dupes.db is 98304 bytes >>> db["t"].create_index(["id"]) <Table t (id)> # dupes.db is 204800 bytes >>> db["t"].indexes [Index(seq=0, name='idx_t_id', unique=0, origin='c', partial=0, columns=['id'])] >>> db["t"].create_index(["id"], index_name="t_idx_t_id_2") <Table t (id)> # 311296 bytes >>> db["t"].create_index(["id"], index_name="t_idx_t_id_3") <Table t (id)> # 417792 bytes >>> db.vacuum() # Still 417792 bytes ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968360538 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968360538 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45uAJa | simonw 9599 | 2021-11-14T20:46:56Z | 2021-11-14T20:46:56Z | OWNER | I'm tempted to not provide an opt-out option either: if you call But... it feels wasteful to create an index that exactly duplicates an existing index. Would SQLite even let you do that or would it notice and NOT double the amount of disk space used for that index? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968360387 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968360387 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45uAHD | simonw 9599 | 2021-11-14T20:45:44Z | 2021-11-14T20:45:44Z | OWNER | What would such an option be called? Some options:
If the user doesn't pass in an explicit name it seems like their intent is "just create me the index, I don't care what name you use" - so actually perhaps the default behaviour here should be to pick a new unique name if that name is already in use. Then maybe there should be an option to disable that - some options there:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968359868 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968359868 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45t_-8 | simonw 9599 | 2021-11-14T20:41:42Z | 2021-11-14T20:41:42Z | OWNER | The "index idx_generators_eia860_report_date already exists" error suggests that the problem here is actually one of an index name collision.
So perhaps |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968359137 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968359137 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45t_zh | simonw 9599 | 2021-11-14T20:37:00Z | 2021-11-14T20:37:00Z | OWNER | This is strange - the code already checks that an index doesn't exist before attempting to create it: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L893-L902 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968210842 | https://github.com/simonw/datasette/issues/1507#issuecomment-968210842 | https://api.github.com/repos/simonw/datasette/issues/1507 | IC_kwDOBm6k_c45tbma | simonw 9599 | 2021-11-14T05:41:55Z | 2021-11-14T05:41:55Z | OWNER | Here's the build with that fix: https://readthedocs.org/projects/datasette/builds/15268498/ It passed and published the docs: https://docs.datasette.io/en/latest/changelog.html |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
ReadTheDocs build failed for 0.59.2 release 1052851176 | |
968210222 | https://github.com/simonw/datasette/issues/1507#issuecomment-968210222 | https://api.github.com/repos/simonw/datasette/issues/1507 | IC_kwDOBm6k_c45tbcu | simonw 9599 | 2021-11-14T05:34:14Z | 2021-11-14T05:34:14Z | OWNER | Here's the new build using Python 3: https://readthedocs.org/projects/datasette/builds/15268482/ It's still broken. Here's one of many issue threads about it, this one has a workaround fix: https://github.com/readthedocs/readthedocs.org/issues/8616#issuecomment-952034858
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
ReadTheDocs build failed for 0.59.2 release 1052851176 | |
968209957 | https://github.com/simonw/datasette/issues/1507#issuecomment-968209957 | https://api.github.com/repos/simonw/datasette/issues/1507 | IC_kwDOBm6k_c45tbYl | simonw 9599 | 2021-11-14T05:31:07Z | 2021-11-14T05:31:07Z | OWNER | Looks like ReadTheDocs builds started failing for |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
ReadTheDocs build failed for 0.59.2 release 1052851176 | |
968209731 | https://github.com/simonw/datasette/issues/1507#issuecomment-968209731 | https://api.github.com/repos/simonw/datasette/issues/1507 | IC_kwDOBm6k_c45tbVD | simonw 9599 | 2021-11-14T05:28:41Z | 2021-11-14T05:28:41Z | OWNER | I will try adding a This might work: ``` version: 2 build: os: ubuntu-20.04 tools: python: "3.9" sphinx: configuration: docs/conf.py ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
ReadTheDocs build failed for 0.59.2 release 1052851176 | |
968209616 | https://github.com/simonw/datasette/issues/1507#issuecomment-968209616 | https://api.github.com/repos/simonw/datasette/issues/1507 | IC_kwDOBm6k_c45tbTQ | simonw 9599 | 2021-11-14T05:27:22Z | 2021-11-14T05:27:22Z | OWNER | https://blog.readthedocs.com/default-python-3/ they started defaulting new projects to Python 3 back in Feb 2019 but clearly my project was created before then. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
ReadTheDocs build failed for 0.59.2 release 1052851176 | |
968209560 | https://github.com/simonw/datasette/issues/1507#issuecomment-968209560 | https://api.github.com/repos/simonw/datasette/issues/1507 | IC_kwDOBm6k_c45tbSY | simonw 9599 | 2021-11-14T05:26:36Z | 2021-11-14T05:26:36Z | OWNER | It looks like my builds there still run on Python 2!
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
ReadTheDocs build failed for 0.59.2 release 1052851176 | |
968207906 | https://github.com/simonw/datasette/issues/1503#issuecomment-968207906 | https://api.github.com/repos/simonw/datasette/issues/1503 | IC_kwDOBm6k_c45ta4i | simonw 9599 | 2021-11-14T05:08:26Z | 2021-11-14T05:08:26Z | OWNER | Error: ``` def test_table_html_filter_form_column_options( path, expected_column_options, app_client ): response = app_client.get(path) assert response.status == 200 form = Soup(response.body, "html.parser").find("form") column_options = [ o.attrs.get("value") or o.string for o in form.select("select[name=_filter_column] option") ]
My solution: start with the query columns, but then add any table columns that were not already returned by the query to the end of the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`?_nocol=` removes that column from the filter interface 1050163432 | |
968192980 | https://github.com/simonw/datasette/issues/1506#issuecomment-968192980 | https://api.github.com/repos/simonw/datasette/issues/1506 | IC_kwDOBm6k_c45tXPU | simonw 9599 | 2021-11-14T02:22:40Z | 2021-11-14T02:22:40Z | OWNER | I think the answer is to spot this case and link to
So maybe the facet selection rendering logic needs to spot this and link correctly to it? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Columns beginning with an underscore do not facet correctly 1052826038 | |
962411119 | https://github.com/simonw/sqlite-utils/issues/336#issuecomment-962411119 | https://api.github.com/repos/simonw/sqlite-utils/issues/336 | IC_kwDOCGYnMM45XTpv | simonw 9599 | 2021-11-06T07:21:04Z | 2021-11-06T07:21:04Z | OWNER | I've never used |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-util tranform --column-order mangles columns of type "timestamp" 1044267332 | |
955367409 | https://github.com/simonw/sqlite-utils/issues/206#issuecomment-955367409 | https://api.github.com/repos/simonw/sqlite-utils/issues/206 | IC_kwDOCGYnMM448b_x | simonw 9599 | 2021-10-30T15:50:39Z | 2021-10-30T15:50:39Z | OWNER | What's the error message? Sometimes I pipe JSON through
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils should suggest --csv if JSON parsing fails 761915790 | |
953508979 | https://github.com/simonw/datasette/issues/1497#issuecomment-953508979 | https://api.github.com/repos/simonw/datasette/issues/1497 | IC_kwDOBm6k_c441WRz | simonw 9599 | 2021-10-28T05:13:49Z | 2021-10-28T05:13:49Z | OWNER | Wrote about this in my weeknotes: https://simonwillison.net/2021/Oct/28/weeknotes-kubernetes-web-components/ |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Publish to Docker Hub failing with "libcrypt.so.1: cannot open shared object file" 1034535001 | |
950417375 | https://github.com/simonw/datasette/issues/1497#issuecomment-950417375 | https://api.github.com/repos/simonw/datasette/issues/1497 | IC_kwDOBm6k_c44pjff | simonw 9599 | 2021-10-24T23:36:54Z | 2021-10-24T23:36:54Z | OWNER | Tried fixing this by pushing a new |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Publish to Docker Hub failing with "libcrypt.so.1: cannot open shared object file" 1034535001 | |
950416802 | https://github.com/simonw/datasette/issues/1497#issuecomment-950416802 | https://api.github.com/repos/simonw/datasette/issues/1497 | IC_kwDOBm6k_c44pjWi | simonw 9599 | 2021-10-24T23:32:39Z | 2021-10-24T23:32:39Z | OWNER | That's because the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Publish to Docker Hub failing with "libcrypt.so.1: cannot open shared object file" 1034535001 | |
950416682 | https://github.com/simonw/datasette/issues/1497#issuecomment-950416682 | https://api.github.com/repos/simonw/datasette/issues/1497 | IC_kwDOBm6k_c44pjUq | simonw 9599 | 2021-10-24T23:31:51Z | 2021-10-24T23:31:51Z | OWNER | One catch: the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Publish to Docker Hub failing with "libcrypt.so.1: cannot open shared object file" 1034535001 | |
950416659 | https://github.com/simonw/datasette/issues/1497#issuecomment-950416659 | https://api.github.com/repos/simonw/datasette/issues/1497 | IC_kwDOBm6k_c44pjUT | simonw 9599 | 2021-10-24T23:31:41Z | 2021-10-24T23:31:41Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Publish to Docker Hub failing with "libcrypt.so.1: cannot open shared object file" 1034535001 | ||
950416460 | https://github.com/simonw/datasette/issues/1497#issuecomment-950416460 | https://api.github.com/repos/simonw/datasette/issues/1497 | IC_kwDOBm6k_c44pjRM | simonw 9599 | 2021-10-24T23:30:10Z | 2021-10-24T23:30:10Z | OWNER | Testing that newly published image:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Publish to Docker Hub failing with "libcrypt.so.1: cannot open shared object file" 1034535001 | |
950416061 | https://github.com/simonw/datasette/issues/1497#issuecomment-950416061 | https://api.github.com/repos/simonw/datasette/issues/1497 | IC_kwDOBm6k_c44pjK9 | simonw 9599 | 2021-10-24T23:27:18Z | 2021-10-24T23:27:18Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Publish to Docker Hub failing with "libcrypt.so.1: cannot open shared object file" 1034535001 | ||
950415822 | https://github.com/simonw/datasette/issues/1497#issuecomment-950415822 | https://api.github.com/repos/simonw/datasette/issues/1497 | IC_kwDOBm6k_c44pjHO | simonw 9599 | 2021-10-24T23:25:45Z | 2021-10-24T23:25:45Z | OWNER | I'm going to attempt to publish |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Publish to Docker Hub failing with "libcrypt.so.1: cannot open shared object file" 1034535001 | |
950415129 | https://github.com/simonw/datasette/issues/1497#issuecomment-950415129 | https://api.github.com/repos/simonw/datasette/issues/1497 | IC_kwDOBm6k_c44pi8Z | simonw 9599 | 2021-10-24T23:21:33Z | 2021-10-24T23:21:33Z | OWNER | That fixed it! Resulting image is 249MB which is a very slight size reduction (I think previous was 259MB (uncompressed). |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Publish to Docker Hub failing with "libcrypt.so.1: cannot open shared object file" 1034535001 | |
950413185 | https://github.com/simonw/datasette/issues/1497#issuecomment-950413185 | https://api.github.com/repos/simonw/datasette/issues/1497 | IC_kwDOBm6k_c44pieB | simonw 9599 | 2021-10-24T23:16:25Z | 2021-10-24T23:18:30Z | OWNER | Debian stable these days is "bullseye" - https://www.debian.org/releases/ - which has the version of SpatiaLite that I was previously pulling in from Sid: https://packages.debian.org/bullseye/libsqlite3-mod-spatialite So upgrading to the 3.9.7-slim-bullseye base image may help. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Publish to Docker Hub failing with "libcrypt.so.1: cannot open shared object file" 1034535001 | |
950412628 | https://github.com/simonw/datasette/issues/1497#issuecomment-950412628 | https://api.github.com/repos/simonw/datasette/issues/1497 | IC_kwDOBm6k_c44piVU | simonw 9599 | 2021-10-24T23:13:20Z | 2021-10-24T23:13:27Z | OWNER | I think the root cause here is that I'm using a Debian Buster base image and then installing SpatiaLite from Debian unstable (sid) - as described in this comment: https://github.com/simonw/datasette/issues/1249#issuecomment-804309510 That's has worked fine in the past, but Sid is unstable - and this seems to be one of those instabilities. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Publish to Docker Hub failing with "libcrypt.so.1: cannot open shared object file" 1034535001 | |
950411417 | https://github.com/simonw/datasette/issues/1497#issuecomment-950411417 | https://api.github.com/repos/simonw/datasette/issues/1497 | IC_kwDOBm6k_c44piCZ | simonw 9599 | 2021-10-24T23:06:45Z | 2021-10-24T23:11:14Z | OWNER | Same errors with 5 41.46 /usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory5 41.46 dpkg: error processing package libc6:amd64 (--configure):5 41.46 installed libc6:amd64 package post-installation script subprocess returned error exit status 1275 41.47 Errors were encountered while processing:5 41.47 libc6:amd645 41.50 E: Sub-process /usr/bin/dpkg returned an error code (1)
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Publish to Docker Hub failing with "libcrypt.so.1: cannot open shared object file" 1034535001 | |
950411912 | https://github.com/simonw/datasette/issues/1497#issuecomment-950411912 | https://api.github.com/repos/simonw/datasette/issues/1497 | IC_kwDOBm6k_c44piKI | simonw 9599 | 2021-10-24T23:09:41Z | 2021-10-24T23:09:41Z | OWNER | Here that is in the Debian bug tracker: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=993755 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Publish to Docker Hub failing with "libcrypt.so.1: cannot open shared object file" 1034535001 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
issue >30