issue_comments
10,495 rows sorted by updated_at descending
This data as json, CSV (advanced)
user 250
- simonw 5,699
- codecov[bot] 111
- russss 39
- psychemedia 27
- abdusco 26
- aborruso 19
- mroswell 19
- chrismp 18
- jacobian 14
- carlmjohnson 14
- RhetTbull 14
- tballison 13
- tsibley 11
- rixx 11
- terrycojones 10
- stonebig 10
- eyeseast 9
- clausjuhl 9
- bobwhitelock 9
- maxhawkins 8
- UtahDave 8
- tomchristie 8
- brandonrobertz 8
- bsilverm 8
- dracos 7
- rgieseke 7
- mhalle 7
- zeluspudding 7
- cobiadigital 7
- frafra 6
- amjith 6
- simonwiles 6
- wragge 6
- fgregg 6
- rayvoelker 6
- dependabot[bot] 6
- khimaros 5
- jaywgraves 5
- MarkusH 5
- dazzag24 5
- SteadBytes 5
- dependabot-preview[bot] 5
- jayvdb 4
- bollwyvl 4
- jefftriplett 4
- ctb 4
- r4vi 4
- zaneselvans 4
- jungle-boogie 4
- ColinMaudry 4
- kbaikov 4
- JBPressac 4
- nitinpaultifr 4
- henry501 4
- benpickles 3
- frankieroberto 3
- obra 3
- janimo 3
- atomotic 3
- ghing 3
- pkoppstein 3
- yozlet 3
- Btibert3 3
- yschimke 3
- philroche 3
- macropin 3
- camallen 3
- wsxiaoys 3
- lovasoa 3
- jsfenfen 3
- glasnt 3
- xrotwang 3
- robroc 3
- dufferzafar 3
- ashishdotme 3
- Segerberg 3
- Kabouik 3
- blairdrummond 3
- jsancho-gpl 3
- kevindkeogh 3
- daniel-butler 3
- FabianHertwig 3
- polyrand 3
- pjamargh 3
- garethr 2
- danp 2
- ftrain 2
- 20after4 2
- chrishas35 2
- ingenieroariel 2
- coleifer 2
- gavinband 2
- aviflax 2
- tholo 2
- cldellow 2
- frankier 2
- tmaier 2
- gfrmin 2
- frosencrantz 2
- eads 2
- leafgarland 2
- Mjboothaus 2
- n8henrie 2
- soobrosa 2
- nathancahill 2
- betatim 2
- bsmithgall 2
- Florents-Tselai 2
- willingc 2
- nattaylor 2
- durkie 2
- raynae 2
- wulfmann 2
- philshem 2
- bram2000 2
- zzeleznick 2
- nickvazz 2
- aaronyih1 2
- jussiarpalahti 2
- lagolucas 2
- chekos 2
- ad-si 2
- smithdc1 2
- gsajko 2
- null92 2
- rachelmarconi 2
- tunguyenatwork 2
- LVerneyPEReN 2
- anotherjesse 1
- jarib 1
- jokull 1
- dsisnero 1
- gijs 1
- gravis 1
- nkirsch 1
- tomdyson 1
- dkam 1
- furilo 1
- fs111 1
- adamwolf 1
- prabhur 1
- dmd 1
- rubenv 1
- Uninen 1
- carsonyl 1
- nryberg 1
- step21 1
- stefanocudini 1
- rcoup 1
- hpk42 1
- annapowellsmith 1
- aslakr 1
- thorn0 1
- yurivish 1
- jmelloy 1
- Krazybug 1
- dvhthomas 1
- sethvincent 1
- meatcar 1
- aitoehigie 1
- michaelmcandrew 1
- drewda 1
- stiles 1
- saulpw 1
- thadk 1
- robintw 1
- ipmb 1
- mikepqr 1
- 0x1997 1
- davidszotten 1
- kevboh 1
- eaubin 1
- yunzheng 1
- heyarne 1
- simonrjones 1
- mcint 1
- justinpinkney 1
- merwok 1
- virtadpt 1
- snth 1
- joshmgrant 1
- bcongdon 1
- nickdirienzo 1
- adamjonas 1
- hannseman 1
- kaihendry 1
- urbas 1
- brimstone 1
- adamchainz 1
- PabloLerma 1
- heussd 1
- RayBB 1
- limar 1
- drkane 1
- Gagravarr 1
- ligurio 1
- dyllan-to-you 1
- justinallen 1
- wdccdw 1
- progpow 1
- ltrgoddard 1
- costrouc 1
- jratike80 1
- ccorcos 1
- qqilihq 1
- QAInsights 1
- secretGeek 1
- fkuhn 1
- jameslittle230 1
- dskrad 1
- kwladyka 1
- Carib0u 1
- fatihky 1
- phoenixjun 1
- JesperTreetop 1
- bapowell 1
- louispotok 1
- chris48s 1
- plpxsk 1
- ChristopherWilks 1
- Maltazar 1
- eumiro 1
- wuhland 1
- foscoj 1
- dvot197007 1
- kokes 1
- rprimet 1
- metab0t 1
- spdkils 1
- sturzl 1
- robmarkcole 1
- jfeiwell 1
- coisnepe 1
- chmaynard 1
- noklam 1
- GmGniap 1
- rdtq 1
- LucasElArruda 1
- duarteocarmo 1
- mattiaborsoi 1
- sarcasticadmin 1
- abeyerpath 1
- b0b5h4rp13 1
- patricktrainer 1
- justmars 1
- jcmkk3 1
- publicmatt 1
- thisismyfuckingusername 1
- MichaelTiemannOSC 1
- kirajano 1
id | html_url | issue_url | node_id | user | created_at | updated_at ▲ | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
973635157 | https://github.com/simonw/datasette/issues/878#issuecomment-973635157 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46CH5V | simonw 9599 | 2021-11-19T01:07:08Z | 2021-11-19T01:07:08Z | OWNER | This exercise is proving so useful in getting my head around how the enormous and complex Here's where I've got to now - I'm systematically working through the variables that are returned for HTML and for JSON copying across code to get it to work: ```python from datasette.database import QueryInterrupted from datasette.utils import escape_sqlite from datasette.utils.asgi import Response, NotFound, Forbidden from datasette.views.base import DatasetteError from datasette import hookimpl from asyncinject import AsyncInject, inject from pprint import pformat class Table(AsyncInject): @inject async def database(self, request, datasette): # TODO: all that nasty hash resolving stuff can go here db_name = request.url_vars["db_name"] try: db = datasette.databases[db_name] except KeyError: raise NotFound(f"Database '{db_name}' does not exist") return db
@hookimpl def register_routes(): return [ (r"/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", Table().view), ] async def check_permissions(datasette, request, permissions): """permissions is a list of (action, resource) tuples or 'action' strings""" for permission in permissions: if isinstance(permission, str): action = permission resource = None elif isinstance(permission, (tuple, list)) and len(permission) == 2: action, resource = permission else: assert ( False ), "permission should be string or tuple of two items: {}".format( repr(permission) ) ok = await datasette.permission_allowed( request.actor, action, resource=resource, default=None, ) if ok is not None: if ok: return else: raise Forbidden(action) async def columns_to_select(datasette, database, table, request): table_columns = await database.table_columns(table) pks = await database.primary_keys(table) columns = list(table_columns) if "_col" in request.args: columns = list(pks) _cols = request.args.getlist("_col") bad_columns = [column for column in _cols if column not in table_columns] if bad_columns: raise DatasetteError( "_col={} - invalid columns".format(", ".join(bad_columns)), status=400, ) # De-duplicate maintaining order: columns.extend(dict.fromkeys(_cols)) if "_nocol" in request.args: # Return all columns EXCEPT these bad_columns = [ column for column in request.args.getlist("_nocol") if (column not in table_columns) or (column in pks) ] if bad_columns: raise DatasetteError( "_nocol={} - invalid columns".format(", ".join(bad_columns)), status=400, ) tmp_columns = [ column for column in columns if column not in request.args.getlist("_nocol") ] columns = tmp_columns return columns ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
973568285 | https://github.com/simonw/datasette/issues/878#issuecomment-973568285 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46B3kd | simonw 9599 | 2021-11-19T00:29:20Z | 2021-11-19T00:29:20Z | OWNER | This is working! ```python from datasette.utils.asgi import Response from datasette import hookimpl import html from asyncinject import AsyncInject, inject class Table(AsyncInject): @inject async def database(self, request): return request.url_vars["db_name"]
@hookimpl
def register_routes():
return [
(r"/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", Table().view),
]
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
973564260 | https://github.com/simonw/datasette/issues/878#issuecomment-973564260 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46B2lk | simonw 9599 | 2021-11-19T00:27:06Z | 2021-11-19T00:27:06Z | OWNER | Problem: the fancy @hookimpl def register_routes(): return [ (r"/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", Table().view), ] ``` This failed with error: "Table.view() takes 1 positional argument but 2 were given" So I'm going to use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
973554024 | https://github.com/simonw/datasette/issues/878#issuecomment-973554024 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46B0Fo | simonw 9599 | 2021-11-19T00:21:20Z | 2021-11-19T00:21:20Z | OWNER | That's annoying: it looks like plugins can't use async def table(request): return Response.html("Hello from {}".format( html.escape(repr(request.url_vars)) )) @hookimpl
def register_routes():
return [
(r"/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)", table),
]
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
973542284 | https://github.com/simonw/datasette/issues/878#issuecomment-973542284 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46BxOM | simonw 9599 | 2021-11-19T00:16:44Z | 2021-11-19T00:16:44Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
973527870 | https://github.com/simonw/datasette/issues/878#issuecomment-973527870 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c46Bts- | simonw 9599 | 2021-11-19T00:13:43Z | 2021-11-19T00:13:43Z | OWNER | New plan: I'm going to build a brand new implementation of It will reuse the existing HTML template but will be a completely new Python implementation, based on I'm going to start by just getting the table to show up on the page - then I'll add faceting, suggested facets, filters and so-on. Bonus: I'm going to see if I can get it to work for arbitrary SQL queries too (stretch goal). |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
972858458 | https://github.com/simonw/datasette/pull/1516#issuecomment-972858458 | https://api.github.com/repos/simonw/datasette/issues/1516 | IC_kwDOBm6k_c45_KRa | codecov[bot] 22429695 | 2021-11-18T13:19:01Z | 2021-11-18T13:19:01Z | NONE | Codecov Report
```diff @@ Coverage Diff @@ main #1516 +/-=======================================
Coverage 91.82% 91.82% Continue to review full report at Codecov.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Bump black from 21.9b0 to 21.11b1 1057340779 | |
972852184 | https://github.com/simonw/datasette/pull/1514#issuecomment-972852184 | https://api.github.com/repos/simonw/datasette/issues/1514 | IC_kwDOBm6k_c45_IvY | dependabot[bot] 49699333 | 2021-11-18T13:11:15Z | 2021-11-18T13:11:15Z | CONTRIBUTOR | Superseded by #1516. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Bump black from 21.9b0 to 21.11b0 1056117435 | |
971575746 | https://github.com/simonw/datasette/pull/1514#issuecomment-971575746 | https://api.github.com/repos/simonw/datasette/issues/1514 | IC_kwDOBm6k_c456RHC | codecov[bot] 22429695 | 2021-11-17T13:18:58Z | 2021-11-17T13:18:58Z | NONE | Codecov Report
```diff @@ Coverage Diff @@ main #1514 +/-=======================================
Coverage 91.82% 91.82% Continue to review full report at Codecov.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Bump black from 21.9b0 to 21.11b0 1056117435 | |
971568829 | https://github.com/simonw/datasette/pull/1500#issuecomment-971568829 | https://api.github.com/repos/simonw/datasette/issues/1500 | IC_kwDOBm6k_c456Pa9 | dependabot[bot] 49699333 | 2021-11-17T13:13:58Z | 2021-11-17T13:13:58Z | CONTRIBUTOR | Superseded by #1514. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Bump black from 21.9b0 to 21.10b0 1041158024 | |
971209475 | https://github.com/simonw/datasette/issues/878#issuecomment-971209475 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c4543sD | simonw 9599 | 2021-11-17T05:41:42Z | 2021-11-17T05:41:42Z | OWNER | I'm going to build a brand new implementation of the I can maybe even run the tests against old |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
971057553 | https://github.com/simonw/datasette/issues/878#issuecomment-971057553 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c454SmR | simonw 9599 | 2021-11-17T01:40:45Z | 2021-11-17T01:40:45Z | OWNER | I shipped that code as a new library, |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
971056169 | https://github.com/simonw/datasette/pull/1512#issuecomment-971056169 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c454SQp | simonw 9599 | 2021-11-17T01:39:44Z | 2021-11-17T01:39:44Z | OWNER | Closing this PR because I shipped the code in it as a separate library instead. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
971055677 | https://github.com/simonw/datasette/pull/1512#issuecomment-971055677 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c454SI9 | simonw 9599 | 2021-11-17T01:39:25Z | 2021-11-17T01:39:25Z | OWNER | https://github.com/simonw/asyncinject version 0.1a0 is now live on PyPI: https://pypi.org/project/asyncinject/ |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
971010724 | https://github.com/simonw/datasette/pull/1512#issuecomment-971010724 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c454HKk | simonw 9599 | 2021-11-17T01:12:22Z | 2021-11-17T01:12:22Z | OWNER | I'm going to extract out the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
970718652 | https://github.com/simonw/datasette/pull/1512#issuecomment-970718652 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c452_28 | codecov[bot] 22429695 | 2021-11-16T22:02:59Z | 2021-11-16T23:51:48Z | NONE | Codecov Report
```diff @@ Coverage Diff @@ main #1512 +/-==========================================
- Coverage 91.82% 89.72% -2.11% | Impacted Files | Coverage Δ | |
|---|---|---|
| datasette/utils/vendored_graphlib.py | Continue to review full report at Codecov.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
970861628 | https://github.com/simonw/datasette/pull/1512#issuecomment-970861628 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c453iw8 | simonw 9599 | 2021-11-16T23:46:07Z | 2021-11-16T23:46:07Z | OWNER | I made the changes locally and tested them with Python 3.6 like so:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
970857411 | https://github.com/simonw/datasette/pull/1512#issuecomment-970857411 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c453hvD | simonw 9599 | 2021-11-16T23:43:21Z | 2021-11-16T23:43:21Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
970855084 | https://github.com/simonw/datasette/issues/1513#issuecomment-970855084 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453hKs | simonw 9599 | 2021-11-16T23:41:46Z | 2021-11-16T23:41:46Z | OWNER | Conclusion: using a giant convoluted CTE and UNION ALL query to attempt to calculate facets at the same time as retrieving rows is a net LOSS for performance! Very surprised to see that. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970853917 | https://github.com/simonw/datasette/issues/1513#issuecomment-970853917 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453g4d | simonw 9599 | 2021-11-16T23:41:01Z | 2021-11-16T23:41:01Z | OWNER | One very interesting difference between the two: on the single giant query page:
The single big query takes 376ms total to render the page, spending 370ms in 5 queries Those 5 queries, if you're interested```sql select database_name, schema_version from databases PRAGMA schema_version PRAGMA schema_version explain with cte as (\r\n select rowid, date, county, state, fips, cases, deaths\r\n from ny_times_us_counties\r\n),\r\ntruncated as (\r\n select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths\r\n from cte order by date desc limit 4\r\n),\r\nstate_facet as (\r\n select 'state' as _facet, state as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\nfips_facet as (\r\n select 'fips' as _facet, fips as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\ncounty_facet as (\r\n select 'county' as _facet, county as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n)\r\nselect * from truncated\r\nunion all select * from state_facet\r\nunion all select * from fips_facet\r\nunion all select * from county_facet with cte as (\r\n select rowid, date, county, state, fips, cases, deaths\r\n from ny_times_us_counties\r\n),\r\ntruncated as (\r\n select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths\r\n from cte order by date desc limit 4\r\n),\r\nstate_facet as (\r\n select 'state' as _facet, state as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\nfips_facet as (\r\n select 'fips' as _facet, fips as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n),\r\ncounty_facet as (\r\n select 'county' as _facet, county as facet_name, count(*) as facet_count,\r\n null, null, null, null, null, null, null\r\n from cte group by facet_name order by facet_count desc limit 3\r\n)\r\nselect * from truncated\r\nunion all select * from state_facet\r\nunion all select * from fips_facet\r\nunion all select * from county_facet ```All of that additional non-SQL overhead must be stuff relating to Python and template rendering code running on the page. I'm really surprised at how much overhead that is! This is worth researching separately. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970845844 | https://github.com/simonw/datasette/issues/1513#issuecomment-970845844 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453e6U | simonw 9599 | 2021-11-16T23:35:38Z | 2021-11-16T23:35:38Z | OWNER | I tried adding
Compared to: Which is 353ms total. The separate queries ran faster! Really surprising result there. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970828568 | https://github.com/simonw/datasette/issues/1513#issuecomment-970828568 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453asY | simonw 9599 | 2021-11-16T23:27:11Z | 2021-11-16T23:27:11Z | OWNER | One last experiment: I'm going to try running an expensive query in the CTE portion. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970827674 | https://github.com/simonw/datasette/issues/1513#issuecomment-970827674 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453aea | simonw 9599 | 2021-11-16T23:26:58Z | 2021-11-16T23:26:58Z | OWNER | With trace. https://covid-19.datasettes.com/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2&_trace=1 shows the following:
It didn't run a count because that's the homepage and the count is cached. So I dropped the count from the query and ran it: https://covid-19.datasettes.com/covid?sql=with+cte+as+(%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A)%2C%0D%0Atruncated+as+(%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A)%2C%0D%0Astate_facet+as+(%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count()+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Afips_facet+as+(%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count()+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Acounty_facet+as+(%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count()+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%0D%0Aselect++from+truncated%0D%0Aunion+all+select++from+state_facet%0D%0Aunion+all+select++from+fips_facet%0D%0Aunion+all+select+*+from+county_facet&_trace=1 Shows 649.4359889999259 ms for the query - compared to 755.78843400001ms for the separate. So it saved about 100ms. Still not a huge difference though! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970780866 | https://github.com/simonw/datasette/issues/1513#issuecomment-970780866 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453PDC | simonw 9599 | 2021-11-16T23:01:57Z | 2021-11-16T23:01:57Z | OWNER | One disadvantage to this approach: if you have a SQL time limit of 1s and it takes 0.9s to return the rows but then 0.5s to calculate each of the requested facets the entire query will exceed the time limit. Could work around this by catching that error and then re-running the query just for the rows, but that would result in the user having to wait longer for the results. Could try to remember if that has happened using an in-memory Python data structure and skip the faceting optimization if it's caused problems in the past? That seems a bit gross. Maybe this becomes an opt-in optimization you can request in your What if we kept the query that returns the rows to be displayed on the page separate from the facets, but then executed all of the facets together using this method such that the Maybe a better optimization would be to move facets to happening via |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970766486 | https://github.com/simonw/datasette/issues/1513#issuecomment-970766486 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453LiW | simonw 9599 | 2021-11-16T22:52:56Z | 2021-11-16T22:56:07Z | OWNER | https://covid-19.datasettes.com/covid is 805.2MB https://covid-19.datasettes.com/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2 Equivalent SQL: https://covid-19.datasettes.com/covid?sql=with+cte+as+%28%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A%29%2C%0D%0Astate_facet+as+%28%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Afips_facet+as+%28%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Acounty_facet+as+%28%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Atotal_count+as+%28%0D%0A++select+%27COUNT%27+as+_facet%2C+%27%27+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte%0D%0A%29%0D%0Aselect++from+truncated%0D%0Aunion+all+select++from+state_facet%0D%0Aunion+all+select++from+fips_facet%0D%0Aunion+all+select++from+county_facet%0D%0Aunion+all+select+*+from+total_count
_facet | facet_name | facet_count | rowid | date | county | state | fips | cases | deaths -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | | | 1917344 | 2021-11-15 | Autauga | Alabama | 1001 | 10407 | 154 | | | 1917345 | 2021-11-15 | Baldwin | Alabama | 1003 | 37875 | 581 | | | 1917346 | 2021-11-15 | Barbour | Alabama | 1005 | 3648 | 79 | | | 1917347 | 2021-11-15 | Bibb | Alabama | 1007 | 4317 | 92 state | Texas | 148028 | | | | | | | state | Georgia | 96249 | | | | | | | state | Virginia | 79315 | | | | | | | fips | | 17580 | | | | | | | fips | 53061 | 665 | | | | | | | fips | 17031 | 662 | | | | | | | county | Washington | 18666 | | | | | | | county | Unknown | 15840 | | | | | | | county | Jefferson | 15637 | | | | | | | COUNT | | 1920593 | | | | | | | |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970770304 | https://github.com/simonw/datasette/issues/1513#issuecomment-970770304 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453MeA | simonw 9599 | 2021-11-16T22:55:19Z | 2021-11-16T22:55:19Z | OWNER | (One thing I really like about this pattern is that it should work exactly the same when used to facet the results of arbitrary SQL queries as it does when faceting results from the table page.) |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970767952 | https://github.com/simonw/datasette/issues/1513#issuecomment-970767952 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453L5Q | simonw 9599 | 2021-11-16T22:53:52Z | 2021-11-16T22:53:52Z | OWNER | It's going to take another 15 minutes for the build to finish and deploy the version with |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970758179 | https://github.com/simonw/datasette/issues/1513#issuecomment-970758179 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453Jgj | simonw 9599 | 2021-11-16T22:47:38Z | 2021-11-16T22:47:38Z | OWNER | Trace now enabled: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet_size=3&_size=2&_nocount=1&_trace=1 Here are the relevant traces:
I modified the query to include the total count as well: https://global-power-plants.datasettes.com/global-power-plants?sql=with+cte+as+%28%0D%0A++select+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+cte+order+by+rowid+limit+4%0D%0A%29%2C%0D%0Acountry_long_facet+as+%28%0D%0A++select+%27country_long%27+as+_facet%2C+country_long+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aowner_facet+as+%28%0D%0A++select+%27owner%27+as+_facet%2C+owner+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aprimary_fuel_facet+as+%28%0D%0A++select+%27primary_fuel%27+as+_facet%2C+primary_fuel+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Atotal_count+as+%28%0D%0A++select+%27COUNT%27+as+_facet%2C+%27%27+as+facet_name%2C+count%28%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte%0D%0A%29%0D%0Aselect++from+truncated%0D%0Aunion+all+select++from+country_long_facet%0D%0Aunion+all+select++from+owner_facet%0D%0Aunion+all+select++from+primary_fuel_facet%0D%0Aunion+all+select+*+from+total_count&_trace=1
To my huge surprise, this convoluted optimization only shaves the sum query time down from 37.8ms to 34.8ms! That entire database file is just 11.1 MB though. Maybe it would make a meaningful difference on something larger? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970742415 | https://github.com/simonw/datasette/issues/1513#issuecomment-970742415 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453FqP | simonw 9599 | 2021-11-16T22:37:14Z | 2021-11-16T22:37:14Z | OWNER | The query takes 42.794ms to run. Here's the equivalent page using separate queries: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet_size=3&_size=2&_nocount=1 Annoyingly I can't disable facet suggestions but keep facets. I'm going to turn on tracing so I can see how long the separate queries took. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970738130 | https://github.com/simonw/datasette/issues/1513#issuecomment-970738130 | https://api.github.com/repos/simonw/datasette/issues/1513 | IC_kwDOBm6k_c453EnS | simonw 9599 | 2021-11-16T22:32:19Z | 2021-11-16T22:32:19Z | OWNER | I came up with the following query which seems to work!
Results look like this: _facet | facet_name | facet_count | rowid | country | country_long | name | owner | primary_fuel -- | -- | -- | -- | -- | -- | -- | -- | -- | | | 1 | AFG | Afghanistan | Kajaki Hydroelectric Power Plant Afghanistan | | Hydro | | | 2 | AFG | Afghanistan | Kandahar DOG | | Solar | | | 3 | AFG | Afghanistan | Kandahar JOL | | Solar | | | 4 | AFG | Afghanistan | Mahipar Hydroelectric Power Plant Afghanistan | | Hydro country_long | United States of America | 8688 | | | | | | country_long | China | 4235 | | | | | | country_long | United Kingdom | 2603 | | | | | | owner | | 14112 | | | | | | owner | Lightsource Renewable Energy | 120 | | | | | | owner | Cypress Creek Renewables | 109 | | | | | | primary_fuel | Solar | 9662 | | | | | | primary_fuel | Hydro | 7155 | | | | | | primary_fuel | Wind | 5188 | | | | | | This is a neat proof of concept. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: CTEs and union all to calculate facets AND query at the same time 1055469073 | |
970718337 | https://github.com/simonw/datasette/pull/1512#issuecomment-970718337 | https://api.github.com/repos/simonw/datasette/issues/1512 | IC_kwDOBm6k_c452_yB | simonw 9599 | 2021-11-16T22:02:30Z | 2021-11-16T22:02:30Z | OWNER | I've decided to make the clever class Simple(AsyncBase): def init(self): self.log = []
class Complex(AsyncBase): inject_all = True
``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for async view classes 1055402144 | |
970712713 | https://github.com/simonw/datasette/issues/878#issuecomment-970712713 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c452-aJ | simonw 9599 | 2021-11-16T21:54:33Z | 2021-11-16T21:54:33Z | OWNER | I'm going to continue working on this in a PR. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970705738 | https://github.com/simonw/datasette/issues/878#issuecomment-970705738 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c4528tK | simonw 9599 | 2021-11-16T21:44:31Z | 2021-11-16T21:44:31Z | OWNER | Wrote a TIL about what I learned using |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970673085 | https://github.com/simonw/datasette/issues/878#issuecomment-970673085 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c4520u9 | simonw 9599 | 2021-11-16T20:58:24Z | 2021-11-16T20:58:24Z | OWNER | New test: ```python class Complex(AsyncBase): def init(self): self.log = []
@pytest.mark.asyncio
async def test_complex():
result = await Complex().go()
# 'c' should only be called once
assert tuple(result) in (
# c and d could happen in either order
("c", "d", "b", "a", "go"),
("d", "c", "b", "a", "go"),
)
try: import graphlib except ImportError: from . import vendored_graphlib as graphlib class AsyncMeta(type): def new(cls, name, bases, attrs): # Decorate any items that are 'async def' methods registry = {} new_attrs = {"_registry": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.__name__ == "resolve": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value # Gather graph for later dependency resolution graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != "self" and not p.startswith("") } for key, method in _registry.items() } new_attrs["_graph"] = graph return super().new(cls, name, bases, new_attrs) def make_method(method): parameters = inspect.signature(method).parameters.keys()
class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names, results=None): print("\n resolve: ", names) if results is None: results = {}
``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970660299 | https://github.com/simonw/datasette/issues/878#issuecomment-970660299 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c452xnL | simonw 9599 | 2021-11-16T20:39:43Z | 2021-11-16T20:42:27Z | OWNER | But that does seem to be the plan that ts = TopologicalSorter(graph)
ts.prepare()
while ts.is_active():
nodes = ts.get_ready()
print(nodes)
ts.done(*nodes)
ts = TopologicalSorter(graph)
ts.prepare()
while ts.is_active():
nodes = ts.get_ready()
print(nodes)
ts.done(nodes)
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970657874 | https://github.com/simonw/datasette/issues/878#issuecomment-970657874 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c452xBS | simonw 9599 | 2021-11-16T20:36:01Z | 2021-11-16T20:36:01Z | OWNER | My goal here is to calculate the most efficient way to resolve the different nodes, running them in parallel where possible. So for this class: ```python class Complex(AsyncBase): async def d(self): pass
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970655927 | https://github.com/simonw/datasette/issues/878#issuecomment-970655927 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c452wi3 | simonw 9599 | 2021-11-16T20:33:11Z | 2021-11-16T20:33:11Z | OWNER | What should be happening here instead is it should resolve the full graph and notice that So maybe the algorithm I'm inheriting from https://docs.python.org/3/library/graphlib.html isn't the correct algorithm? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970655304 | https://github.com/simonw/datasette/issues/878#issuecomment-970655304 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c452wZI | simonw 9599 | 2021-11-16T20:32:16Z | 2021-11-16T20:32:16Z | OWNER | This code is really fiddly. I just got to this version: ```python import asyncio from functools import wraps import inspect try: import graphlib except ImportError: from . import vendored_graphlib as graphlib class AsyncMeta(type): def new(cls, name, bases, attrs): # Decorate any items that are 'async def' methods registry = {} new_attrs = {"_registry": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.__name__ == "resolve": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value # Gather graph for later dependency resolution graph = { key: { p for p in inspect.signature(method).parameters.keys() if p != "self" and not p.startswith("") } for key, method in _registry.items() } new_attrs["_graph"] = graph return super().new(cls, name, bases, new_attrs) def make_method(method): @wraps(method) async def inner(self, _results=None, kwargs): print("inner - _results=", _results) parameters = inspect.signature(method).parameters.keys() # Any parameters not provided by kwargs are resolved from registry to_resolve = [p for p in parameters if p not in kwargs and p != "self"] missing = [p for p in to_resolve if p not in self._registry] assert ( not missing ), "The following DI parameters could not be found in the registry: {}".format( missing ) results = {} results.update(kwargs) if to_resolve: resolved_parameters = await self.resolve(to_resolve, _results) results.update(resolved_parameters) return_value = await method(self, results) if _results is not None: _results[method.name] = return_value return return_value
class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names, results=None): print("\n resolve: ", names) if results is None: results = {}
@pytest.mark.asyncio async def test_complex(): result = await Complex().go() # 'c' should only be called once assert result == ["c", "b", "a", "go"] ``` This test sometimes passes, and sometimes fails! Output for a pass: ``` tests/test_asyncdi.py inner - _results= None resolve: ['a'] ts.get_ready() returned nodes: ('c', 'b') resolve_nodes ('c', 'b') (current results = {}) awaitables: [<coroutine object Complex.c at 0x1074ac890>, <coroutine object Complex.b at 0x1074ac820>] inner - _results= {} LOG: c inner - _results= {'c': None} resolve: ['c']
ts.get_ready() returned nodes: ('c',)
resolve_nodes ('c',)
(current results = {'c': None})
awaitables: []
End of resolve(), returning {'c': None}
LOG: b
ts.get_ready() returned nodes: ('a',)
resolve_nodes ('a',)
(current results = {'c': None, 'b': None})
awaitables: [<coroutine object Complex.a at 0x1074ac7b0>]
inner - _results= {'c': None, 'b': None}
LOG: a
End of resolve(), returning {'c': None, 'b': None, 'a': None}
LOG: go
resolve: ['a'] ts.get_ready() returned nodes: ('b', 'c') resolve_nodes ('b', 'c') (current results = {}) awaitables: [<coroutine object Complex.b at 0x10923c890>, <coroutine object Complex.c at 0x10923c820>] inner - _results= {} resolve: ['c'] ts.get_ready() returned nodes: ('c',) resolve_nodes ('c',) (current results = {}) awaitables: [<coroutine object Complex.c at 0x10923c6d0>] inner - _results= {} LOG: c inner - _results= {'c': None} LOG: c End of resolve(), returning {'c': None} LOG: b ts.get_ready() returned nodes: ('a',) resolve_nodes ('a',) (current results = {'c': None, 'b': None}) awaitables: [<coroutine object Complex.a at 0x10923c6d0>] inner - _results= {'c': None, 'b': None} LOG: a End of resolve(), returning {'c': None, 'b': None, 'a': None} LOG: go F =================================================================================================== FAILURES =================================================================================================== _______________ test_complex _________________
tests/test_asyncdi.py:48: AssertionError ================== short test summary info ================================ FAILED tests/test_asyncdi.py::test_complex - AssertionError: assert ['c', 'c', 'b', 'a', 'go'] == ['c', 'b', 'a', 'go'] ``` I figured out why this is happening.
The code decides to run If If |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970624197 | https://github.com/simonw/datasette/issues/878#issuecomment-970624197 | https://api.github.com/repos/simonw/datasette/issues/878 | IC_kwDOBm6k_c452ozF | simonw 9599 | 2021-11-16T19:49:05Z | 2021-11-16T19:49:05Z | OWNER | Here's the latest version of my weird dependency injection async class: ```python import inspect class AsyncMeta(type): def new(cls, name, bases, attrs): # Decorate any items that are 'async def' methods _registry = {} new_attrs = {"_registry": _registry} for key, value in attrs.items(): if inspect.iscoroutinefunction(value) and not value.name == "resolve": new_attrs[key] = make_method(value) _registry[key] = new_attrs[key] else: new_attrs[key] = value
def make_method(method): @wraps(method) async def inner(self, kwargs): parameters = inspect.signature(method).parameters.keys() # Any parameters not provided by kwargs are resolved from registry to_resolve = [p for p in parameters if p not in kwargs and p != "self"] missing = [p for p in to_resolve if p not in self._registry] assert ( not missing ), "The following DI parameters could not be found in the registry: {}".format( missing ) results = {} results.update(kwargs) results.update(await self.resolve(to_resolve)) return await method(self, results)
bad = [0] class AsyncBase(metaclass=AsyncMeta): async def resolve(self, names): print(" resolve({})".format(names)) results = {} # Resolve them in the correct order ts = TopologicalSorter() ts2 = TopologicalSorter() print(" names = ", names) print(" self._graph = ", self._graph) for name in names: if self._graph[name]: ts.add(name, self._graph[name]) ts2.add(name, self._graph[name]) print(" static_order =", tuple(ts2.static_order())) ts.prepare() while ts.is_active(): print(" is_active, i = ", bad[0]) bad[0] += 1 if bad[0] > 20: print(" Infinite loop?") break nodes = ts.get_ready() print(" Do nodes:", nodes) awaitables = [self._registryname for name in nodes] print(" awaitables: ", awaitables) awaitable_results = await asyncio.gather(*awaitables) results.update({ p[0].name: p[1] for p in zip(awaitables, awaitable_results) }) print(results) for node in nodes: ts.done(node)
foo = Foo()
await foo.other()
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
970554697 | https://github.com/simonw/datasette/issues/782#issuecomment-970554697 | https://api.github.com/repos/simonw/datasette/issues/782 | IC_kwDOBm6k_c452X1J | simonw 9599 | 2021-11-16T18:32:03Z | 2021-11-16T18:32:03Z | OWNER | I'm going to take another look at this: - https://github.com/simonw/datasette/issues/878 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
970553780 | https://github.com/simonw/datasette/issues/782#issuecomment-970553780 | https://api.github.com/repos/simonw/datasette/issues/782 | IC_kwDOBm6k_c452Xm0 | simonw 9599 | 2021-11-16T18:30:51Z | 2021-11-16T18:30:58Z | OWNER | OK, I'm ready to start working on this today. I'm going to go with a default representation that looks like this:
I'll implement |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
970544733 | https://github.com/simonw/datasette/issues/1509#issuecomment-970544733 | https://api.github.com/repos/simonw/datasette/issues/1509 | IC_kwDOBm6k_c452VZd | simonw 9599 | 2021-11-16T18:22:32Z | 2021-11-16T18:22:32Z | OWNER | This is mainly happening here: - https://github.com/simonw/datasette/issues/782 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Datasette 1.0 JSON API (and documentation) 1054243511 | |
970266123 | https://github.com/simonw/datasette/issues/1012#issuecomment-970266123 | https://api.github.com/repos/simonw/datasette/issues/1012 | IC_kwDOBm6k_c451RYL | bollwyvl 45380 | 2021-11-16T13:18:36Z | 2021-11-16T13:18:36Z | CONTRIBUTOR | Congratulations, looks like it went through! There was a bit of a hold-up on the JupyterLab ones, but it's semi automated: a dependabot pr to warehouse and a CI deploy, with a click in between. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
For 1.0 update trove classifier in setup.py 718540751 | |
970188065 | https://github.com/simonw/datasette/issues/1505#issuecomment-970188065 | https://api.github.com/repos/simonw/datasette/issues/1505 | IC_kwDOBm6k_c450-Uh | Segerberg 7094907 | 2021-11-16T11:40:52Z | 2021-11-16T11:40:52Z | NONE | A suggestion is to have the option to choose an arbitrary delimiter (and quoting characters ) |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Datasette should have an option to output CSV with semicolons 1052247023 | |
969621662 | https://github.com/simonw/datasette/issues/448#issuecomment-969621662 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45y0Ce | simonw 9599 | 2021-11-16T01:32:04Z | 2021-11-16T01:32:04Z | OWNER | Tests are failing and I think it's because the facets come back in different orders, need a tie-breaker. https://github.com/simonw/datasette/runs/4219325197?check_suite_focus=true |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969616626 | https://github.com/simonw/datasette/issues/1176#issuecomment-969616626 | https://api.github.com/repos/simonw/datasette/issues/1176 | IC_kwDOBm6k_c45yyzy | simonw 9599 | 2021-11-16T01:29:13Z | 2021-11-16T01:29:13Z | OWNER | I'm inclined to create a Sphinx reference documentation page for this, as I did for |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Policy on documenting "public" datasette.utils functions 779691739 | |
969613166 | https://github.com/simonw/datasette/issues/1012#issuecomment-969613166 | https://api.github.com/repos/simonw/datasette/issues/1012 | IC_kwDOBm6k_c45yx9u | simonw 9599 | 2021-11-16T01:27:25Z | 2021-11-16T01:27:25Z | OWNER | Requested here: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
For 1.0 update trove classifier in setup.py 718540751 | |
969602825 | https://github.com/simonw/datasette/issues/1012#issuecomment-969602825 | https://api.github.com/repos/simonw/datasette/issues/1012 | IC_kwDOBm6k_c45yvcJ | simonw 9599 | 2021-11-16T01:21:14Z | 2021-11-16T01:21:14Z | OWNER | I'd been wondering how to get new classifiers into Trove - thanks, I'll give this a go. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
For 1.0 update trove classifier in setup.py 718540751 | |
969600859 | https://github.com/simonw/datasette/issues/1511#issuecomment-969600859 | https://api.github.com/repos/simonw/datasette/issues/1511 | IC_kwDOBm6k_c45yu9b | simonw 9599 | 2021-11-16T01:20:13Z | 2021-11-16T01:20:13Z | OWNER | See: - #830 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Review plugin hooks for Datasette 1.0 1054246919 | |
969582098 | https://github.com/simonw/datasette/issues/448#issuecomment-969582098 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45yqYS | simonw 9599 | 2021-11-16T01:10:28Z | 2021-11-16T01:10:28Z | OWNER | Also note that this demo data is using a SQL view to create the JSON arrays - the view is defined as such:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969578466 | https://github.com/simonw/datasette/issues/448#issuecomment-969578466 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45ypfi | simonw 9599 | 2021-11-16T01:08:29Z | 2021-11-16T01:08:29Z | OWNER | Actually with the cache warmed up it looks like the facet query is taking 150ms which is good enough. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969572281 | https://github.com/simonw/datasette/issues/448#issuecomment-969572281 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45yn-5 | simonw 9599 | 2021-11-16T01:05:11Z | 2021-11-16T01:05:11Z | OWNER | I tried this and it seems to work correctly:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969557008 | https://github.com/simonw/datasette/issues/448#issuecomment-969557008 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45ykQQ | simonw 9599 | 2021-11-16T00:56:09Z | 2021-11-16T00:59:59Z | OWNER | This looks like it might work:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969557972 | https://github.com/simonw/datasette/issues/448#issuecomment-969557972 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45ykfU | simonw 9599 | 2021-11-16T00:56:58Z | 2021-11-16T00:56:58Z | OWNER | It uses a CTE which were introduced in SQLite 3.8 - and AWS Lambda Python 3.9 still provides 3.7 - but I've checked and I can use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969449772 | https://github.com/simonw/datasette/issues/448#issuecomment-969449772 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45yKEs | simonw 9599 | 2021-11-15T23:48:37Z | 2021-11-15T23:48:37Z | OWNER | Given this query: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads?sql=select%0D%0A++j.value+as+value%2C%0D%0A++count%28*%29+as+count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++id%2C%0D%0A++++++file%2C%0D%0A++++++clicks%2C%0D%0A++++++impressions%2C%0D%0A++++++text%2C%0D%0A++++++url%2C%0D%0A++++++spend_amount%2C%0D%0A++++++spend_currency%2C%0D%0A++++++created%2C%0D%0A++++++ended%2C%0D%0A++++++target_names%0D%0A++++from%0D%0A++++++ads_with_targets%0D%0A++++where%0D%0A++++++%3Ap0+in+%28%0D%0A++++++++select%0D%0A++++++++++value%0D%0A++++++++from%0D%0A++++++++++json_each%28%5Bads_with_targets%5D.%5Btarget_names%5D%29%0D%0A++++++%29%0D%0A++%29%0D%0A++join+json_each%28target_names%29+j%0D%0Agroup+by%0D%0A++j.value%0D%0Aorder+by%0D%0A++count+desc%2C%0D%0A++value%0D%0Alimit%0D%0A++31&p0=people_who_match%3Ainterests%3AAfrican-American+culture
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969446972 | https://github.com/simonw/datasette/issues/448#issuecomment-969446972 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45yJY8 | simonw 9599 | 2021-11-15T23:46:13Z | 2021-11-15T23:46:13Z | OWNER | It looks like the problem here is that some of the tags occur more than once in the documents: So they get counted more than once, hence the 182 count for something that couldn't possibly return more than 172 documents. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969442215 | https://github.com/simonw/datasette/issues/448#issuecomment-969442215 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45yIOn | simonw 9599 | 2021-11-15T23:42:03Z | 2021-11-15T23:42:03Z | OWNER | I think this code is wrong in the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969440918 | https://github.com/simonw/datasette/issues/448#issuecomment-969440918 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45yH6W | simonw 9599 | 2021-11-15T23:40:17Z | 2021-11-15T23:40:35Z | OWNER | Applied that fix to the Should never get 182 results on a page that faceting against only 172 items. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969436930 | https://github.com/simonw/datasette/issues/448#issuecomment-969436930 | https://api.github.com/repos/simonw/datasette/issues/448 | IC_kwDOBm6k_c45yG8C | simonw 9599 | 2021-11-15T23:31:58Z | 2021-11-15T23:31:58Z | OWNER | I think this SQL recipe may work instead:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
_facet_array should work against views 440222719 | |
969433734 | https://github.com/simonw/datasette/issues/519#issuecomment-969433734 | https://api.github.com/repos/simonw/datasette/issues/519 | IC_kwDOBm6k_c45yGKG | simonw 9599 | 2021-11-15T23:26:11Z | 2021-11-15T23:26:11Z | OWNER | I'm happy with this as the goals for 1.0. I'm going to close this issue and create three tracking tickets for the three key themes: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Decide what goes into Datasette 1.0 459590021 | |
968904414 | https://github.com/simonw/datasette/pull/1508#issuecomment-968904414 | https://api.github.com/repos/simonw/datasette/issues/1508 | IC_kwDOBm6k_c45wE7e | codecov[bot] 22429695 | 2021-11-15T13:20:49Z | 2021-11-15T13:20:49Z | NONE | Codecov Report
```diff @@ Coverage Diff @@ main #1508 +/-=======================================
Coverage 91.82% 91.82% Continue to review full report at Codecov.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Update docutils requirement from <0.18 to <0.19 1053655062 | |
968470212 | https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968470212 | https://api.github.com/repos/simonw/sqlite-utils/issues/329 | IC_kwDOCGYnMM45ua7E | simonw 9599 | 2021-11-15T02:49:28Z | 2021-11-15T02:49:28Z | OWNER | I was going to replace all of the I'll keep the This avoids the whole issue of needing to rewrite parameters, and solves the immediate problem which is consuming CSV files with bad column names. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Rethink approach to [ and ] in column names (currently throws error) 1005891028 | |
968458837 | https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968458837 | https://api.github.com/repos/simonw/sqlite-utils/issues/329 | IC_kwDOCGYnMM45uYJV | simonw 9599 | 2021-11-15T02:21:15Z | 2021-11-15T02:21:15Z | OWNER | I'm not going to implement a fix that rewrites the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Rethink approach to [ and ] in column names (currently throws error) 1005891028 | |
968453129 | https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968453129 | https://api.github.com/repos/simonw/sqlite-utils/issues/329 | IC_kwDOCGYnMM45uWwJ | simonw 9599 | 2021-11-15T02:07:46Z | 2021-11-15T02:07:46Z | OWNER | If I replace What should the following do?
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Rethink approach to [ and ] in column names (currently throws error) 1005891028 | |
968451954 | https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968451954 | https://api.github.com/repos/simonw/sqlite-utils/issues/329 | IC_kwDOCGYnMM45uWdy | simonw 9599 | 2021-11-15T02:05:29Z | 2021-11-15T02:05:29Z | OWNER |
I'm not going to do this, it's unnecessary extra complexity and it means the function that fixes the column names needs to have access to the current |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Rethink approach to [ and ] in column names (currently throws error) 1005891028 | |
968450579 | https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968450579 | https://api.github.com/repos/simonw/sqlite-utils/issues/339 | IC_kwDOCGYnMM45uWIT | simonw 9599 | 2021-11-15T02:02:34Z | 2021-11-15T02:02:34Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`table.lookup()` option to populate additional columns when creating a record 1053122092 | ||
968435041 | https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968435041 | https://api.github.com/repos/simonw/sqlite-utils/issues/339 | IC_kwDOCGYnMM45uSVh | simonw 9599 | 2021-11-15T01:44:42Z | 2021-11-15T01:44:42Z | OWNER |
Maybe this:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`table.lookup()` option to populate additional columns when creating a record 1053122092 | |
968434594 | https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968434594 | https://api.github.com/repos/simonw/sqlite-utils/issues/339 | IC_kwDOCGYnMM45uSOi | simonw 9599 | 2021-11-15T01:43:10Z | 2021-11-15T01:43:10Z | OWNER | What should I call this parameter? Django has a similar feature where it calls them |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`table.lookup()` option to populate additional columns when creating a record 1053122092 | |
968434425 | https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968434425 | https://api.github.com/repos/simonw/sqlite-utils/issues/339 | IC_kwDOCGYnMM45uSL5 | simonw 9599 | 2021-11-15T01:42:36Z | 2021-11-15T01:42:36Z | OWNER | Here's the current signature of I'm going to add a second positional argument which can provide a dictionary of column->value to use when creating the original table and populating the initial row. If the row already exists, those columns will be ignored entirely. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`table.lookup()` option to populate additional columns when creating a record 1053122092 | |
968401459 | https://github.com/simonw/sqlite-utils/pull/322#issuecomment-968401459 | https://api.github.com/repos/simonw/sqlite-utils/issues/322 | IC_kwDOCGYnMM45uKIz | simonw 9599 | 2021-11-15T00:26:42Z | 2021-11-15T00:26:42Z | OWNER | This relates to the fact that dictionaries, lists and tuples get special treatment and are converted to JSON strings, using this code: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L2937-L2947 So the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add dict type to be mapped as TEXT in sqllite 979612115 | |
968384988 | https://github.com/simonw/sqlite-utils/pull/324#issuecomment-968384988 | https://api.github.com/repos/simonw/sqlite-utils/issues/324 | IC_kwDOCGYnMM45uGHc | simonw 9599 | 2021-11-14T23:25:16Z | 2021-11-14T23:25:16Z | OWNER | Yes this was absolutely the intention! Thanks, I wonder how often I've made that mistake in other projects? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Use python-dateutil package instead of dateutils 988013247 | |
968384005 | https://github.com/simonw/sqlite-utils/issues/331#issuecomment-968384005 | https://api.github.com/repos/simonw/sqlite-utils/issues/331 | IC_kwDOCGYnMM45uF4F | simonw 9599 | 2021-11-14T23:19:29Z | 2021-11-14T23:20:32Z | OWNER | Tested it like this, against a freshly built My if name == "main":
db = sqlite_utils.Database(memory=True)
table = cast(sqlite_utils.db.Table, db["foo"])
table.insert({"id": 5})
print(list(db.query("select * from foo")))
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mypy error: found module but no type hints or library stubs 1026794056 | |
968381939 | https://github.com/simonw/sqlite-utils/issues/331#issuecomment-968381939 | https://api.github.com/repos/simonw/sqlite-utils/issues/331 | IC_kwDOCGYnMM45uFXz | simonw 9599 | 2021-11-14T23:06:20Z | 2021-11-14T23:06:20Z | OWNER | Thanks - I didn't know this was needed! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mypy error: found module but no type hints or library stubs 1026794056 | |
968380675 | https://github.com/simonw/sqlite-utils/issues/332#issuecomment-968380675 | https://api.github.com/repos/simonw/sqlite-utils/issues/332 | IC_kwDOCGYnMM45uFED | simonw 9599 | 2021-11-14T22:57:56Z | 2021-11-14T22:57:56Z | OWNER | This is a great idea. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`sqlite-utils memory --flatten` option to flatten nested JSON 1028056713 | |
968380387 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968380387 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45uE_j | simonw 9599 | 2021-11-14T22:55:56Z | 2021-11-14T22:55:56Z | OWNER | OK, this should fix it. |
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968371112 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968371112 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45uCuo | simonw 9599 | 2021-11-14T21:57:43Z | 2021-11-14T22:21:31Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968361671 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968361671 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45uAbH | simonw 9599 | 2021-11-14T20:54:53Z | 2021-11-14T21:01:14Z | OWNER | I'm leaning towards Or "ignore" sounds like it might not create the index if the name exists, but we want to still create the index but pick a new name. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968362285 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968362285 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45uAkt | simonw 9599 | 2021-11-14T20:59:44Z | 2021-11-14T20:59:44Z | OWNER | I think I'll attempt to create the index and re-try if it fails with that error. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968362214 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968362214 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45uAjm | simonw 9599 | 2021-11-14T20:59:15Z | 2021-11-14T20:59:15Z | OWNER | How to figure out if an index name is already in use?
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968361409 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968361409 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45uAXB | simonw 9599 | 2021-11-14T20:52:55Z | 2021-11-14T20:52:55Z | OWNER | Looking at the method signature: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L1518-L1524
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968361285 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968361285 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45uAVF | simonw 9599 | 2021-11-14T20:51:57Z | 2021-11-14T20:51:57Z | OWNER | SQLite will happily create multiple identical indexes on a table, using more disk space each time: ```pycon <Table t (id)> # dupes.db is 98304 bytes >>> db["t"].create_index(["id"]) <Table t (id)> # dupes.db is 204800 bytes >>> db["t"].indexes [Index(seq=0, name='idx_t_id', unique=0, origin='c', partial=0, columns=['id'])] >>> db["t"].create_index(["id"], index_name="t_idx_t_id_2") <Table t (id)> # 311296 bytes >>> db["t"].create_index(["id"], index_name="t_idx_t_id_3") <Table t (id)> # 417792 bytes >>> db.vacuum() # Still 417792 bytes ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968360538 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968360538 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45uAJa | simonw 9599 | 2021-11-14T20:46:56Z | 2021-11-14T20:46:56Z | OWNER | I'm tempted to not provide an opt-out option either: if you call But... it feels wasteful to create an index that exactly duplicates an existing index. Would SQLite even let you do that or would it notice and NOT double the amount of disk space used for that index? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968360387 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968360387 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45uAHD | simonw 9599 | 2021-11-14T20:45:44Z | 2021-11-14T20:45:44Z | OWNER | What would such an option be called? Some options:
If the user doesn't pass in an explicit name it seems like their intent is "just create me the index, I don't care what name you use" - so actually perhaps the default behaviour here should be to pick a new unique name if that name is already in use. Then maybe there should be an option to disable that - some options there:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968359868 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968359868 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45t_-8 | simonw 9599 | 2021-11-14T20:41:42Z | 2021-11-14T20:41:42Z | OWNER | The "index idx_generators_eia860_report_date already exists" error suggests that the problem here is actually one of an index name collision.
So perhaps |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968359137 | https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968359137 | https://api.github.com/repos/simonw/sqlite-utils/issues/335 | IC_kwDOCGYnMM45t_zh | simonw 9599 | 2021-11-14T20:37:00Z | 2021-11-14T20:37:00Z | OWNER | This is strange - the code already checks that an index doesn't exist before attempting to create it: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L893-L902 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils index-foreign-keys fails due to pre-existing index 1042569687 | |
968210842 | https://github.com/simonw/datasette/issues/1507#issuecomment-968210842 | https://api.github.com/repos/simonw/datasette/issues/1507 | IC_kwDOBm6k_c45tbma | simonw 9599 | 2021-11-14T05:41:55Z | 2021-11-14T05:41:55Z | OWNER | Here's the build with that fix: https://readthedocs.org/projects/datasette/builds/15268498/ It passed and published the docs: https://docs.datasette.io/en/latest/changelog.html |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
ReadTheDocs build failed for 0.59.2 release 1052851176 | |
968210222 | https://github.com/simonw/datasette/issues/1507#issuecomment-968210222 | https://api.github.com/repos/simonw/datasette/issues/1507 | IC_kwDOBm6k_c45tbcu | simonw 9599 | 2021-11-14T05:34:14Z | 2021-11-14T05:34:14Z | OWNER | Here's the new build using Python 3: https://readthedocs.org/projects/datasette/builds/15268482/ It's still broken. Here's one of many issue threads about it, this one has a workaround fix: https://github.com/readthedocs/readthedocs.org/issues/8616#issuecomment-952034858
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
ReadTheDocs build failed for 0.59.2 release 1052851176 | |
968209957 | https://github.com/simonw/datasette/issues/1507#issuecomment-968209957 | https://api.github.com/repos/simonw/datasette/issues/1507 | IC_kwDOBm6k_c45tbYl | simonw 9599 | 2021-11-14T05:31:07Z | 2021-11-14T05:31:07Z | OWNER | Looks like ReadTheDocs builds started failing for |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
ReadTheDocs build failed for 0.59.2 release 1052851176 | |
968209731 | https://github.com/simonw/datasette/issues/1507#issuecomment-968209731 | https://api.github.com/repos/simonw/datasette/issues/1507 | IC_kwDOBm6k_c45tbVD | simonw 9599 | 2021-11-14T05:28:41Z | 2021-11-14T05:28:41Z | OWNER | I will try adding a This might work: ``` version: 2 build: os: ubuntu-20.04 tools: python: "3.9" sphinx: configuration: docs/conf.py ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
ReadTheDocs build failed for 0.59.2 release 1052851176 | |
968209616 | https://github.com/simonw/datasette/issues/1507#issuecomment-968209616 | https://api.github.com/repos/simonw/datasette/issues/1507 | IC_kwDOBm6k_c45tbTQ | simonw 9599 | 2021-11-14T05:27:22Z | 2021-11-14T05:27:22Z | OWNER | https://blog.readthedocs.com/default-python-3/ they started defaulting new projects to Python 3 back in Feb 2019 but clearly my project was created before then. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
ReadTheDocs build failed for 0.59.2 release 1052851176 | |
968209560 | https://github.com/simonw/datasette/issues/1507#issuecomment-968209560 | https://api.github.com/repos/simonw/datasette/issues/1507 | IC_kwDOBm6k_c45tbSY | simonw 9599 | 2021-11-14T05:26:36Z | 2021-11-14T05:26:36Z | OWNER | It looks like my builds there still run on Python 2!
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
ReadTheDocs build failed for 0.59.2 release 1052851176 | |
968207906 | https://github.com/simonw/datasette/issues/1503#issuecomment-968207906 | https://api.github.com/repos/simonw/datasette/issues/1503 | IC_kwDOBm6k_c45ta4i | simonw 9599 | 2021-11-14T05:08:26Z | 2021-11-14T05:08:26Z | OWNER | Error: ``` def test_table_html_filter_form_column_options( path, expected_column_options, app_client ): response = app_client.get(path) assert response.status == 200 form = Soup(response.body, "html.parser").find("form") column_options = [ o.attrs.get("value") or o.string for o in form.select("select[name=_filter_column] option") ]
My solution: start with the query columns, but then add any table columns that were not already returned by the query to the end of the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`?_nocol=` removes that column from the filter interface 1050163432 | |
968192980 | https://github.com/simonw/datasette/issues/1506#issuecomment-968192980 | https://api.github.com/repos/simonw/datasette/issues/1506 | IC_kwDOBm6k_c45tXPU | simonw 9599 | 2021-11-14T02:22:40Z | 2021-11-14T02:22:40Z | OWNER | I think the answer is to spot this case and link to
So maybe the facet selection rendering logic needs to spot this and link correctly to it? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Columns beginning with an underscore do not facet correctly 1052826038 | |
967801997 | https://github.com/simonw/datasette/issues/1380#issuecomment-967801997 | https://api.github.com/repos/simonw/datasette/issues/1380 | IC_kwDOBm6k_c45r3yN | Segerberg 7094907 | 2021-11-13T08:05:37Z | 2021-11-13T08:09:11Z | NONE | @glasnt yeah I guess that could be an option. I run datasette on large databases > 75gb and the startup time is a bit slow for me even with -i --inspect-file options. Here's a quick sketch for a plugin that will reload db's in a folder that you set for the plugin in metadata.json. If you request /-reload-db new db's will be added. (You probably want to implement some authentication for this =) ) https://gist.github.com/Segerberg/b96a0e0a5389dce2396497323cda7042 |
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Serve all db files in a folder 924748955 | |
967747190 | https://github.com/simonw/datasette/issues/1380#issuecomment-967747190 | https://api.github.com/repos/simonw/datasette/issues/1380 | IC_kwDOBm6k_c45rqZ2 | glasnt 813732 | 2021-11-13T00:47:26Z | 2021-11-13T00:47:26Z | CONTRIBUTOR | Would it make sense to run datasette with a fswatch/inotifywait on a folder, then? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Serve all db files in a folder 924748955 | |
967181828 | https://github.com/simonw/datasette/issues/1380#issuecomment-967181828 | https://api.github.com/repos/simonw/datasette/issues/1380 | IC_kwDOBm6k_c45pgYE | Segerberg 7094907 | 2021-11-12T15:00:18Z | 2021-11-12T20:02:29Z | NONE | There is no such option see https://github.com/simonw/datasette/issues/43. But you could write a plugin using the datasette.add_database(db, name=None) https://docs.datasette.io/en/stable/internals.html#add-database-db-name-none |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Serve all db files in a folder 924748955 | |
964205475 | https://github.com/simonw/sqlite-utils/issues/26#issuecomment-964205475 | https://api.github.com/repos/simonw/sqlite-utils/issues/26 | IC_kwDOCGYnMM45eJuj | fgregg 536941 | 2021-11-09T14:31:29Z | 2021-11-09T14:31:29Z | CONTRIBUTOR | i was just reaching for a tool to do this this morning |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mechanism for turning nested JSON into foreign keys / many-to-many 455486286 | |
962411119 | https://github.com/simonw/sqlite-utils/issues/336#issuecomment-962411119 | https://api.github.com/repos/simonw/sqlite-utils/issues/336 | IC_kwDOCGYnMM45XTpv | simonw 9599 | 2021-11-06T07:21:04Z | 2021-11-06T07:21:04Z | OWNER | I've never used |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-util tranform --column-order mangles columns of type "timestamp" 1044267332 | |
962259527 | https://github.com/simonw/sqlite-utils/pull/337#issuecomment-962259527 | https://api.github.com/repos/simonw/sqlite-utils/issues/337 | IC_kwDOCGYnMM45WupH | urbas 771193 | 2021-11-05T22:33:02Z | 2021-11-05T22:33:02Z | NONE | Smokes, it looks like there was a bug in click 8.0.2 (fixed in 8.0.3: https://github.com/pallets/click/issues/2089). Meaning this PR is not needed. Closing. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Default values for `--attach` and `--param` options 1046271107 | |
851567204 | https://github.com/simonw/datasette/issues/1284#issuecomment-851567204 | https://api.github.com/repos/simonw/datasette/issues/1284 | MDEyOklzc3VlQ29tbWVudDg1MTU2NzIwNA== | mroswell 192568 | 2021-05-31T15:42:10Z | 2021-11-04T03:15:01Z | CONTRIBUTOR | I very much want to make: https://list.SaferDisinfectants.org/disinfectants/listN have this URL: https://list.SaferDisinfectants.org/ I'm using only one table page on the site, with no pagination. I'm not using the home page, though when I tried to move my table to the home page as mentioned above, I failed to figure out how. I am using cloudflare, but I haven't figured out a forwarding or HTML re-write method of doing this, either. Is there any way I can get a prettier list URL? I'm on Vercel. (I have a wordpress site on the main domain on a separate host.) |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Feature or Documentation Request: Individual table as home page template 845794436 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
issue >1000